Tear down a lapsed anonymous Organization one step at a time#2196

An agent that signs up through POST /api/agent/signup gets an Organization for a day. Until now nothing took it back. This PR makes Cloud tear a lapsed one down on its own: the Projects leave their Servers, the Organization goes, and no row or anon- user names it afterwards.

What changes

• A cron schedule-anonymous-sweep runs every 15 minutes and sends anonymous/teardown.requested for each Organization an anonymous agent alone owns whose every agent token has lapsed.
• tear-down-anonymous-organization takes one step per run, keyed per Organization with concurrency 1 and 3 retries. A run with Projects left sends the next event itself.
• Teardown runs as a system caller. It removes each Project through callStore with approval: "not_required", so an agent-caller approval check (FU1) never gates it.
• A claim, a teardown and token auth decide "lapsed" with one clock and one predicate, tokenRuns: expires_at > clock_timestamp(). Postgres now() is the transaction's start time, so a claim committing mid-transaction could pass a check teardown had already failed. clock_timestamp() is the moment of the check. Teardown re-checks under the Organization's row lock inside the transaction that deletes it, and before each Project removal. A claim that commits first makes teardown return with nothing destroyed. A claim checked after teardown saw the lapse is refused.
• Teardown tries the removal first. Only when the Servers hold it up does Cloud probe them. A new table, anonymous_teardown, times that wait on the database clock: waiting_since when the wait began, silent_since at the first probe that reached no Server since the last answer, warned_at once it warned. Any answer clears silent_since, whether to a probe or to the pairing's removal: a Server that authenticates to the removal but drops or fails a step before it confirms stamps organization_pairing.removal_answered_at. A Server that never authenticates never counts as answering. If Cloud can't write that stamp, the step fails and Inngest retries it. A retry (Inngest attempt above 0) probes but never forgets, because its failed attempt may have heard an answer Cloud couldn't record; the next sweep decides. Cloud forgets the Servers only after an hour of silence counted from the latest of silent_since, the current pairing's created_at and its removal_answered_at, so a pairing made since starts its own hour, logged as anonymous_server_forgotten, then re-derives the state before it removes anything. A sandbox that has waited a day warns once. Cancelled removal marks play no part, and a Project whose removal never wrote a mark still starts the clock.
• The sweep's work list folds the owner check into its SQL as not exists, one query instead of one per Organization.
• Rows with no foreign key to the Organization go in the same transaction as the Organization row: analytics_site and analytics_salt before it, organization_change after it (the cascade's statement triggers write it during the delete; every Organization removal now sweeps it). This Organization's anonymous users go last, when no other membership names them.
• A person who co-owns the Organization keeps it. The sweep's SQL and the step's isOwnedOnlyByAgents both check it.
• A pairing removal now clears the Machine's push target along with its cloud Management Client. A Server too old to hold a push target still unpairs.

Data shape and step order

readSandboxState returns one of gone, claimed (a person owns it, alone or alongside the agent), live (an anonymous owner's token still runs by the database clock) or lapsed with the first anonymous owner, the slug, whether a Cluster is paired, and the Projects the Store lists. Lapsed is one way: a token only grows through a claim, and a claim locks the Organization row that the lapsed read locks, so a sandbox lapsed to one is lapsed to the other.

Each run re-derives that state and does one step:

1. gone, claimed or live: nothing.
2. lapsed with Projects: re-check, remove each Project, re-read, report removed_projects or waited_on_servers.
3. lapsed and empty: re-check, then removeOrganization as the anonymous owner with the sandbox's rows deleted in its transaction, or waited_on_servers while a Server still confirms.
4. After waited_on_servers or a Store refusal: probe the Servers and update the silence clock. If the hour of silence is up, forget them, re-derive the state, and retry the removal.

Bets

• core/DESIGN.md 7, Cloud drives, never owns. The forget path is the one place this PR drops a Cluster from Cloud's side. It never fires on a transport failure alone, and never on a token's expiry alone. Graceful removal runs first and asks every Server. Only Servers that every probe for an hour since the last answer failed to reach are forgotten, timed by the database clock, and forgetServers refuses while any Server answers, so one answering Server holds the whole Cluster. The Machine keeps every Service, Volume and its WireGuard peers. If Cloud vanished mid-teardown the Machine is still correct, only its Cloud pairing stays until the next step. Forgetting ends Cloud's bookkeeping for a seat nobody can claim any more. It does not reset the Cluster. The red flag names transport failure used to forget a Cluster; here one failed probe keeps the pairing, and a sustained silence after the agent's own token lapsed is the signal.
• core/DESIGN.md 6, Machine-local authority. Projects leave through a Store removal and a Deploy, so each Machine removes its own copies. The Organization goes only after the Servers confirm, or once they are forgotten.
• core/DESIGN.md 4, Partial results are outcomes. waited_on_servers is a step result, not an error, and the next sweep continues from it. A Server that answers but never confirms surfaces as one warning after a day, not a cap: the Machine is still correct, and only Cloud's row waits.
• dashboard/DESIGN.md holds the design system. No UI changes.

Outside the file boundary

• core/crates/ployz-store/src/teardown.rs and tests/removal.rs: remove_organization left config_removal_mark rows behind, which the "nothing left" check found. The Rust test pins the mark is gone.
• dashboard/src/modules/inngest/events.ts: the new event type and its constructor.
• dashboard/src/modules/identity/claim.server.ts, claim-required.server.ts and caller.server.ts: the claim and token auth read liveness with the shared tokenRuns predicate, the database clock, instead of Date.now().
• dashboard/drizzle/20261017203205_anonymous_teardown: the anonymous_teardown table, cascading with its Organization, and organization_pairing.removal_answered_at.
• dashboard/src/modules/machines/pairing-removal.server.ts: clears the push target, and records a Server that authenticated without confirming, and fails the step when that record can't be written; the lock-order docstring names the Organization row lock.
• dashboard/src/modules/organization/organization-removal.server.ts: removeOrganization takes a deleting wrapper around the row delete, locks the Organization row right after the enrollment lock, and sweeps organization_change after the delete.
• dashboard/src/modules/machines/forget-servers.server.ts: the docstring names the one case Cloud forgets on its own.

Follow-ups

• An index on config_removal_mark stays deferred. The Rust Store owns that migration.
• A claim that lands between the probe's transaction and forgetCluster is guarded only by forgetServers' own mayManage check. Closing it fully would hold a lock across a Store call.

Verify

dashboard/src/modules/anonymous/teardown.inngest.postgres.test.ts holds one case each for a live token, a claimed Organization (driven through claimOrganization), a person co-owning alongside the agent, an empty Organization with the information_schema "nothing left" count, a Project then its Organization in two runs, a Project with running Services removed with no approval row, and the rows committed with the Organization delete. Round 2 adds:

• a claim that commits while teardown waits on the Organization: the Organization, its Project, its Store config and its Server pairing stay;
• a claim checked after teardown saw the lapse is refused;
• an answer restarts the silence clock, one failed probe never forgets, and a cancelled removal mark never counts;
• an app clock jumped a day ahead does not forget early, and a gates test pins every teardown time decision to the database clock;
• a Store that refuses every removal, so no mark is written, still forgets a Server silent for an hour, then finishes;
• a claim landing during the wait is never forgotten;
• the day-long wait warns once across repeated runs;
• two concurrent runs both reaching waited_on_servers with a Server converge.

Round 3 adds, each failing before its fix:

• a Server that answers its pairing removal, then drops before confirming, keeps its pairing and the Organization;
• after the old pairing is forgotten, a new pairing seeded with enrollStoreServer (a fixture, not the CLI enrollment path) and a fresh secret and Machine survives its first failed probe;
• organization-ownership.postgres.test.ts lists as belonging to no Organization only tables without an organization_id; anonymous_teardown and 23 others move to a list of Organization-owned tables that need no change log.

Round 4 adds, each failing before its fix:

• a Server that authenticates to its pairing removal, then fails inspect, restarts the silence clock: the run ends waited_on_servers and the pairing and Organization stay;
• a trigger that rejects the removal_answered_at write fails the run, and the retry, with writes restored and the Server unreachable again, ends waited_on_servers instead of forgetting it.

organization-token.server.postgres.test.ts pins token auth to the database clock and pairing-removal.server.postgres.test.ts pins the cleared push target. Each round-2 test failed before its fix except the strengthened claim, pairing-revoke and concurrent-run cases, which guard behaviour the old code already had; receipts sit under /tmp/p4fix/out/r2/<finding>/{fail,pass}.txt /tmp/p4fix/out/r3/<finding>/{fail,pass}.txt and /tmp/p4fix/out/r4/<finding>/{fail,pass}.txt on codex-vm.

Discovery at 2,500 lapsed Organizations among 10,000 users, interleaved on one machine, 7 samples a run: before p50 3.1-3.7 s, after p50 42-45 ms. At 50 lapsed Organizations: 96-98 ms before, 43-51 ms after. The one listing query is now the floor.

Focused checks ran from dashboard/: pnpm typecheck, pnpm lint, pnpm vitest run src/modules/anonymous src/modules/inngest src/modules/organization src/modules/machines src/modules/identity with src/db/organization-ownership.postgres.test.ts, src/modules/runtime/ployz-sdk-boundary.test.ts and src/routes/api/cli/-cli.handler.postgres.test.ts, and cargo test -p ployz-store --test removal from core/.

Done scenario, run at round 1's head 152e3dc30, on a Boat Server (bare Ubuntu 24.04, box-node-94af34bcfea765a3) with the verify dashboard, at this head, from a shell holding only curl and the ployz binary: signup via curl (Organization anon-v2djj2wv), the Server enrolled with PLOYZ_TOKEN=<anon> ployz server add, Project p4 with a traefik/whoami Service deployed and running, expires_at backdated two hours, one anonymous/teardown.requested fired by hand at 17:54Z. The first run removed the Project, the Server confirmed (its config_removal_mark reached removed), and the whoami container was gone. The 18:00Z sweep found the Organization still lapsed and its run removed it in two seconds. The Server answered throughout, so the forget path never opened: the worker log holds no warning and no anonymous_server_forgotten. After: 0 rows for the Organization across all 73 tables with an organization_id column in public and analytics, no anonymous user, no anon- slug, no organization_change row, the Server's cloud and CLI management clients cleared, caddy and corrosion still running, the Machine still answering ployz server ls locally. The lapsed token is refused by the CLI (unauthenticated) and by /api/agent/me (404).