Fence Volume copy requests with a per-Volume floor on the Machine#1722

_{P2 Badge} Clear persisted floors without requiring Docker

When the daemon starts while Docker is unavailable, daemon.rs:222-234 sets containers to None even though machine.db and its volume_floors table remain on disk. In that state depart_storage returns at lines 691-693 before reaching this new cleanup, so Leave and enrollment do not clear the old cluster's floors; after Docker returns, a newly enrolled cluster whose generation is below the retained value has every copy request refused as stale. The floor store needs to be cleared independently of the optional Docker runtime.

Useful? React with 👍 / 👎.

Leaving this as is. With Docker absent at start, depart_storage already skips demoting the roots themselves, so a Leave in that state departs nothing, and the floors follow the same rule. Clearing floors alone would still leave the old roots behind. Lane gens also come from one Cloud-wide sequence that only rises, so a new cluster on the same Cloud starts above any old floor. A Docker-less Leave that departs nothing is tracked as a follow-up together with the gen reset on a Cloud rebuild.

Real, and deferred. It needs Docker down at Leave and then a cluster whose gen sequence is lower than the kept floor, which means another Cloud or a rebuilt database, since one Cloud's sequence only grows. It is the same gap as Leave on a Machine without Docker, which already skips demoting roots. Both go in one follow-up that opens the Machine store without Docker.

_{P2 Badge} Add stale to the dashboard refusal decoder

Once a capability-aware client raises a Machine's floor above zero, the daemon returns the new stale reason for the dashboard's existing gen-less requests. However, dashboard/src/modules/volume-run/volume-run.server.ts:435-441 still accepts only the previous seven reasons, so decodeCopyRefused fails and line 520 classifies the refusal as a transport SwitchAttemptError; the Inngest step then retries instead of returning a CopyAnswer for the run to resolve. Update the runtime decoder and message handling when exposing this reason.

Useful? React with 👍 / 👎.

Not reachable in this slice. Nothing on s1 sends a Floor want or a gen, so no floor rises above zero and the daemon never answers stale here. The dashboard decoder learns stale in #1725 (5e307cdb4), the same commit that starts sending the lane gen. The stack lands together, and daemons only pick this up at a release.

Not reachable in the landed stack. Nothing raises a floor before #1725: that slice is the first where the dashboard sends a gen, and its refusal decoder already lists stale (volume-run.server.ts, the CopyRefusedDetails reasons and the case "stale" message). Released daemons get floors only after the whole stack lands.

_{P1 Badge} Keep the floor guard through each holder stop

When a stopped holder is already in volume_holder_ids, a newer run can raise the floor and start that holder after this check but before stop_behind_ingress inspects it. The old cut then observes the newly started holder as running, drains and stops it, and only later returns stale, leaving the replacement service down. The floor check must be serialized with the decision to stop each holder rather than dropping its guard before the stop begins.

Useful? React with 👍 / 👎.

Fixed in 9a4617a19 ("Recheck a cut's gen after the drain, before the holder stops"). The cut checks the floor again under the exclusive turn, after the drain. A raised floor puts the holder back in ingress and answers Stale without stopping it.

_{P2 Badge} Reconcile the glossary with the new floor model

This resolves the domain model around a Postgres gen and a machine-db floor, but core/CONTEXT.md:455-457 still defines the Machine-side Lease as a Pool record with a different wire shape that survives reset, while dashboard/CONTEXT.md:43-45 still says Volume Runs are admitted against that lease record. Those specifications now direct later slices toward mutually incompatible persistence and protocol behavior, so the glossaries need to be amended together with this design decision.

AGENTS.md reference: AGENTS.md:L8-L8

Useful? React with 👍 / 👎.

Agreed, docs only. Noted as a follow-up: core/CONTEXT.md and dashboard/CONTEXT.md still describe the Lease. Both glossaries move to the Postgres gen plus the machine.db floor after this stack lands.

_{P1 Badge} Prevent Floor wants from overtaking writer starts

A gen-1 Writer request can pass this final admission check and then pause in the fault hook, plugin call, or Docker inspection while a gen-2 Floor want bypasses switch_mutation and raises the floor. The older request can subsequently execute run_start, starting an obsolete holder after the newer lane owns the Volume; a following gen-2 Writer request may then treat that unrelated running holder as StartState::Running and never start its intended Container. Writer side effects need protection from, or another admission check against, concurrent Floor raises.

Useful? React with 👍 / 👎.

Fixed in af4c2d295 ("Recheck a Writer's gen against the floor before it starts its holder"). The create and start now run under one exclusive turn that checks the floor first, and answer Stale when a newer lane raised it.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Recheck the floor at each Docker stop.

This check releases the floor turn before stop_behind_ingress inspects and drains the holder. If the holder was in the initial list but stopped, a newer run can raise the floor and start that holder before the inspect. The older cut then sees it running and stops it, interrupting the newer run. Recheck gen under the floor lock at the stop's admission turn, and keep that lock through the stop. Release any stopping mark if the recheck refuses the cut. The test with a second holder does not cover this same-holder interleaving.

✅ Confirmed as addressed by @nick-potts

Fixed in 9a4617a19 ("Recheck a cut's gen after the drain, before the holder stops"). The cut checks the floor again under the exclusive turn, after the drain. A raised floor puts the holder back in ingress and answers Stale without stopping it.

@nick-potts Thanks. The supplied code rechecks gen after the drain under the exclusive admission turn. If the generation is stale, it clears the stopping mark and returns Stale without stopping the holder. This addresses the reported interleaving.

✅ Review thread resolved.

You are interacting with an AI system.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Return Stale before returning Busy for a fenced cut.

If a newer holder appears after volume_holder_ids takes its list and raises the floor during the drain, volume_holder_running returns true here. The cut returns Busy before the final floor check, although its gen is below the floor. Recheck the floor on this Busy path, and on the earlier is_running refusal path, so the caller receives Stale(floor) when the cut has been fenced.

Not changing this. A fenced cut that answers Busy changes no state. The root stays the writer and the newer lane owns it. settle sleeps on Busy and resends the same request, and admit_gen refuses that resend as Stale, so the run ends through the stale branch. The cost is one BUSY_WAIT.

@nick-potts Thanks for the clarification. Busy leaves the root as a writer. With the retry behavior you described, settle resends after BUSY_WAIT, and admit_gen rejects the fenced request as Stale. The extra wait does not break fencing. I withdraw this finding.

✅ Review thread resolved.

You are interacting with an AI system.

_{P2 Badge} Keep replay expectations at the unfenced reasons

When this verification runs against this commit, all three replay RPCs at line 847 omit gen, while volume-run.server.ts:514 also performs the preceding moves without a gen or Floor request. Both the requests and persisted floors therefore remain at the default zero, so the daemon reaches the transition planner and returns the original mismatch, mismatch, and no_transition reasons rather than stale; the assertion at line 853 consequently fails every run. Keep the old expectations in this slice or explicitly raise a floor and send an older generation in these replay requests.

AGENTS.md reference: AGENTS.md:L5-L5

Useful? React with 👍 / 👎.

Right for this slice alone, and leaving it. Cloud starts sending the lane's gen in the next slice (#1713), so only from there do the moves raise the floor and the replays answer stale. This script is a manual Incus harness, not a CI job, and the stack lands as one unit. The check runs at the stack top, where the floor is in place.

_{P1 Badge} Hold the floor lock until the holder start completes

When a gen-1 Writer reaches this check, the temporary FloorTurn is dropped as soon as admit returns. A concurrent gen-2 Floor request takes no admission turn, so it can raise the floor while create_container_admitted or containers.start continues; the subsequent gen-2 Writer can then see the obsolete holder running and skip starting its intended Container. Fresh evidence against the prior resolution is that the new exclusive admission turn does not serialize floor_copy; keep the floor guard through the create/start side effects.

Useful? React with 👍 / 👎.

The gen-1 admit is linearized before the floor raise, so the result is the legal serial order: gen-1's start completes, then the Floor, then gen-2. A gen-2 Writer waits behind gen-1 on the Volume's switch lock (local_volume_copy.rs:48-64) and treats a running holder as Running by design (local_volume_copy.rs:294-302), so holding the floor lock through the start would not change what gen-2 sees. A Floor by itself starts nothing.

_{P1 Badge} Hold the floor lock through the fenced holder stop

After an old cut passes this check, its FloorTurn is dropped before containers.stop runs. Because a newer Floor request bypasses the admission turn, it can raise the floor and observe the holder still running in that window, after which the old cut stops the holder that the newer lane elected to keep. Fresh evidence against the prior resolution is that the added admission lock cannot exclude floor_copy; retain the floor guard through the Docker stop or otherwise make the check and stop atomic with floor raises.

Useful? React with 👍 / 👎.

Fixed in #1857. The fenced stop now holds the floor lock from the admit through the Docker stop, so a Floor raised mid-stop waits for it and the Deploy's EnsureContainer sees the holder stopped and starts it.