Sign up
View stargazersSign up

Warm Move 8: Move and Release#1462

merged37
main
warm-move/move-release
openedOct 8, 2026byNNick
57 files changed
View on GitHubGitHub import enabledPR last imported 20d ago

What

ployz volume move data --to web-2 moves a Volume and the Service that mounts it to another Server. ployz volume release data starts a stopped Service again on its old Server. Both are new kinds of the run-volume Inngest function from #1460, planned from the copies each Machine reports.

A Move copies the Volume in rounds while the Service runs, then copies the Service's image. Then it stops the Service (withdraw, freeze), sends the final snapshot, and hands the Volume over (hand_over). The target accepts it, promotes it to the writable root, and starts the Service from the source's spec. Finally the source closes its copy into a read-only mirror.

hand_over (step 08) is the point of no return:

  • A failure before 08 undoes the move. The run thaws the source, clears the target's final, and fails with the step's error.
  • A failure from 09 on goes forward only. The run fails with volume move data --to web-2 again continues from there, and a second Move picks up at Accept, Promote, Start, or Close, depending on what the copies show.

Also in this PR, carried forward from #1460:

  • Volume.ForgetLease, a lease-fenced Machine verb. Deleting a mirror, and the orphan sweep, now clear the slot's lease record.
  • Plugin Mount refuses fast while a switch verb is calling Docker, instead of queuing behind Close until it times out.
  • VolumeStorage::docker takes &HeldMutation, so a Docker call without the mutation lock no longer compiles.
  • The no-writer refusal names the declared Volume (data), not the Docker volume name.
  • The SDK gains copyContainerImage and inspectContainer.
  • Freeze's docker stop, and the docker start of Thaw and of a handed Start, run in a task the Machine owns, keyed by the lease record. The task holds the mount grant until Docker returns, and Docker runs once per record. The verb answers busy after 3 s while Docker still works, and the run sleeps 5 s and asks again at the same position, up to 120 times, without spending a retry. Every switch verb keeps the usual 10 s RPC deadline. DESIGN.md bet 10 records the task.

User docs: the new docs/user/services/move-a-volume.md, and volumes.md "When its server goes down".

Why

Moving a stateful Service between Servers used to mean a manual dump and restore with downtime for the whole copy. With the mirror from #1460 already on the target, the Service is down only for the final delta and the start.

Start reads the Service's spec from the source Machine through inspectContainer, not from the watch frame. The watch frame redacts environment values, so a spec taken from it started postgres with no POSTGRES_PASSWORD, and postgres crash-looped on the target (vc run 1). The spec stays inside step 11-start, so no Inngest step output records secret values.

Verification

  • Core gates on fe026e737: cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features --locked -- -D warnings, and cargo nextest run -p ployzd -p ployz-core -p ployz -p ployz-store --all-features --locked (2140 passed, 53 skipped).
  • Dashboard gates on fe026e737, with the native SDK rebuilt from that head: pnpm lint, pnpm typecheck, and pnpm test (1099 passed).
  • Every guard was red first, then mutated back to red. That covers the point of no return, the undo before 08, the stale lease, a late step after a retry, target exclusion on retry, ForgetLease, Mount versus Close, the no-writer name, and the environment on Start.
  • The Machine-owned Container step was red first in three tests: a caller that gives up mid docker start or docker stop leaves the task running with its grant until Docker returns, a replay with the same record waits on the task and Docker starts once, and a replay with a stale record is refused as before. Mutations catch each regression. Running the step inside the handler again fails 5 tests, dropping the grant before docker start fails 8, letting a replay run Docker again fails 1, and counting busy as a failed attempt in the run fails 2 dashboard tests.
  • The 130 s StartHandedContainer deadline from fix round 1, and the per-verb deadlines added after it, are reverted by new commits (a78ff7a7e through 291e1aece). fe026e737 sends every switch verb with the usual TARGET_RPC_TIMEOUT.
  • verify-cluster --cloud on three Incus Machines with a real dashboard and Inngest, on fe026e737: all 17 checks pass. They cover these cases:
    • single writer at every kill point from 05 to 12, with the source read-only from record 07 on;
    • undo when the withdraw expires, and a late step refused after a retry;
    • deploys refused mid-Move, with the Start held by a 30 s docker start, and the refusal naming the authored Volume;
    • Release after a stopped Move, then a delta sync;
    • the real environment on Start;
    • a Start held 15 s and a Start held 150 s, past the old 130 s deadline, each ending done with one docker start and one writer at every sample;
    • a failed Start, then a second Move ending done;
    • a Thaw held 15 s after a killed run, with Release ending done, the source running, and one writer at every sample;
    • a plain Deploy.
  • CI on fe026e737 passed through workflow_dispatch (run 37786816596), because the branch conflicts with main in the generated config-wasm.mjs and the PR's own CI does not trigger until that is merged.

Merge Danger

  • A migration widens volume_run_kind_check to add move and release. It is additive, and old rows stay valid.
  • Start now needs the source Machine to answer, since it reads the spec there. Close already needs it, so a Move with its source down was already stuck at step 12.
  • A plain deploy of a Service in the middle of a switch is refused by the planner with the no-writer message, not by the daemon's VolumeSwitching.

Checks

1 failing · 13 successful · 1 skipped
Some checks failed

Failing1

3ef6247

Added 18 commits

  • Make the plugin's Docker calls take the held mutation01d4a60
  • Let the SDK read a Container's spec with its environment values0bbf0b6
  • Expect the declared Volume name in the plain-mount refusals1689b1e
  • Start the moved Service from its source spec, environment included175ccfb
  • Add copyContainerImage to the SDK19a8e67
  • Type the round receive and the holder step without casts33734a7
  • Add ployz volume move and ployz volume release34f3b39
  • Add ForgetLease and open the switch verbs to the SDK5727d98
  • Run Move and Release: hand over past the point of no return, undo before it5913490
  • Forget the lease record when a mirror is deleted591af97
  • Document moving a volume, and that a down server's volume stays put5abab84
  • Let a switch step fail without ending the run, and find the Service container6bf1b11
  • Refuse plugin Mount fast while a switch verb is calling Docker797de25
  • Keep the slot of a Volume whose removal is not yet deployed7c4ae92
  • Plan Move and Release from the copies8709420
  • Name the watch frame's holder spec as redacted, and keep Start's reply out of the stepbc32210
  • Name the declared Volume in the no-writer refusalbc53ec9
  • Format the ForgetLease testsd5bcc34
GgitguardianBotOct 8, 2026
Synced from GitHub

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian idGitGuardian statusSecretCommitFilename
37990433TriggeredGeneric Password175ccfbebb4ea654f88e17d09f5db56ec4b8d16adashboard/src/modules/volume-run/volume-run.postgres.test.tsView secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

NNickOct 8, 2026

FAIL (verifier: Codex gpt-6.1-sol @high)

FAIL at head bc3221046351b2a043ab514a659008aaa8a41d15 because the required authored Volume name fix still emits vol-<UUID>. Reproduce by holding AdoptLease during a retry of volume move echo, staging echodb.env.VC_TOUCH, and deploying; the refusal recommends volume restore vol-47ea1f1a-27ad-4db7-adba-5d795592c5ee --from machine-1, whose Volume argument returns not_found on inspection although echo exists. The formatter at core/crates/ployz-core/src/volume_switch.rs:706 only strips the namespace, while Cloud lowering uses vol-<resource UUID>; preserve or map the authored name and test that actual lowering. All required core and dashboard gates passed, with 2,133 core tests and 1,097 dashboard tests, and both selected guard mutations failed behavioral assertions and passed after restoration. One real-ZFS Cloud session passed 19 checks, including all eight runner-kill points, undo, stale leases, target exclusion, mid-Move Deploy refusal, Release, delta Sync, deletion cleanup, and environment/pull-policy preservation, but failed the extra slow-Start recovery check. Four actual 15-second Docker Starts preserved at most one writable copy and one running container, then the Move failed safely; the forward retry was not_started after local Inngest shut down on an I/O timeout, so required retry convergence remains unproven. Cancelled-as-lost and generic AdoptLease diagnostics are tracked follow-ups here; the slow-Start convergence proof remains a verification blocker, and the one-session cap prevents a second run in this round. GitGuardian's secret-fsn-1 POSTGRES_PASSWORD is synthesized test fixture data, not a real credential, and the verification cluster was torn down successfully.

Added 6 commits

  • Test that a Cloud Volume without a writer is refused by its authored name8d305c4
  • Name a Cloud Volume as authored when a Deploy finds no writerdb4b1bc
  • Test that the caller waits out a Machine's Start of a handed Container449846f
  • Name the Machine's volume plugin bound and give each switch verb a deadline566f969
  • Wait out a Machine's Start of a handed Container10e6bb6
  • Regenerate the config wasm loader9d938cf
NNickOct 8, 2026

FAIL (verifier: Opus @high)

Head: 9d938cfacbcedcb46209cd352e722b881364b1fd

Defect: the fix moved only StartHandedContainer onto the 130 s deadline (volume_switch_deadline in core/crates/ployz/src/sdk.rs). Thaw still runs under the 10 s caller deadline, and it does the same granted docker start (start_granted in core/crates/ployzd/src/volume_plugin/switch_source.rs). Its mount grant is an RAII guard, so it drops when the caller's deadline cancels the handler. The docker start child keeps running, but if it reaches the plugin Mount after about 10 s, the Mount is refused, and the same thing happens on every replay.

Reproduction: the Codex vc probe thaw_takes_15_seconds, at this head.

  1. Stop a Move frozen before 08: delay HandOver 55 s, kill the runner, and cancel the run.
  2. Wrap the source's /usr/bin/docker so that start sleeps 15 s.
  3. Run ployz volume release bravo.

Result: all four Thaw attempts ran docker start, and the run ended failed with "the run failed". The source was left thawing, its record was 5:13.0.0:open, and no db container ran on either Machine. A second Release succeeded only after the wrapper was removed. So while docker start is slow, neither the undo nor Release restores the source as writer. This is the round-1 (c) defect: it was fixed for Start but not for Thaw. The fix is to give Thaw, and Freeze, the plugin-bounded deadline.

Everything else holds:

  • The gate is green except for two flakes that pass on rerun: ployzd management_connect ifname, and dashboard-shell window teardown.
  • Both mutations are caught: removing target exclusion fails plan.test, and undoing after the point of no return fails the volume-run.postgres tests.
  • The vc run shows at most one writable copy and one running container in every sample. Older-lease Withdraw and Start are refused with stale_lease. Release followed by sync is a delta, and a second Move reuses the mirror. Slow and held Starts complete on the first try, with the real env and pull_policy never.

The other two scripted vc FAILs are harness artifacts:

  • C3 asserts readonly at the instant 06 opens, but Freeze records open before docker stop and readonly=on.
  • C6's third hold lasts 8 s, which is shorter than detection plus the deploy plus a 5 s sleep. So the container that appears is the Move's own Start, and the refused deploy had every node not_attempted.

The GitGuardian hit is test fixture data, not a credential: secret-${machine name} comes from the fake at volume-run.postgres.test.ts:185 and is asserted at :621.

NNickOct 8, 2026

FAIL (verifier: Codex gpt-6.1-sol @high)

At head 9d938cfacbcedcb46209cd352e722b881364b1fd, Thaw still receives the 10-second caller deadline in core/crates/ployz/src/sdk.rs:149, although its Machine plugin call can take 120 seconds. Reproduce by stopping a Move before HandOver, wrapping source docker start with sleep 15, and requesting Release: run 99cb2992-7484-440a-96f9-4cf10491caef failed with the run failed, leaving writer thawing, lease 5:13.0.0:open, and no running database. Four attempts timed out and Docker logged mount refusals for the thawing writer; removing the delay let Release 58f45900-a2ba-4e9f-99e4-642401ab17e7 finish, so safe recovery after removing the delay does not solve a consistently slow source restart. Both round-1 fixes pass live: authored-volume refusals, a real 15-second Docker Start, a 15-second pre-admission Start hold, failed-Start forward retry, preserved source environment, and no image pulls. Core gates passed 2,135 tests, dashboard gates passed 1,097 tests, and both handed-Thaw and older-lease mutations failed behaviorally and passed after restoration. The single ZFS/Cloud session reported 22 PASS and 3 FAIL: C3's premature readonly assertion and C6's expired-hold sample are harness false positives confirmed by raw samples and not_executed Deploy documents, while slow Thaw is a product failure; normal Release and delta reuse passed, and teardown exited 0. Diagnostic follow-ups remain for lost cancellation, generic AdoptLease failure, and the daemon's physical-name fallback, while slow Freeze is unprobed and the release soak remains required. GitGuardian's POSTGRES_PASSWORD: "secret-fsn-1" finding at test line 621 is generated fixture data from the mocked inspectContainer, not a real credential.

Added 13 commits

  • Decide each Volume switch verb's deadline in its allowlist96e68e5
  • Test that Freeze and Thaw are awaited as long as the Machine works on thembe3e951
  • Await Freeze and Thaw as long as the Machine's volume plugin may takef412eae
  • Test that a Volume's Container step outlives the caller that asked for it1f85374
  • Run a Volume's Container step in a task the Machine ownsfb5f64e
  • Ask a Volume's Container step again while the Machine still works on it70c22c3
  • Revert "Decide each Volume switch verb's deadline in its allowlist"0000f3a
  • Revert "Test that Freeze and Thaw are awaited as long as the Machine works on them"1f2a9bb
  • Revert "Await Freeze and Thaw as long as the Machine's volume plugin may take"a78ff7a
  • Revert "Test that the caller waits out a Machine's Start of a handed Container"0284e68
  • Revert "Wait out a Machine's Start of a handed Container"291e1ae
  • Answer every Volume switch verb within the usual Machine RPC deadlinefe026e7
  • Merge origin/main into warm-move/move-release3ef6247
NNickOct 8, 2026

PASS+NOTES (verifier: Opus @high)

Head: 3ef624786f3b43e942533798d636e8a0e55010d1

Freeze's docker stop, Thaw's docker start and the handed Start's docker start now run in one Machine-owned task per Volume, which holds the mutation lock and the mount grant until Docker returns, so a caller that gives up drops only a waiter. A same-record replay attaches to the running task or reads the per-record memo, an older record is refused through the fence, and the runner turns busy into a 5 s sleep for at most 120 checks without spending an Inngest retry. Three local mutations each failed a behavioral test and were restored: removing AdmitHandedStart's running-step refusal hung a_handed_start_keeps_its_grant_after_its_caller_gives_up_and_starts_once, letting any request attach to the running step turned a_stale_thaw_is_refused_while_a_start_runs from stale_step into busy, and dropping the busy catch in containerStep failed both "asked again" Move and Release tests with "freeze on fsn-1: unavailable: busy". The gate at this head passed fmt, clippy, nextest (2140/2140), lint and typecheck. pnpm test hit host ENOSPC on its first run and an unrelated afterAll teardown timeout in read.changes.postgres.test.ts on the rerun, with all 1092 tests passing and every file passing in at least one of the two runs. The main merge touched billing, the CLI captures and the regenerated config-wasm.mjs only, and "secret-fsn-1" is still only a test fixture. Note: the fixer's vc run (17/17 on fe026e737) is sound, but slow_thaw_retry_safe never starts a second Release (no SLOW_THAW_RETRY line, same run id), so it proves only same-run busy re-asks. Follow-ups are tracked, not blockers: the global mutation lock now stays held for Docker's whole run (1), the Docker memo lives in plugin memory (2), and docker stop has no Machine-side bound (3).

NNickOct 8, 2026

PASS+NOTES (verifier: Codex gpt-6.1-sol @high)

Head 3ef624786f3b43e942533798d636e8a0e55010d1.

The Machine-owned Freeze, Thaw and handed Start keep their guards until Docker returns, fence stale records and attach same-record replays to existing work; the Move remains forward-only after handover and retry retains target exclusion. All head gates pass, with 2140 core tests and 1092 dashboard tests, and all three local mutations fail their predicted behavioral assertions and pass after restoration. I audited the fixer's 17-pass vc run at fe026e737; I reran only slow_thaw_retry_safe because the original check could pass without retrying Thaw, and the targeted run at this head proves an in-flight same-record replay plus runner restart during a 45-second start completes the original Release with exactly one Docker start and at most one sampled writer. Tracked follow-ups are the global plugin mutation lock blocking other Volumes, the Docker memo being lost on plugin restart, and an unbounded Machine-side Docker stop. The advertised ten-minute wait also understates wall time because each of the 120 bounded polls can spend three seconds awaiting Busy in addition to the five-second sleeps. The main merge leaves PR 8 logic unchanged, and GitGuardian's secret-fsn-1 remains fixture-only.

Cchatgpt-codex-connectorBotOct 8, 2026
Synced from GitHub

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

ReviewStatusCommitReview trigger
📝 Code Review✅ Completed 2026-10-08T20:04:56.401785Z3ef6247Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Cchatgpt-codex-connectorBotreviewed · 2 inline threadsOct 8, 2026
Synced from GitHub

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ef624786f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

dashboard/src/modules/volume-run/volume-run.tsline 7Open thread
……@@ -4,13 +4,15 @@ export const VOLUME_RUN_STATES = ["requested", "running", "done", "failed", "can
55export const ACTIVE_VOLUME_RUN_STATES = ["requested", "running"] as const satisfies readonly VolumeRunState[];
66
7-export const VOLUME_RUN_KINDS = ["mirror", "sync", "delete_mirror"] as const;
7+export const VOLUME_RUN_KINDS = ["mirror", "sync", "delete_mirror", "move", "release"] as const;
Cchatgpt-codex-connectorBoton line 7Oct 8, 2026

P1 Badge Update the Volume Run glossary for Move and Release

Adding move and release here changes the established Volume Run domain behavior, but dashboard/CONTEXT.md:43-44 still defines a Volume Run as only Mirror, Sync, or Delete Mirror. Amend that glossary entry so the canonical domain documentation describes the newly shipped kinds and their ownership semantics.

AGENTS.md reference: AGENTS.md:L7-L7

Useful? React with 👍 / 👎.

dashboard/src/modules/volume-run/volume-run.server.tslines 330–333Open thread
……@@ -296,14 +309,71 @@ export const sendSwitch = <R extends VolumeSwitchRequest>(
330+ const holders = frame.containers.filter((container) =>
331+ container.machine_id === writer.id
332+ && container.kind === "service_container"
333+ && container.resolved_spec.volumes.some(({ source }) => source.kind === "provisioned" && source.name === ctx.dockerVolume));
Cchatgpt-codex-connectorBoton lines 330–333Oct 8, 2026

P1 Badge Refuse moves for services with other persistent volumes

When the selected Service mounts this Volume plus another provisioned Volume, this predicate accepts it because it only checks that the requested Volume is among the mounts. startHanded later recreates the complete resolved_spec on the target, but only the requested Volume was copied and promoted; container creation therefore creates a fresh empty dataset for the other provisioned Volume. This can restart the Service against empty data after the handover, while the real data remains on the source. Reject these Service specs before the move starts, or move/register every attached persistent Volume as one operation.

Useful? React with 👍 / 👎.