Add SetVolumeCopy, one verb for every Volume copy transition#1689

merged19
volume-cut/p4-slot
volume-cut/p5-add
openedOct 12, 2026byNNick
View on GitHubGitHub import enabledPR last imported 21d ago

Summary

This is PR 5 of 7 in the Volume RPC cut. It adds SetVolumeCopy, the one verb a Machine serves for every step of a Volume run. Nothing sends it yet, so no user sees a change from the verb itself. PR 6 switches Cloud to it, and PR 7 deletes the old verbs.

One change is visible to Docker callers now. While a stopped writer's root sits in its mirror slot, plugin Get answers from the slot through the ployz:cut marker. Docker keeps its record of the Volume through the cut. On main, an inspect in that window made moby purge the record.

Cloud sends { name, expect, want }. expect is what Cloud believes the copy is, and want is where the copy should end up. The Machine checks the want first. If the copy is already there, it answers the copy and does nothing. It checks the expect next, and refuses mismatch with no effect if they differ. Otherwise it runs one cell of the transition table. A late or resent request therefore lands only on the state it was planned against.

Changes

  • ployz-core/src/volume_copy.rs. The wire types and plan(), one pure function over the transition table. expect is Writer{epoch}, Mirror{newest, slot} or None. want is Writer{at, epoch, start}, Mirror{from, target, drain}, Snapshot{name} or Forget. A start is Kept{container_id} or Create{namespace, resolved_spec, creation_key}, named CopyStart in TypeScript. The reply is { copy: Option<VolumeCopy> }. A refusal is an RpcError whose details hold SetVolumeCopyRefused{reason, copy}. The reasons are mismatch, no_transition, busy, no_capacity, behind, written and no_source.
  • VolumeCopy. Root gains epoch, a WriterEpoch kept in the ZFS property ployz:epoch. A missing property reads 0. Slot gains pull and slot. slot is a SlotId, the guid of the slot parent dataset, sent as a string. A Mirror expect must match both newest and slot, so a slot pulled again after a Forget refuses a promote planned against the old one.
  • Catalog. The SetVolumeCopy row has capability ployz.volume.copy.set.v1 and scope Container.
  • machine_api/local_volume_copy.rs. The daemon runs the table and the Container half under switch_mutation. An admitted request runs detached under an owned guard. A caller that hangs up mid-cut can no longer let the next request plan against a half-done cut.
  • volume_plugin/copy.rs. Seven internal routes, Volume.Copy.{Inspect,Cut,Snapshot,Forget,Writer,Pull,Destroy}, own the ZFS work. Each checks which datasets exist, acts, and answers the copy, so a resend after a crash converges.
  • The cut. stop_behind_ingress drains and stops the writer's Container. The cut lists the Volume's holders again under its exclusive turn and refuses busy if one started during the drain. The plugin then sets readonly, takes the target snapshot, builds the slot with a refquota and renames the root into it. It refuses written if anything landed after the target.
  • Writer. A Writer want can raise a root's epoch at its snapshot. A lower epoch refuses no_transition, and a higher one behind the snapshot refuses behind. A start counts as done when any holder of the Volume runs, so a late Writer never starts a second Container on a live root. A Kept start whose Container Docker no longer has refuses no_transition. Create finds its Container by creation_key through find_creation, so a replayed create starts nothing new.
  • Pull. A writer serves its Volume's bound at GET /volume-bound/<name> on the send port. A pull into an existing slot raises the slot's refquota to that bound, and reserves the difference first. A full pool refuses no_capacity. A pull for another target while a receive runs refuses busy. A resend after the pull landed finishes it, whatever its expect says.
  • Fault points. Each transition has one. They are SetVolumeCopy.WriterToMirror, .WriterToSnapshot, .WriterToForget, .MirrorToWriter, .MirrorToMirror, .MirrorToForget and .NoneToMirror.
  • SDK. volume_switch passes SetVolumeCopyReply and the typed refusal through. A new napi method, removeContainer(machine, container), sends RemoveContainer with force: false and maps NotFound to Ok. generated/payloads.d.ts, catalog.json and the capability list are regenerated. The regen also drops ployz.volume.clear-final.v1.
  • SDK wrapper. index.js wraps the native Client by hand, so it forwards removeContainer too. tests/config-contract.mjs now reads index.d.ts and asserts that every declared Client method is forwarded. CI's sdk job runs it.
  • Dashboard fixtures. The copy fixtures carry epoch, slot and pull.

Verification

The gate commands ran at 697a34766. Every result below is measured unless it says otherwise.

  • cargo fmt --check exited 0.

  • cargo clippy -p ployz-core -p ployzd -p ployz --all-targets -- -D warnings exited 0.

  • cargo nextest run -p ployzd -p ployz-core -p ployz-testkit -E 'test(volume_plugin) | test(volume_copy) | test(volume_send) | test(set_volume_copy) | test(faults) | test(capabilities) | test(volume_switch) | test(machine_api) | test(departure)' passed 281.

  • cargo nextest run -p ployz -E 'test(volume_switch_tests)' passed 3.

  • cargo test -p ployz --test sdk_payloads passed 3.

  • pnpm typecheck in dashboard/ exited 0. The volume vitest suites cannot run in this worktree because ployz-sdk.node is not built there. PR 6 runs them.

  • every_request_of_a_move_replayed_late_through_the_planner_keeps_one_writer replays every request of a move, an undo before the cut and an undo after the cut, forward and reversed. It holds at most one writable root throughout, and it passes in the nextest run above. Its holder model lives in the test, so how closely it matches the daemon is inferred.

  • Each review fix was reverted and nextest rerun. The test failed without the fix for the detached guard, the any-holder rule, the landed pull, the busy pull, the slot bound and the gone Kept start. The re-list of holders under the cut's turn has no revert check, so that its test catches the bug is inferred.

  • In the first fix round, with the slot comparison in expect_matches replaced by true, cargo nextest run -E 'test(volume_plugin) | test(volume_copy)' exited 100 with 2 failed. Both were the slot-id tests.

  • The 2-VM harness at p7 893530e10 failed every case at the move's last step with client.removeContainer is not a function. The method existed in Rust and index.d.ts but not in index.js, and vitest uses a double. Fixed at 6c2548f12. node crates/ployz-sdk/tests/config-contract.mjs passes with the fix and fails with it reverted, listing removeContainer. Measured.

Not run on real Machines yet. The 2-VM harness, the kill matrix and the real-ZFS soak run once at the top of the stack.

Merge Danger

No users run ZFS mirrors yet, and no released daemon was ever cut with these verbs. None of these breaks has a shim.

  • The new verb and capability is ployz.volume.copy.set.v1. Cloud at PR 6 requires it.
  • VolumeCopy changes shape. Root gains epoch, and Slot gains pull and a required slot. The Mirror expect also gains a required slot. A Cloud built on PR 5 alone does not match this daemon, so PR 5 and PR 6 land together.
  • The SDK capability list loses ployz.volume.clear-final.v1.
  • A pull into a slot smaller than the writer's bound now refuses no_capacity on a full pool. It used to fail later with EDQUOT as Internal. Inferred.
  • A late Writer with Create that lands after the Service was scaled to zero recreates the old Container. That needs a request seconds late to outlive a finished run, with a user scale-down in between. Accepted. Inferred.
  • A late first-round pull with expect: None after an undo can recreate a slot on B. That slot never becomes a writer. Inferred.
  • The WriterToSnapshot and MirrorToForget fault points test convergence after a restart, not a resume in the middle of the effect. Accepted.
  • No fault point sits between the epoch write and the start. A resend finds the root at the new epoch, plans WriterToWriter and starts. Inferred from the planner.
  • The plugin Mount lease clause still refuses while an old-flow lease is Open. PR 7 removes it.

Checks

2 failing · 14 successful · 1 skipped
Some checks failed

Failing2

6c2548f

Added 18 commits

  • Name the slot incarnation in a Mirror expect so a slot pulled again after a Forget refuses a stale promote0129fbd
  • Finish a slot's own landed pull for a Mirror want whatever the expect says, so a resend after the landing prunes the slot and clears the pull0bb930b
  • Let a Writer want raise the epoch of a root at its snapshot so a cut from an ended run refuses193f070
  • Serve SetVolumeCopy on the daemon: the transition table and the cut304eb65
  • Add the SetVolumeCopy wire shape and its planner31e950e
  • Give the dashboard's copy fixtures the epoch, slot, and pull fields VolumeCopy now requires697a347
  • Pass SetVolumeCopy replies through the SDK and add remove_container6ac0033
  • Format the SetVolumeCopy holder check and its tests with rustfmt.6db876a
  • Regenerate the SDK capability list for SetVolumeCopy823acab
  • Run an admitted SetVolumeCopy detached with the switch guard, so a caller that hangs up mid-drain cannot let the next request plan against a running cut943c3f2
  • Name every plan arm in the Mirror test helper and move the indexing expect to the test that indexes, so clippy passes.a3dae0b
  • Replay every SetVolumeCopy of a move and both undo orders late through the planner, and undo after the cut by forgetting B's slot before A takes the writer at a raised epoch.b434cfa
  • Count a Writer start as running when any holder of the Volume runs, and refuse a Kept start Docker no longer has as NoTransitionbe6613e
  • Regenerate the SDK payloads for the slot id in a Mirror expectc5e4696
  • Raise an existing slot's refquota to the writer's bound on every pull, so a receive fits a Volume that grew.c715c39
  • Give the volume plugin one internal route per SetVolumeCopy effectdfc5b46
  • Refuse a pull for another target Busy while a receive runs, before the pull label moves.e72015e
  • List the Volume's holders again under the cut's exclusive turn, so a Container started during the drain refuses the cut Busyf1cf0ce
Synced from GitHub

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

yaml
1reviews:
2 auto_review:
3 drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Added 1 commit

  • Forward removeContainer through the SDK's Client wrapper, and check every declared Client method is forwarded6c2548f