Require Device approval for Cluster joins under enforcement#3939

Why

Hosted Ployz Cloud must be clearly safer than self-hosting. Root on np1, the server running Cloud, must not be able to add a Machine or Device to a customer's enforcing Cluster, or get its Sealing Key wrapped to a key it picks.

Today an Operator can register caller-chosen keys, a joiner installs Cloud's unsigned Devices, and a mesh request without an Operator header becomes Full. This change closes those three paths when Device enforcement is enabled, and preserves today's unsigned join when it is disabled. It is stacked on #3926 at e58f75f583d0374270995f2164c87041b7b7abb9.

The authority shape is Signed<JoinApproval>: Cluster Sealing public key, immutable Machine allocation and WireGuard key, sealing recipient, network, approved Devices, admitting Machine IDs, signer, issued_at, and expires_at. A Machine accepts it only if its own ID is named. Machine IDs keep approval inside its Cluster even when several Clusters in one Organization share a Sealing public key. This implementation names exactly one admitting Machine: one local consumption owner can enforce one use without cross-Machine consensus.

DESIGN choices touched:

• Stable promise: request fields are optional; unsigned enrollment is unchanged with enforcement off. The new join capability is explicit. Approved key capsules carry the signature inside their encrypted plaintext, so removing unsigned response fields cannot downgrade an honest approved capsule. An older joiner refuses that capsule; upgrade the joining CLI/daemon before using approved joins.
• 5, Two-tier identity: approvals bind durable Machine IDs and keys, not Server names.
• 6, Machine-local authority: each admitting Machine checks its own trusted Devices, ID, enforcement setting, and durable consumption. Mesh observations grant no Full authority. One issuer avoids a barrier or shared writer across Machines.
• 7, Cloud drives, never owns: Cloud relays proposals and signed envelopes; it cannot mint approval or expand its Device roster.
• 8, Evidence over claims: consumed admission is reported as join_already_admitted, with inspection guidance rather than another signing attempt.
• 9, Lean on proven primitives: reuse #3926's signing envelope, lifetime check, and single ConsumptionLog owner with its persisted clock floor; retain HPKE for key transport.
• 10, Client-first, daemon when forced: the CLI signs and selects direct Full connections. Verification, consumption, and transport scope remain daemon safety boundaries.

Summary

Register, Enroll, and ShareSealingKey refuse unapproved requests under enforcement. Register consumes before key wrapping or publication. The existing approval log retains consumed Machine IDs permanently, alongside restart operations and the shared clock floor. Approvals have the same 15-minute maximum lifetime and 2-minute future-issuance skew as #3926.

server add signs automatically when the local Device key is available. Copy-paste joins print a code for ployz device approve --join CODE; rerunning the same join completes it. Automatic signing checks the entire expected allocation, identity, keys, and policy before signing a pending response. Ambiguous proposals require an explicit admitting Server.

Only a local Unix socket or an authenticated Full management connection grants Full. Mesh TCP gets Operator access, including forwarding and Volume transfer. Full CLI operations dial each target directly, preserving the shared session refusal gate. Explicit SSH Device commands act on that Server; bootstrap the first Device separately on each Server.

Bootstrap scope: a fresh joiner without an approval behaves as on main and trusts the Devices Cloud sent until a Device re-approves it. Such a Cloud-faked off-mode joiner cannot obtain admission or a Sealing Key from an enforcing Machine. Verified SSH or a short code matched in the browser is a follow-up bootstrap pin. Requiring approval for every join by default remains the open “first Device for dashboard-only customers” decision; this PR does not decide it. Both server guides describe this boundary.

Evidence

Final-head proof used verify-server on a two-Machine Boat Cluster with local Ployz Cloud and the exact checkout SDK. Revision: 19d48d9e2d20a77ff1fe51211d4a04508893ce4c. Both running daemon hashes were bb7818831a001a51235e99745df9a57851446662d1ba0ee5fc020b80d9f7cca1. Evidence is retained in core/.verify/20261106T131747-34b3c2ab89/evidence/, including every command, exit status, stdout and stderr. Both Machines and Cloud were torn down afterward.

Commands below ran from the checkout; H=core/scripts/verify-cluster and R=core/.verify/20261106T131747-34b3c2ab89/cluster.json.

Approved join and injected Device

The initial unsigned Register for Machine fb84aaec80474786bb089a5540e3b8df was refused. Then:

Retrying that exact Register returned an assignment and wrapped Sealing Key. Before Enroll, I replaced its unsigned registration.devices with attacker public key fd1724385aa0c75b64fb78cd602fa1d991fdebf76b13c58ed702eac835e9f618, keeping the approved encrypted capsule. Enroll succeeded. A filtered read of /var/lib/ployz/machine.json on the joiner returned:

The injected Device was absent. Requests and replies are saved as fresh-register-request.json, live-approved-registration.json, live-enroll-injected-devices.json, and live-enroll-injected-result.json. Presenting the signed approval to the other Machine was refused: the approval names an admitting Machine this request does not match (wrong-admitting-machine.json). Normal Cloud server add --no-install --storage none --name machine-live-proof ... then completed the pairing.

Replaying Register returned forbidden, reason join_already_admitted, with no wrapped key. After systemctl restart ployz on the admitting Machine, the same request returned the same refusal (final-register-replay-after-restart.json).

Operator refusals and mesh scope

From the registered peer, I used the checkout CLI against tcp://[fdcc:f8a3:3f12:6ab6:1fcb:e8c:890c:1f09]:7569, without an Operator header. Valid-shaped Register used Machine ID aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, attacker key [2;32], subnet 10.210.77.0/24, and endpoint [2001:db8::77]:51820. Register, unsigned Founder Enroll, and ShareSealingKey to the real registered peer's matching key each returned:

None returned a wrapped key or registration. From that same peer:

The persisted admitting record still showed enforcement enabled and only device-nc5e4qhd7p. Forwarded inspection through that peer connection still worked:

Full and incremental Volume transfer

Normal CLI commands created Project join-proof, Service writer (nginx:1.29-alpine), and a 1 GB Volume data mounted at /data, then deployed. Both Machines had enforcement on during transfer.

SetVolumeCopy Snapshot/Mirror/Inspect requests and responses are in the command log. Reading the destination's existing read-only ZFS dataset returned both lines:

Final verify-cluster doctor exited 0: both Machines ran the expected daemon hash and observed both participating peers as up.

Enforcement off

I disabled enforcement on the admitting Machine through the approved Device, reset the disposable second Machine with Leave, and ran the ordinary Cloud command without the Device key:

This retains the unsigned Cloud roster behavior. Enforcement was enabled again on both Machines before the mesh and Volume checks.

One manual approved capsule expired while I rebased; Enroll correctly refused it. The fresh approved/injected-Device proof above passed. Update readiness also reported one peer during intentional join resets; the final doctor passed after restoring both peers. These transient verification states are not reported as product failures.

Focused checks and reviews

• cargo fmt --check: pass.
• cargo clippy -p ployz-core -p ployzd -p ployz -p ployz-testkit --all-targets -- -D warnings: pass.
• ployzd registration tests: 65 pass, including approved join, unsigned Register/Enroll/Share, wrong admitting Machine/recipient, injected Devices, replay after reopen, lifetime bounds, shared clock floor and real TCP scope.
• Existing approval consumption tests: 6 pass; authority tests: 19 pass; CLI Device tests: 12 pass; shared ended-session regression: 1 pass.
• Dashboard typecheck and targeted lint: pass; enrollment-route tests: 22 pass.
• SDK=exact bash core/scripts/build-cloud-sdk.sh: pass, including 38 SDK tests.
• /deslop and /no-comments: complete. Comment review deleted 0 comments; corrected one misplaced doc comment. No constraint encodings or unresolved comment findings.
• Two Codex gpt-6.1-sol agents reviewed separate angles. Comment/signing review found the pending-proposal substitution race; fixed by comparing the full requested allocation before automatic signing and refusing ambiguous manual proposals. Runtime review found the direct Full client's shared-session refusal gap and a missing refusal capability; both fixed and rerun. Its independent Boat run proved full/incremental transfer and mesh refusal at pre-rebase cd564861ffaab629cd4a14ed6033476c47c45a41 and was torn down. The parent repeated those runtime checks on the final rebased head above. Formal full-PR review remains for the draft; no non-Codex model ran.

Round 1 fixes

Round 1 closes the paths a hostile control plane still had after the first review. It adds two Membership checks and fixes two bugs this proof found. Commits be6f76833b through bab4c9b4f1 sit on top of 19d48d9e2d.

Signed membership. Under enforcement, a Machine peers with, gossips with, and admits a Machine only if one of its own Devices signed that Machine's join approval. The signed approval is published in the Machine row. The replicated store filters rows through the local record before WireGuard, gossip or admission reads them. The Members present when enforcement is turned on are pinned by ID, key and subnet. That pinned set is the durable trusted starting set. A Machine also pins each approved Member the first time it accepts it. So device rm of the Device that signed a join no longer cuts that Server off. ployz server rm remains the way to remove it.

Joiner bootstrap. A joiner holding an approval takes its Devices and starting peers only from the signed body (devices, bootstrap). It ignores Cloud's visible_peers and devices.

Replay and lost replies. An exact replay of a completed Enroll returns the stored result before any freshness check, so it succeeds after the approval expires. The admitting Machine stores the sealed Register and Share results with the consumed approval. Retrying the same Register returns the identical result, even after a daemon restart. A Register for the same Machine with another key is refused as join_already_admitted. The Machine never seals twice to different keys.

Allocation and code. Under enforcement, Register refuses a subnet that overlaps a current row before proposing. The CLI prints Approving Server NAME on SUBNET with endpoints … (join code …) before it signs. The join code is 20 base32 characters (100 bits) of the Machine ID and key digest. ShareSealingKey checks the approval against the recipient row's real allocation, not against itself. Pending proposals are capped at 128, and expired ones are pruned.

Transport scope. Full scope over the mesh is refused only when enforcement is on. With it off, mesh callers get Full as on main, including ployz exec across Servers. The refusal names a direct connection or SSH. device rm and device enforce list a Server with no direct connection under omitted and exit 3.

Cloud capability. cloud.rs selects the join path by details.capability, not by error text. This proof found that the CLI's refusal sanitiser dropped any 409 body with a details field. So automatic signing never ran through real Cloud HTTP (enroll HTTP 409: enrollment rejected). The sanitiser now keeps details only when it equals exactly {"capability":"ployz.machine.join.device-approval.v1"}. Any other details still discards the message (c6035613fb).

Accepted residual

This residual is accepted for 0.4 and tracked in #3941 (F=2, S=6, estimate 6–10 files and 200–350 lines). A second accepted residual, the trust-on-first-use starting set, is tracked in #3945 (F=2, S=6) and described under Round 3. Suppose Cloud withholds the sealed capsule during an approved join. The new Machine then joins with no Sealing Key, under Devices Cloud chose, with enforcement off. Honest Machines peer with it because a customer Device approved its ID and key. It cannot decrypt sealed secrets. It cannot get a new Machine admitted by any honest Machine. Cloud can still run workloads on it, which deploy authorization must address separately. The fix is for the new Machine to prove acceptance under the real Sealing Key before honest Machines activate its row. docs/user/servers/add-a-server.md states the same boundary. stripped_capsule_cannot_authorize_a_second_machine_on_an_honest_member documents it in tests.

I skipped the optional post-join CLI check. MachineDetails carries no enforcement flag, so the check needs a new wire field and SDK type, not a few lines.

Focused checks

• cargo fmt --check: pass.
• cargo clippy -p ployz-core -p ployzd -p ployz -p ployz-testkit --all-targets -- -D warnings: pass.
• ployzd tests for machine::, corrosion::store and machine_api: 428 pass. That includes the 19 join-approval tests. The new removing_the_signing_device_keeps_an_admitted_machine_peered fails without the pin. The new share_refuses_an_approval_for_another_allocation_of_the_recipient covers the Share allocation check.
• ployz tests for device, cloud and exec: 600 pass. The cloud_enroll and cloud:: set (55 tests) passes after the sanitiser fix. New tests are join_summary_names_the_server_subnet_and_endpoints_before_signing and join_approval_capability_survives_the_refusal_sanitiser_and_only_from_details.
• ployz-core join tests: 3 pass, including the 100-bit code.
• /deslop changed nothing. /no-comments removed one doc comment that restated its function.

Runtime proof

I used verify-server on a four-Machine Boat run with local Ployz Cloud (--cloud). The CLI was built with verify-faults for ployz debug local-rpc. Manifest: core/.verify/20261106T160514-4345d47860/cluster.json, revision bab4c9b4f117302f7896a769751fdbb1b6962d83. The checks ran on the daemon from c6035613fb. Daemon code is identical at bab4c9b4f1, whose last commit is docs only. All three remaining daemons hashed f5dbf56102817241042cd77fd64d0974fe3c5278065323686c49ca37d660e49a. The driver is core/.verify/round1-head/drive.py, one subcommand per check. Its assertions, commands and output are in core/.verify/round1-head/evidence/ (steps.log lists every PASS and FAIL). The run was torn down afterwards.

Raw debug local-rpc calls over --connect ssh:// reach a daemon's Unix socket with Full scope. They stand in for root on np1 driving Machines directly.

Approved join peers with everyone. Enforcement was on for all four Machines. machine-3 left, then cli --device -- server add root@… --name machine-3b … ran through Cloud. The first attempt hit the sanitiser bug above (enroll HTTP 409: enrollment rejected). After the fix, the CLI printed Approving Server machine-3b on 10.210.4.0/24 with endpoints [2001:41d0:a:1b3c:100::2c]:51820 (join code 2ntohggvzxdfqn3ipojy)., then signed and joined. machine-3b persisted device_enforcement: true. Its WireGuard peers included machine-1, machine-2 and machine-4, and each of them peered machine-3b.

Lost Register reply. machine-4 left, and its new ID was registered on machine-1 with raw RPCs.

Stripped capsule, unapproved Machine. I took that registration, set sealing_key to null and devices to an attacker key, and sent it to machine-4 as Enroll. machine-4 joined with device_enforcement: false. Inspect reported sealing: {"kind":"needs_access"}, and no Sealing Key file existed. machine-1, machine-2 and machine-3b each had machine-4's WireGuard key as a peer. A Register for an unapproved Y (eeee…, key [9;32]) succeeded on machine-4 and was refused on machine-1. Thirty seconds later, Y's row was in machine-1's raw Corrosion table.

Yet list_machines on machine-1, machine-2 and machine-3b did not list Y, and none of them had a WireGuard peer for Y's key.

device enforce with a Server unreachable. I ran power off machine-2, then cli --device -- --json device enforce. It exited 3. machine-1 and machine-3b reported enforced: true. machine-2 was listed under failures (unavailable, machine-2 did not answer; its rows are missing.), with omitted: []. A Server that has just gone down is attempted and fails. The omitted list is for a Server the CLI has no direct connection to, which full_mutations_report_servers_without_a_direct_connection_as_omissions covers. The docs now say which list each case lands in. device rm behaved the same way (exit 3). After power on, device enforce exited 0.

Forwarding and Volume transfer, enforcement on. For fixture setup only, I turned enforcement off and created Project sec04 with Service web (nginx:1.29-alpine). It mounts a 1 GB Managed Volume data at /data. Deploy placed it on machine-3b. I turned enforcement back on before every check.

• service port-forward web 51931:80 served sec04-round1-proof from the Volume.
• volume mirror data --to machine-1 --wait finished as run 54e8e4bf… with state: done. The mirror dataset ployz/ployz-mirror/sec04-production_vol-ce536efb…/fs on machine-1 held sec04-round1-proof.

Enforcement off matches main. I ran exec -T web -- cat /data/proof connected only to machine-1 over SSH, so the call to machine-3b crossed the mesh. With enforcement on it exited 1 with This Server enforces Device approval. Connect directly as an approved Device, or over SSH. and the next step Connect directly to the target Server as an approved Device, or over SSH. After device enforce --off, the same command returned sec04-round1-proof. exec through Cloud also worked with enforcement off, and again with it on (direct connections).

Turning enforcement off also made the fake Y row count as a Server again, as every row does on main. device enforce --off then listed Y as omitted. I removed it with server forget unapproved-y --confirm dead before continuing.

Not proven here. Creating a deploy under enforcement still meets #3926's restart gate. So the fixture deployed with enforcement off, as in the earlier evidence. No non-Anthropic reviewer ran this round.

Round 2 fixes

Round 2 closes two bugs and one risk from the second review. Commits 573a9380d3 and 2ed821dec1 sit on top of bab4c9b4f1.

Saved bootstrap peers pass the membership check. A Machine that joined the unsigned way keeps Cloud's bootstrap rows. Before this fix, a row that was absent from the replicated table when enforcement turned on survived in that list. After a restart it went into WireGuard and the Corrosion bootstrap without any check. LocalMachineRecord::bootstrap() now returns only rows the record accepts. The WireGuard start, the Corrosion start and the boot pass all read it. A network rebuild also drops bootstrap peers that the record no longer accepts, which covers the wiped-store fallback that merges bootstrap peers into the snapshot.

The Device signs the starting set. Before this fix, turning enforcement on read the unenforced replicated Machines table and trusted every row in it. The Device approved only {enforced: true}. The new data shape is Signed<StartingSet>: operation, members (each a Machine ID, WireGuard key and subnet), issued_at, expires_at and signer, signed under a new enforce purpose. ployz device enforce asks each Server for its own allocation over the existing Full connection. It prints Trusting N Servers from the start: name (subnet, id), …, then signs. A row that no daemon answers for is never in the set. Each Machine refuses the request in four cases: no set, a signer outside its own Devices, a set that leaves out that Machine (which scopes it to one Cluster), or a set past the 15-minute lifetime. It then consumes the operation in the existing ConsumptionLog, behind the shared clock floor. Finally it persists the members as trusted_machines. While enforcement is on, a new set adds to the trusted Members. The request gained starting_set, so the capability moves to ployz.machine.set-device-enforcement.v2. A CLI from #3926 refuses cleanly against this daemon, and this CLI names an older daemon as missing. Turning enforcement on now needs this computer's Device key, as docs/user/servers/manage-servers.md states.

Enroll replay only while Joining or Participating. into_resetting keeps the stored enrollment. Resending the original Enroll after Leave committed used to return Replay and report Enrolled for a Machine being destroyed. It now returns a conflict, as it did before this PR.

The capability catalog test also lacked the three join capabilities from the first round. They are now recorded, and each says how older Machines are handled.

DESIGN choices touched: Stable promise (the capability bump, with clean refusal both ways), 6, Machine-local authority (each Machine verifies the set against its own Devices and allocation, and consumes it locally), 7, Cloud drives, never owns (Cloud can no longer pick the trusted rows), 9, Lean on proven primitives (the same Signed<T>, lifetime check and consumption owner as joins and restarts).

Focused checks

• cargo fmt --check: pass.
• cargo clippy -p ployz-core -p ployzd -p ployz -p ployz-testkit --all-targets -- -D warnings: pass.
• ployzd tests for machine::, corrosion::store, machine_api, network and boot_pass: 565 pass. machine_record: 36 pass. ployz tests for device, cloud and cluster: 384 pass. ployz-core: 578 pass.
• New tests. a_saved_bootstrap_peer_outside_the_starting_set_is_not_offered_after_restart fails with the filter removed (it lists both IDs). enforcement_trusts_only_the_signed_starting_set_not_rows_registered_before_it covers fix 2. enforcement_refuses_a_missing_foreign_misscoped_or_reused_starting_set also covers fix 2. resending_the_original_enroll_while_resetting_conflicts failed before its fix. starting_set_summary_names_each_trusted_server_before_signing covers the CLI output.
• /deslop: no changes. Comment review ran in-thread, because a delegate may not start children. It removed one doc comment that restated its function. No Comment Sicko child ran.

Runtime proof

I used verify-server on a three-Machine Boat run with local Ployz Cloud (--cloud) and the verify-faults CLI. The manifest is core/.verify/20261106T172341-146a608707/cluster.json, at revision 2ed821dec13a3f60eff2d30d1f6aacd8865ac538, the PR head. The driver is core/.verify/round2/drive.py, and its PASS and FAIL lines are in core/.verify/round2/evidence/steps.log. The run was torn down afterwards and its evidence kept.

Saved bootstrap peer. I stopped machine-3, appended a row Z (dddd…, key [5;32], subnet 10.210.242.0/24) to its saved Join bootstrap in machine.json, and restarted it. Enforcement was off. Corrosion's config then listed [fdcc:505:505:505:505:505:505:505]:7570 as a gossip bootstrap peer, which is the bug's baseline. After enforcement and systemctl restart ployz, Corrosion no longer listed Z, and machine-3 had no WireGuard peer for Z's key. Z was still in machine.json, so the filter kept it out, not a deletion. With enforcement off, the snapshot rebuild had already pruned Z's WireGuard peer, so the WireGuard baseline was only transient (the one FAIL line in steps.log). The Corrosion baseline reproduced the bug.

Starting set. I ran Register for Y (eeee…, key [9;32]) on machine-1, which was unenforced, through raw debug local-rpc. All three Machines then listed Y. Then cli --device -- --json device enforce printed:

It exited 3 with Y under omitted. Each Machine's trusted_machines held exactly those three IDs. Twenty seconds later, none of them listed Y or had a WireGuard peer for Y's key. A second device enforce signed a fresh set and exited 0.

Approved join, port-forward, Volume transfer, enforcement on. machine-2 left. Then cli --device -- server add … --name machine-2b printed Approving Server machine-2b on 10.210.3.0/24 … (join code ztpvryhoimhicdm2ujvu)., signed and joined. It inherited enforcement, and machine-1 and machine-3 both peered it. service port-forward web 51931:80 served sec04-round2-proof from the Volume on machine-3. volume mirror data --to machine-1 --wait finished, and the mirror dataset on machine-1 held sec04-round2-proof. Cloud's server ls listed three Servers up. The final doctor exited 1 only because machine-2 rejoined under a new Machine ID that the manifest does not know (expected 3 reachable Machines).

The fixture (Project sec04, Service web, a 1 GB Volume) was deployed before enforcement, as in round 1. Deploy under enforcement still meets #3926's restart gate. No non-Anthropic reviewer ran this round.

Round 3 fixes

Round 3 closes one bug from the third review and makes a second finding visible. Commits a9615420e2 and 15b700c083 sit on top of 2ed821dec1.

A refused peer leaves WireGuard even while the store catches up. Round 2 filtered refused rows out of the bootstrap candidates, but it did not take an installed peer off the interface. A store that has never held another Machine's row selects PeerSet::Bootstrap, and peer_changes then made no removals at all. Take a peer Z that an unsigned join installed earlier, with enforcement turned on by a set that leaves Z out. Z kept its WireGuard key and routes through a store recreation, and through a daemon restart, whose first apply is always Bootstrap. Now peer_changes takes a keeps_unlisted predicate from LocalMachineRecord::keeps_unlisted_peer. In Bootstrap, an installed peer missing from the plan stays only when that predicate keeps it. With enforcement off it keeps every peer, as on main. With enforcement on it keeps only a trusted Member's key, from the signed starting set or an approved member this Machine already pinned. The routes of a removed peer go too. Snapshot is unchanged.

Under enforcement this removes any installed key that no accepted row or trusted Member names, not only Z. One tradeoff follows. A Machine whose signed join reached the store after this Machine last read it is not yet pinned here. If this Machine's store is then recreated, it drops that peer until the store refills through a pinned Member. That matches the rule that an enforcing Machine peers only with Machines its own Devices signed.

The starting set is trust on first use. It is only as trustworthy as Ployz Cloud is at the moment you turn enforcement on, the same as SSH's first connect. device enforce signs the Machine ID, WireGuard key and subnet each Server reports. If Cloud swaps a Server's key while the set is gathered, the Device signs a key Cloud controls. This round makes that checkable but does not close it. The Trusting N Servers line now shows each Server's WireGuard key fingerprint, machine-1 (10.210.0.0/24, 9bf8…, key 3tneseul2uuppwo6). That is 16 lowercase base32 characters, 80 bits of a domain-separated SHA-256 of the key. sudo ployz status on a Server now prints a wireguard key line with its own fingerprint. Its JSON gains this_machine.wireguard_key and this_machine.fingerprint. docs/user/servers/manage-servers.md states the trust rule and the check, and docs/user/cli/overview.md documents the new status line. The fix, gathering over pinned SSH or requiring a fingerprint confirmation, is #3945 (F=2, S=6).

DESIGN choice touched: 6, Machine-local authority. The catch-up path now asks the Machine's own record which unlisted peers it may keep, rather than keeping every one.

Focused checks

• cargo fmt --check: pass. cargo clippy -p ployzd --all-targets: no warnings.
• cargo test -p ployzd --no-fail-fast: the lib suite passes with 1539 tests, and so do machine_record, management_connect, network_plane, network_namespace, dial_stdio and startup_refusal. Two binary-target tests failed for reasons unrelated to this diff. volume_plugin::tests::without_systemd_a_restarted_plugin_binds_its_socket_and_departs_a_machine_without_zfs failed once under parallel load and passed 3 of 3 alone. process_smoke::join_leaves_a_journal_line_and_records_the_restart refuses to enroll over this dev host's existing unlabelled ployz Docker network.
• Fingerprint checks. cargo clippy -p ployz-core -p ployz --all-targets -- -D warnings: pass. a_wireguard_key_fingerprint_is_stable_and_tells_keys_apart pins the value for [3; 32] to 3tneseul2uuppwo6, which I computed independently in Python, so a CLI and a daemon of different versions print the same fingerprint. status_on_a_server_shows_the_fingerprint_device_enforce_prints renders the status line. starting_set_summary_names_each_trusted_server_before_signing now expects the fingerprint. ployz tests matching status, device and link: 156 pass. The fingerprint change ran no Boat: it changes CLI output and a local inspect read, not Machine behaviour.
• New tests. startup_removes_an_installed_peer_enforcement_refuses covers the first apply after a restart. a_rebuild_from_a_fresh_store_removes_only_peers_enforcement_refuses covers a rebuild with held_another_machine == false, with enforcement on and off. Both failed with keeps_unlisted_peer forced to true (tests.rs:456 and :485) and pass with the fix.
• /deslop: no changes. I reviewed the comments in-thread and kept only the docs on new items, plus one clause added to the snapshot_peers doc.

Runtime proof

I used verify-server on a two-Machine Boat run with local Ployz Cloud (--cloud). The manifest is core/.verify/20261106T175226-25e1d066b1/cluster.json, at revision a9615420e282b76d9cb38f4785d4a5a760a9280a, the PR head. The run was torn down afterwards and its evidence kept, including plant-z.sh, the script that planted Z.

1. I deployed web (nginx) to machine-1 before enforcement. cli --device -- device enforce printed Trusting 2 Servers from the start: machine-1 (10.210.0.0/24, 9bf8aaef…), machine-2 (10.210.1.0/24, 39109404…). Both records showed device_enforcement: true with exactly those two trusted_machines.
2. On machine-2, the joiner, I stopped ployz. I appended a Z row (7777…, a fresh wg genkey key, subnet 10.210.77.0/24) to its saved Join bootstrap in machine.json. Then I installed Z by hand with wg set ployz-wg peer ptgoqf… allowed-ips 10.210.77.0/24,fdcc:7777::1/128 and ip route replace 10.210.77.0/24 dev ployz-wg. I stopped ployz-corrosion and moved store.db to /root/store-backup/, so the restarted store held no other Machine. Before the restart, wg show ployz-wg listed machine-1's peer and Z, and ip route listed both /24s.
3. I ran partition on machine-2, so gossip could not refill the store, then systemctl start ployz.socket ployz.service at 17:56:32. At 17:56:40, wg show ployz-wg listed only machine-1's peer (JHxaQB…, 10.210.0.0/24), and ip route had no 10.210.77.0/24. At 17:57:19 the new store's machines__crsql_pks held only machine-2's own ID, so every rebuild in that window ran as Bootstrap. Z was still absent.
4. After partition off, machine-2 renewed its handshake with machine-1. ping 10.210.0.1 lost 0 of 3 packets. Cloud's server ls listed both Servers up. The refilled store held machine-1 and machine-2 only. cli --device -- service port-forward web 18431:80 answered HTTP/1.1 200 OK with <title>Welcome to nginx!</title>.

The safety check refused an rm of the store, so I moved store.db instead, which serves the same purpose. A negative control on the round 2 daemon did not run. The unit tests above show that the old peer_changes keeps Z on both paths. No non-Anthropic reviewer ran this round.

Round 4 fixes

Round 4 closes one regression from round 3. Commit b246115e4b sits on top of 15b700c083.

A snapshot no longer removes trusted peers before the store answers for them. Round 3 let Snapshot remove every installed peer that no accepted row named. After store.db is recreated, one unsigned Machine row arriving first makes held_another_machine true. The filtered snapshot was then treated as complete, and it removed the installed approved peers before their rows arrived. Those peers carry the gossip that brings the rows back, so the Machine could stay cut off. A refused row that swaps a trusted ID's key did the same to the legitimate key.

keeps_unlisted_peer now takes answered: the IDs a complete snapshot has an accepted row or a removal for, or None on the Bootstrap path. Under enforcement, an installed peer stays when its key belongs to a trusted Member whose ID the snapshot has not answered for. A missing or refused row is no answer. Two things remove the peer:

• server rm deletes the accepted row. The new ReplicatedStore::removed_machines reads IDs that machines__crsql_pks still holds and machines no longer does. It runs only under enforcement on the Snapshot path.
• A new accepted row for that ID names another key.

With enforcement off, keeps_unlisted_peer returns answered.is_none(). Bootstrap keeps every peer and Snapshot removes every unlisted peer, as on main. peer_changes now derives routes the same way for both sets: kept installed peers plus updates. That keeps a kept peer's routes and equals the old result in every case where nothing extra is kept.

DESIGN choice touched: 6, Machine-local authority. The Machine's own record still decides which peers stay. It now also waits for its own store's evidence of removal before it drops a trusted Member.

Filed, not fixed. #3946 (F=2, S=5): a route can outlive its peer. apply_peers removes the peer and then deletes the route. If the daemon stops in between, startup rebuilds self.routes from the remaining peers only, so the orphan route is never deleted and blackholes that subnet. A failed delete_route is also ignored. Main has the same bug. The fix is to reconcile the actual routes on ployz-wg against the desired set and keep failed deletions for retry.

Focused checks

• cargo fmt --check: pass. cargo clippy -p ployzd --all-targets -- -D warnings: pass.
• cargo nextest run -p ployzd --no-fail-fast: 1876 of 1877 pass. process_smoke::join_leaves_a_journal_line_and_records_the_restart refuses to enroll over this dev host's unlabelled ployz Docker network, the same failure as round 3.
• New tests in network/linux/tests.rs run rows through accepts_machine the way the store does. an_unsigned_row_arriving_first_keeps_the_installed_trusted_peers and a_refused_key_swap_keeps_the_trusted_key both fail with round 3's semantics, where any complete snapshot removes unlisted peers, and pass with the fix. removing_an_approved_machine_removes_its_peer guards the other direction. corrosion::store_tests::a_deleted_machine_row_reads_as_removed checks the removed-IDs query against SQLite.
• /deslop changed one comment, which used a colon as a connector. I reviewed the comments in-thread and kept only the docs on changed items.

Runtime proof

I used verify-server on a two-Machine Boat run with local Ployz Cloud (--cloud). The manifest is core/.verify/20261107T000456-df1a35ac81/cluster.json, at revision b246115e4be15c5137f9bd861de36529d4a8f9d2, the PR head. Evidence is in that run's evidence/, including corro.py, which read and wrote machine-2's Corrosion API on 127.0.0.1:7571. The run was torn down afterwards.

1. cli --device -- device enforce printed Trusting 2 Servers from the start: machine-1 (10.210.0.0/24, e40e2b75…, key rzx763tv2mkagluq), machine-2 (10.210.1.0/24, 163ab817…, key 35rr4msmjpdfqvp2). Both Servers reported enforcement on.
2. I ran partition on machine-2 so gossip could not refill its store. At 00:10:33 I stopped ployz and the ployz-corrosion container, moved store.db to /root/store-backup/, and started ployz again. At 00:10:54 the new store held only machine-2's own ID. wg show ployz-wg listed machine-1's peer e+FLgu… with 10.210.0.0/24.
3. At 00:11:02 I inserted one unsigned row (8888…, random key, 10.210.88.0/24, no approval) into machine-2's store. machines__crsql_pks then held another Machine, so rebuilds ran as Snapshot. At 00:11:32 wg show still listed machine-1's peer, ip route still had 10.210.0.0/24 dev ployz-wg, and the unsigned row had no peer.
4. After partition off, at 00:12:29 the store held machine-1's row again. Both sides showed a handshake 40 seconds old. ping 10.210.0.1 lost 0 of 3. Cloud's server ls listed both Servers up, and doctor exited 0.

Negative control. I built 15b700c083, the round 3 head, and installed only its ployzd on machine-2. The running hash was 9f6ca914…. I repeated steps 2 and 3. At 00:16:52 wg show ployz-wg listed no peers and ip route show dev ployz-wg was empty. One minute after partition off, the store still lacked machine-1's row and ping 10.210.0.1 lost 2 of 2. So the round 3 daemon cut the mesh and could not heal it. I then restored the checkout daemon (63ff1297…). At 00:18:59 it had peered machine-1 again, the store had refilled, ping lost 0 of 3, and doctor exited 0.

Not run on Boat: the refused key swap and server rm of an approved Machine. The unit tests above cover them. A key swap injected on a live Machine would gossip to its peer and could win last-writer-wins there, which would break the lab Cluster. server rm needs a third Machine. No non-Anthropic reviewer ran this round.

Round 5 fixes

Round 5 closes one bug and one risk from the fifth review. Commit ff70fe4e31 sits on top of b246115e4b.

A deletion after a refused row is always seen. Take trusted peer A, installed. A refused key-swap row for A arrives. The rebuild keeps A and, in round 4, marked the filtered snapshot settled. Then server rm deletes A's row. The refused row was already filtered out, so the Machine snapshot did not change. MachineView published nothing, rebuild_when_changed skipped the identical snapshot, and nobody read the removal evidence again. A and its routes stayed.

The review suggested carrying removal evidence inside the snapshot. I took a simpler shape. A rebuild that keeps an installed peer that the plan does not list is not settled. peer_changes now reports settled, which is false while any unlisted peer is kept. rebuild_when_changed then rebuilds on the next 1-second tick and asks the store again. This removes the whole class of bug, not just this path. Retention is the only state that waits on outside evidence, and now it can never be cached as settled. The snapshot type is unchanged, so its ten readers and their tests are untouched. The cost is one small query per second while a trusted peer is kept on trust alone. That lasts while a store catches up, or while someone keeps a refused row for a trusted ID in the store.

The four rules hold:

• An untrusted key is never kept. keeps_unlisted_peer is unchanged.
• A trusted peer stays until its accepted row is deleted or re-keyed. An accepted row for its ID answers for it. So does a removal for its ID.
• A deletion is always seen. While a peer is kept, every tick re-reads the evidence.
• Enforcement off matches main. No peer is kept on Snapshot, so settled reads as before, and no removal query runs.

The removal query asks only about the peers in question. ReplicatedStore::removed_machines now takes IDs. It reads machines__crsql_pks for those IDs only and skips the query when the list is empty. LocalMachineRecord::unlisted_trusted_machines picks them: trusted Members whose key is installed on the device and whose ID no accepted row names. It returns nothing with enforcement off. In a steady Cluster the list is empty, so no query runs.

DESIGN choice touched: 6, Machine-local authority. The Machine's own record still decides which peers stay. It now re-checks its own store's evidence until that decision settles.

Filed, not fixed. #3951 (F=2, S=5). Under enforcement, Cloud's server rm answers not found when Cloud's entry Machine holds a refused row for that Server. It showed up during the proof below.

Focused checks

• cargo fmt --check: pass. cargo clippy -p ployzd --all-targets: no warnings.
• cargo nextest run -p ployzd --no-fail-fast: 1877 of 1878 pass. process_smoke::join_leaves_a_journal_line_and_records_the_restart refuses to enroll over this dev host's unlabelled ployz Docker network, the same failure as rounds 3 and 4.
• New test removing_a_machine_after_a_refused_key_swap_removes_its_peer. The RecreatedStore harness now gates each rebuild on the last settled snapshot, as rebuild_when_changed does. The steps are a refused key-swap row, a rebuild, then a deletion with nothing else changed. The test fails with round 4's settle rule, which I checked by forcing kept.is_empty() to true, and passes with the fix. The three round 4 tests still pass. a_deleted_machine_row_reads_as_removed now also checks that an ID nobody asked about is not returned.
• /deslop dropped one needless .map(drop), rewrote a return doc, and removed a redundant rebuild call from the new test. I reviewed the comments in-thread and kept only docs on new or changed items.

Runtime proof

I used verify-server on a three-Machine Boat run with local Ployz Cloud (--cloud). The manifest is core/.verify/20261107T004230-bc2aa900aa/cluster.json, at revision ff70fe4e31eb2d26c94fd090037ade6ed8e48456, the PR head. Evidence is in that run's evidence/: swap.py, which read and wrote a Machine's Corrosion API, and round5-machine-2.log. The run was torn down afterwards.

The observer is machine-2, not machine-1. Cloud's removal reads the Cluster through machine-1. With the refused row on machine-1, Cloud's server rm answered Server machine-3 was not found, and through machine-2 over SSH it answered No such Server. (#3951). So I restored machine-1's row and moved the injection to machine-2.

1. cli --device -- device enforce printed Trusting 3 Servers from the start, naming machine-1, machine-2 and machine-3. All three reported enforcement on.
2. I ran partition on machine-2. At 01:03:21 I replaced machine-3's row in machine-2's store with a copy that has a random key and no approval. At 01:03:33 machine-2's wg show still listed machine-3's trusted key XJ1WkH… (10.210.2.0/24) and machine-1's key, and its routes held both /24s. The swapped key had no peer.
3. I ran cli --device -- server rm machine-3 --confirm machine-3 through Cloud and approved its approval. It printed Removed Server machine-3. at 01:03:59. machine-1's store then had machine-3's key in machines__crsql_pks and no row.
4. At 01:05:33, before the partition lifted, machine-2 still held the refused row and machine-3's peer. The other rows in its store (machine-1, machine-2) had digest b02221c4808f04fe.
5. I ran partition off machine-2. By 01:06:56 the deletion had reached machine-2, and its other rows still had digest b02221c4808f04fe, so the filtered snapshot was unchanged. wg show ployz-wg listed only machine-1's peer oGEOSw… (10.210.0.0/24), and ip route show dev ployz-wg had only 10.210.0.0/24.
6. At 01:07:07 machine-1's wg show listed machine-2's peer U7rgjW… (10.210.1.0/24). ping both ways lost 0 of 3. Cloud's server ls listed machine-1 and machine-2 up.

Not run: a negative control on the round 4 daemon. That needs a second server rm. The new unit test fails on round 4's settle rule. No non-Anthropic reviewer ran this round.

Merge Danger

Door: one-way for consumed approvals. Machine IDs remain consumed after expiry. Preserve that state during rollback; deleting it reopens replay. The shared private consumption record gains joins, so older #3926 daemons cannot interpret that field. Approved capsules require the updated joiner; legacy unsigned joins remain available only where enforcement is off.

Blast Radius: enrollment and management transport. Full actions now need direct authenticated target connections. Existing Machines must enable enforcement individually; mixed Clusters retain the authority of every unenforced admitting Machine. This PR does not enable enforcement by default, pin the first Device for dashboard-only users, or implement signed Deploy workflows.

Round 2 renames the enforcement capability to ployz.machine.set-device-enforcement.v2 and requires a signed starting set to turn it on. trusted_machines now holds {id, public_key, subnet}. A round-1 record that stored full allocations still parses. Round 1 adds trusted_machines, membership_cluster and enrollment to the local Machine record, and an optional join_approval to Machine rows. Older daemons ignore the row field. They do not filter membership, so a mixed Cluster keeps the authority of every older Machine. Stored join results now hold the sealed reply, so the approval consumption record grows by one reply per admitted Machine.