Device-signed CLI restart under a Full-only enforcement switch#3926

Why

Hosted Ployz Cloud must be clearly safer than self-hosting. Someone with root on np1 must not be able to change anything on a customer's Server unless the customer's own Device key approved it. This is the first end-to-end slice of that: ployz service restart and ployz service stop sign each Container's steps with the Device key, and a Machine under Device enforcement acts on an Operator caller's restart only with that approval.

Bets this touches, and the choice each changed:

• Stable promise. SetDeviceEnforcement is a new catalog row (ployz.machine.set-device-enforcement.v1, Full scope) and the approval on EnsureContainer is a new capability (ployz.restart.device-approval.v2). No existing route changed meaning. A daemon predating this slice ignores the approval field and restarts as before, and ployz device enforce gates on the contract and lists an older Server as missing. Both rows are frozen in capabilities.rs.
• 6. Machine-local authority. The Machine verifies the signature against its own Device list, checks every bound field against the request in hand, and keeps its own record of consumed steps. Cloud carries the approval but cannot mint, alter or reuse one.
• 7. Cloud drives, never owns. Enforcement is off by default and only a Full caller (a Device) can turn it on or off. Under it, Cloud's own restart, stop, start and deploy of a Service are refused with a refusal naming the capability, until each carries its own approval. The CLI restart and stop now carry those approvals.
• 9. Lean on proven primitives. Ed25519 through ed25519-dalek with strict verification, over an RFC 8785-style canonical JSON body under a purpose prefix. No new crypto.
• 10. Client-first, daemon when forced. The CLI mints and signs. The daemon holds only what the client cannot be trusted with: the switch, the verification and the consumption record.

Enforcement must not be relied on until the separate PR that stops headerless mesh TCP callers receiving Full scope lands.

Summary

Under enforcement, every Operator lifecycle request on an existing Container needs approval. Restart and stop approvals never authorize Out or Gone. Running(New) is refused entirely, including adoption, until deploy authorization lands. Enforcement off and Full SSH calls retain their behavior. An unavailable Device key sends an unsigned request; only a Machine refusal produces the short Device note.

Each approval names one Container ID. Restart signing happens inside each rolling batch, immediately before that copy's steps. The Machine enforces a maximum signed lifetime of 15 minutes and two minutes of tolerance on issue time and expiry. It parses sorted, distinct Machines and a fixed 64-byte signature.

Initial slice evidence

Focused checks on this head:

Daemon behaviour tests (local_tests.rs, mod device_enforcement):

• enforcement off leaves an unsigned Operator restart as it was
• an Operator restart passes only with an approved Device's approval of it: unsigned, forged by a non-Device key, tampered body, other Service, other Machines and expired each refuse with ployz.restart.device-approval.v1 in the details and no /stop reaching Docker; the valid one stops the Container
• a used approval runs each step once even after the daemon restarts: Stop then Start pass once, replays refuse, a service rebuilt over the same record refuses both replays, a fresh approval passes
• a Full caller restarts without an approval
• the switch answers its new state and the record keeps it

CLI test (exec_tests/user_commands.rs): a Machine that refuses the Hold answered before acting, so service restart and service stop give the refusal alone, send no bring-in, and leave the copy In.

Boat proof (scripts/verify-cluster up --cloud, one Machine, daemon checkout at revision 505d2d999b, this PR's head, built with --features ployz/verify-faults,ployzd/verify-faults so ployz debug volume-rpc can send a raw EnsureContainer through Cloud as the Device CLI's Operator slot). Project verify, Service web on nginx:1.29-alpine, Container 685518d2f7b7…. cli below is the Organization Token through Cloud, cli --device the seeded owner's CLI with its approved Device. Each case checks docker inspect on the Machine before and after.

Enforcement off, the Organization Token restart passes as today:

The Device turns enforcement on, and the unsigned Operator restart is refused naming the capability, Container untouched:

The Device-signed CLI restart restarts the Container and the Machine records the operation:

Hand-minted approvals (a Python signer over the same canonical body; the genuine one passing below proves it matches the Rust encoding) sent as raw EnsureContainer wants through Cloud:

Enforcement off again, the Organization Token restart passes, and doctor is clean afterwards:

One observation from the proof: a step is recorded before the Machine acts, so a Stop the workflow then refuses (sent before its Hold) spends that step and the next attempt needs a new approval. That is the fail-closed choice the record makes on purpose; the CLI always sends Hold first.

Two independent Boat runs at the same revision, each on its own worktree and --cloud with two Machines, covered what the run above did not:

• Two Machines. Two replicas, one per Machine. Enforcement on both; the Organization Token restart, stop, start and deploy refused on both naming the capability, the Device-signed restart restarted both and each Machine recorded the same operation with its own Container; the Organization Token's device enforce --off refused as needing Full access; --connect ssh:// restarted both with no record written; enforcement off restored every Cloud path. A first deploy of a brand-new Service is also refused, at the Start of its Container.
• Older daemon. Both Machines on release 0.3.16: device enforce names both Servers and exits partial; the Device-signed restart restarts both and the release daemon logs nothing about the approval. Machine-1 moved to the checkout daemon: enforcement is off after the upgrade, turning it on gates only machine-1, the signed restart restarts both and only machine-1 writes restart-approvals.json.

Both runs found the same wrong sentence: a refused Operator restart ended with "It is left Held and not routed" while the copy stayed In and serving. The CLI treated the refusal like a lost Hold and tried to bring the copy back In. Fixed in the last commit, with the CLI test above. The two runs also found that device enforce reports a too-old daemon as "did not answer", the same as the pre-existing device approve and device ls (#3928, F 4 × S 3), and that a refused deploy's retry: hint repeats a failing command (#3929, F 3 × S 2). Both are below the fix bar and left alone. The doc now gives the real reason deploys are refused.

Round 1 fixes

The lifecycle gate now classifies Held, Out, Stopped, Running(Existing) and Gone exhaustively. Restart and stop sign each Container separately. A captured approval cannot affect another copy. New-creation requests, including adoption of an exited same-slot Container, cannot use restart authorization.

The consumption log persists expiry_time_floor and uses max(now, expiry_time_floor) for time checks and pruning, including expiry tolerance. It keeps the latest expiry seen for a reused operation ID, invalidates its cache after write errors, and is owned once by the shared RecordOwner. Refusal reasons travel as error causes. The OperationId comment no longer claims to distinguish a retry from a replay.

Focused checks on the fix source:

All listed checks passed on head 20ae8e51465e8e3830b5a57bae369e0e0297c6d5.

Regression coverage includes unsigned Out then Gone on a serving Container, Running(New) adoption of an exited Container, pruning followed by clock rollback and log reopen, a malformed Device key still sending an unsigned restart, a captured approval rejected on a second copy, Operator refusal at LocalMachine::set_device_enforcement, bounded lifetime and skew, malformed signatures and Machines, reused operation expiry, and a write that lands before returning an error. Deslop completed. Two Codex no-comments passes removed internal narration and replaced the new indexing suppression with checked access.

Boat proof used verify-server, --cloud, one Boat Machine and two nginx copies. Evidence is retained in core/.verify/20261106T125101-4518213567/evidence/commands.jsonl in the implementation checkout. The checkout daemon and CLI ran the final production source. The final manifest names head 20ae8e51465e8e3830b5a57bae369e0e0297c6d5; doctor verified daemon SHA-256 3d634ed18bf500071c839cb1a40d895b29c52152c8025a0210432727b39429f2 and CLI SHA-256 f465a737b0f18b8b8ec6f8c6821c7b6c1d8f957b834a5051bbb22d7b11d5c174. Enforcement was on for the refusal and signed cases.

The unsigned and signed paths were checked again after updating the checkout binaries. The interim lab log's time-floor field was renamed while preserving all consumption records; the regression tests separately prove reading this PR's original map-only log without changing its contents.

Remaining boundaries are explicit. Cloud can still choose which Cluster the CLI connects to, so the Device can sign for a Cluster the user did not intend. Displaying Cluster and Server targets before signing and pinning the Cluster per context remain follow-ups. UpdateContainerLimits, Leave, Forget, RemoveVolume and SetVolumeSize are still Operator-scoped without this Device gate. They are later slices; this PR does not protect every way an Operator can stop a Service.

Round 2 fixes

A captured restart approval previously admitted Out { allow_empty: true } and then Gone. The new daemon behavior test reproduced the first effect on the old head as Ensured with rotation Out; the same test now passes with both requests refused and the Container still running.

WorkflowKind now distinguishes Restart (Hold, Stop, Start) from Stop (Hold, Stop). Neither permits Out or Gone. The CLI signs the requested kind. Removal has no signed Operator workflow in this slice and remains refused under enforcement; it cannot borrow either approval. The changed contract advertises ployz.restart.device-approval.v2, replacing v1. An older parser refuses the new Stop kind rather than interpreting it as Restart. Historical evidence above names the earlier v1 contract.

Admission returns the want pinned to the verified ContainerId, and the RPC handler executes that returned want. An empty target under enforcement refuses before execution. Regression tests replace A after admission and verify B stays running, refuse A's approval once its slot holds B, and refuse an empty slot. With enforcement off, admission returns the original want unchanged.

Bets changed by these fixes are the Stable promise (v2 names the changed contract), 5. Two-tier identity (the slot selects a Container but never substitutes for its ID), 6. Machine-local authority (the Machine authorizes only the signed steps and verified ID), and 10. Client-first, daemon when forced (the CLI chooses and signs the kind; the Machine enforces it).

Focused checks on e58f75f583d0374270995f2164c87041b7b7abb9 all passed:

Deslop completed. The requested no-comments review ran on Codex gpt-6.1-sol over the fix diff, with zero deletions, flags or changed suppressions. Independent security review remains with the coordinating task. This PR remains draft.

Boat proof used verify-server, one Machine, --cloud, the checkout CLI and daemon with verify-faults, and an exact checkout SDK. The manifest revision equals that head. Evidence remains in core/.verify/20261106T133235-c2ae0ace3f/evidence/, including commands.jsonl, round-2-summary.md, the signed envelope, focused check logs and exact-sdk.log.

On postgres:18 Container 9a8f945bed40…, with an image-created anonymous Volume and a sentinel file:

• Enforcement on, the same approved Device's signed restart envelope relayed through Cloud as an Operator was refused for Out and Gone with v2 and cause the approval names a workflow this request does not match. Docker remained running with StartedAt 2026-11-06T13:39:51.727551008Z; the same anonymous Volume and r2-volume-survives sentinel survived.
• Device-signed service restart db passed, with StartedAt 13:40:44.797224823Z. Device-signed service stop db passed; Docker reported exited with FinishedAt 13:40:50.985488673Z.
• Enforcement off, unsigned Cloud start and restart passed. Docker returned running with StartedAt 13:41:20.231440923Z, the same Volume and sentinel. An unsigned restart also passed before enabling enforcement. Doctor passed before and after.

The slot-race cases were exercised by the local daemon behavior tests, not forced on Boat. No requested Boat case was skipped. Doctor verified daemon SHA-256 308ec5e3e801dc54fb48f55ee9b1e54f7fcec36c17ac18a2fc5d59633a3ce81d and CLI SHA-256 f8da0b74da8fb55f7ca0215f22fb13ad10809263c0ad426f1f3420b901f96bde.

Deleting the consumption log locally resets its history; doing that requires root on the customer's Machine, which already holds Full, and is outside this PR's threat boundary.

Merge Danger

Door: two-way

Enforcement is off by default and turning it off restores ordinary Operator lifecycle behavior. The signed approval format belongs to this unshipped first slice. Machine records and RPC additions remain additive. The log reads the previous map-only form and adds a persisted time floor.

Blast Radius: lifecycle

On enforcing Machines, unsigned Operator lifecycle changes and every Running(New) request are refused. Approved Device CLI restart and stop work per Container. Headerless mesh TCP Full access and the later Operator mutation slices remain outside this PR.