Upgrade only along the Release Channel the customer signed (F2)#4086

open
main
t3/sec04-f2-signed-channel
opened1d agobyNNick
View on GitHubGitHub import enabledPR last imported 22h ago

F2 of the 0.4 security program, spec in docs/agents/plans/security-0.4/f-plan.md ("The upgrade channel", F2, Appendix A gaps 1 and 2).

Merge gate. F2 merges only after the operator applies the release-signing protections and removes Administration from the getployz Apps. The parent thread holds the merge. Don't queue this PR.

What changes

A person approves their Cluster's Release Channel once, from a Device or with one passkey prompt. Cloud keeps that signed approval and sends it with every upgrade. A Machine that enforces device approval admits an Operator upgrade only along the approved channel, as the Machine itself resolves it.

mermaid
1sequenceDiagram
2 participant P as Person (Device or passkey)
3 participant C as Ployz Cloud
4 participant M as Machine (enforcing)
5 P->>C: Signed<UpgradeChannel>{cluster, channel, generation} (Rule purpose)
6 C->>C: keep it if newer than the kept one
7 C->>M: RequestMachineUpgrade{release, channel: approval}
8 M->>M: verify signer, Cluster, generation ≥ mark (synced)
9 M->>M: release == channel, or == what the channel resolves to now
10 M-->>C: Accepted, or PermissionDenied naming ployz.machine.upgrade.channel.v1
diff
1 request_upgrade
2 replay(attempt id)
3 admission lock
4 require_management_access
5+ release_choice Full → AnyRelease; Operator, enforcement off → ChannelResolved
6+ verify UpgradeChannel Rule purpose, trusted Device or passkey's own prompt, this Cluster
7+ mark_upgrade_channel generation ≥ mark (equal only for the same channel), synced
8+ → Signed(channel)
9 try_installation
10 request_locked
11 admit(choice)
12+ Signed(c): release is c, or the exact version c resolves to now, else OffApprovedChannel
  • Engine. ployz_core::authority::UpgradeChannel under Purpose::Rule. RequestMachineUpgradeRequest.channel is optional and skipped when absent. The new Always capability is ployz.machine.upgrade.channel.v1. The coverage table moves RequestMachineUpgrade to Signed(&[Upgrade]). Each Machine keeps the newest approval it accepted per Cluster in its approval record.
  • CLI. ployz server upgrade-policy set --channel is now a compiled verb. On enforcing Servers it signs with this Device's key, using a generation past both the kept one and the clock, and posts it to POST /api/cli/upgrade-channel. It stops when it can't tell whether the Servers enforce.
  • Cloud.
    • organization_server_upgrades.channel_approval holds the approval (migration server_upgrade_channel_approval). An older or equal generation is refused in the upsert itself.
    • The approval is sent only with a request for the channel it names.
    • A Server's forbidden refusal ends the attempt as failed with its message, not as unreachable.
    • On enforcing Servers, the dashboard asks for the passkey on a Releases change, before Upgrade, and before turning on Upgrade automatically, until a matching approval is kept.
  • Docs. docs/user/servers/manage-servers.md covers "Choose Stable or Beta releases" and the new "Upgrades on servers that enforce device approval".

Where this differs from the plan text:

  • The flag stays --channel; the outcome text says --releases.
  • The signed approval goes through new CLI routes, because published commands can't carry an object argument. A compiled verb overrides the published server upgrade-policy set.
  • Cloud sends the approval to every Machine, since an older daemon ignores the field.
  • The off-channel refusal is a new OffApprovedChannel. ReleaseChoice::Signed carries ReleaseChannel, not MachineRelease.
  • authorization_tests.rs drops LIVE_ONLY. Without an approval, the upgrade cases are refused before the standard-paths check, so they run in-process.

Bets touched

  • Stable promise. Additive only: an optional field, a new capability, and an older daemon that ignores it. Clusters that don't enforce are unchanged.
  • Bet 6, Machine-local authority. The Machine verifies the approval against its own trusted keys and Sealing key, resolves the channel itself, and keeps its generation mark across restarts.
  • Bet 7, Cloud drives, never owns. Cloud still picks when and which Servers upgrade. It carries the approval but can't make one.
  • Bet 8, evidence over claims. Every refusal names ployz.machine.upgrade.channel.v1, and Cloud records the Server's own refusal as the attempt's error.
  • Bet 3, bounded imperative commands. upgrade-policy set refuses before it signs when the outcome is unknown.

Residuals

Each is named once. Under the Cloud security bar:

  • Cloud picks when each Machine upgrades, in what order, and whether at all.
  • Cloud can keep sending an older approval the person also signed to a Machine that never saw the newer one. Withholding the newer one is a drop or a reorder.
  • Resending an existing attempt returns its earlier outcome without checking an approval again. It starts nothing new.
  • The passkey prompt doesn't show the body, so a compromised dashboard could word the prompt for one channel and get the other signed. That is a mislabel of an action the person approved.
  • The CLI's enforcement probe reads through Cloud. A lying probe makes the CLI sign nothing or sign for the wrong Cluster, which is a drop.
  • A Full downgrade to a daemon that predates the generation mark loses the mark.
  • The Bound routes and the other items in f-plan "What remains".
  • This holds only while Cloud's operator can't publish a signed release or move a channel pointer. Hence the merge gate above.

Proof

Unit and contract

  • ployz-core:
    • authority::upgrade pins the golden body for the Device and passkey forms, byte-equal with the dashboard's channelApprovalBody.
    • A Device and a passkey's own prompt both approve.
    • Another Cluster, an unknown key, and a browser key under a valid session grant are each refused.
    • A lower generation, or an equal one for another channel, is refused.
  • ployzd:
    • installer::upgrade: under Signed(Stable), beta and the next beta's exact version are refused, and Stable's exact version is admitted.
    • local_machine::upgrade: enforcing with no approval is refused. Generation 2 then 1 is refused, also after a reload. An equal generation naming another channel is refused, and so are another Cluster's approval and a missing Sealing key. Enforcement off ignores the field.
    • The coverage matrix and capability tests cover the new capability.
  • The in-process RPC matrix can't send a valid approval with an off-channel release: a temp data dir fails the standard-paths check before admit. Live lanes 3 and 4 cover that path.
  • Dashboard:
    • Storing the approval and sending it with the request and every resend.
    • No approval along a channel changed without one, and a refused attempt recorded with the Server's message.
    • A stale approval refused, and two first approvals racing. The race test fails on the earlier code.
    • The prompt fires only when the Servers enforce and nobody approved this channel for this Cluster.
    • The CLI routes GET/POST upgrade-channel.
  • verify-upgrade-attempt model: 24 of 24 pass. The new check sits before try_installation and changes no receipt, marker, lock or settling reader, so the model needs no change.

Live lanes

verify-cluster --cloud: 3 Machines on this head, with a seeded Cloud and the lab worker. The release channel was a test one (v0/stable = 0.3.17, v0/beta = 0.3.18-beta.1). A compromised Cloud was played by a harness that holds the Organization Token and Cloud's database. It reads the kept approval and sends RequestMachineUpgrade as an Operator through ployz debug volume-rpc (a verify-faults CLI). A CDP virtual authenticator stood in for the passkey.

LaneResult
1. Enforcement offA Cloud upgrade is admitted as before. The harness's beta with no approval is accepted (target 0.3.18-beta.1).
2. Enforcing, then approve with the CLIWith no approval, refused forbidden, naming ployz.machine.upgrade.channel.v1. ployz server upgrade-policy set --channel stable signs and keeps one approval. Upgrade all through Cloud then admits all 3. Each journal logs admitting an upgrade along the approved Release Channel channel=stable generation=…282.
3. Off-channel nameThe harness's beta with the kept Stable approval is refused: beta isn't on the approved stable Release Channel, with next: ployz server upgrade-policy set --channel beta from an approved device. The version stays 0.3.17.
4. Off-channel exact version0.3.18-beta.1 is refused. The exact 0.3.17 is admitted.
5. PasskeyOne prompt switches to Beta (signer: passkey, one assertion), and machine-1 admits it. Switching back to Stable signs a newer generation, and machines 1 and 2 accept it. Replaying the saved Beta approval to machine-1 is then refused for its generation. machine-3 never saw the newer one and admits the replay: the residual above.
6. AutomaticTurning on Upgrade automatically with a matching approval asks for no prompt and changes no generation. A real rollout could not run: Cloud's channel is the public one, at 0.3.17, so no Server is behind.
7. Kill between the mark and the receiptRequests cut off at 0.05, 0.2 and 0.4 s, then resent under the same attempt id. Each resend is accepted or replayed, and the attempt ends succeeded.
8. Docsmanage-servers.md matches what lanes 2 and 5 printed.
Unsigned changeSetting Cloud's channel to beta in the database without an approval: Cloud sends none, the Server refuses with no valid Release Channel approval for this Cluster, and Cloud records the attempt failed with that message.

The dashboard, at the end of the run: one passkey prompt to Beta, one back to Stable (65 s, video). Afterwards machine-1 admits stable and refuses beta with the kept approval.

StableBeta, after the promptAutomatic on

Found live and fixed in ec23ead117: a Server's refusal reaches Cloud as RPC code forbidden, not permission_denied. Before the fix, the refusal fell through as a step error, and the attempt didn't end with the Server's reason.

Could not run:

  • A real upgrade to a new release. The release public key is fixed outside tests, so a lab-signed release can't install, and real releases are Nick's. Admission is what changed, and lanes 2 to 7 cover it.
  • The dashboard's "Upgrade failed" line, and a screenshot of Upgrade on a Server that is behind. Neither shows unless a Server is behind, which needs a newer release on Cloud's channel.
  • Lane 1 on a trunk daemon. Perf below ran trunk on machine-3 and showed it ignores the new field.

Perf

The new check verifies one signature and fsyncs the mark only when it changes. Verifying a statement takes 39 µs in release.

End to end, a Stable request with the kept approval through Cloud, 10 interleaved runs per Machine, median:

RoundTrunk daemon (machine-3, doesn't enforce)This head, machine-1 (enforcing)This head, machine-2 (enforcing)
1564.5 ms613 msnot run
2540 ms557.5 ms493 ms

Both rounds are within 50 ms of trunk. The two head Machines differ by more than head and trunk do, so most of the gap is per-Machine noise. Trunk took the extra channel field without trouble.

Merge Danger

Door: two-way for code. The migration adds one nullable column.

Blast Radius: enforcing Clusters. On a Cluster that enforces device approval, upgrades through Cloud are refused until someone approves a channel. That is intended, and the dashboard asks before the first one. Clusters that don't enforce are unchanged.

Checks

1 failing · 15 successful · 6 skipped
Some checks failed

Failing1

169fc60
Synced from GitHub

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

External image Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 659f61e5-f3fe-452f-8c30-17e956e91aae

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
Pre-merge checks: Inconclusive 2

Failed Failed checks: 2 inconclusive

Check nameStatusExplanationResolution
Title checkPendingThis assessment has not completed. The check will be retried on the next pre-merge check run.
Description checkPendingThis assessment has not completed. The check will be retried on the next pre-merge check run.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Synced from GitHub

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian idGitGuardian statusSecretCommitFilename
39918940TriggeredGeneric High Entropy Secret5352d3e8bf04305696149613c2d7f7a5726a0142core/crates/ployz-core/src/authority/private_access_tests.rsView secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.