Deploys wait on route acks from a Server that never joined, and fail after 300 s#3936
The daemon builds the set of proxies that must confirm a deploy's routes in route_acks::required_ackers (core/crates/ployzd/src/route_acks.rs:194-208). Its rule is: "Only explicit Down evidence counts: a Machine with no evidence is unknown, not Down."
A machines row for a Server that never joined the Cluster has no daemon version and no membership state. Since it is never reported Down, it stays a required acker forever. Its ack never arrives, so every deploy in that Cluster waits ACK_WAIT (300 s) and fails with read routes failed. cause: proxy on MY-SERVER did not confirm, until someone runs ployz server forget <id> --confirm dead.
How such a row appears today: a refused server add of a Server that belongs to another Cluster (#3787). Cloud registers the Server through the Entry Machine before the CLI refuses. #3787's own fix may stop new rows being written, but rows left by older CLIs, and any other path that registers a Server that then never arrives, still block every deploy.
Reproduced on Boat at main ad6fa4e809 (2 Machines): with or without Sealed variables, deploys exit 1 after 305 s; after the forget they pass in 9-15 s.
Proposal: required_ackers skips a row that never arrived (no daemon version and no membership state; compare MachineObservation::unarrived). Sealed admission and Volume planning already skip such rows.
Things to weigh:
- This narrows what
ReadRoutesconfirms (stable promise rule 4); the PR has to argue it. - The same acker set feeds the LastIn guard (
machine/local_machine/rotation.rs:600-626) andboot_pass.rs:507. - During a mixed-version window, older owners still refuse at LastIn.
- It touches bet 8 (evidence).
Tests: ployzd/src/route_acks_tests.rs (machines = local plus a Registered row with no state entry, so the acker set excludes it), plus a LastIn case in machine_api/rotation_fleet_tests.rs.
F 2 × S 6 = 12 (Q184). Split from #3787 (design option B).