Start a Server's ingress once port 80 is free#1683

_{P2 Badge} Avoid retrying Docker while holding the publication guard

When an ingress retry reaches start_stranded, this scan is still holding the MachinePublicationGuard acquired above. A Docker start can block while the daemon performs network/runtime setup, so during that interval machine, container, and volume publications all wait on the same guard; unrelated deploy observation barriers can consequently time out after 30 seconds. Arrange the retry so the potentially long Docker call does not monopolize the publication lock while preserving the participation recheck.

Useful? React with 👍 / 👎.

Fixed in f8d63c8ad. The scan now decides under the guard, publishes, drops the guard, then starts. Moving the start out is safe because a reset that takes the guard meanwhile removes every managed container. The started container's event triggers the next scan, which publishes it.

_{P2 Badge} Serialize the ingress retry with policy updates

When server set --accepts-ingress=false commits after this local snapshot is read but before start_stranded runs, the scan can still start the ingress after the update returns. update_admitted mutates the RecordOwner without acquiring machine_publication (machine/local_machine.rs:636-650), so the publication guard does not make this eligibility check current through the start, contrary to the invariant in core/DESIGN.md:463-465. Serialize the check/start with local policy mutations or perform an equivalent fresh admission check immediately before starting.

AGENTS.md reference: AGENTS.md:L5-L5

Useful? React with 👍 / 👎.

Narrowed in 01abf3c3c. The scan now reads the record again right before the start, after publication. A policy change that commits during publication now holds. That leaves only the Docker start call itself as a window. A loss there leaves a running ingress on a Machine that no longer accepts ingress. That is already what server set --accepts-ingress=false does to a running ingress today: it leaves it in place, which predates this PR and is tracked as a follow-up. So full serialization with update_admitted isn't worth a lock across Docker.

_{P2 Badge} Preserve rollback for unrecognized active runtime states

When a stop-first replacement was planned while the old container was active but the execution-time inspect returns ContainerRuntimeObservation::Unknown—for example, after Docker adds a new active status—this path now stops the container unconditionally but records old_stopped as false. If creating or starting the replacement then fails, compensation never restarts the old container, leaving the service down even though this deploy stopped it. Either avoid stopping an unrecognized state or conservatively treat a successful stop as requiring rollback.

Useful? React with 👍 / 👎.

Fixed in 69c99bad6. The hand-back is now an exhaustive match on the observed runtime. Unrecognized runtimes count as active and are restarted, so are Running, Paused and Restarting. Created, Exited and Dead are handed back only for the ingress, and Removing is not handed back. The new test stop_first_hands_back_an_unrecognized_old_container_when_its_create_fails failed before the fix.

_{P2 Badge} Avoid holding the global retry mutex through Docker I/O

When a stranded ingress start is slow or hangs during inspection, network attachment, or startup, this guard remains held across every Docker await in start_stranded. Because the stranded mutex is shared by the entire ContainerRuntime, ordinary starts, stops, and removal cleanup for unrelated containers also wait for it, so one ingress retry can stall deployments and user-requested lifecycle operations across the Machine even though the publication guard is now released. Limit this serialization to the candidate container or record the held state before performing Docker I/O.

Useful? React with 👍 / 👎.

Fixed in 7194a40e2. Each stranded container now gets its own gate, which the start holds through its Docker calls. The shared lock is held only for brief bookkeeping. A stop on the same container takes the gate after recording its hold, so it still waits for the in-flight start and then holds. Other containers never touch the gate. A new test shows that forget on another container finishes while a stranded start is blocked.

_{P2 Badge} Serialize retries with concurrent ingress deployment

The candidate comes from the pre-publication observed snapshot, but after the guard is dropped there is no fresh inventory check or serialization with an explicit start of a different ingress ID. During an upgrade from an inactive failed ingress, planning creates the replacement without marking the old container stopping; this retry can therefore start the old ingress first, take port 80, and make the replacement's start fail. Revalidate that this is still the newest ingress, or serialize automatic and deployment ingress starts, to preserve the newly documented newest-local-ingress invariant.

AGENTS.md reference: AGENTS.md:L5-L5

Useful? React with 👍 / 👎.

Fixed in 988f52afc. The scan reads a count of container creates before its inventory, and the retry rechecks it just before the start call. If a Deploy created a replacement after the scan began, the retry backs off and the next scan decides. The test is a_container_created_since_the_scan_voids_the_start, and it fails without the check. A create that lands inside the Docker call itself can still lose the race. That's the same narrow window I accepted for the policy race. Losing it fails that Deploy while the old ingress keeps serving, and the next Deploy stops the old ingress first. I chose not to add a lock shared by deployment starts and the retry.

_{P2 Badge} Move stranded starts off the observation loop

Although the publication guard is dropped, this still awaits all inspect, network-connect, and start Docker I/O inside watch_docker_events' single sync_observations callback. If Docker takes more than 30 seconds to answer this retry, no subsequent container events or rescans are consumed, so unrelated deploys waiting on the 30-second barrier in cluster/container_observations.rs can time out because their state is never published. Dispatch the retry outside the observation loop while retaining the per-container gate.

Useful? React with 👍 / 👎.

Fixed in 988f52afc. The scan now spawns the retry as its own task, so later events and rescans never wait on it. The per-container gate is taken with try_lock, so a scan skips while a retry is in flight and retries can't pile up. A stop on the same container still waits for the in-flight start, then holds.

_{P2 Badge} Block retries while a create is already in flight

When a create has incremented creates but is still awaiting Docker, a scan can read that new value, list the old inventory before Docker makes the new container visible, and then see the same value in start_stranded. It can consequently start the old failed ingress while a new ingress is being created, causing the new ingress's explicit start to lose port 80. Fresh evidence beyond the earlier concurrency report is that the added counter is bumped only before the Docker await, so it cannot identify a create already in flight when the scan begins; track in-flight creates or serialize this decision with endpoint_creates to preserve the invariant in core/DESIGN.md:465-466.

AGENTS.md reference: AGENTS.md:L5-L5

Useful? React with 👍 / 👎.

Fixed in 8c9649b2c. Creates now count both started and finished, and a guard counts a create as finished on drop. The scan reads the finished count, and the retry backs off unless the started count still equals it. So a create that was running when the scan began voids the retry, even if it finishes before the retry runs. The test a_create_in_flight_when_the_scan_began_voids_the_start covers all three cases: a create still running, one that finished after the scan began, and a later scan that does start. It fails if the scan reads the started count.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Coordinate the create check with the ingress start.

If a newer ingress create begins after self.docker.creates() != creates is checked, start_inspected can still start the old ingress first. The old ingress can take port 80, cause the newer ingress start to fail, and remain running on later scans. Coordinate the retry with ingress creation or replacement intent so a newer create cannot pass this check while the old start proceeds.

✅ Addressed in commit 8c9649b

Leaving this as is, deliberately. Since 8c9649b2c the retry backs off for any create that started before or during its scan. What remains is a create that starts inside the few milliseconds between that check and Docker's start call. If the retry loses that race, the newer ingress fails to start and the Deploy that created it fails and says so. The old ingress keeps serving port 80, so the Machine still has ingress. Running the Deploy again stops the old ingress before starting the new one. Closing that window needs a lock shared by Deploy-driven ingress starts and the retry. That's a cross-module lock for a millisecond race whose worst case is one failed Deploy you can retry, and I chose not to add it.

@nick-potts Understood. Thanks for explaining the trade-off. The remaining race can cause a reported Deploy failure, but the old ingress keeps serving and the Deploy can be retried. I accept your decision not to add a cross-module lock for this window.

You are interacting with an AI system.

_{P2 Badge} Serialize the create check through the stranded start

When a new ingress create begins immediately after this predicate evaluates, created_nothing_since returns true and start_inspected can still await network attachment before starting the old ingress; meanwhile the new container can be created, after which the old ingress may seize port 80 and make the deployment's explicit start fail. Fresh evidence beyond the earlier create-race report is that the counter is only sampled here and create_container is not excluded until the Docker start completes, so the documented “no container was created since that scan began” guarantee still has a TOCTOU window. Hold the create gate across this decision/start or otherwise serialize the two operations.

AGENTS.md reference: AGENTS.md:L5-L5

Useful? React with 👍 / 👎.

I'm not changing this. Making the create wait for the retry's start wouldn't close the race. A create that begins after the check would wait until the old ingress had started, then create the new one. The new ingress's start would still find port 80 taken. What collides is the old start and the new start, not the create. Closing it fully would need Deploy to tell the daemon it means to replace the ingress, before it plans the replacement. If the retry loses this millisecond window, one Deploy fails and says so. The old ingress keeps serving, and rerunning the Deploy stops it first, so the Machine never loses ingress. The same reasoning is on CodeRabbit's matching thread.