Cancelling a deploy during a slow create leaves its Container running#3919
What happens
Cancelling a deploy while a Server is still creating a Container leaves that Container running, even though the Server is reachable.
- A deploy's create step takes longer than 10 s on the Server (for example
docker startis slow). ployz deployment cancel <n>while it runs. The Deployment endscancelled.- The run's cleanup (discard of the failed start) times out after 10 s, waiting behind the slow create, and only logs
couldn't remove a failed start's Container ... target Machine RPC timed out. - The Server's create runs detached from its RPC, so it finishes after the cancel. The Container starts and stays running as a Pending copy beside the live one.
The next successful deploy removes it (as docs/user/troubleshooting/deployments.md describes), so nothing is lost, but it uses memory and CPU meanwhile and ployz ps lists an extra copy of the Service.
Expected
A cancelled Deployment leaves no Container of its own running, or the deploy result names the copy it couldn't remove.
Repro
Seen on main 38666a9841 on a disposable 1-Server Cluster (MY-SERVER): a 1-replica redis Service, its docker start stalled about 7 minutes by a lab wrapper, cancel at +13 s; after the stall ended, ployz ps showed the cancelled Deployment's copy running, rotation Pending, beside the live copy.
Found while reproducing #2218 (whose fix shortens the discard timeout problem but doesn't cover a cancel).
F: 4 S: 3 F×S: 12