Cancelling a deploy during a slow create leaves its Container running#3919

opened7d agobyNNick
GitHub writeback is off. Issues and comments you change here stay in GitCafe. Sync settings

What happens

Cancelling a deploy while a Server is still creating a Container leaves that Container running, even though the Server is reachable.

  1. A deploy's create step takes longer than 10 s on the Server (for example docker start is slow).
  2. ployz deployment cancel <n> while it runs. The Deployment ends cancelled.
  3. The run's cleanup (discard of the failed start) times out after 10 s, waiting behind the slow create, and only logs couldn't remove a failed start's Container ... target Machine RPC timed out.
  4. The Server's create runs detached from its RPC, so it finishes after the cancel. The Container starts and stays running as a Pending copy beside the live one.

The next successful deploy removes it (as docs/user/troubleshooting/deployments.md describes), so nothing is lost, but it uses memory and CPU meanwhile and ployz ps lists an extra copy of the Service.

Expected

A cancelled Deployment leaves no Container of its own running, or the deploy result names the copy it couldn't remove.

Repro

Seen on main 38666a9841 on a disposable 1-Server Cluster (MY-SERVER): a 1-replica redis Service, its docker start stalled about 7 minutes by a lab wrapper, cancel at +13 s; after the stall ended, ployz ps showed the cancelled Deployment's copy running, rotation Pending, beside the live copy.

Found while reproducing #2218 (whose fix shortens the discard timeout problem but doesn't cover a cancel).

F: 4 S: 3 F×S: 12