Sign up
View stargazersSign up

At 0 bytes free, a cached-image deploy fails with a raw containerd mount error and Corrosion goes down, with no 'disk is full' sentence#3911

opened7d agobyNNick
GitHub writeback is off. Issues and comments you change here stay in GitCafe. Sync settings

Found in the 0.3.17 soak, frozen sha 99a623f800b349f71a37af96cb91dce2658b7a63 (ployz: hit local 5d8fd3c9, ployzd: hit local 9cf0b05b, sdk: hit local 348922e9), local Cloud, four Boat Servers (Ubuntu 24.04).

What happened

On MY-SERVER, root was filled to 0 bytes free, root's reserve included (fallocate to the free size, then dd until No space left on device).

  1. An uncached image deploy fails as documented: full: Server MY-SERVER's disk is full: 0 MB free under /var/lib/containerd. Good.
  2. A deploy of a cached image (an env change to a running Service) fails differently: web: create Container failed. cause: ... mount callback failed on /tmp/containerd-mount…: mkdir …/dev: no space left on device. It doesn't name the Server or say its disk is full, unlike the pull path.
  3. During that deploy, Corrosion on MY-SERVER exits (code 1, then three restarts exiting 101). The deploy adds ! Couldn't check ingress: Corrosion HTTP request failed: … Connection refused (os error 111) (on Server MY-SERVER), and ployz ps fails with error: Corrosion HTTP request failed / cause: Connection refused. Nothing points at the full disk.
  4. ployzd restarts Corrosion (WARN Corrosion wasn't running; ployzd started it again). It stays up once the failed deploy's cleanup frees about 58 MB. After freeing the disk, ployz deploy succeeds on all four Servers and every row agrees. No data was lost that I could see.

ployz server ls does warn Server MY-SERVER's disk is nearly full: 0 MB free for Docker, which helps.

Expected

docs/user/troubleshooting/deployments.md "My server's disk is full" covers only the pull failure. A create failure from ENOSPC, and Corrosion going down on a full disk, should say Server MY-SERVER's disk is full too, rather than a raw containerd mount error and a Corrosion connection refusal.

Reproduces on v0.3.16?

Not run.

F: 2 S: 4 F×S: 8

Blocks the 0.3.17 cut: no. It needs a disk with 0 bytes free, root's reserve included, and it recovers once space is freed.