perf(verify): bake scenario workload images into the guest image#1802

merged1
main
verify-fast/m2-bake-scenario-images
openedOct 13, 2026byNNick
View on GitHubGitHub import enabledPR last imported 21d ago

Scenario checks deploy postgres:16-alpine and alpine:3.20, and each Machine pulled them from Docker Hub during the check. Postgres alone took 44 to 55 s per Machine. The guest image now bakes them in, along with the other scenario images: nginx:1.29-alpine, nginx:1.27-alpine and traefik/whoami:v1.10.3 (the last two come from the dashboard merge-removals fixture). Services use PullPolicy::Missing, so a baked image is not pulled again.

The image key hashes prepare-guest.sh, so the bake invalidates on its own. Keys move from 00256bfc9619 to ccd98fe396d4 for the stable release, and from 5562763db7ae to 7683089b822b for 0.3.0-beta.1. The first run after merge rebuilds the image once.

Measured on one real Incus run (--daemon 0.3.0-beta.1, n=1, loaded host). The guest lists every baked image at boot. A postgres deploy that had to pull (image removed first) took 47.96 s. The same deploy with the baked image took 7.18 s, and Docker logged no pull events during it.

Cost. The image grows from 1035 MiB to 1329 MiB on the dir pool (about 294 MiB). The bake ran about 516 s on the loaded host, once per key. Launch time of the bigger image against the old one was not measured base against head. The only number is the head launch, 26.9 s under load.

The verify-server skill now lists the baked workload images.

Checks

15 successful ¡ 1 skipped
All checks passed
f995dd7
Synced from GitHub

Review in Change Stack →

✨ Finishing Touches: 💡 1
đŸ› ī¸ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
đŸ§Ē Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix ¡ Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Synced from GitHub

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

ReviewStatusCommitReview trigger
📝 Code Review✅ Completed 2026-10-13T23:26:08.997670Za45f4a5Draft marked ready
â„šī¸ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Added 1 commit

  • perf(verify): bake scenario workload images into the guest imagef995dd7
Cchatgpt-codex-connectorBotreviewed ¡ 1 inline threadOct 13, 2026
Synced from GitHub

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a45f4a5f7c

â„šī¸ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.agents/skills/verify-server/SKILL.mdline 34Outdated ¡ a45f4a5Open thread
â€Ļâ€Ļ@@ -31,7 +31,7 @@ scripts/verify-cluster doctor "$run"
3232`up` returns an absolute manifest path on stdout. Each invocation creates a separate cluster, SSH credentials, CLI context, Config Store and application workspace. The manifest records each Machine's daemon source, version and binary hashes, and the run's artifact hashes and timings. Incus is the initial provider; its image, bridge and VM commands stay in `scripts/verify/incus.py`.
3333
34-The first run for an image recipe prepares Ubuntu with Docker, real ZFS, Corrosion, Caddy and nginx. Later runs boot clones with cached dependencies. Published daemons and their systemd units stay installed in release images; checkout runs install their own binaries and units. The host CLI drives both and reads each Entry Machine over SSH. Machine identities are created after cloning. Compilation overlaps image preparation and VM boot; installation and enrollment wait for the required binaries. Phase timings overlap; `total` records elapsed startup time.
34+The first run for an image recipe prepares Ubuntu with Docker, real ZFS, Corrosion, Caddy and the scenario workload images (nginx, Postgres, Alpine, whoami), so checks do not pull them from Docker Hub. Later runs boot clones with cached dependencies. Published daemons and their systemd units stay installed in release images; checkout runs install their own binaries and units. The host CLI drives both and reads each Entry Machine over SSH. Machine identities are created after cloning. Compilation overlaps image preparation and VM boot; installation and enrollment wait for the required binaries. Phase timings overlap; `total` records elapsed startup time.

P2 Badge Remove the remaining explicit scenario pulls

For two-Machine volume verification, the baked postgres:16-alpine image still does not eliminate registry access: test-verify-volume-move.py:460 and test-verify-warm-move.py:1118 both execute docker pull -q postgres:16-alpine during setup. Docker documents pull as downloading an image from a registry, so these checks can still fail or stall when Docker Hub is unavailable despite the new claim that checks no longer pull there (Docker CLI reference). Remove those explicit pulls and rely on/inspect the baked image.

Useful? React with 👍 / 👎.