Sign up
View stargazersSign up

With no Server taking web traffic, domain status says 'No ingress Server has a public IP', and only after a sync#3354

opened20d agobyNNick
GitHub writeback is off. Issues and comments you change here stay in GitCafe. Sync settings

Found in the 0.3.4 pre-cut soak (area deploy, id deploy-2).

Build: main c3277a38df + local 0.3.4 version bump; release CLI 0.3.4; 3 Boat Machines (public IPv6 each) enrolled in a local Cloud.

Steps

Project soakd with web serving a generated domain (web-2.<org>.ployz.test) and a custom domain, both ready.

1$ ployz server set machine-1 --accepts-ingress=false --yes # same for machine-2, machine-3
2$ ployz --json server ls # every Server: accepts_ingress false, public_ip set
3$ ployz domain ls # polled every 10s for 90s
4web-2.ada.ployz.test web 80 ready - # every poll
5$ ployz domain check web-2.ada.ployz.test
6status = needs attention
7reason = No ingress Server has a public IP
8$ ployz domain ls
9... needs attention No ingress Server has a public IP # both domains

Two problems

  1. Wrong reason. Every Server has a public IP. None takes web traffic. The status-side copy for that case already exists: No Server receives traffic (with AddServer). But Cloud's probeIngressServers (dashboard/src/modules/cluster-domain/sync.server.ts) filters accepts_ingress && public_ip !== null and returns no_public_ip when that is empty. So "no ingress Servers" reads as "no public IP". Because a Cluster Domain's status wins over #3251's standalone count, Cloud orgs never see NoServers for this.
  2. Stale for up to an hour. ployz server set --accepts-ingress=false doesn't request a Cluster Domain sync. The dashboard toggle does (server-web-traffic.inngest.ts). So domain ls kept saying ready until something else (domain check, a deploy, the hourly cron) ran the sync.

Expected

domain ls says No Server receives traffic soon after the last Server stops taking web traffic, whether that happens from the CLI or the dashboard.

Notes

  • Same filter in v0.3.3 (git show v0.3.3:dashboard/src/modules/cluster-domain/sync.server.ts), so not a regression.
  • Turning web traffic back on (--accepts-ingress=true) restored ready.
NNick18d ago

More evidence from soak-036 (main 6c849d95a3 + 0.3.6 bump; area compat). A related case: the status is stale after a Server's public IP changes, not only when no Server takes traffic.

  • Local Cloud, SERVERS=2 DAEMON=0.3.5 on Boat. The first Cluster Domain probe ran at 03:20:19, while machine-2 still advertised the Boat node's shared IPv4. The stored traffic row: {"down":[{"name":"machine-2","address":"<node IPv4>"}],"kind":"probed",...}.
  • ployz server set machine-2 --public-ip <its IPv6> followed (as the docs ask for a host behind NAT, #3352). Nothing re-probed.
  • From then until 04:14, ployz domain ls said needs attention Ingress isn't running on machine-2. Deploy to bring it back., and every deploy printed ! web-2.ada.ployz.test is not published yet…. Meanwhile curl --resolve through each Server's IPv6 returned 200, and ingress on machine-2 had been up the whole time. Six deploys didn't clear it, and neither did the 04:00 hourly sync. A server rm of another Server at 04:14 triggered a sync, and the domain read ready.
  • sync-cluster-domain runs only on cluster-domain/sync.requested: a deploy whose ingress changed, a single-domain query, or the hourly cron. server set --public-ip sends none. The code is unchanged since v0.3.5, so this is not a regression.

Suggestion: request a Cluster Domain sync when a Server's public IP or ingress role changes.