[Bug]: Nightly retries unreachable SSH environments much harder than Alpha (≈3× tunnel readiness failures, 2× local ports)#15616

Triage

Thanks for the careful numbers, @smnbss. Having per-build counts from the trace files made this much easier to dig into.

What I found

I compared Alpha on 2 Oct (v0.0.45, 6c8fed35) with 0.0.46-nightly.20261003.2623 (fed41fa8). Between those two, packages/ssh/src/tunnel.ts only changed imports, and the readiness cadence is the same: a probe every 100ms, up to 20s for a new forward (SSH_READY_TIMEOUT_MS), and a 2s check before an existing forward is dropped (ensureTunnelEntry).

The relevant change in that range is #14897. When a connection keeps failing, reconnects now back off with jitter up to 5 minutes instead of stopping at 16 seconds. The first retry comes a little sooner (about 1–2s instead of 3s), and bringing the app to the foreground still resets the backoff, as it did on Alpha.

Here are your numbers converted to hourly rates, reading "until ~18:24" as 18.4 hours and the evening window as 6 hours:

So the 3 Oct evening window is clearly hotter, but the full Nightly day on 4 Oct is quieter than Alpha. During the spike, about one new local port every 5 minutes per host (147 ports over 6h across 2 hosts) lines up with the new 5-minute cap rather than a tighter loop.

Also, those loopback GETs aren't SSH connection attempts. launchOrReuseRemoteServer runs before reserveLocalTunnelPort, so connect timeouts, MagicDNS failures, and the 15-minute SSH command timed out never show up as http.client GET to 127.0.0.1. They come from waitForHttpReady on a forward that already exists. A full 20s wait is about 200 probes, which is close to the ~195 per port you saw. The evening average works out to about 20 GETs per failed waitForHttpReady span (7,317 / 369), which matches the 2s stale-tunnel check.

This is related to #11287 and #4144 but isn't the same report.

To narrow down what made that evening hot, a short sanitized trace excerpt would help:

• ssh/tunnel.launchOrReuseRemoteServer failures that never reserve a port
• each failed shared.httpReadiness.waitForHttpReady span, with its timeoutMs and attempts
• whether an application-active wakeup was resetting the backoff during the spike

The exact Alpha version string from that app would help too.

Likely fix area

• The reconnect backoff from #14897, especially how foreground and wakeup resets interact with it while a host stays unreachable.
• The stale-tunnel check in ensureTunnelEntry and waitForHttpReady in packages/ssh/src/tunnel.ts, if the excerpt shows forwards being recreated faster than the backoff allows.

A maintainer will decide on the fix direction.

I am seeing similar SSH recovery delays with a macOS T3 Code Nightly desktop client connecting to a Linux VPS over Tailscale.

The user reports losing the connection while the app is in the background and waiting for reconnection on return. They think it may sometimes occur while the app is visible too; that part is not yet verified.

Evidence from October 20, 2026 (times below are UTC+05:00):

• Desktop traces contain 12 ssh/tunnel.closeTunnelEntry events and 12 successful ssh/tunnel.startSshTunnel spans that day.
• Failures include SSH command timed out after 900000ms, SSH command timed out after 60000ms, and ssh: connect to host <tailscale-ip> port 22: Operation timed out.
• Examples: a launch/reuse operation beginning at 12:33:18 failed with the 900000ms timeout; a pairing-token operation beginning at 13:20:14 also failed with that timeout. These are span start times, not necessarily the times the errors were emitted. Sleep can extend wall-clock durations.
• Several tunnel-close events align with Mac wake activity: DarkWake at 21:20:27 and tunnel close at 21:20:27; wake at 23:22:40 and close at 23:22:46; wake at 23:40:17 and close at 23:40:26.
• When checked October 21, the systemd-managed VPS service was still running with ActiveEnterTimestamp=Mon 2026-10-19 16:15:20 UTC and NRestarts=0. A direct Tailscale ping succeeded in 148ms. This does not establish network availability during the failures.

The currently installed desktop version is 0.1.2-nightly.20261021.3473; the exact desktop build used during October 20 is unknown. The persistent VPS service process references 0.0.46-nightly.20261012.3021. The desktop also launches/manages a separate remote server on loopback port 3774, so that persistent service's version and uptime should not be treated as the version or uptime of every SSH-managed instance.

This is observational evidence, not a deterministic reproduction. It does not establish whether Tailscale reachability, sleep/wake recovery, or T3's SSH recovery logic initially caused the disconnect. Related: #21235 describes background-window reconnect churn, but a match to that trigger is not confirmed here.

No configuration changes or service restarts were performed during these checks. Raw logs are omitted because they contain pairing credentials.

Prepared with GPT-6.1-Sol through the Codex harness in T3 Code, using t3 triage and read-only desktop/VPS checks.

I have data for the third question in the triage, whether an application-active wakeup was resetting the backoff. It comes from a host that is unreachable for good rather than flaky.

Setup: macOS desktop, 0.0.46-nightly.20261011.2994. One saved SSH environment points at a host whose name no longer resolves. The trace files run from 2026-10-20 16:59 to 2026-10-23 11:33, local time.

What the traces show (desktop.trace.ndjson*, spans de-duplicated by span id):

• 2,564 failed desktop.ipc.sshEnvironment.ensureEnvironment calls. All but one fail with ssh: Could not resolve hostname …, so the failure happens before any port is reserved or tunnel started. That matches your first bullet: the work never reaches waitForHttpReady.
• Up to 423 attempts an hour while the app is in use.
• The gaps between attempts climb as #14897 intends (about 1.8 → 3 → 8 → 11 → 20 → 38 s), then repeatedly fall back to about 1–2 s. I counted a reset each time the gap fell back under 3 s after having grown past 4 s, which happened about 370 times. The delay reached before a reset had a median of 13 s and a p90 of 65 s. So the ladder rarely gets near the 5-minute cap.

Why I think it's the foreground wakeup: on current main (d7a8d6f), every visibilitychange to visible offers application-active (apps/web/src/connection/platform.ts). resetsRetryBackoff is true for it (packages/client-runtime/src/connection/wakeups.ts), and the supervisor then calls resetRetryLadder() after waitForRetrySignal (supervisor.ts). A DNS failure is classed as ConnectionTransientError (sshPreparationError), so the host never leaves the retry path. The traces don't record wakeup reasons, so the attribution is inferred, not proven. The reset cadence does follow how often the window gets focused, though.

Proposal: keep the immediate retry when the app comes back, so reconnecting still feels instant, but let application-active reset the ladder at most once per RETRY_MAX_DELAY_MS window. network-changed keeps resetting right away. An unreachable host then gets one attempt per return to the app and otherwise climbs to the cap. I've put this up as a small PR so it's concrete: #24064. It's one direction among several, so I'm happy to change it or close it if you'd prefer another.