fix(web): offer retry while an environment reconnects#23180

Problem

When a paired environment disconnects, web and desktop chat show a reconnecting banner without a retry action. Stopping the paired server reproduces the missing action while automatic retry backoff grows.

Change

Expose the existing environment retry command as “Retry now” during reconnecting. First connection still hides the action, and offline/error states retain “Reconnect”. Include synthetic stopped-thread fixtures and a transport fault proxy for reproducing the two reconnecting states.

Scope and approval

Fixes #22024

Triaged bug via the automated issue flow: #22024. This restores access to an existing retry capability in chat; Settings already offers it inside Details. No retry policy, protocol, provider or native mobile behavior changes.

Verification

Post-review fixture correction: 48b1ee75e4 handles upstream response errors by destroying the downstream response. Node 24.21 runtime check: before, two truncated responses hung until curl’s 5-second timeout; after, truncation is reported immediately and the proxy stays running. Syntax, formatting and targeted lint passed. Product code and capture SHAs are unchanged.

Merged current main b5a226e49c with a merge commit; final head d4424abcb3. The banner and connection runtime were unchanged on main since capture; post-merge scoped connection tests pass 75/75 and targeted lint passes. Captures retain their original product SHAs below.

The round-three request to remove evidence tooling conflicts with the task’s explicit requirement to commit synthetic seed data and scripts; fixtures are retained for that requirement. No product defect was identified.

• Reproduced on Linux Chromium at 70e5e384da with two real dev servers: stop the paired server, observe reconnecting without retry.
• Scoped supervisor, registry and presentation coverage: 75 tests passed in the final independent review. Earlier supervisor and ChatView logic run: 75 passed.
• Targeted product/fixture lint, fixture formatting and syntax checks, and web Knip unused-file analysis passed.
• Before merging main, web typecheck had two existing errors in useChatShortcuts.browser.test.tsx:53; the implementer confirmed the same failures on the base with the fix stashed.
• Eight matched web recording pairs cover backoff and active attempts at 1440/390 widths in light/dark. Final independent proof audit: SUFFICIENT.
• Desktop/Electron remains unverified and requires Julius’s acceptance. Relay/tunnel execution was not checked. Native mobile is unchanged. Clocks/jitter were not frozen; sparse video frames, historical proxy-load uncertainty and narrow baseline recovery limits are disclosed below. No source defects remain from three review rounds; desktop verification remains a routing condition.

Connection retry evidence for #22024

Before product: 70e5e384da. After product: 56809818ac. Final branch tip: d70a9233b7. Original pre-capture fixture ancestry restored; CI fix retained under apps/web/test/fixtures/connection-retry/. No production changes were added during capture. The base checkout used the same synthetic seed module and input data copied from the fixture directory.

Historical fixture provenance correction

The 23:37 consolidation obscured pre-capture commits. Restored their original ancestry without changing their timestamps, then replayed only the owned unpublished CI fix. Final tree is byte-for-byte identical to the prior CI-fixed HEAD; the consolidation tree equals the original 23:29 fixture tree. This is a history correction, not a recapture or a claim that today's fixture bytes existed before every recording. No screenshot assets are committed.

The table gives the latest original fixture commit already present at each recording start, according to original committer timestamps and capture-start metadata. A commit timestamp proves availability, not which revision an already-running proxy process loaded. No process-load attestation survives for every clip; do not infer proxy parity from seed/data equality. Both columns share the fixture inputs as documented, with distinct host identity/storage. Narrow light A2 predates the stable environment-ID additions; narrow dark A2 uses the fixed-header proxy era. Their baseline recovery limitations remain disclosed.

From c5d2089769 through e07f7e933d, seed.mjs and init.mjs are byte-identical. e07f7e933d changes only removal of the unused clock data field and reflection of requested CORS preflight headers in the proxy. Data files are therefore not byte-identical across e07f7e933d, though all effective seed fields are equal after ignoring the unused clock. The seed does not read that clock and clocks were never frozen. c5d2089769 also added stable environment IDs and changed lifecycle reset/setup; narrow light A2 used the earlier revision and cannot be assigned those later bytes retrospectively. CI fix moves the fixture to apps/web/test/fixtures/connection-retry/, adjusts relative imports and lint syntax, and preserves the production tree.

SHA-256 of historical fixture bytes (paths then apps/web/fixtures/connection-retry/; current path above):

Current CI-fixed fixture hashes:

• init.mjs: 85c133cc568cede287c08e53cc5f5ceacd86f35f034da191e82059f15eb362c2
• seed.mjs: d4f5e3a0accc1accf81406e00b37384b91b980339db3e69b46b2e2f653dfd077
• data.json: d8621d5a92571f18aa3e2bcaeb300087da767ce287e6c8a19652c8c1d634bd5f
• proxy.mjs: 1b7689844838d9e6d0b49a0e99daeb0ee20d38a64994873ae5ca13e7ecc2d7b5

Scope and environment

Corrected matrix: /tmp/issue-22024-proof-audit-pass1-corrected.md. Eight affected web pairs: backoff (A1) and active subsequent establishment (A2), each at 1440×1000 and 390×844, light and dark. Every row has before/after real-app recordings and supplemental matched stills. No provider turn was started.

T3 preview server browser only; Chrome 154 on Linux x86_64, DPR 2, browser zoom 1. System font stack: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif; no web fonts reported by document.fonts. Separate default/incognito profiles and separate base/head checkouts; isolated fresh host homes plus an isolated remote B. Synthetic stopped empty thread, stable project/thread IDs and label; explicit stable environment IDs were added in the c5d2089769 fixture revision, after narrow light A2; populated through existing pairing and RPC APIs. No real conversation screens or provider credentials used.

Both columns use the same host UTC wall clock and shared fault/recovery switches. Clock and Math.random were not frozen: actual retry deadlines differ and are reported below. Timing claims use the recorded supervisor state/deadline, not a claimed deterministic random delay. CSS viewports are exact; T3 saved wide PNGs at 1280×889, narrow PNGs at 780×1688. Recordings are 2880×2000 wide or 780×1688 narrow. Sparse frame encoding can end close to the last paint; wall-clock operation logs and supervisor traces establish request/state ordering; only decoded MP4 presentation timestamps establish visible video timing. They are not synchronized to millisecond precision.

Only the owned remote B transport was failed/held by a localhost proxy; host A remained available. These are direct localhost browser-to-server connections. Relay, tunnel, physical touch, native mobile, and Electron execution are not claimed.

What else differs: host ports and primary environment IDs/profile storage are necessarily distinct; browser caret, hover/focus and timing can contribute small pixels. Added Retry now shortens diagnosis text and raises the expanded narrow stack. All retained wide stills have the sidebar closed. Narrow A2 adds the same synthetic update-failure banner to both columns and expands the real stack. No screenshot assets are committed.

Matrix

A1 baseline waits for its scheduled retry while head requests a new attempt before the old head deadline. A2 head is retried twice while establishment is held; retry resets the supervisor attempt counter, so a temporary initial-connecting presentation is expected. The red update-failure banner in narrow A2 is synthetic and may remain after the connection warning clears.

Diff method: absolute unregistered RGB difference (diff0) amplified ×8 with clipping. Threshold panel is white where max absolute channel delta is >16/255, black otherwise. Metrics below use full original saved PNGs, no masking; displayed comparison panels use an identical lower-half crop of each original. This is explanatory visual evidence, not a pixel-equality assertion.

1440-light — A1

Recording starts (UTC): before 23:26:10.621, after 23:26:11.652. Operation log below gives UTC and wall-clock offsets from each column’s recording-start metadata (B/A), not MP4 PTS or exact visible activation times.

• 23:26:11.665 — B +1.04s / A +0.01s: fail.
• 23:26:45.826 — B +35.20s / A +34.17s: before A1 still; actual {"phase":"backoff","attempt":5,"retryAt":1792625224820,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:26:46.096 — B +35.48s / A +34.44s: after A1 still; actual {"phase":"backoff","attempt":5,"retryAt":1792625226948,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:26:46.152 — B +35.53s / A +34.50s: restore.
• 23:26:46.271 — B +35.65s / A +34.62s: retry.

Before recording · After recording

Before — 70e5e384da

Before 1440-light

After — 56809818ac (fixture only on top)

After 1440-light

Before and after 1440-light, amplified diff0 and threshold

Full-frame threshold: 537/1,137,920 pixels (0.0472%). Raw diff0 bounding box: [328, 717, 944, 776].

Redacted supervisor milestones (milliseconds relative to trace subscription; subscription precedes video start). All rows target synthetic remote B; no credential or request URL is included.

1440-light-A2 — A2

Recording starts (UTC): before 23:15:42.414, after 23:15:44.022. Operation log below gives UTC and wall-clock offsets from each column’s recording-start metadata (B/A), not MP4 PTS or exact visible activation times.

• 23:15:44.034 — B +1.62s / A +0.01s: fail.
• 23:15:47.082 — B +4.67s / A +3.06s: hold.
• 23:15:49.012 — B +6.60s / A +4.99s: before A2 still; actual {"phase":"connecting","attempt":3,"retryAt":null,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:15:49.256 — B +6.84s / A +5.23s: after A2 still; actual {"phase":"connecting","attempt":3,"retryAt":null,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:15:49.587 — B +7.17s / A +5.57s: retry1 operation logged.
• 23:16:10.001 — B +27.59s / A +25.98s: retry2 operation logged.
• 23:16:13.521 — B +31.11s / A +29.50s: recovery operation logged.

Decoded after-video PTS, read sequentially with OpenCV CAP_PROP_POS_MSEC (zero-based frames): first click ring frame 18 at 6.948133s; initial-connecting/reset presentation frame 30 at 7.378000s; second click ring frame 34 at 30.035367s; second reset frame 47 at 30.764633s; connection warning absent by frame 49 at 32.973800s. These are first observed frames, not exact event-dispatch/acceptance instants. The second click is 4.056367s later than the operation-log offset 25.979s; no constant wall-clock-to-video correction is asserted. Supervisor traces below independently establish accepted resets. The operation at A +29.50s restores transport; it is not the visible recovery frame. Sparse video cannot establish exact paint latency.

Before recording · After recording

Before — 70e5e384da

Before 1440-light-A2

After — 56809818ac (fixture only on top)

After 1440-light-A2

Before and after 1440-light-A2, amplified diff0 and threshold

Full-frame threshold: 532/1,137,920 pixels (0.0468%). Raw diff0 bounding box: [328, 722, 937, 776].

Redacted supervisor milestones (milliseconds relative to trace subscription; subscription precedes video start). All rows target synthetic remote B; no credential or request URL is included.

1440-dark-A1 — A1

Recording starts (UTC): before 23:29:06.060, after 23:29:06.859. Operation log below gives UTC and wall-clock offsets from each column’s recording-start metadata (B/A), not MP4 PTS or exact visible activation times.

• 23:29:06.869 — B +0.81s / A +0.01s: fail.
• 23:29:37.168 — B +31.11s / A +30.31s: before still; actual {"phase":"backoff","attempt":5,"retryAt":1792625386905,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:29:37.422 — B +31.36s / A +30.56s: after still; actual {"phase":"backoff","attempt":5,"retryAt":1792625399699,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:29:37.484 — B +31.42s / A +30.62s: restore.
• 23:29:37.873 — B +31.81s / A +31.01s: keyboard focus; focused text 'Retry now'.
• 23:29:37.927 — B +31.87s / A +31.07s: keyboard Enter retry.

Before recording · After recording

Before — 70e5e384da

Before 1440-dark-A1

After — 56809818ac (fixture only on top)

After 1440-dark-A1

Before and after 1440-dark-A1, amplified diff0 and threshold

Full-frame threshold: 497/1,137,920 pixels (0.0437%). Raw diff0 bounding box: [720, 717, 830, 739].

Redacted supervisor milestones (milliseconds relative to trace subscription; subscription precedes video start). All rows target synthetic remote B; no credential or request URL is included.

1440-dark-A2 — A2

Recording starts (UTC): before 23:31:06.844, after 23:31:07.677. Operation log below gives UTC and wall-clock offsets from each column’s recording-start metadata (B/A), not MP4 PTS or exact visible activation times.

• 23:31:07.689 — B +0.84s / A +0.01s: fail.
• 23:31:10.108 — B +3.26s / A +2.43s: hold.
• 23:31:13.578 — B +6.73s / A +5.90s: before still; actual {"phase":"connecting","attempt":3,"retryAt":null,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:31:13.844 — B +7.00s / A +6.17s: after still; actual {"phase":"connecting","attempt":3,"retryAt":null,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:31:13.894 — B +7.05s / A +6.22s: retry1 operation logged.
• 23:31:26.410 — B +19.57s / A +18.73s: retry2 operation logged.
• 23:31:26.808 — B +19.96s / A +19.13s: restore.

Before recording · After recording

Before — 70e5e384da

Before 1440-dark-A2

After — 56809818ac (fixture only on top)

After 1440-dark-A2

Before and after 1440-dark-A2, amplified diff0 and threshold

Full-frame threshold: 519/1,137,920 pixels (0.0456%). Raw diff0 bounding box: [328, 722, 937, 776].

Redacted supervisor milestones (milliseconds relative to trace subscription; subscription precedes video start). All rows target synthetic remote B; no credential or request URL is included.

390-light-A1 — A1

Recording starts (UTC): before 23:30:06.031, after 23:30:06.565. Operation log below gives UTC and wall-clock offsets from each column’s recording-start metadata (B/A), not MP4 PTS or exact visible activation times.

• 23:30:06.593 — B +0.56s / A +0.03s: fail.
• 23:30:36.840 — B +30.81s / A +30.27s: before still; actual {"phase":"backoff","attempt":5,"retryAt":1792625450351,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:30:37.060 — B +31.03s / A +30.50s: after still; actual {"phase":"backoff","attempt":5,"retryAt":1792625450902,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:30:37.113 — B +31.08s / A +30.55s: restore.
• 23:30:37.198 — B +31.17s / A +30.63s: retry.

Before recording · After recording

Before — 70e5e384da

Before 390-light-A1

After — 56809818ac (fixture only on top)

After 390-light-A1

Before and after 390-light-A1, amplified diff0 and threshold

Full-frame threshold: 1,039/1,316,640 pixels (0.0789%). Raw diff0 bounding box: [256, 1434, 394, 1461].

Redacted supervisor milestones (milliseconds relative to trace subscription; subscription precedes video start). All rows target synthetic remote B; no credential or request URL is included.

390-light-A2 — A2

Recording starts (UTC): before 23:08:55.832, after 23:08:56.086. Operation log below gives UTC and wall-clock offsets from each column’s recording-start metadata (B/A), not MP4 PTS or exact visible activation times.

• 23:08:56.096 — B +0.26s / A +0.01s: fail.
• 23:08:58.992 — B +3.16s / A +2.91s: hold.
• 23:09:02.043 — B +6.21s / A +5.96s: before A2 still; actual {"phase":"connecting","attempt":3,"retryAt":null,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:09:02.256 — B +6.42s / A +6.17s: after A2 still; actual {"phase":"connecting","attempt":3,"retryAt":null,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:09:02.547 — B +6.71s / A +6.46s: retry1 operation logged.
• 23:09:20.021 — B +24.19s / A +23.93s: retry2 operation logged.
• 23:09:20.558 — B +24.73s / A +24.47s: recovery operation logged.

Baseline natural recovery was not established inside this retained clip; head retry/recovery is observed. In the dark narrow run, a later baseline preflight recovery failed in the original proxy configuration; the fixture now reflects requested CORS headers. This limits the baseline recovery claim, not the captured connecting-attempt state or head action.

Before recording · After recording

Before — 70e5e384da

Before 390-light-A2

After — 56809818ac (fixture only on top)

After 390-light-A2

Before and after 390-light-A2, amplified diff0 and threshold

Full-frame threshold: 47,462/1,316,640 pixels (3.6048%). Raw diff0 bounding box: [45, 697, 735, 1461].

Redacted supervisor milestones (milliseconds relative to trace subscription; subscription precedes video start). All rows target synthetic remote B; no credential or request URL is included.

390-dark-A1 — A1

Recording starts (UTC): before 23:11:40.047, after 23:11:42.386. Operation log below gives UTC and wall-clock offsets from each column’s recording-start metadata (B/A), not MP4 PTS or exact visible activation times.

• 23:11:42.397 — B +2.35s / A +0.01s: fail.
• 23:12:18.906 — B +38.86s / A +36.52s: before A1 still; actual {"phase":"backoff","attempt":5,"retryAt":1792624350638,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:12:19.113 — B +39.07s / A +36.73s: after A1 still; actual {"phase":"backoff","attempt":5,"retryAt":1792624346967,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:12:19.735 — B +39.69s / A +37.35s: restore.
• 23:12:20.555 — B +40.51s / A +38.17s: retry.

Before recording · After recording

Before — 70e5e384da

Before 390-dark-A1

After — 56809818ac (fixture only on top)

After 390-dark-A1

Before and after 390-dark-A1, amplified diff0 and threshold

Full-frame threshold: 1,085/1,316,640 pixels (0.0824%). Raw diff0 bounding box: [238, 1418, 411, 1474].

Redacted supervisor milestones (milliseconds relative to trace subscription; subscription precedes video start). All rows target synthetic remote B; no credential or request URL is included.

390-dark-A2 — A2

Recording starts (UTC): before 23:13:17.171, after 23:13:17.730. Operation log below gives UTC and wall-clock offsets from each column’s recording-start metadata (B/A), not MP4 PTS or exact visible activation times.

• 23:13:17.740 — B +0.57s / A +0.01s: fail.
• 23:13:22.585 — B +5.41s / A +4.86s: hold.
• 23:13:24.345 — B +7.17s / A +6.62s: before A2 still; actual {"phase":"connecting","attempt":3,"retryAt":null,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:13:25.008 — B +7.84s / A +7.28s: after A2 still; actual {"phase":"connecting","attempt":3,"retryAt":null,"lastFailure":{"reason":"remote-unavailable"}}.
• 23:13:25.603 — B +8.43s / A +7.87s: retry1 operation logged.
• 23:13:48.077 — B +30.91s / A +30.35s: retry2 operation logged.
• 23:13:48.541 — B +31.37s / A +30.81s: recovery operation logged.

Baseline natural recovery was not established inside this retained clip; head retry/recovery is observed. In the dark narrow run, a later baseline preflight recovery failed in the original proxy configuration; the fixture now reflects requested CORS headers. This limits the baseline recovery claim, not the captured connecting-attempt state or head action.

Before recording · After recording

Before — 70e5e384da

Before 390-dark-A2

After — 56809818ac (fixture only on top)

After 390-dark-A2

Before and after 390-dark-A2, amplified diff0 and threshold

Full-frame threshold: 81,966/1,316,640 pixels (6.2254%). Raw diff0 bounding box: [27, 697, 753, 1461].

Redacted supervisor milestones (milliseconds relative to trace subscription; subscription precedes video start). All rows target synthetic remote B; no credential or request URL is included.

Unchanged controls and unverified surfaces

These are source controls, not additional browser-pass claims:

• C1 initial connecting: action gate still suppresses action; presentation distinguishes initial from subsequent attempts.
• C2 offline/available/error/unsupported: the same gate retains Reconnect; blocked mapping is unchanged in presentation. No separate offline/error runtime capture was made.
• C3 connected/missing thread/catalog label: ChatView guard is unchanged. Recovery is observed in affected runs. Other guard states were not separately driven.
• C4 grace/reset and C5 update suppression: same timer and guard. Recordings begin connected and continue beyond grace. Reset/switch and running-update controls were inspected in source, not separately executed.
• C6 Disconnect: same delayed eligibility and local exclusions. Its coexistence with the new button is captured in A1; Disconnect activation and desktop-local targeting were not executed.
• C7 stack/branch mismatch: both ChatView assembly branches spread the same system items; stack renderer uses item actions in attached/floating placements. Narrow expanded composition is executed; branch-mismatch placement is source-only.
• C8 Settings diagnosis: existing separate Retry now unchanged; not executed here.
• D1 Electron: UNVERIFIED, unavailable. Responsive narrow browser is not native mobile. Native mobile files were unchanged.
• T1 relay/tunnel, additional providers, agent/scheduled entry points: not executed. No additional branch changed in this diff. Only actual direct remote B targeting is observed.

Validation and hygiene

Focused existing tests: presentation 4 passed; supervisor “explicit retry interrupts the current backoff” 1 passed (29 skipped). Node syntax checks passed for init.mjs, seed.mjs and proxy.mjs. Commit hook formatted the fixture. No repo-wide checks or tests mirroring the visual implementation were added.

All published stills and sampled recording contact sheets were inspected for conversation content and credentials. Only synthetic empty-thread content is present. Pairing links, tokens, cookies, signed asset URLs, raw operational logs and traces remain private; no such files were uploaded or committed. Evidence artifacts are outside the worktree. No push, PR creation, or watcher was started.

FRICTION

• Scope: corrected audit reduced unchanged-control recordings to source controls and required both affected supervisor states at W4.
• Historical provenance: original committed fixture history restored; process-load revision is not independently attested for every clip. Narrow A2 uses older proxy behavior, as disclosed.
• Timeline: operation-log offsets and video PTS are separate; wide light A2 second-click discrepancy corrected above.
• Fixture lifecycle: retryNow on a connected supervisor probes; reload was needed to reset episodes cleanly. Proxy gained owned-socket error handling and reflected CORS preflight headers. Trials affected by fixture faults were discarded, except the explicitly limited baseline recovery in narrow A2.
• Browser tools: intermittent T3 transport disconnects and tiny hover-sensitive stack-toggle timeouts; stack expansion used a real DOM button click through T3 preview evaluate. Failed tool calls were not counted as activations.
• Evidence: wide snapshot downsampling and sparse recorder frames required post-recovery snapshots/holds and replacement clips. Sidebar noise was removed by closing it equally. No implementation change was needed.
• Coverage: desktop unavailable; relay/tunnel/native touch not exercised.

Final handoff checks: all 40 public asset URLs downloaded successfully with ordinary GET and matched local files byte-for-byte. HEAD requests were rejected; no public-link failure remains. The seed module and data.json match between the two checkouts by SHA-256. All owned server/proxy sessions were stopped (71217, 48212, 57337, 14431), and ports 8556, 7976, 16596, 16016, 14273, 14274 and 14275 were confirmed free. Working tree is clean. The initial supplied nondeterministic screenshot and failed/interrupted trial captures are excluded from this evidence set.

Correction verification artifacts

• provenance-hashes.json
• asset-sha256.txt
• cv2-pts.json
• wide-light-A2-pts.png

All 16 new standalone full-frame diff images were downloaded with ordinary curl GET and matched local bytes. Their threshold counts match the original eight-row manifest. The historical hash manifest retains all eight original commits, original committer times, Git blob IDs and SHA-256 values; files absent in an early revision are explicitly null. The asset hash manifest covers all 40 original assets plus the 16 new diff images.

Scoped fixture validation rerun after history correction: all three Node syntax checks, Vite+ lint with unused-disable reporting, and four-file format check passed. No product edits or recaptures were performed. Working tree remains clean.

Implemented with Claude Opus 5.5 (Claude Agent harness); triage, proof and review with GPT-6.1 Sol (Codex harness).

Fixes pingdotgg/t3code#22024