fix(client-runtime): recover subscriptions that stop on a healthy connection#24400

Problem

An environment can stop updating in one client while its socket still accepts commands. A durable subscription that fails alone is never restarted, leaving sidebar rows and thread state stale until the client reloads. A controlled malformed shell-stream frame reproduces this on current main: an unsettle command succeeds while the sidebar keeps the thread settled.

Change

Resubscribe durable streams on the same session after an RPC transport failure or a stream decoder SchemaError. Reuse the existing exponential retry delay, capped at 30 seconds, and cancel pending retries when the session changes. Shell and thread subscriptions start at their existing 250 ms delay; other subscriptions default to one second. Finite Find searches retain their prior failure handling instead of automatically resubscribing. Durable transport/decoder retries use a separate counter that resets on session replacement, so valid metadata preceding another bad chunk cannot keep retries at the minimum delay. Isolated stream failures on one long-lived session accumulate toward the 30-second cap. Existing defect observers and domain-error handling remain in place.

Scope and approval

Fixes https://github.com/pingdotgg/t3code/issues/4589 GitCafe: #4589

Triaged bug via the automated issue flow. This fixes subscription recovery in the shared client runtime without adding settings, persisted state, contracts, or provider behavior.

Verification

• Focused RPC client and runtime suites: 35/35 passed. Transport/decoder recovery tests fail with the base implementation. Two additional red tests reproduce finite-search decoder errors being swallowed and valid-first-value failures resetting the backoff; both pass with the fix. Tests also cover growth through the 30-second cap and reset/cancellation on session replacement.
• Client-runtime typecheck: no errors. Changed-file lint: no new findings; existing polling-loop warnings remain. Formatting and diff whitespace checks pass.
• Real app on Linux in Chrome: injected a malformed shell frame, observed a new subscription on the same open socket, and saw the accepted unsettle command update the sidebar without reload. Normal settling and full socket reconnect also work. The fault is controlled injection through the real decoder; the original incident's natural trigger is not established.

Integration: merged current main 6e79211743 into head 9446bbefd1. All three changed files are byte-identical to reviewed/captured 86f313103d. Both fresh isolated reviewers returned MERGE; the new F1/B1 evidence audit returned SUFFICIENT. Historical V1–V6 remain evidence at their original SHAs, not execution at current HEAD. The lead independently verified Find failure and explicit Retry on the same OPEN socket. The final auditor independently rebuilt it too: the Search failed/Retry region matched exactly; 734 pixels elsewhere differed in elapsed-age/focus styling, so whole-image rebuild equality is not claimed.

Historical first-failure evidence

Full evidence index, timed steps, transcripts and reproduction drivers. Controlled decoder fault injection; both sides use the real web app and server. BASE 27e0c684c5, HEAD 383a473f2e.

Chrome 154 on Linux, DPR 2, 16 px root font. Desktop CSS viewport 1440×900 (PNG export capped at 1280×800); responsive web 390×844 (PNG 780×1688). Light/dark as labeled. CSS animations, transitions and caret painting disabled; reduced-motion media emulation unavailable. Corrected RGB checks prove each of the twelve endpoint pairs is identical at least 550 ms apart.

V1: Shell and normal reconnect control — 1440×900 light

Steps: settle → fault current shell subscription → accepted unsettle; BASE stays settled, HEAD updates without reload.

Before recording · After recording · Timed narration

V2: Shell — 390×844 light

Steps: settle → fault current shell subscription → accepted unsettle; BASE stays settled, HEAD updates without reload.

Before recording · After recording · Timed narration

V3: Shell — 1440×900 dark

Steps: settle → fault current shell subscription → accepted unsettle; BASE stays settled, HEAD updates without reload.

Before recording · After recording · Timed narration

V4: Shell — 390×844 dark

Steps: settle → fault current shell subscription → accepted unsettle; BASE stays settled, HEAD updates without reload.

Before recording · After recording · Timed narration

V5: Selected thread — 1440×900 light

Steps: fault selected-thread subscription → accepted title change; BASE stays synchronizing, HEAD clears the synchronization banner.

Before recording · After recording · Timed narration

V6: Terminal resume — 1440×900 light

Steps: write BEFORE → fault terminal streams → accepted write AFTER; HEAD reattaches with resume:true and retains term-1.

Before recording · After recording · Timed narration

What else changed: the intended differences are sidebar group/count, selected-thread synchronization status, and resumed terminal output. Amplified comparisons also show small wordmark and sidebar-edge raster differences between browser profiles; these are capture differences outside the fix. The index documents every row and supersedes the initial faulty RGBA stability metadata.

Finite-search and repeated-failure evidence

Current-build index, transcripts and uploaded reproduction drivers · Execution matrix. BEFORE b1622197fa, AFTER 86f313103d. Real web app, CSS 1440×900 light, DPR 2, loaded fonts; PNG export 1280×800, recordings 2880×1800. No provider turn. These controlled faults run through the RPC decoder.

F1 — finite Find failure, manual Retry, completion and close/reopen

Normal control query normal-control completed with zero matches on both builds. Then fault-query was entered with receive interception armed. Two immediate query subscriptions appeared on each build; the active one is BEFORE request 87 / AFTER 76. Both initial real Chunk reads were intercepted. BEFORE automatically replaced request 87 with 89, 90 and 91, and stayed Searching. AFTER displayed Search failed / Retry and made no later automatic replacement for about 9.7 seconds. Its socket stayed OPEN. Injection was disarmed before clicking Retry; fresh request 78 completed with zero matches, then Close find → Ctrl+F → query completed again. BEFORE was disarmed and recovered through close/reopen.

Narration / video times: BEFORE normal query 00:36.3, fault query 00:41.9, retries 00:42.9 / 00:45.0 / 00:49.0, close/reopen query 00:57.3. AFTER normal query 00:20.1, fault query 00:25.7, manual Retry 00:35.7, close/reopen query 00:41.6. Initial idle footage includes a locator retry; it is not an app search delay.

Recordings: BEFORE recording · AFTER recording.

What else changed: 684 RGB-different pixels (552 with any channel >16), all inside exported rectangle [1115,61,1195,85): Searching is replaced by the wrapped Search failed label and Retry control. No other pixels differ. Both independently repeated error endpoints are exact RGB matches, separated by 6,000ms BEFORE and 5,944ms AFTER: BEFORE duplicate / AFTER duplicate.

Completed zero-match close/reopen endpoints (pixel-identical across builds):

B1 — repeated shell faults, healthy gap, cap and session replacement

Each fault targeted the current orchestration.subscribeShell request after genuine snapshot/synchronized values. The first nine retries stayed on BEFORE socket 5 and AFTER socket 4. A 3s healthy gap after retry 2 included successful unsettle/settle commands and real shell updates; the next AFTER delay remained 1s. Transcript timing measures fault → replacement Request, not request → request. Browser scheduling tolerance is visible in measured values.

Narration / video times: use the fault→replacement times below. Each recovery sends a real settle/unsettle command with successful receipt and visible sidebar/composer change. After the ninth recovery, a tenth fault is injected; only the app socket is closed 1s later. BEFORE has already retried at base delay, while AFTER has a pending capped retry. The new socket receives valid shell values, is faulted once, and returns to base delay. Both are observed past the old fault +32s, followed by successful normal unsettle → settle → unsettle. At about 02:24 both receive an additional matched settle/unsettle for final stills. Long static waits remain in the recording; the transcript is the primary timing evidence.

Recordings: BEFORE recording · AFTER recording.

BEFORE: tenth fault 00:08.7, socket close 00:09.7; new socket 6, request 194→196, 252.0ms at 00:14.4→00:14.7. Old socket emitted zero shell Requests after close through 00:40.7 (old fault +32.0s).

AFTER: tenth fault 01:38.7, socket close 01:39.7; new socket 5, request 147→150, 251.8ms at 01:42.7→01:42.9. Old socket emitted zero shell Requests after close through 02:10.7 (old fault +32.0s).

All accepted-retake dispatch receipts succeeded (21 BEFORE / 25 AFTER, including app-generated receipts); no failed command is part of this retake. Eleven faults per build have genuine first values recorded before injection. Vite sockets were never closed.

Narration for the endpoint pair: final normal recovery is complete and both clients show the same unsettled empty stopped thread. Static stills prove matching presentation and recovery, not retry timing.

What else changed: zero pixels across the retained final before/after endpoints. Both independent duplicate pairs are exact RGB matches, separated by 18,021ms BEFORE / 18,009ms AFTER: BEFORE duplicate / AFTER duplicate. An earlier pair had a fluctuating one-pixel sidebar edge (maximum channel delta 2) and was discarded; it was not accepted using a threshold or masked. Retained timestamp metadata · RGB report.

Unverified: the natural incident trigger, Electron/native clients, remote/relay/tunnel connections, and unmounted subscription consumers. The local responsive web captures do not imply native mobile verification.

Models and harnesses: GPT-6.1-Sol via Codex (issue lead, review and evidence); Claude Opus 5.5 via Claude Agent (regression tests, implementation and independent review).