fix(client): slow networks no longer drop working connections#19426

open8
main
t3/slow-links-keep-connections
opened8d agobyTtheo
View on GitHubTwo-way sync enabledPR last imported 2d ago

Healthy slow connections can be dropped when a health check or cached T3 Connect ticket takes more than three seconds. The shared connection supervisor now gives quick probes six seconds and extends them when raw WebSocket messages still arrive, with an absolute fifteen-second cap. Cached ticket requests also get six seconds, including learned LAN and tailnet routes.

Frame counting runs before the RPC read loop, so a full subscription buffer cannot hide incoming traffic. The message listener belongs to the socket scope and is removed on release. Mobile returns from an inactive state without a background visit reset retries and check for a better route without probing or resubscribing shell/thread streams. Genuine background resumes retain their existing probe/reconnect behavior. Effect's five-second ping cadence, direct endpoint identity checks, and current main's token-preservation rules are unchanged.

Maintainer takeover of #19426, building on @t3dotgg's work and @nick-potts's original contribution in #18704. All contributor commits are preserved; current main was merged and follow-ups simplify the tests, socket ownership, and comments.

Verification on headless Linux, Node 24, at aa7bb7087e:

  • Focused tests: 421 tests / 19 files → 419 tests / 19 files, passing. Eight mutations demonstrate failures for the old quick timeout, missing frame extension, wrong retry window, excessive cap, missing raw listener, old ticket timeout, inactive refocus probe, and refocus resubscription. Client-runtime, mobile, web, and desktop typechecks pass; all changed files pass lint/format and whitespace checks.
  • Actual loopback WebSocket timing evidence: production RpcSession and EnvironmentSupervisor with Node's WebSocket client and a controlled server. Substituting the base supervisor replaces the healthy socket at 3,014 ms; the rebuilt head keeps the same socket through an 8,006 ms probe reply while messages arrive. A silent socket is replaced at 6,018 ms; continuous incoming messages cannot extend a stalled probe beyond 15,022 ms. All owned loopback servers closed. This is protocol-level evidence with synthetic configuration, not a browser recording or a real WAN reproduction.
  • Both independent full-diff adversarial reviewers found no substantive issues in round 1. One independently reran 224 tests across seven relevant files. Human review of authorization-path and runtime default changes remains required; this audit does not supply approval.
  • Trade-off: detecting silence through a quick probe now takes six seconds. Incoming messages can defer replacement to fifteen seconds. A stalled cached ticket uses six seconds of the existing fifteen-second setup budget, leaving roughly nine seconds for renewal and the next ticket request on that route.
  • Native device access is disabled in this environment. Real slow-network T3 Connect/Tailscale connections, iOS Control Center/Face ID and background lifecycle, Android, macOS, Windows, hosted browser, and Electron desktop were not exercised. Device/network verification and human approval remain outstanding. Current-head CI passes every executed job, including Check, Test, Test Web, Build and Typecheck, Lint, and eight server shards. Correctness and CodeRabbit pass; Approvability remains neutral. A duplicate fingerprint job was cancelled before a successful replacement. This PR remains unmerged.

Tests: GPT-6-Astra (high); implementation: GPT-6.1-Sol (high); adversarial reviews: Claude Opus 5.5 and GPT-6-Sol through T3 Code delegated tasks. Coordinated with GPT-6.1-Sol through the Codex harness in T3 Code.

Checks

All checks passed

Added 4 commits

  • fix(mobile): slow networks no longer drop working connections6478d22
  • fix(client): data still arriving keeps a slow health check alivecf51630
  • fix(client): tighten slow-link probe extension and keep refocus retriese453eff
  • fix(client): count frames on arrival and keep the default ping intervalf79ea35
Synced from GitHub

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — This PR changes default connection and cached-ticket timeouts and modifies central WebSocket, mobile lifecycle, and authorization behavior. Those production defaults and authorization-path changes require human review despite the focused tests and otherwise targeted bug-fix scope.

You can add or adjust custom eligibility rules. Learn more.

Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire5.0 KiB5.0 KiB−7 B (−0.1%)6.8 KiB✅
CodexThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
CodexLive turn WebSocket wire1.2 KiB1.2 KiB−7 B (−0.6%)2.0 KiB✅
CodexLive turn WebSocket decoded20.9 KiB20.9 KiB0 B (0.0%)29.3 KiB✅
CodexLive turn messages220 (0.0%)8✅
ClaudeTotal thread wire5.0 KiB5.0 KiB+24 B (+0.5%)6.8 KiB✅
ClaudeThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
ClaudeLive turn WebSocket wire1.2 KiB1.2 KiB+24 B (+2.0%)2.0 KiB✅
ClaudeLive turn WebSocket decoded21.2 KiB21.2 KiB+41 B (+0.2%)29.3 KiB✅
ClaudeLive turn messages12+1 (+100.0%)8✅

Baseline: dd5c7bd · PR result: b57afe0 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Synced from GitHub

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Path: .coderabbit.config.ts
  • Review profile: CHILL
  • Plan: Team
  • Run ID: b2e1ff20-b7fc-4b55-b6da-50daaf8c4e85
📥 Commits

Reviewing files that changed from the base of the PR and between b57afe0fa54f4769fdcdf3d38e67bd8c08e74b3e and aa7bb7087eb8390248c6fd8ea06329a21bc105ca.

📒 Files selected for processing: 16
  • apps/mobile/src/connection/app-state-wakeups.test.ts
  • apps/mobile/src/connection/app-state-wakeups.ts
  • docs/internals/connection-runtime.md
  • packages/client-runtime/src/authorization/layer.test.ts
  • packages/client-runtime/src/authorization/service.ts
  • packages/client-runtime/src/connection/registry.test.ts
  • packages/client-runtime/src/connection/supervisor.test.ts
  • packages/client-runtime/src/connection/supervisor.ts
  • packages/client-runtime/src/connection/wakeups.ts
  • packages/client-runtime/src/operations/commands.performance.test.ts
  • packages/client-runtime/src/rpc/session.test.ts
  • packages/client-runtime/src/rpc/session.ts
  • packages/client-runtime/src/state/server.test.ts
  • packages/client-runtime/src/state/serverUsage.test.ts
  • packages/client-runtime/src/state/shell-sync.test.ts
  • packages/client-runtime/src/state/threads-sync.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

📝 Walkthrough

The connection runtime now counts incoming WebSocket frames and uses frame activity when timing session probes. Mobile wakeups distinguish brief refocus from background visits. The cached-endpoint socket timeout increases from three to six seconds.

Changes: Connection runtime

Layer / File(s)Summary
RPC received-frame counting
packages/client-runtime/src/rpc/session.ts, packages/client-runtime/src/rpc/session.test.ts, packages/client-runtime/src/connection/registry.test.ts, packages/client-runtime/src/operations/*.test.ts, packages/client-runtime/src/rpc/client.test.ts, packages/client-runtime/src/state/*.test.ts, apps/web/src/cloud/linkEnvironment.test.ts
RpcSession counts WebSocket message events and exposes the count through receivedFrames. Session mocks and fixtures provide the new field. A test checks that a Pong is counted while the config-stream consumer is blocked.
Probe timing and frame-driven extensions
packages/client-runtime/src/connection/supervisor.ts, packages/client-runtime/src/connection/supervisor.test.ts, docs/internals/connection-runtime.md
Quick probes use a six-second window. Incoming frames can extend a probe deadline, up to 15 seconds from probe start; retry signals can shorten the deadline and set its extension window. Tests cover probe timing, and the connection-runtime documentation describes the behavior.
Cached-endpoint authorization timeout
packages/client-runtime/src/authorization/service.ts, packages/client-runtime/src/authorization/layer.test.ts
The cached-endpoint socket timeout increases to six seconds. Tests cover delayed ticket responses and a pending cached-endpoint request that proceeds through relay bootstrap after six seconds.
Mobile refocus wakeups
apps/mobile/src/connection/app-state-wakeups.ts, apps/mobile/src/connection/app-state-wakeups.test.ts, apps/mobile/src/connection/platform.ts, packages/client-runtime/src/connection/wakeups.ts, packages/client-runtime/src/connection/supervisor.test.ts, packages/client-runtime/src/state/shell-sync.test.ts, packages/client-runtime/src/state/threads-sync.test.ts, docs/internals/connection-runtime.md
Mobile returns without a background timestamp now emit application-refocused. The runtime classifies it as an application-active wakeup. Tests cover retry behavior, connected-session behavior, and state wakeup handling.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

mermaid
1sequenceDiagram
2 participant WebSocket
3 participant RpcSession
4 participant ConnectionSupervisor
5 WebSocket->>RpcSession: message event increments received-frame count
6 ConnectionSupervisor->>RpcSession: read receivedFrames at probe deadline
7 alt New frames arrived and the 15-second cap remains
8 ConnectionSupervisor->>ConnectionSupervisor: extend deadline by current probe window
9 else No new frames or the cap is reached
10 ConnectionSupervisor->>ConnectionSupervisor: handle probe timeout
11 end

Possibly related PRs

  • pingdotgg/t3code#18704: Both pull requests change probe timing and mobile refocus behavior. This PR includes socket-lifetime frame counting, a 15-second probe cap, and related refocus handling.

Suggested reviewers: juliusmarminge

Merge Risk: ⚪ Minimal · up to aa7bb

No concrete merge-blocking issue remains; proceed with normal checks.

Pre-merge checks: Passed 4
Check nameStatusExplanation
Title checkPassedThe title clearly and concisely describes the main change: preventing healthy slow client connections from being dropped.
Linked Issues checkPassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes checkPassedCheck skipped because no linked issues were found for this pull request.
Description checkPassedThe description is detailed and covers the problem, implementation, scope, verification results, limitations, trade-offs, and contributor or maintainer context. It does not use the template section he…
Full details: Description check

Explanation

The description is detailed and covers the problem, implementation, scope, verification results, limitations, trade-offs, and contributor or maintainer context. It does not use the template section headings, but the required information is present and the description is relevant and substantially complete.

✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Added 1 commit

  • docs(client): say why a refocus does not resubscribeb57afe0

Taking over this one as maintainer for batch ssh-relay-remote-10. I will preserve the contributor commits, merge current main, simplify the implementation and tests, and push follow-ups here. Slow-link and native verification limits will stay explicit; this remains unmerged pending human approval.

Added 3 commits

  • Merge remote-tracking branch 'origin/main' into t3/slow-links-keep-connections9249b2b
  • test(connection): focus slow-link and refocus coveragecd95cbc
  • fix(client): scope slow-link frame tracking to socket lifetimeaa7bb70