[Bug]: An environment's threads silently stop updating until the client is restarted#4589

I'm seeing all sorts of issues with the recent nightlies along the same lines (doesn't matter if I have new sidebar on or off).

Often can't settle a thread, new threads aren't always added to the sidebar and the only way to get back to them is hitting the new button - but that means I also can't create new threads.

Woudlnt' be surprised if ti was somehow related to the above one (I'm assumign I have a thread in a broken state that's preventing other actions).

Confirmed instance with server-side evidence, from a case where the server runs on a different machine than the client — so the two sides could be inspected independently.

Thread e22c69b9-f1c5-4b14-8a6f-8f0084752556 ("Investigate Host-Specific Favicon Issue"). Client on host A, execution environment on host B.

The server has it settled. From host B's state.sqlite:

Ten settle commands, every one accepted. The user clicked repeatedly because the UI never changed — two bursts, ~20 minutes apart:

Every corresponding row in orchestration_command_receipts is accepted, with no rejected rows. The first settle did the work; the other nine re-emitted with the original settledAt and updatedAt, exactly as decider.ts:489-505 intends.

This settles the earlier ambiguity in this issue: it is not a client-side pre-send block. The commands reach the server, the server accepts them, the thread is settled — and the client never reflects it.

The server did emit the shell events. threadUpsertOrRemove (ws.ts:725-750) refetches the current projected row and emits thread-upserted after each of these. Its one silent-drop path is a projection refetch that fails twice ("If both attempts fail, log and drop the stream item", ws.ts:663-684) — and that path logs a warning. journalctl for the server has zero occurrences of shell projection refetch failed over its whole uptime, which covers both bursts. So the events were emitted and the client discarded or never received them.

Also ruled out while narrowing this:

• Client shell cache cross-contamination. Sequence spaces are per-environment and differ by an order of magnitude here (host B head is 20761; the client's other, local environment is at 195842). A snapshot sequence leaking across environments would make item.sequence > snapshot.snapshotSequence (shell.ts:149-158) discard every event from the lower-numbered environment permanently. But the shell cache is keyed by environmentId and re-verifies stored.environmentId === environmentId on load (apps/web/src/connection/storage.ts:464-476), so this is not it.
• Server-side snapshot sequencing. computeSnapshotSequence takes the min across required projectors in the same transaction as the projection reads.

What remains is a client whose shell subscription has stopped applying events for one environment while the socket stays healthy enough to round-trip commands on that same connection. The connection is not "down" in any way the client can currently notice — which is why nothing recovers it and why only a full restart (fresh HTTP snapshot + resubscribe) clears it.

Two properties make this unrecoverable rather than merely wrong:

1. The user's only tool, clicking Settle, is a guaranteed no-op once the server already agrees — by design.
2. Reconciliation is reachable only on initial load, foreground wake, or reconnect. A stalled-but-connected subscription hits none of them.

Worth noting the client had been running ~45 minutes when the first burst happened, and the server was under sustained load at the time (VcsStatusBroadcaster git fetch timeouts firing continuously), if that helps anyone reproduce it.

Follow-up: the same client is stale for the whole environment, not just the settled thread — and other machines connected to the same server show correct state. That isolates it to one client's subscriptions and points at a specific mechanism.

The tell: RPC works while subscriptions do not. The ten settle commands were accepted, and the client received their responses — SidebarV2.attemptSettle toasts on failure and nothing was toasted, so the mutations resolved as successes. Commands and subscription pushes share one WebSocket, so this is not transport-level. The RPC channel is healthy; the subscriptions on it are not delivering.

Where that can happen. In subscribeDynamic (packages/client-runtime/src/rpc/client.ts:211-222), a transport failure on a subscription stream logs and drains:

Draining ends the inner stream. Because it sits inside Stream.switchMap over sessions, nothing more is produced until sessions emits again. And supervisor.session is only written when a connection lease is established (connection/supervisor.ts:532) or cleared on teardown (:243).

So if a subscription stream fails with an RPC client error while the underlying session stays alive — a blip that does not tear down the lease — every affected subscription for that environment is permanently dead, with no retry and no error surfaced. New RPC calls on that same session continue to work, which is exactly why the app looks connected and commands still round-trip.

That matches every observation here:

• whole environment frozen in this client, other clients fine
• commands accepted and answered on the same socket
• no reconnect, because nothing was disconnected
• only a full client restart recovers it — a restart makes a new session, which is the one thing that resubscribes
• a thread displaying as Working long after it finished, per its stale last-known shell row

Note the contrast with the branch immediately below it: an expected failure with retryExpectedFailureAfter set does retry (shell.ts passes "250 millis"). Only the transport branch drains unconditionally. If the intent is "the supervisor will bring us back", that holds only when the failure also tears down the lease.

Two things would help independently of which layer gets the fix:

1. Something must be able to ask for reconciliation without a restart. Today reconciliation is reachable only on initial load, foreground wake, or reconnect, and a drained-but-connected subscription hits none of them.
2. A subscription that has stopped delivering is invisible. The shell state machine still reports live, so no indicator, no toast, nothing distinguishes it from a quiet environment.

I have a PR up for the first (#4593), which adds a resync-requested wakeup routed into the same resubscribe stream that subscribeDynamic merges into sessions — so requesting a resync does resubscribe a drained subscription. That was written as a mitigation for the settle symptom before this mechanism was understood; it happens to recover this case, but it only fires from the settle path, so it is not a general fix for the drain.

One more observation that pins the mechanism down, because it distinguishes a dead subscription from a dead connection.

While the client was in this state, a new thread was created successfully and never appeared in the sidebar. Server-side it exists and is running normally:

So on one connection, at the same moment:

• RPC works — thread creation round-tripped, as did ten settle commands earlier
• The pre-existing shell subscription is dead — the new thread produces a thread-upserted that never lands, so the sidebar does not list it
• A newly created thread-detail subscription works — the thread opens and streams

That asymmetry is the signature of the drain in subscribeDynamic (packages/client-runtime/src/rpc/client.ts:211-222). The drain is per subscription stream: a stream that hit the transport branch is over and waits for sessions to emit, while subscribeToSession() runs fresh for any subscription started afterward and works fine against the same still-healthy session. A connection-level failure could not produce this — it would take out the new subscription too.

Practical blast radius: every subscription established before the blip is dead for that environment, and everything opened afterward works. The app therefore reads as selectively broken rather than disconnected — the sidebar frozen (stale rows, threads stuck showing Working long after they finished, new threads missing) while any thread opened now behaves normally. Other clients against the same server are unaffected, which is what first ruled out the server here.

Nothing surfaces any of this. The shell state machine still reports live, so there is no indicator, no toast, and no way for the user to tell a frozen environment from a quiet one — which is why the reachable-for-humans symptom was "Settle does nothing".

The narrow fix is at the drain itself: retry the transport branch with backoff rather than waiting for a session change that never comes when the lease survives. Note the branch directly below it already does this for expected failures when retryExpectedFailureAfter is set — shell.ts passes "250 millis". Only the transport branch drains unconditionally.

A liveness signal seems worth having regardless. A subscription that has stopped delivering is currently indistinguishable from an idle one, at every layer above it.

Root-cause fix opened as #4602 — a durable subscription that loses its transport now asks the supervisor to re-establish, instead of draining and waiting for a session that nothing produces. Reconnect requests are coalesced per session, since a dead transport takes every subscription for an environment at once and each retryNow aborts an in-flight establishment attempt.

#4593 (reconcile on settle) remains useful as a targeted recovery for the settle path, but #4602 is the actual bug.

Two adjacent gaps found while narrowing this, neither addressed by either PR:

• Unary request (rpc/client.ts:107) has no timeout, so a call dispatched into a hung session never settles. In the web app that can latch attemptSettle's in-flight guard (SidebarV2.tsx:1694) permanently, since the key is cleared in a finally that never runs — which would make every later click return before any check, send, or toast.
• The only liveness probe runs on an application-active wakeup, so a desktop window that stays visible never health-checks its connection.

Following #4602 and #4405 from a production incident on a long-lived fork deployment (desktop client, two providers): sharing three additional failure modes we hit in the same subsystem, since neither PR covers them.

1. Terminal misses retry forever. Deleting a thread whose subscription was mid-flight leaves orchestration.subscribeThread failing with OrchestrationGetSnapshotError: Thread … was not found — classified as an expected failure and retried at the fixed delay for the rest of the session. We counted 12,000+ consecutive misses (~4/s for 80+ minutes) against a healthy connection. #4405's backoff shrinks the storm but keeps it eternal; a terminal classification (all-Fail causes whose errors all identify a permanent miss → stop, mark the thread deleted via the existing deleted-thread path) ends it.

2. Input construction failures kill the subscription fiber silently. In subscribeDynamic, makeInput runs outside the catchCause envelope. Its real work includes the HTTP snapshot load plus reducer application, so a transient failure there dies out of Stream.runForEach with no log, no retry, no error state — RPC works, streams never come back until restart. Matches the "RPC works while subscriptions do not" signature described upthread. Moving the input step inside the same classifier makes it recoverable without touching transport semantics.

3. A dead stream renders as healthy work. With fibers dead, the sidebar keeps "Working" (with a ticking timer) and chat shows optimistic echoes only. We added pure liveness derivations (platform-neutral, ready for both web and mobile) and a distinct Reconnecting presentation whenever the stream that would deliver the completion isn't alive. Related to #5742's stuck-Working symptoms.

Happy to send these as small focused PRs (terminal handling; input-envelope; Reconnecting surfaces) and/or stack on #4602/#4405 whatever shape you prefer. Fork branch with working code + tests available on request.

Windows data point (desktop 0.0.45, Windows 11, local environment, ~15 live Claude/Codex threads, window open for several hours), closed as a duplicate of this in #18481.

• A project added via Add project > Local folder was created server-side but never appeared in the sidebar, search, or Settings > Archive project list; re-adding failed with "Active project ... already exists".
• A thread created in that project via the server API never appeared either.
• Several threads kept showing "Working" after their turns completed on the server.
• RPC kept working in the same window (the re-add command round-tripped and returned the invariant error), consistent with dead subscriptions on a healthy connection.
• Recovery without stopping agents: the main-window renderer exiting abnormally triggered the app's render-process-gone reload; all state appeared at once. On Windows there is no menu bar, so no user-facing reload exists (Ideas: #18482).

iOS data point (TestFlight build).

I start a turn from the iPhone app, the server finishes it, and the thread never shows the agent's output. Force-quitting and reopening the app shows it right away. It happens about once a day across 20+ sessions a day.

The app stays in the foreground the whole time and shows Connected, so this looks like the same frozen-subscription problem on mobile rather than a resume-after-background issue.

Happy to test #24400 on iOS if a build is available.