Sign up
View stargazersSign up

[Bug]: a stalled command pipeline gives an endless spinner, and the message typed while it stalls is dropped#23661

opened19d agobyVvifar

Use this form for broken behavior, regressions, crashes, or reliability problems. Feature requests belong in Discussions. Search existing issues first and keep the report focused on one problem.

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

Not sure

(The change is in packages/client-runtime, which the Area dropdown does not list, and in apps/web.)

Steps to reproduce

  1. Put the host under sustained load, so that a command it accepts never comes back. On the machine this was observed on, the host was serving a 25.5 GB SQLite state database, the load average was 68-116 on 10 cores, and the server's event loop was stalling for 2-24 s at a time.
  2. Open a thread and press Enter to send a message.
  3. While that send is still in flight, type a second message and press Enter.
  4. The second message is discarded. The composer shows a spinner that never resolves, and no error appears.

The same stall, observed from another machine over Tailscale, showed a command that got no answer for 163 seconds while small requests to the same host still answered in 6-126 ms.

Expected behavior

  • A send that the host does not answer should fail with an error the user can see, rather than waiting forever.
  • A message typed while a send is in flight should be queued and sent when the composer is free, not dropped.
  • A connection whose command pipeline has stalled should be reported as stalled, even though the socket itself is still healthy.

Actual behavior

1. A request waits forever. packages/client-runtime/src/rpc/client.ts has no timeout, deadline, or Effect.timeout anywhere in its 446 lines, so requestGuarded waits for a reply that never comes. The only bound in the transport is the socket-level ping timeout in packages/client-runtime/src/rpc/protocol.ts:

ts
1// Pings go out every 5 seconds, and a server busy for a few seconds stops
2// answering them. Allow 15 seconds without any frame before dropping the socket.
3export const PING_TIMEOUT = "15 seconds";

That is a no-traffic check. A host under load keeps answering small requests, so the socket stays alive and the one command that is stuck is invisible to it. This is why the host above answered small requests in 6-126 ms while a command got no answer for 163 s.

2. The composer drops Enter, silently. apps/web/src/components/ChatView.tsx:4665-4679 returns early when the send guard fires:

ts
1if (
2 !readEnvironmentScope(environmentId, AuthOrchestrationOperateScope) ||
3 !activeThread ||
4 (isSendBusy && !sendQueuesBehindSetup) ||
5 isConnecting ||
6 isRevertingCheckpoint ||
7 !clientSettingsHydrated ||
8 threadDetailLoading ||
9 readPendingEnvironmentChange() !== null ||
10 sendInFlightRef.current ||
11 feedbackUploadsInFlightRef.current.has(routeThreadKey)
12) {
13 notifyDirectAnnotationAttached();
14 return;
15}

sendQueuesBehindSetup only covers the worktree-setup case, not a general in-flight send. The text the user typed is never sent, never queued, and never reported: the only sign is the spinner.

3. Liveness cannot see a stalled command pipeline. The 15-second heartbeat answers "is the connection still carrying frames?", not "is the host still doing the work we ask of it?".

Impact

Blocks work completely. The user cannot send a message and cannot queue one, and nothing tells them the message was lost.

Version or commit

0.1.1

Environment

macOS 27.0.1, arm64. Observed against a host serving a 25.5 GB statev2.sqlite under load 68-116 on 10 cores. The client-side behaviour is load-independent: any request the host does not answer is affected.

Logs or stack traces

Measured on the affected host, from the server's own trace log over a 10-minute window:

1MCP tool calls: 1,857 in ~10 minutes (peak 506 in one minute)
2DB lock waits > 100 ms: 1,608 (518 seconds of lock wait in that window)
3correlation calls/min vs
4 total lock wait/min: r = 0.751

The reply to the command was stuck behind a database write lock that waited up to 6,622 ms. The command itself was accepted quickly - ws.rpc.orchestration.dispatchCommand p50 5 ms, and every command that started also ended (43/43). The reply path is what stalled.

Attachments

A prepared fix is in TBD: a request deadline with a named EnvironmentRpcTimeoutError, a local composer queue, and a stall tracker that reports the longest-outstanding command.

  • #17765 - server-side HTTP handler timeouts. The client side of the same gap; that issue is about the server bounding its handlers, this one is about the client bounding its requests.
  • #18155 - keeps a query pending across a session handoff. The opposite of a timeout.
  • #9318 - the same onSend guards swallow a pending-user-input answer. Closest existing report of the guard behaviour, but for a different path.
  • #11673 - a client-side queue for messages sent while a turn is running on the server. That queue holds a message the server has already accepted. This issue is about a message the client could not send at all, because the client itself is the thing that is busy.
  • #21539 - a Windows host that disconnects during 36-44 s server stalls, with a 12.5 GB database. Cites the same 15-second no-frame heartbeat. Closed as not reproducible; the diagnosis there was about the stall source, not about the client being unable to see it.
  • #22212, #22213 - reword the ping-timeout disconnect reason and report the server's own event-loop stall on reconnect. Both are about the socket-level heartbeat; neither adds command-level liveness.
  • #22659, #23000 - the 26.5 GB state database and slow thread-shell/recovery queries. The load that produces the stall this issue is about.