perf(server): sending, editing and steering queued messages no longer read a long thread's whole history#29521

Approvability

Verdict: Approved at 02b32d9

Macroscope's review found this PR approvable — Thread-history reads become more selective, with supporting indexes and settled-node filtering for steering and Stop. The migration adds indexes without changing stored customer content or product defaults.

^{You can add or adjust custom eligibility rules. Learn more.}

Review in Change Stack →

No actionable comments were generated in the recent review.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Baseline: 0847a44 · PR result: 02b32d9 · Source CI: success

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Reporter of #29468 here. I compared the command projection-read sets on fresh copy-on-write clones of the affected 4.2 GB database, using PR head 02b32d9 with migration 078 applied. Send, edit and promote improve substantially. Cancel improves less, and acknowledge and one status-only background read stay expensive.

Projection-read timings (first read on fresh clones, live page layout; v0.1.6 vs this branch):

Thread shape: about 25k turn items (most of them command_execution, dynamic_tool and reasoning), 17k nodes, 2k runs and 5.9k messages. On the real data, plans use all four new indexes. Old and new SQL return identical rows and order for items, nodes and provider threads. Message order is also identical for all 2,024 user messages, 852 of which have no user_message item. For dispatch, page misses fell from about 90k to 21.6k, about 76% fewer.

Still open, with suggested fixes (line numbers at 02b32d9):

1. queued-run.cancel and completion-delivery acknowledge still read every node (about 2.2 s for cancel's node read). dispatchQueuedRunCancel (Orchestrator.ts:9675) and the acknowledge/dispose path (Orchestrator.ts:2446) call getThreadRecords with nodes and no omitSettledLeafNodes. Cancel also loads no user_message items, so the ordering query still runs, now bounded by the index (about 0.2–0.3 s). Cancel is called in loops from thread.settle (:3316) and the stop path (:10798). omitSettledLeafNodes: true looks safe here, because cancellation reads root nodes. Cancel could also omit settled subagents. Acknowledge/dispose must keep the target task and any tasks needed to rebuild the remaining delivery message.
2. The turn-item status filter can't use the active-item partial index. turnItemMatch emits status IN (SELECT value FROM json_each(?)) (ProjectionStore.ts:3036), which never matches migration 073's WHERE status IN ('pending','running','waiting'). Status-only reads therefore walk the whole thread. Idle compaction (IdleCompactionService.ts:53) did this 50 times in about 80 minutes, taking about 4.2 s each in the first-read benchmark. Emitting the literal list when the filter is the active set brings that read to about 25 pages and 4 ms.
3. Messages by role scan the thread's whole message history. At about 0.7 s, this is the largest remaining read on send. role IN (SELECT value FROM json_each(?)) (ProjectionStore.ts:3507) can't use the role = 'user' partial index, so the planner walks messages_thread_created_idx. Two options: a literal role = 'user' for the user-only filter, keeping the existing behavior for other filters, or a (thread_id, role, created_at) index.
4. Smaller reads that still scale with history: runs, attempts, provider turns, subagents and context transfers, about 0.14–0.3 s each on the first read. Context transfers are read in full (ProjectionStore.ts:3608), although send's transfer checks only need pending ones. A caller-specific pending filter could help; merge-back and finalization still need historical transfers.
5. Migration 078 took 59.7 s on the live-layout clone, versus about 5.2 s on a warm compacted copy. The cache was not forcibly reset. A startup progress note may be useful.

Other measurements:

• Commit (EventSink.commitCommand) takes about 7 ms.
• Effect decode of what a send loads takes about 0.1–0.15 s.
• The new indexes add a few µs per upsert on this data.

Roughly half of the measured reader time was outside the three commands compared above. The heavy background readers were idle compaction, MCP loadCaller, startNextQueuedRun and subagent finalization. With 4 reader slots, the traces show 111 s of slot waits, up to 14 s for a single read. That may be worth a separate issue.

Another data point from a much bigger thread: about 41k turn items / 37k nodes, in a 28 GB database. On 0.1.7-nightly.20261103.4119 one send to that thread took 354 s under host load. Roughly: message-order lookup 97 s, provider threads 78 s, nodes 65 s, turn items 63 s, messages 20 s, subagents 18 s.

The worse part is the knock-on effect. While that read was open the WAL couldn't checkpoint, and the writer sat waiting 335 s, so every other thread froze with it. These are exactly the queries this PR rewrites, so we'd really like to see it land.

Heads up: it conflicts with main now, since migration 078 is already taken there.

Another data point from a much bigger thread: about 41k turn items / 37k nodes, in a 28 GB database. On 0.1.7-nightly.20261103.4119 one send to that thread took 354 s under host load. Roughly: message-order lookup 97 s, provider threads 78 s, nodes 65 s, turn items 63 s, messages 20 s, subagents 18 s.

The worse part is the knock-on effect. While that read was open the WAL couldn't checkpoint, and the writer sat waiting 335 s, so every other thread froze with it. These are exactly the queries this PR rewrites, so we'd really like to see it land.

Heads up: it conflicts with main now, since migration 078 is already taken there.

You continue using the thread after so much slowdown? How?

It's an automated pipeline thread: subagent reports and scheduled tasks post to it (about 260 sends in 31 h), so nobody sits waiting on the composer. Most of those sends are much faster than that one, about 2,500 s of history reads across the 260; the 354 s send was the worst, under heavy host load. What actually hurts is the knock-on effect: while one of those reads is open the writer is blocked, so every other thread on the server freezes with it.

In case it helps get this lookup fixed sooner: #30873 adds the same index on the message-ordinal statement, with a json_valid guard so the index build and writes don't fail on a row whose payload isn't valid JSON. It is green and mergeable on current main (migration 081). Happy to close it if you'd rather land this PR instead.