Sending, editing or steering queued messages takes 7-15 seconds in a long thread#29468

What happened

In one long-lived thread, each send takes several seconds before the message appears in the thread. Editing a queued message and promoting a queued message to steer are just as slow. Over the same trace window, these three commands had a median of 6.8 s (p95 10.8 s, n=28) on this thread, against 1.6 s (p95 4.3 s, n=15) on all other threads of the same server.

Usage pattern: the thread is a long-running coordinator, about 8 days old. It delegates work to child threads through the MCP delegate-task tools and receives scheduled wake-up messages. The server holds 734 threads, 704 of them delegated task threads, across 16 projects. Several Claude sessions usually run at once.

Size of the affected thread:

Collection	Rows	Payload
turn items	23,670 (1,085 `user_message`, 3,573 `assistant_message`)	62 MB
nodes	16,237 (15,839 completed)	15 MB
messages	5,445	6 MB
subagents	772	3 MB
provider turns	1,812	2 MB
runs	1,867

statev2.sqlite is 4.2 GB.

Diagnosis

Source references are to the v0.1.6 tag; see the note at the end of this section about main.

All three command paths read the thread's history without a window before they plan. The reads sit on the command's dispatch path, so the user sees the full read time as send latency. Each path uses a different field and filter set (v0.1.6 line references):

message.dispatch: dispatchMessage → getProjectionWithPendingEvents → readCommandProjection → ProjectionStore.getThreadRecords(...) with 12 collections, turnItemTypes: ["user_message","error"], messageRoles: ["user"], omitSettledLeafNodes, omitSettledSubagents (Orchestrator.ts ~L5261, ~L1113-1228; ProjectionStore.ts ~L5382).
queued-run.edit: getThreadRecords(threadId, ["runs","messages","turnItems"], { turnItemTypes: ["user_message"], messageRoles: ["user"] }) (Orchestrator.ts ~L9654).
queued-message.promote-to-steer: getThreadRecords with 10 collections, turnItemTypes: [] (no turn items loaded), no settled node or subagent exclusions (Orchestrator.ts ~L9246), followed by dispatchSteerIntoRun, which reads again.

Each read reaches readCanonicalProjection. When more than one message is loaded, it also runs a separate message-ordering query (ProjectionStore.ts ~L3570).

The trace shows these reads ran on the reader connection (sql.transaction with db.connection: "reader", lock wait ≈ 0 ms), so the cost is the reads themselves, not lock contention. The heaviest statements:

Command (total)	Statement	Time
`message.dispatch` (15.0 s)	ordering: `SELECT json_extract(payload_json,'$.messageId'), MIN(ordinal) FROM ..._turn_items WHERE thread_id=? AND type IN ('user_message','assistant_message') GROUP BY json_extract(...)`	4.95 s
	`SELECT payload_json FROM ..._turn_items WHERE thread_id=? AND type IN (SELECT value FROM json_each(?)) ...`	4.00 s
	`SELECT payload_json FROM ..._nodes AS node WHERE node.thread_id=? AND NOT (...)`	2.46 s
	`SELECT payload_json FROM ..._provider_threads WHERE thread_id=? OR owner_node_id IN (...)`	1.85 s
`queued-run.edit` (8.2 s)	turn items with `type IN (SELECT value FROM json_each(?))` (1,085 matching rows)	5.30 s
	ordering query	2.22 s
`queued-message.promote-to-steer` (8.9 s)	ordering query	3.48 s
	all nodes of the thread	2.13 s
	the follow-up read in `dispatchSteerIntoRun`	2.06 s

EXPLAIN QUERY PLAN for the ordering query: SEARCH ... USING INDEX orchestration_v2_projection_turn_items_thread_run_idx (thread_id=?) + USE TEMP B-TREE FOR GROUP BY. type is not part of that index, so the plan filters every turn item of the thread by type. Most of those rows are large command_execution (25 MB) and dynamic_tool (12 MB) payloads excluded by these queries' type filters. The existing partial index turn_items_user_message_idx ON ..._turn_items(thread_id, ordinal, turn_item_id) WHERE type = 'user_message' does not match type IN (...) or type IN (SELECT value FROM json_each(?)).

Note on main: I only skimmed it and did not compare these exact field sets. dispatchMessage there still starts with getProjectionWithPendingEvents, and readCommandProjection requests the same collections. #28301 removed only the extra rereads after settle, snooze or title updates.

Suggested fix

Drop the separate ordering query on command paths. In v0.1.6 these three handlers do not appear to rely on message array order: queue order sorts runs (QueuedRunOrder.ts), steer resolves messages by ID, completion dedupe checks membership, and latest-user selection sorts by timestamp. Promote can opt out of ordering explicitly. For dispatch and edit, the ordering query takes MIN(ordinal) over both user_message and assistant_message items, while these paths load only user_message (and error) items. Deriving ordinals from the loaded items is safe only when each loaded user message's minimum ordinal is on a loaded user_message item. Scheduled wake-ups and other system-originated user messages are the cases to check, and a test should assert that the derived order matches the current query. Keep the default for other callers.
Make the type filters indexable. For the ordering query, prefer a stored message_id column with a partial index (thread_id, message_id, ordinal) WHERE type IN ('user_message','assistant_message'), rewriting SELECT and GROUP BY to use it. A smaller step is an expression partial index on json_extract(payload_json,'$.messageId'); in a synthetic test (not the bundled SQLite build) that removed the temporary GROUP BY B-tree. The bundled-SQLite benchmark is still open. For the filtered payload reads, type IN (SELECT value FROM json_each(?)) cannot match a partial index predicate. Use literal per-caller type lookups with matching partial ordinal indexes (for example two type = ? lookups), or test a (thread_id, type, ordinal) composite. In that synthetic test the composite helped the ordering query but was not chosen for the json_each form.
Fixes 1 and 2 target about 9 s of the 15 s send, and less of promote. The rest is nodes (2.46 s; the settled-node NOT (...) predicate still reads 15,839 completed nodes) and provider threads (1.85 s; thread_id = ? OR owner_node_id IN (...)). Making the settled-node predicate indexable, and turning the OR ... IN into a UNION or join, would address those.
Bounded command reads, with care. Load only what each command uses, but keep what ordinary sends need:
- a per-thread run-ordinal high-water mark covering all stored runs and pending run events: nextRunOrdinal takes the max over loaded runs (Orchestrator.ts ~L443), so dropping cancelled or rolled-back runs could reuse run IDs;
- the latest completed or handoff source run;
- runs referenced by a continuation (manual continuation requires the latest executed run);
- delegated-parent runs;
- dedupe checks, and the history summaries used for titles.
Load other history only when a command asks for it, and fail loudly when a referenced continuation, steer target or delegated parent is missing from the bounded set. Planning mostly needs IDs, ordinals, roles and statuses, so reading those columns instead of full payload_json on these paths would also shrink the reads.
Optional: a slim per-thread command cache. It must be updated by every committed event, including effect-worker writes (writeIfRunCurrent), which the dispatch lock does not cover. It must also be checked against the event sequence inside the write transaction, and fall back to the indexed read when stale.

Steps to reproduce

Use one thread for a long time, so it collects thousands of runs and tens of thousands of turn items. Heavy tool use helps (many command_execution / dynamic_tool items), and delegated tasks and scheduled wake-ups add runs quickly.
Send a message to that thread. Separately, queue a message while a run is active, then edit it, or promote it to steer.
Compare the delay with the same actions on a short thread. In server.trace.ndjson, look at orchestrationV2.dispatch.once → Orchestrator.dispatchMessage / dispatchQueuedRunEdit / dispatchQueuedMessagePromoteToSteer → ProjectionStore.readCanonicalProjection and its sqlite children.

Synthetic repro: seed one thread with about 1,800 completed runs, about 16k completed nodes and about 24k turn items (most of them command_execution / dynamic_tool with 2-5 KB payloads), then time message.dispatch and queued-run.edit on it.

Version

0.1.6 (stable desktop app; the local server reports serverVersion 0.1.6). Source references are to the v0.1.6 tag. Triage CLI version 0.1.2 (the CLI's only).

Environment

macOS 26 (Darwin 25.6.0), arm64; T3 Code (Beta) desktop app with its bundled local server, connected from the desktop app; Claude provider, several sessions at once; 127,840 turn items across all threads, database migration 76

Evidence

shell

#18438: message sends decode every node of the thread first (closed; an earlier, smaller form of this cost); #21840: opening a long thread blocks the writer (closed; moved snapshot reads to the reader); #24895: history-wide reads on large databases (open); #28301: removes extra rereads during dispatch (merged; does not remove the first read)

Fix applied or workaround

Start a new thread and archive the long one. Restarting the app does not help, because the cost follows the size of the thread's history.

Filed by

Filed with help from Claude Code (Claude Opus 5.5) running the T3 Code triage playbook.

Nnkoynov5d ago

Same cost on 0.1.7-nightly.20261103.4119 (Linux x64, headless server, statev2.sqlite 28.5 GB). The thread is a coordinator that gets scheduled wake-ups and runs subagents: 3,722 runs and 36,791 turn items with 207 MB of payload.

Over 16.8 h of trace on 4119, message.dispatch to that thread: n=103, p50 2.1 s, 77 over 1 s, 23 over 5 s, max 108 s. That one thread has 23 of the server's 43 dispatches over 5 s.

The three slowest (83 s, 106 s and 108 s) are each one readCanonicalProjection in a reader transaction. The turn_items payload read alone took 41–72 s, and heap growth was 100–135 MB per dispatch. After it came nodes (13–23 s) and messages (6–16 s).

One difference from your trace: here the other reads did wait. While those dispatches ran, ProjectionStore.getThreadsWithPullRequests (the minute PR sweep), the projects list and some node reads show db.lock_wait_ms of 30–105 s. So the whole server stalls for as long as the dispatch takes. For example, at 16:43 UTC the dispatch held its read from 16:43:18 to 16:45:03, and the sweep that started at 16:43:24 waited 98.9 s of its 99.5 s.

A second server on the same build has a 3.4 GB database and short threads. Its message.dispatch p90 is 0.98 s (max 1.8 s).

Measured with Claude Opus 5.5 in OpenCode.

HHarshSharma9x93d ago

Data point from a small database: 0.1.7, statev2.sqlite 2.3 GB, 608 threads, the slowest thread only 2.8 MB / 785 turn items. message.dispatch still took 36 s: readCanonicalProjection spent 31.5 s inside the write transaction (turn_items read 16.6 s) during an I/O spike from another process. While another process held a long read, sends took 314–382 s. The same read takes 0.1–0.3 s on an idle host. So the dispatch-path history read turns any host I/O contention directly into send latency, even when thread size is modest.

DDropje972d ago

Same on 0.1.8-nightly.20261106.4275 (Linux x64, background service, statev2.sqlite 45 GB). Two coordinator threads have 3,400–4,100 runs, 21–27k nodes and 31–37k turn items. Over 20 minutes of production slow-read logs, dispatchMessage spent 185 s in reader statements, 112 s of it in ProviderContinuationService delegated-completion wakes. Those hold the thread's dispatch lock while they read, so a user send on the same thread waits behind them.

On a copy of that database, readCommandProjection for one send returns 28,164 rows (4,109 runs, 4,109 attempts, 6,250 nodes, 4,043 provider turns, 4,259 user messages, 2,610 transfers, 1,157 user/error items) and takes 6–8 s. A staged read we now run locally takes 0.7 s, with identical dispatch results:

In one read transaction, keep the full thread, all runs, bound provider sessions and related provider threads. Many decisions use run history in different orders, so we did not narrow runs.
Read by key what the branches consume:
- user messages named by active/target runs, by queued runs and by the command;
- target roots, with the same settled-leaf filter;
- active attempts, plus all attempts of the target runs (restart ordinal);
- provider turns of those attempts and roots, and turns of no attempt when a target's activeAttemptId is null;
- items linked to the command's message;
- the highest user/error item, which keeps nextTurnItemOrdinal's floor;
- for delegated completions, the delivery's tasks with omitSettledSubagents, plus the turns of the run its mailbox message joined;
- pending runtime requests and pending transfers into the thread.
Title seeding: two EXISTS probes instead of loading every user message.
modelSwitchSince: rank eligible runs by runRanAfter and fetch linked user items, in batches of 8, until the first run with a shown item or a handoff.
Fall back to the existing broad read, inside the same snapshot, for:
- manual, usage-limit and restart continuations;
- /goal;
- v1 imports;
- a pending fork or merge into the thread;
- a model or provider change (the thread, its active provider thread or an active run differs from the desired selection);
- a command batch that already holds events for the thread.
startNextQueuedRun works the same way. It also falls back when the latest handoff-source run's instance differs.

Before changing the code we recorded 249 original-flow tests: full ordered events, effects, receipts and errors across the branch matrix, including nested pending events and a race before getNextTurnItemOrdinal. They pass unchanged. One detail from review: when merging item cohorts in JS, break ordinal ties by comparing turn_item_id as UTF-8 bytes (SQLite BINARY), not with JS <.

Produced with Claude Code (Claude Opus 5.5), with Codex (gpt-6.1-sol) for research and review.