fix(server): route projection reads through the reader connection#23867

open2
main
fix/issue-23687
opened18h agobyJJulius Marminge
View on GitHubTwo-way sync enabledPR last imported 15h ago

Problem

Pure projection reads use the writer connection, so MCP thread listing can fail while another connection holds a SQLite write transaction. On current main, holding BEGIN IMMEDIATE throughout a real MCP t3_thread_list request caused a 5,027 ms wait and OrchestratorMcpFailure.

Change

Route the five projection reads through the existing NodeSqliteClient.readOnly helper. Multi-query reads retain their snapshot transactions, and reads inside an existing write transaction retain its writer connection. Reuses vifar's prepared implementation from #23689, with author credit in the commit.

Scope and approval

Fixes #23687

Triaged bug via the automated issue flow: #23687.

Verification

  • Five regression cases failed on main with expected ['reader'], received ['writer']; all pass with the fix.
  • Final scoped run: vp test run apps/server/src/orchestration-v2/ProjectionStore.test.ts packages/shared/src/nodeSqliteClient.test.ts β€” 2 files, 55 tests passed.
  • Scoped lint and server typecheck passed.
  • Independent Codex and Claude reviews both returned MERGE; both proof audits returned SUFFICIENT.

Evidence: sanitized runtime transcript. Real dev server used an isolated, seeded 55.6 MB database; the browser paired and loaded successfully. The same MCP request was made while an external write transaction remained open until the response:

ProofBefore: 2c8bdf6177After: 7ca41e9da5
HTTP MCP thread list under write lockFailure after 5,027 msSuccess after 54 ms
Server transaction traceWriter routing; contended transaction failed after 5,013.65 msReader transaction completed in 11.66 ms

Server-only change; the capture matrix is empty. No rendering or client-contract changes require screenshots or recordings. The reporter's 25.5 GB macOS workload was not replicated; the connection-routing and lock-contention defect was reproduced on Linux with the real server.

Models: GPT-6.1-Sol and Claude Opus 5.5, through Codex and Claude harnesses in T3 Code.

Checks

Some checks failed

Added 2 commits

  • fix(server): pure projection reads run on the reader connection7ca41e9
  • test(server): pure projection reads run on the reader connectionffcf9a0
Synced from GitHub

Approvability

Verdict: Approved at 7ca41e9

Macroscope's review found this PR approvable β€” This focused server fix routes five pure projection reads through the existing read-only SQLite worker, reducing contention while leaving write behavior and returned data unchanged. It also adds regression coverage verifying each affected operation uses the reader connection.

You can add or adjust custom eligibility rules. Learn more.

Synced from GitHub

Review in Change Stack β†’

No actionable comments were generated in the recent review. πŸŽ‰

ℹ️ Recent review info
βš™οΈ Run configuration
πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between 2c8bdf6177159d8fcd12a897d7c9311f6eb93237 and 7ca41e9da546c518d1b48d7190dca0998bdcf434.

πŸ“’ Files selected for processing: 2
  • apps/server/src/orchestration-v2/ProjectionStore.test.ts
  • apps/server/src/orchestration-v2/ProjectionStore.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

πŸ“ Walkthrough

Five ProjectionStore read operations now use the reader connection. A parameterized test checks that each operation uses that connection.

Changes: ProjectionStore reads

Layer / File(s)Summary
Route reads through the reader connection
apps/server/src/orchestration-v2/ProjectionStore.ts
getRecoveryThreadIds, getThreadRecords, getSettlementCandidates, getShellSnapshot, and getThreadShell now run through NodeSqliteClient.readOnly. Existing result mapping and error handling remain unchanged.
Test reader connection selection
apps/server/src/orchestration-v2/ProjectionStore.test.ts
A parameterized test uses a file-backed SQLite database and span attributes to verify that all five reads use the reader connection.

Priority: ⬆️ High

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix Β· Severity of issue fixed: High

Possibly related issues

  • Issue 23378: This change routes getShellSnapshot through the reader, but does not change the desktop bootstrap retry budget.
  • Issue 21840: This change routes five ProjectionStore reads through the reader; the getThreadSnapshotWindow path described in that issue remains outside this change.

Suggested reviewers: t3dotgg

Merge Risk: βšͺ Minimal Β· up to 7ca41

The reads can use the reader connection without a demonstrated new failure path. No merge-blocking issue remains after normal checks.

Pre-merge checks: Passed 4
Check nameStatusExplanation
Title checkPassedThe title "fix(server): route projection reads through the reader connection" matches the main change. The raw summary shows five projection-store read methods now use NodeSqliteClient.readOnly, which…
Description checkPassedThe description follows the template and covers the problem, change, scope and approval, and verification. The Problem section explains that pure projection reads use the writer connection, which make…
Linked Issues checkPassedIssue #23687 asks that five pure ProjectionStore reads use the reader connection instead of the writer. The diff at head (versus merge base 2c8bdf6177) applies NodeSqliteClient.readOnly to all five: g…
Out of Scope Changes checkPassedThe whole-PR changes touch apps/server/src/orchestration-v2/ProjectionStore.ts and its test file, ProjectionStore.test.ts. The ProjectionStore.ts diff adds one import of NodeSqliteClient and wraps onl…
Full details: Title check

Explanation

The title "fix(server): route projection reads through the reader connection" matches the main change. The raw summary shows five projection-store read methods now use NodeSqliteClient.readOnly, which is the reader connection. The title is a conventional commit with a clear scope and a specific action. It does not cover the multi-query snapshot handling or the write-transaction exception, and that detail is not required.

Full details: Description check

Explanation

The description follows the template and covers the problem, change, scope and approval, and verification. The Problem section explains that pure projection reads use the writer connection, which makes MCP thread listing fail under write contention, and it gives reproduction data. The Change section names the five reads, the NodeSqliteClient.readOnly helper, and the snapshot and writer-transaction exceptions. Scope links issue #23687 and the reused prepared work in #23689. Verification gives the failing and passing regression results, the scoped test command with its counts, lint and typecheck results, and before and after runtime evidence. It also states that the 25.5 GB macOS workload was not replicated and that no screenshots are needed. The Scope section does not explicitly state maintainer approval of the direction, and the template asks for that when it is a bug fix linked to an automated triage flow. This gap is minor, so the check passes.

Full details: Linked Issues check

Explanation

Issue #23687 asks that five pure ProjectionStore reads use the reader connection instead of the writer. The diff at head (versus merge base 2c8bdf6177) applies NodeSqliteClient.readOnly to all five: getRecoveryThreadIds (startup recovery query), getThreadRecords (readCanonicalProjection inside sql.withTransaction, used by MCP readThread), getSettlementCandidates (auto-settlement sweep), getShellSnapshot (thread list), and getThreadShell (single shell row). The helper is defined in packages/shared/src/nodeSqliteClient.ts and is unchanged by this PR. Its documented behavior matches the issue: reads run on the reader worker, and reads inside an open write transaction keep the writer. Each read keeps its existing transaction and error mapping. The issue's requirement that writer-bound reads remain on the writer is preserved by readOnly's design. The regression test file ProjectionStore.test.ts is changed at head, but its diff could not be read in this review. The PR description reports five regression cases that failed on main and pass with the fix. The code change alone establishes the routing requirement. Runtime evidence in the PR description is author-reported and was not independently verified here.

Full details: Out of Scope Changes check

Explanation

The whole-PR changes touch apps/server/src/orchestration-v2/ProjectionStore.ts and its test file, ProjectionStore.test.ts. The ProjectionStore.ts diff adds one import of NodeSqliteClient and wraps only the five named reads with NodeSqliteClient.readOnly. The surrounding error mapping and decoding remain unchanged. No unrelated read, write, or schema logic changed in the inspected diff. The test file change is supporting verification for the same five reads, per the raw summary. Its diff could not be read, so its exact contents are not verified here, but it is not a demonstrated unrelated change.

✨ Finishing Touches: πŸ’‘ 1
πŸ“ Generate docstrings
  • Commit to this branch
  • Create a new PR
πŸ§ͺ Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
πŸ› οΈ Fix failing CI checks: πŸ’‘
  • Commit to this branch
  • Create a new PR
  • Autofix Β· Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Synced from GitHub

Thread transfer impact

βœ… Thread transfer remains within every enforced ceiling.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire5.0 KiB5.0 KiBβˆ’7 B (βˆ’0.1%)6.8 KiBβœ…
CodexThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiBβœ…
CodexLive turn WebSocket wire1.2 KiB1.2 KiBβˆ’7 B (βˆ’0.6%)2.0 KiBβœ…
CodexLive turn WebSocket decoded20.9 KiB20.9 KiB0 B (0.0%)29.3 KiBβœ…
CodexLive turn messages220 (0.0%)8βœ…
ClaudeTotal thread wire5.0 KiB5.0 KiB0 B (0.0%)6.8 KiBβœ…
ClaudeThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiBβœ…
ClaudeLive turn WebSocket wire1.2 KiB1.2 KiB0 B (0.0%)2.0 KiBβœ…
ClaudeLive turn WebSocket decoded21.2 KiB21.2 KiB0 B (0.0%)29.3 KiBβœ…
ClaudeLive turn messages110 (0.0%)8βœ…

Baseline: 2c8bdf6 Β· PR result: 7ca41e9 Β· Source CI: failure

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Synced from GitHub

Thanks for taking this up, and for crediting me β€” glad it landed somewhere useful.

Two things from my side, both optional.

1. A companion test that isn't in this PR. The five cases here assert that the read runs on the reader connection. The case that actually matters in production is one step stronger: the read has to finish while a writer is holding the write lock. That is one file and one commit, and it passes on your head unchanged:

1700dd5263 β€” branch perf/lock-wait-test-for-23867 in the fork, so it cherry-picks as a single commit.

The writer holds its transaction open for the whole read, so the overlap is guaranteed by construction rather than by timing. With the lock held, the read completes on the reader in ~13 ms and the same read on the writer cannot acquire a connection at all:

1MEASURE read-on-reader: connection=reader lock_wait_ms=0.026 lock_held_for_ms=11 uncommitted_rows_visible=0
2MEASURE read-on-writer: outcome=TimeoutError lock_held_for_ms=3002 connections_acquired=0

2. A typecheck trap, in case you port anything else of mine. My version of the routing test needed twelve typecheck fixes, and the reason they went unnoticed locally is worth passing on: vp run --filter @t3tools/server typecheck matches no package and exits 0, so it reports nothing at all. The package is t3; cd apps/server && npx tsc --noEmit is the command that actually runs. I checked your head β€” 7ca41e9da5 is clean under it, with no errors and no warnings, so this should not affect anything you have here.