# \[Bug\]: the other ProjectionStore reads take the write lock instead of the reader connection · Issue \#23687 · pingdotgg/t3code [View on GitCafe](https://git.cafe/pingdotgg/t3code/issues/23687) Repository: [pingdotgg/t3code](https://git.cafe/pingdotgg/t3code) Visibility: public State: backlog Priority: 0 ## Description [Discussion and activity](https://git.cafe/pingdotgg/t3code/issues/23687) --- Use this form for broken behavior, regressions, crashes, or reliability problems. Feature requests belong in [Discussions](https://github.com/pingdotgg/t3code/discussions/categories/ideas). Search existing reports, discussions, and documented workflows before starting work. - [x] I searched existing issues and did not find a duplicate. - [x] I included enough detail to reproduce or investigate the problem. ### Area apps/server ### Steps to reproduce 1. Run a host whose state database is large and whose writer is busy. On the host this was measured on, `statev2.sqlite` was **25.5 GB**, the load average was 68-116 on 10 cores, and the server's event loop was stalling for 2-24 s at a time. 2. Serve MCP reads against that host. `OrchestratorMcpService.readThread` goes through `ProjectionStore.getThreadRecords`; the thread list goes through `getShellSnapshot`; the shell row for one thread goes through `getThreadShell`; startup recovery goes through `getRecoveryThreadIds`; the auto-settlement sweep goes through `getSettlementCandidates`. 3. Watch the `db.connection` and `db.lock_wait_ms` attributes the server records on its statement spans. ### Expected behavior A pure read should not take the write lock. The server already has a reader connection - `nodeSqliteClient` starts a reader worker for a file-backed database, and `NodeSqliteClient.readOnly` routes a read to it - and WAL lets a reader read the last committed snapshot while a writer holds the lock. ### Actual behavior All five of those reads run in a plain `sql.withTransaction`, so they run on the **writer** connection. The writer is a synchronous `DatabaseSync` on the main thread, so a read that takes the write lock blocks the event loop and every write queues behind it. **This is the same defect as #21840, and this issue covers the reads #21869's scope defers.** A triage note on #21840 named four `ProjectionStore` reads that still default to the writer on main, measured on a 36.7 GB `statev2.sqlite` with 8-12 second event-loop stalls each, and offered to file them separately. They are filed here. The MCP path is the one that bites hardest. Measured on the host above, from the server's own trace log over a 10-minute window: ``` MCP tool calls: 1,857 in ~10 minutes (peak 506 in one minute) DB lock waits > 100 ms: 1,608 (518 seconds of lock wait in that window) correlation calls/min vs total lock wait/min: r = 0.751 sqlite ops by connection: 18,408 (writer, read) <-- reads also go through the writer 234 (writer, write) sql.transaction by connection: writer n=15,839 mean=35.9ms reader n=8 mean=497.7ms ``` **78.5% of every SQLite query in that window ran inside an MCP call.** The worst single lock wait was **6,622 ms**, on: ```sql SELECT payload_json FROM orchestration_v2_projection_nodes WHERE node_id = ? ``` and seven transactions waited 6,326-6,554 ms. The reader connection was barely used: **8 transactions in the whole window, averaging 497.7 ms** - the reader pool starved exactly when the writer was busy, which is the only time it matters. ### Independent reproduction Reproduced independently on 0.1.2-nightly.20261022.3526 (`f054d1b`), macOS 27.0.1, M4 Pro, against a 12.1 GB `statev2.sqlite`, benchmarking the same query on both connections with the same rows out: | | query | event-loop delay | worst probe RTT | worst queued-write delay | writes committed during the read | |---|---|---|---|---|---| | writer (today) | 13,163 ms | **13,170 ms** | 13,419 ms | 13,433 ms | **0** | | reader | 13,374 ms | **12.7 ms** | **1.5 ms** | **11.7 ms** | **52-54** | One detail from that reproduction corrects the framing above, and it widens it: the captured writer transaction ran for 9,841 ms with `db.lock_wait_ms = 0.001` and a 9,133 ms event-loop delay in the same window. The delay was **inside the synchronous query**, not in acquiring the connection. The writer is a `DatabaseSync` on the main thread, so it stalls the event loop even when it never waits for the lock - the lock wait is not the only path to the stall. ### Impact **Major degradation or frequent failure.** The read path is the server's hottest read. Under write load it queues behind the writes, and the reply to an MCP caller is delayed by the lock wait - which is what turns a slow host into a client that appears to hang. ### Version or commit 0.1.1 ### Environment macOS 27.0.1, arm64. Observed on a host serving a 25.5 GB `statev2.sqlite` under load 68-116 on 10 cores. The triage note on #21840 measured the same reads on a 36.7 GB database, with 8-12 second event-loop stalls. ### Logs or stack traces The `db.connection` and `db.lock_wait_ms` attributes are the measurement. The counts above are from the server's own trace log. ### Attachments A prepared fix is in #23689: the five reads are wrapped in `NodeSqliteClient.readOnly`, and the tests show a read completing in 13 ms on the reader connection while a writer holds the write lock, against the same read on the writer connection failing to acquire a connection at all. ### Related issues - #21840 - the accepted, triaged issue for the projection reads taking the write lock. This issue covers the reads its scope and the triage note defer. - #21869 - the pull request that moves the windowed snapshot read to the reader. This pull request covers the other reads, and does not overlap with it. - #23000 - slow thread-shell reads. The same reads, measured from the query cost side; this issue is about the lock, not the query plan. - #23378 - the desktop's fatal HTTP 500 when the backend is busy for more than about 10 s after startup. The same cold-start wait, seen from the desktop's side. - #15971 - the reader pool exhausted by a session scan. The reader pool, starved. - #21799 - checkpointing the WAL without blocking the event loop. The write side of the same contention. - #23661 - command replies stalled by database write-lock waits during MCP activity. The client-side half of the same stall; this issue is the server-side half.