statev2.sqlite grows without bound (33 GB in 7 weeks): retention/compaction for orchestration events and turn items#28903

Second data point, macOS, build of main ac8dce0cc3 (v0.1.6), one heavy user, ~11 weeks of history.

• statev2.sqlite was 15.06 GB; VACUUM INTO brought it to 9.74 GB. The 4.6 GB freelist came from the 6-hourly event compaction (#21154): it deletes superseded events, but on a database created before auto_vacuum=INCREMENTAL became the default, nothing hands those pages back. After the vacuum (with auto_vacuum=INCREMENTAL), DatabaseStorage.reclaim keeps the freelist at 0.
• Live data still grows about 1.5 GB a week. Over the week before the vacuum, page_count stayed at 3,676,931 while 357k free pages were reused.
• dbstat on the 15 GB file: orchestration_events 4.7 GB (1.07M rows) plus 1.1 GB of event indexes; orchestration_v2_projection_turn_items 2.1 GB; orchestration_v2_projection_nodes 0.9 GB.
• Event payload bytes: turn-item.updated 1.86 GB, node.updated 0.69 GB, provider-turn.updated 0.15 GB. Compaction keeps the first and the latest turn-item.updated for each item, so a finished tool call's output is stored twice: in the projection row and in the latest event.
• The largest single category is Pi read tool output: 660 MB in turn items. Of that, 325 MB is 961 image reads (base64 PNG screenshots, up to 3.8 MB each), and the events hold the same ~331 MB again.
• By thread origin: delegated-task child threads hold 55% of event bytes, MCP-launched threads 26% and user threads 19%.

Retention that would bound this for us while keeping settled and archived history readable: purge deleted threads (#17845); once a thread is settled, drop the item events whose final state the projection rows already hold; and store large tool-output and image blobs once, by reference, instead of inline in both the event and the projection.

Data point on 0.1.7-nightly.20261102.4093 (Linux x86_64, headless, systemd user service), measured 2026-11-03. statev2.sqlite is 12.83 GB, the oldest deleted-thread event I saw is from about three weeks ago, and it's already been through t3 doctor (auto_vacuum 2, freelist_count 0).

Supersession compaction is already near its floor here:

• There are 611,805 turn-item.updated events for 465k turn items, and 759,045 node.updated events for 460k nodes.
• Those two event types hold 1.86 GB and 1.08 GB of payload_json characters.
• The last six completed compaction runs removed 674–7,674 events each, under 0.5% of the log, so more supersession alone won't bound this.

Turn item payload by type (characters):

Most of the data isn't recent: 69% of turn items belong to threads with no turn-item update in the past 7 days (by row count, not bytes).

Space inside allocated pages: orchestration_events had about 1.8 GB (29%) unused about an hour after the doctor run. With rows this size on 4 KiB pages, I suspect little of it can be recovered, which fits the 3.42% reduction in the rehearsal on #17845.

For anyone else measuring this: on SQLite 3.37.2, dbstat with aggregate=1 wrapped its size columns at 32 bits, so tables over 4 GiB came back too small or negative. Per-table totals only added up to the file size once I added 2^32 back for the two wrapped tables.

The time t3 doctor took on this database is in #29941.

Another data point: macOS, t3 v0.1.7-nightly.20261103.4119, one user. I measured it on 2026-11-03 with read-only queries.

• statev2.sqlite is 18 GB: 4,705,290 pages of 4 KiB. freelist_count is 0 and auto_vacuum is 0 (NONE), so VACUUM gets back no space.
• orchestration_events has 1.58M rows.
• I took a sample of the last 100k events (sequence > 1480000):

• At this rate the event log grows by about 1.8 GB for each 100k events.
• A full dbstat scan did not finish in 5 minutes. The T3 server restarted 2 times while the scan ran. I did not confirm the cause, but #23482 describes the same lock pattern.

On this machine the database fills the disk faster than any other app data. I support retention for settled threads and storing large tool output once.

Another data point: Linux x86_64, 0.1.7-nightly.20261103.4106, one user running ~5–8 agent threads in parallel most of the day. Read-only aggregates only.

• statev2.sqlite: 39.7 GB (9,689,297 pages of 4 KiB), freelist_count 0, so VACUUM won't shrink it.
• Growth: 18.5 GB on 10-20 → 39.7 GB on 11-03, about 1.5 GB/day.
• By table (dbstat):

• Rows: 5,237,656 events, 1,221,385 turn items.

Events are ~51 % of the file and the four event indexes another ~10 %, so retention or compaction of settled threads' events would cover most of it.

Another data point: Linux x86_64, server 0.1.4-nightly.20261027.3796 (systemd user service, since updated to 0.1.7-nightly.20261104.4151), one heavy user with about 30 concurrent Claude sessions. All numbers come from read-only queries on 2026-11-04.

• statev2.sqlite is 40.7 GB: 9,944,311 pages of 4 KiB, freelist_count 0, auto_vacuum 0. VACUUM would reclaim nothing.
• orchestration_events: the highest rowid is 6.84M. Using rowids as a proxy, about 1M events were written on 10-08, 3M by 10-17, 5M by 10-22 and 6M by 10-27. That is roughly 100k events a day lately.
• Event types among the 40k most recent events: turn-item.updated 19,010, node.updated 13,782, provider-turn.updated 2,938, subagent.updated 1,183, message.updated 1,128.
• Average payload_json size is 3.3 KB; the largest is 236 KB.
• Other tables, by rowid: orchestration_v2_projection_turn_items about 1.26M, orchestration_v2_projection_nodes about 892k, orchestration_command_receipts about 1.0M.
• There are 10,985 v2 threads, of which 10,970 are not archived.
• The effect on 0.1.4: primary-key reads reached a p99 of 15 s and a maximum of 57 s, and the server froze for 20–40 s (see #26224). On 0.1.7 the p99 dropped to under 80 ms, but the database is the same size and still growing.

Another data point, plus a question about what's left after #29806.

Setup. macOS (Mac Studio, 32 GB RAM), 0.1.7-nightly.20261103.4138, one heavy user running many agent threads in parallel. All numbers are read-only, aggregate-only sqlite3 -readonly scans taken while T3 was running.

Size and growth

• statev2.sqlite: 33.4 GB (8,148,267 pages × 4 KiB), auto_vacuum = 0, freelist ~0.12 GB. It grew 2.2 GB in the last ~18 h. The disk has 12 GiB free (98% used).
• Retained event JSON per week: 0.48 → 1.87 → 2.79 → 3.76 GB, with ~1,900 new threads a week. 99.4% of the 6,117 threads were active in the last 30 days.
• Breakdown from a dbstat pass at 29.3 GB:

#29806 should take care of most of the first row. It wasn't in my build yet, so I'll post before/after numbers once it has run. Here's what I think it leaves, measured on this DB:

1. Deleted threads keep their rows. Two threads deleted here still held 542,376 events (1.02 GB of event JSON) and ~0.42 GB of projection rows, with identical counts before and after a completed compaction pass. #29806 will prune their old events like any others. As far as I can tell from main, though, nothing deletes their projection rows (#17845).
2. Large tool output dominates the projections. Turn-item projection rows ≥ 4 KB hold ~2.8 GB, about 71% of that table's payload. In the event log, rows ≥ 4 KB are 7% of rows but 38% of the bytes. #28167 (keep large tool output out of SQLite) targets exactly this.
3. Freed space can't be reclaimed on a full disk. Pruning puts pages on the freelist. On an auto_vacuum = NONE database, getting them back needs t3 doctor, which runs a full VACUUM with T3 stopped and needs roughly the live data size in free space. On a nearly full disk, that's the one moment it can't run.
4. No way to act now. The prune only deletes behind a mark that is at least a day old. doctor runs the same pruneEventStore(), so on a database without a mark it prunes nothing and then VACUUMs all 33 GB.

Small things that would help, smallest first

• An on-demand prune, e.g. by event age, so a user with a full disk can start it right away. I believe the snapshot fallback for resumes already covers the client side, but you know that path better.
• Letting doctor write the compacted copy to another volume (VACUUM INTO <path> plus a swap), so reclaiming doesn't need headroom on the same disk. A further step could reuse T3's remote layer: the compacted copy, or a vault file, could go to a volume on another of the user's environments (LAN, Tailscale or T3 Connect). As far as I can tell, nothing moves database files between environments today, so this would be new. It would fit as a later step on top of the local-path option, in line with the "remote ready" model.
• Deleting a thread removes its projection rows, or at least the deletion dialog doesn't imply it frees space.

A design question for later: moving inactive history out of the live DB. For example, into a separate file that stays searchable and restorable, so the live DB holds only active work. On this DB it would free about 1.1 GB today: settled threads idle for 7+ days hold ~20% of the projection payload, and threads idle for 30+ days hold almost none. Here the growth comes from volume. On installs that run for many months, where history accumulates, it would matter more. Is that a direction you'd want, or is prune + external tool output + deletion purge the intended model?

Adding a data point: our state database is now 24.4 GB (decimal), 8.6 days after a fresh start, averaging roughly 2.8 GB/day.

An earlier read-only breakdown at 23.0 GB showed:

• 60.6% streamed-event history, 37.8% saved thread data and supporting tables, and 1.6% other storage (shares include indexes).
• Of the payload attributable to threads, 54.7% belonged to child/subagent threads and 45.3% to top-level threads.
• Long tool output and screenshots were stored twice: in events and saved thread content. Tool output accounted for about half of stored payload.

Deleting or archiving threads does not free database space on our build.

We were on 0.1.4-nightly.20261027 and moved to v0.1.7 today for event pruning.

Is thread retention (forgetting old threads) planned, and is there a recommended interim practice?

Posted on behalf of Lars by Claude Opus 5.5 on T3 Code.

The one-page reclaim loop reads and writes far more than it frees

Separate from the retention gap above: on a database that already has auto_vacuum=INCREMENTAL and a large freelist, DatabaseStorage.reclaim (apps/server/src/persistence/DatabaseStorage.ts, still the same on main at d5fc29fe5cb6 and in 0.1.8-nightly.20261106.4288) is itself the steady disk load.

The comment says the driver steps once per statement, so the loop calls PRAGMA incremental_vacuum(1) up to 1024 times a minute, with a PRAGMA freelist_count before each one. That premise is wrong for node:sqlite's DatabaseSync, which is what the server uses. One exec("PRAGMA incremental_vacuum(N)") frees N pages in a single statement.

Measured on Node's own DatabaseSync (Linux, Node with --experimental-sqlite), on a throwaway 8,030-page database with 4,000 free pages, nothing else touching it:

So the per-page loop is about 90× the work of one statement for the same pages, and it never catches a freelist of hundreds of thousands of pages: 1,024 pages a minute is roughly 4 MB a minute, about 6 GB a day.

On a live 5.7 GB database (1,486,689 pages, 801,554 of them free, auto_vacuum=2, server 0.1.8-nightly.20261105.4234), the server's own trace for a quiet stretch was 16,200 sqlite spans, of which 8,064 were incremental_vacuum(1) and 8,071 were freelist_count. The other 65 spans were ordinary lookups. The same shape holds across the other rotated trace files: several thousand vacuum statements per file, each 1–9 ms.

incremental_vacuum relocates a live page from the end of the file into the free page and truncates. With 800k free pages scattered through a 5.7 GB file, each of those statements is a random read plus a write, and the process's own /proc/<pid>/io confirms the volume. The t3 serve process (up since 5 Nov, never restarted for this) read and wrote:

The cgroup's own io.stat is lower (6.7, 15.2 and 6.6 GB read on 4, 5 and 6 Nov), so some of the process figure is re-read of pages the kernel counted once. Either way it is orders of magnitude above the 4 MB a minute the loop actually frees.

Two one-off minutes are much larger than the loop explains and look like a scan: 62 GB read in the 04:14 minute on 6 Nov, and 2.7 GB in the 02:21 minute. The six-hourly compaction (pruneEventStore) walks orchestration_events even when it deletes nothing, and no trace file survives from those minutes, so I can't attribute them more tightly.

Suggested change: one PRAGMA incremental_vacuum(1024) per pass instead of 1,024 statements of one page. That keeps the one-minute bound and the yield between passes, and frees the same pages in about a ninetieth of the work. Capping the freelist the loop will chase (or only reclaiming right after compaction frees pages) would stop it paging a multi-GB file all day for 4 MB of progress.

Happy to retest a build. I have not restarted the server or vacuumed the live database.

Another data point, with one requirement that hasn't been stated yet.

• Host: Linux team host, t3 0.1.8-nightly.20261106.4288 (auto-updated today), one heavy owner running many concurrent agent threads, plus remote environments.
• Size: statev2.sqlite is 40.8 GB (40,791,117,824 bytes; file size only, no database reads). It is the largest single item on a 344 GB root filesystem that is 83% full. T3 worktrees under ~/.t3/worktrees add another 41 GiB on the same disk.

Requirement: retention and space reclaim have to work while the server is busy. On this host the server is never idle. There's always an in-flight turn, a delegated child, or a scheduled task. So any fix that needs "stop the server, then prune/compact" (a stopped-server t3 db compact, or t3 doctor, which pauses and restarts the service) can't be scheduled here. What would work:

1. Online retention that prunes events for archived/settled/deleted threads, and events older than a window, in small bounded transactions that never hold the write lock long enough to stall the event loop (#30065, #23482).
2. Incremental reclaim that is rate-limited and bounded per step, instead of the per-page loop described in the latest comment.
3. A prune and reclaim pass tied to the restarts T3 already does for its own updates. Those restarts happen anyway, so a bounded pass there costs no extra downtime. It would also help #21715, since update copies would be smaller.
4. A read-only t3 db stats that reports size, prunable bytes and the freelist without users running ad hoc sqlite3 -readonly scans.

Happy to share more aggregate numbers on request. No row contents or private data were read.

Correction and growth rate for the data point above: the 40.8 GB figure was measured earlier today. It is now 41.5 GB (41,546,035,200 bytes) about five hours later, so roughly +0.75 GB in 5 h (~3.6 GB/day) under normal concurrent use. File size only.

@ViperJuice You have great suggestions!

Mine grew 15GB more in just a few days. I can't keep this up. Looks like it's pushing the entire chat history for every update or something because it's way too big.

I'm strained on DB size atm because I can't buy more SSDs. On top of that, my worktrees are already causing storage constraints. I wish T3 Code worktrees were hardlinks instead of copies of files :/.

#32941 merged in 8969e1d8d0433be9e598d122431c54c69e0aa83c. It partly addresses ask 3 here: return the space after pruning.

Databases created before automatic shrinking keep the space that history cleanup frees. Settings > Maintenance on web and desktop now has a Reclaim button for them. It runs the t3 doctor repair, which also turns on automatic shrinking, and then the desktop app or background service starts T3 Code again. Servers started by hand and the mobile app still show the t3 doctor command. It does not change how fast live data grows.

Verified on a seeded copy of real data: the file went from 1.40 GB to 144 MB, auto_vacuum went from 0 to 2, and all 87 threads were kept. CI passed on the merged head.

Another data point, from read-only dbstat (mode=ro) on a Linux host running t3 v0.1.6 as a systemd service. One user runs many concurrent Claude desks.

• statev2.sqlite is 8.0 GB with 0 free pages, so VACUUM would reclaim nothing.
• Breakdown: orchestration_events 4.2 GB plus ~1.0 GB of indexes; projection_turn_items 1.4 GB; projection_nodes 0.4 GB. That's ~92% of the file.
• Growth: under 10 MB/day until the desks moved into T3 on 4 Nov, then 360–405 MB/day, then 943 MB of event JSON on 8 Nov (433k rows). About 1.5–2 GB/day on disk with indexes.
• Over the last 24 h, turn-item.updated was 441 MB of 135k events and node.updated was 126 MB. Streamed tool calls and command output are stored as a full snapshot on every update. dynamic_tool (483 MB) and command_execution (310 MB) are the largest turn-item payloads.

What would help me most, in line with the asks above:

1. Keep only the latest snapshot per turn item and node, or compact *.updated events once the projection is current.
2. Cap stored tool and command output size, or make it configurable.

Is the history cleanup that pairs with the new Reclaim button in a release I can update to, and does it cover turn items as well as events?

Another data point, macOS, 0.1.8-nightly.20261109.4429: statev2.sqlite is 61.6 GB and filled the disk today (119 MB free).

From dbstat and pragmas, read-only:

So about half the file is freelist that is never returned to the disk (see also #32952). The database has 7,006 threads, 6,455 of them subagent threads; #33578 is about those staying unsettled.

Follow-up data point and a proposal: keep the history, but move it out of the hot database.

Growth on this host: 33 GB (Nov 1) → 41.5 GB (Nov 7) → 54.9 GB (Nov 10) → 56.6 GB (Nov 11), roughly 4–4.5 GB/day. freelist_count is 0 and auto_vacuum is 0, so the new Reclaim button (#32941) has nothing to return here, and running its t3 doctor repair on 55 GB would be a long outage (#29941).

Requirement: for us, retention by deletion isn't acceptable. Thread history is a record we go back to. What we need is a size bound that keeps the data.

Proposal: a cold-storage tier for threads.

1. Archive out of the hot DB: threads that are archived, or settled and untouched for N days (configurable), are exported to one compressed bundle per thread (its events and projections) under a configurable directory. That directory could be on a cheaper, slower disk.
2. Keep a stub: the hot DB keeps a small row per archived thread (title, project, dates, size, bundle path and checksum), so lists and search still find it.
3. Rehydrate on demand: opening an archived thread re-imports it, or reads it read-only from the bundle.
4. Online and bounded: export and import run in small transactions while the server is busy, with a dry-run report of what would move and how much space it frees.

Side effects: a small hot DB means faster startup, faster pre-update copies (#21715, #32952), and less exposure to the write stalls in #23482 and #30065.

Same here on the stable desktop app (0.1.3, macOS): statev2.sqlite hit 15.4 GB in about a month.

• orchestration_events is 7.8 GB (2.8M rows) plus 1.8 GB of indexes
• projection_turn_items is ~3.5 GB, projection_nodes ~0.9 GB
• freelist is only ~100 MB, so VACUUM doesn't help
• almost none of my threads are deleted, so it's live history growing, not leftovers

Most new events are still turn-item.updated, then node.updated. A retention setting or a t3 db prune command would really help, since there's no safe way to shrink it right now.

Thanks for writing this up. It asks for new behaviour rather than reporting a defect, so it does not belong in the bug tracker and I'm closing it as not planned here. Feature ideas are discussed in Discussions: please post it there.