fix(server): compact superseded history in the background#19435

merged2

main

t3/db-compaction-1-background

View on GitHubTwo-way sync enabledPR last imported 9d ago

The server never deleted superseded orchestration history, so databases grow without bound. One maintainer database reached 114 GB, most of it repeated whole-payload updates of the same thread, message, node or PR link.

The server now runs compactEventStore one minute after startup and every six hours after that. Each pass deletes events whose state a newer retained event already carries:

Whole-thread events (rename, archive, settle, pin, model changes…): only the newest per thread is kept. Visit and mark-unread events supersede only each other, so a later visit never removes the rename that carries the thread's activity time.
Messages, nodes and PR links: the oldest and newest event per entity are kept. Clients add an entity where they first see it, so keeping the first event preserves message order for a client that reconnects from an older cursor.
thread.created, thread.deleted, runs and turn items are never compacted.

No replay boundary or snapshot fallback is needed: every deleted row has a newer row for the same key, so any replay from an older cursor still ends in the same state and order.

New databases are created with auto_vacuum = INCREMENTAL, and the server returns freed pages to disk online in small yielding batches. Existing databases get that with t3 doctor (#19436). Compaction pages are 50 rows, because a 500-row page with large payloads blocked the shared connection for ~90 ms.

A retried MCP metadata update whose events were compacted now answers from the current thread instead of failing.

Takes over #19249.

🤖 Generated with Claude Code

Fixes #18559

Devin Review

Checks

20 successful · 13 skipped

All checks passed

Ggithub-actionsBot16d ago

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Codex	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Codex	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Codex	Live turn WebSocket decoded	20.9 KiB	20.9 KiB	0 B (0.0%)	29.3 KiB	✅
Codex	Live turn messages	2	2	0 (0.0%)	8	✅
Claude	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Claude	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Claude	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Claude	Live turn WebSocket decoded	21.2 KiB	21.2 KiB	0 B (0.0%)	29.3 KiB	✅
Claude	Live turn messages	2	2	0 (0.0%)	8	✅

Baseline: fa1184d · PR result: e5cbab0 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

Codex decoded thread snapshot: 108.5 KiB
Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

MmacroscopeappBotreviewed · 1 inline thread16d ago

apps/server/src/persistence/DatabaseStorage.tsline 27Outdated · cf8be3bOpen thread

…	…	@@ -0,0 +1,73 @@
	24	+	* retained event with a higher sequence, so a client replaying from an older
	25	+	* cursor still reaches the same state; no replay boundary is published.
	26	+	*/
	27	+	export const layer = Layer.effect(

MmacroscopeappBoton line 2716d ago

This new service owns acquisition, scheduling, and construction, but puts all of it inside Layer.effect rather than defining a make. Could you move the generator into a private const make = Effect.gen(...) and construct the layer with Layer.effect(DatabaseStorage, make)? This keeps the service's dependencies visible at the construction boundary and follows the canonical service-module shape.

^{Posted via Macroscope — Effect Service Conventions}

JJulius Marminge15d ago

Fixed in 8f1be1c: the generator is now a private make and the layer is Layer.effect(DatabaseStorage, make) (carried up through #19437).

MmacroscopeappBot16d ago

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR not approvable — The PR adds default-on background compaction and SQLite page reclamation, including deletion of durable superseded history and changed replay retention behavior. Its persistence and runtime lifecycle impact is broader than a small isolated fix, even with targeted tests covering replay and reclamation.

No code changes detected at e5cbab0. Prior analysis still applies.

^{You can add or adjust custom eligibility rules. Learn more.}

CcoderabbitaiBot16d ago

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration

Configuration used: Path: .coderabbit.config.ts
Review profile: CHILL
Plan: Team
Run ID: de2a88b8-da3a-489a-b0cb-7d198becaf40

📥 Commits

Reviewing files that changed from the base of the PR and between cd8a1ef30fe7df73e7e93e7f89b5c97e3446f7d3 and e5cbab0b0c6d0d9f4fb5ebfa08b317a594da6b5a.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.config.ts
Review profile: CHILL
Plan: Team
Run ID: 4a66dcd7-e1c4-4a7d-b91c-f79b653d3222

📥 Commits

Reviewing files that changed from the base of the PR and between cf8be3ba762b57e81a5f52d41ba8e6de720eb69c and 8f1be1c0a8c8b7c17b345ab685f2ee8a6e1f744f.

📒 Files selected for processing: 3

apps/server/src/orchestration-v2/FoundationPersistence.test.ts
apps/server/src/persistence/DatabaseStorage.ts
apps/server/src/server.ts

Included review availability: This review used your included allowance. 9 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 10 reviews per hour.

📝 Walkthrough

The server now schedules event-store compaction and incremental SQLite page reclamation. Compaction retention rules preserve selected thread and entity events. Metadata updates use the thread shell when dispatch returns no metadata-updated event.

Changes: Event Compaction and Storage Maintenance

Layer / File(s)	Summary
Compaction retention rules `apps/server/src/orchestration-v2/ProjectionMaintenance.ts`, `apps/server/src/orchestration-v2/FoundationPersistence.test.ts`, `apps/server/src/orchestration-v2/runtimeLayer.test.ts`, `apps/server/src/ws.test.ts`	Compaction handles read-state events separately, retains the oldest and newest keyed entity updates, and removes obsolete pull-request-link updates. Tests cover retained events, projection rebuilding, and replay after compaction.
Scheduled database maintenance `apps/server/src/persistence/Sqlite.ts`, `apps/server/src/persistence/DatabaseStorage.ts`, `apps/server/src/persistence/DatabaseStorage.test.ts`, `apps/server/src/server.ts`	New databases enable incremental auto-vacuum. `DatabaseStorage` schedules event compaction and page reclamation, and the server provides its layer. The storage test checks page reclamation and retained table data.

Thread Metadata Update Fallback

Layer / File(s)	Summary
Select the metadata update result `apps/server/src/mcp/ThreadMetadataMcpService.ts`	When dispatch returns no metadata-updated event, the update path retrieves the thread shell. It returns an orchestration error if the selected thread is null.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

mermaid

1	sequenceDiagram
2	participant DatabaseStorage
3	participant ProjectionMaintenance
4	participant EventStoreV2
5	participant Sqlite
6	DatabaseStorage->>ProjectionMaintenance: Run scheduled event-store compaction
7	ProjectionMaintenance->>EventStoreV2: Scan and compact events
8	DatabaseStorage->>Sqlite: Check auto_vacuum and freelist
9	DatabaseStorage->>Sqlite: Reclaim pages incrementally

Suggested reviewers: t3dotgg

Merge Risk: ⚪ Minimal · up to 8f1be

No actionable merge-blocking issue was established in the reviewed changes. The scheduled maintenance and metadata fallback appear ready to merge after normal checks.

Pre-merge checks: ^Passed 3 · ^Failed 1

^Failed Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Description check	Warning	The description clearly explains the database-growth problem and the compaction, retention, vacuum, and MCP changes. It does not include the required section structure, explicit maintainer approval or…	Add the required Problem, Change, Scope and approval, and Verification sections. Include the triaged issue or approval discussion with explicit maintainer approval, and document the focused tests or manual checks performed with their observ…

^Passed Passed checks: 3 passed

Check name	Status	Explanation
Title check	Passed	The title clearly and concisely describes the primary change: background compaction of superseded server history.
Linked Issues check	Passed	`#18559` requires a production caller for `ProjectionMaintenanceV2.compactEventStore`. `DatabaseStorage.layer` resolves `ProjectionMaintenanceV2`, waits one minute after startup, and repeats compaction …
Out of Scope Changes check	Passed	The changes remain connected to `#18559`. The 50-row batches, page reclamation, incremental auto-vacuum setup, replay coverage, and MCP fallback support safe operation after event deletion and address t…

Full details: Description check

Explanation

The description clearly explains the database-growth problem and the compaction, retention, vacuum, and MCP changes. It does not include the required section structure, explicit maintainer approval or scope rationale, or focused verification results.

Resolution

Add the required Problem, Change, Scope and approval, and Verification sections. Include the triaged issue or approval discussion with explicit maintainer approval, and document the focused tests or manual checks performed with their observed results and any unverified items.

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

JJulius Marmingereviewed15d ago

Added 2 commits14d ago

fix(server): compact superseded history in the background3704707
refactor(server): DatabaseStorage layer uses a make constructore5cbab0

AAdEx-Partners-DE14d ago

A Windows data point from a large database, for anyone wondering what this does on an existing install. The fork I run has carried this change since 16 Oct. The numbers below are read-only queries on its statev2.sqlite at 16:55 UTC, compared with a measurement at 00:54 UTC the same day, before the build had it.

The compactor has caught up. By its own rule (keep the oldest and newest event per entity, delete the ones in between), almost nothing is left to delete:

	events now	still deletable
`message.updated`	32,863 for 32,773 messages	0
`node.updated`	133,397 for 78,651 nodes	22
`thread.pull-request-synced`	6	(1,990 were written in 19 hours on 12/13 Oct, see #18559)

The file did not shrink, as #19436 says it won't. 3,274,887,168 bytes before, 3,299,213,312 now. auto_vacuum is 0, so the background reclaim does nothing here. The free list holds 49,789 pages of 4 KB, about 204 MB or 6% of the file. That is the floor of what t3 doctor would give back on this install; a VACUUM may recover more from half-empty pages. I have not run doctor.

What is left is mostly turn items. Event payload is 1,077 MB, and turn-item.updated is 671 MB of it (174,144 events for 110,243 items). All events except the newest per item come to 63,901 events and 139 MB. The comment in compactEventStore says why they stay (slimmed payloads reference their prompts), so this is not a request to delete them; it only shows where the remaining growth is on an install that runs many long agent turns. By table: events 1,605 MB, projection_turn_items 673 MB, projection_nodes 162 MB.

No stall visible, with one gap. The pass one minute after the 16:20 UTC restart (about 16:21:20) left no SQL span over 300 ms between 16:21:05 and 16:24:40, and no server.eventLoop.stall span. That pass probably had little left to do. The first, large pass ran at an earlier restart whose trace has rotated away, and "Compacted orchestration history" goes to the logger only, so I cannot give the deleted count or the duration of that first pass.

Not checked: other installs, and what doctor would actually reclaim here.

Would a span around each compaction pass (deleted rows and duration as attributes) be welcome, so people with large databases can report the first pass from their trace file?

Posted by Claude (Opus 5.5, Claude Code via T3 Code) on behalf of @AdEx-Partners-DE.