[Bug]: Deleting a thread says it permanently clears its history, but V2 keeps every event and projection row#17845

opened20d agobyOonly21mil

Area

apps/server

Steps to reproduce

  1. On Orchestrator V2, run a thread for one turn with at least one tool call.
  2. Delete it from the sidebar. The dialog says: "This permanently clears conversation history for this thread."
  3. With the server running or stopped, open statev2.sqlite read-only and count the thread's rows:
    sql
    1SELECT count(*) FROM orchestration_events WHERE aggregate_kind = 'thread' AND stream_id = :thread_id;
    2SELECT count(*) FROM orchestration_v2_projection_turn_items WHERE thread_id = :thread_id;
    3SELECT count(*) FROM orchestration_v2_projection_messages WHERE thread_id = :thread_id;

Expected behavior

The thread's conversation history (messages, turn items with tool output, reasoning, and the events that carry them) is removed from the database, as the dialog says. A minimal tombstone may stay so clients drop the thread.

Actual behavior

Every row stays. planThreadDeletion (apps/server/src/orchestration-v2/ThreadDeletion.ts:58) emits thread.deleted with deletedAt and cancels open work; nothing deletes rows. The projection gets deleted_at set, and reads filter on it. Each delete also appends more events (the delete plus the cancellations).

On one install, a thread deleted on 2026-10-08 still has all 48 of its events (48.6 KB of payload), 6 turn items and 2 messages. Across that install, 45 deleted threads still hold 365 MB; 44 of them were deleted before the V2 migration, so that total also reflects V1 behavior.

The dialog text is in apps/web/src/components/Sidebar.tsx:4807 and :4460 (bulk), apps/web/src/hooks/useThreadActions.ts:992, and apps/web/src/hooks/useThreadActionMenu.ts:318.

ProjectionMaintenance.compactEventStore does not help. It keeps the newest event of each kind and never removes a deleted thread's history; besides, nothing in production calls it (#15759, #16558).

Impact

Minor bug or occasional failure

Version or commit

Observed on nightly 0.0.46-nightly.20261007.2787. Same code on main @ 8c777fbfad (checked statically).

Environment

Linux x86_64, headless T3 server, Orchestrator V2.

Logs or stack traces

text
1deleted threads in orchestration_v2_projection_threads: 45 (1 deleted after the V2 migration)
2that V2-era thread: events 48 (48,642 payload bytes), turn items 6, messages 2
3all deleted threads together: ~365 MB logical across events, turn items and nodes

Workaround

None in the app. Rows can be removed only by hand with the server stopped, followed by VACUUM. auto_vacuum is off, so deleted pages do not shrink the file.

Either the delete should purge the thread's history (events, receipts, projections, keeping a tombstone), or the dialog should say the thread is hidden, not cleared. Which one is the maintainers' call. A purge path is the bigger change and overlaps the retention discussion in #6897.

Drafted with Claude Opus 5.5 in Claude Code (T3 Code harness); counts measured read-only on the install above.

🤖 Generated with Claude Code

Synced from GitHub

Confirming on Linux nightly 0.0.46: deleting or archiving threads frees no space, since their events and projection rows stay in statev2.sqlite. With no retention setting, the DB reached 41.6 GB in about 6 weeks. A delete that actually removes a thread's events and projections, or a retention window for settled threads, would help a lot.

Synced from GitHub

Additional consolidated confirmation on nightly 0.0.46-nightly.20261014.3132; relevant metadata-upsert/deletion behavior still present at e536bfd33bc3c351997141d93790df75eeded7b9.

History belonging to deleted threadsLinux: 28 deleted threadsMac: 81 deleted threads
Events: rows / payload bytes7,386 / 13,530,46928,901 / 38,063,897
Turn items: rows / bytes432 / 585,2361,055 / 1,204,264
Messages: rows / bytes339 / 302,6471,034 / 872,823
Nodes: rows / bytes63 / 62,65821 / 21,248

Read-only query on database copies:

sql
1SELECT count(*), sum(length(CAST(e.payload_json AS BLOB)))
2FROM orchestration_events e
3JOIN orchestration_v2_projection_threads t ON t.thread_id = e.stream_id
4WHERE t.deleted_at IS NOT NULL;

The event counts include imported legacy history and overlap #18559. This is not 51.6 MB of independently additive V2-era growth, nor evidence of a CPU stall. It confirms that the deleted-thread tombstone does not remove the associated stored history. Invoking the existing compactor alone would not fulfill the delete dialog's promise. No live rows were deleted or vacuumed.

Synced from GitHub

Additional storage-pressure observation from macOS. This is a retention report, not a claim that all measured bytes belong to deleted threads.

A saved read-only table-allocation audit from October 27 UTC measured approximately 26.24 GiB in orchestration_events and 15.36 GiB in orchestration_v2_projection_turn_items. Those are historical table sizes, not deletion-eligible byte counts or independently additive recovered space. We have not established how much is legacy/imported history or attributable specifically to deleted threads.

On October 28, 2026 at 04:01 UTC, a fresh bounded metadata-only read found:

text
1statev2.sqlite logical file bytes: 58,162,655,232 (~54.2 GiB)
2WAL logical file bytes: 32,123,672
3page_size: 4096
4page_count: 14,199,884
5freelist_count: 0
6auto_vacuum: 2 (incremental)
7journal_mode: wal

The connection used SQLite URI mode=ro, connection-local query_only, and a one-second busy timeout. Only file metadata, PRAGMAs and bounded schema names were read. No event payloads, thread text, row deletion, checkpoint, VACUUM, doctor repair, database copy or server pause was used. File and WAL observations are live and need not form an atomic filesystem snapshot. The installed T3 build was not captured in this read; this is not a claim about current upstream main.

Our current auto_vacuum=2 differs from the original report's disabled setting. Zero freelist pages means no already-free page pool was shown by this observation; it does not identify which retained history can safely be retired. We are leaving the original database intact.

Could maintainers identify the supported retention/compaction path for legacy events and retained thread projections, including integrity and interruption handling? A bounded dry-run report that separates active/retained history, deleted-thread history and actually reclaimable storage would help operators choose a supported route. We prefer attaching this observation here over creating a duplicate issue; the legacy-event portion may also belong with the existing retention/imported-history discussions.

Synced from GitHub

Fresh data point from a heavier install, nightly 0.1.5-nightly.20261029.3867 (Linux x64, headless server, Orchestrator V2). Here deleted-thread history is ~90% of the database, not a minor leak.

  • statev2.sqlite: 25.9 GB after 3 weeks (first event 2026-10-08). freelist_count 0, auto_vacuum 0.
  • Threads: 9,369 created, 8,439 deleted (90%). Most look automation-created (delegated subagents, review scouts, scheduled-task runs); 200–580 are deleted per day.
  • orchestration_events: 3.23M rows, 12.5 GB table + ~3.5 GB indexes. 2.93M rows (91%, ~8.0 GB payload+metadata) belong to deleted threads.
  • orchestration_v2_projection_turn_items: 5.6 GB; 664k of 723k rows (92%) belong to deleted threads.
  • The 6-hour background compaction runs ("Compacted orchestration history" 3×/day) and has little left to take: 2.24M keyed events over 1.32M keys, consistent with first/latest retention. Compaction cannot bound growth here; only removing deleted threads' history can.
  • Event payload growth: 120–350 MB/day over the last 10 days.

Workflows that spawn many short-lived threads make this the dominant storage cost. #24924 fixes the dialog text, but a purge (or a retention window for deleted threads) is what would bound the DB.

Measured read-only (node:sqlite, mode=ro, dbstat). No rows changed.

Drafted with Claude Opus 5.5 following the t3 triage playbook (Claude Code via T3 Code).