Background service exhausts V8 heap and aborts after ~1-2 days of continuous uptime under sustained session load#12584

Triage

Confirmed as a real background-service V8 heap abort, not a one-off OOM from a single oversized payload and not a misconfigured unit. Reproduced twice on v0.0.42 (719a76ca1d) with the same main-thread stack. Current main @ dfbb11bdd still has the long-lived retention paths that fit this shape. I did not find a duplicate.

Related, not the same ticket:

• #5410 / open #9824 — idle desktop Private Bytes from immortal Effect parent spans (~25–30 MB/h with no provider work). That leak is still unfixed on main (forkParked still inherits the caller span). It is a contributor, not this crash: idle growth alone does not reach a V8 ceiling in 30–48h.
• #12305 (on main, after 0.0.42) — bound provider event-log serialization.
• #12523 (on main, after 0.0.42) — drop oversized PR diff cache entries.
• #12345 — keep tool payloads out of completion queues (merged to t3code/codex-turn-mapping, not on main).
• #11514 — bound getThreadProjection with paging (open).
• #8796 — delete provider_session_runtime rows when a thread is deleted (open).

None of those is a multi-day busy-service OOM under hundreds of live sessions.

What the report shows

Two SIGABRTs, both v8::internal::V8::FatalProcessOutOfMemory on the server thread during Heap::CollectGarbage / AllocateRawWithLightRetrySlowPath:

RSS 230 MB after 90 minutes, then fatal after a day-plus. That is retention, not a single huge allocation. The 15–18 GiB cgroup peaks include children (claude, git, extra node). The abort itself is the server process V8 heap.

Workload matches the expensive path: many long-lived provider sessions, frequent turn churn, lots of git/worktree work, ~16.7k orchestration events, ~430 live session bindings. 13h57m CPU / 29.8h wall on the second window.

The generated user unit still ships with no heap cap:

https://github.com/pingdotgg/t3code/blob/dfbb11bdd7c3f1a5575cb55d3e3abb12be025727/apps/server/src/cloud/bootService.ts#L95-L125

Environment= is only T3CODE_HOME and the unit name. No NODE_OPTIONS, no --max-old-space-size. KillMode=mixed + Restart=always is why recovery is ~25s and why the cgroup teardown SIGKILLs in-flight provider/git children. OOMPolicy=continue only covers kernel OOM of a child, not a V8 abort of the main process.

What the process still retains

Several independent sites can grow for the life of t3 serve. Together they fit “small at 90 minutes, dead after 1–2 days” on a busy orchestration host.

1. Adapter session maps keep every completed turn.

Claude’s in-memory session is a process-lifetime Map, and each finished turn is appended — not replaced:

items is the turn’s tool/content payload. The array is only rewritten on rollback. Codex / Cursor / OpenCode / Grok / Antigravity keep the same kind of Map<ThreadId, …SessionContext>. stopSession deletes the entry; if the session is still considered live, the history stays.

2. The inactivity reaper will not touch a “busy” or background-live session.

https://github.com/pingdotgg/t3code/blob/dfbb11bdd7c3f1a5575cb55d3e3abb12be025727/apps/server/src/provider/Layers/ProviderSessionReaper.ts#L42-L95

Default idle window is 30 minutes. It skips stopped, anything with activeTurnId, and anything with backgroundLiveness (working / monitoring — subagent fleets, workflow runs, Monitor loops). On this host that is the intended workload, so ~430 live bindings can be real in-process sessions, not leftover SQLite rows.

ThreadBackgroundLiveness is in-memory and only clears on task.completed (terminal status) or session.exited. A monitor/subagent that never emits a terminal event keeps the session off-limits to the reaper.

Open #8796 is the other half: deleted threads stop the session but leave the provider_session_runtime row. That inflates the binding count; it does not by itself keep adapter maps warm.

3. Live event / worker queues are unbounded.

Every adapter publishes on Queue.unbounded<ProviderRuntimeEvent>(). Ingestion, checkpoint, PR, and command reactors use makeDrainableWorker (TxQueue.unbounded). A slow git/diff/ingestion worker holds every queued event, including diffs and tool payloads, until it drains. #12345 is the same class of bug and is not on main.

4. Immortal parent spans (#5410) are still on main.

forkParked still does Effect.forkScoped(effect) with the caller’s span:

https://github.com/pingdotgg/t3code/blob/dfbb11bdd7c3f1a5575cb55d3e3abb12be025727/apps/server/src/serverActivation.ts#L12-L26

Local file tracing is always on (server.trace.ndjson, Info + timing). Under idle load this was 25–30 MB/h; under this host’s turn/git churn it will be worse. #9824 is the proposed detach. Not merged.

5. Full thread projections are still unbounded (#11514).

orchestration.getThreadProjection still materializes every turn/payload/checkpoint for a long thread. A reconnect or snapshot refresh of many busy threads is a heap spike on top of the leak.

16.7k orchestration events in SQLite is not large by itself. It only hurts if those rows (or their payloads) are decoded and held — replay, projection, live catch-up, or the completion-queue path above.

What this is not

• Not kernel OOM of a child. The stack is V8 FatalProcessOutOfMemory on the server thread.
• Not “raise the heap and we are done.” --max-old-space-size only delays the abort. The generated unit also has no place to put it without a systemd drop-in.
• Not already fixed in a release newer than 0.0.42. #12305 / #12523 help log/cache spikes; they do not bound session maps, reaper skips, unbounded workers, or parent spans.

Suggested direction

Do not start with a larger heap. Bound the process-lifetime sets:

1. Drop completed-turn payloads from adapter session maps (or keep a small rolling window). Resume/rollback should read history from the provider store, not from an ever-growing turns[].
2. Land #9824 so parked reactors do not pin startup spans for the life of the service.
3. Bound the runtime-event / DrainableWorker queues (drop or coalesce when the consumer is behind; bring #12345 onto main).
4. Land #11514 so projection reads cannot materialize a whole thread.
5. Land #8796 so deleted threads do not accumulate bindings. Separately: sample provider.session.reaper.skipped-background-work / skipped-active-turn on this class of host — stuck liveness is how 430 “live” sessions stay in the Map.

Graceful restart is a follow-up, not the fix. KillMode=mixed will still SIGKILL the cgroup if the main process aborts.

What would help on the next occurrence

The offer of --heapsnapshot-near-heap-limit=1 is the right next artifact. Also useful, if cheap:

• process.memoryUsage() / v8.getHeapStatistics() on a 15–30 min timer (heapUsed vs external vs rss).
• Counts: provider_session_runtime by status, adapter.listSessions() vs that table, and reaper skip reasons from the server log.
• Whether server.trace.ndjson is rotating (traceMaxBytes is 10 MiB) or the file is huge.

Raising the heap via a systemd drop-in is a valid delay, not a diagnosis:

Then systemctl --user daemon-reload && systemctl --user restart <unit>. Snapshots land in the service cwd (%h). Redact before attaching.

No workaround is required for the reporter — Restart=always is already holding. Filing this is correct; a long-running t3 serve should not hit the V8 ceiling under normal sustained use.

Data from another long-running host, from T3's own trace spans and a source read. No heap contents are shared.

Runtime

• 0.0.46-nightly.20261008.2833 (a6ec88f7), SEA linux-x64, Node 26.8.2, V8 14.6.202.34, in a container. Our build carries a small local patch (MCP wiring and provider instructions only); it touches none of the files named below.
• Since a graceful restart on 2026-10-15 at 08:00 UTC: NODE_OPTIONS=--max-old-space-size=12288 --heapsnapshot-near-heap-limit=1 --diagnostic-dir=<dir>. RSS was about 1.0 GiB shortly after.
• At our next rebuild we move to a current nightly, which includes #19305, and raise the cap to about 20 GiB.

What we saw

• Snapshot residue (an earlier process). One snapshot from the in-app SIGUSR2 handler at RSS 2926 MiB. The server.heapSnapshot span took 8.3 s and succeeded, writing a 267 MB file. The next server.eventLoop.stall span, 23 s later, had rssMb 4337. RSS settled near 5.5 GiB (high-water mark about 5.9 GiB) and stayed there. Stalls of 3-5 s recurred from then on.
• Slope (a clean process, 6 GiB cap, no snapshot). A sawtooth of about ±500 MB of anonymous memory. The troughs rose about 0.3 GiB/h; trough-to-peak was up to about 0.9 GiB/h. RSS alone cannot separate retained objects from heap headroom.

Source note

v8.writeHeapSnapshot leaves V8's HeapObjectsMap allocated and object-move tracking on for the life of the process. HeapProfiler::TakeSnapshot sets is_tracking_object_moves_ = true (deps/v8/src/profiler/heap-profiler.cc:167 at nodejs/node v26.8.2, f2f2c2f2). Only ClearHeapObjectMap (:344-353) turns it off, and Node's writeHeapSnapshot path never calls it. Every later scavenge and compaction then reports each moved object to the profiler. So the SIGUSR2 handler (apps/server/src/observability/HeapSnapshot.ts:24, :46-49 at 26fffe5a) can permanently raise RSS and per-GC cost on a long-running server. This is from source; we have not measured how much of our rise is the map and how much is the allocator keeping memory.

Suggestions

1. Take the in-app snapshot through an in-process node:inspector Session: HeapProfiler.takeHeapSnapshot, then HeapProfiler.disable, which calls ClearObjectIds (deps/v8/src/inspector/v8-heap-profiler-agent-impl.cc:333-341, same node commit). Otherwise, document that the residue lasts until restart.
2. Put heapUsed, heapTotal, external, arrayBuffers and the heap limit on the server.eventLoop.stall span next to rssMb (apps/server/src/observability/EventLoopMonitor.ts:146, :151 at 26fffe5a). That is the triage ask's data, taken at the moment that matters, and it separates GC-near-limit stalls from sync-I/O stalls without a snapshot.

Disk

Disk, not heap. Our statev2.sqlite is 10.24 GB; orchestration_events is 6.48 GB of it (63%) over about 1.70M rows, and orchestration_v2_events is empty. Since migration 055 orchestration_events also holds every v2 event, so we cannot yet say how much is v1 history. compactEventStore (apps/server/src/orchestration-v2/ProjectionMaintenance.ts:242 at 5a4e9980) has no caller outside tests, which matches #18559.

Offer

We will take no further manual snapshot. If the near-limit snapshot fires, we will post a summary here: retainer classes and their sizes, never heap contents.

Checked current main 89fe509232 on Linux x86_64 / Node 24.21.0 with the real web/server dev app and isolated copied state (3 stopped threads, 341 events). Browser-driven reads completed 200 shell/empty-thread requests and 200 populated-thread requests, all HTTP 200, without a V8 abort. This is a short read-load check, not a reproduction of 430 live sessions over 30–48 hours; it does not establish that the bug is fixed.

Runtime transcript, commands, results and limitations.

No matching duplicate or existing fix PR found. Main includes #19305, but its native SQLite retention fix does not establish a fix for this V8 heap abort.

Does this still occur on a current nightly? Please include the exact build, provider mix, approximate concurrent active sessions and turns/hour, and whether growth occurs while idle too. For a recurrence, a summary of heapUsed/heapTotal/external/arrayBuffers versus RSS and the largest retained object classes/retainer paths would distinguish the suspected causes. Please do not post raw heap snapshots; they can contain conversation content and secrets. The follow-up host report already notes snapshot-related overhead, so no additional manual snapshot is requested.

Still happening on t3 v0.1.0: SEA build dated Oct 15, linux-x64, Node v26.8.2, bundling effect 4.0.4, headless user service. I haven't tried a current nightly yet. There have been five V8 heap aborts since Oct 24, and I have a reproduction for what looks like the main cause on this host: a request whose client disconnects early keeps its whole encoded JSON response in memory.

At every abort, the last full Mark-Compact freed almost nothing (for example 4091.1 -> 4088.0 MB) against the default 4,192 MB limit.

Heap breakdown on the live server

I collected these through the inspector: process.memoryUsage() and V8 heap space statistics every minute. No snapshot was taken.

Almost all of the growth is in large_object_space (7-13 MB/min) and ArrayBuffers (4-8 MB/min); old space stays flat at about 230 MB.

The sampling heap profiler (128 KiB interval) attributes 802 of 909 MB (88%) of still-live sampled allocations over its first hour to the JSON encoder in effect@4.0.4 http-api/HttpApiBuilder.js, getResponseEncode, which calls JSON.stringify(e) and then text(...). That is the allocation site, not a retainer path. I didn't take a heap snapshot, so what keeps these objects reachable is still open.

Reproduction

I forced a full GC (HeapProfiler.collectGarbage) before and after each step:

Completed responses are released. For aborted requests, the string from JSON.stringify and its UTF-8 buffer stay reachable, and they were still held after later GCs.

What it isn't

I couldn't reproduce it in a standalone Effect 4.0.4 server. The test served a ~6 MB JSON body after an uninterruptible 1.5 s delay and aborted 10 requests at 500 ms. I ran it with a plain router and with HttpApi schema responses, with and without a recording tracer, and with the global compression middleware T3 installs, on Node 24 and on Node 26. Every variant released everything after GC.

So the retention looks specific to T3's server layers around these endpoints, rather than to Effect's HTTP stack. Candidates are the auth middleware, annotateEnvironmentRequest, traceLocalHandlerWork and the trace exporter. I haven't narrowed it further.

Why it compounds

/api/orchestration/shell returns every unarchived thread. I have 1,744, of which only 9 are archived. It took a median of 6.4 s earlier, and 44 s once the heap was large.

1. Clients with timeouts abandon most calls; mine used 8 s and 15 s.
2. Each abandoned call leaks about 20 MB.
3. The heap grows, so GC and the endpoint both slow down.
4. Slower responses mean more calls are abandoned, and the cycle repeats.

That matches my time-to-abort dropping from 161.7 h, with about a dozen threads, to 9-15 h once I had 1,100+ threads.

Answers to the questions above

• Build: t3 v0.1.0 (Oct 15 build, effect 4.0.4).
• Current nightly: not tested. main now uses effect 4.0.5, but its Node HTTP server and response changes are only URL redaction in logs, so the client-disconnect path looks unchanged.
• Providers: Claude Code (through a local API gateway) and Codex.
• Load: in the 10 h before the last abort, 478 turns (average 48/h, peak 122/h) and a peak of 21 concurrent runs.
• Idle growth: not measured, because the server was never idle.
• Retained objects: large strings and their UTF-8 ArrayBuffer backing stores from response encoding, almost all in large_object_space. Retainer paths are unknown.
• Snapshots: no raw heap snapshot was taken or shared.

Probably separate, from T3's own trace spans

• A repository and pull request discovery sweep runs every 2-4 minutes. Each pass is about 13,200 spans and 4,100 git commands across 542 worktrees.
• Two traces never end and keep gaining child spans: one at about 1,400/min (Codex provider retry and SQLite) and one at about 575/min (ThreadLiveEventCoalescer).

On my side, I'm removing the client timeouts and most of my shell calls, which should stop the growth for me. The server-side retention on client disconnect would remain.

Thanks for the abort reproduction. I followed it on current main 68339a0846 (effect 4.0.5) and on your build, v0.1.0 81493efb71 (effect 4.0.4). I ran v0.1.0 under both Node 24.22.1 and Node 26.8.2. I could not reproduce the retention.

Setup: t3 serve from source on a fresh base dir with --inspect, 1 project, and 1,744 unarchived threads created through the server. GET /api/orchestration/shell returned 2.6–2.8 MB in 0.3–0.4 s, with a bearer token from t3 auth session issue. I forced a full GC (HeapProfiler.collectGarbage, twice) before each reading.

arrayBuffers stayed at 0.5 MB after every abort phase. Transcript, drivers: seed-threads.mjs, measure-aborts.mjs, measure-bursts.mjs, issue-mcp-token.ts.

So the path that keeps the response alive is probably something on your host that this setup lacks. To close the gap:

1. How the aborting clients reach the server. Loopback, LAN or tailnet IP, Tailscale Serve, or T3 Connect. Also their auth: a bearer from t3 auth session issue, a browser session cookie, or a pairing token. And the client itself (curl, fetch, your own tool), with the exact command or code that aborts.
2. Telemetry. Is OTLP trace or metrics export configured (the telemetry export settings, or T3CODE_OTLP_TRACES_URL / T3CODE_OTLP_METRICS_URL / T3CODE_OTLP_LOGS_URL)? If so, roughly what endpoint (local collector or remote)?
3. One retainer path. On a test process, not the live one: take one heap snapshot after a few aborted requests and open it in Chrome DevTools. Pick one of the large retained strings (~5 MB) and give the class and property names along its Retainers chain up to the first GC root. Names only, no contents. That chain is what tells us which layer holds it.
4. A fresh base dir. Does it still reproduce with t3 serve --base-dir <empty dir> --port <free port> on the same host and binary, with your aborting client pointed at that server?

The PR discovery sweep and the two never-ending traces you list look like separate issues. #25172 already tracks the PR sync sweep's memory growth.

Thanks for running it. I found the retainer, and I think it explains why your runs stayed clean. The leak needs the client's disconnect to be noticed after the handler has returned, while the response is being compressed and sent. Every abort in your runs landed while the handler was still running, so the fiber was interrupted cleanly and answered 499.

Retainer path, from the live process

I read this from the running v0.1.0 server through the inspector, not a snapshot. I used Runtime.queryObjects on ServerResponse, then the closure of makeHandler's close listener to reach each request fiber. Names and sizes only.

After a forced GC, 9 ServerResponses were alive with writableEnded: true, writableFinished: false and destroyed: true:

• 5 × GET /api/orchestration/threads/:threadId:
  • status 200, content-encoding: gzip, content-length about 7.3 MB;
  • none of the body reached the socket, and the socket closed without an error;
  • one end callback is still queued;
  • the request fiber isn't done, has interruptible: false with an interrupt pending, and is suspended on that callback under toHandled's sendResponse continuation.
• 4 × GET /api/orchestration/shell with status 499: their fibers completed. I don't know what keeps these four response objects reachable, and they don't hold a body.

The chain for the 200s:

1. The server scope. makeHandler runs each request with Fiber.runIn(..., options.scope).
2. The request fiber.
3. sendResponse's mapEager continuation, which closes over the original, uncompressed response.
4. Its Uint8Array body: a 54.4M-character JSON string and its 54.4 MB of UTF-8 bytes. The 7.3 MB gzip output is held by the pending end call.

That's about 116 MB per stuck request, which matches the large_object_space and arrayBuffers growth in my earlier comment.

Why it happens

In effect 4.0.4, toHandled sends the response uninterruptibly, including pre-response handlers such as the compression middleware. Suppose the socket's close is handled in that window:

1. makeHandler requests an interrupt, which is deferred.
2. handleResponse calls nodeResponse.end(body, cb) on the destroyed response. That's the Uint8Array branch for 1 MB or more, or Raw.
3. A destroyed response never emits finish, so Node never calls cb, and the fiber waits forever.

A disconnect during the synchronous end of the handler (JSON.stringify of the full snapshot) is only noticed at the next I/O turn, and that is the async zlib call. Here the responses are 54 MB before compression, on a busy event loop, so the window lasts seconds and my client's 15 s timeout lands in it. Your shell responses were 2.7 MB in 0.3 s, so the window was a few milliseconds.

I've opened a fix in Effect: Effect-TS/effect#10004. writeResponse now skips the write and releases the body when the response is already destroyed. Writes that wait on end's callback also resume when the response closes, which covers a disconnect during the send itself. Its deterministic test is an uninterruptible route that responds after the client's socket closes. On Effect's main, the Raw case hangs and the Uint8Array case fails with a content-length mismatch. T3 would pick up the fix with an Effect bump.

To reproduce it in T3, request a thread whose snapshot is tens of MB with GET /api/orchestration/threads/:threadId and accept-encoding: gzip. Time a completed request first, then abort a little before the response would arrive. The standalone version is the test in Effect-TS/effect#10004.

Your questions

1. Clients: loopback 127.0.0.1:3773, HTTP/1.1. The client is Node fetch (undici, accept-encoding: gzip, deflate) with AbortSignal.timeout(15000). Auth is a bearer token obtained by exchanging a t3 auth pairing create credential.
2. Telemetry: no OTLP export. None of the T3CODE_OTLP_* or OTEL_EXPORTER_OTLP_* variables is set, and there are no telemetry settings. Only the local server.trace.ndjson is written.
3. Retainer path: above, from the live process instead of a test process.
4. Fresh base dir: not tried. The retention depends on response size and timing rather than base-dir state, and the Effect test reproduces it without T3.

Separately, my client was requesting /api/orchestration/threads/:threadId?turnLimit=1. That route has no query schema, so turnLimit is ignored and the full snapshot comes back, which is why these responses were 54 MB.