[Bug]: Hidden subagent child threads are auto-settle candidates, so the sweep evaluates and polls threads the user cannot see or settle#19657

Confirmed on main: neither getSettlementCandidates (ProjectionStore.ts) nor isAutoSettlementCandidate (ThreadSettlementService.ts) looks at lineage.relationshipToParent, so hidden subagent threads go through every sweep, and the ones with a branch but no PR link each trigger their own pullRequestFor lookup. Pinned children are already filtered out by the store query, so the waste comes from the unpinned, idle children.

This is a cost problem, not a correctness one. The children do settle through the inactivity timer eventually; the trouble is that they keep piling up faster than it drains them, and you can't see or act on them in the UI. Accepting it as a performance bug.

Possible directions:

• Leave relationshipToParent === "subagent" threads out of the candidate set and settle them along with their parent (or when their run finishes).
• Or at least skip the PR lookup for children and reuse the parent's result for the same branch.

Thanks for the before and after sweep counts.

Independent confirmation from the Linux nightly audits (0.0.46-nightly.20261014.3132), with the candidate-selection path rechecked on e536bfd33bc3c351997141d93790df75eeded7b9:

• 183 threads passed the settlement-candidate filter in one snapshot; 151 were hidden provider-native subagent child threads (thread:provider:*). Of those, 127 had a branch.
• Those 151 had no runs and only their delegated-prompt user message. As the maintainer comment explains, inactivity eventually retires them, but they remain candidates until then.
• A retained ~2.7-hour trace window contained 55,826 ThreadSettlementServiceV2.settleThread spans. That is an observable cost beyond GitHub calls: per-candidate settlement spans contributed roughly 16–23% of trace bytes across the audit windows.

This is candidate evaluation and trace-volume evidence, not 151 GitHub requests each minute: the code groups lookups and uses caches. Background sweeps together occupied about 45% of the trace bytes in the captured workload, shortening the local trace history to about 2.5 hours; the broader tracing issue is #8764. No server-wide CPU storm was observed.

The inherited pins come from [makeSubagentChildThread](https://github.com/pingdotgg/t3code/blob/70d60e03a7a35c6fa131825bbfa05ec893e69912/packages/provider-core/src/server/subagentProjection.ts#L57-L87). It starts from ...input.parentThread and resets archive, settle, snooze and the PR links, but not pinnedAt. Every child of a pinned thread is created pinned, with the parent's exact timestamp, and re-pinning the parent doesn't change existing children.

Pinned children make this worse than a slow drain. The [settlement query](https://github.com/pingdotgg/t3code/blob/70d60e03a7a35c6fa131825bbfa05ec893e69912/apps/server/src/orchestration-v2/ProjectionStore.ts#L5620) and [isAutoSettlementCandidate](https://github.com/pingdotgg/t3code/blob/70d60e03a7a35c6fa131825bbfa05ec893e69912/apps/server/src/orchestration-v2/ThreadSettlementService.ts#L136) skip them, so they never settle. But the [PR sync filter](https://github.com/pingdotgg/t3code/blob/70d60e03a7a35c6fa131825bbfa05ec893e69912/apps/server/src/orchestration-v2/ThreadPullRequestService.ts#L164-L171) doesn't, so each one still gets a branch lookup every minute.

On my install (0.1.1-nightly.20261017.3238), 114 of 160 unsettled threads were pinned children of two lead threads. The server's main thread sat at about 100% CPU, and requests took over 15 s. After unpinning them over the WebSocket API (thread.unpin) and settling the idle ones, 11 unsettled threads were left and CPU dropped to about 25%.

Suggested fix: reset pinnedAt (and pinOrderKey) in makeSubagentChildThread.

Another data point from a Linux nightly (0.1.1-nightly.20261018.3335, Orchestrator V2, ~25 concurrent agents mostly via delegate_task).

The candidate query from this issue on my statev2.sqlite (read-only):

So the sweep set is ~99% hidden subagent threads. The cost shows up in server.trace.ndjson: in one ~2 minute file, 2,117 runGitCommand spans, and walking each one up its parents, 1,438 root at ThreadPullRequestServiceV2.synchronize and 576 at branchPullRequest. synchronize ran 24 times in ~4 minutes, averaging 3.4 s and ~60 git commands per run. That is ~15-17 git processes per second from the server. The t3 serve main thread sits at ~55% CPU, about half of it system time. Clients connected to that server show "syncing messages" for a long time when opening threads.

Related: #18618 (PR discovery builds full thread shells). With subagent children excluded from the candidate set, this box would drop from ~2,650 to ~23 candidates.

Another data point from macOS, desktop app nightly 0.1.1-nightly.20261020.3401, Orchestrator V2.

One more cost from the sweep that I did not see above: each settleThread reads the server settings and their secrets again. In one trace file (about 90 s, 7 sweeps):

866 of the 1,002 materializeSettingsSecrets spans have the parent chain settleThread < sweep, so it is one settings and secret read per candidate thread. Over 80 minutes these three spans ran 27k to 31k times each and made up about 39% of the trace bytes. The trace grows about 7 MB per minute, so the 11 rotated files hold only about 15 minutes of history.

Excluding subagent threads from the sweep (#19665) would remove most of this. Reading settings once per sweep instead of once per thread would also help.

Data gathered with Claude Code (Opus 5.5).

Reproduced on current main 70e5e384da, Linux x86_64, Node 24.21.0, web dev server with isolated .t3 state.

Created a real Codex native child through the web composer, waited for both child and parent to finish, then compared the sidebar, read-only reporter SQL and sweep traces. The sidebar showed 3 top-level threads; SQL returned those 3 plus 1 hidden, unpinned subagent. The next full sweep increased from 6 to 8 settleThread span records and from 3 to 4 pullRequestFor records. These are evaluation/lookup spans, not remote requests or git-process counts. The fixture had no branch; branch discovery overhead was not tested.

Runtime transcript

Current-main sidebar after the native child completed

The existing maintainer takeover #19665 already targets this issue. No competing PR was created. This session verified the main-branch reproduction, not that PR's fixed behavior. Windows/Electron, mobile, Claude and delegated-task variants were not tested. The isolated dev server was stopped.

Adding to @hcaiano's note above about settleThread re-reading settings and secrets: here are latency numbers and what the slow reads wait on. Linux desktop nightly 0.1.2-nightly.20261022.3541 (source 044f07257c), Orchestrator V2.

In an 8.5-minute trace window, each full sweep evaluated about 1,390 candidates. That came to about 2,100 settleThread calls and 400 pullRequestFor lookups per sweep, with 0 auto-settle dispatches across all 6 sweeps. Sweeps took 2.5 s to 80 s. Because the schedule is spaced("1 minute") after the drain, the long ones run nearly back to back.

• 11,017 of 11,799 materializeSettingsSecrets spans are children of settleThread. The settings file is cached, but the exported getSettings (serverSettings.ts:1158) runs materializeSettingsSecrets (:810) on every call, which reads each redacted secret with readFile (ServerSecretStore.ts:174-187). settleThread (ThreadSettlementService.ts:325) calls it per candidate, even though sweep() already read settings at :310.
• The slow secret reads are not waiting on SQLite, and they are not event-loop stalls. 18 to 235 other spans started and finished inside each read of 500 ms or more, writer statements inside them executed in 0.1 ms median, and no server.eventLoop.stall fell in the window. The likely cause is queueing on the default 4-thread libuv pool: 8 concurrent settles, each doing about 3 sequential readFiles, plus fileSystem.exists per lookup candidate.
• These three spans are about 31% of server.trace.ndjson bytes. That file rotates through 10 x 10 MiB in about 9 minutes here.

#19665 would shrink the candidate set, but it keeps the per-thread getSettings. Reading settings once per sweep and passing them to settleThread would remove this cost either way.

Data gathered with Claude Code (Opus 5.5).

Measured on Linux 0.1.3-nightly.20261023.3592 with about 1,700 unsettled threads, 1,694 of them delegated subagent threads (1,589 finished). Server traces over 3 minutes: 2,091 ThreadSettlementServiceV2.settleThread calls, every one under ThreadSettlementServiceV2.sweep; 2,217 serverSettings.materializeSettingsSecrets (about 96 ms each, three to four ServerSecretStore.get file reads per call); and 415 of about 600 lookupBranchPullRequest calls plus most runGitCommand spawns came from the sweep. The server main thread sat near 100% CPU.

Settling the 1,061 finished subagent threads cut settleThread from about 2,000 to about 700 per 2 minutes. Two cheap fixes would help even without excluding hidden threads: read settings once per sweep rather than once per candidate (settleThread calls settingsService.getSettings, which re-materializes secrets each time), and skip the branch lookup for subagent threads that carry no PR link.

Another data point, where the sweep load came right before the desktop app exited.

Setup: Linux (Arch, Hyprland), desktop nightly 0.1.3-nightly.20261023.3553, Orchestrator V2. One orchestrator thread runs about 6 worker threads, plus Claude Agent-tool review subagents and a few delegate_task children.

Candidate set: read-only query on statev2.sqlite, not archived or deleted, with settledAt null:

So about 98 % of what the sweep evaluates are hidden, finished subagent threads.

Traces:

• From about 10:26 to 10:40 local, each 10 MB server.trace.ndjson file rotated every 1–2 minutes.
• Each file held about 1,575 ThreadSettlementServiceV2.settleThread spans, all under ThreadSettlementServiceV2.sweep, plus about 1,700 serverSettings.materializeSettingsSecrets. So the retained history covered only about 15 minutes.
• At 08:40:28Z event-loop-stalls.ndjson logged delayMaxMs: 9763, at 45 % utilisation. The previous stall that morning was 2.7 s.
• The desktop process stopped tracing at 08:40:15Z, and the app scope ended at 08:40:52Z. There was no crash output, no OOM kill and no core dump.

After the restart: on 0.1.3-nightly.20261024.3604 the pattern is unchanged, at 1,560 settleThread spans in one trace file covering 08:59–09:02Z.

https://github.com/pingdotgg/t3code/pull/19665 (subagent threads settle with their parent) would remove almost all of this load for us. It has been conflicting since 2026-10-19; a rebase and merge would help a lot.

Another data point, with a different cost: on a self-managed GitLab, the leftover children alone used up the per-user API rate limit.

Environment. T3 Code desktop 0.1.2 on Windows 10 (local server 0.1.2), Orchestrator V2, glab 1.124.0. Self-managed GitLab with throttle_authenticated_api at 2400 requests per hour per user.

What happened. Settling a parent thread leaves its subagent children unsettled. In statev2.sqlite, 251 unsettled, unarchived threads had a branch. 248 of them were children (lineage.parentThreadId set), and 226 of those had a settled parent. One settled parent alone left 40 unsettled children on its branch. 245 of the 248 had creationSource: "provider", and 3 had mcp.

Both one-minute loops pick these children up because they filter on the thread's own fields only. That's true in v0.1.2 and still true on main:

• Settlement sweep: getSettlementCandidates in ProjectionStore.ts (settledOverride IS NULL, no parent check).
• Branch PR discovery: the thread filter in ThreadPullRequestService.ts (settledOverride !== "settled" && settledAt === null).

On GitLab, each MR summary costs 2 glab api calls, the MR and the project (see #24190). The summary cache keeps a success for 60 s, so the entry has expired by the next sweep.

Rate, from glab spans in server.trace.ndjson over 19 minutes:

That's about 1700 requests an hour from T3 alone, before anything else the user does on GitLab. With glab telemetry on, each launch costs a second request (POST /api/v4/usage_data/track_event). The account hit the hourly limit and GitLab returned 429. On a 429, glab mr list sleeps until RateLimit-Reset and retries up to 6 times. T3 kills it at its 30 s timeout, and the next sweep starts it again.

Before and after settling only the 226 orphaned children with t3_thread_organize (action: settle), with no other change:

The workaround doesn't last. Later the same day there were 26 unsettled children with a branch again, 5 of them under a settled parent.

As described, #19665 covers this case: idle children settle once their parent settles, and a settled child drops out of both the settlement sweep and branch discovery. It's still open and conflicts with main.

Investigated with t3 triage, by Claude Opus 5.5 in Claude Code.

Another data point, from a Linux server under WSL2 with Orchestrator V2, and many agents mostly started through delegate_task. The server is a personal fork build of upstream nightly 0.1.3-nightly.20261027.3754. None of its patches change which threads the sweeps select. One of them adds locking around the PR read cache. The Mac client showed "reconnecting · did not answer" and then "Syncing messages..." for up to 40 s per thread.

Before the change, 378 unarchived, unsettled threads with a branch went through the PR sweep every minute. 330 of them were hidden subagent threads. Their parents broke down as:

server.trace.ndjson showed 404 runGitCommand spans a minute. ThreadPullRequestServiceV2.synchronize started 74% of them and ThreadSettlementServiceV2.sweep started 5%. Over 20 s, the server's main thread used 3.8 s of user CPU and 5.6 s of system CPU. event-loop-stalls.ndjson entries from the same days show the matching pattern: high cpuSystemMs and 110k to 165k minor page faults per stall, with almost no major faults.

I then settled every hidden subagent thread whose parent was already settled (248 of them, by lineage.relationshipToParent = "subagent") through the t3_thread_organize MCP tool. That's the end state #19665 would reach on its own. Same server, same load, measured 8 minutes later:

The settlement sweep's own git count rose from 19 to 46 a minute during the window right after settling. I haven't checked whether that settles back down. The 116 subagent threads still in the sweep belong to parents that are still open, so #19665 wouldn't cover them yet.

Additional triage evidence: automatic settlement can make child threads show Settled while the parent still shows their subagents as Running.

Environment

• Client: T3 Code desktop on Omarchy, version 0.1.6-nightly-20261030.319 (user reported).
• Server: Mac mini, macOS Darwin 25.6.0 arm64, version 0.1.6-nightly.20261030.3940.
• Connection: remote desktop client to Mac mini over Tailscale.
• Provider: Claude Agent.

Observed behavior

The parent showed three subagents running, while opening each child showed it as settled. During investigation, one subagent completed, leaving two with the mismatch.

Database evidence

All timestamps below are UTC:

Both children have settledOverride: "settled". Their thread.settled events have actor server and command IDs beginning server:auto-settle:. Both parent subagent records have null completed_at.

Diagnosis

Automatic settlement occurred before subsequent recorded child activity. This confirms conflicting server-side settlement and subagent states; it does not establish that the underlying agents had finished.

In ThreadSettlementService.ts at server tag v0.1.6-nightly.20261030.3940, isAutoSettlementCandidate checks active runs and pending background tasks but has no explicit subagent-lineage exclusion. The exact settlement trigger and missing activity guard have not been fully established.

Reproduction

Observed in an existing Claude parent with native subagents; no controlled reproduction performed. Open the parent’s running subagent list, then inspect the corresponding auto-settled child threads.

No configuration changes, source patches, or database writes were applied.

Investigated and drafted by Codex (GPT-6), via t3 triage.