After a restart, threads with active turns stayed on "Working" for about 9 minutes with no provider process (#25962). The effect daemon has 4 workers shared by every effect type. A title generation can run for up to about 546 s (180 s CLI timeout × 3 tries), and restart recovery requeues title generations. Five of them took all four workers, so the threads' provider-turn.start effects were not claimed until a title finished.
Reproduced on main with a dev server: the text-generation CLI was a stand-in whose exec never returns. Titles were regenerated on four threads, then a follow-up was sent. All four workers were running titles, and the follow-up's provider-turn.start was still pending 20 s later, with the thread starting and no reply.
The daemon now runs at most concurrency - 1 title generations at a time (3 of the default 4 workers). Every other effect type always has a worker. The cap is part of the shared claim predicate in EffectOutbox.ts, so claimNext and nextClaimableAt both apply it. A title held back by the cap neither gets claimed nor reports a past deadline, so idle workers sleep until a running title settles and wakes them. The count runs inside the atomic claim UPDATE, so concurrent workers cannot exceed the cap. With one worker there is no cap, as before. drain, runRecoveryOnce and direct runOnce() callers keep today's uncapped behaviour.
Fixes pingdotgg/t3code#25962
starts a provider turn while slow title generations are running (FoundationPersistence.test.ts) runs the daemon with three blocked title generations and asserts a provider-turn.start still runs. It fails on main with provider-turn.start was never claimed: thread title generations occupied every effect worker, and passes here.holds pending titles at the running-title cap without waking idle workers asserts that a capped title is not claimed and that nextClaimableAt returns none for it. It fails if the cap is dropped from nextClaimableAt, which would otherwise make idle workers re-poll every 25 ms.vp run check:changed passes: format, lint, knip, typecheck of 3 packages, 7 test files.T3CODE_DEV_SEED=0), using the same scenario. Turns ran on Claude Haiku, and title generation went to a stand-in Codex CLI whose exec hangs:Before, main 67661cfbb5:
| 1 | == before 67661cfbb5 03:18:57 regenerate titles on 4 threads (fake codex exec hangs) |
| 2 | == 03:19:00 t3 thread send mcp:98c7d725-8f6a-4e86-a8af-bf3edc01e21a 'Reply with only the word again' |
| 3 | Sent. |
| 4 | == 03:19:25 outbox, 20 s after the send |
| 5 | rowid thread effect_type status attempts created completed |
| 6 | ----- ------------ --------------------- ------- -------- ------------ --------- |
| 7 | 9 mcp:98c7d725 thread-title.generate running 1 03:18:43.878 |
| 8 | 10 mcp:98c7d725 thread-title.generate pending 0 03:18:57.413 |
| 9 | 11 mcp:cd45e4cc thread-title.generate running 1 03:18:57.599 |
| 10 | 12 mcp:e642a9b0 thread-title.generate running 1 03:18:57.752 |
| 11 | 13 mcp:4a134f19 thread-title.generate running 1 03:18:57.911 |
| 12 | 14 mcp:98c7d725 provider-turn.start pending 0 03:19:05.177 |
| 13 | == 03:19:31 t3 thread show (main) |
| 14 | Reply with only the word ok (1) |
| 15 | mcp:98c7d725-8f6a-4e86-a8af-bf3edc01e21a starting claudeAgent/claude-haiku-5-5 |
| 16 | |
| 17 | [assistant] |
| 18 | ok |
| 19 | |
| 20 | [user] |
| 21 | Reply with only the word again |
| 22 | == provider sessions |
| 23 | auth_sessions |
| 24 | orchestration_v2_projection_provider_session_bindings |
| 25 | orchestration_v2_projection_provider_sessions |
| 26 | projection_thread_sessions |
| 27 | provider_session_runtime |
After, 7a973e09c0, the fix commit; the later commits on this branch add a test and merge main:
| 1 | == after 7a973e09c0 03:16:20 regenerate titles on 4 threads (fake codex exec hangs) |
| 2 | == 03:16:23 t3 thread send mcp:d7cace28-d32b-44a8-8a9f-eac9c278d46f 'Reply with only the word again' |
| 3 | Sent. |
| 4 | == 03:16:48 outbox, 20 s after the send |
| 5 | rowid thread effect_type status attempts created completed |
| 6 | ----- ------------ --------------------- --------- -------- ------------ ------------ |
| 7 | 9 mcp:d7cace28 thread-title.generate running 1 03:16:05.442 |
| 8 | 10 mcp:d7cace28 thread-title.generate pending 0 03:16:20.374 |
| 9 | 11 mcp:f407cec7 thread-title.generate running 1 03:16:20.529 |
| 10 | 12 mcp:f1b9a763 thread-title.generate running 1 03:16:20.670 |
| 11 | 13 mcp:06a59adf thread-title.generate pending 0 03:16:20.801 |
| 12 | 14 mcp:d7cace28 provider-turn.start succeeded 1 03:16:27.941 03:16:28.039 |
| 13 | 15 mcp:d7cace28 checkpoint.capture succeeded 1 03:16:29.817 03:16:30.279 |
Implemented with Claude Opus 5.5 (Claude Code, T3 Code issue flow), reviewed by GPT-6.1 Sol (Codex) and Claude Opus 5.5.
🤖 Generated with Claude Code
Added 5 commits
Verdict: Needs Human Approval
Macroscope's review found this PR requires human approval — The daemon now limits title generation to three of its default four workers, preserving capacity for turn work. A human must confirm the default tradeoff between title-generation throughput and turn responsiveness.
You can add or adjust custom eligibility rules. Learn more.
✅ Thread transfer remains within every enforced ceiling.
| Provider | Metric | Main baseline | This PR | Impact | PR ceiling | |
|---|---|---|---|---|---|---|
| Codex | Total thread wire | 5.0 KiB | 5.0 KiB | 0 B (0.0%) | 6.8 KiB | ✅ |
| Codex | Thread snapshot wire | 3.8 KiB | 3.8 KiB | 0 B (0.0%) | 4.9 KiB | ✅ |
| Codex | Live turn WebSocket wire | 1.2 KiB | 1.2 KiB | 0 B (0.0%) | 2.0 KiB | ✅ |
| Codex | Live turn WebSocket decoded | 21.6 KiB | 21.6 KiB | 0 B (0.0%) | 24.4 KiB | ✅ |
| Codex | Live turn messages | 2 | 2 | 0 (0.0%) | 8 | ✅ |
| Claude | Total thread wire | 5.1 KiB | 5.1 KiB | 0 B (0.0%) | 6.8 KiB | ✅ |
| Claude | Thread snapshot wire | 3.8 KiB | 3.8 KiB | 0 B (0.0%) | 4.9 KiB | ✅ |
| Claude | Live turn WebSocket wire | 1.3 KiB | 1.3 KiB | 0 B (0.0%) | 2.0 KiB | ✅ |
| Claude | Live turn WebSocket decoded | 21.9 KiB | 21.9 KiB | 0 B (0.0%) | 24.4 KiB | ✅ |
| Claude | Live turn messages | 2 | 2 | 0 (0.0%) | 8 | ✅ |
Baseline: f120c51 · PR result: 19265c5 · Source CI: success
10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.
No actionable comments were generated in the recent review.
bf0b930b-f8ea-4fdc-8c13-6fd0965e21c0Reviewing files that changed from the base of the PR and between f120c51134c6b5f20b42f51cda84e3e6da681a5f and 19265c53a046c154f7194db9ccd683942e8a0560.
apps/server/src/orchestration-v2/EffectOutbox.tsapps/server/src/orchestration-v2/EffectWorker.test.tsapps/server/src/orchestration-v2/EffectWorker.tsapps/server/src/orchestration-v2/FoundationPersistence.test.tsapps/server/src/orchestration-v2/SteeringCompletion.integration.test.tsapps/server/src/orchestration-v2/runtimeLayer.test.tsapps/server/src/orchestration-v2/testkit/AcpRegistryReplayFixtures.integration.test.tsapps/server/src/serverRuntimeStartup.autoPull.test.tsapps/server/src/serverRuntimeStartup.test.tsIncluded review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.
The outbox now supports an optional limit on concurrently running title-generation effects. The worker daemon passes that limit to claim and deadline queries when concurrency exceeds one. Tests cover the limit, provider-turn scheduling, and updated worker method calls.
Changes: Title-generation scheduling
| Layer / File(s) | Summary |
|---|---|
Outbox claim and deadline filtering apps/server/src/orchestration-v2/EffectOutbox.ts | claimNext and nextClaimableAt accept an optional title-generation limit. The outbox applies it to title-generation candidates based on the number of currently running title-generation effects. |
Worker filter propagation apps/server/src/orchestration-v2/EffectWorker.ts | The worker accepts claim filters and forwards them to the outbox. When concurrency exceeds one, the daemon sets the title-generation limit to concurrency minus one and uses it for claims and deadline lookups. |
Scheduling tests and method call sites apps/server/src/orchestration-v2/*, apps/server/src/serverRuntimeStartup*.test.ts | Persistence tests cover capped title-generation claims, provider-turn scheduling, and deadline results. Tests and mocks call runOnce and nextClaimableAt as functions. |
Priority: External image Normal
Estimated code review effort: 3 (Moderate) | ~20 minutes
Change: Bug fix · Severity of issue fixed: Medium
| 1 | sequenceDiagram |
| 2 | participant EffectWorkerDaemon |
| 3 | participant OrchestrationEffectWorkerV2 |
| 4 | participant EffectOutboxV2 |
| 5 | EffectWorkerDaemon->>OrchestrationEffectWorkerV2: runOnce with title-generation limit |
| 6 | OrchestrationEffectWorkerV2->>EffectOutboxV2: claimNext with limit |
| 7 | EffectOutboxV2-->>OrchestrationEffectWorkerV2: claimable effect |
| 8 | EffectWorkerDaemon->>OrchestrationEffectWorkerV2: nextClaimableAt with same limit |
| 9 | OrchestrationEffectWorkerV2->>EffectOutboxV2: nextClaimableAt with limit |
| 10 | EffectOutboxV2-->>OrchestrationEffectWorkerV2: next claimable time |
Possibly related issues
pingdotgg/t3code issue 23877 — This change limits concurrent title-generation effects, but does not address the reported runtime cleanup or timeout cause.Suggested labels: bugfix
Suggested reviewers: t3dotgg, yash-singh1
19265No merge-blocking scheduling issue was established.
| Check name | Status | Explanation |
|---|---|---|
| Title check | Passed | The title clearly and concisely describes the main fix: slow title generations no longer consume all effect workers. |
| Description check | Passed | The description is complete and relevant. It explains the problem, reproduction, implementation, scope, linked issue, focused tests, validation results, runtime evidence, and remaining untested scenar… |
| Linked Issues check | Passed | The change addresses the coding requirement in #25962. EffectWorker.ts gives the daemon a concurrency - 1 title-generation limit when more than one worker exists. EffectOutbox.ts applies the sam… |
| Out of Scope Changes check | Passed | The changes stay within #25962. The API updates in EffectOutbox.ts and EffectWorker.ts support the claim filter. The test call-site updates and new persistence tests verify the changed behavior. N… |
Explanation
The description is complete and relevant. It explains the problem, reproduction, implementation, scope, linked issue, focused tests, validation results, runtime evidence, and remaining untested scenario.
Explanation
The change addresses the coding requirement in #25962. EffectWorker.ts gives the daemon a concurrency - 1 title-generation limit when more than one worker exists. EffectOutbox.ts applies the same limit in the atomic claim predicate and in nextClaimableAt. This leaves a worker available for provider-turn.start and prevents capped titles from causing repeated deadline polls. The new FoundationPersistence.test.ts tests cover provider-turn progress and the cap behavior. Recovery and direct runOnce() paths remain uncapped as described.
Explanation
The changes stay within #25962. The API updates in EffectOutbox.ts and EffectWorker.ts support the claim filter. The test call-site updates and new persistence tests verify the changed behavior. No unrelated production or UI changes are present in the reviewed diff.
Comment @coderabbitai help to get the list of available commands.