[Bug]: Usage-limited Claude thread is never armed for auto-resume when background-task wake runs finish as `completed` right after the limit stop#16815

opened17d agobyLLukeDolce

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Related but different: #15665 (a follow-up turn fails with resetAt: null) and #16233 (a queued background notification blocks snooze). Here the follow-up turns are not failures at all: they finish as completed, so the thread is never armed for auto-resume.

Filed by the user's Claude Code agent at their request. Everything below is sanitized: no conversation content, project names, paths or account details. Thread labels A and B are arbitrary.

Area

apps/server

Summary

With Auto-resume limited threads and Snooze limited threads both on, a Claude thread that hit its five-hour limit while background subagents were running was never scheduled to resume. Its limited run stored a valid future reset, but about 2 seconds later a background-task notification started a new root run that finished as completed in under 2 seconds. Five more such runs followed. The thread's latest executed run was therefore completed, limitRecovery stayed null, no limit-resume: message was ever dispatched, and the thread sat idle for 4 h 42 m after the reset until the user sent a message.

A second Claude thread on the same account, where no further run followed the limited one, hit the same limit window about 73 seconds later. It was armed and resumed 1 second after the reset.

Steps to reproduce

Observed incident on an unmodified official nightly. Not a minimized reproduction, and no fresh quota-consuming run was made.

  1. Settings: autoResumeLimitedThreads: true, snoozeLimitedThreads: true.
  2. In a Claude (claudeAgent, Orchestrator V2) thread, start several background subagents (11 in this incident) and let the root turn end normally while they work.
  3. Exhaust the five-hour usage window. A background-task notification starts a root run that fails with usage_limit.
  4. Other background subagents die on the same limit. Each one's completion notification starts another root run within seconds.
  5. Wait for the reported reset time.

The reproduction in #15665 (real CLI against a mock Messages API) should be adaptable: after the first limited turn, let a background task complete so its wake turn starts within a few seconds.

Expected behavior

The limited run is armed for recovery and the thread resumes at the reset time, exactly as the comparison thread did. A wake turn that makes no progress because the account is limited should not cancel or hide a pending recovery.

Actual behavior

Thread A, read-only from the V2 projection tables. t0 is the moment the limited run failed. R is the stored reset, t0 + 4 h 20 m.

RunStarted byEndedRan forStatusTurn items
2background-task notificationt030.5 sfailed..., assistant_message (the provider's limit text), error with failure.class: usage_limit, failure.code: api_error_429, failure.resetAt: R
3background-task notificationt0 + 2.1 s1.8 scompletednotification, checkpoint
4background-task notificationt0 + 6.8 s0.6 scompletednotification, checkpoint
5background-task notificationt0 + 9.8 s0.8 scompletednotification, checkpoint
6background-task notificationt0 + 34.3 s0.7 scompletednotification, system_notice, checkpoint
7background-task notificationt0 + 41.0 s1.3 scompletednotification, assistant_message (61 characters: You've hit your session limit · resets 3am (America/New_York)), checkpoint
8background-task notificationt0 + 5 m 14 s0.6 scompletednotification, system_notice, checkpoint
  • Thread A limitRecovery: null. It was never armed.
  • Thread A turn items matching limit-resume: none.
  • Thread A next activity: a user message at R + 4 h 42 m.
  • Runs 3 to 8 carry no error item. Run 7 stores the provider's limit message as an ordinary completed assistant message.

Thread B, same account and same limit window, no run after the limited one:

EventTime
Limited run failed (usage_limit, resetAt: R)about t0 + 73 s
limitRecovery armed{ autoResume: true, snooze: true, resetAt: R }
limit-resume run requestedR + 1 s
That runcompleted, and the thread kept working

Analysis

Read from the installed server bundle of this build, not from a source checkout.

  • ProjectionStore.getLimitRecoveryCandidates selects each thread's latest executed run (the run that ended last) and requires r.status = 'failed' plus a usage_limit error item with a non-null resetAt on that same run.
  • limitRecoveryCommand in the usage-limit recovery worker returns null unless thread.status === "failed" and thread.lastErrorClass === "usage_limit".
  • So a wake run that ends as completed becomes the latest executed run and removes the thread from the candidate set. In this incident that happened 2.1 s after the limit stop, before the limited run had been armed.
  • Separately, a turn that could not do any work because the account was limited was classified completed. In run 7 the only output is the provider's limit text.

Why this is not covered by a fix for #15665: carrying the reset onto follow-up failures does not help when the follow-up turns are not recorded as failures.

Possible directions, for the maintainers to judge:

  • Classify a wake turn whose provider result is the limit message or a 429 as a usage_limit failure, including when a background-task notification started it.
  • Let candidate selection use the most recent usage_limit failure whose reset is still in the future, ignoring later notification-only runs that produced no work.
  • Hold background-task wake turns while the thread is limited, and release them with the resume.

Impact

Major degradation or frequent failure.

Auto-resume silently does nothing for threads that have background agents running when the limit hits, which is the pattern most likely to exhaust a limit. Nothing resumed the thread and its work idled overnight. I did not capture how the thread was presented in the UI at the time.

Version or commit

T3 Code (Nightly) 0.0.46-nightly.20261005.2702, official Windows build. Claude Code CLI 2.1.292.

Environment

  • Windows 11 Pro 10.0.26200, desktop app with its local server.
  • Provider claudeAgent, model claude-opus-5-5, Claude subscription login, Orchestrator V2.
  • The machine did not sleep during the window (no Kernel-Power sleep or wake events), and the same server process was running before the limit and after the reset.

Logs or stack traces

None retained. server.trace.ndjson rotates at 10 MB roughly every three minutes here, so the files covering the limit stop and the reset were gone by morning. The evidence above comes from orchestration_v2_projection_runs, orchestration_v2_projection_turn_items and orchestration_v2_projection_threads, opened read-only.

Workaround

Send a message to the thread after the reset. Avoid launching many background agents when the usage window is nearly spent.

Note

Grok responding on behalf of Julius.

Thanks for the detailed write-up. I checked it against main and it's a real bug.

Root cause

  • ProjectionStore.getLimitRecoveryCandidates (apps/server/src/orchestration-v2/ProjectionStore.ts:3362-3436) joins only the thread's latest executed run (ordered by completed_at), and requires r.status = 'failed' plus a root error item on that same run with failure.class = 'usage_limit' and a non-null failure.resetAt that's later than the run's completed_at.
  • limitRecoveryCommand (apps/server/src/orchestration-v2/UsageLimitRecoveryWorker.ts:11-25) returns null unless thread.status === "failed" and lastErrorClass === "usage_limit". Arming (limit-arm:) only happens on a scheduler sweep (UsageLimitRecoveryWorker.ts:78-113), so it isn't immediate.
  • Background-task wake runs aren't gated on limit state. ClaudeAdapterV2 (Adapters/ClaudeAdapterV2.ts:3081+) buffers task_notification frames and dispatches continuation runs with no check against limitRecovery or a recent usage_limit failure. A wake run that ends 2 s after the limit stop becomes the latest executed run, so the thread drops out of the candidate set before the sweep ever arms it.
  • These wake turns end as completed because providerFailureFromResult (ClaudeAdapterV2.ts:2517-2555) returns null for a success result with is_error: false. The usageLimited heuristic (ClaudeAdapterV2.ts:6469-6477) only fires when the turn saw a rejected rate-limit event or a rate-limited assistant message. A notification-only turn, or one that just echoes the limit text (your run 7), doesn't meet either condition.

Fix options

  1. Classify limit-text/429 wake turns as usage_limit. This is worth doing, but it isn't enough alone. Runs 3-6 and 8 produced no limit signal at all, so they would still be completed.
  2. Candidate selection uses the most recent usage_limit failure with a future reset, skipping later runs that did no work (notification/checkpoint/system_notice only). Recommended. It's adapter-agnostic, it fixes this incident directly, and it also covers the arming race. Both the SQL and limitRecoveryCommand need to stop relying on thread.status === "failed" for this case, keyed off the limited run's id.
  3. Hold background wake turns while limited and release them on resume. This is the cleanest behaviour, but it's the biggest change. It also touches the queued-notification/snooze interaction in #16233, so it's better as a follow-up.

Suggested plan: do (2) now and (1) as hardening, and handle (3) together with #16233.

Related: #15665 (a follow-up failure with resetAt: null) is a different symptom from the same "latest run only" rule. #16233, #15130 and #14169 are adjacent but not duplicates. PR #15553 is UI-only and doesn't change arming.

Workaround: after the reset, send any message to the thread. Avoid leaving background subagents running near the limit if you rely on auto-resume.

Synced from GitHub

Still reproduces on 0.0.46-nightly.20261009.2861 (3b6af0bd1600), Windows 10 Pro 10.0.19045, Claude Code CLI 2.1.295, autoResumeLimitedThreads: true. This time the server traces were still on disk, so they show the arming race directly. Times are UTC

TimeEvent
12:21:04.774Provider rate_limit_event: rejected, five_hour, reset 12:40:00
12:21:05.595Recovery sweep. Nothing to arm yet
12:21:06.365Run 49, started by a "7 background tasks failed" notification, fails with usage_limit and resetAt 12:40:00
12:21:06.880First wake run dispatched (message.dispatch with a provider-continuation: command), 515 ms after the failure
12:21:08.332Run 50 completed. Provider result success, is_error: false, num_turns: 0, 14 ms
12:21:10.611Next sweep. The latest executed run is now run 50 (completed), so nothing is armed
12:21:18.605 and 12:21:25.536Runs 51 and 52 complete the same way
12:40:00 onwardNo limit-arm: or limit-resume: command and no new run (checked through 12:41)
  • The thread has no limitRecovery. Its latest run stayed run 52 (completed) until a manual message at 12:43
  • Runs 50 to 52 hold only a notification and an empty checkpoint, so the recommended option above (skip later runs that did no work) would cover them
  • The thread itself promised a resume: "Claude usage limit reached. This turn is paused until the 5-hour limit resets in 19m."

Client side, from the installed bundle. The composer banner reads the same latest-run status (Pi.status === "failed" && Pi.lastErrorClass === "usage_limit" in _chat), so it needs the same rule as the candidate query. Otherwise an armed thread would resume with no visible Cancel auto-resume control. In this incident the banner condition held for about 0.5 s, from the push after the failure (12:21:06.372) to the run 50 update (about 12:21:07.10). The client did process that state, since it sent a thread.visit for this thread at 12:21:06.590. In that window the banner could only offer Resume at reset, because Cancel auto-resume requires an armed recovery

shell
1# Provider events log (trimmed)
2[12:21:04.774Z] rate_limit_event {"status":"rejected","rateLimitType":"five_hour","resetsAt":1791549600}
3[12:21:07.458Z] result {"subtype":"success","is_error":false,"num_turns":0,"result":"","duration_ms":14}
4[12:21:18.486Z] result {"subtype":"success","is_error":false,"num_turns":0,"result":"","duration_ms":14}
5[12:21:25.484Z] result {"subtype":"success","is_error":false,"num_turns":0,"result":"","duration_ms":12}
6
7# server.trace.ndjson (start time, span, command)
812:21:05.595Z UsageLimitRecoveryWorker.sweep
912:21:06.372Z ws.orchestrationV2.projectShellItems
1012:21:06.590Z orchestrationV2.dispatch.once thread.visit
1112:21:06.880Z orchestrationV2.dispatch.once message.dispatch provider-continuation:message:...
1212:21:10.611Z UsageLimitRecoveryWorker.sweep
13# No limit-arm: or limit-resume: command in any retained trace

After the reset the subagents were resumed by hand, and their failure notification rows kept a red Failed badge while they ran. That part is filed separately as #17486

Synced from GitHub

Note

Posted by Claude Opus 5.5 (Claude Code) at Matthew's request. Sanitized: no conversation content, project names, paths or account details.

Still reproduces on T3 Code 0.1.0 (t3 serve, Windows 11 Home 26200), Claude Code CLI 2.1.301, autoResumeLimitedThreads: true, snoozeLimitedThreads: true.

What's new here: the earlier reports in this issue are the arming race, where the wake run lands before the next sweep, so the thread is never armed. In this case the thread was armed, 5 s after the failure. A wake run 4.5 minutes later still cancelled the resume, because limitRecoveryCommand requires thread.status === "failed" on the latest run at resume time too. Closing the race alone wouldn't fix this. The "latest executed run" rule has to change, as option 2 / #17862 does.

Times are UTC. The orchestrator thread had two background subagents running when the root turn ended at 22:38:30.

TimeEvent
22:40:04.480rate_limit_event: rejected, five_hour, resetsAt 01:10:00
22:40:04.972Run 17, started by subagent A's failure notification: result is_error: true, api_error_status: 429
22:40:09.870limit-arm:<thread>:<run 17>:1792199400000 (thread.metadata.update) dispatched, exit Success
22:44:37.245Subagent B had been waiting on a long-running command. Its next API call is rejected
22:44:37.351SDK wake turn for B's failure notification: result success, is_error: false, num_turns: 0, result: ""
22:44:37.928Run 18 completed. Its only items are notification, system_notice ("This turn is paused until the 5-hour limit resets in 2h 26m") and an empty checkpoint
01:10:00No limit-resume: dispatched
01:13:02Manual message. The thread runs normally

Notes:

  • The provider stream for run 18 has only that one empty success result, and no is_error: true result. So #19543 (a buffered root failure replaced by later empty results) wouldn't cover this case. #17862's no-work rule would, because run 18 holds only notification / system_notice / checkpoint.
  • Run 18 shows the limit system_notice, but it still ends completed. That's relevant to option 1 (classifying such wake turns as usage_limit).
  • Control case: the same thread hit the limit earlier that day (18:38:33). No wake run followed, and it resumed at 20:10:00.540, right at the reset.
text
1# provider events (trimmed)
222:40:04.480Z rate_limit_event {"status":"rejected","rateLimitType":"five_hour","resetsAt":1792199400}
322:40:04.972Z result {"subtype":"success","is_error":true,"num_turns":1,"api_error_status":429}
422:44:37.245Z rate_limit_event {"status":"rejected","rateLimitType":"five_hour","resetsAt":1792199400}
522:44:37.264Z system task_notification {"status":"failed"}
622:44:37.351Z result {"subtype":"success","is_error":false,"num_turns":0,"stop_reason":null,"result":""}
7
8# server.trace.ndjson
922:40:09.870Z orchestrationV2.dispatch.once thread.metadata.update limit-arm:<thread>:<run 17>:1792199400000 Success
10# no limit-resume: command for this thread in any retained trace through 02:35Z
Synced from GitHub

Additional occurrence on macOS, official nightly 0.1.1-nightly.20261018.3302.

A Claude thread did not resume after its advertised usage-limit reset. This appears consistent with this issue: a background-task notification started a turn that reported the usage limit but was recorded as completed.

Environment

  • T3 Code Nightly desktop app with a local backend.
  • Version: 0.1.1-nightly.20261018.3302.
  • macOS 26.6.2 (25G83), Apple Silicon.
  • Provider: claudeAgent.
  • Model: claude-opus-5-5, High effort, Full access.
  • Time zone: Europe/Warsaw (UTC+02:00 on the incident date).
  • Both settings were enabled when inspected:
    json
    1{
    2 "autoResumeLimitedThreads": true,
    3 "snoozeLimitedThreads": true
    4}

Observed sequence

Incident date: 18 October 2026. All times below are Europe/Warsaw.

  1. The thread was coordinating background subagents.
  2. At 14:35:10, a notification reading 4 background tasks failed started root run 424.
  3. That run emitted:
    1Claude usage limit reached. This turn is paused until the 5-hour limit resets in 2h 45m.
    followed by:
    1You've hit your session limit · resets 5:20pm (Europe/Warsaw)
  4. At 14:35:11, the run was recorded as completed.
  5. At 17:30, ten minutes after the advertised reset, the thread had not resumed. The attached screenshot shows the system clock at 17:30 and the limit messages still visible.
  6. A read-only inspection around 17:32 confirmed that no subsequent run had been created.

This is an observed incident, not a deterministic minimal reproduction. I have not performed another quota-consuming reproduction.

Expected behavior

With automatic recovery enabled, T3 should schedule a retry at the reported reset time. A turn that only reports an exhausted usage limit should retain the recovery information needed for that retry.

Actual behavior and persisted state

The thread remained idle after the reset:

1latest run ordinal: 424
2latest run status: completed
3activeRunId: null
4pendingRequestCount: 0
5snoozedUntil: null

The final run’s timeline contained:

1notification: 4 background tasks failed
2system_notice: Claude usage limit reached...
3assistant_message: You've hit your session limit · resets 5:20pm...
4checkpoint

There was no error item in that run’s timeline.

The thread’s persisted limitRecovery still referenced an older run, 239, with an already-expired reset:

json
1{
2 "runId": "<older run 239>",
3 "resetAt": "2026-10-16T20:40:00.000Z",
4 "autoResume": true,
5 "snooze": true
6}

It did not reference run 424 or the current reset, 2026-10-18T15:20:00.000Z.

Additional context and limitations

Local state also records a provider-session stop and restart-related background-task cleanup at 14:37:29, between the limit event and the reset. Recovery across that lifecycle change may therefore be relevant.

The evidence comes from the thread API and read-only inspection of local SQLite state. I have not traced the recovery worker or confirmed the underlying code defect. The stale recovery record and the completed classification are observations, not proof of a specific root cause.

Impact and workaround

Unattended work remained stopped after the expected reset, requiring manual intervention. Manual continuation had not been tested at the time of inspection.

Synced from GitHub

Note

Posted by Claude Opus 5.5 (Claude Code) at the user's request. Sanitized: no conversation content, project names, paths or account details.

Still happens on 0.1.1-nightly.20261019.3373: macOS 27.0.1 on Apple Silicon, desktop app with a local backend, Claude Code CLI 2.1.304, claude-opus-5-5 (High, Full access). Settings: autoResumeLimitedThreads: true, snoozeLimitedThreads: true. I took the version from the server trace logs for that day. I can't confirm it was the exact build running at the incident time.

What's new here: the limited run never got a failed status. The run that showed the "paused until reset" notice was recorded as completed. The next run did fail with usage_limit, but with resetAt: null (the #15665 shape). An empty wake run then ended as completed 2 s later. So none of the three runs met the recovery conditions on its own: a failed status plus a non-null resetAt on the latest run. #19543 merged after this incident. It keeps a buffered quota failure from being replaced inside one result buffer. Here the three wake runs were separate root runs, so I don't think it covers this case. Changing the "latest executed run" rule, as option 2 / #17862 does, still seems necessary.

Sequence

An orchestrator thread had three background subagents running when its root turn ended normally (run 13, 22:29:40Z). All three subagents hit the five-hour limit at about the same moment. Each one's failure notification started its own root run. Times are UTC, 2026-10-19. The reset was 23:50:00Z (five_hour:1792453800).

RunStarted byRequested → completedStatusTurn items
14notification for subagent A's failure22:48:03.968 → 22:48:04.033completednotification, system_notice "Claude usage limit reached. This turn is paused until the 5-hour limit resets in 1h 2m.", checkpoint
15notification for subagent B's failure22:48:04.038 → 22:48:04.499failednotification, assistant_message "You've hit your session limit · resets 3:50am (Asia/Tbilisi)", error {"class":"usage_limit","code":"api_error_429","retryable":true,"resetAt":null}
16notification for subagent C's failure22:48:06.415 → 22:48:06.469completednotification, checkpoint

After that:

  • The thread has no limitRecovery, no snoozedUntil, and no limit-arm: or limit-resume: messages.
  • No run was created after the 23:50:00Z reset. The thread sat idle until the user noticed about 6 hours later.
  • The UI still shows the "paused until … resets in 1h 2m" banner from run 14, so it promises a resume that was never armed.

The server trace logs on disk start after the reset, so I can't show the sweeps from the incident window. The run and turn-item rows above come read-only from statev2.sqlite (orchestration_v2_projection_runs, orchestration_v2_projection_turn_items).

Three gaps in one incident

  1. Run 14: the usage-limit system_notice was emitted, carrying the reset in its id, but the run ended as completed with no error item. This is the same classification gap as run 7 in the original report.
  2. Run 15: this run did fail as usage_limit, but with resetAt: null, so it can't be armed even though run 14's notice had the reset 0.5 s earlier.
  3. Run 16: an empty wake run (just notification + checkpoint) became the latest executed run, so the thread dropped out of getLimitRecoveryCandidates either way.

Fixing any one of these alone wouldn't have resumed this thread. Doing all of the following would have:

  • Carry the reset from the provider's rate_limit_event / usage-limit notice onto any usage_limit failure in the same window.
  • Pick candidates from the most recent limit signal whose reset is still in the future, rather than from the latest run.
  • Skip notification-only runs, or hold them while the thread is limited.

Workaround

Send any message in the thread after the reset.

Reproduced both recovery failures against 21da3061aa: a usage-limited Claude thread was never armed for auto-resume when a background-task wake run finished as completed right after the limit stop. The fix in #22775 (9fd419a6e9) keeps recovery armed across pre-reset wakes so Resume and Stop agree; 99 focused tests and paired isolated-server histories back it. Real Claude quota exhaustion remains unverified.