fix(server): usage-limit auto-resume retries a continuation refused right after the reset#33737
Problem
When usage-limit auto-resume sends "Continue where you left off." at the reset and the provider refuses that continuation with usage_limit and no reset later than now, recovery stops for good. Codex records the refusal with resetAt: null; a provider can also repeat the window that just expired. The recovery worker and its candidate query drop any failure without a reset later than the run's end, so the thread stays failed until the user continues it by hand.
Reproduced on main with the real orchestrator, event sink, SQLite projection and recovery sweep: run R0 fails usage_limit with a future reset, auto-resume arms and dispatches the continuation R1 at the reset, R1 fails usage_limit with a null or expired reset, and no continuation of R1 is ever created.
Change
- A continuation that the provider refuses with no later reset now retries 2, 5 and then 15 minutes after the refusal. The fourth refusal in a row gives up and leaves the thread in today's failed state. The count comes from the existing
usageLimitContinuationOfRunIdchain, so there is no new persisted field, setting or wire field. - One shared rule (
usageLimitRecoveryResetAtinpackages/shared) decides the retry time. Arm validation, continuation validation, Stop, the recovery worker and the candidate SQL all use it. The SQL bounds the chain with three indexed lookups, so a thread that gave up is no longer a candidate. - A "Resume at reset" chosen for the refused run carries over to its retry. An automatic arm keeps a manual snooze that ends later than the retry time.
- The web banner and the mobile card show the armed retry time with Cancel auto-resume, and Wake now while the retry owns a snooze. After Cancel, Stop or giving up, they show the provider's reset as before. Worktree cleanup holds a thread while its retry is armed.
- The thread sidebar guide now says that a refused scheduled continuation is retried a few times.
Scope and approval
Fixes pingdotgg/t3code#33708
Verification
- New rows in
apps/server/src/orchestration-v2/runtimeLayer.test.ts(suiteusage-limit recovery,retries a usage-limit continuation the provider refuses with …):no-reset,expired-reset,no-reset under a per-thread resume, andexpired-reset past a later manual snooze. They check that no continuation exists 1 ms before each deadline and that one exists at it. After the fourth refusal they check that the thread is not re-armed and is no longer returned bygetLimitRecoveryCandidates, with global auto-resume on and off.- On the test-only commit
8bfc0a1c0d:2 failed,AssertionError: a refused continuation with no later reset should be retried after a backoff: expected undefined to not equal undefined. - At
cbe1b7d96b: all rows pass.
- On the test-only commit
- New rows in
apps/server/src/storageCleanup.test.ts: a missing or expired provider reset with an armed retry is held; the same with auto-resume off is not. - New rows in
packages/client-runtime/src/state/threadExecution.test.tsfor the notice reset: armed retry, cancelled, after Stop, an owned snooze still pending, and one that has passed. - Focused suites pass:
runtimeLayer,UsageLimitRecoveryWorker,ProjectionStore,Orchestrator.control-reads,DelegatedCompletionDelivery,TaskCancelUsageLimit.integration,storageCleanup,threadExecution,UsageLimitRecoveryCard,ChatView.logic.vp run check:changedpasses: format, lint, knip, typecheck of 35 packages, 14 test files, react compiler.
| State | Before | After | Diff0 | DiffT |
|---|---|---|---|---|
| Web banner and mobile card: refused continuation, retry armed | not attached | not attached | — | — |
| Web banner and mobile card: after Cancel or Stop | not attached | not attached | — | — |
| Web banner and mobile card: retry owns a snooze (Wake now) | not attached | not attached | — | — |
| SHA | 934c7f8d8e | cbe1b7d96b |
Not checked: before/after captures of the web banner and mobile card in these states are not attached yet. The client change is covered only by the threadExecution rows above. A live provider refusal at a real reset was not exercised; the provider failure is injected as provider events.
AI-assisted; verified with the tests and tools listed above.
Fixes pingdotgg/t3code#22026
Checks
Added 6 commits
- fix(server): a refused usage-limit continuation retries with backoff0fe78b1
- fix(web,mobile): show the armed usage-limit retry and its cancel control567a198
- fix: offer a refused continuation's retry time only while it is armed5bc331d
- fix(server): usage-limit retries stop for good and keep a later snooze85c2d6e
- test(server): reproduce #33708 refused usage-limit continuation never re-arms8bfc0a1
- Merge remote-tracking branch 'origin/main' into fix/issue-33708cbe1b7d
Approvability
Verdict: Approved at cbe1b7d
Macroscope's review found this PR approvable — Adds bounded backoff retries for already-enabled usage-limit recovery, with matching notices and cleanup protection. Automatic recovery remains off by default, and retries respect existing snoozes and thread-state safeguards.
You can add or adjust custom eligibility rules. Learn more.
Thread transfer impact
✅ Thread transfer remains within every enforced ceiling.
| Provider | Metric | Main baseline | This PR | Impact | PR ceiling | |
|---|---|---|---|---|---|---|
| Codex | Total thread wire | 5.0 KiB | 5.0 KiB | 0 B (0.0%) | 6.8 KiB | ✅ |
| Codex | Thread snapshot wire | 3.8 KiB | 3.8 KiB | 0 B (0.0%) | 4.9 KiB | ✅ |
| Codex | Live turn WebSocket wire | 1.2 KiB | 1.2 KiB | 0 B (0.0%) | 2.0 KiB | ✅ |
| Codex | Live turn WebSocket decoded | 21.5 KiB | 21.5 KiB | 0 B (0.0%) | 24.4 KiB | ✅ |
| Codex | Live turn messages | 1 | 1 | 0 (0.0%) | 8 | ✅ |
| Claude | Total thread wire | 5.0 KiB | 5.0 KiB | 0 B (0.0%) | 6.8 KiB | ✅ |
| Claude | Thread snapshot wire | 3.8 KiB | 3.8 KiB | 0 B (0.0%) | 4.9 KiB | ✅ |
| Claude | Live turn WebSocket wire | 1.2 KiB | 1.2 KiB | 0 B (0.0%) | 2.0 KiB | ✅ |
| Claude | Live turn WebSocket decoded | 21.9 KiB | 21.9 KiB | 0 B (0.0%) | 24.4 KiB | ✅ |
| Claude | Live turn messages | 1 | 1 | 0 (0.0%) | 8 | ✅ |
Baseline: 4c36119 · PR result: cbe1b7d · Source CI: success
Scenario and decoded snapshot size
10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
- Codex decoded thread snapshot: 108.5 KiB
- Claude decoded thread snapshot: 108.8 KiB
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.
No actionable comments were generated in the recent review.
Walkthrough
-
The changes add shared helpers for usage-limit recovery timing and bounded retries after refused continuations. Server recovery and cleanup use derived reset times. Mobile and web views use the same timing for recovery notices and snooze handling.
Changes: Usage-limit recovery
Layer / File(s) Summary Shared recovery timing
packages/shared/src/orchestrationV2ThreadError.ts,packages/client-runtime/src/state/threadExecution.ts,packages/client-runtime/src/state/threadExecution.test.tsShared helpers derive provider reset times, continuation retry times, continuation chains, and notice reset times. The client runtime re-exports the notice helper, and tests cover its timing results. Continuation retry scheduling
apps/server/src/orchestration-v2/ProjectionStore.ts,apps/server/src/orchestration-v2/UsageLimitRecoveryWorker.ts,apps/server/src/orchestration-v2/ProjectionStore.test.ts,apps/server/src/orchestration-v2/runtimeLayer.test.tsRecovery candidates include qualifying refused continuations and can carry forward per-thread auto-resume settings. Tests cover retries after 2, 5, and 15 minutes, snooze deadlines, and the retry limit. Server recovery state
apps/server/src/orchestration-v2/Orchestrator.ts,apps/server/src/storageCleanup.ts,apps/server/src/storageCleanup.test.tsOrchestration validates and initializes recovery with derived reset times and continuation history. Storage cleanup uses derived notice reset times. Tests cover missing and expired provider resets. Recovery surfaces
apps/mobile/src/features/threads/ThreadDetailScreen.tsx,apps/mobile/src/features/threads/UsageLimitRecoveryCard.tsx,apps/web/src/components/ChatView.tsx,docs/user/thread-sidebar.mdMobile and web views derive reset times for snooze ownership and recovery notices. The documentation describes retry behavior and conditions that cancel or pause retries. Estimated code review effort: 4 (Complex) | ~45 minutes
Change: Bug fix · Severity of issue fixed: Medium
Suggested reviewers:
shivamhwp,t3dotgg
Merge Risk: External image Minimal · up to cbe1b
-
The change adds bounded retries for refused usage-limit continuations, with consistent timing across the server, web and mobile surfaces. No actionable merge-blocking issue was found.
Pre-merge checks: Passed 3 · Failed 1
-
Failed Failed checks: 1 warning
Check name Status Explanation Resolution Description check Warning The description explains the problem, change, and verification in detail, but it does not include the required maintainer approval comment for this broader behavior change. It also states that the req… Add a link to the triaged issue or approval discussion and include the maintainer comment that approves the direction and scope. Attach before-and-after screenshots of the web banner and mobile card states required by the template, includin… Passed Passed checks: 3 passed
-
Check name Status Explanation Title check Passed The title clearly describes the main change: retrying usage-limit continuations that the provider refuses after the reset. Linked Issues check Passed Active direct target: #33708. Its coding requirements are: (1) a usage-limit continuation refused with no later reset keeps auto-resume armed; (2) retries use bounded backoff; (3) recovery gives up in…Out of Scope Changes check Passed Each changed area supports the retry objective of #33708:
- Shared rule and server logic (orchestrationV2ThreadError.ts, ProjectionStore.ts, UsageLimitRecoveryWorker.ts, Orchestrator.ts): implement th…Full details: Linked Issues check
-
Explanation
Active direct target:
#33708. Its coding requirements are: (1) a usage-limit continuation refused with no later reset keeps auto-resume armed; (2) retries use bounded backoff; (3) recovery gives up into the existing failed state; (4) retries respect cancel, settle, snooze, and archive.Requirements 1-3 [
#33708]:refusedContinuationRetryAtin packages/shared/src/orchestrationV2ThreadError.ts uses 2, 5, and 15 minute steps and returns null after the third step.usageLimitRecoveryResetAtcounts consecutive refusals alongusageLimitContinuationChain. In ProjectionStore.ts,getLimitRecoveryCandidatesadds a refused-continuation branch. It computes the retry time from the refusal count and excludes a thread when a fourth refusal exists (c3.run_id IS NULL). The worker and Orchestrator use the same shared rule, and the worker carries a per-thread auto-resume to the refused run.Requirement 4 [
#33708]: The candidate SQL keepsdeleted_at IS NULL,archivedAt IS NULL,settledOverride IS NOT 'settled', and the snooze check. For Cancel, the new carry-over branch requireslimitRecovery.autoResume = 1, so a cancelled retry is not a candidate even when global auto-resume is on. Stop uses the same shared rule.Author-reported evidence: runtimeLayer rows for no-reset, expired-reset, per-thread resume, and manual snooze pass at cbe1b7d96b. The test-only commit 8bfc0a1c0d fails as expected. I did not run these tests. This verdict relies on the diff and the author's report.
Minor uncertainty: the shared helper falls back to
requestedAtwhencompletedAtis null, while the SQL falls back to threadupdatedAt. This affects only a failed run with nocompletedAt. This review did not establish that such a run occurs.The optional 30-60 s grace delay in the issue is a suggestion. The issue states that a delay alone does not fix the defect. Its absence is not a failure.
Full details: Out of Scope Changes check
-
Explanation
Each changed area supports the retry objective of
#33708:- Shared rule and server logic (orchestrationV2ThreadError.ts, ProjectionStore.ts, UsageLimitRecoveryWorker.ts, Orchestrator.ts): implement the retry, count, give-up, and carry-over behavior.
- Tests (ProjectionStore.test.ts, runtimeLayer.test.ts, storageCleanup.test.ts, threadExecution.test.ts): cover the changed server and client paths.
- Web and mobile UI (ChatView.tsx, ThreadDetailScreen.tsx, UsageLimitRecoveryCard.tsx, threadExecution.ts): show the armed retry time and the Cancel and Wake controls. The issue states that the UI gives no indication of an abandoned recovery. After Cancel, Stop, or give-up, the UI shows the provider reset as before.
- storageCleanup.ts: holds a thread while its retry is armed. This prevents cleanup from removing state that the retry needs.
- docs/user/thread-sidebar.md: documents the retry behavior.
No changed file was shown to be unrelated to
#33708. Before/after UI captures are not attached. This limits verification of the UI, not scope.
-
Full details: Description check
-
Explanation
The description explains the problem, change, and verification in detail, but it does not include the required maintainer approval comment for this broader behavior change. It also states that the required before-and-after UI screenshots are not attached.
Resolution
Add a link to the triaged issue or approval discussion and include the maintainer comment that approves the direction and scope. Attach before-and-after screenshots of the web banner and mobile card states required by the template, including the armed retry, after Cancel or Stop, and retry-owned snooze states.
-
Finishing Touches
-
Generate docstrings
-
- Commit to this branch
- Create a new PR
Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
External image Recent review info
-
External image Run configuration
-
- Source mode: Auto
- Repository: pingdotgg/t3code
- Branch: fix/issue-33708
- File: .coderabbit.config.ts
- Commit: cbe1b7d
- Layers: Repository TS config
- Review profile: CHILL
- Plan: Team
- Run ID:
e31b8622-60a2-4772-9c65-fcf5408a230c
External image Commits
-
Reviewing files that changed from the base of the PR and between 4c361199f47f6ea32a2c14835df392640d0f7c63 and cbe1b7d96b84ea71b8f3b9a3857124e9b8edabd9.
External image Files selected for processing: 14
-
apps/mobile/src/features/threads/ThreadDetailScreen.tsxapps/mobile/src/features/threads/UsageLimitRecoveryCard.tsxapps/server/src/orchestration-v2/Orchestrator.tsapps/server/src/orchestration-v2/ProjectionStore.test.tsapps/server/src/orchestration-v2/ProjectionStore.tsapps/server/src/orchestration-v2/UsageLimitRecoveryWorker.tsapps/server/src/orchestration-v2/runtimeLayer.test.tsapps/server/src/storageCleanup.test.tsapps/server/src/storageCleanup.tsapps/web/src/components/ChatView.tsxdocs/user/thread-sidebar.mdpackages/client-runtime/src/state/threadExecution.test.tspackages/client-runtime/src/state/threadExecution.tspackages/shared/src/orchestrationV2ThreadError.ts
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.
External image Recent review info
-
External image Run configuration
-
- Source mode: Auto
- Repository: pingdotgg/t3code
- Branch: fix/issue-33708
- File: .coderabbit.config.ts
- Commit: cbe1b7d
- Layers: Repository TS config
- Review profile: CHILL
- Plan: Team
- Run ID:
e31b8622-60a2-4772-9c65-fcf5408a230c
External image Commits
-
Reviewing files that changed from the base of the PR and between 4c361199f47f6ea32a2c14835df392640d0f7c63 and cbe1b7d96b84ea71b8f3b9a3857124e9b8edabd9.
External image Files selected for processing: 14
-
apps/mobile/src/features/threads/ThreadDetailScreen.tsxapps/mobile/src/features/threads/UsageLimitRecoveryCard.tsxapps/server/src/orchestration-v2/Orchestrator.tsapps/server/src/orchestration-v2/ProjectionStore.test.tsapps/server/src/orchestration-v2/ProjectionStore.tsapps/server/src/orchestration-v2/UsageLimitRecoveryWorker.tsapps/server/src/orchestration-v2/runtimeLayer.test.tsapps/server/src/storageCleanup.test.tsapps/server/src/storageCleanup.tsapps/web/src/components/ChatView.tsxdocs/user/thread-sidebar.mdpackages/client-runtime/src/state/threadExecution.test.tspackages/client-runtime/src/state/threadExecution.tspackages/shared/src/orchestrationV2ThreadError.ts
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.
-