fix(server): usage-limit auto-resume retries a continuation refused right after the reset#33737

open6
main
fix/issue-33708
opened6h agobyJJulius Marminge
View on GitHubTwo-way sync enabledPR last imported 2h ago

Problem

When usage-limit auto-resume sends "Continue where you left off." at the reset and the provider refuses that continuation with usage_limit and no reset later than now, recovery stops for good. Codex records the refusal with resetAt: null; a provider can also repeat the window that just expired. The recovery worker and its candidate query drop any failure without a reset later than the run's end, so the thread stays failed until the user continues it by hand.

Reproduced on main with the real orchestrator, event sink, SQLite projection and recovery sweep: run R0 fails usage_limit with a future reset, auto-resume arms and dispatches the continuation R1 at the reset, R1 fails usage_limit with a null or expired reset, and no continuation of R1 is ever created.

Change

  • A continuation that the provider refuses with no later reset now retries 2, 5 and then 15 minutes after the refusal. The fourth refusal in a row gives up and leaves the thread in today's failed state. The count comes from the existing usageLimitContinuationOfRunId chain, so there is no new persisted field, setting or wire field.
  • One shared rule (usageLimitRecoveryResetAt in packages/shared) decides the retry time. Arm validation, continuation validation, Stop, the recovery worker and the candidate SQL all use it. The SQL bounds the chain with three indexed lookups, so a thread that gave up is no longer a candidate.
  • A "Resume at reset" chosen for the refused run carries over to its retry. An automatic arm keeps a manual snooze that ends later than the retry time.
  • The web banner and the mobile card show the armed retry time with Cancel auto-resume, and Wake now while the retry owns a snooze. After Cancel, Stop or giving up, they show the provider's reset as before. Worktree cleanup holds a thread while its retry is armed.
  • The thread sidebar guide now says that a refused scheduled continuation is retried a few times.

Scope and approval

Fixes pingdotgg/t3code#33708

Verification

  • New rows in apps/server/src/orchestration-v2/runtimeLayer.test.ts (suite usage-limit recovery, retries a usage-limit continuation the provider refuses with …): no-reset, expired-reset, no-reset under a per-thread resume, and expired-reset past a later manual snooze. They check that no continuation exists 1 ms before each deadline and that one exists at it. After the fourth refusal they check that the thread is not re-armed and is no longer returned by getLimitRecoveryCandidates, with global auto-resume on and off.
    • On the test-only commit 8bfc0a1c0d: 2 failed, AssertionError: a refused continuation with no later reset should be retried after a backoff: expected undefined to not equal undefined.
    • At cbe1b7d96b: all rows pass.
  • New rows in apps/server/src/storageCleanup.test.ts: a missing or expired provider reset with an armed retry is held; the same with auto-resume off is not.
  • New rows in packages/client-runtime/src/state/threadExecution.test.ts for the notice reset: armed retry, cancelled, after Stop, an owned snooze still pending, and one that has passed.
  • Focused suites pass: runtimeLayer, UsageLimitRecoveryWorker, ProjectionStore, Orchestrator.control-reads, DelegatedCompletionDelivery, TaskCancelUsageLimit.integration, storageCleanup, threadExecution, UsageLimitRecoveryCard, ChatView.logic. vp run check:changed passes: format, lint, knip, typecheck of 35 packages, 14 test files, react compiler.
StateBeforeAfterDiff0DiffT
Web banner and mobile card: refused continuation, retry armednot attachednot attached——
Web banner and mobile card: after Cancel or Stopnot attachednot attached——
Web banner and mobile card: retry owns a snooze (Wake now)not attachednot attached——
SHA934c7f8d8ecbe1b7d96b

Not checked: before/after captures of the web banner and mobile card in these states are not attached yet. The client change is covered only by the threadExecution rows above. A live provider refusal at a real reset was not exercised; the provider failure is injected as provider events.

AI-assisted; verified with the tests and tools listed above.

Fixes pingdotgg/t3code#22026

Checks

24 successful · 13 skipped
All checks passed
cbe1b7d

Added 6 commits

  • fix(server): a refused usage-limit continuation retries with backoff0fe78b1
  • fix(web,mobile): show the armed usage-limit retry and its cancel control567a198
  • fix: offer a refused continuation's retry time only while it is armed5bc331d
  • fix(server): usage-limit retries stop for good and keep a later snooze85c2d6e
  • test(server): reproduce #33708 refused usage-limit continuation never re-arms8bfc0a1
  • Merge remote-tracking branch 'origin/main' into fix/issue-33708cbe1b7d
Synced from GitHub

Approvability

Verdict: Approved at cbe1b7d

Macroscope's review found this PR approvable — Adds bounded backoff retries for already-enabled usage-limit recovery, with matching notices and cleanup protection. Automatic recovery remains off by default, and retries respect existing snoozes and thread-state safeguards.

You can add or adjust custom eligibility rules. Learn more.

Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire5.0 KiB5.0 KiB0 B (0.0%)6.8 KiB✅
CodexThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
CodexLive turn WebSocket wire1.2 KiB1.2 KiB0 B (0.0%)2.0 KiB✅
CodexLive turn WebSocket decoded21.5 KiB21.5 KiB0 B (0.0%)24.4 KiB✅
CodexLive turn messages110 (0.0%)8✅
ClaudeTotal thread wire5.0 KiB5.0 KiB0 B (0.0%)6.8 KiB✅
ClaudeThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
ClaudeLive turn WebSocket wire1.2 KiB1.2 KiB0 B (0.0%)2.0 KiB✅
ClaudeLive turn WebSocket decoded21.9 KiB21.9 KiB0 B (0.0%)24.4 KiB✅
ClaudeLive turn messages110 (0.0%)8✅

Baseline: 4c36119 · PR result: cbe1b7d · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Synced from GitHub

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The changes add shared helpers for usage-limit recovery timing and bounded retries after refused continuations. Server recovery and cleanup use derived reset times. Mobile and web views use the same timing for recovery notices and snooze handling.

Changes: Usage-limit recovery

Layer / File(s)Summary
Shared recovery timing
packages/shared/src/orchestrationV2ThreadError.ts, packages/client-runtime/src/state/threadExecution.ts, packages/client-runtime/src/state/threadExecution.test.ts
Shared helpers derive provider reset times, continuation retry times, continuation chains, and notice reset times. The client runtime re-exports the notice helper, and tests cover its timing results.
Continuation retry scheduling
apps/server/src/orchestration-v2/ProjectionStore.ts, apps/server/src/orchestration-v2/UsageLimitRecoveryWorker.ts, apps/server/src/orchestration-v2/ProjectionStore.test.ts, apps/server/src/orchestration-v2/runtimeLayer.test.ts
Recovery candidates include qualifying refused continuations and can carry forward per-thread auto-resume settings. Tests cover retries after 2, 5, and 15 minutes, snooze deadlines, and the retry limit.
Server recovery state
apps/server/src/orchestration-v2/Orchestrator.ts, apps/server/src/storageCleanup.ts, apps/server/src/storageCleanup.test.ts
Orchestration validates and initializes recovery with derived reset times and continuation history. Storage cleanup uses derived notice reset times. Tests cover missing and expired provider resets.
Recovery surfaces
apps/mobile/src/features/threads/ThreadDetailScreen.tsx, apps/mobile/src/features/threads/UsageLimitRecoveryCard.tsx, apps/web/src/components/ChatView.tsx, docs/user/thread-sidebar.md
Mobile and web views derive reset times for snooze ownership and recovery notices. The documentation describes retry behavior and conditions that cancel or pause retries.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: shivamhwp, t3dotgg

Merge Risk: External image Minimal · up to cbe1b

The change adds bounded retries for refused usage-limit continuations, with consistent timing across the server, web and mobile surfaces. No actionable merge-blocking issue was found.

Pre-merge checks: Passed 3 · Failed 1

Failed Failed checks: 1 warning

Check nameStatusExplanationResolution
Description checkWarningThe description explains the problem, change, and verification in detail, but it does not include the required maintainer approval comment for this broader behavior change. It also states that the req…Add a link to the triaged issue or approval discussion and include the maintainer comment that approves the direction and scope. Attach before-and-after screenshots of the web banner and mobile card states required by the template, includin…
Passed Passed checks: 3 passed

Check nameStatusExplanation
Title checkPassedThe title clearly describes the main change: retrying usage-limit continuations that the provider refuses after the reset.
Linked Issues checkPassedActive direct target: #33708. Its coding requirements are: (1) a usage-limit continuation refused with no later reset keeps auto-resume armed; (2) retries use bounded backoff; (3) recovery gives up in…
Out of Scope Changes checkPassedEach changed area supports the retry objective of #33708:
- Shared rule and server logic (orchestrationV2ThreadError.ts, ProjectionStore.ts, UsageLimitRecoveryWorker.ts, Orchestrator.ts): implement th…
Full details: Linked Issues check

Explanation

Active direct target: #33708. Its coding requirements are: (1) a usage-limit continuation refused with no later reset keeps auto-resume armed; (2) retries use bounded backoff; (3) recovery gives up into the existing failed state; (4) retries respect cancel, settle, snooze, and archive.

Requirements 1-3 [#33708]: refusedContinuationRetryAt in packages/shared/src/orchestrationV2ThreadError.ts uses 2, 5, and 15 minute steps and returns null after the third step. usageLimitRecoveryResetAt counts consecutive refusals along usageLimitContinuationChain. In ProjectionStore.ts, getLimitRecoveryCandidates adds a refused-continuation branch. It computes the retry time from the refusal count and excludes a thread when a fourth refusal exists (c3.run_id IS NULL). The worker and Orchestrator use the same shared rule, and the worker carries a per-thread auto-resume to the refused run.

Requirement 4 [#33708]: The candidate SQL keeps deleted_at IS NULL, archivedAt IS NULL, settledOverride IS NOT 'settled', and the snooze check. For Cancel, the new carry-over branch requires limitRecovery.autoResume = 1, so a cancelled retry is not a candidate even when global auto-resume is on. Stop uses the same shared rule.

Author-reported evidence: runtimeLayer rows for no-reset, expired-reset, per-thread resume, and manual snooze pass at cbe1b7d96b. The test-only commit 8bfc0a1c0d fails as expected. I did not run these tests. This verdict relies on the diff and the author's report.

Minor uncertainty: the shared helper falls back to requestedAt when completedAt is null, while the SQL falls back to thread updatedAt. This affects only a failed run with no completedAt. This review did not establish that such a run occurs.

The optional 30-60 s grace delay in the issue is a suggestion. The issue states that a delay alone does not fix the defect. Its absence is not a failure.

Full details: Out of Scope Changes check

Explanation

Each changed area supports the retry objective of #33708:

  • Shared rule and server logic (orchestrationV2ThreadError.ts, ProjectionStore.ts, UsageLimitRecoveryWorker.ts, Orchestrator.ts): implement the retry, count, give-up, and carry-over behavior.
  • Tests (ProjectionStore.test.ts, runtimeLayer.test.ts, storageCleanup.test.ts, threadExecution.test.ts): cover the changed server and client paths.
  • Web and mobile UI (ChatView.tsx, ThreadDetailScreen.tsx, UsageLimitRecoveryCard.tsx, threadExecution.ts): show the armed retry time and the Cancel and Wake controls. The issue states that the UI gives no indication of an abandoned recovery. After Cancel, Stop, or give-up, the UI shows the provider reset as before.
  • storageCleanup.ts: holds a thread while its retry is armed. This prevents cleanup from removing state that the retry needs.
  • docs/user/thread-sidebar.md: documents the retry behavior.

No changed file was shown to be unrelated to #33708. Before/after UI captures are not attached. This limits verification of the UI, not scope.

Full details: Description check

Explanation

The description explains the problem, change, and verification in detail, but it does not include the required maintainer approval comment for this broader behavior change. It also states that the required before-and-after UI screenshots are not attached.

Resolution

Add a link to the triaged issue or approval discussion and include the maintainer comment that approves the direction and scope. Attach before-and-after screenshots of the web banner and mobile card states required by the template, including the armed retry, after Cancel or Stop, and retry-owned snooze states.

Finishing Touches

Generate docstrings

  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
External image Recent review info

External image Run configuration

External image Commits

Reviewing files that changed from the base of the PR and between 4c361199f47f6ea32a2c14835df392640d0f7c63 and cbe1b7d96b84ea71b8f3b9a3857124e9b8edabd9.

External image Files selected for processing: 14

  • apps/mobile/src/features/threads/ThreadDetailScreen.tsx
  • apps/mobile/src/features/threads/UsageLimitRecoveryCard.tsx
  • apps/server/src/orchestration-v2/Orchestrator.ts
  • apps/server/src/orchestration-v2/ProjectionStore.test.ts
  • apps/server/src/orchestration-v2/ProjectionStore.ts
  • apps/server/src/orchestration-v2/UsageLimitRecoveryWorker.ts
  • apps/server/src/orchestration-v2/runtimeLayer.test.ts
  • apps/server/src/storageCleanup.test.ts
  • apps/server/src/storageCleanup.ts
  • apps/web/src/components/ChatView.tsx
  • docs/user/thread-sidebar.md
  • packages/client-runtime/src/state/threadExecution.test.ts
  • packages/client-runtime/src/state/threadExecution.ts
  • packages/shared/src/orchestrationV2ThreadError.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.