Sign up
View stargazersSign up

[Bug]: A successful turn on another account releases queued work held by a usage limit on the original account#21705

opened23d agobyDdancingmadkefka

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Summary

A usage-limit failure holds queued work only until any later run executes on the thread. A successful turn on another account clears the hold, even though the queued messages still select the exhausted account and its reset is still in the future. The next queued message then starts automatically, switches the thread back to the exhausted account, and fails on the same limit.

Keeping a queued message's selected account is intentional. This report does not ask to reroute those messages to another account.

Related:

  • #21313 / open PR #21314: queued PR-watch updates should follow the takeover provider. #21314 reroutes app-owned monitor notifications and makes the usage-limit hold provider-specific. The messages here have no notification source, so they keep their saved selection, and the new provider check still reads the thread's latest executed run (details below).
  • #17932, closed by #18467: thread-bound scheduled tasks now take the thread's selection at dispatch. Both messages here were dispatched while the thread was still on A, so they carry A on main too.

Steps to reproduce

  1. Configure Codex instances A and B. A is exhausted; B has quota.
  2. In a thread on A, queue a message while an A turn is running. Let that turn fail with usageLimitExceeded and a future reset time. The queued A message stays queued.
  3. With no active run, send a new message on B. Let it finish successfully.
  4. Watch the queue start on its own, without resuming the queue or retrying A.

In the incident, the queued messages were thread-bound webhook scheduled-task wakes, saved with the A selection. The live sequence happened twice. The steps have not been rerun end to end on a clean installation.

Expected behavior

Completing work on B should not by itself clear A's known usage-limit hold. Keep the A-selected message queued until A's limit no longer applies or the user explicitly retries or resumes that work. Keep the message's selected account.

Actual behavior

UTC times on October 19, 2026:

EventResult
00:57 and 01:05A messages 158 and 159 are queued behind running A run 157.
01:09:22, A run 157Usage-limit failure, reset October 24 at 20:40:41 UTC. 158 and 159 stay queued.
10:41:38, B run 160Completes successfully.
10:41:43–10:41:46, A run 158Starts automatically, then fails with usageLimitExceeded.
10:52:15, B run 163Completes successfully.
10:52:20–10:52:22, A run 159Starts automatically, then fails with usageLimitExceeded.

Each direct B send switched the thread to B. Each queued A start switched it back to A. So every successful B continuation was followed within seconds by a limited thread again. The limit errors are genuine A errors. No false quota classification or wrong sidebar status is claimed.

Source investigation

At main 1451e7cb23034d29774338608368a15d8845df36, read but not built or run:

  • usageLimitBlockedRun looks only at the thread's latest executed run. After B succeeds it returns null, although A's last execution is still a usage-limit failure.
  • startNextQueuedRun uses that result as its usage-limit gate. Its comment states the purpose: "Starting the queue would send every waiting message and drop it from the queue as each one fails." The run that clears the gate belongs to a different account and says nothing about A's quota.
  • The generic failure hold a few lines later is already instance-aware, but it never runs for usage limits because the gate above returns first.

PR #21314 (read, not run) replaces the gate with usageLimitHoldsQueue, which passes A's latest session error to usageLimitBlockedRun(projection.runs, ...) and compares the result's provider with A. The PR does not change usageLimitBlockedRun, so in this sequence it still returns null after B's success and the A message starts. The PR's new usage-limit tests hold an A message while A's failure is the latest run and reroute a monitor wake. Neither covers a B success before an A-selected message starts.

Impact

Major degradation or frequent failure

Each queued message is consumed by a run that fails immediately on A's limit, and each B continuation can trigger another futile A attempt that pulls the thread back to A. The user has to find and reconcile the consumed messages by hand.

Version or commit

Observed on 0.0.46-nightly.20261010.2935. The latest-run guard is unchanged on main 1451e7cb23034d29774338608368a15d8845df36.

Environment

Windows 11 desktop server. Codex app-server 0.163.0 as reported by its protocol user agent. Two authenticated Codex provider instances: A exhausted, B available.

Workaround

Before continuing on B, cancel or edit queued messages that still select A. Move recurring automatic messages to the thread and account that should receive them.

Verification limits

Observed twice, with native provider errors and durable run timestamps. The guard transition was reproduced with the installed production helper and synthetic run records. No clean-install end-to-end reproduction or current-main build was run. Drafted with AI assistance (Claude Opus 5.5 and Codex); the script output was run and checked.

Production-helper reproduction

The script extracts the installed helpers without reimplementing their policy. Run it against an extracted server-binCli-*.mjs bundle. It shows the guard disappearing after an unrelated B success, even when A's session error is passed in, which is the input #21314 would supply. It does not start a provider or exercise the whole queue dispatcher.

javascript
1import fs from 'node:fs';
2import assert from 'node:assert/strict';
3const source = fs.readFileSync(process.argv[2], 'utf8');
4const start = source.indexOf('function latestRootProviderFailure(');
5const end = source.indexOf('function usageLimitRunPresentedAsLatest(', start);
6assert(start >= 0 && end > start, 'Expected production helpers not found');
7const { usageLimitBlockedRun } = new Function('toEpochMillis', source.slice(start, end) + '\nreturn {usageLimitBlockedRun};')(Date.parse);
8const limited = {id:'run1',ordinal:1,providerInstanceId:'A',status:'failed',rootNodeId:'root1',startedAt:'2026-10-19T01:00:00Z',completedAt:'2026-10-19T01:01:00Z'};
9const queued = {id:'run2',ordinal:2,providerInstanceId:'A',status:'queued',startedAt:null,completedAt:null};
10const success = {id:'run3',ordinal:3,providerInstanceId:'B',status:'completed',rootNodeId:'root3',startedAt:'2026-10-19T10:00:00Z',completedAt:'2026-10-19T10:01:00Z'};
11const failure = {id:'error1',type:'error',status:'failed',runId:'run1',nodeId:'root1',ordinal:1,updatedAt:limited.completedAt,failure:{class:'usage_limit',message:'account A limited',resetAt:'2026-10-24T20:40:41Z'}};
12assert.equal(usageLimitBlockedRun([limited,queued],[failure],null)?.id,'run1');
13assert.equal(usageLimitBlockedRun([limited,queued,success],[failure],null),null);
14assert.equal(usageLimitBlockedRun([limited,queued,success],[failure],'account A limited'),null);
15console.log('Before B success: A queue blocked. After B success: no usage-limit queue block, even when A session retains its error.');
JJulius Marminge23d ago

Note

Grok responding on behalf of Julius.

Confirmed on main, so this is accepted as a bug. The queue's usage-limit hold exists so waiting messages aren't each sent and failed against the limit (Orchestrator.ts:1298-1309). It checks usageLimitBlockedRun, which looks only at the thread's latest executed run (orchestrationV2ThreadError.ts:78-87). Once a run on B succeeds, that run is the latest, the check returns nothing, and an A-selected queued message starts against A's limit.

This is separate from #21313, which is about queued monitor updates following the provider the user switched to. Open #21314 makes the hold provider-specific, but its new check still calls usageLimitBlockedRun on all runs and compares the result's provider. After a B success it returns nothing for A too, so it wouldn't hold the queue in this sequence.

Keeping each queued message on the account it was saved with is intended, as you said.

PPolarBearEs20d ago
Synced from GitHub

Observed the same sequence across providers on 0.1.2-nightly.20261021.3473 (Linux desktop host, Android Chrome client):

  • Claude Opus 5.5 hit its session limit with one message queued.
  • I switched to GPT-6.1-Sol and sent a continuation. Sol completed the work successfully.
  • The queued message then started automatically, switched the thread back to Opus, and failed on the still-active session limit.