[Bug]: Codex turn hangs silently during provider 429 retries; session left wedged in systemError until the next manual message#17503

Additional evidence from October 15, 2026 identifies one concrete cause of the long 429 retry wait described here. This covers the retry-wait portion of this issue. It does not reproduce the separate systemError recovery problem.

A live Codex 0.162.1 turn used a Responses-compatible CLIProxy route. The proxy retained saved quota cooldowns after upstream quota had recovered. It returned HTTP 429 with error.code: "model_cooldown", no error.type, and Retry-After: 384882. Codex reported Reconnecting... 1/5 with willRetry: true. The turn remained open until it was interrupted manually, about 12 minutes after it started.

The important distinction is that T3's running state matched the provider's reported state. Codex had announced another attempt and had not emitted a terminal result. The observed error notification contained no retry deadline. This evidence does not establish an incorrect T3 state transition, although making a prolonged provider wait understandable and recoverable remains relevant to this issue.

We reproduced the HTTP error behavior outside T3 with the installed Codex 0.162.1 binary and a loopback HTTP fixture server. No upstream model request or quota exhaustion was needed.

1. Point a custom Codex Responses provider at the fixture server's loopback /v1 endpoint. The replay used wire_api = "responses", request_max_retries = 0, and stream_max_retries = 5.
2. For its POST /v1/responses, return the response below. The body and headers were exported from the proxy's actual Go error implementation.
3. Run codex exec with a trivial prompt and model gpt-6-astra. Count requests and start the observation timer when the fixture receives the first request.
4. Repeat with the same body and HTTP status, removing only Retry-After.

The replay produced these results:

The multi-day duration is supported by source inspection, not by leaving the test running for days. In the pinned Codex 0.162.1 source:

• The 429 classifier recognizes error.type: "usage_limit_reached" as a terminal usage limit. This untyped model_cooldown body instead becomes RetryLimit.
• RetryLimit uses the server's retry advice.
• The Responses retry loop emits the reconnect notice and sleeps until the supplied deadline, without capping it to the local backoff.

Cross-reference #16288 for the separate problem of classifying a terminal HTTP 429 retry-exhaustion error and exposing Limited/Resume. This incident stalled before reaching that terminal stage. It does not verify or invalidate the fix for #16288, and it does not establish the cause of every silent retry described in this issue.

The stale cooldown state belongs to the proxy. Clearing only cooldowns contradicted by current upstream quota, then interrupting and resuming the original T3 turn, restored progress and produced a completed turn with tool results. A local proxy change that omits long automatic retry hints was tested, but it has not been deployed and does not prevent stale cooldowns from recurring.

The human reporter requested this follow-up. GPT-6 Astra through Codex performed the investigation and replay; Claude Opus 5.5 through the Claude agent route provided diagnosis and independent code review. Account identities, credentials, hostnames, private paths, thread IDs, and task content are omitted.

Tested current main 70e5e384da on Ubuntu Linux, web dev app at 1280×800, Codex 0.163.0, isolated T3 state and a separate Codex home.

Replayed the October 15 comment's model_cooldown HTTP 429 body using a loopback Responses endpoint (request_max_retries = 0, stream_max_retries = 5):

• With Retry-After: 384882, the app displayed Retrying provider (1/5) throughout a 31-second turn. Clicking Stop interrupted it. The wait reproduced, but it was not silent.
• With the same body and no Retry-After, the provider turn failed in 94 ms with exceeded retry limit, last status: 429 Too Many Requests. The app showed Usage limit reached, with no running spinner.

These replay attempts fail the initial model request; they do not reproduce the original post-tool failure or establish systemError session recovery. The original exhausted-account route was unavailable here.

Does the silent/post-tool case still occur on a current build? Please provide the current T3 and Codex versions and a sanitized provider notification sequence from the last tool result through thread/status/changed, error, and turn/completed (including willRetry and timestamps). A replay fixture that reaches the post-tool systemError case would also let us verify the remaining recovery behavior. Leave out credentials and task contents.

Attached: current-main retry notice and terminal usage-limit state.

Yes, the silent case still happens on a current build, and it is worst for delegated tasks. On 2026-10-20, three Codex children started with delegate_task stayed "running" for about 2h10m after a 429. The parent got no wake and no result. A human found them only by asking the parent for status.

Versions: T3 0.1.1-nightly.20261020.3411 (or the 20261019 build just before it; I did not confirm which one was running at that minute), Codex 0.163.0, Linux. Codex goes through a local CLIProxyAPI route with one ChatGPT Plus account.

Sanitized sequence (from T3's provider event log and the proxy log, UTC):

So the turn is not wedged. Codex sleeps until the proxy's retry deadline, about 4 hours here. I did not capture the proxy's response headers this time. The proxy log matches the model_cooldown path in the October 15 comment, and the silence matches a wait until the reset. In this case the cooldown was correct, not stale: the upstream message said "try again at 12:05 AM", which was 00:05 UTC.

I also checked the other delegated runs on this host (765 since early October). The same pattern caused stalls on 10-15 (about 1h50m) and 10-16 (44 min). Each time the parent saw only "running".

What would help delegated work. The thread UI shows "Retrying provider (1/5)", but a parent agent never sees that UI. It only reads task_status or waits for a wake. Some options:

1. When a retry wait is longer than a few minutes, end the turn as a usage limit and keep the reset time, as the no-Retry-After path already does. Then a parent can switch to another provider at once.
2. Or report the retry state through task_status (for example workState: "retrying", with the attempt and the retry deadline), and wake the parent when a child enters a long retry wait.
3. Fill retryDelayMs (or a resetAt) from the retry notice, so tools can see the deadline.

I read the stored events and logs. I did not replay this with a fixture.

Thanks for testing current main and for the replay results.

Does the silent / post-tool case still occur? Yes. I'm still hitting it on recent builds, and @snnbotchway's comment above shows the same thing on 0.1.1-nightly.20261020.3411 with Codex 0.163.0: delegated tasks sat at "running" for about 2h10m after a post-tool 429 with willRetry: true.

Notification sequence. The best one I have is still the 2026-10-09 incident in the issue body (T3 0.0.45, Codex v0.153.4): last tool result at 09:50:47 UTC, then silence, then thread/status/changed {type: "systemError"} → error (exceeded retry limit, last status: 429 Too Many Requests) → turn/completed (failed) at 09:51:27, then no provider events until I re-sent the message at 10:00:15.

Replay fixture. I don't have one that reaches the post-tool systemError case yet. My untested guess at the shape: the first POST /v1/responses streams a tool call normally, and every request after the tool result returns 429 without Retry-After, so the retry ladder runs out quickly.

Suggested direction. I'd like T3 to back off and retry automatically on a 429 / usage limit instead of sitting there silently: wait for Retry-After or the reset time when it's known, otherwise use bounded exponential backoff, and once retries run out, fail clearly. The retry state should be visible too, including to parent agents through task_status for delegated tasks. This lines up with options 2 and 3 in @snnbotchway's comment.