# fix\(codex\): stalled HTTP 429 retries end with an explicit failure instead of hanging · PR \#23842 · pingdotgg/t3code

[View on GitCafe](https://git.cafe/pingdotgg/t3code/pull/23842)

Repository: [pingdotgg/t3code](https://git.cafe/pingdotgg/t3code)

Visibility: public

State: open

Source repository: Unavailable fork

Source branch: upstream-pr/codex-429-retry

Target branch: main

Head commit: 3782d8adf38639ba27e23738fa425e3cba2111d5

[Discussion and reviews](https://git.cafe/pingdotgg/t3code/pull/23842)

[Commits](https://git.cafe/pingdotgg/t3code/pull/23842/commits?format=markdown)

[Files changed](https://git.cafe/pingdotgg/t3code/pull/23842/diffs?format=markdown)

## Description

---

## Problem

The Codex CLI retries a rate-limited (HTTP 429) model request itself. The only thing the app-server sends is an `error` notification with `willRetry: true`. If the response carries a long `Retry-After`, Codex sleeps for that whole time and sends nothing else. The T3 Code turn then stays `running` with no further events. Any delegated task waiting on it also hangs, and the user gets no failure, reset time or recovery hint. In the case reported in #17503, a post-tool 429 retry left the turn running for hours.

Expected behaviour: the turn either makes progress or ends in a clear, retryable failure that includes the rate-limit reset time when it is known.

Refs #17503

## Change

The change is limited to the Codex adapter (`apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts`) and a new pure helper module (`apps/server/src/provider/codexRateLimitRetry.ts`). There are no contract, client or settings changes.

**Retry policy.** Each `willRetry: true` notice classified as HTTP 429 arms a deadline:

- At most 5 retry attempts.
- With no `Retry-After`, the expected delay is exponential backoff: 5s, doubling each attempt, capped at 5 minutes.
- If `Retry-After` is present (delta-seconds or HTTP-date), it is the expected delay. The HTTP-date form is measured from the notice time.
- The per-attempt deadline is the expected delay plus 60s for the retried request to show progress.
- A `Retry-After` above 5 minutes fails the turn immediately, reporting the reset time.
- A 6th attempt, or a missed deadline, also fails the turn.
- Any progress (deltas, items, compaction) or a newer retry notice clears or re-arms the deadline.

Worst case before giving up:

- No `Retry-After`: about 455s (65 + 70 + 80 + 100 + 140).
- `Retry-After` at the 5-minute cap on every attempt: 1800s.

**Classification.** The status comes from `codexErrorInfo`. When Codex sends no HTTP status (for example `responseStreamDisconnected`), a `429` / `Too Many Requests` in the error text also counts. An explicit non-429 status always takes precedence.

**Ending the turn.** When the bound is hit, the adapter:

1. Marks the turn as rate-limit exhausted.
2. Sends `turn/interrupt` to the native turn and to every active descendant (subagent) turn. Each request is bounded to 10s. Descendants are re-scanned under the existing terminalization permit, so a nested turn started during teardown is either stopped or handled by the existing late-subagent path.
3. Fails the root turn with a retryable usage-limit failure, code `rateLimitRetryExhausted`, including `resetAt` when known.

Late `turn/completed` or `error` frames for that turn are ignored, so they cannot overwrite the failure. A subagent turn closes its retry item with the same failure.

**Unconfirmed stop.** If Codex does not acknowledge the interrupt (it times out or errors), the turn is not reported idle while Codex may still be retrying. It stays active until Codex sends `turn/completed`, or for at most 60s more. It is then failed and the thread disposition is set to `broken`, so the next turn runs `thread/resume` instead of reusing a loaded thread that may still be retrying. A subagent thread in this state is published as `error`. The same applies to a descendant whose own interrupt was not acknowledged (at the deadline or at the end of the 60s bound): it is marked abandoned, so its thread is published as `error` rather than `idle`. A later user Stop that Codex does acknowledge for such a descendant counts as a confirmed stop, so that thread returns to `idle`. If `turn/completed` arrives while the abandonment waits for the terminalization permit, the abandonment re-checks the containment state and leaves the confirmed end in place. A user Stop that Codex does not acknowledge keeps the same `broken` disposition.

The PR has these commits (plus a merge of `main`):

1. The bound itself.
2. Hardening of the bound: Retry-After HTTP-date, textual 429, compaction, interrupt-before-terminal, lineage, unconfirmed stop, and the subagent failure.
3. Descendants whose 429 interrupt is unconfirmed are marked abandoned instead of `idle`.
4. A user Stop acknowledged for a late descendant counts as a confirmed stop.
5. The abandonment re-checks the containment state after acquiring the permit.

## Scope and approval

This is a reliability fix for one problem: a Codex turn hanging silently after an HTTP 429. It links #17503, which **has not been triaged by maintainers yet**. I'm happy to wait for triage, or to narrow the scope (for example, to the first commit only) if you prefer.

## Verification

**How the problem was confirmed.** The adapter tests replay Codex app-server transcripts. One has a tool call followed by an `error` notice with `willRetry: true` and HTTP 429, then silence. Against the unchanged adapter at `044f07257` (the Codex adapter and its tests are identical at `22840ab9`, the `main` this branch now merges), I ran the new 429 tests (`vp test run src/orchestration-v2/Adapters/CodexAdapterV2.test.ts -t 429`):

- The `ends a stalled post-tool 429 retry with an explicit failure` cases failed with `expected [] to have a length of 1 but got +0`: no terminal event was ever produced after the deadline.
- The run never finished and was killed by a 600s timeout. This is the same silent hang as in the issue.

**How it was tested.** Each file was run separately from `apps/server` with `vp test run <file>` (Node 24.22.0):

| File | Result |
| --- | --- |
| `src/orchestration-v2/Adapters/CodexAdapterV2.test.ts` | 97/97 passed |
| `src/provider/codexRateLimitRetry.test.ts` | 7/7 passed |
| `src/orchestration-v2/testkit/CodexReplayFixtures.integration.test.ts` | 24/24 passed |
| `src/provider/Drivers/CodexDriver.test.ts` | 11/11 passed |
| `src/orchestration-v2/DelegatedCompletionDelivery.test.ts` | 26/26 passed |
| `src/provider/codexUsageLimits.test.ts` | 15/15 passed |

- The 429 subset (`-t 429`) was run 3 times: 26 passed each time.
- At the first commit alone: `CodexAdapterV2.test.ts` 79/79, `codexRateLimitRetry.test.ts` 5/5.
- `vp fmt --check` on the 4 changed files was clean. `vp lint` gave 0 errors; its 2 `no-unused-vars` warnings are already present on `main`.

**What the tests cover and what was observed:**

- Stalled 429 retries, covering backoff, short `Retry-After`, HTTP-date `Retry-After`, textual 429, and an over-cap `Retry-After`. The turn fails at exactly notice time + deadline on `TestClock`, with code `rateLimitRetryExhausted` and `resetAt`. `turn/interrupt` is sent before the terminal is published.
- A retry that resumes into context compaction keeps running.
- Retry-policy unit tests for backoff, the cap, the attempt limit, and Retry-After parsing (delta-seconds, HTTP-date, over the cap).
- Lineage: the root and active descendants are interrupted, including a nested turn that starts during teardown or after the last scan. The root terminal does not report `childTurnsContinue`.
- Unconfirmed stop:
  - The turn stays active when the interrupt times out or errors.
  - It is failed as `broken` after the 60s bound.
  - A late `turn/completed` is ignored.
  - A nested turn started during the wait is stopped.
  - An abandoned subagent thread becomes `error`, not `idle`.
  - A descendant whose interrupt fails at the deadline, or a late descendant that cannot be stopped within the bound, is published as `error`, not `idle`.
  - A late descendant whose unconfirmed 429 interrupt is later confirmed by a user Stop is published as `idle`.
  - A Stop Codex never acknowledged keeps `broken`.
- A late final `error` during the interrupt does not replace the rate-limit failure.
- A subagent retry that runs out of attempts closes with the usage-limit failure.

The tests use `TestClock` and wait on observable milestones (the retry notice, the interrupt request). There are no real-time waits.

**Not verified:**

- Not run against a live Codex CLI / app-server, and no real upstream 429 was triggered. Coverage is replay transcripts and unit tests only.
- Ran on Node 24.22.0, not the pinned Node version. The full typecheck (`vp run typecheck`, 0/32 cache hits) passed with 0 errors across 32 tasks; `node scripts/release-smoke.ts` passed (6/6) on the final head; the full test suite and the browser suite were not run.
- Only `contextCompaction` counts as progress among the early-returning item handlers. `subAgentActivity` / `userMessage` items do not re-arm the deadline.
- The containment re-check in commit 5 guards a race between `turn/completed` and the permit acquisition that the replay harness could not reproduce deterministically, so it has no dedicated failing-before test.
- A turn in the unconfirmed-stop state has no timer beyond the 60s bound, and it does not send background-terminal termination requests.

Agent use: this change was written and tested locally with AI coding agents (Grok Bot). The commands and results above come from those local runs.