Codex idle threads stay active when turn/completed is missed#28028

opened10d agobyAAlb11747

What happened

A Codex run can remain active after its native thread reports idle if T3 does not receive turn/completed. The adapter ignores the idle notification, so it never reconciles the terminal turn that Codex already records. Later messages can remain queued behind the active run.

This investigation began with a Windows incident in which a delegated run remained active for over fourteen hours. Its native log stopped after polling a background command; startup recovery eventually cancelled it. The retained logs do not establish whether Codex became idle in that incident, so the deterministic failure below is narrower than a confirmed diagnosis of that original stall.

Diagnosis

On upstream 50c1c5dc536493826fc66e220598f38b831d5bde, CodexAdapterV2 handles turn/completed but has no handler for thread/status/changed. Replaying a started turn followed by idle, with the native turn history reporting the exact turn as terminal, leaves T3's turn running. Existing normal completion and process-exit paths work.

Steps to reproduce

Use the Codex adapter's protocol replay harness:

  1. Accept turn/start and emit turn/started for a native turn.
  2. Emit thread/status/changed with status.type = idle, omitting turn/completed.
  3. Make thread/turns/list report that exact turn as completed.
  4. Advance the test clock. Before the proposed fix, no history request or terminal event occurs, and the turn stays active.

This deliberately simulates a missing completion notification; it is not a natural reproduction of the original provider stall.

Version and environment

Deterministic reproduction on upstream 50c1c5dc5364, Windows, Node 24.21.0, using the real adapter and protocol replay client. No paid agent or live user database was involved. The original incident used 0.1.4-nightly.20261028.3841.fork.1 on Windows.

Proposed fix and validation

The prepared fix waits one second after an idle notification, then pages through turn metadata if the same context is still active. It settles only an exact native turn that Codex reports as terminal, preserving completion, failure, interruption, and retry handling. Normal completion within that grace period makes no request. The lookup runs outside the protocol reader and shares a ten-second deadline across all pages. The review regression also covers an older completed goal turn behind a newer orphan continuation.

Prepared branch: https://github.com/Alb11747/t3code/tree/fix/codex-missed-terminal

Diff: https://github.com/pingdotgg/t3code/compare/main...Alb11747:fix/codex-missed-terminal

Six of seven initial regression cases failed before implementation; all seven pass with the fix. An eighth case reproduces the goal-continuation history gap on the original PR head and passes with pagination. The full Codex adapter suite passes 130 tests. vp run check:changed upstream/main passes format, lint, knip, generated-code checks, affected typechecks, and 131 tests. The capacity-retry fixtures also include idle notifications.

Runtime replay transcript from the adapter, with the completion notification omitted:

text
1provider turn: running
2authoritative read: thread/turns/list
3provider turn: completed
4settled: completed

Original upstream CI passed on 43ccc5b. Its fork checks failed an unrelated, untouched storage-cleanup assertion against Git 2.56: https://github.com/Alb11747/t3code/actions/runs/41181249257. Follow-up checks are linked from PR #28029.

The fix is available for maintainers to consider or reimplement in PR #28029. It does not claim to recover a provider that still reports inProgress, or a case where neither idle nor completion is reported.

Maria subsequently verified unchanged d778fe3 with real Codex turns in the Linux web client. A transparent wrapper dropped exactly one real completion notification: main remained working with its follow-up queued, while the PR recovered the exact terminal state and ran the follow-up. Normal completion, user stop, and a subsequent turn worked. The injected loss is still distinct from a naturally occurring lost notification, and the original incident's precise cause remains unproven.

  • #25000 involves turn/completed being received before late command completions; this repro omits that terminal event.
  • #24300 concerns a dead or backlogged shared transport. This recovery needs a responsive provider and an authoritative terminal turn.

Filed by

GPT-6.1 Sol through the Codex harness in T3 Code, with independent Sol design and code review.