Codex run stays running forever when command items complete after turn/completed#25000

opened2d agobyJJulius Marminge

What happened

A Codex thread's run stayed running for 11+ hours after its turn had completed. Nothing settled it except restarting the T3 server.

Provider event log (Codex app-server, shared session), times UTC on 2026-10-25:

  • 09:28:51 the agent starts a long commandExecution (ops fleet release …, several releases in one shell call). A second command starts and completes at 09:28:52.
  • 09:28:54 agentMessage with phase: final_answer.
  • 09:28:55 thread/status/changed → idle, then turn/completed.
  • 09:29:55 and 09:31:29 two item/completed events for commands from that turn arrive after turn/completed.
  • 09:58:55 T3 sends thread/unsubscribe, which succeeds.

T3 still lists the run as active (activeRunId = ordinal 3147). Another 100+ messages (scheduled ticks, webhook runs, delegated task completions) queued behind it and never started.

Expected

turn/completed settles the run. Late item/completed events for that turn update the items but do not hold the run open, and the queue continues.

Related

  • #24743: a run never settles when a Codex turn errors during a steer. Same symptom, different trigger.
  • #24782 changed running task results to tell the agent to end its turn, which makes turns that end while their commands are still running more common.

Environment

T3 Code preview server on Linux, Codex app-server (codex-cli 0.163.0), runtime mode full-access. The thread is a long-lived scheduled dispatcher with about 2,900 runs.

Tested 2050a12b6d with the real web dev server on Linux, Codex app-server 0.163.0, gpt-6-astra, full access, and fresh isolated state.

Started a 45-second command and ended the turn immediately, then repeated with parallel 35- and 55-second commands plus a quick command. The commands emitted item/completed after turn/completed; both runs had already settled as completed. A follow-up message ran while the commands were outstanding, and all completion-notification runs also settled. Runtime transcript.

Which exact T3 preview build or commit was running during the incident? Please attach a redacted server trace around 09:28:55–09:31:29, including any run-finalization or checkpoint errors. The provider sequence reproduces here, but the stuck run does not; the build and server-side settlement events are needed to distinguish the failing path.

Retested current main dd5b5401e1 with the real web dev server on Linux, fresh isolated state, Codex app-server 0.163.0, gpt-6-astra, full access.

A 45-second command completed 42 seconds after turn/completed; its run had already settled. A second attempt started parallel 35- and 55-second commands plus a quick command. Both delayed commands completed after that turn settled. The second message started while the first command was outstanding; all five runs, including completion notifications, finished successfully. Runtime transcript, transcript extractor.

The late-event sequence reproduces, but the stuck run does not. Please provide the incident's exact preview build or commit and a redacted server trace around 09:28:55–09:31:29, including run-finalization and checkpoint errors.

Retested on current main 96fc7b9551 by replaying the incident sequence through the real Codex adapter, run execution, effect worker and checkpoint capture (CodexOrchestratorReplayHarness). A long command and a short command start in the turn, then final_answer and turn/completed arrive, then the long command's item/completed arrives late, with a second message queued behind the run. I ran three variants:

  • The late completion arrives before the queued run starts: both runs complete.
  • The late completion arrives during the queued run's turn: both runs complete.
  • The continuation worker is on: the late completion dispatches a "Background command completed" message, and T3 sends turn/start for it.

The late item/completed did not hold any run open.

The incident's 09:58:55 thread/unsubscribe is exactly 30 minutes after turn/completed. That means T3 saw the turn end and never started another Codex turn on that thread. The likely path is that the run's finalization never finished. A completed turn leaves the run waiting until its checkpoint.capture effect finishes, and later provider-turn.start effects for the thread queue behind it.

To confirm this, please share two things from the incident server's statev2.sqlite, if it still has them:

  1. The status and completed_at of run ordinal 3147 on the stuck thread. Was it the run whose turn ended at 09:28:55, or a later one?
  2. That thread's orchestration_v2_effect_outbox rows created between 09:28 and 09:59 UTC: effect_type, status, attempt_count, lease_expires_at, last_error.