Sign up
View stargazersSign up

[Bug]: Run never settles when a Codex turn ends with an error during a steer; interrupt/stop hang and every queued run blocks behind it#24743

opened2d agobyJJulius Marminge

Summary

A Codex turn that ends with a provider error while T3 is steering input into it leaves the T3 run active forever. Every later message, scheduled tick, and webhook run then queues behind it. Interrupt is accepted but changes nothing, and a Stop (t3_thread_interrupt with stop: true) never returns. Only restarting the T3 server clears it.

Environment

  • T3 0.1.2-preview.20261022.3516, Linux server
  • Codex provider (gpt-6.1-sol), long-running dispatcher thread that delegates tasks and gets "Delegated task … finished" notices steered into its active turn

Sequence (provider event log, 2026-10-24 UTC)

123:02:11.789 item/started mcpToolCall t3-code.task_status
223:02:12.376 outgoing {"id":307,"method":"turn/steer","params":{"expectedTurnId":"01a17024-…", "input":[{"text":"Delegated task … finished"…}]}}
323:02:12.477 incoming {"id":307,"result":{"turnId":"01a17024-…"}}
423:02:12.478 item/started + item/completed userMessage (the steered notice)
523:02:12.946 thread/status/changed {"status":{"type":"systemError"}}
623:02:12.946 error "exceeded retry limit, last status: 429 Too Many Requests" (responseTooManyFailedAttempts)
723:02:12.947 turn/completed {"turn":{"id":"01a17024-…", "items":[], …}}
823:02:18–25 item/completed for three commandExecutions and the mcpToolCall that were in flight

After this:

  • The T3 thread still showed status: running with activeRunId = that run, and no items were added for 6 hours.
  • In-flight items showed failed in T3 at 23:02:12.000Z, even though Codex later reported them completed.
  • 50+ runs queued behind it, and none started.
  • t3_thread_interrupt (no stop) returned interrupt_requested twice, hours apart, and nothing changed.
  • t3_thread_interrupt with stop: true hung for about 40 minutes without returning.
  • systemctl --user restart t3code.service cleared it: activeRunId became null and the queue could run again.

Expected

  • turn/completed after systemError (or any provider error) settles the run as failed, even when a steer landed in the same turn.
  • Interrupt and Stop on a run whose provider turn has already ended settle it immediately, instead of waiting for a provider acknowledgement that can never come.
  • Ideally, a watchdog fails a run that has had no provider activity for a long time while its provider reports no active turn.

Related: #24300 (shared Codex app-server backlog/death not detected).

JJulius Marminge2d ago

Tested Linux web/server on 2fd28bf938 with Codex 0.163.0. A normal steer completed. With a real delegated child finishing during the parent's shell call, a schema-valid replay of systemError → non-retryable 429 error → failed turn/completed settled the parent as failed immediately; it did not remain active. This injected the provider failure rather than exhausting an upstream account. The focused adapter replay also passed on main.

Please provide the full decoded error and turn/completed JSON from 23:02:12.946–.947, including turn.status, turn.error, and codexErrorInfo, plus the incident's codex --version. Prompt text and tool arguments can be redacted. The abbreviated terminal frame in the report omits the fields needed to check whether it was accepted by the protocol decoder. An invalid error shape in an initial replay was dropped and left a run active, so that attempt was discarded.

Replay transcript and persisted result · Replay driver

Delegated completion followed by a settled failed parent