[Bug]: Provider turn projection stays running after T3 ends the turn#16557

opened25d agobyDDropje97

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

Two paths lead to the same inconsistent provider-turn projection:

  1. Failed run. Start a provider turn and let provider event ingestion fail before the turn is terminalized normally. An injected stream or ingestion error makes this deterministic in an integration test. After failure finalization finishes, inspect the run, attempt, root node and provider turn.
  2. Startup recovery of a native Codex subagent. Start a Codex native subagent and restart the T3 server while its runless root is active. After startup recovery, inspect the child's root node and the provider turn whose node is that root (its runAttemptId is null).

Expected behavior

When T3 terminalizes a root or run, either through failure finalization or through process-loss recovery, the provider turn under that root also becomes terminal. Independent background work keeps its own lifecycle.

Actual behavior

The run and root become failed or cancelled, but the provider turn stays running. Nothing revisits these rows later, so they survive further restarts.

Both omissions are still in current main:

Impact

Minor bug or occasional failure

Version or commit

Observed on the official preview build 0.0.46-preview.20261002.2598 (b37fcea). Checked against the source of main @ 9503155; I have not run main itself.

Environment

Linux x86_64 (Debian 13 container), headless T3 server with browser clients, Orchestrator V2, Claude and Codex providers.

Logs or stack traces

shell
1Failed run:
2 run: failed
3 root node: root_turn / failed
4 provider turn: running
5
6Native Codex subagent after startup recovery:
7 runAttemptId: null
8 root node: root_turn / cancelled
9 provider turn: running
10 provider thread: idle
11 background roster: []
12 (one runless reasoning node under that root also stays running)

Screenshots, recordings, or supporting files

No response

Workaround

None in T3. We noticed this because an external maintenance check that waits until no provider turn is running never saw the server as idle. It now ignores provider turns whose root has ended, while still blocking on runs, attempts, sessions, background rosters and items.

Related but different: #15542 (provider keeps executing after failed ingestion), #15567 (tracking of resumed native children after restart).

Synced from GitHub

Correction to the related work above: open PR #14434 (against the V2 branch) already fixes the second path. It adds runless native child provider turns to the recovery projection and cancels them. Unless I missed it, that leaves the failed-run finalization path (writeFinalRunEvents) unfixed.

Reproduced the native Codex child path on Linux x64 with the real web dev app at main a12d6ed942. A native child was running sleep 300 when the isolated server was restarted. After startup recovery completed, its runless root was cancelled while its provider turn was still running (runAttemptId: null). An earlier stale provider turn survived the next restart too. The failed-ingestion path already has a partial fix in #22846; that PR explicitly excludes startup recovery.

Both reported paths reproduced with the real web dev server and Codex on Linux at a12d6ed942. After integrating current main, focused regressions still fail for interrupted-provider cleanup failure and native-child startup recovery. Updated the existing #22846 with both repairs, current-main red/green proof, scoped lint/typechecks, and the explicitly versioned runtime transcripts. No duplicate PR was opened.

Note

🤖 Claude Opus 5.5 responding on behalf of Theo

Closing: this is covered by #22846, which landed. Reopen if an important part is still missing.