# fix\(server\): Codex recovers after its app-server process is killed · PR \#22791 · pingdotgg/t3code

[View on GitCafe](https://git.cafe/pingdotgg/t3code/pull/22791)

Repository: [pingdotgg/t3code](https://git.cafe/pingdotgg/t3code)

Visibility: public

State: open

Source repository: pingdotgg/t3code

Source branch: fix/issue-16502

Target branch: main

Head commit: 1afd783c5e4ee4764ccc24bbe1649d25be539cee

[Discussion and reviews](https://git.cafe/pingdotgg/t3code/pull/22791)

[Commits](https://git.cafe/pingdotgg/t3code/pull/22791/commits?format=markdown)

[Files changed](https://git.cafe/pingdotgg/t3code/pull/22791/diffs?format=markdown)

## Description

---

## Problem

When the shared `codex app-server` process dies (for example the kernel OOM killer sends `SIGKILL`), T3 keeps its Codex session cached. Every later Codex turn and every brand-new Codex thread or delegated task then fails with `CodexAppServerTransportError: ... 'read-process-exit-status' failed`, and no new app-server is started until T3 restarts. Reproduced on main [ed3f965e07](https://github.com/pingdotgg/t3code/commit/ed3f965e07) with the dev server and web client: first Codex turn completes, `kill -KILL <native app-server pid>`, then the follow-up turn and a new Codex thread both fail and no app-server process reappears.

## Change

- `effect-codex-app-server`'s client exposes `awaitTermination`, which fails when the protocol records the process termination. That covers a numeric exit code and also a signal death, which arrives as `read-process-exit-status`.
- `CodexAdapterV2` watches it for each session and fails the session's event queue with `ProviderAdapterEventStreamError`. Already-buffered notifications are still delivered first. The existing `ProviderSessionManager` failure path then fails the running turn, if any, and releases the session, so the next turn or thread opens a fresh app-server.
- `OrchestratorMcpToolkit.integration.test.ts`: the `delegated_task_status` recording ends with the recorder closing the app-server while the parent turn is still open. The test keeps acting as that parent, so its loader now drops the recorded exit instead of replaying a process death.

## Scope and approval

Fixes #16502
GitCafe: https://git.cafe/pingdotgg/t3code/issues/16502
Triaged bug via the automated issue flow.

## Verification

- New tests in `CodexAdapterV2.test.ts`, "CodexAdapterV2 process exit", cover numeric exit and SIGKILL, each with an idle and an active turn. They assert that a dead process fails the event stream with `ProviderAdapterEventStreamError`. With main's `CodexAdapterV2.ts`/`client.ts`, all 4 fail with `AssertionError: the session manager must receive process death so it can retire the shared process`. With this change, all 4 pass.
- Every test file that uses a Codex replay transcript passes: 13 files, 182/182. That includes `OrchestratorMcpToolkit.integration.test.ts` 2/2, which fails on the first fix commit without the loader change. `vp run check:changed` passes: format, lint, knip, typecheck, tests.
- Real-app check (Linux, Codex CLI 0.163.0, `vp run dev` + web client, GPT-6-Astra); full [runtime transcript](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/76972aafa4616c06/transcript.txt):

| Step | Before ([ed3f965e07](https://github.com/pingdotgg/t3code/commit/ed3f965e07)) | After (fix branch) |
|---|---|---|
| SIGKILL app-server, follow-up turn on same thread | ![before follow-up fails](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/908dfefa8caf9421/before-2-followup.png) | ![after follow-up succeeds with a new app-server](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/f1d8d2e84ccd327c/after-2-followup.png) |
| SIGKILL app-server, brand-new Codex thread | ![before new thread fails with read-process-exit-status](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/12d53eaf7abd3858/before-3-new-thread.png) | ![after new thread succeeds](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/dfdb540eaa9ba0cf/after-3-new-thread.png) |
| SIGKILL during a running turn, then a new turn | n/a | ![running turn fails with event stream closed](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/54fe778f61851210/after-4-inflight-kill.png) ![next turn recovers](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/719a2a21c579c689/after-5-recovered.png) |

After each kill on the fix branch, a new `codex app-server` pid appeared, and `read-process-exit-status` never showed up as a turn-start failure.

Not checked: macOS and Windows hosts, and a systemd-oomd whole-scope kill. A delegated Codex child was not driven separately; it uses the same shared session and `ensureThread` path as the new-thread case.

Model: Claude Opus 5.5 (triage, review fixes) and GPT-6.1-Sol (original fix). Harness: Claude Code and Codex in T3 Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


---

## Independent review and proof audit (maintainer takeover, head [1afd783](https://github.com/pingdotgg/t3code/commit/1afd783c5e4ee4764ccc24bbe1649d25be539cee))

**Reviews.** Two read-only reviewers looked at the full diff against main.
- Claude Sonnet 5.5: **MERGE**.
- Claude Opus 5.5: **MERGE**.

Both confirm:
- Buffered notifications always reach the session before the failure. The handlers run inline on the protocol reader fiber, termination is signalled last, and `Queue.fail` keeps already-buffered items.
- A normal close, scope close or server shutdown is not misread as a crash.
- All Codex threads on an instance share one app-server, so one death fails every running turn on that instance and every thread recovers on one fresh process.
- The replay-loader change is a legitimate fixture fix.
- https://git.cafe/pingdotgg/t3code/pulls/23631 merges cleanly in either order.

Low, non-blocking notes:
- Events emitted from forked fibers after the death are dropped: the goal-continuation grace release, or a Stop that races the crash. Those runs show failed rather than completed or interrupted, which is still better than the permanent wedge on main.
- An idle death now writes the session error to every attached thread.
- The new adapter tests assert after a single poll.
- With https://git.cafe/pingdotgg/t3code/pulls/23631: a thread whose last turn ran in Full access may lose its remembered permission profile after a restart. That comes from https://git.cafe/pingdotgg/t3code/pulls/23631's resume-based design. Rerun its profile test across a restart after both land.

**Proof audit: SUFFICIENT.** The rows below are new. Before is merge-base [ed3f965](https://github.com/pingdotgg/t3code/commit/ed3f965e0768f9a77729a17c5145a90ff3ab6210), after is head. Each ran the real T3 server from that checkout. Codex returned 429 tonight, so turns were driven by a **scripted stand-in `codex app-server`** that speaks the same JSON-RPC over stdio. Rows R1 and R2 stay covered by the author's real-Codex run above.

| Row | Before (main) | After (this PR) |
|---|---|---|
| R3 SIGKILL during a running turn | Turn still `running` after 70 s, next turn stays `queued`, no new app-server | Turn fails in 0.3 s (`ProviderAdapterEventStreamError`), next turn completes on a new pid (1972592 → 1974071) |
| R4 `exit(17)` mid-turn | Turn still `running` after 70 s | Turn fails in 0.4 s, last error `exited with code 17`, next turn on a new pid (1974071 → 1974968) |
| R4 `exit(17)` while idle | Next turn fails with `ProviderAdapterTurnStartError` (code 17), no respawn | Next turn completes on a new pid (1974968 → 1975188) |
| R5a Two threads share one app-server; SIGKILL while T1 runs and T2 is idle | T1 still `running` after 70 s; T2's follow-up fails with `read-process-exit-status` | T1 fails in 0.2 s, T2 gets no false failure, one release for the shared session, both follow-ups complete on one new pid (1975188 → 1975815) via `thread/resume` |
| R5b Both threads running at the kill | n/a (main cannot respawn after R5a) | Both turns fail in 0.1 s, one release, both follow-ups complete on one new pid (1975815 → 1976466) |
| R6 Interrupt, archive, SIGTERM the server | Interrupted / ready / stopped | Same; no new `ProviderAdapterEventStreamError`, session ends `stopped` |

The release reason is inferred from session status: `releaseStatusFor` maps only `runtime_error` to `error` (`ProviderSessionManager.ts:238`). The trace does not log the reason itself.

- Capture transcript: [transcript.txt](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/45daa8d572852068/transcript.txt)
- Auditor's independent R4 rerun (after and before): [rerun.txt](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/214e7145cf174c84/rerun.txt)
- Driver (stand-in app-server, wrapper, RPC driver): [driver.tar.gz](https://gh-file-drop-api-prod-mi5fy3sowv63ufte.pinglabs.workers.dev/f/1dcddc1512f3e146/driver.tar.gz), sha256 `1f86ff3f3f7767bb9e58b3feb63f386847f232f2693b5b873c6eb00986d85def`. The bundle does not write the `settings.json` that points Codex `binaryPath` at the stand-in, so set that by hand.

Still unverified:
- A real-Codex run of R3-before and of R4–R6 (the model returned 429).
- macOS and Windows hosts.

<details><summary>Verification log</summary>

- 2026-10-22, fs-04, head `1afd783c5e4ee4764ccc24bbe1649d25be539cee`, merge-base `ed3f965e0768f9a77729a17c5145a90ff3ab6210`.
- Reviews: Sonnet 5.5 MERGE, Opus 5.5 MERGE.
- Proof audit pass 1: R1–R3 covered by the real run; R3-before, R4, R5 and R6 uncovered.
- Captures: R3, R4, R5a, R5b and R6, before and after, real server with the stand-in app-server. All pass on head; main hangs or fails as described.
- Proof audit pass 2: driver sha256 and both SHAs checked, every row checked against the transcript, R4 rerun on both sides. **SUFFICIENT.**
- No code changes, commits or pushes were made during this verification.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)