[Bug]: Codex sub-agents keep failing after the shared App Server is killed by SIGKILL#16502

Triage

Thanks @BenWortley for the detailed report and the cause chain, it made this quick to trace. Reading current main (ac8e9453ca), this looks like a Codex session that stays stuck after the shared app-server dies. It's separate from #12904 and #15359.

What I found

• How the error is produced. Effect's ChildProcess.exitCode fails when Node reports a signal and a null exit code (Process interrupted due to receipt of signal: 'SIGKILL'). makeTerminationError in packages/effect-codex-app-server/src/_internal/stdio.ts maps that failure to CodexAppServerTransportError (read-process-exit-status), and only maps a numeric code to CodexAppServerProcessExitedError. The protocol keeps that error, so every later RPC fails with it, and ensureThread wraps it as ProviderAdapterEnsureThreadError ("Provider turn failed to start"). The four quick retries are the effect worker: five attempts with backoff starting at 100ms.
• The error tag alone may not be the deciding factor. On the new orchestrator, the only catch of CodexAppServerProcessExitedError / CodexAppServerInputStreamEndedError is terminateBackgroundTerminal in CodexAdapterV2.ts. A numeric exit would fail the next thread/start with ProcessExitedError and still leave the same dead process in place, so remapping SIGKILL would change the log line without starting a new app-server.
• Why no new app-server is spawned.
  • Codex sets supportsMultipleProviderThreadsPerSession, so every Codex thread for an instance shares one derived session id (provider-session:…:shared). A new delegate_task attaches to that session instead of opening another.
  • The adapter event stream is Queue.take on a queue that isn't ended or failed when the child dies, and ProviderSessionManager only drops a session when that stream ends or fails. The Codex client also doesn't wire onTermination. So the dead client stays the live session, and open returns it instead of spawning.
  • Idle release is 30 minutes and each attach resets it. If the killed turn still has running commands, tools, or subagents in memory, hasPendingBackgroundWork can pin the session for up to 4 hours. That fits "no recovery over 22 minutes" and "restarting the T3 server fixes it". Claude uses a separate per-thread session, which is why it keeps working.
• Related items. #12904 was this signal-exit symptom on the old runtime and was closed when the new orchestrator landed. #15359 and #15406 are about desktop backend log wording for the same Effect exitCode behavior, not this shared Codex process.

Likely fix area

The Codex adapter / ProviderSessionManager session lifecycle. One option is to treat any app-server death, including a signal, as session death: end or fail the adapter event stream and the in-flight turns so the shared session is released and the next thread spawns a fresh codex app-server. Adjusting the error tag in stdio.ts could go alongside that, but probably wouldn't be enough on its own.

Workaround: restarting the T3 server, as you found.

A maintainer will decide on the fix direction.

Another occurrence, this time hitting an in-flight turn (steer/interrupt path) rather than a new thread:

• T3 0.0.46-nightly.20261011.2955 (72fbad7b), Codex CLI 0.162.1, Linux x64 (headless, t3 serve), shared custom Codex provider instance. Host was under heavy memory pressure (no swap, global OOM kills nearby); OOM is the likely killer but not confirmed for this PID.
• The shared codex app-server died mid-turn (last notification was a completed reasoning item, no turn/completed). T3 kept the run running.
• ~23 min later a follow-up message failed provider-turn.steer five times: ProviderAdapterSteerRunError → CodexAppServerTransportError (read-process-exit-status) → Process interrupted due to receipt of signal: 'SIGKILL'.
• Stop: ProviderAdapterInterruptError against the same dead transport; the fallback then marked the run interrupted. The next message's run failed to start, and the shared provider session is still projected as ready.
• Source at that commit matches the analysis above: CodexAppServerClient.make doesn't wire the protocol's onTermination, so the CodexAdapterV2 event stream never ends and ProviderSessionManager keeps reusing the dead shared runtime.

Suggested extra regression case for #17046: kill the app-server after turn/start but before turn/completed, then assert the in-flight run goes terminal, steer/interrupt fail fast (not 5× retries), and the next message spawns a fresh app-server.

Hitting this one constantly on a 16GB ram host where systemd-oomd kills the codex app server constantly.

Additional confirmed Linux occurrence, including T3 0.1.1-nightly.20261016.3203: a systemd-oomd kill of the shared Codex App Server leaves the T3 server healthy but Codex work stuck until manual recovery. Adding evidence here rather than opening a duplicate of this issue or #17045.

Environment

• Linux x86_64, Omarchy/Arch, kernel 7.2.5-3-omarchy, systemd 261.2.
• Approximately 30 GiB RAM and 30 GiB swap.
• Desktop client connected to a single headless t3 serve systemd user service.
• Codex CLI 0.162.1, using a custom OpenAI-compatible backend with features.instant_interrupt=false.
• The first two incidents below occurred on 0.0.46-nightly.20261014.3132. The package was updated at 09:14 local time; the backend restarted at 09:17:44, and the last incident occurred on 0.1.1-nightly.20261016.3203 (release commit e3b35d1b2440b104f0bc2e357377c348121234dd).

Three scope kills on October 16, 2026

Times are UTC−03:00. Memory figures and killed counts are the systemd journal's own reporting, not an independently enumerated count of unique OS processes.

Sanitized excerpt from the last incident:

The earlier same-day failure was captured in the T3 trace as:

At 09:35, after the third kill:

• t3code.service was still active/running, and its HTTP endpoint returned 200.
• No active t3code-codex-* scope remained; the killed scope had Result=oom-kill.
• Thirteen runs were still projected as running: the main conversation and twelve delegated tasks. No pending/running effect-outbox entries or unresolved runtime requests were present.
• The UI subsequently displayed: The provider could not start this turn. Retry the turn; if it keeps failing, check the provider setup and server logs.

Using Stop on the main conversation terminalized all thirteen runs as interrupted. Restarting the T3 service then passed probeCodexAppServerProvider and requestAllCodexModels, and the user resumed the conversation. No database rows were edited manually.

Mitigations tested before the third kill

The third incident happened despite launching the same MCP versions directly instead of through three retained npx parents, setting thread_unload_delay_secs=1, and setting native Codex agents.max_threads=4 (read back as max_concurrent_threads_per_session=4). That native limit does not bound independent T3 delegate_task runs.

A separate no-inference JSONL probe (initialize, two ephemeral thread/start calls) measured:

Summed RSS can double-count shared pages. This demonstrates startup overhead and that Codex can release these MCP children when explicitly unsubscribed; it does not establish a T3 memory leak or prove which processes held all memory at the actual kill. thread/loaded/list became empty after the explicit unsubscribe test.

Related: #15979 covers MCP residency and was closed after idle unloading was added. The installed T3 bundle still uses a 30-minute idle timeout; the latest scope was killed after only ~11 minutes, so this occurrence does not establish that the timeout implementation is broken. The distinct confirmed recovery problem remains: a dead shared Codex runtime and its runs are not promptly retired/replaced. #18139 appears relevant to that recovery path. This was a whole-scope systemd-oomd kill, not proof of the single-tool kernel-OOM propagation case discussed in #18329.

Private prompts, project data, full provider logs, credentials, and local connection addresses are omitted.

Another confirmed occurrence on Linux on Oct 18: the kernel OOM-killed a Node subprocess inside the shared Codex scope. With OOMPolicy=stop, systemd stopped the whole scope (Result=oom-kill), including the app-server.

Threads stayed “working” for over an hour, and the provider session remained ready despite no Codex process being alive. A later steer reported Codex App Server process exited with code 0; stopping and retrying the turn then failed to start it.

This also supports covering exit code 0 in the recovery fix: the OOM cause was visible in the scope status, not the provider exit code.

Another occurrence, on the 0.1.0 stable desktop build (Arch t3code-bin 0.1.0, Electron-bundled server). Two things here that the earlier reports don't cover: how it shows up for delegated tasks from the parent's side, and that an unrelated new thread was still failing almost an hour later.

Environment: Linux x86_64 (Omarchy/Arch), 61 GiB RAM + 61 GiB swap, systemd-oomd enabled, Codex CLI 0.162.1, GPT-6.1 Sol. Several Claude parent sessions were delegating tasks to Codex via delegate_task.

Timeline (local time)

• ~20:33: five delegated Codex tasks start on the shared app-server.
• 20:56:10: systemd-oomd kills the shared t3code-codex-<hash>.scope. Journal: systemd-oomd killed 805 process(es) in this unit, Consumed 2h 39min CPU time over 1h 2min wall clock time, 21.9G memory peak, 9G memory swap peak. Most of that was tool children (builds/tests) of the parallel Codex tasks, not the app-server itself.
• 20:58 to 21:45: 25 failed spans, every one against the same dead scope. No new Codex scope was ever started. 24 were orchestrationV2.providerTurnStart.start and one was orchestrationV2.deliverContextHandoffs. Same chain as in the report: ProviderAdapterEnsureThreadError / Failed to inject historical context → CodexAppServerTransportError ('read-process-exit-status') → Process interrupted due to receipt of signal: 'SIGKILL'.
• 21:42: the five in-flight delegated runs finally go failed, 46 minutes after their process died. They stayed running the whole time.
• 21:45:36: a brand-new top-level Codex thread, with no relation to the delegated work, fails to start about 5 ms after the first message. The UI shows "Provider turn failed to start". So 49 minutes after the kill, every new Codex thread on the instance was still dead.
• Two other Codex runs that were in flight at the kill were still projected as running when I checked afterwards. They never went terminal.

What it looks like from the delegating parent

• New delegations after the kill failed within about 2 s, and the parent got status: "failed" with latestTerminalSummary set to the raw transport string Codex App Server transport operation 'read-process-exit-status' failed. Nothing told the parent that the process had been killed, or that a retry on the same provider would fail.
• The parents did the reasonable thing and retried on Codex (<task>-retry). Those failed the same way in about 2 s. Then they re-delegated the work to a different provider.
• Net result in the sidebar: a "Failed" copy of each task next to a working copy, and no way to tell whether the failed ones had finished. The in-flight ones hadn't. They were killed mid-turn, so their worktrees may hold partial changes.

Still present on main

As of 13aa882c22, CodexAdapterV2 still builds the session event stream as Stream.fromEffectRepeat(Queue.take(events)) with nothing ending that queue when the child exits, which matches the triage above. #18139 and #18764 both look like they'd fix this. Neither is merged yet, and #18329 (OOM-specific) has conflicts. Happy to test a build of either PR against a real oomd kill.

Two small asks beyond recovery, since they shaped what the parent agents did:

1. Report a signal death to delegate_task callers as "the provider process was killed (SIGKILL)" rather than the transport-operation name, so an agent doesn't burn a retry on the same dead process.
2. When the app-server dies, end its in-flight runs right away as interrupted, rather than leaving them "running" for most of an hour and then plain "failed". That makes it clear the work didn't finish.

Workaround is still restarting T3.

Another occurrence, on Windows, with a cause that isn't running out of memory, so it isn't specific to Linux.

Environment: Windows 11 Pro (26200), T3 desktop 0.1.1-nightly.20261017.3238, Codex CLI 0.163.0 installed globally with npm, default Codex home, no shadow home, no launch args.

Trigger: An agent in another thread ran taskkill /F /IM node.exe to clear a hung test runner. The npm codex.cmd shim runs node …\@openai\codex\bin\codex.js, so the shared codex app-server was killed with every other Node process. There's no systemd scope or OOM killer on Windows; any broad process kill does it. Agents use taskkill /IM node.exe fairly often on Windows.

What's different from earlier reports:

• Exit code 1, not a signal. On Windows a forced kill sets exit code 1, so the chain is ProviderAdapterEnsureThreadError → CodexAppServerProcessExitedError: Codex App Server process exited with code 1. This shows that when the exit is reported as a number, T3 still keeps reusing the dead session, which is what the triage predicted. Changing how a signal is reported won't fix it; only ending the session when the process exits will.
• The Codex status check stays green. checkCodexProviderStatus / probeCodexAppServerProvider start their own short-lived app-server, so they keep succeeding and the provider looks healthy in Settings. Meanwhile every turn fails within milliseconds.
• No new process is started (verified). I temporarily pointed binaryPath at a wrapper script that logged each launch. The only launch was the status check after the settings change. Delegated Codex turns run at the same time started no process and failed with the same code-1 error.
• Settings changes don't release the session. Neither turning the Codex provider instance off and back on nor changing its binaryPath cleared it. The next turn still got the dead shared session. This looks related to #19025. Only a T3 restart clears it.
• No diagnostics are kept. The client discards app-server stderr (runDrain(handle.stderr)), and Codex's own log (~/.codex/logs_2.sqlite) stops at the kill. The only way to trace this was matching timestamps against the other thread's tool calls.

Timeline (local time): Codex's last log entry was at 00:01:05, the taskkill ran at 00:02:41, and the first Codex turn failed at 00:07:15. It was still failing at 00:26, after four test delegations; no Codex app-server process was alive.

As far as I can tell, #21013 and #20992 would cover this too, because both trigger on any end of the connection to the app-server, regardless of how the exit is reported.

Written by Claude Opus 5.5 (claude-opus-5-5), running in Claude Code inside T3 Code, at the request of the user who hit this. The investigation and the tests above were done on their machine.

Another occurrence on Linux, same cause as the Oct 18 OOM report above.

Environment: Ubuntu (x86_64), T3 Code 0.1.0 running as the t3code.service systemd user service, Codex CLI 0.163.0 (npm global install, default Codex home, no shadow home, no launch args).

Trigger: At 12:03:27 UTC the kernel OOM killer killed processes in the shared Codex scope (t3code-codex-*.scope, memory peak 13G, probably from agent workloads running in that scope). systemd then stopped the whole scope (Result: oom-kill). The Codex log ends with a graceful shutdown at 12:03:28, so T3 reports exited with code 0 instead of a SIGKILL.

Symptom: For the next 8+ hours, every Codex turn and every new delegated Codex task failed in about 5 ms with ProviderAdapterEnsureThreadError caused by CodexAppServerProcessExitedError: Codex App Server process exited with code 0. No new codex app-server was spawned. The short-lived Codex probes (skills, model list, provider status, thread title generation) kept working, so the provider showed as healthy. Turning the provider instance off and on in settings did not help. Only restarting t3code.service recovers it.

Follow-up to my comment above: after restarting t3code.service at 20:34 UTC, it happened again 31 minutes later. One detail I haven't seen mentioned yet: T3 makes its own Codex processes the kernel's first OOM target, even when the memory hog is outside T3.

At 21:05:24 UTC there was a global OOM (constraint=CONSTRAINT_NONE, global_oom). From the kernel's task dump at that moment:

The kernel killed the 0.9 GB process in the Codex scope (Killed process 1805091 (MainThread) ... oom_score_adj:800), not the 9.9 GB container process. systemd then stopped the whole t3code-codex-*.scope (Result: oom-kill), and the Codex App Server went with it, which leads to this bug.

So on a shared machine, any memory pressure from outside T3 is likely to land on the shared Codex App Server. With oom_score_adj=800 on its scope and systemd stopping the whole scope when any process in it is OOM-killed, every Codex thread breaks until T3 restarts. The fixes in #18764 / #21013 would make this recoverable. It may also be worth separating the app-server process from the agent's child processes, or not stopping the whole scope on a single OOM kill (e.g. OOMPolicy=continue), so that one killed child doesn't take down the shared server.

Environment as before: Ubuntu x86_64, 31 GB RAM, T3 Code 0.1.0 (t3code.service), Codex CLI 0.163.0.

Confirmed the same session-lifecycle failure on T3 Code 0.1.1, commit fe27e8e3dc12, Codex CLI 0.163.0, Linux x64. This occurrence retained a protocol parse error:

The shared session stayed ready. Subsequent attempts reused its failed transport and failed before a new turn could start. An independently started app-server could read the saved native thread, so deleting the conversation was unnecessary.

The public source at the tested commit shows the missing lifecycle connection:

• protocol.ts, handleTermination records terminationFailure, fails pending requests, signals termination, and supports an onTermination callback.
• client.ts does not connect that callback. Its returned client also does not expose a termination effect.
• CodexAdapterV2.ts creates an independent event queue and returns a stream that takes from it. Transport failure never fails that queue.
• ProviderSessionManager.ts already has cleanup for a failed provider event stream. Without that failure, open keeps returning the cached runtime.

A deterministic regression test reproduced the missing propagation without a model request or external connection:

1. Construct the real Codex client over in-memory Stdio, with queues for input and output.
2. Construct the real makeCodexAdapterV2, supply a client factory that returns that client, and open a test session. Use synthetic IDs and stub unrelated dependencies.
3. Inject the synthetic input unexpected stdout\n. In a separate test, end the input queue to simulate stdout EOF.
4. Collect session.events with a 500 ms test deadline and inspect the terminal error.

The 500 ms deadline detects the open stream; it is not a performance measurement. Both baseline failures and both patched passes were checked.

The tested fix exposes a replayable termination effect from the protocol through the client, then watches it in the adapter's scope. When the transport fails, the watcher fails the adapter event queue. The session manager can then release the failed runtime and open a replacement for later work. The adapter-side watcher is equivalent to the following, with raw.termination being the proposed new client member:

This must cover fatal parse failures as well as process exit and EOF. Restricting recovery to selected process-exit error tags would leave the parse-failure case broken. A termination observer also needs to see a failure that occurred before it subscribed.

After replacing the failed live runtime, thread/read and thread/resume succeeded for the original saved native thread. Conversation history and project files were preserved. Stale task state was cleaned separately. This verification did not replay the full workload or prove automatic recovery of interrupted work. Recovery must not silently resubmit an in-flight request whose outcome is unknown.

The actual invalid wire line was not retained, so its producer and the initial parse-failure trigger remain unconfirmed. This report establishes the subsequent failure to release the shared session. It does not establish an OOM event or require logging raw prompts, credentials, or wire payloads.

The approaches described in #20992 and #21013 appear relevant. The malformed-JSON test would be useful alongside their process-exit coverage. Neither PR was used for the local validation above.