Sign up
View stargazersSign up

[Bug]: Terminal "timed out negotiating with the code-mode host" on every call while the VM's Codex app-server is backlogged#24322

opened6d agobyJJulius Marminge

What happens

On the Freestyle VMs that run issue-triage leads (Codex gpt-6.1-sol through T3 Code, 4 leads per VM), the terminal tool fails with "timed out negotiating with the code-mode host" on every call for a stretch, then recovers. The lead cannot run commands, read artifacts or use cafe, and aborts its round.

Evidence

  • 219 open friction entries from leads between 2026-10-21 and 2026-10-23 (120 on 10-22, 97 on 10-23), wording "Terminal host negotiation timed out twice", "Code-mode host negotiation timed out on every call", "Command and MCP transport repeatedly timed out".
  • The affected VMs are the same ones whose shared Codex app-server was backlogged at the time (fs-16 33 entries, fs-23 28, fs-02 24, fs-06 19, fs-35 17), see #24300 for that failure: thread/start answered hours late or never. MCP tool calls timed out alongside the terminal ("Terminal and orchestration calls repeatedly timed out").
  • A lead that fails this way retries twice and then reports blocked; nothing on the VM is down (t3code.service up, /.well-known/t3/environment answers 200, load normal).

Expected

The code-mode host handshake should not depend on a provider app-server's stdio stream being drained, or if it does, it should fail fast with a message that names the cause instead of a negotiation timeout. If it is the same backlog as #24300, say so here and the fix there covers it.

Where

Freestyle VMs fs-01..fs-40, Ubuntu, T3 Code server from the preview rollout of 2026-10-22, codex-cli 0.163.0. Provider event logs under ~/.t3/userdata/logs/provider/ on the VM show the outgoing requests without responses.

JJulius Marminge4d ago

Tested 53896aea76 on Ubuntu fs-07 with Codex 0.163.0 and GPT-6.1-Sol through the real web app. Terminal probes passed in empty state, a snapshot with 18 stopped threads, and four overlapping Codex sessions (six commands total, all exit 0). These short-lived runs did not recreate the reported backlog or negotiation timeout. Reproduction transcript.

Please attach one timestamped terminal negotiation failure and the provider request/response events from the same time window, with credentials and prompt content removed. Include any repeatable workload that induces the backlog beyond starting four concurrent sessions. The aggregate counts do not establish whether this is the same failure as #24300.

JJulius Marminge4d ago

Retested current main dd5b5401e1 on Ubuntu fs-08 through the real web app, with Codex 0.163.0 and GPT-6.1-Sol. Three sequential terminal calls in empty state, eight calls across four overlapping sessions, and one call with a snapshot of 25 stopped threads all exited 0. No negotiation timeout or app-server backlog appeared. Timestamped command transcript.

Still needed: one timestamped negotiation failure with the provider request/response events from the same window, credentials and prompt content removed, plus the workload that induces the backlog. These successful short runs do not establish whether this shares a cause with #24300.

JJulius Marminge3d ago

Not reproduced on main 96fc7b9551, Ubuntu fs-34, codex-cli 0.163.0.

The message is raised inside Codex, not T3: the app-server spawns codex-code-mode-host as its own child, writes a ClientHello on that child's stdin and gives up after 30 s without a HostHello (connection.rs, LOCAL_HOST_STARTUP_TIMEOUT). That handshake uses private pipes and never touches the app-server's stdout to T3, so a T3-side read backlog like #24300 should not cause it. I checked this directly with a driver speaking JSON-RPC to codex app-server, with its stdout pipe shrunk to 4 KiB:

  1. Reader stalled 90 s with the live host reused: 4/4 exec calls exit 0.
  2. Reader stalled 90 s with the host SIGKILLed at the start, forcing a respawn and handshake during the stall: 4/4 exit 0.
  3. Three threads on one app-server, reader stalled 150 s, about 7.3 MB of output queued, host first spawned during the stall: 12/12 exit 0.
  4. Whole tree in a cgroup scope held over MemoryHigh: with mild throttling, 8/8 exit 0; with severe throttling the app-server never answered initialize at all, rather than failing negotiation.

No negotiation error in 36 exec calls. Transcript, drivers: driver.py, driver_scale.py, throttle.sh, hog.py.

To tell why the host does not answer within 30 s, the next failure needs, from that machine while it is failing:

  • ps -o pid,ppid,etime,stat,cmd for the shared codex app-server and every codex-code-mode-host under it (is a host running, how old, what state);
  • cat of memory.events and memory.pressure in the app-t3code-agents.slice cgroup (whether high is climbing);
  • the WARN/ERROR rows from ~/.codex/logs_2.sqlite for that window, prompt content removed;
  • whether codex was upgraded while that app-server was running.