[Bug]: One unresponsive provider holds all four effect workers and stalls every thread#16636

Confirmed on current main (517188b). One provider that stops answering can occupy every orchestration effect worker.

Root cause.

• Fixed pool of four workers.

  • apps/server/src/orchestration-v2/EffectWorker.ts:781 sets DEFAULT_EFFECT_WORKER_CONCURRENCY = 4.
  • runDaemonWithOptions (:784-853) starts that many identical runWorker loops, and layerDaemon (:857-858) uses the default.
  • The loops are not partitioned by provider. Per-thread ordering in EffectOutbox.ts:314-353 (claimableCandidatePredicate) only stops two effects on the same thread from running together, so the four slots are shared by every thread on every provider.

• No execution deadline. runOnce (EffectWorker.ts:681-686) awaits Effect.raceFirst(executor.execute(...), cancellation). The only other branch is an explicit cancellation, with no timeout.

  • provider-turn.start → ProviderTurnStartService.start and provider-turn.interrupt → ProviderTurnControlService.interrupt (dispatch at EffectWorker.ts:152-204) add no timeout either. ProviderTurnControlService.ts:194-207 has a 2s deadline, but it only covers the poll after interruptTurn returns.
  • In the Codex adapter, client.request("turn/start", ...) (CodexAdapterV2.ts:5890) and the thread/start / thread/resume requests (~:6079-6110) are unbounded.
  • packages/effect-codex-app-server/src/client.ts has no default request timeout.
  • Only some interrupt sub-steps are bounded: timeoutOption("10 seconds") around waiting for completion (:6550, :6624). The bare turn/interrupt request at :6561 is not.

• The lease is never enforced while the process is alive.

  • claimNext (EffectOutbox.ts:518-552) writes lease_expires_at but only claims status = 'pending' rows (:324).
  • Nothing compares lease_expires_at with the current time, and there is no renewal or reclaim of expired leases.
  • Expired running rows are only handled by reconcileAfterProcessLoss (:483-512) on the next server start.
  • workerId is per process (EffectWorker.ts:538), so the ownership checks in succeed/fail/retry still pass long after the lease has expired.

  In practice the 30s lease is metadata only. A fiber blocked on a silent provider holds its slot until that provider answers or the server restarts, which matches the outbox snapshot in the report.

Suggested fix direction. Both parts from the report seem right, and they work independently:

1. Give provider-control effects an execution deadline that fails the effect on timeout. Either raise it above the observed p95/max for provider-turn.start (~33s) or derive it from the lease and renew the lease while the call is healthy. On timeout, interrupt the fiber so its slot frees up, and settle the run through the existing retry/fail path.
2. Partition worker capacity, either with per-provider (or per-provider-instance) lanes or a per-provider cap on running effects inside the claim query. That way one wedged provider can't take every slot.

A unit test with a stub executor whose provider-turn.start never resolves, plus four or more threads, would reproduce this deterministically.

Related.

• #15979 (the Codex app-server condition that triggered it)
• #13392 (closed) and #16441, which cover runs stuck in starting with different triggers
• #16502 (Codex sub-agent failures after the shared app-server dies)

I didn't find an open PR that adds a deadline or per-provider lanes.

Another occurrence on 0.1.0 (Linux, t3 serve), and in our case no Codex involved: every stuck effect was on a Claude Agent provider instance (through a local Anthropic-compatible proxy that kept answering normally; it served only 46 short requests during the stall, all 200 under 10 s).

Timeline (UTC, 2026-10-18), from orchestration_v2_effect_outbox:

• The server log is silent from 00:41 to 03:37 except for one environment api request failed with errorTag: 'Interrupt' at 00:40:59. At 03:37:32 everything drains at once: dozens of queued thread.auto-settle attempts fail with "changed before automatic settlement".
• The Claude process of thread A received the user message only at 03:37:38, as a "Provider context handoff". It answered 4 s later. So the provider was idle the whole time, and the turn start was waiting on a worker.
• No OOM, no restart, and no stuck git/glab children afterwards. It cleared by itself after about 3 h, unlike #20419.
• Over three days, 433 of 439 provider-turn.start effects finished in under 15 s; the other 6 are the rows above. So a deadline on the effect itself, not only on the lease, would separate these cleanly.

The trigger seems to be a run cancelled or interrupted right before (thread A run cancelled at 00:27, steer/interrupts from an automation client). After that, the next start on the same thread never resolved until a worker freed up.

Workaround we run meanwhile: a watcher that reads the outbox every 15 s. It alerts on any effect pending or running for more than 15 s (checkpoint: 60 s). After 30 s more it kills the Claude CLI process of that thread, matched via --session-id in provider_threads, so the start fails and frees its worker.

Another occurrence, Claude only, on nightly 0.1.1-nightly.20261018.3315 (Linux). Here the blocking call is identified.

Twelve Claude threads had stalled while their agents slice was throttled at MemoryHigh (the claude processes were alive but barely running). Stop was sent to all 12 at 13:41:22. From the trace and orchestration_v2_effect_outbox:

• All 12 provider-turn.interrupt effects were queued within about a second.
• Only four started. They took 140.5, 220.3, 132.9 and 209.9 s.
• Each of the other eight started 20–43 ms after an earlier one finished, so never more than four ran at once. All twelve took 3 min 43 s.
• 24 preview.cleanup / terminal.cleanup effects for those threads waited 217–219 s behind them. They normally take about 0 s.

The time was spent in ClaudeAdapterV2.interruptTurn awaiting existing.query.interrupt, the SDK's interrupt control request. That call has no deadline: the SDK resolves it only when the CLI sends its control_response. The existing 10 s bound covers only the close that follows. Each of the four spans ended exactly 2.0 s, the SDK's close grace, after its CLI finally answered. Those CLIs had been silent for up to 369 s.

#21303 bounds that one call and always continues to the close, so Stop ends the turn as interrupted. It combines #16419 and #16439 on current main. In a live check with a SIGSTOPped real Claude CLI, the interruptTurn span went from 90.7 s (it only ended after SIGCONT) to 12.0 s, and the frozen CLI was killed 17 s after Stop. It doesn't touch the worker pool itself, so it fixes one source of this stall, not the issue.

Reproduced on the current nightly, 0.1.2-nightly.20261021.3499 on Linux (t3 serve, default four effect workers), with the outbox snapshot and server log taken while it was stuck: secret gist (outbox-snapshot.txt, server-log.txt, plus the trace spans and the stub).

Setup: a Codex instance whose app-server stays alive but never answers thread/start (a small proxy in front of codex-cli 0.163.0 that drops it, codex-wedged.js in the gist), four Codex threads, then one Claude thread, on a separate server with a throwaway home.

• Pending Claude provider-turn.start: queued at 13:29:27.885Z, attempt count 0.
• The four running effects, all provider-turn.start on Codex threads, claimed at 13:29:20.093, 13:29:21.014, 13:29:21.912 and 13:29:22.803Z. At the snapshot (13:39:56Z) they were 636, 636, 635 and 634 s old, about 605 s past their 30 s leases, with attempt count 1.
• The Claude start was pending for 629 s at the same moment.

Each of the four was blocked in a single CodexAppServerProtocol.request (thread/start under ProviderTurnStartService.loadFromProvider). Each span ran about 639 s and ended only when I killed the app-server at 13:39:59.533Z. Nothing timed out or reclaimed the expired leases in the meantime. The Claude start was claimed 18 ms later (13:39:59.551Z), 631.7 s after it was queued, and its turn then completed normally (checkpoint captured at 13:40:03Z).

Same here on T3 0.1.1 (t3 serve) with Codex 0.162.1, in a Linux VM with 4 GB and no swap. From about 15:30 UTC on 23 October four provider-turn.start effects went to running and stayed there; their leases expired and were never reclaimed, so 55 turn starts and 53 title generations waited behind them, and no turn finished on any thread until we stopped the runs and restarted T3 at 21:52 UTC.

The trigger was the agents slice holding at MemoryHigh (2.0 GB of 4 GB): 3.6 million high events, 96% of its time stalled, Codex's threads in __mem_cgroup_handle_over_high, while the VM had 846 MB free. In the slice were the app-server with about 38 loaded threads, most waiting on command approvals, and 31 processes of one MCP plugin, about 78 MB each.

Adding swap (Ubuntu's zram) has kept the slice from stalling since, even with 30 turns waiting on approvals at once. It only removes this trigger: the four workers never freed themselves, so any provider that stops answering can still stall every thread.