Sign up
View stargazersSign up

A running effect with an expired lease blocks every later run on its thread, and a server restart doesn't clear it#25636

opened3d agobyJJulius Marminge

What happened

On nucbox-1 (0.1.3-preview.20261026.3702), the scheduled Codex thread Issue Triager (mcp:launch:870f4a9a-16a6-4ff8-a844-a7c824708e01:issue-dispatcher-v1) started no run for 12 hours. Its last run ended its turn at 09:14Z, which is #25000. The new part is that a server restart didn't clear the hang:

  1. 09:14:17Z: the run's turn/completed arrives. A command item completes after it (#25000). At 09:44:17Z T3 sends thread/unsubscribe for the thread. That is the last provider event for this thread.
  2. 09:44:18Z: the shared Codex session provider-session:provider-instance:codex:shared was seen stopped, bound to the delegate thread "Codex smoke test 2" (thread:delegated-task:command%3Amcp%3Acf3cf8b8-be40-4945-8320-23164dc5230b%3Adelegate-task%3Aae96ee0f-c297-4119-ac3d-a0aec3c64d38, finished at 06:45Z, last updated 09:44:18.983Z).
  3. 18:32–18:33Z: the run was stopped by hand. That created a delegated-tasks.stop effect at 18:33:04Z.
  4. 18:45:21Z: the server restarted. delegated-tasks.stop is replay-safe, so it was requeued and claimed again by the new process. It is still running with attempt_count 2, lease owner orchestration-v2:1170978 (the current server process) and lease_expires_at 18:45:52Z. It never completed.
  5. Every later effect on the thread is blocked behind it:
    • run 4333 (a direct message, 18:48:27Z): its provider-turn.start stayed pending, attempt_count 0, until the run was interrupted at 20:33:42Z ("interrupted before its provider turn started"). No log line, no Codex process.
    • run 4349: same, interrupted at 20:37:34Z.
    • run 4351 (scheduled tick, 20:37:39Z): starting, startedAt null, effect pending, attempt_count 0, still so at 21:30Z.
    • two later delegated-tasks.stop effects (20:33:44Z, 20:37:36Z): pending, attempt_count 0.
    • 111 runs queued between 09:09Z and 18:44Z never started. They were cancelled by hand between 18:32Z and 21:18Z. Runs 4355–4359 are still queued.
  6. Codex itself was fine. A fresh Codex delegate (codex-liveness-probe-20261026a) ran at 20:35Z in 16 s, and a new Codex thread started at 21:19Z has completed 9 runs on the same shared session, which is now ready.

After the restart the thread shell reported status: cancelled, activeRunId: null, so nothing showed it was wedged.

Expected

  • An effect whose lease has expired is reclaimed or failed. It shouldn't block the thread forever.
  • A stuck delegated-tasks.stop doesn't stop new runs on the parent thread from starting.
  • After a restart, queued runs on a thread with no active run start.

Where (origin/main fe507c5a1e)

  • apps/server/src/orchestration-v2/EffectOutbox.ts:318-357 claimableCandidatePredicate: a candidate is never claimable while any effect on the same thread has status = 'running'. lease_expires_at isn't checked.
  • EffectOutbox.ts:125-136 lists delegated-tasks.stop as replay-safe, so reconcileAfterProcessLoss (:492) requeues it on restart instead of cancelling it.

Related, not the same

  • #25000: the original run that never settles (step 1).
  • #24743: a stuck run blocks the queue, but a restart cleared it there. Here the restart didn't.
  • #24300: shared Codex app-server backlog. It doesn't apply here; the app-server answered other threads immediately.
  • #16441: a run stays starting when recovery state can't be read. That's a different cause; here the start effect is never claimed.