Sign up
View stargazersSign up

Accepting a child's completion delivery reverts a completed parent run to waiting and stalls its thread#25757

opened2d agobyJJulius Marminge

What happens

A parent run whose turn ends while some of its delegated children are still running goes to waiting, then completed once its checkpoint is captured. When a child finishes in that window, accepting its completion delivery (notification.delivery.accept) plans a run.updated for the parent from a read taken while it was still waiting. If checkpoint capture commits completed first, the accept writes that stale snapshot back: the run returns to waiting with no completedAt or checkpointId. A waiting run counts as blocking, so its own delivery run and every later run on the thread queue behind it. Nothing times out; the thread is stuck until the server restarts.

Seen on (nucbox-1, 2026-10-27)

Issue Triager thread, Codex gpt-6.1-sol, 7 app-owned remote delegates started by run 436:

Time (UTC)Event
03:16:50run 436 starts (a scheduled tick), dispatches 7 delegate_task leads
03:17:56–03:18:174 leads finish; their results arrive as 3 steers into the live turn (all accepted, tasks delivered)
03:18:55.66the turn ends with 3 leads still running; run 436 becomes waiting
03:18:55.99checkpoint captured; run 436 becomes completed
03:18:56.79the delivery's run.updated writes the stale waiting snapshot back (completedAt: null, delivery generation 4)
03:18:56lead issue-lead-16629-r21 finishes
03:19:03completion delivery reserved: generation 4, message ordinal 1374, as the user message of run 438 (queued); task states claimed
03:34, 03:58the other two leads finish and join the same delivery (taskIds has 3 entries)
04:19run 436 is still waiting, run 438 still queued, runs 437–453 queued behind them

Run 436's projection at 04:19:

json
1"status": "waiting",
2"completedAt": null,
3"delegatedCompletion": {
4 "disposition": "open",
5 "nextGeneration": 5,
6 "delivery": { "generation": 4, "messageId": "…:ordinal:1374:317f6ae4-…", "taskIds": ["…issue-lead-16629-r21", "…issue-lead-22461-r2", "…issue-lead-25558-r7"] }
7}

The run attempt and root node are completed, all 7 subagents are completed, and the thread's effect outbox has no pending or leased effects (1,003 succeeded), so this is not #25636.

Expected

Delivery acceptance updates only the parent's delegatedCompletion bookkeeping and keeps a lifecycle or checkpoint change committed after its read, as acknowledge and dispose already do since #25672.

Related

  • #24326 / #24297: child results arriving after the parent's turn (delivery path)
  • #24743, #25000: other ways a run never settles and blocks the queue
  • #25644: the delivery run cannot be reordered either
JJulius Marminge2d ago

Correction: this is a duplicate of pingdotgg/t3code#25000, not a separate design flaw in how waiting runs block the queue.

Run 436's run.updated history shows it did complete, then an old snapshot reopened it:

Time (UTC)StatusNote
03:18:55.662waitingturn ended, 3 children still running
03:18:55.992completedcheckpoint captured, completedAt set
03:18:56.785waitingchild issue-lead-16629-r21 finished; its completion handling wrote a snapshot read before the checkpoint completed the run, with completedAt: null and delivery generation 4
04:32:26cancelledT3 restart recovery

That stale write is the race fixed by the open PR pingdotgg/t3code#25666 (git.cafe #25667), "preserve completed runs when accepting child deliveries". The queued delivery run only got stuck because the parent had been wrongly put back into waiting. Closing as a duplicate.

JJulius Marminge2d ago

Reopening, and correcting my duplicate close above. #25000 is a different bug: its run stayed running after Codex sent command item/completed events following turn/completed. This one is the stale-write race in delivery acceptance, so I've rewritten the description around it.

  • #25667 fixes the accept path that stalled this thread, and now closes this issue.
  • #25662 fixes the same race on the wake-policy path and references this issue.
  • #25672 (merged) covered acknowledge and dispose.