[BUG] V2 waiting run keeps showing Agent is working and offers Stop for finished work when checkpoint capture stalls#15124

Evidence addendum (logic repro, no provider CLI needed):

phaseFor(waiting) returns running per packages/shared/src/agentAwareness.ts:99-105, and derivePendingBackgroundWork treats waiting as settled per packages/shared/src/orchestrationV2PendingBackgroundWork.ts:37-43,109-116. Successful turns are persisted as waiting at apps/server/src/orchestration-v2/RunExecutionService.ts:633-634 until CheckpointCaptureService.ts:90-104 flips them. Repro script output:

Visual diagram prepared locally at /tmp/opencode/shot.svg and /tmp/opencode/v2-waiting-evidence.html for the reporter to screenshot and drag into this issue (no desktop browser connected in this session, so no PNG captured here).

Triage

Thanks for the careful source walk-through, @NK-Works! What the logic repro shows is the designed post-success state, not a misclassification.

How waiting is supposed to behave

A successful provider turn is saved as waiting until checkpoint capture commits completed (persistedStatus in RunExecutionService). While a run is in that state:

• Awareness and the composer treat waiting as still working, and Stop stays available. projectThreadAwarenessV2 maps waiting to the running phase ("Agent is working"), and derivePhase and deriveCanInterruptRunningThread do the same. apps/web/src/session-logic.test.ts checks this.
• waiting is in SETTLED_FOR_BACKGROUND_WAIT_RUN_STATUSES on purpose, so background tasks that are still running can show up before the checkpoint lands. derivePendingBackgroundWork only includes items that are still active (pending, running, or waiting), not finished work. The test "returns pending work when the latest run is waiting (post-success, pre-checkpoint)" covers this.
• If that list includes work that will wake the agent, like a subagent or a monitor, the thread is shown as idle rather than stuck on the checkpoint waiting state. The background stop strip appears, and the thread doesn't read as "Agent is working" (presentThreadShell, see the test "parks runtime idle over stale checkpoint waiting when the roster is nonempty").

So the tests you ran pass because they lock this behavior in. A short "working" window while capture is in flight is expected. active_run_id also excludes waiting, so this isn't an interruptible provider turn.

What would be a real bug

This report doesn't show a run that stays stuck after capture gives up. checkpoint.capture retries, and after the worker's maximum attempts (5 by default) the outbox row fails without moving the run off waiting.

Because waiting still blocks the thread, it would keep looking busy, and queued turns wouldn't start. Stop would target that run, but dispatchRunInterrupt rejects it as "not interruptible" when there's no running provider turn and no active background work. That failure to finish the run is a separate problem from showing the short in-flight state as "working".

If you've seen a thread stay on waiting after capture stopped retrying, please comment with:

• The run status
• The checkpoint.capture effect row (status, attempt count, and error)
• Roughly how long it stayed that way

A maintainer will decide on the fix direction once that's clear.

Thanks for the detailed triage — verified your refs against main@fed41fa88 and you're right that the short in-flight window is by design. Withdrawing the broad "waiting should not show working" claim:

• apps/web/src/session-logic.ts:1013 maps waiting to running, locked by session-logic.test.ts:103-106.
• packages/client-runtime/src/state/models.ts:168-171 parks runtime at idle when the background roster is nonempty, locked by entities.test.ts:302 ("parks runtime idle over stale checkpoint waiting"). So with background work present the thread does not read as "Agent is working" — my report overstated that case.
• EffectWorker.ts:508 defaults maxAttempts to 5 for the checkpoint.capture outbox row.

Narrowing to the stuck case you describe, which I have not reproduced live in this session: a run remaining in waiting after checkpoint.capture exhausts retries, where the outbox row fails without moving the run off waiting, the thread keeps looking busy, queued turns do not start, and Stop targets the run but dispatchRunInterrupt rejects it as not interruptible (Orchestrator.ts:7779 path).

I don't have a stuck run here to pull the three items from (run status, checkpoint.capture effect row status/attempts/error, duration). If it helps, I can gather them from a reporter machine with read-only queries against statev2.sqlite (runs + effect outbox rows for checkpoint.capture:<runId>) plus the surrounding server.trace.ndjson — or, if you prefer, I can re-title this issue to that stuck-after-retries shape and drop the in-flight presentation part. Let me know which evidence format you want and I'll supply it.

Evidence gathered while you review (code-construction + local snapshot, no live stuck run here):

1. Stuck mechanism exists by construction (verified on main@fed41fa88)

• Failure path emits no run transition: CheckpointCaptureService.ts:90-104 returns CheckpointCaptureExecutionError when the capture target is incomplete (missing run/rootNode/scope, scope mismatch, no provider thread). Only the success path (:136-174) commits run.updated (waiting -> completed) plus checkpoint.captured. The existing test exercises this: CheckpointCaptureService.test.ts:316-321 executes with a missing scope and asserts the error is returned before attempting the real capture.
• Retry exhaustion also leaves the run untouched: EffectWorker.ts:688-691 calls outbox.fail once attemptCount >= maxAttempts (default 5, :508) without any run-status write. So a persistently failing checkpoint.capture:<runId> effect ends as failed in the outbox while the run row stays waiting — the exact stuck shape you describe (busy-looking thread, queued turns blocked, Stop targeting a non-interruptible run per Orchestrator.ts:7779).

2. Local snapshot has no stuck rows (queries included for reporters)

Read-only copy of this machine's statev2.sqlite (copied to /tmp, never opened read-write):

Queries a reporter can rerun against their statev2.sqlite (read-only open recommended):

Plus the surrounding server.trace.ndjson window for the stuck runId (the checkpoint.capture attempts and the final fail).

3. What I still lack

A live thread actually stuck in waiting with its failed checkpoint.capture row (effect status, attempt count, last_error) and how long it stayed that way. If you can point me at a repro that forces capture to fail persistently (e.g. a scope-mismatch fixture or a failing CheckpointStore.capture), I'll drive it here and attach the run + outbox rows. Otherwise, happy for this issue to be re-titled to the stuck-after-retries shape.

Deterministic repro now in hand (temporary test, since removed — not committed):

Temp test: failed capture leaves the run in waiting with zero events committed

Setup mirrors CheckpointCaptureService.test.ts: thread + run in waiting applied to an in-memory ProjectionStore, then CheckpointCaptureService.execute against a missing scope (incomplete target, same class of failure as a persistently failing capture):

• execute returns CheckpointCaptureExecutionError (Failure exit).
• The mocked EventSink.commitCommand is never called (0 commits — no run.updated, no checkpoint.captured).
• getThreadProjection still shows the run as waiting.

Result under Node 24 (repo tests need node:sqlite):

Combined with EffectWorker.ts:688-691 (outbox.fail after maxAttempts, default 5 at :508, with no run write), a checkpoint.capture effect that keeps failing ends as failed in orchestration_v2_effect_outbox while the run row stays waiting indefinitely — thread looks busy, queued turns stay blocked, Stop targets a run dispatchRunInterrupt rejects (Orchestrator.ts:7779).

Live snapshot (this machine, read-only copy): no stuck rows currently — runs: 3 cancelled / 35 completed / 5 failed / 2 running / 0 waiting; checkpoint.capture effects: 35 succeeded, 0 failed. Reporter queries from my previous comment still apply for any machine that does hit it (run status + effect row status/attempts/last_error + duration).

Live triage evidence from T3 Code 0.0.46-nightly.20261011.2967, commit 8d199b4f7b91, on macOS arm64 (25.6.0), Node 26.8.2.

A Codex provider turn finished on October 10, but its run remained waiting until recovery on October 11—approximately 25 hours. Immediately before recovery, the checkpoint.capture outbox row was running, attempt count 4, last error NULL, last updated 2026-10-11T13:25:53.503Z. Repeated Stop attempts returned “Run … is not interruptible.” Five queued messages remained held.

This differs from the exhausted-retries case discussed above: the effect was still running, not failed after five attempts.

Two other threads in the same checkout, one Claude and one Codex, also stalled during provider-turn.start, without obtaining a provider turn ID. Source at the installed commit shows baseline capture precedes provider start, and baseline/checkpoint capture share a per-workspace lock. This makes a blocked capture holding up the other launches a plausible explanation; it is not a proven complete causal trace.

Local Git status timed out while scanning four untracked, cloud-only iCloud duplicate directories. Excluding just those directories in .git/info/exclude reduced git status --porcelain=v1 --untracked-files=all to 0.11 seconds. No tracked files or working-tree files were deleted.

After backing up state and gracefully restarting T3, checkpoint capture succeeded on attempt five and the waiting run completed. Normal restart recovery cancelled the two stalled launches; no pending/running effects remain in these three threads. Queued work remains held for manual resume.

Recovery did not directly edit the database or patch T3 source. This is evidence for handling stalled checkpoint work and making Stop/recovery actionable, rather than changing the normal short-lived waiting presentation.

Investigated by Codex (GPT-6.1 Sol, medium reasoning) via t3 triage.