Queued PR-watch wakeups survive unwatch, merge, and a newer head#21009

opened25d agobyPporoburu

PR watch notifications queued during an active turn can outlive unwatch, merge, and a newer head

Observed with T3 Code server 0.1.0 on Ubuntu 24.04 using the Codex provider. After an active agent turn unwatched and merged a PR, previously queued check notifications still started additional turns. A failure for an older head also arrived after its replacement head had passed. The completed work and the original failed run remain distinct; the extra turn is the defect.

Reproduction:

Start a sufficiently long agent turn and watch an open PR.
Let a check finish while that turn remains active, so the watch update queues behind it.
Before the active turn finishes, unwatch and merge the PR. Alternatively push a new head and let it pass before the older notification is delivered.
Finish the active turn. The stale check notification can still start another provider turn.

Expected: unwatch, merge, or a newer observed head invalidates an undelivered check wake belonging to the old watch/head. Preserve the historical notification and original check result, but do not spend another provider turn on it. An unrelated user message and a genuinely fresh comment must remain deliverable.

Source trace at 13aa882c22d584de5388dfce3b3ca413387ba059:

PullRequestWatchReactor.ts dispatches a watch sync with its generation.
Orchestrator.ts correctly rejects a host read that arrives after unwatch. An accepted update becomes an ordinary message.dispatch with queue_after_active.
The persisted notification source is only { kind: "monitor" }. It does not retain the PR key, watch generation, or observed head for queue invalidation.
startNextQueuedRun starts that queued message without validating its watch origin. This is a different race from the existing test that unwatches before the sync command lands.

Suggested scope: keep the existing watcher and event queue. Carry typed PR/watch/head identity into its queued notifications; cancel or supersede obsolete undelivered wakes through the existing queued-run cancellation owner. A second watcher or agent-side polling loop would duplicate authority. An additional small race guard should reject an in-flight open-PR wake when the current linked snapshot is already merged or closed; that guard alone does not fix notifications accepted earlier.

Focused regression cases should enqueue behind a real active run, then unwatch, merge, or advance the observed head; drain the existing effect worker and assert no extra provider turn. Preserve independent PRs, a new watch generation, user messages, current comments, and the intentional one-time closed notification. Also exercise stop/restart while a host read is in flight. No sleeps or repo-wide test run are needed.

I searched existing issues for unwatch, stale watch, and queued watch notifications. #19183 concerns batching multiple watched PRs and appears adjacent rather than the same cancellation defect. No installed bundle, live database, credentials, or runtime settings were changed to investigate this report. The queue-origin proposal and the small race-guard patch have not been executed against an upstream development checkout.

JJulius Marminge25d ago

Note

Grok responding on behalf of Julius.

Confirmed on main. dispatchPullRequestWatchSync checks that the watch is still on only when the host read lands. After that, the wake becomes an ordinary queue_after_active message whose notification source is just { kind: "monitor" }, with no PR, watch, or head identity. startNextQueuedRun doesn't recheck the watch, and unwatch or merge doesn't cancel queued wakes, so a stale wake can still start a turn. This isn't covered by #19183, which is about batching. Keeping this open as a bug.

Ssmagnetize11d ago

Another observation of this on a newer build, with the Claude provider, plus how often it happens.

Build: T3 Code 0.1.6-nightly.20261030.3904, Windows 11 x64 desktop, Claude Agent SDK (claude-opus-5-5), full-access runtime.

One thread on 2026-10-31, times in UTC. The thread was in one long active turn from before 06:38 until about 08:49.

Time	Event
06:38:26	`watch_pull_request` on PR A
06:54:47	PR A merges
06:54:53	`unwatch_pull_request` on PR A
08:41:41	`watch_pull_request` on PR B
08:46:07	PR B merges
08:46:11	`unwatch_pull_request` on PR B
08:49:20	Update on PR A starts a turn: checks passed, plus the Codex review-summary comment
08:49:29	The agent unwatches PR A a second time
08:49:41	Another update on PR A starts a turn
08:49:51	Update on PR B starts a turn: checks passed
08:50:00	Another update on PR B starts a turn: checks and two comments

This matches the queued queue_after_active path described above: the PR A updates waited almost two hours behind the active turn and were all released after both unwatches.

Each released update still says "This PR stays linked and watched, so do not link, watch, or list it again", which is no longer true at that point.

Frequency. Over 21 days of one machine's Claude transcripts for one repository:

377 of 1,433 PR-watch updates arrived after their PR had merged.
328 of those 377 arrived after the thread's own unwatch_pull_request, a median of 3.7 minutes after the merge.
Each one started a full provider turn of roughly 400k context tokens that could only say the PR had already merged.

Workaround meanwhile. A repository UserPromptSubmit hook drops the prompt when it is only a PR-watch update, the thread's last watch call for that PR was an unwatch made after GitHub's merge time, and the update reports only passing checks or a conflict. It removes about two thirds of these late turns. It cannot safely judge updates that carry comments, which is why invalidating the queued wake inside T3 is the real fix.

Ddannydtk8d ago

Another observation on T3 Code 0.1.6 (macOS arm64 desktop app, Cursor provider, claude-opus-5-5). It adds the queue-growth case, wakes from an earlier watch on the same PR, and edited comments re-sent as new ones.

Setup: one long-lived thread watching 5 PRs in one repo (a 3-PR stack, A–C, plus 2 standalone, D–E), all landing through a GitHub merge queue.

Time (UTC)	Event
16:35–18:05	69 PR-watch wakes queued across the 5 PRs, about one per PR per 2-minute sweep, while each turn ran 1–5 min
18:07	Watches on A–C end (trace `pull request watch ended`, reason `stopped`) and are later started again
18:59	A, D, E merge (reason `merged`); `unwatch_pull_request` returns `wasWatching: false`
19:29	B, C merge (reason `merged`)
18:59–19:44	24 wakes requested 16:59–17:22 start provider turns
19:47	A user message waits behind them until 19:54
19:52–19:53	41 wakes still queued for the merged PRs; the agent cancels them by hand

What the database shows (orchestration_v2_projection_runs joined to message:pr-watch:* messages):

84 wakes total: 43 started a turn and 41 were cancelled by hand. The longest gap from requestedAt to startedAt was 2h20m (17:22:29 → 19:42:48).
12 wakes started after their PR's watch had ended on merge.
22 wakes from the watch that ended at 18:07 were delivered after it ended, even though the PRs were watched again with new startedAt values. Once queued, a wake isn't tied to a watch generation.
Older heads arrive after newer ones: PR C got a turn at 19:27 about checks on a head replaced over an hour earlier, while wakes for two newer heads were already queued.
The wake text (from #24781) still says "This PR stays linked and watched" after merge and unwatch, because it is written at queue time.

Edited comments re-sent as new comments (possibly its own issue): a review bot that edits one status comment ("reviewing" → "finished") produced 12 separate wakes, each listing that same issuecomment-… link under "N new comments". pullRequestWatch.ts uses editedAt ?? createdAt as the comment's time, so each edit moves it past remarksThrough again. Expected: an edit to an already-reported comment isn't reported as a new comment, or is labelled as an edit and coalesced.

Still on main: at main today, nextQueuedRun → queuedRunsInDeliveryOrder (unchanged since v0.1.6) starts the next queued run without checking whether a monitor wake's watch is still on. A sweep's wake also never replaces an earlier queued wake for the same PR. So whenever wakes are added faster than turns finish, which matches #26987, every stale wake eventually runs.

Workaround: cancel queued "Update on pull request" runs by hand after unwatching.

Produced by Claude Code (claude-opus-5-5) via t3 triage.

Ggenesiscz6d ago

Another observation on T3 Code 0.1.7 (macOS arm64 desktop, Claude provider), at a larger scale, with a source check on current main (337c5858c).

Case. One orchestrator thread watched about 25 PRs from 06:04 UTC and was busy for most of the day. Every check or comment change on a watched PR queued its own run. The agent then called unwatch_pull_request for every PR (54 calls, 12:58 to 15:58 UTC). Afterwards list_thread_pull_requests showed watching: false for all of them, and every link carried endedWatch and no watch.

At 16:00 UTC, 115 PR-watch runs were still queued on that thread (requested 11:31 to 13:22 UTC). Each one would have started its own turn, and each still said "which T3 Code is watching for you" and "This PR stays linked and watched". The agent removed them with 115 separate t3_queue cancel calls (16:00:40 to 16:05:04 UTC).

Over the day the thread had 466 PR-watch runs (message:pr-watch:%): 350 completed, so 350 full-context turns were woken by PR notices; 115 were cancelled by hand; 1 is the running turn.

Still on main:

Unwatch only clears watch on the link: handlers.ts:524 dispatches thread.pull-request.watch with watching: false (handlers.ts:281-291), a plain thread mutation whose reducer sets watch to undefined (Orchestrator.ts:3933). No queued-run.cancel follows. A watch that ends by itself goes through dispatchPullRequestWatchSync, which also only records the mutation (Orchestrator.ts:3024-3040).
The queued notice cannot be matched to its PR or watch: its source is only { kind: "monitor" } (pullRequestWatch.ts:252, orchestrationV2.ts:1267).
Delivery does not check the watch: nextQueuedRun takes the first queued run (Orchestrator.ts:823-827), and startNextQueuedRun checks archive, delete, active runs, usage limits, runtime mode and worktree state, but not the watch (Orchestrator.ts:1649-1700). The "watching for you" text is fixed when the notice is queued (pullRequestWatch.ts:237).

Both open PRs would have prevented this case: #30894 cancels at watch teardown, #21239 revalidates at delivery. The per-PR pile-up and the one-call-per-run cleanup are the #26987 side; I added the numbers for those there.

Written by Claude Opus 5.5 in T3 Code for the reporter, from read-only queries on the local database.