fix(server): PR watches report reruns that fail again and failed runs another GitHub event hides#32319

open5

main

fix/issue-32277

View on GitHubTwo-way sync enabledPR last imported 7h ago

Problem

A PR watch stays silent when a failed check is rerun and fails again between two sweeps (2 minutes apart), because the watch remembers failing checks by name only. On GitHub, a workflow that runs on both push and pull_request runs each check twice per head; the reader kept the newest run across both events, so a failed push run read as pending while the pull_request run ran, then as passed once it passed. The agent never heard about the failure and could be told "All checks passed". #17344 left this second case open on purpose until the reader had the run's event.

Reproduced on main 4cb6945dc6 by running the real evaluatePullRequestWatch and pullRequestChecksFromContexts on the issue's sequences (transcript below).

Change

Watch: the watch also stores the failed runs it told on the current head, as check name plus run URL, in a new optional failedRuns field. A failing check is news when its name was not failing or when its (name, URL) run has not been told on this head. The list resets on a new head and is restored when a watch is re-armed. A watch saved before the field existed compares by name only on its first read, so an upgrade causes no repeat wakes. A check with no URL falls back to its name.
GitHub reader: the check query also selects workflowRun.event. dedupeChecks keeps the newest run per workflow, check and event, as before, so a rerun still replaces its run. It then reports the worse run across events: a failure over a running run, a running run over a pass, and a pass over a run cancelled next to it. Between two failed runs it shows the one that finished last; when the finish times are equal, the event name breaks the tie. The check keeps required if either run had it. Hosts that report no event (Forgejo, Bitbucket, commit statuses) keep today's behaviour.

Scope and approval

Fixes pingdotgg/t3code#32277

Triaged via the automated issue flow.

Verification

Red on main, green here:
- apps/server/src/orchestration-v2/pullRequestWatch.test.ts › "reports a rerun that fails again between two reads". It also covers a status URL shared with another check, and a run told earlier on the head that shows up again.
- packages/source-control-github/src/server/gitHubPullRequestJson.test.ts › "keeps a failed push run failing while the pull_request run of it runs or passes": four rollup orders, plus required coming from the losing run.
- gitHubPullRequestJson.test.ts › "does not fail a check whose run of one event was cancelled beside a passing run".
- packages/source-control-core/src/server/pullRequestChecks.test.ts › "reports the same run for two events' failures finished at once, whichever is listed first".
Guard: pullRequestWatch.test.ts › "does not repeat a failure an older watch told by name".
Mutation-checked: matching on URL alone, dropping the required merge, and dropping the tie-break each fail a test.
vp test run on pullRequestWatch, runtimeLayer, pullRequestChecks, gitHubPullRequestJson and GitHubPullRequestApi: 338 passed. vp run check:changed passes: format, lint, knip, graphql codegen check, typecheck of 36 packages, 242 tests.
Server-only, no visible surface, so there are no captures. Runtime transcript of the real watch evaluator and GitHub reader, run on main's and this branch's sources with a synthetic driver:

text

1	== main 4cb6945dc6
2	A. PR watch, one head, sweeps 2 minutes apart
3	lint fails (job/1), rerun fails again (job/2) wakes per sweep: ["checks-failed"] -> []
4	control: pending seen between the two failures wakes per sweep: ["checks-failed"] -> [] -> ["checks-failed"]
5	B. GitHub reader, workflow on push and pull_request, one head
6	push failed + pull_request running checks: [["lint","pending"]] state: pending
7	push failed + pull_request passed checks: [["lint","success"]] state: passing
8	B through the watch: push fails, then pull_request passes wakes per sweep: [] -> ["checks-passed"]
9
10	== head 3c0110de88
11	A. PR watch, one head, sweeps 2 minutes apart
12	lint fails (job/1), rerun fails again (job/2) wakes per sweep: ["checks-failed"] -> ["checks-failed"]
13	control: pending seen between the two failures wakes per sweep: ["checks-failed"] -> [] -> ["checks-failed"]
14	B. GitHub reader, workflow on push and pull_request, one head
15	push failed + pull_request running checks: [["lint","failure"]] state: failing
16	push failed + pull_request passed checks: [["lint","failure"]] state: failing
17	B through the watch: push fails, then pull_request passes wakes per sweep: ["checks-failed"] -> []

Synthetic driver (not committed)

1	// Synthetic runtime driver for pingdotgg/t3code#32277: runs the real watch evaluator and GitHub
2	// check reader from a checkout. Usage (from <checkout>/apps/server): node issue-32277-driver.ts <checkout>
3	const wt = process.argv[2];
4	const { evaluatePullRequestWatch } = await import(`${wt}/apps/server/src/orchestration-v2/pullRequestWatch.ts`);
5	const { pullRequestChecksFromContexts } = await import(`${wt}/packages/source-control-github/src/server/gitHubPullRequestJson.ts`);
6	const sha = "abc123";
7	const watch0 = { startedAt: "2026-11-07T00:00:00Z", headSha: sha, failedChecks: [], passed: false, passedChecks: [], owedChecks: [], remarksThrough: "2026-11-07T00:00:00Z", remarkIds: [], conflicting: false, wakes: 0, checksAwaitedSince: null };
8	const detail = (checks: unknown[]) => ({ capabilities: {}, headSha: sha, checks, mergeability: "mergeable", viewer: "me", author: null });
9	const sweeps = (label: string, reads: unknown[][]) => {
10	let w: any = watch0; const out: string[] = [];
11	for (const r of reads) { const rep = evaluatePullRequestWatch(w, detail(r), [], 0); out.push(JSON.stringify(rep.changes.map((c: any) => c.kind))); w = rep.next; }
12	console.log(`${label.padEnd(52)} wakes per sweep: ${out.join(" -> ")}`);
13	};
14	const lint = (url: string, status: string) => ({ name: "lint", status, description: null, url, required: true });
15	console.log("A. PR watch, one head, sweeps 2 minutes apart");
16	sweeps("lint fails (job/1), rerun fails again (job/2)", [[lint("https://ci.example/job/1", "failure")], [lint("https://ci.example/job/2", "failure")]]);
17	sweeps("control: pending seen between the two failures", [[lint("https://ci.example/job/1", "failure")], [lint("https://ci.example/job/2", "pending")], [lint("https://ci.example/job/2", "failure")]]);
18	const unset = "0001-01-01T00:00:00Z";
19	const run = (event: string, status: string, conclusion: string \| null, url: string, completedAt: string) => ({
20	__typename: "CheckRun", name: "lint", status, conclusion, startedAt: "2026-11-07T00:00:00Z", completedAt, detailsUrl: url,
21	checkSuite: { workflowRun: { event, workflow: { name: "CI" } } }, workflowRun: { event }, workflowName: "CI",
22	});
23	const pushFailed = run("push", "COMPLETED", "FAILURE", "https://ci.example/push/1", "2026-11-07T00:01:00Z");
24	const prRunning = run("pull_request", "IN_PROGRESS", null, "https://ci.example/pr/1", unset);
25	const prPassed = run("pull_request", "COMPLETED", "SUCCESS", "https://ci.example/pr/1", "2026-11-07T00:02:00Z");
26	const read = (ctx: unknown[]) => pullRequestChecksFromContexts(ctx);
27	console.log("B. GitHub reader, workflow on push and pull_request, one head");
28	for (const [label, ctx] of [["push failed + pull_request running", [pushFailed, prRunning]], ["push failed + pull_request passed", [pushFailed, prPassed]]] as const) {
29	const r = read([...ctx]);
30	console.log(`${label.padEnd(52)} checks: ${JSON.stringify(r.checks.map((c: any) => [c.name, c.status]))} state: ${r.checksState}`);
31	}
32	sweeps("B through the watch: push fails, then pull_request passes", [read([pushFailed, prRunning]).checks as unknown[], read([pushFailed, prPassed]).checks as unknown[]]);

Not checked: a live GitHub repository with a dual-trigger workflow.

AI-assisted; verified with the tests and tools listed above.

Checks

24 successful · 9 skipped

All checks passed

Added 5 commits3d ago

fix(server): a watch tells failed runs apart by check name and URL3c0110d
fix(server): report check failures a rerun or a push run would hide3c3cd50
fix(server): a watch keeps the failed runs it told for the head9a03aa7
test: reproduce #32277 missed check failurese3b3b1b
fix(server): break cross-event check ties by event so a watch's run cannot flipec13a32

MmacroscopeappBotapproved these changes3d ago

MmacroscopeappBot3d ago

Approvability

Verdict: Approved at 3c0110d

Macroscope's review found this PR approvable — PR watches now recognize failed reruns and retain failures that overlapping GitHub events previously hid. The change is confined to check reporting and existing watch notifications, with no changes to permissions or product defaults.

^{You can add or adjust custom eligibility rules. Learn more.}

Ggithub-actionsBot3d ago

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Codex	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Codex	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Codex	Live turn WebSocket decoded	21.5 KiB	21.5 KiB	0 B (0.0%)	24.4 KiB	✅
Codex	Live turn messages	1	1	0 (0.0%)	8	✅
Claude	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Claude	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Claude	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Claude	Live turn WebSocket decoded	21.9 KiB	21.9 KiB	0 B (0.0%)	24.4 KiB	✅
Claude	Live turn messages	1	1	0 (0.0%)	8	✅

Baseline: 4cb6945 · PR result: 3c0110d · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

Codex decoded thread snapshot: 108.5 KiB
Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

CcoderabbitaiBot3d ago

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

GitHub check processing now retains workflow events and selects check results across events by status severity. Pull-request watches now track failed runs by check name and URL, including across restarted watches.

Changes: Pull-request check reporting

Layer / File(s) Summary

GitHub check event aggregation
packages/source-control-github/src/server/pullRequestReads.graphql, packages/source-control-github/src/server/pullRequestReads.graphql.ts, packages/source-control-github/src/server/gitHubPullRequestJson.ts, packages/source-control-core/src/server/pullRequestChecks.ts, packages/source-control-github/src/server/GitHubPullRequestApi.test.ts, packages/source-control-github/src/server/gitHubPullRequestJson.test.ts, packages/source-control-core/src/server/pullRequestChecks.test.ts GitHub check entries now include workflow events. Deduplication selects the newest run per event, then prefers higher-severity results across events and preserves the required flag. Tests cover event ordering, failed runs, and cancelled runs.

Pull-request watch failed-run history
packages/contracts/src/threadPullRequest.ts, apps/server/src/orchestration-v2/pullRequestWatch.ts, apps/server/src/orchestration-v2/Orchestrator.ts, apps/server/src/orchestration-v2/PullRequestWatchReactor.ts, apps/server/src/orchestration-v2/pullRequestWatch.test.ts Watch state now records failed check name-and-URL identities and retains them for the current head. A new watch carries forward the history when defined. Tests cover new runs, previously reported runs, and older watch state.

Layer / File(s)	Summary
GitHub check event aggregation `packages/source-control-github/src/server/pullRequestReads.graphql`, `packages/source-control-github/src/server/pullRequestReads.graphql.ts`, `packages/source-control-github/src/server/gitHubPullRequestJson.ts`, `packages/source-control-core/src/server/pullRequestChecks.ts`, `packages/source-control-github/src/server/GitHubPullRequestApi.test.ts`, `packages/source-control-github/src/server/gitHubPullRequestJson.test.ts`, `packages/source-control-core/src/server/pullRequestChecks.test.ts`	GitHub check entries now include workflow events. Deduplication selects the newest run per event, then prefers higher-severity results across events and preserves the required flag. Tests cover event ordering, failed runs, and cancelled runs.
Pull-request watch failed-run history `packages/contracts/src/threadPullRequest.ts`, `apps/server/src/orchestration-v2/pullRequestWatch.ts`, `apps/server/src/orchestration-v2/Orchestrator.ts`, `apps/server/src/orchestration-v2/PullRequestWatchReactor.ts`, `apps/server/src/orchestration-v2/pullRequestWatch.test.ts`	Watch state now records failed check name-and-URL identities and retains them for the current head. A new watch carries forward the history when defined. Tests cover new runs, previously reported runs, and older watch state.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: fixfon, saphid, bil0000

Merge Risk: External image Minimal · up to 3c011

No identified issue blocks merging after normal checks.

Pre-merge checks: ^Passed 4

Check name	Status	Explanation
Title check	Passed	The title is a conventional commit that names both halves of the change: PR watches now report reruns that fail again, and failed runs hidden by another GitHub event are now reported. Both match the r…
Description check	Passed	The description follows the repository template. It has Problem, Change, Scope and approval, and Verification sections. The Problem section gives the cause, reproduction steps, and the environment (ma…
Linked Issues check	Passed	Linked issue `#32277` is the only direct, active target. Its coding requirements are: 1. Watch wakes on a rerun that fails again under a new run URL [`#32277`]. In apps/server/src/orchestration-v2/pullRe…
Out of Scope Changes check	Passed	All changed areas trace to `#32277` or to required support for it: - pullRequestWatch.ts: run-URL tracking and the `failedRuns` field, which implement the watch requirement. - pullRequestChecks.ts: per…

Full details: Title check

Explanation

The title is a conventional commit that names both halves of the change: PR watches now report reruns that fail again, and failed runs hidden by another GitHub event are now reported. Both match the raw summary, the PR objectives, and the linked issue. The title is long at 96 characters, but it stays on the main change.

Full details: Description check

Explanation

The description follows the repository template. It has Problem, Change, Scope and approval, and Verification sections. The Problem section gives the cause, reproduction steps, and the environment (main commit 4cb6945dc6). The Change section explains the watch and GitHub reader fixes and the compatibility behavior. Verification lists focused tests, mutation checks, and a runtime transcript, and it states what was not checked (a live dual-trigger repository). The Scope section links the issue (pingdotgg/t3code#32277) and says it was triaged through the automated flow, but it does not include the maintainer approval comment the template asks for. The template also asks for the model and harness to be named when an agent was used; the description says only 'AI-assisted' and does not name the model or harness. These are gaps, but the description is largely complete.

Full details: Linked Issues check

Explanation

Linked issue #32277 is the only direct, active target. Its coding requirements are:

Watch wakes on a rerun that fails again under a new run URL [#32277]. In apps/server/src/orchestration-v2/pullRequestWatch.ts, newlyFailed also selects a failing check whose (name, URL) pair is absent from the new failedRuns list. The list is kept per head and resets on a head move. A check with no URL uses name-only comparison, and a watch without failedRuns compares by name on its first read. These match the issue's proposed fix.
GitHub reader keeps the worse run across push and pull_request events [#32277]. In packages/source-control-github/src/server/gitHubPullRequestJson.ts, workflowRun.event is read and passed to dedupeChecks. In packages/source-control-core/src/server/pullRequestChecks.ts, the newest run per workflow, check and event is kept first. Across events the code then applies eventSeverity: failure over pending, pending over success, success over cancelled or other. Dedupe preserves the required flag from either run.
Between two failed runs, the run that finished last is shown, with event name as tie-break [#32277]. The later/tied logic in dedupeChecks implements this and makes the result independent of list order.
Hosts without an event keep existing behaviour. The event is null for those entries, so the per-event key is unchanged.

The linked issue's cited tests cover these cases. The PR also lists them as passing on this head. I did not run the tests. I confirmed the implementation paths in the diffs for the three main source files. The generated GraphQL file and the failedRuns contract field are supporting changes for these requirements.

Full details: Out of Scope Changes check

Explanation

All changed areas trace to #32277 or to required support for it:

pullRequestWatch.ts: run-URL tracking and the failedRuns field, which implement the watch requirement.
pullRequestChecks.ts: per-event dedupe and cross-event severity merge, which implement the GitHub requirement.
gitHubPullRequestJson.ts, pullRequestReads.graphql, and pullRequestReads.graphql.ts: select and carry workflowRun.event. The generated GraphQL file follows the graphql source change, so these are required for the event data.
packages/contracts/src/threadPullRequest.ts: adds the optional failedRuns field to the persisted watch state.
Orchestrator.ts and PullRequestWatchReactor.ts: carry failedRuns across re-arm and compare it in watchesEqual. These support the watch state requirement.
Test files (pullRequestWatch.test.ts, pullRequestChecks.test.ts, gitHubPullRequestJson.test.ts, GitHubPullRequestApi.test.ts): cover the new behaviour. The GitHubPullRequestApi.test.ts fixtures add workflowRun.event to existing check data, which the new reader requires.

I found no change unrelated to #32277. The PR description's validation and docs-only items are not code changes.

Finishing Touches

Generate docstrings

Commit to this branch
Create a new PR

Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

External image Recent review info

External image Run configuration

Source mode: Auto
Repository: pingdotgg/t3code
Branch: fix/issue-32277
File: .coderabbit.config.ts
Commit: 3c0110d
Layers: Repository TS config
Review profile: CHILL
Plan: Team
Run ID: `2c00bb7f-caa2-4cc5-8991-25d38b3a8789`

External image Commits

Reviewing files that changed from the base of the PR and between 4cb6945dc60ed03f567dc5daafd083e52308e82b and 3c0110de88fc63f5c5ac57631900a4da579d0be8.

External image Files selected for processing: 12

`apps/server/src/orchestration-v2/Orchestrator.ts`
`apps/server/src/orchestration-v2/PullRequestWatchReactor.ts`
`apps/server/src/orchestration-v2/pullRequestWatch.test.ts`
`apps/server/src/orchestration-v2/pullRequestWatch.ts`
`packages/contracts/src/threadPullRequest.ts`
`packages/source-control-core/src/server/pullRequestChecks.test.ts`
`packages/source-control-core/src/server/pullRequestChecks.ts`
`packages/source-control-github/src/server/GitHubPullRequestApi.test.ts`
`packages/source-control-github/src/server/gitHubPullRequestJson.test.ts`
`packages/source-control-github/src/server/gitHubPullRequestJson.ts`
`packages/source-control-github/src/server/pullRequestReads.graphql`
`packages/source-control-github/src/server/pullRequestReads.graphql.ts`

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

Rroyalaid8h ago

GitLab follow-up for this change: #33671 targets this branch. GitLab's single Pipeline check keeps its URL when failed jobs are retried in place, so the name-plus-URL run key here misses a retry that fails again. #33671 adds an optional runId to checks (GitLab: pipeline ID plus finished_at) and keys failed runs on it, falling back to the URL. Details in https://github.com/pingdotgg/t3code/issues/32277#issuecomment-6757020424.