A webhook delivery the server answered with HTTP 202 was lost if the server restarted (crash, desktop update, npx t3 restart) before its run started. Immediate (non-coalesced) deliveries were only queued in memory behind the task's permit. After a restart, the delivery log row stayed accepted forever, the prompt never ran, and a relay replay came back duplicate.
Repro on 8d3b061121 with the dev server: one webhook task without dedupe settings, two concurrent POST /api/hooks/<task>/<token> (both 202), SIGKILL the server, restart. 75 s later both deliveries were still accepted, scheduled_task_webhook_pending was empty, and nothing had dispatched.
acceptImmediateWebhookDedupe now writes the delivery to scheduled_task_webhook_pending (state dispatching, due now, empty coalesce_key). The write happens in the same transaction that logs accepted, using the table coalesced deliveries already use.runWebhookDelivery. The coalescing scheduler worker now shares that drain as drainPendingWebhooks. After a restart, the existing webhook-coalescing worker picks up the row and re-runs it with the same deterministic command id, so a re-run is idempotent.webhookQueued counter is removed.delivery_id. releaseFailedDelivery deletes its pending row in the same transaction as the claim release, and the drain's cleanup deletes only completed rows. Together these keep a redelivery from colliding with, or being deleted by, the old row's cleanup.coalesce_key <> '').Fixes #18810 GitCafe: https://git.cafe/pingdotgg/t3code/issues/18810 Triaged bug via the automated issue flow. Reproduced on Linux by the maintainer in this comment and again by the triage run above.
vp test run apps/server/src/scheduledTasks: 8 files, 82 tests passed.an accepted delivery queued behind a run still launches after a restart failed on main with the accepted delivery was never launched after the restart and passes here.a replaced task's old drain leaves the new incarnation's delivery alone fails without the permit check (expected 'dispatch_failed' to equal 'accepted') and passes with it.an immediate delivery queued behind a run is not shown as an open window covers the open-window flag.dispatching.webhook-coalescing worker drained both and attempted the launch.dispatch_failed: The run failed to start. (ThreadLaunchError … failed during resolve-project) instead of staying accepted.triggerWebhook admission, and the unit tests cover relay claim release and redelivery.Server-only change; no UI surface.
Model and harness: Claude Opus 5.5 (with Claude Sonnet 5.5 as second reviewer) in Claude Code via T3 Code.
🤖 Generated with Claude Code
Real server (node apps/server/src/bin.ts --base-dir <fresh tmp dir>), a valid scratch project, and real claudeAgent turns. Each row: SIGKILL the server while deliveries are queued, then restart it. Before = main 8d3b061121, after = head e3c21846c2.
| Row | Before (8d3b061121) | After (e3c21846c2) |
|---|---|---|
| R1: immediate delivery queued, SIGKILL, restart | Both POSTs 202, pending table empty at kill. 120 s after the restart, still accepted with 0 threads: lost (log) | Both rows dispatching at kill. After the restart both launched and completed, 1 thread each (log) |
| R2: two concurrent immediate deliveries behind an in-flight one | The in-flight run was cancelled by restart recovery, and r2-a/b never launched (log) | Rows at kill: running, dispatching, dispatching. All 3 completed with 1 thread each (project total 3). Re-running the running row reused its thread (log) |
| R3: coalesced delivery (unchanged path) | The pending row survived and produced 1 completed run (log) | Same as before (log) |
R4: replay of the same body.id after restart (in R1/R2) | duplicate, but the prompt had run 0 times | duplicate, and the prompt ran exactly once |
| R5: task replaced between accept and restart | Unit-test-only | Unit-test-only: a replaced task's old drain leaves the new incarnation's delivery alone |
cd750c4e583fd9eb75591f776dc7f310f7e61dd21d7c0a3ee910f895da9eb2ef. Helpers: burst.mjs, mint-mcp-token.mjs, gitwrap-git.sh. Summary: SUMMARY.md.claudeAgent / claude-haiku-5-5 (Claude Code 2.1.303); the prompt forbids tools and edits. Isolated --base-dir per side and row, ports 47912–47917. bin.ts has no --home-dir or --no-seed and never seeds from ~/.t3.running row must not create a second thread. The after-R2 run shows it does not.burst.mjs posts the deliveries concurrently and SIGKILLs the server on the last 202. A running provider turn does not hold the per-task lock on either side; only the launch does, so the "in-flight run" is a launch in progress. The kill-time dumps show deliveries still queued in every capture.{"deliveryId":…}. The duplicate outcome is visible only in the delivery log, which the transcripts dump.threads_with_marker column reads 0 everywhere because of a broken text match. It is not evidence; the per-delivery thread counts and project totals are.Added 3 commits
Verdict: Needs Human Approval
Macroscope's review found this PR requires human approval — This PR changes the production webhook execution path by persisting immediate deliveries and routing them through a new shared drain for ordering and restart recovery. The associated state and concurrency changes span acceptance, capacity, completion, and task replacement behavior, warranting human review.
You can add or adjust custom eligibility rules. Learn more.
✅ Thread transfer remains within every enforced ceiling.
| Provider | Metric | Main baseline | This PR | Impact | PR ceiling | |
|---|---|---|---|---|---|---|
| Codex | Total thread wire | 4.9 KiB | 5.0 KiB | +30 B (+0.6%) | 6.8 KiB | ✅ |
| Codex | Thread snapshot wire | 3.8 KiB | 3.8 KiB | 0 B (0.0%) | 4.9 KiB | ✅ |
| Codex | Live turn WebSocket wire | 1.2 KiB | 1.2 KiB | +30 B (+2.5%) | 2.0 KiB | ✅ |
| Codex | Live turn WebSocket decoded | 20.8 KiB | 20.9 KiB | +41 B (+0.2%) | 29.3 KiB | ✅ |
| Codex | Live turn messages | 1 | 2 | +1 (+100.0%) | 8 | ✅ |
| Claude | Total thread wire | 5.0 KiB | 5.0 KiB | 0 B (0.0%) | 6.8 KiB | ✅ |
| Claude | Thread snapshot wire | 3.8 KiB | 3.8 KiB | 0 B (0.0%) | 4.9 KiB | ✅ |
| Claude | Live turn WebSocket wire | 1.2 KiB | 1.2 KiB | 0 B (0.0%) | 2.0 KiB | ✅ |
| Claude | Live turn WebSocket decoded | 21.2 KiB | 21.2 KiB | 0 B (0.0%) | 29.3 KiB | ✅ |
| Claude | Live turn messages | 2 | 2 | 0 (0.0%) | 8 | ✅ |
Baseline: 8d3b061 · PR result: e3c2184 · Source CI: success
10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.
No actionable comments were generated in the recent review. 🎉
916b96fc-efd8-4edb-8bfa-591412de36e9Reviewing files that changed from the base of the PR and between 8d3b061121fe65669d5d43d011b48cd3027a4108 and e3c21846c23fc6f412914a3590c73b94e3004e5e.
apps/server/src/scheduledTasks/ScheduledTaskService.tsapps/server/src/scheduledTasks/ScheduledTaskService.webhook.test.tsapps/server/src/scheduledTasks/webhookDedupe.tsIncluded review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.
Webhook dispatch now uses persisted pending rows instead of an in-memory queue counter. Immediate and periodic dispatch share a drain, and tests cover restart recovery, task recreation races, and completion bookkeeping.
Changes: Webhook dispatch
| Layer / File(s) | Summary |
|---|---|
Persisted admission and queue state apps/server/src/scheduledTasks/webhookDedupe.ts, apps/server/src/scheduledTasks/ScheduledTaskService.ts | Immediate dedupe encodes and stores a pending row. Capacity checks count persisted rows in pending, dispatching, or running states. Pending-window queries require a nonempty coalesce key. |
Shared pending-row drain apps/server/src/scheduledTasks/ScheduledTaskService.ts | Immediate dispatch and the periodic worker use a shared drain under the task permit. The drain removes completed rows and marks and releases failed deliveries. |
Recovery and dispatch validation apps/server/src/scheduledTasks/ScheduledTaskService.webhook.test.ts | Tests cover restart recovery, pending-window state, task recreation during a drain, retries, and completion bookkeeping without duplicate launches. |
Priority: ➖ Normal
Estimated code review effort: 4 (Complex) | ~45 minutes
Change: Bug fix · Severity of issue fixed: Medium
| 1 | sequenceDiagram |
| 2 | participant ScheduledTaskService |
| 3 | participant PendingWebhookRows |
| 4 | participant TaskRunner |
| 5 | ScheduledTaskService->>PendingWebhookRows: Select persisted rows in order |
| 6 | PendingWebhookRows-->>ScheduledTaskService: Return pending delivery rows |
| 7 | ScheduledTaskService->>TaskRunner: Dispatch a valid prompt |
| 8 | TaskRunner-->>ScheduledTaskService: Return run result |
| 9 | ScheduledTaskService->>PendingWebhookRows: Remove completed row or mark and release failed row |
Suggested reviewers: t3dotgg
e3c21Accepted deliveries have a persisted recovery path after restart. No issue identified here needs resolution before merging.
| Check name | Status | Explanation |
|---|---|---|
| Title check | Passed | The title clearly and concisely describes the main fix: accepted webhook deliveries now survive a server restart. |
| Linked Issues check | Passed | The PR addresses the coding requirements in #18810. acceptImmediateWebhookDedupe persists the task and request payload in scheduled_task_webhook_pending before the accepted response transaction co… |
| Out of Scope Changes check | Passed | The changes stay within #18810. They modify scheduled webhook persistence, draining, capacity accounting, relay redelivery cleanup, task-incarnation handling, and related automated tests. The shared d… |
| Description check | Passed | The description includes complete Problem, Change, Scope and approval, and Verification sections. It provides reproduction steps, linked issue and approval context, focused test results, runtime evide… |
Explanation
The PR addresses the coding requirements in #18810. acceptImmediateWebhookDedupe persists the task and request payload in scheduled_task_webhook_pending before the accepted response transaction completes. drainPendingWebhooks processes persisted rows for immediate and coalesced deliveries. The startup worker can resume rows after restart. The drain uses the delivery-derived command ID, and completion and failure paths update or remove durable state. Relay claim release deletes the old pending row before redelivery. The empty coalesce key excludes immediate deliveries from open windows. Tests cover restart recovery, relay retry behavior, queue-window reporting, and replacement-task isolation.
Explanation
The changes stay within #18810. They modify scheduled webhook persistence, draining, capacity accounting, relay redelivery cleanup, task-incarnation handling, and related automated tests. The shared drain supports the linked issue's direct, tunnel, and relay admission paths. No unrelated product or UI changes are shown.
Explanation
The description includes complete Problem, Change, Scope and approval, and Verification sections. It provides reproduction steps, linked issue and approval context, focused test results, runtime evidence, limitations, and the required model and harness information.
Comment @coderabbitai help to get the list of available commands.