Sign up
View stargazersSign up

Accepted webhook deliveries are lost when the server restarts before they run#18810

opened15d agobySsaphid

Bug

A webhook delivery that the server has answered with HTTP 202 can be lost if the server restarts before the delivery runs.

  1. Create a webhook scheduled task.
  2. Send two deliveries while the first one's dispatch is still starting (the per-task permit is held).
  3. Restart the server (desktop update, npx t3 restart, crash) before the second delivery gets the permit.
  4. The second delivery's log entry stays accepted forever and its prompt never runs. Replaying the same relay delivery returns duplicate, so the sender cannot recover it either.

This applies to direct, tunnel and T3 Connect relay deliveries. The first delivery has the same gap if the restart lands between the accepted log write and the start of its run.

Cause

ScheduledTaskService (apps/server/src/scheduledTasks/ScheduledTaskService.ts, around lines 1611-1655 on 511d20c6f2) writes the accepted delivery log, returns 202, and runs the delivery in a fiber forked into the service scope behind webhookPermit. That queue exists only in memory. Shutdown interrupts the fiber, and startup never looks at deliveries that are still accepted. The relay dedup claim (around line 1495) is already persisted, so a retry is treated as a duplicate.

A deterministic reproduction against the real service and an in-memory migrated SQLite database: hold the first launch with a Deferred, accept a second delivery, close and rebuild the service over the same database. Both rows stay accepted, a replay of the second returns duplicate, and the second prompt is never dispatched.

Possible fixes (needs a direction from maintainers)

  • Durable pending queue: persist the rendered prompt and a pending dispatch state before answering 202, drain pending deliveries through a scoped worker at startup and during normal operation, reuse the delivery-derived command/message IDs so a replay is idempotent, and mark completion durably. Replay payloads would need their own storage, because the delivery log is truncated and pruned.
  • Fail visibly instead: at startup, mark leftover accepted deliveries as failed ("server restarted before the run started") and release their relay claims so the relay can retry. Smaller, but a direct sender that already got 202 still loses the request.
  • Answer later: only answer 202 once the run has started, and return a retryable status while the task is busy. This changes the webhook contract (senders see 429/503 under load).

The first option is the complete fix but is a larger change to the scheduled task model, so I am filing this instead of opening a PR.

Found during an agent-driven audit (GPT-6.1 Sol, high) of upstream main 511d20c6f2. Filed by Claude Opus 5.5 in Claude Code via T3 Code on behalf of Alex Southwell.

JJulius Marminge6d ago

Reproduced on Linux with current main 70e5e384da. In an isolated web dev instance, created an immediate webhook task with no dedupe settings, sent two concurrent POSTs (both HTTP 202), and crashed the server before dispatch. After restart, both delivery rows remain accepted, the task has zero runs, and the pending queue is empty. The persisted coalescing queue on main does not cover this immediate-delivery path.