[Bug]: Recurring automations accumulate queued prompts while threads are blocked#16818

Thanks for the detailed report. I checked it against main, and the accumulation and the misleading success status both come from the code.

Root cause

1. No per-task outstanding check. runTask in apps/server/src/scheduledTasks/ScheduledTaskService.ts:715-726 only guards against an in-flight dispatch (activeRuns). For a bound thread, it calls threadManagement.sendToThread({ mode: "queue" }) (:816-831), and the reservation is released in Effect.ensuring (:867-873) as soon as admission returns. next_run_at then advances (:845, :854-861). Nothing checks whether a run from an earlier occurrence (scheduledTaskId) is still queued, so each interval adds another copy.
2. Success means "admitted," not "dispatched." runSucceeded = result._tag === "Success" (:834-836) records last_run_status = 'succeeded' once the enqueue call returns, even if the queued run never starts.
3. Inconsistent admission checks. Queue promotion (apps/server/src/orchestration-v2/Orchestrator.ts:1243-1269, startNextQueuedRun) refuses to start while a queued run is queueHeld or usageLimitBlockedRun(...) is non-null. Fresh admission doesn't do the same:
   • With an active run, queue_after_active inherits queueHeld (Orchestrator.ts:4937-4941), but the usage-limit state isn't checked.
   • With no active run, decideMessageDispatch turns queue_after_active into start_run (apps/server/src/orchestration-v2/CommandPolicy.ts:460-462, and start_immediately at :469-471) without checking either a held queue or a usage-limit block. So a scheduled prompt can skip past held items, or start a run that fails right away against the limit.

Stale "Working": I didn't find a mechanism in the scheduler for this. Duplicate admission alone doesn't explain it. If you can share the run statuses for the thread while it shows Working (from the thread's run list or logs), that would help.

Proposed fix

• Allow one outstanding occurrence per task. Before admitting a trigger === "scheduled" occurrence, skip or coalesce it if the bound thread already has a queued (or not-yet-started) run with the same scheduledTaskId, and still advance next_run_at. Keep explicit Run now and webhook deliveries as they are (webhooks already have WEBHOOK_MAX_QUEUED_PER_TASK).
• Make admission checks consistent. Apply the same held-queue and usageLimitBlockedRun checks in decideMessageDispatch / message.dispatch that startNextQueuedRun uses. A blocked thread should then queue the message (marked held) instead of starting a run, while still following the configured quota-recovery policy.
• Separate queued from dispatched. Record a queued result, or something similar, for the run, separate from succeeded/dispatched, so the automation UI shows when a prompt is still waiting.

Workaround

Disable the recurring automation to stop the queue from growing, then clear the duplicate prompts from the thread's queue by hand.

Related

#16393 (the open skip-if-busy feature request) overlaps with the first fix item. An outstanding-occurrence guard covers part of what it asks for, so the two could be designed together. #16726 (closed, about reusing one thread per run for agent scheduled tasks) touches the same runner but is a separate issue.

I have a fix for the first item of the proposed fix (one outstanding occurrence per task) and would like to send it as a focused PR. A few behavior choices need your call first.

What it does

• Before a timed occurrence is dispatched, the scheduler checks whether the task's thread still has an active run (preparing, queued, starting, running, or waiting). For a bound task that's the bound thread. For a task that launches a fresh thread per run, it's the thread its last timed run launched. While there is an active run, the occurrence is held.
• When that work finishes, the task runs once, however many slots passed, and then continues on its schedule. This coalesces instead of skipping, so the occurrence isn't lost and nothing piles up in the queue.
• Run now and webhook deliveries are unchanged.

How it differs from your suggestion, and the questions

1. Which runs hold the task. You suggested checking for a queued run with the same scheduledTaskId. That id only lives in the message payload JSON. I check run status per thread instead, which uses the existing (thread_id, status) index and also covers restart continuations. The side effect is that any active turn in a bound thread holds the task, including the user's own work. That is the stricter behavior from #18038. Is that acceptable, or should only the task's own prompts count?
2. Coalesce vs skip. After the busy run, should the held occurrence run once (what I have), or be skipped until the next regular slot?
3. Tasks that launch a fresh thread per run. These don't queue, but a long run can end up with a second one running alongside it in a new thread. The fix holds these too, which needs one nullable column (last_thread_id). Include it, or keep this PR to bound threads?
4. Showing the held state. The task list says Next run when the current one finishes · due 17 min ago instead of a next-run time in the past. Include it, or leave it out?

Items 2 and 3 of your proposed fix (consistent admission checks, and recording queued separately from dispatched) are not included. They would be separate PRs.

Evidence. The scheduler change has service-level tests that use the test clock and real SQLite projections. They cover a slot that comes due during a long run (it runs once, when the run finishes), a restart continuation still holding the slot, a task rebound while its launched run is still working, and Run now not hiding a busy timed run. The code is in the closed #19381, which also bundled a cron feature (now proposed separately in #19798). The fix PR would contain only this change.

Posted by Claude (Opus 5.5, Claude Code via T3 Code) on behalf of @thymikee.

Another reproduction on v0.1.0 (macOS, Claude provider, desktop app against the local server), with a different trigger: the bound thread was not usage-limited, only busy with one long turn.

What happened

An agent created an interval task (every 30 min) bound to its own thread via schedule_task, then kept working in a single turn for ~2.5 h. Run records for the thread:

Every interval queued a new copy while run 1 was still active, which matches root cause 1 above. The task prompt said "if a turn is already working, do nothing". That guard can never fire, because a queued prompt only starts after the active turn ends.

Gap not covered by this issue: deleting the task does not clear its queued runs

The agent deleted the task at 07:24:10 (delete_scheduled_task). Runs 4 to 7 were already queued, so they were still delivered over the next 48 s. That made it look as if the deleted task was still firing. deleteTask (ScheduledTaskService.ts:1075 on main) removes the row and the webhook rate and permit state, but does not cancel or drop queued thread runs that carry its scheduledTaskId. The suggested workaround here ("disable the automation") has the same limit.

Suggested addition to the fix: when a task is deleted or disabled, cancel its not-yet-started queued runs in the bound thread, or at least report how many remain. The one-outstanding-occurrence guard from @thymikee would have capped this case at one stale wake-up instead of four.

On question 1 in the comment above: in this case the run holding the task was the agent's own work in the same thread, not the task's own prompt. So checking "any active run in the bound thread" would have prevented it, and checking only "a queued run with the same scheduledTaskId" would have let one copy through.

Investigated with t3 triage; posted by Claude (Opus 5.5, Claude Code via T3 Code) on behalf of @yosofbadr.

Another reproduction on T3 Code 0.1.1, with Codex and recurring tasks bound to existing threads.

We have two tasks, running every 5 and 15 minutes. Both target threads were busy with long-running goal work. Scheduled prompts entered their message queues instead of steering the active turns. I had to manually promote the queued messages to steering.

Inspection of the installed runtime found that ScheduledTaskService.runTask calls threadManagement.sendToThread with hard-coded mode: "queue" for bound threads.

Preventing duplicate occurrences would help, but it would not fully address this use case: periodic progress checks must reach an active turn without manual action.

Would you accept an optional delivery mode for thread-bound schedules, exposed through the UI and MCP create/update tools?

• queue: preserve the existing behaviour.
• auto: start an idle thread or steer an active turn, subject to provider and run-state support.
• Combine repeated pending occurrences from the same timer.
• Preserve explicit Stop, held-queue, usage-limit, and approval boundaries.
• Show queued delivery separately from execution or completion.

Is this covered by planned work on this issue, or should I open a separate feature request for scheduled steering?

Reproduced on Linux x64 in the real web dev app and server at a12d6ed942. A Codex turn ran sleep 600; an enabled one-minute recurring task targeted that same thread. Three automatic intervals created three queued copies while the original run remained active. The task recorded succeeded after each admission.

This confirms the busy-thread variant from the comments. The literal quota-limit case was not verified: this VM uses Codex API-key authentication, and the rate-limit probe requires ChatGPT authentication. The Working label was accurate during this reproduction; it does not establish the reported stale Working state.

Runtime transcript. Recording: task already configured at the start; first queue admission at 00:51, second at 01:51. No Run now clicks. The test task was then disabled and the original turn stopped.

Two recurring prompts queued behind one active Codex turn

The held-queue variant also reproduces at a12d6ed942: Stop left runs 2–4 queued with queueHeld: true. Re-enabling the one-minute task allowed the next timed occurrence (run 5) to start and complete while those three remained held. No Send now or Run now action was used.

Runtime transcript. The browser later lost its connection under VM load, so the persisted run transcript is the evidence for this attempt.

Another reproduction, with a different cost angle: Claude provider, an interval task every 5 minutes bound to a long-lived orchestrating agent's own thread.

When a scheduled turn or a conversation turn runs longer than the interval, the next occurrence queues behind it and starts seconds after the previous turn finishes. The agent sees that it just ran and stops, but each of those occurrences still costs a full model turn on a long thread: about 1.3–1.5M tokens per skipped turn, almost all of it cache reads. On one day, 39 of 101 scheduled turns were these duplicates. Over 16 days there were at least 140.

The fix proposed above (one outstanding occurrence per task) would remove all of them: skip rather than queue when the bound thread already has an active or queued run. Latest-only coalescing would work just as well, provided a manual Run now still runs.

Additional reproduction on 2026-10-31: Claude subscription usage limit + automatic snooze + recurring automation, with 60 queued messages visible in the thread.

The reporter supplied two desktop screenshots captured at 19:07 local time (Europe/Vienna). They show:

• An explicit Claude weekly usage-limit response: “Usage limit reached. Retry after tomorrow at 16:00.”
• The thread is snoozed with roughly 21 hours remaining, and the composer shows the usage-limit auto-resume banner.
• Repeated queued cards for the same “Fleet health and stuck CI recovery” automation. The collapsed queue says “Show all 60 queued”. The expanded capture shows several identical automation cards plus an ordinary user message.
• Usage-limit responses at 19:05 and 19:06. These establish repeated failed attempts, but the captures alone do not prove which scheduler occurrences caused each attempt.

This matches this issue; no duplicate issue or competing PR is needed. Related #26987 describes a mixed automatic-notification backlog, while this reproduction specifically shows a subscription-limited, snoozed thread.

Current source and fix status

Checked upstream main at 15fe283272:

1. ScheduledTaskService.runTask reserves activeRuns only during dispatch. For a bound thread, sendBound passes scheduledTaskId and mode: "queue" to sendToThread.
2. Every timed occurrence uses a fresh fire key (task id + current timestamp + trigger), hence distinct command/message ids. Ordinary command idempotency cannot deduplicate different scheduled intervals.
3. A successful admission sets lastRunStatus to succeeded, increments the run count, advances nextRunAt, and releases the reservation. There is still no check here for an earlier queued occurrence from that same task. Snooze/quota recovery does not itself stop this scheduler path from admitting another occurrence.

The focused existing fix is #23609, head 46ec89c730cc887bb49a35b697b3580ed4d21add. It currently has merge conflicts with main. Its per-task queued-occurrence guard targets this accumulation directly; its description explicitly records that a literal subscription-limit reproduction was not exercised. These screenshots provide additional real-world evidence for that trigger, but do not validate the fix against it.

Suggested maintainer validation

After refreshing #23609, run the focused scheduler regression and verify a real or controlled recognized Claude usage-limit pause across multiple timer intervals: at most one queued occurrence per task, nextRunAt continues advancing, and the thread is not woken before its reset time. Check two task ids independently, a held queue after Stop, restart, and normal delivery after release. Preserve explicit Run now and webhook requests. Existing backlog cleanup and the distinction between queue admission and execution remain separate concerns; the guard does not retroactively remove 60 already-queued items.

Workaround: pause the recurring automation to stop new copies, then review and cancel duplicate queued occurrences while retaining user messages. This investigation did not modify the reporter’s live thread, queue, automation, or database. App version and host-server commit are not established by the captures.

Investigated and posted with GPT-6.1-Sol through the Codex harness, on behalf of @vitalyiegorov.

Correction: this investigation has not independently confirmed a T3 bug. The original wording overstated what the observations established.

This installation has both an hourly T3 task bound to an existing thread and custom external automation that explicitly sends new messages through the CLI when work finishes. Those external sends explain much of the repeated resurfacing. A fresh message waking a snoozed or settled thread does not, by itself, establish a snooze defect.

For completeness, the read-only records from T3 Code 0.1.5 on macOS, Claude provider, also show a native scheduled-delivery occurrence:

• T+0: “Snooze until reset” is accepted and saved with a future weekly reset deadline; auto-resume is enabled.
• T+19 minutes: An hourly scheduled occurrence dispatches a message. The same command emits thread.unsnoozed, clears the persisted snooze, and starts a run that fails on usage_limit / api_error_429 in under one second.
• The scheduled task records lastRunStatus: succeeded, although the admitted run fails.

The captured history contains 37 usage-limit failures: eight from scheduled occurrences and 29 from external CLI messages. These counts describe this mixed setup, not 37 independently established scheduler failures.

No queued backlog was present at capture, so this does not independently reproduce the queue accumulation reported here. Whether the native scheduled-delivery occurrence violates the intended quota/queue policy remains a separate question requiring an isolated reproduction. No proposed fix was validated.

The investigation was historical and read-only; no settings, schedules, queues, or application code were changed. Thread titles, identifiers, prompts, paths, and work details have been omitted.

Another way to reach the same backlog, plus one behaviour this issue doesn't cover yet: queued copies keep arriving after the scheduled task is deleted.

Setup

• Claude provider, one long-lived orchestrator thread.
• One recurring task bound to it with schedule_task (interval, everyMs: 1800000, bindToCurrentThread: true).

What happened

1. The thread showed an AskUserQuestion question and waited for the answer. While the question was open, every interval queued another copy of the task's prompt; none ran.
2. Once the question was answered, the queued copies arrived one after another, back to back. This happened several times during a day, each time a question was open.
3. When the work was done, the thread deleted the task with delete_scheduled_task. The call returned deleted: true, and list_scheduled_tasks returned [] for this project, for the other projects on the same repo, and for the scratch project.
4. More than 20 more copies of the same prompt reached the thread after that, one per turn, roughly as fast as each turn ended. They seem to be the copies queued while questions were open, and nothing marks them as belonging to a task that is gone.

Expected

• A pending user question counts as "blocked" for a recurring task: at most one outstanding occurrence, as this issue suggests for usage limits.
• Deleting a scheduled task also drops that task's undelivered occurrences from the thread queue, or marks them stale so they are not delivered.

Impact

Each stale copy starts a full turn that does nothing. The user had to read the same "no job to delete" answer over and over, and the only way to stop them was to wait for the queue to empty.