refactor(server): share scheduling for tasks and limit recovery#12795

merged6
provider-limits/failed-turn-transcript
provider-limits/shared-scheduler
openedSep 20, 2026byJJulius Marminge
View on GitHubTwo-way sync enabledPR last imported 9d ago

Limit retries added a separate 30-second polling loop alongside Scheduled Tasks. Extract the existing five-second task scheduler into one shared server primitive and register both features with it.

The scheduler checks due work on registration, then every five seconds. It owns source lifetimes, error isolation, and preventing overlapping sweeps. Slow task work does not delay limit recovery. Each feature keeps its existing persisted state and dispatch guards, so retries survive restarts without becoming user-visible Scheduled Tasks. Limit recovery uses one focused SQLite query for unarmed failures affected by preferences and due enabled retries. Future, cancelled, archived, settled, and pending-request threads are excluded before decoding. Scheduler ticks no longer rebuild shell snapshots, count transcript history, or traverse fork ancestors.

Stacked on #12783.

Verification: 81 focused tests pass, covering the shared scheduler, overdue work after restart, SQL recovery filtering, cancellation, snooze deadlines, provider-session errors, latest root failures, and 15 dispatch guards. Scoped server typecheck and targeted lint pass. SQLite query-plan inspection confirms indexed latest-run/root-error lookups. The query still checks thread metadata every five seconds; CPU usage on large databases has not been benchmarked. No client presentation changes; automated behavior tests provide the evidence.

Model: GPT-6. Harness: Codex.

Summary by CodeRabbit

  • New Features

    • Added centralized scheduling for recurring background work, including scheduled tasks and usage-limit recovery.
    • Coordinated due work across restarts while preventing overlapping executions.
    • Usage-limit recovery now identifies eligible failed threads and respects recovery timing, auto-resume, and snooze settings.
  • Bug Fixes

    • Improved resilience so failures in one scheduled process do not stop other scheduled work.
    • Prevented overlapping background executions during slow processing.
  • Tests

    • Added coverage for scheduling reliability, restart recovery, timing, cleanup, and recovery eligibility.

Integration verification: the shared scheduler constructor is private, fixing the unused export reported by CI. All five scheduler lifecycle and integration tests pass, including immediate overdue recovery, source isolation, interruption, and no overlapping work. Scoped server typecheck passes.

Built with GPT-6 in Codex.

apps/server/src/scheduling/Scheduler.tslines 60–60Outdated · 9c1aaadOpen thread
……@@ -0,0 +1,63 @@
57+ );
58+ // One clock for all due-work sources. A slow source cannot block another
59+ // source or overlap itself, and no extra missed-tick backlog is queued.
60+ yield* Effect.sleep("5 seconds").pipe(Effect.andThen(tick), Effect.forever, Effect.forkScoped);
MmacroscopeappBoton lines 60–60Sep 20, 2026

🟡 Medium scheduling/Scheduler.ts:60

The scheduler performs no work for the first five seconds after startup, so overdue tasks and due usage-limit recoveries are unnecessarily delayed. Line 60 enters sleep before the first tick; run tick once before starting the sleeping loop.

No longer relevant as of f5cc6ed2573c9e2c9aa81a6d69bd11530fe3a768

Fixed in b875b89: each source checks due work immediately when it registers, then uses the shared five-second scheduler. A tick during scheduler construction would run before sources exist, so registration is the useful startup trigger. The focused tests now wait for startup receipts before advancing the clock; all 50 scheduler/task tests and the server typecheck pass.

apps/server/src/scheduling/Scheduler.tsline 21Outdated · b875b89Open thread
……@@ -0,0 +1,64 @@
18+>()("t3/scheduling/Scheduler") {}
19+
20+export const layer = Layer.effect(
21+ Scheduler,

This service module owns construction, so expose a real make before layer rather than constructing it inline in Layer.effect. Extract the current construction effect as export const make = Effect.gen(...), then define export const layer = Layer.effect(Scheduler, make).

No diff: this requires moving the construction body and changing the layer declaration.

Posted via Macroscope — Effect Service Conventions

Fixed in 024025a: Scheduler exposes its construction as make, and all consumers use the Scheduler module namespace. The 50 focused scheduler/task tests, server typecheck, and targeted lint pass.

……@@ -7,7 +7,7 @@ import {
88import * as Effect from "effect/Effect";
99import * as Layer from "effect/Layer";
10-import * as Schedule from "effect/Schedule";
10+import { Scheduler } from "../scheduling/Scheduler.ts";

Import this service module as a namespace so the service boundary retains its public shape. Use import * as Scheduler from "../scheduling/Scheduler.ts", then acquire yield* Scheduler.Scheduler.

No diff: the import and consumer are in separate hunks.

Posted via Macroscope — Effect Service Conventions

Fixed in 024025a: Scheduler exposes its construction as make, and all consumers use the Scheduler module namespace. The 50 focused scheduler/task tests, server typecheck, and targeted lint pass.

apps/server/src/scheduledTasks/ScheduledTaskService.tsline 33Outdated · b875b89Open thread
……@@ -31,6 +30,7 @@ import * as SqlClient from "effect/unstable/sql/SqlClient";
3130
3231import * as ThreadLaunchService from "../orchestration-v2/ThreadLaunchService.ts";
3332import * as ThreadManagementService from "../orchestration-v2/ThreadManagementService.ts";
33+import { Scheduler } from "../scheduling/Scheduler.ts";

Import this service module as a namespace so consumers use its public service boundary. Use import * as Scheduler from "../scheduling/Scheduler.ts", then acquire yield* Scheduler.Scheduler.

No diff: the import and consumer are in separate hunks.

Posted via Macroscope — Effect Service Conventions

Fixed in 024025a: Scheduler exposes its construction as make, and all consumers use the Scheduler module namespace. The 50 focused scheduler/task tests, server typecheck, and targeted lint pass.

apps/server/src/orchestration-v2/runtimeLayer.tsline 2Outdated · b875b89Open thread
……@@ -1,4 +1,5 @@
11import { workerLive as usageLimitRecoveryWorkerLive } from "./UsageLimitRecoveryService.ts";
2+import { layer as schedulerLayer } from "../scheduling/Scheduler.ts";

orchestration-v2/runtimeLayer.ts:2

Import the service module as a namespace rather than aliasing its layer export. Use import * as Scheduler from "../scheduling/Scheduler.ts" and provide Scheduler.layer.

No diff: the import and use are in separate hunks.

No longer relevant as of 024025a0145ebd899e775f24af2550ad1bda64a2

Fixed in 024025a: Scheduler exposes its construction as make, and all consumers use the Scheduler module namespace. The 50 focused scheduler/task tests, server typecheck, and targeted lint pass.

Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire—4.9 KiB—6.8 KiB✅
CodexThread snapshot wire—3.7 KiB—4.9 KiB✅
CodexLive turn WebSocket wire—1.2 KiB—2.0 KiB✅
CodexLive turn WebSocket decoded—20.4 KiB—29.3 KiB✅
CodexLive turn messages—2—8✅
ClaudeTotal thread wire—4.9 KiB—6.8 KiB✅
ClaudeThread snapshot wire—3.7 KiB—4.9 KiB✅
ClaudeLive turn WebSocket wire—1.2 KiB—2.0 KiB✅
ClaudeLive turn WebSocket decoded—20.8 KiB—29.3 KiB✅
ClaudeLive turn messages—2—8✅

Baseline: unavailable · PR result: 4dc66ff · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 106.1 KiB
  • Claude decoded thread snapshot: 106.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Synced from GitHub

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: pingdotgg/t3code/.coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: 304e4e8c-28b1-48d2-9335-1aeabaccb419

📥 Commits

Reviewing files that changed from the base of the PR and between f5cc6ed2573c9e2c9aa81a6d69bd11530fe3a768 and f34dc5e3d9b85e2c21355441a1132ceb450dcd30.

📒 Files selected for processing (2)
  • apps/server/src/orchestration-v2/ProjectionStore.test.ts
  • apps/server/src/orchestration-v2/ProjectionStore.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

The PR adds a shared Effect scheduler with scoped registrations, serialized execution, failure handling, and five-second polling. Scheduled-task and usage-limit recovery services use it. The projection store supplies filtered recovery candidates. Tests cover concurrency, cleanup, restart, and recovery behavior.

Changes

Shared scheduler adoption

Layer / File(s)Summary
Scheduler implementation and behavior
apps/server/src/scheduling/Scheduler.ts, apps/server/src/scheduling/Scheduler.test.ts
Adds scoped registration, five-second polling, per-source serialization, failure handling, scope cleanup, and scheduler behavior tests.
Limit-recovery candidate projection
apps/server/src/orchestration-v2/ProjectionStore.ts, apps/server/src/orchestration-v2/ProjectionStore.test.ts, apps/server/src/orchestration-v2/ProviderTurnControlService.test.ts
Adds the candidate type and store method. SQL and in-memory implementations filter candidates by thread state, reset timing, recovery state, and configuration.
Service registration and production wiring
apps/server/src/scheduledTasks/*, apps/server/src/orchestration-v2/UsageLimitRecoveryService.ts, apps/server/src/orchestration-v2/runtimeLayer.ts, apps/server/src/orchestration-v2/ThreadLaunchService.test.ts
Moves scheduled-task and usage-limit recovery polling to the shared scheduler. Production and test layers provide Scheduler.layer.
Cross-service integration validation
apps/server/src/scheduling/Scheduler.integration.test.ts
Tests scheduled-task execution and persisted usage-limit recovery at reset time and after restart.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Refactor

Suggested reviewers: saphid

Sequence Diagram(s)

mermaid
1sequenceDiagram
2 participant Scheduler
3 participant ScheduledTaskService
4 participant UsageLimitRecoveryService
5 participant ProjectionStore
6 participant OrchestrationService
7 Scheduler->>ScheduledTaskService: Run registered due-task work
8 ScheduledTaskService->>OrchestrationService: Dispatch due task
9 Scheduler->>UsageLimitRecoveryService: Run registered recovery work
10 UsageLimitRecoveryService->>ProjectionStore: Get limit-recovery candidates
11 UsageLimitRecoveryService->>OrchestrationService: Dispatch recovery retry
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check nameStatusExplanation
Docstring Coverage✅ PassedDocstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 12 files.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
Title check✅ PassedThe title clearly summarizes the primary change: sharing one server scheduler between scheduled tasks and limit recovery.
Description check✅ PassedThe description explains what changed, why the shared scheduler is needed, implementation details, verification results, and UI impact. The checklist section from the template is not included, but the…
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

Synced from GitHub

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR introduces a new production scheduler and changes the timing, lifecycle, concurrency, and persistence-query behavior of both scheduled tasks and usage-limit recovery. Because it restructures shared background infrastructure and alters existing runtime workflows, human review is warranted.

No code changes detected at 4dc66ff. Prior analysis still applies.

You can add or adjust custom eligibility rules. Learn more.

apps/server/src/orchestration-v2/runtimeLayer.tsline 1Outdated · 62c0cf7Open thread
……@@ -1,4 +1,5 @@
11import * as UsageLimitRecoveryService from "./UsageLimitRecoveryService.ts";

orchestration-v2/runtimeLayer.ts:1

Import this Effect worker module as a namespace so its public boundary remains visible: use import * as UsageLimitRecoveryService from "./UsageLimitRecoveryService.ts", then reference UsageLimitRecoveryService.workerLive.

No diff: the import and consumer are in separate hunks.

No longer relevant as of 62c0cf705d7f3212cc3dff4f048c64681199cf17

Already fixed in the current integrated head 62c0cf705d7: runtimeLayer imports UsageLimitRecoveryService as a namespace and references UsageLimitRecoveryService.workerLive. Scoped server typecheck and lint pass.

apps/server/src/scheduling/Scheduler.tsline 20Outdated · 62c0cf7Open thread
……@@ -0,0 +1,61 @@
17+ }
18+>()("t3/scheduling/Scheduler") {}
19+
20+const make = Effect.gen(function* () {

make has regressed to a private constructor even though this module owns the service construction. Export it as the canonical service constructor.

suggestion
1export const make = Effect.gen(function* () {

Posted via Macroscope — Effect Service Conventions

Keeping make private because required CI knip explicitly failed on this unused export. Both production consumers use Scheduler.layer, which owns construction and exposes the service requirements; there is no external consumer of a direct constructor. Exporting an unused API or suppressing the unused-export check would add no functionality. The five scheduler lifecycle/integration tests and scoped server typecheck pass.

Added 6 commits

  • perf(server): query only actionable usage-limit recovery19602ca
  • refactor(server): follow scheduler service conventions2d94436
  • fix(server): keep scheduler construction private4dc66ff
  • test(server): ensure scheduler skips overlapping ticks74072e7
  • fix(server): check due work when scheduler sources registerb126b04
  • refactor(server): share scheduling for tasks and limit recoveryb1796d4