fix(server): preserve message order across lineage reads#24442

open2
main
fix/issue-24335
opened4d agobyJJulius Marminge
View on GitHubTwo-way sync enabledPR last imported 4d ago

Problem

A user message can receive a later run ordinal than an agent message dispatched after it. With a file-backed database, the user message’s subagent-lineage lookup yields before the thread lock, allowing the later agent message to overtake it. The reporter’s queued-cancellation replay fails and can cancel the wrong run.

Change

Serialize message intake per thread before the lineage lookup. Keep the existing parent-before-child thread lock order and allow unrelated threads to proceed independently. Add a deterministic regression test for the overtaking sequence.

Scope and approval

Fixes https://github.com/pingdotgg/t3code/issues/24335

GitCafe: #24335

Triaged bug via the automated issue flow. This fixes dispatch ordering without changing contracts, persistence, providers, or clients.

Verification

  • The regression fails on base with agent-first ordering and passes after the fix. The control-read suite passes 12/12; 11 focused server suites pass 187 tests. Both independent code reviewers returned MERGE with no blocking findings.
  • The reporter’s file-backed queued_cancelled_while_active replay fails 3/3 before and passes 3/3 after. Server typecheck has no errors; scoped lint has no new findings.
  • Actual isolated dev servers exercise a genuine Claude native background continuation. Before, the later agent continuation takes run 2 and the user takes run 3. After, the user takes run 2 and the agent takes run 3; cancelling run 3 never starts it, leaves run 2 active, and run 2 completes.
  • Runtime evidence uses a documented one-shot reader-worker gate to force the overlap. It proves ordering under that overlap, without measuring uninstrumented frequency. Runtime verification used the Claude adapter; other adapters were not exercised. No client render path changed, so the capture matrix is empty.

Evidence index: before/after transcripts, named red/green tests, database observations, commands, and reproduction drivers. All evidence-only artifacts are uploaded separately from the code.

Implemented with Claude Opus 5.5 through the Claude Agent harness; reviewed and verified with GPT-6.1 Sol through Codex and Claude Opus 5.5 through the Claude Agent harness. Issue lead: GPT-6.1 Sol through Codex.

Checks

22 successful · 7 skipped
All checks passed
f94282f

Added 2 commits

  • test(server): reproduce message ordering regression for issue 2433539a8049
  • fix(server): messages to a thread keep arrival order when a lineage read yieldsf94282f
Synced from GitHub

Approvability

Verdict: Approved at f94282f

Macroscope's review found this PR approvable — This is a contained server-side concurrency fix that serializes same-thread message intake before lineage reads, preventing later messages from overtaking earlier ones while preserving parallelism across unrelated threads. It includes a deterministic regression test and no schema, deployment, security, billing, or static-analysis configuration changes.

You can add or adjust custom eligibility rules. Learn more.

Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire5.0 KiB5.0 KiB−7 B (−0.1%)6.8 KiB✅
CodexThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
CodexLive turn WebSocket wire1.2 KiB1.2 KiB−7 B (−0.6%)2.0 KiB✅
CodexLive turn WebSocket decoded20.9 KiB20.9 KiB0 B (0.0%)29.3 KiB✅
CodexLive turn messages220 (0.0%)8✅
ClaudeTotal thread wire5.0 KiB5.0 KiB0 B (0.0%)6.8 KiB✅
ClaudeThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
ClaudeLive turn WebSocket wire1.2 KiB1.2 KiB0 B (0.0%)2.0 KiB✅
ClaudeLive turn WebSocket decoded21.2 KiB21.2 KiB0 B (0.0%)29.3 KiB✅
ClaudeLive turn messages220 (0.0%)8✅

Baseline: d53873b · PR result: f94282f · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Synced from GitHub

Review in Change Stack →

No actionable comments were generated in the recent review.

External image Recent review info
External image Run configuration
External image Commits

Reviewing files that changed from the base of the PR and between d53873b6a4799e87afd63c8d288ef0089f5a3748 and f94282faa30e1091c1fc8c29f85e6387c06a3920.

External image Files selected for processing: 3
  • apps/server/src/orchestration-v2/Orchestrator.control-reads.test.ts
  • apps/server/src/orchestration-v2/Orchestrator.ts
  • apps/server/src/orchestration-v2/testkit/ProviderReplayHarness.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

Walkthrough

Message dispatch now serializes intake per thread before checking subagent lineage. A deterministic test delays a lineage read and verifies that a later agent message does not receive an earlier run ordinal.

Changes: Message intake ordering

Layer / File(s)Summary
Serialize message intake
apps/server/src/orchestration-v2/Orchestrator.ts
Message dispatch acquires a per-thread intake lock before checking for an app-owned subagent parent. User-created and MCP messages also acquire the parent thread’s dispatch lock when a parent is found.
Test delayed lineage reads
apps/server/src/orchestration-v2/testkit/ProviderReplayHarness.ts, apps/server/src/orchestration-v2/Orchestrator.control-reads.test.ts
The replay harness accepts a projection-store wrapper. The test pauses a targeted getThread call, dispatches user and agent messages, and verifies their run order.

Priority: External image Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Low

Sequence Diagram(s)

mermaid
1sequenceDiagram
2 participant User
3 participant Agent
4 participant Orchestrator
5 participant KeyedLock
6 participant ProjectionStore
7 User->>Orchestrator: message.dispatch
8 Orchestrator->>KeyedLock: acquire thread intake lock
9 Orchestrator->>ProjectionStore: getThread for lineage lookup
10 Agent->>Orchestrator: message.dispatch while lookup is pending
11 Orchestrator->>KeyedLock: wait for the same thread intake lock
12 ProjectionStore-->>Orchestrator: lineage result

Possibly related PRs

  • pingdotgg/t3code#21813: Both changes modify message-dispatch locking and include parent-lock handling in the dispatch path.

Suggested labels: bugfix, size:M

Suggested reviewers: maria-rcks, macroscopeapp

Merge Risk: External image Minimal · up to f9428

No concrete issue remains that should block merging. The ordering regression test was not independently run here.

Pre-merge checks: Passed 4
Check nameStatusExplanation
Title checkPassedThe title clearly and concisely summarizes the primary server fix: preserving message order across lineage reads.
Description checkPassedThe description includes all required sections. It explains the race, the implementation, issue approval, focused verification, limitations, and evidence.
Linked Issues checkPassed[#24335] Objective: keep a user or MCP message's dispatch order ahead of a later agent-queued message for the same thread on the file-database path, so the user message becomes run 1 and the agent mes…
Out of Scope Changes checkPassedThe changes described and visible in the diff are limited to the dispatch-ordering fix and its test support:
- Orchestrator.ts: the per-thread intake lock and the restructured message.dispatch path. T…
Full details: Linked Issues check

Explanation

[#24335] Objective: keep a user or MCP message's dispatch order ahead of a later agent-queued message for the same thread on the file-database path, so the user message becomes run 1 and the agent message run 2.

Evidence at head, from the whole-PR diff of apps/server/src/orchestration-v2/Orchestrator.ts:

  • A new KeyedLock messageIntake is created per orchestrator layer.
  • For every message.dispatch, the handler takes messageIntake.withLock(command.threadId, ...) before the appOwnedSubagentParentThreadId lineage read. The read therefore runs while the thread's intake is held. A later message for the same thread waits on intake and cannot overtake it.
  • Inside intake, a user or MCP message resolves the parent thread. If a parent exists, the dispatch runs under threadDispatch.withLock(parentThreadId, dispatch). The child lock is then taken inside that, so the order is intake(child), parent, child. The existing parent-before-child thread lock order is kept.
  • Other message sources run dispatch under intake alone. Unrelated threads use different intake keys and are not serialized against each other.

Test evidence: the head file apps/server/src/orchestration-v2/Orchestrator.control-reads.test.ts imports Deferred and Fiber, which matches the described deterministic test. The added test body was not read, because the changed-file diff was unavailable and the head file read was truncated before the new test. The regression result rests on the PR description and change summary for that part.

Residual risk, not established either way: a parent-thread operation that calls a child message.dispatch through the public dispatch while holding the parent lock would take intake(child) under the parent lock. That is the reverse of the user path. No such path was inspected, so no deadlock finding is made.

Full details: Out of Scope Changes check

Explanation

The changes described and visible in the diff are limited to the dispatch-ordering fix and its test support:

  • Orchestrator.ts: the per-thread intake lock and the restructured message.dispatch path. This implements #24335 directly.
  • Orchestrator.control-reads.test.ts: the deterministic overtaking regression test. It uses Deferred and Fiber, which the test needs to hold the lineage read. This supports the fix.
  • ProviderReplayHarness.ts: an optional wrapProjectionStore hook, per the change summary. It is used only to gate the projection read in the test. This supports the test.

No unrelated behavior, contract, persistence, provider, or client change was reported or observed. The ProviderReplayHarness.ts and test-file hunks were not read directly, because the changed-file diff was unavailable. This verdict relies on the change summary for those files.

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.