Sign up
View stargazers13Sign up

Queued native child continuation output is dropped after root completion#19228

opened10d agobyNnot-compromised

Native child continuation events can be dropped after a root turn finishes, even on current main with synchronous SQLite. This is the separate defect discussed by the maintainer in #19211.

Confirmed on main commit 11fd54fcbbfb39f824bb4c908d428c20d801267c, Node 24.21.0. Only the test file changes; no SQLite worker or new provider-event fields are involved. Server typecheck passes.

A focused RunExecutionService test queues these normalized events while the adapter's live pending-work probe already returns false:

  1. A child is running when the root terminal arrives.
  2. The child's subagent completion arrives, emptying the ingestion tracking sets.
  3. That child's native continuation starts, sends GRANDCHILD_OK, and completes.

shouldStopProviderEventIngestion consults the live probe at step 2 and closes before consuming step 3. Both retained-root and lost-root ownership cases discard all continuation events. Expected observations are running, GRANDCHILD_OK, completed; actual observations are []. Both tests fail by assertion immediately, without sleeps, polling, timeout changes, or database delays.

Apply the focused patch below to the pinned main revision, then run:

sh
1vp test run apps/server/src/orchestration-v2/RunExecutionService.test.ts --testNamePattern='retains queued child output'

This is a service-level reproduction with the ingestion persistence dependency mocked, distinct from a physical-client reproduction. The full-worker investigation also reproduced the missing response in the real OpenCode HTTP replay with SQLite persistence. It does not establish the historical phone incident's exact cause.

Focused regression patch for current main
diff
1diff --git a/apps/server/src/orchestration-v2/RunExecutionService.test.ts b/apps/server/src/orchestration-v2/RunExecutionService.test.ts
2index d877922a6..9a0e40d63 100644
3--- a/apps/server/src/orchestration-v2/RunExecutionService.test.ts
4+++ b/apps/server/src/orchestration-v2/RunExecutionService.test.ts
5@@ -12,9 +12,11 @@ import {
6 type OrchestrationV2DomainEvent,
7 type OrchestrationV2ExecutionNode,
8 type OrchestrationV2ProviderThread,
9+ type OrchestrationV2ProviderTurn,
10 type OrchestrationV2Run,
11 type OrchestrationV2RunAttempt,
12 type OrchestrationV2Subagent,
13+ type OrchestrationV2StoredEvent,
14 type OrchestrationV2TurnItem,
15 ProviderDriverKind,
16 ProviderInstanceId,
17@@ -1687,6 +1689,92 @@ it.effect("ends a nested native subagent and its thread when the root run is int
18 }),
19 );
20
21+it.effect.each([false, true])(
22+ "retains queued child output when the adapter is already idle and root ownership lost=%s",
23+ (rootOwnershipLost) =>
24+ Effect.gen(function* () {
25+ const childEvents = yield* Ref.make<ReadonlyArray<string>>([]);
26+ yield* runBackgroundItemScenario(
27+ "ordered-background-continuation",
28+ (ids) => {
29+ const childProviderThreadId = ProviderThreadId.make(
30+ "provider-thread:ordered-background-continuation:child",
31+ );
32+ const snapshot = (populated: boolean, child = false) => ({
33+ type: "provider_thread.updated" as const,
34+ driver,
35+ providerThread: {
36+ id: child ? childProviderThreadId : ids.providerThreadId,
37+ driver,
38+ providerInstanceId: ProviderInstanceId.make("codex"),
39+ providerSessionId: null,
40+ appThreadId: child ? ids.childThreadId : ids.threadId,
41+ ownerNodeId: null,
42+ nativeThreadRef: null,
43+ nativeConversationHeadRef: null,
44+ status: "idle" as const,
45+ firstRunOrdinal: 1,
46+ lastRunOrdinal: 1,
47+ handoffIds: [],
48+ forkedFrom: null,
49+ pendingBackgroundTasks: populated
50+ ? [{ taskId: "child", kind: "subagent" as const, childThreadId: ids.childThreadId }]
51+ : [],
52+ createdAt: DateTime.makeUnsafe(0),
53+ updatedAt: DateTime.makeUnsafe(0),
54+ } satisfies OrchestrationV2ProviderThread,
55+ });
56+ const childTurn = (status: "running" | "completed") => ({
57+ type: "provider_turn.updated" as const,
58+ driver,
59+ threadId: ids.childThreadId,
60+ providerTurn: {
61+ id: ProviderTurnId.make("provider-turn:ordered-background-continuation:child"),
62+ providerThreadId: ProviderThreadId.make(
63+ "provider-thread:ordered-background-continuation:child",
64+ ),
65+ runAttemptId: null,
66+ ordinal: 2,
67+ status,
68+ } as OrchestrationV2ProviderTurn,
69+ });
70+ return [
71+ childThreadCreatedEvent(ids),
72+ snapshot(true),
73+ snapshot(true, true),
74+ subagentEvent(ids, "running"),
75+ rootTerminalEvent(ids, "completed"),
76+ subagentEvent(ids, "completed"),
77+ snapshot(false),
78+ childTurn("running"),
79+ {
80+ type: "message.updated",
81+ driver,
82+ message: { threadId: ids.childThreadId, runId: null, text: "GRANDCHILD_OK" },
83+ } as ProviderAdapter.ProviderAdapterV2Event,
84+ childTurn("completed"),
85+ snapshot(false, true),
86+ snapshot(false),
87+ ];
88+ },
89+ {
90+ keepEventStreamOpen: true,
91+ hasPendingBackgroundWorkForThread: () => Effect.succeed(false),
92+ rejectRootProviderThreadUpdates: rootOwnershipLost,
93+ onIngest: (event) =>
94+ Ref.update(childEvents, (events) =>
95+ event.type === "provider_turn.updated"
96+ ? [...events, event.providerTurn.status]
97+ : event.type === "message.updated"
98+ ? [...events, event.message.text]
99+ : events,
100+ ),
101+ },
102+ );
103+ assert.deepEqual(yield* Ref.get(childEvents), ["running", "GRANDCHILD_OK", "completed"]);
104+ }),
105+);
106+
107 it.effect("keeps ingesting a late background command item completion after root terminal", () =>
108 Effect.gen(function* () {
109 const observed = yield* runBackgroundItemScenario("bg-command", (ids) => [
110@@ -4331,6 +4419,11 @@ function runBackgroundItemScenario(
111 ReadonlyArray<{ readonly id: TurnItemId; readonly runId: RunId }>
112 >;
113 readonly onSubscribe?: Effect.Effect<void>;
114+ readonly onIngest?: (event: ProviderAdapter.ProviderAdapterV2Event) => Effect.Effect<void>;
115+ readonly hasPendingBackgroundWorkForThread?: (
116+ thread: OrchestrationV2ProviderThread,
117+ ) => Effect.Effect<boolean>;
118+ readonly rejectRootProviderThreadUpdates?: boolean;
119 },
120 ) {
121 return Effect.gen(function* () {
122@@ -4363,6 +4456,7 @@ function runBackgroundItemScenario(
123 ingestNormalized: (input) =>
124 Effect.gen(function* () {
125 const event = input.event;
126+ yield* options?.onIngest?.(event) ?? Effect.void;
127 if (event.type === "turn_item.updated") {
128 yield* Ref.update(observed, (current) => [
129 ...current,
130@@ -4375,6 +4469,18 @@ function runBackgroundItemScenario(
131 `subagent:${event.subagent.status}`,
132 ]);
133 }
134+ if (event.type === "provider_thread.updated") {
135+ if (
136+ options?.rejectRootProviderThreadUpdates === true &&
137+ input.writeIfProviderThreadOwner !== undefined
138+ )
139+ return [];
140+ return [
141+ {
142+ event: { type: "provider-thread.updated", payload: event.providerThread },
143+ } as OrchestrationV2StoredEvent,
144+ ];
145+ }
146 return [];
147 }),
148 }),
149@@ -4391,6 +4497,11 @@ function runBackgroundItemScenario(
150 providerSessionId: ProviderSessionId.make(`session:${key}`),
151 session: {
152 events: Stream.empty,
153+ ...(options?.hasPendingBackgroundWorkForThread === undefined
154+ ? {}
155+ : {
156+ hasPendingBackgroundWorkForThread: options.hasPendingBackgroundWorkForThread,
157+ }),
158 subscribeEvents: Effect.gen(function* () {
159 yield* options?.onSubscribe ?? Effect.void;
160 const events = Stream.fromIterable(makeEvents(ids));

Written by GPT-6.1 Sol in Codex.

JJulius Marminge10d ago

Note

Grok responding on behalf of Julius.

Confirmed on main: once the root turn has completed, this ingestion stream can close between a child's subagent completion and that child's native continuation, so the continuation's output is never ingested.

In apps/server/src/orchestration-v2/RunExecutionService.ts, shouldStopProviderEventIngestion runs after every event (Stream.takeUntilEffect). After a completed root terminal it only keeps the stream open while activeChildProviderTurns, activeChildSubagents or activeBackgroundTurnItems are non-empty, or while hasPendingBackgroundWorkForThread returns true. The child's subagent completion empties those sets. If the thread-scoped probe already reports idle, or providerThreadOwnerLost is set, the function returns true right there. The child's continuation (provider_turn.updated running → message → completed) arrives after the stream has closed, so it's dropped. Nothing in the stop check expects a child turn to start up again after its subagent item settles.

Notes for a fix:

  • The stream needs a signal that a child continuation is still coming, either from the adapter or by holding briefly after an owned child's subagent completes. Making the live probe more permissive alone would undo the owner-loss release.
  • The reporter's two-case test (root ownership retained or lost) is a good regression test to adopt.

Evidence: confirmed in code (stop logic as of b9afba59f). I didn't run the reporter's test. The service-level repro uses a mocked ingestor, and the original phone incident isn't shown to have this cause.

JJulius Marminge2d ago

Checked on Linux, Node 24.21.0, main 70e5e384da. No matching duplicate among 950 open issues.

The supplied test patch needs context updates on current main. With those updates, vp test run apps/server/src/orchestration-v2/RunExecutionService.test.ts --testNamePattern='retains queued child output' fails both ownership cases exactly as reported: expected ['running', 'GRANDCHILD_OK', 'completed'], received []. Failure log, adapted test patch.

The existing opencode2_nested_background HTTP/SSE replay passes on the same main with synchronous SQLite: replay log.

I also started the real web/server app with isolated seeded state and sent the fixture's nested-background prompt using the installed OpenCode 1.18.18 / Big Pickle. It remained at Thinking without reaching root completion or native child activity. The first attempt was interrupted by the dev file watcher during test startup; a repeat without tests running also did not reach the reported sequence. This is inconclusive, not evidence that the reported ordering is fixed.

Live web attempt before cancellation

Can you provide a current-main HTTP/SSE replay or app-level steps that fail with synchronous SQLite, including the exact OpenCode version, model, nesting configuration, and how to force the adapter-ahead-of-ingestion ordering? The checked-in fixture records OpenCode 2.0.18 with experimental.subagent_depth: 2; it currently passes here. The service-level failure is confirmed, but the real-app reproduction remains unverified.