fix(server): recover Codex child models after metadata lookup fails#22829

draft8

main

fix/issue-15250

opened22d agobyJJulius Marminge

6 files changed

View on GitHubTwo-way sync enabledPR last imported 2d ago

Problem

Fixes #15250.

Change

A native Codex child's first metadata lookup can race an empty rollout. If both read and fallback resume fail, its Lineage model stays “Not reported” even after metadata becomes available. Retry the lookup up to six times with bounded backoff, stopping when the child's actual selection is known or its lifecycle no longer permits recovery. Newer settings/reroute metadata wins over an older lookup; the parent model is never substituted.

Scope and approval

Fixes #15250 GitCafe: https://git.cafe/pingdotgg/t3code/issues/15250 Triaged bug via the automated issue flow.

Evidence index includes correlated native IDs, wire/projection timestamps, exact same-parent task_status errors, card/UI text, completion records, rebuild instructions, temporary fault script/patch, rendering environment and clean restoration proof. Server-only exception: scoped tests and real runtime transcripts replace a pixel matrix. Runtime injection is a published temporary artifact, not committed product code or a committed database seed; the deterministic replay fixture is committed.

Validation: scoped report. Focused adapter suite: 76 passed (prior auditor's run). Scoped formatting passed; lint exited 0 with two unused-variable warnings; server-only typecheck exited 0 with Effect suggestions. Two independent reviews, Codex and Claude, reported MERGE (lead-supplied reports; not rerun by the evidence assembler). At the final lead check, CI reported 21 successful, 2 pending (Collect PR targets and Prepare PR size config), and 0 failing checks. Final proof audit: SUFFICIENT; its timestamp wording finding is corrected in the evidence index.

Verification

Reproduced against the actual base 1688f96568, then verified at 4dd9adf5c0, using Codex 0.163.0 and a real isolated dev server. Parent Sol/low spawned native Luna/high children. Before, a natural empty-rollout read error and a temporarily rejected successful resume result left the child model null through completion. After, attempt 1 hit natural empty-rollout/list_turns errors; attempt 2's successful wire results were rejected by the temporary fault, which then released; attempt 3 read succeeded at 11:35:43.716 UTC. The persisted selection corrected to Luna/high by 43.778 and remained through completion at 11:38:48.129. Healthy control also retained Luna/high through completion. No settings/reroute notification supplied the correction.

Before: failed metadata lookup	After: recovered metadata
Before: native child model Not reported	After: recovered native child GPT-6-Luna high

Rendering: T3 preview Chromium/Linux, 1280×800 CSS viewport, light, measured DPR 2, no emulation, default font scaling. The after image shows the running recovered card; completion retention is established by the persisted snapshot and final UI text. Across captures, isolated history/titles/timers changed, displayed diff counts included temporary instrumentation, and an update notification was dismissed. Product changes are limited to the server adapter and its tests; existing client renderers and contracts are unchanged.

P/I race and interruption behavior are covered by deterministic replay. B/V retry bounds and invalid-result variants are code-inspected with executable coverage limits disclosed in the index. Native task_status still accepts app-owned tasks only; the same-parent native probe returns task_not_found. Unverified: Claude children, mobile card, remote/relay/tunnel and multi-environment behavior. No pixel-diff or completed-card screenshot claim.

Model: GPT-6.1-Sol. Harness: Codex in T3 Code.

Superseded: the evidence index, validation report and Verification section above were captured at 4dd9adf5c0, before the merge from main. The evidence below replaces them. It was recaptured at aa57d7d3ba, and the implementation is unchanged at head f3117126dc, which adds only the evidence scripts and a knip entry for them.

Evidence

Both sides ran a real t3 start server with an isolated --base-dir, the same driver, and a STUBBED Codex app-server (real Codex was rate-limited). Before is the merge-base 8d3b061121; after is aa57d7d3ba. The stub replaces Codex through providerInstances.codex.config.binaryPath, and its thread/read and thread/resume replies follow a fixture. The client value is the subagent model / modelSelection in orchestration_v2_projection_subagents.payload_json, read read-only. The client shows "Not reported" when it is null.

Driver and stub: apps/server/scripts/evidence/issue-15250/ (run.mjs, stub-codex-app-server.mjs, README.md).
Fixture matrix (audit pass 1), captures, raw logs, audit pass 2.

In the table, R is thread/read and S is thread/resume. Times are seconds after the child registered.

Row	Stub fixture	Before `8d3b061121`	After `aa57d7d3ba`	Client value, before → after
R1	Lookup succeeds first time	1R 0S	1R 0S	`gpt-6-luna` → `gpt-6-luna`
R2a	Attempt 0: R and S fail, later R succeeds	1R 1S, then nothing	2R 1S; retry R at +1.04 succeeds	`null` (Not reported) → `gpt-6-luna`
R3	R and S always fail	1R 1S	6R 6S at 0 / 1.04 / 3.04 / 7.05 / 15.05 / 31.05, then quiet for 68 s, including after the parent turn completed	`null` → `null` (stable, no retry storm)
R3b	R always hangs	1R, no S	6R, no S (0 → 56 s)	`null` → `null`
R4a	Child interrupted after attempt 0	1R 1S	1R 1S; no retry at +1 s	`null` (interrupted) at both
R5	`model/rerouted` arrives mid-retry	1R 1S	1R 1S; no further lookups	`gpt-5.6-sol`, one update, at both
R6	R returns an empty model, S succeeds	1R 1S	1R 1S	`gpt-6-luna` at both

Environment: fs-24, Linux 6.1.102 x86_64, node v26.12.1. Ports 127.0.0.1:43832 (before) and 43833 (after). Stubbed Codex app-server, isolated homes under /tmp.

What else changed between before and after: git diff --stat 8d3b061121 f3117126dc lists CodexAdapterV2.ts, CodexAdapterV2.test.ts, the three evidence files and one line in knip.jsonc. Nothing else.

Verification log

Audit pass 1 (Claude Opus 5.5): derived the matrix R1–R7 from the diff and named how a running server reaches each row. It raised one design question, R4f: retries stop once a child ends interrupted, failed or cancelled. A child that ends before any lookup succeeds therefore never shows its model. Audit pass 2 calls this unchanged from before. Audit pass 1 reports one race where it differs: when the child is interrupted while a successful lookup is in flight, before applies the model and after drops it. The PR's interruption test asserts the after behaviour on purpose. A maintainer should confirm it's intended.
Audit pass 2 (Claude Opus 5.5): SUFFICIENT. It checked every row against the raw results.json and checked both diff claims. It rebuilt R2a and R3 from the committed run.mjs on fresh checkouts at 8d3b061121 and 89d72c1ff2, and both matched within about 20 ms: R2a went from 1R 1S null to 2R 1S gpt-6-luna, and R3 went from 1R 1S to 6R 6S. Caveat: the raw logs from the first pass record checkout paths, not SHAs. The rebuild is what ties those results to SHAs.
Tests: vp test run apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts passed 78/78 at aa57d7d3ba. vp lint and vp fmt --check on the changed files show no errors, only two existing unused-variable warnings in CodexAdapterV2.ts. knip --include files,dependencies and the apps/server exports pass both exit 0 at f3117126dc.
CI: at 89d72c1ff2, 25 checks passed and 2 failed: Lint and Check. The local knip run flagged the two evidence scripts as unused files; f3117126dc adds them as knip entries and knip now passes locally. CI is pending on f3117126dc.
Not verified:
- R2b, R2c, R4b–R4e, R6b and R7: not required, and capture ran out of budget. Each is a stub fixture away.
- R4f and R5b: timing-sensitive races, not attempted on a running server. R5b is covered by an existing unit test.
- No web or mobile screenshot: no browser use was authorised. The client value comes from the projection row the clients render.
- Not tested against real Codex, which was rate-limited.
- Remote, relay and tunnel connections, and multiple environments, were not tested.
Model: Claude Opus 5.5 (lead, auditors, capturer). Harness: Claude Code in T3 Code.

Checks

5 failing · 17 successful · 14 skipped

Some checks failed

Failing5

Build and TypecheckGitHub
Failed
CheckGitHub
Failed
LintGitHub
Failed
Test Server 3GitHub
Failed
Test WebGitHub
Failed

Added 2 commits22d ago

test(server): add takeover tests for native Codex child model recoveryc0abf4f
fix(server): retry native Codex child model lookup4dd9adf

Ggithub-actionsBot22d ago

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Codex	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Codex	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Codex	Live turn WebSocket decoded	21.6 KiB	21.6 KiB	0 B (0.0%)	24.4 KiB	✅
Codex	Live turn messages	2	2	0 (0.0%)	8	✅
Claude	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Claude	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Claude	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Claude	Live turn WebSocket decoded	21.9 KiB	21.9 KiB	0 B (0.0%)	24.4 KiB	✅
Claude	Live turn messages	2	2	0 (0.0%)	8	✅

Baseline: f6503f0 · PR result: d3d0823 · Source CI: failure

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

Codex decoded thread snapshot: 108.5 KiB
Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Added 6 commits9d ago

Merge branch 'main' into fix/issue-15250635ad67
Merge remote-tracking branch 'origin/main' into fix/issue-15250aa57d7d
test(server): add runtime evidence driver for #1525089d72c1
chore(server): register #15250 evidence scripts as knip entriesf311712
Merge remote-tracking branch 'origin/main' into HEADe51cff5
Merge branch 'main' into fix/issue-15250d3d0823