fix(server): keep Codex overlay SQLite in the shared home#21947

closed5

main

codex-09/sqlite-home

View on GitHubTwo-way sync enabledPR last imported 18h ago

Managed Codex accounts keep authentication in private overlays, but linking database files into those overlays can split SQLite database and WAL ownership. Default overlay SQLite storage to the shared home and stop mirroring root SQLite database and sidecar families. Empty or whitespace environment values use that default; nonempty overrides and native configuration precedence remain intact.

Refuse legacy SQLite links before shadow writes, executable acquisition or managed sign-in. Preserve every database, sidecar and link for manual recovery, and carry fixed recovery guidance with the original diagnostic cause through provider boundaries. This intentionally interrupts startup for older linked overlays on every platform, including Linux/macOS; human approval of that upgrade policy remains required. Both stack layers need to land together for terminal coverage and recovery instructions.

Rebuilt from Timpan4's #19861. The complete original was compared with the replacement: canonical storage/provider safety is here; terminal selection, typed terminal errors and user recovery guidance are in #21948. Helper-specific suites and redundant matrices were discarded: 119 added cases across eight files became 44 behavioral cases across six existing files, plus the settings fixture update.

Validation: 365 tests pass across six focused suites, plus the scoped settings case; scoped lint/format and server typecheck pass. The retained tests fail on unchanged production. Real Codex 0.156.1, using actual managed-runtime configuration, completed simultaneous synthetic turns with distinct threads in one canonical database and private authentication. Native controls verified explicit/configuration precedence and empty/whitespace defaults. Installer acquisition and credentials were injected; inference was loopback-only. The upper layer additionally has current-head native proof of terminal-selected home defaults.

Not verified: real OAuth, Windows split-WAL/Known Folder behavior, macOS, packaged Electron/SEA, UI or mobile. The Linux WAL-preservation test does not reproduce Windows split WAL. Preflight does not coordinate later filesystem changes. Shared/contracts typechecks and server export checks passed earlier on unchanged code.

Models/harness: GPT-6 Astra tests; GPT-6.1 Sol implementation; independent Opus 5.5 and GPT-6 Sol reviews, through T3 Code.

Fixes #20261

Checks

23 successful · 15 skipped · 1 neutral

All checks passed

Added 5 commits23d ago

test(server): cover blank Codex SQLite home overrides1a22f41
fix(server): keep Codex overlay SQLite storage canonical24df5fb
fix(server): default blank Codex SQLite overrides to shared home7bb8b1a
refactor(server): classify Codex recovery by its structured causef13b0c7
test(server): cover canonical Codex SQLite storage and recoveryffe1f97

MmacroscopeappBot23d ago

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — The change modifies the default SQLite location for Codex auth overlays and intentionally interrupts startup when legacy links are detected, affecting existing managed and native runtime paths. It also introduces a static-analysis suppression in a test, so the behavioral and diagnostic-policy changes warrant human review.

^{You can add or adjust custom eligibility rules. Learn more.}

Ggithub-actionsBot23d ago

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Codex	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Codex	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Codex	Live turn WebSocket decoded	20.9 KiB	20.9 KiB	0 B (0.0%)	29.3 KiB	✅
Codex	Live turn messages	2	2	0 (0.0%)	8	✅
Claude	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Claude	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Claude	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Claude	Live turn WebSocket decoded	21.2 KiB	21.2 KiB	0 B (0.0%)	29.3 KiB	✅
Claude	Live turn messages	2	2	0 (0.0%)	8	✅

Baseline: b34501a · PR result: 7bb8b1a · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

Codex decoded thread snapshot: 108.5 KiB
Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

CcoderabbitaiBot23d ago

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.config.ts
Review profile: CHILL
Plan: Team
Run ID: 0473e1fc-190d-4b89-bfba-e9718eb679f4

📥 Commits

Reviewing files that changed from the base of the PR and between b34501a466d9e6302c41c88a69eca2770a5c5995 and 7bb8b1a74a2a36d6d5ccac8589bd91c73256704e.

📒 Files selected for processing: 10

apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts
apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts
apps/server/src/provider/CodexManagedRuntime.test.ts
apps/server/src/provider/CodexManagedRuntime.ts
apps/server/src/provider/Drivers/CodexDriver.test.ts
apps/server/src/provider/Drivers/CodexDriver.ts
apps/server/src/provider/Drivers/CodexHomeLayout.test.ts
apps/server/src/provider/Drivers/CodexHomeLayout.ts
apps/server/src/provider/Drivers/CodexManagedProvider.ts
apps/server/src/textGeneration/CodexTextGeneration.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

📝 Walkthrough

Codex home preparation now detects SQLite-related symlinks in shadow homes, avoids linking SQLite files, and reports recovery guidance for affected setup paths. Codex environment handling also normalizes home variables on Windows and sets the SQLite home for auth-overlay mode.

Changes: Codex SQLite shadow-home handling

Layer / File(s)	Summary
Home layout rules and SQLite inspection `apps/server/src/provider/Drivers/CodexHomeLayout.ts`, `apps/server/src/provider/Drivers/CodexHomeLayout.test.ts`	Environment helpers normalize Codex home keys on Windows and set a shared SQLite-home default in auth-overlay mode. Shadow-home materialization detects SQLite symlinks and excludes SQLite files and sidecars from linking. Tests cover regular SQLite files, legacy and dangling links, and live WAL data.
Runtime and driver setup `apps/server/src/provider/CodexManagedRuntime.ts`, `apps/server/src/provider/CodexManagedRuntime.test.ts`, `apps/server/src/provider/Drivers/CodexDriver.ts`, `apps/server/src/provider/Drivers/CodexDriver.test.ts`	Runtime and driver setup use Codex-specific environment helpers and detect recovery-required errors. Setup errors use fixed recovery guidance instead of exposing home paths, while retaining causes. Tests cover recovery failures and environment expectations.
Adapter and provider error surfaces `apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts`, `apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts`, `apps/server/src/provider/Drivers/CodexManagedProvider.ts`, `apps/server/src/textGeneration/CodexTextGeneration.ts`	Adapters and managed providers apply Codex environment merging and recovery guidance. Adapter tests cover error details and spawned environment values. Runtime-resolution text-generation errors now retain their cause.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

mermaid

1	sequenceDiagram
2	participant CodexManagedRuntime
3	participant CodexHomeLayout
4	participant NodeFilesystem
5	CodexManagedRuntime->>CodexHomeLayout: inspectCodexShadowSqlite(layout)
6	CodexHomeLayout->>NodeFilesystem: read shadow-home entries and check links
7	NodeFilesystem-->>CodexHomeLayout: return entries or filesystem error
8	CodexHomeLayout-->>CodexManagedRuntime: return inspection result or recovery error

Possibly related PRs

pingdotgg/t3code#19861: Addresses the Codex SQLite shadow-home behavior, including SQLite-home environment handling and rejection of legacy SQLite links.

Suggested reviewers: maria-rcks

Merge Risk: ⚪ Minimal · up to 7bb8b

The change keeps Codex SQLite storage in the shared home and refuses legacy SQLite links with recovery guidance. No merge-blocking issue was identified. The author notes that the intentional startup interruption for older linked overlays still needs human policy approval.

Pre-merge checks: ^Passed 3 · ^Failed 1

^Failed Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Description check	Warning	The description provides detailed problem, implementation, verification, and limitation information. However, it does not use the required section headings and does not provide explicit maintainer app…	Add the required Problem, Change, Scope and approval, and Verification headings. In Scope and approval, link the triaged issue or include the explicit maintainer approval comment for this direction and scope. If the fix qualifies as an obvi…

^Passed Passed checks: 3 passed

Check name	Status	Explanation
Title check	Passed	The title clearly and concisely describes the primary change: keeping Codex overlay SQLite storage in the shared home.
Linked Issues check	Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	Passed	Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description provides detailed problem, implementation, verification, and limitation information. However, it does not use the required section headings and does not provide explicit maintainer approval for the stated cross-platform startup interruption policy.

Resolution

Add the required Problem, Change, Scope and approval, and Verification headings. In Scope and approval, link the triaged issue or include the explicit maintainer approval comment for this direction and scope. If the fix qualifies as an obvious focused bug fix, explain that exemption instead.

Fix all pre-merge checks with AI

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

Ddarkov1919d ago

Windows data for the "Not verified: Windows split-WAL" gap, from a managed ChatGPT account (T3-owned shadow home), plus a check that CODEX_SQLITE_HOME covers every Codex database on Windows.

Environment: Windows 11 x64 (build 26300), T3 Code Nightly 0.1.3-nightly.20261023.3553, Codex CLI 0.164.0. One managed Codex instance; its shadow home is <t3-userdata>/providers/codex/<instance>/shadow.

Failure. The provider card showed "Could not check Codex right now. Retry, or reconnect in provider settings." The server trace had:

1	ProviderDriverError: Provider driver 'codex' failed to create instance '<instance>': Failed to probe Codex skills for '<project>'
2	[cause]: ProviderSetupError: Cannot create Codex shadow home entry 'thread_history_1.sqlite-shm' because '<shadow>\thread_history_1.sqlite-shm' already exists and is not a symlink.

Every base *.sqlite in the shadow home was a link into ~/.codex. Ten sidecars were regular files: -shm/-wal for goals_1, logs_2, memories_1, queue_1 and thread_history_1. Two WALs were nonempty: logs_2 (4.6 MB) and thread_history_1 (3.4 MB). The shared thread_history_1.sqlite-wal was a separate 9.9 MB file at the same time, so both homes had their own WAL for the same database.

Recurrence. I quit T3, moved the 10 regular sidecars aside (no deletion) and restarted. Within about 14 seconds the managed Codex had recreated regular -shm/-wal files in the shadow home for goals_1, logs_2, memories_1 and queue_1. thread_history_1 and state_5 got sidecar links instead, because the shared home had those sidecars when the shadow home was rematerialized. So on Windows, which family ends up split depends on which shared sidecars exist at materialization time. Moving the files aside only clears the error until the next run.

CODEX_SQLITE_HOME on Windows. I ran codex app-server (0.164.0) with CODEX_HOME and CODEX_SQLITE_HOME set to two empty directories, then sent initialize, thread/start and one turn/start. Codex was pointed at a closed loopback openai_base_url, so no inference ran. Results:

CODEX_SQLITE_HOME received all six databases with their sidecars: goals_1, logs_2, memories_1, queue_1, state_5 and thread_history_1 (thread_history_1 appears only after a turn starts).
CODEX_HOME received no *.sqlite* files at all.

This supports #21947's direction for Windows. With overlay SQLite in the shared home, Codex never opens a database through a shadow-home link, so SQLite has nothing to place beside a link. That doesn't apply to #23472 alone: as noted above, keeping the regular sidecars leaves the split WAL in place on Windows.

Not checked: Windows without Developer Mode, the sqlite_home config key (only the env var was tested), and older Codex versions. I can share the probe script if it helps.

JJulius Marminge1d ago

This appears redundant with #29574, which addresses the same split SQLite/WAL defect and includes terminal and legacy-overlay coverage. This earlier PR has not been updated since October 19, and main has since changed its shadow-home code. Thanks to Julius and Timpan4 for the earlier work, and to nkcbuilds for carrying the fix forward. The PR remains open because it belongs to an active stack.

JJulius Marminge1d ago

This is a duplicate of #29574, where nkcbuilds carries the SQLite/WAL fix through provider terminals and legacy-overlay recovery too. This earlier branch has not been updated since October 19, and main has since changed the shadow-home code. GitCafe requires handling the active stack before this PR can be closed. Thanks, Julius and Timpan4, for the earlier work and investigation.

JJulius Marminge1d ago

Thanks for this. #29574 makes the same change and is the one being kept.

#29574 carries the same shared SQLite storage fix through provider terminals as well, covering a launch path this PR leaves to its companion layer. It also refuses ambiguous mixed SQLite state before changing the overlay.

Your shared-storage approach and preservation of legacy recovery data remain in that replacement. Keeping the complete fix together gives us one implementation to review.

Closing as a duplicate.