[Bug]: Codex shadow home splits each SQLite database from its WAL, corrupting shared state and disabling the default Codex instance#24486

opened19d agobyMMans230

Before submitting

I searched existing issues and did not find a duplicate.
I included enough detail to reproduce or investigate the problem.

Scope. This is the step after #5817 and its closed duplicates #17005 / #20261. Those cover materializeCodexShadowHome rejecting a regular shadow entry and disabling that instance. Here materialization succeeds (or is bypassed entirely) and the shared database is then corrupted, because the base file is shared while the -wal/-shm are not. The failure then surfaces on a different instance — the default codex provider, which has no shadow home at all — and repeats every few minutes.

Area

apps/server

Summary

shadowHomePath (mode authOverlay) mirrors every top-level entry of the shared Codex home into the instance home as a symlink, including the SQLite state databases.

SQLite resolves the base file through that symlink, but derives the -wal / -shm names from the path it was handed. So each shadow instance writes its own transaction log beside the link while all instances append to one shared base file:

1	~/.codex/state_5.sqlite one physical database, 3 live connections
2	~/.codex-t3/codex-2/state_5.sqlite-wal instance 1 transaction log
3	~/.codex-t3/codex-3/state_5.sqlite-wal instance 2 transaction log

Two connections to one database with different WAL directories each see a stale snapshot and checkpoint over the other's pages. Codex detects the result as a damaged database, tries to move it into db-backups\sqlite-*, fails because a sibling process holds the file, and exits 1.

Steps to reproduce

Documented multi-account layout: shared home ~/.codex, second account with config.shadowHomePath: ~/.codex-t3/codex-2.
Start the shared codex instance and the shadow instance and let both run (each app-server opens state_5.sqlite through its own home).
SQLite creates state_5.sqlite-wal / -shm in the shadow home as regular files; the shared home keeps its own pair for the same base file.
Reload providers or restart T3 Code.
The instance that owns the home Codex tries to repair exits 1; T3 shows it as failed.

Observed on the default codex instance:

1	Codex app-server provider probe failed: Codex App Server process exited with code 1.

Running the probe by hand showed why:

1	Codex local database at C:\Users\Administrator\.codex\state_5.sqlite appears damaged.
2	Moving it into a backup folder so the app server can rebuild it from saved data.
3	Error: failed to initialize sqlite state runtime under C:\Users\Administrator\.codex:
4	failed to move damaged sqlite state database files into a backup folder:
5	The process cannot access the file because it is being used by another process. (os error 32);
6	original error: failed to initialize state runtime at C:\Users\Administrator\.codex

At that moment ~/.codex/state_5.sqlite could not be opened for write by any other process, and the only Codex process alive was a codex app-server child of the T3 server. After stopping it and giving each instance its own SQLite files, the default instance probed ready on the next rebuild.

Expected behavior

One physical SQLite database must have exactly one -wal/-shm pair, shared by every connection that opens it. "Keep auth.json private while sharing state" must not put two live connections on one database file with different WAL directories.

Actual behavior

The shared database is repeatedly damaged and rebuilt.
~/.codex/db-backups accumulated six sqlite-* folders in 20 minutes:

1	sqlite-1792856184-0 2026-10-24 08:36:24
2	sqlite-1792857124-0 2026-10-24 08:52:04
3	sqlite-1792857124-1 2026-10-24 08:52:04
4	sqlite-1792857163-0 2026-10-24 08:52:43
5	sqlite-1792857295-0 2026-10-24 08:54:55
6	sqlite-1792857388-0 2026-10-24 08:56:28

File sizes confirm the split: shared state_5.sqlite 667,648 B with a 4,177,712 B -wal; shadow codex-2 state_5.sqlite 368,640 B with its own 4,148,872 B -wal; shadow codex-3 had a symlinked state_5.sqlite and a regular 4,165,352 B -wal.
A shadow instance can read the database without the shared instance's committed frames. On the same installation the shadow state_5.sqlite-wal was 0 bytes while the shared home held 86,552 bytes of WAL frames for that database (also noted in #20261).

Impact

Silent corruption of the Codex state database, plus a rebuild loop while both instances are in use.
Cross-instance outage: a shadow-home instance disables the default provider, which is not configured with a shadow home at all. Here codex showed status: error while codex-2 / codex-3 still reported ready, which makes it look like an unrelated auth or CLI problem.
Only multi-account users are affected, and they stay affected permanently, because every Codex run recreates the split sidecars.

Version or commit

T3 Code (Nightly) 0.1.3-nightly.20261023.3553, server.asar 114,821,431 bytes.
Codex CLI 0.163.0, T3-managed (~/.t3/tools/codex/0.163.0/bin/codex.exe).

Environment

Windows Server 2022 (10.0.20348, AMD64), T3 Code desktop nightly with the local server.
Shared home C:\Users\Administrator\.codex; instance homes C:\Users\Administrator\.codex-t3\codex-2 and codex-3.
Instance config: { "driver": "codex", "displayName": "Codex 2", "enabled": true, "config": { "shadowHomePath": "~/.codex-t3/codex-2" } }. No binaryPath, no setupMode.
codex login status → Logged in using ChatGPT for all three homes.

Root cause

apps/server/src/provider/Drivers/CodexHomeLayout.ts:

1	const resolveCodexHomeLayout = fn("resolveCodexHomeLayout")(function* (config) {
2	const sharedHomePath = resolveHomePath(path, home, config.homePath);
3	const shadowHomePath = config.shadowHomePath.trim();
4	if (shadowHomePath.length === 0) return {
5	mode: "direct",
6	sharedHomePath,
7	effectiveHomePath: config.homePath.trim().length > 0 ? sharedHomePath : void 0,
8	continuationKey: `codex:home:${sharedHomePath}`
9	};
10	return {
11	mode: "authOverlay",
12	sharedHomePath,
13	effectiveHomePath: path.resolve(expandHomePath(shadowHomePath, home)),
14	continuationKey: `codex:home:${sharedHomePath}`
15	};
16	});

materializeCodexShadowHome then links the shared home's entries into effectiveHomePath. Nothing excludes the *.sqlite families, so the databases become links.

The SQLite behaviour is the part that makes this unrecoverable. Opening a database through a symlink does not relocate the sidecars: -wal/-shm are derived from the path passed in, so they land in the shadow home. The base file is shared; the write-ahead log is not. WAL is a per-database recovery log, so the pairing must never be split.

Minimal proof, no T3 involved (Node 24 built-in node:sqlite):

1	const fs = require("node:fs");
2	const { DatabaseSync } = require("node:sqlite");
3
4	const shared = new DatabaseSync("base/state.sqlite");
5	shared.exec("PRAGMA journal_mode=WAL; CREATE TABLE t(a INTEGER); INSERT INTO t VALUES (1);");
6
7	// what materializeCodexShadowHome creates
8	fs.symlinkSync("base/state.sqlite", "shadow/state.sqlite", "file");
9
10	// what the shadow instance's app-server then opens
11	const shadow = new DatabaseSync("shadow/state.sqlite");
12	shadow.exec("INSERT INTO t VALUES (2)");
13	// SqliteError: no such table: t

The second connection cannot see the first connection's committed row, because that row is only in base/state.sqlite-wal and it is looking for shadow/state.sqlite-wal.

Why the existing sidecar work does not cover this

REPLACEABLE_SHARED_RUNTIME_ENTRIES and the \.sqlite-(shm|wal)$ predicate discussed in #5817 only decide whether ensureSymlink replaces a regular shadow entry instead of throwing. That removes the conflict error. It does not change the fact that the base file is shared while the WAL is not — it makes the split permanent instead of fatal.
I verified this directly: with a local build that treats every conflicting shadow entry as replaceable, codex-2 and codex-3 both reported ready and no longer logged CodexShadowHomeEntryConflictError — while state_5.sqlite kept being rebuilt into db-backups and the default codex provider died with exited with code 1. The symptom disappeared; the corruption did not.
#19861 ("keep Codex SQLite state in one home") addressed exactly this and was closed on 2026-10-19 without merging, so #5817 currently has no implementation in flight.

Proposed fix

Either is sufficient; both keep one physical database with one WAL:

Point SQLite at the shared home for auth overlays. CODEX_SQLITE_HOME is present in the pinned Codex binary (0.163.0 contains CODEX_SQLITE_HOME and the sqlite_home config key), so the child environment for authOverlay instances can set it to sharedHomePath. CODEX_HOME stays the shadow home, so auth.json / models_cache.json remain private, while the database and its WAL live together in the shared home. This is what #19861 did.
Do not mirror the SQLite families at all, and let each instance own its databases in effectiveHomePath. Cheaper, and it matches the fact that state_5.sqlite is private runtime state rather than shared user data; sessions/, skills/, plugins/, config.toml stay shared through links.

Whichever is chosen, the guard should be structural rather than a filename allowlist:

No *.sqlite / *.sqlite-wal / *.sqlite-shm / *.sqlite-journal (any case, future names) may be a symlink inside a shadow home.
Pre-existing shadow homes that already contain such links must be refused with an actionable message, not silently re-pointed — pointing the base file at the shared home while leaving a private WAL is the corruption this report describes.
A regression test asserting that state_5.sqlite and state_5.sqlite-wal resolve to the same directory for every instance.

Workaround

Stop the instances, then give each account its own SQLite state while keeping sessions and configuration shared:

diff

1	"codex-2": {
2	"config": {
3	- "shadowHomePath": "~/.codex-t3/codex-2"
4	+ "homePath": "~/.codex-t3/codex-2"
5	}
6	}

resolveCodexHomeLayout then returns mode direct, so nothing is mirrored. Remove the SQLite links from the instance homes first (fs.unlinkSync on the links only — never on the targets) and let Codex rebuild each database from the saved session data. auth.json, models_cache.json and every *.sqlite* become private; sessions, skills, plugins, config.toml keep working through the existing links.

After doing that here: codex, codex-2, codex-3 all ready with 7 models, codex app-server against ~/.codex starts with empty stderr, and no new db-backups\sqlite-* folder has appeared since.

#5817 — materialization conflict / provider disabled (open; different failure point).
#17005, #20261 — closed as duplicates of #5817.
#19861 — proposed fix, closed unmerged.

MMoazMustafa-stack19d ago

Confirming this on a second Windows 11 Home install (single shadow-home instance, ~/.codex-t3/personal, Codex signed in with ChatGPT).

Startup failure. The instance is disabled with:

1	Driver 'codex' failed to create instance: Cannot create Codex shadow home entry 'memories_1.sqlite-shm' because '...\.codex-t3\personal\memories_1.sqlite-shm' already exists and is not a symlink.

The same message later named logs_2.sqlite-shm. Every SQLite database in the shared home produces it: goals_1, logs_2, memories_1, queue_1, state_5, and thread_history_1 all got regular -shm/-wal files beside their symlinks.

It recurs. Moving the strays aside lets the instance start, but they come back within minutes while T3's codex.exe app-server is alive, and the next materialization fails again. Cleaning only holds with T3 fully closed.

Corruption, as described here. On the last occurrence the shadow home also contained a regular state_5.sqlite (~1.69 MB) that diverged from the shared one (~1.68 MB), and ~/.codex/db-backups/ gained four sqlite-* rebuild folders in about 14 hours (Oct 24 18:26, 22:44 x2, and Oct 25 00:41).

Suggested direction (matches the expected behavior above): one physical database must have exactly one -wal/-shm pair. Either don't symlink the SQLite files into the shadow home (use a separate per-instance database, or point Codex's SQLite directory at the shared home), or never create a second connection path to the shared files. Failing that, materialization should tolerate and clean up sidecar files instead of disabling the instance.

Kkyzer102312d ago

Confirming the recurring shadow-home startup failure on another Windows installation.

Environment:

Windows 11 Pro, build 26200 (x64).
T3 Code desktop 0.1.5.
Codex CLI 0.165.0.
An additional managed ChatGPT/Codex provider, alongside the default Codex provider.

The additional provider still showed an authenticated account, but its status was “Could not check Codex right now. Retry, or reconnect in provider settings.” The default Codex CLI remained signed in.

The server trace identified the underlying failure:

text

1	CodexShadowHomeEntryConflictError: Cannot create Codex shadow home entry
2	'goals_1.sqlite-shm' because '<managed-shadow>/goals_1.sqlite-shm'
3	already exists and is not a symlink.

At failure:

goals_1.sqlite in the shadow home was a symlink to the shared Codex home.
The shadow goals_1.sqlite-shm was a regular 32,768-byte file.
The shadow goals_1.sqlite-wal was a regular, empty file.
Corresponding SHM/WAL files existed in the shared home.

Backing up those two local sidecars and restoring their links to the shared counterparts allowed T3's shadow-home materialization, provider probe, and ChatGPT model discovery to succeed again. The existing sign-in was retained.

This has happened before on this installation, so the manual repair is only recovery from the current failure. No claim of observed database corruption here; the confirmed symptoms are the recurring filesystem conflict and provider-check failure. The SQLite storage change proposed in #24426 appears relevant.

Follow-up: testing of #24426 at 1831f29a0c2a49e7005e9a7a24d6b8c2eccbd0c2 is reported on the PR. The three focused suites passed (19 tests passed, 12 skipped), but a separate synthetic Windows/SQLite test still reproduced split WAL ownership for an existing symlinked database.

HHaget-55211d ago

Confirming this on another Windows install. It happened three times in one day (November 1, 2026) and disabled the secondary instance each time.

Environment

Windows 11 Pro, build 26300 (x64), Developer Mode on.
T3 Code desktop, executable version 0.1.2.
Codex CLI 0.164.0.
Two Codex instances: the default codex (shared home ~/.codex) and a second ChatGPT account (codex_<uuid>). The second one first used setupMode: "managed", and later used a plain shadowHomePath pointing at the same per-instance shadow directory under userdata/providers/codex/<id>/shadow.

What happened The second instance was disabled or failed to open its session with:

1	CodexShadowHomeEntryConflictError: Cannot create Codex shadow home entry 'memories_1.sqlite-shm' because '…\providers\codex\codex_<uuid>\shadow\memories_1.sqlite-shm' already exists and is not a symlink.

The entry it named changed between failures: goals_1.sqlite-shm (Oct 25), memories_1.sqlite-shm (twice), and state_5.sqlite-shm. Each time the shadow directory had a symlinked X.sqlite next to regular-file X.sqlite-shm / X.sqlite-wal. The sidecars had been created after materialization, apparently whenever the matching sidecar was missing from ~/.codex at materialization time. In every case I checked, the WAL was 0 bytes.

In one case the conflict showed up as "Provider session failed to open" in the thread. In another, the provider card showed "Disabled" even though the toggle was on.

Workaround applied (not a fix) I moved the regular sidecar files out of the shadow home. Then I created file symlinks for every -shm / -wal pointing at ~/.codex with mklink (PowerShell's New-Item -ItemType SymbolicLink requires admin even with Developer Mode), including links whose targets did not exist yet. After that, materialization succeeded and the instance came back. I expect this to break again, because SQLite deletes sidecars on close, and through the shadow path that removes the link rather than the target.

Expected A shadow instance should never create private SQLite sidecars for a shared database, and an empty or regenerable -shm / -wal should never disable the instance.

Nnkcbuilds10d ago

I opened #29574 with a fix for this, including the existing-overlay case that remained reproducible on #24426.

The key difference is the upgrade path: new managed shadows no longer mirror SQLite files and use a canonical CODEX_SQLITE_HOME, while an existing Windows shadow is inspected before materialization changes anything. A clean legacy DB symlink can move forward using the shared SQLite home, but a mixed layout such as thread_history_1.sqlite -> shared plus a real shadow thread_history_1.sqlite-wal/-shm is refused with recovery guidance rather than deleting or silently abandoning possible uncheckpointed state.

Validation on Windows included the exact linked-DB/private-sidecar failure, a real uncheckpointed node:sqlite WAL with byte-for-byte preservation assertions, provider-status/auth preservation, token-renewal regressions, and the installed Codex CLI 0.166.0. With the old symlink layout Codex recreated shadow WAL/SHM; with the proposed CODEX_SQLITE_HOME routing it produced zero shadow SQLite sidecars.

PR: #29574

Ssuper1486661d ago

Confirming this on another Windows install. It happens every time T3 Code restarts.

Environment

Windows 11 Pro, build 26300 (x64)
T3 Code (Nightly) desktop
A shadow-home Codex instance at ~/.codex-t3/personall, alongside the default Codex instance and the Codex desktop app (which use ~/.codex directly)

Startup failure

1	Driver 'codex' failed to create instance: Cannot create Codex shadow home entry 'goals_1.sqlite-shm' because 'C:\Users\<user>\.codex-t3\personall\goals_1.sqlite-shm' already exists and is not a symlink.

Files that had become real files in the shadow home (the base .sqlite files were all still symlinks):

goals_1.sqlite-shm, goals_1.sqlite-wal
memories_1.sqlite-shm, memories_1.sqlite-wal
state_5.sqlite-shm, state_5.sqlite-wal

All the -wal files were 0 bytes, so moving them out was safe. The same thing happened earlier the same day with goals_1, memories_1 and queue_1. After I moved those out, queue_1's sidecars were re-linked as symlinks, because they existed in ~/.codex at link time. goals_1's were recreated as real files, because they didn't. So whether a restart fails depends on which sidecars happen to exist in ~/.codex when T3 materializes the shadow.

The same setup on macOS never hits this, which fits SQLite following symlinks on Unix but not on Windows. Looking forward to #29574.

IInvixGG1d ago

Confirming on the stable 0.1.8 release with Codex CLI 0.167.1, on Windows x64. The setup is one shadow-home instance next to a direct instance on %USERPROFILE%\.codex.

Regular files in the shadow home, next to symlinked base DBs:

goals_1.sqlite-shm / goals_1.sqlite-wal
logs_2.sqlite-shm / logs_2.sqlite-wal
memories_1.sqlite-shm / memories_1.sqlite-wal
queue_1.sqlite-shm / queue_1.sqlite-wal

state_5.sqlite had diverged further. The base DB itself was a real file in the shadow home, with its own -shm/-wal, about a week old. So the shadow instance had been on a separate state DB from the shared one, without any error. It surfaced only when an unrelated db-backups/ conflict disabled the instance (details on #5817). I didn't capture when the base file stopped being a link.

macOS, same T3 and Codex versions: in the shadow home, every *.sqlite-wal/-shm is still a symlink into ~/.codex, and the only real files are the private and shadow-local entries (auth.json, auth.lock, models_cache.json, log, memories, tmp). That matches the unix VFS resolving the link before it names the sidecars.

In 0.1.8, materializeCodexShadowHome still links only the -wal/-shm files that exist in the shared home at setup time, so nothing has changed here since the earlier reports. #29574's approach (one CODEX_SQLITE_HOME for every instance, with no SQLite links in the shadow home) would remove this whole class of problem.

MMrSco1d ago

Confirming on the current stable release, with a worse outcome than a rebuild loop: thread_history_1.sqlite became malformed and its thread_items table is unreadable.

Environment

Windows 11, T3 Code 0.1.9, bundled Codex CLI 0.166.0.
Two Codex accounts. One uses %USERPROFILE%\.codex directly. The other is a separate provider instance, so T3 runs it from %USERPROFILE%\.t3\userdata\providers\codex\<instance-id>\shadow.

Steps and evidence

In the shadow home nearly every entry is a symlink to the same name in .codex. That includes state_5.sqlite, thread_history_1.sqlite and logs_*.sqlite, plus whichever -wal/-shm files existed in .codex at setup time.
When a sidecar was missing at setup, Codex in the shadow home creates it as a regular file beside the link. A Codex process that exits without a clean close leaves it there.
Every later start of that instance fails:
1 Cannot create Codex shadow home entry 'state_5.sqlite-shm' because '%USERPROFILE%\.t3\userdata\providers\codex\<instance-id>\shadow\state_5.sqlite-shm' already exists and is not a symlink.
Meanwhile the other account writes the same database file through .codex, with its own WAL. For thread_history_1.sqlite:
- Commits on the shadow side grew the database to 61,535 pages and wrote that size into the file header.
- The .codex side later truncated the file to 61,444 pages.
- SQLite now reports the database as malformed. The thread_items table and its indexes can't be read.
- On the same day, every commit in the stale shadow WAL recorded a 61,535-page database.

Impact

The second account's provider can't start until the regular sidecars are removed by hand.
Shared Codex thread history is permanently damaged, beyond what Codex rebuilds automatically.

Root cause (apps/server/src/provider/Drivers/CodexHomeLayout.ts on main, identical in v0.1.9)

L391–408: the entries are a snapshot of readDirectory(sharedHome) minus a short denylist (L33, L35). Each *.sqlite file is linked on its own; nothing treats a database and its sidecars as one unit.
L251–260: a regular file at a shared name throws CodexShadowHomeEntryConflictError unless the name is in REPLACEABLE_SHARED_RUNTIME_ENTRIES (L37–41).
Whether a sidecar becomes a link or a real file depends only on whether it existed in .codex at that moment.
SQLite documents the result: one file opened under two names gets two WAL files, with "behavior that is undefined". Only the unix VFS resolves symlinks to the canonical name (since 3.10.0), which is why macOS isn't affected (howtocorrupt §2.6).

Suggested fix: the approach in #29574.

Codex 0.166.0 honours CODEX_SQLITE_HOME and the sqlite_home setting (codex-rs/state/src/lib.rs:145, codex-rs/config/src/config_toml.rs:376). Every runtime database sits at the root of that directory (codex-rs/state/src/sqlite.rs:48-53).
Pointing it at the shared home and never linking *.sqlite* keeps each database together with its single WAL, while auth.json stays private.
Linking sidecars ahead of time can't fix this, because SQLite deletes and recreates -wal/-shm, and deleting through the link removes the link.
Existing shadow homes where linked databases sit next to regular sidecars should be refused with recovery steps, as #29574 does, not cleaned up automatically. A shadow WAL can hold committed frames.

Workaround: stop T3 and every Codex process, then move (don't delete) the regular *-wal/*-shm files out of the shadow folder.

This only clears the startup error. On the next setup, state_5's sidecars came back as links because they existed in .codex at the time.
thread_history_1 had no sidecars in .codex, so it got no links and is still exposed.

Minor, separate: the shadow folder also contains a dangling link, .codex-provisioning-<hash>.guard. Codex's Windows sandbox creates that randomly named, delete-on-close file in .codex (codex-rs/windows-sandbox-rs/src/no_reparse_dir.rs:211). Setup linked it while it existed, and nothing removes links whose target is gone.

Aaniketshaw748-hub8h ago

Confirming on the current nightly, Windows 11 Home.

Environment

T3 Code desktop nightly 0.1.10-nightly.20261112.4580, local server
Codex CLI 0.167.2
Windows 11 Home build 26300 (x64)
Default codex instance on ~/.codex, plus a second managed ChatGPT instance (codex_<uuid>, shadow under userdata/providers/codex/<id>/shadow)

Symptom Sign-in for the second account completes (OAuth callback page says it's finishing the connection; settings show "Signed in as …"), but status stays at "Needs attention · Could not check Codex right now. Retry, or reconnect in provider settings."

Trace materializeCodexShadowHome / CodexHomeLayout.ensureSymlink fail about every 5 minutes, continuously from 18:24 to 19:08 UTC on 2026-11-12, alternating between two entries:

1	CodexShadowHomeEntryConflictError: Cannot create Codex shadow home entry 'logs_2.sqlite-shm'
2	because '<shadow>\logs_2.sqlite-shm' already exists and is not a symlink.
3	CodexShadowHomeEntryConflictError: Cannot create Codex shadow home entry 'state_5.sqlite-shm'
4	because '<shadow>\state_5.sqlite-shm' already exists and is not a symlink.

Shadow layout at failure

Base DBs goals_1, logs_2, memories_1, queue_1, state_5 .sqlite are all symlinks to ~/.codex.
Their -shm (32,768 bytes) and -wal (0 bytes) are regular files, all -wal created at the same moment.
thread_history_1.sqlite-shm/-wal are symlinks (they existed in ~/.codex at link time), matching the "depends on which sidecars exist when linking" observation above.
~/.codex currently has only logs_2 and thread_history_1 -shm files and no db-backups/, so no rebuild corruption yet on this machine.

Workaround Quit T3, move the regular *.sqlite-shm/-wal files out of the shadow dir, reopen. Waiting on #29574 for the real fix.

Diagnosed and written by Claude Opus 5.5 (Claude Code) via t3 triage.

[Bug]: Codex shadow home splits each SQLite database from its WAL, corrupting shared state and disabling the default Codex instance#24486

Before submitting

Area

Summary

Steps to reproduce

Expected behavior

Actual behavior

Impact

Version or commit

Environment

Root cause

Why the existing sidecar work does not cover this

Proposed fix

Workaround

Related