Sign up
View stargazers13Sign up

Multiple servers share one --base-dir and each resumes the others' live threads#14115

opened26d agobyMmchisolm0

What happened

Continuing an existing Claude thread produced several agents working at once. The agent reported that multiple copies of the session were running, and the thread looked like it spanned both old and new threads. First noticed from the iOS app; the thread below was started from a Tailscale-connected MacBook and ran on a Mac mini.

That machine had four T3 Code servers running against one --base-dir. Three of them independently spawned their own agent process for the same thread.

Diagnosis

Nothing enforces that one --base-dir is served by one server. When several are, each spawns its own provider session for the same thread, and the user sees one of them.

Verified by parentage, not inferred. All three of these are live claude processes for one thread on one day, each with a different server as its parent:

1pid ppid started parent
216705 28646 09:56:40 desktop app (Nightly), port 3776
363219 53721 10:58:03 CLI 0.0.43-nightly.20260928.2375, port 3774
481957 35731 11:17:03 CLI 0.0.43-nightly.20260922.2123, port 3775

How several servers come to share a base dir

Two routes observed, both unguarded.

Route 1: the manual update command starts a server

apps/web/src/versionSkew.ts:117 hands users whose server cannot update itself:

ts
1/** The command to hand users whose server cannot update itself. */
2export function manualServerUpdateCommand(targetVersion: string): string {
3 return `npx t3@${targetVersion}`;
4}

That has no subcommand, and at apps/server/src/bin.ts:57 the root command runs a server:

ts
1Command.make("t3", { ...sharedServerCommandFlags }).pipe(
2 Command.withDescription("Run the T3 Code server."),
3 Command.withHandler((flags) => runServerCommand(flags)),
4 Command.withSubcommands([...]),
5);

So the documented way to update a server starts an additional one instead. Following it once per nightly built a four-server ladder.

Route 2: two launches race and both win

On a second machine (Linux), two servers of the same build run against one base dir, started three seconds apart, both orphaned to PPID 1, with no service unit installed and no manual update involved. Nothing was updated by hand there, so a fix aimed only at Route 1 would not reach this machine.

Why both routes succeed silently

In apps/server/src/cli/config.ts (resolveServerConfig), web mode resolves an unspecified port with findAvailablePort(ServerConfig.DEFAULT_PORT), and packages/shared/src/Net.ts:195 falls through when that port is busy:

ts
1findAvailablePort: (preferred) =>
2 Effect.gen(function* () {
3 if (preferred > 0 && (yield* isPortAvailableOnLoopback(preferred))) {
4 return preferred;
5 }
6 return yield* reserveLoopbackPort();
7 }),

A busy default port is the normal signal that a server already serves this base directory. It is treated only as a port collision: the new server takes the next free port and serves the same database, with no warning. I searched apps/server and packages for a lock file, port registry or instance id and found none, so the one-server-per-base-dir assumption appears unenforced rather than enforced-and-broken.

One mechanism by which a second server adopts a live thread

In apps/server/src/serverRuntimeStartup.ts:535 (reconcileProviderSessions):

ts
1const orphanedThreads = threads.filter(
2 (thread) =>
3 thread.session !== null &&
4 (thread.session.status === "starting" ||
5 thread.session.status === "running" ||
6 thread.session.activeTurnId !== null ||
7 (thread.session.status === "ready" && preparedThreadIds.has(thread.id))) &&
8 !liveThreadIds.has(thread.id),
9);

liveThreadIds comes from providerService.listSessions(), the calling process's own sessions. Shared state records no server identity for a thread, so a thread genuinely live in server A is indistinguishable from an orphan when server B looks at it.

This accounts for adoption at server startup. It does not account for every instance I observed: on the affected thread, four sessions began within 7 milliseconds (16:16:52.391/.396/.396/.398) at a moment when no server was starting. I could not identify the code path for that and am not going to guess at one.

The state is not self-correcting

apps/server/src/cli/update.ts:601 only warns:

1A server started by hand is still running at {origin} (pid {pid}).
2Stop it and start it again to pick up {targetVersion}.

It never stops it, and findForegroundServer locates the stray through readPersistedServerRuntimeState, which holds a single pid, so with three strays it reports at most one. I cleaned this condition up by hand around Sep 11; every server now running postdates that cleanup.

Cost

Duplicate resumes are not free: each re-reads the transcript before doing any work. On the affected thread, 28 session starts cost 2.59M tokens in first-call context reads (median 88k, worst 186k) against a thread total of 40M, so roughly 6.5% of that thread's usage went to resumes, most of them unnecessary.

That is real waste, but it is not the headline, and I want to correct an earlier impression rather than leave it standing: I first attributed a large subscription quota drain to this bug, and measurement did not support it. The dominant cost of these threads is ordinary agent work. The argument for fixing this is correctness, not spend: duplicate agents run real turns, in the same worktree, under bypassPermissions, invisibly to the user.

Steps to reproduce

  1. Start a server: t3 serve --base-dir <DIR> (no --port). It binds the default port.
  2. Run the command the UI gives for a manual server update, npx t3@<version>, or start a second server with the same flags. Expected a refusal or a warning; instead it silently binds the next free port and serves the same state.
  3. Open a thread and submit a turn that runs long enough to still be active a few seconds in.
  4. Observe more than one provider process for that one turn, with different servers as parents (ps -Ao pid,ppid,command | grep claude), and clustered session.started events for the single thread id in <DIR>/userdata/logs/provider/events.<threadId>.log.

Step 2 need not be deliberate: on one machine it was the UI's own update instruction, on another two launches three seconds apart did it with no user action.

Version

Servers 0.0.43-nightly.20260928.2375, .20260922.2123, .20260921.2058, and the T3 Code (Nightly) desktop app. Source analysis against main at d15210cd. npx t3 separately reports 0.0.42, the stable latest tag.

Environment

Three machines, each with its own base dir. The correlation with server count is clean:

machineservers on one base dirthreads with clustered session starts
Mac mini (darwin arm64, Darwin 27.0.0, Node v26.8.2)43
Linux desktop21, predating the current pair
MacBook (desktop app only)10

The single-server machine is unaffected. The Linux machine's two servers both started this morning and no turn has been submitted there since, so it is exposed but not yet exercised.

Provider Claude Code (claudeAgent), model claude-opus-5-5[1m], permissionMode: bypassPermissions, defaultThreadEnvMode: worktree. Surfaces: iOS app, a Tailscale-connected MacBook, and the local desktop app. continueThreadsAfterServerUpdate is true on both multi-server machines and unset on the single-server one, so I cannot separate its contribution from server count here.

Evidence

shell
1# Thread 606f5fd7, session.started timestamps across one day.
2# Clusters are tight; the 3.5s group at the top is the reported incident.
314:56:39.236 14:56:40.940 14:56:42.760 3 in 3.5s
414:57:12.950 14:57:12.951 2 in 1ms
515:58:03.085
616:13:35.920 16:13:50.745 16:13:50.745 2 in the same millisecond
716:16:52.391 16:16:52.396 16:16:52.396 16:16:52.398 4 in 7ms
816:17:03.136
9
10# Same thread, the three claude processes still alive, by parent server:
1116705 28646 Sep 28 09:56:40 desktop app (Nightly), port 3776
1263219 53721 Sep 28 10:58:03 CLI 0.0.43-nightly.20260928.2375, port 3774
1381957 35731 Sep 28 11:17:03 CLI 0.0.43-nightly.20260922.2123, port 3775
14
15# The four servers, one base dir (update-command ladder):
1656473 Sep 21 <runtime>/0.0.43-nightly.20260921.2058/t3 serve --port 3773 --base-dir <DIR>
1735731 Sep 23 <runtime>/0.0.43-nightly.20260922.2123/t3 serve --port 3775 --base-dir <DIR>
1853721 Sep 28 <runtime>/0.0.43-nightly.20260928.2375/t3 serve --port 3774 --base-dir <DIR>
1928646 Sep 24 T3 Code (Nightly).app … (port 3776)
20
21# Linux machine: two servers of ONE build, three seconds apart, no user action:
222464729 1 Sep 28 09:20:23 <runtime>/0.0.43-nightly.20260928.2375/t3 serve --port 3773 --base-dir <DIR>
232465796 1 Sep 28 09:20:26 <runtime>/0.0.43-nightly.20260928.2375/t3 serve --port 3774 --base-dir <DIR>
24LISTEN 127.0.0.1:3773 pid=2464729
25LISTEN 127.0.0.1:3774 pid=2465796
26
27# Worst single incident, Sep 14, a different shape: 10 session.started
28# over 67s at even ~10s spacing, more like a restart loop than simultaneity.
2916:16:51.709 … 16:17:58.924
30
31# Provider processes orphaned by superseded servers, still alive days later:
32 4388 35731 Sep 24 14:52 claude … (3d 19h)
3317524 56473 Sep 24 20:09 claude … (3d 13h)
3451566 56473 Sep 27 01:21 claude … (1d 08h)
3554163 28646 Sep 24 15:10 claude … (3d 18h)

Counting method: a thread is counted only when two or more session.started events fall within 60 seconds of each other. Threads with several starts spread over hours are normal returns to idle work and are excluded; counting those inflates the numbers several-fold.

All four servers also interleaved writes into one server.trace.ndjson, rotating it ten times in one day.

Related issues

#5447 is the nearest existing report and is not a duplicate: there the extra session is a claude-haiku-4-5 title-generation helper executing the quoted prompt, spawned by a single server. Here there is no helper: full claudeAgent sessions run on the thread's configured model, spawned by distinct server processes sharing a base directory. Searches for multi-instance, shared base-dir, stale-server-after-update and orphan-reconcile phrasings returned no other matches.

Fix applied or workaround

Stopped the two superseded CLI servers on the four-server machine, leaving the current nightly CLI and the desktop app's embedded server. That is still two servers on one base dir, so the condition is reduced, not removed; fully clearing it means not running the desktop app and a CLI server against the same base dir at all. No source was patched and no database writes were made.

This is a delay rather than a workaround. I did the same cleanup around Sep 11 and the ladder rebuilt itself within days, because the update instruction regenerates it.

Suggested directions, in the order I would expect them to help:

  • Make npx t3@<version> not start a server. Either give the root command a help or update handler, or have manualServerUpdateCommand emit npx t3@<version> update.
  • Refuse to start, or attach to the existing server, when another already serves the requested --base-dir. A busy default port should not be resolved by silently choosing another. A lock file in the base dir would also cover the launch race, which no fix to the update command can reach.
  • Record an owning server identity (pid plus boot id, or a lease in shared state) on running sessions, and exclude threads owned by a live server from the orphan reconcile.
  • Stop the superseded server and its provider children when applying an update, rather than printing a warning about one of them.

Filed by

claude (Opus 5) via t3 triage, Claude Code harness.

Mmchisolm026d ago
Synced from GitHub

Follow-up repro on the Mac mini: after stopping the two older CLI servers, only the current nightly CLI server and the Nightly desktop app's embedded server remained, sharing the same base directory. From the MacBook's desktop app, connected remotely to the Mac mini, I resumed an existing thread hosted there.

That one resume produced two session.started events 2.7 seconds apart. The corresponding Claude provider processes were direct children of the two different Mac mini servers. No server launch or update was needed for this turn. Two servers on the host were enough to reproduce the duplicate agents; the MacBook was the remote client.

Ff4llenz26d ago
Synced from GitHub

Another data point for several servers on one base dir, from a route the issue does not list: a launchd service plus a desktop-managed SSH launch.

Version 0.0.43-nightly.20260926.2282, macOS. A Mac mini was the remote machine and the desktop app ran on a MacBook.

  • t3 service (launchd) and the desktop's SSH environment (t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3, recorded as managed in ~/.t3/ssh-launch/<id>/) ran against the same ~/.t3.
  • The SSH launch did not reuse the service and started a second server. REMOTE_LAUNCH_SCRIPT in packages/ssh/src/tunnel.ts reuses a server only when server-runtime.json names a live loopback server that answers a readiness probe within REMOTE_REUSE_READY_TIMEOUT_MS (2 s). I did not capture why that check failed here.
  • When the service restarted and 3773 was taken, findAvailablePort (packages/shared/src/Net.ts) moved it to a random port.
  • Each server ran its own cloudflared tunnel, and both tunnels had the same tunnel name.
  • Both servers resumed the same threads. There was a claude --resume <same session> process under each server, so two agents worked in the same git worktrees.
  • Afterwards server-runtime.json was missing while one server was still running. The runtime-state cleanup in apps/server/src/server.ts removes the file on any server's shutdown without checking who wrote it.

Workaround: run exactly one server per base dir. Stop or remove one of the two.

Related: #5749, #6097, #8442.

Mmchisolm026d ago
Synced from GitHub

I reduced the affected machines to one server per base dir. A thread that had refused new prompts because another session was running now accepts prompts and completed a new turn. Thanks @f4llenz for calling out the workaround.

BBlackskyliner25d ago
Synced from GitHub

Note: This comment was drafted with AI assistance (Claude, Opus 5.5 Medium) from my own process listings and runtime files. I reviewed it before posting. All data below comes from my machine.

Additional reproduction: Linux remote host over SSH, 3 servers on one base dir

Same condition as described here, on a remote dev host.

  • Remote: Ubuntu 22.04.5 LTS, kernel 5.15.0-191-generic
  • Client: T3 Code (Nightly) desktop on macOS, 0.0.43-nightly.20260929.2428
  • Connection: desktop SSH remote (the client socket on the remote is owned by sshd)
  • continueThreadsAfterServerUpdate: true (in ~/.t3/userdata/settings.json)

Servers (all PPID 1, all --base-dir ~/.t3)

1PID PPID STARTED BUILD PORT
23016299 1 Sat Sep 19 08:10:16 0.0.43-nightly.20260917.1880 3773
31386590 1 Wed Sep 30 06:08:11 0.0.43-nightly.20260929.2428 3774
41843403 1 Wed Sep 30 13:05:43 0.0.43-nightly.20260929.2428 3775
  • 3016299: superseded build, still alive 11 days later.
  • 1386590 / 1843403: same build as the desktop client, started ~7 h apart with no update in between. The second server was not update-triggered. This matches the SSH reconnect pattern in #13521 / Route 2 here.

Runtime file tracks only the newest server

1~/.t3/userdata/server-runtime.json
2{"version":1,"pid":1843403,"port":3775,"startedAt":"2026-09-30T13:05:51.911Z"}

Only 1843403 had an established client connection. The two older servers are no longer referenced anywhere visible, but they keep running and keep their provider sessions.

The UI-connected server had zero provider children

Every live claude process belonged to a server the UI was not connected to:

1PID PPID STARTED --resume
21026748 3016299 Tue Sep 29 12:33:04 a18d9006-…
31543595 3016299 Wed Sep 30 09:19:55 5ab40021-…
41593209 3016299 Wed Sep 30 10:52:33 1cecb2ee-… <- duplicate
51769801 1386590 Wed Sep 30 11:53:16 e3904573-…
61769822 1386590 Wed Sep 30 11:53:16 1cecb2ee-… <- duplicate
  • Session 1cecb2ee-… was resumed by two different servers, about 1 h apart, in the same worktree.
  • 1769801 and 1769822 started in the same second under 1386590. This fits a startup reconcile adopting threads that 3016299 still owned.
  • Each session's MCP endpoint (--mcp-config … http://127.0.0.1:<port>/mcp) points at its own parent server, not at the server the UI uses.

User-visible symptom

The thread shows as running, but no output arrives in the UI. The work continues invisibly.

Ghost sessions execute real work

1593209 (duplicate of 1cecb2ee, parent 3016299) was still spawning shell commands after the newest server started:

11602146 1593209 10:58:57 bash … until docker images … | grep -q docling-worker; do sleep 5; done
21844685 1593209 13:06:00 bash … until <glab api pipelines/816/jobs …>; do sleep 30; done …

The session was polling a CI pipeline and waiting on Docker images under bypassPermissions, in a worktree also used by the second copy. None of this was visible in the UI.

Workaround until now

killall claude on the remote plus a restart of T3 Code. This clears the provider processes but not the stale servers, so the condition recurs.

Targeted cleanup: keep only the server named in server-runtime.json and stop the others including their children (pkill -TERM -P <pid>; kill -TERM <pid>).

Observations relevant to a fix

  1. server-runtime.json being overwritten by each new server means the launcher and the update path lose track of older servers entirely. This reinforces the suggestion to use a lock file or lease instead of a single-pid record.
  2. On an SSH remote, stale servers accumulate across both updates (build 20260917 still alive) and reconnects (two servers of the client's own build). A fix only for npx t3@<version> would not cover this host.
  3. continueThreadsAfterServerUpdate: true here as well. This is consistent with the correlation in the original report, but I cannot isolate its contribution.
  4. A server with a connected client and zero provider sessions, while other servers hold that thread's sessions, is a detectable state. It could at least surface a warning in the UI.
AAlexBabescu16d ago
Synced from GitHub

Confirmed another occurrence during t3 triage on 2026-10-08.

A Linux background service running 0.0.46-nightly.20261008.2819 crashed at 21:16:21 UTC after a browser drag timeout. The service restarted at 21:16:26. Meanwhile, the macOS desktop client reconnected over SSH and launched 0.0.46-nightly.20261008.2833 on port 3774 at 21:16:30.

Read-only process and file-descriptor inspection confirmed both servers running concurrently:

ServerPIDPortVersion
systemd user service29060613773.2819
Desktop SSH launcher29070683774.2833

Both processes had the same <T3_HOME>/userdata/statev2.sqlite, its WAL, and its shared-memory file open. Desktop tunnel logs confirmed the client used port 3774. The service was still enabled for startup.

With the user's approval, ran systemctl --user disable --now t3code.service. Port 3773 closed. The port 3774 server kept its original PID and returned HTTP 200.

The crash recovery cancelled four running threads. Recent conversation data remained in storage. We have not proved that the competing servers caused the user's apparent history reversion.

Environment: Linux x86_64, kernel 7.0.0-38-generic; macOS arm64 client, Darwin 27.0.0; Codex provider; SSH connection. No source patch or direct database write was made.

Investigated by Codex, GPT-6, via t3 triage.

VViperJuice13d ago
Synced from GitHub

Read-only exact-release corroboration for pingdotgg/t3code#14115 from Consiliency/dotfiles#90, nightly 0.0.46-nightly.20261011.2980 (ed0f7b908b2e7b3a22e6fd82a73794c2fe6ee06e): the user-service MainPID and CLI are both this release, while the shared runtime descriptor names a different live same-UID process on a different loopback port and lacks serviceManaged. The intended service listener remains correctly loopback-only and owner-owned. No restart, process termination, descriptor edit or database change was performed by this audit.

The exact-tag server.ts persists its selected address to the single config.serverRuntimeStatePath on activation; release cleanup calls clearPersistedServerRuntimeState on that same path. The exact-tag serverRuntimeState.ts persist/clear functions do not protect the file with an instance-ownership comparison. This explains the descriptor health failure without treating its envelope version: 1 as an application release.

We are keeping service health admission closed. Does an upstream-supported attended route exist to hand the descriptor back to the persistent service after all competing-environment work has finished, without a restart or manual descriptor edits? Otherwise the single-server service/T3 Connect recipe in the linked discussion appears to require a coordinated owner-client handoff and an attended service restart. Collaborators here use a separate login identity and must retain that separation.

There is a separate new restart-admission gap in pingdotgg/t3code#18284: unfinished guide generation (including superseded heads) is not globally exposed by the existing database/per-PR inspection APIs. That must also be accounted for before declaring this runtime idle. No private infrastructure identifiers, credentials or process arguments are included here.

VViperJuice13d ago
Synced from GitHub

Exact-source reconnect clarification at ed0f7b908b2e7b3a22e6fd82a73794c2fe6ee06e (nightly2980): packages/ssh/src/tunnel.ts:641-649 stops the previously launched pid when the shared descriptor names any ready loopback server and launch state is managed. It does not compare the descriptor pid with its own launched pid. Thus a reconnect can stop the SSH-launched server even while the record still names that same server; a descriptor change is unnecessary. The resulting restart can drop in-memory review-guide work, including superseded runs, and does not repair service ownership of the shared record.

This was independently checked against public source and the verbatim launch-script control flow in disposable isolation; no live reconnect or process stop was performed. Until a supported repair is available, our owner selected preserving running work and waiting for complete all-generation inspection (pingdotgg/t3code#18284), with no restart/source activation. Consiliency/dotfiles#95 is being corrected to preserve existing connections until complete idle evidence is available and avoid new connections/reconnects; reconnect is not an activity check.

Additional lifecycle qualification: the desktop setEnabled(false)/remove-route path reaches disconnectSsh, disconnectEnvironment and REMOTE_STOP_SCRIPT (packages/client-runtime/src/connection/registry.ts:1073-1079, packages/ssh/src/tunnel.ts:1722-1752, stop script:737-758). A deliberate disconnect kills its launched managed server without an activity check. Stale-tunnel recovery also runs the stop path before relaunching. Thus do not tell an owner with running work to disconnect as a preventive measure. Leave existing connections unchanged until a complete qualified idle inspection passes; a dropped transport alone is not idle evidence. This was checked against exact public source and the verbatim stop script in disposable isolation, without a live desktop disconnect.

Vvirenpepper12d ago
Synced from GitHub

Another data point: a macOS host running both the desktop app and the background service on one ~/.t3, with a second MacBook connected as a peer.

Drafted with Claude from my own process listings and transcript; I reviewed it.

pidparentstartedclaude args
74633T3 Code (Alpha).app (desktop, embedded server)Oct 11 11:08--session-id=afe81ad2…
57708t3 serve 0.0.45 (service, via t3 __service-launcher)Oct 12 11:48--resume=afe81ad2…

Both processes wrote to the same Claude transcript (one afe81ad2….jsonl). From then on, every prompt from either machine reached both.

One detail that isn't in this thread yet, and is why it was costly: at the moment the second server resumed the session, the transcript recorded the first process's still-running background tasks (a workflow and a subagent) as stopped / "didn't finish", and the thread's original first prompt was queued again. The resumed agent read that as an unfinished task and restarted the same multi-agent workflow against the same git checkout, so two sets of agents edited one working tree until one of them noticed.

A prompt from the peer while the desktop-owned session was idle did not spawn a duplicate. The duplicate appeared while that session was mid-turn with background tasks running.

JJulius Marminge11d ago

Note

Grok responding on behalf of Julius.

Fixed by merged #16102: servers now take an exclusive lock on the state directory before opening the database, so multiple servers can no longer share one --base-dir / T3 home.