Sign up
View stargazers13Sign up

[Bug]: Service unit stays pinned to the runtime that installed it, so deleting old runtimes breaks the next restart#17490

opened16d agobySSvejoDev

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. On Linux, run t3 service install with version A. The systemd unit gets ExecStart=~/.t3/runtime/versions/A/t3 __service-launcher.
  2. Update the server several times with Update server from a client (A → B → C → D). Each update hands off to the running launcher. service-state.json ends up with activeVersion: D and update.fromVersion: C.
  3. Delete old runtime directories, keeping only those service-state.json refers to (active plus fromVersion), the retention rule proposed in #8345 and the workaround people describe there. Version A is deleted.
  4. Wait for the service process to exit. For us, an OOM kill. systemctl --user restart t3code.service does the same.

Expected behavior

The service restarts on the active runtime. The unit should not depend on a runtime directory that nothing in service-state.json records.

Actual behavior

systemd cannot run the unit's ExecStart binary, so every restart fails with 203/EXEC until the start limit is hit. The server stays down until someone edits the unit or runs t3 service install with the active version.

The unit still pointed at 0.0.43-nightly.20260917.1851 three weeks and several remote updates later. The launcher had been running from that directory the whole time, so nothing showed the problem until the process restarted.

Cause on main @ ec80933ac8cd02fec5c97b342462ccc9567cdb1e:

  • bootService.ts#L607-L630: the unit's program is pinnedRuntimePaths(..., input.cliVersion), the runtime of the CLI that ran install.
  • selfUpdate.ts#L313-L332: a remote update calls launcher.requestUpdate and never re-renders the unit. Only t3 update from the CLI does (cli/update.ts, target.install(...)).
  • service-state.json records activeVersion and update.{fromVersion,targetVersion} but not the version the unit executes, so any cleanup based on that file, manual or automatic, can delete it.

Possible fixes, for maintainers to choose: re-render the unit (without restarting) after a remote update commits, or record the unit's runtime version in service-state.json so retention keeps it. Either should be settled before #8345's pruning ships, since its proposed rule would delete this runtime.

Impact

Major degradation or frequent failure

Version or commit

0.0.46-nightly.20261008.2819 (active); unit pinned to 0.0.43-nightly.20260917.1851. Code references are to main @ ec80933.

Environment

Ubuntu 24.04.4 LTS, x86_64, kernel 6.8.0, systemd 255, systemd user service, self-contained runtime. Connected from the macOS desktop app over Tailscale Serve.

Logs or stack traces

shell
1$ systemctl --user cat t3code.service | grep ExecStart
2ExecStart=~/.t3/runtime/versions/0.0.43-nightly.20260917.1851/t3 __service-launcher
3
4$ ls ~/.t3/runtime/versions
50.0.46-nightly.20261005.2676 0.0.46-nightly.20261005.2702 0.0.46-nightly.20261007.2787 0.0.46-nightly.20261008.2819
6
7$ cat ~/.t3/runtime/service-state.json
8{ "protocol": 3, "activeVersion": "0.0.46-nightly.20261008.2819",
9 "update": { "fromVersion": "0.0.46-nightly.20261005.2702", "targetVersion": "0.0.46-nightly.20261008.2819", "status": "committed" } }
10
11$ journalctl --user -u t3code.service
1214:41:39 t3code.service: Failed with result 'oom-kill'.
1314:41:44 Started t3code.service - T3 Code server.
1414:41:44 t3code.service: Main process exited, code=exited, status=203/EXEC
15...
1614:42:10 t3code.service: Start request repeated too quickly.
1714:42:10 Failed to start t3code.service - T3 Code server.

Workaround

Point ExecStart at the active runtime and restart:

sh
1sed -i 's#versions/0.0.43-nightly.20260917.1851/t3#versions/0.0.46-nightly.20261008.2819/t3#' ~/.config/systemd/user/t3code.service
2systemctl --user daemon-reload && systemctl --user reset-failed t3code.service && systemctl --user start t3code.service

Running t3 service install with the active runtime's CLI should also re-render the unit. Before deleting old runtimes, check which one the unit's ExecStart uses.

Related: #8345 (runtime accumulation and the retention rule in step 3); #14609 (remote self-update doesn't update the CLI symlink; same pattern of the remote path skipping what t3 update does, but a different artifact).

Investigated and filed by Claude Code (claude-opus-5-5).

Sspiky02plateau14d ago
Synced from GitHub

Same on macOS with the launchd service.

The LaunchAgent ProgramArguments still points at the runtime that ran t3 service install on Sep 25, after several remote updates since:

shell
1$ /usr/libexec/PlistBuddy -c "Print :ProgramArguments:0" ~/Library/LaunchAgents/com.t3tools.t3code.service.plist
2~/.t3/runtime/versions/0.0.43-nightly.20260925.2251/t3
3
4$ jq '{activeVersion, update: {fromVersion: .update.fromVersion, status: .update.status}}' ~/.t3/runtime/service-state.json
5{ "activeVersion": "0.0.46-nightly.20261011.2967",
6 "update": { "fromVersion": "0.0.46-nightly.20261008.2813", "status": "committed" } }

The launcher process runs from 0.0.43-nightly.20260925.2251, the server from 0.0.46-nightly.20261011.2967. ~/.t3/runtime/versions holds 20 runtimes, 6.8 GiB. Pruning by the #8345 rule (active plus fromVersion) would delete the runtime the LaunchAgent executes, so the next launchctl kickstart or reboot would fail the same way.

Version: 0.0.46-nightly.20261011.2967, macOS 26.5.2, Apple Silicon.

JJasonHassold11d ago
Synced from GitHub

Ran into this right after cleaning up old runtimes to work around #8345. Deleting the old versions removed the one the service unit pointed to, and T3 stopped connecting until I reinstalled the service.

Seems like these two should be fixed together, so that cleanup never deletes the version the service is using.

JJulius Marminge3d ago

Reproduced on Ubuntu/systemd using 70e5e384da, built as three local version labels, in an isolated service account. Installed 0.1.1, then used Settings → Connections to update to 0.1.2 and 0.1.3. Both committed; the unit still executed 0.1.1 while state recorded only 0.1.3 and 0.1.2. Removing 0.1.1 and restarting produced ExecMainStatus=203 / 203/EXEC.

Current automatic cleanup preserves the running launcher, so it did not cause this failure; state-only manual cleanup still does. Runtime transcript. macOS was not verified.

JJulius Marminge3d ago

Fix in #23247. On 0df1ba9eae, the three focused test files pass 63/63. The isolated Ubuntu/systemd runtime check retained launcher A after remote updates through C and state-based cleanup, then restarted successfully into server C.

Existing installations need a service reinstall with the fixed CLI and a restart into the fixed launcher before relying on launcherVersion; remote updating alone does not migrate older launchers. macOS launchd was not verified.

Aarchitpai3d ago
Synced from GitHub

Two more data points, one on macOS (launchd) and one on Ubuntu (systemd). Both show a user-visible symptom that is not listed here yet: t3 service status reports a false "needs repair".

Symptom

1T3 Code service
2 Status: needs an update or repair
3 Next: Run `t3 service install` to repair it.

The server itself was healthy on both machines (port 3773 returned 200, and clients connected).

Environment

macOS 27.0 arm64 (launchd)Ubuntu, kernel 7.0.0 x64 (systemd --user)
Unit pinned to0.0.46-nightly.20261013.3053 __service-launcher0.1.1-nightly.20261020.3386 __service-launcher
Server actually running0.1.2-nightly.20261021.3486 t3 serve0.1.2-nightly.20261021.3486 t3 serve
Update pathin-app/remote updates only0.1.1 → 0.1.2-nightly.…3473 → …3486 (remote), service-state.json shows activeVersion 3486, status: committed

On macOS, the plist EnvironmentVariables.PATH also still had a stale runtime dir (versions/0.0.46-nightly.20261004.2657, a different build from the one in ProgramArguments). In addition, ~/.local/bin/t3 still pointed at 0.0.46 (#14609). Running that old CLI against the newer DB printed Database migration history diverges from this build (migrations 62–67 unknown).

What did and did not help

  • t3 service restart (run with the 3486 binary): no change. The status still says "needs an update or repair", because restart does not re-render the unit.
  • t3 service install (run with the 3486 binary): fixed on both machines. The status is now installed · t3@0.1.2-nightly.20261021.3486, and the launcher and server both run from 3486.

As far as I can tell, the status check (bootService.ts) compares the rendered unit with the unit on disk. So the stale ExecStart/ProgramArguments is the only failing condition. This matches the root cause above. Users will probably see this status message before they hit the 203/EXEC failure. It would be good if the fix for this issue also made the status say why (for example, "service unit points at runtime X, active is Y").

Diagnosed with t3 triage (Claude Code, Opus 5.5) on both machines. I wrote this comment with Claude Code.