[Bug]: `t3 update` across a service-launcher protocol bump leaves the service crash-looping ("Service state is invalid or unsupported") while reporting success#12627

Triage

Confirmed against current apps/server. This is a real bug, not a duplicate of #11934.

#11940 gated the in-client update path when the running launcher is behind SERVICE_LAUNCHER_PROTOCOL (now 3) and documented local t3 update as the way to replace the launcher. That CLI path has no protocol check, and the service switch runs in the old process.

What the code does

1. t3 update (apps/server/src/cli/update.ts) installs the target archive, then calls BootService.install({ start: true }) in this process with cliVersion: targetVersion. The comment at ~L546 is explicit: it does not go through the target’s own CLI (so downgrades to older command sets still work).
2. install() (apps/server/src/cloud/bootService.ts ~L861) writes ~/.t3/runtime/service-state.json as { protocol: SERVICE_LAUNCHER_PROTOCOL, activeVersion: cliVersion } — protocol is the running CLI’s compile-time constant — and points the unit at the new t3 __service-launcher.
3. The new launcher’s readServiceState / decodeServiceState accept only its own protocol and throw Service state is invalid or unsupported. There is no 2→3 migration. launchd/systemd then respawn every 5s. install() treats a successful bootstrap/start as done and t3 update prints success.

That matches the leftover document:

serviceStateActiveVersion / serviceStateHasPendingUpdate already read across protocol revisions for downgrade/pending-update guards. Decode used at launch does not. Same write path on Linux (systemd), even though this report is launchd/macOS.

apps/server/src/cli/update.test.ts does not cover a protocol bump. Remote preflight tests do (servicePreflight.test.ts blocks protocols 1 and 2).

Workaround (confirmed by the report)

After PATH is repointed at the new binary:

The new CLI rewrites service-state.json with protocol 3 and the service comes up.

Suggested fix

• Have the launcher accept and migrate compatible older state (2→3 is the same two fields). That also unsticks hosts already crash-looping.
• And/or have t3 update delegate the service switch to the target CLI when protocols differ, or refuse with the existing preflight message, and not report success unless the service is actually listening.

Do not treat “run t3 update again from 0.0.42” as the fix; that is the failing path.

Accepting as a bug.

Reproduced on Linux (headless systemd user service), today, via npx t3@latest update --channel nightly — plus a note that the in-client prompt path is what got us updating.

Host: NixOS x86_64, background service as systemd user unit t3code.service (ExecStart=…/runtime/versions/<ver>/t3 __service-launcher), headless environment used only remotely from the macOS desktop app (0.0.42).

Sequence:

• Service had been running 0.0.38 for ~1.5 weeks. Desktop client 0.0.42 showed reconnect errors plus an "update remote environment" prompt.
• Clicked the GUI prompt: 16:09:47 unit stops, 16:09:48 back up on 0.0.42 — works fine (relay tunnel connections register, server on 127.0.0.1:3773, GUI usable again). 0.0.38 → 0.0.42 is protocol 2 → 2, so no bump, no breakage.
• Then on the host: npx --yes t3@latest update --channel stable (npm latest = 0.0.42), then npx --yes t3@latest update --channel nightly.
• 16:10:19 unit stops. ~/.t3/runtime/service-state.json left as { "protocol": 2, "activeVersion": "0.0.43-nightly.20260929.2428" }, ExecStart repointed at the nightly binary, update reports success.
• 16:10:21–16:10:37 four start attempts, each exits 1 in ~0.4 s with [service-launcher] Service state is invalid or unsupported.; 16:10:42 start-limit-hit (StartLimitBurst=5, RestartSec=5). Nothing on 3773 from that point; the desktop app shows Relay could not reach the environment endpoint (endpoint_request_failed).

Protocol-3 requirement confirmed in the shipped binary (extracted around decodeServiceState in versions/0.0.43-nightly.20260929.2428/t3):

Workaround used (service restored, NRestarts=0, clients pairing again): edit ~/.t3/runtime/service-state.json protocol 2→3, remove the stale ~/.t3/runtime/.service-stopping left by the SIGTERM'd launcher (not verified whether that part matters), then systemctl --user reset-failed t3code.service && systemctl --user start t3code.service. t3 service install from the new binary is the cleaner variant; the file edit is just the minimal one for a headless box.

One extra angle for the fix: npx --yes t3@latest update --channel nightly necessarily performs the switch from the stable CLI (0.0.42 today), so every stable→nightly hop through npm latest hits this until a stable with a protocol-3 launcher ships. Worth erroring out (or delegating to the target CLI) on targetProtocol != runningProtocol.

Hit on the stable channel now: v0.0.43 stable (0.0.42 → 0.0.43) on a Linux systemd user service. Since v0.0.43 went out without #12629, any stable host that follows the in-client message ("This release requires a newer T3 Code service launcher. Update it on the server machine.") ends up here.

Sequence (Ubuntu, kernel 7.0, t3code.service user unit):

1. The desktop client (0.0.43) tried to self-update the remote server and failed with the preflight message above. The server was on 0.0.42 with a protocol-2 launcher.
2. On the host: t3 update 0.0.43 --yes (run through the 0.0.42 CLI at ~/.local/bin/t3) printed Background service restarted on 0.0.43.
3. The unit's ExecStart now pointed at versions/0.0.43/t3 __service-launcher, but service-state.json was { "protocol": 2, "activeVersion": "0.0.43" }. The launcher crash-looped with [service-launcher] Service state is invalid or unsupported., and t3 service status reported service-stopped.
4. t3 service install (now from the 0.0.43 CLI) rewrote the state as { "protocol": 3, "activeVersion": "0.0.43" } and the service came up.

The document matches the no-update-record, two-field case that adoptOlderInstallState in #12629 handles, so that PR would have covered this. Merging it and cutting a patch release would help anyone still on 0.0.42 stable. Until then, the in-client message could say to run t3 service install after t3 update.

Also reproduced on Linux (systemd user service), stable channel, 0.0.42 → 0.0.44, reached from the desktop app:

1. Desktop 0.0.44 → "Update server" on a remote environment running 0.0.42 fails with: Server update failed: This release requires a newer T3 Code service launcher. Update it on the server machine. (ServerSelfUpdateError from server.updateServerWithProgress). The toast gives no command to run.
2. On the server: t3 update 0.0.44 --yes (run from the 0.0.42 binary, the only one installed) prints Background service restarted on 0.0.44.
3. Unit ExecStart now points at versions/0.0.44/t3 __service-launcher, but service-state.json is {"protocol": 2, "activeVersion": "0.0.44"}. The launcher exits 1 with Service state is invalid or unsupported. every 5 s until systemd hits StartLimitBurst and marks the unit failed. Every client is disconnected.

Workaround confirmed: ~/.t3/runtime/versions/0.0.44/t3 service install (the new CLI) rewrites the state as protocol: 3, restarts the service, and the desktop reconnects.

So the documented recovery path the desktop error points to (update it on the server machine) runs straight into this bug for every protocol-2 → protocol-3 upgrade.

Also reproduced on Arch Linux (systemd user service), stable channel, 0.0.42 → 0.0.45 — 2026-10-03. The fix PR has not merged, so current stable still bites.

Host: Arch Linux 7.1.9-arch1-2, x86_64, t3code.service systemd user unit (ExecStart=~/.t3/runtime/versions/<ver>/t3 __service-launcher), StartLimitIntervalSec=300 / StartLimitBurst=5. Accessed remotely from the iOS app and the desktop client over T3 Connect (managed cloudflared tunnel).

Sequence:

1. Host ran 0.0.42 (protocol-2 launcher). t3 update to 0.0.45 stable reported success and repointed the unit: ExecStart=…/runtime/versions/0.0.45/t3 __service-launcher
2. ~/.t3/runtime/service-state.json after the update — exactly the mismatch described above:
   json

   1	{ "protocol": 2, "activeVersion": "0.0.45" }

3. boot-service.log fills with [service-launcher] Service state is invalid or unsupported. ×5, then:

   1	systemd[1239]: t3code.service: Scheduled restart job, restart counter is at 5.
   2	systemd[1239]: t3code.service: Start request repeated too quickly.
   3	systemd[1239]: t3code.service: Failed with result 'start-limit-hit'.
   4	systemd[1239]: Failed to start T3 Code server.

4. Nothing on the local port, Relay client stopped / Released the managed tunnel on shutdown — every paired client dropped.

Two things this adds to the existing reports:

• systemd gives up permanently; launchd does not. The unit ships StartLimitBurst=5 / StartLimitIntervalSec=300, so after five 5s-spaced crashes systemd stops retrying and parks the unit in failed. There is no self-healing and no notification — the outage is silent and total until a human runs systemctl --user reset-failed t3code.service && systemctl --user start t3code.service. Rebooting the host just replays the same five-crash-then-give-up cycle.
• The user-visible symptom is "my phone can't connect to my server." No client-side error names the launcher or the protocol; the mobile app just can't reach the environment, because the relay/managed tunnel is owned by the dead service. Diagnosing it requires knowing to look at the systemd unit and boot-service.log on the server box — which is exactly the machine the user may not be sitting at.

Workaround confirmation (alternative to t3 service install): hand-writing the state document in the shape BootService.install() itself emits is sufficient, no reinstall needed:

Result: service active on t3@0.0.45, HTTP answering on loopback, relay client re-registered 4 tunnel connections, mobile reconnected. This reinforces the suggested launcher-side migration — protocol 2 → 3 is the same two fields, so accepting and rewriting the older document would have made all of this a non-event.

Still unfixed as of today: gh pr view 12629 → state: OPEN, mergedAt: null; latest commit touching apps/server/src/serviceLauncher.ts is d8655ed2 (2026-09-14), i.e. neither v0.0.45 (latest stable, 2026-10-02) nor 0.0.46-nightly.20261003.2638 contains a migration. Every host still on a protocol-2 launcher will hit this on its next update, and the in-client prompt ("This release requires a newer T3 Code service launcher. Update it on the server machine.") actively routes users through the broken CLI path.

_{Triage notes prepared with pi (Claude) on the affected host; evidence collected locally, secrets and home paths redacted.}

Same failure reproduced on Linux (systemd), so it is not launchd-specific.

Two Ubuntu 24.04 hosts, systemd user service, root user, 2026-10-08:

1. Service installed on t3 0.0.42 with t3 service install.
2. Ran /root/.t3/runtime/versions/0.0.42/t3 update 0.0.46-nightly.20261008.2801 --channel nightly --yes. It printed Background service restarted on 0.0.46-nightly.20261008.2801.
3. The 0.0.42 updater wrote ~/.t3/runtime/service-state.json as {"protocol": 2, "activeVersion": "0.0.46-nightly.20261008.2801"} and pointed the unit at the nightly binary.
4. The nightly launcher logged [service-launcher] Service state is invalid or unsupported. five times. systemd then reported "Start request repeated too quickly" and the unit failed. A .service-stopping marker stayed in ~/.t3/runtime/.
5. t3 service status (nightly) printed Status: needs an update or repair ... [service-stopped].

Extra finding: t3 service install with the nightly binary rewrote the state to protocol 3, but it still printed BootServiceCommandError: Background setup failed while starting the service (exit code 1). The repair worked. The start failed because systemd's start limit was still tripped. install does not run systemctl --user reset-failed before it starts the unit.

Workaround that worked on both hosts:

Impact: the remote host stays offline after the update until someone runs these commands over SSH.

Source check: SERVICE_LAUNCHER_PROTOCOL went from 2 to 3 in b84f63bb13 (#11940), and decodeServiceState in apps/server/src/cloud/serviceProtocol.ts rejects any other protocol.

Reported with Claude Code (Sonnet subagent).

Reproduced the launcher failure on current main b5a226e49c on Linux with Node 24.21.0. Ran the real serviceLauncher.main() entry point through node --input-type=module, using temporary T3CODE_HOME directories and the reported version/state. With a complete synthetic runtime and matching .install-complete sentinel:

The protocol-3 control deliberately exits after printing its marker; protocol 2 fails before spawning it. This verifies the launcher rejection, not the full macOS update/success-message sequence. No live service or userdata was changed; launchd was unavailable on this Linux VM.

Existing open fix #19449 addresses this exact legacy install-state failure. No duplicate PR opened. Keeping this issue open: that PR still calls for human approval and real legacy-service/platform validation.