[Bug]: Service update leaves T3 Connect relay targeting stale origin port#7458

I apologize for the lack of human input on this, I have no idea where to even start with this issue

This reproduced again roughly 80 minutes after the first repair, and it also affected a separate desktop-managed macOS environment at the same time.

New evidence:

• The Linux/Coder T3 service remained active and healthy on 127.0.0.1:3773.
• Its managed Cloudflared process remained active, but the relay configuration had changed to another dead ephemeral origin (127.0.0.1:37337).
• Restarting only t3code.service caused CloudManagedEndpointRuntime to create a new tunnel/connector and restored the public relay health endpoint to HTTP 200.
• Separately, the macOS 0.0.33 desktop backend was healthy on local port 3773, but its T3 Connect endpoint returned the same endpoint_request_failed error after the desktop app and connector had been running continuously for about two days.
• Restarting T3 Code desktop triggered CloudManagedEndpointRuntime.reconcileConfig, replaced the managed connector, and restored that relay health endpoint to HTTP 200 as well.

This recurrence suggests the issue is not limited to a partially upgraded systemd service. It appears that relay configuration can drift to a dead ephemeral origin while the actual backend and connector processes remain healthy. No environment IDs, relay hostnames, tunnel IDs, or credentials are included here.

I'm having this same issue, it happens very frequently and have to restart the service to recover. There is some config difference between the server port and the remotely managed tunnel. For me, it was 32913.

Reproduced on a desktop-managed macOS environment, including after updating and restarting into T3 Code Nightly 0.0.36-nightly.20260827.1205.

This case differs slightly from the stale local-origin-port case in the original report: the active managed tunnel is healthy and demonstrably reaches the current backend on port 3773, but relay status/connect still returns endpoint_request_failed without a request arriving at the tunnel.

Reproduction / recovery attempts

1. Confirmed only one embedded T3 server and one managed cloudflared process were running; removed a duplicate backend/tunnel created during troubleshooting.
2. Updated from Nightly 0.0.34 through 0.0.36-nightly.20260827.1205.
3. Performed a complete unlink rather than only disabling the tunnel:
   • disabled Publish agent activity;
   • disabled T3 Connect;
   • verified npx t3 connect status --json returned linked: false, cloudUserId: null, relayUrl: null, and publishing off.
4. Re-enabled T3 Connect, producing a fresh managed tunnel, then restored activity publishing.
5. Fully quit/restarted T3 Code and retried with a freshly reloaded relay-environment descriptor.

The failure persists:

Diagnostics

• T3 backend is listening on the expected port 3773.
• Managed cloudflared 2026.5.2 has four active HA connections.
• cloudflared_tunnel_request_errors is 0.
• A direct HTTPS request to the managed hostname reaches the current backend and returns HTTP 200.
• A direct unauthenticated POST to /api/t3-connect/health reaches the backend and returns the expected HTTP 400, proving the public hostname routes to the live origin.
• After the fresh link/restart, relay status/connect failures do not increment cloudflared_tunnel_total_requests and do not appear in the server trace. This suggests failure/stale state before the relay request reaches the managed endpoint, rather than an origin-port mismatch.
• The same account's mobile client initially showed invalid_dpop; the complete unlink/relink should have replaced that stale credential, but desktop relay connectivity remains blocked by endpoint_request_failed.

Latest trace ID from Nightly 1205:

Earlier trace IDs from the same investigation:

No environment IDs, managed hostnames, tunnel IDs, account IDs, or credentials are included here. This was collected through npx t3 triage --agent codex plus direct local checks.

Reproduced on a plain single-machine Linux setup — no VM/Coder hop and no desktop app involved, so the drift isn't limited to the multi-host update scenarios above.

Environment

• T3 Code 0.0.38 (stable), user-level systemd service (t3code.service, Restart=always)
• Linux x64 (Arch, kernel 7.2.3-arch1-2), Node v26.8.1
• Launch method: plain npx t3 in an interactive terminal (which also manages the background service)

What happened

The background service crash-looped for several restart cycles after a fresh npx t3 install (unrelated root cause — the node-pty install-script issue tracked in #7475), then finally came up and stayed running. That surviving process landed on a different local port than the last time the environment had linked with the relay (the CLI's own port picker falls back to a random available port whenever the default 3773 is occupied, which it was during the crash-loop churn).

From then on:

• The T3 server was demonstrably alive and listening on its new port.
• boot-service.log showed the startup path logging "T3 Connect desired link reconciled on startup" with no error — i.e. the reconcile call it makes on every boot did run and did not fail.
• Despite that, cloudflared kept logging Unable to reach the origin service ... dial tcp 127.0.0.1:<old-port>: connect: connection refused against the previous session's port, indefinitely (observed for 20+ minutes with zero recovery, until we intervened).
• A remote client (a second T3 Code instance connecting through T3 Connect) saw exactly Relay could not reach the environment endpoint (endpoint_request_failed) the whole time.

Fix that worked: systemctl --user restart t3code.service. The restart spawned a brand-new relay-client/tunnel connector (new tunnel ID observed in the logs) which immediately registered against the current port with no further warnings. Same workaround already documented above for the VM/desktop cases.

Possible mechanism (from reading apps/server/src/cloud/ManagedEndpointRuntime.ts and apps/server/src/cloud/http.ts on this version): the per-boot reconcile (reconcileDesiredCloudLink) does re-provision the relay side with the current origin, but CloudManagedEndpointRuntime.reconcileConfig's dedup key (runtimeConfigKey) only hashes providerKind + connectorToken + tunnelId + tunnelName — it never includes the origin host/port. So when the same tunnel identity is reused across a reconcile, the already-running local cloudflared tunnel run process is left untouched even if the origin it should be forwarding to has changed, and there's nothing else here that forces it to reconnect or verifies it actually converged on the new target. That would explain why a full service restart (which tears down and respawns the connector unconditionally) reliably fixes it, while the routine startup-time reconcile does not.

Diagnosed via t3 triage (Claude Sonnet 5).

Reproduced on macOS (launchd service), 0.0.44, with a deterministic trigger

Environment: T3 Code 0.0.44 (CLI, service runtime and desktop app), macOS arm64 (Darwin 27.0.0), Node v26.8.2, cloudflared 2026.5.2 (managed install). Background service via launchd (t3 serve on 127.0.0.1:3773), desktop app with local environment disabled, linked to T3 Connect with a managed tunnel.

Repro:

1. Have the background service running and linked to T3 Connect (tunnel origin 127.0.0.1:3773, remote clients work).
2. In a terminal, run bare t3 as the same user with the default T3CODE_HOME. It starts a second web-mode server on an ephemeral port against the same state dir.
3. Wait ~10 s, then Ctrl-C it.
4. Remote clients now show T3 Connect · Reconnecting: Relay could not reach the environment endpoint (endpoint_request_failed). indefinitely.

Diagnosis: the ad-hoc server shares the service's secrets and environment link, so its startup path (registerManagedCloudTunnelRecovery(localOrigin), apps/server/src/server.ts:868) asks the relay to reconcile the tunnel ingress to its own ephemeral port. When it exits, the ingress stays on that port. The service only registers its origin at startup, so it never corrects it, and its cloudflared child keeps forwarding to the dead port.

Evidence (UTC, 2026-10-01):

• 09:58:14 service: T3 Connect managed tunnel recovery registered (port 3773)
• 09:58:14–09:58:48 trace spans with server.mode=web, server.port=51865; environment.cloud.registerManagedCloudTunnelRecovery succeeds at 09:58:24
• 09:58:48 releaseManagedTunnelOnShutdown fails: upstream_unavailable, trace ID 20b251b9f693c559ae637a4922753a65
• 09:58:54–10:03:35 service's cloudflared: Unable to reach the origin service … dial tcp 127.0.0.1:51865: connect: connection refused … originService=http://127.0.0.1:51865
• 10:03:08–10:03:37 second bare t3 on port 52657; registerManagedCloudTunnelRecovery fails with relay returned HTTP 504 (Ray ID a43aa3258f08fd81-SIN), yet from 10:03:39 the ingress is originService=http://127.0.0.1:52657, so the origin change was applied despite the 504
• lsof: only t3 serve listens, on 127.0.0.1:3773; t3 connect status shows exposure enabled and link provisioned

Workaround: t3 service restart re-registers 3773. Verified here: after the restart at 10:16:16 the service logged T3 Connect managed tunnel recovery registered at 10:17:24 and cloudflared reported no further origin errors.

Suggestions: an ad-hoc t3 should not re-register the managed tunnel origin when a service owns the T3 home (overlaps #6097), and/or the service should re-reconcile its origin when its connector reports repeated connection refused to a port that is not its own.

Produced via t3 triage by Claude Opus 5.5 (claude-opus-5-5) in Claude Code.

Reproduced on Linux (Fedora 44, x64) with the systemd service on 0.0.46-nightly.20261008.2849. In my case the stale port has a concrete trigger: a second T3 server launched over SSH by the macOS desktop app.

What happens

1. The t3code.service server listens on 127.0.0.1:3773 and owns the T3 Connect managed tunnel.
2. The macOS desktop app connects to this host over SSH and starts its own server via ~/.t3/ssh-launch/<id>/run-t3.sh. It shares the same T3 home, so it also shares the same T3 Connect link and tunnel.
3. 3773 is busy, so findAvailablePort (apps/server/src/cli/config.ts) gives the SSH-launched server 3774. On startup it calls registerManagedTunnelRecovery with origin 127.0.0.1:3774, and the relay switches the tunnel's ingress to 3774.
4. The SSH-launched server exits when the Mac disconnects. Nothing points the relay back at 3773. The service server only re-registers its origin at its own startup.
5. From then on, cloudflared (still running under the 3773 server) logs dial tcp 127.0.0.1:3774: connect: connection refused for every request, and mobile shows Relay could not reach the environment endpoint (endpoint_request_failed).

Evidence

• The service log shows the 3774 ingress errors on 18 separate days between Sep 6 and Oct 11, matching the times the SSH-launched server was started.
• The SSH-launched server's server.log shows Listening on http://127.0.0.1:3774 followed by T3 Connect managed tunnel recovery registered (last at 23:17, it stopped at 23:24). The phone failed from then on.
• Requests reached 3773 successfully in the window where the service server had registered last. So the relay ingress flips to whichever server registered most recently.
• systemctl --user restart t3code.service fixes it right away: the public /api/t3-connect/health reaches the backend again (HTTP 400 for an unauthenticated probe) and the 3774 errors stop.

Repro

1. Run T3 as a service on Linux host A (port 3773) with T3 Connect enabled.
2. From the macOS desktop app, add host A as an SSH environment, so it launches a second server on 3774.
3. Close the desktop app or let the Mac sleep, so the SSH server exits.
4. Open host A from the mobile app through T3 Connect: endpoint_request_failed.

Possible directions: the SSH-launched server shouldn't run the managed tunnel or register an origin when another server on the host already owns the link. Or the tunnel owner could re-assert its origin when cloudflared reports origin connection failures.

Mobile trace IDs: f68867e720322ef82c798d215ec5567e, e8e8d71d4868e441ae24c8e1f0937fde

Diagnosed and written by Claude Opus 5.5 (Claude Code) via t3 triage.

Same root cause as the SSH-launched server report above, with two additions.

1. Any second server on the same T3 home triggers it, not just the SSH launcher. Running t3 serve in a terminal against the same T3 home is enough. It came up on a random port (127.0.0.1:36413), ran registerManagedCloudTunnelRecovery / activateManagedTunnel, and the relay ingress moved to 36413.

2. The shutdown hook runs but doesn't restore the previous origin. After stopping the second server, releaseManagedTunnelOnShutdown ran, yet the relay ingress stayed on 36413. cloudflared then logged connection refused for every request, including /api/t3-connect/mint-credential. So besides the directions already suggested, the release path could hand the ingress back to the previous owner (here 127.0.0.1:3773) instead of leaving it dangling.

Also reproduces on 0.0.45 stable, so it isn't nightly-only. systemctl --user restart t3code.service fixes it.

Environment: T3 Code 0.0.45 (CLI + service runtime), Linux 7.0.0 x64, Node v26.8.2, cloudflared 2026.5.2 (managed).

Diagnosed via t3 triage with Claude Code (Claude Opus 5.5).

Reproduced on macOS (launchd service), 0.0.46-nightly.20261012.3021: running t3 start while the service was up started a web-mode server on 127.0.0.1:60565, which ran activateManagedTunnel and then releaseManagedTunnelOnShutdown ~12 s later. The ingress stayed on 60565 (cloudflared: dial tcp 127.0.0.1:60565: connect: connection refused) until the service was restarted.

This looks fixed as of 0.0.46-nightly.20261014.3105: acquireServerOwnership (exclusive lock on userdata/server-owner.sqlite) is now the first step of layerMakeServer, before the cloud link is activated. Re-ran the repro on 0.0.46-nightly.20261015.3171 with the service running: t3 start now exits immediately with

and the tunnel ingress stays on 3773 (no connection refused in the service log afterwards).

Another occurrence, moved here from #20119 (closed as a duplicate of this).

Setup: WSL2 Ubuntu 24.04 running T3 as the systemd --user service (t3code.service, serviceManaged: true), linked to T3 Connect through the managed cloudflared. Clients are the macOS desktop app and Android. Server 0.0.46-nightly.20261015.3176; the unit's ExecStart launcher is 0.0.46-nightly.20261007.2787.

Timeline (UTC):

• 2026-10-09 12:35:38: first boot-service.log line with originService=http://127.0.0.1:3774 (Incoming request ended abruptly: context canceled). The service itself listened on 127.0.0.1:3773 throughout.
• 9 to 16 October: 354 log lines reference 127.0.0.1:3774. Early ones are context canceled; later ones are Unable to reach the origin service ... dial tcp 127.0.0.1:3774: connect: connection refused.
• 2026-10-16 02:11:31: last 3774 error.
• 2026-10-16 02:15:41: the service restarted after a WSL outage. Since then, server-runtime.json reads port: 3773 with the live pid and no new 3774 errors have appeared. I have not confirmed the ingress was re-registered, as opposed to unused.

Stale runtime file: before the restart, ~/.t3/userdata/server-runtime.json advertised port 3774 and a pid that no longer existed, while the live service was on 3773.

Second runtime: ~/.t3/wsl-runtime/ holds two runtime installs (sha256-d8c7… installed 7 October 18:40, sha256-7052… 8 October 00:00, local time). During the bad period a t3 --bootstrap-fd 0 process from there ran with its own cloudflared alongside the service. It is not running now, so I could not trace its parent. The install times fall a day before the first 3774 ingress line, which fits the #5749 trigger, but I have not confirmed which client launched it.

Workaround: systemctl --user restart t3code.service.

The duplicate-server trigger documented here is fixed on current main by 570542662a, with older-service protection added in 353d8c55d8. A second server can no longer share the same T3 home and overwrite its tunnel origin. Focused ownership tests pass, and a local stub confirmed that re-registration sends the changed port. Hosted relay behavior remains unverified. Recommend closing the documented trigger as fixed; a recurrence after updating and restarting warrants fresh listener/origin evidence.