[Bug]: SSH environment stays reconnecting after desktop update reuses an older remote server#16984

Thanks for the detailed report. Your logs point to a specific mechanism in the code. When the desktop and the reused remote server are on different builds, the pairing token is minted by one build and redeemed by the other. Reading the code, an Oct 4 server can't read tokens minted by an Oct 7 build.

What the reconnect does (main @ 300f7f9)

1. The launch script runs the runner once, which installs the desktop's pinned archive version. That's why the Oct 7 runtime showed up on the remote (tunnel.ts#L568-L573).
2. If ~/.t3/userdata/server-runtime.json names a live server whose port answers the readiness probe, that server is adopted as external and reused. Nothing compares its version to the desktop's (tunnel.ts#L630-L672). Only the managed branch restarts a server when the runner script changes (#L673-L686).
3. The pairing token is then minted by running auth pairing create --base-dir ~/.t3 through the new runner. It is written to the shared database, not issued by the running server (tunnel.ts#L723-L734, #L1689-L1697). It gets the default AuthStandardClientScopes (cli/auth.ts#L85-L104).

Why the Oct 4 server returns 500

• On Oct 7 (03:30 UTC), the permission split (#9786 to #9791, #10298) added new scopes: settings:write, providers:manage, environment:maintain, preview:operate, diagnostics:read, terminal:read, source-control:write, filesystem:read and filesystem:write. All of them are now in the standard client scopes (contracts/auth.ts#L205-L219).
• The Oct 4 build (v0.0.46-nightly.20261004.2652 = 4ee6bfd) only knows 8 scope literals (auth.ts#L89-L98). Its pairing-link row decodes scopes strictly with Schema.fromJsonString(AuthEnvironmentScopes) (AuthPairingLinks.ts#L18-L30).
• So consumeAvailable finds the row, but decoding its RETURNING row fails with a PersistenceDecodeError (#L261-L283). That error is wrapped as BootstrapCredentialConsumeAvailableError (PairingGrantStore.ts#L512-L519), then becomes ServerAuthBootstrapCredentialValidationError (EnvironmentAuth.ts#L531-L541) and finally a 500 access_token_issuance_failed (http.ts#L381-L383). That is exactly the error chain in your log.
• #10298 handled older clients talking to newer servers. It didn't cover a newer CLI writing grants that an older server has to decode.
• Each reconnect mints a fresh token the same way, so the loop never recovers. Restarting the server on the Oct 7 build fixes it because that build knows every scope.

This was confirmed by reading the code. I haven't reproduced it end to end.

Not #16797. The boolean bind (${requestedScopes === undefined}) arrived with #10298 and isn't in the Oct 4 query (AuthPairingLinks.ts @ 4ee6bfd). The Oct 7 server that worked for you does have it.

Why the UI only says "reconnecting": every SSH preparation failure that isn't a cancellation, including this 500, is mapped to ConnectionTransientError("remote-unavailable") (platform.ts#L170-L182). That error goes into backoff, and backoff shows as reconnecting (presentation.ts#L45-L50). The chat banner title is just " is reconnecting" (ChatView.tsx#L3118).

Fix direction

• Detect a version mismatch on reuse. Compare the reused server's version with the runner's archive version. Then either restart it on the matching build or block with a clear "remote server is X, desktop expects Y" message. Avoid killing busy servers (#16477).
• Alternatively, mint the token through the running server, or request only scopes it understands, so the issuer and consumer always agree.
• Forward compatibility: drop unknown scope strings when decoding stored grants instead of failing. That prevents the next scope addition from causing a 500 on older servers, though it can't fix builds already shipped.
• Treat a 5xx from bootstrap-bearer-session as a blocking error that shows the reason, not a silent transient retry.

Workaround: restart the remote T3 server on the build that matches the desktop. That's what you did, keeping the same --base-dir and port.

Related: #16797 / #16730 (a different consume-query failure), #16477, #5749, #13341 and #13521 (SSH reuse/restart behavior; more reuse means more chances of version skew), #13717 and #11287 (other ways to get stuck reconnecting).

Same failure on my side. This adds one case that the issue does not cover: the remote server runs as a background service.

Setup

• Desktop: 0.0.46-nightly.20261012.3010.
• Remote server: 0.0.46-nightly.20261005.2702, started by the background service launcher.

Evidence

• POST /oauth/token on the remote returns 500 with PersistenceDecodeError in AuthPairingLinkRepository.consumeAvailable:decodeRow.
• The pairing link row in auth_pairing_links has 13 scopes, for example filesystem:write, preview:operate and source-control:write.
• The string "filesystem:write" is not in the 2702 binary. It is in the 3010 binary.

Why a plain restart did not help

1. The desktop installed 3010 on the remote, but service-state.json still had activeVersion set to 2702.
2. t3 service restart started 2702 again.
3. t3 update run over SSH has no terminal. It downloads the release and sets activeVersion to 3010. It does not restart the service.
4. A second t3 service restart started 3010, and the 500 error stopped.

Workaround for a service install

1. Make sure no agent turn runs on the remote.
2. Run t3 update.
3. Run t3 service restart.

Suggestion When the desktop reuses a remote server, it should compare the server version with the version it installed. If they differ, it should show a clear message such as "remote server is X, desktop expects Y". The message should also tell the user to run t3 update and t3 service restart if the server is a background service.