# feat\(server\): install provider CLI updates in the background · PR \#17585 · pingdotgg/t3code [View on GitCafe](https://git.cafe/pingdotgg/t3code/pull/17585) Repository: [pingdotgg/t3code](https://git.cafe/pingdotgg/t3code) Visibility: public State: open Source repository: pingdotgg/t3code Source branch: t3/background-provider-updates Target branch: main Head commit: bb7ffa1beea6b63056d14424eacff203fa4575e4 [Discussion and reviews](https://git.cafe/pingdotgg/t3code/pull/17585) [Commits](https://git.cafe/pingdotgg/t3code/pull/17585/commits?format=markdown) [Files changed](https://git.cafe/pingdotgg/t3code/pull/17585/diffs?format=markdown) ## Description --- Provider CLI updates only happened when someone clicked **Update** on each environment. **Settings → General → Update automatically** (new, on by default, per environment) now has each server install them in the background when nobody would notice. **When it installs** - No client interaction for 2 minutes, on the monotonic clock. An install disconnects nobody; a turn sent during one waits for it to finish. The web reporter now listens in the capture phase, so editors that stop key propagation still count. - No run preparing, starting, running, or waiting on an approval or question. - No provider installer running on this host, including a manual one or one started by another T3 server. - A pass runs every 10 minutes and installs one instance at a time. Each `instance@version` is tried at most once per 6 hours. - **Update automatically** needs **Provider update checks** on. While checks are off, the switch is disabled on web and mobile and says so. Dev servers skip it, since they share the host's CLIs. An unreadable `settings.json` keeps it off. **What it installs** - The newest version the update check already vetted, pinned (`npm install @openai/codex@0.162.1`, not `@latest`), and never one the compatibility manifest marks broken or unsupported. Only native self-updaters and Homebrew run unpinned. - Under the lock it re-probes the CLI and skips if it already reaches the target, so it never redoes another server's update or downgrades a newer beta. - Background installs are silent: no update state goes to clients, so there is no toast, pill, or streamed output. Results go to the server log. **Install safety** - One numbered lock file per install target in the user cache dir (`t3/provider-install/.lock.N`), shared by every T3 server on the host. Exclusive creation is the only step that picks an owner: a server writes its record to a temp file and hard-links it as generation N+1, which fails if anyone else got there first, so two servers reclaiming a stale lock at once cannot both win and nothing is ever deleted out from under an owner. The record holds the server's pid, then the installer wrapper's, plus a per-run owner id; the server stays the owner through verification, and only a surviving installer keeps it after that. A lock whose pids are all dead, or a record older than an hour, counts as free. A successor that takes over an aged record whose pid was still alive re-reads that record after publishing its own generation and backs off if it changed, while a resumed owner refreshes its record before checking for a newer generation and sending "go": at least one side sees the other, so a server paused past the hour cannot install alongside its successor (at worst both back off and retry). Rewrites go through a temp file and rename, so a full disk never leaves an empty, held lock. - Turn starts wait while any install runs, so a CLI is never replaced as it launches. The run is persisted as starting first, so the updater's last check under the lock sees it and backs off. - The installer runs detached through a small Node wrapper with its output in a log file. The wrapper starts nothing until the server has durably recorded its pid and sends "go"; if the server dies first, or "go" comes more than 30 seconds late (a paused server resuming), it exits without touching the target. Once started, quitting or crashing the server no longer kills the install. The wrapper itself kills the installer tree after 30 minutes, so a hung install is bounded even with its server gone, which keeps the one-hour age cap safe. Main killed the installer on scope close and after 5 minutes, which could cut an npm extract in half. - A restarted server waits for a running installer, then refreshes the providers. - Manual updates and **Update everything** (#18053) use the same runner, now one per server instead of one per connection, and the same lock. This replaces the earlier version of this PR (+8.6k lines). Its host-wide reader/writer lease, install supervisor, launch-permit admission across every probe, scheduled-task blockers, update origins in the protocol, and per-driver ownership changes are gone. **Verified** - A non-dev server with no client connected, an isolated npm prefix, and Codex 0.160.0. The pass at +22 minutes found the window open and logged "Updating provider in the background". The lock file held the installer pid, `ps` showed `npm install @openai/codex@0.162.1`, Codex on disk became 0.162.1, the run logged success, and the lock file was removed. Earlier passes stopped at the window check. - SIGKILL of a second server 160 ms after its installer started: the installer was reparented, finished ("changed 2 packages in 7s"), and its lock kept the installer pid until it exited. A restarted server reported the provider behind while a lock's installer was alive, and current within 3 s of it exiting. - Lock protocol, on Linux with independent Node processes driving the real `ProviderInstallLock` and detached runner against an isolated prefix (fake installer that detects overlapping writers): 200 two-server install races with no overlap; 100 rounds of 8 simultaneous stale reclaimers with exactly one winner each; parent SIGSTOP+SIGKILL before and after recording the wrapper pid (wrapper exits, a second server installs); SIGKILL after "go" (lock held until the installer exits); a paused server resumed after the 30-second "go" deadline starts nothing; a lock held by an unrelated live pid frees after an hour; the wrapper's own backstop kills the installer and its descendants with the server dead. 200 forced rounds of a paused owner resuming while a successor takes over: 83 owner wins, 65 successor wins, 52 mutual backoffs, never two installers. The wrapper itself also ran under Electron 44.4.5's bundled runtime (`ELECTRON_RUN_AS_NODE=1`, on a Blacksmith Ubuntu runner): "go" runs the installer and forwards its exit code, the installer does not inherit `ELECTRON_RUN_AS_NODE`, closing stdin without "go" installs nothing, and the wrapper's backstop kills the installer and its descendant. Blacksmith: 85 lock/maintenance/turn-start/node-runtime tests, server and shared typechecks, lint, format. - After merging main (b72740852f): live with an isolated HOME, a pinned automatic install, deferral behind an active turn, waiting on the install lock, and a failed install followed by a successful retry. Blacksmith: 252 tests, then 68 lock/maintenance/turn-start tests after the lock ownership fix. - After merging main again (2893437f26, head 87aad2b47c): a real source server with an isolated home and Codex 0.162.0 in a temp npm prefix, latest 0.163.0. Client activity and a running thread each kept the install off. Once idle, one real `npm install` ran. A turn sent during it stayed `starting` until verification, then ran. The WebSocket snapshot moved to 0.163.0 / current. With **Update automatically** off, with **Provider update checks** off, and with a corrupt `settings.json`, nothing installed. An installer exiting 17 left a failed state, released the lock, and the next turn started. A hung installer was killed by the backstop (shortened at runtime to 7 s). With "go" withheld, the real 30 s deadline started nothing. Killing the server mid-install left the detached install running, and the restarted server saw exactly one install. Two servers sharing one prefix ran exactly one installer, and both reported 0.163.0. Blacksmith: 168 + 180 focused tests, server typecheck, lint, format. - Blacksmith: the touched server tests plus `ws`, `runtimeLayer`, and the `mcp/` tests (214 and 404 tests), 57 web test files (602 tests), and server, web, mobile, and contracts typechecks. Lint and format are clean. **Not verified:** Windows (`.cmd` shims through the wrapper, installer descendants, `taskkill /T`, turn starts during an install), the packaged desktop app end to end (the wrapper itself was run under Electron's runtime), and the mobile setting on a device. These need a Windows machine and a phone. **Known gaps:** a server cannot see another server's already-running turn on the same host; that server's new turns do wait on the lock. A hung installer holds turn starts until the 30-minute backstop. Muse is not installed in the background; its own launcher updates it. ![Before (main): Settings → General has no automatic update row](https://uploads-production-47e4.up.railway.app/files/183d18e1-1b82-4c7f-8702-b168b798ca7b/before-settings-general.png) ![After: Update automatically sits below Provider update checks, on by default](https://uploads-production-47e4.up.railway.app/files/41bae673-9d04-424d-8db1-05452d8e610f/after-settings-general.png) Current head (8bb20296d1), seeded dev data: ![Current head, desktop light: Update automatically on, directly below Provider update checks](https://uploads-production-47e4.up.railway.app/files/23db27d3-cb00-445f-854d-c32b8f09eb5c/pr-17585-general-desktop-default.png) ![Current head, desktop dark: the same two rows, both on](https://uploads-production-47e4.up.railway.app/files/269dadac-5bf0-42b6-9a0f-1ea68278f45f/pr-17585-general-desktop-dark.png) ![Current head, 390px phone width: both rows stack with their switches on](https://uploads-production-47e4.up.railway.app/files/cddf9709-e2c2-4c81-b75c-7ed1d2b25835/pr-17585-general-phone-default.png) With **Provider update checks** off, **Update automatically** was still toggleable even though the server ignores it then (36021ba04f fixes this): ![Before 36021ba04f: Provider update checks off, Update automatically still on and toggleable with no explanation](https://uploads-production-47e4.up.railway.app/files/2ec5b7ed-a398-4bd7-8732-11ab2c8efcff/pr-17585-checks-off-before.png) ![After 36021ba04f: Update automatically is disabled and reads "Turn on Provider update checks to update automatically."](https://uploads-production-47e4.up.railway.app/files/bad98a3a-7107-454b-b349-cfff0a9b357a/pr-17585-checks-off-after.png) Made with Opus 5.5 in Claude Code, with Sonnet 5.5 and GPT-6.1-Sol (Codex) subagents, running in T3 Code.