[Bug]: On Windows, every git start blocks the server's event loop, freezing it for seconds#25245

Independent Windows evidence: native spawn stalls and a local-primary heartbeat timeout

I can independently confirm the native-spawn blocking symptom, with a sampled backend stack and a separately captured local connection failure. Posting here rather than opening a duplicate.

Environment / build disclosure: Windows 11 Pro, build 26300; desktop executable reports T3 0.1.2; shipped Electron 44.9.3 / Node 24.22.0; Git 2.40.1.windows.1. The server bundle is locally modified with asynchronous Windows executable lookup and SQLite read routing. This is not a clean-stock reproduction. Its packaged Effect spawner still calls native NodeChildProcess.spawn(...) directly on the backend event loop. The user regularly runs dozens of agent threads, but the exact active count during this capture was not established.

Observed evidence

A 100 Hz V8 profile of the live backend on 2026-10-26, approximately 01:20:59–01:22:00 UTC:

The sampled call path was:

The native call site was verified in the running installation's server bundle. The public process runner still delegates launch to the Effect spawner.

These are statistical wall-time samples, not CPU utilization: they can include native waiting/descheduling. Consecutive samples in the same frame may cover back-to-back calls, so 7.38 seconds is not claimed to be one measured spawn call. The Git spans overlap and are not an exact or additive process-launch count. Profiler startup acknowledgement time is not included in the spawn attribution.

To check whether slow launch exists without profiling, a separate finite fixture using the same shipped Electron executable launched only git --version:

This is only two launches per phase under uncontrolled host load, not a benchmark proving zero profiler effect. It does show that slow synchronous launch also exists with the profiler off.

Real disconnect: failure started before focusing T3

In a later window, with no CPU capture active, an out-of-process probe and foreground-state sampling captured:

The client ping timeout is 15 seconds, and the disconnect path distinguishes this ping-timeout detail.

For this event, backend response failure preceded foreground return by 13.45 seconds. The observed “Syncing messages…” then “reconnecting” sequence therefore does not establish that focusing the window caused the failure.

The metadata endpoint also performs asynchronous filesystem work, so its timeout alone is not a pure event-loop measurement. The CPU profile and disconnect are different windows; we have not proven that a particular Git launch caused this exact heartbeat loss.

Suggested T3-side investigation / changes

1. Isolate native child creation from the event loop serving RPC and heartbeats. Evaluate a long-lived launch worker/broker rather than merely wrapping spawn() in a Promise/Effect. Preserve stdout/stderr streaming, backpressure, exit status, cancellation, and process-tree ownership. A launch worker may block on Windows, but the primary backend should remain responsive.
2. Reduce duplicate background Git launches per worktree / branch. Coalesce in-flight read requests, reuse valid cached results, and stagger sweeps. Keep freshness checks before settlement or other state-changing decisions. This complements #24098 and should not require lowering user agent-task concurrency.
3. Measure creation delay separately from command runtime and queue wait. Record bounded aggregates for native spawn return latency, pending launches and event-loop lag. Increasing RPC concurrency or extending the 15-second timeout alone does not remove synchronous blocking.
4. Test responsiveness under deliberately delayed child creation. Assert that independent RPC / heartbeats remain responsive while the launch worker is delayed, and that command output, cancellation and cleanup retain their behavior.

A small spawn-host direction appeared in #18219, but that PR is closed/unmerged and described Linux support; I have not verified a Windows equivalent. #21289 concerns Linux fork/RSS cost, which should not be assumed to explain Windows native creation latency.

The underlying Windows cause remains unknown; these measurements do not establish Defender, disk, memory growth, or a T3 database lock as the cause. No raw profiles, conversation contents, credentials, environment-variable values, private repository paths or host identifiers are attached.

Prepared and submitted by Codex at the user's request. No running T3 tasks were stopped or restarted for this investigation.

Another Windows datapoint: stalls about once a minute, and what detached: true does and doesn't fix

Posting here rather than in a new issue. It's the same mechanism on a different machine, plus one partial mitigation measured inside T3.

Environment / build disclosure: Windows 11 Enterprise (build 26200), 16 logical cores, 62 GB RAM, Git for Windows 2.49.0, T3 Code (Nightly) 0.1.7 and 0.1.8 builds. It's a managed corporate laptop. Endpoint security software inspects every process creation, so each launch costs more than on a clean machine. The last section uses a locally patched server bundle (a preload that changes spawn options for some git calls). Everything before that section is stock.

What the stall log shows (stock build)

• From 08:30 to 13:00, event-loop-stalls.ndjson logged 25–46 stalls per 30 minutes. The median stall was 2.5–3.9 s and the worst was 10.9 s. Every record has utilization: 1, with 2–6 s of system CPU per window.
• In one 27.7-minute trace window, the server ran 484 processes (17.5/min), 453 of them git. The processRunner.runProcessCore spans for git had a median of 4.0 s and a p90 of 7.3 s. In the same minutes, a plain cmd /c exit from a terminal took 0.26–0.77 s. So most of each span is probably time spent queued behind the blocked loop, not the git work itself.
• That window also re-ran the full source-control discovery set four times (gh, glab, az, jj and three other CLI probes). That adds launches at exactly the moment launches are expensive.
• The per-launch cost on this machine also varies a lot within one boot. It's under a second most of the time and 3–10 s in bad stretches. That matches the "varies over time" observation in the original report.

The SQLite spans are mostly queueing, not database time

This matters for anyone who lands here from #26224 / #30065 / #29468.

• In the same trace, sqlite spans had a p90 of 2.4 s and a max of 12.7 s, and BEGIN/COMMIT averaged about 300 ms each.
• I ran the same read statements from a separate read-only connection to the same statev2.sqlite, while the server was stalling. They took 0.3–23 ms.
• So on this machine the slow DB spans mostly measure the blocked event loop, not SQLite.

The database had also grown to 5.56 GB, with 46% free pages and auto_vacuum = NONE (a migrated install, as in #30066). I ran VACUUM offline with auto_vacuum = INCREMENTAL, which shrank it to 2.90 GB. The stalls continued afterwards: 11 in the first ~12 minutes after restart, median 3.7 s, worst 6.9 s. The restart also re-applied the git patch below, so neither change on its own stopped the stalls.

Partial mitigation: detached: true for read-only git calls (locally patched)

• On win32, the Effect Node spawner defaults to detached: false, windowsHide: options.windowsHide ?? !detached (@effect/platform-node-shared 4.0.5, nodeChildProcessSpawner). libuv turns windowsHide with piped stdio into CREATE_NO_WINDOW, and that still creates a console host for every git child.
• With detached: true, the child gets no console, so no console host is created.

I measured this outside T3 first. A console-less Node parent (as T3's backend is) timed spawn → exit, interleaved and rotated. The machine was healthy, early in the boot:

Then I tried it inside T3, with a preload that sets detached: true only for read-only leaf subcommands (rev-parse, show-ref, for-each-ref, status, ls-files, config --get, worktree list, …) run through mingw64\bin\git.exe.

• T3's own runProcessCore git spans dropped from a median of 1.24–1.42 s to 0.83–0.84 s (−33 to −41%). The p90 improved less (−5 to −22%).
• These were back-to-back sessions, not a controlled A/B.
• The stalls don't go away. The spawn() call is still synchronous on the event loop; it's just shorter.

One hazard: detached removes the console only for the direct child. A console-less child that starts console programs (hooks, credential helpers, ssh, ! aliases) gives each of them a new console. cmd /c ping under detached took 17–19 s in both runs. So detached only suits leaf read-only calls, not git calls that can start helpers.

What would help on Windows

1. A Windows version of #30209's approach: launch through a helper process so spawn() can't block the RPC loop. Of the fixes in this thread and its linked PRs, it's the only one that removes the stalls rather than shortening them.
2. In the meantime, detached: true (and mingw64\bin\git.exe, which preferGitForWindowsBinary already selects) for read-only leaf git calls. That cut the median launch cost by about a third to a half here.
3. Don't re-run the full source-control discovery probes on every reconnect while launches are slow. Otherwise slow launches cause a reconnect, the reconnect causes more launches, and those make launches slower.

I can share the trace-analysis scripts (Node, read the server.trace.ndjson* and stall files) if they would help.

Windows datapoint: CPU contention turns the per-start block into seconds, and a priority A/B isolates it

Setup:

• Windows 11, 16 logical CPUs, 0.1.8-nightly.20261109.4399.
• 148 worktrees and 403 unarchived threads.
• Antivirus real-time scanning off.

Probe: a separate, nearly idle Node process on the same host (monitorEventLoopDelay plus a 100 ms ticker). Every 2 s it times only the synchronous part of spawn('git', ['--version']).

A/B test: the same workload both times, two Playwright suites at 8 workers each for 240 s, with T3 left running. Only the test processes' priority class changed:

• With the host idle, the same sync spawn takes 4–10 ms.
• With antivirus real-time scanning on, an earlier session on this machine measured 134–550 ms per start.

Meanwhile the server starts about 300 git processes a minute while otherwise idle (runGitCommand about 309/min, readConfigValue about 135/min, from PR lookup and settlement; see #24098). So under contention the main thread spends seconds per second inside CreateProcess.

Takeaway: the stall is the main thread competing for CPU and process-creation time with everything else on the host. Moving spawns off the event loop (#25245) and cutting the spawn volume (#24098) would both decouple T3's responsiveness from host load. Lowering the priority of other workloads is a user-side workaround, not a fix.

Windows datapoint: the stall at startup makes the desktop give up on its own backend

Same mechanism on another machine. One new consequence: the desktop's 180 s readiness check fails while the server's event loop is busy.

Environment: Windows 11 Pro (build 26200), 24 logical CPUs, 64 GB RAM, Git for Windows 2.55.0.windows.3, T3 Code (Nightly) desktop 0.1.9-nightly.20261111.4501. Defender real-time scanning is on. About 28 worktree directories across a few projects.

What happened: On the first launch after the nightly update, the backend started (port 3773) but never answered /.well-known/t3/environment in time. server-child.log:

During those 3 minutes, every request to the environment endpoint failed while waiting on a small file read:

event-loop-stalls.ndjson: There are dozens of entries over roughly 55 minutes. Each 30 s sample shows utilization 0.9 to 1.0, delayMaxMs of 2 to 8 s and cpuSystemMs of 15,000 to 31,000. That means the server process spent most of each window in kernel time:

server.trace.ndjson, first 15 minutes after launch:

StorageCleanup.cleanWorktrees (inside StorageCleanup.sweep) and ThreadPullRequestServiceV2.synchronize both started at boot and ran for 979 s and 839 s. Later in the session, git starts peaked at 389 a minute. Across the last ~2 hours of traces there were 3,922 runGitCommand spans totalling 6,265 s.

Windows recorded no application crash events, and the renderer never logged render-process-gone. To the user it looked like a crash: the app reported a backend failure and was relaunched several times that day.

Suggestion beyond the spawn cost itself: At startup, either hold back the PR and worktree sweeps until the backend reports ready, or answer /.well-known/t3/environment without waiting behind them. Otherwise a git-heavy startup reads as a dead backend.