Sign up
View stargazers13Sign up

Antigravity runtime temp-dir cleanup starves the server's file I/O for minutes, and successful title generations report a timeout and retry#23877

opened23h agobyLluquibu

What happened

On Windows, while I open and use my conversations, the desktop app shows "<host> is reconnecting" and "T3 Code on <host> did not answer". The window stays open and the chats are visible, but the connection to the local backend stays degraded for many minutes. I first saw this on 0.1.1 and reproduced it on 0.1.2-nightly.20261022.3541. Setting UV_THREADPOOL_SIZE=16 (verified in the backend's environment) did not help.

The backend keeps listening on 127.0.0.1:3773. OPTIONS / answers 204 in about 28 ms during the outage, but GET /.well-known/t3/environment times out. In one 5-minute sample: 4× 200, then 20 consecutive 3 s timeouts.

Diagnosis

The outage lasts exactly as long as T3 is deleting an Antigravity runtime's PyInstaller temp directory. My text generation model is Antigravity, so every thread title runs the cycle below. My chat provider is Codex.

  1. ThreadTitleRegenerationService calls AntigravityTextGeneration.generateThreadTitle. AntigravityDriver.makeRuntime creates a per-run run-* directory under <stateDir>/antigravity-tmp/<hash>/. The release step is fileSystem.remove(directory, { recursive: true, force: true }) (apps/server/src/provider/Drivers/AntigravityDriver.ts:223-240).
  2. agy_acp_server.exe unpacks its PyInstaller payload into run-*/_MEI*: at least 6,456 files, counted 17 s after the directory appeared (about 1 GB according to #11204).
  3. The ACP prompt succeeds in 3–7 s. When the runtime scope closes, the recursive remove queues thousands of delete requests on the server's shared libuv pool. During the outage, all 16 pool workers were blocked in NtSetInformationFile(FileDispositionInformationEx) on files under run-*/_MEI*, read at 21:42 UTC during the second cleanup. A runtime snapshot during the first cleanup counted 2,494 pending FSReqCallbacks.
  4. Every other async fs call waits behind those deletes. /.well-known/t3/environment waits on ServerSecretStore.get, a plain readFile, so the descriptor times out, and the client shows reconnecting / did not answer. fs.statSync in the backend returned in 0 ms while an async fs.stat of the same directory did not finish in 2 s.
  5. The runtime scope that holds the temp dir is closed inside Effect.timeoutOption(ANTIGRAVITY_TIMEOUT_MS) (180 s; inner Effect.scoped at apps/server/src/textGeneration/AntigravityTextGeneration.ts:253, timeout at :270). Cleanup time therefore counts against the timeout. A title that was generated successfully is reported as "Antigravity text generation timed out." and thrown away.
  6. Initial titles retry twice (ThreadTitleRegenerationService.ts:126-129), so one title costs three launches and three cleanups. The outbox effect also survives backend restarts: it was on attempt_count 4 of 5 (EffectWorker maxAttempts 5). On restart, removeAntigravityRuntimeTempDirs sweeps the leftover tree at driver creation, which stalls startup the same way. It took 14.5 s on one launch and 49 s on the previous one, and desktop.backendProcess.probeReadiness took 16.8 s and 51.9 s.

On this machine the three cleanups took about 7.7, 9.3 and 4.2 minutes: about 21 minutes of a stalled server for one title. For comparison, fs.rm of 1,000 freshly written small files in a separate Node process took under 1 s. The cost comes from the size of the Antigravity tree. ESET Endpoint Security is installed and may add to it; that was not isolated.

Possible directions, for the maintainers to judge:

  • Do the cleanup outside the timed region, and do not discard a completed generation because cleanup is slow.
  • Delete the tree off the shared libuv pool, or with bounded concurrency, or detached from request-serving work.
  • Avoid one full extraction per text-generation call.

Steps to reproduce

  1. Windows 11, T3 Code desktop 0.1.2-nightly.20261022.3541, Antigravity provider installed and signed in.
  2. Settings → text generation model: Antigravity (gemini-3.8-flash-high here).
  3. Start a new thread with any provider and send the first message. This enqueues thread-title.generate.
  4. Watch %USERPROFILE%\.t3\userdata\antigravity-tmp\<hash>\run-*\_MEI* fill with thousands of files, then get deleted after the prompt returns.
  5. While it is being deleted, poll GET http://127.0.0.1:3773/.well-known/t3/environment: it times out and the app shows "reconnecting". server.trace.ndjson shows session/prompt Success, followed minutes later by AntigravityTextGeneration.generateThreadTitle Failure "timed out", then a retry.

How long each outage lasts depends on how fast the tree is deleted. Not tried on a second machine.

Version

0.1.2-nightly.20261022.3541 (commit 044f072), desktop app with its local backend. Same symptom seen on 0.1.1; the mechanism was not verified there. The t3 triage CLI was also 0.1.2-nightly.20261022.3541.

Environment

Windows 11 Pro 10.0.26200 x64, 22 logical CPUs, 31.5 GiB RAM. Desktop app (Nightly) with local backend (Electron, ELECTRON_RUN_AS_NODE). ESET Endpoint Security running. Chat provider Codex; text generation Antigravity gemini-3.8-flash-high.

Evidence

shell
1# Backend libuv workers during the outage (UV_THREADPOOL_SIZE=16), 21:42 UTC, read with GetThreadContext.
2# 16 of 16 workers, all the same call; 3 shown:
3Wait/Executive NtSetInformationFile class=64 FileDispositionInformationEx flags=0x13 ~\.t3\userdata\antigravity-tmp\<hash>\run-QdmTr1\_MEI00007f702\google3\third_party\py\mcp\server\auth\errors.py
4Wait/Executive NtSetInformationFile class=64 FileDispositionInformationEx flags=0x13 ~\.t3\userdata\antigravity-tmp\<hash>\run-QdmTr1\_MEI00007f702\google3\third_party\py\numpy\f2py\_backends\__init__.py
5Wait/Executive NtSetInformationFile class=64 FileDispositionInformationEx flags=0x13 ~\.t3\userdata\antigravity-tmp\<hash>\run-QdmTr1\_MEI00007f702\google3\third_party\py\setuptools\_distutils\command\build_py.py
6# Control: an idle plain Node process with 16 workers shows them as Wait/Unknown (NtWaitForAlertByThreadId).
7
8# Backend active resources at 21:28:47 UTC (attempt 1 cleanup): FSReqCallback 2494; statSync(userdata) 0 ms; async stat(userdata) >2000 ms
9
10# server.trace.ndjson (UTC)
1121:25:01.572 -> 21:48:24.833 ThreadTitleRegenerationService.execute 1403260 ms Success
1221:25:01.866 -> 21:33:23.105 AntigravityTextGeneration.generateThreadTitle 501239 ms Failure "Antigravity text generation timed out."
1321:25:34.596 -> 21:25:37.862 session/prompt 3266 ms Success
1421:33:25.109 -> 21:43:29.537 AntigravityTextGeneration.generateThreadTitle 604428 ms Failure "Antigravity text generation timed out."
1521:34:04.655 -> 21:34:08.675 session/prompt 4020 ms Success
1621:43:33.590 -> 21:48:24.804 AntigravityTextGeneration.generateThreadTitle 291214 ms Failure "Antigravity text generation timed out."
1721:44:03.226 -> 21:44:09.843 session/prompt 6617 ms Success
18# ServerSecretStore.get between 21:25 and 21:35: p50 177 ms, max 294323 ms
19# environment.metadata.descriptor: repeated 10 s Interrupted from 21:26 to 21:32
20# Startup: removeAntigravityRuntimeTempDirs 14549 ms (49124 ms on the previous launch)
21# The descriptor returned 200 in 6-9 ms right after the title effect completed at 21:48:24.

Related issues

#11204, #9650 and #12239 reported Antigravity _MEI folders being left behind and filling the disk. This issue is about the cost of deleting the per-run tree while the server is running, not about leftovers. #16932 had the same pool-starvation symptom from pipe reads; that was fixed by #17386, and the code is present in this build. #22586 is a different Windows stall (fff rescan on the event loop).

Fix applied or workaround

Nothing was changed on the machine during triage. Workaround: switch the text generation model away from Antigravity. Raising UV_THREADPOOL_SIZE to 16 did not help.

Filed by

Claude Code (claude-opus-5-5) via t3 triage

LLaguirre-Axoft18h ago
Synced from GitHub

Same pool starvation, reached through a second path: the managed runtime install/update validation. On this machine it caused an 11-minute "reconnecting / did not answer" outage, and then the update was reported as failed even though validation succeeded.

What happened

In Settings → Providers → Antigravity, I started the update from the managed runtime 1.1.1 to 1.3.0. Within seconds, every thread and the Providers page showed "<host> is reconnecting" and "T3 Code on <host> did not answer". The backend kept listening on 127.0.0.1:3773 but GET /.well-known/t3/environment hung. Text generation on this machine is Claude, not Antigravity, so the title path described above was not involved.

Diagnosis

  1. AntigravityInstallation.install downloads and extracts 1.3.0, then runs validate, which starts the new agy_acp_server.exe in %TEMP%\t3-antigravity-validate-*. PyInstaller unpacks its payload into _MEI* there: 3,426 files in 686 directories, 140 MB.
  2. Validation succeeded in 15.6 s. When its scope closed, the profile directory removal (AntigravityInstallation.ts:356-371) started the recursive delete. It ran at about 10 files/s, and every other async fs call waited behind it, including ServerSecretStore.get and the trace writer.
  3. The validation scope is closed inside the 90 s timeout: validate(...).pipe(Effect.scoped, Effect.timeout(VALIDATION_TIMEOUT)) at AntigravityInstallation.ts:657-666. The cleanup took about 11 minutes, so the install failed with TimeoutError once the delete finished. It never wrote .install-complete.json, the staging directory was removed, and active.json still points at 1.1.1.
  4. The user-facing message is the generic "Could not install Antigravity. Check free disk space and directory access, then try again." Retrying repeats the same outage and the same failure. This matches #18551's symptom (successful validation, failed install), with slow cleanup instead of EBUSY as the cause. The retry added by #18472 does not help here.

The directions above apply to this path too: run the validation profile cleanup outside the timed region, and keep the delete from occupying the shared pool.

Steps to reproduce

  1. Windows 11, ESET Endpoint Security running, T3 Code desktop 0.1.2 with its local backend, managed Antigravity runtime 1.1.1 installed.
  2. Settings → Providers → Antigravity, start the update to 1.3.0.
  3. Watch %TEMP%\t3-antigravity-validate-*\_MEI* drain while polling GET http://127.0.0.1:3773/.well-known/t3/environment. It hangs until the directory is gone, and the app shows "reconnecting" the whole time.
  4. When the directory is gone, the endpoint answers again and the install reports the disk space / directory access error. The previous runtime stays active.

Version

0.1.2 (tag v0.1.2, commit 044f072), desktop app with its local backend. The t3 triage CLI was 0.1.2-nightly.20261022.3541. AntigravityInstallation.ts is unchanged on main through v0.1.3-nightly.20261023.3553.

Environment

Windows 11 Pro 10.0.26200 x64, 22 logical CPUs. ESET Endpoint Security running; Defender real-time protection off. Antigravity managed runtime 1.1.1 (sha 47cb50ee…) updating to 1.3.0.

Evidence

shell
1# server.trace.ndjson (UTC)
203:01:23.940 provider.install.start
303:01:23.943 GET https://dl.google.com/agy-extensions/releases/windows/agy-acp-server-1.3.0-windows-x86_64.zip
403:01:29.978 -> 03:01:45.537 AntigravityInstallation.validate 15.6 s Success
503:01:23.940 -> 03:13:10.832 AntigravityInstallation.install 706.9 s Failure
6 AntigravityInstallationError: Could not install Antigravity. Check free disk space and directory access, then try again.
7 [cause]: TimeoutError: Operation timed out after '1m 30s'
8
9# ServerSecretStore.get (plain readFile) during the cleanup
1003:01:49.628 -> 03:02:34.489 44.9 s
1103:02:53.1 -> 03:10:40.6 467.5 s (8 concurrent calls, all the same)
12# environment.metadata.descriptor: 10 s Interrupted, repeatedly, from 03:02:19
13# First 200 again at 03:13:09, in 9 ms, right after the directory was gone.
14
15# Temp dir drain, local sampling
16%TEMP%\t3-antigravity-validate-FMgz2l\_MEI0000a7882: 3,426 files / 686 dirs / 140 MB at start
173,292 -> 3,076 files in 21 s
18
19# Process sample during the outage
20backend: 16 ms CPU over 5 s; 465 "other" I/O ops per 10 s; 6 threads in Wait/Executive
21ekrn.exe (ESET): 1.42 M read ops, 7.7 s kernel + 2.6 s user CPU per 10 s

Fix applied or workaround

Nothing was changed on the machine. The backend recovered by itself when the delete finished. Workaround: do not retry the update from the UI on this machine until the cleanup moves out of the timed region; each attempt costs the same outage and fails the same way.

Filed by

Claude Code (claude-opus-5-5) via t3 triage

LLaguirre-Axoft17h ago
Synced from GitHub

Follow-up from the same machine: the slow delete is specific to the T3 Code executable, not to the files, Node, or Electron.

Same script (fs.promises.rm(dir, { recursive: true, force: true })), same files copied from the live Antigravity _MEI tree, Electron binaries run with ELECTRON_RUN_AS_NODE=1:

ExecutableFresh small filesAntigravity _MEI files
node 22.23.3665 to 2,614 files/s10,775 to 11,658 files/s
node 24.15.01,561 files/s9,432 files/s
VS Code Code.exe (Electron 43.7.7, Node 24.21.0)3,170 files/s10,632 files/s
T3 Code (Beta).exe 0.1.2 (Electron 44.9.3, Node 24.22.0)6 to 8 files/s7 files/s

The T3 Code rate matches the drain rate seen during the outages, so any recursive delete the backend does on this machine runs about 1,000x slower than the same delete from another process. ESET Endpoint Security 13.1 is the only real-time scanner here (Defender real-time protection off), centrally managed. During the outage ekrn.exe did about 1.4 M read ops per 10 s. Both binaries are validly signed. Not confirmed from ESET's own logs, so the exact module is unknown. Users on managed endpoints with similar policies will see the same cost, which makes doing the cleanup outside the shared pool and outside timed regions more important.

Filed by Claude Code (claude-opus-5-5) via t3 triage