Sign up
View stargazersSign up

[Bug]: Backend event loop blocks for minutes inside synchronous fff mixedSearch waiting on watcher lock (home-directory project) — shows as "<label> is reconnecting"#11306

openedSep 11, 2026byMmimen

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Add the user's home directory (/Users/<name>) as a T3 project on macOS. Home contains ~25 GB under Library/Caches plus constantly written directories (.t3, .claude, .ccs).
  2. Run several concurrent agent threads so the backend is busy and files under home change continuously.
  3. Open the Files panel for that project, or let the Files panel refresh.
  4. Watch the composer banner.

Expected behavior

A synchronous file-search call should not be able to block the backend event loop for minutes. A busy file watcher should not make the Files panel call hang.

Actual behavior

The desktop backend stops responding entirely: no WebSocket frames, no trace spans, no provider log writes, health endpoint unreachable. The desktop shows "<label> is reconnecting / Trying again" and "Not connected: message not sent". The freeze lasts 1 to 6 minutes, then either recovers on its own or the user relaunches the app.

Six freezes in ~20 hours on 0.0.41-nightly.20260911.1520 (and .20260910.1507), each visible as a gap in server.trace.ndjson where every span source stops at once:

Freeze start (UTC)DurationEnded by
00:51:0390suser relaunch
01:08:4155supdater relaunch
20:20:09385srecovered
20:35:43157srecovered

Root cause (from sample of the backend during the 20:35 freeze)

All 4099 main-thread samples show the Node event loop blocked inside a synchronous FFI call into libfff_c, waiting on a parking_lot read lock. The lock is held by fff's own watcher thread, which is reading files to sniff for binary content.

1Thread main (100% of samples):
2 uv_run -> uv__work_done -> node::fs::AfterStat -> JS callback
3 -> ffi-rs -> fff_search_mixed
4 -> parking_lot::raw_rwlock::RawRwLock::lock_shared_slow
5 -> __psynch_cvwait
6
7Thread fff-watcher-own (holds the lock):
8 fff_search::file_picker::FilePicker::add_new_file
9 -> fff_search::types::FileItem::detect_binary_per_byte -> read()
10
11Thread "notify-rs debouncer loop":
12 fff_search::background_watcher::handle_debounced_events
13 -> RawRwLock::lock_exclusive_slow (also waiting)
14
15Thread fff-scan:
16 fff_search::scan::ScanJob::run
17 -> bigram_filter::sniff_binary_for_non_indexable -> open()/read()

Two things combine:

  1. apps/server/src/workspace/WorkspaceSearchIndex.ts calls finder.mixedSearch(...) (line ~236) and finder.isScanning() synchronously on the event loop. Any lock contention inside the native library stalls the whole server.
  2. The finder is created with enableFsRootScanning: true, enableHomeDirScanning: true (line ~172), so a home-directory project indexes and watches the entire home tree. Every burst of writes under home (agent transcripts, caches, T3's own logs) makes the watcher take the write lock and read each new file while a listEntries / search call waits.

The 25,000-entry listEntries call for the home project takes ~13 s even when it does not deadlock (WorkspaceSearchIndex.waitForIndexReady 12993 ms in the trace).

Workaround

A ~/.ignore excluding Library/, .cache/, .t3/, .claude/, .ccs/, node_modules/ etc. shrinks the watched set (the ignore crate honors .ignore outside git). Removing the home project stops it entirely.

Suggested fixes

  • Run mixedSearch / isScanning off the event loop (worker thread or async FFI), or bound them with a timeout that fails the single request instead of the process.
  • Do not default enableHomeDirScanning / enableFsRootScanning to true. Or apply a large default exclusion set when the workspace root is $HOME or /.
  • Related: #10918 (no way to exclude directories), #10917 (hardcoded 15 s scan timeout), #7231 (a slow probe tearing down the session, which is the client-side symptom of this).

Environment

  • T3 Code (Nightly) 0.0.41-nightly.20260911.1520, also on .20260910.1507
  • macOS 26.4 (25E246), Apple Silicon (M5)
  • @ff-labs/fff-node 0.9.4
  • Local desktop backend, direct connection, no remote environment
  • state.sqlite 1.9 GB (not the cause, noted for context)

Full sample output available on request (269 KB). Diagnosed with Claude Fable 5.1 in T3 Code.

MmimenSep 12, 2026
Synced from GitHub

Follow-up with a second sample and source verification. Two additions.

1. Same deadlock via fff_destroy

A 20-minute freeze (17:20 to 17:40 PDT, ended by app relaunch). Main thread was in WorkspaceSearchIndex's idle-TTL release path this time, not a search:

1main (4276/4276 samples):
2 uv__run_timers -> JS -> ffi-rs -> fff_destroy
3 -> parking_lot::RawRwLock::lock_exclusive_slow -> __psynch_cvwait
4
5"notify-rs debouncer loop" (holds the picker write lock):
6 handle_debounced_events -> FilePicker::add_new_file
7 -> FileItem::detect_binary_per_byte -> read() (stuck in read)
8
9fff-watcher-own: lock_exclusive_slow (waiting)
10fff-scan: ScanJob::new_rescan -> lock_shared_slow (waiting)

So both mixedSearch (first sample) and destroy (this one) block the event loop behind the watcher's per-file read. fff-core/src/types.rs detect_binary_per_byte opens the file and reads it to EOF with no size cap; add_new_file calls it while handle_debounced_events holds shared_picker.write(). Any large or slow file (network-backed cloud placeholder, for example) parks the whole server.

2. The ~/.ignore workaround does not apply to the watcher on 0.9.4

Checked v0.9.4 source. In background_watcher.rs, should_include_file only consults git when the root is a repo; otherwise it uses the compiled-in is_non_code_directory list (node_modules, __pycache__, venv, target/*, Library/Caches, Library/Application Support, Library/Containers, Library/Group Containers). .ignore / .gitignore are honored by the initial WalkBuilder scan only. Current fff main (0.10.x) routes watcher events through the walker's ignore rules, so bumping @ff-labs/fff-node would at least give non-git roots a working exclusion list. It does not fix the unbounded read under the lock, or the synchronous FFI call on the event loop.

Net: on the shipped build, a non-git workspace root has no user-configurable exclusions for the watcher, and one slow file read anywhere under it can freeze the backend for as long as the read takes.

Jjihjihk16d ago
Synced from GitHub

Related incident on Linux with T3 Code 0.0.42: opening a broad directory through the T3 UI was followed by the host becoming unusable, and the machine was restarted. Reporting this here because the home-directory workspace and file-index trace look related; I have not confirmed the native-lock root cause described above.

Environment

  • Headless t3 serve on Ubuntu 24.04.4 LTS, accessed through the T3 UI.
  • 4 vCPUs, approximately 15 GiB usable RAM, no swap.
  • Workspace root was the user's home directory, containing multiple repositories and Git worktrees.

Observed sequence, 2026-10-12 UTC

  1. At 19:00:43, a projects.readFile request tried to open a skill directory as a file and returned WorkspacePathNotFileError.
  2. Immediately afterward, projects.listEntries took 15,400 ms, including 14,378 ms in WorkspaceSearchIndex.list and 14,013 ms in WorkspaceSearchIndex.runSearch.
  3. The user reported that the machine became unusable after opening the directory view. The journal records shutdown at 19:02:38 and the next boot at 19:03:39.

The recorded list request completed successfully. There is no matching kernel OOM-kill entry in the previous boot's journal, so this is not a confirmed OOM crash, and the trace alone does not prove the cause of the later unresponsiveness. I have not retried the broad scan on this working machine.

Expected behavior Opening a home-directory workspace should keep unrelated sessions responsive. Broad indexing should have bounded work, cancellation, and isolation from the server event loop, with a warning or explicit opt-in before scanning very large roots.

Only timing and environment details are included here; private workspace names and raw application logs are omitted.