# Speed up project search over misclassified binary files \(\#64549\) · gitcafe/zed

[View on GitCafe](https://git.cafe/gitcafe/zed/commit/8bbcd839686303a299b7b5135117058e9d66e64a)

Repository: [gitcafe/zed](https://git.cafe/gitcafe/zed)

Visibility: public

Requested revision: 8bbcd839686303a299b7b5135117058e9d66e64a

Requested commit: 8bbcd839686303a299b7b5135117058e9d66e64a

Commit: 8bbcd839686303a299b7b5135117058e9d66e64a

Tree: 15fd9a738bf95c0a9ed4f6edbaa48abc851b3a2a

Author: Kirill Bulatov

Committer: GitHub

## Message

```
Speed up project search over misclassified binary files (#64549)

Based on
https://github.com/zed-industries/zed/issues/38799#issuecomment-5758593194
finding.

Project search treated any NUL-free file as text, then read it whole,
ran chardetng over every byte (~34 MiB/s) and decoded it into a String
before searching. For raw assets (.yuv/.raw/.onnx/.glb/.tif in #38799)
that cost ~8 s and ~2.6× the file size in RAM per worker.

- Bound chardetng to a 1 MiB window starting at the first non-ASCII
byte.
- Stream-decode legacy/UTF-16 files in project search (`DecodingReader`)
instead of materializing them.
- Scan the single-line text pre-filter in 64 KiB blocks instead of
`read_line`.
- Run the control-byte check even without NULs; treat all high bytes as
text so legacy encodings (windows-1251, GBK, ...) stay text.
- Add magic numbers for TIFF, glTF, ELF, Mach-O, gzip/xz/zstd, SQLite,
HDF5, NumPy, safetensors, GGUF, fonts, media containers and others.
- Move the classifier into a `file_content` crate and use it for git
commit blobs too, replacing the separate NUL-only `is_binary_content`.

Measured on a 200 MiB NUL-free raw file: 8.5 s → ~1.5 s per file, no
whole-file allocation.

Release Notes:

- Improved project search speed and memory usage over misclassified
binary files
```

## Parents

- [d284cbbc87b55d659303106ced627c8612369827](https://git.cafe/gitcafe/zed/commit/d284cbbc87b55d659303106ced627c8612369827?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zed/tree/8bbcd839686303a299b7b5135117058e9d66e64a?format=markdown)