# Correct UTF-16 saving and add heuristic encoding detection \(\#45243\) · gitcafe/zed

[View on GitCafe](https://git.cafe/gitcafe/zed/commit/1bc3fa81543f6159a5f607a9b01c83c2ae5c309a)

Repository: [gitcafe/zed](https://git.cafe/gitcafe/zed)

Visibility: public

Requested revision: 1bc3fa81543f6159a5f607a9b01c83c2ae5c309a

Requested commit: 1bc3fa81543f6159a5f607a9b01c83c2ae5c309a

Commit: 1bc3fa81543f6159a5f607a9b01c83c2ae5c309a

Tree: 3211d7813a160800395fd3b5de913e257f7fde16

Author: Ichimura Tomoo

Committer: GitHub

## Message

```
Correct UTF-16 saving and add heuristic encoding detection (#45243)

This commit fixes an issue where saving UTF-16 files resulted in UTF-8
bytes due to `encoding_rs` default behavior. It also introduces a
heuristic to detect BOM-less UTF-16 and binary files.

Changes:
- Manually implement UTF-16LE/BE encoding during file save to avoid
implicit UTF-8 conversion.
- Add `analyze_byte_content` to guess UTF-16LE/BE or Binary based on
null byte distribution.
- Prevent loading binary files as text by returning an error when binary
content is detected.

Special thanks to @CrazyboyQCD for pointing out the `encoding_rs`
behavior and providing the fix, and to @ConradIrwin for the suggestion
on the detection heuristic.

Closes #14654

Release Notes:

- (nightly only) Fixed an issue where saving files with UTF-16 encoding
incorrectly wrote them as UTF-8. Also improved detection for binary
files and BOM-less UTF-16.
```

## Parents

- [22916311cd2b7d98d89d1ba72b5a567373472e2d](https://git.cafe/gitcafe/zed/commit/22916311cd2b7d98d89d1ba72b5a567373472e2d?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zed/tree/1bc3fa81543f6159a5f607a9b01c83c2ae5c309a?format=markdown)