# language_models: Remove unused language model token counting \(\#54177\) · gitcafe/zed

[View on GitCafe](https://git.cafe/gitcafe/zed/commit/2eafa6e6aa471a9930a5b8c0209afa66030fb41a)

Repository: [gitcafe/zed](https://git.cafe/gitcafe/zed)

Visibility: public

Requested revision: 2eafa6e6aa471a9930a5b8c0209afa66030fb41a

Requested commit: 2eafa6e6aa471a9930a5b8c0209afa66030fb41a

Commit: 2eafa6e6aa471a9930a5b8c0209afa66030fb41a

Tree: 023cd1d0f750742acc7964039d52a6014eda5d79

Author: Ben Brandt

Committer: GitHub

## Message

```
language_models: Remove unused language model token counting (#54177)

Drop the `count_tokens` API and related implementations across
providers, and remove the unused `tiktoken-rs` dependency.

I was going to update the dependency becuase they finally released a fix
we needed. But then I realized we only used this api in one place, the
Rules library. And for most models it would have been wildly incorrect
becuase we use tiktoken, i.e. OpenAI tokenizers, for almost every model,
which is going to give incorrect results.

Given that, I just removed these because the difference in how we get
these has caused plenty of confusion in the past.

Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- N/A
```

## Parents

- [254c27d5113f4cc97fadd37b56836ebec8e113d7](https://git.cafe/gitcafe/zed/commit/254c27d5113f4cc97fadd37b56836ebec8e113d7?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zed/tree/2eafa6e6aa471a9930a5b8c0209afa66030fb41a?format=markdown)