# Improve token counting for OpenAI models \(\#30242\) · gitcafe/zed

[View on GitCafe](https://git.cafe/gitcafe/zed/commit/3a3d3c05e862c48bb2c79e27c50f08556ec349b5)

Repository: [gitcafe/zed](https://git.cafe/gitcafe/zed)

Visibility: public

Requested revision: 3a3d3c05e862c48bb2c79e27c50f08556ec349b5

Requested commit: 3a3d3c05e862c48bb2c79e27c50f08556ec349b5

Commit: 3a3d3c05e862c48bb2c79e27c50f08556ec349b5

Tree: 2b18bac2215c8e4017fe95da634be967a7017767

Author: Ben Brandt

Committer: GitHub

## Message

```
Improve token counting for OpenAI models (#30242)

tiktoken_rs is a bit behind (and even upstream tiktoken doesn't have all
of these models)

We were incorrectly using the cl100k tokenizer for some models that
actually use the o200k tokenizers. So that is updated.

I also made the match arms specific so that we do a better job of
catching whether or not tiktoken-rs accurately supports new models we
add in.

I will also do a PR upstream to see if we can move some of this logic
back out if tiktoken better supports the newer models.

Release Notes:

- Improved tokenizer support for openai models.
```

## Parents

- [ee56706d158a6d2da9c2f2961bb8a133eafe1780](https://git.cafe/gitcafe/zed/commit/ee56706d158a6d2da9c2f2961bb8a133eafe1780?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zed/tree/3a3d3c05e862c48bb2c79e27c50f08556ec349b5?format=markdown)