# language_models: Update tiktoken-rs to support newer models \(\#30951\) · gitcafe/zed

[View on GitCafe](https://git.cafe/gitcafe/zed/commit/57424e47434e56bb489eb6a55687f68e4d4f6bea)

Repository: [gitcafe/zed](https://git.cafe/gitcafe/zed)

Visibility: public

Requested revision: 57424e47434e56bb489eb6a55687f68e4d4f6bea

Requested commit: 57424e47434e56bb489eb6a55687f68e4d4f6bea

Commit: 57424e47434e56bb489eb6a55687f68e4d4f6bea

Tree: 5da5958907edbf9bec6265f46de4a76ff16f2d89

Author: Ben Brandt

Committer: GitHub

## Message

```
language_models: Update tiktoken-rs to support newer models (#30951)

I was able to get this fix in upstream, so now we can have simpler code
paths for our model selection.

I also added a test to catch if this would cause a bug again in the
future.

Release Notes:

- N/A
```

## Parents

- [2b6dab91972725c7e540cccb6adf1cb015421158](https://git.cafe/gitcafe/zed/commit/2b6dab91972725c7e540cccb6adf1cb015421158?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zed/tree/57424e47434e56bb489eb6a55687f68e4d4f6bea?format=markdown)