# language_models: Make Copilot Chat resilient to new model vendors and add tokenizer-based token counting \(\#37118\) · gitcafe/zed

[View on GitCafe](https://git.cafe/gitcafe/zed/commit/2e36e9782ee2235e97d765387ceb0fed57a406af)

Repository: [gitcafe/zed](https://git.cafe/gitcafe/zed)

Visibility: public

Requested revision: 2e36e9782ee2235e97d765387ceb0fed57a406af

Requested commit: 2e36e9782ee2235e97d765387ceb0fed57a406af

Commit: 2e36e9782ee2235e97d765387ceb0fed57a406af

Tree: 736dee0f40f658d4d84a84ee22a3529b38247a83

Author: Umesh Yadav

Committer: GitHub

## Message

```
language_models: Make Copilot Chat resilient to new model vendors and add tokenizer-based token counting (#37118)

While working on fixing this: #37116. I reliased the current
implementation of github copilot is not truly resilient to upstream
changes.

This PR enhances GitHub Copilot Chat to be forward-compatible with new
AI model vendors and improves token counting accuracy by using
vendor-specific tokenizers from the GitHub Copilot API. The system
previously failed when GitHub added new model vendors like xAI with
deserialization errors, and token counting wasn't utilizing the
vendor-specific tokenizer information provided by the API. The solution
adds an Unknown variant to the ModelVendor enum with serde other
attribute to gracefully handle any new vendors GitHub introduces,
implements tokenizer-aware token counting that uses the model's
specified tokenizer mapping o200k_base to gpt-4o with fallback, adds
explicit support for xAI models with proper tool input format handling,
and includes comprehensive test coverage for unknown vendor scenarios.
Key changes include adding the tokenizer field to model capabilities,
implementing the tokenizer method on models, updating tool input format
logic to handle unknown vendors, and simplifying token counting to use
the vendor's specified tokenizer or fall back to gpt-4o. This ensures
Zed's Copilot Chat integration remains robust and accurate as GitHub
continues expanding their AI model provider ecosystem.

Release Notes:

- Enhanced model vendor compatibility to automatically support future AI
providers and improved token counting accuracy using vendor-specific
tokenizers from the GitHub Copilot

---------

Signed-off-by: Umesh Yadav <git@umesh.dev>
```

## Parents

- [1751bf4cdbe8b6a55546d290c7f2a0f47721b041](https://git.cafe/gitcafe/zed/commit/1751bf4cdbe8b6a55546d290c7f2a0f47721b041?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zed/tree/2e36e9782ee2235e97d765387ceb0fed57a406af?format=markdown)