# Merge pull request \#20885 from ziglang/simplify-tokenizer · gitcafe/zig

[View on GitCafe](https://git.cafe/gitcafe/zig/commit/eb1a199dff2b54271bd275c2528bdd898bf1d4eb)

Repository: [gitcafe/zig](https://git.cafe/gitcafe/zig)

Visibility: public

Requested revision: eb1a199dff2b54271bd275c2528bdd898bf1d4eb

Requested commit: eb1a199dff2b54271bd275c2528bdd898bf1d4eb

Commit: eb1a199dff2b54271bd275c2528bdd898bf1d4eb

Tree: 0cef5904cef4e638dc67248384663c6779b5e293

Author: Andrew Kelley

Committer: GitHub

## Message

```
Merge pull request #20885 from ziglang/simplify-tokenizer

std.zig.tokenizer: simplification and spec conformance
```

## Parents

- [059856acfc9f87d723a90af6a4214e128b8cae2e](https://git.cafe/gitcafe/zig/commit/059856acfc9f87d723a90af6a4214e128b8cae2e?format=markdown)

- [c2b8afcac9e427102370dc5bac8c3d9621eee6d8](https://git.cafe/gitcafe/zig/commit/c2b8afcac9e427102370dc5bac8c3d9621eee6d8?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zig/tree/eb1a199dff2b54271bd275c2528bdd898bf1d4eb?format=markdown)