# Merge pull request \#8918 from ziglang/stage1-tokenizer · gitcafe/zig

[View on GitCafe](https://git.cafe/gitcafe/zig/commit/c12704a3395a11c28569ebb160e6ac33091d983a)

Repository: [gitcafe/zig](https://git.cafe/gitcafe/zig)

Visibility: public

Requested revision: c12704a3395a11c28569ebb160e6ac33091d983a

Requested commit: c12704a3395a11c28569ebb160e6ac33091d983a

Commit: c12704a3395a11c28569ebb160e6ac33091d983a

Tree: cf2e0055c53eac499b1fb66e2472a2036286afd6

Author: Andrew Kelley

Committer: GitHub

## Message

```
Merge pull request #8918 from ziglang/stage1-tokenizer

stage1: rework tokenizer to match stage2
```

## Parents

- [3f5ca3920a58d44a018ff2a2e277e60813e20d5a](https://git.cafe/gitcafe/zig/commit/3f5ca3920a58d44a018ff2a2e277e60813e20d5a?format=markdown)

- [d888fa12a88ae7e5332de669b3c3a1dac57f5c19](https://git.cafe/gitcafe/zig/commit/d888fa12a88ae7e5332de669b3c3a1dac57f5c19?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zig/tree/c12704a3395a11c28569ebb160e6ac33091d983a?format=markdown)