# stage1: tokenizer: backport recent stage2 changes · gitcafe/zig

[View on GitCafe](https://git.cafe/gitcafe/zig/commit/9de2c1b76e2b38e35bb17cd0a3943a61c730da76)

Repository: [gitcafe/zig](https://git.cafe/gitcafe/zig)

Visibility: public

Requested revision: 9de2c1b76e2b38e35bb17cd0a3943a61c730da76

Requested commit: 9de2c1b76e2b38e35bb17cd0a3943a61c730da76

Commit: 9de2c1b76e2b38e35bb17cd0a3943a61c730da76

Tree: 1e23f3926d2757e546107bf75f913a2fe75f3c3d

Author: Andrew Kelley

Committer: Andrew Kelley

## Message

```
stage1: tokenizer: backport recent stage2 changes

This backports ec10595b65490126e1b7d231f655be6d7d198acf stage2 tokenizer
logic to stage1.

```

## Parents

- [01e0a4b2f06d60a58b3d2df410776b164d05c1ec](https://git.cafe/gitcafe/zig/commit/01e0a4b2f06d60a58b3d2df410776b164d05c1ec?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zig/tree/9de2c1b76e2b38e35bb17cd0a3943a61c730da76?format=markdown)