# stage2: improve compile errors from tokenizer · gitcafe/zig

[View on GitCafe](https://git.cafe/gitcafe/zig/commit/24c432608f6b07020fa0b18fc9c868ad6abd9b15)

Repository: [gitcafe/zig](https://git.cafe/gitcafe/zig)

Visibility: public

Requested revision: 24c432608f6b07020fa0b18fc9c868ad6abd9b15

Requested commit: 24c432608f6b07020fa0b18fc9c868ad6abd9b15

Commit: 24c432608f6b07020fa0b18fc9c868ad6abd9b15

Tree: f9650d8c9aa36d6fddd45b1f70847304da16c0f5

Author: Andrew Kelley

Committer: Andrew Kelley

## Message

```
stage2: improve compile errors from tokenizer

In order to not regress the quality of compile errors, some improvements
had to be made.

 * std.zig.parseCharLiteral is improved to return more detailed parse
   failure information.
 * tokenizer is improved to handle null bytes in the middle of strings,
   character literals, and line comments.
 * validating how many unicode escape digits in string literals is moved
   to std.zig.parseStringLiteral rather than handled in the tokenizer.
 * when a tokenizer error occurs, if the reported token is the 'invalid'
   tag, an error note is added to point to the invalid byte location.
   Further improvements would be:
   - Mention the expected set of allowed bytes at this location.
   - Display the invalid byte (if printable, print it, otherwise
     escape-print it).

```

## Parents

- [3f680abbe2c4d2eeefd0eb73b8af25d1768e6ceb](https://git.cafe/gitcafe/zig/commit/3f680abbe2c4d2eeefd0eb73b8af25d1768e6ceb?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zig/tree/24c432608f6b07020fa0b18fc9c868ad6abd9b15?format=markdown)