# Tokenizer bug fixes and improvements · gitcafe/zig

[View on GitCafe](https://git.cafe/gitcafe/zig/commit/c50f3003874d561aa696feb71f2089562307cdca)

Repository: [gitcafe/zig](https://git.cafe/gitcafe/zig)

Visibility: public

Requested revision: c50f3003874d561aa696feb71f2089562307cdca

Requested commit: c50f3003874d561aa696feb71f2089562307cdca

Commit: c50f3003874d561aa696feb71f2089562307cdca

Tree: 63990eafe12e6fa5045851d865d389fbe0386fc6

Author: gooncreeper

Committer: Veikka Tuominen

## Message

```
Tokenizer bug fixes and improvements

Fixes many error messages corresponding to invalid bytes displaying the
wrong byte. Additionaly improves handling of UTF-8 in some places.

```

## Parents

- [9d38e82b5c0f28ea6a2d8d31ebd73b6e2a8aad26](https://git.cafe/gitcafe/zig/commit/9d38e82b5c0f28ea6a2d8d31ebd73b6e2a8aad26?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zig/tree/c50f3003874d561aa696feb71f2089562307cdca?format=markdown)