# std.zig.tokenizer: simplify line-based tokens · gitcafe/zig

[View on GitCafe](https://git.cafe/gitcafe/zig/commit/90075345519f165a0164c52b62b78453068c6ce6)

Repository: [gitcafe/zig](https://git.cafe/gitcafe/zig)

Visibility: public

Requested revision: 90075345519f165a0164c52b62b78453068c6ce6

Requested commit: 90075345519f165a0164c52b62b78453068c6ce6

Commit: 90075345519f165a0164c52b62b78453068c6ce6

Tree: ca83ba36ab298b92db9be64cb46a42a1a608166b

Author: Ian Johnson

Committer: Veikka Tuominen

## Message

```
std.zig.tokenizer: simplify line-based tokens

Closes #21358
Closes #21360

This commit modifies the `multiline_string_literal_line`, `doc_comment`,
and `container_doc_comment` tokens to no longer include the line ending
as part of the token. This makes it easier to handle line endings (which
may be LF, CRLF, or in edge cases possibly nonexistent) consistently.

In the two issues linked above, Autodoc was already assuming this for
doc comments, and yielding incorrect results when handling files with
CRLF line endings (both in Markdown parsing and source rendering).

Applying the same simplification for multiline string literals also
brings `zig fmt` into conformance with
https://github.com/ziglang/zig-spec/issues/38 regarding formatting of
multiline strings with CRLF line endings: the spec says that `zig fmt`
should remove the CR from such line endings, but this was not previously
the case.

```

## Parents

- [d6d09f4ea7cc590035bf8af6617f93767423c691](https://git.cafe/gitcafe/zig/commit/d6d09f4ea7cc590035bf8af6617f93767423c691?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zig/tree/90075345519f165a0164c52b62b78453068c6ce6?format=markdown)