# std.mem: Split \`tokenize\` into 3 versions by delimiter type: full, any, and scalar · gitcafe/zig

[View on GitCafe](https://git.cafe/gitcafe/zig/commit/9da3a9733db037d8391eb2b4ccb6c7db774a5dd5)

Repository: [gitcafe/zig](https://git.cafe/gitcafe/zig)

Visibility: public

Requested revision: 9da3a9733db037d8391eb2b4ccb6c7db774a5dd5

Requested commit: 9da3a9733db037d8391eb2b4ccb6c7db774a5dd5

Commit: 9da3a9733db037d8391eb2b4ccb6c7db774a5dd5

Tree: d89ffa89da45157daaebd9ddd34a5a5af4749d5f

Author: Ryan Liptak

Committer: Ryan Liptak

## Message

```
std.mem: Split `tokenize` into 3 versions by delimiter type: full, any, and scalar

This allows users to choose which version they need for their particular use case, as the previous default (now the 'any' version) was (1) not always the desired type of delimiter and (2) performed worse than the scalar version if the delimiter was a single item.

```

## Parents

- [bda645d911fd79c5aba1f93c0877c7eb915ed709](https://git.cafe/gitcafe/zig/commit/bda645d911fd79c5aba1f93c0877c7eb915ed709?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zig/tree/9da3a9733db037d8391eb2b4ccb6c7db774a5dd5?format=markdown)