# Evaluate zeta2 context retrieval and edit predictions \(\#41921\) · gitcafe/zed

[View on GitCafe](https://git.cafe/gitcafe/zed/commit/91d631c22920518dfe025a40dbcbd1bbb867784c)

Repository: [gitcafe/zed](https://git.cafe/gitcafe/zed)

Visibility: public

Requested revision: 91d631c22920518dfe025a40dbcbd1bbb867784c

Requested commit: 91d631c22920518dfe025a40dbcbd1bbb867784c

Commit: 91d631c22920518dfe025a40dbcbd1bbb867784c

Tree: acdb897038fe395397727ad3ae33b4e96ccec4fc

Author: Oleksiy Syvokon

Committer: GitHub

## Message

```
Evaluate zeta2 context retrieval and edit predictions (#41921)

This PR implements the `zeta-cli eval` command. It will:

- Run the edit prediction model if there are no cached results
- Compute precision/recall/F1 for context retrieval at the line level:
every retrieved line of context is counted as a true positive (correct
retrieval), false positive (retrieved something that was not expected),
or false negative (didn't retrieve an expected line)
- Compute similar metrics for edit predictions
- Pretty-print results, highlighting the difference between actual and
expected when printing to tty

Other changes:
- `zeta-cli predict` accepts a `--format` argument with options `md`,
`json`, `diff`
- Code restructure

Release Notes:

- N/A

---------

Co-authored-by: Piotr Osiewicz <24362066+osiewicz@users.noreply.github.com>
Co-authored-by: Agus Zubiaga <agus@zed.dev>
```

## Parents

- [054d2e15247169f41368b3d9c1eb4ebfb99426d8](https://git.cafe/gitcafe/zed/commit/054d2e15247169f41368b3d9c1eb4ebfb99426d8?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zed/tree/91d631c22920518dfe025a40dbcbd1bbb867784c?format=markdown)