# agent: Migrate thread storage to SQLite with zstd compression \(\#31741\) · gitcafe/zed

[View on GitCafe](https://git.cafe/gitcafe/zed/commit/c874f1fa9d70c889602e54a8f6ac3ad8c2a2734a)

Repository: [gitcafe/zed](https://git.cafe/gitcafe/zed)

Visibility: public

Requested revision: c874f1fa9d70c889602e54a8f6ac3ad8c2a2734a

Requested commit: c874f1fa9d70c889602e54a8f6ac3ad8c2a2734a

Commit: c874f1fa9d70c889602e54a8f6ac3ad8c2a2734a

Tree: 3705d0b9c8caf72ae04d20d6a78ca7f602de3d7a

Author: Oleksiy Syvokon

Committer: GitHub

## Message

````
agent: Migrate thread storage to SQLite with zstd compression (#31741)

Previously, LMDB was used for storing threads, but it consumed excessive
disk space and was capped at 1GB.

This change migrates thread storage to an SQLite database. Thread JSON
objects are now compressed using zstd.

I considered training a custom zstd dictionary and storing it in a
separate table. However, the additional complexity outweighed the modest
space savings (up to 20%). I ended up using the default dictionary
stored with data.

Threads can be exported relatively easily from outside the application:

```
$ sqlite3 threads.db "SELECT hex(data) FROM threads LIMIT 5;" |
    xxd -r -p |
    zstd -d |
    fx
```

Benchmarks:
- Original heed database: 200MB
- Sqlite uncompressed: 51MB
- sqlite compressed (this PR): 4.0MB
- sqlite compressed with a trained dictionary: 3.8MB


Release Notes:

- Migrated thread storage to SQLite with compression
````

## Parents

- [9a9e96ed5a4d3a31cf075a31c3df3186c19496c5](https://git.cafe/gitcafe/zed/commit/9a9e96ed5a4d3a31cf075a31c3df3186c19496c5?format=markdown)

[Source at this commit](https://git.cafe/gitcafe/zed/tree/c874f1fa9d70c889602e54a8f6ac3ad8c2a2734a?format=markdown)