# feat: store packet\- and byte-weighted MAAD measures · PR \#112 · flambo/atlantis

[View on GitCafe](https://git.cafe/flambo/atlantis/pull/112)

Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis)

Visibility: public

State: merged

Source repository: flambo/atlantis

Source branch: maad/06-weighted-pipeline

Target branch: maad/05-weighted-estimator

Head commit: da1fe86c996136d7df3ccb81325523159a618144

[Discussion and reviews](https://git.cafe/flambo/atlantis/pull/112)

[Commits](https://git.cafe/flambo/atlantis/pull/112/commits?format=markdown)

[Files changed](https://git.cafe/flambo/atlantis/pull/112/diffs?format=markdown)

## Description

---

> [!NOTE]
> 🤖 **Claude Opus 5.5 on behalf of Oliver**

## What changes

The pipeline stores packet- and byte-weighted MAAD next to the existing address-count MAAD for every bucket, scope, and address side. Flows that report 0 packets are dropped before aggregation. For now you switch measures on the file detail page. A dashboard view of the weighted measures will follow in a PR on top of #119.

## Flows to exercise

**Setup.** Build a small CSV fixture in an empty directory `$FIX` (all addresses are documentation ranges):

```bash
cat > $FIX/flows.csv <<'EOF'
received,src,dst,packets,bytes,protocol
0,192.0.2.1,198.51.100.1,4,400,TCP
10,192.0.2.1,198.51.100.9,0,0,TCP
20,192.0.2.2,198.51.100.1,1,60,UDP
300,192.0.2.3,198.51.100.3,0,40,UDP
3600,192.0.2.1,198.51.100.1,2,120,TCP
EOF
cat > $FIX/mapping.json <<'EOF'
{"timestamp_format":"unix","timestamp_timezone":"UTC","columns":{"time_received":"received","src_ip":"src","dst_ip":"dst","packets":"packets","bytes":"bytes","protocol":"protocol"},"protocol_map":{"TCP":6,"UDP":17},"source_id":{"value":"edge"}}
EOF
cat > $FIX/pipeline.json <<EOF
{"database_path":"$FIX/fixture.sqlite","timezone":"UTC","locality":[{"type":"prefixes","prefixes":["192.0.2.0/24"]}],"inputs":[{"input_kind":"csv","path":"$FIX/flows.csv","mapping_path":"$FIX/mapping.json"}]}
EOF
```

1. **Pipeline and the zero-packet drop.** Run `RUST_LOG=info ./scripts/netflow-db.sh pipeline --config $FIX/pipeline.json --maad-workers 2`.
   - Expect: `3 complete, 0 partial, 10 unknown` coverage. The logs show `dropped zero-packet flows before aggregation` with `dropped_zero_packet_flows=1` for buckets `0` and `300`.
   - `SELECT bucket_start, flows, packets FROM traffic_stats WHERE granularity='5m' AND ip_version=4 AND src_locality='all' AND dst_locality='all' AND bucket_start <= 300` returns `0|2|5` and `300|0|0`. Bucket 300 stays observed (complete coverage, zero traffic) even though its only flow was dropped.
   - `SELECT measure, COUNT(*) FROM address_structure_stats WHERE structure_kind='structure' GROUP BY measure` returns equal counts for `addresses`, `packets`, and `bytes`.
2. **Verify.** On a dataset-mode database (datasets registry with `datasets.json`, per `docs/user/setup-pipeline.md`), run `./scripts/netflow-db.sh verify <db> --dataset-id <id> --require-data --require-maad-data --require-processed --require-rollup-parity --require-no-raw-ip`. Expect `OK`. To see the per-key check fail, delete a single `bytes` structure row for one bucket and side. Expect a nonzero exit and `address_structure_stats is missing bytes structure rows for source … bucket … <side>`.
3. **`--maad-workers`.** `--maad-workers 0` (or `-1` or `many`) is rejected by argument parsing. Omit the option to keep the default of available parallelism capped at 8. The worker count is not part of the product identity, so changing it on an existing database does not force a rebuild.
4. **File page.** Run `bun run dev` against the database. Open a file detail page and switch the measure between addresses, packets, and bytes. Spectrum is hidden for packets and bytes.
5. **API.** `/api/netflow/structure-stats?measure=packets` returns rows. `/api/netflow/spectrum-stats?measure=packets` returns 400.

## Edge cases and decisions

- **Zero-packet flows are dropped, not rounded up to 1.** This was the owner's decision. The rule applies to native `nfcapd` and CSV input (sorted and unsorted CSV) and to daily-active-source qualification. A dropped flow adds nothing to traffic, protocols, ports, or address sets. It also adds nothing to the `addresses` measure, so an address seen only in zero-packet flows disappears from every product. Bucket coverage is unchanged. The pipeline logs the dropped flow count for each bucket, counted as `flow_count` rather than records.
- **Missing packet count ≠ reported zero.** Some CSV sources have no packets column, or a row has an empty cell. Those rows keep contributing. Their addresses become zero-weight for `packets` and are counted in `zeroWeightAddrs`. Without this rule, a CSV dataset with no packets column would publish as empty.
- **Weighted measures:** Addresses with a summed bytes value of 0 are left out of `bytes`, as are packets-unknown CSV addresses from `packets`. Their count is in `metadata_json.zeroWeightAddrs`. Prefix validity still uses distinct-address counts. Weighted measures have no spectrum. The owner judged it degenerate, and a DB CHECK enforces its absence.
- **Product identity:** The pipeline contract goes to v7, and the measure set is part of the identity, so this needs a fresh database. The zero-packet rule ships inside the same version bump. Drizzle `0005` marks existing D1 rows as `addresses`.
- **Verify** checks measure presence per full scope key (source, granularity, bucket, IP version, locality pair, side), not global counts.
- **Cost (accepted by owner):** On a real day with the full stack, wall time was 24:37 vs 16:41 on main, and peak RSS was 5.9 GB vs 3.3 GB. Storage is 658 MB/day vs 125 MB. #119 compacts MAAD storage before the reprocess.

## Verification

- Automated: `cargo fmt --all --check`, `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings`, `bun run format`, `bun run lint`, `bun run typecheck`, `bun run test:db`. `bun run test:web` covers the web layer from the original PR; the review fixes don't touch web code.
- New regressions:
  - native zero-only and mixed flows sharing a source address
  - CSV sorted and unsorted, zero-only and mixed, plus a CSV with no packets column
  - an end-to-end CSV `pipeline` CLI run that asserts the drop logs and the observed zero bucket
  - publish profile drop count
  - verify with equal global counts but mismatched keys
  - `--maad-workers` parsing and pool-resize rejection
- Remaining manual checks:
  - flows 4–5 in a browser
  - one real-day `pipeline` + `verify --require-maad-data` on a fresh database, checking the zero-packet drop log volume