feat: store packet- and byte-weighted MAAD measures#112
Note
🤖 Claude Opus 5.5 on behalf of Oliver
What changes
The pipeline stores packet- and byte-weighted MAAD next to the existing address-count MAAD for every bucket, scope, and address side. Flows that report 0 packets are dropped before aggregation. For now you switch measures on the file detail page. A dashboard view of the weighted measures will follow in a PR on top of #119.
Flows to exercise
Setup. Build a small CSV fixture in an empty directory $FIX (all addresses are documentation ranges):
bash
| 1 | cat > $FIX/flows.csv <<'EOF' |
| 2 | received,src,dst,packets,bytes,protocol |
| 3 | 0,192.0.2.1,198.51.100.1,4,400,TCP |
| 4 | 10,192.0.2.1,198.51.100.9,0,0,TCP |
| 5 | 20,192.0.2.2,198.51.100.1,1,60,UDP |
| 6 | 300,192.0.2.3,198.51.100.3,0,40,UDP |
| 7 | 3600,192.0.2.1,198.51.100.1,2,120,TCP |
| 8 | EOF |
| 9 | cat > $FIX/mapping.json <<'EOF' |
| 10 | {"timestamp_format":"unix","timestamp_timezone":"UTC","columns":{"time_received":"received","src_ip":"src","dst_ip":"dst","packets":"packets","bytes":"bytes","protocol":"protocol"},"protocol_map":{"TCP":6,"UDP":17},"source_id":{"value":"edge"}} |
| 11 | EOF |
| 12 | cat > $FIX/pipeline.json <<EOF |
| 13 | {"database_path":"$FIX/fixture.sqlite","timezone":"UTC","locality":[{"type":"prefixes","prefixes":["192.0.2.0/24"]}],"inputs":[{"input_kind":"csv","path":"$FIX/flows.csv","mapping_path":"$FIX/mapping.json"}]} |
| 14 | EOF |
- Pipeline and the zero-packet drop. Run
RUST_LOG=info ./scripts/netflow-db.sh pipeline --config $FIX/pipeline.json --maad-workers 2.- Expect:
3 complete, 0 partial, 10 unknowncoverage. The logs showdropped zero-packet flows before aggregationwithdropped_zero_packet_flows=1for buckets0and300. SELECT bucket_start, flows, packets FROM traffic_stats WHERE granularity='5m' AND ip_version=4 AND src_locality='all' AND dst_locality='all' AND bucket_start <= 300returns0|2|5and300|0|0. Bucket 300 stays observed (complete coverage, zero traffic) even though its only flow was dropped.SELECT measure, COUNT(*) FROM address_structure_stats WHERE structure_kind='structure' GROUP BY measurereturns equal counts foraddresses,packets, andbytes.
- Expect:
- Verify. On a dataset-mode database (datasets registry with
datasets.json, perdocs/user/setup-pipeline.md), run./scripts/netflow-db.sh verify <db> --dataset-id <id> --require-data --require-maad-data --require-processed --require-rollup-parity --require-no-raw-ip. ExpectOK. To see the per-key check fail, delete a singlebytesstructure row for one bucket and side. Expect a nonzero exit andaddress_structure_stats is missing bytes structure rows for source … bucket … <side>. --maad-workers.--maad-workers 0(or-1ormany) is rejected by argument parsing. Omit the option to keep the default of available parallelism capped at 8. The worker count is not part of the product identity, so changing it on an existing database does not force a rebuild.- File page. Run
bun run devagainst the database. Open a file detail page and switch the measure between addresses, packets, and bytes. Spectrum is hidden for packets and bytes. - API.
/api/netflow/structure-stats?measure=packetsreturns rows./api/netflow/spectrum-stats?measure=packetsreturns 400.
Edge cases and decisions
- Zero-packet flows are dropped, not rounded up to 1. This was the owner's decision. The rule applies to native
nfcapdand CSV input (sorted and unsorted CSV) and to daily-active-source qualification. A dropped flow adds nothing to traffic, protocols, ports, or address sets. It also adds nothing to theaddressesmeasure, so an address seen only in zero-packet flows disappears from every product. Bucket coverage is unchanged. The pipeline logs the dropped flow count for each bucket, counted asflow_countrather than records. - Missing packet count ≠reported zero. Some CSV sources have no packets column, or a row has an empty cell. Those rows keep contributing. Their addresses become zero-weight for
packetsand are counted inzeroWeightAddrs. Without this rule, a CSV dataset with no packets column would publish as empty. - Weighted measures: Addresses with a summed bytes value of 0 are left out of
bytes, as are packets-unknown CSV addresses frompackets. Their count is inmetadata_json.zeroWeightAddrs. Prefix validity still uses distinct-address counts. Weighted measures have no spectrum. The owner judged it degenerate, and a DB CHECK enforces its absence. - Product identity: The pipeline contract goes to v7, and the measure set is part of the identity, so this needs a fresh database. The zero-packet rule ships inside the same version bump. Drizzle
0005marks existing D1 rows asaddresses. - Verify checks measure presence per full scope key (source, granularity, bucket, IP version, locality pair, side), not global counts.
- Cost (accepted by owner): On a real day with the full stack, wall time was 24:37 vs 16:41 on main, and peak RSS was 5.9 GB vs 3.3 GB. Storage is 658 MB/day vs 125 MB. #119 compacts MAAD storage before the reprocess.
Verification
- Automated:
cargo fmt --all --check,cargo clippy --workspace --all-targets --all-features --locked -- -D warnings,bun run format,bun run lint,bun run typecheck,bun run test:db.bun run test:webcovers the web layer from the original PR; the review fixes don't touch web code. - New regressions:
- native zero-only and mixed flows sharing a source address
- CSV sorted and unsorted, zero-only and mixed, plus a CSV with no packets column
- an end-to-end CSV
pipelineCLI run that asserts the drop logs and the observed zero bucket - publish profile drop count
- verify with equal global counts but mismatched keys
--maad-workersparsing and pool-resize rejection
- Remaining manual checks:
- flows 4–5 in a browser
- one real-day
pipeline+verify --require-maad-dataon a fresh database, checking the zero-packet drop log volume
Checks
- SuccessfulPlaywright E2EGitHub
- SuccessfulWeb ChecksGitHub
- SuccessfulRust Pipeline ChecksGitHub
Added 8 commits
- feat(netflow-db): store packet- and byte-weighted MAAD measuresb987c2d
- feat(web): choose the MAAD measure on file details2dd7548
- perf(netflow-db): compute a scope's MAAD measures together and in parallel4595418
- perf(netflow-db): decode the next nfcapd batch while publishing the current one6cca5a1
- fix(netflow-db): surface later-batch errors only after the current batch publishesd2d4a5b
- fix(netflow-db): drop and log zero-packet flows before aggregation1898b00
- feat(netflow-db): add --maad-workers pipeline optionda1fe86
- fix(netflow-db): verify MAAD measure presence per scope keyf32aba1