# feat\(netflow-db\): add measure-weighted MAAD estimator · PR \#111 · flambo/atlantis

[View on GitCafe](https://git.cafe/flambo/atlantis/pull/111)

Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis)

Visibility: public

State: merged

Source repository: flambo/atlantis

Source branch: maad/05-weighted-estimator

Target branch: maad/04-ipv6

Head commit: b7aff01ad8acc14e1a927e3961e42e320c2c34b7

[Discussion and reviews](https://git.cafe/flambo/atlantis/pull/111)

[Commits](https://git.cafe/flambo/atlantis/pull/111/commits?format=markdown)

[Files changed](https://git.cafe/flambo/atlantis/pull/111/diffs?format=markdown)

## Description

---

> [!NOTE]
> 🤖 **Claude Opus 5.5 on behalf of Oliver**

## ELI5

Teaches the MAAD analysis to weight each address by how much traffic it sent (packets or bytes), instead of counting every address once.

## Why

Address-count MAAD misses volume anomalies. In our study, one bogus flow claiming 465 G packets was invisible to address-count MAAD but drove packet-weighted D1 to 0. A source-address spike showed the opposite pattern. The two views complement each other.

## Flows to exercise

1. **Weighted CLI vs the upstream oracle.** Setup: a release build (`cargo build --release -p atlantis-netflow-db`), an upstream MAAD checkout, and an `address,packets` CSV. Run `python3 scripts/local/validate_maad.py --weighted --rust target/release/netflow-db --haskell /path/to/MAAD name=addr_packets.csv`. Expected: every structure row and D0/D1/D2 match within the validator's default 1e-10 tolerance.
2. **Invalid weights.** Add a row with weight `0`, a negative weight, or `NaN` and run `netflow-db maad --weighted`. Expected: the command exits non-zero and names the offending address.
3. **Unweighted output unchanged.** `netflow-db maad` without `--weighted` produces byte-identical JSON to the base branch.

## Decisions and edge cases

- Matches upstream `--csv --meas-col`. Distinct-address counts decide prefix validity and pruning. Summed weights drive moments, D2, and D1 entropy. Weights on duplicate addresses are summed.
- Weights must be finite and > 0, and their total must be finite. There is no weighted spectrum, because upstream's spectrum collapses on weighted measures.
- **Scale invariance.** Multiplying every weight by a constant does not change the result. If a q's raw powers overflow or underflow, or a moment's partition sum is not finite, that q is recomputed from masses rescaled within each moment. Example: `10.0.0.0` and `10.0.0.128` with equal weights give D2 = 1/17 whether the weights are 1, 1e160, or 1e-200. Before this fix, 1e160 gave NaN and 1e-200 gave 0. Ordinary inputs keep the raw path, so unit weights still reproduce the unweighted result exactly.
- **Masses that normalize to zero.** A positive weight can be too small to register against the total, such as 1e-300 next to 1e50. It contributes 0 to the entropy (the 0·log 0 = 0 limit), so D1 stays finite instead of becoming NaN.
- Performance: all measures share one prefix walk, each distinct mass is raised to a power once per q, and q values run in parallel.

## Follow-up

#112 covers the integration work: handling zero-packet flows, storing weighted results, and showing them in the dashboard. This PR only adds the estimator and the CLI/validator surface.

## Verification

- Automated: `cargo fmt --all --check`, `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings`, and `cargo test -p atlantis-netflow-db maad`. The tests cover:
  - unit weights reproduce the unweighted result exactly;
  - permutation and duplicate invariance;
  - a hand-computed case;
  - an independent ordered-map reference;
  - scale invariance across 1e-300…1e300;
  - finite D1 when masses normalize to zero.
- The unweighted JSON was byte-identical to `main` on fixtures and 20 real windows. Weighted output matched the oracle to about 1e-15.
- Remaining manual verification: none. Flow 1 is an optional spot check for reviewers who have an upstream MAAD checkout.

---
Made by Claude Opus 5.5 (with Opus 5.5 subagents) via Claude Code.