# feat\(netflow-db\): sync in-process MAAD estimator · PR \#93 · flambo/atlantis [View on GitCafe](https://git.cafe/flambo/atlantis/pull/93) Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis) Visibility: public State: merged Source repository: flambo/atlantis Source branch: t3code/review-upstream-maad Target branch: t3code/refine-rust-ci-config Head commit: 0c0ff5a3e5d90e03f5599b867beb24c4593b22ba [Discussion and reviews](https://git.cafe/flambo/atlantis/pull/93) [Commits](https://git.cafe/flambo/atlantis/pull/93/commits?format=markdown) [Files changed](https://git.cafe/flambo/atlantis/pull/93/diffs?format=markdown) ## Description --- ## Problem The experimental Rust MAAD estimator still used the earlier global atomic/spillover prefix range and reported the old subprocess contract. Current upstream MAAD filters each candidate prefix level through Patricia-tree ancestry and uses different variance aggregation, so our persisted results could diverge. ## Solution ### Product impact - Match upstream's fixed /8 through /24 candidate range, per-prefix atomic/nearly-full filtering, invalid-ancestor pruning, and dropping of empty levels. - Persist the exact retained prefix lengths and bump the MAAD result/identity contract to version 2 so incompatible databases cannot be mixed. - Add a typed `MaadConfig` for q bounds/step, prefix bounds, and fullness threshold. - Keep the pipeline default at q-step 1/8 (33 structure rows), intentionally avoiding upstream's newer 1/16 default and its doubled storage cost. ### Fixes and internal changes - Correct standard-deviation aggregation to upstream's `sqrt(sum(d2)) / n`. - Handle empty, singleton, duplicate, sparse, and fully filtered address sets safely. - Reject invalid/unbounded q grids, require generated q=0 and q=2 rows, and cap configurable grids at 1,025 values. - Ignore sub-epsilon floating-point noise when selecting decreasing spectrum segments, so a mathematically linear structure curve does not produce spurious spectrum rows. - Record the in-process backend and complete default MAAD configuration in pipeline product identity. - Add focused reference, numerical-oracle, pruning, configuration, schema, identity, and spectrum-stability tests. - Run five pinned Haskell-derived golden fixtures through the public Rust CLI in normal PR CI. - Provide a standard-library local validator for arbitrary external address files without committing private data or requiring Haskell in CI. Upstream reference: `chris-misa/maad@3ae7536`. The q-step is the intentional configuration exception. ## Reviewer fast path 1. Run `cargo test -p atlantis-netflow-db --test maad_conformance --locked`; expect five golden cases to pass in roughly 0.04 seconds of test runtime. 2. Put at least two IPv4 addresses, one per line, in a temporary file. 3. Run `./scripts/netflow-db.sh maad `. 4. Expect schema version 2, explicit `metadata.prefixLengths`, 33 structure rows for non-empty valid input, and finite sparse-input output. 5. Run a fresh pipeline with `run_maad: true`; expect `pipeline_product.config_json.maad` to identify backend `in-process`, contract version 2, and q-step `0.125`. Important edge cases are invalid/full ancestor pruning, empty valid-level sets, duplicate addresses, non-aligned q grids, linear structure curves, and product-identity incompatibility. Changing these semantics requires a fresh product database. ## Cross-language conformance The optimized upstream Haskell executable was built from `3ae7536` with only `deltaQ` changed from 1/16 to 1/8. Both executables received the same deduplicated IPv4 files through their public JSON interfaces. Row counts and retained prefix levels were compared exactly; numeric fields used absolute + relative tolerance `1e-10`. - 13/13 real-address cases passed: 10 UOregon-derived source/destination sets (16,526–220,718 unique addresses), upstream's CAIDA 100k fixture, and two deterministic sets derived from Cloudflare's published IPv4 ranges. - The largest real-case numeric difference was `1.60e-12`; all row counts and retained prefix levels matched exactly. - 85 synthetic/metamorphic runs passed directly, with zero order, duplicate, shuffle, or XOR-translation failures. - Upstream Haskell exits on 21 empty/fully-filtered sparse variants; Rust intentionally returns a safe empty result. - Five variants of one perfectly uniform synthetic set differ only in spectrum row count: Haskell retains one row because of a `3.55e-15` rounding wobble, while Rust suppresses that sub-epsilon artifact. Structure and dimension values still agree. PR CI now retains five small synthetic inputs and their independent Haskell JSON outputs, covering clustered, random, mixed, ancestor-pruning, and balanced-branch behavior. The integration test adds about 0.04 seconds and does not install or execute Haskell. Real/UOregon address data remains local and uncommitted; `docs/maad-conformance.md` documents repeatable local validation. ## Verification - `bun run format` - `bun run lint` - `bun run typecheck` - `bun run test:db` (97 Rust unit tests and 5 integration tests) - PR golden conformance test: five fixtures in 0.04 seconds - Local live-oracle smoke: all five committed inputs pass Rust vs Haskell in about 0.2 seconds with prebuilt binaries - Temporary full validation: 13 real cases and 111 synthetic/metamorphic runs - Manual end-to-end pipeline run not performed; focused pipeline identity coverage exercises the persisted contract. Implemented with GPT-5.6 Sol in T3 Code through the Codex harness, with worker subagents for the MAAD core, pipeline identity, conformance harness, dataset extraction, numerical-stability regression, CI fixtures, local validator, and adversarial review.