feat(netflow-db): sync in-process MAAD estimator#93

Problem

The experimental Rust MAAD estimator still used the earlier global atomic/spillover prefix range and reported the old subprocess contract. Current upstream MAAD filters each candidate prefix level through Patricia-tree ancestry and uses different variance aggregation, so our persisted results could diverge.

Solution

Product impact

• Match upstream's fixed /8 through /24 candidate range, per-prefix atomic/nearly-full filtering, invalid-ancestor pruning, and dropping of empty levels.
• Persist the exact retained prefix lengths and bump the MAAD result/identity contract to version 2 so incompatible databases cannot be mixed.
• Add a typed MaadConfig for q bounds/step, prefix bounds, and fullness threshold.
• Keep the pipeline default at q-step 1/8 (33 structure rows), intentionally avoiding upstream's newer 1/16 default and its doubled storage cost.

Fixes and internal changes

• Correct standard-deviation aggregation to upstream's sqrt(sum(d2)) / n.
• Handle empty, singleton, duplicate, sparse, and fully filtered address sets safely.
• Reject invalid/unbounded q grids, require generated q=0 and q=2 rows, and cap configurable grids at 1,025 values.
• Ignore sub-epsilon floating-point noise when selecting decreasing spectrum segments, so a mathematically linear structure curve does not produce spurious spectrum rows.
• Record the in-process backend and complete default MAAD configuration in pipeline product identity.
• Add focused reference, numerical-oracle, pruning, configuration, schema, identity, and spectrum-stability tests.
• Run five pinned Haskell-derived golden fixtures through the public Rust CLI in normal PR CI.
• Provide a standard-library local validator for arbitrary external address files without committing private data or requiring Haskell in CI.

Upstream reference: chris-misa/maad@3ae7536. The q-step is the intentional configuration exception.

Reviewer fast path

1. Run cargo test -p atlantis-netflow-db --test maad_conformance --locked; expect five golden cases to pass in roughly 0.04 seconds of test runtime.
2. Put at least two IPv4 addresses, one per line, in a temporary file.
3. Run ./scripts/netflow-db.sh maad <file>.
4. Expect schema version 2, explicit metadata.prefixLengths, 33 structure rows for non-empty valid input, and finite sparse-input output.
5. Run a fresh pipeline with run_maad: true; expect pipeline_product.config_json.maad to identify backend in-process, contract version 2, and q-step 0.125.

Important edge cases are invalid/full ancestor pruning, empty valid-level sets, duplicate addresses, non-aligned q grids, linear structure curves, and product-identity incompatibility. Changing these semantics requires a fresh product database.

Cross-language conformance

The optimized upstream Haskell executable was built from 3ae7536 with only deltaQ changed from 1/16 to 1/8. Both executables received the same deduplicated IPv4 files through their public JSON interfaces. Row counts and retained prefix levels were compared exactly; numeric fields used absolute + relative tolerance 1e-10.

• 13/13 real-address cases passed: 10 UOregon-derived source/destination sets (16,526–220,718 unique addresses), upstream's CAIDA 100k fixture, and two deterministic sets derived from Cloudflare's published IPv4 ranges.
• The largest real-case numeric difference was 1.60e-12; all row counts and retained prefix levels matched exactly.
• 85 synthetic/metamorphic runs passed directly, with zero order, duplicate, shuffle, or XOR-translation failures.
• Upstream Haskell exits on 21 empty/fully-filtered sparse variants; Rust intentionally returns a safe empty result.
• Five variants of one perfectly uniform synthetic set differ only in spectrum row count: Haskell retains one row because of a 3.55e-15 rounding wobble, while Rust suppresses that sub-epsilon artifact. Structure and dimension values still agree.

PR CI now retains five small synthetic inputs and their independent Haskell JSON outputs, covering clustered, random, mixed, ancestor-pruning, and balanced-branch behavior. The integration test adds about 0.04 seconds and does not install or execute Haskell. Real/UOregon address data remains local and uncommitted; docs/maad-conformance.md documents repeatable local validation.

Verification

• bun run format
• bun run lint
• bun run typecheck
• bun run test:db (97 Rust unit tests and 5 integration tests)
• PR golden conformance test: five fixtures in 0.04 seconds
• Local live-oracle smoke: all five committed inputs pass Rust vs Haskell in about 0.2 seconds with prebuilt binaries
• Temporary full validation: 13 real cases and 111 synthetic/metamorphic runs
• Manual end-to-end pipeline run not performed; focused pipeline identity coverage exercises the persisted contract.

Implemented with GPT-5.6 Sol in T3 Code through the Codex harness, with worker subagents for the MAAD core, pipeline identity, conformance harness, dataset extraction, numerical-stability regression, CI fixtures, local validator, and adversarial review.