perf(netflow-db): stream binary nfdump output#90

Stacked on #89. Review this PR against codex/rust-netflow-pipeline.

Problem

The Rust pipeline still paid substantial conversion overhead around nfdump output, and publish-time address unions used ordered sets even though address order is not part of the persisted contract. Profiling identified these as two separate bottlenecks.

Solution

• pin flamboh/nfdump at 16348e79 with an ATLANTIS-specific compact binary output contract
• decode and aggregate that stream directly in Rust with explicit framing, diagnostics, timeout, and overflow handling
• remove the superseded text reducer and normalization path
• represent unique addresses with an order-free AddressSet, sorting only inside MAAD where prefix analysis actually requires order
• add low-overhead phase instrumentation around daily preparation, decoding, aggregation, rollup publication, SQLite writes, and MAAD
• verify that address insertion order cannot change persisted products

Measured impact

Binary nfdump fork

The clean pre-fork reference spent 494.469s decoding one full day. The binary-output fork reduced the measured decode phase to 264.021s:

• 230.448s less decode time
• 46.6% faster decode phase
• estimated full-day runtime of 14m56s instead of 18m49s when holding the uncontended non-decode phases constant
• about 3m53s saved per day, reducing the annual workload estimate from roughly 4.8 to 3.8 machine-days

The binary candidate's directly observed wall time was 18m43s, but that run overlapped another CPU- and memory-heavy pipeline. Its prepare and publish phases were heavily inflated, so the phase-isolated estimate is the honest total-time comparison.

The candidate and pre-fork reference databases matched exactly across all five data tables for 1,152 processed rows. Their provenance fingerprints intentionally differ because the decoder implementation changed.

HashSet address accumulator

A later isolated Shiraz experiment targeted the now-dominant publish phase:

• publish: 498.958s → 456.807s (-42.150s, -8.45%)
• peak RSS: 5,078,828 KiB → 4,438,388 KiB (-640,440 KiB, -12.6%)
• complete observed day: 687.94s (11m27.94s)

An exact, zero-tolerance comparison found identical row counts and zero mismatches across all six persisted tables. These figures come from a separate experiment; wall and prepare time are cache-sensitive, so the two experiments' total-time improvements should not be mechanically added together.

Validation

• bun format
• bun lint
• bun typecheck
• bun run test
  • 62 web tests
  • 86 Rust unit tests
  • 4 compare-CLI tests
• full-day pre/post binary decoder comparison
• isolated full-day HashSet profile and exact database comparison on Shiraz

Built, profiled, and verified with Codex (GPT-5 family) using the Codex desktop/CLI agent harness, with Luna agents used as infrequent monitors for the persistent profiling runs.