feat(netflow-db): rewrite pipeline in Rust#89

Follow-up performance work is stacked in #90.

Problem

The NetFlow database pipeline was split across a large Python surface and companion native reducers. That made the processing contract harder to reason about, test, and optimize as one system.

Solution

• replace the Python pipeline with a typed Rust workspace covering configuration, ingestion, canonical buckets, rollups, SQLite persistence, exports, preparation, and verification
• preserve the existing web-facing database contract and add a comparison CLI for candidate/reference validation
• reduce directly from borrowed nfdump rows instead of allocating a generic observation and channel message per flow
• use bounded concurrency for capture decoding and first-run provenance hashing
• avoid copying singleton address sets and canonical row data during publication
• replace ordered-tree-heavy MAAD construction with sorted linear prefix scans
• update CI, local scripts, setup documentation, and package commands for the Rust implementation
• remove the superseded Python modules and tests

Measured performance

These figures compare PR #89's optimized Rust pipeline with its initial Rust implementation. They are not a Python-versus-Rust benchmark.

The full-day A/B processed UOregon 2025-11-03: 576 nfcapd files, 61.637 GiB, 941,171,446 combined flows, three logical sources, and 864 five-minute buckets. Both sides used fresh SQLite databases and the same process-tree profiler.

The full-day result saves 43m33s per processed day. A simple 365-day extrapolation reduces the workload from roughly 15.8 to 4.8 machine-days per year.

Independent experiments

• dedicated reducer on a real 64 MiB capture: 9.48s → 3.84s median (2.47×), with roughly 39× fewer voluntary context switches
• decode concurrency on 48 real captures: 46.38s → 31.60s moving from 4 to 12 timestamps (-31.9%); 16 timestamps regressed
• borrowed singleton and row publication: 30.98s → 25.96s (-16.2%)
• MAAD over 71,698 real unique IPv4 addresses: 390.8ms → 97.7ms mean (4.00×), with byte-identical JSON
• fresh provenance hashing over 61.64 GiB: 669.59s serial → 57.83s with 24 workers (11.58×), with all 1,152 fingerprints identical

The optimized database passed the full verifier. Comparison found zero candidate-only, reference-only, or mismatched rows across 119,130 statistics rows and 1,152 provenance rows, with a maximum numeric and MAAD delta of exactly 0.0.

This is a reproducible warm-cache CPU-throughput result, not a cold-storage claim. One full profiled A/B was run because the initial side takes more than an hour; an earlier unprofiled baseline independently measured about 62.4 minutes. Publication-grade confidence would require multiple alternating full-day pairs on an otherwise idle host.

Impact

The pipeline now has one canonical Rust domain model and one implementation path from source discovery through persisted statistics. This reduces cross-language coordination and gives subsequent performance work a clear, profiled seam.

Validation

• bun format
• bun lint
• bun typecheck
• bun run test
  • 62 web tests
  • 82 Rust unit tests
  • 4 compare-CLI tests
• full-day database verifier: data, MAAD, provenance, rollup parity, and no raw IPs
• exact six-table and provenance comparison

Built, profiled, and verified with Codex (GPT-5 family) using the Codex desktop/CLI agent harness.