feat(netflow): singularity alert feed#96
Singularity (MAAD) scores how anomalous each IP address is within a trace's address population, but ATLANTIS had no way to watch a live capture tree and surface those anomalies continuously.
This adds a rolling alert feed: a new long-running netflow-db feed <dataset> subcommand polls a dataset's nfcapd tree, and for each completed five-minute window unions the distinct IPv4 addresses across members, scores them with a Rust port of Singularities.hs, and records addresses beyond calibrated alpha thresholds into alerts.sqlite beside the dataset's product database (rolling ~7-day retention). The dashboard reads that file per dataset on a new /alerts page.
What's in here
singularity.rs: Rust port of MAAD's per-address alpha scoring, plus anetflow-db singularityCLI for stdin/file scoring. Conformance-verified against the Haskell reference on 24 real uOregon windows (7.6M address comparisons, zero violations; max alpha deviation 7.7e-15).feed.rs: the feed loop — registry-driven config, bounded window discovery with a grace rule for incomplete windows, per-window transactional writes, retention pruning,--backfill,--once.- Default thresholds
high = 2.0,low = 0.3, calibrated on 200 windows spanning 13 months (methodology + sensitivity indocs/agent/singularity-calibration.md). A typical window records ~20 alerts (p10–p90: 14–25). Both tails alert: high alpha = isolated/sparse address-space region, low alpha = deep inside a dense cluster. - Web:
/api/alerts+ a dataset-scoped/datasets/<id>/alertspage (Dashboard | Alerts tabs) — an address-centric feed: one row per anomalous address over a selectable horizon (1h/6h/24h/7d), showing its peak alpha in that horizon, last-seen, times-flagged persistence, and a "new" badge for fresh entrants; sorted by severity (distance past threshold) with a recency toggle, tail filter, and 30 s auto-refresh while visible. Windows are aggregation evidence, not page structure. The alerts DB is opened read-only; presence ofalerts.sqliteis the feature flag. D1 deployments reportfeed.present = false— this is local-driver-only for now.
Decisions worth reviewing
- The alert DB is an ephemeral rolling buffer owned by the feed, deliberately not a pipeline product database — no dataset identity or coverage semantics, safe to delete anytime.
- Feed depends on the dataset's registry config only, not on a pipeline run having produced
netflow.sqlite. - Backfill deeper than retention is clamped (processing windows that would be pruned in the same pass is wasted work).
- A window whose files all fail to read is skipped and retried next pass, never marked processed; a regression test covers the previous stall-on-gap behavior.
How to exercise
- Build the pinned nfdump fork (
./vendor/scripts/compile-nfdump.sh), then runnetflow-db feed <dataset-id> --nfdump target/nfdump/libexec/nfdump --backfill 2h --onceagainst a dataset fromdatasets.json. Expect onewindowsrow per completed five-minute window indata/<dataset>/alerts.sqlite, ~14–25 alerts each on uOregon-like traffic. - Start the web app (
ATLANTIS_DB_DRIVER=sqlite bun run dev:web), open/datasets/<id>/alerts: feed sections newest-first, tail badges with tooltips, "Feed idle"/"Live" status per window age; datasets without a feed show the start-command card. - Cross-check scoring:
netflow-db singularity <file-of-ipv4s>againstvendor/maadSingularities -1 <file>.
Verification
bun run format/lint/typecheckclean;bun run test:db(131 lib tests + integration, incl. 8 feed + 7 singularity) andbun run test:web(87 tests, 15 new) pass.- Conformance + calibration report:
docs/agent/singularity-calibration.md. - Manual end-to-end run over six real uOregon windows (both members, ~345k addresses per window, 5–19 s per window) verified alert rows, retention pruning, and the dashboard rendering them.
Built by Claude (Fable 5) orchestrating gpt-5.6-sol (xhigh) implementation subagents via Claude Code.
Checks
Added 14 commits
- feat(netflow-db): scaffold singularity and feed subcommandsc1fc839
- feat(web): scope alerts to dataset routes with dashboard summary0a85f02
- feat(web): explain alert tails with badge tooltips10cc36f
- feat(web): drop alerts summary card from dashboard16e6141
- feat(web): restructure alerts into address-centric feed22ec066
- feat(web): tabular alert rows and visible thresholds45aea59
- feat(web): show peak and latest alpha per alert address6a1e1a5
- feat(netflow-db): implement singularity scoring and alert feed8683354
- fix(web): judge alert novelty against retained history874dd72
- feat(web): first-seen column, plain thresholds, always-on refresh9a04c6f
- feat(netflow-db): accept minute backfill durationsac32223
- feat(web): alert feed page and APIaf37f52
- feat(web): default alerts to high tail and drop row pillscba5c9c
- feat(netflow-db): calibrate singularity alert thresholdsefff45b