perf(netflow): precompute dashboard rollups (#59)

## Summary

Precompute coarse NetFlow rollups for dashboard cache misses and route coarse `/api/netflow/stats` requests to those rollups.

- Adds `netflow_stats_aggregate_v2` for `30m`, `1h`, and `1d` NetFlow metrics.
- Populates rollups from both streaming and batch pipeline paths using existing aggregate bucket state.
- Uses the rollup table for non-5m dashboard stats requests, leaving `5min` drilldown/file paths on `netflow_stats_v2`.
- Adds time-first range indexes for NetFlow, IP, protocol, structure, spectrum, and dimension tables.
- Updates schema verification and route/pipeline tests for the new table.

## Why

Dashboard cache misses currently ask SQLite to scan 5-minute rows and compute local-time buckets for coarse views. The pipeline already materializes local-time bucket boundaries for IP/protocol/MAAD aggregates, so NetFlow can reuse that same aggregate calendar and avoid expensive request-time grouping.

## Migration Steps

Existing v2 databases need one data migration plus idempotent index creation.

1. Create `netflow_stats_aggregate_v2` with columns matching `netflow_stats_v2` plus `granularity`, and primary key `(source_id, granularity, bucket_start, ip_version)`.
2. Backfill from existing tables by joining the aggregate calendar from `ip_stats_v2` to raw 5m NetFlow rows:

```sql
INSERT OR REPLACE INTO netflow_stats_aggregate_v2 (
  source_id, granularity, bucket_start, bucket_end, ip_version,
  flows, flows_tcp, flows_udp, flows_icmp, flows_other,
  packets, packets_tcp, packets_udp, packets_icmp, packets_other,
  bytes, bytes_tcp, bytes_udp, bytes_icmp, bytes_other
)
SELECT
  calendar.source_id,
  calendar.granularity,
  calendar.bucket_start,
  calendar.bucket_end,
  ns.ip_version,
  SUM(ns.flows), SUM(ns.flows_tcp), SUM(ns.flows_udp), SUM(ns.flows_icmp), SUM(ns.flows_other),
  SUM(ns.packets), SUM(ns.packets_tcp), SUM(ns.packets_udp), SUM(ns.packets_icmp), SUM(ns.packets_other),
  SUM(ns.bytes), SUM(ns.bytes_tcp), SUM(ns.bytes_udp), SUM(ns.bytes_icmp), SUM(ns.bytes_other)
FROM (
  SELECT source_id, granularity, bucket_start, bucket_end
  FROM ip_stats_v2
  WHERE granularity IN ('30m', '1h', '1d')
) AS calendar
JOIN netflow_stats_v2 AS ns
  ON ns.source_id = calendar.source_id
 AND ns.bucket_start >= calendar.bucket_start
 AND ns.bucket_start < calendar.bucket_end
GROUP BY calendar.source_id, calendar.granularity, calendar.bucket_start, calendar.bucket_end, ns.ip_version;
```

3. Add these indexes if missing:

```sql
CREATE INDEX IF NOT EXISTS idx_netflow_stats_v2_bucket_source
  ON netflow_stats_v2(bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_netflow_stats_aggregate_v2_granularity_bucket_source
  ON netflow_stats_aggregate_v2(granularity, bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_ip_stats_v2_granularity_bucket_source
  ON ip_stats_v2(granularity, bucket_start, source_id);
CREATE INDEX IF NOT EXISTS idx_protocol_stats_v2_granularity_bucket_source
  ON protocol_stats_v2(granularity, bucket_start, source_id);
CREATE INDEX IF NOT EXISTS idx_structure_stats_v2_granularity_bucket_source
  ON structure_stats_v2(granularity, bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_spectrum_stats_v2_granularity_bucket_source
  ON spectrum_stats_v2(granularity, bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_dimension_stats_v2_granularity_bucket_source
  ON dimension_stats_v2(granularity, bucket_start, source_id, ip_version);
```

4. Run `ANALYZE` after migration.

Local migration checks on copied DBs:

- UGR16: 404 MB DB, migration `~0.86s`, `8,008` rollup rows, size delta `~5.7 MB`.
- UOregon v2: 2.6 GB DB, migration `~6.3s`, `71,820` rollup rows.

## Validation

- `bun format`
- `bun lint`
- `bun typecheck`
- `bun run test` in `apps/web`: 33 passed
- `./scripts/run-with-nix-if-available.sh uv run --extra test pytest tests/python`: 81 passed, 1 skipped
- Manual UOregon v2 copy migrated and smoke-tested through dev server on `/datasets/uoregon`
187c4f6d8aoliver committed on 5/11/2026, 7:26:18 AM· committed by GitHubparents1df7d6909fc34b
11 files changedLine totals unavailable