## Summary
Precompute coarse NetFlow rollups for dashboard cache misses and route coarse `/api/netflow/stats` requests to those rollups.
- Adds `netflow_stats_aggregate_v2` for `30m`, `1h`, and `1d` NetFlow metrics.
- Populates rollups from both streaming and batch pipeline paths using existing aggregate bucket state.
- Uses the rollup table for non-5m dashboard stats requests, leaving `5min` drilldown/file paths on `netflow_stats_v2`.
- Adds time-first range indexes for NetFlow, IP, protocol, structure, spectrum, and dimension tables.
- Updates schema verification and route/pipeline tests for the new table.
## Why
Dashboard cache misses currently ask SQLite to scan 5-minute rows and compute local-time buckets for coarse views. The pipeline already materializes local-time bucket boundaries for IP/protocol/MAAD aggregates, so NetFlow can reuse that same aggregate calendar and avoid expensive request-time grouping.
## Migration Steps
Existing v2 databases need one data migration plus idempotent index creation.
1. Create `netflow_stats_aggregate_v2` with columns matching `netflow_stats_v2` plus `granularity`, and primary key `(source_id, granularity, bucket_start, ip_version)`.
2. Backfill from existing tables by joining the aggregate calendar from `ip_stats_v2` to raw 5m NetFlow rows:
```sql
INSERT OR REPLACE INTO netflow_stats_aggregate_v2 (
source_id, granularity, bucket_start, bucket_end, ip_version,
flows, flows_tcp, flows_udp, flows_icmp, flows_other,
packets, packets_tcp, packets_udp, packets_icmp, packets_other,
bytes, bytes_tcp, bytes_udp, bytes_icmp, bytes_other
)
SELECT
calendar.source_id,
calendar.granularity,
calendar.bucket_start,
calendar.bucket_end,
ns.ip_version,
SUM(ns.flows), SUM(ns.flows_tcp), SUM(ns.flows_udp), SUM(ns.flows_icmp), SUM(ns.flows_other),
SUM(ns.packets), SUM(ns.packets_tcp), SUM(ns.packets_udp), SUM(ns.packets_icmp), SUM(ns.packets_other),
SUM(ns.bytes), SUM(ns.bytes_tcp), SUM(ns.bytes_udp), SUM(ns.bytes_icmp), SUM(ns.bytes_other)
FROM (
SELECT source_id, granularity, bucket_start, bucket_end
FROM ip_stats_v2
WHERE granularity IN ('30m', '1h', '1d')
) AS calendar
JOIN netflow_stats_v2 AS ns
ON ns.source_id = calendar.source_id
AND ns.bucket_start >= calendar.bucket_start
AND ns.bucket_start < calendar.bucket_end
GROUP BY calendar.source_id, calendar.granularity, calendar.bucket_start, calendar.bucket_end, ns.ip_version;
```
3. Add these indexes if missing:
```sql
CREATE INDEX IF NOT EXISTS idx_netflow_stats_v2_bucket_source
ON netflow_stats_v2(bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_netflow_stats_aggregate_v2_granularity_bucket_source
ON netflow_stats_aggregate_v2(granularity, bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_ip_stats_v2_granularity_bucket_source
ON ip_stats_v2(granularity, bucket_start, source_id);
CREATE INDEX IF NOT EXISTS idx_protocol_stats_v2_granularity_bucket_source
ON protocol_stats_v2(granularity, bucket_start, source_id);
CREATE INDEX IF NOT EXISTS idx_structure_stats_v2_granularity_bucket_source
ON structure_stats_v2(granularity, bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_spectrum_stats_v2_granularity_bucket_source
ON spectrum_stats_v2(granularity, bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_dimension_stats_v2_granularity_bucket_source
ON dimension_stats_v2(granularity, bucket_start, source_id, ip_version);
```
4. Run `ANALYZE` after migration.
Local migration checks on copied DBs:
- UGR16: 404 MB DB, migration `~0.86s`, `8,008` rollup rows, size delta `~5.7 MB`.
- UOregon v2: 2.6 GB DB, migration `~6.3s`, `71,820` rollup rows.
## Validation
- `bun format`
- `bun lint`
- `bun typecheck`
- `bun run test` in `apps/web`: 33 passed
- `./scripts/run-with-nix-if-available.sh uv run --extra test pytest tests/python`: 81 passed, 1 skipped
- Manual UOregon v2 copy migrated and smoke-tested through dev server on `/datasets/uoregon`11 files changedLine totals unavailable