# perf\(netflow\): precompute dashboard rollups \(\#59\) · flambo/atlantis

[View on GitCafe](https://git.cafe/flambo/atlantis/commit/187c4f6d8affb40dc60a7f6d20b668e0e65dc9af)

Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis)

Visibility: public

Requested revision: 187c4f6d8affb40dc60a7f6d20b668e0e65dc9af

Requested commit: 187c4f6d8affb40dc60a7f6d20b668e0e65dc9af

Commit: 187c4f6d8affb40dc60a7f6d20b668e0e65dc9af

Tree: 056f2b88f83c63ed0629adb377fcbd4313ca02f5

Author: oliver

Committer: GitHub

## Message

````
perf(netflow): precompute dashboard rollups (#59)

## Summary

Precompute coarse NetFlow rollups for dashboard cache misses and route coarse `/api/netflow/stats` requests to those rollups.

- Adds `netflow_stats_aggregate_v2` for `30m`, `1h`, and `1d` NetFlow metrics.
- Populates rollups from both streaming and batch pipeline paths using existing aggregate bucket state.
- Uses the rollup table for non-5m dashboard stats requests, leaving `5min` drilldown/file paths on `netflow_stats_v2`.
- Adds time-first range indexes for NetFlow, IP, protocol, structure, spectrum, and dimension tables.
- Updates schema verification and route/pipeline tests for the new table.

## Why

Dashboard cache misses currently ask SQLite to scan 5-minute rows and compute local-time buckets for coarse views. The pipeline already materializes local-time bucket boundaries for IP/protocol/MAAD aggregates, so NetFlow can reuse that same aggregate calendar and avoid expensive request-time grouping.

## Migration Steps

Existing v2 databases need one data migration plus idempotent index creation.

1. Create `netflow_stats_aggregate_v2` with columns matching `netflow_stats_v2` plus `granularity`, and primary key `(source_id, granularity, bucket_start, ip_version)`.
2. Backfill from existing tables by joining the aggregate calendar from `ip_stats_v2` to raw 5m NetFlow rows:

```sql
INSERT OR REPLACE INTO netflow_stats_aggregate_v2 (
  source_id, granularity, bucket_start, bucket_end, ip_version,
  flows, flows_tcp, flows_udp, flows_icmp, flows_other,
  packets, packets_tcp, packets_udp, packets_icmp, packets_other,
  bytes, bytes_tcp, bytes_udp, bytes_icmp, bytes_other
)
SELECT
  calendar.source_id,
  calendar.granularity,
  calendar.bucket_start,
  calendar.bucket_end,
  ns.ip_version,
  SUM(ns.flows), SUM(ns.flows_tcp), SUM(ns.flows_udp), SUM(ns.flows_icmp), SUM(ns.flows_other),
  SUM(ns.packets), SUM(ns.packets_tcp), SUM(ns.packets_udp), SUM(ns.packets_icmp), SUM(ns.packets_other),
  SUM(ns.bytes), SUM(ns.bytes_tcp), SUM(ns.bytes_udp), SUM(ns.bytes_icmp), SUM(ns.bytes_other)
FROM (
  SELECT source_id, granularity, bucket_start, bucket_end
  FROM ip_stats_v2
  WHERE granularity IN ('30m', '1h', '1d')
) AS calendar
JOIN netflow_stats_v2 AS ns
  ON ns.source_id = calendar.source_id
 AND ns.bucket_start >= calendar.bucket_start
 AND ns.bucket_start < calendar.bucket_end
GROUP BY calendar.source_id, calendar.granularity, calendar.bucket_start, calendar.bucket_end, ns.ip_version;
```

3. Add these indexes if missing:

```sql
CREATE INDEX IF NOT EXISTS idx_netflow_stats_v2_bucket_source
  ON netflow_stats_v2(bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_netflow_stats_aggregate_v2_granularity_bucket_source
  ON netflow_stats_aggregate_v2(granularity, bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_ip_stats_v2_granularity_bucket_source
  ON ip_stats_v2(granularity, bucket_start, source_id);
CREATE INDEX IF NOT EXISTS idx_protocol_stats_v2_granularity_bucket_source
  ON protocol_stats_v2(granularity, bucket_start, source_id);
CREATE INDEX IF NOT EXISTS idx_structure_stats_v2_granularity_bucket_source
  ON structure_stats_v2(granularity, bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_spectrum_stats_v2_granularity_bucket_source
  ON spectrum_stats_v2(granularity, bucket_start, source_id, ip_version);
CREATE INDEX IF NOT EXISTS idx_dimension_stats_v2_granularity_bucket_source
  ON dimension_stats_v2(granularity, bucket_start, source_id, ip_version);
```

4. Run `ANALYZE` after migration.

Local migration checks on copied DBs:

- UGR16: 404 MB DB, migration `~0.86s`, `8,008` rollup rows, size delta `~5.7 MB`.
- UOregon v2: 2.6 GB DB, migration `~6.3s`, `71,820` rollup rows.

## Validation

- `bun format`
- `bun lint`
- `bun typecheck`
- `bun run test` in `apps/web`: 33 passed
- `./scripts/run-with-nix-if-available.sh uv run --extra test pytest tests/python`: 81 passed, 1 skipped
- Manual UOregon v2 copy migrated and smoke-tested through dev server on `/datasets/uoregon`
````

## Parents

- [1df7d69f84ddfb0ce013a2751d381c2843a2c3e9](https://git.cafe/flambo/atlantis/commit/1df7d69f84ddfb0ce013a2751d381c2843a2c3e9?format=markdown)

- [09fc34bb21bf3c0f65d3a7a5000a58e061dbb6f7](https://git.cafe/flambo/atlantis/commit/09fc34bb21bf3c0f65d3a7a5000a58e061dbb6f7?format=markdown)

[Source at this commit](https://git.cafe/flambo/atlantis/tree/187c4f6d8affb40dc60a7f6d20b668e0e65dc9af?format=markdown)