# feat\(netflow\): add per-dataset dashboards and pipeline config \(\#40\) · flambo/atlantis

[View on GitCafe](https://git.cafe/flambo/atlantis/commit/d02684e012b31ff0b00060268ab0153f56ae6c60)

Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis)

Visibility: public

Requested revision: d02684e012b31ff0b00060268ab0153f56ae6c60

Requested commit: d02684e012b31ff0b00060268ab0153f56ae6c60

Commit: d02684e012b31ff0b00060268ab0153f56ae6c60

Tree: 3bb3269525930688f56a09aa092aeca989bd89bf

Author: oliver

Committer: GitHub

## Message

```
feat(netflow): add per-dataset dashboards and pipeline config (#40)

<!-- greptile_comment -->

<h3>Greptile Summary</h3>

This PR introduces a multi-dataset architecture for both the Python processing pipeline and the SvelteKit web app. The pipeline gains a `--dataset` flag and a `datasets.json` registry, while the web app adds a dataset picker home page and per-dataset dashboard routes at `/datasets/[dataset]`. The Python backend uses a careful deferred-import pattern in `pipeline.py` to ensure the dataset selection env var is set before `common` is initialized, which is correct and well-structured.

Key issues found:

- **`datasets.json` committed but also added to `.gitignore`**: Once a file is tracked by git, `.gitignore` entries have no effect. The committed file also contains real machine-specific absolute paths (`/research/obo/netflow_datasets/*`), contradicting the docs. Running `git rm --cached datasets.json` is needed to actually untrack it.
- **Hardcoded dataset-specific start date**: `handleResetView` in `DatasetDashboardPage.svelte` resets to `'2025-02-11'`, a UOregon-specific date that will be incorrect for other datasets like UGR16.
- **`datasetDbCache` has no invalidation path**: The server-side SQLite connection cache in `server/datasets.ts` is never cleared. If the pipeline recreates a database (e.g. `FIRST_RUN=True`), the web app will serve stale data until restarted.
- **Duplicate navigation links**: The "Datasets" and "Home" nav entries both point to `/`.

<h3>Confidence Score: 3/5</h3>

- Safe to merge for functionality, but the committed datasets.json with real paths is a git hygiene issue that should be resolved first.
- The core multi-dataset logic in both the Python pipeline and the web app is well-implemented and the deferred-import pattern for dataset selection is correct. However, the accidental commitment of datasets.json with real machine-specific paths (despite documenting that it shouldn't be committed) creates a persistent git tracking problem, and the hardcoded UOregon-specific reset date in DatasetDashboardPage will cause subtle UX issues for other datasets.
- datasets.json (git tracking issue), netflow-webapp/src/lib/components/netflow/DatasetDashboardPage.svelte (hardcoded start date), netflow-webapp/src/lib/server/datasets.ts (DB cache invalidation)

<h3>Important Files Changed</h3>

| Filename | Overview |
|----------|----------|
| netflow-db/common.py | Major addition: multi-dataset registry loading, validation, and lookup helpers. Module-level initialization order is correct (DATASET_REGISTRY → DEFAULT_DATASET → ACTIVE_DATASET). Deferred imports in pipeline.py correctly rely on NETFLOW_DATASET being set before common is first imported. |
| netflow-db/pipeline.py | Adds --dataset flag; sets NETFLOW_DATASET env var before deferred import of common, which ensures the correct dataset config is used. Refactored discovery/processing helpers into inner functions. Clean and correct. |
| netflow-webapp/src/lib/server/datasets.ts | New server-side dataset registry module; includes mtime-based caching for the registry file. DB connection cache is indefinite with no invalidation path if the database file is recreated by the pipeline. |
| datasets.json | Committed with real machine-specific absolute paths and simultaneously added to .gitignore — adding to .gitignore does not untrack an already-committed file, so changes will still be tracked by git. |
| netflow-webapp/src/lib/components/netflow/DatasetDashboardPage.svelte | New per-dataset dashboard page wrapping all existing charts; handleResetView hardcodes '2025-02-11' as a default start date that is specific to the UOregon dataset and will be wrong for other datasets. |
| netflow-webapp/src/routes/+layout.svelte | Minor update; "Datasets" nav link duplicates the "Home" link (both point to /). |

</details>

<sub>Last reviewed commit: da9782a</sub>

> Greptile also left **4 inline comments** on this PR.

<!-- /greptile_comment -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

* **New Features**
  * Multi-dataset support with a dataset registry, per-dataset databases, dataset index, and per-dataset dashboard pages with interactive, reorderable charts
  * Dataset selection propagated across file views, charts, and APIs; CLI/web option to target a specific dataset
  * New dataset preparation utility to normalize/import external NetFlow archives

* **Documentation**
  * Guides for datasets.json configuration and multi-dataset pipeline usage with examples

* **Tests**
  * Unit tests for the datasets API

* **Chores**
  * Updated env template, .gitignore and package deps; added test tooling
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
```

## Parents

- [b24fa55684ffd496b0f5d2e6849ffb21f97c85b1](https://git.cafe/flambo/atlantis/commit/b24fa55684ffd496b0f5d2e6849ffb21f97c85b1?format=markdown)

- [27ba3cc01f12ddb2e028fa9a75f547a190e4c010](https://git.cafe/flambo/atlantis/commit/27ba3cc01f12ddb2e028fa9a75f547a190e4c010?format=markdown)

[Source at this commit](https://git.cafe/flambo/atlantis/tree/d02684e012b31ff0b00060268ab0153f56ae6c60?format=markdown)