# feat: classify endpoint locality and filter by traffic direction · PR \#108 · flambo/atlantis

[View on GitCafe](https://git.cafe/flambo/atlantis/pull/108)

Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis)

Visibility: public

State: merged

Source repository: flambo/atlantis

Source branch: maad/02-locality-direction

Target branch: maad/01-10m-buckets

Head commit: fd5f115fdc833e1706341d4846d1d9d7616dc93e

[Discussion and reviews](https://git.cafe/flambo/atlantis/pull/108)

[Commits](https://git.cafe/flambo/atlantis/pull/108/commits?format=markdown)

[Files changed](https://git.cafe/flambo/atlantis/pull/108/diffs?format=markdown)

## Description

---

> [!NOTE]
> 🤖 **Claude Opus 5.5 on behalf of Oliver**

## ELI5

Instead of guessing "inside vs outside" from a network-specific anonymization flag, each dataset now lists rules for what counts as internal, and the dashboard can filter traffic by direction: incoming, outgoing, inside-to-inside, or neither.

## Why

Endpoint "visibility" came from the anonymizer's ToS bits, which only works on our network and misclassifies known literal internal hosts. Direction (ingress/egress/lateral) is what analyses actually need.

## Implementation

- Per-dataset `locality` rules: `tos_anonymized` (the old behavior), `prefixes` (CIDRs), `addresses` (inline list or file). An endpoint is internal if any rule matches. Validation is strict; IPv4-mapped IPv6 is canonicalized.
- Direction from (src, dst): egress, ingress, lateral, transit (external→external, kept so bad records stay visible).
- Storage keeps its shape: `src_locality`/`dst_locality` (internal/external/all) replace `src_visibility`/`dst_visibility`.
- Product identity records rule counts and SHA-256 digests (never the addresses), so editing rules yields a new product.
- Address files are read only for the dataset being run, so one unreadable file doesn't break other datasets.
- PR #105's `daily_active_sources` selection migrated to locality.
- Web: shared direction filter (all/ingress/egress/lateral/transit) on every chart; hidden for datasets without rules (`datasets.has_locality`).

## Review path

1. **Registry mode.** Add to a dataset in `datasets.json`:
   ```json
   "locality": [
     { "type": "tos_anonymized" },
     { "type": "addresses", "path": "data/private/internal.txt" }
   ]
   ```
   Run `pipeline --dataset <id>` for one day. Expected: rows use `src_locality`/`dst_locality`; the four directions sum exactly to `all`; `datasets.has_locality = 1`. Relative paths resolve from the repository root.
2. **Config mode.** Put `"locality": [...]` at the top level of a pipeline config (a per-dataset `locality` inside a config is rejected), with a relative address file next to the config. Run `pipeline --config <path>` from another directory. Expected: the file resolves from the config's directory, and every dataset in the product gets `has_locality = 1`.
3. **Dashboard.** The direction filter changes every chart; a dataset without rules shows no filter.
4. **Obsolete local DBs.** Leave a pre-locality `netflow.sqlite` (with `src_visibility` columns and no `has_locality`) under `data/`, including one that reuses a current dataset ID. Expected: the dashboard lists only current products; the old DB is skipped and doesn't shadow the current one or break the listing.

## Edge cases and decisions

- Local discovery skips any SQLite file that lacks `bucket_coverage` or `datasets.has_locality`, or that has a stats table without both locality columns. It doesn't error on them.
- A locality filter (`--src-locality`/`--dst-locality`, `daily_active_sources`) without rules is rejected, because every endpoint would be external.
- Private address lists belong under the gitignored `data/`.
- ⚠️ Drizzle migration `0003` keeps only `all`/`all` rows: literal/anonymized can't be mapped to internal/external. Migration `0004` adds `datasets.has_locality`.
- **Production D1 validation and cutover are pending.** Production will be cut over to a fresh database in the infra stack rather than migrated in place. This PR does not validate against production D1.

## Verification

- Automated: `bun run format`, `bun run lint`, `bun run typecheck`, `bun run test:web` (includes the pre-locality/duplicate-ID discovery regression and the migrations for all five stats tables), `bun run test:db` (rule evaluation, direction mapping, fingerprint changes, registry errors, rollup parity, config-mode `has_locality` and config-relative address files), `bun run test:e2e` (passes; the fixture now uses the shared local schema and checks the served dataset metadata), `cargo fmt --check`, `cargo clippy -D warnings`.
- Manual: run a real one-day dataset through the dashboard direction filter; production D1 cutover (pending, see above).

---
Made by Claude Opus 5.5 (with Opus 5.5 subagents) via Claude Code.