# feat\(discovery\): add filesystem discovery window \(\#36\) · flambo/atlantis

[View on GitCafe](https://git.cafe/flambo/atlantis/commit/4444fbdcac19d1bd836209434f1004d7c9f445eb)

Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis)

Visibility: public

Requested revision: 4444fbdcac19d1bd836209434f1004d7c9f445eb

Requested commit: 4444fbdcac19d1bd836209434f1004d7c9f445eb

Commit: 4444fbdcac19d1bd836209434f1004d7c9f445eb

Tree: 51656cf27f013d35b60de3bf28f7936fce96849e

Author: oliver

Committer: GitHub

## Message

````
feat(discovery): add filesystem discovery window (#36)

<!-- greptile_comment -->

<h3>Greptile Summary</h3>

This PR introduces a configurable `--discovery-window-days` flag that decouples filesystem discovery from the reprocessing window, replacing the previous full recursive directory tree-walk with direct date-directory enumeration. The change reduces unnecessary I/O on normal runs by scanning only a bounded window of day directories (`router/YYYY/MM/DD/`) while keeping unlimited historical mode available via `--discovery-window-days 0`.

Key changes:
- `iter_scan_days(discovery_window_days)` and `get_discovery_start_dt(discovery_window_days)` added to `discovery.py` to compute the bounded set of day directories to inspect.
- `scan_filesystem()` refactored from a recursive `iterdir()` walk to targeted `day_dir.glob('nfcapd.*')` calls per day in the window.
- `--discovery-window-days N` CLI flag added to `pipeline.py`; defaults to `reprocess_window_days` when not supplied.
- Both windows use consistent calendar-day semantics (local midnight) and clamp to `DATA_START_DATE`.

The core logic is sound and correct. One minor issue: the `sync_processed_files_table` docstring still claims to scan "all nfcapd files," which no longer reflects the bounded discovery window behavior.

<h3>Confidence Score: 5/5</h3>

- This PR is safe to merge. The core logic is correct and well-tested, with proper parameter threading through the call stack.
- The refactoring from recursive tree-walk to bounded day-directory enumeration is sound. The two-window design (discovery and reprocessing) is correctly implemented with consistent semantics and proper clipping to `DATA_START_DATE`. All explicit parameter passing in `main()` ensures correct behavior. Only one minor issue remains: a docstring update to reflect bounded discovery behavior. This does not affect functionality.
- No files require special attention — the docstring issue in `netflow-db/discovery.py` is a straightforward one-line fix.

<h3>Flowchart</h3>

```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([pipeline.py main]) --> B{--discovery-window-days<br/>provided?}
    B -- No --> C[discovery_window_days<br/>= reprocess_window_days]
    B -- Yes --> D[discovery_window_days<br/>= arg value]
    C --> E[run_discovery]
    D --> E
    E --> F[sync_processed_files_table<br/>discovery_window_days, reprocess_window_days]
    F --> G[scan_filesystem<br/>discovery_window_days]
    G --> H[get_discovery_start_dt]
    H --> I{discovery_window_days == 0?}
    I -- Yes<br/>unlimited --> J[start = DATA_START_DATE midnight]
    I -- No<br/>bounded --> K[start = today_midnight<br/>- N days<br/>clamped to DATA_START_DATE]
    J --> L[iter_scan_days<br/>yield each day → today]
    K --> L
    L --> M[for each router × each day<br/>build router/YYYY/MM/DD path]
    M --> N{day dir exists?}
    N -- No --> O[skip]
    N -- Yes --> P[glob nfcapd.* <br/>yield file_path, router, timestamp]
    P --> Q[(processed_files DB<br/>INSERT OR IGNORE)]
    F --> R[gap detection<br/>reprocess_window_days only]
    R --> Q
```

<!-- greptile_failed_comments -->
<details><summary><h3>Comments Outside Diff (1)</h3></summary>

1. `netflow-db/discovery.py`, line 195-196 ([link](https://github.com/flamboh/netflow-analysis/blob/5e8fe91c3e716e83ef1561c651db4a7a23c8a1a5/netflow-db/discovery.py#L195-L196)) 

   The function-level docstring still describes the scan as applying to **all** nfcapd files, but the function now respects a bounded `discovery_window_days` parameter. Update to reflect the new behavior:

</details>

<!-- /greptile_failed_comments -->

<sub>Last reviewed commit: 5e8fe91</sub>

<!-- /greptile_comment -->
````

## Parents

- [8693f7eda7545d57604076e6686d1f0d0154ab36](https://git.cafe/flambo/atlantis/commit/8693f7eda7545d57604076e6686d1f0d0154ab36?format=markdown)

- [5e8fe91c3e716e83ef1561c651db4a7a23c8a1a5](https://git.cafe/flambo/atlantis/commit/5e8fe91c3e716e83ef1561c651db4a7a23c8a1a5?format=markdown)

[Source at this commit](https://git.cafe/flambo/atlantis/tree/4444fbdcac19d1bd836209434f1004d7c9f445eb?format=markdown)