<!-- greptile_comment -->

<h3>Greptile Summary</h3>

This PR introduces a configurable `--reprocess-window-days` CLI flag (default 30, `0` = unlimited) that gates which files and days the pipeline automatically revisits, replacing a previously hard-coded 30-day constant in `handle_stale_days`. The change centralises cutoff logic in a new `get_reprocess_cutoff_dt` helper in `discovery.py` and threads the parameter through all five processor modules and both pipeline phases.

**Key issues found:**
- **Real files blocked by reprocessing window** — Both `get_files_needing_processing` and `get_pending_files` apply the window cutoff to all rows, including real files (`file_exists = 1`). The design spec (`plans/reprocessing-window-plan.md`) explicitly requires that only synthetic gap placeholders (`file_exists = 0`) be gated by the window, while real files must always remain eligible. A late-arriving upload older than 30 days will be silently excluded from processing under default settings.
- **Full table scan before Python-side filtering** — The SQL queries no longer include any timestamp constraint, fetching the entire unprocessed backlog into memory before applying the window filter and limit in Python. For databases with large historical records this will degrade pipeline performance; adding `AND (file_exists = 1 OR timestamp >= ?)` directly to the SQL when the window is not unlimited would be more efficient.

<h3>Confidence Score: 2/5</h3>

- Not safe to merge as-is — real late-arriving files will be silently skipped under default settings, contradicting the stated design and recovery workflow.
- The overall structure is sound and the propagation of the new parameter through the pipeline is clean. However, there is a meaningful logic bug in the two core file-selection functions in discovery.py: the reprocessing window is applied to real files as well as gap placeholders, which contradicts the explicit design requirement that real files always remain eligible. This would silently prevent late-arriving uploads older than 30 days from being ingested, making the feature incomplete relative to its own specification.
- netflow-db/discovery.py — specifically get_pending_files (line 287) and get_files_needing_processing (line 370) need the cutoff guard conditioned on `file_exists = 0` only.

<h3>Important Files Changed</h3>

| Filename | Overview |
|----------|----------|
| netflow-db/discovery.py | Core logic file with two bugs: both get_pending_files and get_files_needing_processing apply the reprocessing window cutoff to real files (file_exists=1) as well as synthetic gap placeholders, contradicting the design spec which requires real files to always be eligible. Also fetches all unprocessed rows into memory before Python-side filtering, which will be slow at scale. |
| netflow-db/pipeline.py | Clean orchestration layer; correctly threads reprocess_window_days through run_discovery and run_processing, and adds the CLI argument with sensible defaults. |
| netflow-db/flow_db.py | Correctly propagates reprocess_window_days to handle_stale_days and get_files_needing_processing; no issues. |
| netflow-db/ip_db.py | Correctly propagates reprocess_window_days; no issues. |
| netflow-db/protocol_db.py | Correctly propagates reprocess_window_days; no issues. |
| netflow-db/spectrum_db.py | Correctly propagates reprocess_window_days; no issues. |
| netflow-db/structure_db.py | Correctly propagates reprocess_window_days; no issues. |
| docs/pipeline-usage.md | New documentation accurately describes the --reprocess-window-days flag and its behavior; no issues. |
| plans/reprocessing-window-plan.md | Design document; clearly captures the intent that real files must always remain eligible regardless of window, which the implementation does not fully honour. |

</details>

<h3>Sequence Diagram</h3>

```mermaid
sequenceDiagram
    participant CLI as pipeline.py (CLI)
    participant D as discovery.py
    participant P as *_db.py (processors)

    CLI->>CLI: parse --reprocess-window-days (default 30)
    CLI->>D: run_discovery(conn, reprocess_window_days)
    D->>D: get_reprocess_cutoff_dt(reprocess_window_days)
    D->>D: sync_processed_files_table(conn, reprocess_window_days)
    note over D: Gap scan starts at max(router_start, cutoff_dt)
    D->>D: get_pending_files(conn, reprocess_window_days)
    note over D: ⚠ Cutoff applied to ALL rows incl. file_exists=1

    CLI->>P: run_processing(conn, tables, limit, reprocess_window_days)
    P->>D: handle_stale_days(conn, table, reprocess_window_days)
    D->>D: get_reprocess_cutoff_dt(reprocess_window_days)
    note over D: Stale days outside window are skipped
    P->>D: get_files_needing_processing(conn, table, limit, reprocess_window_days)
    D->>D: get_reprocess_cutoff_dt(reprocess_window_days)
    note over D: ⚠ Cutoff applied to ALL rows incl. file_exists=1
    P->>P: process files / rebuild day aggregates
```

<sub>Last reviewed commit: 4393ae4</sub>

> Greptile also left **3 inline comments** on this PR.

<!-- /greptile_comment -->