<!-- greptile_comment -->

<h3>Greptile Summary</h3>

This PR hardens two related areas of the netflow database pipeline: (1) it prevents automatic resets of stale days older than 30 days in `discovery.py`, protecting historical data from unintended reprocessing when late-arriving files are discovered; (2) it raises the minimum IP count threshold from 10 to 100 in both `spectrum_db.py` and `structure_db.py`, making MAAD analysis more statistically meaningful; (3) it makes the CSV output parser in both modules resilient to headers being present or absent; and (4) it adds a new one-shot recovery utility `restore_spectrum_from_backup.py` for restoring `spectrum_stats` rows from a SQLite backup.

Key issues found:

- **SQL injection in `restore_spectrum_from_backup.py`**: `month_clause()` directly interpolates command-line-supplied month strings into SQL using f-strings. While this is an admin-only script, the values are never sanitized or parameterized, making it vulnerable to injection if input is not carefully controlled.
- **Double ATTACH of backup database**: `apply_restore` calls `gather_counts` (which ATTACHes and DETACHes the backup DB), then immediately ATTACHes it again. If `gather_counts` throws after ATTACH but before DETACH, the second ATTACH in `apply_restore` will fail with a SQLite error.
- **`processed_at` overwrite without `IS NULL` guard**: The final `UPDATE processed_files SET processed_at = CURRENT_TIMESTAMP` in `apply_restore` lacks the `AND processed_at IS NULL` condition present in all analogous calls elsewhere in the codebase (`batch_mark_processed`, `mark_file_processed`), causing every fully-complete row in the database (not just restored ones) to get its `processed_at` bumped.
- **`BEGIN IMMEDIATE` after implicit read transaction**: Calling `gather_counts` on the same connection before `BEGIN IMMEDIATE` may leave SQLite in a deferred transaction state, causing `BEGIN IMMEDIATE` to raise `OperationalError: cannot start a transaction within a transaction`.

<h3>Confidence Score: 2/5</h3>

- Not safe to merge without fixes — the restore script has logic bugs that can corrupt processed_at timestamps across the entire processed_files table and will likely throw a SQLite error on the BEGIN IMMEDIATE call.
- The changes to discovery.py, spectrum_db.py, and structure_db.py are well-reasoned and low-risk. However, restore_spectrum_from_backup.py — a new file that performs destructive database surgery — contains multiple correctness bugs: an unguarded processed_at update that affects unrelated rows across the whole DB, a BEGIN IMMEDIATE that may fail due to a prior implicit read transaction on the same connection, a repeated ATTACH/DETACH of the backup database that can fail on error paths, and unparameterized SQL from CLI input. These are not theoretical edge cases; the processed_at overwrite will execute on every run with --apply.
- netflow-db/restore_spectrum_from_backup.py requires careful attention before running with --apply.

<h3>Important Files Changed</h3>




| Filename | Overview |
|----------|----------|
| netflow-db/discovery.py | Adds a 30-day cutoff guard in handle_stale_days to prevent automatic resets of historical stale days; also enriches the returned summary dict with new fields. Logic is sound, but the "stale_days_reset" count in the summary dict is set to len(stale_days_recent) before the loop that may skip some resets on error. |
| netflow-db/restore_spectrum_from_backup.py | New one-shot restore script with SQL injection risk via month_clause, a stale database attachment in gather_counts that is re-attached inside apply_restore (double-attach), and a processed_at update that may inadvertently mark already-complete rows as updated. |
| netflow-db/spectrum_db.py | Raises MIN_IPS_FOR_SPECTRUM threshold from 10 to 100, and makes CSV parsing robust to missing headers by iterating all lines and skipping non-numeric ones via ValueError catch. Changes are correct and safe. |
| netflow-db/structure_db.py | Mirrors spectrum_db.py changes: raises MIN_IPS_FOR_STRUCTURE to 100 and makes CSV header parsing robust. Symmetric and safe. |

</details>



<h3>Flowchart</h3>

```mermaid
flowchart TD
    A[restore_spectrum_from_backup.py --apply] --> B[gather_counts\nDry-run metrics]
    B --> B1[ATTACH old DB]
    B1 --> B2[COUNT suspect rows\nCOUNT backup rows\nDiff analysis]
    B2 --> B3[DETACH old DB]
    B3 --> C{--apply flag?}
    C -- No --> Z[Print dry-run counts\nExit]
    C -- Yes --> D[apply_restore]
    D --> D1[gather_counts again\nre-ATTACH old DB]
    D1 --> D2[ATTACH old DB again\nBegin IMMEDIATE txn]
    D2 --> E[CREATE TEMP suspect_5m\nfrom spectrum_stats]
    E --> F[UPDATE processed_files\nspectrum_stats_status = NULL\nprocessed_at = NULL]
    F --> G[DELETE spectrum_stats\nwhere processed_at LIKE month%]
    G --> H[CREATE TEMP restore_source\nfrom old.spectrum_stats\nwith REPLACE hyphen→underscore]
    H --> I{backup_rows_different_current > 0?}
    I -- Yes --> J[CREATE TEMP restore_diff\nUPDATE spectrum_stats\nfrom diff]
    I -- No --> K
    J --> K[INSERT missing rows\nfrom restore_source]
    K --> L[UPDATE processed_files\nspectrum_stats_status = 1]
    L --> M[UPDATE processed_files\nprocessed_at = NOW\nwhere ALL statuses set\n⚠️ no processed_at IS NULL guard]
    M --> N[COMMIT]
    N --> O[Verify: COUNT hyphen router rows]
    O --> P[DETACH old DB\nReturn counts]

    style D1 fill:#f99,stroke:#c00
    style M fill:#f99,stroke:#c00
    style B1 fill:#ffd,stroke:#aa0
```

<sub>Last reviewed commit: 6f17d1d</sub>

<!-- greptile_other_comments_section -->

<!-- /greptile_comment -->