<!-- greptile_comment -->

<h3>Greptile Summary</h3>


- Refactors database schema for `structure_db.py` and `spectrum_db.py` to separate source and destination IP analysis data into distinct JSON columns (`structure_json_sa`/`structure_json_da` and `spectrum_json_sa`/`spectrum_json_da`)
- Adds IPv4/IPv6 version support with new `ip_version` column and updates primary keys to include version for future compatibility
- Changes spectrum data format from raw dict to structured list of objects with 'alpha' and 'f' keys to match frontend `SpectrumPoint` interface expectations

<h3>Important Files Changed</h3>


| Filename | Overview |
|----------|----------|
| netflow-db/structure_db.py | Refactored table schema to separate source/destination structure functions, added IPv6 support, and changed data format to match frontend interfaces |
| netflow-db/spectrum_db.py | Updated schema with separate SA/DA spectrum columns, added ip_version support, and modified spectrum computation to return structured point data |

<h3>Confidence score: 3/5</h3>


- This PR requires careful review due to significant database schema changes and data format modifications
- Score reflects concerns about breaking changes to database schema, potential performance impacts from computing spectra twice per file, and possible frontend compatibility issues with the new data formats
- Pay close attention to both database files as they introduce schema changes that may require database migration and frontend updates

<h3>Sequence Diagram</h3>

```mermaid
sequenceDiagram
    participant Main as "Main Process"
    participant Discovery as "Discovery Module"
    participant DB as "SQLite Database"
    participant Pool as "Process Pool"
    participant Worker as "Day Worker"
    participant MAAD as "MAAD Binaries"
    participant FS as "Filesystem"

    Main->>Discovery: "sync_processed_files_table()"
    Discovery->>FS: "scan_filesystem()"
    FS-->>Discovery: "file list"
    Discovery->>DB: "INSERT new files"
    Discovery->>DB: "identify and insert gaps"
    
    Main->>Discovery: "get_files_needing_processing()"
    Discovery->>DB: "SELECT unprocessed files"
    DB-->>Discovery: "file list"
    Discovery->>Discovery: "group_files_by_day()"
    Discovery-->>Main: "files grouped by day"

    Main->>Pool: "map(process_day_worker, day_tasks)"
    
    loop For each day
        Pool->>Worker: "process_day_worker((router, day_start, files))"
        
        loop For each file in day
            Worker->>Worker: "process_file()"
            Worker->>FS: "nfdump -r file"
            FS-->>Worker: "IP addresses"
            Worker->>Worker: "extract source/dest IPs"
        end
        
        Worker->>Worker: "compute_aggregates()"
        
        loop For each granularity
            Worker->>MAAD: "Singularities/StructureFunction binary"
            MAAD-->>Worker: "spectrum/structure data"
        end
        
        Worker->>DB: "INSERT 5m and aggregate results"
        Worker->>DB: "batch_mark_processed()"
        Worker-->>Pool: "processing stats"
    end
    
    Pool-->>Main: "all worker results"
    Main->>Main: "aggregate final stats"
```

<!-- greptile_other_comments_section -->

**Context used:**

- Context from `dashboard` - AGENTS.md ([source](https://app.greptile.com/review/custom-context?memory=7aea0312-a002-45e6-8e0a-d2380aa2087c))
- Context from `dashboard` - .cursor/rules/svelte.mdc ([source](https://app.greptile.com/review/custom-context?memory=591a6ea0-1733-46da-9c1e-60a7c778920e))

<!-- /greptile_comment -->