<!-- greptile_comment -->

<h3>Greptile Summary</h3>


This PR refactors the NetFlow data processing pipeline and fixes timezone handling across the application.

**Backend Changes:**
- Extracted shared environment loading and database utilities into `common.py` module
- Created `discovery.py` module for filesystem scanning, gap detection, and tracking via `processed_files` table
- Added `pipeline.py` as unified entry point orchestrating discovery and processing phases
- Refactored all processor modules (`flow_db.py`, `ip_db.py`, `protocol_db.py`, `spectrum_db.py`, `structure_db.py`) to use the new shared modules

**Frontend Changes:**
- Created `timezone.ts` module with `PSTDateComponents` interface for consistent PST/PDT handling
- Refactored all chart components to use PST components instead of JavaScript Date objects
- Fixed Spectrum Chart tick mark rendering bug by using `ctx.tick.value` instead of `ctx.index`
- Moved spectrum address type selection from chart component to centralized `PrimaryFilters` component

**Key Improvements:**
- Eliminates duplicate environment loading code across 5+ Python files
- Centralizes file discovery and gap tracking for consistent processing
- Ensures timezone-correct display regardless of viewer's local timezone
- Fixes Chart.js grid line callbacks that were accessing incorrect context properties

<h3>Confidence Score: 4/5</h3>


- This PR is safe to merge with one minor edge case in timezone conversion logic
- The refactoring is well-structured and eliminates significant code duplication. The spectrum chart fix correctly addresses the Chart.js API change. However, there is one logical issue with month arithmetic in timezone.ts that could cause errors near month boundaries
- Pay close attention to `netflow-webapp/src/lib/utils/timezone.ts` lines 138-143 for the month calculation issue

<h3>Important Files Changed</h3>




| Filename | Overview |
|----------|----------|
| netflow-db/common.py | New module centralizing environment loading, database helpers, and path utilities for shared use across processors |
| netflow-db/discovery.py | New module handling filesystem scanning, gap detection, and processed_files table synchronization |
| netflow-db/pipeline.py | New unified entry point orchestrating file discovery, gap detection, and processing across all tables |
| netflow-webapp/src/lib/utils/timezone.ts | New timezone utilities module providing PST/PDT-aware date handling using PSTDateComponents interface |
| netflow-webapp/src/lib/components/charts/SpectrumStatsChart.svelte | Fixed tick mark callbacks using ctx.tick.value, moved address type selection to filters, integrated timezone utilities |
| netflow-webapp/src/lib/components/filters/PrimaryFilters.svelte | Added spectrum address type radio controls for centralized chart option management |

</details>



<h3>Sequence Diagram</h3>

```mermaid
sequenceDiagram
    participant User
    participant PrimaryFilters
    participant PageSvelte as +page.svelte
    participant SpectrumChart as SpectrumStatsChart
    participant ChartUtils as chart-utils.ts
    participant Timezone as timezone.ts
    participant Pipeline as pipeline.py
    participant Discovery as discovery.py
    participant Common as common.py
    participant Processors as flow_db/ip_db/etc

    Note over User,SpectrumChart: Frontend: Spectrum Chart Tick Fix
    User->>PrimaryFilters: Select address type (sa/da)
    PrimaryFilters->>PageSvelte: spectrumAddressTypeChange event
    PageSvelte->>SpectrumChart: Pass addressType prop
    SpectrumChart->>Timezone: dateStringToEpochPST()
    Timezone-->>SpectrumChart: epoch seconds
    SpectrumChart->>Timezone: epochToPSTComponents()
    Timezone-->>SpectrumChart: PSTDateComponents
    SpectrumChart->>SpectrumChart: Render chart with ctx.tick.value fix
    Note over SpectrumChart: Fixed: use Math.round(ctx.tick.value) for bucket index

    Note over User,Processors: Frontend: Date Handling Refactor
    ChartUtils->>Timezone: parseLabelToPSTComponents()
    Timezone-->>ChartUtils: PSTDateComponents
    ChartUtils->>Timezone: formatDateAsPSTDateString()
    Timezone-->>ChartUtils: PST date string
    Note over Timezone: All charts now use PST components instead of Date objects

    Note over Pipeline,Processors: Backend: Processing Pipeline Refactor
    Pipeline->>Common: load_env_file()
    Common-->>Pipeline: Environment loaded
    Pipeline->>Discovery: sync_processed_files_table()
    Discovery->>Discovery: scan_filesystem()
    Discovery->>Discovery: identify_gaps()
    Discovery-->>Pipeline: Discovery stats
    Pipeline->>Discovery: get_files_needing_processing()
    Discovery-->>Pipeline: List of pending files
    loop For each table
        Pipeline->>Processors: process_pending_files()
        Processors->>Common: get_db_connection()
        Common-->>Processors: DB connection
        Processors->>Discovery: mark_file_processed()
        Discovery-->>Processors: Success
        Processors-->>Pipeline: Processing stats
    end
    Note over Pipeline,Processors: Centralized env loading and file discovery
```

<!-- greptile_other_comments_section -->

<!-- /greptile_comment -->