# feat\(database\): migration process and data_start_date updated \(\#14\) · flambo/atlantis

[View on GitCafe](https://git.cafe/flambo/atlantis/commit/2170a101181c89da09f2c3a17edf528707c0ef8d)

Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis)

Visibility: public

Requested revision: 2170a101181c89da09f2c3a17edf528707c0ef8d

Requested commit: 2170a101181c89da09f2c3a17edf528707c0ef8d

Commit: 2170a101181c89da09f2c3a17edf528707c0ef8d

Tree: 8e5bce51e74f9e43d417acb995eb3c9be85206ab

Author: oliver

Committer: GitHub

## Message

````
feat(database): migration process and data_start_date updated (#14)

<!-- greptile_comment -->

<h3>Greptile Summary</h3>


This PR updates the `DATA_START_DATE` from March 2024 to February 2025 to exclude corrupt/unusable data, and adds a comprehensive migration script to clean the database and backfill the `processed_files` tracking table.

**Key changes:**
- Updated `DATA_START_DATE` constant with clear documentation about data corruption
- Added filtering in `discovery.py` to skip files before the new start date
- Created `migrate_processed_files.py` with backup functionality, dry-run mode, and verification
- Enhanced logging across all processing modules with table-specific prefixes (`[flow_stats]`, `[ip_stats]`, etc.)

**Minor issue found:**
- Documentation inconsistency: The docstring on line 7 mentions "Feb 11" as backfill start, but the constant on line 25 is set to Feb 1, 2025

<h3>Confidence Score: 4/5</h3>


- This PR is safe to merge after clarifying the Feb 1 vs Feb 11 date discrepancy
- The changes are well-structured with proper backup mechanisms and dry-run testing. The migration script follows best practices with phased execution and verification. Only concern is the minor documentation inconsistency about start dates that should be clarified.
- Review `netflow-db/migrate_processed_files.py` to confirm whether Feb 1 or Feb 11 is the correct backfill start date

<h3>Important Files Changed</h3>




| Filename | Overview |
|----------|----------|
| netflow-db/common.py | Updated DATA_START_DATE from 2024-03-01 to 2025-02-01 with clear documentation about data corruption |
| netflow-db/discovery.py | Added filtering to skip files before DATA_START_DATE during filesystem scan |
| netflow-db/migrate_processed_files.py | New migration script to clean post-Oct 2025 data and backfill processed_files table from netflow_stats; includes backup functionality |

</details>



<h3>Sequence Diagram</h3>

```mermaid
sequenceDiagram
    participant User
    participant MigrationScript as migrate_processed_files.py
    participant SQLite as Database
    participant FileSystem as NetFlow Files

    User->>MigrationScript: Run migration (with --dry-run or not)
    
    alt Dry Run Mode
        MigrationScript->>SQLite: Query current row counts
        MigrationScript->>SQLite: Count rows to delete (>= Oct 1, 2025)
        MigrationScript->>SQLite: Count backfill rows from netflow_stats
        MigrationScript->>User: Display what would change (no modifications)
    else Actual Migration
        MigrationScript->>SQLite: Create database backup
        SQLite-->>MigrationScript: Backup created
        
        Note over MigrationScript,SQLite: Phase 1: Delete post-Oct 1 data
        loop For each table
            MigrationScript->>SQLite: DELETE FROM table WHERE timestamp >= Oct 1
        end
        MigrationScript->>SQLite: COMMIT
        
        Note over MigrationScript,SQLite: Phase 2: Ensure table exists
        MigrationScript->>SQLite: init_processed_files_table()
        
        Note over MigrationScript,SQLite: Phase 3: Backfill processed_files
        MigrationScript->>SQLite: INSERT processed_files from netflow_stats (Feb 1 - Oct 1)
        MigrationScript->>SQLite: Mark all 5 tables as processed (status=1)
        MigrationScript->>SQLite: COMMIT
        
        Note over MigrationScript,SQLite: Phase 4: Verification
        MigrationScript->>SQLite: Query final row counts
        SQLite-->>MigrationScript: Return counts
        MigrationScript->>User: Display migration complete
    end
    
    Note over FileSystem: Meanwhile, discovery.py filters files
    FileSystem->>MigrationScript: Files with timestamp < Feb 1, 2025
    MigrationScript->>MigrationScript: Skip (filtered out by DATA_START_DATE check)
```

<!-- greptile_other_comments_section -->

<!-- /greptile_comment -->
````

## Parents

- [e132f1feee49087b4c2892355ab8e46161c5d93c](https://git.cafe/flambo/atlantis/commit/e132f1feee49087b4c2892355ab8e46161c5d93c?format=markdown)

- [49a376c79c1f47a236a4b5f64c5ee6a6038afe14](https://git.cafe/flambo/atlantis/commit/49a376c79c1f47a236a4b5f64c5ee6a6038afe14?format=markdown)

[Source at this commit](https://git.cafe/flambo/atlantis/tree/2170a101181c89da09f2c3a17edf528707c0ef8d?format=markdown)