README.md
AirLock DB
AirLock DB is a local-first migration tool for air-gapped environments. It reads legacy JSON, JSONL, NDJSON, and XML files, validates every record against a strict schema, writes accepted records to a SQLite database, and produces one self-contained HTML report.
It makes no network requests. The report embeds its CSS and SVG, so it opens on a workstation with no internet access.
What it does
- Streams input one record at a time, so memory stays bounded as files grow.
- Reads JSON, JSONL, NDJSON, XML, or rows from a local SQLite database.
- Validates against the built-in financial-transaction model, or any client layout defined in a JSON or YAML config.
- Never crashes on a bad row. Rejected rows are reported with per-field reasons and the run continues.
- Reports schema drift: missing fields, undocumented extras, type mismatches, and new enum values.
- Writes accepted records to SQLite, a JSONL migration file, or both, with
skiporreplaceconflict handling for SQLite. - Optionally pseudonymizes sensitive fields with a run-scoped key that is never stored.
- Provides an interactive repair session for rejected rows, with crash-safe checkpoints.
Requirements
- Python 3.10 or newer
- Pydantic 2.6 or newer and earlier than 3
- PyYAML 6.x, needed only for YAML schema configs
- SQLite, included with Python
Install
On a connected machine:
| 1 | python -m pip install . |
In an air-gapped environment, stage the wheels in an approved local wheelhouse, then install without an index:
| 1 | python -m pip install --no-index --find-links C:\offline\wheelhouse . |
From a source checkout, python main.py works directly if Pydantic 2 is installed.
Quick start
| 1 | python main.py --input examples/messy_transactions.json --constraints examples/constraints.json --output-dir output |
The run prints a summary and writes its files:
| 1 | Records checked: 32 |
| 2 | Valid: 8; invalid: 24 |
| 3 | SQLite: output/target_migration.db (committed) |
| 4 | Report: output/airlock_report.html |
Open output/airlock_report.html to inspect the chart, drift counts, and issue table. Use a fresh output directory for each run. Existing rows are treated as duplicates or conflicts.
Common tasks
Migrate several sources
Repeat --input. Each path can be a supported file or a directory scanned recursively.
| 1 | airlock-db --input C:\legacy\batch-a --input C:\legacy\batch-b.jsonl --output-dir C:\migration\run-01 |
Migrate from a legacy SQLite database
Read rows straight from a local database, without exporting files first. Column names must match the target schema.
| 1 | airlock-db --source-db C:\legacy\records.sqlite --output-dir C:\migration\run-02 |
When the database holds several tables, choose one with --source-table.
Export a JSONL migration file
Write the accepted records as a clean JSONL file instead of a SQLite database. Use both to get both targets.
| 1 | python main.py --input examples/messy_transactions.json --constraints examples/constraints.json --output-format jsonl --output-dir jsonl-output |
Validate a client-specific schema
Replace the built-in model with a JSON or YAML layout. The bundled example is a support-ticket schema.
| 1 | python main.py --input legacy/tickets.json --config examples/schema_config.json --output-dir ticket-output |
Run with pseudonymized fields
Account IDs, counterparties, and descriptions become one-run tokens. Report details are redacted.
| 1 | airlock-db --input C:\legacy\export.jsonl --anonymize --output-dir C:\migration\run-02 |
Repair rejected rows
Write a repair log during migration, then review and fix rows. At each prompt: e edit, p set field, r remove field, d drop, q finish.
| 1 | python main.py --input examples/messy_transactions.json --constraints examples/constraints.json --error-log --output-dir output |
| 2 | python main.py repair --error-log output/airlock_errors.jsonl --constraints examples/constraints.json |
Full behavior, including resume and checkpoint rules: docs/repair.md.
Test with real public data
A 10-row sample of actual UCI Online Retail records runs through the dynamic-schema path. The same notes cover converting and running MoneyData, a real anonymized bank ledger, through the built-in schema. Attribution and license terms are included.
| 1 | python main.py --input examples/real_data/online_retail_sample.jsonl --config examples/real_data/online_retail_schema.json --anonymize --output-dir real-data-output |
See examples/real_data/README.md.
Benchmark throughput
Measures JSONL parsing, dynamic-schema validation, SQLite writes, and report generation on synthetic records.
| 1 | python scripts/benchmark_migration.py --rows 100000 --runs 3 --batch-size 500 |
Exit codes
| Code | Meaning |
|---|---|
0 | No record or source issues, and SQLite committed |
1 | Completed with issues: rejected rows, an incomplete source, a skipped conflicting ID, or a storage failure. The report is still written when the output directory permits it |
2 | Command-line, configuration, output-directory, requested repair-log, or report-writing error |
Documentation
- CLI reference: every option, output file, size limit, conflict rule, anonymization detail, and the constraints and dynamic-schema formats.
- Repair workflow: the session keys, staging, resume behavior, and privacy rules.
- Roadmap: planned upgrades and how each one is verified.
- Real-data examples: public datasets, attribution, and conversion notes.
- Verification skill: the end-to-end checks for this repo and a reusable output verifier.
License
MIT. See LICENSE.