AirLock DB

Local-first migration tool for air-gapped environments. It reads legacy files, validates every record against a strict schema, writes accepted records to SQLite or a JSONL file, and produces one self-contained HTML report.

• Makes no network requests.
• Embeds CSS and SVG in the report, so it opens on a workstation with no internet access.
• Never crashes on a bad row. Rejected rows get per-field reasons and the run continues.

Contents

• Features
• Install
• Quick start
• Commands
• Common tasks
• Exit codes
• Documentation
• License

Features

Install

Requirements:

• Python 3.10 or newer
• Pydantic 2.6 or newer, below 3
• PyYAML 6.x, installed with the package and read only for YAML schema configs
• SQLite, included with Python

On a connected machine:

In an air-gapped environment, stage the Pydantic and PyYAML wheels in an approved local wheelhouse, then install without an index:

The guide covers staging the wheelhouse, the manifest check, and the move to the offline machine.

From a source checkout, python main.py works directly if Pydantic 2 is installed. Every airlock-db command below also runs as python main.py.

Quick start

Output:

Open output/airlock_report.html to see the chart, drift counts, and issue table.

Use a fresh output directory for each run. Existing rows count as duplicates or conflicts.

Commands

schema-sentry is a second name for the same entry point.

Key options

Every option is in the CLI reference.

Output files

Common tasks

Read inputs

Several sources

Repeat --input. Each path is a supported file or a directory, scanned recursively.

CSV or TSV

Header names must match the schema. The delimiter defaults by file suffix. Override it and the encoding with --csv-delimiter and --csv-encoding.

Compressed files

Gzip files and zip archives stream directly and are never extracted. Every zip member with a supported suffix is validated.

A legacy SQLite database

Read rows directly, with no file export. Column names must match the target schema.

• Choose tables with --source-table. Repeat it to read several tables in one run.
• Read a filtered view with one --source-query per database.

Write outputs

JSONL migration file

Write accepted records as clean JSONL instead of SQLite. Use --output-format both for both targets.

Pseudonymized fields

Account IDs, counterparties, and descriptions become one-run tokens. Report details are redacted.

Handle large migrations

Resume an interrupted run

Commit and save progress every N records. Rerun the same command with --resume to continue where the run stopped.

• Both options need the SQLite target.
• --resume cannot combine with --anonymize, because pseudonyms are run-scoped.
• --resume skips writes only when the source file is unchanged (same size and modification time).

Check a large source before writing

Counts and issue codes print to the terminal. No target, report, or summary is written.

Preview before the full run

Stop after N accepted records. The output directory holds only the scanned prefix. The report, run summary, and any repair log record that the run was a preview.

A preview is not a handoff. verify reports the unscanned records as missing, so rerun without --sample before verifying.

Work with schemas

Validate against a client schema

Replace the built-in model with a JSON or YAML layout. The bundled example is a support-ticket schema.

Draft a schema from your data

infer reads the source with the normal readers and drafts a config from what is in the files. It uses conservative types and marks fields that need review.

Check a schema before you run

check loads a schema, constraints file, or fixed-width layout with the migration's own loaders, then prints the counts and fingerprints. It reads no input and writes nothing.

A bad file exits 2 before any source is read.

Verify a handoff

Before a target database leaves the machine, confirm it still matches its sources. verify re-reads the sources, rebuilds every stored row with the migration's own rules, and compares per-record digests against the committed database, which it opens read-only.

• Pass the same source and schema flags as the original run.
• Exit code 0 means the target matches the sources under the recorded conflict policy. Exit code 1 means a mismatch or a missing target. Exit code 2 means bad flags or config files.
• A JSONL-only run is verified from target_migration.jsonl. When both targets exist, verify checks the database, which is canonical.
• A changed, missing, or extra row fails the check, lists the row IDs, and writes airlock_verify.json into the output directory.
• Anonymized fields stay out of the digests, because their tokens use a run-scoped key that is never stored.

Repair rejected rows

Write a repair log during migration, then review and fix rows.

Before you repair:

• Do not add --anonymize to a migration whose log you plan to repair. Anonymized logs omit rejected values and are not repairable.
• Pass the same --config and --constraints used for the migration. A fingerprint mismatch stops the session.
• The database defaults to target_migration.db beside the error log. Change it with --output-db.

Prompt keys:

Related options:

• repair --emit-decisions decisions.jsonl writes an inert template with one skip line per repairable entry. Edit it to script fixes. It refuses to replace an existing file without --overwrite.
• repair --decisions decisions.jsonl applies a decisions file.
• repair --error-log output/airlock_errors.jsonl --summary prints entry, repairability, raw-row, and issue-category counts without opening a session or writing.
• python main.py report --output-dir output rebuilds the report with a Repair outcomes section.

Resume and checkpoint rules are in the guide.

Try real data and measure speed

Public dataset sample

A 10-row sample of real UCI Online Retail records runs through the dynamic-schema path.

examples/real_data/README.md also covers converting and running MoneyData, a real anonymized bank ledger, through the built-in schema. It includes attribution and license terms.

Benchmark throughput

Measures JSONL parsing, dynamic-schema validation, SQLite writes, and report generation on synthetic records.

Soak a large migration

Generates a synthetic JSONL source, migrates it with the real CLI, and asserts the stored row count, summary counts, wall time, and peak memory against an optional ceiling. Results land in soak_verification.json.

• On the development machine, 1,000,000 rows migrate in about 37 seconds (27k records/s) at a 41 MiB peak. Peak memory is the same at 20,000 rows, so buffers stay flat as row count grows.
• Peak RSS is exact for a single run on Linux or macOS, where the platform reports the finished child's high-water mark.
• Other setups sample the live process every 50 ms.
• A platform that cannot report memory leaves it unmeasured instead of failing the ceiling check.
• CI runs a 100,000-row variant with the same ceiling.

Exit codes

Documentation

License

MIT. See LICENSE.