dmp-mvp

Phase 1 MVP for a decentralized, layer-sharded transformer training system.

What This Repo Is

This project is a localhost-first prototype of the DMP design:

• shard a transformer across multiple nodes
• exchange versioned boundary state between adjacent shards
• study staleness, compression, snapshots, and rollback
• compare the distributed run against a centralized baseline

The current implementation is intentionally narrower than the full design document. Real networking, trust systems, proof-of-contribution, and volunteer-node deployment are out of scope for the first milestone.

Current Status

• repo scaffolded
• toy transformer implemented
• shard equivalence proven
• single-window runtime path verified
• test suite passing (37 passed)
• eval artifact locked
• centralized baseline runner implemented
• first frozen baseline run completed
• distributed comparison runner implemented
• first zero-delay distributed comparison run completed
• zero-delay distributed final eval loss matches the frozen baseline (2.063348965211348)
• delayed-network comparison sweeps completed for latency, jitter, and reordering within the staleness budget
• all successful delayed sweeps remain eval-curve identical to the zero-delay control and frozen baseline
• the 1 ms / max_staleness=0 edge still fails fast with a stale-boundary error as designed
• staleness-weighted delayed sweeps now produce measurable eval-loss deltas against baseline
• end-to-end compression sweeps now produce measurable eval-loss deltas against baseline and record boundary density stats
• boundary-activation error feedback is now rejected explicitly; the older EF artifacts are retained only as diagnostic evidence
• supported non-EF compression tuning shows that keeping more boundary values matters more than raising quantization precision on this toy setup
• first combined staleness-plus-compression matrix is complete for the supported top50/6-bit point
• top50/4-bit versus top50/6-bit combined-delay comparison is complete, and the gap stays small once staleness becomes the dominant source of damage
• exact sparse payload byte reporting is now wired into the experiment summaries
• current byte-aware recommendation: top50/4-bit for better quality per byte, top50/6-bit for a slightly safer quality margin under jitter
• hardware-facing default has now moved to top50/6-bit, because the first external-worker attach path should bias toward transport robustness over the last 42 payload bytes
• first multi-process shard smoke now reproduces forward logits, repeated train steps, and checkpoint/version advancement exactly across separate localhost worker processes
• the process runtime now also supports delay, staleness decay, and compression on the real worker handoff path
• the process runtime now rolls all workers back to the last consistent checkpoint after a partial commit failure
• the process runtime can now relaunch a dead worker from the last committed checkpoint and retry the interrupted step cleanly
• the process runtime can now attach to externally hosted shard workers over TCP instead of assuming the parent owns every worker process
• the process experiment entrypoint can now print a ready-to-run launch plan for external workers and the coordinator command
• first process-side comparison sweep is complete at the top50/4-bit operating point, and the degradation ordering matches the simulator (compression-only < 1 ms decay < 2 ms decay)
• process-side top50/4-bit versus top50/6-bit comparator is now complete, and 6-bit keeps the safer quality margin once jitter/reordering enters the path

Current Test Command

Project Layout

Key Artifacts

• DMP_Final_Project_Document.docx
• PHASE1_EXECUTION_PLAN.md
• PROGRESS.md

Immediate Next Step

Use the new process path as the bridge out of pure simulation, now that it covers exact forward execution plus repeated exact train steps across separate workers and can apply the same boundary perturbations directly:

• centralized reference artifacts:
  • baseline/baseline_loss_curve.csv
  • baseline/baseline_config.json
• zero-delay distributed control artifacts:
  • experiments/distributed_loss_curve.csv
  • experiments/distributed_run_summary.json
• delayed comparison artifacts:
  • experiments/distributed_delay_1ms_summary.json
  • experiments/distributed_delay_2ms_summary.json
  • experiments/distributed_delay_jitter_reorder_summary.json
• staleness-weighted artifacts:
  • experiments/distributed_decay_1ms_summary.json
  • experiments/distributed_decay_2ms_summary.json
  • experiments/distributed_decay_jitter_reorder_summary.json
• compression artifacts:
  • experiments/distributed_compression_top25_8bit_summary.json
  • experiments/distributed_compression_top50_8bit_summary.json
  • experiments/distributed_compression_top50_6bit_summary.json
  • experiments/distributed_compression_top50_4bit_summary.json
  • experiments/distributed_compression_top25_12bit_summary.json
• combined tradeoff artifacts:
  • experiments/distributed_combined_top50_6bit_decay_1ms_summary.json
  • experiments/distributed_combined_top50_6bit_decay_2ms_summary.json
  • experiments/distributed_combined_top50_6bit_decay_jitter_reorder_summary.json
  • experiments/distributed_combined_top50_4bit_decay_1ms_summary.json
  • experiments/distributed_combined_top50_4bit_decay_2ms_summary.json
  • experiments/distributed_combined_top50_4bit_decay_jitter_reorder_summary.json
• first multi-process smoke artifact:
  • experiments/process_window_summary.json
• first external-worker smoke artifact:
  • experiments/process_window_external_top50_6bit_summary.json
• diagnostic-only unsupported EF artifacts:
  • experiments/distributed_compression_top25_8bit_ef_summary.json
  • experiments/distributed_compression_top10_4bit_ef_summary.json

The next implementation step is not another simulator sweep. It is carrying the now recovery-capable, external-worker process runtime from localhost into the first real LAN multi-machine smoke at the top50/6-bit operating point.