docs: rework setup docs and simplify first-time configuration#87

Problem

The setup documentation mixed user, contributor, and generated content, predated the Rust pipeline rewrite (#89–#93), and assumed the lab environment. An external user following the docs hit dead ends: a quick start that omitted the mandatory nfdump fork build, a fresh checkout that rendered a 500 error page, a datasets.json.example with lab-specific entries (including a CSV dataset that dataset mode cannot ingest), and no troubleshooting for the failures a first setup actually produces.

Solution

Documentation reorganization (as before):

• Separate user, code, and agent-generated documentation under docs/user, docs/code, and docs/agent.
• Rewrite the pipeline docs around the Rust atlantis-netflow-db crate and the pinned nfdump fork; replace the Python and uv requirements with the pinned Rust toolchain.

First-time-user pass on top of it:

• Honest README quick start for nfcapd users: submodule init, nfdump fork build, --nfdump, a bounded first import, and compile-time warnings.
• docs/user/troubleshooting.md keyed on the exact errors of a first setup (missing nfdump, system nfdump rejection, missing member directories, "No datasets found", bun install without Node.js, empty charts).
• docs/user/requirements.md splits the required tools by concern (dashboard, pipeline, nfdump fork build); the README quick start is numbered steps with an explicit configuration step.
• Dataset docs lead with a minimal example and explain field semantics; developer-only CI checks moved out of the user path.
• fix(web): zero discovered databases is now a valid state. The index page renders setup guidance instead of a 500 error page.
• fix(web): dev no longer boots the Cloudflare workerd emulator unless ATLANTIS_DB_DRIVER=d1 is exported. This unblocks systems that cannot run the workerd binary (NixOS without nix-ld) and removes the missing-.env → unmigrated-local-D1 500.
• feat(netflow-db): datasets.json shrinks to three required fields (dataset_id, root_path, sources). db_path defaults to data/<dataset-id>/netflow.sqlite (the dashboard's auto-discovery location) and an unset default_start_date is inferred as the earliest ingested local day, recomputed each run. Explicit configuration wins for both.

Flows to exercise

• Fresh checkout, no data/ and no .env, bun run dev:web → "No datasets found" guidance page instead of a 500; no workerd process starts.
• ATLANTIS_DB_DRIVER=d1 bun run dev:web → workerd emulation and local D1 as before (needs applied migrations).
• datasets.json with only dataset_id, root_path, sources → pipeline writes data/<dataset-id>/netflow.sqlite; dashboard opens on the earliest ingested day.
• A dataset with explicit db_path/default_start_date → both values are used verbatim (selection-variant databases keep working).

Edge cases: a run that ingests nothing keeps the deterministic 2025-02-01 fallback (column is NOT NULL); the start-date inference probes 5m granularity so a partial first day still counts; ingesting earlier days later moves the inferred date back.

Validation

• bun run format, bun run lint, bun run typecheck
• bun run test:db (106 passed; new tests for the db_path default, inferred start date, explicit-config-wins, and empty-ingest fallback)
• bun run test:web (63 passed; new test for the empty-dataset state)
• Documented commands, flags, env vars, and tables checked against the code. An independent GPT-5.6 review verified the original docs claims.
• Manual: dashboard empty state not yet eyeballed in a browser.

Written by GPT-5.6-sol with the Codex harness in T3 Code. Refreshed for the Rust pipeline and reworked for external users by Claude Fable 5 in Claude Code (pipeline inference implemented by an Opus 4.8 subagent).