# Files changed · PR \#117 · flambo/atlantis [View on GitCafe](https://git.cafe/flambo/atlantis/pull/117/diffs) Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis) Visibility: public [feat\(infra\): add self-hosted container stack](https://git.cafe/flambo/atlantis/pull/117.md) Version: 0 Head commit: a8f21a42c57c42ecbdc9b3aa183989446d5e7180 Base commit: 75182f5692cbb4979c7e3b03bc8a681a6e740ad5 Comparison base: 75182f5692cbb4979c7e3b03bc8a681a6e740ad5 Showing up to 20 entries from this revision. If the pull request changes, reopen the first page before continuing. ## \.env.example Status: modified; +2; −0 ```diff @@ -14,3 +14,5 @@ # Set one database path or use automatic discovery at data/*/netflow.sqlite. # LOCAL_SQLITE_PATH=/absolute/path/to/netflow.sqlite # DATABASE_PATH=/absolute/path/to/netflow.sqlite +# Or discover */netflow.sqlite under another directory instead of data/. +# LOCAL_DATA_DIR=/absolute/path/to/data ``` ## \.github/workflows/ci.yaml Status: modified; +5; −1 ```diff @@ -23,6 +23,10 @@ - run: cp datasets.json.example datasets.json + - uses: actions/setup-node@v6 + with: + node-version-file: .node-version + - uses: oven-sh/setup-bun@v2 with: bun-version-file: package.json @@ -49,7 +53,7 @@ run: bun run --cwd apps/web typecheck && bun run --cwd infra typecheck - name: Test + run: bun run test:web && bun run test:infra - run: bun run test:web - name: Build run: bun run build:web ``` ## \.node-version Status: modified; +1; −1 ```diff @@ -1,1 +1,1 @@ -24.4.1 +24.18.1 ``` ## AGENTS.md Status: modified; +1; −1 ```diff @@ -7,7 +7,7 @@ - `tools/netflow-db`: Rust pipeline (`atlantis-netflow-db` crate) for ingestion, aggregation, verification, and analysis-window exports. Native `nfcapd` ingestion uses the pinned `nfdump` fork in `vendor/nfdump`. - `apps/web`: Svelte 5/SvelteKit 3 dashboard and API routes. It reads local SQLite during development and Cloudflare D1 in deployment. - `apps/landing`: Astro marketing and SEO site. -- `infra`: Alchemy v2 stacks (Effect programs) that deploy the dashboard. `cloudflare.ts` deploys the worker and its D1 database; `shared.ts` holds names and paths common to all stacks. +- `infra`: Alchemy v2 stacks (Effect programs) that deploy the dashboard. `cloudflare.ts` deploys the worker and its D1 database; `self-hosted.ts` runs the SQLite dashboard as a Docker container on a self-hosted machine that users reach through SSH port forwarding; `shared.ts` holds names and paths common to all stacks. - `vendor/*`: Third-party analysis submodules. Treat these as read-only; build repo-local binaries through the scripts in `vendor/scripts/`. - `data/`, `.env`, and `datasets.json`: Machine-local inputs and generated databases. Keep paths and dataset contents out of commits. - `docs/user`: User setup and operation documentation. ``` ## apps/landing/\.node-version Status: modified; +1; −1 ```diff @@ -1,1 +1,1 @@ -22.16.0 +24.18.1 ``` ## apps/web/Dockerfile Status: added; +58; −0 ```diff @@ -1,0 +1,58 @@ +# syntax=docker/dockerfile:1 + +ARG BUN_VERSION=1.3.11 +ARG NODE_VERSION=24.18.1 + +FROM oven/bun:${BUN_VERSION}-slim AS bun + +FROM node:${NODE_VERSION}-bookworm-slim AS workspace + +ARG DEBIAN_FRONTEND=noninteractive +RUN apt-get update \ + && apt-get install --yes --no-install-recommends g++ make python3 \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=bun /usr/local/bin/bun /usr/local/bin/bun + +WORKDIR /app +COPY package.json bun.lock ./ +COPY patches patches +COPY apps/web/package.json apps/web/package.json +COPY apps/landing/package.json apps/landing/package.json +COPY infra/package.json infra/package.json + +FROM workspace AS build + +RUN bun install --frozen-lockfile --ignore-scripts --filter @atlantis/web + +COPY apps/web apps/web + +RUN cd apps/web && ATLANTIS_DB_DRIVER=sqlite bun run build + +FROM workspace AS production-deps + +RUN bun install --frozen-lockfile --ignore-scripts --production --filter @atlantis/web \ + && npm run build-release --prefix node_modules/better-sqlite3 \ + && rm -rf node_modules/better-sqlite3/build/Release/obj node_modules/better-sqlite3/deps \ + && node -e "new (require('better-sqlite3'))(':memory:').prepare('select 1').get()" + +FROM node:${NODE_VERSION}-bookworm-slim AS runtime + +ARG RUNTIME_UID=1000 +ARG RUNTIME_GID=1000 + +ENV NODE_ENV=production \ + HOST=0.0.0.0 \ + PORT=3000 \ + LOCAL_DATA_DIR=/data + +WORKDIR /app +COPY --from=production-deps /app/node_modules node_modules +COPY --from=build /app/apps/web/package.json package.json +COPY --from=build /app/apps/web/build build + +RUN mkdir -p /data + +USER ${RUNTIME_UID}:${RUNTIME_GID} +EXPOSE 3000 +CMD ["node", "build"] ``` ## apps/web/Dockerfile.dockerignore Status: added; +18; −0 ```diff @@ -1,0 +1,18 @@ +* +!package.json +!bun.lock +!patches +!infra/package.json +!apps/landing/package.json +!apps/web +apps/web/node_modules +apps/web/build +apps/web/dist +apps/web/.svelte-kit +apps/web/test-results +apps/web/playwright-report +apps/web/tests +apps/web/Dockerfile +apps/web/Dockerfile.dockerignore +**/.env +**/.env.* ``` ## apps/web/package.json Status: modified; +1; −0 ```diff @@ -26,6 +26,7 @@ "@eslint/js": "^9.18.0", "@internationalized/date": "^3.12.0", "@playwright/test": "1.52.0", + "@sveltejs/adapter-node": "6.0.0-next.12", "@sveltejs/kit": "3.0.0-next.27", "@sveltejs/load-config": "^0.2.3", "@sveltejs/vite-plugin-svelte": "^7.3.1", ``` ## apps/web/src/env.ts Status: modified; +5; −0 ```diff @@ -11,6 +11,11 @@ description: 'Fallback for LOCAL_SQLITE_PATH', schema: optional }, + LOCAL_DATA_DIR: { + description: + 'Directory scanned for /netflow.sqlite products; replaces the data/ and ../../data defaults', + schema: optional + }, DEFAULT_DATASET: { description: 'Dataset ID selected when a request does not name one', schema: optional ``` ## apps/web/src/lib/server/db/local-files.ts Status: modified; +4; −2 ```diff @@ -1,4 +1,4 @@ -import { DATABASE_PATH, LOCAL_SQLITE_PATH } from '$app/env/private'; +import { DATABASE_PATH, LOCAL_DATA_DIR, LOCAL_SQLITE_PATH } from '$app/env/private'; async function resolvePath(value: string): Promise { if (value === ':memory:') { @@ -17,7 +17,9 @@ const fs = await import('node:fs/promises'); const path = await import('node:path'); - const roots = [path.resolve(process.cwd(), 'data'), path.resolve(process.cwd(), '../../data')]; + const roots = LOCAL_DATA_DIR + ? [await resolvePath(LOCAL_DATA_DIR)] + : [path.resolve(process.cwd(), 'data'), path.resolve(process.cwd(), '../../data')]; const dbPaths = new Set(); for (const root of roots) { ``` ## apps/web/tests/lib/server/datasets.test.ts Status: modified; +32; −0 ```diff @@ -548,6 +548,38 @@ await expect(datasets.getDatasetConfig('legacy')).rejects.toThrow(/Unknown dataset 'legacy'/); }); + it('skips pre-locality databases found under LOCAL_DATA_DIR', async () => { + const workspace = fs.mkdtempSync(path.join(os.tmpdir(), 'datasets-local-data-dir-')); + const dataDir = path.join(workspace, 'mounted-data'); + const legacyDir = path.join(dataDir, 'legacy'); + const currentDir = path.join(dataDir, 'current'); + fs.mkdirSync(legacyDir, { recursive: true }); + fs.mkdirSync(currentDir, { recursive: true }); + seedPreLocalityDatasetDb( + path.join(legacyDir, 'netflow.sqlite'), + 'legacy', + 'Legacy Only', + 'router-legacy' + ); + seedDatasetDb(path.join(currentDir, 'netflow.sqlite'), 'current', 'Current', 'router-current'); + process.chdir(os.tmpdir()); + vi.stubEnv('LOCAL_DATA_DIR', dataDir); + + const datasets = await loadDatasetsModule(); + + await expect(datasets.listDatasetSummaries()).resolves.toEqual([ + { + datasetId: 'current', + label: 'Current', + defaultStartDate: '2025-03-01', + discoveryMode: 'static', + hasLocality: false, + isDefault: true + } + ]); + await expect(datasets.getDatasetConfig('legacy')).rejects.toThrow(/Unknown dataset 'legacy'/); + }); + it('refreshes local dataset discovery after files move', async () => { const workspace = fs.mkdtempSync(path.join(os.tmpdir(), 'datasets-refresh-')); const alphaDir = path.join(workspace, 'data', 'alpha'); ``` ## apps/web/tests/lib/server/db/local-files.test.ts Status: added; +42; −0 ```diff @@ -1,0 +1,42 @@ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { discoverLocalSqlitePaths } from '../../../../src/lib/server/db/local-files'; + +describe('local SQLite discovery', () => { + let dataDir: string; + + beforeEach(() => { + dataDir = mkdtempSync(join(tmpdir(), 'atlantis-local-files-')); + for (const dataset of ['beta', 'alpha']) { + mkdirSync(join(dataDir, dataset)); + writeFileSync(join(dataDir, dataset, 'netflow.sqlite'), ''); + } + mkdirSync(join(dataDir, 'backups')); + writeFileSync(join(dataDir, 'backups', 'old.sqlite'), ''); + }); + + afterEach(() => { + vi.unstubAllEnvs(); + rmSync(dataDir, { recursive: true, force: true }); + }); + + it('scans LOCAL_DATA_DIR for dataset products', async () => { + vi.stubEnv('LOCAL_DATA_DIR', dataDir); + + await expect(discoverLocalSqlitePaths()).resolves.toEqual([ + join(dataDir, 'alpha', 'netflow.sqlite'), + join(dataDir, 'beta', 'netflow.sqlite') + ]); + }); + + it('prefers LOCAL_SQLITE_PATH over LOCAL_DATA_DIR', async () => { + vi.stubEnv('LOCAL_DATA_DIR', dataDir); + vi.stubEnv('LOCAL_SQLITE_PATH', join(dataDir, 'beta', 'netflow.sqlite')); + + await expect(discoverLocalSqlitePaths()).resolves.toEqual([ + join(dataDir, 'beta', 'netflow.sqlite') + ]); + }); +}); ``` ## apps/web/vite.config.ts Status: modified; +3; −0 ```diff @@ -1,4 +1,5 @@ import { fileURLToPath } from 'node:url'; +import adapterNode from '@sveltejs/adapter-node'; import tailwindcss from '@tailwindcss/vite'; import { sveltekit } from '@sveltejs/kit/vite'; import { vitePreprocess } from '@sveltejs/vite-plugin-svelte'; @@ -56,6 +57,7 @@ export default defineConfig(({ command, mode, isPreview = false }) => { const driver = resolveDatabaseDriver(command, mode, isPreview); + const nodeBuild = command === 'build' && driver === 'sqlite'; return { plugins: [ @@ -63,6 +65,7 @@ databaseDriver(driver), sveltekit({ preprocess: vitePreprocess(), + adapter: nodeBuild ? adapterNode() : undefined, env: { dir: '../..' } ``` ## bun.lock Status: modified; +38; −0 ```diff @@ -44,6 +44,7 @@ "@eslint/js": "^9.18.0", "@internationalized/date": "^3.12.0", "@playwright/test": "1.52.0", + "@sveltejs/adapter-node": "6.0.0-next.12", "@sveltejs/kit": "3.0.0-next.27", "@sveltejs/load-config": "^0.2.3", "@sveltejs/vite-plugin-svelte": "^7.3.1", @@ -94,6 +95,7 @@ }, }, "patchedDependencies": { + "alchemy@2.0.0-beta.79": "patches/alchemy@2.0.0-beta.79.patch", "@alchemy.run/frontend-frameworks@2.0.0-beta.79": "patches/@alchemy.run%2Ffrontend-frameworks@2.0.0-beta.79.patch", }, "packages": { @@ -617,6 +619,8 @@ "@sveltejs/acorn-typescript": ["@sveltejs/acorn-typescript@1.0.13", "", { "peerDependencies": { "acorn": "^8.9.0" } }, "sha512-wgKggnhZVL9Bfx1OaKKTrYY9BFRk6C8UAkQNUcIv1+llzYrIqy+RZm5HPKzn0NpEBvTVhTqB4kQyllZywsRBRQ=="], + "@sveltejs/adapter-node": ["@sveltejs/adapter-node@6.0.0-next.12", "", { "dependencies": { "rolldown": "^1.2.3" }, "peerDependencies": { "@sveltejs/kit": "^3.0.0-next.0" } }, "sha512-wheNAOlAqLoXvGeSxJChM4jMEAKoMIcBCE18jGbTUdDDENPLefTmX55ACeUOkFqaVeZbAwKVHKifKZeyVNR9Yg=="], + "@sveltejs/kit": ["@sveltejs/kit@3.0.0-next.27", "", { "dependencies": { "@rolldown/pluginutils": "^1.0.1", "@standard-schema/spec": "^1.1.0", "cookie": "^2.0.1", "devalue": "^5.9.0", "esm-env": "^1.2.2", "magic-string": "^1.1.0", "mrmime": "^2.0.1", "sirv": "^3.0.2" }, "peerDependencies": { "@opentelemetry/api": "^1.0.0", "@sveltejs/vite-plugin-svelte": "^7.0.0", "svelte": "^5.56.4", "typescript": "^6.0.0", "vite": "^8.0.12" }, "optionalPeers": ["@opentelemetry/api", "typescript"], "bin": { "svelte-kit": "svelte-kit.js" } }, "sha512-m2R6GvcMhiAbYVKhzRf5mzLSvcXEuy/H0Va72XvxfEl1ZglrSC/321GKYa6idDHPojuRzkyh9EB0jBZmAwf6+Q=="], "@sveltejs/load-config": ["@sveltejs/load-config@0.2.3", "", {}, "sha512-VT3qmUb8pRV2QrZjd8iAmtg8lf4W0TIjZbvXtz5MKei/q96teWZgGJyyidJzOjzZzvdq616eSRVeMYIQChUTAQ=="], @@ -1717,6 +1721,8 @@ "@rollup/pluginutils/picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="], + "@sveltejs/adapter-node/rolldown": ["rolldown@1.2.10", "", { "dependencies": { "@oxc-project/types": "=0.151.0", "@rolldown/pluginutils": "^1.0.0" }, "optionalDependencies": { "@rolldown/binding-android-arm-eabi": "1.2.10", "@rolldown/binding-android-arm64": "1.2.10", "@rolldown/binding-darwin-arm64": "1.2.10", "@rolldown/binding-darwin-x64": "1.2.10", "@rolldown/binding-freebsd-x64": "1.2.10", "@rolldown/binding-linux-arm-gnueabihf": "1.2.10", "@rolldown/binding-linux-arm64-gnu": "1.2.10", "@rolldown/binding-linux-arm64-musl": "1.2.10", "@rolldown/binding-linux-ppc64-gnu": "1.2.10", "@rolldown/binding-linux-s390x-gnu": "1.2.10", "@rolldown/binding-linux-x64-gnu": "1.2.10", "@rolldown/binding-linux-x64-musl": "1.2.10", "@rolldown/binding-openharmony-arm64": "1.2.10", "@rolldown/binding-win32-arm64-msvc": "1.2.10", "@rolldown/binding-win32-x64-msvc": "1.2.10" }, "bin": { "rolldown": "./bin/cli.mjs" } }, "sha512-OxkA08pSryMK7B3XiFA09B4OJ1xJMPgIYCBMY2xchzpqgBGsV1o0DetPAE+Sl3N3L4oCPiEzmHVSOj7iR04Zog=="], + "@sveltejs/kit/cookie": ["cookie@2.0.1", "", {}, "sha512-yuToqVvRrj6pfDXREyQAAv8SkAEk/8GS3jQRTiUMm66TVtBYmqQeoEjL2Lmq8Rpo6271vH76InTChTitEAm65w=="], "@sveltejs/kit/devalue": ["devalue@5.9.4", "", {}, "sha512-sPAT4pztbu6586/hrhOnMKS17IJrvg12mXiSPSS3W5qDeN2RGgvZ0diZCm31dBbnevfVmujNO3IM2wrS4Y2Rhg=="], @@ -1979,6 +1985,38 @@ "@puppeteer/browsers/yargs/string-width": ["string-width@8.3.0", "", { "dependencies": { "get-east-asian-width": "^1.5.0", "strip-ansi": "^7.1.2" } }, "sha512-ZbmZM0JCihQN91dWnxoipT2KOEyHqEyfRXUyjuRhW8b/xnqPDoq4gWEVApTVa9db2wN8mmoikgFBbjh71+cGeQ=="], + "@sveltejs/adapter-node/rolldown/@oxc-project/types": ["@oxc-project/types@0.151.0", "", {}, "sha512-J1yXrIlNDZVzE3ada310xeAw7nH8yCAyLPuUIsjKatFPmfn5bS1oW+cM+QsGOtVWd5nhSpbwZWx/rue+r5Z+PA=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-android-arm-eabi": ["@rolldown/binding-android-arm-eabi@1.2.10", "", { "os": "android", "cpu": "arm" }, "sha512-bp9svZb+QurZeh+8H4BhrZkifEB0YBNvTVzNSJnJQkj4NrRwmQoDUCGP0vSN7PbvLeM7l1tK6GXL8mrTiH2myg=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-android-arm64": ["@rolldown/binding-android-arm64@1.2.10", "", { "os": "android", "cpu": "arm64" }, "sha512-wm6Dld3RXUAZ/gRWKyUy+4W1B5CB5UeFaOzsSWJWEdxZXHH8rCYiZ5dGe6oJmhsunAPWzL7FZV+VtvmN5Ye2eA=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-darwin-arm64": ["@rolldown/binding-darwin-arm64@1.2.10", "", { "os": "darwin", "cpu": "arm64" }, "sha512-UbEfXq/AqGNgRTV3ik+X/iR6mUxu2QdYAadwRxJWquUGnW6gDqdP1FtLtFXRow7RJx0ssRwi80XAPr4r+4DtsA=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-darwin-x64": ["@rolldown/binding-darwin-x64@1.2.10", "", { "os": "darwin", "cpu": "x64" }, "sha512-7f5h17q5KZVx/ji1vb8OTq31ch1O2I7K8NPIr44GkyWTApXMIsmhWqZfgpOH10xeauqghDAvGlZktasCkcF6Eg=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-freebsd-x64": ["@rolldown/binding-freebsd-x64@1.2.10", "", { "os": "freebsd", "cpu": "x64" }, "sha512-ynOk/eEYhC6ZB2xCGvKrEOwE58oBy9LnrAqtkrDF9Fz1VTaNdGZTsV0VarJdhPwb+sOJTGjCLwcuyRJZ1dnMcQ=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-linux-arm-gnueabihf": ["@rolldown/binding-linux-arm-gnueabihf@1.2.10", "", { "os": "linux", "cpu": "arm" }, "sha512-ERrAs185meZZhGan7a4l3RiiJK1ArSDlHdST++uvSxe+FDbR4TwUPahT/cbZJvaG6fIpDpF78surN+tX708Y4Q=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-linux-arm64-gnu": ["@rolldown/binding-linux-arm64-gnu@1.2.10", "", { "os": "linux", "cpu": "arm64" }, "sha512-KN7OHKD0J3jy1UzBwZWPxpwhODf9IARUIJcrH+yLYKOcmegZ8luEUM38lDP1bDVj40yP6PsSzCqOJF76vljFnQ=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-linux-arm64-musl": ["@rolldown/binding-linux-arm64-musl@1.2.10", "", { "os": "linux", "cpu": "arm64" }, "sha512-8l9wP8O+wa8zD6iw6egSfzVtu7oZVfH3hlUsMM4MwbLMhxleqeoXbZzjddyK3YyNlwLhqznq3tF7PkNJ8T/V2w=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-linux-ppc64-gnu": ["@rolldown/binding-linux-ppc64-gnu@1.2.10", "", { "os": "linux", "cpu": "ppc64" }, "sha512-SeXNKeQzA5kLhz/J0CH6ZP0/HJ3v1xm/0YbiYpE0kK7emfRC2OIGGIaE14xzkISEGv2aYuUSpiLiU5Gbq+OI0A=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-linux-s390x-gnu": ["@rolldown/binding-linux-s390x-gnu@1.2.10", "", { "os": "linux", "cpu": "s390x" }, "sha512-mtht0nR+y8/hart4175Ll15w7lY8dg7CtQ+j2FDNTsDRspOWTK/2V3l0aj9sIj7XmvqxT8Yli/wq22e7feTTWg=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-linux-x64-gnu": ["@rolldown/binding-linux-x64-gnu@1.2.10", "", { "os": "linux", "cpu": "x64" }, "sha512-FSM94nGd55NYo48usCyM/nHfUKRnqc9+b0vJNuKV0oCCpIp/OGims7rO1Nv/DkFkt0S/s2rxsJ2kkS8J3HcpeA=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-linux-x64-musl": ["@rolldown/binding-linux-x64-musl@1.2.10", "", { "os": "linux", "cpu": "x64" }, "sha512-C3YxNB16myRLs7o+B+6PnQ6jBsdIS4+AE4Ah8glVGhDpEv9AOvxhZ/1duAb4B0UGczEK/lBbccksd8VI+p6zfw=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-openharmony-arm64": ["@rolldown/binding-openharmony-arm64@1.2.10", "", { "os": "none", "cpu": "arm64" }, "sha512-571TlE/F1eeTjjdjYAMMMPs1Mfv3MtX6s3+ZKVU6HiUjZ5Njc6c/qzNy/8K3zALTZnaw3JQVYrHxvNfjm43KAg=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-win32-arm64-msvc": ["@rolldown/binding-win32-arm64-msvc@1.2.10", "", { "os": "win32", "cpu": "arm64" }, "sha512-QXW+ZWaiqs2c7Fi++D/SsW07LTPcUrncxcskJGfGNBoaLik1IU6fJymz4HsqwEO0u5Iq11yTO0B/mc4cPk7jrQ=="], + + "@sveltejs/adapter-node/rolldown/@rolldown/binding-win32-x64-msvc": ["@rolldown/binding-win32-x64-msvc@1.2.10", "", { "os": "win32", "cpu": "x64" }, "sha512-5FQFGgah17YeMtG1Yd5a+rMxQpTksyNXxRtKz06FVTaQw3RKYUJQbUoKk0/5jrXBpDo+7makNP7UHA2LQyH64A=="], + "@tailwindcss/node/lightningcss/lightningcss-android-arm64": ["lightningcss-android-arm64@1.32.0", "", { "os": "android", "cpu": "arm64" }, "sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg=="], "@tailwindcss/node/lightningcss/lightningcss-darwin-arm64": ["lightningcss-darwin-arm64@1.32.0", "", { "os": "darwin", "cpu": "arm64" }, "sha512-RzeG9Ju5bag2Bv1/lwlVJvBE3q6TtXskdZLLCyfg5pt+HLz9BqlICO7LZM7VHNTTn/5PRhHFBSjk5lc4cmscPQ=="], ``` ## docs/code/architecture.md Status: modified; +11; −1 ```diff @@ -36,7 +36,17 @@ The local dashboard reads SQLite databases with `better-sqlite3`. It opens these databases in read-only mode. The deployed dashboard reads the `DB` Cloudflare D1 binding from `cloudflare:workers`. `infra/cloudflare.ts` defines the worker and the D1 database as an Alchemy stack. Alchemy builds the worker with its own SvelteKit adapter and applies the D1 migrations during a deploy. + +The self-hosted dashboard runs the SQLite driver on a Docker host. `infra/self-hosted.ts` defines the `atlantis-self-hosted` stack: a Docker context that reaches the host over SSH, an image built from `apps/web/Dockerfile` on that host, and a container named `atlantis-self-hosted-web-` (`atlantis-self-hosted-web` for `prod`). The image holds the adapter-node build. The container mounts the host data directory at `/data`, publishes its port on the host loopback only, and has a 1 GB memory limit. Users reach it through SSH port forwarding. + +The image tag is a content hash of the platform, build arguments, Dockerfile, and build context. `Dockerfile.dockerignore` is an allowlist, and `infra/build-context.ts` walks only the allowlisted paths, so large ignored trees such as captures, `target`, and `node_modules` are never read. A changed input replaces the container; an unchanged deploy leaves it running. + +The data mount is writable because the pipeline publishes WAL-mode databases. SQLite can read a WAL database only if it can create the `-shm` and `-wal` files next to it, even for a read-only connection. The dashboard still opens every database read-only with `query_only`, so it never writes data. +The container names its image as a plain `:` string. Alchemy compares a container's properties at plan time only when every property is resolved, and an output of an image that is being rebuilt stays unresolved until apply, so the plan would record an in-place update that never swaps the image. The container instead binds to the image's `imageId` output. The binding orders the container after the image on create and before it on destroy, and it is not one of the properties that the container compares. The plan also builds the new image while it compares the image, so the tagged image exists before apply replaces the container. + +`.node-version`, the image, and the CI web job pin Node.js 24.18.1. Node.js 24.19.0 and later abort the process when the garbage collector frees a `better-sqlite3` statement ([nodejs/node#65446](https://github.com/nodejs/node/issues/65446)). `shell.nix` and the CI e2e job stay on an earlier Node.js 24, because Playwright 1.52 hangs while loading its config on Node.js 24.18.1; Playwright 1.53 fixes this. + The build selects one database driver. `apps/web/src/lib/server/db/d1.ts` reads D1, and `apps/web/src/lib/server/db/sqlite.ts` reads the pipeline SQLite files. Server code imports the driver as `#db`. [Development](development.md#choose-the-database-driver) explains the selection. The landing site has no database. It builds static files in `apps/landing/dist`. @@ -60,4 +70,4 @@ - Chart.js - Rust 1.97.1 - SQLite and Cloudflare D1 +- Bun 1.3.11 -- Bun 1.2.16 ``` ## docs/code/development.md Status: modified; +10; −6 ````diff @@ -47,10 +47,10 @@ The dashboard has two database drivers. Each build and each development server includes only one of them. -| `ATLANTIS_DB_DRIVER` | Driver | Default for | -| -------------------- | --------------------------------------------------------------------------------------------- | ----------------------------------- | -| `sqlite` | `src/lib/server/db/sqlite.ts` reads `data//netflow.sqlite` or `LOCAL_SQLITE_PATH` | `vite dev`, `vite preview` (always) | -| `d1` | `src/lib/server/db/d1.ts` reads the `DB` binding from `cloudflare:workers` | `vite build` | +| `ATLANTIS_DB_DRIVER` | Driver | Default for | +| -------------------- | ---------------------------------------------------------------------------------------------------------------- | ----------------------------------- | +| `sqlite` | `src/lib/server/db/sqlite.ts` reads `data//netflow.sqlite`, `LOCAL_DATA_DIR`, or `LOCAL_SQLITE_PATH` | `vite dev`, `vite preview` (always) | +| `d1` | `src/lib/server/db/d1.ts` reads the `DB` binding from `cloudflare:workers` | `vite build` | Server code imports the driver as `#db`. The `imports` field in `apps/web/package.json` maps `#db` to the D1 driver under the `atlantis-d1` export condition, and to the SQLite driver otherwise. `apps/web/vite.config.ts` adds that condition to the server environments and keeps `cloudflare:workers` external when the driver is `d1`. Both drivers implement `DatabaseDriver` in `src/lib/server/db/driver.ts`. @@ -59,12 +59,16 @@ ```bash bun run dev:web # SQLite bun run build:web # D1 server bundle, no adapter -ATLANTIS_DB_DRIVER=sqlite bun run build:web # SQLite server bundle, no adapter +ATLANTIS_DB_DRIVER=sqlite bun run build:web # Node server in apps/web/build bun run --cwd apps/web preview # SQLite build, then vite preview ``` -The web app has no SvelteKit adapter. `bun run build:web` checks that the D1 bundle compiles. The Cloudflare worker is built by Alchemy during a deploy, which injects its own adapter into the `sveltekit()` call. [Operations](../user/operations.md#deploy-the-dashboard) describes the deploy. +A D1 build has no SvelteKit adapter. `bun run build:web` checks that the D1 bundle compiles. The Cloudflare worker is built by Alchemy during a deploy, which injects its own adapter into the `sveltekit()` call. [Operations](../user/operations.md#deploy-the-dashboard) describes the deploy. +A SQLite build uses `@sveltejs/adapter-node` and writes a Node server to `apps/web/build`. Start it with `node build` from `apps/web`. The self-hosted deployment runs this build in a container. [Operations](../user/operations.md#deploy-the-self-hosted-dashboard) describes it. + +The build leaves `paths.origin` unset. SvelteKit 3 replaced adapter-node's runtime `ORIGIN` variable with this build-time option, and a fixed origin would break SSH port forwarding, where each user picks a local port. Adapter-node then builds the request URL from the `Host` header and the `https` protocol, so a request to `http://localhost:8080` has the origin `https://localhost:8080`. The dashboard has no form actions, remote functions, or mutating endpoints, so SvelteKit's CSRF origin check never runs. Before you add a `POST` form, set `PROTOCOL_HEADER` or `paths.origin` so that the origin check sees the browser's real origin. + `preview` always rebuilds with SQLite before it serves, because `vite preview` runs the server in Node and cannot load the D1 bundle. The D1 driver runs only in a deployed worker. `vite dev` rejects `ATLANTIS_DB_DRIVER=d1`. `alchemy dev` does not help here, because it runs SvelteKit's server code in Node and exposes bindings only on `platform.env`. It does not provide the `cloudflare:workers` module. To test D1 behavior, deploy a personal stage: ```` ## docs/user/datasets.md Status: modified; +8; −8 ````diff @@ -68,8 +68,8 @@ ```json { - "dataset_id": "campus", - "root_path": "/data/netflow/campus", + "dataset_id": "dataset-a", + "root_path": "/data/netflow/dataset-a", "sources": [{ "source_id": "router-a", "members": ["router-a"] }], "locality": [ { "type": "prefixes", "prefixes": ["192.0.2.0/24", "2001:db8::/32"] }, @@ -122,8 +122,8 @@ ```json [ { - "dataset_id": "campus-a", - "root_path": "/data/netflow/campus", + "dataset_id": "dataset-a", + "root_path": "/data/netflow/example", "source_ids": ["router-a"], "locality": [ { "type": "prefixes", "prefixes": ["198.18.0.0/16", "198.19.0.0/16"] } @@ -132,11 +132,11 @@ "kind": "daily_active_sources", "ip_prefix": "198.18.0.0/16" }, - "db_path": "data/campus-a/netflow.sqlite" + "db_path": "data/dataset-a/netflow.sqlite" }, { - "dataset_id": "campus-b", - "root_path": "/data/netflow/campus", + "dataset_id": "dataset-b", + "root_path": "/data/netflow/example", "source_ids": ["router-a"], "locality": [ { "type": "prefixes", "prefixes": ["198.18.0.0/16", "198.19.0.0/16"] } @@ -145,7 +145,7 @@ "kind": "daily_active_sources", "ip_prefix": "198.19.0.0/16" }, - "db_path": "data/campus-b/netflow.sqlite" + "db_path": "data/dataset-b/netflow.sqlite" } ] ``` ```` ## docs/user/operations.md Status: modified; +86; −2 ````diff @@ -1,8 +1,8 @@ # Operations -Use these procedures to verify and publish a database. This document also gives the available Cloudflare deployment commands. +Use these procedures to verify and publish a database. This document also gives the Cloudflare and self-hosted deployment commands. -The D1 and deployment sections apply to the hosted ATLANTIS deployment and need Cloudflare access. A local installation does not use them. +The D1 and Cloudflare deployment sections apply to the hosted ATLANTIS deployment and need Cloudflare access. A local installation does not use them. ## Verify a SQLite database @@ -154,7 +154,91 @@ ```bash bunx wrangler@4.141.0 d1 time-travel restore atlantis-db --bookmark= ``` + +## Deploy the self-hosted dashboard + +This runs the dashboard as a Docker container on a machine you control. Users reach it through an SSH tunnel; it is not exposed to the network. + +### Prerequisites + +On the host: + +- Docker Engine, usable by your account without `sudo` (for example through the `docker` group). +- A data directory with `/netflow.sqlite` for each dataset. + +On your machine: + +- The repository with `bun install` done. +- Docker's command-line client. A local Docker daemon is not needed; the image builds on the host. +- SSH key access to the host: `ssh docker info` must succeed without a password prompt. Host aliases from `~/.ssh/config` work. +- `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID`, as for the [Cloudflare stack](#set-up-cloudflare-access). Alchemy keeps the deploy state in Cloudflare; no Cloudflare resources are created. +### Configure + +Export these variables: + +| Variable | Required | Meaning | +| ---------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `ATLANTIS_SELF_HOSTED_DOCKER_HOST` | yes | The host as an SSH URL, for example `ssh://user@host.example.com`, `ssh://user@host.example.com:2222`, or an `~/.ssh/config` alias as `ssh://my-host`. | +| `ATLANTIS_SELF_HOSTED_DATA_DIR` | yes | Absolute path of the data directory on the host. | +| `ATLANTIS_SELF_HOSTED_PORT` | no | Port on the host's loopback interface. Default `8080`. `ssh ss -ltn` lists ports already in use. | +| `ATLANTIS_SELF_HOSTED_DATA_USER` | no | `:` the container runs as. Default `1000:1000`. Use the owner of the data files (`id -u` and `id -g` on the host). | + +The container must run as the data owner because SQLite creates `-shm` and `-wal` files next to each database even when it only reads it. + +### Deploy + +```bash +bun run deploy:self-hosted --stage +``` + +The deploy shows its plan and asks for approval; add `--yes` to skip the prompt. It prints the tunnel command for users. `bun run plan:self-hosted --stage ` previews the change; it builds the image on the host but leaves the container alone. + +Deploy again after pulling new code. An unchanged checkout leaves the running container alone. + +Check that the container is healthy: + +```bash +ssh docker ps --filter name=atlantis-self-hosted-web +``` + +### Connect + +Users need an SSH account on the host. Run the tunnel command that the deploy printed, for example: + +```bash +ssh -N -L 8080:127.0.0.1:8080 user@host.example.com +``` + +Then open `http://localhost:8080`. To use a different local port, change the first number and open that port instead. The tunnel stays open until you stop `ssh`. + +### Tear down + +```bash +bun run destroy:self-hosted --stage +``` + +This removes the container, its current image, and the Docker context. The data directory is untouched. Images from earlier deploys remain; remove them on the host with `docker image rm atlantis-self-hosted-web-:`. + +### Add or replace datasets + +Copy each database to `//netflow.sqlite` on the host. The dashboard picks up new and replaced databases on the next request, without a restart. + +Copy only a database that no process is writing. To replace a live database, use `sqlite-maintenance`; see [Publish a local SQLite database](#publish-a-local-sqlite-database). + +### Run the pipeline on the host + +Use the [Docker wrapper](setup-pipeline.md#docker-setup) from a checkout on the host. It writes to the checkout's `data/` directory, so point `ATLANTIS_SELF_HOSTED_DATA_DIR` there: + +```bash +ssh +git clone https://github.com/flamboh/atlantis.git atlantis +cd atlantis +./scripts/netflow-db-docker.sh --capture-root /data/netflow/example pipeline ... +``` + +See [Set up the data pipeline](setup-pipeline.md) for `datasets.json` and the pipeline commands. + ## Deploy the landing site 1. Build the site with its public URL. ```` ## docs/user/requirements.md Status: modified; +3; −3 ```diff @@ -10,7 +10,7 @@ Running the pipeline with Docker needs only Git and Docker on the host. The image build supplies the Rust toolchain, the nfdump build tools, and the pinned fork, and it does not need initialized Git submodules. -Docker covers the pipeline only; the dashboard runs natively. +To run the dashboard in Docker on a server, see [Deploy the self-hosted dashboard](operations.md#deploy-the-self-hosted-dashboard). Otherwise the dashboard runs natively. ## Dashboard @@ -19,8 +19,8 @@ | Tool | Version | Source of truth | | ------- | ------------------------- | --------------- | | Git | Current supported version | Git releases | -| Bun | 1.2.16 | `package.json` | +| Bun | 1.3.11 | `package.json` | +| Node.js | 24.18.1 | `.node-version` | -| Node.js | 24.4.1 | `.node-version` | Node.js is necessary even though Bun installs the packages. The development server runs under Node.js, and `bun install` needs Node.js on `PATH` to download the prebuilt SQLite driver. Without it, the install prints a `better-sqlite3` warning and the dashboard cannot open a database (see [Troubleshooting](troubleshooting.md)). ``` ## docs/user/setup-pipeline.md Status: modified; +2; −2 ````diff @@ -72,8 +72,8 @@ ```bash ./scripts/netflow-db.sh pipeline \ --nfdump target/nfdump/libexec/nfdump \ - --dataset campus-a \ - --dataset campus-b \ + --dataset dataset-a \ + --dataset dataset-b \ --start-date \ --end-date ``` ```` [Next page](https://git.cafe/flambo/atlantis/pull/117/diffs?after=ImRvY3MvdXNlci9zZXR1cC1waXBlbGluZS5tZCI&expectedVersion=0&format=markdown)