Sign up
View stargazersSign up

docs(specs): renderer spike for the capture engine first slice (step 0)#1823

opened16d agobyMMatt Donovan

Note

Claude responding on behalf of Matt.

Step 0 of the capture engine first-slice plan (#1803). Nothing blocks it.

The capture engine's first slice will record every network exchange of a background page render, so a Capture can show what the browser fetched. Before anyone writes that code, the project needs to know how the browser's own network reporting behaves inside the hidden window Birdbrain renders pages in. The plan lists seven questions that only a real run can answer, such as whether turning the reporting on breaks the app's automated end-to-end tests, and whether a page embedded from another site shows up at all.

This issue is a short experiment. The agent changes the renderer on a throwaway branch, runs it against small local test pages, and writes up the answers together with the raw recordings. Only the write-up and the recordings merge. Two answers go to the maintainer: if a page embedded from another site goes unreported, the next step stops until a new design exists; if the details of the page's secure connection never arrive, the next step leaves them out of the record.

Agent brief: rulings, scope, acceptance criteria, verification and starting files

Maintainer rulings this issue builds on (2026-10-10)

  • Ruling 35: D1 to D7 of the first-slice plan are the shape step 1 freezes, including certificateTransparencyCompliance, serverSignatureAlgorithm, encryptedClientHello, mac and a top-level boundTlsAbsent. Question 6 checks D7's field names against what Chromium 152 sends before step 1 freezes them.
  • Ruling 32: the probe reruns as a CI test on each Electron change; step 9 commits the probe script, and the maintainer opens the PR that adds the CI job. Question 7 decides whether that script expects history.length to be equal between its two passes or one apart.
  • Rulings 36 and 71: the rulings fold into the plan, and the maintainer reads #1803 and approves it on its PR. The roadmap's gates table holds CE-P steps 1 to 10 for that approval; step 0 does not wait for it. #1803 has since merged with the approved label.

Scope

  • On a scratch branch cut from origin/main, change renderPageInHiddenWindow so it attaches the debugger, loads about:blank, sends Network.enable with the probe's limits (maxTotalBufferSize: 50e6, maxResourceBufferSize: 10e6), loads the target, and writes every Network event that wc.debugger emits as a method and params pair, in arrival order, to a file outside the repository. Add the scratch hooks the questions need: local fixture servers, a network-blocking rule in a fixture consent list, a test-only setCertificateVerifyProc on the render's session, and attempts to drop the about:blank history entry. None of this merges; the scratch branch is thrown away. Generate the test certificate and its key at run time: no private key goes in the doc or the JSON, because the repository's secret scan (.gitleaks.toml, run by .github/workflows/security.yml) fails on one.
  • Answer the plan's seven questions, quoted from its step 0:
    1. Do e2e/recapture.spec.ts and e2e/recapture-consent.spec.ts still pass with the domain on? This tests the comment from #180.
    2. Which requests reach the fixture server during wc.savePage? The answer fixes where collection stops.
    3. How does a request the consent filter cancels arrive: loadingFailed with canceled, with blockedReason, or another way?
    4. Do the requests of a cross-site iframe (for example, a page on 127.0.0.1 framing one on localhost), and both hops of a main document redirected between two loopback ports, appear on wc.debugger? Does the iframe run in its own renderer process?
    5. Can a fixture produce Network.requestServedFromCache or a fromDiskCache response in a fresh partition?
    6. Served over HTTPS with a generated certificate, accepted through a test-only setCertificateVerifyProc on the render's session, does the main document's Network.responseReceived carry securityDetails, and do its field names match D7 on Chromium 152? The spike records the keyExchange, keyExchangeGroup, and mac values it sees.
    7. What does the target read as history.length? A scratch run for this plan read 2 after about:blank and 1 without it (Probe rerun on Electron 44.5.1). Can the renderer remove the about:blank entry before the target's document commits, for example with wc.navigationHistory.clear(), so the target reads 1? If no way works, step 9 records the difference as a known limitation in the D16 validation record and in features.mdx beside the capability, and D15's script expects it.
  • The two stop conditions, which go to the maintainer (plan, step 0 "Done when"):
    • If question 4 shows an iframe's requests missing, step 1 stops: the inventory needs a design for child targets, which goes to the maintainer.
    • If question 6 shows no securityDetails, step 1 leaves boundTls out and Bound TLS Details go back to the maintainer.
  • Other outcomes the doc records without stopping anything: if question 1 fails, step 9 needs an Electron-run harness shaped like the probe; if question 5 finds no way to produce a cache event, that validation item goes to the maintainer (plan, Risks item 4); if question 7 finds no way to drop the entry, step 9 records the limitation as quoted above.
  • Write docs/specs/YYYY-MM-DD-capture-engine-renderer-spike.md (the date it is written) and commit the recorded event streams beside it as JSON (proposed name docs/specs/YYYY-MM-DD-capture-engine-renderer-spike-events.json). Step 7 copies that JSON into src/packages/transaction-record/tests/fixtures/, so a scratch branch cannot be the source. Fixture content is invented.

Acceptance criteria

  • git diff --name-only --no-renames origin/main...HEAD lists only the spike doc and its event-stream JSON file or files, all under docs/specs/.
  • The doc names the commit the scratch branch was cut from, the Electron, Chromium and Node versions the run reported (process.versions), the operating system, and whether #1693 and #1726 had merged at that commit.
  • The doc holds the scratch change to src/main/services/backgroundRenderer.ts as a diff block, the fixture servers' code, and every command run with its exit code, so a reader can rerun the spike without the scratch branch.
  • Each of the seven questions has its own heading holding the question, the run that answers it, the observed output and a one-line verdict. Every verdict comes from a run the doc names; a question no run could answer says so and why.
  • Question 1's verdict states pass or fail for each of the two e2e specs with the domain on, with the Playwright exit code.
  • Question 2's verdict lists each request the fixture server logged between the start and the end of wc.savePage, or states that it logged none.
  • Question 3's verdict names the event and the values of canceled and blockedReason that a request cancelled by the fixture list's network rule produced, or states that no event arrived.
  • Question 4's verdict states, separately, whether the cross-site iframe's requests and each hop of the cross-port redirect appeared on wc.debugger, and whether the iframe ran in a renderer process other than the main frame's, with how that was read.
  • Question 5's verdict names each fixture tried and whether any produced Network.requestServedFromCache or a fromDiskCache response.
  • Question 6's verdict lists every field of the plan's D7 table as present, absent or named differently in the recorded securityDetails, and gives the keyExchange, keyExchangeGroup and mac values seen, including whether keyExchange was the empty string.
  • Question 7's verdict gives the history.length the target read with and without the about:blank load, names each removal method tried, and states whether any made the target read 1.
  • The JSON parses, and for each run holds its name, the question it answers and the recorded events as { method, params } pairs in arrival order (the input step 7's collector reads). Every http, https, ws or wss URL in it has the host 127.0.0.1 or localhost.
  • If either stop condition holds, that question's verdict says so in its first sentence, and the PR body asks the maintainer for the ruling the plan names.
  • Vale reports no errors on the changed lines.

Verification

  • On the scratch branch, for question 1: pnpm build, then env -u ELECTRON_RUN_AS_NODE -u WAYLAND_DISPLAY XDG_SESSION_TYPE=x11 xvfb-run --auto-servernum npx playwright test e2e/recapture.spec.ts e2e/recapture-consent.spec.ts (the form in .claude/skills/run-birdbrain/SKILL.md, "Test"; the env prefix matters only on a Wayland desktop). Port 19845 must be free (the same skill, "Gotchas"). The output goes in the doc; CI never runs the scratch change.
  • On the PR branch:
    • .claude/hooks/vale-prose.sh --base origin/main
    • git diff --name-only --no-renames origin/main...HEAD
    • jq empty docs/specs/<date>-capture-engine-renderer-spike-events.json
    • jq -r '.. | strings' docs/specs/<date>-capture-engine-renderer-spike-events.json | grep -oE '(https?|wss?)://[^[:space:]"]+' | grep -Ev '^(https?|wss?)://(127\.0\.0\.1|localhost)([:/?#]|$)' prints nothing (it scans URLs anywhere in a string, headers and stack frames included)
    • pnpm preflight at head

Starting files

  • docs/plans/2026-10-10-capture-engine-first-slice.md:778 (step 0), :151 (the probe rerun on Electron 44.5.1; the history.length scratch run at :184-189), :293 (D4, the counts the answers feed), :392 (D7, the Bound TLS field table), :563 (D15), :971 (step 7, which reads the JSON), :1011 (step 9, which uses the answers), :1172 (Risks, items 1 to 6 and 9 map to the questions)
  • docs/specs/2026-10-02-capture-engine-probe-spike.md:110 ("The probe": a standalone Electron script that listens to Network events on wc.debugger and tallies them; Network.enable with the limits at :209)
  • docs/specs/2026-05-17-chrome-debugger-tls-spike.md:31 (section 2.1, the securityDetails fields)
  • src/main/services/backgroundRenderer.ts:376 (renderPageInHiddenWindow), :393 (the fresh recapture-<uuid> partition), :465-468 (the debugger attached with no event domain enabled), :497 (wc.loadURL), :519 (wc.savePage), and :48-53 (the comment from #180 above createNetworkIdleTracker, which question 1 tests)
  • src/main/services/recapture.ts:10 (RenderedPage) and :52 (DEFAULT_TIMEOUT_MS)
  • src/main/services/consentBlocker.ts:28-34 (the BIRDBRAIN_CONSENT_LISTS fixture seam). The list e2e/recapture-consent.spec.ts:35 serves holds one cosmetic rule only, so question 3 needs a network-blocking rule in the spike's own fixture list.
  • e2e/recapture.spec.ts:19-21 (a loopback fixture server) and e2e/fixtures/electronApp.ts
  • src/shared/ipc.ts:723 (RecaptureEnqueuePayload) and src/preload/index.ts:90-93 (recapture.enqueue), for queueing a Recapture of a fixture page from the app driver in .claude/skills/run-birdbrain/SKILL.md

Evidence-affecting: no. The PR adds only files under docs/specs/, which is on neither tier of docs/specs/2026-07-31-evidence-affecting-paths-assessment.md (the include list at :181-331; the backstop matcher scripts/jev-lens/backstop.mjs finds no entry for either file). The scratch change to src/main/services/backgroundRenderer.ts (blocking tier, :204) never merges. The steps that act on the answers carry the label themselves.

Order and lanes

  • Wave 0. Waits for nothing.
  • Holds no lane: no migration, no manifest schema change, no Case Archive bump.
  • Hot files: none merged. The scratch edit is to backgroundRenderer.ts, which #1693, #1726, #1695 and #1499 change before the capture engine's writer steps (the roadmap's hot-file order); because it never merges it costs them no rebase. If #1726 has merged at the commit the branch is cut from, the fixture runs turn on its test-only loopback allowance, and the doc says so.
  • Blocks CE-P step 1 (CE-5, the verifier and manifest schema step; not yet filed) and CE-P step 7 (CE-7 first part, the exchange collector and Bound TLS conversion; not yet filed), which copies the JSON into its fixtures. Its answers also set where collection stops, the test certificate allowance and the history.length handling in CE-P step 9 (the wiring step; not yet filed).

Out of scope

  • Merging any change to the renderer, the unit tests or the e2e specs.
  • The exchange collector, the WARC writer and any schema change (CE-P steps 1, 7, 8 and 9).
  • Committing the probe script or adding its CI job (CE-P step 9 and the maintainer's own PR, ruling 32).
  • Designing child-target collection, or ruling on Bound TLS Details, if a stop condition holds: both go to the maintainer.
  • Editing the first-slice plan to record the answers; later steps read the spike doc.
  • The launched-browser probe for stage 2 of the capture engine, which has its own brief.
Bbirdbrain-agent16d ago
Synced from GitHub

Reopened after #1840 merged the spike report, because two decisions it left for the maintainer are still open.

First, requests made inside a cross-site frame, which runs in a process of its own, never reach the renderer's network listener; that meets the plan's first stop condition, so the maintainer must rule how the first slice accounts for them before the next step starts. Second, starting the page from a blank document makes its history read two entries, and the two ways the spike found to bring it to one each cost something: a script redirect changes one request header from "none" to "cross-site", and clearing the history only takes effect after the page finishes loading. The image the spike saw fetched but missing from the saved page is filed separately as #1843.

MMatt Donovan10d ago

Note

Written by claude-opus-5-5 via Claude Code on behalf of Matt.

Maintainer ruling of 2026-10-17 on the first of this issue's two open decisions: how the capture engine's first slice accounts for requests made inside a frame from another site.

The question offered three options: (a) the engine also attaches to each such frame's process and records its requests, so the record of the page's network exchanges covers the whole page; (b) the record covers the main frame's process only and says how many cross-site frames it did not cover; (c) per-site processes are turned off for the hidden window. The maintainer chose (a), and asked that an agent first add a short design for it to the first-slice plan for approval, before step 1 starts.

That design is on pull request #1902. The second decision, how the slice handles the extra history entry a blank first page adds, stays open, so this issue stays open.

Where the question and the design are
  • The stop condition: docs/specs/2026-10-11-capture-engine-renderer-spike.md, "Rulings this asks for", item 1, and Question 4.
  • The question as put to the maintainer, with the three options and the recommendation of (a), was relayed through the capture engine programme's question list on 2026-10-17; the answer came back the same day.
  • The design: decision D17 in docs/plans/2026-10-10-capture-engine-first-slice.md on PR #1902, which waits for the maintainer's approval.