fix(preview): server recordings capture the tab instead of re-encoding the JPEG screencast#26684

closed1
main
GitHub fork: fix/server-recording-tab-capture
opened7d agobyVVantrongs
View on GitHubTwo-way sync enabledPR last imported 6d ago

Problem

Agent recordings of server-browser tabs go through the JPEG screencast: Chromium compresses each frame, CDP sends it as base64, and an encoder page decodes it again before MediaRecorder encodes the canvas. At the 60 fps setting and a 1600×900 viewport (3200×1800 frames), main records 17–19 fps here and keeps 3.5–3.7 cores busy. #26683 has the measurements on three machines, including with PR #25075's faster delivery (45 fps on a 32-thread host, 10 fps on an i7-4790S).

Change

Tabs in a persistent profile are recorded from the frames Chrome composites, through chrome.tabCapture, the way the desktop app records its tabs:

  • ServerBrowserContexts writes a small recorder extension (MV3, tabCapture and debugger, the latter only for chrome.debugger.getTargets(), which maps the tab's CDP target to its tab ID) and loads it into persistent profiles. A fixed key gives it a fixed ID for --allowlisted-extension-id, without which getMediaStreamId fails. recorderPage() opens the extension's page beside the tab, or returns null where extensions do not run.
  • startRecording uses that page as the encoder. recordingEncoderScript gets capture(): the tab's video track goes through MediaStreamTrackProcessor, and each VideoFrame is drawn by the existing paint, cursor and click ring included, into the same canvas and MediaRecorder.
  • Tab capture records the window, into which Chrome scales an emulated viewport of another size, so the window's content area follows the recorded tab's viewport (Browser.setContentsSize) at the start and after every viewport change. Tab capture also keeps one frame size and letterboxes into it unless its size bounds exceed 1 px and differ in aspect ratio (SelectResolutionPolicyFromCandidates); with bounds of 3×2 to 7680×7680 the frames follow the window as screencast frames follow the viewport, and the encoder scales the cursor from the CSS width it is told.
  • Incognito and isolated tabs (off-the-record contexts, where extensions do not run) and desktop tabs keep the screencast path unchanged.

Tab capture needs full Chrome, which chrome-headless-shell is not: it rejects getDisplayMedia and does not load an unpacked extension. So PreviewBrowser installs Chrome for Testing's chrome 156.0.8078.70 instead of the shell of the same version:

  • The zips are about 200 MB instead of about 120 MB. HtmlRender launches the same executable with --headless=new, which full Chrome runs.
  • The macOS zips hold five relative links inside the app bundle. The unpacker now creates links whose target is relative and has no .. segment, and still rejects everything else.
  • After a successful install, the old tools/chrome-headless-shell directory is removed.
  • chrome links four more libraries directly than the shell (libcairo, libcups, libpango, and NSS's libsmime3), so t3 browser setup installs libcairo2, libcups2t64/libcups2 and libpango-1.0-0 as well. Its AppArmor profile is renamed to t3-chrome-for-testing for the new path; setup checks only that the profile file exists, so a host set up for the shell gets the new one on the next t3 browser setup.

Limits:

  • Tabs of one profile share a window. If two of them record at once with different viewports, the one that started first is scaled into the other's window, with its cursor off, until its own viewport changes again.
  • Recordings of incognito, isolated and desktop tabs are as slow as before; PR #25075 still applies to them. It changes the same functions, so one of the two will need a rebase; I can rebase this one onto it.
  • With PR #24959, a browser the host provides would have to be full Chrome for tab capture.

Scope and approval

#26683 reports the defect with measurements and has not been triaged yet. This is a substantial fix (it changes the browser T3 downloads), so it needs that triage; I am opening it next to the issue because a working change makes the tab-capture direction easier to judge.

Verification

  • Recording, end to end. A script (not committed) drove this branch's ServerBrowserContexts and recordingEncoderScript with the real Chrome for Testing 156 on Linux x64, recording the issue's reproduction page (text plus a box moving every frame) at a 1600×900 viewport and the 60 fps setting, with the agent cursor moving ten times a second, each run 4 s after 0.5 s of warm-up. "Before" was main's screencast delivery into the same encoder with chrome-headless-shell 156. Runs alternated on a busy host (load average 5–14), so the cores are indicative:

    FramesfpsCores (Chrome + Node)
    main (screencast)3200×180017.1, 17.5, 18.6, 19.23.5–3.7
    This branch (tab capture)3200×180052.8, 54.2, 55.4, 56.02.7–3.0
  • Geometry. A red dot at CSS (100, 100) sits at frame pixel (200, 200) in the captured frames, as in Page.captureScreenshot, at the first capture, at a second capture in the already fitted window, and when another tab opens mid-capture. With the agent cursor held on the dot, the recorded cursor stays on it before and after the viewport changes from 1600×900 to 800×800 mid-recording. Frames follow 800×800, 390×844, 1920×1080 and 2160×3840 (1600×1600, 780×1688, 3840×2160, 4320×7680); for 1–6 frames during each change they have an intermediate size, as screencast frames do.

  • The recorded tab stays visible with requestAnimationFrame at 61 per second while the recorder page is open, and the live view's screencast delivers 52 frames a second while it is captured, against 60–61 before and after.

  • Tests:

    • ServerBrowser.test.ts: a tab the recorder page can capture records through capture() with the frame-size bounds and the 60 fps setting, fits the window, starts no screencast, and refits the window and tells the encoder the new width after a viewer resize (this part fails without the call in the resize handler). A tab the recorder cannot see fails the recording and closes the recorder page.
    • ServerBrowserContexts.test.ts: only persistent profiles load the extension, recorderPage() returns null for an isolated context, and a page that fails to load is closed.
    • PreviewBrowser.test.ts: links inside the browser's directory are installed, links to a parent directory, an absolute path or nothing reject the archive, and the old shell directory is removed after an install.
    • vp run check:changed: format, lint, knip, typecheck and the changed files' tests pass. vp test run on src/preview, src/cli and src/htmlRender passes except ServerBrowserStream.test.ts (6) and cli/pair.test.ts (1). The six stream tests fail the same way on main here, under Node 24 and 26 (WebSocket close 1006), and replace ServerBrowser with a mock. pair.test.ts is a Tailscale pairing test that this change does not touch.
  • I checked the six zips' sizes and SHA-256, the macOS zips' links, and the Linux binary's NEEDED libraries against the Debian packages.

  • Not checked: installing and recording on macOS, Windows and arm64, and t3 browser setup on Debian or Ubuntu.

Model and harness: Claude Opus 5.5, Claude Code in T3 Code

Checks

20 successful · 17 skipped
All checks passed
611afa8

Added 1 commit

  • fix(preview): server recordings capture the tab instead of re-encoding the JPEG screencast611afa8
Synced from GitHub

Review in Change Stack →

No actionable comments were generated in the recent review.

External image Recent review info
External image Run configuration
External image Commits

Reviewing files that changed from the base of the PR and between 9826dbb9904b2591bfbf60d1317486b59132f56b and 611afa8b63bdf2be09407e8c78cb045fbc25a033.

External image Files selected for processing: 12
  • apps/server/src/cli/browser.ts
  • apps/server/src/preview/PreviewBrowser.test.ts
  • apps/server/src/preview/PreviewBrowser.ts
  • apps/server/src/preview/PreviewBrowserHost.test.ts
  • apps/server/src/preview/PreviewBrowserHost.ts
  • apps/server/src/preview/ServerBrowser.test.ts
  • apps/server/src/preview/ServerBrowser.ts
  • apps/server/src/preview/ServerBrowserContexts.test.ts
  • apps/server/src/preview/ServerBrowserContexts.ts
  • apps/server/src/preview/ServerBrowserPage.ts
  • docs/user/html-renders.md
  • docs/user/remote-access.md

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

The preview browser now installs Chrome for Testing. Persistent browser profiles can use a recorder extension to capture tab frames, while desktop tabs and unavailable tab capture use the CDP screencast path.

Changes: Browser preview and recording

Layer / File(s)Summary
Install Chrome for Testing
apps/server/src/preview/PreviewBrowser.ts, apps/server/src/preview/PreviewBrowser.test.ts, apps/server/src/cli/browser.ts, apps/server/src/preview/PreviewBrowserHost.ts, apps/server/src/preview/PreviewBrowserHost.test.ts, docs/user/html-renders.md, docs/user/remote-access.md
Downloads, archive extraction, executable discovery, and host configuration now target Chrome for Testing. Extraction permits safe relative symlinks and installation removes the former headless-shell directory. Tests and browser download-size documentation are updated.
Provision recorder extension
apps/server/src/preview/ServerBrowserContexts.ts, apps/server/src/preview/ServerBrowserContexts.test.ts, apps/server/src/preview/ServerBrowser.ts
Persistent-profile contexts write and load the recorder extension and provide a recorder page. The browser context configuration supplies the extension directory. Tests cover extension setup and recorder-page navigation.
Capture and encode tab frames
apps/server/src/preview/ServerBrowser.ts, apps/server/src/preview/ServerBrowserPage.ts, apps/server/src/preview/ServerBrowser.test.ts
Non-desktop tabs use recorder-page capture when available. The encoder processes captured frames and updates its viewport after resize. Screencast recordings retain their own session cleanup, and tests cover capture constraints, resizing, and capture failure.

Priority: External image Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

mermaid
1sequenceDiagram
2 participant ServerBrowser
3 participant ServerBrowserContexts
4 participant RecorderPage
5 participant TabCaptureAPI
6 participant MediaStreamTrackProcessor
7 participant Encoder
8 ServerBrowser->>ServerBrowserContexts: Request recorderPage for context
9 ServerBrowserContexts-->>ServerBrowser: Return recorder page
10 ServerBrowser->>RecorderPage: Start capture for target tab
11 RecorderPage->>TabCaptureAPI: Request stream for target tab
12 TabCaptureAPI-->>RecorderPage: Return video stream
13 RecorderPage->>MediaStreamTrackProcessor: Read video frames
14 MediaStreamTrackProcessor-->>RecorderPage: Provide frames
15 RecorderPage->>Encoder: Display frames and record

Suggested labels: bugfix, size:L

Suggested reviewers: juliusmarminge, maria-rcks, t3dotgg

Merge Risk: External image Minimal · up to 611af

No actionable merge-blocking issue was established; the change is ready for normal checks.

Pre-merge checks: Passed 3 · Failed 1

Failed Failed checks: 1 warning

Check nameStatusExplanationResolution
Description checkWarningThe description explains the problem, implementation, limitations, and verification in detail. However, the Scope and approval section states that issue #26683 is not triaged and does not provide the …Add a link to the triaged issue or approval discussion, including the maintainer's explicit approval comment for the direction and scope. Update the description after triage or document the approved exception.
Passed Passed checks: 3 passed
Check nameStatusExplanation
Title checkPassedThe title clearly identifies the main change: server recordings capture tabs instead of re-encoding JPEG screencasts.
Linked Issues checkPassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes checkPassedCheck skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description explains the problem, implementation, limitations, and verification in detail. However, the Scope and approval section states that issue #26683 is not triaged and does not provide the required triage or explicit maintainer approval for this substantial browser-download and recording behavior change.

Resolution

Add a link to the triaged issue or approval discussion, including the maintainer's explicit approval comment for the direction and scope. Update the description after triage or document the approved exception.

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Synced from GitHub

closing at @StiensWout's request under Establish the problem and scope first.

A substantial recording-backend and browser-distribution repair is proposed before the linked performance failure and intended direction have received maintainer triage.

Have maintainers triage issue #26683 and approve the intended tab-capture/browser-distribution direction and necessary platform/recording scope before reconsideration. Preserve the demonstrated server-recording frame-rate and CPU problem.

thanks for trying to help improve t3 code. if you think this closure was a mistake, please open a fresh pr addressing the requirement above, and link back to this one.