fix(server): reduce dropped frames in agent preview recordings#25075

open4

main

fix/issue-24979

View on GitHubTwo-way sync enabledPR last imported 6d ago

Problem

Agent preview recordings lose moving frames at large viewports because screencast acknowledgements wait for image decoding. The reporter’s frame-numbered animation/scroll fixture at 1600×900 CSS and DPR 2 reproduced the loss on main: three recordings contained 35, 31, and 26 encoded frames.

Change

Acknowledge screencast frames immediately and decode one frame at a time, retaining only the newest pending frame. Drain delivery before stopping the encoder. Replace the per-byte JavaScript base64 conversion with Chromium’s native decoder.

Scope and approval

Fixes https://github.com/pingdotgg/t3code/issues/24979

Triaged bug via the automated issue flow. GitCafe: #24979.

Verification

Final merged-head verification: vp run check:changed passed format, lint, both knip checks, GraphQL generation, typecheck of two packages, and 46 tests in two files.
Red-first acknowledgement test fails on main with zero acknowledgements instead of three while decode is blocked. The focused suite has 30 passing tests, including newest-pending-frame delivery before stop; removing the drain makes that test fail.
Three real-app recording pairs improved encoded frames 35/31/26 → 78/85/83, with distinct active barcode IDs 32/28/24 → 74/81/80. Active-phase frame survival rose from about 31–46% to 75–86%. Recorded page paints separately changed 77/87/54 → 95/105/95, versus approximately 360 unrecorded. The third baseline run overlapped machine load; all raw runs are retained.
No full 30/60 fps or sharpness improvement is established. JPEG quality, bitrate policy, output sizing and frame-rate settings are unchanged. Initial seed pairs are pixel-identical; dynamic frames have compression differences and different update cadence.
Linux Chromium 156, DPR 2, configured 30 fps, plain Node 26.12.1; isolated dev app and unchanged reporter fixture. Native recording-stop claimed every artifact. No customer data is used.
Unverified: macOS Electron/CDP reporter surface, actual 60 fps Electron UI setting, exact cursor-ring timing, final-frame survival in MediaRecorder, remote artifact transfer and attachment-player playback. L4 resize and L6 lifecycle coverage are partial; L6 ran at 30 fps. The real web client disables the Electron-only rate selector.

Both independent reviews found no confirmed code blocker. The proof audit independently rebuilt the idle capture with zero changed pixels out of 1,920,000. Full comparison index, raw metrics, scripts and action timestamps.

The encoder microbenchmark produced identical bytes and reduced base64 conversion from 16.5 ms to 0.7 ms: driver, malformed-input diagnostic, results.

Evidence below was captured before merging current main. The merge did not change any of the three preview source/test files. Before is 5f1f604733; after is ab79eb7f54. Diff0 uses absolute RGB delta at zero threshold; DiffT uses max-channel delta >8, amplified 8×. These stills show matched initial state; the linked videos prove timing.

L1-cycle-1 — 1600×900, DPR 2, white fixture

BEFORE UTC 2026-10-25T22:08:38.986Z → 2026-10-25T22:08:48.991Z; AFTER UTC 2026-10-25T22:24:52.916Z → 2026-10-25T22:25:02.920Z.

Reset; unrecorded10s control then reset/start; 2s idle,3s animation,3s scroll,2s idle; native stop.

AFTER measurement UTC 2026-10-25T22:24:52.916Z → 2026-10-25T22:25:02.920Z; page paints 95. Exact tool chronology: actions.json.

BEFORE recording | AFTER recording

AFTER: 78 encoded frames, 10.758133s, 3200×1800, 2425227bit/s, 3262926bytes; strict PTS. temporal sheet · native text pair

Initial settled seed (before start):

Before	After	Diff0	DiffT
L1-cycle-1-seed.png 5f1f604733	L1-cycle-1-seed.png ab79eb7f54	L1-cycle-1-diff0.png	L1-cycle-1-diffT.png

L1-cycle-2 — 1600×900, DPR 2, white fixture

BEFORE UTC 2026-10-25T22:09:03.692Z → 2026-10-25T22:09:13.706Z; AFTER UTC 2026-10-25T22:25:17.677Z → 2026-10-25T22:25:27.686Z.

Reset; unrecorded10s control then reset/start; 2s idle,3s animation,3s scroll,2s idle; native stop.

AFTER measurement UTC 2026-10-25T22:25:17.677Z → 2026-10-25T22:25:27.686Z; page paints 105. Exact tool chronology: actions.json.

BEFORE recording | AFTER recording

AFTER: 85 encoded frames, 10.524967s, 3200×1800, 2826971bit/s, 3720840bytes; strict PTS. temporal sheet · native text pair

Initial settled seed (before start):

Before	After	Diff0	DiffT
L1-cycle-2-seed.png 5f1f604733	L1-cycle-2-seed.png ab79eb7f54	L1-cycle-2-diff0.png	L1-cycle-2-diffT.png

L1-cycle-3 — 1600×900, DPR 2, white fixture

BEFORE UTC 2026-10-25T22:09:41.357Z → 2026-10-25T22:09:51.372Z; AFTER UTC 2026-10-25T22:25:48.271Z → 2026-10-25T22:25:58.284Z.

Reset; unrecorded10s control then reset/start; 2s idle,3s animation,3s scroll,2s idle; native stop.

AFTER measurement UTC 2026-10-25T22:25:48.271Z → 2026-10-25T22:25:58.284Z; page paints 95. Exact tool chronology: actions.json.

BEFORE recording | AFTER recording

AFTER: 83 encoded frames, 9.106967s, 3200×1800, 2919419bit/s, 3324983bytes; strict PTS. temporal sheet · native text pair

Initial settled seed (before start):

Before	After	Diff0	DiffT
L1-cycle-3-seed.png 5f1f604733	L1-cycle-3-seed.png ab79eb7f54	L1-cycle-3-diff0.png	L1-cycle-3-diffT.png

L3-idle-first — 800×600, DPR 2, white fixture

BEFORE UTC 2026-10-25T22:10:07.441Z → 2026-10-25T22:10:07.995Z; AFTER UTC 2026-10-25T22:26:00.976Z → 2026-10-25T22:26:01.134Z.

Reset800×600 idle; start/same-active-start, stop; restart new session then stop. Idle event-driven recording has short encoded duration despite wall-clock wait.

BEFORE recording | AFTER recording

AFTER: 4 encoded frames, 0.323333s, 1600×1200, 1522342bit/s, 62497bytes; strict PTS. temporal sheet · native text pair

Initial settled seed (before start):

Before	After	Diff0	DiffT
L3-idle-first-seed.png 5f1f604733	L3-idle-first-seed.png ab79eb7f54	L3-idle-first-diff0.png	L3-idle-first-diffT.png

L3-idle-restart — 800×600, DPR 2, white fixture

BEFORE UTC 2026-10-25T22:10:09.641Z → 2026-10-25T22:10:09.923Z; AFTER UTC 2026-10-25T22:26:01.479Z → 2026-10-25T22:26:01.636Z.

Reset800×600 idle; start/same-active-start, stop; restart new session then stop. Idle event-driven recording has short encoded duration despite wall-clock wait.

BEFORE recording | AFTER recording

AFTER: 2 encoded frames, 0.265900s, 1600×1200, 1838014bit/s, 62044bytes; strict PTS. temporal sheet · native text pair

Initial settled seed (before start):

Before	After	Diff0	DiffT
L3-idle-restart-seed.png 5f1f604733	L3-idle-restart-seed.png ab79eb7f54	L3-idle-restart-diff0.png	L3-idle-restart-diffT.png

L4-interaction-resize — 1280×800 → 800×600, DPR 2, white fixture

BEFORE UTC 2026-10-25T22:11:37.718Z → 2026-10-25T22:11:47.736Z; AFTER UTC 2026-10-25T22:26:10.891Z → 2026-10-25T22:26:20.905Z.

Reset1280×800; native start; hover/click/drag, scrolling, key press and resize to800×600; continue10s fixture cycle, HUD click then stop.

AFTER measurement UTC 2026-10-25T22:26:10.891Z → 2026-10-25T22:26:20.905Z; page paints 167. Exact tool chronology: actions.json.

BEFORE recording | AFTER recording

AFTER: 222 encoded frames, 13.300300s, 2560×1600, 1893649bit/s, 3151108bytes; strict PTS. temporal sheet · native text pair

Initial settled seed (before start):

Before	After	Diff0	DiffT
L4-interaction-resize-seed.png 5f1f604733	L4-interaction-resize-seed.png ab79eb7f54	L4-interaction-resize-diff0.png	L4-interaction-resize-diffT.png

L5-one-viewer-snapshot — 1600×900, DPR 2, white fixture

BEFORE UTC 2026-10-25T22:19:20.779Z → 2026-10-25T22:19:30.816Z; AFTER UTC 2026-10-25T22:29:01.707Z → 2026-10-25T22:29:11.752Z.

Reset1600×900; one viewer; native start and GO10s fixture cycle; one screenshot mid-motion; motion resumes, stop.

AFTER measurement UTC 2026-10-25T22:29:01.707Z → 2026-10-25T22:29:11.752Z; page paints 81. Exact tool chronology: actions.json.

BEFORE recording | AFTER recording

AFTER: 91 encoded frames, 10.929500s, 3200×1800, 2467458bit/s, 3372676bytes; strict PTS. temporal sheet · native text pair

Initial settled seed (before start):

Before	After	Diff0	DiffT
L5-one-viewer-snapshot-seed.png 5f1f604733	L5-one-viewer-snapshot-seed.png ab79eb7f54	L5-one-viewer-snapshot-diff0.png	L5-one-viewer-snapshot-diffT.png

L6-fresh-short-session — 1600×900, DPR 2, white fixture

BEFORE UTC 2026-10-25T22:16:25.232Z → 2026-10-25T22:16:28.666Z; AFTER UTC 2026-10-25T22:26:42.048Z → 2026-10-25T22:26:45.426Z.

Reset1600×900; start fixture, stop while moving; new recording session and repeat; separate close during recording discards artifact.

BEFORE recording | AFTER recording

AFTER: 15 encoded frames, 3.684167s, 3200×1800, 644496bit/s, 297862bytes; strict PTS. temporal sheet · native text pair

Initial settled seed (before start):

Before	After	Diff0	DiffT
L6-fresh-short-session-seed.png 5f1f604733	L6-fresh-short-session-seed.png ab79eb7f54	L6-fresh-short-session-diff0.png	L6-fresh-short-session-diffT.png

L6-stop-during-motion — 1600×900, DPR 2, white fixture

BEFORE UTC 2026-10-25T22:16:20.651Z → 2026-10-25T22:16:24.177Z; AFTER UTC 2026-10-25T22:26:37.703Z → 2026-10-25T22:26:41.187Z.

Reset1600×900; start fixture, stop while moving; new recording session and repeat; separate close during recording discards artifact.

BEFORE recording | AFTER recording

AFTER: 18 encoded frames, 3.855733s, 3200×1800, 628678bit/s, 304084bytes; strict PTS. temporal sheet · native text pair

Initial settled seed (before start):

Before	After	Diff0	DiffT
L6-stop-during-motion-seed.png 5f1f604733	L6-stop-during-motion-seed.png ab79eb7f54	L6-stop-during-motion-diff0.png	L6-stop-during-motion-diffT.png

Models: claude-opus-5-5 and gpt-6.1-sol. Harnesses: Claude Agent and Codex through T3 Code.

Checks

22 successful · 10 skipped

All checks passed

Added 4 commits10d ago

test(server): recordings keep acking screencast frames while the encoder decodes1bea84b
Merge main into fix/issue-249793ba653f
perf(server): recordings decode screencast frames with Uint8Array.fromBase64ab79eb7
fix(server): recordings ack screencast frames without waiting on the encoderd9e6fe9

MmacroscopeappBotapproved these changes10d ago

MmacroscopeappBot10d ago

Approvability

Verdict: Approved at 3ba653f

Macroscope's review found this PR approvable — This is a localized preview-recording bug fix that improves screencast backpressure handling, retains the newest pending frame, and drains delivery during shutdown. It includes targeted tests and a contained decoder optimization without changing APIs, defaults, deployment behavior, or sensitive code.

^{You can add or adjust custom eligibility rules. Learn more.}

Ggithub-actionsBot10d ago

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	5.0 KiB	5.0 KiB	+7 B (+0.1%)	6.8 KiB	✅
Codex	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Codex	Live turn WebSocket wire	1.2 KiB	1.2 KiB	+7 B (+0.6%)	2.0 KiB	✅
Codex	Live turn WebSocket decoded	20.9 KiB	20.9 KiB	0 B (0.0%)	29.3 KiB	✅
Codex	Live turn messages	2	2	0 (0.0%)	8	✅
Claude	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Claude	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Claude	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Claude	Live turn WebSocket decoded	21.2 KiB	21.2 KiB	0 B (0.0%)	29.3 KiB	✅
Claude	Live turn messages	2	2	0 (0.0%)	8	✅

Baseline: d5b921c · PR result: 3ba653f · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

Codex decoded thread snapshot: 108.5 KiB
Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

CcoderabbitaiBot10d ago

Review in Change Stack →

No actionable comments were generated in the recent review.

External image Recent review info

External image Run configuration

Source mode: Auto
Repository: pingdotgg/t3code
Branch: fix/issue-24979
File: .coderabbit.config.ts
Commit: 3ba653f
Layers: Repository TS config
Review profile: CHILL
Plan: Team
Run ID: 1cf9ca85-6cb1-4934-9b17-0c0b59499f3f

External image Commits

Reviewing files that changed from the base of the PR and between d5b921ce4bf0d687e5cedbf956839dd5ebabeaa2 and 3ba653fc4e586b11c096892c156f013d81526791.

External image Files selected for processing: 3

apps/server/src/preview/ServerBrowser.test.ts
apps/server/src/preview/ServerBrowser.ts
apps/server/src/preview/ServerBrowserPage.ts

Limit details: You’ve used all 10 included reviews currently available.

Walkthrough

Recording now acknowledges screencast frames before encoding them and keeps only the newest waiting frame while encoding. Stopping waits for frame delivery to finish. The frame decoder also uses Uint8Array.fromBase64.

Changes: Recording frame delivery

Layer / File(s)	Summary
Asynchronous frame delivery and shutdown `apps/server/src/preview/ServerBrowser.ts`, `apps/server/src/preview/ServerBrowserPage.ts`, `apps/server/src/preview/ServerBrowser.test.ts`	The recording path acknowledges frames before encoding and replaces pending frames with the newest arrival. Stop waits for delivery to drain. Tests cover acknowledgement, superseded frames, and stopping during decoding. Frame base64 decoding now uses `Uint8Array.fromBase64`.

Priority: External image Normal

Estimated code review effort: 3 (Moderate) | ~18 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

mermaid

1	sequenceDiagram
2	participant ScreencastSession
3	participant RecordingDeliveryLoop
4	participant FrameEncoder
5	participant RecordingStop
6	ScreencastSession->>RecordingDeliveryLoop: deliver frame and acknowledge session
7	RecordingDeliveryLoop->>FrameEncoder: encode current frame
8	RecordingDeliveryLoop->>RecordingDeliveryLoop: replace pending frame with newest arrival
9	RecordingStop->>RecordingDeliveryLoop: wait for delivery to drain

Suggested reviewers: maria-rcks, ethereal49

Merge Risk: External image Minimal · up to 3ba65

The recording changes are ready to merge after normal checks; the investigated decoder and final-frame concerns do not block recording.

Pre-merge checks: ^Passed 4

Check name	Status	Explanation
Title check	Passed	The title is concise, uses conventional commit style, and clearly identifies the main change: reducing dropped frames in agent preview recordings.
Description check	Passed	The description includes all required sections. It explains the problem, implementation, scope approval, focused verification, observed results, limitations, and agent details.
Linked Issues check	Passed	Issue `#24979` requires a recording-path fix for viewport-dependent dropped frames. `ServerBrowser.ts` now acknowledges each screencast frame before delivery, retains only the newest pending frame, and …
Out of Scope Changes check	Passed	The changed production code is limited to screencast acknowledgement, frame delivery and shutdown ordering, and JPEG base64 decoding. The test changes support these behaviors and mock the required rec…

Full details: Linked Issues check

Explanation

Issue #24979 requires a recording-path fix for viewport-dependent dropped frames. ServerBrowser.ts now acknowledges each screencast frame before delivery, retains only the newest pending frame, and drains delivery before encoder stop. ServerBrowserPage.ts uses Uint8Array.fromBase64 for native decoding. ServerBrowser.test.ts covers immediate acknowledgements, newest-frame retention, and stop-time draining. The reported sharpness and full frame-rate improvements are observations, not separate acceptance criteria; the PR correctly does not claim to establish them.

Full details: Out of Scope Changes check

Explanation

The changed production code is limited to screencast acknowledgement, frame delivery and shutdown ordering, and JPEG base64 decoding. The test changes support these behaviors and mock the required recording lifecycle. These changes directly support issue #24979. No unrelated production behavior is changed.

Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Usage-based review receipt

Mode: Continue automatically
Reviewed files: 3
Charged: $0.75
View usage details

Note

This review was completed with usage-based billing: files reviewed beyond your plan's included limits are billed at $0.25/file. View usage-based billing.

_{Comment @coderabbitai help to get the list of available commands.}