Sign up
View stargazersSign up

feat(cua): float the window the agent drives over chat#16531

draft1
t3code/cua/4-setup
t3code/cua/5-pip
opened27d agobyJJulius Marminge
19 files changed
View on GitHubTwo-way sync enabledPR last imported 5h ago

While a computer use turn runs, the web and desktop clients float the window the agent is driving over chat, like an agent-opened device. This is the live preview card from #10899.

  • The server captures the window the agent last targeted (the desktop before it names one) every 1.5 seconds and streams small frames over subscribeCuaWindowPreview.
  • The loop runs only while a client watches and a turn with Cua attached is in flight. It stops, and releases its native handle, as soon as either ends, so a closed card costs the host nothing.
  • Between turns the card keeps its last frame. It shows the app's icon and window title, and says why when the host cannot capture.
  • It follows Auto-show floating preview. A browser or device the agent already floats keeps its place.
  • It works for every provider and host OS, because the capture uses Cua's own get_window_state.

The capture now follows V2 provider turns (provider-turn.updated) instead of the V1 provider service.

  • While a run drives the computer, the composer shows Using your computer with Stop, in every provider (from #15006).

Verification

  • Focused tests: the capture loop starts and stops with subscribers and turns; the card's selector finds the newest computer use call in any provider; mini-player store and ChatView logic.
  • Checked by hand: the 0.34 SDK capture call against this Mac's driver. Without Screen Recording it fails with "could not create image from display", which is the reason the card shows.
  • Not yet verified in a running client. A recording will follow.

Built with Claude Opus 5.5 in T3 Code, through the Claude Code harness.

🤖 Generated with Claude Code

Checks

22 successful · 6 skipped
All checks passed
40c264a
Ggithub-actionsBot27d ago
Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire—5.0 KiB—6.8 KiB✅
CodexThread snapshot wire—3.8 KiB—4.9 KiB✅
CodexLive turn WebSocket wire—1.2 KiB—2.0 KiB✅
CodexLive turn WebSocket decoded—21.6 KiB—24.4 KiB✅
CodexLive turn messages—2—8✅
ClaudeTotal thread wire—5.0 KiB—6.8 KiB✅
ClaudeThread snapshot wire—3.8 KiB—4.9 KiB✅
ClaudeLive turn WebSocket wire—1.3 KiB—2.0 KiB✅
ClaudeLive turn WebSocket decoded—21.9 KiB—24.4 KiB✅
ClaudeLive turn messages—2—8✅

Baseline: unavailable · PR result: 40c264a · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

JJulius Marminge14d ago

Taking over this one for batch composer-chat-18. Preflight result: blocked; draft retained, no push or replacement.

This is based on t3code/cua/4-setup, not main. Current main lacks apps/server/src/cua, packages/contracts/src/cua.ts, provider-core/server/cuaToolPresentation, McpProviderSessions.cuaDriver, and the CUA driver dependency/session setup. Those are prerequisite services, not preview implementation details. Rebuilding them here would absorb work outside the five authorized PRs, and a mocked capture loop would not verify the shipped feature.

The lifecycle defect is confirmed in this head: reconcile deletes the thread preview when its final subscriber leaves, including its live flag. Reopening during the same running turn creates live: false; no new running event arrives, so capture stays stopped until another turn. Keep authoritative turn/session state independent of subscriber lifetime, while releasing the native capture handle as soon as no subscriber remains. A first subscription after a turn has already started must also derive current liveness.

To unblock:

  1. Land or explicitly provide the CUA setup/services against main. Keep this PR a preview layer rather than duplicating the foundation.
  2. Rewrite the lifecycle coverage using deterministic service/event barriers: start an attached CUA turn, attach two subscribers, detach both, reattach during that same turn, and require fresh frames without another running event. Also subscribe first after a turn has started, settle/Stop, and verify capture handles stop/release while the last frame remains visible between turns.
  3. On a host with a working CUA driver and screen-capture permissions, open/close/reopen the web card during a real computer-use turn. Verify targeted-window changes, unavailable-permission state, auto-show preference, existing browser/device preview precedence, and remote-client frame delivery. Record the flow; repeat supported OS capture paths before claiming all-host support.

This head has no running-client recording; the original SDK probe reached a Screen Recording permission failure. No new tests or adversarial implementation reviews were run because the prerequisite boundary prevents a responsible main-based rebuild. Nothing was merged.

Assessment: GPT-6.1-Sol, Codex harness in T3 Code.

Added 1 commit

  • feat(cua): float the window the agent drives over chat40c264a