Sign up
View stargazersSign up

feat(cua): float the window the agent drives over chat#16531

draft1
t3code/cua/4-setup
t3code/cua/5-pip
opened23d agobyJJulius Marminge
19 files changed
View on GitHubTwo-way sync enabledPR last imported 1h ago

While a computer use turn runs, the web and desktop clients float the window the agent is driving over chat, like an agent-opened device. This is the live preview card from #10899.

  • The server captures the window the agent last targeted (the desktop before it names one) every 1.5 seconds and streams small frames over subscribeCuaWindowPreview.
  • The loop runs only while a client watches and a turn with Cua attached is in flight. It stops, and releases its native handle, as soon as either ends, so a closed card costs the host nothing.
  • Between turns the card keeps its last frame. It shows the app's icon and window title, and says why when the host cannot capture.
  • It follows Auto-show floating preview. A browser or device the agent already floats keeps its place.
  • It works for every provider and host OS, because the capture uses Cua's own get_window_state.

The capture now follows V2 provider turns (provider-turn.updated) instead of the V1 provider service.

  • While a run drives the computer, the composer shows Using your computer with Stop, in every provider (from #15006).

Verification

  • Focused tests: the capture loop starts and stops with subscribers and turns; the card's selector finds the newest computer use call in any provider; mini-player store and ChatView logic.
  • Checked by hand: the 0.34 SDK capture call against this Mac's driver. Without Screen Recording it fails with "could not create image from display", which is the reason the card shows.
  • Not yet verified in a running client. A recording will follow.

Built with Claude Opus 5.5 in T3 Code, through the Claude Code harness.

šŸ¤– Generated with Claude Code

Checks

22 successful Ā· 6 skipped
All checks passed
e3af01d
Ggithub-actionsBot23d ago
Synced from GitHub

Thread transfer impact

āœ… Thread transfer remains within every enforced ceiling.

ā„¹ļø No successful main baseline artifact is available yet. This run establishes the initial measurement.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wireā€”5.0 KiBā€”6.8 KiBāœ…
CodexThread snapshot wireā€”3.8 KiBā€”4.9 KiBāœ…
CodexLive turn WebSocket wireā€”1.3 KiBā€”2.0 KiBāœ…
CodexLive turn WebSocket decodedā€”21.6 KiBā€”24.4 KiBāœ…
CodexLive turn messagesā€”2ā€”8āœ…
ClaudeTotal thread wireā€”5.0 KiBā€”6.8 KiBāœ…
ClaudeThread snapshot wireā€”3.8 KiBā€”4.9 KiBāœ…
ClaudeLive turn WebSocket wireā€”1.3 KiBā€”2.0 KiBāœ…
ClaudeLive turn WebSocket decodedā€”21.9 KiBā€”24.4 KiBāœ…
ClaudeLive turn messagesā€”2ā€”8āœ…

Baseline: unavailable Ā· PR result: e3af01d Ā· Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

JJulius Marminge11d ago

Taking over this one for batch composer-chat-18. Preflight result: blocked; draft retained, no push or replacement.

This is based on t3code/cua/4-setup, not main. Current main lacks apps/server/src/cua, packages/contracts/src/cua.ts, provider-core/server/cuaToolPresentation, McpProviderSessions.cuaDriver, and the CUA driver dependency/session setup. Those are prerequisite services, not preview implementation details. Rebuilding them here would absorb work outside the five authorized PRs, and a mocked capture loop would not verify the shipped feature.

The lifecycle defect is confirmed in this head: reconcile deletes the thread preview when its final subscriber leaves, including its live flag. Reopening during the same running turn creates live: false; no new running event arrives, so capture stays stopped until another turn. Keep authoritative turn/session state independent of subscriber lifetime, while releasing the native capture handle as soon as no subscriber remains. A first subscription after a turn has already started must also derive current liveness.

To unblock:

  1. Land or explicitly provide the CUA setup/services against main. Keep this PR a preview layer rather than duplicating the foundation.
  2. Rewrite the lifecycle coverage using deterministic service/event barriers: start an attached CUA turn, attach two subscribers, detach both, reattach during that same turn, and require fresh frames without another running event. Also subscribe first after a turn has started, settle/Stop, and verify capture handles stop/release while the last frame remains visible between turns.
  3. On a host with a working CUA driver and screen-capture permissions, open/close/reopen the web card during a real computer-use turn. Verify targeted-window changes, unavailable-permission state, auto-show preference, existing browser/device preview precedence, and remote-client frame delivery. Record the flow; repeat supported OS capture paths before claiming all-host support.

This head has no running-client recording; the original SDK probe reached a Screen Recording permission failure. No new tests or adversarial implementation reviews were run because the prerequisite boundary prevents a responsible main-based rebuild. Nothing was merged.

Assessment: GPT-6.1-Sol, Codex harness in T3 Code.

Added 1 commit

  • feat(cua): float the window the agent drives over chate3af01d