Sign up
View stargazersSign up

feat(cua): show Cua Driver calls as computer use in every provider#16529

draft1
t3code/cua/2-providers
t3code/cua/3-presentation
opened23d agobyJJulius Marminge
18 files changed
View on GitHubTwo-way sync enabledPR last imported 55m ago

Cua tool calls showed up as raw mcp__cua-driver__click rows. They now read like Codex's own computer use in every harness: the target app's name in a plain title ("Clicked in Safari"), its icon, and a computer source.

  • The shared MCP presentation recognizes the cua-driver server, so Claude, Codex, Cursor, the ACP family, OpenCode 1.x, OpenCode 2, and Pi all reach it through one helper.
  • Completed calls teach the thread which app owns each pid, so later calls that only name a pid or window still name the app.
  • The title table covers every tool Cua 0.38 exposes on any platform, pinned by a test. A run_steps batch is attributed to the window its last step names.
  • App icons resolve on every host: Spotlight and sips on macOS; launcher entries and the hicolor theme on Linux (from #10899); and on Windows, the executable's associated icon, found through running processes or the Start Menu, rendered to a cached PNG with PowerShell.

Verification

  • Focused tests: Cua presentation across every provider's tool-name shape, title table against the 0.38 catalogue, Linux icon lookup, and the adapter and provider suites (2001 tests).
  • Not verified: the Windows icon path on a real Windows host.

Built with Claude Opus 5.5 in T3 Code, through the Claude Code harness.

🤖 Generated with Claude Code

Checks

23 successful · 6 skipped
All checks passed
06d2d39
Ggithub-actionsBot23d ago
Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire—5.0 KiB—6.8 KiB✅
CodexThread snapshot wire—3.8 KiB—4.9 KiB✅
CodexLive turn WebSocket wire—1.2 KiB—2.0 KiB✅
CodexLive turn WebSocket decoded—21.5 KiB—24.4 KiB✅
CodexLive turn messages—1—8✅
ClaudeTotal thread wire—5.1 KiB—6.8 KiB✅
ClaudeThread snapshot wire—3.8 KiB—4.9 KiB✅
ClaudeLive turn WebSocket wire—1.3 KiB—2.0 KiB✅
ClaudeLive turn WebSocket decoded—21.9 KiB—24.4 KiB✅
ClaudeLive turn messages—2—8✅

Baseline: unavailable · PR result: 06d2d39 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

JJulius Marminge11d ago

Batch other-providers-22: blocked for native-platform verification; no takeover code pushed.

The current diff adds Windows executable discovery and PowerShell/System.Drawing icon extraction, plus Cua tool presentation across provider adapters. This headless Linux VM has no Windows/macOS host or Cua Driver installation, so I cannot validate the actual app identity → icon → asset route → client path on those platforms. The existing body also explicitly leaves Windows unverified.

A maintainer with a Windows host must run Cua Driver calls through a current provider package, verify running-process and Start Menu icon lookup (including a missing icon), and capture the resulting computer-use row in a real client. Repeat the macOS icon regression path, then provide matching before/after client screenshots. The adapter integration must be checked against current main, including the extracted provider packages. This draft remains open; no unverified platform changes were pushed or represented as ready.

Added 1 commit

  • feat(cua): show Cua Driver calls as computer use in every provider06d2d39