Sign up
View stargazersSign up

feat(cua): host Cua Driver on macOS, Linux, and Windows#16527

draft1
main
t3code/cua/1-host
opened23d agobyJJulius Marminge
43 files changed
View on GitHubTwo-way sync enabledPR last imported 7h ago

Supersedes #15006 and #10899. This stack (#16527 → #16528 → #16529 → #16530 → #16531 → #16532) ships computer use and agent browser tabs on macOS, Linux, and Windows together.

Computer use needs Cua Driver running on the environment's host. #15006 asked users to install it themselves and only worked on macOS; Cua's own #10899 bundled it but only wired macOS end to end.

This layer hosts the pinned Cua Driver 0.38.0 on macOS, Linux, and Windows. The @trycua/cua-driver SDK stays on 0.37.0, the newest on npm (0.38.0's npm publish failed upstream); the SDK only checks that the driver's contract, tool-schema, capability, and MCP protocol versions match, and those are identical in 0.37.0 and 0.38.0.

  • Desktop builds bundle it for every target: one universal macOS executable, and per-arch Linux and Windows bundles. Electron starts it as an embedded host, so macOS grants Accessibility and Screen Recording to T3 Code, not to a second app. It hands the MCP descriptor to its primary backend over the existing private control pipe.
  • A standalone server (npx t3) downloads the same release into T3's home on first use, checksum- and size-verified, with path-traversal checks on the archive. T3CODE_CUA_DRIVER_PATH still overrides it.
  • The release table (asset names, sizes, SHA-256) lives in @t3tools/shared/cuaDriverRelease, so the build script and the server cannot drift. @trycua/cua-driver moves to the pnpm catalog, so desktop and server share one version.

Nothing attaches the driver to agents yet; that is the next layer. The setting stays off by default.

Ported from #10899 by Francesco Bonacci, with Julius's Effect-convention fixes from #11709, rebased onto the V2 orchestrator.

  • The SDK loads in place from the app archive. A small patch to @ubjs/node (the SDK's native loader) points dlopen at the .asar.unpacked sibling, the same fix fff already carries. Only the SDK's platform packages are unpacked, so Windows stays under its 80-file installer budget.

Verification

  • Focused tests: CuaDriver (start, revoke, crash, bounded shutdown), CuaDriverInstallation (verified install, checksum rejection, unsafe archive), DesktopCuaDriver, desktop telemetry and backend tests, and build-script staging for all five archive layouts. Server, desktop, contracts, and shared typecheck clean.
  • Checked by hand on macOS arm64 with 0.34.0: Cua's EmbeddedCuaDriverHost starts, returns its MCP descriptor and socket, and listApps answers through it. 0.38.0 has not been rechecked by hand.
  • The archive checksums and sizes match Cua's published checksums.txt and assets for 0.38.0.
  • skips the primary native probe for cross-architecture Windows payloads in build-desktop-artifact.test.ts fails on main too, so this PR did not cause it.

Built with Claude Opus 5.5 in T3 Code, through the Claude Code harness.

🤖 Generated with Claude Code

Checks

22 successful · 6 skipped
All checks passed
1df82d3
Ggithub-actionsBot23d ago
Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire5.0 KiB5.0 KiB+6 B (+0.1%)6.8 KiB✅
CodexThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
CodexLive turn WebSocket wire1.2 KiB1.3 KiB+6 B (+0.5%)2.0 KiB✅
CodexLive turn WebSocket decoded21.6 KiB21.6 KiB0 B (0.0%)24.4 KiB✅
CodexLive turn messages220 (0.0%)8✅
ClaudeTotal thread wire5.0 KiB5.0 KiB0 B (0.0%)6.8 KiB✅
ClaudeThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
ClaudeLive turn WebSocket wire1.3 KiB1.3 KiB0 B (0.0%)2.0 KiB✅
ClaudeLive turn WebSocket decoded21.9 KiB21.9 KiB0 B (0.0%)24.4 KiB✅
ClaudeLive turn messages220 (0.0%)8✅

Baseline: 9826dbb · PR result: 1df82d3 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

JJulius Marminge10d ago

Taking over the host-layer assessment for maintainer batch desktop-app-12. I will check whether it can be rebuilt from current main as focused distribution, host lifecycle, and desktop integration layers without expanding into the later agent-attachment PRs. The existing draft and native qualification gaps stay explicit; no merge is planned.

JJulius Marminge10d ago

Thanks @juliusmarminge for establishing the host lifecycle, native SDK/permission constraints, pinned release facts, and packaging work in this draft.

I rebuilt this against current main as a smaller review sequence, with fresh test and implementation agents and independent Opus/Sol reviews:

  • #21694: pinned SDK/release manifest and verified standalone installation.
  • #21695: opt-in server lifecycle, settings, and private desktop handoff.
  • #21696: Electron-owned hosting, permission restart, and primary-only private report routing.
  • #21697: verified release staging, native ASAR loading, and minimal runtime resources.

I compared the entire original diff against the stack. All host-only capabilities are accounted for. The knip allowlist changes were unnecessary with the rebuilt dependency graph. Unreachable System Settings permission branches were removed because the current contract only accepts full-disk-access; the real SDK permission prompt and restart IPC remain. Unused SDK libraries were removed from desktop binary resources and the Windows server sidecar. Standalone SDK installation remains supported. The original 39 added behavior cases became 32 focused cases, with real compressed-archive and sidecar packaging coverage.

Three full-stack review rounds are complete; the final round is clean. The rebuilt Linux download/install/acquire/MCP path, cleanup, single-executable loading, full CLI executable build, and actual headless Electron ASAR native loading have passed. The PR bodies distinguish these proofs from full packaged GUI qualification.

Signed macOS TCC/cursor behavior, installed Windows UIA/NSIS, and Linux GUI capture/input and GNOME extension activation still need human qualification. Packaging adds resources before there is a client entry point, so its body recommends landing it with the attachment feature. Later layers of the original series are outside this takeover and were not modified.

Closing this draft in favor of the linked replacement stack. Nothing was merged.

Implementation: Codex gpt-6.1-sol; tests: Codex gpt-6-astra; reviews: Claude claude-opus-5-5 and Codex gpt-6-sol. Harness: T3 Code delegated tasks.

Added 1 commit

  • feat(cua): host Cua Driver on macOS, Linux, and Windows1df82d3