fix(cua): stabilize native window previews and Windows app identity#16975

Computer-use PiP could show unrelated desktop pixels after a failed window capture, flash while replacing images, and remain open after a turn ended. Capture now stays bound to the selected window, retries transient failures, decodes replacement frames before painting them to a persistent canvas, and dismisses the computer preview when the turn settles. Windows packaged apps now retain their own names and native icons instead of appearing as ApplicationFrameHost.exe.

The subscriber-driven loop caps active capture at 4 FPS, leaves idle time after slow captures, suppresses identical frames, and drops static-window polling to 1 FPS after three unchanged captures. A changed frame restores the normal rate; renewed motion can take up to one second to detect. Drivers advertising display_only use an observation-only capture that skips action-registry PNG decoding and snapshot replacement. Older drivers keep the typed screenshot path. No SDK ABI change is required.

Stacked on #16532.

Windows integration

Window identities are keyed by PID and window handle, so Calculator and Settings remain distinct when Windows hosts them in one process. PiP captions and icons travel with the captured frame, display-only inspections do not redirect the preview, and native icons resolve packaged-app AUMIDs through their manifests. Duplicate detection includes app identity as well as image bytes and window metadata.

Cua Driver 0.35.4 and later start the embedded daemon without a console window on Windows (trycua/cua#5136), so the temporary source patch and the T3CODE_CUA_DRIVER_SDK_LIBRARY SDK override this PR used to carry are gone.

Windows test build 9007 was exercised interactively: Calculator operations stayed in the background, its name/icon matched the preview, closing it showed an unavailable state, and Theo confirmed no flashing. After integration with the newer stack head: 133 focused tests pass, one is skipped, and two packaging tests fail identically on the parent commit (Linux cargo executable lookup on Windows and the cross-architecture native probe). Server/web typechecks and targeted lint pass. The combined stack has not received another packaged interactive pass.

Windows integration by GPT-6.1-Sol through the Codex harness.

Upstream Linux fixes

The native Linux fixes this PR used to carry as a source patch shipped upstream in Cua Driver 0.34.1 (trycua/cua#4895): the Wayland opt-in reaches embedded hosts, GNOME windows are captured from their own actor with compositor downscaling (WinRects API 10), the GNOME helper reuses one session-bus connection, GNOME's changing PNG Creation Time chunk is stripped, and get_window_state accepts display_only on Linux, macOS, and Windows with the same schema, refusal, and response fields.

The stack pins Cua Driver 0.38.0 from #16527, and this PR packages Linux like macOS and Windows: the published driver archive is bundled, and the SDK's native library comes from npm. The source patch, scripts/build-cua-linux.sh, SDK library replacement, the Linux CLI's bundled-driver lookup, and the extra Rust toolchain and X11/Wayland dev packages in CI are removed.

One piece of the old patch is not upstream: it hid GNOME's stage-wide agent cursor during window-targeted actions and after each action ended. That cursor follows upstream behavior; 0.38.0 still does not hide it. 0.38.0 also refuses to start its GNOME helper on GNOME 50 and newer (trycua/cua#5565), so window previews there fall back to what the driver can capture without it.

Evidence from the patched build

Measured on the patched 0.34.0 build, whose native changes 0.34.1 now ships. 0.34.1 itself has not been reprofiled.

A covered Helium window benchmark measured median native capture at 61.1 ms normally versus 38.5 ms display-only (19 measured calls per mode after warmup). For static content, normal 4 FPS capture used 14.3% driver CPU / 13.3% GNOME CPU / 4.8% device GPU busy; display-only adaptive polling used 1.8% / 6.9% / 3.8%, with 23 captures and one changed frame over 20.8 seconds. For animated content, capture stayed at 4 FPS while driver CPU fell from 14.8% to 6.1%. The persistent D-Bus transport reduced median native capture from 381 ms to 58 ms and raised capture rate from 1.29 to 4 FPS.

Installed build 2757: the covered Helium window appears correctly in the actual PiP. A 30-second static run captured 33 frames and averaged 2.65% device GPU busy and 41.7% tracked CPU (down from 7.3%/62.2% on 2755); an animated run captured 119 frames (3.96/sec) at 9.48% GPU and 62.4% CPU (down from 13.0%/72.9%). Closing PiP stopped capture completely: zero calls for 20 seconds. Tracked CPU includes T3 processes, GNOME, Cua, and Helium; one core = 100%.

Validation

• Focused tests: apps/server/src/cua, apps/desktop/src/cua, packages/shared/src/cuaDriverHostOptions.test.ts, scripts/build-desktop-artifact.test.ts, scripts/lib/cli-external-packages.test.ts: 166 pass. The one failure, "skips the primary native probe for cross-architecture Windows payloads", also fails on main.
• Typecheck clean for scripts, server, shared, and desktop; targeted lint clean.
• The upstream 0.34.1 linux-x86_64-binary archive matches the pinned SHA-256 on Linux x64, reports cua-driver 0.34.1, and its get_window_state schema advertises display_only with the description our capability probe matches.

Still draft: before/after video evidence is outstanding, and an installed 0.38.0 build has not been exercised on GNOME or Windows.

Implemented with GPT-6 Astra through the Codex harness; the Cua Driver bumps and removal of both source patches by Claude Opus 5.5 through Claude Code.

🤖 Generated with Claude Code

Application mentions

The composer can now find installed applications from the selected environment when typing @, including Helium on Windows. Selections retain the environment and launch identity, render as native app chips, and tell providers to resolve the current window before interacting. The inventory uses Cua Driver's cross-platform listApps API and stays disabled when Computer Use is off. Native icons retain a fallback until the image loads and when loading fails.

Validation: 184 focused tests pass across app inventory, reference projection, composer editing, icon loading, and authorization. Server/web typechecks passed. Windows build 9009 was exercised with an @Helium mention: the provider received the correct executable and environment, launched Helium, navigated releases and discussions, and closed its test window. macOS/Linux inventory behavior has fixture coverage but has not been exercised interactively.

Application mentions implemented with GPT-6.1-Sol through the Codex harness.