feat(speech): filter silence and transcribe live audio chunks#30961

Problem

Web and desktop dictation currently sends non-speech audio to ASR, cuts bounded recordings by duration/quiet-frame energy, and waits until Stop to process non-native-streaming models. Long recordings therefore leave all their decoding work until the end, and silence can produce unwanted text.

Change

Run Silero VAD v6.2 on the selected transcription environment's CPU through the installed transcribe-cpp 0.4.1 API. One recurrent session scores each frame once for both filtering and pause planning. Capture filtering uses Handy's time-based pre-roll, onset and tails: 450 ms prefill, 60 ms onset, 450 ms offline tail and 1650 ms native-streaming tail, rounded up to model frames. Partial final frames and short stopped utterances retain their real samples.

Eligible non-native-streaming models use pause cuts near 30 seconds, a 40-second reference window, short-final protection and hard cuts when needed, bounded by the loaded model's existing output-limit headroom. Completed chunks decode during recording and appear as stable committed previews; Stop finishes the remaining tail and returns the complete ordered transcript. Native streaming keeps its native stream, and models with internal long-form support, such as Whisper, keep that path. Existing bounded offline error retries remain intact.

Silence filtering is a personal web/desktop preference, On by default. AUDIO consent remains Off by default, and this preference never grants microphone access, changes the named destination or enables cleanup. Disabling filtering retains raw PCM while pause planning remains applicable. Browser capture, destination binding, ownership, cancellation, recording/audio limits and bounded socket acknowledgements are preserved. Filtering runs after upload and does not reduce upload bandwidth.

The MIT-licensed 1,241,056-byte Silero helper is acquired through the existing hash-verified model store at a pinned revision, outside ASAR and outside the ASR picker. The worker reuses/resets its VAD session and releases native memory with its process. The helper is acquired concurrently with ASR installation from either setup or Settings. Every installation ensures the one shared helper; a verified cache entry causes no network download. Setup discloses its 1.2 MB size and reuse. Install-time acquisition uses the normal model downloader without the recording-specific deadline. A missing helper gets a 15-second recording-time retry; optional VAD failures warn and replay preserved raw audio using bounded ASR processing. ASR errors remain distinct.

Scope and approval

This is a focused extension of the STT capability in Julius's engine/consent stack, #30625 and #30623. It targets takeover/14882-voice-consent directly at 215cd5dce8cdd007d1cfa6eb67ba209663c69c82, because shared server processing and the web/desktop recording preference belong together. The takeover explanation records why original #14882 was superseded; it was closed, not merged. No separate maintainer approval is claimed.

No mobile feature, compatibility/migration layer, native recorder, auxiliary language detection, backend picker, model catalog refresh, vocabulary enhancement, Moonshine rollover, output-budget/truncation/repetition policy or unrelated recovery fix is included. Independent #30758, #30821, #30855 and #30869 remain separate and unmodified.

Verification

• Failing-first worker tests on the original base exposed silence reaching ASR and the inability of batch models to produce live chunks. Processing/worker coverage exercises speech edges and the longer streaming tail, silence, very short audio, partial frames, filtering disabled, native/internal-long-form eligibility, smaller limits, hard cuts, live ordering/final tails, cancellation/reset, unavailable/failed VAD and distinct ASR failures. Existing client tests cover both filter values, the selected destination rather than primary/draft host, separate cleanup consent and grant revocation.
• Focused eight-suite run: 91 tests passed. Final vp run check:changed upstream/takeover/14882-voice-consent passed formatting, lint, both Knip passes, GraphQL validation, all 36 affected typechecks and 110 adjacent tests across nine executed suites. React Compiler selection passed with no matching component cases. Commit hooks also passed.
• Real Windows CPU inference through the production worker used hash-verified Silero and Moonshine Tiny with transcribe.cpp samples. Padded JFK retained the full sentence; jobs-silence, all-zero audio and a very short quiet head returned empty. A 52-second repeated-JFK recording emitted stable updates at 30 and 48 seconds before Stop, then returned ordered final text. Native Moonshine Streaming Tiny plus Silero returned the full JFK sentence and empty silence. This verifies inference and lifecycle, not perfect recognition accuracy; the tiny model's normal wording/punctuation errors remain visible. Native runtime transcript.
• The production server bundle built successfully. Its actual packed __speech-worker entry loaded CPU ASR and Silero, returned empty text for silence, and exited. Packed-worker transcript.
• test-t3-app integrated pass used isolated snapshot-seeded state with two projects and stopped threads. It exercised the real client/controller/authenticated socket/service/CPU Silero/ASR, with a synthetic microphone and recorded PCM injected at the browser capture boundary. Filtering defaulted On without AUDIO consent; toggling did not change AUDIO/CLEANUP grants. Stable text appeared during Recording; Stop inserted the ordered transcript. The next recording sent filtering false, and revoking AUDIO closed its socket without a second finished result. Cleanup remained Off. No paid provider calls or live database writes. Client observations and final result, short recording showing live preview and Stop.
• Direct retained-PCM follow-up with real CPU Silero and ASR, using an external transparent observer around the unchanged production worker: padded JFK is 16.000 s / 256,000 samples. Batch filtering On forwards 11.456 s / 183,296 samples (4.544 s removed), with exactly the same full transcript as Off; Off preserves PCM byte-for-byte. Native filtering On forwards 12.768 s / 204,288 samples (3.232 s removed), including the entire original JFK waveform plus its longer tail, and yields the complete sentence. Both paths forward zero samples/calls for one second of generated zeros and the real 5.331-second jobs-silence.wav. These are measured inputs immediately before actual ASR calls, not scripted probabilities. Additional observation: native filtering Off forwarded every sample but returned empty text on this padded clip; no excluded stream-recovery changes were made. Measurements and method, padded source audio, batch retained audio, native retained audio. Observer filename ordering was corrected before native measurements completed; the method record discloses this scratch-harness issue.
• Native empty-result diagnosis: reproduced without the observation wrapper, both through T3's unchanged worker and directly through transcribe-cpp 0.4.1. Direct Moonshine Streaming Tiny recognizes the complete padded JFK sentence at 13 seconds; after more trailing silence, its 15-second feed replaces full/committed/tentative text with empty strings. finalize() retains the empty result. The original unpadded clip and the retained VAD clip both finish with the complete sentence. The engine re-decodes its full hypothesis on feeds and replaces the old text, so this is native decoder behavior on this clip, not lost PCM or T3 dropping the final text. VAD's shorter tail avoids it here. The underlying reason the model chooses an empty hypothesis has not been established. No custom Moonshine recovery, rollover or stitching was added. Direct-engine/worker trace.
• Install-time acquisition follow-up: the existing detached-install regression failed first because only ASR was acquired. Focused service/store verification now passes 17 tests, including concurrent helper acquisition, detached-request ownership, optional helper failure without breaking ASR, and verified-cache reuse without HTTP requests. The final base-scoped check passed 110 adjacent tests and all other selected checks. A real isolated setup pass selected a genuinely missing Whisper Tiny while leaving Moonshine installed, downloaded missing Silero before Whisper finished, and reached Ready to dictate. The missing-selected-model state was set through the server API; the normal Settings UI only offers Use model after installation. The real Settings Download control also reused an identical cached helper (size/hash/mtime unchanged); its ASR weights were preseeded to avoid an unnecessary second large download. No additional microphone access occurred during either installation flow. Install observations.

Setup screenshots: before / after.

• Timeout correction: install-time Silero acquisition no longer inherits the recording startup deadline. The extended detached-install regression failed first when the old 15-second cap interrupted the helper. With a virtual TestClock, installation now remains active beyond 16 seconds, preserves exclusive ownership and completes when the helper finishes. The complementary recording regression verifies that startup waits at 14,999 ms, acquisition is interrupted at 15,000 ms, and raw-audio feed/finalization still return the transcript. All 15 speech-service tests and the base-scoped changed checks pass (110 adjacent tests). This is deterministic service-level timing proof; no new slow-network benchmark or native/packaged-platform run is claimed.
• CI budget note: the Agent context budget gate compares with current main, not this PR's consent base. The unchanged base/head values are 5153 / 6883 / 47538; main at 9e85538535c8553901351112ab02dd1dd62db5ee has 5133 / 6863 / 47498. The reported +20/+20/+40-byte increases are inherited. The focused AgentContextBudget.test.ts passes (one test), confirming measured agent context matches the unchanged budget. This PR edits neither the budget file nor agent instructions/tool definitions. Resolving this gate needs an upstream stack update or a maintainer's explicit acceptance; no budget edit, main merge, policy change or acceptance label was applied. Macroscope's separate “Needs Human Approval” notice concerns the requested default-on filtering and is not a reported correctness failure.
• Verification limits: the shared desktop/web UI was exercised in the Browser panel, not a rebuilt Electron installer. Full Electron/ASAR and SEA packages, native macOS/Linux, GPU execution, physical microphone capture and a live second environment/relay/tunnel session were not exercised. Existing chosen-destination tests cover host routing; this is not a claim of live remote proof.

Before:

Voice recognition settings before

After:

Voice recognition settings with silence filtering

Evidence is attached to a GitHub prerelease in the fork; no PR-only assets or plans are committed.

Implemented with GPT-6.1 Sol through the Codex harness in T3 Code.

Note for takeover agents

Maria's and Shivam's agents, do not takeover this PR. This is a part of the STT stack which is in development. Do not update this branch or push any changes. You can test, review and provide feedback.

Julius' agents, the above restrictions don't apply to you. You can update this branch or make any fixes and changes to fit it into the voice transcription stack.