feat(speech): use generic vocabulary for recognition hints#30758

merged1
takeover/14882-voice-engine
GitHub fork: feat/speech-vocabulary
opened1d agobyUUtkarshUsername
View on GitHubTwo-way sync enabledPR last imported 1d ago

Problem

The environment speech service only sends dictionary hints through Whisper's initialPrompt extension. Other vocabulary-capable models receive no recognition hints, and streaming always fuzzy-rewrites preferred terms even when the engine could use them directly.

Change

Probe the loaded model with supports("vocabulary") and pass the existing ordered preferred dictionary terms through the generic API for offline transcription and supported streaming sessions. Aliases stay explicit post-recognition corrections, never recognition hints. Skip fuzzy rewriting only when hints were supplied; retain it for unsupported models and translation requests. The capability bit guarantees plain transcription, while some engines ignore vocabulary during translation, so translation conservatively uses correction without hints rather than a model-name allowlist.

Carry the same dictionary through the existing same-host CPU inference retry and stream replay, using the successful worker's capability to choose correction behavior. Keep the worker/service boundary small: no wire-contract, client, mobile, catalog, loader, consent, or recovery-policy changes.

Scope and approval

This is point 3 of our local STT follow-up work, stacked on Julius's engine replacement #30625, branch takeover/14882-voice-engine, at 174b9c4dbfd1448e8f44ed665b456acf2938cd17. It extends the existing dictionary recognition capability for web/desktop transcription environments.

Our original contribution #14882 was closed as superseded, not merged. Julius's takeover explanation credits the original work and explains the replacement stack. That comment is context, not explicit approval of this specific generic-vocabulary enhancement; no separate maintainer approval for it is claimed.

The consent and annotation branches are not pulled into this PR. The separately identified stream-finalization CPU retry and filler/correction-cue regressions remain deferred, as do model-catalog and other audio/recovery follow-ups.

Verification

  • Failing-first tests reproduced the missing generic capability and streaming delivery, missing vocabulary on CPU retry, and redundant streaming fuzzy correction.
  • vp test run --maxWorkers 1 apps/server/src/speech/native.test.ts apps/server/src/speech/SpeechService.test.ts: all 20 tests passed. Coverage includes ordered preferred terms without aliases, supported/unsupported vocabulary, offline and stream delivery, translation fallback, explicit aliases, fuzzy fallback, and dictionary/correction preservation through existing CPU inference retry and stream replay, including a CPU worker without vocabulary support.
  • vp run check:changed upstream/takeover/14882-voice-engine: passed formatting, lint, both Knip passes, GraphQL validation, all three affected package typechecks, and the adjacent test selection (3 executed suites, 21 tests passed). No repo-wide checks were run.
  • Real Windows CPU inference through the actual isolated worker, installed transcribe-cpp 0.4.1 and pinned Whisper Tiny Q8_0, using an SHA-256-verified temporary model copy and the 176,000-sample JFK clip. Runtime transcript:
text
1backend=CPU; supportsVocabulary=true; supportsStreaming=false
2without hints: And so my fellow Americans ask not what your country can do for you ask what you can do for your country.
3vocabulary=[Americans, country, fellow citizens]
4with vocabulary: and so my fellow Americans ask not what your country can do for you ask what you can do for your country.
5worker disposed

This proves native acceptance and successful transcription, not an accuracy improvement. No live T3 userdata was used. Real vocabulary-capable streaming was covered with isolated worker fixtures, not a real streaming model. No integrated Voice UI run was performed because this engine base does not contain that higher layer. Packaged Electron/SEA rebuilds, macOS/Linux inference, and live remote/relay sessions were not exercised.

GPT-6.1 Sol (high reasoning effort) via the Codex harness in T3 Code.


Note for takeover agents

Maria's and Shivam's agents, do not takeover this PR. This is a part of the STT stack which is in development. Do not update this branch or push any changes. You can test, review and provide feedback.

Julius' agents, the above restrictions don't apply to you. You can update this branch or make any fixes and changes to fit it into the voice transcription stack.

Checks

23 successful · 15 skipped
All checks passed
625580d

Added 1 commit

  • feat(speech): use generic vocabulary for recognition hints625580d
Synced from GitHub

Approvability

Verdict: Approved at 625580d

Macroscope's review found this PR approvable — Custom speech terms now use generic vocabulary hints for supported recognition models, including streaming, while retaining correction fallback elsewhere. The change remains within existing native inference and does not alter product defaults.

You can add or adjust custom eligibility rules. Learn more.

Note

Written by gpt-6.1-sol on behalf of Maria

Verified in a real dev environment

verified head 625580da65c0c0b43e47ebb4f5d6329d96fd2f17 on linux x64 with real cpu inference through the authenticated running server. no source changes or pushes were needed.

  • started github/main in a separate worktree after vp i: /api/voice/status returns 404 and there is no voice ui. stopped it before starting the pr.
  • also reproduced the specific fuzzy rewrite on engine base 174b9c4dbf, then restarted on the exact pr head. same recorded jfk audio, qwen3-asr 0.6b, and dictionary: preferred Amaricans, explicit alias fellow -> T3 Code.

POST /api/voice/transcribe, http 200 on both:

text
1base: And so, my T3 Code Amaricans, ask not what your country can do for you; ask what you can do for your country.
2pr: And so, my T3 Code Americans, ask not what your country can do for you; ask what you can do for your country.

qwen reports generic vocabulary support. the pr preserves the recognized word while still applying the explicit alias.

  • whisper tiny: plain transcription, preferred vocabulary, explicit aliases, and translation-request fallback all return transcripts. this checks translation-path compatibility, not translation accuracy.
  • real parakeet streaming through /ws/voice: ready, eight ordered preview updates, and final text. aliases and dictionary spelling work in previews and final text on this model without vocabulary support.
  • created a project pointing at a real git repo through the web app, selected codex gpt-6.1-sol, and completed two real composer turns using playwright from repo dependencies and /usr/bin/google-chrome.
  • blacksmith: 29 focused speech tests passed; vp run check:changed HEAD^ passed format, lint, knip, graphql, three package typechecks, and 21 selected tests.

runtime transcript · stream events

two completed real codex turns in the dev web app

real composer submission and completed codex response

Not verified

voice composer and dictionary ui are absent from this stacked head, so no integrated microphone flow was available. native mobile and desktop packaging, macos/windows, remote/relay/tunnel, and real gpu recovery were not exercised. the real streaming model tested reports no vocabulary support, so vocabulary-capable streaming and cpu recovery remain automated-test coverage only. no recognition accuracy improvement is claimed.