Sign up
View stargazersSign up

Feat/voice postprocessing api#33007

closed3
main
GitHub fork: feat/voice-postprocessing-api
opened14h agobyOOuna-the-Dataweaver
33 files changed
View on GitHubTwo-way sync enabledPR last imported 6h ago

Problem

Transcript cleanup currently goes through a coding-agent provider, which means that for simple text-only rewrite it calls agent with full coding prompt, and starts full instance of provider. Users cannot choose a direct model API for this step. Considering that OpenAI and Anthropic 100$+ subs come with api key, and that rewrite is small token wise, it makes sense to expose path for BYOK cleanup for transcripts. For OpenCode, it offers POST /api/experimental/generate/text endpoint with simple generation, which is both faster and uses less tokens compared to full build agent invocation.

Change

Add optional cleanup backends for OpenAI, Anthropic, OpenAI-compatible endpoints, and OpenCode raw text generation.

  • OpenRouter is a preset under OpenAI-compatible endpoints.
  • Direct API settings include model discovery, manual model entry, OpenAI/Anthropic reasoning controls, and explicit saved-key status.
  • OpenCode reuses the configured provider’s server and authentication. It does not require a separate cleanup API key.
  • Cleanup keeps the existing prompt, alias correction, and raw-transcript fallback.

Contract, server, client-runtime, credential-storage, and settings changes support these backend choices.

Scope and approval

This configures the existing optional transcript-cleanup step: its execution backend, model, and reasoning level. Cleanup remains off by default, and the existing agent-backed path remains the default backend.

STT and model hosting are unchanged. Configuration controls are exposed in the web/desktop Voice settings.

Verification

  • vp run check:changed passed: 265 tests passed, with 2 existing platform skips, plus affected-package typechecks, formatting, lint, knip, and React Compiler checks.
  • Protocol tests cover provider credentials, reasoning payloads, model-list pagination, incomplete responses, and OpenCode generation without an agent session.
  • Regression tests cover provider-specific catalogs, ignored key removals, OpenCode model defaults, and unsaved keys when switching environments.
  • Browser coverage confirms model catalogs do not reload while typing an API key.
  • Live OpenAI model discovery returned 96 models, and cleanup succeeded with explicit low reasoning.
  • Live cleanup through the configured OpenCode provider returned “Hello, world.”
  • OpenAI and OpenCode cleanups tested manually.

UI evidence:

Implemented and reviewed with GPT-6.1 Sol + Astra using Codex harness in T3 Code.

Checks

23 successful · 13 skipped · 2 neutral
All checks passed
4788b96

Added 3 commits

  • fix(voice): isolate cleanup catalogs and credential drafts4788b96
  • feat(voice): add cleanup reasoning and model discovery7475530
  • feat(voice): add direct API transcript cleanup89ec96d
apps/server/src/speech/SpeechCleanupApi.tslines 77–90Open thread
……@@ -0,0 +1,323 @@
87+ }),
88+ ),
89+ has_more: Schema.optionalKey(Schema.Boolean),
90+ last_id: Schema.optionalKey(Schema.String),
MmacroscopeappBoton lines 77–9014h ago

🟠 High speech/SpeechCleanupApi.ts:77

Anthropic model discovery fails for valid model-list pages when capabilities or last_id is null, because ModelPage rejects those values and the response decode fails. Make both optional fields nullable; the existing checks already handle null as absent.

diff
1- capabilities: Schema.optionalKey(
2- Schema.Struct({
3+ capabilities: Schema.optionalKey(
4+ Schema.NullOr(
5+ Schema.Struct({
6 effort: Schema.optionalKey(
7 Schema.Record(
8 Schema.String,
9 Schema.Union([Schema.Boolean, Schema.Struct({ supported: Schema.Boolean })]),
10 ),
11 ),
12- }),
13+ }),
14+ ),
15 ),
16@@
17- last_id: Schema.optionalKey(Schema.String),
18+ last_id: Schema.optionalKey(Schema.NullOr(Schema.String)),
🤖 Copy this AI Prompt to have your agent fix this:
text
1In file @apps/server/src/speech/SpeechCleanupApi.ts around lines 77-90:
2
3Anthropic model discovery fails for valid model-list pages when `capabilities` or `last_id` is `null`, because `ModelPage` rejects those values and the response decode fails. Make both optional fields nullable; the existing checks already handle `null` as absent.
MmacroscopeappBot14h ago
Synced from GitHub

Approvability

Verdict: Would Approve

Macroscope's review found this PR approvable — Adds explicitly configured direct API and OpenCode transcript-cleanup options while preserving existing defaults and secret handling. The Anthropic catalog-decoding finding remains subject to the separate correctness gate.

Needs human approval because:

  • 1 blocking correctness issue found at or above your repo's Minimum Blocking Severity

Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.

CcoderabbitaiBot14h ago
Synced from GitHub

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The pull request adds direct OpenAI, Anthropic, compatible, and OpenRouter speech cleanup, plus OpenCode raw-generation cleanup. It adds provider settings, credential storage, model discovery, and web configuration for these paths.

Changes: Speech Cleanup Providers

Layer / File(s)Summary
Cleanup settings and credential contracts
packages/contracts/src/speech.ts, packages/contracts/src/settings.ts, packages/contracts/src/settingsSecretValue.ts, apps/server/src/settingsSecrets.ts, apps/server/src/serverSettings.ts, apps/server/src/serverSettings.test.ts		Contracts add cleanup provider, model, reasoning, endpoint, and credential settings. The server maps voice API credentials to secret storage. Tests cover redaction, provider removal, and independent provider settings.
Cleanup API and model discovery
packages/contracts/src/environmentHttp.ts, packages/client-runtime/src/voice-input/*, apps/server/src/speech/SpeechCleanupApi.ts, apps/server/src/speech/http.ts, apps/server/src/server.ts, apps/server/src/speech/SpeechCleanupApi.test.ts, apps/server/src/speech/http.test.ts		The server adds provider-specific cleanup requests and model discovery. The authenticated route and client runtime expose model catalogs. Tests cover request formats, credentials, model responses, and errors.
Post-processing routing and OpenCode raw generation
packages/provider-core/src/server/textGeneration.ts, packages/shared/src/serverSettings.ts, apps/server/src/speech/postProcessing.ts, apps/server/src/textGeneration/TextGeneration.ts, packages/provider-opencode/src/server/*, apps/server/src/speech/postProcessing.test.ts, apps/server/src/textGeneration/OpenCode2TextGeneration.test.ts, packages/shared/src/serverSettings.test.ts, packages/provider-opencode/src/server/driver.test.ts		Post-processing routes to the configured API provider, agent generation, or OpenCode direct generation. OpenCode direct generation uses the raw endpoint and rejects unsupported direct cleanup on OpenCode 1.x.
Cleanup provider settings and scoped credential handling
apps/web/src/components/settings/CleanupApiSettings.tsx, apps/web/src/components/settings/VoiceSettingsPanel.tsx, apps/web/src/components/settings/scopedSettings.ts, apps/web/src/components/settings/useScopedSettings.ts, apps/web/src/components/settings/settingsSearch.ts, apps/web/src/components/settings/CleanupApiSettings.browser.test.tsx, apps/web/src/components/settings/scopedSettings.test.ts, docs/user/voice-input.md		The web interface configures cleanup providers and loads model catalogs. Scoped settings handle redacted credential responses and save callbacks. The documentation describes direct API and OpenCode cleanup configuration.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

mermaid
1sequenceDiagram
2 participant SpeechPostProcessing
3 participant SpeechCleanupApi
4 participant ProviderEndpoint
5 SpeechPostProcessing->>SpeechCleanupApi: Submit cleanup prompt
6 SpeechCleanupApi->>ProviderEndpoint: Send provider-specific request
7 ProviderEndpoint-->>SpeechCleanupApi: Return completion response
8 SpeechCleanupApi-->>SpeechPostProcessing: Return trimmed text

Possibly related issues

  • pingdotgg/t3code issue 32049: The pull request adds an OpenCode raw-generation cleanup path, but the provided summary does not establish that it resolves the reported prompt/output-schema mismatch.

Suggested reviewers: juliusmarminge, yordis

Merge Risk: External image Minimal · up to 4788b

This change adds optional transcript cleanup providers that are off by default, and no merge-blocking issue was found. The agent-backed default path is unchanged.

Pre-merge checks: Passed 4

Check nameStatusExplanation
Title checkPassedThe title "Feat/voice postprocessing api" names the main change: a new voice transcript post-processing (cleanup) backend that uses direct provider APIs. It is short and specific. It uses a branch-sty…
Description checkPassedThe description follows the repository template sections (Problem, Change, Scope and approval, Verification). The Problem section explains why agent-based cleanup is costly and why a direct API path i…
Linked Issues checkPassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes checkPassedCheck skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "Feat/voice postprocessing api" names the main change: a new voice transcript post-processing (cleanup) backend that uses direct provider APIs. It is short and specific. It uses a branch-style prefix and abbreviates the content, but a teammate scanning history can identify the primary change. It is not vague, unrelated, or misleading.

Full details: Description check

Explanation

The description follows the repository template sections (Problem, Change, Scope and approval, Verification). The Problem section explains why agent-based cleanup is costly and why a direct API path is needed. The Change section lists the backends and related layers. The Scope and approval section states that the change configures the existing optional cleanup step and that the default backend is unchanged. It does not link a triaged issue or maintainer approval, which the template requires for new capabilities. The Verification section lists specific tests and live checks with observed results. The title in the description also does not follow the conventional commit format the template suggests. The description is mostly complete, but the approval link is missing and the template's conventional commit title guidance is not followed.

Resolution

Add a link to the triaged issue or the maintainer discussion that approved this direction and scope, including the approval comment. Alternatively, explain why the change qualifies as focused configuration of an established capability. Consider also renaming the PR title to a conventional commit form in plain language, such as "feat(server): add direct API backends for voice transcript cleanup".

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

External image Recent review info

External image Run configuration

External image Commits

Reviewing files that changed from the base of the PR and between ade473018d32003658b7d38693e083ee2e314bd8 and 4788b960fa53814a6bcf14b9d98e480322cd6b56.

External image Files selected for processing: 33

  • apps/server/src/server.ts
  • apps/server/src/serverSettings.test.ts
  • apps/server/src/serverSettings.ts
  • apps/server/src/settingsSecrets.ts
  • apps/server/src/speech/SpeechCleanupApi.test.ts
  • apps/server/src/speech/SpeechCleanupApi.ts
  • apps/server/src/speech/http.test.ts
  • apps/server/src/speech/http.ts
  • apps/server/src/speech/postProcessing.test.ts
  • apps/server/src/speech/postProcessing.ts
  • apps/server/src/textGeneration/OpenCode2TextGeneration.test.ts
  • apps/server/src/textGeneration/TextGeneration.ts
  • apps/web/src/components/settings/CleanupApiSettings.browser.test.tsx
  • apps/web/src/components/settings/CleanupApiSettings.tsx
  • apps/web/src/components/settings/VoiceSettingsPanel.tsx
  • apps/web/src/components/settings/scopedSettings.test.ts
  • apps/web/src/components/settings/scopedSettings.ts
  • apps/web/src/components/settings/settingsSearch.ts
  • apps/web/src/components/settings/useScopedSettings.ts
  • docs/user/voice-input.md
  • packages/client-runtime/src/voice-input/environment.ts
  • packages/client-runtime/src/voice-input/index.ts
  • packages/contracts/src/environmentHttp.ts
  • packages/contracts/src/settings.ts
  • packages/contracts/src/settingsSecretValue.ts
  • packages/contracts/src/speech.ts
  • packages/provider-core/src/server/textGeneration.ts
  • packages/provider-opencode/src/server/driver.test.ts
  • packages/provider-opencode/src/server/driver.ts
  • packages/provider-opencode/src/server/v2/OpenCode2Client.ts
  • packages/provider-opencode/src/server/v2/textGeneration.ts
  • packages/shared/src/serverSettings.test.ts
  • packages/shared/src/serverSettings.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

JJulius Marminge6h ago

Thanks for this, it pointed in the right direction. Rather than a voice-only cleanup backend, we're making direct model APIs a core concept that all text generation can use: thread titles, commit and PR text, branch names, and transcript cleanup. That landed as pingdotgg/t3code#33147, which reuses your OpenAI-compatible and Anthropic request and model-discovery code and keeps you as the source of the idea. The OpenCode raw /generate/text path would speed up OpenCode's runner for every task, so it's worth its own PR against that runner if you want to pick it up. Closing in favor of #33147.

OOuna-the-Dataweaver4h ago
Synced from GitHub

so it's worth its own PR against that runner if you want to pick it up.

Yep, thanks, generic reusable BYOK text generation is even better approach here. I'll pick up the opencode generation after you merge #33147 so it goes on top of existing code.