Sign up
View stargazersSign up

fix(server): usage prices long-context requests at above-tier rates#25429

open2
main
fix/issue-25414
opened6d agobyJJulius Marminge
4 files changed
View on GitHubTwo-way sync enabledPR last imported 4d ago

Problem

Usage estimated requests above LiteLLM's long-context thresholds at base rates. A Codex rollout with 300,000 input tokens (280,000 cached) and 1,000 output tokens on gpt-6-sol cost $0.106 instead of $0.207; Claude's 100K tier and priority long-context rates were also ignored.

Change

Parse LiteLLM's context tiers and select the rate from each request's uncached, cached, and cache-creation input total before aggregating costs. Apply the same tier to category costs, cache savings, and speed premiums. Existing raw rate documents and per-request scan caches are repriced on read; custom price overrides remain flat.

Scope and approval

Fixes https://github.com/pingdotgg/t3code/issues/25414

Triaged bug via the automated issue flow. GitCafe: #25414. Reproduced in the real web app against an explicitly isolated server home. Scope is server pricing and its regression tests.

Verification

  • Red tests first: Codex $0.106 versus $0.207, Claude above 100K, and priority above 272K failed on main. Reproduction transcript.
  • Focused changed-file tests: 68 passed across UsageService, usagePricing, and usageAggregation; covers threshold boundaries, mixed short/long requests, overrides, missing rates, and persisted cache restoration.
  • Real server/browser, same rollout and isolated --home-dir: cost $0.106 → $0.207; 301K processed tokens, 280K cached, 20K uncached, and 1K output unchanged. After categories: input $0.080, cache read $0.112, output $0.015. Before response · After response · Reproduction seed.
  • Evidence index; server-only change, capture matrix [], independently audited sufficient.
  • vp run check:changed: format, lint, GraphQL validation, and focused tests passed. The overall run failed on a stale local TypeScript executable and knip's mobile Metro configuration loader/unrelated unused export. A retry was stopped when VM load reached 25. Complete check output.
  • After removing stale generated compiler shims, vp run --filter t3 typecheck passed (exit 0). The overall changed-file run still has no complete green result.
  • Web, desktop, mobile, and remote clients consume the same server estimate; no client or wire changes. No platform-specific verification is needed for this server calculation.

Models: GPT-6.1 Sol through Codex and Claude Opus 5.5 through Claude Code, in T3 Code.

Checks

21 successful · 10 skipped · 2 neutral
All checks passed
f5a43a8

Added 2 commits

  • test(server): usage prices long-context requests at above-tier rates (red)93b6571
  • fix(server): usage prices long-context requests at above-tier ratesf5a43a8
apps/server/src/usage/usagePricing.tsline 214Open thread
……@@ -141,22 +183,51 @@ function sameTokenRates(a: TokenRates | null, b: TokenRates | null): boolean {
150212 !sameTokenRates(fast, b.fast ?? fast) ||
151- !sameTokenRates(ultrafast, b.ultrafast ?? ultrafast)
213+ !sameTokenRates(ultrafast, b.ultrafast ?? ultrafast) ||
214+ (b.contextTiers.length > 0 && !sameContextTiers(contextTiers, b.contextTiers))
MmacroscopeappBoton line 2146d ago

🟡 Medium usage/usagePricing.ts:214

A bare alias is rejected even when both providers agree on its standard context-tier price, so priceUsage reports short standard requests as unpriced. sameContextTiers treats a missing fast or ultrafast rate as a conflict instead of filling it from the other matching tier; merge those missing speed variants before checking for conflicts.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
text
1In file @apps/server/src/usage/usagePricing.ts around line 214:
2
3A bare alias is rejected even when both providers agree on its standard context-tier price, so `priceUsage` reports short standard requests as unpriced. `sameContextTiers` treats a missing `fast` or `ultrafast` rate as a conflict instead of filling it from the other matching tier; merge those missing speed variants before checking for conflicts.
MmacroscopeappBot6d ago
Synced from GitHub

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — This production change alters customer-visible usage pricing, cache savings, speed premiums, and persisted usage repricing for long-context requests. Its billing/metering impact, together with an unresolved Medium pricing concern around aliases, warrants human review.

Needs human approval because:

  • 1 blocking correctness issue found at or above your repo's Minimum Blocking Severity

Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.

Ggithub-actionsBot6d ago
Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire5.0 KiB5.0 KiB0 B (0.0%)6.8 KiB✅
CodexThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
CodexLive turn WebSocket wire1.2 KiB1.2 KiB0 B (0.0%)2.0 KiB✅
CodexLive turn WebSocket decoded20.9 KiB20.9 KiB0 B (0.0%)29.3 KiB✅
CodexLive turn messages220 (0.0%)8✅
ClaudeTotal thread wire5.0 KiB5.0 KiB−7 B (−0.1%)6.8 KiB✅
ClaudeThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
ClaudeLive turn WebSocket wire1.2 KiB1.2 KiB−7 B (−0.6%)2.0 KiB✅
ClaudeLive turn WebSocket decoded21.2 KiB21.2 KiB0 B (0.0%)29.3 KiB✅
ClaudeLive turn messages220 (0.0%)8✅

Baseline: 1316d48 · PR result: f5a43a8 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

CcoderabbitaiBot6d ago
Synced from GitHub

Review in Change Stack →

No actionable comments were generated in the recent review.

External image Recent review info
External image Run configuration
External image Commits

Reviewing files that changed from the base of the PR and between 1316d48b0c68151afd8b47eb0c11a114cf0b4daa and f5a43a82bb3c5ab4e4343bd66357b71638a8cf5b.

External image Files selected for processing: 4
  • apps/server/src/usage/UsageService.test.ts
  • apps/server/src/usage/usageAggregation.test.ts
  • apps/server/src/usage/usagePricing.test.ts
  • apps/server/src/usage/usagePricing.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

Walkthrough

Usage pricing now parses long-context rate tiers and selects rates from each record’s prompt-token total and speed. Tests cover tier thresholds, fallback rates, aggregation, and service behavior when the rates endpoint returns either success or failure.

Changes: Long-context usage pricing

Layer / File(s)Summary
Parse context-tier rates
apps/server/src/usage/usagePricing.ts		ModelRate now includes context tiers. Rate parsing reads tier-specific standard and speed rates, applies base-rate fallbacks for omitted fields, and sorts tiers by threshold.
Select rates by prompt size
apps/server/src/usage/usagePricing.ts, apps/server/src/usage/usagePricing.test.ts		Alias merging includes context tiers. Pricing and cache-savings calculations select rates using prompt-token totals and record speed. Tests cover Claude and GPT thresholds, priority rates, missing tier fields, flat-rate models, and price overrides.
Verify aggregate and service pricing
apps/server/src/usage/usageAggregation.test.ts, apps/server/src/usage/UsageService.test.ts		Aggregation tests check that higher rates apply to the long request. Service tests check the expected $0.207 cost when the rates endpoint returns 200 or 500.

Priority: External image Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Low

Suggested reviewers: t3dotgg, maria-rcks

Merge Risk: External image Minimal · up to f5a43

No concrete issue in the supplied evidence prevents merging after normal checks.

Pre-merge checks: Passed 4
Check nameStatusExplanation
Title checkPassedThe title is concise, conventional, and clearly describes the server pricing fix for long-context requests using above-tier rates.
Description checkPassedThe description includes complete Problem, Change, Scope and approval, and Verification sections. It provides the issue link, implementation scope, focused test results, runtime evidence, and known ve…
Linked Issues checkPassedIssue [#25414] requires above-threshold pricing for GPT-family, Claude, and priority requests without changing token totals. parseRateTable reads above_Nk_tokens tiers, fills omitted tier fields f…
Out of Scope Changes checkPassedThe whole-PR diff contains changes only in server usage pricing and its regression tests: apps/server/src/usage/usagePricing.ts, usagePricing.test.ts, usageAggregation.test.ts, and `UsageService…
Full details: Description check

Explanation

The description includes complete Problem, Change, Scope and approval, and Verification sections. It provides the issue link, implementation scope, focused test results, runtime evidence, and known verification limitations.

Full details: Linked Issues check

Explanation

Issue [#25414] requires above-threshold pricing for GPT-family, Claude, and priority requests without changing token totals. parseRateTable reads above_Nk_tokens tiers, fills omitted tier fields from base rates, and preserves speed-specific tiers. ratesAt selects a tier when combined uncached, cached, and cache-creation input tokens exceed the threshold. priceUsage applies that tier to input, cache, output, category costs, and speed premiums. cacheSavingsUsd uses the same tier. Overrides remain flat. The added tests cover Claude above 100K, GPT above 272K, priority rates, threshold boundaries, fallback fields, overrides, mixed request aggregation, and persisted UsageService pricing. The changed files are within the issue's apps/server scope.

Full details: Out of Scope Changes check

Explanation

The whole-PR diff contains changes only in server usage pricing and its regression tests: apps/server/src/usage/usagePricing.ts, usagePricing.test.ts, usageAggregation.test.ts, and UsageService.test.ts. The implementation and tests directly support issue [#25414]. No client, wire, unrelated feature, or unrelated cleanup changes are present.

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.