Sign up
View stargazersSign up

test: delete 11k low-signal tests so CI runs faster and flakes less#22557

merged1
main
t3/purge-low-value-tests
opened8d agobyTtheo
100+ files changed
View on GitHubTwo-way sync enabledPR last imported 8d ago
Thanos snap

Our test suite got huge and noisy. Release runs fail most of the time on flaky browser tests, and a lot of tests check things types already guarantee, mirror the code, or wait on real timers. They cost time and give no signal.

This deletes them: about 11,000 of 25,928 tests (43%). 456 test files are deleted outright and 1,195 more are trimmed. Summed test-file time drops 59%.

How the tests were picked

  • A swarm of Claude Haiku 5.5 agents read every test file against one rubric. Delete: type-redundant, trivial logic, basic DOM behavior, real or fake timers, mock-call mirrors, smoke tests, duplicate table cases, and wall-clock perf asserts. Keep: orchestration and event-sourcing correctness, provider protocol translation, auth and permission guards, persistence and migrations, external-format parsing, and deterministic race fixes.
  • About half the files (987) got two independent passes. There a test is deleted only when both agree, and a third Haiku vote settled the ~1,900 disagreements. The other files got one pass.
  • Deletions are applied with the TypeScript AST, matched by test title, not by hand. Imports and helpers left unused afterwards are removed the same way.

Follow-on cleanup

  • knip --exports failed on ~420 exports that only deleted tests used. Those are un-exported now, and ~48 declarations nothing calls are deleted, so there is no production behavior change. builtInProviderAdapterDrivers.ts only fed a test, so it is gone and adding-a-provider.md is updated to match.
  • Orphaned test fixtures are removed. provider-testing and marketing lose their now-empty test scripts, and provider-testing drops dev deps it no longer uses.
  • Kept on purpose: both reactCompilerCoverage.test.ts files (documented perf guard), and slimmed copies of bundledDev.test.ts and fileEditorVirtualization.test.ts, which guard a Vite upgrade and a patched dependency.
  • Updated the test:perf:v2-wire file list, the mobile react-test list, the knip entry for sign-macos.ts, and doc references to deleted files.

Verification

  • Typecheck (all packages), lint, fmt check, and knip:check pass locally on the rebased head.
  • Local before/after, same commands and machine (vitest JSON reporter):
PackageTests before → afterWall time before → after
server7,478 → 4,8401,040s → 564s
web (unit + browser)8,509 → 4,240132s → 53s
mobile3,478 → 2,19934s → 27s
client-runtime2,261 → 1,2608s → 6s
desktop1,665 → 93420s → 5s
shared1,842 → 1,2355s → 3s
contracts695 → 1391s → 1s
total25,928 → 14,847summed file time 1,205s → 490s
  • The server failures that remain locally (pinnedRuntime, ServerEnvironment, ProviderInstanceRegistry, AcpSessionRuntime.processTree) fail the same way on main on this machine. One web browser test, TelemetryExportSettings, timed out once under load; this PR does not touch it.
  • shardWeights.json drops deleted files and scales trimmed ones by their measured speedup, so server shards stay balanced.
  • Review restored 22 auth boundary tests the swarm had cut as "timing" (they use the deterministic Effect clock): session and websocket credential expiry, desktop bootstrap expiry and rotation, MCP credential liveness and thread isolation, the one-scope-per-RPC guard, and a few scope-narrowing checks.
  • Browser tests pre-bundle @base-ui/react/autocomplete. With fewer tests, Vite's dependency scan no longer found it and reloaded mid-run, which broke a test iframe.
  • main currently fails typecheck and web tests on a stale mermaid import in MessagesTimeline.browser.test.tsx (from #21472). This PR removes that warmup along with the Mermaid tests it served.

Reviewed with sol-loop: 2 rounds with GPT-6.1 Sol. Sol verdict: approved

Created with Claude Opus 5.5 (test selection by Claude Haiku 5.5) in Claude Code.

🤖 Generated with Claude Code

Checks

24 successful · 9 skipped · 1 neutral
All checks passed
1061079
MmacroscopeappBot8d ago
Synced from GitHub

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — This is primarily a large test-pruning and dead-export cleanup change, with limited intended runtime impact. It also modifies authentication-directory files and removes authentication-related coverage, so the sensitive-area criteria require human review.

Notes:

  • All code in this push has already been reviewed. Approvability was decided on eligibility alone.

You can add or adjust custom eligibility rules. Learn more.

Ggithub-actionsBot8d ago
Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire5.0 KiB5.0 KiB+7 B (+0.1%)6.8 KiB✅
CodexThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
CodexLive turn WebSocket wire1.2 KiB1.2 KiB+7 B (+0.6%)2.0 KiB✅
CodexLive turn WebSocket decoded20.9 KiB20.9 KiB0 B (0.0%)29.3 KiB✅
CodexLive turn messages220 (0.0%)8✅
ClaudeTotal thread wire5.0 KiB5.0 KiB−24 B (−0.5%)6.8 KiB✅
ClaudeThread snapshot wire3.8 KiB3.8 KiB0 B (0.0%)4.9 KiB✅
ClaudeLive turn WebSocket wire1.2 KiB1.2 KiB−24 B (−2.0%)2.0 KiB✅
ClaudeLive turn WebSocket decoded21.2 KiB21.2 KiB−41 B (−0.2%)29.3 KiB✅
ClaudeLive turn messages21−1 (−50.0%)8✅

Baseline: c3540a7 · PR result: 1061079 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Added 1 commit

  • test: delete 11k low-signal tests so CI runs faster and flakes less1061079