Sign up
View stargazersSign up

fix(search): honest token-drop, Cyrillic normalization, multi-word brand boost#143

open3
v2
orch/search-followups
openedJul 7, 2026byAAviddaram
4 files changed
View on GitHubTwo-way sync enabledPR last imported 20d ago

Three P2 follow-ups from Devin's review of merged PR #132. Engine files only (core.ts, text.ts, tests).

  • Unmatched-token honesty: a token is dropped only when it has zero postings AND zero alias/fuzzy candidates. 'creatine vitamin d3' (creatine unstocked, real term) now returns honest empty instead of d3-only results presented as a full match. Pure noise tokens ('d','c') still drop.
  • Cyrillic normalization: NFKD was splitting й into и + combining breve ('нойр' → 'нои'+'р'), breaking Mongolian recall. Now folds Latin diacritics only and recomposes NFC — 'нойр'/'ё' stay intact, café→cafe still works.
  • Multi-word brand boost: canonical-brand boost was token-wise Array.includes, so multi-word canonicals (e.g. 'jarrow formulas') never boosted. Now phrase-matches; lifts the real brand over a token-matching decoy.

Harness (§6/§8 fixture): all prior GOOD/Cyrillic/dosage/honest-empty acceptance queries byte-identical or improved. Tests 29/29 search, packages/api 63/63, apps/server 5/5.

Reindex note (unchanged from #132): rebuild the DO snapshot via product.rebuildSearchIndex after deploy.

Open in Devin Review

Note

  • Replaces the per-token filter in searchMiniSearchIndex with selectMatchableTokens, which keeps tokens that have Latin alias expansions even when they produce zero direct hits, so unmatched known tokens are correctly required rather than silently dropped.
  • Fixes normalizeSearchText in text.ts to strip combining marks only from Latin-script characters, preserving Cyrillic diacritics (e.g. й, ё) while still folding Latin accents (e.g. café → cafe).
  • Fixes brand canonical scoring in scoreSearchResult to use phrase-index matching via getPhraseTokenIndex, so multi-word brand names (e.g. "Jarrow Formulas") receive the brand boost correctly.
  • Behavioral Change: multi-token queries that previously dropped tokens with no direct hits will now retain those tokens if they expand via Latin aliases, changing result sets for affected queries.

Macroscope summarized 534eefd.

Checks

3 successful · 1 skipped
All checks passed
09280c7

Added 3 commits

  • fix(search): boost multi-word canonical brand names09280c7
  • fix(search): preserve Cyrillic combining letters in normalizeSearchText3aec584
  • fix(search): keep unmatched-but-known query tokens required, not silently dropped99ab050
CcoderabbitaiBotJul 7, 2026
Synced from GitHub

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 49323a38-28e8-4513-9683-0cbbf3830314

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch orch/search-followups

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

packages/api/src/lib/product-search/core.tslines 176–185Outdated · 534eefdOpen thread
……@@ -172,6 +173,17 @@ const resultMatchesFilters = (
182+ (token) =>
183+ miniSearch.search(token, searchOptions).length > 0 ||
184+ expandLatinAliases(token).length > 1,
185+ );
Ddevin-ai-integrationBoton lines 176–185Jul 7, 2026

🔍 Known-token detection is asymmetric between alias keys and alias values

The new selectMatchableTokens at packages/api/src/lib/product-search/core.ts:176-185 keeps a token if expandLatinAliases(token).length > 1, which is true only when the token is a key in LATIN_SEARCH_ALIASES (e.g. "creatine" → length 4). However, alias values like "kreatin" or "креатин" return length 1 from expandLatinAliases because the lookup is key-based only (packages/api/src/lib/product-search/text.ts:165). This means "creatine vitamin d3" correctly returns empty (creatine is kept, AND fails), but "kreatin vitamin d3" would silently drop "kreatin" and return d3 results — the exact leak FIX 1 is meant to prevent, just via a misspelling variant. Whether this matters depends on how often users type alias forms vs canonical forms.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.