Sign up
View stargazersSign up

Let search and AI crawlers in, ping IndexNow on deploy, and add an AI info page#105

merged5
stack/critique-usecases
stack/seo-tech
openedOct 9, 2026byAAviddaram
16 files changed
View on GitHubTwo-way sync enabledPR last imported 20d ago

Production crawlers find no sitemap and robots.txt says little. This names search and AI crawlers in robots.txt and keeps them out of the app, adds an IndexNow key plus a ping after production deploys, gives sitemap entries a lastmod, fills in missing titles, descriptions and og tags, and adds /ai and /llms-full.txt with plain facts for assistants.

1robots.txt Googlebot, Bingbot, OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot... allow /, disallow /api /rpc /d/ /deliveries /sign-in
2deploy.yml main only: verify key file → POST sitemap URLs to api.indexnow.org (continue-on-error)
3vite.config.ts sitemap lastmod = last commit touching src
4routes/ai.tsx facts from plan catalog, WebPage JSON-LD

Layer 4 of 7.

Checks

1 failing · 3 successful · 2 skipped
Some checks failed

Failing1

1a5ec02

Added 5 commits

  • Name search and AI crawlers in robots.txt and keep them out of the app153a180
  • Add /ai and /llms-full.txt: plain facts for AI assistants1a5ec02
  • Fill in titles, descriptions and og tags the audit found missing7b4fd3f
  • Give sitemap entries a lastmod from the last page-source commitf316011
  • Ping IndexNow with the sitemap after each production deployf692dc2
CcoderabbitaiBotOct 9, 2026
Synced from GitHub

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 63c369e2-7b81-4177-95c0-636f2b6c4e94

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.