feat(docs): expand robots.txt with full AI crawler coverage#43843
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
Great work expanding the robots.txt with comprehensive AI crawler coverage! 🎉 The new entries are well-organized — 9 additional bots added with clear identifiers — and
|
🤖 PR Triage
Summary: DRAFT. Adds 9 missing AI crawler entries to
|
There was a problem hiding this comment.
Pull request overview
This PR updates the documentation site’s robots.txt and its corresponding Playwright test to explicitly allow additional crawler user agents, expanding the list of bots granted access while keeping the sitemap reference intact.
Changes:
- Added explicit
User-agent+Allow: /blocks for 9 additional crawler user agents indocs/public/robots.txt. - Updated the Playwright expectation in
docs/tests/robots-txt.spec.tsto match the expandedrobots.txtcontents.
Show a summary per file
| File | Description |
|---|---|
| docs/public/robots.txt | Adds additional explicit allow entries for more crawler user agents and retains sitemap index reference. |
| docs/tests/robots-txt.spec.ts | Updates the expected robots.txt string to match the expanded crawler list. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Low
|
🎉 This pull request is included in a new release. Release: |
The GEO audit flagged missing explicit
Allowdirectives for major AI crawlers, leaving indexing permission ambiguous for citation engines (ChatGPT, Claude, Perplexity, etc.).Changes
docs/public/robots.txt— Added 9 missing AI crawler entries:Claude-SearchBot,Bingbot,cohere-ai,DuckAssistBot,xAI-Bot,Amazonbot,AI2Bot,YouBot,CCBotdocs/tests/robots-txt.spec.ts— Updated expected content to match the expanded bot list