GEO Improvement: Add robots.txt to docs site
Source audit: GEO Audit Report — 2026-07-06
Audit date: 2026-07-06
Run: §28809885240
Finding
robots_txt score: 0/18 — found: false. The docs site at github.github.com/gh-aw/ has no robots.txt file. AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) have no explicit permission to crawl and index this site, leaving 18 points on the table — the single largest gap in the audit.
Why this matters
robots.txt is the primary signal AI crawlers check before indexing content. Without explicit Allow rules for AI bots, citation engines like ChatGPT, Claude, and Perplexity may skip the docs site entirely, making the project invisible in AI-generated answers. Fixing this is estimated to raise the docs site GEO score from 43 → 61/100 (+18 pts), lifting all 20 sitemap pages simultaneously.
Suggested fix
- Create a
robots.txt file at the root of the docs site (github.github.com/gh-aw/robots.txt) with explicit Allow directives for all major AI bots:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Bingbot
Allow: /
User-agent: cohere-ai
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: xAI-Bot
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: AI2Bot
Allow: /
User-agent: YouBot
Allow: /
User-agent: CCBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://github.github.com/gh-aw/sitemap.xml
- If the docs site is generated by a static site generator (e.g., Docusaurus, Jekyll, GitHub Pages), add the file to the
static/ or public/ directory so it is deployed to the site root.
- Verify the file is accessible at
https://github.github.com/gh-aw/robots.txt after deployment.
- Re-run
gh aw geo audit to confirm the score improvement.
Generated by 🌍 GEO Optimizer Daily Audit · 38.1 AIC · ⌖ 10 AIC · ⊞ 5K · ◷
GEO Improvement: Add
robots.txtto docs siteSource audit: GEO Audit Report — 2026-07-06
Audit date: 2026-07-06
Run: §28809885240
Finding
Why this matters
robots.txtis the primary signal AI crawlers check before indexing content. Without explicitAllowrules for AI bots, citation engines like ChatGPT, Claude, and Perplexity may skip the docs site entirely, making the project invisible in AI-generated answers. Fixing this is estimated to raise the docs site GEO score from 43 → 61/100 (+18 pts), lifting all 20 sitemap pages simultaneously.Suggested fix
robots.txtfile at the root of the docs site (github.github.com/gh-aw/robots.txt) with explicitAllowdirectives for all major AI bots:static/orpublic/directory so it is deployed to the site root.https://github.github.com/gh-aw/robots.txtafter deployment.gh aw geo auditto confirm the score improvement.