Robots.txt is a plain-text file at the root of a domain (/robots.txt) that tells crawlers which parts of the site they are allowed to fetch, using per-bot rules like User-agent and Disallow.
Why it matters
Robots.txt used to be a purely defensive SEO file; in the AI era it is a strategic one. It is where you decide whether AI crawlers, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, can access your content at all. Block them and you protect your content from training, but you also opt out of AI citations and disappear from AI-generated answers. For a SaaS chasing AI visibility, accidentally disallowing these bots is a silent kill switch: your GEO work produces nothing because the engines physically cannot read your pages. Note that robots.txt controls crawling, not indexing, a disallowed URL can still appear in results if others link to it.
How to use it
- Audit your file today: confirm GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot are not disallowed on the content you want cited.
- Reference your XML sitemap with a
Sitemap:line so every crawler finds your URL list immediately. - Use Disallow to save crawl budget on junk paths (search results, parameters, admin), never to hide sensitive data, since the file itself is public.
Related terms
Paul-Marie is the founder of Readyt, the Reddit growth platform for SaaS. He has generated 16K€+ in pre-sales in 2 months using nothing but Reddit, and now helps founders turn Reddit threads into their #1 acquisition channel.
