robots.txt Is a Crawler Directive, Not a Security Control — What It Can and Cannot Do
robots.txt is a text file that requests well-behaved crawlers avoid or allow specific paths. It is guidance for crawling, not a security boundary — it cannot prevent a browser or a non-compliant crawler from reaching a page, so sensitive content must be protected by authentication instead.
A robots.txt file at the root of a domain tells crawlers which paths they may or may not fetch. It is read by well-behaved crawlers — including traditional search engines and some AI crawlers — and ignored by everyone else.
What robots.txt can do
- Allow or disallow crawling of specific paths.
- Reference your XML sitemap location.
- Set rules per crawler (Googlebot, GPTBot, ClaudeBot, PerplexityBot).
- Reduce wasted crawl budget on low-value paths.
What it cannot do
robots.txt cannot hide a page from people, block a non-compliant crawler, or remove a page that is already indexed. If a URL must not be public, the correct tool is authentication or a noindex directive — not a disallow line, which can actually prevent the noindex from being discovered.
Allowing AI crawlers
Allowing AI crawlers such as GPTBot, ClaudeBot, or PerplexityBot can help your content be discovered and cited by answer engines; disallowing them prevents your content from being used for training or citation. The trade-off is editorial, not technical.
Frequently asked questions
Does robots.txt hide my pages?
No. It only asks compliant crawlers to skip them. A disallowed URL can still be indexed if linked from elsewhere, and nothing stops a human or a non-compliant crawler from viewing it.
Should I disallow to remove an indexed page?
No. A disallow can block the crawler from seeing your noindex tag, leaving the page indexed. Use a noindex meta or header and let the page be crawled so the directive is read.
About the author
RankoraXPress Team — Search Visibility Editorial
The RankoraXPress editorial team publishes original daily briefs on search visibility — covering SEO, AEO, GEO, and developer SEO. Every brief is written in-house and never scraped or copied from third-party sources.
Expertise: SEO, AEO, GEO, and developer SEO