Developer SEO
    Free · in-browser · no signup

    Robots.txt Generator

    Build a robots.txt with allow/disallow groups, sitemap reference, and AI bot rules. Never a security boundary.

    Direct answer

    The Robots.txt Generator builds a valid robots.txt with User-agent groups, Disallow/Allow rules, a Sitemap directive, and optional AI crawler rules for GPTBot, ClaudeBot, and PerplexityBot.

    Try the tool

    robots.txt

    User-agent: *
Allow: /

# AI crawlers — allow for discovery, disallow for scraping content
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: CCBot
Disallow: /

Sitemap: https://rankoraxpress.com/sitemap.xml

    What this tool generates

    It produces a robots.txt file with a default User-agent: * group, your disallow paths, a Sitemap: directive, and — when enabled — explicit allow rules for AI crawlers. The output is plain text you can save as /robots.txt.

    Robots.txt is a directive, not a security boundary

    robots.txt tells well-behaved crawlers what they may fetch. It does not prevent access — malicious or non-compliant crawlers can ignore it. Never use robots.txt to protect private content; use authentication and authorization instead.

    How to use the output

    Save the generated text as /robots.txt at the root of your domain. Reference your XML sitemap so crawlers can discover all indexable URLs. Do not accidentally block CSS, JavaScript, or images that are required for rendering.

    How to use the Robots.txt Generator

    1. 1

      Set the default User-agent group

      Start with a User-agent: * group that applies to all crawlers, then add specific groups for AI bots like GPTBot or ClaudeBot if you want different rules.

    2. 2

      Add your disallow paths

      Disallow private, admin, or non-public routes such as /admin, /login, and internal API paths. Do not disallow CSS, JS, or images needed for rendering.

    3. 3

      Reference your sitemap

      Add a Sitemap: directive pointing to your https sitemap.xml so crawlers can discover all indexable URLs.

    4. 4

      Save the file at the root

      Upload the generated text as /robots.txt at the root of your domain and verify it is reachable at https://yourdomain.com/robots.txt.

    Examples

    Allow public content, block admin

    User-agent: * / Disallow: /admin / Disallow: /login / Sitemap: https://example.com/sitemap.xml — the standard pattern for a public site.

    Allow AI crawlers explicitly

    Add a User-agent: GPTBot group with Allow: / to signal that OpenAI's crawler may fetch your public content for potential citation.

    Common problems & fixes

    Accidentally blocking CSS or JavaScript

    Fix: Never disallow /assets/, /static/, or file extensions like .css and .js — blocking them can break rendering and indexing.

    Using robots.txt to protect private data

    Fix: robots.txt is not a security boundary. Protect private content with authentication and authorization; disallowed paths are still publicly reachable.

    No Sitemap directive

    Fix: Add a Sitemap: line so crawlers discover your indexable URLs automatically instead of relying on link discovery alone.

    Frequently asked questions

    What is robots.txt?

    robots.txt is a text file at the root of a domain that tells crawlers which paths they may or may not fetch. It is a directive, not a security control.

    Does robots.txt block access?

    No. It requests that well-behaved crawlers avoid listed paths. It does not prevent access by users or non-compliant crawlers.

    Should I allow AI crawlers?

    It depends on your goals. Allowing GPTBot, ClaudeBot, and PerplexityBot can help your content be surfaced and cited by AI answer engines; disallowing them prevents your content from training or being cited by those systems.

    Related free tools

    Audit your site in seconds

    Run a free SEO, AEO, and GEO audit. See exactly what to fix and why.

    No signup required · RankoraXPress