Robots.txt Generator

How to Generate and Deploy a Valid Robots.txt File

Follow this 5-step operational checklist to configure and deploy crawl directives without risking search de-indexation.

1. Select a Preset Blueprint: Choose a starting template (Standard SEO, Block AI Scrapers, WordPress, or E-Commerce) to automatically populate verified default paths.
2. Configure the AI Crawler Shield: Toggle permissions for automated LLM crawlers like GPTBot, ClaudeBot, and Google-Extended to protect proprietary intellectual property from uncompensated model training.
3. Specify Internal Paths to Disallow: Enter sensitive directories, cart checkouts, internal search results, or parameterized URLs that search spiders should not spend crawl budget exploring.
4. Validate with the Path Tester: Use the built-in simulator to verify that your business-critical landing pages return ALLOWED while staging routes return BLOCKED.
5. Deploy to Your Server Root: Download robots.txt and upload it directly to your domain's root directory (https://yourdomain.com/robots.txt).

Robots Exclusion Protocol Syntax: Directives & Wildcards

Search engine bots parse robots.txt line-by-line according to RFC 9309 standards. Understanding the fundamental directives prevents catastrophic sitewide crawl blocks:

• User-agent: Identifies which crawler the subsequent rules apply to. An asterisk (User-agent: *) targets all spiders globally, while specific names (User-agent: Googlebot) override global rules for that crawler.
• Disallow: Specifies URL paths that the designated bot is forbidden to request. An empty value (Disallow:) explicitly permits crawling of all content.
• Allow: Explicitly permits access to a specific subfolder or file located inside an otherwise disallowed parent directory (e.g., Allow: /wp-admin/admin-ajax.php).
• Wildcard Operator (*): Matches any sequence of characters in a URL string. For instance, Disallow: /*?*sort= blocks all parameterized sorting variants.
• End-of-String Operator ($): Anchors the match to the very end of the URL. E.g., Disallow: /*.pdf$ blocks spiders from crawling PDF file extensions.
• Sitemap: Declares the absolute URL of your XML sitemap index. Sitemaps should always use full https:// protocols and can be listed multiple times.

The AI Scraper Landscape: Which Bots Should You Block?

The emergence of generative AI and automated retrieval agents has created distinct classes of web bots with different business impacts:

• Training Bots (GPTBot, ClaudeBot, CCBot): These scrapers extract content to train future foundation models. Blocking them protects copyright and proprietary content without impacting Google Search rankings.
• Conversational Search Agents (ChatGPT-User, PerplexityBot): These crawlers fetch web pages in real time when an end-user explicitly prompts an AI assistant with your brand or topic. Blocking them prevents users from receiving direct answers containing links to your site.
• Google-Extended: A standalone token introduced by Google that allows webmasters to opt out of Gemini and Vertex AI training datasets without affecting Google Search crawl frequency or SEO ranking.
• Aggressive Data Scrapers (Bytespider): Crawlers operated by ByteDance that frequently execute high-frequency scraping requests that consume server CPU and bandwidth.

Platform Deployment Guide: Next.js, WordPress, & Shopify

Deploying your robots.txt file depends on your hosting architecture and content management system:

• Next.js App Router (13, 14, 15): Place a static robots.txt file in your /public directory, or export a dynamic robots() function from src/app/robots.ts returning a typed MetadataRoute.Robots object.
• WordPress: By default, WordPress generates a virtual robots.txt dynamically. You can customize it directly inside SEO plugins like Rank Math (General Settings > Edit robots.txt) or Yoast SEO (Tools > File Editor).
• Shopify: Shopify automatically serves a default robots.txt. Liquid developers can customize rules by creating a robots.txt.liquid snippet in the theme template directory.
• Apache / Nginx / Static Hosting: Upload the generated robots.txt file directly into the public_html or document root folder of your web server. If you are migrating obsolete URLs or restructuring folders, pair this with our 301 redirect rule generator to preserve backlink equity instead of disallowing paths.

Fatal Robots.txt Mistakes That Damage Search Visibility

• Accidental Sitewide Disallow: Deploying Disallow: / to production after copying staging configurations blocks all search indexers from crawling your entire domain.
• Disallowing Pages with "noindex" Tags: If you disallow a URL in robots.txt, Googlebot is blocked from crawling the page and will never see the noindex tag, causing the URL to remain in search results.
• Blocking CSS and JavaScript Assets: Disallowing paths like /css/, /js/, or /assets/ prevents Google from rendering your page layout, hurting Core Web Vitals and mobile-friendly audits.
• Case Sensitivity Errors: Robots.txt paths are strictly case-sensitive. E.g., Disallow: /Admin/ will not block /admin/.
• Subdirectory Placement: Placing robots.txt at https://example.com/blog/robots.txt is completely invalid; crawlers only check the domain root.