Robots.txt generator.

What is a robots.txt generator?

A robots.txt generator builds the robots.txt file that tells search engine crawlers which parts of your website they may crawl and which parts they must skip. Add your rules in the form above, and the robots.txt generator outputs a valid file you can upload to your site root in seconds. No syntax memorization, no formatting mistakes.

How robots.txt actually works

When Googlebot arrives at your site, the first file it requests is robots.txt from your domain root. This plain text file lists user agents, the crawlers the rules apply to, followed by allow and disallow directives for URL paths. A rule like "Disallow: /admin/" tells every crawler to skip everything under the admin directory. The crawler obeys voluntarily. Robots.txt is a protocol of cooperation, not a security mechanism. Anyone can ignore it, and malicious bots routinely do.

This distinction shapes every decision you make with the file. Use robots.txt to manage crawl budget and keep junk pages out of the crawl queue. Never use it to hide sensitive content. Anything that must stay private needs authentication, not a disallow rule. Our generator makes the syntax foolproof, but the strategy behind the rules still needs human judgment.

Robots.txt versus noindex: when to use each

This is the most misunderstood pair in technical SEO. Robots.txt blocks crawling. The noindex meta tag blocks indexing. They solve different problems and they conflict with each other. If you disallow a page in robots.txt, Googlebot never crawls it, which means it never sees the noindex tag on it. The page can then linger in the index indefinitely, showing as a URL without a title or description.

The rule is simple. To keep a page out of Google's index, use noindex and let Google crawl it. To save crawl budget on pages that do not need crawling at all, like faceted filter URLs or internal search results, use robots.txt disallow. Never apply both to the same URL. When our team audits a site, conflicting robots and noindex directives are among the first issues we check, because they are common and entirely self inflicted.

How to use this robots.txt generator

The form starts with sensible defaults for common blocks. To add your own rule, type the user agent, usually an asterisk for all crawlers, enter the path to allow or block, and toggle between Allow and Disallow. Click Add rule and the generated file updates instantly. Rules group automatically by user agent, keeping the output clean and readable.

Add your sitemap URL in the field below the rules. This single line dramatically improves discovery, especially for new or frequently updated sites. When the file looks right, copy it and upload it as robots.txt to your domain root, so it resolves at yoursite.com/robots.txt. Then open Google Search Console's robots.txt tester and verify that important pages show as allowed and junk pages show as blocked before you consider the job done.

What to block and what to never block

Block liberally: admin dashboards, login and registration pages, shopping carts and checkouts, internal site search results, tag and filter URLs that generate infinite combinations, staging and development subdomains, and API endpoints. Every one of these consumes crawl budget while contributing nothing to rankings. On large sites, blocking parameter hell, the endless URL variations from sorting and filtering, can free up dramatic amounts of crawl capacity for the pages that matter.

Never block your CSS, JavaScript or image files. Google needs these to render pages the way users see them. Blocking resources triggers "blocked resources" warnings in Search Console and can distort how Google evaluates your page quality. Also never block your sitemap, your homepage, or any page you want indexed. It sounds obvious, but inherited rules from old developers cause exactly this damage more often than anyone admits. After generating your file, read every line once more with fresh eyes.

Crawl budget: why it matters for larger sites

Google allocates each site a crawl budget based on its authority and server capacity. Small sites rarely hit the limit. Sites with tens of thousands of URLs do, constantly. When junk pages consume the budget, important new pages wait days or weeks for their first crawl, and updated pages keep stale content in the index. A tight robots.txt is the cheapest crawl budget optimization available. Our generator helps you write it correctly, and our technical SEO audits identify exactly which URL patterns deserve to be blocked on your site.

Advanced robots.txt patterns

Beyond basic allow and disallow, the file supports wildcards that handle complex sites. An asterisk matches any sequence of characters, so "Disallow: /*?" blocks every URL containing a question mark, killing parameter URLs in one line. The dollar sign anchors the end, so "Disallow: /*.pdf$" blocks PDFs specifically. These patterns demand care: one greedy wildcard can block half your site. Test each pattern against real URLs in Search Console before deploying.

You can also set crawl delay for specific bots, though Google ignores it. Bing and some other crawlers respect it. If a aggressive bot hammers your server, a targeted user agent block solves the problem faster than any delay directive. Identify the bot from your server logs, add its exact user agent string, and disallow everything for it. Keep a comment in the file noting why, so future developers do not remove a rule they do not understand.

Common robots.txt mistakes

The nightmare scenario is "Disallow: /" on a live site, which blocks the entire domain. It usually arrives via a staging file copied to production during a migration. Always verify robots.txt immediately after any deploy or migration. The second classic is case sensitivity: "Disallow: /Admin/" does not block "/admin/". Paths match exactly, so audit your real URL casing before writing rules.

Another frequent error is blocking with the wrong user agent syntax, like "User-agent: google" instead of the correct "Googlebot". Unknown agent names match nothing, so the rules silently do nothing and the site owner believes they are protected. Our generator uses the standard asterisk default to avoid this trap. For the complete specification, Google's robots.txt documentation covers every directive and edge case.

Robots.txt for B2B SaaS sites: specific patterns

B2B SaaS sites generate distinctive crawl traps. Documentation portals with versioned URLs create near duplicate sets for every release. Help centers with search and filter combinations spawn thousands of thin pages. Marketing sites with UTM tagged URLs let crawlers discover the same page under dozens of addresses. Pricing pages with currency parameters multiply further. A robots.txt built for a SaaS site blocks these patterns specifically: version archives beyond the current docs, help center search results, and any URL carrying tracking parameters.

The docs versioning case deserves care. Old documentation versions serve existing customers, so you may want them indexed for support queries. But each version multiplies your crawlable URL count. The balanced approach keeps the current version fully crawlable, noindexes outdated versions so they can still serve direct visitors, and uses robots.txt only for the truly useless variants like print views and JSON endpoints. Map your URL patterns first, then write the rules to match reality.

Monitoring and maintaining your robots.txt

Robots.txt is not a write once file. Every site redesign, migration, CMS change and new section can invalidate old rules or demand new ones. Build two habits. First, check the file in Search Console monthly and review the crawl stats for blocked URLs that should be allowed. Second, add robots.txt review to your launch checklist for every deploy. The five minutes it takes have saved countless sites from the staging file disaster.

Keep the file self documenting with comments. A line like "# Blocked 2026-03: faceted nav was consuming 40 percent of crawl budget" tells the next developer exactly why the rule exists and when to revisit it. Version control the file alongside your code so changes get reviewed like any other infrastructure change. When crawl issues get complex, with international subfolders or JavaScript rendered content, our team handles the full technical audit so your rules match your architecture instead of fighting it.

Robots.txt vs meta robots vs noindex: when to use each

Use this decision rule and you will never confuse the three again. Use robots.txt disallow when you want to save crawl budget on URLs nobody should crawl: internal search results, faceted filters, admin pages. Use noindex when you want a page out of Google's index but you are fine with Google crawling it: thank you pages, thin tag archives, staging pages that must stay reachable. Deliver the noindex through the meta robots tag or the X Robots Tag HTTP header, since both work for HTML and non HTML files like PDFs.

The critical conflict to avoid is combining disallow with noindex on the same URL. A disallowed page never gets crawled, so Google never sees its noindex tag, and the URL can sit in the index as a bare link for months. Pick one mechanism per URL based on your goal. Crawl control means robots.txt. Index control means noindex with crawling allowed. Removal of already indexed URLs means noindex first, then optional disallow after they drop. Google's robots.txt specification documents exactly how crawlers interpret each directive. When the choice affects revenue pages, talk to our team or let our technical SEO team map every URL pattern to the right mechanism before any rule ships.

How to test your robots.txt in Google Search Console

Every robots.txt change deserves a test before you trust it. Open Google Search Console, go to Settings, then the robots.txt section. Google shows the exact file it fetched, when it fetched it, and flags syntax it cannot parse. Because Google caches robots.txt, typically for up to 24 hours, this screen also tells you whether Google has seen your latest edit yet. If it shows the old file, wait for the next fetch rather than assuming your change is live.

Next, verify individual URLs with the URL Inspection tool. Enter any important page and check the crawl status: it states plainly whether the URL is allowed or blocked by robots.txt. Test a sample from each category: homepage, a money page, a blocked admin path, a blocked parameter URL. If a page you need indexed shows as blocked, fix the rule immediately. This two step check, file level in Settings plus URL level in Inspection, catches nearly every robots.txt disaster before it costs traffic. Google's Search Console help walks through both screens. Pair it with a run through our free SEO tools, and for a deeper safety net our technical audits include automated robots.txt monitoring that alerts you the moment a deploy breaks the file.

Robots.txt for WordPress, Shopify and Next.js

Each platform handles robots.txt differently, and the wrong assumption breaks the file. WordPress generates a virtual robots.txt automatically. The single most important setting is Settings then Reading, where "Discourage search engines from indexing this site" adds a sitewide disallow that people forget to uncheck after launch. For custom rules on WordPress, use an SEO plugin or place a physical file in the site root, which overrides the virtual one.

Shopify controls robots.txt through a robots.txt.liquid template in your theme code, so you edit the template rather than uploading a file. Next.js ships no default at all: add a static file at public/robots.txt or generate one with app/robots.ts. A minimal safe starting point for most sites looks like this:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Whatever the platform, the file must resolve at the domain root and return plain text. After any platform migration, recheck the file, because migrations are the number one source of robots.txt disasters. We cover platform migration pitfalls on our blog. Google documents the full robots.txt specification including sitemap declarations. If your setup spans subdomains or a headless frontend, talk to our technical SEO team before launch day.

Crawl budget optimization for large sites

Crawl budget is the number of URLs Googlebot will crawl on your site in a given period. Google defines it through two factors: crawl capacity, what your server can handle, and crawl demand, how much Google wants your pages. Small sites never think about it. Sites with more than 50,000 URLs live or die by it. When low value URLs consume the budget, new pages wait weeks for discovery and price or inventory updates go stale in the index.

The highest return fixes are all robots.txt work. Block internal search result pages, faceted navigation combinations, session IDs and tracking parameters, and legacy versioned documentation. Each blocked pattern returns crawl capacity to pages that earn revenue. Measure the effect in Search Console's crawl stats report: blocked URL crawls should fall while crawls of important pages rise. Google's crawl budget guide details exactly how capacity and demand interact. For sites where crawl waste hides in JavaScript rendering or infinite calendars, our technical SEO audits run the full log file analysis, or contact us and we will rewrite the rules to match your architecture.