Robots.txt Generator
Create a robots.txt file and test URL paths against it.
robots.txt tells well-behaved crawlers what not to crawl. It is not access control: blocked pages stay publicly reachable, and a page can still appear in search results if other sites link to it.
Test a URL path
Write a robots.txt file and check what it blocks
Robots.txt Generator builds the plain-text file that tells crawlers which parts of a site to stay out of. You set up user-agent groups with Allow and Disallow rules, add your sitemap addresses, and the file appears below as you type.
Test a URL path then checks any path against those rules for a crawler you name. Everything happens on this page; nothing you enter is sent anywhere.
How to create a robots.txt file
- Choose a starting point next to Start from: Allow all, Block all, or Block some folders — the example loaded when the page opens.
- In each group, fill in User agent: * means every crawler, and names like Googlebot are suggested.
- Click Add rule, pick Disallow or Allow, and type a path starting with /, such as /checkout/. Add a user-agent group when one crawler needs different rules.
- Enter any Sitemap URLs, one full address per line.
- Clear anything marked Error, then click Copy or Download robots.txt and upload the file to the top level of your site, so it opens at an address like https://www.example.com/robots.txt.
Reading the file and the test verdict
- One block per group: a User-agent line and its rules in the order you added them, with Sitemap lines at the end.
- Rule-free groups are written as a bare “Disallow:”, which blocks nothing.
- Errors and notes: flag a missing user agent, a path not starting with / or *, a sitemap that isn’t a full URL, or a space in a path. A group with no user agent is left out of the file until you fill one in.
- The verdict: Allowed or Blocked, naming the deciding rule and its group, or saying that no rule matched.
Situations where robots.txt helps
- Low-value pages: ask crawlers to skip admin screens, carts or internal search results.
- Before a launch: make sure a leftover “Disallow: /” from a staging site isn’t shutting crawlers out.
- Per-crawler rules: give a bot such as CCBot its own group, which works only if it honours the file.
- Sitemap discovery: list your sitemap so crawlers can find it; the Sitemap Generator can create one.
Writing paths that match what you intend
- End a folder path with a slash. Disallow: /admin/ covers that folder, while Disallow: /admin also catches /admin-login.
- Paths are case-sensitive, so /Private/ and /private/ need separate rules.
- Test the exact path a visitor would request, including any query string, such as /search?q=shoes.
How crawlers read the rules
A request to crawlers, not a lock
robots.txt is guidance that well-behaved crawlers choose to follow. A blocked URL still opens for anyone who has the address, and a crawler that ignores the file can fetch it anyway. The file is public too, so listing a hidden folder in it advertises where that folder is.
Which rule wins
The tester follows RFC 9309, the robots.txt standard: within the group that applies, the longest matching path wins, and Allow beats Disallow at equal length. That’s how the folders preset blocks /admin/ yet allows /admin/public/. In a path, * stands for any characters and $ marks the end, so /*.pdf$ covers addresses ending in .pdf. /robots.txt itself is always allowed.
Robots.txt Generator: common questions
Will a Disallow rule keep a page out of search results?
Not reliably. Compliant crawlers won’t fetch the page, but a search engine may still list its URL, usually without a description, when other pages link to it. To keep a page out, let it be crawled and add a noindex robots meta tag — the Meta Tag Generator can write one — or put it behind a login.
If Googlebot has its own group, do the * rules still apply to it?
No. A crawler uses the group that names it (ignoring case) and falls back to * only when none does, so repeat any * rules it should still follow. The tester’s verdict says which group was used.
Can I test a robots.txt file I already have?
Not by pasting it in — the tester only checks rules built on this page, so recreate your existing rules here first.
Is anything I enter uploaded?
No. The file is assembled in your browser and Download robots.txt saves it from the page directly.