robots.txt Generator
Generate robots.txt for your website with allow/disallow rules and sitemap.
Written by Anil Gautam · Updated
How do Allow and Disallow rules decide what is blocked?
The default file has one group for all crawlers, User-agent: *, with Allow: / and Disallow lines for /admin/, /api/ and /private/, followed by a Sitemap line. Google, like other crawlers that follow the robots.txt standard, applies the most specific matching rule, meaning the one with the longest path, and Allow wins a tie. For /admin/users both Allow: / and Disallow: /admin/ match, and the 7-character Disallow path beats the 1-character Allow, so the page is blocked. Line order does not change that result, so the tool's habit of listing Allow lines first is harmless.
Paths match from the start of the URL path and are case-sensitive. Disallow: /admin/ blocks /admin/users but leaves /admin itself and /Admin/ open, so add Disallow: /admin if the bare address should be covered too, keeping in mind that it also catches /administrator. Google also reads * as a wildcard and $ as the end of a URL, so Disallow: /*.pdf$ blocks every address ending in .pdf. The + Add button starts every new row as /, and a Disallow: / left unedited blocks the entire site for that group.
Why does adding a Googlebot group unblock pages?
A crawler obeys only the group whose User-agent line matches it most specifically and ignores the others. Add user-agent group creates a Googlebot group with no rules, and once it exists Googlebot stops reading the * group, so /admin/, /api/ and /private/ become open to Google. If you want Google to skip /search/ as well, the Googlebot group needs Disallow lines for /admin/, /api/, /private/ and /search/, because nothing carries over from the * group. A group whose User-agent box is left empty is written as *.
What should I check before uploading robots.txt?
Crawl-delay is a single value that the tool writes into every group. Google ignores that directive, while some other crawlers, Bingbot among them, honour it. The Sitemap line needs a full URL starting with https://; the sitemap.xml Generator can build the file if you do not have one yet.
Download saves the text as a file named robots.txt. After uploading, run its address through the HTTP Headers Checker to confirm it returns status 200, because Google treats a 404 as if no rules existed and may crawl everything. The file is assembled in your browser and nothing is sent to a server.