Where should robots.txt go?

At the root of your domain, served at https://yoursite.com/robots.txt.

Does Disallow remove pages from Google?

It blocks crawling, not indexing. To remove a page from results, use noindex meta or remove it from your sitemap.

Utilities Tool

Generate robots.txt for your website with allow/disallow rules and sitemap.

Privacy-first workflows

Fast and simple

100% Free, no signup

Group 1

Allow paths

Disallow paths

Sitemap URL

Crawl-delay (seconds, optional)

robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

How it works

STEP 01

Add user-agent groups (* for all crawlers, Googlebot for Google only).

STEP 02

List allow and disallow paths.

STEP 03

Add your sitemap URL.

STEP 04

Download and place at the root of your domain as /robots.txt.

You might also need

FAQ