Generate and check a robots.txt with crawler presets, and understand what Disallow does and does not prevent — it is not access control and not deindexing.
1 group
1 things worth knowing.
Absolute URLs, including the scheme and host. Unlike Allow and Disallow, which are paths, a relative Sitemap line is ignored.
User-agent: *
Allow: /
robots.txt is a request that well-behaved crawlers honour and nothing else does. It is not access control, and a disallowed URL can still appear in results if other sites link to it.
robots.txt controls crawling, not indexing, and it is not access control. Blocking a URL here does not remove it from search results and actively prevents a noindex tag on that page from ever being discovered. The file is public, so listing a path you want hidden advertises it. Anything that must not be read needs authentication.
robots.txt Generator runs entirely on your device — nothing is fetched or uploaded.
You’re on the Free plan. Compare plans
robots.txt is misunderstood in two specific ways, and both make it onto the page rather than into a footnote. It is not access control: the file is public, it lists the paths you would rather people not visit, and a crawler that ignores it faces no obstacle. Anything that must not be read needs authentication.
It is also not a way to remove a page from search results. A disallowed URL can still be indexed — from links elsewhere — and because the crawler may not fetch it, it cannot see a `noindex` tag either. Disallowing a page you want deindexed is the reliable way to keep it in the results with no description. The `noindex` header or meta tag is the tool for that, and it requires the page to be crawlable.
The AI crawler presets are included with the honest caveat: they work by naming user agents, so they only affect crawlers that identify themselves and choose to comply. The list of names changes, and one that does not identify itself is unaffected by anything in this file.
The checker flags what genuinely breaks: `Disallow` lines before any `User-agent`, which belong to nothing; a sitemap reference that is not an absolute URL, which is required; and `Crawl-delay`, which Google has never supported and ignores.
Nothing is fetched. The file is generated and parsed in your browser, so this cannot tell you whether a crawler is obeying it.
No. A disallowed URL can still be indexed from links elsewhere, and because the crawler may not fetch it, it cannot see a noindex tag either — so you get a result with no description. Use a noindex header or meta tag, and leave the page crawlable so it can be read.
The opposite. The file is public and lists the paths you would rather were not visited, which makes it a map. A crawler that ignores it faces nothing. Anything that must not be read needs authentication.
Only for crawlers that identify themselves honestly and choose to comply. The presets name the known agents, that list changes, and a crawler that does not identify itself is unaffected by any of it.
Not for Google, which has never supported it. Some other crawlers honour it. For Google, crawl rate is managed in Search Console rather than in this file.
Yes. The Sitemap directive requires a full URL including the scheme and host, unlike the path-relative Disallow rules. A relative path there is the most common error in a robots.txt file.