Allow All Crawlers
Allows search engines to crawl the website normally.
Robots.txt Settings
Presets
Generate a robots.txt file to see validation notes.
Generated Robots.txt
Important
Robots.txt can guide search engine crawlers, but it does not protect private content. Do not use robots.txt as a security tool.
Where To Upload
Upload the file to your root domain, for example:
https://cheesebridge.com/robots.txt
Search engines may interpret robots.txt rules differently. Always test important changes using search engine webmaster tools.
What Is A Robots.txt File?
A robots.txt file is a text file placed at the root of a website to give instructions to search engine crawlers. It can tell bots which parts of a site they are allowed to crawl and which paths should be avoided.
What Is A Robots.txt Generator?
A robots.txt generator helps create the correct text file without writing rules manually. It can generate common rules for allowing all crawlers, blocking all crawlers, adding sitemap links, disallowing private paths and creating website-specific crawl instructions.
Common Robots.txt Rules
Block All Crawlers
Requests that crawlers avoid the entire website. Useful for staging sites, but not a security measure.
Disallow Admin Pages
Prevents crawlers from wasting crawl budget on admin, login or internal paths.
Add Sitemap URL
Helps search engines discover your sitemap location more easily.
Custom User-agents
Creates rules for specific crawlers such as Googlebot, Bingbot or other named bots.
Crawl Delay
Suggests a delay between crawler requests. Not all crawlers support crawl-delay.
Example Robots.txt File
User-agent: *
Allow: /
Sitemap: https://cheesebridge.com/sitemap.xmlRobots.txt Is Not Security
A robots.txt file does not hide or protect private information. Anyone can visit
/robots.txt and read the rules. If a page contains private content, it should be protected with
proper authentication, server-side access controls or removed from the public web.
Frequently Asked Questions
Where should robots.txt be placed?
It should be placed at the root of your domain, such as https://cheesebridge.com/robots.txt.
Can robots.txt block Google?
Yes, you can request that Googlebot does not crawl certain paths, but this does not guarantee deindexing.
Does robots.txt remove pages from Google?
No. Robots.txt controls crawling, not necessarily indexing. Use noindex or removal tools for deindexing.
Should I include my sitemap?
Yes, adding a sitemap URL is a common best practice.
Does crawl-delay work for Google?
Google does not use crawl-delay in robots.txt. Other crawlers may support it.
Does this generator upload my rules?
No. The robots.txt generator runs locally in your browser using JavaScript.