Robots.txt Generator
Guide search engine bots on how to crawl your site.
Loading Smart Logic...
Quick Tip
Choose the user agent: * for all crawlers, or a specific bot name.
About This Tool
Why use Robots.txt Generator?
Build a robots.txt file by choosing a user agent, listing the paths to disallow and adding your sitemap URL. The file goes at the root of your domain — example.com/robots.txt — and nowhere else; crawlers do not look in subdirectories.
Robots.txt controls crawling, not indexing. A page that is blocked here can still appear in search results if other sites link to it, because Google cannot read the page to see a noindex tag. To keep a page out of the index, allow crawling and use a noindex meta tag, or require authentication.
Key Benefits
Correct syntax
User-agent, Disallow and Sitemap lines in the right order and format.
Sitemap line included
The simplest way to tell every crawler where your sitemap lives.
Per-agent rules
Write one block for all crawlers, or target a specific bot.
How to use Robots.txt Generator
- 1
Choose the user agent: * for all crawlers, or a specific bot name.
- 2
List the directories to disallow, one per line, starting with a slash.
- 3
Add your sitemap URL so crawlers can find it.
- 4
Copy the result into a file named robots.txt at the root of your site.
Best use cases
Create a robots.txt for a new site.
Block admin, cart or internal search paths from crawling.
Point crawlers at a newly published sitemap.
Write a rule for one specific bot.
Robots.txt Generator FAQs
Does robots.txt stop a page being indexed?
No. It stops crawling. A blocked URL can still be listed if other pages link to it — usually without a description. Use a noindex meta tag on a crawlable page instead.
Where does robots.txt go?
At the root of the domain, for example https://example.com/robots.txt. It applies to that host and protocol only.
Should I block CSS and JavaScript?
No. Google renders pages to judge them, so blocking assets can make your page look broken to the crawler and hurt how it is assessed.
What does Disallow: / do?
It asks the named crawler not to crawl anything on the site. Use it deliberately — leaving it in place after a launch is a common and costly mistake.
Related SEO Tools
More free seo tools that work well alongside robots.txt generator.