Web tool
robots.txt Builder
Allow the site, block a few paths, and copy the robots.txt text.
Processed in your browser. The text is not sent to our servers.
How to build a robots.txt file
- Leave “Allow the whole site” on when crawlers may read the public pages.
- List any path to block, one per line, starting with /.
- Add a sitemap URL if you have one, then copy the file.
Allow: / with Disallow: /admin tells a crawler the site is open except that path. The page does not upload the file or fetch your site.
Frequently asked questions
Does this test a live site?
No. It writes the file from the choices you make. It does not request the URL.
What if I leave the blocked paths empty?
With the allow box on, the file allows the whole site. Turn the box off and leave the paths empty to allow everything with a blank Disallow line.
Related web tools
Keep working without leaving the site.
UTM builder
Add source, medium, and campaign to a URL.
Open toolMeta length
Count title and description characters against the usual limits.
Open toolURL parser
Split a URL into protocol, host, path, query, and hash.
Open toolHTTP status
Read the meaning of a common status code, such as 404.
Open toolUser agent
Read the browser, system, and device from a user agent string.
Open toolMIME types
Look up a file extension or a MIME type.
Open toolQuery builder
Add name and value pairs to a URL.
Open toolMeta tags
Write a title, description, and canonical link to copy.
Open tool