Free Robots.txt Generator
Build a correct robots.txt in minutes. Start from a template, set Allow and Disallow rules per crawler, block AI training bots if you want to, and add your sitemap. It runs entirely in your browser, so nothing is uploaded.
Templates replace your current rules, then you can edit every line.
Paths should start with a /. Leaving a Disallow empty means allow everything for that crawler.
Use the full URL, including https. Sitemap lines always go at the end of the file.
User-agent: * Disallow:
Upload the file to the root of your site, so it lives at yoursite.com/robots.txt. It is advice for polite crawlers, not access control, so keep anything truly private behind a login.
Upload the finished file to the root of your site, then check it with your search console before you rely on it.
A finished robots.txt in four steps.
Start from a template, shape the rules, add your sitemaps, then copy or download the file. The output updates as you edit.
Start from a template
Pick a starting point: allow everything, block everything, block admin and private areas, or block AI training bots. You can edit whatever it fills in.
Shape your rule groups
Each group targets one crawler. Pick a user agent from the list or type your own, then add Allow and Disallow rows for the paths you care about.
Add your sitemaps
Paste the full URL of your XML sitemap, and add more lines if you have several. Crawlers use these to find your pages faster.
Copy or download
The finished robots.txt updates live as you edit. Copy it in one tap or download the file, then upload it to the root of your site.
A generator that writes the file properly.
robots.txt looks trivial until a stray line blocks your whole site. This tool keeps the structure right and flags the easy mistakes before they cost you traffic.
Templates for the common cases
Allow everything, block everything, keep crawlers out of admin and checkout pages, or block AI training bots like GPTBot and CCBot, each set up as its own group.
Correct grouping, always
Groups are separated by blank lines, every rule sits under its user agent, and Sitemap lines go at the end. The file comes out the way crawlers expect to read it.
Gentle guidance as you type
The tool nudges you when a path is missing its leading slash, and reminds you that an empty Disallow means allow everything, so small mistakes get caught early.
Private by design
It runs entirely in your browser. Nothing you type is sent to a server, so you can draft rules for an unlaunched site without telling anyone about it.
Technical SEO, done for you.
A robots.txt is one line item on a much longer list: sitemaps, redirects, structured data, page speed and a site built to rank in the first place. If you would rather hand that whole list to someone, that is the kind of web development and technical SEO work we do at Techliphant, on new builds and on sites that need fixing.
Working on your on-page SEO too? Try the meta tag generator.
Common questions.
It is a plain text file that tells crawlers which parts of your site you would rather they did not fetch. Each group names a user agent and lists Allow and Disallow rules for it. Well-behaved crawlers like Googlebot and Bingbot read it before crawling and follow the rules.
At the root of your site, so it is reachable at yoursite.com/robots.txt. Crawlers only look there. A robots.txt inside a subfolder is ignored, and the filename must be exactly robots.txt in lowercase.
No, and this catches a lot of people out. Blocking a page stops crawling, but the URL can still appear in results if other sites link to it. To keep a page out of the index, let it be crawled and add a noindex meta tag or header instead. Blocking and noindexing the same page does not work, because the crawler never sees the tag.
Yes, the major engines support both. A * inside a path matches any sequence of characters, so Disallow: /*?sort= blocks any URL with that parameter. A $ pins the rule to the end of the URL, so Disallow: /*.pdf$ blocks PDFs without touching pages that merely contain .pdf in the middle.
That is a judgement call. Blocking crawlers like GPTBot, ClaudeBot, Google-Extended and CCBot keeps your content out of future training sets, which some publishers want. The trade-off is that your content may also be less likely to be cited or surfaced by AI assistants that people increasingly use to find things. The template here sets up the groups, and you decide which ones to keep.
Only partly. Google ignores the Crawl-delay line entirely, and you set its crawl rate in Search Console instead. Bing and Yandex do respect it. If server load is the worry, it can still be worth adding, just do not expect it to change what Googlebot does.
Yes. A robots.txt only speaks for its own host, so blog.yoursite.com and shop.yoursite.com each need their own file at their own root. The one on www.yoursite.com says nothing about them, and http and https versions are treated separately too.
No. The generator runs entirely in your browser using JavaScript. Your rules, paths and sitemap URLs never leave your device, which matters if you are drafting rules for a site that has not launched yet.
Yes. A robots.txt is one small piece of technical SEO, alongside sitemaps, redirects, structured data, page speed and how your site is built in the first place. If you want a site that is set up properly for search from day one, or an audit of one that is not, that is the kind of web work we do at Techliphant.
Private by design: this generator runs entirely in your browser, so nothing you type is sent anywhere or stored. Your rules and sitemap URLs stay on your own device. It is provided free for everyday use, and it is worth testing your finished file in your search console.
Ready when you are
Let's build something exceptional.
Tell us about your business, your stack, and the problem you are trying to solve. We respond with a clear next step usually a 30-minute discovery call, no fluff.