Robots.txt is a plain text file that sits at the root of your website and tells crawlers which parts of the site they may fetch. It is one of the oldest conventions on the web, it is read by everything from Googlebot to the AI crawlers training the latest chatbots, and it is also one of the easiest ways to accidentally wreck your own rankings. One wrong line in robots.txt can remove an entire site from Google. This guide explains how the file works, what is worth blocking, what to leave alone, and the mistakes we keep finding on real audits.
What robots.txt actually does
When a well-behaved crawler visits your site, the first thing it requests is yoursite.com/robots.txt. The file contains crawl directives: simple rules saying which user agents (crawlers) may fetch which paths. A typical robots.txt file looks like this: a User-agent line naming the bot, then Disallow and Allow lines listing paths, and usually a Sitemap line pointing to your XML sitemap.
Here is a complete, sensible robots.txt for a typical WordPress business site: a User-agent: * line applying the rules to everyone, Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php, and a Sitemap: line. Three rules, and for many sites that is genuinely all it should say.
Two things matter about the mechanism. First, it controls crawling, not indexing; that distinction does a lot of work later in this guide. Second, the rules are voluntary. Reputable crawlers follow them, but nothing forces compliance. It is a "please keep off the grass" sign, not a fence, which is why our plain-English guide to technical SEO files it under communication with search engines rather than security.
What's worth blocking
For most business sites, the honest answer is: very little. Good candidates for a disallow rule are pages that waste crawl attention without ever deserving a ranking. Internal search results pages, which can generate infinite URL variations. Cart, checkout and account areas of a store. Filtered and sorted duplicates of category pages. Staging or testing paths that accidentally became reachable. And admin areas, though your CMS usually protects those anyway.
The common thread: block pages that are genuinely useless to searchers and exist in unbounded quantities. On large sites this protects your crawl budget, the attention Google is willing to spend on you. On a twenty-page tradie site, crawl budget is a non-issue, and the best file is usually the shortest one.
What to leave alone
Never block CSS and JavaScript files. Google renders pages like a browser, and if the file hides your styling and scripts, Google sees a broken page and judges you on it. Do not block pages you want ranked, obviously, but also be careful with sweeping folder rules that catch them by accident; a single Disallow: / removes everything. And do not use crawl directives to hide thin or duplicate pages that are already indexed; blocking them just freezes them in the index as-is, because Google can no longer crawl the page to see your noindex tag.
The rule of thumb: robots.txt answers "may you fetch this URL?", nothing more. If the question you are really asking is "should this appear in Google?", the answer belongs in a noindex tag, not here.
Robots.txt and AI crawlers
The file has a new job in 2026: it is where you decide how AI systems access your content. OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended and PerplexityBot all identify themselves and respect its directives, so you can allow or refuse them per bot. Some publishers block them to keep their content out of training data. For most businesses, we recommend the opposite: AI assistants are now a discovery channel, and being readable is a precondition for being cited in AI answers. Whichever way you lean, make the choice deliberately: check your file today and see whether a developer or plugin has already made it for you, because some security and CDN tools now block AI crawlers by default. There is also a newer companion file designed specifically for AI systems; our guide to llms.txt covers what it is and whether it is worth adding alongside it.
Robots.txt is not security
This deserves its own section because it is the most dangerous misunderstanding of all. Robots.txt is a public file; anyone can read yours right now by typing the URL. Listing a "secret" folder in it does not hide the folder, it advertises the folder, and malicious bots ignore the file entirely. Worse, a blocked URL can still end up in Google: if other sites link to it, Google may index the bare address without crawling it. Google's own documentation is explicit that robots.txt is not a mechanism for keeping pages out of search results. Anything genuinely private belongs behind a password, and anything you want de-indexed needs a noindex tag or removal, not a crawl directive.
Common mistakes, and how to check yours
The failures we find on audits are rarely exotic. A developer launches a redesign with the staging site's Disallow: / still in place, and rankings evaporate over the following weeks. A folder rule written for one section quietly catches another, because rules match by prefix and a missing trailing slash widens the net. Or the file blocks a page that carries a noindex tag, so the noindex is never seen and the page lingers in results anyway. Every one of these ships silently; nothing warns you at the moment the damage is done.
Checking takes five minutes. Load yoursite.com/robots.txt and read every disallow line, asking "do I actually want this hidden from crawlers?". Then open the Pages report in Google Search Console, which flags URLs that are "blocked by robots.txt" or "indexed, though blocked". We run this exact check inside our technical SEO checklist, and it is a standard part of our technical SEO service. If anything in your file looks wrong and you are not sure why, ask us before you edit; five minutes of review beats a month of lost traffic.
