Crawl budget is one of those SEO terms that generates far more anxiety than it deserves. The short version, straight from Google: most websites do not need to worry about it at all. But if you run a large site, an online store with thousands of product URLs, or a site where new pages take weeks to appear in Google, crawl budget is worth understanding properly. This guide explains what it is, who genuinely needs to care, where crawling gets wasted, and what to fix if your site is in the group that does.
What crawl budget actually is
Googlebot, the crawler that fetches pages for Google's index, cannot crawl the entire web constantly. It has to decide how many pages to fetch from your site and how often. Crawl budget is the practical result of that decision: the amount of crawling attention your site gets in a given period.
Google's documentation describes it as the product of two forces. Crawl capacity is the ceiling: how many connections Googlebot can open to your site without slowing it down for real visitors. If your server responds quickly and reliably, the ceiling rises; if it slows or throws errors, Googlebot backs off. Crawl demand is the appetite: how much Google wants to crawl your URLs, driven by how popular they are, how often they change, and how stale Google's copy has become. Put simply, crawl budget grows when your site is fast and your pages are worth revisiting, and shrinks when either is untrue.
The figure is different for every site, and it moves. Publish often, earn links, keep the server quick, and Googlebot visits more; let the site drift, slow down, or fill up with near-identical URLs, and the visits thin out. Nothing about it is fixed or guaranteed, which is exactly why the word budget fits so well.
Does your site actually need to care?
For most businesses, no. Google is direct about this in its crawl budget documentation: it is a concern for very large sites, roughly a million unique pages and up, and for medium-to-large sites, around ten thousand pages and up, whose content changes very rapidly. It also matters for sites where Search Console flags a large share of URLs as Discovered, currently not indexed, which means Google knows the pages exist but has not got around to crawling them.
If your site is a few hundred pages, Googlebot can crawl the whole thing without breaking a sweat, and hours spent optimising crawl budget are hours taken from work that would actually move rankings. The exceptions are predictable: eCommerce catalogues with faceted navigation, publishers with huge archives, classifieds and listing sites, and the enterprise SEO tier generally, where URL counts run into the millions and crawling really is a constraint worth managing.
The honest test: if you can count your pages in hundreds, crawl budget is not your problem. If you count them in hundreds of thousands, or new content routinely takes weeks to be indexed, keep reading, because the fixes below are for you.
Where crawl budget gets wasted
On large sites, the crawler's time leaks away into URLs that should never have been crawlable in the first place. Google's own list of crawl-wasters starts with faceted navigation and session identifiers: every combination of filters, sorts and parameters mints a new URL, and a catalogue of two thousand products can quietly generate two million crawlable addresses. Add on-site duplicate content, soft 404 pages that return a friendly page instead of an error status, infinite calendar and pagination spaces, and long redirect chains, and Googlebot can spend most of its visit fetching nothing of value.
A concrete example makes the scale obvious. Take a store with twenty filter options across size, colour, brand and price. Allow them to combine freely and you have more URL permutations than products by several orders of magnitude, every one of them crawlable unless something says otherwise, and almost none of them worth indexing.
The cost is not a penalty; it is delay. Crawling wasted on junk URLs is crawling not spent on the new product page or updated service page you actually want indexed. That is why the symptom of a crawl budget problem is almost always lag: content that takes weeks to appear, or updates that take weeks to be reflected in results.
How to check what Googlebot is doing
You do not have to guess. The Crawl stats report in Google Search Console, tucked under Settings, shows total crawl requests, average response time, and breakdowns by response code and file type. A rising average response time or a growing share of 404s and redirects is an early warning. Pair it with the Pages report: a large and growing pile of URLs at Discovered, currently not indexed is the classic signature of crawl demand outstripping crawl budget. Server log analysis goes deeper still, showing exactly which URLs Googlebot fetches and how often, which is standard work in a serious technical SEO engagement.
One caution when reading the numbers: more crawling is not the goal in itself. Google is explicit that a higher crawl rate is not a ranking signal, so the aim is not a bigger number in the report but the right pages being fetched promptly. A site can be crawled half as often and rank better, provided the crawling lands where it counts.
How to fix it: cut waste, raise capacity
The playbook follows the two levers. First, cut the waste. Block filter and search-result URLs in robots.txt so crawlers never enter the maze. Consolidate duplicates and return proper 404 or 410 status codes for pages that are gone, rather than soft 404s. Flatten redirect chains so each hop stops costing a fetch. Keep your XML sitemap clean and current, because it is one of the strongest signals of which URLs deserve crawling, and accurate lastmod dates tell Googlebot what changed.
Choose the blocking tool deliberately. Robots.txt stops crawling but not indexing, so a blocked URL can still appear in results off the back of links alone, while noindex has to be crawled to be seen. For crawl waste specifically, robots.txt is usually the right lever, because the whole point is stopping the fetch from happening at all.
Second, raise the ceiling. Faster, more reliable hosting lifts crawl capacity directly, since Googlebot fetches more when responses come back quickly and without errors. That is the same server speed work that improves user experience and Core Web Vitals, so it pays twice. And remember the demand side responds to quality: Google crawls more of sites whose pages are popular, current and worth indexing, which means pruning low-value pages helps crawling and rankings at the same time.
Running a big site? Our technical SEO service includes crawl analysis, log review and index-bloat cleanup for large and eCommerce sites. Talk to us and we will show you where Googlebot's time is going.
