Who needs to worry about crawl budget, and what wastes it

Which sites need to care

Google's crawl budget guide is written for large and fast-changing sites. It gives two rough guides. One is sites with 1 million or more unique pages whose content changes moderately often, about once a week. The other is sites with 10,000 or more unique pages whose content changes very rapidly, daily.

A third case is sites with many URLs marked in Search Console as discovered but currently not indexed. Google calls its page counts a rough estimate and says they are not exact thresholds. If your site is smaller and stable, this guide is probably not your bottleneck.

Google says perceived inventory is the factor a site owner controls most. Duplicate or unwanted URLs inflate it, which is why the fixes below focus on them.

The two parts: capacity and demand

Crawl capacity limit caps the total time your server spends holding connections open for Google. Every site starts with the same default, conservative limit. It rises when the site responds consistently and response times stay stable or improve. It falls when the site slows down, returns 5xx errors, or sends rate-limiting signals such as HTTP 429. Google gives no numeric latency or error-rate thresholds.

Crawl demand is how much Google wants to crawl. For Googlebot, it depends on site size, update frequency, page quality and relevance. The main factors are perceived inventory, popularity and staleness. Perceived inventory is the one you control most. Site moves can also temporarily increase demand.

The capacity limit is shared across Google's crawlers, so high demand from one can reduce capacity for others.

What wastes crawl

Google says duplicate URLs and URLs you do not want crawled waste crawl time. Its guide names these causes.

• Duplicate URLs, including pages that duplicate information found on linked pages.
• Removed or unimportant pages that are still linked or listed.
• Soft 404 pages, which Google says will continue to be crawled.
• Long redirect chains, which Google says have a negative effect on crawling. It gives no maximum chain length.
• Slow responses and server errors, which lower the capacity limit.

How Google says to manage it

• Consolidate duplicate content.
• Block unimportant URLs with robots.txt when they cannot be consolidated.
• Return 404 or 410 for permanently removed pages.
• Eliminate soft 404s. The Page Indexing report in Search Console helps find them.
• Keep sitemaps current and include lastmod for updated content.
• Avoid long redirect chains.
• Speed up pages and support 304 Not Modified responses.
• Watch for availability problems during crawling.

Common misreadings

Noindex does not save crawl budget. Google still requests the page and then drops it, so the crawl time is spent. Google says not to use noindex to manage crawl budget.

Robots.txt is for pages you never want crawled. Google says not to use it to temporarily free budget for other pages, because it will not shift that budget unless you are already hitting your capacity limit.

A 404 does not make Google forget a URL. Google says a 404 is a strong signal not to crawl it again. Blocked URLs stay in the crawl queue longer.

What to do

First check whether this applies. Compare your unique page count and update rate with the figures above, and look in Search Console for many URLs that are discovered but not indexed.

If it applies, start with the cheapest wins from Google's own list: fix soft 404s, shorten redirect chains, remove dead URLs from sitemaps and internal links, and consolidate duplicates. Then check server speed and errors, since those set the capacity limit.

After a fix, recheck the Page Indexing report in Search Console to confirm the soft 404s and unwanted URLs are gone.

Keep notes on what you changed and when. That makes it easier to link a later change in crawling to a specific fix.

Sources

1. [1]
   Managing crawl budget for large sites

   Google Search Central. Size guidance, capacity limit, crawl demand, causes of waste, management steps and the noindex, robots.txt and 404 statements.
2. [2]
   How to specify a canonical URL with rel=canonical and other methods

   Google Search Central. Consolidating duplicates also helps crawl efficiency.
3. [3]
   Introduction to robots.txt

   Google Search Central. Robots.txt manages crawl traffic.
4. [4]
   Block Search indexing with noindex

   Google Search Central. Noindex needs a crawl to be seen; Page Indexing report shows noindex pages.

Sources were last checked on October 13, 2026. Documentation changes, so confirm any detail that matters to a decision against the source itself.

Indexive Research. (2026, August 19). Who needs to worry about crawl budget, and what wastes it. Indexive AI. https://indexive.ai/research/crawl-budget-explained