indexive
ServicesSolutionsIndustriesResearch
Log inFree execution audit
GuidePublished August 19, 2026·Sources checked October 14, 2026·22 min read

Who needs to worry about crawl budget and what wastes it

Crawl budget is the set of URLs Google can and wants to crawl, set by a server-driven capacity limit and by demand. Google says its guide is meant for sites with 1 million or more pages changing weekly, 10,000 or more pages changing daily, or many URLs stuck in Discovered - currently not indexed. For everyone else, a current sitemap and a healthy server are enough. This guide covers what wastes crawling, how to read the Crawl Stats report and server logs, and how to tell a crawling problem from an indexing problem.

By Indexive Research · every source is linked at the end
Key points
1 million+
unique pages, changing weekly, where Google says its crawl budget guide applies
10,000+
unique pages, changing daily, the second size Google names
1 to 2 days
longest Google suggests serving 503 or 429 to slow Googlebot

Who needs to worry about crawl budget

Most sites do not. Google's crawl budget guide says so in its first paragraph: if your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day they're published, you don't need the guide, and keeping your sitemap current and checking the Page indexing report is adequate.

The guide names three kinds of site it is written for, and it calls the numbers rough estimates rather than exact thresholds.

Site profileGoogle's rough size and change rate
Large sites1 million or more unique pages, content changes about once a week
Medium or larger sites with fast-changing content10,000 or more unique pages, content changes daily
Sites with an indexing symptomA large portion of total URLs classified in Search Console as Discovered - currently not indexed
Table 1. Who Google's crawl budget guide is written for

The third row is the one a site owner can check today. Open the Page indexing report in Search Console and look at how many URLs sit under Discovered - currently not indexed. Google's report definitions describe that status as a URL Google found but did not crawl, typically because Google wanted to crawl it but expected the crawl to overload the site, so it rescheduled. The last crawl date is empty for these URLs, which is how you tell them apart from Crawled - currently not indexed, where Google did fetch the page and then decided not to index it.

Google's 2017 post, What Crawl Budget Means for Googlebot, said a site with fewer than a few thousand URLs is, most of the time, crawled efficiently. That post is marked as possibly outdated and points to the current documentation, so treat Table 1 as the current statement.

A smaller site can still have pages that never show up, but the cause is usually that Google crawled them and chose not to index them, or that nothing links to them. The diagnostic procedure near the end separates those cases, because crawl budget tuning helps with only one.

How Google defines crawl budget

Google defines crawl budget as the set of URLs it can and wants to crawl. The guide splits that into two parts, a crawl capacity limit and crawl demand, and says that if demand is low, Google crawls your site less even when the capacity limit is nowhere near reached.

One detail changes how large sites should plan. Google's crawling infrastructure treats each unique hostname as a separate site. The guide's example is that https://www.example.com/ and https://code.example.com/ have separate crawl budgets.

Crawl capacity limit

The capacity limit, also called hostload, caps the total time your server spends holding connections open for Google, taking into account both the number of parallel connections and how long each lasts. Every site starts with the same conservative default. If there is demand to crawl more and the site stays healthy, Google raises the limit automatically.

The guide lists what moves the limit. If the site responds consistently and response times, including latency and time to first byte, stay stable or improve, the limit goes up. If the site slows down, returns 5xx errors, or sends rate-limiting signals such as HTTP 429, the limit goes down and Google crawls less. Google's own finite resources also cap it, and the limit is shared across all of Google's crawlers, so heavy demand from one crawler can reduce what is available to another.

Google's Googlebot page says Googlebot shouldn't access your site more than once every few seconds on average, with short spikes possible.

Crawl demand

Each Google crawler has its own demand; AdsBot, for example, generally wants more when a site runs dynamic ad targets. For Googlebot, the guide says demand varies with the site's size, update frequency, page quality, and relevance compared with other sites. It names three factors:

  • Perceived inventory. Without guidance, Google tries to crawl all or most of the URLs it knows about. If many are duplicates or URLs you don't want crawled, that crawling time is wasted. Google calls this the factor you can control the most.
  • Popularity. URLs that are more popular on the internet tend to be crawled more often to keep them fresh.
  • Staleness. Google's systems want to recrawl documents often enough to notice changes. Site-wide events such as a site move can also raise demand while Google reprocesses content under the new URLs.

What the definition does not claim

Crawling is not indexing. The guide says that for Search, a crawled page still has to be evaluated and consolidated before it is considered for the index. Crawling is not ranking either: the 2017 post states that although crawling is necessary for being in results, it's not a ranking signal, and an increased crawl rate will not necessarily lead to better positions. Google's CDN post adds a note for big launches: Google allows higher crawl rates on sites it infers are CDN-backed, but each new URL still has to be served by your origin once to warm the cache, so launching a million pages at once means heavy origin load for a few days.

What wastes crawl budget

The 2017 post listed the low-value URL categories Google found most harmful, in order of significance: faceted navigation and session identifiers, on-site duplicate content, soft error pages, hacked pages, infinite spaces and proxies, and low quality and spam content. The current guide reduces the cause to perceived inventory: Google crawls what it knows about, and URLs that deserve no crawling push out the ones that do.

Session identifiers work like filters: every visit can mint new URLs for the same content. Infinite spaces such as calendars never run out of URLs to offer, and hacked or spam pages inflate the inventory with URLs you did not create. For duplicates, Google's advice is to consolidate, and to use robots.txt for variants that cannot be consolidated.

Faceted navigation

Google's December 2024 faceted navigation post calls it by far the most common source of overcrawl issues that site owners report, and says most could have been avoided. The mechanism is arithmetic. If one category page has 5 colors, 6 sizes, and 8 brands, and each facet can be unset or set to one value, that is (5+1) x (6+1) x (8+1) = 378 URL variants for one category, before sort orders and pagination multiply it again.

The post offers two approaches. If you don't need the filtered URLs indexed, block them with robots.txt or use URL fragments (#) for filters, which search engines generally ignore. If you need them crawled, use the standard & separator between parameters, keep the order of filters consistent, and return a 404 status code for combinations that have no results rather than redirecting to a generic not-found page. It adds that rel="canonical" helps consolidate signals but takes time to be picked up, and that rel="nofollow" on filter links discourages crawling only if every link to those pages, internal and external, carries it.

User-agent: *
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sort=

Sitemap: https://www.example.com/sitemap.xml
Example robots.txt for a filter parameter pattern (adapt the parameter names to your site)

Test patterns like this against real URLs first. A rule that is slightly too broad can block the category pages themselves, and a blocked URL's canonical tag or noindex can't be seen.

Soft 404s and errors that look like content

Google's status code documentation says that when a 2xx page's content suggests an error, such as an empty page or an error message, Search Console shows a soft 404. The crawl budget guide adds that soft 404 pages continue to be crawled and waste budget. An expired product page that returns 200 with a short apology is a typical case. Return a real 404 or 410 instead.

Redirect chains and page resources

The guide says long redirect chains have a negative effect on crawling. Googlebot generally follows up to 10 hops, and the Crawl Stats report counts each request in a chain separately. Resources count too. Google's post on crawling page resources says fetching the JavaScript and CSS needed to render a page chips away at the crawl budget of the hostname that hosts them. The rendering service caches those resources for up to 30 days regardless of HTTP caching headers, so use as few resources as feasible and be cautious with cache-busting parameters, since a changed resource URL may be fetched again even if its contents are identical.

The same post says serving resources from another hostname shifts the crawl budget concern to that host, with a December 6, 2024 update that it can slow page performance, so it is not recommended for critical JavaScript and CSS. Large non-critical files such as video or downloads are better candidates. Disallowing rendering resources in robots.txt can leave Google unable to understand the page.

Blocks by your own CDN or firewall

The CDN post describes a failure that looks like a crawl budget problem and isn't: a firewall that blocklists Google's IP addresses. Hard blocks include network timeouts, which Google says cause URLs to be removed from the index and can also drag down crawl rate, and error messages served with HTTP 200, which Google may treat as duplicates of one another. A soft block is a bot-verification interstitial. For temporary blockage and interstitials alike, Google says 503 or 429 is the preferred signal, so content is not removed from the index.

How status codes change crawling

Google's documentation on HTTP status codes and the crawl budget guide together give the following behavior.

ResponseWhat Google does
200 with real contentPasses content on for processing. Does not guarantee indexing
200 with an error or empty pageReported as a soft 404 in Search Console; keeps being crawled and wastes budget
301 or 308Strong signal that the target is canonical; Googlebot follows up to 10 hops
302 or 307Weak signal that the target is canonical
304 Not ModifiedThe content matches the last crawl; no body is sent, so bandwidth is saved
404 or 410Indexed URLs are removed. Crawl frequency for new 404 URLs gradually decreases. A strong signal not to crawl the URL again
401 or 403Do not use these to limit crawl rate
429Treated as a server overload signal, handled like a 5xx
500Crawl rate decreases in proportion to the number of URLs returning errors; persistent errors lead to removal from the index
502, 503 and other 5xxGoogle's crawlers temporarily slow down; indexed URLs are kept at first but eventually dropped if errors persist; crawl rate increases gradually after 2xx responses resume
Table 2. Status codes and their documented effect on Google's crawling

Two rows deserve emphasis. A 404 or 410 is the clearest way to say a URL is gone: Google won't forget a URL it knows about, but a 404 is a strong signal not to crawl it again, whereas a robots.txt block keeps the URL in the crawl queue much longer. And 429 and 5xx are the codes that slow the whole host, which is the mechanism behind the emergency lever covered below and the reason a flaky origin can quietly throttle crawling of a healthy site.

robots.txt as a crawl dependency

Googlebot checks robots.txt before it crawls, and a failing robots.txt file stops crawling. The Crawl Stats help page gives the sequence. A response of 200 (valid, invalid, or empty file) or 403, 404, or 410 counts as success, and a 404 means no restrictions. A 429 or 5xx counts as unsuccessful. For the first 12 hours of failure Google stops crawling the site but keeps requesting the file. From 12 hours to 30 days it uses the last successfully fetched copy. After 30 days, if the homepage is available Google acts as though there is no robots.txt and crawls without restraint, and if the homepage is not available it stops crawling.

Google's robots.txt specification page adds that Google generally caches robots.txt for up to 24 hours, enforces a 500 KiB size limit and disregards content past it, and ignores fields other than user-agent, allow, disallow, and sitemap.

Reading the Crawl Stats report

The Crawl Stats report lives under Settings in Search Console, and Google's help page says it is aimed at advanced users: a site with fewer than a thousand pages should not need it. It is available only for root-level properties, meaning a Domain property or a URL-prefix property at the root. The data is built from the URLs Google actually requested, not canonical URLs.

SectionWhat it measuresWhat to look for
Total crawl requestsAll crawl requests for URLs on your site, successful or not, including requests for resources hosted on your site. Duplicate requests for one URL count individuallyA sudden step down after a robots.txt change, or a spike after unblocking or adding a section
Total download sizeBytes downloaded during crawling. A cached resource shared by many pages is requested only onceGrowth in bytes without growth in useful pages
Average response timeMean response time across all resources fetched, each resource counted separatelyA rising trend, which tells Google to back off
Host statusWhether Google met availability problems in the last 90 days, across robots.txt fetching, DNS resolution, and server connectivityAnything but green. Red means a significant problem in the last week
By responseResponses grouped by type as a percentage of requests, not URLsMost responses should be 200 or other good codes unless you are moving the site
By file typeThe type returned, as a percentage of responses, not of bytesLarge shares of images, scripts, or feeds you would not expect to be crawled heavily
By purposeDiscovery (URL never crawled before) or Refresh (a recrawl of a known page)Many new pages but little Discovery, or heavy Refresh of pages that never change
By Googlebot typeSmartphone, Desktop, Image, Video, Page resource load, AdsBot, StoreBot, otherWhich agent caused a spike
Table 3. Crawl Stats report sections and how to read them

Host status

Host status has three values. Green means no significant availability issues in the past 90 days. A second value means at least one significant issue occurred in the last 90 days but more than a week ago, and the error might have been transient. A third means a significant issue within the last week. Each of the three categories has a chart with a dotted red threshold line; Google's example is DNS resolution failing for more than 5% of requests on a given day.

Reading it without being misled

  • Each hop in a redirect chain appears as a separate request, and percentages are of requests rather than URLs.
  • Counts can exceed your server logs. Google says this can occur when robots.txt has been unavailable for too long, because Google counts crawls it might have made but did not. Requests that never reached your server, reported as Page could not be reached, also never appear in your logs.
  • Example URLs under each grouping are representative samples, not a complete list, and a domain-level property lists only the top 20 child domains with traffic in the past 90 days.

The help page also explains crawl-rate swings. Spikes follow unblocking a large section, adding a large new section, or adding many Dynamic Search Ads targets (AdsBot crawls URLs about every two weeks). Drops follow a new broad robots.txt rule, slower responses, more server errors, or content that changes rarely or is judged lower quality.

Server log analysis

Google describes raw access logs as the best source for analyzing what its crawlers are requesting, since they have an entry for every URL requested by browsers and crawlers alike, with the Crawl Stats report second best (Crawling December, resources post). Logs show what the report summarizes away: exact URLs and parameters, user agents, and response sizes.

Verify the visitor first

User agent strings are trivially faked, so confirm that a visitor claiming to be Googlebot is Google before counting it. Google's verification guide gives two methods. For a one-off check, run a reverse DNS lookup on the IP from your logs, confirm the returned domain is googlebot.com, google.com, or googleusercontent.com, then run a forward lookup on that name and confirm it returns the same IP. For bulk work, match IPs against the published ranges in JSON files in CIDR notation.

host 66.249.66.1
# expect a name like crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com
# expect 66.249.66.1
Example: reverse DNS then forward DNS (the guide's sample IP)
Crawler typeReverse DNS maskIP range file
Common crawlers (Googlebot)crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com/static/crawling/ipranges/common-crawlers.json
Special-case crawlersrate-limited-proxy-***-***-***-***.google.com/static/crawling/ipranges/special-crawlers.json
User-triggered fetchers***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com/static/crawling/ipranges/user-triggered-fetchers.json and related files
Table 4. Google's published crawler verification data

The paths are relative to developers.google.com. The page lists further files for Google-run user-triggered fetchers and user-triggered agents.

What to look for

Google publishes no benchmark for a healthy log, so this reading comes from its documentation. A healthy picture is verified Googlebot requests dominated by 200 and 304 responses for URLs you want indexed, steady response times, and new URLs getting a first visit soon after they appear in the sitemap or in internal links. Questions worth asking of a verified log:

  • What share of requests goes to URLs with parameters, session identifiers, or filter combinations you do not want indexed? That share is the perceived inventory problem, measured.
  • How many requests return 3xx, 404, 410, or soft-404 candidates, and do the same URLs reappear day after day?
  • Do 429 or 5xx responses cluster at certain times? Those lower the capacity limit for the whole host.
  • Do 304 responses appear at all? If every recrawl of an unchanged page returns a full 200, the server is not answering conditional requests.
  • Do your most important URLs get any verified visits in the period?
grep 'Googlebot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn
Example: status code counts for lines containing Googlebot in a combined-format access log (verify the IPs separately)

That command filters on user agent text, which can be spoofed, so verify the IPs before drawing conclusions.

Making crawling more efficient

Google's guide groups its advice into managing URL inventory and making pages efficient to load. Shrink the inventory first, since that is the factor Google says you control most.

Shrink the inventory

  • Consolidate duplicate content so crawling goes to unique content rather than unique URLs.
  • Block unimportant URLs with robots.txt. Google's guide recommends this for infinite scrolling pages that duplicate information on linked pages and for differently sorted versions of a page. Google's robots.txt introduction frames the file as a way to manage crawler traffic, not a mechanism for keeping pages out of Google.
  • Return 404 or 410 for permanently removed pages and fix soft 404s.
  • Link to final URLs, not through redirects or to parameterized variants.
  • Keep sitemaps current with accurate lastmod values.

Sitemaps and lastmod

Google's sitemap documentation caps one sitemap at 50 MB uncompressed or 50,000 URLs. It uses lastmod only if the value is consistently and verifiably accurate, reflecting the last significant update, and says a copyright date change doesn't count. Google ignores priority and changefreq.

<url>
  <loc>https://www.example.com/guides/crawl-budget</loc>
  <lastmod>2026-09-14T08:30:00+00:00</lastmod>
</url>
Example sitemap entry with a lastmod that reflects the page's last meaningful change

Bing's lastmod post shows how this fails in practice: among sites with at least one Bing-indexed URL, 18% of known sitemaps had incorrectly set lastmod values, most often the sitemap's generation date on every URL. Bing says it may disregard dates that are consistently set to the current date.

HTTP caching

Google's HTTP caching post says about 0.026% of its total fetches were cacheable ten years ago and 0.017% by December 2024. Its crawlers support ETag with If-None-Match, and Last-Modified with If-Modified-Since. Google strongly recommends ETag because its value is unstructured and less error-prone, and suggests setting both. If the ETag matches, the server should return 304 with no body, saving compute and bandwidth.

GET /guides/crawl-budget HTTP/1.1
Host: www.example.com
User-Agent: Googlebot/2.1 (+http://www.google.com/bot.html)
If-None-Match: "34aa387-d-1568eb00"

HTTP/1.1 304 Not Modified
ETag: "34aa387-d-1568eb00"
Cache-Control: max-age=94043
Example conditional request and response (illustrative values, following the pattern in Google's post)

For Last-Modified, Google asks for the HTTP date format, for example Fri, 4 Sep 1998 19:15:56 GMT, and says setting max-age in Cache-Control, while optional, helps crawlers decide when to recrawl. Google suggests a new validator for significant content changes, not for a copyright year update.

Speed and compression

Faster, steadier responses raise the capacity limit directly. Google's crawler overview says its crawlers support HTTP/1.1 and HTTP/2 and the content encodings gzip, deflate, and Brotli. The Googlebot page says that for Search, Googlebot crawls the first 2 MB of a supported file type and the first 64 MB of a PDF, applied to uncompressed data, so content past that point in a very large HTML file is not read.

What does not work

TacticWhy it falls short, per the sources
Crawl-delay in robots.txtGoogle's robots.txt documentation says fields such as crawl-delay aren't supported, and the 2017 post says Googlebot doesn't process it. Bing does honor it
noindex as crawl controlGoogle still requests the page and then drops it, so crawling time is spent anyway
nofollow on internal linksAny crawled URL counts toward budget, so a nofollow link does not prevent crawling if any other link, on your site or elsewhere, is not nofollowed
Blocking in robots.txt to shift budgetGoogle won't move the freed budget to other pages unless it is already hitting your capacity limit
Requesting a higher crawl rateNot possible, per the Crawl Stats help page
Repeated indexing requestsThe recrawl documentation says there is a quota and that asking again for the same URL won't get it crawled faster
Table 5. Common crawl controls that do not do what people expect

Slowing Googlebot in an emergency

Google's guide to reducing crawl rate lists causes of a sharp increase: inefficient URL configuration such as faceted navigation, sorting, or a calendar with many date-specific URLs, and a Dynamic Search Ads target, which brings AdsBot.

The short-term lever is to return 500, 503, or 429 instead of 200 for crawl requests. Google says this reduces the crawl rate across the whole hostname, including pages that still return content, and that the rate recovers automatically once errors fall. With 503 or 429 you can add a Retry-After header, in seconds or as a date.

HTTP/1.1 503 Service Unavailable
Retry-After: 120
Content-Type: text/plain

Temporarily unavailable
Example overload response

Duration matters. The reduce-crawl-rate page suggests a couple of hours up to one or two days, and warns that if Googlebot sees these codes on the same URL over multiple days, the URL may be dropped from the index. Another Google page, the Crawl Stats help, allows two or three days. Given that difference, the conservative reading is safer. Reducing crawl also has costs: Googlebot discovers fewer new pages, refreshes existing pages less often, and keeps removed pages in the index longer, and Google Ads campaigns may be paused or ads may stop serving.

If returning errors isn't feasible, Google offers a special request through the Googlebot report form in Search Console, stating the optimal crawl rate, and evaluation may take several days. The older Search Console crawl rate limiter tool is gone: Google announced it would deprecate the tool on January 8, 2024, explaining that Googlebot already slows down automatically and almost immediately when a server persistently returns HTTP 500 for a range of URLs or when response times grow, while the tool could take over a day to take effect. Google kept the Googlebot report form for unusual activity and emergencies, and said the fastest way to reduce crawl rate is to instruct Googlebot through server responses.

Bing, IndexNow, and AI crawlers

Bing's controls

Unlike Google, Bing honors crawl-delay. A Bing post from 2012, To crawl or not to crawl, explains that Crawl-delay is not a rate but the size of a time window, from 1 to 30 seconds, during which Bingbot crawls only once. With a delay of 5, Bing says, it slices the day into five-second windows, for a maximum of around 17,280 pages a day. Bing generally does not recommend it, since fewer pages crawled can mean less content indexed. It applies per host, so each subdomain's robots.txt is managed separately.

A 2009 Bing post on crawl delay rated the effect on index freshness as slow at 1, very slow at 5, and extremely slow at 10, and advised against anything higher. The 2012 post also describes hourly crawl rates in Bing Webmaster Tools and says a robots.txt crawl-delay overrides that setting. Both posts are old and Bing's current Crawl Control help page could not be loaded, so check the Webmaster Tools interface for the present form of the setting.

Bing's July 2025 sitemap post says it revisits sitemaps typically at least once per day, treats lastmod as a key signal for prioritizing or skipping recrawls, and ignores changefreq and priority.

IndexNow

IndexNow is a ping protocol: you tell participating engines that a URL has been added, updated, or deleted, so they can prioritize crawling it. Per its home page, it has support from Microsoft Bing, Naver, Seznam.cz, Yandex, and Yep; Google is not on that list. The documentation specifies a key of 8 to 128 characters hosted as a UTF-8 text file, single-URL submission through a query string, and a JSON POST carrying up to 10,000 URLs. A 200 response only means the engine received the URL, not that it was indexed.

POST /indexnow HTTP/1.1
Host: api.indexnow.org
Content-Type: application/json; charset=utf-8

{
  "host": "www.example.com",
  "key": "abcd1234abcd1234",
  "urlList": [
    "https://www.example.com/guides/crawl-budget"
  ]
}
Example IndexNow POST body (the key and host are placeholders)

For Google, discovery and recrawl run through sitemaps, links, and URL Inspection, and Google's recrawl documentation says crawling can take anywhere from a few days to a few weeks.

AI crawler load

The best public measurement of crawler load that I could fetch is Cloudflare's 2025 Radar Year in Review, published December 15, 2025. Its scope is Cloudflare's own network, so it reflects Cloudflare customers, not the whole web, and it measures request counts, not server cost per request.

  • Googlebot originated 4.5% of HTML requests, a share slightly larger than all other AI bots together at 4.2%. Cloudflare counts Googlebot in its AI crawler overview because it crawls for both search indexing and AI training.
  • Googlebot's share began the year at just under 2.5%, peaked at 11% in late April, and ended with a 5% share.
  • AI user action crawling, which visits sites in response to user questions, grew by over 15x in 2025.
  • The crawl-to-refer ratio, launched on Radar on July 1, divides HTML requests from a platform's crawlers by HTML requests carrying that platform's referer. Anthropic's reached as high as 500,000:1 during an erratic January to May; Google's started the year just over 3:1, peaked at 30:1 in April, and fell back to 3:1 by mid-July.

The ratio is volatile and describes crawling relative to referrals, not the load any one site sees. For crawl budget, the practical point is our own inference from Google's documented mechanism: Google lowers its capacity limit when response times rise, and your server's capacity is shared by every crawler, so heavy AI crawler load that slows your origin can also reduce Googlebot's crawling. Which AI crawlers to allow is a robots.txt and firewall decision, covered in the companion guide on AI crawlers and robots.txt.

A diagnostic procedure

Google publishes no single flowchart, so this procedure is our reading of its documentation. It sorts a symptom into a crawling problem or an indexing problem, because the fixes are not interchangeable.

  1. Check scope. If new pages are crawled the same day they are published and the site is well below the sizes in Table 1, stop tuning crawl budget and treat missing pages as an indexing question.
  2. Run a sample of important unindexed URLs through URL Inspection. Note the status and whether a last crawl date exists.
  3. If the status is Discovered - currently not indexed (no last crawl date), treat it as a crawling problem. Google's definition says it expected the crawl to overload the site.
  4. Open Crawl Stats. Is host status green? Is average response time rising? Are 429 or 5xx responses present? If yes, fix the server first, since this lowers the capacity limit for every URL. If URL Inspection shows Hostload exceeded, the guide says to add server resources if that makes sense for your business.
  5. In your verified logs, measure what share of Googlebot requests goes to URLs that should not be crawled: parameters, filters, sessions, soft 404s, redirect chains. If that share is large, shrink the inventory with the steps under Shrink the inventory, and recheck after Google has had time to recrawl.
  6. If the inventory is clean and the server healthy but important pages are still waiting, check discoverability: are they in the sitemap with accurate lastmod, and do indexable pages link to them with crawlable links?
  7. If the status is Crawled - currently not indexed (there is a crawl date), treat it as an indexing problem. Google fetched the page and decided against indexing it for now; the help text says it may or may not be indexed later and that you should not resubmit it. More crawling won't change that. Check for duplicates and canonical conflicts and improve the content; the guide notes that quality, uniqueness, and popularity influence the resources allocated to a site.
  8. If pages are indexed but stale, look at Refresh versus Discovery in the purpose table, then at lastmod accuracy and whether the server answers conditional requests with 304.

What to do now

Work in this order and stop when the symptom clears. Confirm you are in scope using Table 1 and the Page indexing report. Check Crawl Stats host status, response time, and 5xx and 429 responses, since server health constrains everything else. Measure waste in verified logs and cut the inventory: block parameterized and infinite-space URLs, return 404 or 410 for gone pages, fix soft 404s, flatten redirects. Make the remaining URLs cheap to recrawl with accurate lastmod and ETag or Last-Modified support. If the status is Crawled - currently not indexed, stop working on crawling and work on the pages.

Reserve the 503 and 429 lever for genuine overload and hold it for no more than a day or two, since it slows the entire host. This advice changes if Google revises its documented thresholds, or if the Page indexing report and URL Inspection show that crawling is not your constraint.

Sources

  1. [1]
    Optimize your crawl budget
    Google Search Central. Definition, thresholds, capacity limit, demand, best practices
  2. [2]
    What Crawl Budget Means for Googlebot
    Google Search Central Blog (January 16, 2017). Low-value URL categories, crawl rate not a ranking signal, crawl-delay, nofollow; marked as possibly outdated
  3. [3]
    Reduce the Googlebot crawl rate
    Google Search Central. Emergency 500, 503, 429 guidance, Retry-After, Googlebot report
  4. [4]
    Crawl Stats report
    Google Search Console Help. Report sections, host status, robots.txt availability rules, troubleshooting
  5. [5]
    How HTTP status codes, and network and DNS errors affect Google Search
    Google Search Central. Status code handling, redirect hops, soft 404, 429 and 5xx
  6. [6]
    Verifying Googlebot and other Google crawlers
    Google Search Central. Reverse DNS masks, forward DNS check, IP range files
  7. [7]
    How Google interprets the robots.txt specification
    Google Search Central. 500 KiB limit, 24 hour cache, unsupported fields, 5xx behavior
  8. [8]
    Introduction to robots.txt
    Google Search Central. robots.txt manages crawler traffic, not a way to keep pages out of Google
  9. [9]
    Crawling December, The how and why of Googlebot crawling
    Google Search Central Blog (December 3, 2024). Rendering resources and crawl budget, WRS 30 day cache, access logs
  10. [10]
    Crawling December, HTTP caching
    Google Search Central Blog (December 9, 2024). ETag, Last-Modified, 304, cacheable fetch share
  11. [11]
    Crawling December, Faceted navigation
    Google Search Central Blog (December 17, 2024). Faceted navigation as the most common overcrawl source and its fixes
  12. [12]
    Crawling December, CDNs and crawling
    Google Search Central Blog (December 24, 2024). CDN crawl rate, cold cache, hard and soft blocks
  13. [13]
    Upcoming deprecation of Crawl Rate Limiter Tool in Search Console
    Google Search Central Blog (November 24, 2023). Removal on January 8, 2024 and what remains
  14. [14]
    Page indexing report
    Google Search Console Help. Discovered and Crawled currently not indexed definitions, soft 404
  15. [15]
    Build and submit a sitemap
    Google Search Central. 50 MB and 50,000 URL limits, lastmod, priority and changefreq
  16. [16]
    Googlebot
    Google Search Central. Crawl pace, 2 MB and 64 MB file limits
  17. [17]
    Google's common crawlers and fetchers overview
    Google Search Central. HTTP/1.1, HTTP/2, content encodings
  18. [18]
    Ask Google to recrawl your URLs
    Google Search Central. Recrawl quota and timing
  19. [19]
    To crawl or not to crawl, that is BingBot's question
    Bing Webmaster Blog (May 3, 2012). Crawl-delay semantics, hourly crawl rates, robots.txt precedence
  20. [20]
    Crawl delay and the Bing crawler, MSNBot
    Bing Webmaster Blog (August 10, 2009). Crawl-delay values and effect on index freshness
  21. [21]
    The Importance of Setting the lastmod Tag in Your Sitemap
    Bing Webmaster Blog (February 1, 2023). Bing study of lastmod accuracy
  22. [22]
    Keeping Content Discoverable with Sitemaps in AI Powered Search
    Bing Webmaster Blog (July 31, 2025). Bing sitemap fetch cadence and lastmod use
  23. [23]
    IndexNow documentation
    IndexNow.org. Key file, submission methods, 10,000 URL limit, response codes
  24. [24]
    IndexNow home page
    IndexNow.org. Participating engines
  25. [25]
    The 2025 Cloudflare Radar Year in Review
    Cloudflare (December 15, 2025). Googlebot and AI bot shares of HTML requests, crawl-to-refer ratio

Sources were last checked on October 14, 2026. Documentation changes, so confirm any detail that matters to a decision against the source itself.

How to cite

Indexive Research. (2026, August 19). Who needs to worry about crawl budget and what wastes it. Indexive AI. https://indexive.ai/research/crawl-budget-explained

Keep reading
Guide · 23 min
What robots.txt and noindex each really control
Guide · 23 min
Internal linking, from crawlable links to site structure
Reference · 23 min
Which AI crawlers honor robots.txt, vendor by vendor

Frequently asked questions.

Straight answers about how we work. Still have a question? The free audit is the fastest way to get one answered for your site.

Probably not. Google's guide is aimed at sites with 1 million or more pages changing weekly, 10,000 or more pages changing daily, or a large share of URLs in Discovered - currently not indexed, and calls those rough estimates.

No. Google's 2017 post says crawling is necessary to appear in results but is not a ranking signal, and a higher crawl rate does not necessarily mean better positions.

No. Google's documentation says crawl-delay isn't supported. Bing does honor it, as a window of 1 to 30 seconds.

Return 503 or 429, optionally with a Retry-After header, for a short period only. Google suggests no more than a day or two, because persistent errors on the same URL over several days can lead to removal from the index.

Discovered means Google knows the URL but has not crawled it, often because it expected the crawl to overload the site. Crawled means Google fetched the page and chose not to index it for now, which is an indexing and quality question rather than a crawl budget one.

See your first 30 days of execution.

A free execution audit: your prioritized backlog, effort estimates, and what we'd ship first. Read-only access, no contract.

By continuing you agree to our Privacy Policy and Terms.
indexive
ServicesResearchContentTechnical SEOImplementationGEO / AI searchPerformanceMigrations
SolutionsDeclining traffic recoveryAI search visibilitySEO backlog executionTechnical debt cleanupContent at scaleHeads of SEOVPs of Marketing & CMOsLean teams with no SEO
IndustriesB2B SaaSDeveloper toolsCybersecurityFintechHealthcare & life sciencesE-commerceMarketplacesManufacturing & industrialProfessional services
CompanyResearchFree auditSecurityContactLog in
© 2026 Indexive AIPrivacyTermsSan Francisco, CA