Crawl budget is the set of URLs Google can and wants to crawl, set by a server-driven capacity limit and by demand. Google says its guide is meant for sites with 1 million or more pages changing weekly, 10,000 or more pages changing daily, or many URLs stuck in Discovered - currently not indexed. For everyone else, a current sitemap and a healthy server are enough. This guide covers what wastes crawling, how to read the Crawl Stats report and server logs, and how to tell a crawling problem from an indexing problem.
Most sites do not. Google's crawl budget guide says so in its first paragraph: if your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day they're published, you don't need the guide, and keeping your sitemap current and checking the Page indexing report is adequate.
The guide names three kinds of site it is written for, and it calls the numbers rough estimates rather than exact thresholds.
| Site profile | Google's rough size and change rate |
|---|---|
| Large sites | 1 million or more unique pages, content changes about once a week |
| Medium or larger sites with fast-changing content | 10,000 or more unique pages, content changes daily |
| Sites with an indexing symptom | A large portion of total URLs classified in Search Console as Discovered - currently not indexed |
The third row is the one a site owner can check today. Open the Page indexing report in Search Console and look at how many URLs sit under Discovered - currently not indexed. Google's report definitions describe that status as a URL Google found but did not crawl, typically because Google wanted to crawl it but expected the crawl to overload the site, so it rescheduled. The last crawl date is empty for these URLs, which is how you tell them apart from Crawled - currently not indexed, where Google did fetch the page and then decided not to index it.
Google's 2017 post, What Crawl Budget Means for Googlebot, said a site with fewer than a few thousand URLs is, most of the time, crawled efficiently. That post is marked as possibly outdated and points to the current documentation, so treat Table 1 as the current statement.
A smaller site can still have pages that never show up, but the cause is usually that Google crawled them and chose not to index them, or that nothing links to them. The diagnostic procedure near the end separates those cases, because crawl budget tuning helps with only one.
Google defines crawl budget as the set of URLs it can and wants to crawl. The guide splits that into two parts, a crawl capacity limit and crawl demand, and says that if demand is low, Google crawls your site less even when the capacity limit is nowhere near reached.
One detail changes how large sites should plan. Google's crawling infrastructure treats each unique hostname as a separate site. The guide's example is that https://www.example.com/ and https://code.example.com/ have separate crawl budgets.
The capacity limit, also called hostload, caps the total time your server spends holding connections open for Google, taking into account both the number of parallel connections and how long each lasts. Every site starts with the same conservative default. If there is demand to crawl more and the site stays healthy, Google raises the limit automatically.
The guide lists what moves the limit. If the site responds consistently and response times, including latency and time to first byte, stay stable or improve, the limit goes up. If the site slows down, returns 5xx errors, or sends rate-limiting signals such as HTTP 429, the limit goes down and Google crawls less. Google's own finite resources also cap it, and the limit is shared across all of Google's crawlers, so heavy demand from one crawler can reduce what is available to another.
Google's Googlebot page says Googlebot shouldn't access your site more than once every few seconds on average, with short spikes possible.
Each Google crawler has its own demand; AdsBot, for example, generally wants more when a site runs dynamic ad targets. For Googlebot, the guide says demand varies with the site's size, update frequency, page quality, and relevance compared with other sites. It names three factors:
Crawling is not indexing. The guide says that for Search, a crawled page still has to be evaluated and consolidated before it is considered for the index. Crawling is not ranking either: the 2017 post states that although crawling is necessary for being in results, it's not a ranking signal, and an increased crawl rate will not necessarily lead to better positions. Google's CDN post adds a note for big launches: Google allows higher crawl rates on sites it infers are CDN-backed, but each new URL still has to be served by your origin once to warm the cache, so launching a million pages at once means heavy origin load for a few days.
The 2017 post listed the low-value URL categories Google found most harmful, in order of significance: faceted navigation and session identifiers, on-site duplicate content, soft error pages, hacked pages, infinite spaces and proxies, and low quality and spam content. The current guide reduces the cause to perceived inventory: Google crawls what it knows about, and URLs that deserve no crawling push out the ones that do.
Session identifiers work like filters: every visit can mint new URLs for the same content. Infinite spaces such as calendars never run out of URLs to offer, and hacked or spam pages inflate the inventory with URLs you did not create. For duplicates, Google's advice is to consolidate, and to use robots.txt for variants that cannot be consolidated.
Google's December 2024 faceted navigation post calls it by far the most common source of overcrawl issues that site owners report, and says most could have been avoided. The mechanism is arithmetic. If one category page has 5 colors, 6 sizes, and 8 brands, and each facet can be unset or set to one value, that is (5+1) x (6+1) x (8+1) = 378 URL variants for one category, before sort orders and pagination multiply it again.
The post offers two approaches. If you don't need the filtered URLs indexed, block them with robots.txt or use URL fragments (#) for filters, which search engines generally ignore. If you need them crawled, use the standard & separator between parameters, keep the order of filters consistent, and return a 404 status code for combinations that have no results rather than redirecting to a generic not-found page. It adds that rel="canonical" helps consolidate signals but takes time to be picked up, and that rel="nofollow" on filter links discourages crawling only if every link to those pages, internal and external, carries it.
User-agent: *
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sort=
Sitemap: https://www.example.com/sitemap.xmlTest patterns like this against real URLs first. A rule that is slightly too broad can block the category pages themselves, and a blocked URL's canonical tag or noindex can't be seen.
Google's status code documentation says that when a 2xx page's content suggests an error, such as an empty page or an error message, Search Console shows a soft 404. The crawl budget guide adds that soft 404 pages continue to be crawled and waste budget. An expired product page that returns 200 with a short apology is a typical case. Return a real 404 or 410 instead.
The guide says long redirect chains have a negative effect on crawling. Googlebot generally follows up to 10 hops, and the Crawl Stats report counts each request in a chain separately. Resources count too. Google's post on crawling page resources says fetching the JavaScript and CSS needed to render a page chips away at the crawl budget of the hostname that hosts them. The rendering service caches those resources for up to 30 days regardless of HTTP caching headers, so use as few resources as feasible and be cautious with cache-busting parameters, since a changed resource URL may be fetched again even if its contents are identical.
The same post says serving resources from another hostname shifts the crawl budget concern to that host, with a December 6, 2024 update that it can slow page performance, so it is not recommended for critical JavaScript and CSS. Large non-critical files such as video or downloads are better candidates. Disallowing rendering resources in robots.txt can leave Google unable to understand the page.
The CDN post describes a failure that looks like a crawl budget problem and isn't: a firewall that blocklists Google's IP addresses. Hard blocks include network timeouts, which Google says cause URLs to be removed from the index and can also drag down crawl rate, and error messages served with HTTP 200, which Google may treat as duplicates of one another. A soft block is a bot-verification interstitial. For temporary blockage and interstitials alike, Google says 503 or 429 is the preferred signal, so content is not removed from the index.
Google's documentation on HTTP status codes and the crawl budget guide together give the following behavior.
| Response | What Google does |
|---|---|
| 200 with real content | Passes content on for processing. Does not guarantee indexing |
| 200 with an error or empty page | Reported as a soft 404 in Search Console; keeps being crawled and wastes budget |
| 301 or 308 | Strong signal that the target is canonical; Googlebot follows up to 10 hops |
| 302 or 307 | Weak signal that the target is canonical |
| 304 Not Modified | The content matches the last crawl; no body is sent, so bandwidth is saved |
| 404 or 410 | Indexed URLs are removed. Crawl frequency for new 404 URLs gradually decreases. A strong signal not to crawl the URL again |
| 401 or 403 | Do not use these to limit crawl rate |
| 429 | Treated as a server overload signal, handled like a 5xx |
| 500 | Crawl rate decreases in proportion to the number of URLs returning errors; persistent errors lead to removal from the index |
| 502, 503 and other 5xx | Google's crawlers temporarily slow down; indexed URLs are kept at first but eventually dropped if errors persist; crawl rate increases gradually after 2xx responses resume |
Two rows deserve emphasis. A 404 or 410 is the clearest way to say a URL is gone: Google won't forget a URL it knows about, but a 404 is a strong signal not to crawl it again, whereas a robots.txt block keeps the URL in the crawl queue much longer. And 429 and 5xx are the codes that slow the whole host, which is the mechanism behind the emergency lever covered below and the reason a flaky origin can quietly throttle crawling of a healthy site.
Googlebot checks robots.txt before it crawls, and a failing robots.txt file stops crawling. The Crawl Stats help page gives the sequence. A response of 200 (valid, invalid, or empty file) or 403, 404, or 410 counts as success, and a 404 means no restrictions. A 429 or 5xx counts as unsuccessful. For the first 12 hours of failure Google stops crawling the site but keeps requesting the file. From 12 hours to 30 days it uses the last successfully fetched copy. After 30 days, if the homepage is available Google acts as though there is no robots.txt and crawls without restraint, and if the homepage is not available it stops crawling.
Google's robots.txt specification page adds that Google generally caches robots.txt for up to 24 hours, enforces a 500 KiB size limit and disregards content past it, and ignores fields other than user-agent, allow, disallow, and sitemap.
The Crawl Stats report lives under Settings in Search Console, and Google's help page says it is aimed at advanced users: a site with fewer than a thousand pages should not need it. It is available only for root-level properties, meaning a Domain property or a URL-prefix property at the root. The data is built from the URLs Google actually requested, not canonical URLs.
| Section | What it measures | What to look for |
|---|---|---|
| Total crawl requests | All crawl requests for URLs on your site, successful or not, including requests for resources hosted on your site. Duplicate requests for one URL count individually | A sudden step down after a robots.txt change, or a spike after unblocking or adding a section |
| Total download size | Bytes downloaded during crawling. A cached resource shared by many pages is requested only once | Growth in bytes without growth in useful pages |
| Average response time | Mean response time across all resources fetched, each resource counted separately | A rising trend, which tells Google to back off |
| Host status | Whether Google met availability problems in the last 90 days, across robots.txt fetching, DNS resolution, and server connectivity | Anything but green. Red means a significant problem in the last week |
| By response | Responses grouped by type as a percentage of requests, not URLs | Most responses should be 200 or other good codes unless you are moving the site |
| By file type | The type returned, as a percentage of responses, not of bytes | Large shares of images, scripts, or feeds you would not expect to be crawled heavily |
| By purpose | Discovery (URL never crawled before) or Refresh (a recrawl of a known page) | Many new pages but little Discovery, or heavy Refresh of pages that never change |
| By Googlebot type | Smartphone, Desktop, Image, Video, Page resource load, AdsBot, StoreBot, other | Which agent caused a spike |
Host status has three values. Green means no significant availability issues in the past 90 days. A second value means at least one significant issue occurred in the last 90 days but more than a week ago, and the error might have been transient. A third means a significant issue within the last week. Each of the three categories has a chart with a dotted red threshold line; Google's example is DNS resolution failing for more than 5% of requests on a given day.
The help page also explains crawl-rate swings. Spikes follow unblocking a large section, adding a large new section, or adding many Dynamic Search Ads targets (AdsBot crawls URLs about every two weeks). Drops follow a new broad robots.txt rule, slower responses, more server errors, or content that changes rarely or is judged lower quality.
Google describes raw access logs as the best source for analyzing what its crawlers are requesting, since they have an entry for every URL requested by browsers and crawlers alike, with the Crawl Stats report second best (Crawling December, resources post). Logs show what the report summarizes away: exact URLs and parameters, user agents, and response sizes.
User agent strings are trivially faked, so confirm that a visitor claiming to be Googlebot is Google before counting it. Google's verification guide gives two methods. For a one-off check, run a reverse DNS lookup on the IP from your logs, confirm the returned domain is googlebot.com, google.com, or googleusercontent.com, then run a forward lookup on that name and confirm it returns the same IP. For bulk work, match IPs against the published ranges in JSON files in CIDR notation.
host 66.249.66.1
# expect a name like crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com
# expect 66.249.66.1| Crawler type | Reverse DNS mask | IP range file |
|---|---|---|
| Common crawlers (Googlebot) | crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com | /static/crawling/ipranges/common-crawlers.json |
| Special-case crawlers | rate-limited-proxy-***-***-***-***.google.com | /static/crawling/ipranges/special-crawlers.json |
| User-triggered fetchers | ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com | /static/crawling/ipranges/user-triggered-fetchers.json and related files |
The paths are relative to developers.google.com. The page lists further files for Google-run user-triggered fetchers and user-triggered agents.
Google publishes no benchmark for a healthy log, so this reading comes from its documentation. A healthy picture is verified Googlebot requests dominated by 200 and 304 responses for URLs you want indexed, steady response times, and new URLs getting a first visit soon after they appear in the sitemap or in internal links. Questions worth asking of a verified log:
grep 'Googlebot' access.log | awk '{print $9}' | sort | uniq -c | sort -rnThat command filters on user agent text, which can be spoofed, so verify the IPs before drawing conclusions.
Google's guide groups its advice into managing URL inventory and making pages efficient to load. Shrink the inventory first, since that is the factor Google says you control most.
Google's sitemap documentation caps one sitemap at 50 MB uncompressed or 50,000 URLs. It uses lastmod only if the value is consistently and verifiably accurate, reflecting the last significant update, and says a copyright date change doesn't count. Google ignores priority and changefreq.
<url>
<loc>https://www.example.com/guides/crawl-budget</loc>
<lastmod>2026-09-14T08:30:00+00:00</lastmod>
</url>Bing's lastmod post shows how this fails in practice: among sites with at least one Bing-indexed URL, 18% of known sitemaps had incorrectly set lastmod values, most often the sitemap's generation date on every URL. Bing says it may disregard dates that are consistently set to the current date.
Google's HTTP caching post says about 0.026% of its total fetches were cacheable ten years ago and 0.017% by December 2024. Its crawlers support ETag with If-None-Match, and Last-Modified with If-Modified-Since. Google strongly recommends ETag because its value is unstructured and less error-prone, and suggests setting both. If the ETag matches, the server should return 304 with no body, saving compute and bandwidth.
GET /guides/crawl-budget HTTP/1.1
Host: www.example.com
User-Agent: Googlebot/2.1 (+http://www.google.com/bot.html)
If-None-Match: "34aa387-d-1568eb00"
HTTP/1.1 304 Not Modified
ETag: "34aa387-d-1568eb00"
Cache-Control: max-age=94043For Last-Modified, Google asks for the HTTP date format, for example Fri, 4 Sep 1998 19:15:56 GMT, and says setting max-age in Cache-Control, while optional, helps crawlers decide when to recrawl. Google suggests a new validator for significant content changes, not for a copyright year update.
Faster, steadier responses raise the capacity limit directly. Google's crawler overview says its crawlers support HTTP/1.1 and HTTP/2 and the content encodings gzip, deflate, and Brotli. The Googlebot page says that for Search, Googlebot crawls the first 2 MB of a supported file type and the first 64 MB of a PDF, applied to uncompressed data, so content past that point in a very large HTML file is not read.
| Tactic | Why it falls short, per the sources |
|---|---|
| Crawl-delay in robots.txt | Google's robots.txt documentation says fields such as crawl-delay aren't supported, and the 2017 post says Googlebot doesn't process it. Bing does honor it |
| noindex as crawl control | Google still requests the page and then drops it, so crawling time is spent anyway |
| nofollow on internal links | Any crawled URL counts toward budget, so a nofollow link does not prevent crawling if any other link, on your site or elsewhere, is not nofollowed |
| Blocking in robots.txt to shift budget | Google won't move the freed budget to other pages unless it is already hitting your capacity limit |
| Requesting a higher crawl rate | Not possible, per the Crawl Stats help page |
| Repeated indexing requests | The recrawl documentation says there is a quota and that asking again for the same URL won't get it crawled faster |
Google's guide to reducing crawl rate lists causes of a sharp increase: inefficient URL configuration such as faceted navigation, sorting, or a calendar with many date-specific URLs, and a Dynamic Search Ads target, which brings AdsBot.
The short-term lever is to return 500, 503, or 429 instead of 200 for crawl requests. Google says this reduces the crawl rate across the whole hostname, including pages that still return content, and that the rate recovers automatically once errors fall. With 503 or 429 you can add a Retry-After header, in seconds or as a date.
HTTP/1.1 503 Service Unavailable
Retry-After: 120
Content-Type: text/plain
Temporarily unavailableDuration matters. The reduce-crawl-rate page suggests a couple of hours up to one or two days, and warns that if Googlebot sees these codes on the same URL over multiple days, the URL may be dropped from the index. Another Google page, the Crawl Stats help, allows two or three days. Given that difference, the conservative reading is safer. Reducing crawl also has costs: Googlebot discovers fewer new pages, refreshes existing pages less often, and keeps removed pages in the index longer, and Google Ads campaigns may be paused or ads may stop serving.
If returning errors isn't feasible, Google offers a special request through the Googlebot report form in Search Console, stating the optimal crawl rate, and evaluation may take several days. The older Search Console crawl rate limiter tool is gone: Google announced it would deprecate the tool on January 8, 2024, explaining that Googlebot already slows down automatically and almost immediately when a server persistently returns HTTP 500 for a range of URLs or when response times grow, while the tool could take over a day to take effect. Google kept the Googlebot report form for unusual activity and emergencies, and said the fastest way to reduce crawl rate is to instruct Googlebot through server responses.
Unlike Google, Bing honors crawl-delay. A Bing post from 2012, To crawl or not to crawl, explains that Crawl-delay is not a rate but the size of a time window, from 1 to 30 seconds, during which Bingbot crawls only once. With a delay of 5, Bing says, it slices the day into five-second windows, for a maximum of around 17,280 pages a day. Bing generally does not recommend it, since fewer pages crawled can mean less content indexed. It applies per host, so each subdomain's robots.txt is managed separately.
A 2009 Bing post on crawl delay rated the effect on index freshness as slow at 1, very slow at 5, and extremely slow at 10, and advised against anything higher. The 2012 post also describes hourly crawl rates in Bing Webmaster Tools and says a robots.txt crawl-delay overrides that setting. Both posts are old and Bing's current Crawl Control help page could not be loaded, so check the Webmaster Tools interface for the present form of the setting.
Bing's July 2025 sitemap post says it revisits sitemaps typically at least once per day, treats lastmod as a key signal for prioritizing or skipping recrawls, and ignores changefreq and priority.
IndexNow is a ping protocol: you tell participating engines that a URL has been added, updated, or deleted, so they can prioritize crawling it. Per its home page, it has support from Microsoft Bing, Naver, Seznam.cz, Yandex, and Yep; Google is not on that list. The documentation specifies a key of 8 to 128 characters hosted as a UTF-8 text file, single-URL submission through a query string, and a JSON POST carrying up to 10,000 URLs. A 200 response only means the engine received the URL, not that it was indexed.
POST /indexnow HTTP/1.1
Host: api.indexnow.org
Content-Type: application/json; charset=utf-8
{
"host": "www.example.com",
"key": "abcd1234abcd1234",
"urlList": [
"https://www.example.com/guides/crawl-budget"
]
}For Google, discovery and recrawl run through sitemaps, links, and URL Inspection, and Google's recrawl documentation says crawling can take anywhere from a few days to a few weeks.
The best public measurement of crawler load that I could fetch is Cloudflare's 2025 Radar Year in Review, published December 15, 2025. Its scope is Cloudflare's own network, so it reflects Cloudflare customers, not the whole web, and it measures request counts, not server cost per request.
The ratio is volatile and describes crawling relative to referrals, not the load any one site sees. For crawl budget, the practical point is our own inference from Google's documented mechanism: Google lowers its capacity limit when response times rise, and your server's capacity is shared by every crawler, so heavy AI crawler load that slows your origin can also reduce Googlebot's crawling. Which AI crawlers to allow is a robots.txt and firewall decision, covered in the companion guide on AI crawlers and robots.txt.
Google publishes no single flowchart, so this procedure is our reading of its documentation. It sorts a symptom into a crawling problem or an indexing problem, because the fixes are not interchangeable.
Work in this order and stop when the symptom clears. Confirm you are in scope using Table 1 and the Page indexing report. Check Crawl Stats host status, response time, and 5xx and 429 responses, since server health constrains everything else. Measure waste in verified logs and cut the inventory: block parameterized and infinite-space URLs, return 404 or 410 for gone pages, fix soft 404s, flatten redirects. Make the remaining URLs cheap to recrawl with accurate lastmod and ETag or Last-Modified support. If the status is Crawled - currently not indexed, stop working on crawling and work on the pages.
Reserve the 503 and 429 lever for genuine overload and hold it for no more than a day or two, since it slows the entire host. This advice changes if Google revises its documented thresholds, or if the Page indexing report and URL Inspection show that crawling is not your constraint.
Sources were last checked on October 14, 2026. Documentation changes, so confirm any detail that matters to a decision against the source itself.
Indexive Research. (2026, August 19). Who needs to worry about crawl budget and what wastes it. Indexive AI. https://indexive.ai/research/crawl-budget-explained
Straight answers about how we work. Still have a question? The free audit is the fastest way to get one answered for your site.
Probably not. Google's guide is aimed at sites with 1 million or more pages changing weekly, 10,000 or more pages changing daily, or a large share of URLs in Discovered - currently not indexed, and calls those rough estimates.
A free execution audit: your prioritized backlog, effort estimates, and what we'd ship first. Read-only access, no contract.