Crawl budget is the amount of crawling a search engine will do on your site in a given timeframe. Google describes it as the combination of crawl capacity (how much your server can handle) and crawl demand (how much Google wants to crawl).
Why it matters
For small sites it is rarely a concern. For large sites, such as stores with many filter combinations, wasted crawls on duplicate or low-value URLs can delay discovery of new or updated pages. Slow responses and server errors reduce the crawl rate.
Example
An online shop has 2,000 products but filters create 400,000 URLs like:
/shoes?color=red&size=9&sort=priceBots spend most visits on these variants. Blocking them in robots.txt or canonicalizing them frees crawling for real product pages.
Common mistakes
- Worrying about crawl budget on a site with a few hundred pages.
- Using noindex to save budget; the page must still be crawled to see the tag.
- Ignoring recurring 5xx errors, which signal the bot to slow down.
Related
Get notified when this term updates
Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.
This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.