Crawl Budget

The number of pages a search engine bot will crawl on your website within a given timeframe, determined by your site's authority and server capacity.

Crawl budget is the combination of crawl rate (how fast Googlebot can crawl without overloading your server) and crawl demand (how much Google wants to crawl your site). For most small-to-medium sites, crawl budget is not a concern.

When Crawl Budget Matters

  • Sites with 10,000+ pages
  • E-commerce sites with extensive product catalogs
  • Sites with significant duplicate or low-quality content
  • Sites with slow server response times

How to Optimize Crawl Budget

  • Remove or noindex low-value pages (tag pages, internal search results)
  • Fix crawl errors and broken links
  • Improve server response times
  • Use XML sitemaps to guide crawlers to important pages
  • Avoid redirect chains and loops

FAQ

Do small websites need to worry about crawl budget?

Generally no. Google can easily crawl sites under 10,000 pages. Focus on crawl budget optimization only if you have a very large site or notice indexing delays.

How do I check my crawl budget?

Use Google Search Console's Crawl Stats report (Settings > Crawl Stats) to see how often and how many pages Googlebot crawls daily.