SEO
Crawl Budget
Quick definition
Crawl budget is the number of pages a search engine bot will crawl on your site within a given period, based on crawl rate limit and crawl demand.
For most small to mid-sized sites, crawl budget isn't a major concern. It becomes critical on sites with millions of URLs, dynamic parameters, or rapid content turnover.
Why Crawl Budget matters
Wasted crawl budget — on duplicate URLs, parameter variants, redirect chains, and infinite calendars — delays indexing of important pages.
How Crawl Budget works in practice
Optimize crawl budget by fixing redirect chains, blocking low-value URLs in robots.txt, consolidating duplicates with canonical tags, and improving site speed.
Best practices
- Block low-value URLs in robots.txt.
- Fix redirect chains and broken internal links.
- Consolidate duplicates with canonicals.
- Improve server response times.