Services · Technical SEO
On a large site, Google will only crawl so many of your URLs.
The work is making sure it spends that time on the pages you care about.
How crawling actually works
How crawl budget works.
Crawl budget is two numbers multiplied: capacity, meaning how hard Google is willing to hit your server, and demand. The documentation says “every site starts with the same default, conservative crawl capacity limit”, so there’s no tier to buy. Capacity rises while your response times hold and drops when the server hands back 5xx or 429 (Google, crawl-budget documentation, 2026).
The budget belongs to a hostname, not a brand. http://www.example.com and code.example.com each get their own allowance and neither knows the other exists.
The 2026 addition is the one we point at first: capacity is shared across every Google crawler. Googlebot, Googlebot-Image, AdsBot and StoreBot all share it. Google puts it as “high demand from one crawler can reduce the capacity available for others,” so a shopping feed competes with the location pages that pay the bills.
So the lever is perceived inventory, the set of URLs Google thinks it has to visit, which it calls “the factor that you can positively control the most.” Left alone it goes after nearly every URL it has heard of. That starts with the filter links down the side of a category page, which Google says “generate infinite URL spaces” (Google, faceted-navigation documentation, 2025). Your new pages queue behind those.
The bit that catches everybody out is in the documentation: each response you return tells Googlebot something different about the URL. And caching is the cheapest way to get more capacity, and almost nobody uses it.
ETag rather than Last-Modified and return a real 304 Not Modified.
Effective capacity goes up
Rendering is a second, quieter crawl. Googlebot hands the HTML to the Web Rendering Service, which goes back out for the JS and CSS, and those trips “chip away from the crawl budget of the hostname that’s hosting the resource.” WRS caches those files for up to 30 days, whatever your cache headers say. If asset filenames change on every deploy, Google fetches the whole set again. Googlebot reads the first 2MB of a file, uncompressed, and 64MB for PDFs (Google, February 2026). And the render queue is slow: a Vercel/MERJ study of 100,000+ Googlebot fetches found that some pages wait hours, even days, to be fully read.
So where do you go and watch this happening? Google answers that itself:
“The best source to analyze what resources Google is crawling is the site’s raw access logs.” Google Search Central documentation
After the logs, Crawl Stats in Search Console splits requests by response code, file type, Googlebot type, and the one that matters, crawl purpose, which is discovery against refresh. It reaches back 90 days.
What’s actually broken
Six failures we check first.
Core Web Vitals are graded at the 75th percentile of real users, so the slowest quarter of your traffic decides the result.
Thresholds: Google, Core Web Vitals documentation, 2026. Across the web, 48% of mobile pages pass all three, and LCP is the weak one at 62% good. HTTP Archive, Web Almanac 2025 (CrUX, June 2025).
Query-string URLs dominate the crawl log
Take Googlebot’s HTML requests for the last 30 days, count the ones carrying a ?, then count how many of those turn up in your XML sitemap. The gap between the two is roughly what your crawl budget is buying.
noindex used as the crawl-budget tool
Count the URLs coming back with a noindex meta tag or an X-Robots-Tag, and remember Google fetches every one of them, reads the directive and then throws the page away. Where a URL is permanently dead, 410 or 404 is the only response that stops Google coming back.
A large “Discovered — currently not indexed” bucket, next to a slow TTFB
Google’s definition: it “wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl.” So read that bucket next to your average response time, and next to any 5xx or 429 spikes.
The <head> closes early and takes the canonical with it
One stray element inside <head>, a <div> or an unescaped script from a tag manager, closes the head early, and everything after it lands in the body doing nothing. That includes the canonical tag, the one telling Google which version of a page is the real one.
hreflang that isn’t reciprocal, or uses relative URLs
hreflang tags tell Google which page is the German version and which is the Spanish one. Open every alternate a localized page lists and check each points back, itself included, fully qualified. Google: “If two pages don’t both point to each other, the tags will be ignored.”
Content behind “load more” with no crawlable link
Category, route and location listings that hand over their results on a click or a scroll are invisible unless those URLs also sit on the page as <a href> elements. Switch JavaScript off and count the links left.
In context
The Franzy relaunch, in numbers.
Franzy · franchise marketplace
Case study · FranzyFranzy is a Charlotte startup that set out to be the Zillow for franchises, and they had an indexing problem in the most literal sense. Thin pages get crawled but not indexed, and thousands of thin URLs quietly dilute crawl budget and sitewide trust. The rich information sat behind a sign-up wall, so what Google saw up front was generic. We rebuilt the inventory first, then what fills it.
organic lift on the directory folder within six months of relaunch
visits a month on the folder that had stalled, up from a few hundred visits a month at the start
to a working content engine, with first results in roughly two months
domain authority gained over the year
This was never a standalone technical project. Taking the crawl and indexing constraint off is what let content do the ranking. Sherpa Auto Transport ran the same way, from fewer than 100 pages indexed to 1,131 pages with traffic today (A2 case study).
What the work involves
Eight steps, in order.
Get the logs
The slowest step on the list. It’s a ticket with whoever owns the CDN, and since most edge providers keep logs for days, the first ask is usually “start retaining these”. Then Googlebot gets verified by reverse DNS, because anybody can spoof a user-agent string. Count on 1–3 weeks.
Build one URL inventory
A full site crawl, the XML sitemaps, the access logs, Search Console and analytics, joined on one URL key. Most of it is getting the joins right: normalizing slashes and case, collapsing parameter orderings, getting timezones to agree. On a 500k-URL site that’s 2–4 weeks.
Classify every URL into a disposition
Canonical and indexable, intended noindex, parameter noise, orphaned, permanently dead. No tool hands you this, because underneath it is a run of arguments with engineering about what a URL is even for.
Decide and stage the removals
Disallow patterns in robots.txt for the parameters, 410 for the dead sets, consolidation for the duplicates. Every pattern runs against the real URL list in staging first, because one over-broad wildcard can take a whole folder out of the index. Count on 1–2 weeks.
Repair the head, the links, and the caching
Canonical, hreflang and meta robots checked in the rendered head, pagination shipped as sequential <a href> links, ETag set and 304s genuinely coming back, asset URLs that stop churning on every deploy. Tiny diffs, a lot of leverage.
Structured data down to eligible types only
Pull the deprecated markup, repair the required properties, and check that what’s marked up is visible on the page. Structured data buys eligibility for a feature and has never bought rank.
Core Web Vitals, from field data
CrUX at the 75th percentile, split mobile and desktop, prioritized by template rather than by URL, because on a site this size you fix a template, not a URL. Usually it comes down to LCP on mobile.
What moves first, and when
Redistribution shows up first in Crawl Stats, as a shift between discovery and refresh crawls, and that takes 4–12 weeks. Index changes trail it again. Crawl Stats keeps 90 days, which is the ceiling on proving what you changed last quarter.
If that reads like plumbing, it’s because it is: joining messy exports, arguing about URL dispositions, waiting for logs to pile up. It’s also what decides whether Googlebot’s next 30 days look different from its last 30.
The decision
An audit versus an inventory.
A crawler runs and a prioritized spreadsheet lands, a few hundred rows sorted by severity: missing meta descriptions, heading order, image alt text, thin-content flags. A fair number even get done.
Six months on, the crawl distribution in Search Console looks exactly as it did at the start, because nothing in the exercise took a single URL out of Google’s inventory.
The work was real; it just wasn’t aimed at the constraint.
The first thing we hand over is a count: how many distinct URLs this hostname exposes, and how Googlebot spread its last 30 days across them. It is slower to start, because the logs take a while, and it produces fewer line items. Decisions then get made about whole classes of URL.
What you watch afterwards isn’t a score.
The request mix in your own logs moves week over week, and the “Discovered — currently not indexed” bucket drains.
Of roughly 14 billion pages studied, the share getting no traffic at all from Google. Ahrefs call that conservative themselves, because their sample leans toward higher-quality content than the web as a whole.
Source: Ahrefs, “96.55% of Content Gets No Traffic From Google,” 2023Sourcing
Where these numbers come from.
URLs with daily-changing content, the point where Google’s guidance says crawl-budget work starts to matter (1M+ at weekly change).
Source: Google, Optimize your crawl budget, 2026How much of a page Googlebot fetches, measured uncompressed. 64MB for PDFs.
Source: Google, Googlebot documentation, Feb 2026Of Google’s fetches are servable from cache today, down from 0.026% a decade ago.
Source: Google Search Central Blog, Crawling December: HTTP caching, 2024LCP, INP and CLS “good” thresholds, assessed at the 75th percentile of real-user field data.
Source: Google Search Central, Core Web Vitals documentation, 2026Of mobile pages pass Core Web Vitals; LCP is the weak metric at 62% good.
Source: HTTP Archive, Web Almanac 2025 (CrUX, June 2025)Median versus p90 wait in Google’s render queue across 100,000+ Googlebot fetches.
Source: Vercel with MERJ, 2024Structured data features Google currently documents. FAQ rich results ended 7 May 2026.
Source: Google Search Central, structured data gallery, June 2026How far back Crawl Stats reaches, and the limit on proving a change in crawl behavior.
Source: Google Search Console Help, Crawl Stats reportGo deeper
Two case studies and Google’s documentation.
Work with A2
We’ll read your logs and tell you where the crawl is going.
Thirty days of verified Googlebot requests joined against your whole URL inventory, so you can see where the crawl goes and what we’d take out first.