On November 11, 2026, the Google Search Central team published a post about moving to a CDN. The starting point: right after putting their site behind a CDN, some site owners see crawler traffic to their server go up, when the CDN was supposed to bring it down.
Of Google's three recommendations, the second one is clear: "Pre-warm your caches before opening the floodgates."
Why Googlebot speeds up after the switch
Google adjusts a site's crawl rate based on server health (the "crawl capacity limit", or hostload):
- when response times go up, Googlebot slows down so it doesn't overload the server;
- when responses are fast and stable, it gradually speeds up, as long as it has pages to explore.
As soon as DNS points to the CDN, connection and response times drop. To Google, it looks like the server just got a major hardware upgrade. So Googlebot raises its rate.
The problem: a new CDN has an empty cache
A higher crawl rate wouldn't be a problem if the CDN absorbed every request. But a CDN that just started has nothing cached yet.
The first request for a URL, at a given point of presence, is a MISS: the CDN fetches the page from the origin, stores it, then serves it. Only later requests for that URL are HITs served without touching the origin.
Google stresses the difference between visitors and crawlers:
- Visitors cluster on a few popular pages (homepage, top articles, promotions). Those pages warm up fast.
- Crawlers do the opposite. Their job is to go through all your URLs, including the ones nobody visits.
The result: Googlebot crawls faster because the CDN responds fast, but every URL that is still cold is a MISS that goes straight to the origin. Until the cache is filled, the CDN doesn't protect the origin. The origin still does the work.
The wrong reaction: blocking all bots
Faced with a spike of automated requests in the logs, it's tempting to turn on blanket bot protection on the CDN or WAF. Google warns against it: Googlebot can't solve CAPTCHAs. Stuck on a challenge page, it backs off, and the affected pages eventually drop out of the index.
If you filter bots, allow verified bots (most CDNs have a ready-made rule for this), or target specific unwanted bots with robots.txt or dedicated rules.
Google's three recommendations
- Allow verified bots in your CDN and firewall settings. Google publishes its IP ranges so you can identify its crawlers.
- Warm the cache before routing all traffic. Programmatically request, through the CDN and at a controlled pace, your highest-traffic pages, the pages for upcoming peaks (promotions, seasonal launches) and your most crawled URLs.
- Brake temporarily with a 429 or 503 if crawling still overloads the origin while the cache warms up. A few hours, two days at most.
The other two recommendations are safety nets. Warming is the only one that fixes the cause: the cold cache.
What warming means in practice
The recommendation fits in one sentence, but doing it right takes some care. These are the same points we cover in the seven golden rules of cache warming.
Cover every URL, not just popular pages. Visitors already warm your top pages. It's the long tail that Googlebot will find cold. The simplest source is your sitemap, which lists exactly the URLs you want crawled.
Warm close to the points of presence. A CDN keeps one cache per point of presence. A page cached in Paris is cold in Frankfurt or Virginia. Googlebot crawls mostly from the United States: if you only warm in Europe, its requests will hit cold edges. We explain this in the post on warming from 12 regions.
Keep a controlled pace. The goal is to protect the origin, not to overload it instead of Googlebot. Warming should spread its requests out.
Check that pages stay cached. Requesting a page doesn't guarantee the CDN keeps it. A Cache-Control: private, a Set-Cookie or a Vary: Cookie is enough for the page to be served once and then forgotten. In that case warming does nothing and Googlebot falls back on the origin.
Warm again after every purge. The problem Google describes isn't limited to the day the CDN goes live. Every purge (deployment, content update, TTL expiry) makes the cache cold again, and Googlebot can come back at that moment.
How CacheBoost applies this recommendation
CacheBoost is a cache warming service. It applies what Google describes, and not only on switch day.
- Your sitemap as the URL list. CacheBoost reads your sitemap and warms every URL in it, including the ones your visitors never warm.
- 12 warming regions. Agents in Europe, the Americas and Asia-Pacific fill the points of presence your visitors and crawlers actually use.
- A limited pace. Each region sends a limited number of simultaneous requests, to fill the cache without saturating the origin.
- Cache check. After every boost, a sample of pages is requested again under the same conditions to verify they are served from cache. When a page isn't kept, the cause is shown. See the cache check after warming.
- Scheduled or triggered boosts. A cron expression to follow TTL expiry, a trigger from your deployment pipeline, or the WordPress plugin after every purge.
- Crawler tracking. SEO and GEO crawler hit tracking shows whether Googlebot and AI crawlers hit a warm or a cold cache.
Apply Google's first recommendation to your own warming traffic too: if your WAF challenges automated requests, allow CacheBoost traffic before the first boost.
A four-step switch procedure
To move to a CDN without overloading the origin:
- Prepare. Set up the CDN, allow verified bots and CacheBoost in your WAF rules, and lower your DNS record TTL a few days ahead so the switch propagates quickly.
- Switch DNS and start a boost right away on your sitemap, from your visitors' regions and from the United States for Googlebot. CacheBoost requests your public domain: it fills the CDN cache as soon as DNS points to it. Google notes that its crawlers don't ramp up immediately: that's the window to use.
- Review the cache check. Fix the pages flagged as not kept, then run another boost.
- Schedule a recurring boost to keep the cache warm after every purge and every expiry.
Cache warming isn't just a performance optimization. Google now explicitly recommends it to keep a new CDN from sending all crawling back to the origin. For the wider picture on search, read why cache warming matters for SEO and GEO.