Google crawlers: how do the various Google bots work?

What are Google crawlers?

Google crawlers can be compared to all the little spiders that will search your website. The goal of these Googlebots is to get as much information about a website as possible. By thoroughly examining each page, Google tries to get the best possible picture of your website's content. By reviewing all of this data, Google then determines which websites and pages are best suited for specific search terms and thus which websites they want to rank highest in the SERP.

Which Google bots are crawling your website?

These Googlebots use Google to read the content on your website:

• Googlebot Smartphone: For many websites, more than 50% of visitors come from a mobile device. To provide mobile users with an optimal experience, Google has launched a separate smartphone bot for this purpose. This Googlebot specifically checks whether certain pages and sections on a website are optimized for mobile users. If this is not the case, Google is likely to show these pages lower in search results when searched via cell phone.
• Googlebot Desktop: Along with Googlebot Smartphone, this is the largest crawler used by Google. However, whereas Googlebot Smartphone specifically looks to see if the content is optimized for mobile use, this bot actually looks at optimization for desktop. So a page that is not optimized for desktop will not rank as well in search results from desktop because of this bot.
• Googlebot Image: This crawler was launched specifically to search the Web for images. The purpose of this bot is not only to search for images, but also to find out the image's function and context. With that information, Googlebot Image then assesses how relevant an image is to a particular search term.
• Googlebot Video: This Google crawler specializes in examining videos. Similar to the Googlebot Image, this bot not only looks at the presence of a video, but also examines the size and context of the video. Based on this data, it then determines the quality and relevance of the video.
• Google Storebot: This Googlebot crawls all pages related to an online store. This includes product pages, shopping carts, payment pages and so on. Google Storebot tries to go through all the steps in a visitor's buying process to get information about prices, payment methods, delivery and service.
• Google Inspection tool: This is the crawler used in various tools for test results. Examples include the Rich Result Test and URL inspection in Google Search Console.
• GoogleOther: GoogleOther is used internally by Google. The reason for launching this crawler is to relieve the other Google bots and thus free up crawl capacity.
• Google Extended: Google Extended is a stand-alone product that can be used to indicate whether a website may be used to enhance generative APIs from Gemini Apps and Vertex AI. Google Extended does not affect a website's inclusion or positioning in search results.
• AdsBot Google: This crawler crawls all pages used in your GoogleAds ads. This looks at the quality of the landing page to ultimately determine how relevant the page is relative to the intent of the keyword being advertised.

How can I keep these Googlebots out?

If you don't want Googlebots to crawl your page(s), you can indicate this in the robots.txt. Here you can use a "disallow" per crawler to indicate that Google does not need to view the entire website or certain pages. You can do this for pages that you don't think are important. Indeed, by excluding these pages from the crawl process, more crawl budget remains for the pages that are important.

Please note that blocking crawlbots can have a major impact on the performance of your website. For example, if you block the GoogleAds bot, your Google Ads ads will no longer be shown from then on. So think carefully before you ban a Googlebot!

Viewing crawl statistics

So behind the scenes, a lot is happening on your website. Fortunately, it is possible to gain insight into the crawl activities of these Google bots. You can find the crawl statistics in Google Search Console. Here you can see exactly which Googlebots have been on your website, what their purpose was, what content they viewed and what status codes the Googlebots encountered.

This last result in particular is interesting to keep an eye on. After all, you want your crawl budget to be used as efficiently as possible and for the Googlebot to crawl as many status code-200 pages as possible.

Want to know more?

In our blog: Crawl statistics in Search Console: what you need to know about them, we go into more detail about the different crawl statistics displayed in Google Search Console.