Crawl Budget Strategy: How to Get Google to Crawl the Right Pages
OMA B.V.
Dieselstraat 1B
6716 BC Ede
Chamber of Commerce: 83301852
A SERP is the page where Google displays search results. AI-SERP features are sections in which Google combines information from multiple sources into a single answer. The best-known example is the AI Overview. This usually appears above the organic results and sometimes includes references to the websites used. Product comparisons, local results, and follow-up questions are also increasingly being presented in an AI-driven format. As a result, users can see a complete answer more quickly, without having to open multiple pages first.
Not every search result contains the same elements. Google adjusts the display based on the search query. For informational queries, a summary is more likely to appear. Commercial search queries more often yield products, stores, or comparisons.
During an analysis, pay attention to the following, among other things:
Determine which elements take up the most space for each keyword. This will show you which results are capturing the user’s attention.
A ranking only tells you where your page stands among the organic results. It doesn’t show the position of other elements. An AI Overview may appear above your result and take up a large portion of the screen. As a result, the click-through rate sometimes drops even though your position remains the same. By analyzing AI SERP features, you can determine whether a drop in traffic is related to changes in the search results page. You’ll also see which competitors appear as sources and which topics your website is still missing.
SEO is no longer just about securing a spot among the regular search results. Visibility in summaries, source citations, and other search features also counts. Google can use a page as a source without showing it in the number one position. Clear answers, demonstrable expertise, and a logical structure therefore become more valuable. So you’re not just measuring rankings and clicks. You’re also looking at brand mentions, sources used, and a feature’s position within the search results.
For simple questions, Google sometimes provides sufficient information right away. The user then doesn’t need to click through. This happens mainly with definitions, brief explanations, and simple step-by-step guides. For more complex topics, a website visit is often still necessary. In such cases, people are looking for examples, pricing, research, or personalized advice. Therefore, consider the expected click behavior for each search intent. A lower number of clicks isn’t always a bad outcome. A visible source attribution can also contribute to brand recognition and trust.
Choose keywords that are important for your revenue, leads, and visibility. Include a variety of search intents so you can see which queries trigger Google’s AI features. For example, use informational questions, comparisons, service-oriented terms, and local searches. Also include keywords for which you already rank well. It’s precisely with those terms that you’ll quickly notice when a new feature affects your click-through rate. Keep the list short at first. A fixed set of key keywords provides more insight than a large list that you rarely monitor.
For each measurement, take a screenshot of the entire search results page. Also note the date, location, device, and search query used. This information is necessary because results vary by user and at different times. Additionally, record where the AI feature is located, how much space it takes up, and which websites are used as sources. Repeat this measurement on a regular schedule. This will give you an overview that allows you to compare changes. Without a baseline measurement, you won’t know afterward which elements have shifted.
See which pages Google includes in the generated answer. Then check what these sources have in common. Pay attention not only to the amount of text, but especially to the way the information is presented.
For example, check:
Here’s how to find out what content Google considers relevant for this topic.
A competitive analysis isn’t about copying text. You want to discover which questions other pages answer and what information is still missing. So, for each important keyword, create an overview of topics, examples, sources, and formats. Also look at tables, images, and practical steps. Does every competitor provide the same general explanation? If so, add your own experiences, research data, or a concrete real-world example. This gives visitors a reason to open your page and read it for longer.
Combine SERP data with Google Search Console and your analytics software. Check the position, impressions, clicks, and click-through rate for each keyword. Also, note whether your website appears as a source in an AI feature. This will help you see the difference between organic ranking and AI visibility.
Pay special attention to these situations:
Start each page with a clear answer to the main question. Then expand on that answer with details that help the visitor further. Use logical headings, short paragraphs, and bullet points when they make the information easier to scan. Incorporate your own expertise, such as examples from client projects, research findings, or insights from specialists. Also, ensure that authorship, publication date, and sources used are clearly visible. This makes the content easier to understand and evaluate, without having to write the text specifically for an algorithm.
Good content is of little value if Google can’t properly process the page. Therefore, make sure important pages are indexable and contain the correct canonical tag. Use internal links to connect relevant topics. Only add structured data when it aligns with the page’s content. Also, pay attention to load time, mobile display, and error codes. These elements don’t automatically guarantee a ranking, but they do prevent technical issues from limiting your visibility.
A one-off check provides little insight. AI-SERP features can change from week to week. Therefore, choose a set time to review your key keywords. A monthly schedule is a good starting point for many websites. Compare the new results with previous screenshots and Search Console data. Note which sources disappear, which ones are added, and how your click-through rate changes. Also keep track of which pages you’ve updated. This will help you better assess which changes may have had an effect.
The biggest mistake is drawing conclusions based on a single search query. Results vary by location, device, and time. Therefore, look at multiple data points. Also, keep in mind that a drop in clicks can have various causes. A new AI feature is just one possibility. Other mistakes include ignoring brand mentions, optimizing every page at once, and blindly following competitors. Start with keywords that demonstrably influence traffic or revenue. That’s where your analysis will yield the most useful information.
Use this order to stay organized:
With this approach, you’ll not only see what Google is changing, but you’ll also discover which changes to your website should be prioritized.
Is your click-through rate dropping even though your rankings are barely changing? Then it’s time to look beyond the familiar blue links. OMA investigates which AI-SERP features appear for your most important keywords, which sources Google uses, and where your website is missing opportunities. We’ll then translate those insights into concrete improvements for your content and technical setup. Contact us to have your AI-SERP visibility analyzed. That way, you’ll know exactly where you stand and what your first step should be.
Written by: Mehdi Achour
Mehdi is an online marketer at OMA and helps clients grow with smart SEO, strong content and data analysis, always based on facts, not wet-finger work.