Optimizing a site for AI: how to prepare it for AI search

Accessibility for the crawlers that assemble AI answers

Google Search’s generative features run on the same pages as regular search. Google states directly that content doesn’t need to meet extra requirements or special additional work to be used in these features — it’s enough for the page to be indexed and eligible to show a snippet in Search.

How to restrict or allow access in robots.txt

Crawl rules for the crawlers involved in Google’s AI features are set the same way as for any other search crawler — through groups and User-agent directives. How the syntax of these rules works is covered in the article on robots.txt; here — only what specifically concerns visibility in generative answers.

If you need to limit exactly what shows up in the snippet or in the answer itself, Google points to separate controls instead of a full crawl block: “robots.txt directives for Googlebot is the control for site owners to manage access.” To limit what’s shown from your pages, Google says to “use nosnippet, data-nosnippet, max-snippet, or noindex controls.” We don’t repeat the full list of different AI services’ crawlers and what their names mean — that table is in the GEO article.

Training AI models and appearing in search are separate settings

At Google these are two separate settings. Googlebot handles crawling for Google Search: rules addressed to this crawler affect Google Search, including recommendations and all of Google Search’s features. Google-Extended is a separate crawler. Google describes it as “a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models,” and states that “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”

So disallowing Google-Extended doesn’t remove a site from Google Search’s answers, while changing the rules for Googlebot does — and it removes it from both modes at once.

In practice, this mistake usually isn’t intentional. For example, a blanket robots.txt rule is meant to block scrapers and aggregators, but its wording ends up blocking Googlebot too. Then the page disappears not just from regular search results, but from every feature built on top of them, including AI features. Checking rules like this is part of a regular technical audit, not a separate task just for generative search.

Step	Status
Confirm the page is indexed and eligible to show a snippet	confirmed by a source
Don’t block, with a blanket rule, the crawler that needs to crawl the site to appear in Search	confirmed by a source
Keep “model training” and “search display” decisions separate when configuring robots.txt	confirmed by a source
Fully control exactly what the system will cite in its answer	market assumption — no source describes this

A consistent description of the company, services, and author

A generative system picks a few pages for its answer, not the whole site. If information about the company, a service, or the author is scattered across several similar pages or contradicts itself, it’s harder for the system to work out which version is correct and which page to show as the source.

Platforms don’t have an official recommendation that says “make your entities consistent for the sake of AI answers.” But Yandex’s recommendations on visibility in Alice AI (Yandex’s AI assistant) answers include a related idea: “expand topic coverage — create material relevant to the primary and secondary intent” — meaning genuinely different new pages, not near-identical copies built around the same question.

• One page, one question — no duplicates for the same intent.
• The name of the company, services, and products is the same across every page of the site.
• The material’s author is credited and doesn’t change from page to page.
• Topics are expanded with new, relevant pages, not copies of existing ones.

The first, second, and third points are a recommendation based on general SEO logic, not a platform’s direct wording. The fourth is confirmed by a source.

In practice, inconsistency looks simple. A site has a service page, a case study card, and several blog articles where the service’s name or approach is described slightly differently. A person reads the text in context and rarely notices the difference. For a system assembling a short answer out of fragments from different pages, it’s easier when the wording is the same everywhere.

The structure of a page that answers the question

Both Google and Yandex describe content quality in similar terms, though they’re talking about different systems. In its guide to optimizing for generative-AI-based features, Google recommends creating helpful, trustworthy, people-first content that isn’t formulaic, and suggests “providing a unique point of view,” noting that “it can be helpful to have a unique viewpoint that stands out.”

Yandex writes that “Alice AI uses pages filled with quality content as sources,” and that when selecting sources, “these aspects matter most: expertise; usefulness; originality; substance.”

This leads to a practical conclusion that the platforms don’t state directly, but that follows logically from their descriptions:

• the answer to the page’s main question comes at the start, not in the middle or the conclusion;
• the page has one job, and it doesn’t repeat another page on the site almost verbatim;
• section headings reflect real sub-questions rather than existing only to pad out the length;
• where it’s appropriate, a fact is backed by a link, not a vague phrase.

Status	What’s confirmed
confirmed by a source	Helpful, people-first, non-formulaic content with its own point of view
confirmed by a source	Expertise, usefulness, originality, and substance of the source page
market assumption	A specific block order (“answer first”) as a mandatory condition for generative search

The difference shows up in practice. If a page opens with the company’s history and only gets to the answer midway through the text, it’s equally hard for a reader and a system to find the main point. If the first paragraph answers the question right away, with details and sources following, the same content is faster to read — for both a person and whatever is preparing a short summary.

Facts with sources that can be checked

Yandex states directly: “Alice AI doesn’t reproduce page content verbatim — except when quoting direct speech.” The system paraphrases rather than copies. That means if a fact on your page is worded vaguely or without a source, the paraphrase risks distorting it even further, and it becomes harder for a reader to check the original point.

That gives us a practical rule: every verifiable claim — a number, a date, a name, a result — should point to a specific source, not a vague phrase like “studies show” with no named study. This isn’t specific to generative search; it’s an ordinary requirement for substantive text that simply becomes more visible once a system is paraphrasing it.

• Cite the original source of a specific fact, not a link to a general topic.
• Don’t phrase an opinion as a proven fact, or the other way around.
• If there’s no exact data, it’s better to say so directly than to write something that merely sounds plausible.

This is a market assumption when it comes to “exactly how this affects making it into the answer,” but not when it comes to facts needing to be verifiable — that’s a requirement for text regardless of whether a person or a system is reading it.

The difference is visible in an example. The claim “most sites lose rankings after a redesign” doesn’t confirm anything without a link to who measured that and how. The claim “Google states that a page must be indexed to appear in AI features” is specific and points to a document you can open and check. The second type of claim is more reliable both for the reader and for the system paraphrasing it.

Markup and structured data, without extra files

The temptation to build something separate for AI — a special file, special markup — comes up often. Google addresses this directly: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features. There’s also no special schema.org structured data that you need to add.”

In its guide to optimizing for generative-AI-based features, Google puts this more broadly: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.” This also covers files like llms.txt: per the same documentation, Google doesn’t require a separate file for AI, and a detailed look at the format is in the article on llms.txt.

If a site already has structured data, generative search doesn’t need a separate set of it: what matters is that the markup matches the page’s visible text — this principle, along with how Schema.org’s types and relationships work, is covered in the article on Schema.org markup.

Status	What’s confirmed
confirmed by a source	Separate schema.org structured data isn’t needed for AI features
confirmed by a source	Special machine-readable files, special markup, and Markdown aren’t needed to appear in AI answers
market assumption	Any third-party “AI markup” services provide a measurable advantage

If the organization, services, or articles on a site are already marked up with Schema.org, that’s enough — there’s no need to duplicate the markup just because a page might end up in a generative answer. What matters more is making sure the markup names the same company and service as the visible text does: without that, the markup won’t actually be consistent with the content.

Index availability and the absence of duplicates

The basic condition for everything else is ordinary availability. Google states that a page must be indexed and eligible to show a snippet in Search; this applies to generative features too. If a page isn’t indexed, the question of markup or answer structure stops mattering.

The second part of this block is the absence of competing pages for the same question. If several pages on a site answer the same query, the system (like regular search) has to pick one of them, and it isn’t always the page you consider the primary one. Platforms don’t describe this as a separate rule for generative search — it’s general SEO logic that simply becomes more visible here.

• Check that key pages are indexed and not accidentally blocked.
• Find pages that answer the same question and decide which one is the primary one.
• Don’t create a new page for a query variation if an existing page already covers the question.

In practice, duplicates usually appear unintentionally, along the way. A blog article, a service card, and an FAQ on the homepage all end up covering the same question — and all three were written independently of each other. In that situation, a generative system picks one version, and not necessarily the most complete one. That’s why reviewing overlapping pages is worth doing regularly, not just once.

None of the search engines or generative services guarantees that a page will appear in a specific answer, even if every step above is done: the decision about which sources to use and how to paraphrase them belongs to the system itself. You can influence what depends on the site — availability, consistency, content quality, and verifiable facts. You can’t control the answer itself.

How AI tools help an SEO specialist’s own work — as opposed to how a site looks to generative search — is a separate topic. That’s covered in the article on AI in SEO.

Frequently asked questions

Do you need a separate file or special markup “for AI”?

No. Google states directly that AI features don’t need special files, special markup, or separate schema.org structured data. More in the “Markup and structured data, without extra files” section.

Can you guarantee a brand will be mentioned in AI answers?

No. The decision about which sources to use and how to paraphrase them belongs to the system itself. Yandex only gathers visibility statistics for Alice AI answers on queries where the site already ranks high enough in regular search results — meaning ordinary search visibility has to come first.

What’s the difference between the Googlebot and Google-Extended settings?

Googlebot controls crawling for Google Search and all its features, including AI features. Google-Extended is responsible only for whether content is used to train future models, and doesn’t affect how the site appears in search. More in the “Training AI models and appearing in search are separate settings” section.

Do you need a separate page “for AI search”?

No. The logic is the same as for regular SEO: one page covers one question, and the fewer near-identical pages a site has, the easier it is for the system to choose which one to use.

Where do you start if a site already has several pages for the same question?

First, determine which page should remain the primary one, and which should be updated, merged, or trimmed. This is a job for a technical and content audit, not a one-off edit to a single page.

Do you need to rewrite all of a site’s old content for this checklist?

Not all of it. Start with the pages that cover questions important to the business and that are competing for attention with other pages on the site. That’s the same priority as in the block on availability and duplicates.