The robots.txt file: how it’s structured, which rules work, and how to check it

What robots.txt is and where it’s located

What the file is for

Crawling is a crawler loading pages. Google describes robots.txt primarily as a way to control crawling: it can be used to limit the number of crawl requests and reduce the load on the site. Typical candidates for disallowing are utility sections, internal search, the cart and other addresses the crawler has no reason to load.

robots.txt doesn’t protect access to pages. The protocol standard states outright that these rules aren’t an authorization mechanism. Google also notes that the rules aren’t honored by every search engine, and recommends protecting sensitive files on the server with a password or other means.

Where it lives and the file’s requirements

Google’s requirements for the file:

• the file is named robots.txt and lives in the root directory: https://example.com/robots.txt;
• there should be only one such file on the site;
• the rules apply only to the protocol, host and port where the file is hosted. A subdomain like https://shop.example.com/ has its own robots.txt;
• the encoding is UTF-8;
• rules are case-sensitive: /Catalog/ and /catalog/ are different paths.

Google’s maximum file size is 500 kibibytes (KiB); content beyond the limit is ignored. Yandex correctly processes a file up to 500 KB, located in the site’s root and served with a 200 OK code. Yandex disallows using Cyrillic characters in robots.txt.

Viewing any site’s file is simple: open the /robots.txt address at the domain’s root in a browser. In a CMS, the file is often generated by a plugin or theme, so check the resulting /robots.txt, not the admin-panel settings.

Crawling and indexing: what robots.txt can and can’t do

A page’s path into search takes three steps: the crawler loads the page, the system processes the content, then it stores information about the page in the index. robots.txt only acts at the first step: Disallow blocks loading, but it doesn’t remove an address from the search engine’s database.

So a blocked page’s address can still end up in the index another way. Google writes that a page disallowed in robots.txt can be indexed via links from other sites and shown in results with no description. Yandex’s help says that pages restricted in robots.txt can still appear in search.

A different tool helps remove a page from search — noindex in a meta tag or HTTP header — and it only works if the page isn’t blocked in robots.txt: otherwise the crawler won’t load the code and won’t see the rule. More on how noindex removes a page from search — in the article on indexing.

robots.txt also doesn’t merge duplicates or pick a page’s primary address. There’s a separate mechanism for that — we covered how to specify the primary address via canonical separately.

robots.txt syntax: groups, directives and rule order

Groups and User-agent

The file consists of groups. Each group starts with a User-agent line — it names the crawler the rules are addressed to — and contains one or more Disallow / Allow rules. The value * applies to every crawler that has no dedicated group. Text after a # character is a comment.

# A group for all crawlers
User-agent: *
Disallow: /search/

# A group only for Google's main crawler
User-agent: Googlebot
Disallow: /search/
Disallow: /drafts/

Google and Yandex pick a group differently:

• Google uses only one group per crawler — the one that names that crawler most specifically. If there are several groups for the same crawler in the file, Google merges their rules into one. The order of groups in the file doesn’t matter, and the case of the User-agent value isn’t significant.
• Yandex picks a group by precedence. If the file has User-agent: Yandex, the User-agent: * group is ignored. If there’s a group for a specific Yandex crawler, neither Yandex nor * is used.

The practical takeaway: if you add a separate group for a crawler, repeat all the general rules from * in it. Otherwise that crawler won’t see them.

Rules for AI services’ crawlers are set up the same way. Which ones exist and what their names mean is covered in the piece on which AI services’ crawlers exist and how to manage them.

Disallow and Allow

Disallow blocks crawling of sections or pages, Allow permits it. A rule’s path is given from the site’s root and starts with /.

User-agent: *
# Disallow crawling of /cart/ and everything inside it
Disallow: /cart/
# Allow one subsection inside the disallowed one
Allow: /cart/help/

A few special cases:

• Disallow: with no path disallows nothing. Yandex treats it as equivalent to Allow: /; Google ignores rules with no path.
• Disallow: / blocks crawling of the entire site.
• If no rule in a group matches an address, that address is allowed.

Which rule wins

If both Allow and Disallow match an address, path length decides, not the order of the lines in the file:

• Google applies the rule with the longer path. If rules conflict at equal length, the least restrictive one wins — that is, Allow.
• The RFC 9309 standard requires using the longest match; at an equal length between Allow and Disallow, Allow should apply.
• Yandex sorts a group’s rules by prefix length from shortest to longest and applies the last one that matches. At equal length, Allow takes priority.

An example:

User-agent: *
Disallow: /catalog/
Allow: /catalog/shoes/

The address /catalog/shoes/boots/ matches both rules. The path /catalog/shoes/ is longer, so the page is allowed. The address /catalog/bags/ only matches Disallow: /catalog/ and is blocked. Swapping the lines’ order wouldn’t change the result for either Google or Yandex.

The * and $ characters

In paths, Google and Yandex support two special characters:

• * — any sequence of characters, including an empty one;
• $ — the end of the address. By default a rule acts as a prefix: Yandex implicitly adds * to the end of the path, and $ cancels that asterisk.

User-agent: *
# Addresses where the query string starts with a sort parameter
Disallow: /*?sort=
# Only addresses that end in.pdf
Disallow: /*.pdf$

Yandex’s help calls these patterns regular expressions, but neither Google nor Yandex describes any special characters beyond * and $. So rely only on these two.

Be careful with the # character: everything after it is a comment. So Yandex would read the rule Disallow: /# as Disallow: / — that is, as blocking crawling of the whole site.

The Sitemap line

The Sitemap line tells crawlers the sitemap’s address. Both Google and Yandex support it. Under Google’s rules, the address must be full, with protocol and host. There can be several Sitemap lines, and they aren’t tied to a specific User-agent group.

Sitemap: https://example.com/sitemap.xml

Which addresses to include in the sitemap and how to submit it to search engines is covered in the article on what to include in a sitemap and how to submit it.

Clean-param — a Yandex directive

Clean-param tells Yandex’s crawler which GET parameters don’t change a page’s content. With this directive, the crawler won’t repeatedly load the same content at addresses that differ only by such parameters.

Syntax:

Clean-param: p0[&p1&p2&..&pn] [path]

• The first field lists the parameters separated by &; the second holds the path prefix the rule applies to.
• The rule’s length is limited to 500 characters, and case is significant.
• The directive is cross-section: it can be placed anywhere in the file.

Yandex can automatically drop some analytics parameters without any entry in robots.txt. Its help specifically lists utm_source, utm_medium, utm_campaign and ysclid. Clean-param is needed for the remaining parameters that don’t change the page’s content.

Google doesn’t support the directive: its list of supported fields only includes user-agent, allow, disallow and sitemap. Where duplicate pages caused by parameters come from, and how search engines pick a primary address among them, is covered in the article on canonical.

Crawl-delay and Host: directives that no longer work

Crawl-delay set a pause between a crawler’s requests. Google doesn’t support this directive. Yandex stopped honoring it on February 22, 2018; crawl speed is now set via the “Crawl rate” setting in Webmaster.

Host told Yandex the site’s primary mirror. On March 20, 2018, Yandex announced it was dropping the directive: the primary mirror is now determined by a page-by-page 301 redirect from the non-primary mirrors. You can delete the Host line or leave it — according to Yandex, the crawler simply ignores it.

Directive table

Directive	What it does	Google	Yandex
User-agent	Starts a group and names the crawler	Supported; the most specific group applies, and groups for the same crawler are merged	Supported; a Yandex or crawler-specific group overrides *
Disallow	Disallows crawling a path	Supported	Supported
Allow	Allows crawling a path	Supported	Supported
Sitemap	Points to the sitemap’s address	Supported, full URL, outside groups	Supported
Clean-param	Lists parameters that don’t change content	Not supported	Supported
Crawl-delay	Used to set a pause between requests	Not supported	Not honored since 22.02.2018
Host	Used to point to the primary mirror	Not in the list of supported fields	Not honored since 20.03.2018, replaced by a 301 redirect

A sample robots.txt with comments

The example below illustrates the syntax. It isn’t a template for a “correct robots.txt” — the set of rules depends on a specific site’s structure. The stylesheets, scripts and images in the example are deliberately left unblocked.

# Rules for all crawlers
User-agent: *
# A utility section
Disallow: /admin/
# Exception: public help pages inside the utility section
Allow: /admin/help/
# Internal search results
Disallow: /search/
# Cart and checkout
Disallow: /cart/
Disallow: /checkout/
# Addresses where the query string starts with a sort parameter
Disallow: /*?sort=

# Parameters that don't change catalog page content (only Yandex honors this)
Clean-param: sid&ref /catalog/

# Sitemap: full address
Sitemap: https://example.com/sitemap.xml

What’s happening here:

• one * group applies to every crawler that has no group of its own;
• /admin/help/ is open even though /admin/ is blocked: the Allow path is longer;
• Clean-param is placed at the end of the file: for Yandex it’s a cross-section directive that can go anywhere, while Google doesn’t honor this line at all;
• Sitemap is given as a full address.

How to check robots.txt

The robots.txt report in Google Search Console

Search Console has a “robots.txt report.” It shows which robots.txt files Google found, when it last crawled them, and what warnings and errors there were. File-fetch statuses: “Fetched,” “Couldn’t fetch – not found (404),” “Couldn’t fetch – any other reason.”

In the report you can view the file’s versions from the past 30 days. After fixing the file, you can request a recrawl: the “More” menu next to the file → “Request a recrawl.”

robots.txt analysis in Yandex Webmaster

The “robots.txt analysis” tool is located under “Tools.” It shows syntax errors and contradictions in the file. In it you can also list specific addresses and check whether they’re allowed or disallowed.

You can check edits before publishing: change the code in the tool’s field and click “Check.” You can check a file without having rights to the site. To see the file’s version history, the site needs to be added to Webmaster and have its ownership confirmed.

The server’s response for /robots.txt

What matters isn’t only the file’s content but also the server’s response code to a request for /robots.txt.

Google:

• 2xx — the file is processed;
• 3xx — Google follows at least five redirects in a chain;
• 4xx, except 429 — Google behaves as if there’s no file, meaning there are no crawling restrictions;
• 5xx and network errors — for the first 12 hours Google doesn’t crawl the site, then for up to 30 days it uses the last available version of the file. If the file stays unavailable longer, Google acts as if there’s no file when the site as a whole is accessible, and stops crawling if there’s a broader accessibility problem;
• caching: the file’s content is usually kept for no more than a day.

Yandex honors the file if the server responds with a 200 OK code, or redirects to another robots.txt that itself returns 200 OK. If the file doesn’t meet the requirements, the site is treated as open for crawling.

Checking sequence:

1. Open /robots.txt in a browser and make sure the file opens and contains the expected rules.
2. Check the file in Search Console and in Webmaster.
3. Run the site’s important addresses — the homepage, sections, key pages — through “robots.txt analysis.”
4. If the file is fine but a page still isn’t in search, look for other causes — there’s a separate article on why a page doesn’t make it into the index.

Post-release checklist

• /robots.txt returns a 200 code and opens in the browser.
• The file has no leftover Disallow: / from a staging version of the site.
• Stylesheets, scripts and images that change how a page renders aren’t blocked.
• Sections that should be in search are allowed — confirmed through a list of addresses in Webmaster.
• The Sitemap line points to the current sitemap with a full address.
• Pages that should leave search aren’t blocked in robots.txt, or the crawler won’t see their noindex.

Common robots.txt errors

Error	What it risks	How to fix it
Disallow: / left over from a staging version of the site	Crawling of the entire site is blocked	Compare robots.txt before and after a release; remove the rule
Disallow: /# or a path that’s too short	For Yandex, /# equals Disallow: /; a too-short prefix rule blocks every address that starts with that path	Check the rule against a list of addresses in “robots.txt analysis”
Stylesheets, scripts or images needed for rendering are blocked	Google warns that pages may be analyzed incorrectly	Only block resources that barely affect how pages are rendered
A page that needs to be removed from search is blocked	The crawler won’t see noindex; the page may stay in search	Open the page for crawling and use noindex
The file isn’t at the root, has a different name, or is for a different subdomain or protocol	The rules don’t apply to the host you need; for Yandex, the file doesn’t meet the requirements	Place robots.txt at the root of each host
File exceeds the limit	Google ignores content beyond 500 KiB; for Yandex, a file over 500 KB doesn’t meet the requirements, and the site is treated as open	Shorten the file: merge rules using *
/robots.txt returns 5xx, 4xx, or a redirect that doesn’t lead to a robots.txt	Google pauses crawling on 5xx and treats 4xx as if there’s no file; Yandex treats the site as open unless the file is served directly, or after a redirect to another robots.txt, with a 200 OK code	Set up the file to be served with a 200 code
Relying on Crawl-delay or Host	The directives don’t work	Crawl speed — in Webmaster’s settings; primary mirror — via a 301 redirect

If errors affect many sections, or appeared after a release, you need a check of the whole site, not just one file. That’s part of technical SEO work — more on how a technical audit of crawl settings works.

Frequently asked questions

Why is a page blocked in robots.txt still in search?

Because robots.txt disallows loading, not indexing. Google may index such a page via external links and show it with no description; Yandex’s help says such pages may still appear in search. More — in the “Crawling and indexing” section.

Do you need a robots.txt file?

No. If there’s no file and the server returns a 4xx code, Google treats it as if there are no restrictions. Without a valid file, Yandex treats the site as open. You need the file when the crawler has no reason to load some of the addresses.

How quickly will search engines see changes?

Google usually caches the file for no more than a day; after an edit, you can request a recrawl in the Search Console report. Yandex’s help doesn’t give a timeframe for picking up changes; the file’s versions are visible in “robots.txt analysis.” Updating the file doesn’t mean the index will change.

Do you need a separate User-agent: Yandex block?

Not necessarily: without it, Yandex’s crawler follows the * group. If such a block exists, Yandex ignores *, so you need to repeat the general rules inside it.

Can you block AI services through robots.txt?

You can set rules for specific crawler names. But the rules aren’t honored by every system and aren’t an authorization mechanism, so you can’t rely on them completely. More on crawler names — in the GEO article linked above.

Are there regular expressions in robots.txt?

Only two special characters: * and $. More — in the “* and $ characters” section.