robots.txt and XML Sitemap Guide: Syntax, Limits and Mistakes

Two small files tell search engines how to crawl your site: robots.txt says where crawlers should not go, and sitemap.xml says which URLs you want them to find. Both are easy to write and easy to get subtly wrong. After this guide you will be able to write valid rules, avoid the classic mistake of blocking your whole site, and build sitemaps that stay inside the documented limits.

What robots.txt is, and is not

robots.txt is a plain-text file at the root of a host, such as https://example.com/robots.txt. Well-behaved crawlers fetch it before crawling and follow its rules. Three points shape everything else.

1. It controls crawling, not indexing. A URL blocked in robots.txt can still appear in search results if other pages link to it, usually without a description, because the crawler could not fetch the page.
2. It is not security. The file is public and compliance is voluntary. Anyone can read it, and a malicious bot can ignore it. Listing /secret-admin/ in robots.txt even advertises its existence. Protect private content with authentication.
3. It applies per host and protocol. https://example.com/robots.txt does not cover https://blog.example.com/ or a different port; each subdomain needs its own file.

Syntax fundamentals

A file consists of groups. Each group starts with one or more User-agent lines and is followed by rules.

# Rules for all crawlers
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

# A stricter group for one named crawler
User-agent: Googlebot-Image
Disallow: /private-images/

Sitemap: https://example.com/sitemap.xml

Key facts:

• Directive names are case-insensitive; paths are case-sensitive. /Photos/ and /photos/ are different.
• Each path is matched as a prefix from the start of the URL path. Disallow: /search blocks /search, /search?q=x and /search-results/.
• An empty Disallow: means nothing is blocked. Disallow: / blocks everything.
• Comments start with #.
• A crawler uses the most specific User-agent group that matches its name and ignores the others; it does not merge a named group with the * group. If you create a group for Googlebot, repeat any general rules you still want applied to it.
• The file must be UTF-8 plain text, served with a 200 status as text/plain.
• Google reads only a limited size of the file (documented as 500 KiB) and ignores the excess, so keep it compact.

Allow versus Disallow precedence

When more than one rule matches a URL, the standard (RFC 9309) and Google's implementation choose the most specific rule, meaning the longest matching path. If an Allow and a Disallow match with equal length, the Allow wins (least restrictive). Example:

User-agent: *
Disallow: /downloads/
Allow: /downloads/public/

/downloads/public/guide.pdf is crawlable because the Allow path is longer; /downloads/internal.pdf is blocked. Some other crawlers apply simpler first-match logic, so for important cases put the more specific rule first as well, which works under both.

Wildcards

Major crawlers support two pattern characters, though the original specification did not define them and tiny parsers may ignore them:

• * matches any sequence of characters.
• $ anchors the match to the end of the URL.

User-agent: *
Disallow: /*?sessionid=
Disallow: /*.pdf$

The first rule blocks any URL containing ?sessionid=. The second blocks URLs that end in .pdf but not /file.pdf?download=1, because $ requires the URL to end there.

Crawl-delay and other directives

Crawl-delay: 10 asks a crawler to wait ten seconds between requests. Support is inconsistent: Googlebot ignores it, while some other crawlers, historically including Bing's, honor it. To slow Google, use the crawl rate settings in its tools, or return 503/429 temporarily when your server is overloaded. Other unofficial directives such as Host or Noindex are not part of the standard, and Google does not support Noindex in robots.txt. The Sitemap directive is widely supported and must use an absolute URL.

noindex versus Disallow

This is the most misunderstood part, and the confusion causes real damage.

• Disallow says "do not fetch this URL".
• noindex says "do not show this page in search results", and it must be seen by the crawler, either as a meta tag or as an HTTP header:

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

If you both disallow a page and put noindex on it, the crawler cannot fetch the page and never sees the noindex, so the URL may stay indexed from external links. To remove a page from search results, allow crawling and serve noindex, wait until it is processed, and only then, if you want, block it. For private content, use authentication; for outdated pages, return 404 or 410. You can check what a URL actually returns with the Meta Tag Checker and HTTP Header Checker, which shows X-Robots-Tag too.

Also note that noindex does not stop non-search bots from reading the page, and neither mechanism hides content from people.

Common mistakes

Leaving staging rules in production

Staging sites commonly use:

User-agent: *
Disallow: /

That is correct for staging and disastrous if copied to production during a launch, because it asks every crawler to stay away from the whole site. Add a deployment check that fetches /robots.txt and fails if it contains Disallow: / on production. Also check meta robots tags and X-Robots-Tag headers, since staging builds often set noindex there too. If you made this mistake, fix the file immediately, then request a recrawl; recovery usually takes some days or longer.

Blocking CSS and JavaScript

Search engines render pages. If you disallow /wp-content/, /assets/ or script and style directories, they cannot render the page as users see it, which can hurt understanding of layout and mobile-friendliness. Allow static resources needed to render. For WordPress, a typical safe file is:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Using robots.txt to fix duplicate content

Blocking parameter URLs is sometimes appropriate to save crawl budget on huge sites, but for duplicate content prefer canonical tags or redirects, which consolidate signals. A blocked URL cannot reveal its canonical tag.

Wrong status codes for the file itself

Google's documented behavior: a 404 on robots.txt means no restrictions; a 5xx means it temporarily treats the site as fully disallowed, and if the error persists for a long period it may fall back to a cached copy or assume no restrictions. A server error on /robots.txt can therefore pause crawling of the entire site. Check the status with the HTTP Status Checker, and test your rules with the robots.txt Checker. Also note redirects from /robots.txt are followed in a limited way; serve it directly.

XML sitemaps

A sitemap lists canonical, indexable URLs you want discovered, with optional metadata. It helps most on large sites, new sites with few inbound links, and pages deep in the structure. It is a hint, not a command: listing a URL does not guarantee that it is crawled or indexed.

Structure

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guides/robots-txt</loc>
    <lastmod>2026-10-04</lastmod>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2026-08-19T09:30:00+00:00</lastmod>
  </url>
</urlset>

Rules:

• <loc> must be an absolute URL with the correct scheme and host, and special characters must be entity-escaped (for instance & becomes &).
• Only list URLs on the same host as the sitemap (with exceptions for verified cross-submission).
• The file must be UTF-8 encoded.
• List only URLs that return 200, are canonical and are indexable. Redirected, blocked, noindex and error URLs waste crawl attention and make reports noisy. Find them with the Broken Link Checker and Redirect Checker.

Limits and sitemap indexes

A single sitemap file may contain at most 50,000 URLs and must be no larger than 50 MB uncompressed. Gzip compression (sitemap.xml.gz) is allowed for transfer, but the limit applies to the uncompressed size. For larger sites, split into several files and reference them from a sitemap index, which can itself list up to 50,000 sitemaps:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemaps/posts-1.xml</loc>
    <lastmod>2026-10-01</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemaps/products-1.xml</loc>
    <lastmod>2026-10-03</lastmod>
  </sitemap>
</sitemapindex>

Splitting by content type (posts, products, categories) rather than by arbitrary count makes it easier to see which section has indexing problems.

lastmod, priority and changefreq

Use <lastmod> only when it reflects the last significant change to the page content, in W3C datetime format. Google has said it uses lastmod when it is consistently accurate, and may disregard it if it is always set to the current time. The <priority> and <changefreq> fields are ignored by Google, so you can omit them.

Telling crawlers where the sitemap is

Add the Sitemap: line to robots.txt, and submit it in the webmaster tools of each search engine you care about. The line is independent of any user-agent group and may appear multiple times.

A launch checklist

1. robots.txt returns 200 with the intended rules, and does not contain a stray Disallow: /.
2. Important CSS, JavaScript and image directories are crawlable.
3. Pages you want removed from search use noindex and are crawlable, not merely disallowed.
4. The sitemap lists only canonical 200 URLs and stays under the limits.
5. The sitemap is referenced from robots.txt and submitted to search engines.
6. Redirects are clean; see HTTP redirects 301 vs 302.
7. A recurring check, such as a website health check, catches regressions after deploys.

Next steps

• Validate rules with the robots.txt Checker
• Verify page directives using the Meta Tag Checker
• Find dead links in your sitemap via the Broken Link Checker
• Confirm the file returns 200 with the HTTP Status Checker
• Read how redirects affect canonical URLs

FAQ

Can robots.txt hide a page from Google?

No. It prevents crawling, but the URL can still be indexed if other sites link to it. To keep a page out of results, use noindex on a crawlable page, authentication, or removal with a 404 or 410 response.

Where does robots.txt need to be?

At the root of each host and protocol, for example https://example.com/robots.txt. Placing it in a subdirectory has no effect, and every subdomain needs its own file.

Do I need a sitemap if my site is small?

Not strictly. A small, well-linked site is found without one. A sitemap is still cheap, helps with new content discovery, and gives you indexing diagnostics in search tools.

Does Google honor Crawl-delay?

No, Googlebot ignores it. Some other crawlers do honor it. If Google is overloading your server, respond with temporary 503 or 429 codes and fix capacity, rather than relying on the directive.

What happens if my sitemap exceeds the limits?

Search engines may reject or truncate it. Split it into multiple files under 50,000 URLs and 50 MB uncompressed each, and reference them from a sitemap index.

Should I list noindex or redirected URLs in my sitemap?

No. The sitemap should contain canonical URLs that return 200 and are meant to be indexed. Including others sends mixed signals and clutters your coverage reports.

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.