Glossary · Internet & Web

robots.txt

robots.txt is a plain text file at a site's root that tells crawlers which paths they may or may not request; it is a guideline, not access control.

Updated October 6, 2026

robots.txt is a text file placed at the root of a domain, such as https://example.com/robots.txt, that lists rules for web crawlers. It uses User-agent, Disallow and Allow directives and can point to a sitemap.

Why it matters

The file shapes what well-behaved crawlers fetch, helping you save crawl budget and keep admin or duplicate areas out of the crawl. It does not secure anything: the file is public and disobedient bots can ignore it. A single wrong line can block your whole site.

Remember that the file applies per host: blog.example.com needs its own robots.txt, and the rules are case sensitive for paths.

Example

User-agent: *
Disallow: /admin/
Allow: /admin/help
Sitemap: https://example.com/sitemap.xml

Common mistakes

  • Writing Disallow: / on production after copying a staging file.
  • Using it to hide private pages; use authentication, and noindex for search removal.
  • Blocking CSS or JavaScript that crawlers need to render the page.

Get notified when this term updates

Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.

Keep reading

Glossary · Internet & WebSitemap (XML Sitemap)A sitemap is a file listing a site's URLs, usually in XML, to help search engines discover and understand the pages you want indexed.Glossary · SEO & MarketingGoogle Search ConsoleGoogle Search Console is a free Google tool that shows how your site performs in Google Search, including queries, clicks, indexing status and errors.Glossary · SEO & MarketingTechnical SEOTechnical SEO is the work of making a site easy for search engines to crawl, render and index, covering speed, status codes, sitemaps and site structure.Guide · Web Basicsrobots.txt and XML Sitemap Guide: Syntax, Limits and MistakesWrite a correct robots.txt, understand Allow and Disallow precedence, noindex versus disallow, and build XML sitemaps within the 50,000 URL and 50 MB limits.8 min readGuide · E-commerceShopify Redirects and URL Structure: Fixes, Limits and Best PracticesHow Shopify URLs are structured and how to create and bulk-import URL redirects, handle changed handles and deleted products, and avoid chains and 404s.7 min readGuide · E-commerceDoes Website Downtime Hurt SEO? What Google Says About 503 and OutagesHow outages affect Google crawling and rankings, why a 503 with Retry-After is the right response, how short and long outages differ, and how to recover.8 min read

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.