robots.txt is a text file placed at the root of a domain, such as https://example.com/robots.txt, that lists rules for web crawlers. It uses User-agent, Disallow and Allow directives and can point to a sitemap.
Why it matters
The file shapes what well-behaved crawlers fetch, helping you save crawl budget and keep admin or duplicate areas out of the crawl. It does not secure anything: the file is public and disobedient bots can ignore it. A single wrong line can block your whole site.
Remember that the file applies per host: blog.example.com needs its own robots.txt, and the rules are case sensitive for paths.
Example
User-agent: *
Disallow: /admin/
Allow: /admin/help
Sitemap: https://example.com/sitemap.xmlCommon mistakes
- Writing
Disallow: /on production after copying a staging file. - Using it to hide private pages; use authentication, and
noindexfor search removal. - Blocking CSS or JavaScript that crawlers need to render the page.
Related
Get notified when this term updates
Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.
यह पेज सामान्य जानकारी है, जिसे हमारी संपादकीय टीम ने लिखा है और दस्तावेज़ीकरण बदलने पर अपडेट किया जाता है। सर्वर और प्रदाता अलग-अलग होते हैं, इसलिए बदलाव पहले सुरक्षित तरीके से जाँचें। देखें: संपादकीय नीति और अस्वीकरण.