A web crawler, also called a spider or bot, is software that systematically browses the web by fetching pages and following their links. Search engines use crawlers such as Googlebot to discover and index content.
Why it matters
Crawlers decide whether your pages can appear in search results. They respect rules in robots.txt, consume server resources and can overload a weak server. Other bots scrape content, probe for vulnerabilities or monitor uptime, so telling them apart matters.
Example
Googlebot requests /, reads the links, queues each new URL and fetches them over time. Your log shows lines such as Googlebot/2.1 in the user agent. A stricter robots rule or slow server can reduce how much it fetches.
Common mistakes
- Trusting the user agent to verify a bot; confirm with reverse DNS or published IP ranges.
- Blocking all bots and removing the site from search.
- Ignoring crawl errors reported by search consoles.
Related
Get notified when this term updates
Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.
यह पेज सामान्य जानकारी है, जिसे हमारी संपादकीय टीम ने लिखा है और दस्तावेज़ीकरण बदलने पर अपडेट किया जाता है। सर्वर और प्रदाता अलग-अलग होते हैं, इसलिए बदलाव पहले सुरक्षित तरीके से जाँचें। देखें: संपादकीय नीति और अस्वीकरण.