Glossary · Internet & Web

Web Crawler (Spider, Bot)

A web crawler is an automated program that visits web pages, follows links and collects content, mainly to build search engine indexes.

Updated October 6, 2026

A web crawler, also called a spider or bot, is software that systematically browses the web by fetching pages and following their links. Search engines use crawlers such as Googlebot to discover and index content.

Why it matters

Crawlers decide whether your pages can appear in search results. They respect rules in robots.txt, consume server resources and can overload a weak server. Other bots scrape content, probe for vulnerabilities or monitor uptime, so telling them apart matters.

Example

Googlebot requests /, reads the links, queues each new URL and fetches them over time. Your log shows lines such as Googlebot/2.1 in the user agent. A stricter robots rule or slow server can reduce how much it fetches.

Common mistakes

  • Trusting the user agent to verify a bot; confirm with reverse DNS or published IP ranges.
  • Blocking all bots and removing the site from search.
  • Ignoring crawl errors reported by search consoles.

Get notified when this term updates

Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.

Keep reading

Guide · E-commerceDoes Website Downtime Hurt SEO? What Google Says About 503 and OutagesHow outages affect Google crawling and rankings, why a 503 with Retry-After is the right response, how short and long outages differ, and how to recover.8 min readError fix · Server errorsApache "client denied by server configuration" (AH01630) FixFix Apache 403 errors logged as "AH01630: client denied by server configuration" by replacing old Order/Allow rules with Require and checking Directory blocks.4 min readError fix · Server errorsApache "Forbidden: You don't have permission to access this resourceApache 403 Forbidden on a new site is usually file permissions, a missing index file, SELinux or a Require rule. Use the error log to pick the right fix quickly.4 min readError fix · Server errorsHTTP 403 Forbidden: Why It Happens and How to Fix ItFix HTTP 403 Forbidden as a visitor or owner: file permissions, missing index, nginx and Apache access rules, WAF and CDN blocks, hotlink and IP restrictions.4 min readGlossary · Internet & WebBrowser CacheThe browser cache is local storage where a browser keeps copies of files it has downloaded so repeat visits can load faster without refetching them.Glossary · Internet & WebCache-ControlCache-Control is an HTTP header that sets caching rules, such as how long browsers and CDNs may reuse a response and whether they may store it.

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.