A bot (short for robot) is a software program that performs automated tasks over the internet, usually much faster and at higher volume than a person could. Bots can be helpful or harmful depending on who runs them and why.
Why it matters
A large share of web traffic is automated. Good bots include search engine crawlers, uptime monitors and feed readers. Bad bots scrape content, stuff credentials into login forms, buy limited stock, post spam or take part in denial-of-service attacks. Site owners need to tell the two apart, because blocking everything harms search visibility while allowing everything harms security and server load.
Example
A server log shows a request with the user agent Googlebot/2.1 and another with a browser-like agent hitting /login 50 times a second. The first can be verified through reverse DNS and is welcome. The second is likely a bot and a good candidate for rate limiting.
Common mistakes
- Trusting the user agent string alone. It is trivially faked, so verify important crawlers by IP or DNS.
- Blocking all bots in a firewall rule and accidentally cutting off search engines or monitoring.
- Confusing a bot with a botnet, which is a network of compromised machines controlled by an attacker.
Related
Get notified when this term updates
Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.
This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.