Glossary · Monitoring

Chaos Engineering

Chaos engineering is the practice of deliberately injecting controlled failures into a system to find weaknesses before real outages expose them.

Updated October 6, 2026

Chaos engineering is experimenting on a system by introducing failures such as stopped servers, added latency or blocked network links, to see whether it behaves as expected. The approach was popularized by Netflix's Chaos Monkey tool, which terminated production instances at random.

Why it matters

Redundancy and failover only count if they work. Many teams find through real outages that a backup node was misconfigured or an alert never fired. Controlled experiments surface these gaps during working hours, with people ready, instead of at 3 a.m. They also test monitoring, runbooks and on-call response. A rehearsal event of this kind is often called a game day.

Example

An experiment written as a hypothesis:

Hypothesis: if one of three web nodes is stopped, error rate stays below 1%
            and an alert reaches on-call within 3 minutes.
Action:     stop web-2 for 10 minutes (staging first, small blast radius)
Abort if:   error rate above 5%
Result:     errors 0.4%, alert arrived in 2 min; health check removed node as intended

Common mistakes

  • Starting in production with no safeguards. Begin small and in staging.
  • Running experiments with no steady-state metric to compare.
  • Skipping the fix. Findings must lead to changes.
  • Surprising the team without agreed abort conditions.

Get notified when this term updates

Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.

Keep reading

Glossary · MonitoringAlert (Monitoring)A monitoring alert is an automatic notification sent when a check or metric crosses a defined condition, such as a site failing to respond.Glossary · MonitoringAlert DeduplicationAlert deduplication merges repeated alerts about the same underlying problem into one incident, so responders are notified once rather than hundreds of times.Glossary · MonitoringAvailabilityAvailability is the share of time a service is working and reachable when users need it, usually expressed as a percentage over a defined period.Guide · MonitoringAlert Fatigue and On-Call Basics for Small TeamsSet severity levels, escalation paths, alert thresholds and runbooks so a small team gets paged for real problems only and still sleeps through the night.8 min readGuide · MonitoringCron Job Heartbeat Monitoring to Catch Silent Failures EarlyLearn how heartbeat monitoring catches silent cron failures, with copy-ready examples for crontab, systemd, GitHub Actions, Docker, WP-Cron, Laravel and more.9 min readGuide · MonitoringA Blameless Incident Postmortem Template With a Worked ExampleWrite a blameless incident postmortem that drives real fixes: timeline, root cause vs contributing factors, 5 whys limits, a copy-paste template and an example.10 min read

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.