Alert Fatigue and On-Call Basics for Small Teams

Alert fatigue is what happens when your monitoring cries wolf often enough that people stop believing it. The phone buzzes, someone glances at it, assumes it is another false alarm, and the one real outage sits unnoticed for an hour. In a small team there is no night shift and no dedicated operations staff, so this problem arrives early and does real damage. This guide covers how to decide what deserves a page, how to route alerts, how to set thresholds, and how to write runbooks so whoever is on call at 3 a.m. knows what to do.

What alert fatigue looks like

You have it if any of these are true:

• Alerts are muted, filtered into a folder or "handled later."
• People say "it usually fixes itself" about a recurring alert.
• The same alert fires several times a week and nobody changes anything.
• Responders cannot tell from the alert text what is wrong or what to do.
• One person receives everything, and is close to quitting the rotation.

The cause is nearly always the same: alerts that are not actionable. An alert that does not need a human right now should not wake a human right now.

The one rule: every page must be actionable and urgent

Before an alert is allowed to page someone, check it against three questions:

1. Does it indicate real user impact, or imminent impact?
2. Does a human need to act on it now, rather than in working hours?
3. Is there something they can do about it?

If any answer is no, downgrade it. A disk at 70% is a ticket. A disk at 95% and filling fast is a page. A single failed check is neither; it is a signal to retry (see how to reduce false alerts).

Define severity levels

Keep the scheme small so people remember it. Four levels are plenty, three are fine for a very small team.

The important decision is where the line sits between "wakes someone up" and "wait until morning." Write it down and agree on it with whoever owns the business, because it is partly a business decision about the cost of downtime. For a view of what an hour of downtime might cost you, see ecommerce downtime cost.

Use thresholds that reflect reality

Require confirmation

Alert on sustained failure, not a single blip. A typical pattern is to require two or more consecutive failed checks, or checks from more than one location, before an alert fires. The trade-off is detection time: confirmation adds delay. The guide on how often to monitor a website shows how to calculate it.

Alert on symptoms, not causes

Page on what users feel: the site does not respond, pages return 5xx errors, checkout fails, response time is far above normal. Do not page on CPU at 85%, memory usage, or a single process restart unless you know it reliably predicts impact. Cause-based metrics are useful on a dashboard and in a ticket queue, and they are what you look at during diagnosis.

Use duration and rate, not just a level

"Response time above 3 seconds" will fire constantly during a normal spike. "Median response time above 3 seconds for 10 minutes" fires when something is really wrong. Add a duration to every threshold, and review them after incidents.

Keep warning and critical thresholds separate

Send warnings (disk 80%, certificate 21 days left) to chat or email and the critical level (disk 95%, certificate 3 days left) to the pager. Both exist, but only one interrupts anyone.

Beware flapping

A service that goes up and down every few minutes can generate dozens of alerts. Configure recovery notifications to require a few consecutive passes, and group repeated alerts into a single incident rather than sending a new message each time.

Escalation: who gets it and what if they do not answer

Even a two-person team needs a written escalation path. A simple one:

1. Primary on-call is notified immediately.
2. If they do not acknowledge within a set time (for example 10 to 15 minutes), secondary on-call is notified.
3. If still no acknowledgment, the owner or manager is notified.

Make acknowledgment explicit: someone replies in the channel or taps acknowledge so everybody knows the alert is being handled and stops duplicating effort.

Use more than one channel for pages. A Slack message at 3 a.m. will not wake anyone with notifications silenced. Phone push, SMS or a call is for critical pages. SiteDownCheck sends alerts by email, webhook, Slack and Discord; a webhook can forward critical alerts into whichever paging or phone service you use. See the uptime monitor.

Make on-call sustainable for a small team

• Rotate. Weekly rotations are common. Nobody should be permanently on call.
• Protect rest. If someone was woken during the night, let them start late the next day. This costs less than the mistakes that tired people make.
• Keep the load measurable. Count pages per week per person. If a person gets more than a handful of out-of-hours pages in a week, fix the alert sources before adding anything else.
• Share the pager with the right people. Developers who built the system can fix root causes, and being paged for your own code is a powerful incentive to make it reliable.
• Consider business-hours-only coverage. For a brochure site or small business, "we respond within one business hour during working hours, and check alerts at 8 a.m." can be a legitimate, honest policy. Say so openly, and choose a monitoring setup that reflects it.
• Agree on compensation or time off in lieu if on-call is a formal duty. That is a conversation, not a tooling problem.

Write runbooks people can follow half asleep

A runbook is a short document attached to an alert that says what it means and what to do. Link it from the alert text. A good one fits on one screen:

RUNBOOK: Website not responding (SEV1)

Meaning:  The main site failed 2 consecutive checks from 2 locations.

1. Confirm: open the site yourself; run the website health check.
   Is it down for everyone, or just you?
2. Check the host's status page and your own server status.
3. Check recent changes: last deploy, plugin updates, config edits.
   If a deploy happened in the last hour, roll it back first.
4. Check disk space, memory, and that web server and database are running:
   df -h ; free -m ; systemctl status nginx php-fpm mysql
5. Check logs for the first error:
   tail -n 100 /var/log/nginx/error.log
6. If not solved in 15 minutes, escalate to: <name/number>.
7. Post an update on the status page now, then every 30 minutes.

After: write a short postmortem within 3 working days.

The specifics will vary by stack, but the structure holds: meaning, first checks, common causes, rollback, escalation, communication. For common causes see why is my website down and the guides for 502, 503 और 504 errors.

Always include how to communicate. Posting a quick acknowledgment on your status page reduces support load while you work. Templates are in status page communication templates.

Review and prune regularly

Alerting is not a set-and-forget job. Once a month, or after any bad week:

1. List every page sent. For each one, ask: was it real? Did the responder need to act?
2. For false or non-actionable alerts, delete them, add confirmation, change the threshold, or downgrade to a ticket.
3. For missed incidents (customers noticed before you did), add a check.
4. For repeated real incidents, fix the root cause. A postmortem turns a repeating page into a project.

Scheduled jobs deserve their own attention, since a job that silently stops produces no error to alert on. Use heartbeat monitoring so the absence of a signal becomes the alert.

अगले कदम

• Set confirmation and channels in your uptime monitor.
• Watch scheduled jobs with cron monitoring.
• Make sure the first responder can run a quick website health check.
• Write up each outage with the incident postmortem generator.
• Read how to reduce false alerts in monitoring.

सामान्य प्रश्न

How many alerts per week is too many?

There is no universal number, but if on-call people regularly get woken for things that need no action, you have too many. Track pages per person and fix the noisiest source first.

Should every alert go to everyone?

No. Send pages to the current on-call person and route lower-severity messages to a shared channel or ticket queue. Broadcasting everything teaches people to ignore it.

Can a one-person business do on-call?

Yes, in a lightweight form. Page yourself only for real outages, send everything else to a morning digest, and publish your response hours so customers know what to expect.

What is the difference between an alert and an incident?

An alert is a notification from a monitor. An incident is the real-world problem, which may trigger many alerts. Group alerts into incidents so you manage one problem, not fifty messages.

यह पेज सामान्य जानकारी है, जिसे हमारी संपादकीय टीम ने लिखा है और दस्तावेज़ीकरण बदलने पर अपडेट किया जाता है। सर्वर और प्रदाता अलग-अलग होते हैं, इसलिए बदलाव पहले सुरक्षित तरीके से जाँचें। देखें: संपादकीय नीति और अस्वीकरण.