When Alert Fatigue Caused Real Incidents: Case Studies Across Ops, Security, and Healthcare

The Therac-25 (1985-1987): When Alarm Suppression Kills

The Therac-25 radiation therapy machine produced software errors that in some configurations delivered massive radiation overdoses. Crucially, the machine displayed an error message ("Malfunction 54") that operators had been taught to dismiss as a benign communication error and reset. The alarm was non-actionable by design in most contexts. Operators had been conditioned to ignore it.

When the same alarm started appearing in a genuinely dangerous context (race condition causing overdose), operators continued to dismiss it and restart the machine, as they had been trained. Six patients received fatal or near-fatal radiation overdoses between 1985 and 1987.

Target Corporation Data Breach (2013): Alerts Present, Action Absent

In November 2013, Target suffered one of the largest retail data breaches in history. Attackers gained access via a third-party HVAC vendor, installed malware on Target's point-of-sale terminals, and exfiltrated 40 million credit card records over several weeks.

A Bloomberg investigation (2014) revealed that Target's $1.6 million FireEye malware detection system had alerted the security team to suspicious activity. Bangalore-based analysts saw the alerts and escalated. The alerts were not acted upon by Target's Minneapolis security team. Target had automatic quarantine capabilities enabled during testing but had disabled the automatic response before the breach, preferring manual review. The manual queue was too noisy.

ICU Alarm Fatigue Sentinel Events: The Joint Commission Pattern

The Joint Commission Sentinel Event Alert 50 (2013) documents a general pattern from reported sentinel events (unexpected patient deaths or serious harm) related to alarm fatigue. The document does not attribute cases to individual patients or hospitals to protect privacy, but describes the pattern: a monitor alarm fires, the nurse either does not hear it (alarm fatigue, alarm silencing), cannot reach the patient in time, or dismisses it as another false positive. The patient deteriorates.

The Joint Commission received 98 sentinel events between January 2009 and June 2012 in which alarm fatigue was cited as a contributing factor. 80 resulted in patient death. The report notes this is likely a significant undercount as reporting is voluntary and sentinel events are often attributed to primary clinical causes without documenting the monitoring-system failure.

Major Outage Amplification: Alert Noise and Detection Lag

Public post-mortems from major technology companies consistently reference alert noise as a contributing factor to delayed detection and extended outage duration. While companies rarely state "we missed the alert because we had alert fatigue", the patterns are consistent:

Composite Success Pattern: 60-90% Noise Reduction in 6 Months

The following is a composite of recurring patterns from published vendor case studies (BigPanda, PagerDuty, incident.io, FireHydrant). Individual company names are not cited to avoid attributing specific numbers to specific companies without permission.

The critical insight: deduplication and correlation alone (steps 1-2) deliver the majority of the reduction with no changes to alerting rules. SLO migration delivers structural reduction in false positives. The quarterly audit sustains the improvement by preventing rule accumulation from reversing the gains.

The detection system worked. In every case, a real signal was present in the alert queue. The failure was not in detection -- it was in the operator's ability to prioritise a true positive among false positives.

Volume was the mechanism. High alert volume is the common cause. Whether ICU alarms, SIEM events, or monitoring pages, the quantity overwhelmed the human's ability to triage accurately.

The consequences scale with the context. In healthcare: patient death. In security: breach with financial and reputational impact. In DevOps: extended MTTR and amplified incident cost. The mechanism is identical; the stakes differ by domain.

Structural fixes outperform training. Every case where teams attempted to fix alert fatigue through operator training alone (JCAHO compliance training, SOC analyst refreshers, DevOps on-call preparation) showed limited improvement. Structural fixes (alarm customisation, deduplication, SLO-based alerting) showed sustained improvement.

No team self-diagnosed. In every documented case, the alert fatigue problem was identified retrospectively -- after the missed detection caused consequences. No team had a proactive alert-noise measurement programme that caught the problem before it caused harm.