Monitoring and Alerting

Monitoring means checking known failure conditions — watching dashboards and thresholds you defined in advance. Observability means understanding system behavior, including failures you did not anticipate. Monitoring tells you something is wrong. Observability helps you understand why.

Both are necessary. The goal of monitoring is to surface symptoms fast enough that the on-call engineer is notified before users are meaningfully impacted. The goal of observability is to make root cause analysis tractable once you are investigating an incident.

Brendan Gregg's USE method provides a systematic approach for identifying performance bottlenecks. For every resource in your system, measure three things: Utilization (how busy is it?), Saturation (is it overloaded?), and Errors (is it failing?).

The USE method is most useful for infrastructure resources. For user-facing services, pair it with the four golden signals (latency, traffic, errors, saturation) to get a complete picture.

Alert fatigue is the failure mode of monitoring. When alerts fire too often, on-call engineers begin to ignore them — and when a real incident occurs, it gets missed. Good alerts are rare, actionable, and unambiguous.

On-call is not a punishment. It is a responsibility distributed across a team. Good on-call culture has three properties: sustainability (engineers are not burned out), effectiveness (incidents are resolved quickly), and learning (every incident improves the system).

A good dashboard answers specific questions at a glance. The most common mistake is building a dashboard that displays everything — which means it communicates nothing. Good dashboards have a clear audience and a clear purpose.