Monitoring & Alerting

Designing a system is one thing; knowing at 3am whether it is healthy is another. Monitoring is the difference between learning about an outage from your own alerts and learning about it from a customer.

The observability pipeline from services to dashboards and pages100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

What monitoring buys

Monitoring is the collection, aggregation and display of quantitative data about a system's health over time. It detects failures as they happen, supplies the data to diagnose them, tracks resource use so capacity is planned early, and proves an SLA was met with evidence rather than recollection.

The four golden signals

Google's SRE practice narrows instrumentation to four measurements per service. Get these right per endpoint and most dashboards become optional.

1. Latency: time to serve a request. Percentiles, not averages: 99 requests at 10 ms and one at 900 ms average to 19 ms, a number no user experienced, and p99 is what a user feels. Count failed requests separately — a fast 500 flatters the graph.
2. Traffic: demand arriving, as requests per second or network I/O. It is the denominator of every rate you quote later.
3. Errors: the rate of requests that fail — 5xx, caught exceptions, connection failures. Rate, not count: 500 errors means nothing until you know whether it came out of 1,000 requests or 10 million.
4. Saturation: how full the service is — CPU, memory, and above all queue depth. It is the early warning for errors: by Little's Law a dependency slowing from 50 ms to 2 s multiplies in-flight requests fortyfold, so pools and queues fill the moment latency moves, minutes before that pressure surfaces as timeouts, shed load and 5xx. That head start is what lets a circuit breaker trip before the pool empties.

Around them sit infrastructure metrics for the hosts underneath, application metrics such as cache hit rate, structured logs where debugging happens, distributed tracing — the only practical way to find a slow hop in a microservices call chain — and the health check a load balancer uses to route traffic.

Metrics, logs and traces cost different amounts

The three stores are not interchangeable, and the difference is cardinality.

A metric with labels is one time series per unique label combination, held resident in the ingester. http_requests_total labelled by 5 endpoints × 3 status classes × 100 pods is 1,500 series, a rounding error. A new label multiplies that count rather than adding to it, because the pod is part of a series' identity: assume roughly 50,000 customers and customer_id turns 1,500 into 50,000 × 5 × 3 × 100 = 75,000,000, so memory runs out long before disk. Any label whose value count grows with users or requests belongs in logs and traces, joined by a trace id, not on a metric.

Traces are priced by volume instead. At 1 billion requests/day ≈ 12,000 QPS, 5 services per request and roughly 1 KB per span, full capture is 12,000 × 5 × 1 KB = 60 MB/s, about 5 TB a day. Head sampling decides at the root and propagates that decision, so 1% costs 50 GB/day and keeps ~120 traces a second — enough to characterise the normal path, and nothing for the one broken request a customer is calling about, because the other 99% were never recorded. Tail sampling keeps every error and everything past p99, but only by transporting and buffering all 5 TB. They are alternatives, not layers: you buy either the cheap ingest or the answered ticket.

Alerting

Alerting notifies a responsible human when monitoring detects a problematic condition, cutting Mean Time To Detect and Respond. An alert is worth writing only if it names a problem a human can act on now, with a runbook; alerts that fire without action train the on-call to ignore the pager. A page interrupts sleep, a ticket waits for morning.

Alert on burn rate, not on thresholds

Take a 99.9% monthly success SLO on 1M requests/day. That is 30M requests a month, so the error budget is 0.1% — 30,000 failed requests, the same 43 minutes of full outage that three nines allows.

Burn rate is the multiple of budget-neutral spend:

burn rate = observed error rate ÷ (1 − SLO)

  2% errors  ->  0.02 ÷ 0.001 =    20x  ->  budget gone in 720 h ÷ 20   = 36 hours
100% errors  ->  1.00 ÷ 0.001 = 1,000x  ->  budget gone in 720 h ÷ 1000 = 43 minutes

Two windows, two responses. Page when a fast window says the budget dies in about two days: 14.4× for an hour spends 14.4 × 1/720 = 2% of the month and zeroes it in 720 ÷ 14.4 ≈ 50 hours. Open a ticket when a slow window says it dies this week: 6× for six hours spends 5%, and 720 ÷ 6 = 120 hours is five days.

The rejected alternative is the static threshold, error rate > 2% for 5 minutes. The for clause exists to swallow blips, so the shortest event it fires on is five minutes at 2% — at 12 QPS, 3,600 requests and 72 errors, 0.24% of the budget. It pages for that exactly as it pages for the sustained 20× burn that ends the month's compliance in 36 hours. One deserves a human out of bed; the other deserves a graph.

The failure mode nobody plans for

The pipeline that tells you the system is down runs on the system: agents ship over the same network into a store in the same region, so a zone failure takes the evidence out with the service and the dashboards go flat rather than red. Flat is worse: "no data" and "no errors" look identical to a threshold rule, so the alert that should fire is the one that cannot.

Two mechanisms close the gap. Write rules so missing data fails rather than passes — absent(up{job="api"}) in Prometheus terms. Then run a dead man's switch: a rule that always fires, heartbeating to an external service that pages you when the heartbeat stops. Its silence is the alarm, and it catches a monitoring stack that died quietly at 2am.

Common tools

Metrics: Prometheus, Datadog, CloudWatch. Logs: ELK, Splunk, Loki. Traces: Jaeger, Zipkin, Tempo. Paging: Alertmanager, PagerDuty, Opsgenie. Dashboards: Grafana, Kibana. The choice matters far less than what you point them at.

In an interview

What is tested is whether operations are part of your design rather than something an SRE team bolts on later: alerts tied to user pain, and knowing what the instrumentation costs to run.

Phrasing that lands: "I would instrument the four golden signals per endpoint and set one SLO per user-facing journey, say 99.9% of reads under 200 ms. Alerts fire on burn rate against that SLO on two windows: a page at 14.4× over an hour, a ticket at 6× over six hours, not on raw CPU. Logs carry the trace id, so a page leads to a trace and the trace leads to the slow span." Then price it: "Full tracing at 12,000 QPS is roughly 5 TB a day, so 1% head sampling if cost is the constraint, or tail sampling if the support queue is — not both, since head sampling has already thrown away what tail sampling exists to keep."

The mistake that loses points is naming tools instead of alert conditions. "Prometheus, Grafana, ELK, Jaeger" answers nothing; the follow-up is always "what pages someone at 3am, and what is that worth?" A candidate with no threshold, no window and no named recipient has described a dashboard nobody watches. Close second is alerting on causes rather than symptoms: CPU at 90% with latency and errors flat is a machine doing its job.

Check yourself

1. You run 1M requests/day against a 99.9% monthly success SLO. A bad deploy pushes the error rate to 5% and you catch it 20 minutes later. How much budget is gone, and should it have paged?

The month is 30M requests, so the budget is 30,000 errors. 1M/day ≈ 12 QPS, so 20 minutes is 12 × 1,200 s ≈ 14,400 requests and 5% of those is about 720 errors — roughly 2.4% of the budget. Small in absolute terms, but the burn rate is 0.05 ÷ 0.001 = 50×, which empties the month in 720 h ÷ 50 ≈ 14 hours. It pages on the rate, not on the 720 errors, which alone look survivable.

2. Support asks which customers saw errors last night. Do you add a customer_id label to the error counter?

No. Assume roughly 50,000 customers: the label multiplies the existing series rather than adding to them, so 50,000 × 5 endpoints × 3 status classes × 100 pods = 75,000,000 active series sit in the ingester's memory and the store falls over before it answers. Put the customer id on log lines and trace spans instead, which are stored per event, and join by trace id. The test: if a label's value count grows with users or requests, it is not a metric label.

3. Your rule needs the condition to hold 5 minutes before firing, and the on-call takes 10 minutes to acknowledge. The SLO is 99.9% monthly. What does that arithmetic say about the window?

A full outage burns at 1,000×, spending the whole 43-minute monthly budget in 43 minutes. Confirmation plus acknowledgement is 15 of those 43 before anyone opens the runbook, leaving 28 minutes to diagnose and fix. Either shorten the fast window — 30 seconds at a very high burn rate, where a false page is cheap — or accept that three nines means "one incident a month, fixed inside half an hour" and staff for it. That is why burn-rate alerting uses two windows rather than one.