How Anomaly Detection Works in Monitoring: Baselines, Seasonality, and Alert Windows
EngineeringNovember 9, 20269 min read

How Anomaly Detection Works in Monitoring: Baselines, Seasonality, and Alert Windows

Your z-score alert just paged you at 3am for a Tuesday traffic dip that happens every week. Here's why.

If you're running Datadog, Prometheus + Grafana, New Relic, or JustAnalytics, you've probably met anomaly detection. The promise of anomaly detection is compelling: let the system learn what's normal and alert you when something deviates. The reality? Badly configured anomaly detection generates more false positives than static thresholds while missing actual incidents.

What "Anomaly Detection" Actually Means

When monitoring tools say "anomaly detection," they're typically implementing one of three approaches:

Static thresholds aren't anomaly detection at all — they're just conditionals. Error rate > 5%? Alert. P99 latency > 500ms? Alert. Simple, but wrong whenever "normal" varies.

Z-score / standard deviation methods compute how far the current value is from the mean, measured in standard deviations. A value 2.5 standard deviations above the mean is flagged as anomalous. Works great for stationary metrics. Fails spectacularly for anything with patterns.

Decomposition-based methods separate a time series into trend, seasonality, and residual components. STL decomposition (Seasonal-Trend decomposition using LOESS) is the most common. Remove the predictable parts, then apply anomaly detection to the residuals. This is what production-grade systems use. In my opinion, it's the only approach worth your time if you're dealing with metrics that have any kind of pattern — which is basically all of them.

The key insight: real production metrics almost never meet the assumptions of basic z-scores. Your traffic has hourly patterns (people are asleep at 3am), daily patterns (weekends differ from weekdays), and monthly patterns (end-of-month billing runs). A z-score computed against a 24-hour rolling window will flag every Monday morning traffic spike as anomalous — because compared to the weekend, it is.

Rolling Baselines: The Foundation

Before you can detect anomalies, you need to define "normal."

Fixed baselines use a single historical period. The average error rate from last week becomes your reference point. Problem: last week might have been abnormal.

Rolling baselines continuously update based on recent data. Exponential moving average (EMA) with a decay factor is common. Adapts to gradual changes but still fails on seasonal patterns.

Seasonal baselines maintain separate "normal" profiles for different time windows. Tuesday at 3pm has a different baseline than Saturday at 3am. The math:

baseline[t] = mean(values at same hour-of-week over past N weeks)
deviation[t] = current_value[t] - baseline[t]
normalized_deviation[t] = deviation[t] / stddev[t]
alert if abs(normalized_deviation[t]) > threshold

The current value is compared not to a global average, but to the typical value for that specific time slot.

Training period matters. Most systems need 2-4 weeks of historical data. Teams that enable anomaly detection on day one often find everything flagged as anomalous — the system doesn't know what normal looks like yet.

Seasonal Decomposition: Handling the Real World

STL decomposition is the workhorse algorithm. Conceptually:

  1. Extract the trend: smooth the data with LOESS to get the long-term direction
  2. Extract seasonality: compute the average deviation from trend for each time slot
  3. Compute residuals: subtract trend and seasonality from the raw data
raw_data = trend + seasonality + residuals
anomaly_detection(residuals)  // not anomaly_detection(raw_data)

This is why Tuesday morning dips don't trigger alerts: the seasonality component captures "Tuesdays are lower than Mondays," so the residual is near zero. Only unexpected deviations — a Tuesday that's low even by Tuesday standards — show up as anomalies.

Multiple seasonalities compound. Real systems have hourly, daily, AND weekly patterns. Handling multiple overlapping seasonal components requires models like TBATS or Prophet that support multi-periodicity.

Holidays break everything. Your system learned that Monday at 9am means traffic spike. Christmas Monday at 9am? Not so much. Options: manually marking holiday periods, using holiday-aware models like Prophet, or accepting that holidays generate false positives you ignore.

Alert Windows: How Long Is Anomalous?

A metric spikes for 30 seconds then returns to normal. Worth alerting?

Short windows (30s-1m) catch incidents fast but generate noise from transient spikes. A single slow query can spike p99 latency for 45 seconds.

Medium windows (3-5m) are the sweet spot. Real incidents persist. Transient garbage-collection pauses and network blips usually resolve within 2 minutes.

Long windows (10-15m) minimize noise at the cost of delayed detection. If your SLO error budget burns 1% per minute of elevated error rate, waiting 15 minutes means you've already burned 15% of your monthly budget.

Cooldown periods prevent alert storms. Without cooldowns, a flapping metric pages you every time it crosses the threshold. Uptime checks have the same problem; see confirmation windows for false-positive downtime alerts.

For teams tracking SLOs, the alert window math should tie to error budget consumption. Burn-rate alerting is covered in our guide to SLO definition and rollout.

Why Naive Z-Scores Page You at 3am

Let's make this concrete. Monitoring request latency with a naive z-score:

rolling_mean = mean(latency over past 24 hours)
rolling_stddev = stddev(latency over past 24 hours)
z_score = (current_latency - rolling_mean) / rolling_stddev
alert if abs(z_score) > 2.5

Monday 3pm: Traffic peaks. Latency is 120ms. Rolling mean is 80ms. Z-score is (120-80)/15 = 2.67. Alert fires. Nothing's wrong — afternoon traffic is just higher.

Tuesday 3am: Traffic is minimal. Latency is 45ms. Rolling mean has drifted to 95ms. Z-score is (45-95)/15 = -3.33. Alert fires for low latency. Nothing's wrong — there's just no traffic.

Wednesday 10am: Actual incident. Database is slow, latency spikes to 400ms. But yesterday's spike is still in the rolling window. Z-score is 3.25. Alert fires — this one's real. But alert fatigue means you assume it's fake and ignore it for 20 minutes.

The fix: seasonal decomposition. Extract hourly and daily patterns, run z-scores on the residuals. Monday 3pm's residual is near zero (the spike is explained by seasonality). Wednesday 10am's residual is still huge. Now your alerts have signal.

What Anomaly Detection Won't Fix

It can't detect novel failures. If your database has never gone down, the anomaly detector has no baseline for "database down." It'll detect symptoms but won't tell you the cause. That's root cause analysis, a different problem.

It can't handle abrupt baseline shifts. You launch a feature that increases request volume 3x. For the next 2-4 weeks, every metric is "anomalous." Some systems have "baseline reset" features; most don't.

It can't distinguish good anomalies from bad. A viral tweet sends traffic up 500%. That's an anomaly. It's also great. Your anomaly detector doesn't know the difference. Honestly, this frustrates me more than anything else on this list — you'd think we'd have solved "good spike vs bad spike" by now, but nope.

It requires sufficient historical data. At least 2 weeks for daily seasonality, 4+ weeks for day-of-week patterns. New services are stuck with static thresholds until you've accumulated history.

Detection also depends on good instrumentation: a deploy that changes metric names or units quietly resets what "normal" means.

Configuring Anomaly Detection That Works

Step 1: Categorize your metrics. Some should never use anomaly detection — they have hard limits. Error rate above 10%, disk usage above 90%, certificate expiry under 7 days — static thresholds work.

Step 2: Identify seasonal patterns. Graph metrics over 4 weeks. Look for hourly patterns (lunch spikes), daily patterns (weekdays vs weekends), and weekly patterns (Monday mornings). Configure matching seasonality.

Step 3: Set persistence windows. 3-5 minutes for most metrics. 1-2 minutes for latency-sensitive paths. 10-15 minutes for background jobs.

Step 4: Tune during low-stakes hours. Enable in staging or on non-paging channels first. Watch for false positives over 1-2 weeks.

Step 5: Build feedback loops. Log whether each alert was actionable. If your actionable rate drops below 70%, you're generating noise. And look — most teams skip this step. I get it. Tracking alert outcomes feels like busywork when you're firefighting. But without it, you're tuning blind.

JustAnalytics' AI Command Center ($25/month add-on, requires Pro or Enterprise) includes anomaly detection, but the principles are the same whether you're using us, Datadog, or rolling your own on Prometheus.

Common Mistakes

Over-alerting on vanity metrics. Pageviews fluctuating 10% is probably not worth a page. Error rate fluctuating 10% is.

Using the same threshold for everything. Error rates and latency have different distributions. Tune per metric.

Ignoring training periods. Don't enable paging on a metric with less than 2 weeks of baseline data.

No cooldowns. A flapping metric will page you every 30 seconds. Set cooldowns of 15-30 minutes. Your on-call will thank you. Or at least stop plotting your demise.

Frequently Asked Questions

What's the difference between static thresholds and adaptive baselines in monitoring?

Static thresholds alert when a metric crosses a fixed number (e.g., error rate > 5%). Adaptive baselines learn what's normal for each time window and alert on deviations. Static thresholds cause false positives during predictable variations like weekend traffic dips. Adaptive baselines handle these automatically but require 2-4 weeks of historical data.

Why do naive z-score alerts fail for seasonal metrics?

Z-scores assume constant mean and variance. Real production metrics have daily patterns (3am traffic is lower than 3pm), weekly patterns (Monday spikes), and monthly patterns (end-of-month billing runs). A z-score computed against a 24-hour rolling average will flag every Monday morning spike as anomalous because it doesn't account for weekday seasonality.

How long should an anomaly persist before triggering an alert?

Most teams set 3-5 minute persistence windows for critical metrics. Shorter windows catch incidents faster but generate more noise. Longer windows reduce noise but delay detection. The right answer depends on your SLO error budget — if you're burning 0.1% of monthly budget per minute of downtime, even 5 minutes matters.

What metrics should use anomaly detection vs static thresholds?

Use static thresholds for metrics with known safe ranges: error rate above 10% is always bad, disk usage above 90% needs attention. Use anomaly detection for metrics where normal varies: request latency, traffic volume, conversion rate. Ask: does the acceptable value change based on time of day, day of week, or recent history?


Try JustAnalytics

All-in-one observability in one under-5KB script: cookieless analytics + error tracking + APM + session replay + uptime + structured logs. Replaces GA4 + Sentry + Datadog + Pingdom + LogRocket. Free tier (100K events/mo), Pro $49/month, or $39/month billed annually.

Start free → · AI Command Center MCP

JP
JustAnalytics Platform TeamContributor

Author at JustAnalytics.

Related posts