Stop Getting Paged at 3am for Nothing: Killing False-Positive Downtime Alerts

The page hits at 3:17am:

CRITICAL: api.yourproduct.com is DOWN (0% uptime from US-East)

You grab the laptop, adrenaline already spiking. Dashboard open. A curl from your machine returns 200 OK. The status page is green. Nothing is wrong.

The site was never down. But you're awake now, and the night is gone.

The worst part is that you can't prove it was a false positive. Maybe it really was down for those 30 seconds? So you close the ticket, go back to bed, and wait for next time. This post is the config pattern that stops that loop: multi-region quorum, retries, and confirmation windows, so a single flaky check path can't wake anyone while real outages still get through within a couple of minutes.

What You'll Have By The End

An uptime alerting setup that actually works:

Multi-region quorum, so several locations must fail before anything alerts
Retry-before-alert to filter transient blips
Confirmation windows so brief hiccups don't page you
Severity tiers, so the marketing site never wakes the on-call engineer
Real outages still detected within a few minutes

You won't miss real downtime. You will stop getting paged for phantom failures. Probably. Your CDN can still do something weird, but this gets you most of the way there.

Prerequisites

An uptime monitoring tool that supports multi-region checks (Pingdom, Better Stack, Datadog Synthetics, or similar)
Access to configure alert conditions and notification rules
Twenty minutes or so, mostly spent second-guessing your thresholds

One caveat before the config blocks: the YAML below is illustrative. Field names differ between tools, and your product's UI may expose the same settings as form fields rather than files. Treat the numbers as starting points, not product defaults.

Step 1: Enable Multi-Region Checks

Most uptime tools can check from a single region, and many setups never move past that. It's a common root cause of false positives.

Here's why: the internet is messy. A packet gets dropped between your monitoring provider's Virginia data center and your AWS load balancer. Your site is fine. That one path through the internet hiccuped. But your monitoring tool doesn't know that. It sees a timeout and fires an alert.

The fix: Check from at least three geographically distributed regions, and require failure from more than one before triggering. Simple in theory. Annoyingly fiddly in practice.

monitors:
  - name: "API Health"
    url: "https://api.yourproduct.com/health"
    check_interval: 30s
    regions:
      - us-east
      - eu-west
      - ap-northeast
    alert_condition:
      type: quorum
      min_failures: 2  # 2 of 3 regions must fail
      within: 60s

The min_failures: 2 line is the key. A single region timeout won't page you. Two regions timing out at once? That's probably real. If you're coming from GA4 and want a unified view, the same consolidation applies here: uptime status sits next to your analytics instead of in a separate tool.

What you should see: Check results from multiple regions. Individual region failures show as warnings, not critical alerts.

Every vendor names this differently ("multi-location", "multi-probe", "consensus"). Same idea, different label. Look for the setting that says how many locations must agree.

Step 2: Add Retry-Before-Alert

Even with quorum, a momentary blip can hit several regions at once. A brief cloud-provider hiccup, a CDN edge problem affecting a few PoPs, or a long garbage collection pause that makes your health endpoint briefly unresponsive.

The fix: Retry failed checks before alerting.

monitors:
  - name: "API Health"
    url: "https://api.yourproduct.com/health"
    check_interval: 30s
    regions:
      - us-east
      - eu-west
      - ap-northeast
    retry:
      count: 2           # retry twice before considering it a failure
      delay: 10s         # wait 10s between retries
      backoff: linear    # 10s, then 20s
    alert_condition:
      type: quorum
      min_failures: 2
      within: 90s        # expanded window to account for retries

The flow: first check fails, wait 10 seconds, retry, still failing, wait 20 seconds, retry again, still failing. Only then is the region marked down and counted toward quorum.

Why this matters: a GC pause that timed out the first check will almost certainly succeed on retry. Another class of phantom outage gone.

What you should see: Retry attempts in your uptime logs. Checks that recover on retry don't show up as incidents.

Step 3: Configure Confirmation Windows

The final layer. Even after quorum and retries, you might get an alert for a 45-second blip that resolved itself. Do you really need to be paged for that at 3am?

Probably not. But you do need to know if it lasts two minutes or more.

The fix: Add a confirmation window.

alert_rules:
  - name: "API Critical"
    trigger:
      monitor: "API Health"
      status: down
      duration: 60s    # must be down for 60 continuous seconds
    notify:
      channels: [pagerduty-oncall]
      message: "API health check failing from multiple regions for 60+ seconds"

The duration: 60s is the confirmation window. The alert won't fire until the service has been marked down for a full minute. Brief blips come and go silently, or at most leave a warning you can read in the morning.

The tradeoff: you're adding 60 seconds to detection time. For most services that's fine. A sub-minute blip at 3am that resolved itself usually isn't worth waking someone for. If you run a trading platform or anything life-critical, use a shorter window and accept more noise.

Pick the window per service: shorter for critical paths, longer for low-priority ones. Tie it to your SLOs if you have real ones. If your SLOs are aspirational, pick numbers that feel right and revisit them after a month of data.

Step 4: Implement Severity Tiers

Not every endpoint deserves a 3am wake-up. Your health check endpoint? Critical. Your blog's RSS feed? Can wait until morning.

The fix: Create severity-based alert rules.

alert_rules:
  - name: "Critical Path - Immediate Page"
    trigger:
      monitors: ["API Health", "Payment Endpoint", "Auth Service"]
      status: down
      duration: 60s
    notify:
      channels: [pagerduty-oncall]
      urgency: high

  - name: "Supporting Services - Morning Review"
    trigger:
      monitors: ["Blog CDN", "Documentation", "Marketing Site"]
      status: down
      duration: 300s  # 5 minutes before alerting
    notify:
      channels: [slack-sre]
      urgency: low
      quiet_hours: true  # suppress between 10pm-8am local

Quiet hours are underrated. Criminally underrated. A marketing site can be down for eight hours overnight without anyone caring, so don't page for it. Some people believe everything needs 24/7 alerting. They're wrong.

Step 5: Monitor Your Monitors

Teams forget this one: your monitoring system can generate false positives itself. A flaky monitoring provider, an expired API key, or rate limiting from too-frequent checks all look like downtime.

The fix: Add meta-monitoring.

meta_monitors:
  - name: "Monitor Health"
    type: internal
    check: check_execution_rate
    alert_if: execution_rate < 95%  # checks not running = bad data
    notify: [slack-sre]

And if you run your own check infrastructure (cron jobs doing health checks, say), make sure those runners are monitored too. Cron and missed-run alerting is exactly the kind of thing an uptime tool should cover for you.

Common Errors and Fixes

Symptom: alert fired but status shows UP

Alert: api.yourproduct.com DOWN
Current status: UP
Last down: 2 minutes ago

Cause: Your confirmation window is shorter than your check interval. The site recovered between checks, but the alert had already fired.

Fix: Set duration to at least 2x your check_interval. If you check every 30 seconds, confirm over at least 60.

Symptom: no checks running for one region

Warning: No check results from ap-northeast in last 5 minutes

Cause: Network issues between the monitoring provider and your region, or degraded regional infrastructure on the provider's side.

Fix: Add a fourth region as backup. If one region goes silent, quorum still works with the remaining three. Check the provider's status page too. They might be having their own outage.

Symptom: retry timeout exceeded

Check failed: retry timeout exceeded (30s)

Cause: Retry delay plus backoff exceeds the timeout for the retry window, so the check gives up before the retries complete.

Fix: Reduce retry delay or increase the timeout. For a 30s interval with two retries at 10s/20s, you need at least a 60s window for retries to finish.

Symptom: quorum never reached

Region us-east: DOWN
Region eu-west: UP
Region ap-northeast: UP
Alert status: none (quorum not met)

This is working as intended. One region failing doesn't meet quorum, so nothing fires. If it keeps happening to the same region, investigate that specific network path. Just don't page for it.

What to Expect

Your numbers will differ, so measure your own baseline first. Count last month's pages that turned out to be nothing, apply the five steps, and count again.

Expect two things. False-positive pages should drop sharply, because single-region timeouts and sub-minute blips are the most common culprits. And detection of real incidents gets slower by roughly your retry time plus your confirmation window, a couple of minutes in the configs above. That's a trade worth making. A pager you trust gets answered faster than one you've learned to ignore.

One class of false positive will still slip through: planned changes, such as DNS migrations or deploys. Put those on a maintenance window so checks pause while you work.

Look, this isn't rocket science. It's just surprisingly easy to get wrong, and single-region defaults make it the path of least resistance.

Next Steps

Now that your uptime alerts are trustworthy, you probably want to:

Build a status page from your uptime data: see building a status page from uptime and incident data
Watch errors, not just uptime: a site can be up and still throwing errors; correlating errors with funnel drop-off shows which ones cost you conversions
Detect third-party outages: sometimes the problem isn't you, it's Stripe or AWS going down

The goal isn't zero alerts. It's zero wasted alerts. Every page should mean "something is actually wrong, and you need to do something about it."

That's when on-call stops being punishment and starts being... well, still not fun. But at least trustworthy. And you might sleep through the night.

Frequently Asked Questions

What causes false positive uptime alerts?

Mostly single-region checks hitting transient network problems: CDN edge node failures, packet loss between your monitoring provider and your server, a router somewhere having a bad day. A check from Virginia times out while London, Tokyo, and Sydney all see your site as healthy. Without multi-region consensus, that single failure triggers an alert for downtime that never existed.

What is multi-region quorum for uptime monitoring?

Multi-region quorum requires failures from multiple geographic locations before triggering an alert. Instead of one failed check = one page, you configure rules like "alert only if 2 of 3 regions fail within 60 seconds." This filters out localized network blips while still catching genuine global outages.

How does retry-before-alert work?

Retry-before-alert adds a second (or third) check before firing a notification. If the first check fails, the system retries, usually after a 10-30 second delay. Only if the retry also fails does the alert fire. This catches momentary blips from garbage collection pauses, brief network hiccups, or transient load spikes that resolve themselves.

What is an alert confirmation window?

A confirmation window waits a set period (30-120 seconds) after detecting failure before alerting. If the service recovers within the window, no alert fires. It prevents pages for brief blips while still catching sustained outages, and combined with retry-before-alert it cuts false positives without much delay in detecting real incidents.

Try JustAnalytics

All-in-one observability in one under-5KB script: cookieless analytics + error tracking + APM + session replay + uptime + structured logs. Replaces GA4 + Sentry + Datadog + Pingdom + LogRocket. Free tier (100K events/mo), Pro $49/month ($39 annual).

Start free → · AI Command Center MCP