One Endpoint Is Killing Your P99: How to Find the Latency Source of Slow API Calls
When p99 spikes and the median doesn't move, one endpoint is usually to blame. Here's how to find the p99 latency source, from browser to database.
Picture a PagerDuty alert at 2:47pm on a Thursday: API latency p99 > 4000ms for 5 minutes. A typical p99 hovers around 800ms. Something has jumped it 5x.
Open your APM tool. The service map shows elevated latency on the main API gateway. Cool. But which endpoint? The APM view lists services sorted by request count. The slow requests are buried somewhere in 47 different endpoints, and the dashboard is averaging across all of them. Median looks fine. P50 looks fine. Even p95 is only slightly elevated.
It's just the p99 that has gone haywire. Which means a small percentage of requests are catastrophically slow, and you have to find the p99 latency source.
Twenty minutes of clicking around narrows it to the /api/users/search endpoint. Another ten minutes finds a specific slow trace. Another fifteen reveals the trace doesn't include the database query timing, because the instrumentation is incomplete: nobody verified that the Prisma spans actually include PostgreSQL execution time. They don't.
By the time the root cause surfaces — a missing index on a JSONB column that only affects searches with special characters — the p99 has already recovered on its own. The incident lasts 45 minutes. Debugging takes 45 minutes. The actual fix (adding an index) takes 3 minutes.
That ratio is backwards.
Why p99 Issues Are Harder to Debug Than p50 Issues
P99 latency problems are sneaky. They only affect 1% of requests, so your dashboards look mostly healthy. Median? Fine. Average? Fine. Most users? Happy. Meanwhile, your biggest customers are quietly seething.
But that 1% is often your worst 1% — the power users with large datasets, the enterprise customers with complex queries, the edge cases your test suite never exercised. And 1% of a million daily requests is still 10,000 slow requests per day.
The debugging challenge is that siloed tools obscure the connection between frontend slowness and backend causes. Your Real User Monitoring (RUM) shows elevated load times. Your APM shows elevated p99. Your database metrics show... nothing obvious, because the slow queries are hidden in aggregate statistics.
To actually find the root cause, you need to:
- See which specific user sessions experienced slowness
- Trace those sessions to specific API calls
- Follow those API calls through your service mesh
- Identify which downstream operation — database, cache, external API — was slow
- Get the actual query or request that caused it
With separate tools (GA4 for analytics, Sentry for errors, Datadog for APM, something else for database monitoring), you're stitching this together manually. CSV exports, timestamp matching, educated guessing.
With unified observability, it's one click path from a user's slow page load to the database query that took 3.2 seconds.
How to Find the P99 Latency Source: An Illustrative Walkthrough
Here's how the same issue could play out with JustAnalytics. The service names, timings, query and screens below are illustrative, not a real customer incident.
2:47pm — Alert fires. The p99 latency alert triggers and lands in your team channel.
2:48pm — Open the Service Map. The service map shows all instrumented services with real-time latency coloring. One service — api-gateway — is yellow instead of green. Something's elevated. Click it.
2:49pm — Identify the endpoint. The endpoint breakdown shows p99 latency by route. Most routes are under 1s. One route — POST /api/users/search — shows p99 of 4,200ms. That's the culprit.
2:50pm — Filter to slow traces. Click into /api/users/search and filter to traces where duration > 3000ms. Fourteen traces in the last 10 minutes. Click the slowest one: 4,847ms.
2:51pm — Follow the span waterfall. The trace shows the full journey:
[Browser] Navigation: /dashboard/search → 4,847ms total
├─ [Next.js SSR] getServerSideProps → 4,623ms
│ └─ [API Client] POST /api/users/search → 4,589ms
│ ├─ [Redis] GET user_search_cache:query_hash → 2ms (cache miss)
│ ├─ [Prisma] prisma.user.findMany → 4,412ms ← HERE
│ └─ [Redis] SET user_search_cache:query_hash → 3ms
└─ [Browser] Hydration → 224ms
The Prisma query took 4.4 seconds. Everything else was fast. That's the root cause.
2:52pm — Get the actual query. Click into the Prisma span to see the statement it ran, ideally with parameters (capturing those is a privacy decision, so check what your setup records):
SELECT * FROM users
WHERE search_metadata @> '{"tags": ["müller"]}'
ORDER BY created_at DESC
LIMIT 50
The search term contains an umlaut. The JSONB @> operator was doing a sequential scan because the GIN index didn't cover this specific access pattern.
2:54pm — Verify the hypothesis. Check the other 13 slow traces. Nine of them have search queries with special characters (umlauts, accents, CJK characters). The pattern is consistent.
2:55pm — Fix deployed. Write a migration adding a proper GIN index with the jsonb_path_ops class, deploy it, and the p99 drops back to normal within minutes.
Total debugging time: 8 minutes, in this walkthrough. The 45-minute incident described at the start is the same class of problem with different tooling.
The Three-Layer Correlation That Makes This Possible
What makes that flow work is what you might call three-layer correlation:
Layer 1: Browser → API. Every request from the browser carries a trace ID. When that request hits your API, the trace ID propagates via headers. This means the frontend "slow page load" and the backend "slow API call" are already linked. You don't need to match timestamps or guess. The Next.js setup guide covers the browser side.
Layer 2: API → Services. If your backend calls other internal services (microservices, workers, whatever), the trace ID propagates through each hop. JustAnalytics is OpenTelemetry-native, so if you're already using OTel instrumentation, the spans carry over.
Layer 3: Services → Data Stores. Database queries, Redis operations, external API calls — each gets captured as a child span with timing data. This is where most p99 issues live. The application code is usually fast. The I/O is usually slow.
The alternative — running separate tools for each layer — breaks these correlations. You can reconstruct them manually, but you're spending 30 minutes on data archaeology instead of 3 minutes on debugging.
What Usually Causes p99 Spikes (While Median Stays Stable)
Here are the most common root causes, in rough order of how often they bite:
Missing indexes on edge-case queries. This was the JSONB example above. The query plan works fine for 99% of inputs. The 1% that triggers a sequential scan destroys your p99.
N+1 patterns that only trigger on large data. You load a user, then load their 3 posts, then load 5 comments per post. Fast when users have 3 posts. Catastrophic when a power user has 500 posts.
Cache misses under specific conditions. Your cache TTL is 1 hour. At minute 61, whoever hits the endpoint first takes the full database query time. If that happens to coincide with a complex query, boom — p99 spike.
Connection pool exhaustion. Your pool has 10 connections. Under normal load, fine. During a traffic burst, requests queue waiting for a free connection. The requests themselves are fast once they execute, but the queue wait shows up in total latency.
External API timeouts. You call Stripe or Twilio or whatever. Usually returns in 200ms. Occasionally takes 5 seconds. Your p99 reflects their p99.
Garbage collection pauses. More common in JVM-based services, but Node.js can hit this too with large heaps. A GC pause freezes everything for 200-500ms, spiking latency for any requests that happened to be in flight.
The common thread: these are all problems that affect a minority of requests. Your tests don't catch them because your test data is clean and small. Your monitoring averages hide them. You only see them when you look at percentiles and when you have full traces.
Setting Up Alerting That Catches p99 Before Users Notice
The goal isn't just debugging fast — it's catching issues before they impact enough users to complain.
Here's a sensible starting shape for the rules (illustrative pseudo-config, not an exact JustAnalytics file format):
# Illustrative alert rules
alerts:
- name: "API p99 Latency Spike"
metric: "api.latency.p99"
condition: "value > baseline * 2 for 3 minutes"
baseline_window: "7d"
filters:
- "service:api-gateway"
severity: warning
channels: ["slack-engineering"]
- name: "API p99 Latency Critical"
metric: "api.latency.p99"
condition: "value > 5000ms for 2 minutes"
filters:
- "service:api-gateway"
severity: critical
channels: ["pagerduty-oncall"]
Two alerts: one relative (2x baseline), one absolute (5 second threshold). The relative alert catches gradual degradation — if your p99 slowly creeps from 800ms to 1600ms over a week, the 2x baseline alert fires even though 1600ms might not feel "critical." The absolute alert is your "something is very broken" backstop.
Use a 7-day baseline window, because weekend traffic patterns differ from weekday. A 24-hour window creates false positives every Monday morning.
The "3 minutes" duration filter is important — it ignores single-point spikes from one-off slow requests. A genuine problem sustains for multiple minutes. A fluke doesn't.
The "Works on My Machine" Problem: Why Local Profiling Misses p99 Issues
A key point: you can't reproduce p99 issues locally.
Local development uses test data — clean, small, predictable. Production has 4 years of accumulated user data including edge cases nobody designed for. That JSONB query that takes 4 seconds in production takes 3ms against your local seed data.
Local development has warm caches. Production has cold cache scenarios you can't easily replicate.
Local development doesn't have concurrent load. The connection pool exhaustion that happens at 500 RPS never triggers when you're the only user.
The only way to debug p99 issues is with production observability. You need traces from real requests, from real users, with real data. Distributed tracing isn't optional anymore. I'll die on this hill: if you're running a production API without traces, you're flying blind and it's only a matter of time before a p99 issue costs you more than the instrumentation would have.
Some teams try to replicate production data locally for debugging. That's a GDPR nightmare waiting to happen. (For EU compliance considerations around analytics and session data, the Schrems III post covers where the law currently stands. TL;DR: don't copy production PII to developer laptops.)
Why Siloed Tools Make This Harder
I want to be specific about what "siloed tools" means, because the term gets thrown around vaguely.
Sentry + Datadog + Separate Database Monitoring:
When a user reports "the search page was slow," you check Sentry for errors (none), Datadog for traces (elevated latency on search endpoint), and your database dashboard for slow queries (nothing obvious in aggregates).
By default, the Sentry error and the Datadog trace don't share identifiers. You can't click from one to the other. You're matching by timestamp and hoping.
The Datadog trace might not include database query details if you didn't instrument that layer. Or the instrumentation exists but strips query parameters for privacy, so you see "SELECT * FROM users WHERE..." without the actual WHERE clause.
The database dashboard shows aggregate query performance. The specific slow query that caused your p99 spike might not appear because it only ran 47 times today, and your dashboard aggregates queries by pattern, not by execution time.
Unified Observability (JustAnalytics or similar):
One trace ID flows from browser to database. Click on the slow request, see the full waterfall, identify the 4-second span, click into it to see the query. One tool, one correlation, no guessing.
That consolidation is the whole pitch: one script instead of five tools, one trace ID instead of timestamp matching.
Where Tracing Still Falls Short
No tool makes this painless, and it's worth knowing the gaps before you lean on traces.
A SQL string isn't a query plan. Seeing the statement tells you what ran, not why it was slow. You still have to run EXPLAIN ANALYZE yourself to tell a sequential scan from an index scan, and that's the step most trace views skip.
Forty-seven slow traces are tedious to click through. Grouping them by root cause span (this many slow on the database span, that many on Redis) is what you actually want, and most tools leave it to you.
Percentile alerts are reactive. By the time p99 crosses a threshold, users have already felt it. A relative baseline alert narrows the gap but doesn't close it.
Frequently Asked Questions
How do you identify which API endpoint is causing p99 latency spikes?
Start with your service map or endpoint latency dashboard filtered to p99. Sort by latency deviation from baseline. The endpoint with the largest p99 increase relative to its median is usually your culprit. In JustAnalytics, click into that endpoint to see individual traces, then filter to only show requests above your p99 threshold to find patterns.
Can you trace a slow API call from browser to database without switching tools?
Yes, with unified observability. JustAnalytics correlates RUM (browser) performance data with backend APM traces using a shared trace ID. A single click on a slow frontend request shows you the full waterfall: browser timing, API latency, downstream service calls, and individual database queries with their execution times.
What usually causes API p99 latency to spike while median stays stable?
Missing database indexes on edge-case queries, N+1 patterns that only trigger on large data sets, cache misses under specific conditions, or connection pool exhaustion during traffic bursts. The median stays stable because most requests hit the happy path. The p99 spikes because a small share of requests hit a slow branch your tests never exercised.
How quickly can JustAnalytics identify the root cause of a latency spike?
It depends on how complete your instrumentation is. With distributed tracing in place from browser to database, the path is short: receive the p99 alert, open the service map, identify the slow endpoint, click into a slow trace, follow the span waterfall to the slowest child span. Most p99 issues come down to one or two slow spans — usually a database query or external API call.
What's the difference between p99 latency and average latency for API monitoring?
Average latency hides outliers by smoothing slow requests into the overall mean. P99 shows the latency experienced by your slowest 1% of requests — often your most important users. A healthy average with bad p99 means most requests are fast but a minority are catastrophically slow, usually due to missing indexes, N+1 queries, or cache misses.
Try JustAnalytics
All-in-one observability in one under-5KB script: cookieless analytics + error tracking + APM + session replay + uptime + structured logs. Replaces GA4 + Sentry + Datadog + Pingdom + LogRocket. Free tier (100K events/mo), Pro $49/month ($39 annual).
Author at JustAnalytics.