How to Cut Observability Data Volume in Half Without Losing the Signal That Matters
GuidesNovember 22, 202611 min read

How to Cut Observability Data Volume in Half Without Losing the Signal That Matters

Your observability bill doubled but debugging didn't improve. Here's how to cut data volume 50% without losing signal.

What This Is Really About

Observability tools charge by ingestion volume. More spans, more logs, more metrics — more money. The pricing model creates a perverse incentive: instrument everything, store everything, query almost nothing.

Most teams instrument first and optimize never. They add tracing to every service, log at DEBUG level in production (just in case), and create metrics with high-cardinality labels because someone might want to slice by user ID someday. (If you're also moving off GA4, see our GA4 migration guide.)

The result is data obesity. You're paying to store information you'll never look at. Worse, the sheer volume makes finding real signal harder — noise you're paying for the privilege of wading through.

But here's the thing most cost-optimization guides miss: you can't just slash blindly. Drop the wrong 50% and you're flying blind when production catches fire. The goal isn't minimum data — it's minimum data that still lets you debug at 3am.

Core Concepts Before We Start

Sampling vs. Filtering vs. Dropping

These aren't the same thing.

Sampling keeps a statistical subset of your data. You might keep 10% of traces, chosen randomly or by rule. You lose individual data points but retain statistical validity. Good for understanding patterns.

Filtering removes data that matches specific criteria. You might drop all health check requests or all logs from your chatbot service. You lose specific categories entirely but keep everything else complete.

Dropping (attribute-level) removes fields from the data you do keep. You keep the trace but strip the request body. Cheaper storage, less debugging context.

Each technique has trade-offs. Most teams need all three working together.

Cardinality and Why It Kills Your Metrics Budget

Cardinality is the number of unique values an attribute can take. A boolean has cardinality 2. A status code has maybe 20. A user ID has millions.

Here's why this matters: metrics systems (Prometheus, Datadog Metrics, Grafana Mimir) store each unique label combination as a separate time series. A counter with labels {service, endpoint, status} might have 50 services × 200 endpoints × 20 statuses = 200,000 time series. Add {userId} and suddenly you've got billions.

The fix isn't avoiding useful labels — it's understanding which labels belong on metrics versus traces. User IDs? Traces. Service names? Metrics. The boundary matters.

The Debugging Test

For every optimization: would this have broken your last three incident investigations?

Seriously. Pull up your last three production incidents. What data did you actually use to find the root cause? That data is sacred. Everything else is negotiable.

Teams often religiously preserve DEBUG logs "in case we need them" while sampling away the error traces that actually matter. That's backwards. Priorities, people.

The Complete Process: Reducing Volume Without Going Blind

Step 1: Audit What You're Actually Using

Before cutting anything, measure usage. You need to know:

  • Which dashboards get opened weekly versus never
  • Which log queries run during incidents
  • Which trace attributes appear in your debug workflows
  • Which metrics drive alerts versus sit dormant

Most observability platforms have query audit logs. Datadog shows dashboard view counts. Grafana tracks panel queries. Your log management tool knows which queries ran.

Dashboards nobody has opened in 90 days, and saved queries that never run, are not data worth preserving. They're clutter.

Step 2: Implement Tiered Trace Sampling

This is where the biggest wins live. Most teams capture 100% of traces. Most teams should capture 5-15% of successful traces and 100% of interesting ones.

Tail-based sampling is the key. With tail-based sampling, you buffer the complete trace until the request finishes, then decide whether to keep it based on the outcome.

A starting set of keep rules (illustrative; tune the thresholds to your own traffic):

sampling:
  rules:
    # Always keep errors
    - name: errors
      match:
        error: true
      sample_rate: 1.0

    # Always keep slow requests
    - name: high_latency
      match:
        duration_ms: "> 2000"
      sample_rate: 1.0

    # Keep traces that hit payment systems
    - name: payment_flows
      match:
        service: ["payment-service", "checkout-api"]
      sample_rate: 1.0

    # Sample everything else aggressively
    - name: default
      sample_rate: 0.05

This keeps 100% of errors, 100% of slow requests, 100% of payment flows, and 5% of normal successful requests. The 5% gives you baseline visibility without paying for 100% of the data.

The OpenTelemetry Collector has a tail_sampling processor that does exactly this, and most commercial vendors offer something similar. Check your vendor's docs for the syntax, since the YAML above is a sketch rather than any one product's config.

The math is simple: if 95% of your requests succeed quickly and aren't payment-related, you keep only a small fraction of total traces while keeping 100% of the signal that matters. Your bill falls roughly in proportion to the share you drop.

Step 3: Drop High-Cardinality Attributes From Metrics

This is the silent budget killer. You add userId to a metric because it seems useful. Suddenly you've got cardinality explosion and your metrics bill tripled.

Attributes to drop from metrics (keep them in traces instead):

  • User IDs, session IDs, request IDs
  • Full URLs (use normalized path templates instead: /users/{id} not /users/12345)
  • Timestamps with high precision
  • Request/response bodies (why are these on metrics at all?)
  • UUIDs of any kind
  • Customer-specific identifiers

The OpenTelemetry Collector's transform processor can strip these:

processors:
  transform:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "user_id")
          - delete_key(attributes, "session_id")
          - delete_key(attributes, "request_id")
          - replace_pattern(attributes["http.target"], "/users/[0-9]+", "/users/{id}")

If you need to query by user ID, that's what traces are for. Metrics are for aggregates. This distinction saves serious money. (Our Django analytics tutorial shows a middleware that records the user ID on each request event.)

Step 4: Tune Log Levels and Filter at Ingestion

Logs are usually the biggest volume driver. And most log volume is garbage.

At the source: Change production configs from DEBUG to INFO or WARN. Your framework's connection pool doesn't need to announce every acquired connection in prod. It's not a personality trait.

At ingestion: Filter what you don't need before it hits storage.

Rules that work:

log_filtering:
  # Drop health checks entirely
  - match:
      path: ["/health", "/ready", "/metrics"]
    action: drop

  # Drop INFO from high-volume services
  - match:
      service: ["api-gateway", "load-balancer"]
      level: INFO
    action: drop

  # Drop connection pool noise
  - match:
      message: /pool.*connection|heartbeat.*received/i
    action: drop

  # Sample DEBUG logs if you must keep them
  - match:
      level: DEBUG
    action: sample
    rate: 0.01

Health check logs are astonishing volume. A pod that answers three health checks every 10 seconds logs 25,920 health check lines per day. Multiply by pods. It adds up.

Dropping health check logs isn't optimization. It's removing obvious waste that should never have been there.

Step 5: Aggregate Metrics Before Export

Instead of sending raw data points, pre-aggregate at the edge.

Bad: send a counter increment to your metrics backend every time an event occurs. At 10K requests/second, that's 10K data points/second.

Good: aggregate locally for 10 seconds, send one batch with the sum. Now each counter sends one data point every 10 seconds for the same information.

OpenTelemetry SDKs do this by default: they aggregate in process and export on an interval through a periodic metric reader. Prometheus exporters aggregate by default. If you're using StatsD, batching is built in.

The aggregation window is a trade-off: longer windows mean less data but coarser resolution. For most use cases, 10-30 second aggregation is fine. You don't need second-by-second request counts unless you're actively debugging. And if you are? Just bump the resolution temporarily. Yes, "temporarily" has a way of becoming permanent. But still.

Step 6: Set Retention Tiers

Not all data deserves the same lifespan.

Hot tier (7-14 days): Full-resolution traces, all logs, high-cardinality metrics. This is your debugging window.

Warm tier (30 days): Sampled traces, aggregated logs (counts by level/service, not full text), rolled-up metrics. Good for trend analysis.

Cold tier (90+ days): Daily aggregates only. Compliance and long-term trends.

Most observability platforms support tiered retention. If yours doesn't, you're overpaying for storage.

The cost difference is dramatic on most vendors: archive storage is typically priced at a small fraction of hot, indexed storage. Check your vendor's published rates for both before deciding what to demote.

Advanced Tips for Large-Scale Teams

Implement Cardinality Limits

Set hard caps on metric cardinality. If a metric exceeds 10K unique label combinations, drop new values or aggregate them into an "other" bucket.

Prometheus has sample_limit. Datadog has cardinality estimators. OpenTelemetry Collector can filter high-cardinality series.

The alternative is surprise bills when someone adds a log-derived label that explodes. Unbounded labels on metrics are rarely worth what they cost.

Use Sampling Cascades

For very high-volume systems, sample in stages:

  1. Edge sampling at 50% (before data leaves the service)
  2. Collector sampling at 20% of what arrives
  3. Backend sampling at 50% of what's stored

Net result: 5% of original volume, but the sampling happens at different stages with different rules. Edge sampling is fast but blind. Collector sampling can be tail-based. Backend sampling can be ML-driven based on anomaly detection.

Profile Your Trace Sizes

Some traces are huge because they include massive payloads. A 50KB request body in every span adds up.

Strip request/response bodies except for errors. For errors, truncate to 10KB. The full 2MB JSON blob rarely helps debugging — the first few hundred bytes usually tell the story. Nobody needs the entire customer record in every span.

Common Mistakes

Sampling errors at the same rate as successes. Errors are rare. Sample them at 100%. This is non-negotiable.

Dropping attributes you need for debugging. Run the debugging test before committing. Check your last three incidents.

Setting retention too short. 7 days feels like enough until you need to investigate something from 10 days ago. 14-30 day hot retention is more realistic for most teams.

Optimizing once and never again. Traffic patterns change. Services get added. Run a data audit quarterly. Our error-analytics correlation guide covers a related habit: tying errors to what users actually do.

Not measuring the before and after. You can't prove ROI without baselines. Capture your current ingestion volumes and costs before making changes.

Frequently Asked Questions

What percentage of observability data can you safely drop without losing visibility?

Many teams can drop around half of their observability data without meaningful visibility loss. The key is intelligent selection: keep 100% of error traces, sample successful requests at 1-10%, drop high-cardinality attributes that don't aid debugging, and filter logs below WARN in production. The data you're dropping wasn't helping you debug anyway.

How do you implement head-based vs tail-based sampling for traces?

Head-based sampling decides at request start (simple but blind to outcomes). Tail-based sampling waits until request completion, keeping traces with errors, high latency, or interesting attributes while discarding successful fast requests. Tail-based costs more compute but keeps better signal. Most teams should use tail-based with head-based fallback for throughput protection.

What are high-cardinality attributes and why do they explode costs?

High-cardinality attributes have many unique values: user IDs, session tokens, request UUIDs, timestamps with millisecond precision. Each unique value creates a new time series in metrics systems. A userId label on a counter with 100K users creates 100K time series. Drop these from metrics, keep them in traces where they're indexed differently.

Should you reduce log levels in production or filter at ingestion?

Both. Reduce at source first — change DEBUG to INFO in production configs. Then filter at ingestion — drop INFO logs from high-volume services that produce most of your log traffic. The combination cuts volume more than either alone, and source reduction is free while ingestion filtering still incurs network costs.

Resources and Next Steps

If your observability bill is eating your infrastructure budget, start with the audit. Know what you're storing, know what you're querying, and cut the gap.

For teams consolidating their stack, the economics get simpler. JustAnalytics bundles analytics, errors, APM, replay, uptime, and logs in one under-5KB script — Free tier at 100K events/month, Pro at $49/month ($39/month billed annually) for 1M events. Pricing is per plan with event quotas rather than per GB.

The goal isn't minimum data. It's minimum data that still lets you debug. Get that balance right and your observability bill drops while your debugging capability stays the same.

That's the win. Now go audit those dashboards nobody opens.


Try JustAnalytics

All-in-one observability in one under-5KB script: cookieless analytics + error tracking + APM + session replay + uptime + structured logs. Replaces GA4 + Sentry + Datadog + Pingdom + LogRocket. Free tier (100K events/mo), Pro $49/month ($39/month billed annually).

Start free → · AI Command Center MCP

JP
JustAnalytics Platform TeamContributor

Author at JustAnalytics.

Related posts