APM for Rust Web Services: Tracing Axum and Actix Without a Vendor Agent
Rust APM tracing with no agent—finally, real observability.
The latency spike hit at 3:47 PM on a Thursday. P99 response times on our Axum API jumped from 45ms to 1,200ms. The metrics dashboard showed the spike, but not the cause. We had structured logs — lots of them — but no trace context linking the slow requests to the slow queries. Somewhere between the HTTP handler and the Postgres connection pool, 1.1 seconds were vanishing.
I'd been putting off proper APM instrumentation for months. The Datadog agent felt like overkill for a three-service setup, and Jaeger meant running another piece of infrastructure. Classic engineer brain: "I'll instrument it later." Later never came. So I kept adding log statements — dozens of them, scattered everywhere, each one a small admission of defeat. That Thursday I finally gave up on println! debugging at scale.
Here's the setup I landed on: Rust's tracing crate for instrumentation, tracing-opentelemetry for the bridge, and OTLP export straight to JustAnalytics. No sidecar. No agent process. The whole thing adds about 200KB to your binary and single-digit microseconds per span.
What We're Building
By the end of this, you'll have distributed tracing running on an Axum or Actix web service. Every incoming request gets a trace. Every database query, every HTTP call to a downstream service, every async task — all linked together with span context that propagates automatically. You'll be able to click on a slow request in your JustAnalytics APM dashboard and see exactly which line of code took 1.1 seconds.
The same approach works for any tower-based Rust service: Axum, Actix (via the tower compatibility layer), tonic for gRPC, warp, or plain hyper.
Prerequisites
- Rust 1.75+ (we're using async traits, which stabilized in 1.75)
- An existing Axum or Actix project (we'll show both)
- A JustAnalytics account with an OTLP endpoint (free tier works fine for testing)
- Basic familiarity with the
tracingcrate — if you've usedtracing::info!()before, you're good
Step 1: Add the Dependencies
Your Cargo.toml needs four crates. Five if you want automatic HTTP header propagation.
[dependencies]
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tracing-opentelemetry = "0.23"
opentelemetry = "0.22"
opentelemetry-otlp = { version = "0.15", features = ["tls-roots", "reqwest-client"] }
opentelemetry_sdk = { version = "0.22", features = ["rt-tokio"] }
# For Axum — automatic span creation on every request
tower-http = { version = "0.5", features = ["trace"] }
The tls-roots feature pulls in Mozilla's root certificates so OTLP export works over HTTPS without you manually configuring a cert bundle. I spent an embarrassing amount of time debugging "connection refused" errors before realizing the default build doesn't include TLS roots.
Run cargo build now. It'll take a minute — OpenTelemetry has a lot of transitive dependencies. Go get coffee. Seriously. The first build pulls in half the internet.
Step 2: Initialize the Tracer
This is the part that looks intimidating but is mostly boilerplate. Create a function that sets up the OpenTelemetry pipeline and wires it into tracing:
// src/telemetry.rs
use opentelemetry::trace::TracerProvider;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::{runtime, trace as sdktrace, Resource};
use opentelemetry::KeyValue;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
pub fn init_tracing(service_name: &str, otlp_endpoint: &str) {
let exporter = opentelemetry_otlp::new_exporter()
.http()
.with_endpoint(otlp_endpoint)
.with_headers(std::collections::HashMap::from([
("Authorization".to_string(),
format!("Bearer {}", std::env::var("JA_API_KEY").unwrap())),
]));
let tracer_provider = opentelemetry_otlp::new_pipeline()
.tracing()
.with_exporter(exporter)
.with_trace_config(
sdktrace::Config::default()
.with_resource(Resource::new(vec![
KeyValue::new("service.name", service_name.to_string()),
])),
)
.install_batch(runtime::Tokio)
.expect("Failed to install OpenTelemetry tracer");
let tracer = tracer_provider.tracer(service_name.to_string());
let telemetry_layer = tracing_opentelemetry::layer().with_tracer(tracer);
tracing_subscriber::registry()
.with(tracing_subscriber::EnvFilter::from_default_env())
.with(telemetry_layer)
.with(tracing_subscriber::fmt::layer())
.init();
}
Call this once at startup, before you bind your HTTP server. The fmt::layer() at the end means you still get console output in development — traces go to JustAnalytics AND to stdout.
One gotcha: install_batch spawns a background tokio task for flushing traces. If you're not inside a tokio runtime when you call this, it'll panic. Hard. No helpful error message, just "there is no reactor running." I wasted an hour on this because I was calling init from a test that didn't have an async runtime. Make sure your main() has the #[tokio::main] attribute or you're explicitly entering a runtime.
Step 3: Instrument Your Axum Router
Axum plays nicely with tower middleware, so adding per-request tracing is two lines:
// src/main.rs
use axum::{routing::get, Router};
use tower_http::trace::TraceLayer;
mod telemetry;
#[tokio::main]
async fn main() {
telemetry::init_tracing("my-api", "https://otlp.justanalytics.app/v1/traces");
let app = Router::new()
.route("/health", get(health_check))
.route("/users/:id", get(get_user))
.layer(TraceLayer::new_for_http());
let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap();
axum::serve(listener, app).await.unwrap();
}
async fn health_check() -> &'static str {
"ok"
}
#[tracing::instrument(skip(id))]
async fn get_user(axum::extract::Path(id): axum::extract::Path<u64>) -> String {
// Your actual handler logic
format!("User {}", id)
}
TraceLayer::new_for_http() creates a span for every request with the HTTP method, path, and status code. The #[tracing::instrument] macro on get_user creates a child span inside the request span — so if get_user calls a database function that's also instrumented, you get the full call tree.
Step 4: Instrument Actix (Alternative)
If you're on Actix instead of Axum, the setup is slightly different. Actix has its own middleware system, but since 4.0 it also supports tower layers via the actix-web-lab crate:
// src/main.rs (Actix version)
use actix_web::{web, App, HttpServer};
use actix_web_lab::middleware::from_fn;
use tracing_actix_web::TracingLogger;
mod telemetry;
#[actix_web::main]
async fn main() -> std::io::Result<()> {
telemetry::init_tracing("my-actix-api", "https://otlp.justanalytics.app/v1/traces");
HttpServer::new(|| {
App::new()
.wrap(TracingLogger::default())
.route("/health", web::get().to(health_check))
})
.bind("0.0.0.0:3000")?
.run()
.await
}
async fn health_check() -> &'static str {
"ok"
}
You'll need tracing-actix-web in your Cargo.toml:
tracing-actix-web = "0.7"
The TracingLogger middleware does the same job as Axum's TraceLayer — creates a root span per request with method, path, status, and latency. Honestly? I prefer Axum's tower integration for new projects. But if you've got an existing Actix codebase, this works fine.
Step 5: Instrument Database Queries
This is where tracing actually saves you. Wrap your database calls in instrumented functions:
#[tracing::instrument(skip(pool), fields(user_id = %id))]
async fn fetch_user_from_db(pool: &sqlx::PgPool, id: u64) -> Result<User, sqlx::Error> {
sqlx::query_as!(User, "SELECT * FROM users WHERE id = $1", id as i64)
.fetch_one(pool)
.await
}
Now when you look at a slow request trace, you'll see exactly how long the database query took versus the handler overhead. The Thursday outage I mentioned? Turned out to be a missing index on a join table. The trace showed a 1,050ms child span on fetch_user_from_db inside a 1,200ms request. A five-minute fix after weeks of "it's probably network latency." That's the kind of visibility you don't get from logs alone.
If you're using SQLx, there's also sqlx-tracing which instruments every query automatically. Worth considering if you have dozens of query sites.
Common Errors and How to Fix Them
"OpenTelemetry trace error: the global default trace provider is already set"
You called init_tracing() twice. This usually happens when running tests — each test case tries to initialize the tracer. Fix by wrapping init in a std::sync::Once or using the tracing-test crate which handles this for you.
"connection refused" or TLS handshake failures on OTLP export
Missing TLS roots. Make sure opentelemetry-otlp has the tls-roots feature enabled. Alternatively, you're hitting an HTTP endpoint with an HTTPS URL or vice versa — double-check the scheme.
Traces appear in the dashboard but spans are disconnected (no parent-child relationships)
Context isn't propagating. If you're making HTTP calls to downstream services, you need to inject trace headers. Use the reqwest-tracing crate or manually inject the traceparent header:
let mut headers = reqwest::header::HeaderMap::new();
// Inject current span context as W3C traceparent
opentelemetry::global::get_text_map_propagator(|propagator| {
propagator.inject_context(&tracing::Span::current().context(), &mut HeaderInjector(&mut headers));
});
High memory usage after running for a few hours
The span batch queue is backing up, usually because the OTLP endpoint is slow or unreachable. Check your exporter configuration — the default batch size and timeout are conservative. If JustAnalytics is temporarily unreachable, spans queue in memory. Reduce max_queue_size if you'd rather drop spans than OOM. (I learned this one in production. Not fun.) For teams running mission-critical services, pair this with uptime monitoring to catch endpoint availability issues before they cascade.
Next Steps
Now that traces are flowing, you'll probably want to add custom metrics (request counts, error rates, business KPIs) alongside the traces. JustAnalytics accepts OTLP metrics on the same endpoint — the setup is nearly identical, just swap tracing for metrics and add the OpenTelemetry metrics SDK. We cover that in our OpenTelemetry metrics guide.
For frontend-to-backend correlation, look into passing a trace ID from your browser sessions. JustAnalytics session replay can attach trace IDs to user interactions, so you can jump from a user's frustrated clicks directly to the backend span that caused the delay. Teams running aggressive click-fraud detection often pair this with ClickzProtect to correlate suspicious traffic patterns with their trace data — especially useful when debugging bot-driven load spikes.
And if you're dealing with the debugging workflow across multiple browser profiles during development, JustBrowser integrates with JustAnalytics to tag traces by browser profile — handy when you're testing auth flows that require separate sessions. For teams building internal tooling, DevOS pairs nicely with this observability stack for developer portal dashboards.
If your Rust services power outbound communications, VeloCalls offers similar OTLP-compatible telemetry for voice APIs, and JustEmails tracks email delivery metrics in the same unified dashboard.
The full setup we use is about 150 lines of Rust and exports traces, metrics, and logs through a single OTLP connection. No agent process. No sidecar. One fewer thing to wake you up at 3 AM.
Could I have figured out the Thursday outage faster with better logging? Maybe. Probably not, honestly. Tracing gives you the call graph for free. Logs make you reconstruct it in your head at 3 AM while half-asleep. Pick your poison — but I know which one I'm choosing now.
Frequently Asked Questions
Do I need a Datadog or Jaeger agent running alongside my Rust service?
No. The OTLP exporter sends traces directly to JustAnalytics (or any OTLP-compatible backend) over HTTPS. There's no sidecar, no local agent process, and no extra container to manage. The tracing-opentelemetry crate handles batching and retry internally. This is the main selling point over agent-based APM — one fewer thing to deploy and monitor.
Will adding tracing slow down my Axum or Actix handlers?
Barely. The tracing crate uses a compile-time disabled-by-default model — spans that aren't subscribed to compile down to no-ops. With an active subscriber and OTLP export, expect single-digit microsecond overhead per span on a typical x86 server. We've measured 4-8µs per span in production. If you're handling 10K requests per second, that's still under 1% of a typical 10ms handler. The batching exporter sends traces asynchronously, so your request path isn't waiting on network I/O.
Can I use this setup for gRPC services built with tonic?
Yes. Tonic is built on top of the same tower ecosystem as Axum, so tower-http's TraceLayer works identically. The only difference is your service definition — instead of axum::Router, you're wrapping a tonic::Server. The tracing and OpenTelemetry setup is unchanged. We've run this exact config on tonic services handling 50K+ RPCs per minute without modification.
How do I correlate frontend errors with backend traces?
Pass a trace ID header from your frontend to your Rust service. JustAnalytics session replay and error tracking can attach a trace_id property to browser events. On the Rust side, extract that header in middleware and inject it as the parent span context. Then your frontend error and backend trace share the same trace ID, and you can jump from a user's rage-click directly to the slow database query that caused it. We cover the frontend wiring in our session replay docs.
Try JustAnalytics
All-in-one observability in one under-5KB script: cookieless analytics + error tracking + APM + session replay + uptime + structured logs. Replaces GA4 + Sentry + Datadog + Pingdom + LogRocket. Free tier (100K events/mo), Pro $49/month ($39 annual).
Author at JustAnalytics.