APM for gRPC and Protobuf Microservices
How to wire distributed tracing into gRPC microservices — interceptor-based span propagation, deadline errors that actually tell you something, and streaming RPC traces that don't blow up your cardinality.
Three weeks ago I watched a P99 latency spike to 4.2 seconds on a service that handles internal payments. The gRPC call was timing out at the client, the server logs showed nothing unusual, and every metric we had said the service was fine. Spoiler: it wasn't fine. A downstream dependency had slowed down, but because we weren't propagating trace context across gRPC boundaries, the spans were orphaned. The trace showed the client waiting. It showed the server responding. The 3.8 seconds in between? Black hole.
That's what happens when you bolt APM onto gRPC services without thinking about the transport. (I should know — I've made this mistake on three different projects now.) HTTP tracing middleware doesn't work. gRPC uses HTTP/2 multiplexing, streaming RPCs, and a metadata system that's similar to HTTP headers but not quite the same. And if you're doing bidirectional streaming — which, let's be honest, is half the reason people pick gRPC — the "one request, one span" model falls apart entirely.
This tutorial walks through wiring distributed tracing into gRPC microservices the right way. We'll use Go (the most common gRPC server language), OpenTelemetry (because it's the standard now), and JustAnalytics as the backend. If you're migrating from GA4 to a unified observability platform, see our GA4 migration guide. The same patterns work with Jaeger, Datadog, or any OTLP-compatible collector — the interceptor code is identical.
Prerequisites
Before we start, you'll need:
- Go 1.21+ (we use generics and the newer slog package)
- A gRPC service with at least one unary RPC and one streaming RPC
- Basic familiarity with Protocol Buffers — you've written a
.protofile before - An OpenTelemetry-compatible APM backend (JustAnalytics, Jaeger, Datadog, etc.) — see our comparison of JustAnalytics vs Plausible vs Fathom for context on where JustAnalytics fits
If you don't have a gRPC service handy, the examples use a simple order service with CreateOrder (unary) and StreamOrderUpdates (server-streaming). I'll include the proto definitions.
Step 1: Add OpenTelemetry dependencies
Install the OTel SDK and the gRPC instrumentation package. The instrumentation package gives you interceptors that handle span creation and context propagation automatically — mostly.
go get go.opentelemetry.io/otel \
go.opentelemetry.io/otel/sdk \
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
You're pulling in four things: the core OTel API, the SDK for creating TracerProviders, an HTTP exporter for OTLP (works with JustAnalytics, Jaeger, Tempo, whatever), and the gRPC instrumentation library. That last one is what saves you from writing interceptor boilerplate.
The version compatibility matrix matters here. As of August 2026, otelgrpc v0.53+ requires OTel SDK v1.28+. If you're on an older SDK, pin otelgrpc to v0.49. I lost an hour to this last month when a go get pulled incompatible versions.
Step 2: Initialize the tracer provider
Before any spans can be created, you need a TracerProvider configured with your exporter. This goes in your main.go or a dedicated telemetry init file.
package main
import (
"context"
"log/slog"
"os"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)
func initTracer(ctx context.Context) (func(), error) {
exporter, err := otlptracehttp.New(ctx,
otlptracehttp.WithEndpoint(os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")),
otlptracehttp.WithHeaders(map[string]string{
"Authorization": "Bearer " + os.Getenv("JA_API_KEY"),
}),
)
if err != nil {
return nil, err
}
res, err := resource.Merge(
resource.Default(),
resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceName("order-service"),
semconv.ServiceVersion("1.4.2"),
semconv.DeploymentEnvironment("production"),
),
)
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(res),
sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
))
return func() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := tp.Shutdown(ctx); err != nil {
slog.Error("tracer shutdown failed", "error", err)
}
}, nil
}
A few things worth noting. The TraceIDRatioBased(0.1) sampler means we sample 10% of traces — adjust based on your volume. At 1M requests/day, that's 100K traces, which is plenty for debugging without blowing up your bill. The ParentBased wrapper ensures that if an incoming request already has a trace context (from another service), we respect its sampling decision.
The propagator setup is easy to forget. Without propagation.TraceContext{}, outgoing gRPC calls won't include the traceparent header, and your traces will be fragmented. I've seen this bug in production more times than I'd like to admit.
Step 3: Wire interceptors into your gRPC server
Here's where the actual magic happens. The otelgrpc package provides stats handlers that create spans for every RPC. Add them when you create your server.
package main
import (
"net"
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
"google.golang.org/grpc"
)
func main() {
ctx := context.Background()
shutdown, err := initTracer(ctx)
if err != nil {
slog.Error("failed to init tracer", "error", err)
os.Exit(1)
}
defer shutdown()
lis, err := net.Listen("tcp", ":50051")
if err != nil {
slog.Error("failed to listen", "error", err)
os.Exit(1)
}
srv := grpc.NewServer(
grpc.StatsHandler(otelgrpc.NewServerHandler()),
)
// Register your service implementations here
// pb.RegisterOrderServiceServer(srv, &orderServer{})
slog.Info("starting gRPC server", "port", 50051)
if err := srv.Serve(lis); err != nil {
slog.Error("server failed", "error", err)
}
}
That grpc.StatsHandler(otelgrpc.NewServerHandler()) line does the heavy lifting. Every incoming RPC now gets a span with the method name, status code, and duration. The span extracts trace context from incoming metadata automatically.
For clients calling other gRPC services, do the same thing:
conn, err := grpc.DialContext(ctx, "inventory-service:50051",
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)
Now when order-service calls inventory-service, the trace context propagates through, and you see the full call chain in your APM dashboard.
Step 4: Handle streaming RPCs without exploding cardinality
Here's where most tutorials stop — and where most production setups break. Streaming RPCs (server-streaming, client-streaming, bidirectional) don't fit the "one request, one span" model.
The naive approach creates a span per message. Don't do this. I've seen teams accidentally create 50,000 spans per minute from a single bidirectional stream that handles real-time updates. Their Datadog bill went from $400/month to $3,200/month in a week. Not a fun conversation.
The right approach: one span for the stream lifecycle, with events for significant messages.
func (s *orderServer) StreamOrderUpdates(req *pb.StreamRequest, stream pb.OrderService_StreamOrderUpdatesServer) error {
ctx := stream.Context()
tracer := otel.Tracer("order-service")
ctx, span := tracer.Start(ctx, "StreamOrderUpdates",
trace.WithAttributes(
attribute.String("order_id", req.OrderId),
attribute.String("stream_type", "server"),
),
)
defer span.End()
var messageCount int64
startTime := time.Now()
for {
update, err := s.getNextUpdate(ctx, req.OrderId)
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
span.SetAttributes(
attribute.Int64("message_count", messageCount),
attribute.String("end_reason", "context_done"),
)
return nil
}
span.RecordError(err)
return err
}
if err := stream.Send(update); err != nil {
span.RecordError(err)
span.SetAttributes(attribute.String("end_reason", "send_error"))
return err
}
messageCount++
// Only record events for significant messages, not every one
if update.Status == pb.Status_DELIVERED {
span.AddEvent("order_delivered", trace.WithAttributes(
attribute.String("delivered_at", update.Timestamp),
))
}
}
span.SetAttributes(
attribute.Int64("message_count", messageCount),
attribute.Int64("stream_duration_ms", time.Since(startTime).Milliseconds()),
)
return nil
}
The key insight: use span.AddEvent() for important milestones, not new spans. Events attach to the parent span without creating new trace IDs. You still get the debugging context ("when exactly did the stream start sending garbage?") without the cardinality bill.
Honestly, I wish someone had explained this to me before I burned a week's worth of observability budget in two days. Learn from my expensive mistakes.
Step 5: Make deadline errors actually useful
gRPC deadlines are one of the best features of the protocol — and one of the worst-instrumented. A DEADLINE_EXCEEDED error tells you the call timed out. It doesn't tell you why, where the time went, or whether the server even received the request.
Add deadline context to your spans:
func (s *orderServer) CreateOrder(ctx context.Context, req *pb.CreateOrderRequest) (*pb.Order, error) {
tracer := otel.Tracer("order-service")
ctx, span := tracer.Start(ctx, "CreateOrder.internal")
defer span.End()
// Record the deadline if one exists
if deadline, ok := ctx.Deadline(); ok {
remaining := time.Until(deadline)
span.SetAttributes(
attribute.Int64("grpc.deadline_ms", remaining.Milliseconds()),
attribute.String("grpc.deadline_at", deadline.Format(time.RFC3339)),
)
// Warn if we're already running low on time
if remaining < 500*time.Millisecond {
span.AddEvent("deadline_warning", trace.WithAttributes(
attribute.String("remaining", remaining.String()),
))
}
}
// Your actual logic here
order, err := s.processOrder(ctx, req)
if err != nil {
span.RecordError(err)
if errors.Is(err, context.DeadlineExceeded) {
span.SetAttributes(attribute.Bool("deadline_exceeded", true))
}
return nil, err
}
return order, nil
}
Now when a call times out, you can see how much deadline remained when it entered the service, whether it was already tight, and exactly where in the call chain the deadline ran out. This turns "we got a timeout" into "the inventory-service call took 2.3s of our 3s deadline, then payment-service took 0.8s, leaving us 0.1s for the rest — not enough."
Teams running performance-sensitive services (ad bidding, real-time pricing, anything with SLAs) should pair APM with click-fraud protection like ClickzProtect — latency spikes from bot traffic hitting your APIs show up as deadline pressure before they show up in conversion rates. Similar pattern applies for call tracking systems; VeloCalls operators often see API timeouts as the first signal of a traffic spike from a new publisher.
Common errors and how to fix them
"context.Background() has no trace context"
You're creating a new background context somewhere in your call chain, which orphans the trace. Always pass the incoming context through your function calls. The fix is usually to add ctx context.Context as the first parameter and stop using context.Background() except at the very top of main().
Spans appear but aren't connected
The propagator isn't set, or you're using the wrong one. Make sure you called otel.SetTextMapPropagator() with propagation.TraceContext{}. If you're calling a service that uses B3 headers (Zipkin-style), add b3.New() to your composite propagator.
"missing required attribute rpc.system"
Some APM backends (including strict OpenTelemetry collectors) require semantic convention attributes. The otelgrpc interceptors add these automatically for intercepted calls, but if you're creating manual spans, add them yourself:
span.SetAttributes(
semconv.RPCSystemGRPC,
semconv.RPCService("OrderService"),
semconv.RPCMethod("CreateOrder"),
)
Streaming spans have zero duration
You're ending the span before the stream completes. Make sure span.End() is deferred and runs only when the stream handler returns, not after the first message.
Traces work locally but not in production
Check that OTEL_EXPORTER_OTLP_ENDPOINT is set and reachable from your production pods. Kubernetes network policies often block outbound traffic by default. Also verify your API key environment variable is populated — silent auth failures are common.
My hot take: if your observability setup requires more than 10 minutes of debugging to get working in a new environment, something's wrong with your tooling, not your skills. Too many APM vendors make this harder than it needs to be. That's why we built JustAnalytics to replace GA4, Sentry, Pingdom, and LogRocket in one script.
Next steps
Once you've got basic tracing working, there's more to explore:
- Error tracking with source maps: If your gRPC services power a frontend, correlate backend traces with frontend errors. Our error tracking guide covers wiring this up across the stack.
- P99 latency debugging: gRPC is fast, but your P99 tells the real story. Check out finding P99 latency sources for techniques that apply directly to gRPC services.
- Alerting on deadline pressure: Set up alerts for when spans regularly consume more than 80% of their deadline — that's a leading indicator of future timeouts, before users start complaining.
- Framework-specific guides: If you're working with web frameworks alongside gRPC, check our Next.js 15 analytics tutorial or Django analytics middleware guide for the frontend/BFF layer.
The full example code is available in our docs. Good luck — and may your traces never be orphaned again.
Frequently Asked Questions
Does this work with gRPC-Web and browser clients?
Yes, but with caveats. gRPC-Web uses HTTP/1.1 or HTTP/2 through an Envoy proxy, so spans from the browser won't automatically propagate W3C trace context headers the same way server-to-server calls do. You'll need to inject the traceparent header manually in your gRPC-Web client code, then your Envoy proxy passes it through to the backend. The backend interceptors in this tutorial will pick it up from there. Most teams skip browser-side gRPC tracing entirely and just trace from the Envoy edge inward.
How do I trace streaming RPCs without creating thousands of spans?
Don't create a span per message — that's the mistake everyone makes first. Create one span for the entire stream lifecycle, then use span events (or logs attached to the span) for individual messages if you need that granularity. Set attributes like message_count, first_message_at, last_message_at on the parent span. You get the debugging context without the cardinality explosion. The Go code in this tutorial shows exactly this pattern.
What about gRPC deadlines vs OpenTelemetry span timeouts?
They're separate concepts that you should track together. gRPC deadlines are transport-level — the client says 'cancel after 5 seconds' and the server respects it. OpenTelemetry spans don't have built-in timeouts, but you should record the deadline as a span attribute (grpc.deadline_ms) and check whether the span ended due to DEADLINE_EXCEEDED. If your span duration approaches the deadline, that's a signal worth alerting on. We cover this in the deadline section.
Can I use this with Protobuf but not gRPC?
If you're using Protobuf over HTTP/JSON (like many REST APIs do for request/response serialization), the gRPC interceptor pattern doesn't apply — there's no gRPC transport. Instead, use standard HTTP middleware for tracing. The span attribute conventions (rpc.system, rpc.method, rpc.service) still work, but you'd set rpc.system to 'http' and add the protobuf message type as a custom attribute. This tutorial focuses on gRPC transport specifically.
Try JustAnalytics
All-in-one observability in one under-5KB script: cookieless analytics + error tracking + APM + session replay + uptime + structured logs. Replaces GA4 + Sentry + Datadog + Pingdom + LogRocket. Free tier (100K events/mo), Pro $49/month ($39 annual).
Author at JustAnalytics.