APM for Scala Play and Pekko Actor Systems
Actor mailboxes break thread-local trace context. Here's how to propagate spans across Pekko actors in a Play Framework app without losing your mind.
The dashboard showed a 3-second P99 latency on our checkout endpoint. The Play controller logged 40ms. The database query logged 12ms. Somewhere between the HTTP request arriving and the response leaving, 2.9 seconds vanished into the actor system.
I spent four hours that Friday with println statements and log timestamps before I realized the problem. Our tracing setup — OpenTelemetry hooked into Play — tracked the HTTP layer fine. But every time a message crossed an actor mailbox, the trace context dropped. The spans existed. They just weren't connected to each other. What looked like a single slow request was actually six orphaned spans floating in the void, unlinked, useless.
Actor mailboxes break thread-local context. That's the fundamental problem. And if you're running Scala Play with Pekko (or classic Akka), you've probably hit this exact wall.
This tutorial shows how to propagate trace context across actor boundaries in a Play Framework application using OpenTelemetry. By the end, you'll have connected traces that follow requests from HTTP handler through actor hierarchies and back — the kind of visibility that would've saved me that Friday.
What you'll have by the end
A Play Framework 3.0 application with:
- OpenTelemetry instrumentation on HTTP routes (automatic via the Java agent)
- Manual context propagation through Pekko actor messages
- Connected distributed traces that survive mailbox hops
- Spans that actually show where your latency lives
Look, this isn't the most elegant solution. Wrapping every message in an envelope feels clunky at first. I resisted it for weeks. But clunky-and-working beats elegant-and-broken when your CEO is asking why checkout is slow.
The whole setup integrates with JustAnalytics via OTLP export. Same patterns work for any OpenTelemetry-compatible backend.
Prerequisites
- Scala 3.3+ and sbt 1.9+
- Play Framework 3.0+ (we're on 3.0.2)
- Pekko 1.0+ (ships with Play 3.0)
- Basic familiarity with actors and the ask pattern
- A JustAnalytics account — free tier covers 100K events/month, plenty for dev
If you're running Play 2.9 with classic Akka, the code is nearly identical — swap org.apache.pekko for akka in imports.
Why thread-local tracing breaks in actors
Most APM tools store trace context in ThreadLocal variables. When an HTTP request arrives, the instrumentation stashes the current span in thread-local storage. Any code running on that thread can grab the context and create child spans.
Actors destroy this assumption.
When you send a message to an actor, it lands in a mailbox. The actor picks it up later — potentially on a different thread, potentially much later, potentially after the original request thread returned to the pool. The thread processing your message has no idea what span started the work.
Here's what happens without explicit propagation:
HTTP Request → Controller (Span A)
↓
actor ! message
↓
Actor receives message (Span B, new trace — no parent)
Span B exists. But it's orphaned. Your APM shows two disconnected traces instead of one request.
The fix: carry the trace context inside the message itself.
Step 1: Add OpenTelemetry dependencies
Update your build.sbt:
// build.sbt
val openTelemetryVersion = "1.36.0"
libraryDependencies ++= Seq(
"io.opentelemetry" % "opentelemetry-api" % openTelemetryVersion,
"io.opentelemetry" % "opentelemetry-sdk" % openTelemetryVersion,
"io.opentelemetry" % "opentelemetry-exporter-otlp" % openTelemetryVersion,
"io.opentelemetry" % "opentelemetry-sdk-extension-autoconfigure" % openTelemetryVersion,
"io.opentelemetry.instrumentation" % "opentelemetry-instrumentation-annotations" % "2.2.0"
)
For automatic HTTP instrumentation, we'll use the OpenTelemetry Java agent at runtime. Download it:
curl -L -o opentelemetry-javaagent.jar \
https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.2.0/opentelemetry-javaagent.jar
Then add to your sbt run configuration or startup script:
sbt -J-javaagent:opentelemetry-javaagent.jar \
-J-Dotel.service.name=your-play-app \
-J-Dotel.exporter.otlp.endpoint=https://otlp.justanalytics.app \
-J-Dotel.exporter.otlp.headers=Authorization=Bearer\ YOUR_API_KEY \
run
At this point, HTTP requests get traced automatically. Play controllers, database calls through JDBC, outbound HTTP — all instrumented.
The problem is everything that happens inside actors. And honestly? The OpenTelemetry docs barely mention actors. I spent way too long searching for "pekko instrumentation" before accepting I'd have to roll my own.
Step 2: Create a traced message envelope
The pattern that actually works: wrap every actor message in an envelope that carries the span context.
// app/tracing/TracedEnvelope.scala
package tracing
import io.opentelemetry.api.trace.Span
import io.opentelemetry.context.Context
case class TracedEnvelope[T](
payload: T,
traceContext: Context
)
object TracedEnvelope {
def wrap[T](payload: T): TracedEnvelope[T] = {
TracedEnvelope(payload, Context.current())
}
def wrapWithSpan[T](payload: T, span: Span): TracedEnvelope[T] = {
TracedEnvelope(payload, Context.current().`with`(span))
}
}
Context.current() grabs whatever trace context exists on the current thread. When you send the message, that context travels with it.
Step 3: Build a traced actor base class
Now we need actors that unwrap the envelope and restore context before processing:
// app/tracing/TracedActor.scala
package tracing
import org.apache.pekko.actor.typed.Behavior
import org.apache.pekko.actor.typed.scaladsl.{AbstractBehavior, ActorContext, Behaviors}
import io.opentelemetry.api.GlobalOpenTelemetry
import io.opentelemetry.api.trace.{Span, SpanKind, Tracer}
import io.opentelemetry.context.Scope
trait TracedBehavior {
protected val tracer: Tracer = GlobalOpenTelemetry.getTracer("pekko-actors")
protected def withTracing[T](
envelope: TracedEnvelope[T],
operationName: String
)(block: T => Unit): Unit = {
val span = tracer.spanBuilder(operationName)
.setParent(envelope.traceContext)
.setSpanKind(SpanKind.INTERNAL)
.startSpan()
val scope: Scope = span.makeCurrent()
try {
block(envelope.payload)
} catch {
case e: Throwable =>
span.recordException(e)
throw e
} finally {
span.end()
scope.close()
}
}
}
The withTracing method does the heavy lifting: extracts the parent context from the envelope, creates a child span, makes it current (restoring thread-local context for any nested operations), runs your logic, and cleans up.
Is this boilerplate? Yeah. Do I wish the JVM had better primitives for this? Also yeah. But it works.
Step 4: Wire it into a real actor
Here's an order processing actor that uses the pattern:
// app/actors/OrderProcessor.scala
package actors
import org.apache.pekko.actor.typed.{ActorRef, Behavior}
import org.apache.pekko.actor.typed.scaladsl.Behaviors
import tracing.{TracedBehavior, TracedEnvelope}
object OrderProcessor extends TracedBehavior {
sealed trait Command
case class ProcessOrder(orderId: String, replyTo: ActorRef[OrderResult]) extends Command
case class ValidateInventory(orderId: String, items: List[String], replyTo: ActorRef[OrderResult]) extends Command
sealed trait OrderResult
case class OrderProcessed(orderId: String) extends OrderResult
case class OrderFailed(orderId: String, reason: String) extends OrderResult
def apply(): Behavior[TracedEnvelope[Command]] = Behaviors.receive { (context, envelope) =>
withTracing(envelope, s"OrderProcessor.${envelope.payload.getClass.getSimpleName}") { command =>
command match {
case ProcessOrder(orderId, replyTo) =>
context.log.info(s"Processing order $orderId")
// Your order logic here
// Any database calls or HTTP requests made here
// will automatically attach to this span
replyTo ! OrderProcessed(orderId)
case ValidateInventory(orderId, items, replyTo) =>
context.log.info(s"Validating ${items.size} items for order $orderId")
// Inventory check logic
replyTo ! OrderProcessed(orderId)
}
}
Behaviors.same
}
}
The actor only accepts TracedEnvelope[Command], not raw commands. This forces callers to wrap messages — no accidental context loss.
Step 5: Send traced messages from controllers
In your Play controller:
// app/controllers/OrderController.scala
package controllers
import javax.inject._
import play.api.mvc._
import org.apache.pekko.actor.typed.{ActorRef, Scheduler}
import org.apache.pekko.actor.typed.scaladsl.AskPattern._
import org.apache.pekko.util.Timeout
import scala.concurrent.{ExecutionContext, Future}
import scala.concurrent.duration._
import actors.OrderProcessor
import actors.OrderProcessor._
import tracing.TracedEnvelope
@Singleton
class OrderController @Inject()(
cc: ControllerComponents,
orderProcessor: ActorRef[TracedEnvelope[OrderProcessor.Command]]
)(implicit ec: ExecutionContext, scheduler: Scheduler) extends AbstractController(cc) {
implicit val timeout: Timeout = 30.seconds
def processOrder(orderId: String): Action[AnyContent] = Action.async {
// TracedEnvelope.wrap grabs the current span context
// (set automatically by the OpenTelemetry Java agent on HTTP entry)
val envelope = TracedEnvelope.wrap(ProcessOrder(orderId, _))
orderProcessor.ask[OrderResult](replyTo =>
TracedEnvelope.wrap(ProcessOrder(orderId, replyTo))
).map {
case OrderProcessed(id) => Ok(s"Order $id processed")
case OrderFailed(id, reason) => InternalServerError(s"Order $id failed: $reason")
}
}
}
The Java agent already created a span for the HTTP request. TracedEnvelope.wrap captures it. When the actor processes the message, withTracing creates a child span linked to that HTTP span.
Your trace now shows: HTTP Request → OrderProcessor.ProcessOrder → (any database calls) → response.
Step 6: Handle actor-to-actor messaging
Things get interesting when actors talk to each other. The same pattern works — pass the envelope along:
// Inside OrderProcessor, delegating to an InventoryActor
case ProcessOrder(orderId, replyTo) =>
context.log.info(s"Processing order $orderId")
// Forward with the SAME trace context
val inventoryEnvelope = TracedEnvelope(
InventoryActor.CheckStock(orderId, items, context.self),
envelope.traceContext // preserve the original context
)
inventoryActor ! inventoryEnvelope
Don't call TracedEnvelope.wrap here — that would capture the actor's thread context, not the original request context. Pass through the envelope's traceContext explicitly.
Common errors and how to fix them
Spans show up but aren't connected. You're probably calling TracedEnvelope.wrap inside the actor instead of passing through the original context. The actor's thread has no knowledge of the HTTP request. Always forward envelope.traceContext for inter-actor messaging.
"No tracer found" or GlobalOpenTelemetry returns a no-op. The Java agent initializes OpenTelemetry, but you're accessing it before the agent runs. Make sure the -javaagent flag comes before other JVM options. Also check that GlobalOpenTelemetry.get() is called after Play starts, not during static initialization.
Traces appear in logs but not in JustAnalytics. Check the OTLP endpoint configuration. Common issues: wrong protocol (use https:// not http://), missing auth header, or the API key has spaces that weren't escaped. Test with curl first:
curl -X POST https://otlp.justanalytics.app/v1/traces \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{}'
You should get a 200 or 400 (empty payload), not a 401 or connection refused.
Memory grows over time. If you're storing trace context in a mutable map keyed by message ID (I've seen this pattern — and yes, I wrote it once), you've got a leak. The envelope approach avoids this — context lives in the message itself and gets garbage collected when the message does. Learn from my mistakes.
What this approach doesn't cover
Scheduled messages via context.scheduleOnce won't carry context unless you wrap them explicitly. Same for timers and Pekko Streams — those need their own propagation patterns. We'll cover Pekko Streams tracing in a follow-up post.
Remote actors (Pekko Cluster) require serializing the trace context. The Context object isn't serializable by default — extract the W3C traceparent header string and include it in your message case class.
And if you're using Pekko's FSM or Persistence modules, the patterns are similar but require state machine-specific adaptations. (I haven't tested Persistence thoroughly. Your mileage may vary.)
Integration with the JustAnalytics stack
Once traces flow correctly, you've got visibility into where time actually goes. But traces alone don't tell you why something is slow or who it affected.
For the complete picture, JustAnalytics ties APM traces to error tracking — when an exception fires inside an actor, the trace shows exactly which span failed and what messages led there. That's the correlation I was missing on that Friday. Four hours into a debugging session, staring at disconnected spans, wishing someone had written this tutorial. So here it is.
If you're running Scala on multiple services, the OpenTelemetry setup works the same across your fleet. For teams managing multiple JVM applications alongside frontend stacks, VeloCalls provides similar observability for voice/call workflows — different domain, same distributed tracing challenges.
If your Scala app makes outbound HTTP calls to APIs protected by fraud detection, ClickzProtect tracks traffic patterns on the other side — useful when your P99 spikes correlate with suspicious inbound traffic you're processing. And for teams testing across multiple cloud accounts, JustBrowser keeps browser profiles separated while you verify traces show up correctly in different environments.
The Pro plan at $49/month ($39/month annual) covers 1M events with 1-year retention, including all the APM and tracing surfaces we've used here — plus error tracking, session replay, uptime monitoring, and structured logs. The free tier (100K events/month) is enough to validate this tutorial setup before committing.
Frequently Asked Questions
Does this work with classic Akka actors or only Pekko?
Both. Pekko is the Apache fork of Akka after Lightbend changed the license in 2022. The tracing patterns here work identically on Akka 2.6.x and Pekko 1.x — the APIs are the same, just different package names. If you're on Akka, swap org.apache.pekko for akka in your imports.
Can I use Kamon instead of OpenTelemetry for this?
Yes. Kamon has solid Pekko instrumentation and handles context propagation out of the box. The trade-off: Kamon is Scala-specific and sends to its own backend (or Datadog/Prometheus via reporters), while OpenTelemetry is vendor-neutral and works with any OTLP-compatible backend including JustAnalytics. If your team is already on Kamon, stay there. If you're starting fresh or want vendor flexibility, OpenTelemetry is the safer bet.
How much latency does this add to actor message processing?
Roughly 50-100 microseconds per message for context injection and extraction. On a system processing 10K messages per second, that's about 0.5-1 second of total overhead across all messages — spread across your actor pool. In practice, we've never seen this show up in flame graphs. Database calls, HTTP round-trips, and serialization dominate. The tracing overhead is noise.
(If tracing overhead is your bottleneck, you've either solved all the hard problems or you're running at a scale where you probably have dedicated observability engineers anyway.)
Will this work with Pekko Cluster and remote actors?
Yes, with extra work. For remote actors, you need to serialize the trace context alongside your message payload. The TracedEnvelope pattern in this post handles local actors. For cluster messaging, inject the W3C traceparent header into your message case class and extract it on the receiving node. Same pattern, just explicit serialization.
Try JustAnalytics
All-in-one observability in one under-5KB script: cookieless analytics + error tracking + APM + session replay + uptime + structured logs. Replaces GA4 + Sentry + Datadog + Pingdom + LogRocket. Free tier (100K events/mo), Pro $49/month ($39 annual).
Author at JustAnalytics.