Ktor + OpenTelemetry: Distributed tracing for microservices without the observability tax

Meta description: Wire OpenTelemetry into Ktor microservices with coroutine-safe span propagation and Grafana Tempo — end-to-end distributed tracing for under $50/month, no Datadog required.

Tags: kotlin microservices backend architecture devops

TL;DR

You do not need Datadog, New Relic, or a five-figure observability contract to get production-grade distributed tracing. Ktor’s plugin architecture and Kotlin coroutines pair cleanly with the OpenTelemetry SDK. Wire in a Grafana Tempo backend and you have end-to-end request tracing across services for under $50/month. This post shows you exactly how to do it.

The observability tax is optional

Most teams conflate “enterprise-grade observability” with “expensive SaaS vendor.” That conflation is costing them money. The OpenTelemetry project — now a CNCF graduated project — has commoditized the instrumentation layer. The only remaining cost is the backend, and open-source options have closed that gap.

The math isn’t close. Datadog’s APM starts at roughly $31 per host per month, and that scales fast in a microservices environment. A self-hosted Grafana Tempo instance on a modest VM with object storage sits well under $50/month for most teams — same W3C TraceContext propagation, same OTLP wire format, query speeds that hold up at scale.

Wiring OpenTelemetry into Ktor

Ktor’s plugin system makes this straightforward. You install the OpenTelemetry SDK once at the application level and propagate spans through the coroutine context — no thread-local hacks required.

Dependencies

implementation("io.opentelemetry:opentelemetry-api:1.38.0")
implementation("io.opentelemetry:opentelemetry-sdk:1.38.0")
implementation("io.opentelemetry:opentelemetry-exporter-otlp:1.38.0")
implementation("io.opentelemetry.instrumentation:opentelemetry-ktor-2.0:2.4.0-alpha")

SDK initialization

val openTelemetry: OpenTelemetry = OpenTelemetrySdk.builder()
    .setTracerProvider(
        SdkTracerProvider.builder()
            .addSpanProcessor(
                BatchSpanProcessor.builder(
                    OtlpGrpcSpanExporter.builder()
                        .setEndpoint("http://tempo:4317")
                        .build()
                ).build()
            )
            .setResource(Resource.create(
                Attributes.of(ResourceAttributes.SERVICE_NAME, "order-service")
            ))
            .build()
    )
    .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
    .buildAndRegisterGlobal()

Installing the Ktor plugin

install(KtorServerTelemetry) {
    setOpenTelemetry(openTelemetry)
}

This intercepts every incoming request, extracts the traceparent header, and creates a root span. Child spans created inside route handlers inherit the trace automatically.

The coroutine context problem — and the fix

Kotlin coroutines do not use thread locals. This is the right design decision for performance, but it means naive OpenTelemetry instrumentation breaks span propagation the moment you launch or async across coroutine boundaries.

The fix is a CoroutineContext element that carries the OTel Context:

class OtelContextElement(val otelContext: io.opentelemetry.context.Context) : CoroutineContext.Element {
    companion object Key : CoroutineContext.Key<OtelContextElement>
    override val key = Key
}

suspend fun <T> withSpan(name: String, block: suspend () -> T): T {
    val tracer = GlobalOpenTelemetry.getTracer("app")
    val currentOtelCtx = coroutineContext[OtelContextElement]?.otelContext
        ?: io.opentelemetry.context.Context.current()
    val span = tracer.spanBuilder(name).setParent(currentOtelCtx).startSpan()
    val newOtelCtx = currentOtelCtx.with(span)
    return try {
        withContext(OtelContextElement(newOtelCtx)) { block() }
    } finally {
        span.end()
    }
}

Every withSpan call now correctly parents to the active trace regardless of which thread the coroutine resumes on.

Backend comparison: what you’re actually choosing between

Backend	Ingestion cost	Retention control	Query language	Self-hosted option
Datadog APM	$31+/host/month	Vendor-controlled	Proprietary	No
Grafana Tempo + Cloud	~$8/GB ingested	Configurable	TraceQL	Yes
Jaeger (self-hosted)	Infrastructure only	Full control	Jaeger UI	Yes
Honeycomb	$130+/month base	Vendor-controlled	BubbleUp	No

Grafana Tempo with S3-compatible object storage (Backblaze B2 or Cloudflare R2) is the sweet spot. TraceQL gives you powerful filtering — { .service.name = "order-service" && duration > 500ms } — without having to learn a proprietary query language.

What the trace actually shows you

In my experience building production systems, the first time engineers see a flame graph spanning three services — HTTP ingress, async Kafka consumer, downstream gRPC call — the conversation about observability investment changes. You stop guessing where latency lives and start measuring it.

With this stack, each trace carries:

Full coroutine-safe span hierarchy
HTTP attributes (method, status, route)
Custom business attributes via span.setAttribute("order.id", orderId)
Cross-service traceparent propagation via outgoing HttpClient interceptors

Where to start

Use the OTLP exporter, not a vendor SDK. The OpenTelemetry SDK is vendor-neutral — instrument once and swap backends without touching application code.

Carry OTel context explicitly in your coroutine context. Thread locals don’t survive coroutine suspension. The CoroutineContext.Element pattern above handles this correctly; build it once and it works everywhere.

Run Grafana Tempo with object storage from day one. The operational overhead is low, the cost ceiling is predictable, and you get full TraceQL access plus Grafana dashboards with no vendor lock-in.