Distributed Tracing: Fix Split Traces Before Your Collector Rollout

Distributed tracing records a request’s end-to-end journey across services as a trace made of spans, so you can see where time is spent and where failures cross service boundaries. OpenTelemetry, with its Collector, is the modern standard for producing and routing that data. Use it when logs and metrics tell you something is slow or broken, but not where.

TL;DR:

• Keep W3C Trace Context and B3 propagation aligned across applications and proxies; otherwise, mesh requests can split into disconnected trace trees.
• Instrument one critical request path and a canary service first, then measure latency and CPU overhead before expanding coverage or adding manual business logic spans.
• Use tail sampling to retain every error and traces slower than five seconds for critical services, but route each trace’s spans to the same Collector.
• Avoid DaemonSet scraping across the cluster, which can multiply metric volume by 20–40x; use a per node target allocator for local targets.

Table of Contents

• Trace and span model: what each piece means
• How context propagation keeps spans correlated
• What tracing shows that logs and metrics cannot
• When to adopt tracing and how to start small
• Instrumentation with OpenTelemetry: auto, manual and the Collector
• Sampling and data-volume control with tail-based policies
• Rollout and deployment: Collector topology, Kubernetes and sizing
• Security and hygiene for Collector hosting and propagation
• Troubleshooting split traces behind Envoy and a service mesh
• Our view: instrumentation is an iteration problem
• How we can help with your tracing rollout
• FAQ
• Sources

Trace and span model: what each piece means

A handful of concepts is enough to read any trace correctly:

• Trace: the full path of one request, identified by a trace ID.
• Span: one timed operation inside that trace, such as an HTTP handler or a database query.
• Span context: the trace ID, span ID and flags that travel with the request.
• Baggage: key-value pairs propagated alongside the context, to be used sparingly.
• Span kinds: client, server, producer, consumer and internal, which describe the role of the operation.

Spans form a tree through parent/child relationships. Where causality is not a simple call, such as a batch job consuming many messages, span links connect related traces without nesting them. Name spans after the operation, not the instance (“GET /orders/{id}”, not “GET /orders/8841”), and attach attributes you will actually filter on: tenant, route template, status code.

How context propagation keeps spans correlated

A trace stays whole only if every hop passes its context forward. The sender injects the trace ID and span ID into a carrier, usually HTTP headers, and the receiver extracts them to create a child span, as described in the context propagation documentation. OpenTelemetry’s default propagator follows the W3C Trace Context specification.

• W3C Trace Context: the traceparent header carries trace ID, parent span ID and sampling flag, while tracestate carries vendor-specific data.
• OTLP: the OpenTelemetry protocol used to ship spans from applications to the Collector, which acts as a neutral ingestion and processing point. The same architecture underpins Jaeger v2’s OpenTelemetry core.
• Multiple propagators: B3 still appears in practice because older libraries, proxies and meshes emit it, so systems often run tracecontext and B3 side by side.

Instrumentation libraries usually handle injection and extraction automatically, which is why propagation bugs surface only at the boundaries they do not cover.

What tracing shows that logs and metrics cannot

Metrics tell you that P99 latency rose. Logs tell you what one service did. A trace gives you the causal chain between them, so you can locate the slow hop in a request that touches a dozen services. It also exposes patterns neither signal captures well: intermittent latency from one downstream dependency, retries stacking up across services, or an asynchronous job that fans out and waits on its slowest branch.

The practical habit is to stamp the trace ID into your logs and attach it as an exemplar to metrics. You then pivot from a latency alert to a representative trace, and from that trace to the exact log lines, rather than correlating timestamps by hand.

When to adopt tracing and how to start small

Tracing pays for itself where the question is “where did the time go?” or “which service broke this request?”. High-value cases include P99 latency, SLO burn analysis, failures that cross several services and complex asynchronous flows. For a monolith with a single database, it adds less.

Start with one critical request path and a canary service, and measure the instrumentation overhead before widening. Tie the effort to outcomes you can verify: faster detection (MTTD), faster resolution (MTTR) and clearer SLO diagnostics. A practical OpenTelemetry guide recommends the same pattern of canary instrumentation and sampling tied to SLOs.

Instrumentation with OpenTelemetry: auto, manual and the Collector

OpenTelemetry decouples instrumentation from the backend, so you instrument once and change backends through configuration, for example to Jaeger or Tempo. That is also why it reduced vendor lock-in. A sensible order of work:

1. Enable auto-instrumentation for your language runtime and frameworks.
2. Watch latency and CPU, since auto-instrumentation overhead should be monitored.
3. Add manual spans around business logic that auto-instrumentation cannot see, such as pricing or fraud checks.
4. Keep provider initialisation separate from instrumentation code, so exporters change per environment without touching business logic.
5. Send everything to the Collector, where receivers, processors and exporters form one central pipeline.

Pro Tip: Use an in-memory exporter in unit tests to assert that your critical spans exist, with the right names and attributes, before they reach production.

Sampling and data-volume control with tail-based policies

Sampling controls ingestion cost, storage and overhead. Tail-based sampling in the Collector decides after a trace completes, so it can keep what matters. The OpenTelemetry demo configuration shows the pattern, and a Datadog walkthrough describes the same approach in practice.

• Always keep error traces.
• Always keep slow traces, for example those above five seconds, from critical and high-criticality services.
• Use a service.criticality attribute to vary probabilistic rates by tier.
• Sample remaining normal traffic at a lower probability.

Combine probabilistic, attribute and latency rules rather than relying on one. Note that tail sampling needs all spans of a trace to reach the same Collector instance, which shapes your topology.

Rollout and deployment: Collector topology, Kubernetes and sizing

Roll out in phases: pick the critical requests first, enable services in groups and review the data before expanding.

1. Run lightweight per-cluster collectors with central routing, the cost-effective pattern described by CNCF.
2. Avoid cluster-wide scraping from a DaemonSet, which can multiply metric volume by 20–40x; use a per-node target allocator so each collector scrapes only local targets.
3. Give collectors adequate memory headroom for production and pin collectors with nodeAffinity.
4. Tune batching, queue sizes and the memory_limiter processor so load is shed gracefully instead of the process hitting an out-of-memory kill.
5. Load test with realistic span volumes before enabling tail sampling.

Security and hygiene for Collector hosting and propagation

Treat the Collector as production infrastructure. Following the hosting best practices, bind endpoints to localhost or the pod IP instead of 0.0.0.0, restrict Collector UIs and use authentication extensions on exposed channels.

The configuration best practices recommend minimising the components you enable to reduce attack surface, storing secrets securely and scrubbing sensitive attributes with a redaction processor. Keep personal data out of baggage, since it travels to every downstream service. Be wary of context arriving from untrusted external callers: sanitise it or start a new trace at the edge. Balance compression, batching and retries against memory use, because aggressive retries can themselves exhaust a Collector.

Troubleshooting split traces behind Envoy and a service mesh

A classic symptom: one request produces disconnected trace trees from the Envoy sidecar and the application. The cause is usually mismatched propagation, with Envoy and the app extracting different headers. The CNCF mesh observability guide documents the fixes.

• Enable B3 propagation alongside W3C Trace Context, or configure Envoy to extract the same headers as your application.
• Let the Collector accept several formats and normalise them into OTLP for one backend.
• Validate by checking that traceparent survives each hop.
• Confirm the Collector has a receiver for every format in use.
• Compare trace IDs end to end for a single test request.

Our view: instrumentation is an iteration problem

Teams rarely fail at tracing because of the concepts. They fail because the feedback loop is slow: instrument, deploy, discover a propagation gap, wait for another sprint. We think the loop shortens when the people tuning the Collector and sampling policy are the same engineers who read the traces, which is how we work, without an account-manager layer in between.

Outside help earns its cost at three moments: architecture review before rollout, mapping SLOs to sampling rules, and operationalising the pipeline so it survives staff changes. We can run an initial assessment to scope a rollout that does not disturb production.

— Pepe F.

How we can help with your tracing rollout

If you want a rollout that is scoped, sized and supported, we work with you directly on:

• an initial assessment of your services, propagation paths and current telemetry;
• architecture and Collector deployment, including Kubernetes topology;
• sampling policy design tied to your SLOs;
• long-term support after go-live.

The next step is simple: request an initial assessment or a CTO advisory engagement, or see our wider engineering services.

FAQ

What is distributed tracing in simple terms?

Distributed tracing follows a single request as it moves through multiple services and records each step as a timed span. Together the spans form a trace that shows where time was spent and where an error occurred.

Is OpenTelemetry required for distributed tracing?

No, but it is the industry-standard framework for instrumentation and lets you switch backends through configuration. Choosing it avoids tying your code to one vendor.

What is the difference between head-based and tail-based sampling?

Head-based sampling decides when a trace starts, while tail-based sampling decides after it completes. The tail approach lets you keep every error and slow trace, as in the OpenTelemetry sample configuration.

Why do my traces split across a service mesh?

Usually the mesh proxy and the application use different propagation headers, so the second hop cannot find its parent. Aligning propagators or normalising formats in the Collector resolves it.

Sources

• Datadoghq
• Context propagation | OpenTelemetry

This article was produced with AI assistance and reviewed for accuracy. It is provided for general information only and is not professional advice.