Telemtra
Sign in Start free
/blog/opentelemetry-sampling-strategies
head + tail sampling
free plan / 1 gb-mo
telemtra.io
// telemetry quality

OpenTelemetry sampling strategies:
head, tail, and what to actually keep.

You usually do not want every span from every request. This is the practical guide to sampling: head vs tail, parent-based vs ratio-based, and the Collector rules that keep your error traces and your slow traces while dropping the noise.

> Start free ~ Browse blog

Why sample at all

Tracing every request at scale is wasteful. Most traces are uninteresting: 200 OK, p50 latency, no errors. The valuable traces are the ones with errors, the ones that exceeded a latency budget, and a representative sample of the rest. Sampling is how you keep storage costs sane while preserving the traces that matter.

Head sampling: cheap and predictable

Head sampling decides at the start of a trace whether to record it. The decision propagates with the trace context, so every service in the trace makes the same call. This is the default and the cheapest option.

Configure it on the SDK side with environment variables. A 10% sampling rate, parent-respected:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.10

parentbased_traceidratio means: respect the parent's sampling decision if there is one (so cross-service traces are consistent), otherwise sample based on the trace ID at 10%.

The downside of head sampling is that you cannot keep "all error traces" — by the time the error happens, the trace was already discarded at the front door.

Tail sampling: smarter, more infrastructure

Tail sampling waits until every span in a trace has arrived, then makes the keep-or-drop decision based on what actually happened. This is where you keep all errors, all slow traces, and a sample of normal traffic.

Tail sampling lives in the Collector, not the SDK, because only the Collector has visibility into the whole trace. The tail_sampling processor is the standard tool:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 1500 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

That config keeps every trace with an error span, every trace slower than 1.5s, and 5% of the remaining traffic. In Telemtra this means your debugging traces are always there when you need them, and your monthly ingest is a fraction of what unfiltered tracing would cost.

Where to put the tail sampler

The tail_sampling processor needs to see every span of a trace, which means all spans for a single trace ID must reach the same Collector instance. There are three common patterns:

  • Single gateway. All apps send to one Collector deployment. Simple, but a single point of failure for telemetry.
  • Load-balanced exporter. Apps send to a load-balancing Collector that uses the loadbalancing exporter to route by trace ID to a fleet of tail samplers. Scales horizontally.
  • Service mesh sampling. Mesh data plane makes the decision; OTel SDK respects it via parent-based sampling.

What to keep in every case

Regardless of strategy, these traces should always be kept:

  • Any trace with a span in ERROR status.
  • Any trace slower than your stated latency objective (often p99).
  • Any trace that touched a new code path you are watching (release canary).
  • A small, randomized probabilistic sample of normal traffic so dashboards stay representative.

What to drop aggressively

  • Health check endpoints (/healthz, /readyz).
  • Static asset requests (/favicon.ico, /static/*).
  • High-frequency internal sidecar polling that produces no useful trace.

You can drop these at the SDK level with a custom sampler, or in the Collector with a filter processor that matches span name or URL.

Sampling logs and metrics

OpenTelemetry sampling is span-focused. Logs and metrics have their own quality controls. For logs, use Data Shaper to drop noisy fields at ingest. For metrics, prefer reducing cardinality (fewer label values) over sampling — a sampled metric is a misleading metric.

Putting it together for Telemtra

A common production setup for an OTel + Telemtra stack:

  • SDKs use parentbased_alwayson so trace context is always preserved across services.
  • A Collector layer applies tail_sampling with error + latency + 5% probabilistic policies.
  • The Collector exports to https://ingest.telemtra.io.
  • Data Shaper rules in Telemtra drop additional log noise that survives the Collector.

Result: every error and slow trace is in Telemtra for debugging, normal traffic is sampled for cost, and dashboards still see a representative slice. See the Collector quickstart and how to avoid noisy observability bills for related setup. Start a Telemtra workspace to receive sampled traces.

Further reading