OpenTelemetry metrics
in production.

OpenTelemetry metrics are easier to get wrong than traces. With traces, mistakes show up as visibly broken waterfalls. With metrics, mistakes show up as quietly wrong dashboards and surprise bills. This post is the short list of practical patterns we recommend to teams adopting OTel metrics in production.

Counter vs gauge vs histogram: which instrument when

OTel has four instrument types that matter in production. Pick the right one and the math works out; pick the wrong one and your dashboards lie.

Counter

Monotonically increasing. Use for things you count: requests received, errors logged, orders placed, bytes written. The value can only go up.

orders_total = meter.create_counter(
    "orders_total",
    description="Number of orders submitted",
)
orders_total.add(1, {"plan": "team"})

Common mistake: using a counter for the current value of something. Counters are deltas (or cumulatives), not snapshots. For "current number of active connections", use a gauge.

UpDownCounter

Can go up or down. Use for inventory-style values where the absolute number matters and you have explicit add and remove events: in-flight requests, queue depth, connections held.

in_flight = meter.create_up_down_counter("http.server.active_requests")
in_flight.add(1)  # on request start
# ... handle request ...
in_flight.add(-1)  # on request end

Gauge (observable)

The current value at observation time, measured by the SDK by calling your callback periodically. Use for things you poll rather than increment: CPU temperature, memory usage, queue length read from the queue itself.

def observe_queue_depth(callback):
    callback.observe(queue.qsize(), {"queue": "orders"})

meter.create_observable_gauge(
    "queue.depth",
    callbacks=[observe_queue_depth],
)

Histogram

Distribution of values over time. Use for latencies, sizes, durations: anything where you care about the shape of the distribution, not just the average. The backend computes percentiles (p50, p95, p99) from the buckets.

checkout_latency = meter.create_histogram(
    "checkout_latency_ms",
    unit="ms",
)
checkout_latency.record(348, {"region": "eu-west-1"})

Common mistake: averaging latencies. A p50 of 100ms and a p99 of 2,000ms is not the same as an average of 150ms. The average hides the tail. Always use histogram, never counter+division, for latency.

The cardinality trap

Cardinality is the number of unique label-value combinations on a metric. It is the single biggest cost driver in any modern metrics backend (commercial or self-hosted) and it is shockingly easy to blow up.

A counter http.server.requests with labels service (10 values), method (5 values), route (50 routes), status_code (10 codes) produces 10 × 5 × 50 × 10 = 25,000 unique series. That's fine. Now add user_id (1 million users) and you have 25 billion series. That's not fine.

The discipline:

• Labels should come from a bounded enum. http.status_code (hundreds of values, bounded by the spec) is fine. http.route (templated, like /orders/{id}, not /orders/12345) is fine.
• Anything unbounded (user ID, request ID, customer ID, build SHA, full URL, IP address) does not belong on a metric. Put it on a span attribute instead, or attach it via an exemplar (next section).
• Cardinality compounds across labels. Adding a label with 100 values to a metric that already has 1,000 series turns it into 100,000 series.
• Verify with backend dashboards: most observability backends (including Telemtra) expose per-metric series counts. Watch for the top-N highest-cardinality metrics weekly.

Exemplars: the underrated metric-to-trace link

Exemplars are sampled trace-IDs attached to metric data points. When a histogram bucket records "this request took 2,400ms", an exemplar captures the trace ID of one such request. Click the slow part of the histogram on a dashboard, jump to the exact trace.

This is the closest thing OTel has to a magic trick. Most teams enable exemplars and never look back. Enable them via SDK config:

# Python
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics._internal.exemplar import (
    AlignedHistogramBucketExemplarReservoir,
)
# (Exemplar reservoir is on by default in current SDK versions when
# a trace context is active. Verify your SDK version and Collector.)

The Collector must be configured to pass exemplars through. With OTLP-in and OTLP-out, exemplars travel by default. With Prometheus-in or Prometheus-out, exemplars require explicit configuration and use the Prometheus exemplar format (which is limited to 128 characters and one trace ID per data point).

Aggregation temporality: delta vs cumulative

Counters and histograms can be reported in two ways:

• Cumulative: the value reported is the total since process start. The backend computes rates by diffing. Default in OTel.
• Delta: the value reported is the increase since the last report. The backend sums directly.

Why this matters: cumulative metrics break across restarts. If a process restarts mid-window, the counter resets and the backend has to detect the reset to compute the correct rate. Most backends handle this well, but it's an extra source of edge cases (gauges that briefly go negative, rates that spike).

Delta is simpler for backends that aggregate at ingest time. If you're targeting a backend that prefers delta (some commercial APMs do), set OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta. Otherwise leave the default.

Collector pipeline patterns for metrics

Three patterns we recommend for metrics-specific Collector configuration:

1. Always batch

processors:
  batch:
    timeout: 10s
    send_batch_size: 8192

Larger batches than for traces are fine; metrics are more amenable to delayed export than spans.

2. Filter known noise

The filter processor drops metrics by name pattern. Useful for dropping kube-state-metrics noise, debug-only metrics, or metrics from sidecar containers you don't own.

processors:
  filter/metrics:
    metrics:
      metric:
        - 'name == "process_cpu_seconds_total" and resource.attributes["container.name"] == "istio-proxy"'

3. Cardinality limiting

The transform processor with OTTL can strip high-cardinality labels at the Collector. Useful as a safety net when application instrumentation is hard to control (third-party libraries, contractor code).

processors:
  transform/drop_user_id:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "user.id")

Prometheus interop without losing your mind

Most teams have existing Prometheus metrics they want to keep. The Collector's prometheus receiver scrapes them; the prometheusremotewrite exporter sends them to a Prometheus-compatible store. You can run OTel-native and Prometheus-scrape side by side.

One gotcha: Prometheus and OTel disagree on naming conventions. Prometheus uses http_server_request_duration_seconds; OTel uses http.server.request.duration with unit s. The Collector normalises one way or the other, but be deliberate about which convention is canonical in your stack. Mixing both in dashboards is confusing.

The one-page checklist

• Use histograms for latency, never counter+division.
• Keep labels on bounded enums; put unbounded identifiers on spans, not metrics.
• Enable exemplars; verify they survive the Collector pipeline.
• Use cumulative temporality unless your backend specifically prefers delta.
• Batch before export; filter noise at the Collector; cap cardinality with transform as a safety net.
• Watch the top-N highest-cardinality metrics weekly.

Frequently asked questions

When should I use a gauge instead of a counter?

Counters count events and only go up; gauges report the current value of something you poll. The instrument section above has the full decision table — and if you're reaching for a counter that needs to go down, that's an UpDownCounter.

What is metric cardinality and why does it matter?

The number of unique label-value combinations on a metric. It compounds multiplicatively across labels and drives cost in every backend — the cardinality trap section above shows how 25,000 series becomes 25 billion with one bad label.

What are exemplars?

Sampled trace IDs attached to metric data points, so a slow histogram bucket links to a real trace. Covered above, including the Collector configuration needed for them to survive the pipeline.

Further reading

See OpenTelemetry signals for the three-signal data model, OpenTelemetry best practices for production for the broader operational picture, and how to avoid noisy observability bills for cost mechanics. Start a Telemtra workspace if you want to point your OTLP metrics exporter at a backend in 60 seconds.