Debug production 500 errors
with distributed tracing.

500 errors are the worst kind of production failure to debug because the response itself is a confession of "something went wrong, I can't tell you what." Without distributed tracing the next steps are guessing: which service? which endpoint? which downstream call? With OpenTelemetry traces in place, the next steps are mechanical. This is the runbook.

We assume you already export OTLP traces from your services and have them landing in a trace backend. Telemtra, Jaeger, Tempo, Datadog APM, doesn't matter for this playbook. The steps work in any tracing UI that shows spans with status codes.

Step 1: Confirm the scope

Before the trace dive, confirm what's failing and how much:

• Which services are returning 500s? Filter traces by http.status_code >= 500 and group by service.name. One service or many?
• Which endpoints? Group the same filter by http.route. One endpoint or many?
• Started when? Eyeball the trace count over time. Continuous low rate, recent spike, or specific deploy cliff?
• Affecting whom? Group by deployment.environment if you have multiple environments. Production only, or staging too?

One of three patterns will be obvious within sixty seconds:

• One service, one endpoint, sharp time cliff → recent deploy or config change. Skip to step 3.
• One service, many endpoints → service-level failure (DB connection, OOM, GC pause, dependency outage). Skip to step 2.
• Many services failing on a shared upstream → one dependency is down and cascading. Find the upstream first (step 2).

Step 2: Open the Service Map

The Service Map (drawn from trace edges) shows which service depends on which, with error rate per edge. The failing service will have red edges from it (it's the one calling things that fail) or red edges into it (callers see it as broken).

The pattern to look for: the red edges converge on one node. That node is the failure source. Common shapes:

• All red edges point at payment-api: payment-api is down, everything calling it sees 500s. Open payment-api.
• All red edges point at db: database is down or saturated. The application services are healthy; the dependency isn't.
• Red edges fan out from checkout-api to multiple downstreams: checkout-api itself is malfunctioning (not the downstreams). Open checkout-api.

Don't skip the Service Map even when you "know" which service is broken. It catches the case where the team-favourite suspect is innocent.

Step 3: Open a representative failing trace

From the failing service, open a recent trace where the root span returned http.status_code = 500. The trace waterfall shows every span in the request, with the slow ones long and the failed ones marked.

What to look at, in order:

1. The span with status = error and an exception event. OpenTelemetry's status field is the primary error signal. Click the span; the attribute panel shows the exception type, message, and stack trace if the SDK captured it.
2. The deepest red span in the waterfall. Errors propagate up the trace, but the root cause is the deepest red span: the one with no failing children. That's where the actual problem is.
3. The span just before the deepest red span. Sometimes the error is in the call following the slow span. Database timeouts, retries, circuit breakers triggering.
4. The HTTP/RPC client spans. If the deepest red span is an outbound HTTP or DB query, the failure is in something outside your application. Note the target, then look at that service's traces.

Step 4: Jump to logs from the failing span

If your applications use the OTel log bridge, every log emitted during the span carries the trace ID. The span detail panel links to those logs directly. No "logs from around that time" searching.

What logs typically reveal that the span didn't:

• The exact SQL query that failed (when the span just says "db query error")
• The full external API response body (when the span only captured the status code)
• Configuration values at the time of the failure (DB hostname, feature flag state)
• Application-level retry counters or circuit breaker state

If the log lines don't appear linked to the span, your application is probably logging via a non-OTel logger and missing trace-context attachment. Fix that after the incident; see OpenTelemetry logs for the bridge setup.

Step 5: Distinguish "broken once" from "broken pattern"

One failing trace tells you what failed in one request. To know whether to roll back, escalate, or treat the trace as informational, you need the pattern.

• Filter traces by the same span attribute that the failing trace had. Same endpoint? Same database? Same external API? Same downstream service? If the filter returns many failing traces, the root cause is structural.
• Compare time windows. Was this attribute returning errors before the incident started? If yes, the cause predates the incident and the incident is a different problem.
• Group by version or build SHA. If service.version is in your resource attributes, you can see whether the new build is the only one failing.

Step 6: Decide and act

By now you should know enough to pick one of four actions:

• Roll back. If the cause is a recent deploy of your code and the previous version was healthy, roll back is the fastest fix. Re-confirm with traces from the rolled-back version after deploy completes.
• Restart or scale. If the cause is resource pressure (DB connection pool exhausted, OOM, runaway thread) in your service, restart cycles the bad state. Scale out if the load is genuine.
• Failover or degrade. If the cause is a downstream dependency, failover to a replica or degrade the feature gracefully. The error trace tells you which downstream; the dependency owner deals with it.
• Communicate and stay calm. If the cause is external (a cloud provider outage, a partner API down), there's nothing to roll back. Status page the incident; the trace data becomes your post-mortem evidence.

What to fix before the next incident

Three improvements that pay back during the next 500-error incident:

1. OTel log bridge on every service. So step 4 actually works. Without it you're guessing which logs go with the failing trace.
2. Resource attributes set consistently. service.name, deployment.environment, service.version. Without these, step 5 is impossible.
3. Span attributes for the obvious dimensions. http.route (templated, not raw URL), http.status_code, db.system.name, db.statement (or hashed for sensitive workloads). The same attributes you'd want on the dashboard you don't have yet.

Further reading

See Trace-first debugging for the trace workflow in Telemtra specifically, Service Map for the dependency-graph step, and OpenTelemetry best practices for production for the instrumentation that makes this runbook possible. Start a Telemtra workspace to point an OTLP exporter at a backend that wires this workflow together.