AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
OpenTelemetry in Production: Make Logs, Metrics, and Traces Correlate
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Telemetry arriving is only the first boundary. Build resource identity, preserve trace context, and diagnose the broken links between metrics, traces, and logs.
TL;DR: OpenTelemetry correlation depends on explicit identity and pipeline contracts. Stable resources connect service-level evidence, while native trace and span fields connect individual logs to request traces. This article shows a complete Collector and OTLP sender example, then separates missing instrumentation context from sampling, retention, and export failures. It explains agent and gateway ownership, pipeline isolation, redaction, queue limits, and authenticated tenant routing. The example uses illustrative protocol records rather than operational measurements, and the production guidance makes the limits of intake acknowledgements and debug export clear.
The incident is visible; the evidence is disconnected
Your checkout latency alert fires immediately. The trace view shows slow payment calls. Application logs contain a useful upstream error, but searching the trace ID finds nothing. Three telemetry streams arrived successfully; the investigation still stalls.
The architectural problem is correlation, not collection. OpenTelemetry gives teams a common transport and data model, but the platform must establish resource identity, preserve request context, and operate three pipelines with explicit failure policies. This article builds a small OTLP intake example and explains the boundaries needed before that pattern reaches production.
Define the joins before choosing the backend
Treat resource identity as a platform contract. Application owners set a stable service name, deployment environment, and service version; platform owners enrich cluster and namespace identity. Validate those values in deployment templates. A gateway cannot safely infer the identity of every caller from its own pod.
Table. Columns: Evidence, Useful join, Dangerous shortcut.
Row 1. Evidence: Metrics; Useful join: Service, environment, bounded operation; Dangerous shortcut: Trace ID as a metric label.
Row 2. Evidence: Traces; Useful join: Trace ID, span ID, resource; Dangerous shortcut: Assuming every request was sampled.
Row 3. Evidence: Logs; Useful join: Resource plus native trace/span fields; Dangerous shortcut: Searching arbitrary body text.
End of table.
The log data model defines TraceId and SpanId fields. A string attribute called trace_id is not automatically the native TraceId. For legacy file ingestion, parsing must explicitly map the application field into the appropriate log context. Prefer SDK logging integration when it can capture active context correctly.
For metrics, use bounded dimensions such as route templates and status classes. Keep request IDs out of the time-series identity. Exemplars can reference representative traces without turning each request into a separate series; availability depends on instrumentation and backend support. Resource conventions align service-level queries, but they do not prove that two individual events belong to the same request.
Image. Investigation decision tree: separate missing identity, sampling, and export failures
Decide where enrichment and failure isolation belong
A node agent is useful for local file collection and host context. A gateway centralizes credentials, routing, and organization-wide processing. The official agent and gateway patterns can coexist.
Choose the topology based on ownership. A namespace team should control application instrumentation; the platform should own intake authentication, resource governance, and destination credentials. Give node collectors only the Kubernetes API permissions needed for their enrichment. Do not grant cluster-admin because metadata is missing.
Keep logs, metrics, and traces as explicit pipelines. Sharing a Collector deployment is convenient, but a log burst can still compete for process memory with traces. Separate deployments when retention policies, throughput, or incident-critical isolation justify the operating cost.
The Collector configuration guide explains component activation and processor order. A configured exporter that is absent from service.pipelines does nothing. Put admission and memory protection early, and batching after transformations. Redaction must precede every external exporter; it does not need to precede a processor that only checks memory.
A complete OTLP intake example
This example uses Collector Contrib 0.162.0, a published release. Prerequisites are Docker with Linux containers, Python 3.11 or newer, and a POSIX shell. Save these files in an otherwise empty directory. It is a protocol and correlation check, not a production architecture or performance experiment.
Save collector.yaml:
Code example. Refer to the written article for the complete code.
Start the isolated container. The receiver listens on all container interfaces, while the host port binds only to loopback:
Code example. Refer to the written article for the complete code.
Save send.py. The deliberately constructed span, log, and gauge demonstrate transport fields; they are illustrative records, not captured application observations.
Code example. Refer to the written article for the complete code.
Run the sender after the Collector starts, then inspect its output:
Code example. Refer to the written article for the complete code.
Verify the shared service resource and the matching native trace/span IDs in the debug export. Metrics should share resource identity, not acquire the trace ID as a label. The OTLP specification defines HTTP paths and JSON encoding, including hexadecimal trace IDs and string-encoded 64-bit integers.
A successful HTTP response is an intake acknowledgement. Check any partial-success rejection fields, Collector export errors, and backend visibility separately. This example has no durable storage and uses debug output instead of a backend. Detailed debug export can reveal sensitive payloads; never enable it indiscriminately in production.
Diagnose a broken correlation link
Consider an illustrative rollout where logs still arrive, but the logging bridge stops copying active span context. Trace IDs remain embedded inside JSON message bodies, and the backend's trace-to-log navigation stops working.
First, inspect a sanitized record before export. If native trace context is missing there, changing the backend query cannot repair instrumentation. Restore the logging bridge, verify asynchronous context propagation, or correct the file parser's mapping. Validate one known request across its span and log rather than comparing aggregate counts.
If native IDs exist, check whether the trace was sampled and retained. A log can legitimately refer to a trace that was never exported or has expired. Coordinate sampling and retention before treating a missing trace as collector loss.
If both signals disappear, inspect the pipeline boundary: receiver acceptance, processor refusals, exporter failures, queue occupancy, and destination authorization. The troubleshooting guide provides the investigation path. Restore connectivity or credentials before increasing buffers. A larger queue buys time; it cannot resolve a permanent rejection.
Production decisions this example intentionally leaves open
Use TLS and authenticated intake, restrict network paths, and give exporters destination-specific credentials. For multitenant collection, a caller-supplied tenant attribute is not an authorization boundary: derive routing from authenticated identity and enforce destination isolation.
Choose queue and retry settings against an explicit outage budget. Durable buffering needs separately configured persistence and storage capacity; it is not an implied property of OTLP. For tail sampling, route spans from the same trace to the same sampling decision point.
Follow the sensitive-data guidance: prevent collection where possible, then redact before export. Deleting one attribute does not remove secrets already embedded in free-text bodies. Review all three signals and every destination.
The useful acceptance criterion is operational: can the on-call engineer move from a service-level symptom to a retained trace and its relevant logs, and explain when that link is unavailable?
FAQ
Does OpenTelemetry store telemetry? The Collector receives, processes, and exports telemetry. Durable query storage requires a backend; configured queues serve a different purpose.
Should every log contain a trace ID? Logs emitted inside a request context can carry it. Startup and background events may have no active span, so resource identity remains essential.
Can all three signals share one Collector? Yes, with separate pipelines. Split deployments when throughput, ownership, or failure isolation makes shared resources a liability.