AI Agent Observability: Tracing and Debugging with OpenTelemetry and LangSmith

You Cannot Fix What You Cannot See

Debugging a traditional API is straightforward: read the logs, check the status code, trace the request. Debugging an AI agent is a different problem entirely. The agent made seven LLM calls, used three tools, spent 45 seconds reasoning, and produced an answer that is subtly wrong. Where did it go off track? Which retrieval returned irrelevant context? Which reasoning step introduced the error?

Without observability, you are flying blind. Agent failures become anecdotal ("it sometimes gives weird answers") rather than systematic. In early 2026, observability tooling for AI agents has matured significantly, and teams that invest in it ship better agents faster.

The Three Pillars for AI Agents

Traditional observability rests on metrics, logs, and traces. AI agent observability extends these concepts with domain-specific requirements.

Traces: The Backbone

Every agent execution should produce a structured trace — a tree of spans showing the complete execution path. Each span captures an LLM call, tool invocation, retrieval operation, or reasoning step.

from opentelemetry import trace

tracer = trace.get_tracer("ai-agent")

async def agent_run(query: str):
    with tracer.start_as_current_span("agent.run") as span:
        span.set_attribute("agent.query", query)

        with tracer.start_as_current_span("agent.plan"):
            plan = await planner.create_plan(query)

        for step in plan.steps:
            with tracer.start_as_current_span(f"agent.step.{step.name}") as step_span:
                step_span.set_attribute("step.tool", step.tool_name)
                result = await step.execute()
                step_span.set_attribute("step.result_length", len(str(result)))

        with tracer.start_as_current_span("agent.synthesize"):
            answer = await synthesizer.generate(query, results)
            span.set_attribute("agent.answer_length", len(answer))
    return answer

Metrics: Cost, Latency, Quality

Agent-specific metrics go beyond request count and error rate:

• Token usage per model per step (for cost tracking)
• Latency breakdown across LLM calls vs tool calls vs retrieval
• Tool success rate — which tools fail most often
• Retrieval relevance scores — are we fetching useful context
• Agent loop count — how many reasoning iterations before completion
• Quality scores — automated evaluation of output quality (LLM-as-judge, reference matching)

Logs: Structured and Semantic

Every LLM call should log the full prompt, completion, model used, token counts, and latency. Every tool call should log inputs, outputs, and errors. These logs, linked to trace IDs, enable deep debugging of specific failures.

LangSmith for Agent Debugging

LangSmith (by LangChain) has become the most widely adopted agent-specific observability platform. It captures traces automatically for LangChain and LangGraph agents and provides a visual debugger for stepping through agent execution.

Key capabilities in the latest version:

• Trace visualization: See the full agent execution tree with expandable spans for each LLM call and tool use
• Dataset and evaluation: Create test datasets from production traces, run evaluations across model changes
• Comparison views: Side-by-side comparison of agent runs to identify what changed when behavior regresses
• Online evaluation: Attach LLM-as-judge evaluators that score production traces automatically

For non-LangChain agents, the LangSmith SDK provides manual tracing that works with any framework.

OpenTelemetry for AI: The Emerging Standard

The OpenTelemetry community has been developing semantic conventions specifically for generative AI. The opentelemetry-instrumentation-openai and similar packages auto-instrument LLM client libraries.

The advantage of OTel over proprietary solutions is integration with your existing observability stack. AI agent traces appear alongside your application traces in Jaeger, Grafana Tempo, or Datadog, providing end-to-end visibility from HTTP request through agent execution to database queries.

# Auto-instrument OpenAI client with OTel
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

OpenAIInstrumentor().instrument()
# All openai.chat.completions.create() calls now emit OTel spans

Arize Phoenix and Alternatives

Arize Phoenix provides open-source agent tracing with a focus on retrieval evaluation — it visualizes embedding spaces and identifies retrieval quality issues. Weights & Biases Weave offers experiment tracking combined with production monitoring. Helicone provides a lightweight proxy that captures all LLM calls with minimal integration effort.

Building an Observability Culture

The tooling is available. The harder part is building the habit. Every agent deployment should include a monitoring dashboard, every failure should be traced back to root cause, and every model change should be validated against evaluation datasets built from production traces. The teams building the most reliable agents in 2026 are the ones treating observability as a first-class engineering discipline, not an afterthought.

Sources:

• https://docs.smith.langchain.com/
• https://opentelemetry.io/docs/specs/semconv/gen-ai/
• https://docs.arize.com/phoenix