Home › Guides › AI agent observability
Tech Explained · 2026What Is AI Agent Observability in 2026? How Tracing Works, Observability vs APM and 6 Steps to Instrument Your Agent
AI agent observability is the practice of recording every step an agent takes, each model call, tool call and retrieval, as linked spans inside one replayable trace. It answers why an agent did something, not just whether it replied. Open source tiers start free at 50,000 events a month, so cost is rarely the blocker.
- A trace is the unit of truth. One user request becomes one trace with nested spans for planning, each tool call and each model call, joined by a parent span id.
- APM tools answer availability, not behaviour. Your existing dashboard will happily show 200 OK on a response that invented a refund policy.
-
Learn the vocabulary, not the vendor. The OpenTelemetry GenAI semantic conventions define
gen_ai.operation.namevalues such aschat,execute_tool,create_agentandinvoke_agent, and every serious backend reads them. - Free tiers are generous. Langfuse Hobby lists 50,000 units a month at no cost, LangSmith Developer lists 5,000 traces, and Datadog lists 40,000 LLM spans.
- Cost attribution is the feature that pays for itself. Token spend per tenant, per tool and per prompt version is what turns a surprise invoice into a line item.
- The conventions are still in Development status. Attribute names can shift, so wrap your instrumentation in a thin layer of your own.
- If you run one prompt and no tools, skip this. Observability earns its keep the moment an agent takes more than two steps on its own.
Your support agent handled 4,000 conversations last week and every one returned HTTP 200. Then a customer forwarded a screenshot where the agent promised a refund window your company does not offer, and your logs held one line: agent_response_sent. No record of which documents it retrieved, which tool it called, which prompt version was live. That gap between "the service was up" and "the service was right" is why AI agent observability exists as a separate discipline in 2026.
What Is AI Agent Observability, and What Makes It Different
AI agent observability is instrumentation designed around non-deterministic, multi-step work. A normal web request has one shape: in, query the database, out. An agent request has a shape it decides at runtime, and two identical inputs can produce different paths. You cannot debug that with a counter and a p95 latency chart. You need the path itself.
Three things get recorded. The structure of the run, as a trace of nested spans. The payloads, meaning the actual prompt, the retrieved chunks, the tool arguments and the output. And the numbers attached to each span: input and output tokens, cost, latency, model name, and whatever business keys you add such as tenant id or conversation id.
Payload capture is what separates this from classic monitoring, and it is also what gets teams into trouble. You are storing customer text in a third party system, which is a data protection decision rather than an engineering one. Decide early whether you redact at the SDK level or self host. Retrofitting that after six months of traces is painful.
Traces, spans and the agent loop
A span is one unit of work with a start time, an end time, attributes and a parent. The root span covers the whole agent invocation. Under it sit a planning model call, a tool execution, maybe a retrieval span, then the model call that writes the answer. Read top to bottom and you have the agent's reasoning path as a timeline. That timeline is what you send a colleague at 11pm instead of a paragraph of guesswork.
One request, one trace, six spans
What the agent actually did, in the order it did it, with tokens attached to each step.
Span and operation names follow the OpenTelemetry GenAI semantic conventions; token counts are illustrative. Checked 1 October 2026.
AI Agent Observability vs APM: What Your Existing Monitoring Misses
Teams usually arrive here with Datadog, New Relic or Grafana already wired up, and the honest question is what actually changes. APM is not wrong about agents. It is answering a different question.
| The question | Traditional APM | Agent observability |
|---|---|---|
| Unit of work | HTTP request, database query, job | Trace of one agent run, nested by step |
| Definition of failure | Error status, exception, timeout | Confident, well formed, wrong answer |
| Latency breakdown | Service and endpoint level | Per model call and per tool call, plus queue time |
| Cost visibility | Infrastructure spend | Input and output tokens priced per span, per tenant |
| Inputs kept | Headers and sampled metadata | Full prompt, retrieved chunks, tool arguments |
| Replay | Not a concept | Re-run the same inputs against a new prompt or model |
| Quality signal | None | Scores from automated evals and human annotation |
| Typical blind spot | Everything inside the model call | Host level resource exhaustion |
The last row matters. Agent observability platforms are poor infrastructure monitors, and APM remains better at CPU, memory and pod restarts. Run both and join them on the same trace id. Collector deployment and alert design, the half of this job that is not AI specific, are taught live in the AZ-400 DevOps track and the AWS DevOps track.
Also read: LLM Evaluation in 2026: How Evals Work, 8 Metrics That Matter and a Pipeline You Can Ship.
6 Agent Failure Modes Your Logs Will Not Catch
Here is what actually breaks, in rough order of how often it bites small teams.
The silent tool loop
The agent calls the same tool eleven times with slightly different arguments, gets nothing useful, and answers anyway. Latency looks fine because each call is fast. Only the span count gives it away.
Watch span count per traceRetrieval returned nothing
Your vector search matched zero chunks above threshold, the prompt template filled the context slot with an empty string, and the model improvised. Without a retrieval span holding the returned documents, this is invisible.
Log chunk ids and scoresContext creep
Conversation history grows turn by turn until input tokens triple and cost per conversation quietly follows. Nothing errors. Your invoice moves before your dashboard does.
Track input tokens per turnPrompt version drift
Someone edited a system prompt on Friday. Quality dropped on Monday. If the prompt version is not an attribute on the span, you are reduced to reading git history against a vague date.
Attribute every spanOne tenant eating the budget
A single heavy customer can burn many times the median cost per conversation. Aggregate cost graphs hide it completely; cost grouped by tenant id surfaces it in one query.
Group cost by tenantTruncated output
The model hit its max output tokens mid sentence and the finish reason said so, but your code only read the text field. Users see a cut off answer; your logs see a successful call.
Record the finish reasonWhat usually goes wrong on your first attempt is simpler than any of these. You instrument the model calls, because the SDK wrapper is one line, and skip the tool calls, because those live in your own functions. Then an incident arrives and the trace shows two confident model calls with a mysterious 900ms gap between them. The gap was your tool. Instrument your own functions on day one, or the trace lies to you by omission.
LLM Observability Tools Compared: Langfuse, LangSmith, Phoenix and Datadog
Four options cover almost every real decision in 2026. The limits below were read on vendor pricing pages on 1 October 2026 and they move often, so treat them as the shape of the market rather than a quote.
| Tool | Free tier | Paid entry | Self host | Convention | Pick it when |
|---|---|---|---|---|---|
| Langfuse | Hobby: 50,000 units a month, 30 day retention, 2 users | Per seat cloud tiers above Hobby | Yes, MIT core except the ee folders | OTel native, accepts OTLP/HTTP | You want prompt management, evals and self host optionality in one place |
| LangSmith | Developer: 5,000 traces a month, 1 seat | Plus at 39 USD per seat a month, 10,000 base traces | Enterprise only | Its own schema, OTel ingest supported | You already build on LangChain or LangGraph and want zero wiring |
| Arize Phoenix | Open source, run it locally in a container | Arize AX Pro listed from 50 USD a month | Yes, source available | OpenInference, Apache 2.0, works on any OTel backend | You need first class retrieval and re-ranking spans for a RAG system |
| Datadog | 40,000 LLM spans a month | Pro from 160 USD a month annual, 100,000 spans, then 3.50 USD per extra 10,000 | No | Reads OTel GenAI attributes | Your infrastructure already lives in Datadog and one pane of glass wins |
| OTel plus your own backend | Free software, you pay for storage | Your cluster and engineering time | Yes, fully | Raw GenAI conventions | You have a platform team and a hard data residency requirement |
With no incumbent, start with Langfuse. Not because it wins every feature comparison, since Phoenix models retrieval better and Datadog correlates with infrastructure better. Because a 50,000 event free tier, an MIT licensed core and an OTLP endpoint let you prototype on cloud, move to self hosted when legal asks, and change no instrumentation code. In year one that optionality beats any single feature. The trade off is running one more stateful service later: Langfuse at volume wants a ClickHouse backend, which is an operations commitment rather than a container you forget about.
The numbers that decide your first month
Free allowances are large enough that budget is almost never the reason teams skip this.
Read from vendor pricing pages and the OpenTelemetry semantic conventions repository on 1 October 2026.
OpenTelemetry GenAI Semantic Conventions: The Vocabulary Worth Learning
Pick the vendor last. Learn the attribute names first, because they are what makes your traces portable. The OpenTelemetry GenAI semantic conventions give every step a standard name through gen_ai.operation.name, with values including chat, embeddings, execute_tool, create_agent and invoke_agent. Token accounting lands on gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and the input count is specified to include cached tokens. On the metrics side, two histograms carry most production dashboards: gen_ai.client.operation.duration and gen_ai.client.token.usage.
One caveat that will save you a refactor. As of the versions published in 2026, these conventions are still marked Development rather than Stable, so attribute names can change without a major version bump. Write a thin wrapper of your own around span creation, five or six lines, so that when an attribute is renamed you edit one file instead of forty call sites.
A second vocabulary is in play. OpenInference, the Apache 2.0 convention set behind Phoenix, distinguishes span kinds such as LLM, TOOL, AGENT, CHAIN and RETRIEVER, with explicit document and tool call types. The GenAI conventions are broader across signals but collapse more of your agent's shape into generic spans. If your system is retrieval heavy, OpenInference describes it more precisely today. Both ride on OpenTelemetry transport, so this is labels, not lock in.
AI agent monitoring: what is actually worth paging someone for
Dashboards are not alerts. Four signals deserve a page: error rate on tool spans, p95 spans per trace crossing your loop threshold, cost per hour above a ceiling you set deliberately, and eval score on a sampled set dropping below a floor. Everything else is a weekly review item. Teams that page on token spikes instead of cost per hour end up ignoring the channel inside a fortnight.
These are the operating habits the MLOps Engineer course drills live, and the agent evaluation and deployment modules of the AI Engineer course build the traced agent that those alerts watch. If you are coming at this from the exam side, the architecture reasoning sits inside CCAR-F and the hands on tool use inside CCDV-F; the full certifications overview shows how they stack.
Also read: What Is Context Engineering in 2026? How It Works, Why Agents Fail and 6 Techniques That Cut Token Costs.
How to Instrument Your First Agent in 6 Steps
This is a weekend of work for one engineer on an existing agent, not a quarter.
Pick a backend and get one trace in
Sign up for a free tier or run Phoenix in a container. Deliverable: one real request visible as a trace in the UI, nothing else changed.
Auto instrument the model client
Add the OTel or vendor instrumentation for your SDK and set the service name. Deliverable: every model call appears with model name, token counts and latency.
Wrap your own tools by hand
Decorate each tool function so it emits an execute_tool span with its arguments and result size. Deliverable: no unexplained gaps in the timeline.
Add the keys you will group by
Attach tenant id, conversation id, prompt version and environment to the root span. Deliverable: cost per tenant answerable in one filter.
Decide redaction and retention
Choose what never leaves your network, and set retention to match your policy rather than the default. Deliverable: a written one page note your security reviewer can sign.
Attach evals to sampled traces
Score 2 to 5 percent of production traces automatically, and annotate the failures by hand weekly. Deliverable: a quality number that moves when you change a prompt.
Step 6 is the one most teams postpone and should not. Tracing tells you what happened; evals tell you whether it was any good. Without the second, you have a beautiful archive of unexamined conversations.
A Worked Scenario: Four Engineers, 1.4 Million Spans a Month
Take an illustrative scenario, not a real customer: a four person team at a Pune logistics SaaS running one support agent across 20,000 conversations a month. Each conversation averages three model calls, four tool calls and a root span, so eight spans. That is 160,000 spans a month, and once retries, a nightly summarisation job and an internal research agent are traced too, the real figure lands nearer 1.4 million.
On Datadog's published rates, 1.4 million LLM spans against a 100,000 span Pro allowance leaves 1.3 million extra, billed at 3.50 USD per 10,000, which is roughly 455 USD a month on top of the 160 USD plan. On Langfuse Hobby they blow through 50,000 units in the first two days, so they either move to a paid cloud tier or self host. Self hosting Langfuse on a modest cluster with ClickHouse costs them perhaps 80 to 120 USD a month in infrastructure plus a day a month of someone's attention.
The honest answer for this team is to sample. Trace 100 percent of the customer facing agent, where incidents cost money, and 10 percent of the internal research agent, where a bad answer wastes an analyst's afternoon. Span volume drops to roughly 300,000 a month and the hosted versus self hosted decision stops being urgent. Sampling is not a compromise you apologise for. It is the right answer once you accept that traces are not equally valuable.
- Tracing your evals. Running 500 eval cases through a traced agent generates eval spans too, and teams routinely double their bill in a single afternoon of experimentation.
- Default retention. Long retention on full payloads is the single largest line item in most invoices. Thirty days of payloads with longer retention on metrics only is usually enough.
- Synchronous export. If your exporter blocks the request path, a backend outage becomes your outage. Batch and export asynchronously.
- Logging secrets. Tool arguments often carry API keys and customer identifiers. Redact at the span level, before export, not in the UI afterwards.
Build the traced, evaluated agent that observability is meant to watch
The AI Engineer course teaches you to build agents that plan, use tools and act, certified on both Microsoft Copilot Studio and Claude Code. You ship RAG and agent systems live over 16 weeks, with evaluation and deployment treated as part of the build rather than an afterthought.
Explore the course
Limitations, and Who Should Not Bother Yet
Skip this entirely if your product is one prompt behind a form with no tools and no memory. Structured logging of the input, the output, the model name and the token counts gives you 80 percent of the value for an afternoon of work, and adding a tracing vendor at that stage is ceremony. The threshold is the second autonomous step: the moment the model decides what to do next, the decision path becomes the thing you need to see.
Three limits are worth stating plainly. Traces explain mechanism, not meaning, so a perfectly instrumented wrong answer still needs a human or an eval to catch it. Payload capture moves customer data into a vendor, which is a compliance conversation at most Indian enterprises and the reason self hosting keeps winning in regulated sectors. And the tooling is young: the conventions are not stable, and Python support runs ahead of TypeScript, Java and Go.
What this does not fix is a badly designed agent. If your agent loops because its tool descriptions are vague, observability shows you the loop in high definition and then waits for you to fix the prompt. Teams sometimes buy a platform hoping it will substitute for design work. It will not.
Also read: Enterprise AI Agents in 2026: Why Only 23% of Pilots Scale, and 6 Skills That Close the Gap.
What I Would Do in Your Position
If you are an engineer trying to get an agent into production this quarter, spend one weekend on steps 1 to 4 with a free Langfuse or Phoenix instance, then do steps 5 and 6 before you let the agent touch a customer. Do not evaluate four platforms first. The instrumentation code is the durable asset and the backend is replaceable, so the cost of choosing the wrong vendor is far lower than the cost of another month without traces.
If you are learning this for a job rather than a deadline, build one small agent with three tools, instrument it properly, break it on purpose and write up the trace that found the bug. That artefact gets you further in an interview than a certificate alone, because it is evidence you have operated an agent rather than demoed one. To build it with feedback instead of alone at midnight, the AI Engineer course is the straight path: 100+ hours live over 16 weeks, agents and RAG through to evaluation and deployment, with a free demo class first if you want to see the teaching.
Related guides
- What Is an MLOps Engineer in 2026? Role, Skills, AI-300 Certification and Career Path in India the job title that owns this instrumentation once it reaches production.
- How to Build Production RAG: Why Your Demo Works and Your Deployment Doesn't the retrieval failures your spans will keep catching, explained at source.
- What Is an AI Agent Harness in 2026? How Orchestration Works, Frameworks Compared and How to Start Building One the layer that creates the spans you are reading.
- MLOps Engineer Salary in India 2026: Advertised Pay by Experience, Company Type and Skill Stack what the production operations skill set is advertised at right now.
- AI Engineer Interview Questions 2026: 18 Real Questions With Model Answers several of these questions are answered best with a trace.
- 7 Claude API Projects That Get You Hired as an AI Developer in 2026 pick one of these to instrument as your portfolio proof.
Frequently asked questions
What is AI agent observability in simple terms?
It is recording every step an agent takes as a linked set of spans so you can replay the run later. One user request becomes one trace, and inside it you see the planning model call, each tool call, each retrieval and the final answer, with tokens, cost and latency attached to each step.
Is LLM observability the same as APM?
No. APM answers whether your service was available and fast. LLM observability answers what the model and its tools actually did, and whether the output was any good. Both matter, and the practical move is to run them side by side and join them on the same trace id.
Which AI agent observability tool should a beginner start with?
Langfuse if you want a hosted free tier that lists 50,000 units a month, or Arize Phoenix if you would rather run a container locally and keep all data on your machine. Both speak OpenTelemetry, so your instrumentation code survives a change of mind later.
Do I need OpenTelemetry to trace an AI agent?
Not strictly, since most vendors have their own SDK. You want it anyway. Using the GenAI semantic conventions means a trace produced by one library reads correctly in another backend, which is the difference between a migration and a rewrite.
How much does LLM observability cost in production?
Less than most teams fear, if you sample. Free tiers in 2026 range from 5,000 traces a month on LangSmith Developer to 40,000 LLM spans on Datadog and 50,000 units on Langfuse Hobby. Beyond that, Datadog lists 3.50 USD per additional 10,000 spans, so a million span month is a few hundred dollars unless you sample the low value traffic.
What should I actually alert on for an agent in production?
Four things: error rate on tool spans, p95 spans per trace crossing your loop threshold, cost per hour above a ceiling you set, and eval score on a sampled set below a floor. Treat everything else as a weekly review metric or the channel gets muted.
Can I keep customer data out of a third party observability tool?
Yes, in two ways. Redact or hash payload fields inside your own span wrapper before export, which keeps structure and numbers while dropping text. Or self host, which Langfuse supports under its MIT licensed core and Phoenix supports as a source available container.
Is observability part of any AI certification syllabus?
Production operations and monitoring of AI workloads appear in the Microsoft AI-300 machine learning operations track and in the architecture reasoning tested by Claude Certified Architect Foundations. The certifications overview maps which exam covers which layer.
About this guide. 360 Digital Transformation is an Anthropic Partner. Other certification bodies, vendors and employers named here are not affiliated with us. Product features and pricing change often; figures cited were checked on 1 October 2026.
Python
Claude API
FAISS
ChromaDB
LangChain
LangGraph
CrewAI
AutoGen (AG2)
