Home › Guides › What is observability
Tech Explained · 2026What Is Observability? How Logs, Metrics and Traces Actually Work in 2026
Observability is the practice of instrumenting a system so you can answer new questions about its behaviour from the data it already emits, without shipping new code. It rests on three signals: logs, metrics and traces. Teams that wire all three through OpenTelemetry often cut telemetry spend sharply, with one CNCF case study reporting 72%.
- Monitoring answers questions you wrote down last quarter. Observability lets you ask a new one at 2am against data already in the pipe.
- Traces are the signal most teams skip and most need. Only a trace tells you which of your nine services ate the 2.3 seconds.
- Learn OpenTelemetry, not a vendor. Four commands gets a Python service emitting traces, and the same instrumentation exports anywhere.
- Cost is a cardinality problem, not a volume problem. One metric label holding a user ID can multiply your bill by a thousand overnight.
- Sample in the Collector, not in your app. Keep every error, sample the boring 95 percent, and below five services skip tracing entirely for now.
It is 2:04am and the payments endpoint at a mid-size Indian fintech is timing out for one request in forty. The two-person platform team has a green CPU dashboard, a green error-rate graph, and a log search returning 180,000 lines that all say upstream request failed. Everything they built to watch the system is working, and none of it can say which service is slow.
What Is Observability, and How It Differs From Monitoring
The term comes from control theory, where a system is observable if you can infer its internal state from its outputs. In software the test is blunter: can you explain why one request was slow using only telemetry the system already emits? If the answer is "we would have to add a log line and redeploy", it is not observable yet, however many dashboards it has.
The real difference is who writes the question. Monitoring encodes questions decided in advance: CPU over 80 percent, error rate over 1 percent, queue depth over 5,000. Keep those, they are what pages you. Observability is asking something new mid-incident, like every request from the Android app that touched payments and took over a second, grouped by database host.
| Monitoring | Observability | |
|---|---|---|
| Question shape | Known in advance, encoded as a dashboard or alert | Asked for the first time during an incident |
| Typical data | Pre-aggregated metrics, fixed dimensions | High-dimension events and traces, per-request attributes |
| You define up front | Thresholds and alert rules | Instrumentation and attribute naming |
| Good at | Telling you something broke, cheaply, at scale | Telling you where and why it broke |
| Fails at | Novel failures, per-customer problems, tail latency | Being cheap if you instrument carelessly |
The alert that wakes you is monitoring. The twenty minutes after you wake up is observability, and that is where most teams are weakest. Certifications have caught up: the AWS DevOps Engineer Professional exam carries a dedicated monitoring and logging domain, which 360DT's live AWS Solutions Architect and DevOps course teaches as hands-on instrumentation work, and Microsoft files the same ground under implement instrumentation in the AZ-400 half of the Azure architect and DevOps program.
How Observability Actually Works, End to End
Four moving parts, and most confusion about observability comes from collapsing them into one word. Instrumentation runs inside your process and creates the data. Collection takes it off the host. Storage keeps it searchable. Query is where a human finally gets an answer.
One slow checkout request, from instrumentation to answer
The trace ID generated at the first service is what stitches three services into one searchable story.
Port and header names follow the OpenTelemetry specification, checked 3 November 2026. Timings are from the worked example in this guide.
Back at 2:04am, the fix is boring once a trace exists. The team filters for traces over one second on the checkout endpoint, opens one, and reads three spans: the API spent 2,310ms, of which payments spent 2,280ms, of which a single SELECT ... FOR UPDATE spent 2,240ms. No new log line. The question "which hop is slow" was answerable because every service passed the same trace ID along.
The one header that makes it work
Context propagation is the whole trick, and it is one HTTP header from the W3C Trace Context standard that OpenTelemetry implements:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
| | | |
version trace ID (32 hex) parent span ID flags (01 = sampled)Drop that header anywhere and the trace breaks into two unrelated fragments. That is the commonest reason a new tracing setup looks useless: a message queue, a custom HTTP client or a gateway is not forwarding traceparent. Check propagation before you blame the backend.
What Is Observability Made Of? Logs, Metrics and Traces
Logs, metrics and traces get called the three pillars, which makes them sound equal. They answer different questions, cost wildly different amounts per useful answer, and the order you adopt them in matters.
| Signal | The question it answers | What drives the cost | Where it hurts | Retention you actually use |
|---|---|---|---|---|
| Metrics | Is it broken, and how badly, right now | Active series, meaning unique label combinations | Cardinality explosion from IDs in labels | 13 months, for capacity trends |
| Logs | What exactly happened inside this one process | Gigabytes ingested and indexed | Debug logging left on in production | 7 to 30 days |
| Traces | Which service and which call ate the latency | Spans ingested, roughly 10 to 40 per request | Keeping 100 percent of boring traffic | 3 to 14 days |
Here is the opinion, and it cuts against the usual advice to start with logs because you already have them. Start with traces. Logs are what most teams already have too much of, and a log line cannot tell you which of nine services was slow because it only ever knows one process. Traces answer the expensive question, and the same instrumentation hands you request-rate and latency metrics almost free. Profiles come last, and only for a service that is genuinely CPU-bound.
OpenTelemetry Tutorial: Instrument a Python Service in Four Commands
OpenTelemetry is the CNCF project that defines the data model, the wire protocol and the SDKs. It matters for one practical reason: instrument once and point the output wherever you like, so moving from a free Grafana Cloud account to a paid vendor is a config change rather than a rewrite. That is the layer worth learning deeply, because it survives every vendor decision you make. Auto-instrumentation gets a Flask or FastAPI service producing real traces without editing application code:
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
export OTEL_SERVICE_NAME=checkout-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
opentelemetry-instrument python app.pyThe bootstrap step scans your installed packages and pulls the matching instrumentation libraries, so a project with Flask and requests gets both. Then opentelemetry-instrument wraps your normal start command, and inbound requests, outbound HTTP calls and database queries produce spans with header propagation handled for you.
That tells you a function was slow. It cannot tell you it was slow for Razorpay specifically, because it does not know your business. Four lines fixes that, and this is where tracing starts paying for itself:
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("verify_payment") as span:
span.set_attribute("payment.provider", provider) # razorpay, payu
span.set_attribute("payment.amount_inr", amount)
span.set_attribute("tenant.tier", tier) # free, pro, enterprise
result = provider_client.verify(order_id)
if not result.ok:
span.set_status(trace.Status(trace.StatusCode.ERROR))Span attributes are cheap and high-dimension, the opposite of metric labels. Putting tenant.tier on a span lets you later ask whether enterprise tenants hit the slow path more often, with nothing pre-aggregated. Putting the same field on a metric is how bills get out of hand. The same span-and-attribute discipline is what makes an LLM pipeline debuggable, which is why the AI Engineer course on RAG and AI agents works at trace level, while model drift and latency monitoring sit in the AI-300 syllabus taught in 360DT's MLOps Engineer program.
Also read: Docker Tutorial for Beginners 2026: Containerise a Python API in 7 Steps, which gives you the kind of service worth instrumenting next.
What Observability Costs in 2026, and the Cardinality Trap
Nobody has a telemetry problem until the invoice arrives. Then the question stops being which tool has the nicest UI and becomes why watching the system costs more than running it. It is almost always one of four things.
An ID in a metric label
A metric carrying user_id creates one active series per distinct value. Ten thousand users is ten thousand series from one metric, and ten thousand series is Grafana Cloud's entire free tier.
Debug logging that shipped
One logger.debug inside a request path at 400 requests a second is about 35 million lines a day. The fix is a log level you can change at runtime, not a redeploy.
Keeping every boring trace
Most of your traces are successful 40ms requests nobody will open. You pay full ingest price for them and learn nothing.
SamplingLong retention on everything
Teams set 90 days once, for compliance that applies to audit logs only, then pay it on traces and debug logs too. Set retention per signal.
RetentionSampling is the lever with the best ratio of saving to regret, provided you pull it in the right place. Head sampling, where the first service decides at random, discards errors at the same rate as successes. Tail sampling runs in the Collector once the whole trace has arrived, so it decides knowing whether the request failed or was slow:
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: keep-all-errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: keep-slow-requests
type: latency
latency: {threshold_ms: 1000}
- name: sample-the-rest
type: probabilistic
probabilistic: {sampling_percentage: 5}Spans retained per day under four sampling strategies
Tail sampling keeps every error and every slow request for about 6 percent of the volume of keeping everything.
Worked from a stated assumption of 50 million spans a day across 40 services, with 1.5 percent of requests erroring or exceeding one second: 0.75M interesting spans kept in full, plus 5 percent of the remaining 49.25M. Policy shape follows the Collector Contrib tail sampling processor, checked 3 November 2026.
- Tail sampling needs the whole trace in one Collector. Run several replicas behind a round-robin load balancer and spans from one trace land on different replicas, so the policies only ever see fragments. You need a load-balancing exporter that routes by trace ID. Teams find this out a week after the savings look great and the traces look broken.
- Dropping a metric label is not retroactive. Billing reflects active series across the period, so today's fix shows up next cycle. Separate audit logs before you touch retention, too, or compliance becomes its own incident.
On tools, here is where I would put money. Learn OpenTelemetry properly and run a managed free tier until it hurts, because 10,000 active series and 50 GB of logs a month covers a useful slice of a small production system. Self-hosting Prometheus, Loki and Tempo is a reasonable second step and a bad first one: you will spend month one operating storage instead of fixing latency. A per-host APM vendor is defensible to roughly thirty services and hard to justify past that.
Case Study: STCLab, From 5% Trace Sampling to Full Coverage
STCLab published its migration on the CNCF blog in December 2025, and the starting position will look familiar. Commercial APM was switched off entirely in development and staging because the licence was too expensive to spend there, and production ran at 5 percent trace sampling. The team was blind in the environments where bugs are cheap to fix, and working from one trace in twenty where they are expensive.
The work moved instrumentation to OpenTelemetry and storage to the open-source LGTM stack: Loki for logs, Grafana for query, Tempo for traces, Mimir for metrics. The reported result was a 72 percent cost reduction against the previous vendor with 100 percent trace coverage across all environments. The second number is the one worth stealing. They did not get cheaper by watching less, they got cheaper by owning the collection layer and spent the saving on coverage. A June 2026 case study of a European mobility platform reports the same mechanism and the same tail sampling pattern, with monthly spend falling from roughly 40,000 euros to 11,000 and mean time to identify from 47 minutes to 9. Both sets of figures are published by the parties involved, so treat the sizes as indicative and the mechanism as reproducible.
The honest caveat
Observability will not rescue a system whose real problem is architecture. If one Postgres instance is the bottleneck for every service you own, traces will tell you so beautifully, every night, and you will still have to shard the database. And on one application with one database, the full three-signal stack is a waste of your next month: structured JSON logs with a request ID, one latency and error dashboard, and an alert on the 95th percentile beat a half-finished platform.
Also read: Kubernetes Troubleshooting in 2026: 10 Pod Errors, Their Real Causes and the Fix for Each for the layer underneath this one, and the 16-week DevOps roadmap if you are assembling the whole skill set.
How to Learn Observability Without a Production System
The real obstacle for most learners in India is that observability looks like it needs live traffic you do not have. You can manufacture a substitute in a weekend: three small services in Docker Compose calling each other, a deliberate time.sleep inside the deepest one, and load from a loop of curl. That is a distributed system with a real latency bug and no obvious culprit.
Then five exercises, in order. Get auto-instrumentation emitting spans. Strip the traceparent header on purpose and watch the trace split in half. Add one manual span carrying business attributes. Run a Collector with tail sampling and confirm your error traces survive. Last, put a cardinality bomb in a metric label and watch the active series count climb. That one teaches more than any exam question on the topic, because you will never do it by accident afterwards.
For structured study, the monitoring and logging domain of DOP-C02 is the closest certification fit, and the 360DT certifications overview shows how it sits beside the Azure and data tracks. Pipeline-level telemetry in the Fabric world belongs to the DP-700 data engineering program, and if your dashboards are for business users rather than on-call engineers, that is a reporting skill from the Power BI and PL-300 data analyst course instead.
Learn instrumentation, logging and monitoring the way DOP-C02 tests it
A live weekend program over 12 weeks that prepares you for both AWS Solutions Architect Associate (SAA-C03) and AWS DevOps Engineer Professional (DOP-C02). It includes hands-on projects, mentor support and placement guidance.
Explore the course
What I Would Do in Your Position
If you want this on your CV inside a month, instrument one service with OpenTelemetry this week, push it to a free managed tier, and write up the cardinality experiment as a short public note. That one artefact separates you from everyone who has only read about the three pillars, and it is what an interviewer will ask follow-up questions about. If you are already on an ops or support team, take the certification route instead: DOP-C02 or AZ-400 gives your employer a reason to fund the learning, and the monitoring domain is the part you use on Monday. Either way, traces first, dashboards later. The live AWS and DevOps course is built in that order, and a free weekend webinar or a demo class is the cheapest way to check the pace.
Related guides
- SAA-C03 vs DOP-C02: Which AWS Certification Should You Take First in 2026? decides which exam to sit if the monitoring domain is what you want.
- AZ-305 vs AZ-400: Which Azure Certification Should You Take First in 2026? answers the same question on the Microsoft side, where instrumentation lives in AZ-400.
- DevOps Engineer Salary in India 2026 shows what this on-call skill set is advertised at by experience and company type.
- MLOps Engineer Salary in India 2026 is the adjacent path if you would rather monitor models than microservices.
- Cloud Engineer Jobs in Chennai 2026 names the GCCs where platform and reliability work is concentrated.
Frequently asked questions
What is observability in simple terms?
Observability means your system emits enough detail to answer a question you did not plan for, without adding code and redeploying. The practical test is whether a slow request can be traced to the service and the call that caused it.
What is the difference between observability and monitoring?
Monitoring checks questions decided in advance, like whether CPU is over 80 percent, and it is what pages you. Observability is investigating a failure nobody predicted, usually through traces. Monitoring says something broke; observability says where and why.
What are the three pillars of observability?
Logs, metrics and traces. Logs record what happened inside one process, metrics aggregate numbers over time so alerting stays cheap, and traces follow one request across every service it touches. Continuous profiling is increasingly treated as a fourth signal.
Is OpenTelemetry free to use?
Yes. OpenTelemetry is an open-source CNCF project covering the SDKs, the Collector and the OTLP wire protocol, with no licence cost. You pay only for whatever stores and queries the data.
How much does observability cost for a small team?
It can start at zero. Grafana Cloud's free tier covered 10,000 active metric series, 50 GB of logs and 50 GB of traces a month at 14-day retention for three users when checked in November 2026. Costs rise with active series and ingested gigabytes, not dashboards.
Do I need distributed tracing if I only run a monolith?
Usually not yet. With one application and one database, structured logs carrying a request ID plus a latency and error dashboard answer most questions. Tracing starts paying off around five services, and is essential once one request crosses team boundaries.
Which observability tools should a beginner learn in 2026?
OpenTelemetry first, because it is vendor-neutral and transfers everywhere. Then one query surface, realistically Grafana with Prometheus, Loki and Tempo behind it. Add the Collector's processors last, especially batching and tail sampling, because that is where cost control lives.
How do I practise observability without production traffic?
Run three small services in Docker Compose that call each other, add a deliberate delay in the deepest one, and drive load with a curl loop. Then break the traceparent header on purpose and watch the trace split. That teaches context propagation faster than any tutorial.
About this guide. 360 Digital Transformation is an Anthropic Partner and a Microsoft Authorized Training Partner. Other certification bodies, vendors and employers named here are not affiliated with us. Tools and versions change quickly; commands and figures cited were checked on 3 November 2026.
Amazon Web Services
