Observability isn't pretty dashboards. It's being able to answer what happened at 3 AM

It's 3:05 AM and nothing has paged anyone. The message comes from the support lead in Brazil instead: a few dozen customers say their orders have been stuck on "processing" since around midnight.

You open the main dashboard. CPU is flat, p99 latency is normal, the error rate sits at a comfortable 0.2%. Everything is green, and customers are still waiting.

So you start asking the questions every incident asks. Which customers? Since when, exactly? What changed around midnight? Where in the order flow are those requests stopping?

Twenty minutes later you're grepping logs that don't include the order ID, scrolling through traces where the failing requests were never sampled, and asking in the team channel whether anyone deployed something.

That gap, between the questions you need answered and what your data can actually answer, is what observability is about.

There's even a name for a wall of panels that stays green through it: watermelon metrics, green on the outside and red on the inside. Each panel is accurate about CPU, latency or error rate, and none of them tracks whether an order makes it through the queue.

Observability vs. monitoring: the difference shows up mid-incident

Monitoring answers questions you already knew to ask. Is the service up? Is latency above the threshold? Is the queue growing?

You define the check in advance, and a dashboard or an alert tells you when it fails. That's useful, and you need it.

Observability is the ability to answer questions you didn't prepare for. Honeycomb defines it as being able to investigate a system "by asking any question about its behavior," without adding new instrumentation "each time a new question comes up."

The stuck orders are a new question. Nobody built a panel for "orders from one payment provider that entered the queue after the 23:50 deploy and were never acknowledged." A system is observable when you can still get to that answer in minutes, by slicing the data you already have.

That's why "we have dashboards for everything" and "we have observability" are different claims: a wall of green panels only covers what you anticipated, and what breaks at 3 AM is, almost by definition, something you didn't.

How telemetry piles up without answering anything

The systems that fail at 3 AM rarely lack data. They have months of metrics, logs and traces. What's missing is a decision about what that data is for, so each piece got there by habit.

Google's SRE book has a blunt rule here: signals that are collected but don't show up in any dashboard or alert are "candidates for removal". You can follow that rule, or go one step earlier and decide what each signal is for before you collect it.

Some of these will probably sound familiar:

• Default dashboards. The vendor or the Helm chart shipped with CPU, memory and request-rate panels. Nobody chose them, so nobody knows which question they answer.
• Logs written for the developer, not the incident. "Processing order..." made sense while you were debugging the feature. Without the order ID, customer, region or version, that line can't be filtered or joined with anything else.
• Traces sampled to save money. Keeping 1% of traces at random cuts the bill, and it also means the rare failing request almost never makes it into the sample, which is exactly the trace you'll want at 3 AM.
• Labels stripped for cardinality. Someone removed the customer label from a metric because it made storage explode. It was a reasonable call, and it's also why you can't tell whether a spike hits everyone or one account.
• Context lost at the queue. The trace follows the request into the API and stops at the message broker, because the trace context never reached the consumer. In the 3 AM scenario, that's exactly where the orders were stuck.

Each of these made sense when someone decided it, and together they add up to a system that produces plenty of telemetry and still can't tell you who is affected, or why.

The same thing happens with resilience patterns: a circuit breaker nobody watches looks like protection on the architecture diagram while it quietly drops traffic.

The five questions every incident asks

This is usually where someone pushes back: if observability is about questions you didn't anticipate, how do you design for it, when nobody can list every failure in advance?

You don't need to. Whatever breaks, the questions during the incident are almost always the same five, and each one depends on data that has to exist before it starts:

1. Who is affected, and who is causing it? Which customers, tenants, regions or internal callers. That needs a customer or tenant ID on your events, plus the identity of whoever is calling, including your own services.
2. Since when? The exact moment it started, not "around midnight." That needs timestamps you can trust across services, and metrics fine-grained enough to show where the line bends.
3. What changed? A deploy, a feature flag, a config change, a shift in traffic. That needs the version and a deploy marker on your telemetry, plus a record of config and flag changes.
4. Where does it break? Which service, which hop, which step in the flow. That needs trace context that survives queues and async jobs.
5. Is it getting better or worse? Whether the blast radius is growing. That needs the same "who" dimensions tracked over time, like error rate per tenant, not just one global number.

If a field isn't there at 3 AM, no query and no tool will produce it. Two public postmortems from the past year show what that looks like.

Who is causing it? In April 2026, intermittent downtime in Bluesky's data plane hit about half of its users for roughly eight hours, and finding the root cause took about four days. A new internal service was sending huge batches of requests from a single data center.

As Jim Calabro, the Bluesky engineer who wrote the postmortem, put it: "we didn't have metrics per-client in the data plane." So the metrics confirmed the outage and gave no way to see which client was behind it.

What changed? During Cloudflare's global outage in November 2025, failures came and went in waves, and the team's first read was a massive DDoS attack. Their status page, hosted outside Cloudflare, went down at the same time by pure coincidence, which made the attack theory look even stronger.

The real cause was a permissions change in a ClickHouse cluster. A query that builds the Bot Management feature file started returning duplicate rows, the file doubled in size, and the proxy hit its 200-feature limit and began returning 5xx errors. The waves came from some database nodes still producing a good file while others produced a bad one.

Every failure showed up in the telemetry; linking those failures to the permissions change was the hard part.

That's what instrumenting with intent means: making sure that whatever the bug turns out to be, those five questions have an answer in the data.

What instrumenting with intent looks like in practice

Start from the questions and work backwards to the telemetry. In practice, that comes down to a handful of concrete decisions.

Prefer wide events to many thin log lines. A single structured event per unit of work (what's usually called a wide event), carrying request ID, customer or tenant, region, version, active feature flags and outcome, answers more questions than twenty lines saying "step 3 done." It's also much easier to filter and group.

Attach the deploy to everything. Every event and span should carry the service version or commit. When "what changed?" comes up, you want to compare before and after the deploy, not ask around in Slack.

Propagate context across every boundary. HTTP calls usually get this from OpenTelemetry's auto-instrumentation. Queues, scheduled jobs and webhooks usually don't, and that's where incidents like the stuck orders hide.

Sample with intent. If you need to sample traces, keep every error and every slow request, and sample only the healthy ones, so the traces you keep are the ones an incident will ask for.

Tail-based sampling does this, and it isn't free: the collector has to hold each trace until it completes before deciding whether to keep it, which costs memory and adds one more component to run. Weigh that against how often you need the traces you're dropping today.

Decide where high-cardinality data lives. Customer IDs don't belong in every metric label, but they have to live somewhere you can query, like structured events or traces. Dropping them everywhere is how you lose "who is affected?"

All of these are decisions about what your code emits, and you can make them with the tools you already have.

A test you can run this week

Take the last real incident your team had. Open the incident channel, not the postmortem, and write down every question someone asked while it was happening: "Is this only on Android?", "Did it start with the 2.14 release?", "How many customers are we talking about?"

Postmortems are written after the fact and tend to skip the dead ends, which is usually where the missing data shows up.

Then try to answer each one with the data you have today, with a timer running.

Note which ones take two minutes, which take twenty, and which can't be answered at all. The ones you can't answer are your instrumentation backlog, ranked by how often they come up.

It's more useful than auditing your dashboards, because it starts from what you needed when things broke.

Where the tooling budget fits

Cost has been the top criterion for choosing an observability tool three years running, cited by 65% of respondents in Grafana Labs' 2026 Observability Survey. It's a sensible filter to apply once you know which questions the tool has to answer.

No platform can produce a tenant ID that was never logged or a trace that was never kept, however well it stores and visualizes everything else.

That's how an expensive stack and a system that can't answer "who is affected?" end up living together for months: the budget went into storing telemetry before anyone decided which questions it had to answer.

Go back to the stuck orders for a moment. With a customer ID on every event, the version on every span and trace context that crosses the queue, the questions from 3:05 AM (which orders, since when, after what change?) take a query or two, and whoever is on call is back in bed before the support lead in Brazil has to send a second message.