While the industry argues over what to call the modern observability stack — observability 2.0 or already 3.0 — three concrete shifts hide behind the terminology. Instrumentation moved out of the code and into the kernel (eBPF). Telemetry standardized around OpenTelemetry. And root cause analysis started migrating from the human to the machine. Each next layer only pays off if the one below it is in order — and the order of investment is exactly what teams get wrong most often.
Monitoring answers «is it working». Observability answers «why not»
Monitoring answers «is the system up and how fast is it». Observability answers «why is it slow and where exactly». In Kubernetes, cross-component incidents cannot be closed without the second one: the symptom «latency went up» cannot be linked to the cause «a node hit MemoryPressure, the kubelet evicted pods, the service endpoints flapped» if all you see is application metrics.
That is why the classic three pillars — metrics, logs, traces — gain a fourth signal in Kubernetes: cluster events and workload metadata. OOMKills, evictions, HPA decisions, scheduler failures, admission rejections — this is the orchestrator's decision log, and without it incident analysis degrades into «we rebooted it and it worked».
Instrumentation moved into the kernel
For years the entry price of distributed tracing was steep: an SDK in every service, a release, a regression cycle. eBPF removed that barrier. OBI — OpenTelemetry eBPF Instrumentation — deploys as a DaemonSet and gives the cluster a baseline within a day, without a single code change: RED metrics, HTTP and gRPC traces, service-to-service flows, database activity, and — an unexpected bonus — TLS traffic visibility without decryption, because interception happens at the syscall level, before encryption. The requirements are modest: Linux kernel 5.8+ and BPF privileges.
eBPF has a ceiling: it sees the network and syscalls, but not business context — which order_id, which tenant. Hence the working rule «Observe first. Instrument later.»: start with an eBPF baseline across the whole cluster, add infrastructure-level auto-instrumentation for the polyglot fleet, and reserve the manual SDK with custom attributes for critical, revenue-impacting paths only. Instrumentation stops being a developer concern and becomes a platform concern.
OpenTelemetry is a contract, not a tool
OTel is a standard: unified SDKs, semantic conventions and a wire protocol. Instrument the application once — export anywhere, from Tempo to a commercial APM, with no lock-in. Semantic conventions (service.name, k8s.pod.name, http.status_code) give all signals a shared vocabulary — and a shared vocabulary plus an end-to-end trace_id is what turns three storage backends into one system.
A key point for brownfield: OTel does not require throwing away your existing Prometheus or VictoriaMetrics. The prometheusremotewrite exporter writes metrics into the same TSDB; scraping stays for legacy targets, the collector is added for new ones. The migration is evolution, not revolution.
The AI layer: the LLM never sees a metric first
The temptation of «send the metrics to GPT and it will find the anomalies» falls apart on real data: too slow, too expensive, non-deterministic. The production pattern is different — filter cheap, escalate smart. Cheap detectors absorb 95–99% of events before any AI: Isolation Forest on metrics (inference takes microseconds), Drain3 template mining on logs, frequency baselines. The LLM receives only the escalated remainder — the «unknown» and the «spikes» — and does what it is good at: reasoning over messy context, linking the signal to recent deploys, drafting the postmortem. This is a division of labor, not a competition: ML does numerical thresholding at scale, the LLM does the reasoning.
The second pillar of the AI layer is change correlation: 70–80% of production incidents are change-related. Deploy markers from ArgoCD, config changes, IAM mutations from the audit log — a separate event stream next to the telemetry. When an anomaly hits, the machine's first question is the same as an experienced SRE's: «what was deployed in the last hour?»
The industry numbers are sobering: leaders cut MTTR by 40–70%, yet only 4% of organizations have fully operationalized AI, and half are stuck in pilots. The paradox: observability budgets grew by 70% while manual toil grew by 30%. The reason — AI bolted on top of a fragmented stack without a unified context layer works through a keyhole instead of seeing the picture.
The order of investment
The layers pay off strictly bottom-up. First, a unified signal layer: OTel, an end-to-end trace_id, the fourth signal from cluster events. Then, coverage without code: an eBPF baseline and platform-provided auto-instrumentation. Next, cheap ML filters and a change-event stream. And only on that foundation — an LLM for root cause. AI in observability is the roof, not the foundation: starting from the roof means paying tokens to scan ten thousand metrics — a job a fifty-line model handles.