Observability and AI explained

Fri, 31 Jul 2026 09:00:00 +0100

“AI observability” gets thrown around a lot, and it can mean one of two very different things. So in this video I map out the whole landscape of AI-related work at Grafana across the entire software development lifecycle — and show how AI and observability come together in two directions: AI for observability (AI that helps you observe your systems) and observability for AI (observing the AI itself).

The framing is deliberate: AI in observability isn’t about replacing engineers. It’s about handing back the exhausting parts of the job — the 3am triage, the repetitive queries, the report nobody wanted to write — so you can spend your energy on the work only humans can do.

I walk through each phase of the lifecycle and where the tools fit:

Planning — clarity about what’s really going on, with Grafana Assistant: not just a chatbot, but an agent that knows your telemetry, writes your queries, and builds your dashboards.
Building — momentum without losing control: Assistant, the open-source GCX CLI, the Grafana MCP server, the specialized Loki/Tempo/k6 MCP servers, and the AI SDK for building your own agents in Go.
Operating — staying calm under load: Investigations for agentic incident triage, Assistant Automations and the Watcher Agent for staying proactive, plus Grafana Machine Learning (Metrics Forecasting, Outlier Detection) and Adaptive Telemetry.
Testing & evaluating — doubt is a form of care: k6 and agentic testing on the systems side, and Agent Observability plus the open-source o11y-bench benchmark on the “observing your AI” side.
Collaborating — bringing it all to where your team already works: Assistant in Slack (and coming to Teams), on mobile, on the desktop, and via API/MCP, plus Assistant Search across your Grafana artifacts.

If you’re new to the Grafana stack itself, start with my open source observability video for the non-AI foundations that all of this builds on.

Resources

My notes on Grafana Assistant
My notes on GCX
My notes on k6
o11ybench.ai — the open leaderboard for AI agents doing observability work

All About AI Observability (The Context Window #03)

Wed, 22 Apr 2026 18:00:00 +0100

In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and the engineers behind the product, Alexander Sniffin and Jack Gordley, for a deep dive into AI Observability in Grafana Cloud. We unpack what it actually is (a new way to instrument your AI apps and see canonical data about them, sitting alongside your existing telemetry), what an evaluation is in this context (and the difference between online evals on live traffic and offline evals on dataset conversations), and we walk through real demos: setting up AI Observability and evaluators, the analytics view, instrumenting a local coding agent, and Jack’s system prompt analysis tool. We also talk about the things nobody else is talking about — like LLM-as-judge as a method, evaluators that catch the most bugs, and the cheating-LLM phenomenon.

Timestamps

00:00:00 — Introductions
00:02:00 — The last month in AI news
00:11:09 — What is AI Observability in Grafana Cloud?
00:19:20 — What is an evaluation?
00:21:05 — The origin of AI Observability
00:25:09 — Demo: Setting up AI Observability and evaluators
00:32:04 — What is LLM as judge?
00:38:43 — Demo: AI Observability Analytics
00:40:52 — AI O11y is based on OpenTelemetry
00:42:17 — Demo: Instrumenting a local coding agent
00:47:18 — Potential future agentic use cases
00:52:00 — Evaluators that catch the most bugs
01:02:23 — Demo: System prompt analysis
01:05:11 — Guess the prompt

Ai Observability on Nicole van der Hoeven

Observability and AI explained

Resources

All About AI Observability (The Context Window #03)

Timestamps

Resources

News from the episode

AI Observability

Mentioned

Ai Observability on Nicole van der Hoeven

Observability and AI explained

Resources

All About AI Observability (The Context Window #03)

Timestamps

Resources

News from the episode

AI Observability

Mentioned

Related reading