The Context Window on Nicole van der Hoeven

Build Apps on Your Data with Grafana Pages (The Context Window #07)

Thu, 05 Nov 2026 18:00:00 +0000

My favourite announcement out of ObservabilityCON was Grafana Pages — a way to build a live, interactive data app right from your Grafana instance, with just prompts and no code. In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by my co-host Tiffany Clifford and by Stephanie and Tom from Grafana’s AI department, who built Pages (it started as a hackathon project to give Grafana Assistant a proper Linux sandbox, and turned into something much bigger). We dig into what Pages actually is, build one live from a prompt, and get into the stuff I find genuinely delightful about it: a page is still real Grafana — same data sources, same RBAC, same production-ready panels — it’s just no longer you arranging things on a grid. We also cover personal data Pages (my sleep-vs-flight tracker, Stephanie’s StairMaster world tour), Pages vs dashboards, how it all works under the hood (React in a locked-down iframe, served from a micro-VM sandbox), and the current limitations and roadmap.

Fun fact dropped mid-stream: the ObservabilityCON keynote itself was a Grafana Page. So was this presentation.

What Grafana Pages is

As Tom put it, a Page is “all of Grafana, minus the panel grid.” It’s the same data sources, the same RBAC, the same underlying panels and visualization libraries — but instead of you arranging panels on a grid (or Assistant trying to squeeze your intent into what a dashboard can represent), you unlock Assistant and let it do the thing modern models are genuinely good at: writing code and building a website. You keep everything that makes Grafana production- and enterprise-ready, carefully worked out over nine years, and you add interactive, tailored, pretty ways to look at your data.

It builds on the Grafana-managed sandboxes announced in the same release — the same underlying layer of Linux compute for AI agents that also powers auto-remediation and investigations.

Seeing it on Grafana Play

The fastest way to understand Pages is to go poke at it on Grafana Play — no login, no Cloud account needed. Stephanie showed a few:

An observability-survey quiz game in comic-book style, pulling real data already on Play — built from one prompt (“make it higher-lower, comic-book style”).
A quarterly-review page, fully interactive with drill-downs by region and customer sentiment — also one prompt, with “make it look pretty” doing most of the design work.
A teammate’s page that pulls your GitHub contribution data and thanks you — and, because it uses the inline Assistant, renders the thank-you in the viewer’s native language (inferred from internal data).

The point Stephanie and Tom kept coming back to: Pages aren’t static. A page can fetch recent data every time you open it and tell you the most noteworthy events in the last 24 hours — so an on-call summary page genuinely updates itself.

Building a Page from a prompt

We built one live on Play against the demo incident/outage data: “Create a page to summarize what is noisy and needs attention from the on-call engineer — give us the top three things.” Assistant loops through alerts, shows the queries it’s running in a Sources tab (no hand-waving about what it’s doing), takes live screenshots of the rendered page as it edits so it actually knows how it looks, and comes back with dynamic top-three items. Making a page really is just that easy now.

Personal data Pages

This is where I got carried away. I hooked up my two smart-ring data sources (Oura and Ultrahuman — both have API endpoints) and, from one prompt, got a whole page summarizing years of sleep: best month, worst month, seasonal breakdowns, the slightly-depressing reveal that I average just under seven hours. I didn’t ask for the blue-purple theming. I also didn’t ask it to correlate my flights (TripIt + Flighty, ten years of data) with my sleep — it just did, because the flight data was there. A second prompt gave me an animated replay of every flight I’ve taken.

Stephanie’s was my favourite: a StairMaster world tour, ranking gyms she’s visited for work by how good the view is (so she doesn’t have to think about being on a StairMaster), complete with an out-of-order-machine mini-game and motivational quotes from ultrarunners. And a summit-conditions page pulling weather data for different mountains.

Pages vs dashboards

So when do you build a page instead of a dashboard? The honest answer from the team is: it’s early, and Pages already cover many dashboard use cases. Tom’s product-brain answer is “make it a page, period” — but if you have a well-understood dashboard that serves you well, there’s no urgent reason to change it, and dashboards still win where you need a dashboards-as-code story (Pages has no as-code path yet). For me, I reach for a page when I want to tell a story with data — a narrative, some annotation, something I’d present to my manager without drowning her in every metric. Pages is great at filtering down to the readable essentials without deleting anything.

A nice pro tip from Tom: if you ever present real data on a big screen, Pages is for you — you can ask for presentation style and still get real panels backed by real data, instead of screenshotting them out. We’re both using Pages for KubeCon talks.

Remixing, editing, and building from a dashboard

There are several ways to create a page:

Ask Assistant directly in chat.
Create new page with a starter example (service health, minimal, etc.) that pre-seeds the prompt with styling hints.
Remix an existing page — reuses it as a reference and changes style, data sources, or format; saved as its own new page.
Start from a dashboard — turn an existing dashboard into a page (handy when the dashboard is the ongoing stream and you want a “how was last quarter” narrative view).

Editing happens through Assistant too: highlight an element and it’s attached as context (it carries the DOM node / data-page-id), so you can ask “how did you get this data?” or request a change to that specific element. Templates are coming — define a house style once and have everyone in your org generate pages or presentations that match.

Sharing today works a lot like Google Docs/Slides: add specific people, generate a link, or drop it in a folder (folder view/edit permissions carry over). But viewers must be logged into the Grafana instance — there is no fully public “share with the world” option yet, and no share-by-email-domain yet either. Both are open feature requests; the team explicitly asked for feedback on whether people want them.

Crucially, data access still respects user permissions. If you can open a page but don’t have access to its data sources, the page opens but shows no data — exactly like dashboards, because the queries still go through your regular data sources with full authorization and RBAC.

How Pages work: React, sandboxes, and iframes

A dashboard is configuration — a (very large) JSON blob that Grafana lays out into the panel grid at runtime. A Page is a layer lower: what the agent produces is React source code for a website, which Grafana renders inside a heavily locked-down, isolated iframe, piping through only carefully chosen data.

But something has to safely compile that TypeScript/JSX and bundle it — running arbitrary model output is exactly the kind of unsafe operation you don’t want on your main infrastructure. So each conversation gets a thin Linux micro-VM. The hard engineering problem was making it fast: nobody waits two or three minutes for an EC2 to spin up before losing interest. The team got it to start in under 10 seconds, with each command finishing in under two.

Tom clarified that “sandbox” is overloaded here — there are two:

The Linux micro-VM used only while building or editing a page. Once built, the output is just HTML in an S3 bucket — no VM involved anymore.
The iframe isolation layer inside Grafana, so AI-generated website content renders safely without leaking your data. You control what hosts a page can call out to, per page (important — not every page should be as shareable as, say, my health data). MCP server connections are supported too, and any viewer gets asked for permission before anything runs on their behalf.

Limitations and roadmap (no promises)

Pages is in public preview across Grafana Cloud (not OSS yet, and it will always require Assistant to drive the prompt → page step). Known rough edges and wishes, with the usual “nothing here is a commitment”:

Multi-editing: today you don’t know if someone else is editing the same page, so you can clobber each other. A “someone else is editing” warning is coming before GA, with proper multiplayer later. (We hit this building the KubeCon deck.)
Public sharing and share-by-domain: open questions, driven by feedback.
Storage: pages and their remixes naturally form a graph of related code artifacts, so a Git-inspired storage model feels like the next natural step.
Higher environment fidelity, templates (rolling out now), and more examples.

If you want it for OSS — which I personally do — leave a comment so the team can champion it internally.

Resources

Grafana Play — try the public Pages (games included), no login required
Grafana Cloud — sign up free; Pages is in public preview
Grafana Community Slack — there’s a Grafana Pages channel for feedback

My notes on Grafana Assistant

Transcript

Intro, guests, and ObservabilityCON recap

Nicole: Hi everyone, welcome to another episode of The Context Window. Today we’re talking about the hottest thing — in my opinion — to come out of ObservabilityCON: Grafana Pages. It’s a way to build a live, interactive data app right from your Grafana instance, with just prompts and no code. I’m not the only one excited about it, so — Tiffany, do you want to introduce yourself, my awesome co-host?

Tiffany: Hi, I’m Tiffany, a developer advocate. Until literally a couple of days ago I was focused on Tempo, Pyroscope, and AI — and now just AI, so I get to dive into all this really cool stuff. This one’s new to me anyway.

Stephanie: Hello, I’m Stephanie. I work in the AI department with Tom. We did a hackathon together for Pages and have been working on getting it released for ObservabilityCON. That’s been fun. Tom?

Tom: Hi, I’m Tom, also in the AI department — I’ve been at Grafana seven years. For the last couple of months Stephanie and I were toying with the idea of giving Assistant a proper Linux sandbox and a proper way to execute code. We didn’t even set out to build Pages — then we realized we could just let it write React code, and that gives you so much better visualizations and tailored outcomes than dashboards.

Nicole: I’m Nicole van der Hoeven, a developer advocate mainly focused on AI right now. And I wanted to mention ObservabilityCON, which just passed — I wasn’t there in person, but a lot came out of it. The keynote on YouTube is really good.

Tom: Can I tell you a secret? The keynote is a Page. And, sneakily, so is this presentation. The three ObservabilityCON themes were autonomous operations — and look, there’s Grafana Pages right there — cloud economics, and context. We’ll focus on Pages.

What Grafana Pages is

Stephanie: We started by working on sandboxing for coding agents, then in a hackathon realized we could use those sandboxes for Grafana Pages. It lets you see your data in more interactive ways — you can prompt Assistant with “I want to look at my business revenue for the past quarter,” or weather on a mountain, and it creates unique interactive ways to look at data.

Tom: If I had to describe it, it’s all of Grafana minus the panel grid. Same data sources, same RBAC, same underlying panels, same visualization libraries. It’s just no longer you arranging them on a grid, or Assistant trying to fuse your thoughts into what a dashboard can represent — we unlock Assistant and let it write code and build a website. But you’re not giving up the things that make Grafana production- and enterprise-ready, carefully worked out over nine years.

Tiffany: And the sandboxes you mentioned — is that the Grafana-managed sandboxes that also got released?

Tom: Those were coined more towards auto-remediation and investigations, where we use sandboxes to look at your source code and check a hypothesis for why something’s broken. But it’s the very same underlying layer of Linux compute for AI agents that powers Pages too.

Grafana Pages examples on Grafana Play

Nicole: The best way to understand Pages is to dive in on Grafana Play — this playground for all things Grafana. No login, no Cloud account. You can just mess around and see what people have already created.

Stephanie: Here’s an observability survey — we run one every year — turned into a game. It’s pulling in real data that’s already on Play, with little animations, streaks, ten questions. Maybe you think reading the survey is dry; a game could help. And this was one prompt: “make me a higher-lower game, comic-book style.” I’m not a front-end engineer — I did not do this.

Here’s a quarterly-review example, also one prompt, all interactive — drill into what the customer is thinking, different regions, fake sales data. I can’t get over how pretty it is. There’s also a Sources tab so you can see the queries behind the data. A teammate made one that pulls your GitHub data and thanks you for your contributions.

Tom: That one uses the inline Assistant, so the page itself can make model generations — it shows the thank-you in the viewer’s native language, guessed from internal data.

Stephanie: The powerful part is it’s not static. You can make a page that summarizes recent data, and it’s not hard-coding the summary — every time you open it, it first fetches recent data and tells you the most noteworthy events in the last 24 hours. I made an on-call rotation page that surfaces the most urgent things every time I look.

Build a Grafana Page from a prompt

Tom: Since it’s so easy to prompt — can we build one from scratch on Play?

Stephanie: You go to Pages, create a new page, pick from starter examples that give Assistant some styling direction so you don’t need a huge prompt.

Nicole: Let’s say: “Create a page to summarize what is noisy and needs attention from the on-call engineer — give us a summary of the top three things.” It’s so cool that making a page is now just that easy.

Stephanie: It’s looping through alerts, figuring out the alert states, and showing its queries in the Sources tab — so it’s not hand-wavy; you can check it’s actually doing what you asked.

Tom: Depending on what you ask, it can take a bit longer. Let’s leave it and jump to some other things.

Personal data Pages: sleep and flight tracking

Nicole: I have two smart rings — Ultrahuman and Oura. I hooked up both data sources (they have API endpoints), and this whole page was one prompt: how much sleep I get each year, best and worst months, by season. I average just under seven hours. I didn’t ask for the blue-purple theming either.

Then it added a travel page — I realized it’s correlating my flights with my sleep, because I have flight data in here too (TripIt and Flighty, ten years). I didn’t ask it to do that.

Tiffany: I was literally looking into TripIt yesterday — and apparently they closed the API in late September to anyone not already using it. I’m so glad you’ve been using it.

Nicole: This replay — this one I did ask for, so two prompts — animates all my flights, new countries, busiest month. Pages has been useful for more than the usual outages.

Grafana Pages vs dashboards

Nicole: When would you build a page versus a dashboard? Are you all-Pages all the time now?

Tom: The product person in me says “make it a page, period.” Honestly we don’t fully know yet — Pages cover many dashboard use cases. If you have a well-understood dashboard that serves you well, there might be no immediate reason to change it. But in general Pages should replace most dashboard use cases, and if they fall short, that’s a reason to talk to us.

Stephanie: For dashboards, if you need a dashboards-as-code story — running the same dashboard across many clusters — we don’t have Pages-as-code right now, so dashboards still win there.

Tom: Pages will not come to OSS without Assistant — the way it works, you have to prompt Assistant to make it into a page. Assistant has a lot of smarts about correlating Grafana data; a vanilla model wouldn’t have the opinionated, docs-backed best practices.

Tom: Pages are very much a Grafana thing. Grafana has always stood for two things — that it’s pretty, and that it’s serious, as in real data, not made up. We tried hard to preserve both: lots of tuning so a small prompt gives a pretty page, and it still retains inspectability and real data rather than looking like LLM slop. If you just want to post something up, tools like Claude artifacts already exist.

Remix a Page or create one from a dashboard

Stephanie: The main way is prompting. There’s also “make your own,” where you prompt it much more specifically about the look. Templates are coming — define a style for your whole org’s presentations. And you can remix a page: it uses the page as a reference and changes style, data sources, or format depending on what you ask; saved as its own new page.

For editing, you highlight an element and it’s attached as context — the DOM node with its data-page-id — so you can ask how it got the data, or request a change to that specific element. You can also start a page from a dashboard.

Nicole: I tend to build a page when I want to tell a story with the data — some narrative, maybe annotation. My dashboards get cluttered; Pages lets me streamline to what’s readable without deleting anything. Great for explaining something to my manager.

Stephanie: To share, you can add specific people, say “anyone with this link,” or put it in a folder so folder permissions apply. There’s no fully public “share with the world” option today — that’s a feature request; let us know in the comments. And data access respects user permissions: if you can open a page but lack access to its data sources, it opens but shows no data, just like dashboards.

Tom: It’s best described as the Google Docs/Slides model — share with specific people or generate a link — but viewers must be logged into the Grafana instance. There’s no public option yet.

How Pages work: React, sandboxes, and iframes

Tom: A dashboard is configuration — a big JSON blob Grafana lays out into the panel grid at runtime. A Page is lower-level: what the agent produces is React source code for a website, which we render in a heavily locked-down, isolated iframe, piping through carefully chosen data.

Something has to safely compile that JSX and bundle it, which is unsafe to do with arbitrary model output — so each conversation gets a thin micro-VM. Making that scalable took most of the platform time: you don’t want to wait two or three minutes for an EC2. We got it to start in under 10 seconds, every command finishing in under two.

The term “sandbox” is overloaded. There are two: the Linux micro-VM used while building or editing a page — once built, it’s just HTML in an S3 bucket, no VM anymore — and the iframe isolation layer inside Grafana, so AI-generated content renders safely without leaking your data.

Stephanie: There’s a setting for what a page can call out to, and you can control it per page — some things, like my health data, shouldn’t be broadly available. MCP server connections are supported too, and viewers are asked for permission before anything runs on their behalf.

Stephanie: Data sources still use the user’s permissions — if you don’t have access, it won’t show data. It builds on battle-tested, enterprise-ready foundations, so authorization and RBAC carry forward.

Limitations, roadmap, and more Page examples

Nicole: Beyond sharing, what other limitations should people be aware of?

Stephanie: Multi-editing is difficult — right now you don’t know if someone else is editing a page, so you can overwrite their work. Until GA, just tell people if you’re editing. We’ll have a “someone else is editing” warning before GA and proper multi-edit later. Templates are rolling out now.

Tom: Pages and remixed pages form a graph of relationships and are code artifacts, so some Git-inspired storage system feels like the natural next step — but it needs careful thinking.

Nicole: I have an unnecessary setup where Alloy on my laptop sends metrics about my Obsidian notes into Loki and Prometheus — and I have a page for all of it: top topic this month, a GitHub-style contribution heat map, a graph view. I didn’t ask for any of the styling; I just said “here’s some data, build me a page.”

Stephanie: My favourite is my StairMaster world tour — I do mountaineering, so as I travel for work I find StairMasters with pretty views. One prompt ranked the gyms by view, added a mini-game for out-of-order machines, and pulled motivational quotes from ultrarunners. I also made one for summit weather conditions on different mountains.

Nicole: There’s a Grafana Play page on Singapore weather and haze that’s beautiful — it shows wind direction and air quality across regions, with a 72-hour replay. I have family in Singapore, so I care about the air quality. Amazing stuff.

If you want to try this, sign up for a free Grafana Cloud account — Pages is in public preview, nothing to contact anyone about — or just play with the ones on Grafana Play. And please give feedback: there’s a Grafana Pages channel in the community Slack and a Reddit post. Comment on the video too, especially if you want it for OSS so we can champion it internally.

Nicole: Thank you all for joining — this was a fun one. The next Context Window livestream is later this month with William Dumont, about Assistant Watchers. Stay tuned.

Tom: Thanks for building this amazing tool — it’s my favourite feature on Grafana now.

Context Engineering Fundamentals (The Context Window #06)

Fri, 02 Oct 2026 18:00:00 +0100

What actually counts as “context” for an AI agent — and how does it find the right things to put in a limited window? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I dig into context engineering fundamentals with Ben Sully and Ioanna Armouti. We treat the context window as an attention budget, walk through Assistant Search and semantic retrieval, embeddings, and how you actually evaluate whether the retrieved context was any good.

What we cover

What counts as context — and why the context window behaves more like an attention budget than just a size limit.
Finding the right context — Assistant Search, semantic retrieval, and embeddings: how an agent surfaces what it needs instead of being handed everything.
Evaluating retrieved context — because retrieval that looks relevant isn’t the same as retrieval that is.
The tool tax and tool deferral — every tool definition costs context, so when do you load it versus defer it?
Progressive disclosure and long-term memory — giving the agent what it needs, when it needs it.
Why removing 80% of a system prompt can change nothing — a nice illustration that more context is not automatically better context.

Resources

An Overview of AI Evaluation (The Context Window #05)

Fri, 26 Jun 2026 18:00:00 +0100

How do you know if an AI agent is actually any good — and whether the change you just made to it made it better or worse? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Yas Ekinci, an engineer on Grafana’s AI team who has spent the last several months focused almost entirely on evaluation. We talk about why trusting AI is so hard (non-deterministic models, oversized PRs, and the shift from writing code to reviewing it), the difference between online and offline evals, how we actually test Grafana Assistant, and o11y-bench, the open observability benchmark we announced at GrafanaCON. We recorded this one ahead of Yas’s two-month leave — so consider it a deep dive banked for the road.

The “final answer problem”

A recurring theme — and one close to my testing heart — is what I call the final answer problem. As a tester I’m used to looking at an outcome and asking “did this pass or fail?” But with AI, a plausible-looking answer isn’t enough. If the process was wrong, you can’t trust the outcome even when it looks right — and a plausible-but-wrong answer is the dangerous case, because nothing on the surface tells you it’s wrong. Yas’s example: a broken Tempo tool that returned no spans, while the agent still answered convincingly by guessing from metadata. Plausible ≠ correct.

This is also why the bottleneck has moved. The cost of writing code has collapsed, so the burden now sits on review. Developers, as I like to put it, are becoming shepherds of intent — less about typing the code, more about making sure the intent is clear and faithfully implemented.

What evals are, and online vs offline

An eval is simply a test of whether an AI agent behaves the way you want — from “who are you?” → “I’m Grafana Assistant” all the way up to multi-step querying and dashboarding. The point is to know with confidence whether a change made the agent better, which is what lets you improve it consistently over time.

There are two complementary flavours, and they map neatly onto testing concepts we already have:

Online evals — scoring a sample of real production conversations, usually with an LLM-as-a-judge. Great as a pulse check (it’ll catch a quality drop when a model provider silently swaps models), but it only tests the deployed variant and is hard to verify. Closest analogy: traditional observability / RUM.
Offline evals — a “golden set” of tasks run during development and CI against a controlled, reproducible environment. You can test every change, prompt, and implementation, and drill into specific skills (PromQL vs LogQL, dashboarding). Closest analogy: integration testing.

How we test Grafana Assistant

Assistant evals run a real agent-plus-tools harness against a real-ish environment (a Grafana with data sources, alerts, and dashboards), then grade the full transcript with “graders” or “verifiers.” The hard tension is that the environment has to be real and reproducible at the same time. Because Assistant is a front-end plugin that runs some tools in the browser, the team uses Playwright to replicate real browser behaviour — which is where I made my pitch for k6’s browser module (now approaching Playwright parity) as a dogfooding option. The internal eval tool is a CLI called LLM Spec, now on its third major iteration after the lessons from o11y-bench.

o11y-bench

o11y-bench is our public benchmark, announced at GrafanaCON, with six task categories: dashboarding, investigation, logs, metrics, and traces. A few things I think make it genuinely interesting:

It’s built on the Harbor framework (also used by terminal-bench), which separates harness from environment from tasks — so you can swap harnesses and compare, say, gcx + Claude Code against the Grafana MCP.
The leaderboard includes open models (Qwen 3.6, Kimi), not just the frontier labs.
It distinguishes pass@3 (got it right at least once in three runs — rough capability) from pass^3 (got it right all three times — reliability). The 10–20% gaps between the two are the whole reliability story in a single number.
It runs against a synthetic environment (Prometheus, Loki, Tempo in Docker) with generated data, so the correct answer is known in advance — and it uses fact-based rubrics, where the grader runs its own known-good queries against the environment and checks the agent’s answer against them. That’s how you catch the subtle, plausible-but-wrong failures.

The internal eval set is a superset that adds scenarios drawn from real (private) conversations and Assistant-specific behaviour — like Slack’s not-quite-markdown formatting — while o11y-bench stays generic and harness-independent, grading on outcome rather than method.

The evaluation loop

Yas’s blog post (the first engineering post on Grafana’s new community Unprompted blog) and our GrafanaCON talk both center on the evaluation loop: measure → make changes (new tools, prompt tweaks, bug fixes) → measure again. The crucial bit is that the loop can be driven by a coding agent — via an LLM Spec Improve skill — because you can’t unit-test natural-language “programming” the way you test code. There’s no clean relationship between the instructions you write and what the model does, so the only way to know the impact of a change is to run the loop.

Where it’s going

We closed on open questions and wishes: higher environment fidelity, more and harder tasks (very likely an o11y-bench 2.0), and getting more contributors — including PMs writing evals to capture desired product behaviour. I floated a k6-extension-style community model for evaluators, and using o11y-bench to support bring-your-own-model, so users can see exactly where a smaller or local model falls short instead of assuming Assistant is broken. Yas’s industry hope is the neat one: if labs tune their models against o11y-bench, observability and Grafana performance improve across the board — free fine-tuning out of the box.

Resources

Transcript

Nicole: Hey everyone. Welcome to another episode of The Context Window. This is a semi-regular podcast — we try to make it every month — and it’s all about the things going on with AI at Grafana. Usually this is live, and you might think looking at this that it’s live, but it’s actually prerecorded, because Yas has a busy schedule ahead and I wanted to get him on here before he left. So hi, I’m Nicole van der Hoeven, a senior developer advocate at Grafana Labs, and I’m assigned to the AI team these days. And one of the people on the AI team is — this person over here. How do you say your last name, actually?

Yas: Oh, it’s Ekinci.

Nicole: Ekinci. I was totally confused because I just found out you speak Dutch. Did not know that.

Yas: I’m based in Belgium, and I was like, “Nicole van der Hoeven” — okay, that sounds familiar enough to me.

Nicole: Yeah, it’s the name of my ex-husband, but it’s okay, we still like him. So tell us what you do on the AI team.

Yas: I’m an engineer on the AI team — I’m getting to five years at Grafana next week. When I joined, this was pre-AI; we had the machine learning team building predictive models, forecasting, that kind of stuff. As the AI wave came, we all got absorbed into this newer, larger AI department. It’s been quite varied — the assistant, the AI agent inside Grafana — but especially the latter half of this year I’ve been more and more focused specifically on evals: how do we figure out the quality and reliability of these things we’re deploying?

AI is a very useful tool. It has utility — it’s not like NFTs. We use it day to day. My job has changed drastically over the last two years. But you still notice it needs guidance and sometimes has reliability issues. The trend is that the guidance has come down — a year or two ago I’d steer an agent 50, 60, 70 times in a session; now it’s more like five or ten. But you still need to do it. If you just blindly vibe-code your way out of this, you’re not going to end up in a good place.

Nicole: It moves so quickly. Whenever someone tells me they tried AI and it didn’t do what they wanted, I ask, “Well, when was that?” Because if it was six months ago — try it again. In this episode we’ll talk all about evals, but first, some announcements. Assistant is now available for OSS and self-hosted Grafana, and I made a video showing how to do that. There’s also a new Medium blog for Grafana Labs called Unprompted — a community-focused blog — and Yas wrote the first engineering post, about building an evaluation loop. And we’ve started AI weekly videos: the AI team has a weekly meeting, Mat Ryer had the idea to publish excerpts, usually the demo part. Just be warned we demo things that may never make it to production — you’re getting it raw, warts and all.

Yas: I need to make sure I don’t wear my PJs anymore for the AI weekly.

Nicole: A bit of quick AI news: Warp is now open source. There are new models — GPT-5.5, Gemini 3.1 Ultra with a two-million-token context window, Anthropic’s Claude Mythos security preview. And there’s a great article by Matt Shumer called “Something Big Is Happening,” where he says, “I am no longer needed for the actual technical work of my job” — which hits hard — and advises leaning into what’s hardest to replace.

Yas: It’s still that guidance thing — you need a human in the loop to guide it. But if you ask me how much code I’ve manually written lately, the actual technical piece is very close to zero. We’ll talk about how that’s different for evals, because evals don’t work as well as coding does there.

Nicole: So why is it so hard to trust AI? Why don’t we just YOLO it and accept everything?

Yas: A mix of things. Everybody knows these models are non-deterministic and not perfect — these aren’t new problems; underneath it’s all machine learning, numbers and math and stats. Every time you talk to it you might get a slightly different response. The second part is it writes a lot of code. You can 10x your output — send ten agents at ten different features and launch all those PRs — but then someone needs to review them, and the PRs are typically big and over-engineered. One thing I always do at the end of a piece of work is ask it to find a simpler way and simplify. Bigger PRs plus so many changes makes it hard to keep a mental model of the whole thing. And interestingly, I’ll use AI to help review a PR — which gets you back to the same problem: if you use AI to review the AI, how much can you trust that?

Nicole: I like the phrase that developers are becoming shepherds of intent — we’re not writing the code so much as making sure the intent is clear and implemented. And this is the difference between process and outcome — what I call the final answer problem. I come from performance testing; I’m used to looking at the outcome and asking did it pass or fail. But now, even if the outcome looks right, if the process was wrong I don’t trust it. And the reverse too: right process, wrong outcome, something else is broken.

Yas: Tests are interesting here, because most of those tests are also AI-generated — and the AI doesn’t shy away from just changing the test to make its branch pass. For any branch I review, I still look at all the changes; I won’t blindly rubber-stamp. That’s probably the biggest bottleneck right now.

Nicole: One approach to trusting AI is human evaluation — thumbs up/thumbs down, or something like LMArena where you compare two answers blind. Useful, but not scalable; you can’t run it in CI.

Yas: On the eval side they’re all complementary. So, what is an eval? It’s testing whether the AI agent behaves like you want. A simple one: ask “who are you?” and expect “I’m Grafana Assistant.” There are much more complicated ones around querying and dashboarding. At its core, an eval is how you figure out if the agent is any good — and the reason we do it is so that when we make changes, we know if it’s better. Answer that with confidence and you have a path to improve consistently over time.

We split into online and offline. Online — we recently launched AI observability for agents — is about measuring how the deployed agent does against real production conversations from real users. Valuable, because you get a score based on real behaviour, but hard to verify; we typically use an LLM as a judge, so it’s not as robust. Offline is used more during development, in CI — much more controlled. You have a golden set of evals covering the behaviour you expect in production, and because you always test the same set, you can see whether changes make it better, and dig into PromQL vs LogQL, or dashboarding. Online is a great pulse check — it can catch quality dropping when Anthropic swaps a model — but it only tests the deployed variant. Offline can test all your changes, your implementation, and all your prompts.

Nicole: It took me a while to translate these, because the testing industry already has this — we call it pre-prod and production testing. Pre-prod is everything from unit to system integration testing in dev and staging; you can really hammer the thing. Production you have to be more careful — synthetic monitoring, real user monitoring — same idea with an AI flavour.

Yas: Definitely. Online is much more like traditional observability — you’re observing a system that happens to be powered by AI, turning natural language into a number you track over time and alert on. Offline is closer to an integration test: you give it a task, it runs the whole agent against a real environment, you get something out, and then separately you grade that outcome.

Nicole: We did an AI observability episode without you — you were at GrafanaCON — so I’ll link that. Let’s talk more about offline evals, since we haven’t covered those.

Yas: Offline is a bit of a beast. There’s a good Anthropic post called “Demystifying evals.” Back when we had simple single-turn LLM calls — incident summary was one of the first — the prompt was “here’s the context of the incident, summarize this,” and grading checked readability, completeness, appropriate length. Easy enough to set up. Agents get more complicated, because every time you run a task the agent can take very different trajectories — five steps one time, seven the next. The agent plus its tools is roughly the harness; it runs against an environment — a real Grafana with data sources, a certain alert and dashboard state — and keeps interacting using the tools until it’s done. Then we take the whole transcript and grade it with graders, sometimes called verifiers, to verify it actually got the task right.

Nicole: So how do you do it for Assistant, given how broad that environment is?

Yas: We have the public benchmark we released recently and our internal evals — same approach, different number of tasks and some different data sources. We want an environment as real as possible, but with the tension that we also need to control it so a benchmark is reproducible. If two people run it and get different scores, which one was right? We started, like everybody, with manual tests — listing data sources to confirm a tool is wired up, some CPU questions, showing logs — with a human as the judge. That’s fine at four or five scenarios; you can’t do it as a human at 500. Assistant is a front-end plugin — some tools run in your browser, which is good for auth and security but makes testing hard — so we use Playwright to replicate real behaviour and pull the telemetry out. The internal eval tool is a CLI we call LLM Spec, and it can run against cloud instances or a local controlled environment.

Nicole: Before the environment — did you consider using k6 instead of Playwright?

Yas: Interesting — yes and no. k6 is also a tool the assistant needs to use, but our goal isn’t to load test; it’s to hook into the plugin and get the messages and telemetry out, so we have a Playwright harness.

Nicole: There’s actually a browser module for k6 going towards feature parity with Playwright.

Yas: I did not know that. Might be interesting.

Nicole: Just dogfood all the things. I’d love to work with you on that. Okay, the environment.

Yas: We test against cloud instances — for incidents especially, it’s a lot of data. And we run LLM Spec against a local controlled environment, which is quite similar to the open benchmark we published.

Nicole: You’ve mentioned o11y-bench — the benchmark announced at the last GrafanaCON. That was fast.

Yas: o11y-bench has tasks across six categories — dashboarding, investigation, logs, metrics, traces. We wanted to bring our internal approaches outward, and it was a learning opportunity: we used a framework called Harbor, also used by the makers of terminal-bench. The nice thing is it separates harness from environment from tasks — so the default harness is a simple system prompt plus the Grafana MCP, but the folks working on GCX run Claude Code plus GCX against o11y-bench and compare GCX with MCP. There’s a leaderboard with a bunch of models, including open ones like Qwen 3.6 and Kimi.

Nicole: They’re getting good. Qwen 3.6 is awesome — not top five, but for an open-weight model, impressive.

Yas: And this is the interesting part: sort on pass@3 and it looks different from pass^3. pass@3 runs each task three times and passes if it gets it right one out of three — rough capability. pass^3 means you need three out of three on every task — reliably solving it, not just getting lucky once. You see 10–20% differences between the two scores.

Nicole: That’s awesome. And this is public, right?

Yas: Yep — the repo’s on GitHub with all the tasks. We run o11y-bench against a synthetic environment — a Prometheus, Loki, and Tempo in Docker — and we generate the data, so every run uses the same data and we know what the answer should be before we even run it. We also do fact-based rubrics: separately from what the agent does, the grading side runs its own known-good queries against the environment and checks whether the agent’s answer matches. That matters because these agents fail in subtle ways. We once made a change to the Tempo tool that broke the spans — the agent got no spans but still answered, fairly realistically, from metadata. Without testing for it you’d never catch it; the text looked plausible.

Nicole: That’s exactly process versus outcome. If the answer were negative two seconds it’s obviously wrong, but a plausible answer is the dangerous one.

Yas: Exactly, and that happens a lot.

Nicole: An early Assistant once told me it created a dashboard, and it didn’t look right — turned out it couldn’t find the data source so it just used dummy data.

Yas: Same thing happened to me locally — I broke all the tools, and it still came up with tool calls it couldn’t execute because of rough instructions in the system prompt.

Nicole: So what’s missing in o11y-bench that means you still need an internal one?

Yas: Two levels. First, some scenarios are based on real conversations or our own internal environments — anything involving private data can’t go public. Second, there are very Assistant-specific things we want to test — Slack, for instance, has its own version of markdown that looks like markdown but isn’t, and early Slack Assistant struggled to write it consistently, so we wrote tests for that. The internal set is a superset; the public set is still part of it. The public set is generic observability tasks — asking questions about your data, dashboarding, basic Grafana API checks — and because the harness is independent, we can test GCX, MCP, our assistant, or any other agent against it, grading on the outcome rather than how it was solved.

Nicole: We talked about your blog post — let’s define the evaluation loop.

Yas: At its core: measure the current state, see where it works and where it doesn’t, make changes — a new tool, tool instructions, the system prompt, bug fixes — then measure again, and again. The key is that loop can be driven by a coding agent, not us manually. If your evals are robust enough to pass a clean feedback signal back, the agent can keep improving until it hits 100%, and then you make harder tasks or increase coverage. At scale it’s nearly impossible for one human to understand the whole system and make targeted improvements, because an agent isn’t programmed as code — it’s programmed with natural language, and there’s no clear relationship between the instructions and what the model does. The only way to probe that is to run a bunch of these loops.

Nicole: Is the loop equally implemented for o11y-bench and LLM Spec?

Yas: It’s implemented internally. o11y-bench is really just the measurement part — we’ve used it for GCX and MCP, but more manually. In LLM Spec we have an LLM Spec Improve skill — you pick the agent, and it runs measure-improve-measure-improve.

Nicole: Where are you going next — no promises?

Yas: I’ve done a lot on the harnesses — the Playwright stuff, and there was an LLM Spec 1.0, then a 2.0 revamp for speed and robustness, then o11y-bench, and the learnings from that went into what I’d call LLM Spec 3.0. The two big challenges now are environment fidelity — more an issue for internal evals — and the number and quality of tasks: how do we cover more of Grafana? We weren’t sure o11y-bench would be a recurring thing, but there’s a lot of interest, so we’ll probably do an o11y-bench 2.0. And I want more people contributing — I was talking at the PM all-hands about getting product managers to write evals, because the tasks capture desired behaviour: when someone asks about my product, how should the agent ideally respond?

Nicole: What I’d love is something like the k6 extension model applied to evaluators. When Grafana acquired k6 — I was on k6 — k6 had extensions and modules, and we got a lot of cool community extensions. I think evaluators would be the same. After my Madrid talk, all the questions about the evaluator section were “can we add this, can we add that, why isn’t this in the template?” — and it could be.

Yas: It would be nice to do that across both o11y-bench and the internal ones — there are open questions on making them work together; internally we’re in Go, o11y-bench is on Harbor in Python, but the tasks and most verifiers are language-independent by design. A fourth thing is getting to a better golden set over time — we haven’t yet automated going from real AI-observability conversations to verifiable offline tasks. As the product changes, your tasks might not stay relevant — imagine k6 didn’t exist and now people ask Assistant about k6. Bridging online and offline in a more automated way would be the fourth big thing.

Nicole: I also have a wish for o11y-bench beyond being a benchmark — with no internal knowledge and no promises: it could be a step toward making Assistant bring-your-own-model. One barrier is that people try Assistant with a tiny local model and conclude it doesn’t work, when really you need certain models. A standardized way to measure that and show users where their model falls short — versus where the latest Opus sits on the leaderboard — lets them make an informed decision.

Yas: I’d go beyond Assistant — we’ve got GCX and MCP too. If you want to use your own models with them, the benchmark should reflect, for observability tasks, which are the best models and the best open models. It gets to your point: people can make a more informed decision.

Nicole: Where are we still missing things as an industry, not just as a company?

Yas: Just recently a new benchmark called DeepSWE was released, and it’s a bit controversial because it flips the ranking from the most popular software-engineering benchmark — where Opus was always leading, here GPT-5.5 leads by a margin. So there’s something to be said about the reliability of the benchmarks themselves, since labs test and sometimes tune their models against them. That might actually be good for o11y-bench — I’d want a lab to tune against it, so we get free fine-tuning out of the box: as newer models use o11y-bench to tune on observability tasks, they get better at using Grafana and solving observability problems.

Nicole: Thank you — we just hit time. It was really informative, and I’m glad we both found time; this got rescheduled for various reasons. A great overview of all the evals we’re doing.

Yas: Thanks — it’s nice to step out of the day-to-day and talk about this stuff.

Nicole: And if you’re watching, leave any eval questions in the comments and I’ll bug Yas to answer them. Thanks everyone for watching.

Yas: Bye.

CLI and MCP Basics for AI Agents (The Context Window #04)

Wed, 03 Jun 2026 18:00:00 +0100

MCP vs CLI: which one should your AI agent actually use? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and three of the engineers and PMs behind Grafana’s agent tooling: Ben Sully (who wrote the Grafana MCP server), and Ward Bekker and Dafydd Thomas from the team building gcx. We get into the two main ways an AI agent can reach the outside world — the Model Context Protocol (MCP) and good old command-line interfaces — and look at when each one makes sense, the tradeoffs around context windows, tool sprawl, token cost, and reliability. We walk through Grafana’s cloud and open-source MCP servers along with gcx, the Grafana Cloud CLI built for agents, do a live side-by-side demo of both creating an SLO, and talk about how we benchmark them against each other with o11y-bench.

Timestamps

00:00:00 — Introductions
00:02:49 — The last month in AI news
00:10:19 — What is Grafana Assistant?
00:12:30 — What does it mean for Assistant to use tools?
00:13:30 — What is MCP?
00:16:56 — What is a CLI in an AI context?
00:19:17 — What are skills?
00:24:37 — Can you use a custom MCP server with Grafana Assistant?
00:25:55 — Why team MCP?
00:28:22 — The context-window cost of MCP tool definitions
00:31:50 — Tool discoverability
00:39:25 — What is gcx, and why a CLI?
00:46:13 — Demo: creating an SLO with MCP vs gcx
00:52:58 — Benchmarking MCP vs gcx with o11y-bench
00:58:10 — How MCP and gcx relate to Grafana Assistant
01:02:35 — k6 MCP vs Grafana MCP
01:03:00 — When to use which (decision guide)
01:04:29 — Adding the OSS Grafana MCP to Assistant
01:05:41 — Turtles all the way down

Resources

MCP and CLI for agents

My Context Horizon piece: Unlearnings from building Grafana Assistant

All About AI Observability (The Context Window #03)

Wed, 22 Apr 2026 18:00:00 +0100

In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and the engineers behind the product, Alexander Sniffin and Jack Gordley, for a deep dive into AI Observability in Grafana Cloud. We unpack what it actually is (a new way to instrument your AI apps and see canonical data about them, sitting alongside your existing telemetry), what an evaluation is in this context (and the difference between online evals on live traffic and offline evals on dataset conversations), and we walk through real demos: setting up AI Observability and evaluators, the analytics view, instrumenting a local coding agent, and Jack’s system prompt analysis tool. We also talk about the things nobody else is talking about — like LLM-as-judge as a method, evaluators that catch the most bugs, and the cheating-LLM phenomenon.

Timestamps

00:00:00 — Introductions
00:02:00 — The last month in AI news
00:11:09 — What is AI Observability in Grafana Cloud?
00:19:20 — What is an evaluation?
00:21:05 — The origin of AI Observability
00:25:09 — Demo: Setting up AI Observability and evaluators
00:32:04 — What is LLM as judge?
00:38:43 — Demo: AI Observability Analytics
00:40:52 — AI O11y is based on OpenTelemetry
00:42:17 — Demo: Instrumenting a local coding agent
00:47:18 — Potential future agentic use cases
00:52:00 — Evaluators that catch the most bugs
01:02:23 — Demo: System prompt analysis
01:05:11 — Guess the prompt

The Context Window on Nicole van der Hoeven

Build Apps on Your Data with Grafana Pages (The Context Window #07)

What Grafana Pages is

Seeing it on Grafana Play

Building a Page from a prompt

Personal data Pages

Pages vs dashboards

Remixing, editing, and building from a dashboard

Sharing and access control

How Pages work: React, sandboxes, and iframes

Limitations and roadmap (no promises)

Resources

Related reading

More from The Context Window

Transcript

Intro, guests, and ObservabilityCON recap

What Grafana Pages is

Grafana Pages examples on Grafana Play

Build a Grafana Page from a prompt

Personal data Pages: sleep and flight tracking

Grafana Pages vs dashboards

Remix a Page or create one from a dashboard

Share Pages and control access

How Pages work: React, sandboxes, and iframes

Limitations, roadmap, and more Page examples

Context Engineering Fundamentals (The Context Window #06)

What we cover

Resources

An Overview of AI Evaluation (The Context Window #05)

The “final answer problem”

What evals are, and online vs offline

How we test Grafana Assistant

o11y-bench

The evaluation loop

Where it’s going

Resources

Related reading

More from The Context Window

Transcript

CLI and MCP Basics for AI Agents (The Context Window #04)

Timestamps

Resources

MCP and CLI for agents

Related reading

More from The Context Window

All About AI Observability (The Context Window #03)

Timestamps

Resources

News from the episode

AI Observability

Mentioned

Related reading