Ai on Nicole van der Hoeven

The Defiant Jazz Incident

Fri, 30 Oct 2026 00:00:00 +0000

The Defiant Jazz Incident is a murder mystery you solve with observability tools. It’s set on the Sleep Floor at Kipsake, a company that rents your sleep: at night, its Dreamers sleep in pods while their sleepwalking selves work the floor. During the Cocoa Hour, the floor’s middle-of-the-night holiday party, the power goes out for five minutes. When it comes back, the night supervisor is dead in the elevator.

Kipsake recorded everything it could: badges, doors, chat, sleep rings, power, microphones, cameras, sleep bands, and personnel records. Your job is to work out who did it, and the only way to do that is through the telemetry.

How it works

The case file is a set of logs, metrics, and traces. A loader sends them to Grafana Loki, Prometheus or Mimir, and Grafana Tempo, either in Grafana Cloud or in a local Docker stack, and you investigate from there with whatever you like: Grafana itself, Grafana Assistant, an MCP server, gcx, or the AI agent of your choice. The data is deliberately messy. Some devices lose clock sync during the outage, some records arrive late, and not everyone on the Sleep Floor remembers what their body did.

The game itself comes in two forms. A terminal version is for demos and playtests: you watch the testimonies and choose what to investigate, while an AI that only sees the telemetry answers your questions. A web version is made to embed in a Grafana dashboard panel right next to the data.

Why a murder mystery?

Investigating an incident is investigating an incident, whether the victim is a production service or a night supervisor. You start with a vague report, a pile of signals that don’t quite agree, and a clock that can’t be fully trusted, and you have to build a timeline you can defend. A case with a known answer is a safe way to practice that, to compare how different tools and AI agents cope with it, and to see how good a question you have to ask before the data answers.

No spoilers

The repository keeps the solution in clearly marked files, and the README tells you which ones to avoid if you want to play. This page deliberately says nothing about who did it.

Try it

The repository has everything: the case brief, the telemetry, the loader, and both versions of the game.

Asimov

Wed, 02 Apr 2025 00:00:00 +0000

Asimov is the companion project to my talk Asimov’s Zeroth Law of Robotics. It’s an AI app built to be observed and tested: The Silent Enterprise, a single-player Star Trek text adventure. You play Data, and Claude is the game master. Every model call, dice roll, and turn shows up as telemetry in a Grafana that you can run on your own machine or in Grafana Cloud, and a k6 test suite, some of it judged by Claude, checks whether the AI follows the rules.

The game master has some known flaws, kept on purpose. It sometimes narrates a rescue that didn’t happen, or “destroys” a drone that’s still standing. The point of the demo is catching those flaws with telemetry, tests, and online evaluators.

The idea

Isaac Asimov’s three Laws of Robotics govern how robots behave. The talk argues for a law that comes before all of them: a robot must be observable, and, in its newest version, testable. AI systems fail differently from traditional software. There’s no single correct output to assert against, a response can be fluent and completely wrong, and models drift. You can’t test what you can’t see, and you can’t trust what you can’t test.

What’s in it

The game. A Go program with a command-line interface and an HTTP API. The game engine owns the rules and the state. Claude interprets the player’s input, makes the game master’s own rolls through a tool, and narrates, so a narration that contradicts the saved state is a defect that can be caught.
Telemetry. Traces, metrics, and logs go over OpenTelemetry to a local Grafana stack (Tempo, Prometheus, and Loki, with the game’s dashboard already loaded) or to Grafana Cloud. Generations can also go to Grafana Cloud’s Agent Observability, where online evaluators judge a sample of them.
Tests. A k6 suite with code checks, LLM judges, dice-trajectory evals, and whole playthroughs. All of it can run for free against a mock game master and judge.
No account needed to start. You can play offline with no API key, and see the local monitoring stack with one command and Docker.

The screenshot above is the game’s local Grafana dashboard.

The talks

I’ve given the talk, in several versions, at KubeCon Europe 2025, Dutch Cloud Native Day 2025, Newcrafts 2025, ExpoQA Madrid 2026, and HUSTEF 2026. The write-ups are linked below. The earlier talks used a Python and Flask version of the app, which has since been rewritten in Go.

Try it

The repository has a quick start for playing offline, a Docker-only path, and release binaries for Windows, macOS, and Linux.

Build Apps on Your Data with Grafana Pages (The Context Window #07)

Thu, 05 Nov 2026 18:00:00 +0000

My favourite announcement out of ObservabilityCON was Grafana Pages — a way to build a live, interactive data app right from your Grafana instance, with just prompts and no code. In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by my co-host Tiffany Clifford and by Stephanie and Tom from Grafana’s AI department, who built Pages (it started as a hackathon project to give Grafana Assistant a proper Linux sandbox, and turned into something much bigger). We dig into what Pages actually is, build one live from a prompt, and get into the stuff I find genuinely delightful about it: a page is still real Grafana — same data sources, same RBAC, same production-ready panels — it’s just no longer you arranging things on a grid. We also cover personal data Pages (my sleep-vs-flight tracker, Stephanie’s StairMaster world tour), Pages vs dashboards, how it all works under the hood (React in a locked-down iframe, served from a micro-VM sandbox), and the current limitations and roadmap.

Fun fact dropped mid-stream: the ObservabilityCON keynote itself was a Grafana Page. So was this presentation.

What Grafana Pages is

As Tom put it, a Page is “all of Grafana, minus the panel grid.” It’s the same data sources, the same RBAC, the same underlying panels and visualization libraries — but instead of you arranging panels on a grid (or Assistant trying to squeeze your intent into what a dashboard can represent), you unlock Assistant and let it do the thing modern models are genuinely good at: writing code and building a website. You keep everything that makes Grafana production- and enterprise-ready, carefully worked out over nine years, and you add interactive, tailored, pretty ways to look at your data.

It builds on the Grafana-managed sandboxes announced in the same release — the same underlying layer of Linux compute for AI agents that also powers auto-remediation and investigations.

Seeing it on Grafana Play

The fastest way to understand Pages is to go poke at it on Grafana Play — no login, no Cloud account needed. Stephanie showed a few:

An observability-survey quiz game in comic-book style, pulling real data already on Play — built from one prompt (“make it higher-lower, comic-book style”).
A quarterly-review page, fully interactive with drill-downs by region and customer sentiment — also one prompt, with “make it look pretty” doing most of the design work.
A teammate’s page that pulls your GitHub contribution data and thanks you — and, because it uses the inline Assistant, renders the thank-you in the viewer’s native language (inferred from internal data).

The point Stephanie and Tom kept coming back to: Pages aren’t static. A page can fetch recent data every time you open it and tell you the most noteworthy events in the last 24 hours — so an on-call summary page genuinely updates itself.

Building a Page from a prompt

We built one live on Play against the demo incident/outage data: “Create a page to summarize what is noisy and needs attention from the on-call engineer — give us the top three things.” Assistant loops through alerts, shows the queries it’s running in a Sources tab (no hand-waving about what it’s doing), takes live screenshots of the rendered page as it edits so it actually knows how it looks, and comes back with dynamic top-three items. Making a page really is just that easy now.

Personal data Pages

This is where I got carried away. I hooked up my two smart-ring data sources (Oura and Ultrahuman — both have API endpoints) and, from one prompt, got a whole page summarizing years of sleep: best month, worst month, seasonal breakdowns, the slightly-depressing reveal that I average just under seven hours. I didn’t ask for the blue-purple theming. I also didn’t ask it to correlate my flights (TripIt + Flighty, ten years of data) with my sleep — it just did, because the flight data was there. A second prompt gave me an animated replay of every flight I’ve taken.

Stephanie’s was my favourite: a StairMaster world tour, ranking gyms she’s visited for work by how good the view is (so she doesn’t have to think about being on a StairMaster), complete with an out-of-order-machine mini-game and motivational quotes from ultrarunners. And a summit-conditions page pulling weather data for different mountains.

Pages vs dashboards

So when do you build a page instead of a dashboard? The honest answer from the team is: it’s early, and Pages already cover many dashboard use cases. Tom’s product-brain answer is “make it a page, period” — but if you have a well-understood dashboard that serves you well, there’s no urgent reason to change it, and dashboards still win where you need a dashboards-as-code story (Pages has no as-code path yet). For me, I reach for a page when I want to tell a story with data — a narrative, some annotation, something I’d present to my manager without drowning her in every metric. Pages is great at filtering down to the readable essentials without deleting anything.

A nice pro tip from Tom: if you ever present real data on a big screen, Pages is for you — you can ask for presentation style and still get real panels backed by real data, instead of screenshotting them out. We’re both using Pages for KubeCon talks.

Remixing, editing, and building from a dashboard

There are several ways to create a page:

Ask Assistant directly in chat.
Create new page with a starter example (service health, minimal, etc.) that pre-seeds the prompt with styling hints.
Remix an existing page — reuses it as a reference and changes style, data sources, or format; saved as its own new page.
Start from a dashboard — turn an existing dashboard into a page (handy when the dashboard is the ongoing stream and you want a “how was last quarter” narrative view).

Editing happens through Assistant too: highlight an element and it’s attached as context (it carries the DOM node / data-page-id), so you can ask “how did you get this data?” or request a change to that specific element. Templates are coming — define a house style once and have everyone in your org generate pages or presentations that match.

Sharing today works a lot like Google Docs/Slides: add specific people, generate a link, or drop it in a folder (folder view/edit permissions carry over). But viewers must be logged into the Grafana instance — there is no fully public “share with the world” option yet, and no share-by-email-domain yet either. Both are open feature requests; the team explicitly asked for feedback on whether people want them.

Crucially, data access still respects user permissions. If you can open a page but don’t have access to its data sources, the page opens but shows no data — exactly like dashboards, because the queries still go through your regular data sources with full authorization and RBAC.

How Pages work: React, sandboxes, and iframes

A dashboard is configuration — a (very large) JSON blob that Grafana lays out into the panel grid at runtime. A Page is a layer lower: what the agent produces is React source code for a website, which Grafana renders inside a heavily locked-down, isolated iframe, piping through only carefully chosen data.

But something has to safely compile that TypeScript/JSX and bundle it — running arbitrary model output is exactly the kind of unsafe operation you don’t want on your main infrastructure. So each conversation gets a thin Linux micro-VM. The hard engineering problem was making it fast: nobody waits two or three minutes for an EC2 to spin up before losing interest. The team got it to start in under 10 seconds, with each command finishing in under two.

Tom clarified that “sandbox” is overloaded here — there are two:

The Linux micro-VM used only while building or editing a page. Once built, the output is just HTML in an S3 bucket — no VM involved anymore.
The iframe isolation layer inside Grafana, so AI-generated website content renders safely without leaking your data. You control what hosts a page can call out to, per page (important — not every page should be as shareable as, say, my health data). MCP server connections are supported too, and any viewer gets asked for permission before anything runs on their behalf.

Limitations and roadmap (no promises)

Pages is in public preview across Grafana Cloud (not OSS yet, and it will always require Assistant to drive the prompt → page step). Known rough edges and wishes, with the usual “nothing here is a commitment”:

Multi-editing: today you don’t know if someone else is editing the same page, so you can clobber each other. A “someone else is editing” warning is coming before GA, with proper multiplayer later. (We hit this building the KubeCon deck.)
Public sharing and share-by-domain: open questions, driven by feedback.
Storage: pages and their remixes naturally form a graph of related code artifacts, so a Git-inspired storage model feels like the next natural step.
Higher environment fidelity, templates (rolling out now), and more examples.

If you want it for OSS — which I personally do — leave a comment so the team can champion it internally.

Resources

Grafana Play — try the public Pages (games included), no login required
Grafana Cloud — sign up free; Pages is in public preview
Grafana Community Slack — there’s a Grafana Pages channel for feedback

My notes on Grafana Assistant

Transcript

Intro, guests, and ObservabilityCON recap

Nicole: Hi everyone, welcome to another episode of The Context Window. Today we’re talking about the hottest thing — in my opinion — to come out of ObservabilityCON: Grafana Pages. It’s a way to build a live, interactive data app right from your Grafana instance, with just prompts and no code. I’m not the only one excited about it, so — Tiffany, do you want to introduce yourself, my awesome co-host?

Tiffany: Hi, I’m Tiffany, a developer advocate. Until literally a couple of days ago I was focused on Tempo, Pyroscope, and AI — and now just AI, so I get to dive into all this really cool stuff. This one’s new to me anyway.

Stephanie: Hello, I’m Stephanie. I work in the AI department with Tom. We did a hackathon together for Pages and have been working on getting it released for ObservabilityCON. That’s been fun. Tom?

Tom: Hi, I’m Tom, also in the AI department — I’ve been at Grafana seven years. For the last couple of months Stephanie and I were toying with the idea of giving Assistant a proper Linux sandbox and a proper way to execute code. We didn’t even set out to build Pages — then we realized we could just let it write React code, and that gives you so much better visualizations and tailored outcomes than dashboards.

Nicole: I’m Nicole van der Hoeven, a developer advocate mainly focused on AI right now. And I wanted to mention ObservabilityCON, which just passed — I wasn’t there in person, but a lot came out of it. The keynote on YouTube is really good.

Tom: Can I tell you a secret? The keynote is a Page. And, sneakily, so is this presentation. The three ObservabilityCON themes were autonomous operations — and look, there’s Grafana Pages right there — cloud economics, and context. We’ll focus on Pages.

What Grafana Pages is

Stephanie: We started by working on sandboxing for coding agents, then in a hackathon realized we could use those sandboxes for Grafana Pages. It lets you see your data in more interactive ways — you can prompt Assistant with “I want to look at my business revenue for the past quarter,” or weather on a mountain, and it creates unique interactive ways to look at data.

Tom: If I had to describe it, it’s all of Grafana minus the panel grid. Same data sources, same RBAC, same underlying panels, same visualization libraries. It’s just no longer you arranging them on a grid, or Assistant trying to fuse your thoughts into what a dashboard can represent — we unlock Assistant and let it write code and build a website. But you’re not giving up the things that make Grafana production- and enterprise-ready, carefully worked out over nine years.

Tiffany: And the sandboxes you mentioned — is that the Grafana-managed sandboxes that also got released?

Tom: Those were coined more towards auto-remediation and investigations, where we use sandboxes to look at your source code and check a hypothesis for why something’s broken. But it’s the very same underlying layer of Linux compute for AI agents that powers Pages too.

Grafana Pages examples on Grafana Play

Nicole: The best way to understand Pages is to dive in on Grafana Play — this playground for all things Grafana. No login, no Cloud account. You can just mess around and see what people have already created.

Stephanie: Here’s an observability survey — we run one every year — turned into a game. It’s pulling in real data that’s already on Play, with little animations, streaks, ten questions. Maybe you think reading the survey is dry; a game could help. And this was one prompt: “make me a higher-lower game, comic-book style.” I’m not a front-end engineer — I did not do this.

Here’s a quarterly-review example, also one prompt, all interactive — drill into what the customer is thinking, different regions, fake sales data. I can’t get over how pretty it is. There’s also a Sources tab so you can see the queries behind the data. A teammate made one that pulls your GitHub data and thanks you for your contributions.

Tom: That one uses the inline Assistant, so the page itself can make model generations — it shows the thank-you in the viewer’s native language, guessed from internal data.

Stephanie: The powerful part is it’s not static. You can make a page that summarizes recent data, and it’s not hard-coding the summary — every time you open it, it first fetches recent data and tells you the most noteworthy events in the last 24 hours. I made an on-call rotation page that surfaces the most urgent things every time I look.

Build a Grafana Page from a prompt

Tom: Since it’s so easy to prompt — can we build one from scratch on Play?

Stephanie: You go to Pages, create a new page, pick from starter examples that give Assistant some styling direction so you don’t need a huge prompt.

Nicole: Let’s say: “Create a page to summarize what is noisy and needs attention from the on-call engineer — give us a summary of the top three things.” It’s so cool that making a page is now just that easy.

Stephanie: It’s looping through alerts, figuring out the alert states, and showing its queries in the Sources tab — so it’s not hand-wavy; you can check it’s actually doing what you asked.

Tom: Depending on what you ask, it can take a bit longer. Let’s leave it and jump to some other things.

Personal data Pages: sleep and flight tracking

Nicole: I have two smart rings — Ultrahuman and Oura. I hooked up both data sources (they have API endpoints), and this whole page was one prompt: how much sleep I get each year, best and worst months, by season. I average just under seven hours. I didn’t ask for the blue-purple theming either.

Then it added a travel page — I realized it’s correlating my flights with my sleep, because I have flight data in here too (TripIt and Flighty, ten years). I didn’t ask it to do that.

Tiffany: I was literally looking into TripIt yesterday — and apparently they closed the API in late September to anyone not already using it. I’m so glad you’ve been using it.

Nicole: This replay — this one I did ask for, so two prompts — animates all my flights, new countries, busiest month. Pages has been useful for more than the usual outages.

Grafana Pages vs dashboards

Nicole: When would you build a page versus a dashboard? Are you all-Pages all the time now?

Tom: The product person in me says “make it a page, period.” Honestly we don’t fully know yet — Pages cover many dashboard use cases. If you have a well-understood dashboard that serves you well, there might be no immediate reason to change it. But in general Pages should replace most dashboard use cases, and if they fall short, that’s a reason to talk to us.

Stephanie: For dashboards, if you need a dashboards-as-code story — running the same dashboard across many clusters — we don’t have Pages-as-code right now, so dashboards still win there.

Tom: Pages will not come to OSS without Assistant — the way it works, you have to prompt Assistant to make it into a page. Assistant has a lot of smarts about correlating Grafana data; a vanilla model wouldn’t have the opinionated, docs-backed best practices.

Tom: Pages are very much a Grafana thing. Grafana has always stood for two things — that it’s pretty, and that it’s serious, as in real data, not made up. We tried hard to preserve both: lots of tuning so a small prompt gives a pretty page, and it still retains inspectability and real data rather than looking like LLM slop. If you just want to post something up, tools like Claude artifacts already exist.

Remix a Page or create one from a dashboard

Stephanie: The main way is prompting. There’s also “make your own,” where you prompt it much more specifically about the look. Templates are coming — define a style for your whole org’s presentations. And you can remix a page: it uses the page as a reference and changes style, data sources, or format depending on what you ask; saved as its own new page.

For editing, you highlight an element and it’s attached as context — the DOM node with its data-page-id — so you can ask how it got the data, or request a change to that specific element. You can also start a page from a dashboard.

Nicole: I tend to build a page when I want to tell a story with the data — some narrative, maybe annotation. My dashboards get cluttered; Pages lets me streamline to what’s readable without deleting anything. Great for explaining something to my manager.

Stephanie: To share, you can add specific people, say “anyone with this link,” or put it in a folder so folder permissions apply. There’s no fully public “share with the world” option today — that’s a feature request; let us know in the comments. And data access respects user permissions: if you can open a page but lack access to its data sources, it opens but shows no data, just like dashboards.

Tom: It’s best described as the Google Docs/Slides model — share with specific people or generate a link — but viewers must be logged into the Grafana instance. There’s no public option yet.

How Pages work: React, sandboxes, and iframes

Tom: A dashboard is configuration — a big JSON blob Grafana lays out into the panel grid at runtime. A Page is lower-level: what the agent produces is React source code for a website, which we render in a heavily locked-down, isolated iframe, piping through carefully chosen data.

Something has to safely compile that JSX and bundle it, which is unsafe to do with arbitrary model output — so each conversation gets a thin micro-VM. Making that scalable took most of the platform time: you don’t want to wait two or three minutes for an EC2. We got it to start in under 10 seconds, every command finishing in under two.

The term “sandbox” is overloaded. There are two: the Linux micro-VM used while building or editing a page — once built, it’s just HTML in an S3 bucket, no VM anymore — and the iframe isolation layer inside Grafana, so AI-generated content renders safely without leaking your data.

Stephanie: There’s a setting for what a page can call out to, and you can control it per page — some things, like my health data, shouldn’t be broadly available. MCP server connections are supported too, and viewers are asked for permission before anything runs on their behalf.

Stephanie: Data sources still use the user’s permissions — if you don’t have access, it won’t show data. It builds on battle-tested, enterprise-ready foundations, so authorization and RBAC carry forward.

Limitations, roadmap, and more Page examples

Nicole: Beyond sharing, what other limitations should people be aware of?

Stephanie: Multi-editing is difficult — right now you don’t know if someone else is editing a page, so you can overwrite their work. Until GA, just tell people if you’re editing. We’ll have a “someone else is editing” warning before GA and proper multi-edit later. Templates are rolling out now.

Tom: Pages and remixed pages form a graph of relationships and are code artifacts, so some Git-inspired storage system feels like the natural next step — but it needs careful thinking.

Nicole: I have an unnecessary setup where Alloy on my laptop sends metrics about my Obsidian notes into Loki and Prometheus — and I have a page for all of it: top topic this month, a GitHub-style contribution heat map, a graph view. I didn’t ask for any of the styling; I just said “here’s some data, build me a page.”

Stephanie: My favourite is my StairMaster world tour — I do mountaineering, so as I travel for work I find StairMasters with pretty views. One prompt ranked the gyms by view, added a mini-game for out-of-order machines, and pulled motivational quotes from ultrarunners. I also made one for summit weather conditions on different mountains.

Nicole: There’s a Grafana Play page on Singapore weather and haze that’s beautiful — it shows wind direction and air quality across regions, with a 72-hour replay. I have family in Singapore, so I care about the air quality. Amazing stuff.

If you want to try this, sign up for a free Grafana Cloud account — Pages is in public preview, nothing to contact anyone about — or just play with the ones on Grafana Play. And please give feedback: there’s a Grafana Pages channel in the community Slack and a Reddit post. Comment on the video too, especially if you want it for OSS so we can champion it internally.

Nicole: Thank you all for joining — this was a fun one. The next Context Window livestream is later this month with William Dumont, about Assistant Watchers. Stay tuned.

Tom: Thanks for building this amazing tool — it’s my favourite feature on Grafana now.

AI Tools We Use Every Day on the Grafana AI Team

Fri, 30 Oct 2026 09:00:00 +0100

What AI tools do the people building AI features actually reach for day to day? I asked around the Grafana AI team to find out which tools have genuinely stuck — the ones we open without thinking — versus the ones that didn’t survive contact with real work.

Resources

My notes on Grafana Assistant

What Does a Grafana AI Offsite Look Like?

Wed, 28 Oct 2026 09:00:00 +0100

A quick behind-the-scenes short: what actually happens when Grafana’s AI team gets together in person for an offsite.

Resources

My notes on Grafana Assistant

Roundtable: How We Really Use AI in Testing (HUSTEF 2026)

Thu, 08 Oct 2026 14:00:00 +0200

On the final day of HUSTEF 2026 in Budapest, I joined the conference’s roundtable on AI in testing — “AI Open Space Experiences: a roundtable with speakers on their experiences with AI testing.” It was five of us on stage: me plus Marta Firlej, Kristoffer Nordström, Callum Akehurst-Ryan, and Maryia Tuleika, with Marta moderating.

The format was refreshingly honest — “how we really use AI, with no bullshit.” Marta had run the same “AI open space” exercise with the Polish testing community, crowdsourcing the actual, un-hyped ways people use AI (information gathering, functionality discovery, generating tests and code, TDD, recruitment, presentation prep, even designing furniture and planning holidays), and then opened it up to the panel and the room. Here’s roughly how I answered her questions:

What do you use AI for? Mostly two things at work: analysing codebases and test code to understand what something actually does, and filling gaps on teams without a dedicated quality engineer — baking my testing knowledge into an AI the team can ask questions of. On the personal side, I’ve run some weird experiments: I’m a lifelong note-taker with decades of context (journals, therapy notes, eight years of WhatsApp history with my partner), so I’ve fed it all into a local, open-weight model and asked things like “what do I claim to believe, and where have I acted contrary to that — cite your evidence?” It’s a surprisingly honest mirror. I’m in a privileged spot with effectively unlimited AI access right now, and I know it won’t last, so I want to find out what’s safe and what isn’t while I can.

How do you stay up to date? Honestly, I don’t — not by chasing model releases or news. I keep up with how other people, especially in testing, are using AI, and I lean on my coworkers (we’re at the forefront of this at work, so stepping outside that bubble is a useful reality check). I do have a scheduled Claude Code task that scrapes my ~34 AI-related Slack channels into a daily digest of just the things I care about.

One piece of advice? Don’t trust AI too much, and don’t believe you’re as far behind as you think. Almost no one has really tried this stuff; it’s genuinely early, everyone’s learning, and your foundational testing skills transfer. Just pick it up, play, and if it’s not for you, move on.

Anything you’d never use AI for? Communicating with people as a proxy. I’ll use it to translate or to help me sound more fluent (I live across multiple languages — Portugal, Dutch, English), but I never want it to craft a whole message I didn’t think of. I don’t want to lose the discipline of actually talking to a human.

Ethical dilemmas? Yes — which is exactly why I keep my use mostly in the context of work, and try not to generate imagery or anything that’s obviously lifting from artists who deserve to be paid for it. There’s no hard line; it’s a sliding scale, and I have to keep questioning myself. (A Mastodon line stuck with me: the strange future where the humans do the menial tasks and the robots write poetry and paint. I try not to let AI stop me from making art.)

The best part of these panels is always the disagreement and the texture — five people who use AI daily and still land in very different places on trust, ethics, and what it’s actually for. Thanks to Marta for moderating and to Kristoffer, Callum, and Maryia for a genuinely good conversation.

Resources

HUSTEF 2026
My HUSTEF keynote: Asimov’s Zeroth Law of Robotics: Testing for AI

Asimov's Zeroth Law of Robotics: Testing for AI (HUSTEF 2026)

Wed, 07 Oct 2026 10:00:00 +0200

This week I gave a keynote at HUSTEF 2026 in Budapest, Hungary. The talk is called Asimov’s Zeroth Law of Robotics: Testing for AI — a testing-focused evolution of a talk I’ve been giving (and rewriting) for a while, previously as Observability for AI at KubeCon EU 2025 in London, Dutch Cloud Native Day 2025 in Utrecht, NewCrafts in Paris, and ExpoQA Madrid 2026. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be testable.

AI systems fail differently than traditional software. There’s no single correct output to assert against. A response can be fluent and completely wrong. Models drift silently over time. And yet we’re shipping AI into production — often without knowing what “passing” even looks like.

Observability is what makes testing possible. When we can see inside our AI systems — trace the chain of reasoning, measure what changed, surface what failed — we can start to write evaluations, define what “good” looks like, and catch regressions we didn’t know to look for.

In this talk, I explore what it means to test AI in the real world, drawing on lessons from building and evaluating an AI assistant for a large-scale observability platform. I’ll cover how evaluations work as the AI equivalent of a test suite, what to measure, how to catch hallucinations, and how to know when your model has regressed.

You cannot test what you cannot see — and you cannot trust what you cannot test.

The big shift in this version is right there in the Zeroth Law itself. In earlier iterations I argued that a robot must be observable. That’s still true — observability is the only way to see the reasoning behind a system and find out what actually happened. But I’ve come to think that pure knowledge isn’t the same as trust. You can know exactly what a system did and still not trust it. The only way to trust something is to put it through its paces: to know how it behaves in the situations you foresee, and even the ones you don’t. So the Zeroth Law has become a robot must be testable.

I framed the whole thing around Asimov’s short story Runaround (from I, Robot) and its robot, Speedy. Speedy’s smuggler owners sent him to collect selenium on Mercury, and he got stuck in a loop — circling the selenium pool, unable to resolve the conflict between obedience (the second law) and self-preservation (the third). His owners couldn’t predict his behaviour and couldn’t reason about it, so they had to do the only thing left: go out into the danger zone and fix him in production. Speedy wasn’t untestable because his owners were incompetent or lacked the tools — he was untestable because he was never built to be tested. That’s the trap I think a lot of AI systems are walking into right now.

The through-line for a testing audience: AI is rewriting how software gets built, but it doesn’t rewrite the discipline of trust. Trust that a system does what we say it does, and that when it breaks we know why and can fix it — that’s always been the job, and it isn’t going anywhere. There’s a whole generation of AI engineers shipping software they haven’t tested and can’t explain, and when someone says “the agent does this,” our job is still to ask how do you know? Testing isn’t just going to survive the age of AI — testing is the reason AI gets to survive, if we make it testable.

I demoed this with my TTRPG side project: an AI Dungeon Master, instrumented and put under test. It turns out that writing evals for something non-deterministic forces you to confront your real requirements. I thought I knew exactly when a dungeon master should call for a dice roll — I GM myself! — but trying to put it into words (“in this situation I’d roll, but in that one I wouldn’t, because I want to keep the story moving”) surfaced decisions I’d never had to make explicit. Those are things we’d normally discover much later, in production. Having to figure them out up front is a win.

The Q&A was genuinely great and ranged well beyond the slides:

“When has an AI been tested enough?” Same answer as for anything else — it’s a mix of risk and budget. If it were a life-critical robot like Speedy, you test every scenario the people relying on it need. For my little demo app, less so.
Testers vs. developers in the age of AI. I said I’d fire developers before testers — provocatively, but the point stands: anyone can write code now. What made a good developer good was never the coding; it was communicating with business and engineers, translating requirements, finding creative-but-safe solutions. Testers already think at that level and translate across teams, which makes them more important in the age of AI, not less.
Defining expected results for non-deterministic systems, multi-turn evals (I give the simulated user a personality — adversarial, cooperative, beginner — rather than a script), and the broader shift from imperative to declarative: we no longer instruct step by step, we declare the end state we want, the way we do with Kubernetes.
Convincing colleagues not to fear AI. I don’t, really. AI is here to stay; the question isn’t whether you approve of it, it’s whether you want to stay in this industry and help make it safe and trustworthy — or sit on the sidelines.

I also recommended the book Slow AI, which lands where I do: not an AI apologist, not a boycott, but a clear-eyed list of what AI is and isn’t good for — and the reminder to keep checking whether a given use of AI is making you more capable or less.

The demo app (the instrumented AI DM) lives on GitHub, and I keep refining it with each iteration of this talk.

I’ll update this post with the recording once HUSTEF publishes it.

Resources

Slides
Demo app on GitHub
Slow AI (book)
Previous versions of this talk: KubeCon EU 2025 · Dutch Cloud Native Day 2025 · NewCrafts 2025 · ExpoQA Madrid 2026

Context Engineering Fundamentals (The Context Window #06)

Fri, 02 Oct 2026 18:00:00 +0100

What actually counts as “context” for an AI agent — and how does it find the right things to put in a limited window? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I dig into context engineering fundamentals with Ben Sully and Ioanna Armouti. We treat the context window as an attention budget, walk through Assistant Search and semantic retrieval, embeddings, and how you actually evaluate whether the retrieved context was any good.

What we cover

What counts as context — and why the context window behaves more like an attention budget than just a size limit.
Finding the right context — Assistant Search, semantic retrieval, and embeddings: how an agent surfaces what it needs instead of being handed everything.
Evaluating retrieved context — because retrieval that looks relevant isn’t the same as retrieval that is.
The tool tax and tool deferral — every tool definition costs context, so when do you load it versus defer it?
Progressive disclosure and long-term memory — giving the agent what it needs, when it needs it.
Why removing 80% of a system prompt can change nothing — a nice illustration that more context is not automatically better context.

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana GCX

Fri, 21 Aug 2026 09:00:00 +0100

Three AI tools people confuse: Assistant, MCP, and GCX. Last time I talked about the easy hand — the MCP server. Now the power hand: GCX, a CLI built for AI agents.

If Assistant is the brain and Grafana MCP is the easy hand, then GCX is the power hand — a CLI built for an agent to drive. (Quick refresher: a CLI is a command line interface — no clicking, just typed commands.)

So when does GCX win?

When you’re using terminal coding agents — Claude Code, Cursor, Codex. It writes code, pulls the exact log or metric, and verifies against it — without leaving the editor.
When you need everything. GCX wraps the entire Grafana Cloud API — SLOs, Synthetics, IRM, k6, Fleet — the esoteric corners MCP skips. The full banana.
When you want automation and CI/CD. Dry-run mode, JSON output, and exit codes an agent can act on — built for GitOps.

Do you need Grafana Assistant for this? No — not even a subscription. Bring your own brain, same as MCP: metrics, logs, dashboards, and alerts run on nothing but a service account token. You only really pay for Assistant if you call gcx assistant directly — which taps its reasoning, billed separately.

So there’s your map: Assistant’s the brain, MCP’s the easy hand, GCX is the power hand. It’s not one versus another — they work really well together to give you the best choice in how to access your Grafana stack.

Transcript

I’m differentiating Assistant, Grafana MCP and GCX and in this one I’m talking about GCX. If Assistant is the brain and Grafana MCP is the easy hand, then GCX is the power hand — a CLI built for AI agents.

Quick refresher: a CLI is a command line interface — no clicking, just typed commands. GCX is a CLI built for an agent to drive.

So when does GCX win?

When you’re using terminal coding agents — Claude Code, Cursor, Codex. It writes code, pulls the exact log or metric, and verifies against it — without leaving the editor.
When you need everything. GCX wraps the entire Grafana Cloud API — SLOs, Synthetics, IRM, k6, Fleet — the esoteric corners MCP skips — the full banana.
When you want to do automation and CI/CD. Dry-run mode, JSON output, exit codes an agent can act on — built for GitOps.

Do you need Grafana Assistant for this? No — not even a subscription. Bring your own brain, same as MCP: metrics, logs, dashboards, alerts run on nothing but a service account token. You only really pay for Assistant if you call gcx assistant directly — which taps its reasoning, billed separately.

So there’s your map: Assistant’s the brain, MCP’s the easy hand, GCX is the power hand. It’s not one versus another. They actually work really well together to give you the best choice in how to access your Grafana stack.

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana MCP

Wed, 19 Aug 2026 09:00:00 +0100

Three AI tools people confuse: Assistant, MCP, and GCX. Last time I talked about the brain — Assistant. Now hand number one: the MCP server, the easy hand.

Grafana MCP — Model Context Protocol — lets any AI client reach into Grafana. The magic part: you bring your own brain.

Using MCP requires two pieces:

The client is your AI app — the brain you bring: claude.ai, ChatGPT, Cursor.
The server is the door into Grafana that exposes the tools.

There are two ways to get that server. The easiest is the Grafana Cloud hosted one — already running, auth built in, just connect and go. But you can also self-host the open-source one on your own infra.

So when do you reach for Grafana MCP?

When you’re using hosted chat tools with no terminal — claude.ai, ChatGPT, your phone. No shell means no CLI, so MCP is the only way in.
When you want to bolt Grafana onto a tool you already use — already in Cursor or Claude? Just add the server. No new workflow.
When you want the best discovery out of the box — the tools describe themselves up front, so even smaller or cheaper models just know what’s available. No trial-and-error.

One honest catch: those tool definitions can eat some context if you load everything — but modern hosts fix that with lazy tool search.

MCP is deliberately curated — it doesn’t expose every corner of Grafana. So what do you do when you need the esoteric stuff? That’s exactly what the power hand, GCX, is for — next up.

Transcript

Three AI tools people confuse: Assistant, MCP, GCX. Last time, I talked about the brain — Assistant. Now hand number one: the MCP server, the easy hand.

Grafana MCP — Model Context Protocol — lets any AI client reach into Grafana. The magic part: you bring that AI client - you bring your own brain.

Using MCP requires two pieces. The client is your AI app — the brain you bring: claude.ai, ChatGPT, Cursor. The server is the door into Grafana that exposes the tools.

Two ways to get that server: the easiest way is to use the Grafana Cloud hosted one — already running, auth built in, just connect and go. But you can also self-host the open-source one on your own infra.

So when do you reach for Grafana MCP?

When you’re using hosted chat tools with no terminal — claude.ai, ChatGPT, your phone. No shell means no CLI, so MCP is the only way in.
When you want to bolt Grafana onto a tool you already use — already in Cursor or Claude? Just add the server. No new workflow.
When you want the best discovery out of the box — the tools describe themselves up front, so even smaller or cheaper models just know what’s available. No trial-and-error.

One honest catch: those tool definitions can eat some context if you load everything — but modern hosts fix that with lazy tool search.

But MCP is deliberately curated — it doesn’t expose every corner of Grafana. So what do you do when you need the esoteric stuff? That’s exactly what the power hand is for. Next up is GCX.

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Mon, 17 Aug 2026 09:00:00 +0100

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. They’re not competitors. Think of it as a brain and two hands. In this short, I start with the brain: Grafana Assistant.

Assistant isn’t just a chatbot — it’s the reasoning layer. It manages context and knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. Instead of clicking around, you tell Assistant what you want to do and it figures out the how.

Reach for the Assistant when the work is visual or exploratory:

Building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
A real incident that needs root-cause analysis across multiple data sources? Assistant. That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in, and I’ll cover the first of those (the MCP server) next.

Transcript

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. I’m going to talk a bit about each one so you know which one is best for your use case.

Spoiler: They’re not competitors. Think of it as a brain and two hands.

Let’s start with Grafana Assistant, which is the brain.

It’s not just a chatbot — it’s the reasoning layer. It manages context, it knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. So instead of you clicking around to do something, you can say what you want to do to Assistant and it figures out the how.

Reach for the Assistant when the work is visual or exploratory.

If you’re building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
You have a real incident where you need root-cause analysis across multiple data sources? Assistant.
- That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in… and I’ll show you those next.

Resources

Grafana Assistant (product page)
My notes on Grafana Assistant
My notes on GCX
Unlearnings from building Grafana Assistant — my Context Horizon piece

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Fri, 31 Jul 2026 09:00:00 +0100

The final day of AI Week is about collaborating — using AI right where your team already works, no context switching required. Grafana Assistant in Slack is now generally available: investigate issues, ask follow-up questions, and query your metrics and logs from the channel where the incident conversation is already happening. And that’s just one place Assistant now meets you — also announced: the Grafana mobile app is GA (with Assistant on mobile in public preview), Assistant Desktop in public preview, Microsoft Teams support in private preview, and context-aware Assistant Search in public preview. Add it all up and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search. That’s a wrap on five days of announcements.

Transcript

It’s the final day of Grafana Labs’ AI Week, and Friday is all about collaborating — using AI right where your team already works, no context switching required.

Today, Grafana Assistant in Slack is generally available. Investigate issues, ask follow-up questions, query your metrics and logs — all from the channel where your incident conversation is already happening.

And that’s just one place Assistant now meets you. Today we also announced: the Grafana mobile app is GA, with Assistant on mobile in public preview. Assistant Desktop is in public preview. Microsoft Teams support is in private preview. And Assistant Search — deeper, context-aware search — is in public preview too.

Add it all up, and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search.

That’s a wrap on five days of announcements — and I know it was a lot. So if you missed a day, or you just want the full picture of where AI at Grafana stands today, I made a video for exactly that: everything you can do with AI in Grafana Cloud, including everything we shipped this week. It’s linked below and pinned in the comments.

One last time: grafana.ai for all of it. Thanks for spending AI Week with me.

Observability and AI explained

Fri, 31 Jul 2026 09:00:00 +0100

“AI observability” gets thrown around a lot, and it can mean one of two very different things. So in this video I map out the whole landscape of AI-related work at Grafana across the entire software development lifecycle — and show how AI and observability come together in two directions: AI for observability (AI that helps you observe your systems) and observability for AI (observing the AI itself).

The framing is deliberate: AI in observability isn’t about replacing engineers. It’s about handing back the exhausting parts of the job — the 3am triage, the repetitive queries, the report nobody wanted to write — so you can spend your energy on the work only humans can do.

I walk through each phase of the lifecycle and where the tools fit:

Planning — clarity about what’s really going on, with Grafana Assistant: not just a chatbot, but an agent that knows your telemetry, writes your queries, and builds your dashboards.
Building — momentum without losing control: Assistant, the open-source GCX CLI, the Grafana MCP server, the specialized Loki/Tempo/k6 MCP servers, and the AI SDK for building your own agents in Go.
Operating — staying calm under load: Investigations for agentic incident triage, Assistant Automations and the Watcher Agent for staying proactive, plus Grafana Machine Learning (Metrics Forecasting, Outlier Detection) and Adaptive Telemetry.
Testing & evaluating — doubt is a form of care: k6 and agentic testing on the systems side, and Agent Observability plus the open-source o11y-bench benchmark on the “observing your AI” side.
Collaborating — bringing it all to where your team already works: Assistant in Slack (and coming to Teams), on mobile, on the desktop, and via API/MCP, plus Assistant Search across your Grafana artifacts.

If you’re new to the Grafana stack itself, start with my open source observability video for the non-AI foundations that all of this builds on.

Resources

My notes on Grafana Assistant
My notes on GCX
My notes on k6
o11ybench.ai — the open leaderboard for AI agents doing observability work

AI Week Day 4: Evaluating (Grafana AI Week — Thursday)

Thu, 30 Jul 2026 09:00:00 +0100

Day four of AI Week is all about evaluating — because building an agent is the easy part; trusting it is where things get hard. The headline: Grafana Agent Observability is now generally available — the tool Grafana built for itself while scaling Assistant, once it grew too complex to evaluate with unit tests. Instrument your agent with the SDK and your conversations, tool calls, traces, latency, token use, and cost all become observable in one place; add evaluators (LLM judges plus deterministic checks) to score quality on live traffic, feed low-scoring conversations into your test suite, and gate CI on experiments comparing a candidate against a baseline. And because evaluator metrics land in Prometheus, you can get paged when a bad prompt change tanks quality. The fun bonus: k6 agentic testing — describe a browser test in plain language and an agent embedded in k6 drives the browser, no script required (they proved it by getting it to beat Wordle).

Transcript

It’s day four of Grafana Labs’ AI Week, and Thursday is all about evaluating — because building an agent is the easy part. Trusting it is where things get hard.

The big news: Grafana Agent Observability is now generally available. This is the tool we built for ourselves while scaling Grafana Assistant, when it grew too complex to evaluate with unit tests. Now it’s available to everyone on Grafana Cloud.

Here’s how it works: instrument your agent with the SDK, and your conversations, tool calls, traces, latency, token use, and cost all become observable in one place. Then add evaluators — LLM judges plus deterministic checks — to score quality on live traffic. Low-scoring conversations feed your test suite, and experiments let you compare a candidate against a baseline before you ship — you can even make it a CI gate. And because evaluator metrics land in Prometheus, you can get paged if a bad prompt change tanks quality in production.

But my favorite part of today? k6 agentic testing. Describe a browser test in plain natural language, and an agent embedded in k6 drives the browser for you — no script at all. And to prove it, we gave it one job: beat Wordle. Go watch that video. I’ll wait.

Everything’s at grafana.ai — see you tomorrow for the finale: using AI where your team already works.

Investigations Arena: how we test Assistant investigations

Thu, 30 Jul 2026 09:00:00 +0100

How do you test an AI agent whose whole job is to investigate incidents — when every investigation is different and “correct” isn’t a single deterministic answer? In this interview I sit down with Will, a software engineer at Grafana who’s spent the last year focused on AI: the Grafana Assistant, investigations, and memory features.

The story here is how the team got from Investigations V1 to a much better V2. V1 hit a ceiling, and the demo environment they’d been testing against turned out to be the problem — the cases were too easy and too far from the messy reality of a production cluster. So they built an arena: think competitive bot programming, but the “game” is investigating real incidents in Grafana’s own production environment. Different AI agents, each with its own strategy and tools, all take a swing at the same incident, and an LLM-as-judge scores their answers against a human-verified root cause.

What’s interesting is that the score wasn’t even the most valuable output at first — the insights were. Watching which strategies worked, which tools returned so much noise that they poisoned the context, and which agents wandered off is what let the team optimize bit by bit into the V2 release. The arena also compares more than models: different harnesses running Sonnet vs. Opus, Claude Code with the Grafana MCP, even just hitting the public API directly with curl.

A couple of my favorite moments: Will explains the guardrails that stop agents from “cheating” (they can only look at data from within the incident’s time window, so they can’t peek at the fix commit that resolved it) — and how the models mostly play fair because they treat it like a game. And there’s a lovely example of emergent creativity: an agent reached for tools the team hadn’t even considered, they turned out to be genuinely useful for investigation, and those tools got promoted into the real harness. Will’s hope for the future is to open this up — an OliBench-style shared benchmark where the community can throw their own investigation agents into the arena and raise quality across the board.

Transcript

Will: Hey, my name is Will. I’m a software engineer at Grafana. I’ve been there for three years, and for the last year I’ve been focusing on AI — the Grafana Assistant, investigations, and memory features.

Nicole: And what’s the latest thing you’ve been working on?

Will: So we developed Investigations V1, and we saw that we hit a ceiling at some point. We came up with a different approach, but this time we needed a good evaluation environment — the demo environment wasn’t enough, because those cases were too easy and too far from the reality of a production environment. So we decided to build an arena.

It’s a little bit like the idea of competitive bot programming, where bots fight against each other on specific tasks or games. In this case, the topic was investigations in our production cluster — real incidents. We started building bots that had different strategies and different tools; these were AI agents. Comparing a diversity of strategies let us surface the best approaches, and from there we could iterate, bring in more test cases from more incidents, and reach higher and higher quality.

The idea is that the bots try to solve the same case, and we use an LLM as a judge to compare the different approaches. We have different criteria, because for each incident we have a root cause that’s been verified by a human. That gives us our ground truth, and the LLM-as-judge compares that ground truth against the answers the agents came up with.

The important part at first wasn’t even the score — it was the insights. This tool was better with this strategy; this one had noise in the results that poisoned the context; the results returned from this specific tool were just too much. So we managed to really optimize bit by bit and come up with Investigations V2, which we just released. It’s a massive improvement compared to our V1 approach.

Nicole: So is it comparing just different models for investigations, or is it also comparing things like Claude Code and the Grafana MCP versus Assistant?

Will: Both, actually. We have different versions of our harness for investigations that we run with Sonnet and with Opus, but we also use the Claude Code harness with the Grafana MCP — and even just using curl to directly hit our public API.

Nicole: Do the models ever cheat?

Will: So far it’s been quite good. We have guardrails. Because we investigate past incidents, we need to make sure the LLM only looks at a specific time window — only from when the incident started — otherwise it would be able to find the pull request that was used to solve the incident. But they’re pretty good at following the rules; they take it as a game a little bit. We even have an agent that checks afterward, with the judge, to make sure they followed every rule.

It’s funny, actually — we had a case using Claude Code where it started to use some tools we hadn’t thought about in the first place, and they turned out to be very useful for investigation. We’d never considered them, so we brought them into our investigation harness and it helped a lot. Giving the agents freedom and creativity in this kind of environment is really useful. That diversity sponsors new ideas that we can put into our own work.

Nicole: Cool. Do you have any plans to publish this and share it so that other people can do it too — kind of like OliBench?

Will: Yeah, I think, why not? I think we can go this way too. I’m sure it would be useful for other agents. And it would be very nice if we could compare agents from other people and other companies — maybe with different harnesses, maybe even different tools — because right now it’s mostly about our MCP, our CLI, and our investigation harness. But if the community comes up with their own investigation agents — and I think I’ve seen a few — it would be fun to put them in the arena and help not just our product, but any product improve in quality.

Resources

My notes on Grafana Assistant

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Thu, 30 Jul 2026 09:00:00 +0900

This week I spoke at KubeCon + CloudNativeCon Japan 2026 in Yokohama, Japan. I gave a talk called The Great Doubt: What Building an AI Agent Taught Us About Trust. Here’s the abstract:

When we started building an AI assistant for observability, we thought the hard part would be making it smart. We were wrong. The hard part was knowing when to trust it.

AI agents hallucinate, forget context, and confidently give wrong answers. Traditional testing doesn’t catch these failures. What’s needed is evaluation grounded in systematic doubt.

This talk shares lessons from shipping an AI agent to production: how to build a golden dataset of use cases you must get right, how to use LLM-as-judge when there’s no ground truth, and how to use OpenTelemetry traces to debug eval failures. You’ll also hear what’s still unsolved: evaluating multi-agent handoffs and closing the feedback loop between what users ask and what your evals cover.

Kyoto School philosopher Nishitani Keiji called this “The Great Doubt” (大疑): questioning every assumption until only what survives is real. For AI agents, that’s not philosophy. It’s the job.

The talk walks through the layers of doubt we worked through while building Grafana Assistant: doubting the agent itself, then doubting our tests, then the LLM-as-judge, then our own scores — and finally doubting whether the doubt was even justified. Each layer needed a different tool, from a golden dataset run in a fully synthetic observability environment, through online evals on live traffic, to per-conversation OpenTelemetry traces that link every eval score back to the exact LLM and tool calls that produced it.

You can find the slides at nicole.to/doubtslides, and you can watch the full talk below:

Resources

Grafana Assistant (generally available for Grafana OSS, Grafana Cloud, and Grafana Play):

Assistant Investigations (Grafana Cloud):

Docs

Agent Observability (Grafana Cloud):

Docs

o11y-bench (OSS):

Site

Investigations Arena:

Video

My notes on Grafana Assistant
My Context Horizon piece Unlearnings from building Grafana Assistant
The Way of Zen by Alan W. Watts
Zen Mind, Beginner’s Mind by Shunryu Suzuki
Religion and Nothingness, Volume 1 by Nishitani Keiji
Philosophers of Nothingness: An Essay on the Kyoto School by James W. Heisig

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Wed, 29 Jul 2026 09:00:00 +0100

Day three of AI Week is about operations and maintenance — because once code reaches production, the operational work is still surprisingly manual: someone’s watching dashboards, deciding whether an error spike is real, and writing up what happened, every single day. Two Grafana Assistant features went GA to take that off your plate. Assistant Automations handle recurring work — save a prompt and run it on a schedule (like a morning brief on your service’s health), each run keeping its own history and able to notify you in Slack. Assistant Investigations is for when something actually breaks: it digs across metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report you can follow along with in a Workspace. You’re offloading vigilance, not accountability — the evidence is always there to review, and you still make the call.

Transcript

It’s day three of AI Week, and Wednesday is all about operations and maintenance. Because here’s the thing: once your code reaches production, the operational work is still surprisingly manual. Someone’s watching dashboards, someone’s deciding whether that error spike is real, and someone’s writing up what happened — every single day.

Today, two features of Grafana Assistant are now generally available to take that burden off your plate: Investigations and Automations.

Assistant Automations handle the recurring work of operating a system. Save an Assistant prompt and run it on a schedule — like a morning brief that summarizes the operational health of your service over the last 24 hours. Every run keeps its own conversation and history, and it can notify you in Slack when it’s done or needs approval.

Assistant Investigations is for when something actually breaks. Investigations digs into issues across your metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report. You can follow along in a Workspace, see what’s been ruled out, and add guidance while it works.

You’re offloading vigilance, not accountability — the evidence and reasoning are always there to review, and you still make the call.

Everything’s at grafana.ai — see you tomorrow for day four: evaluating your agents.

AI Week Day 2: Building (Grafana AI Week — Tuesday)

Tue, 28 Jul 2026 09:00:00 +0100

Day two of AI Week is all about building — the AI tools that help you write software with confidence. Two big announcements went generally available: gcx (the Grafana CLI) and the Grafana MCP server. Both connect your coding agents — Claude Code, Cursor, and friends — to your Grafana telemetry. The MCP server gives agents opinionated, ready-made tools for the common cases (querying metrics and logs, reading and updating dashboards); gcx is the more flexible, less opinionated option, built for agents from the ground up with read-only guardrails and its own bundle of agent skills. Both work for OSS and Grafana Cloud. The point: when an agent writes your code, it shouldn’t be guessing — it should be looking at how your system actually behaves in production. That’s telemetry-driven development.

Transcript

It’s day two of Grafana Labs’ AI Week, and Tuesday is all about building — the AI tools that help you build software with confidence.

Today we have two big announcements: gcx and the Grafana MCP server are both now generally available.

Both of them connect your coding agents — think Claude Code or Cursor — to your Grafana telemetry. So what’s the difference?

The Grafana MCP server gives your agent opinionated, ready-made tools for the most common use cases: querying your metrics and logs, reading and updating dashboards, that kind of thing. You can run it yourself, or use the hosted MCP server with your Grafana Cloud stack — so yes, it works for both OSS and Grafana Cloud.

gcx is the Grafana CLI, and it’s the more flexible, less opinionated option. Agents can explore it and build their own workflows, and it was designed for agents from the ground up, with read-only guardrails so your agent can’t touch anything you didn’t approve. It works with Cloud stacks and self-managed instances, and it even ships with its own bundle of agent skills.

Why does this matter? Because when an agent writes your code, it shouldn’t be guessing — it should be looking at how your system actually behaves in production. That’s telemetry-driven development.

Head to grafana.ai for everything we’re shipping this week, and I’ll see you tomorrow for day three.

AI Week Day 1: Planning (Grafana AI Week — Monday)

Mon, 27 Jul 2026 09:00:00 +0100

It’s day one of Grafana Labs’ AI Week, and Monday is all about planning — getting the clarity you need on what’s going on in your system so you can figure out what to build. Kicking things off is Workspace for Grafana Assistant: a full-page view of Assistant so your conversations and generated graphs aren’t crammed into the little right-hand sidebar. It’s a much less cluttered place to run investigations and to be agent-first. On top of that, Assistant is now easier to try — enable it in Grafana 13 for OSS without linking a Grafana Cloud account, or try it with no signup or download at all on Grafana Play.

Transcript

Hi everyone! I’m Nicole van der Hoeven and I’m really excited to welcome you to AI Week. Today is Monday, which is planning day, where we give you the tools you need to get clarity on what even is going on in your system so that you can figure out what to build.

Now we’re kicking things off with a feature called Workspace for Grafana Assistant. Now if you’ve used Assistant before, you’ll know that it comes up on the little sidebar on the right side of Grafana. Well, that can start to get cramped if you have multiple conversations or if you’ve asked Assistant to generate multiple graphs for you. Well, we’ve heard you, and now we’re giving you a full page view of Assistant. So you’ll have all of your conversations on the left, and this is also an excellent place to run investigations from. You can also zoom in on any of those graphs, and it’s just a much less cluttered view if you want to be agent-first.

Speaking of Assistant, the second thing I want to tell you is that you know, Assistant was already available everywhere. But now, Assistant is available… in more places than everywhere? And also more easily. So Assistant was already available for OSS, but you had to link your Grafana Cloud account, which was not so handy if you wanted to try it out. But not anymore. Now if you download Grafana 13 and enable Assistant, then you’ll just be able to go and try it out and chat with Assistant without doing anything else. It’s still not free or open source, but it IS a lot easier to try it out.

And if you don’t want to download anything, that’s what Grafana Play is for. Grafana Assistant is now available to try out on Grafana Play without any signup or any download required. Try it out at play.grafana.org. I really love seeing it there. Tune in tomorrow where we’re going to talk about everything we’re announcing to help you start to build.

My reMarkable Obsidian workflow

Fri, 03 Jul 2026 17:00:00 +0100

I’m a big note-taker — but what most people don’t realise is that digital notes have only ever been a small part of that. I love analog notes, and lately I’ve loved meshing the analog with the digital using E Ink tablets. The problem with the reMarkable has always been the workflow: there hasn’t been an easy way to get everything back into Obsidian, which — as you know — is kind of important for me.

In this video I show how there’s finally a clean way to tie it all together: reMarkable → Obsidian as a PDF → searchable, OCR’d text.

The setup: a lot of E Ink

I’ve tried a lot of E Ink tablets and readers. My everyday carry is the reMarkable Paper Pro Move, and I keep a full-size reMarkable Paper Pro at my desk. I love them more than the Kobo or Kindle Scribe lines — but the deciding factor was always going to be whether I could get my handwriting into Obsidian without friction.

Step 1 — Sync the PDF into Obsidian

The piece that makes this work is a community plugin: reMarkable Sync by Tim Dommett.

In Obsidian, go to Community plugins → Browse, search for reMarkable, install and enable it.
The first time, you’ll register the plugin — it sends you to your reMarkable page, gives you a one-time code, and you paste that in and hit register.
In settings, choose a sync sub-folder. I put mine under plugins/reMarkable.
Set the sync interval — I have mine sync every 30 minutes, but you can go more frequently, manually, or trigger it on demand from the command palette (reMarkable Sync: Sync documents).

Once that’s done, anything I write on the reMarkable lands in my vault as a PDF — diagrams, cursive, print, colours and all. In the video I write out a page (neural nets, as explained by my partner Noel), trigger a manual sync, and watch it appear in Obsidian.

Step 2 — OCR the handwriting into real text

A PDF in Obsidian is a start, but it isn’t searchable or editable. That’s where OCR (optical character recognition) comes in — turning the handwritten image into text that behaves like any other note: searchable, linkable, modifiable. I keep the PDF too (I like preserving the colours and the handwriting, especially when someone else wrote on my tablet), but I want the durable text version as well.

I automate this with OpenClaw — my AI agent, Iris, runs a skill on a heartbeat that watches my reMarkable folder, notices new files as they drop in, and OCRs them automatically. It even fills in properties I like and, because it knows my context (and sometimes has the transcript too), it can describe diagrams more completely than I’d expect. It correctly read “Neural nets as explained by Noel” and “get diagrams from WhatsApp” — whether I wrote in print or cursive.

That setup is admittedly overkill, and not something I’d recommend to everyone. So in the video I also show the more accessible version:

The accessible alternative: Claude Cowork / Claude Desktop

If you’d like to automate OCR without a full agent setup, the Claude Desktop app (with Cowork) is a great option — especially if you already keep notes, since your vault is exactly the kind of context that makes AI immediately useful.

The prompt I use, roughly:

You are in my Obsidian vault, in the folder plugins/reMarkable. Look for the very latest PDF I’ve put in there and OCR it. If there’s a diagram, describe the diagram. If there’s text — handwritten cursive or print — transcribe it. Print all of your findings into a note in the plugins/Claude output folder, using the same file name as the original PDF.

It reads the latest PDF, describes the diagram (it got “neuron” and “perceptron” — words that were never actually written), and writes the note. Interestingly it did as well as my own setup, because under the hood we were both using the same model (4.8).

Then turn it into a skill. I tell Claude: “Please turn all of these instructions into a skill and call it OCR.” It writes the skill, preserving the paths, and I hit save. Now next time I have a reMarkable PDF to transcribe, I just run /OCR and name the file. If you want to go further, you can even ask it to make this a scheduled task.

(Tip: I use Spokenly to dictate these instructions to Claude rather than type them.)

Why this matters

This simple workflow has made it so I can actually use my reMarkables. I love typing notes on my laptop, but when I’m face-to-face with someone — in a talk, at a conference — a screen between me and the speaker feels distancing, and a little rude. I feel more present without it. And the little Paper Pro Move is far easier to carry than a laptop or iPad.

Now that I know everything ends up in Obsidian anyway, I get to pick whatever note-taking method I feel like in the moment — typing, reMarkable, or fully analog. That same OCR process (minus the reMarkable Sync plugin) also works for OCRing photos of pages from my paper traveller’s journal.

If you’d like to go deeper on how I use AI in my note-taking, check out my AI note-taking workflows video — I cover the different audio transcription tools I use there too.

Arigatō gozaimashita! Mata kondo ne~

Resources

🖊️ reMarkable Sync by Tim Dommett — the Obsidian community plugin that syncs reMarkable documents into your vault as PDFs
🧠 My notes on OpenClaw — the agent runtime Iris runs on
🎥 My AI note-taking workflows — the companion video on transcription and AI capture
🗣️ Spokenly — the dictation app I use to talk to Claude
💛 My Patreon

An Overview of AI Evaluation (The Context Window #05)

Fri, 26 Jun 2026 18:00:00 +0100

How do you know if an AI agent is actually any good — and whether the change you just made to it made it better or worse? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Yas Ekinci, an engineer on Grafana’s AI team who has spent the last several months focused almost entirely on evaluation. We talk about why trusting AI is so hard (non-deterministic models, oversized PRs, and the shift from writing code to reviewing it), the difference between online and offline evals, how we actually test Grafana Assistant, and o11y-bench, the open observability benchmark we announced at GrafanaCON. We recorded this one ahead of Yas’s two-month leave — so consider it a deep dive banked for the road.

The “final answer problem”

A recurring theme — and one close to my testing heart — is what I call the final answer problem. As a tester I’m used to looking at an outcome and asking “did this pass or fail?” But with AI, a plausible-looking answer isn’t enough. If the process was wrong, you can’t trust the outcome even when it looks right — and a plausible-but-wrong answer is the dangerous case, because nothing on the surface tells you it’s wrong. Yas’s example: a broken Tempo tool that returned no spans, while the agent still answered convincingly by guessing from metadata. Plausible ≠ correct.

This is also why the bottleneck has moved. The cost of writing code has collapsed, so the burden now sits on review. Developers, as I like to put it, are becoming shepherds of intent — less about typing the code, more about making sure the intent is clear and faithfully implemented.

What evals are, and online vs offline

An eval is simply a test of whether an AI agent behaves the way you want — from “who are you?” → “I’m Grafana Assistant” all the way up to multi-step querying and dashboarding. The point is to know with confidence whether a change made the agent better, which is what lets you improve it consistently over time.

There are two complementary flavours, and they map neatly onto testing concepts we already have:

Online evals — scoring a sample of real production conversations, usually with an LLM-as-a-judge. Great as a pulse check (it’ll catch a quality drop when a model provider silently swaps models), but it only tests the deployed variant and is hard to verify. Closest analogy: traditional observability / RUM.
Offline evals — a “golden set” of tasks run during development and CI against a controlled, reproducible environment. You can test every change, prompt, and implementation, and drill into specific skills (PromQL vs LogQL, dashboarding). Closest analogy: integration testing.

How we test Grafana Assistant

Assistant evals run a real agent-plus-tools harness against a real-ish environment (a Grafana with data sources, alerts, and dashboards), then grade the full transcript with “graders” or “verifiers.” The hard tension is that the environment has to be real and reproducible at the same time. Because Assistant is a front-end plugin that runs some tools in the browser, the team uses Playwright to replicate real browser behaviour — which is where I made my pitch for k6’s browser module (now approaching Playwright parity) as a dogfooding option. The internal eval tool is a CLI called LLM Spec, now on its third major iteration after the lessons from o11y-bench.

o11y-bench

o11y-bench is our public benchmark, announced at GrafanaCON, with six task categories: dashboarding, investigation, logs, metrics, and traces. A few things I think make it genuinely interesting:

It’s built on the Harbor framework (also used by terminal-bench), which separates harness from environment from tasks — so you can swap harnesses and compare, say, gcx + Claude Code against the Grafana MCP.
The leaderboard includes open models (Qwen 3.6, Kimi), not just the frontier labs.
It distinguishes pass@3 (got it right at least once in three runs — rough capability) from pass^3 (got it right all three times — reliability). The 10–20% gaps between the two are the whole reliability story in a single number.
It runs against a synthetic environment (Prometheus, Loki, Tempo in Docker) with generated data, so the correct answer is known in advance — and it uses fact-based rubrics, where the grader runs its own known-good queries against the environment and checks the agent’s answer against them. That’s how you catch the subtle, plausible-but-wrong failures.

The internal eval set is a superset that adds scenarios drawn from real (private) conversations and Assistant-specific behaviour — like Slack’s not-quite-markdown formatting — while o11y-bench stays generic and harness-independent, grading on outcome rather than method.

The evaluation loop

Yas’s blog post (the first engineering post on Grafana’s new community Unprompted blog) and our GrafanaCON talk both center on the evaluation loop: measure → make changes (new tools, prompt tweaks, bug fixes) → measure again. The crucial bit is that the loop can be driven by a coding agent — via an LLM Spec Improve skill — because you can’t unit-test natural-language “programming” the way you test code. There’s no clean relationship between the instructions you write and what the model does, so the only way to know the impact of a change is to run the loop.

Where it’s going

We closed on open questions and wishes: higher environment fidelity, more and harder tasks (very likely an o11y-bench 2.0), and getting more contributors — including PMs writing evals to capture desired product behaviour. I floated a k6-extension-style community model for evaluators, and using o11y-bench to support bring-your-own-model, so users can see exactly where a smaller or local model falls short instead of assuming Assistant is broken. Yas’s industry hope is the neat one: if labs tune their models against o11y-bench, observability and Grafana performance improve across the board — free fine-tuning out of the box.

Resources

Transcript

Nicole: Hey everyone. Welcome to another episode of The Context Window. This is a semi-regular podcast — we try to make it every month — and it’s all about the things going on with AI at Grafana. Usually this is live, and you might think looking at this that it’s live, but it’s actually prerecorded, because Yas has a busy schedule ahead and I wanted to get him on here before he left. So hi, I’m Nicole van der Hoeven, a senior developer advocate at Grafana Labs, and I’m assigned to the AI team these days. And one of the people on the AI team is — this person over here. How do you say your last name, actually?

Yas: Oh, it’s Ekinci.

Nicole: Ekinci. I was totally confused because I just found out you speak Dutch. Did not know that.

Yas: I’m based in Belgium, and I was like, “Nicole van der Hoeven” — okay, that sounds familiar enough to me.

Nicole: Yeah, it’s the name of my ex-husband, but it’s okay, we still like him. So tell us what you do on the AI team.

Yas: I’m an engineer on the AI team — I’m getting to five years at Grafana next week. When I joined, this was pre-AI; we had the machine learning team building predictive models, forecasting, that kind of stuff. As the AI wave came, we all got absorbed into this newer, larger AI department. It’s been quite varied — the assistant, the AI agent inside Grafana — but especially the latter half of this year I’ve been more and more focused specifically on evals: how do we figure out the quality and reliability of these things we’re deploying?

AI is a very useful tool. It has utility — it’s not like NFTs. We use it day to day. My job has changed drastically over the last two years. But you still notice it needs guidance and sometimes has reliability issues. The trend is that the guidance has come down — a year or two ago I’d steer an agent 50, 60, 70 times in a session; now it’s more like five or ten. But you still need to do it. If you just blindly vibe-code your way out of this, you’re not going to end up in a good place.

Nicole: It moves so quickly. Whenever someone tells me they tried AI and it didn’t do what they wanted, I ask, “Well, when was that?” Because if it was six months ago — try it again. In this episode we’ll talk all about evals, but first, some announcements. Assistant is now available for OSS and self-hosted Grafana, and I made a video showing how to do that. There’s also a new Medium blog for Grafana Labs called Unprompted — a community-focused blog — and Yas wrote the first engineering post, about building an evaluation loop. And we’ve started AI weekly videos: the AI team has a weekly meeting, Mat Ryer had the idea to publish excerpts, usually the demo part. Just be warned we demo things that may never make it to production — you’re getting it raw, warts and all.

Yas: I need to make sure I don’t wear my PJs anymore for the AI weekly.

Nicole: A bit of quick AI news: Warp is now open source. There are new models — GPT-5.5, Gemini 3.1 Ultra with a two-million-token context window, Anthropic’s Claude Mythos security preview. And there’s a great article by Matt Shumer called “Something Big Is Happening,” where he says, “I am no longer needed for the actual technical work of my job” — which hits hard — and advises leaning into what’s hardest to replace.

Yas: It’s still that guidance thing — you need a human in the loop to guide it. But if you ask me how much code I’ve manually written lately, the actual technical piece is very close to zero. We’ll talk about how that’s different for evals, because evals don’t work as well as coding does there.

Nicole: So why is it so hard to trust AI? Why don’t we just YOLO it and accept everything?

Yas: A mix of things. Everybody knows these models are non-deterministic and not perfect — these aren’t new problems; underneath it’s all machine learning, numbers and math and stats. Every time you talk to it you might get a slightly different response. The second part is it writes a lot of code. You can 10x your output — send ten agents at ten different features and launch all those PRs — but then someone needs to review them, and the PRs are typically big and over-engineered. One thing I always do at the end of a piece of work is ask it to find a simpler way and simplify. Bigger PRs plus so many changes makes it hard to keep a mental model of the whole thing. And interestingly, I’ll use AI to help review a PR — which gets you back to the same problem: if you use AI to review the AI, how much can you trust that?

Nicole: I like the phrase that developers are becoming shepherds of intent — we’re not writing the code so much as making sure the intent is clear and implemented. And this is the difference between process and outcome — what I call the final answer problem. I come from performance testing; I’m used to looking at the outcome and asking did it pass or fail. But now, even if the outcome looks right, if the process was wrong I don’t trust it. And the reverse too: right process, wrong outcome, something else is broken.

Yas: Tests are interesting here, because most of those tests are also AI-generated — and the AI doesn’t shy away from just changing the test to make its branch pass. For any branch I review, I still look at all the changes; I won’t blindly rubber-stamp. That’s probably the biggest bottleneck right now.

Nicole: One approach to trusting AI is human evaluation — thumbs up/thumbs down, or something like LMArena where you compare two answers blind. Useful, but not scalable; you can’t run it in CI.

Yas: On the eval side they’re all complementary. So, what is an eval? It’s testing whether the AI agent behaves like you want. A simple one: ask “who are you?” and expect “I’m Grafana Assistant.” There are much more complicated ones around querying and dashboarding. At its core, an eval is how you figure out if the agent is any good — and the reason we do it is so that when we make changes, we know if it’s better. Answer that with confidence and you have a path to improve consistently over time.

We split into online and offline. Online — we recently launched AI observability for agents — is about measuring how the deployed agent does against real production conversations from real users. Valuable, because you get a score based on real behaviour, but hard to verify; we typically use an LLM as a judge, so it’s not as robust. Offline is used more during development, in CI — much more controlled. You have a golden set of evals covering the behaviour you expect in production, and because you always test the same set, you can see whether changes make it better, and dig into PromQL vs LogQL, or dashboarding. Online is a great pulse check — it can catch quality dropping when Anthropic swaps a model — but it only tests the deployed variant. Offline can test all your changes, your implementation, and all your prompts.

Nicole: It took me a while to translate these, because the testing industry already has this — we call it pre-prod and production testing. Pre-prod is everything from unit to system integration testing in dev and staging; you can really hammer the thing. Production you have to be more careful — synthetic monitoring, real user monitoring — same idea with an AI flavour.

Yas: Definitely. Online is much more like traditional observability — you’re observing a system that happens to be powered by AI, turning natural language into a number you track over time and alert on. Offline is closer to an integration test: you give it a task, it runs the whole agent against a real environment, you get something out, and then separately you grade that outcome.

Nicole: We did an AI observability episode without you — you were at GrafanaCON — so I’ll link that. Let’s talk more about offline evals, since we haven’t covered those.

Yas: Offline is a bit of a beast. There’s a good Anthropic post called “Demystifying evals.” Back when we had simple single-turn LLM calls — incident summary was one of the first — the prompt was “here’s the context of the incident, summarize this,” and grading checked readability, completeness, appropriate length. Easy enough to set up. Agents get more complicated, because every time you run a task the agent can take very different trajectories — five steps one time, seven the next. The agent plus its tools is roughly the harness; it runs against an environment — a real Grafana with data sources, a certain alert and dashboard state — and keeps interacting using the tools until it’s done. Then we take the whole transcript and grade it with graders, sometimes called verifiers, to verify it actually got the task right.

Nicole: So how do you do it for Assistant, given how broad that environment is?

Yas: We have the public benchmark we released recently and our internal evals — same approach, different number of tasks and some different data sources. We want an environment as real as possible, but with the tension that we also need to control it so a benchmark is reproducible. If two people run it and get different scores, which one was right? We started, like everybody, with manual tests — listing data sources to confirm a tool is wired up, some CPU questions, showing logs — with a human as the judge. That’s fine at four or five scenarios; you can’t do it as a human at 500. Assistant is a front-end plugin — some tools run in your browser, which is good for auth and security but makes testing hard — so we use Playwright to replicate real behaviour and pull the telemetry out. The internal eval tool is a CLI we call LLM Spec, and it can run against cloud instances or a local controlled environment.

Nicole: Before the environment — did you consider using k6 instead of Playwright?

Yas: Interesting — yes and no. k6 is also a tool the assistant needs to use, but our goal isn’t to load test; it’s to hook into the plugin and get the messages and telemetry out, so we have a Playwright harness.

Nicole: There’s actually a browser module for k6 going towards feature parity with Playwright.

Yas: I did not know that. Might be interesting.

Nicole: Just dogfood all the things. I’d love to work with you on that. Okay, the environment.

Yas: We test against cloud instances — for incidents especially, it’s a lot of data. And we run LLM Spec against a local controlled environment, which is quite similar to the open benchmark we published.

Nicole: You’ve mentioned o11y-bench — the benchmark announced at the last GrafanaCON. That was fast.

Yas: o11y-bench has tasks across six categories — dashboarding, investigation, logs, metrics, traces. We wanted to bring our internal approaches outward, and it was a learning opportunity: we used a framework called Harbor, also used by the makers of terminal-bench. The nice thing is it separates harness from environment from tasks — so the default harness is a simple system prompt plus the Grafana MCP, but the folks working on GCX run Claude Code plus GCX against o11y-bench and compare GCX with MCP. There’s a leaderboard with a bunch of models, including open ones like Qwen 3.6 and Kimi.

Nicole: They’re getting good. Qwen 3.6 is awesome — not top five, but for an open-weight model, impressive.

Yas: And this is the interesting part: sort on pass@3 and it looks different from pass^3. pass@3 runs each task three times and passes if it gets it right one out of three — rough capability. pass^3 means you need three out of three on every task — reliably solving it, not just getting lucky once. You see 10–20% differences between the two scores.

Nicole: That’s awesome. And this is public, right?

Yas: Yep — the repo’s on GitHub with all the tasks. We run o11y-bench against a synthetic environment — a Prometheus, Loki, and Tempo in Docker — and we generate the data, so every run uses the same data and we know what the answer should be before we even run it. We also do fact-based rubrics: separately from what the agent does, the grading side runs its own known-good queries against the environment and checks whether the agent’s answer matches. That matters because these agents fail in subtle ways. We once made a change to the Tempo tool that broke the spans — the agent got no spans but still answered, fairly realistically, from metadata. Without testing for it you’d never catch it; the text looked plausible.

Nicole: That’s exactly process versus outcome. If the answer were negative two seconds it’s obviously wrong, but a plausible answer is the dangerous one.

Yas: Exactly, and that happens a lot.

Nicole: An early Assistant once told me it created a dashboard, and it didn’t look right — turned out it couldn’t find the data source so it just used dummy data.

Yas: Same thing happened to me locally — I broke all the tools, and it still came up with tool calls it couldn’t execute because of rough instructions in the system prompt.

Nicole: So what’s missing in o11y-bench that means you still need an internal one?

Yas: Two levels. First, some scenarios are based on real conversations or our own internal environments — anything involving private data can’t go public. Second, there are very Assistant-specific things we want to test — Slack, for instance, has its own version of markdown that looks like markdown but isn’t, and early Slack Assistant struggled to write it consistently, so we wrote tests for that. The internal set is a superset; the public set is still part of it. The public set is generic observability tasks — asking questions about your data, dashboarding, basic Grafana API checks — and because the harness is independent, we can test GCX, MCP, our assistant, or any other agent against it, grading on the outcome rather than how it was solved.

Nicole: We talked about your blog post — let’s define the evaluation loop.

Yas: At its core: measure the current state, see where it works and where it doesn’t, make changes — a new tool, tool instructions, the system prompt, bug fixes — then measure again, and again. The key is that loop can be driven by a coding agent, not us manually. If your evals are robust enough to pass a clean feedback signal back, the agent can keep improving until it hits 100%, and then you make harder tasks or increase coverage. At scale it’s nearly impossible for one human to understand the whole system and make targeted improvements, because an agent isn’t programmed as code — it’s programmed with natural language, and there’s no clear relationship between the instructions and what the model does. The only way to probe that is to run a bunch of these loops.

Nicole: Is the loop equally implemented for o11y-bench and LLM Spec?

Yas: It’s implemented internally. o11y-bench is really just the measurement part — we’ve used it for GCX and MCP, but more manually. In LLM Spec we have an LLM Spec Improve skill — you pick the agent, and it runs measure-improve-measure-improve.

Nicole: Where are you going next — no promises?

Yas: I’ve done a lot on the harnesses — the Playwright stuff, and there was an LLM Spec 1.0, then a 2.0 revamp for speed and robustness, then o11y-bench, and the learnings from that went into what I’d call LLM Spec 3.0. The two big challenges now are environment fidelity — more an issue for internal evals — and the number and quality of tasks: how do we cover more of Grafana? We weren’t sure o11y-bench would be a recurring thing, but there’s a lot of interest, so we’ll probably do an o11y-bench 2.0. And I want more people contributing — I was talking at the PM all-hands about getting product managers to write evals, because the tasks capture desired behaviour: when someone asks about my product, how should the agent ideally respond?

Nicole: What I’d love is something like the k6 extension model applied to evaluators. When Grafana acquired k6 — I was on k6 — k6 had extensions and modules, and we got a lot of cool community extensions. I think evaluators would be the same. After my Madrid talk, all the questions about the evaluator section were “can we add this, can we add that, why isn’t this in the template?” — and it could be.

Yas: It would be nice to do that across both o11y-bench and the internal ones — there are open questions on making them work together; internally we’re in Go, o11y-bench is on Harbor in Python, but the tasks and most verifiers are language-independent by design. A fourth thing is getting to a better golden set over time — we haven’t yet automated going from real AI-observability conversations to verifiable offline tasks. As the product changes, your tasks might not stay relevant — imagine k6 didn’t exist and now people ask Assistant about k6. Bridging online and offline in a more automated way would be the fourth big thing.

Nicole: I also have a wish for o11y-bench beyond being a benchmark — with no internal knowledge and no promises: it could be a step toward making Assistant bring-your-own-model. One barrier is that people try Assistant with a tiny local model and conclude it doesn’t work, when really you need certain models. A standardized way to measure that and show users where their model falls short — versus where the latest Opus sits on the leaderboard — lets them make an informed decision.

Yas: I’d go beyond Assistant — we’ve got GCX and MCP too. If you want to use your own models with them, the benchmark should reflect, for observability tasks, which are the best models and the best open models. It gets to your point: people can make a more informed decision.

Nicole: Where are we still missing things as an industry, not just as a company?

Yas: Just recently a new benchmark called DeepSWE was released, and it’s a bit controversial because it flips the ranking from the most popular software-engineering benchmark — where Opus was always leading, here GPT-5.5 leads by a margin. So there’s something to be said about the reliability of the benchmarks themselves, since labs test and sometimes tune their models against them. That might actually be good for o11y-bench — I’d want a lab to tune against it, so we get free fine-tuning out of the box: as newer models use o11y-bench to tune on observability tasks, they get better at using Grafana and solving observability problems.

Nicole: Thank you — we just hit time. It was really informative, and I’m glad we both found time; this got rescheduled for various reasons. A great overview of all the evals we’re doing.

Yas: Thanks — it’s nice to step out of the day-to-day and talk about this stuff.

Nicole: And if you’re watching, leave any eval questions in the comments and I’ll bug Yas to answer them. Thanks everyone for watching.

Yas: Bye.

CLI and MCP Basics for AI Agents (The Context Window #04)

Wed, 03 Jun 2026 18:00:00 +0100

MCP vs CLI: which one should your AI agent actually use? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and three of the engineers and PMs behind Grafana’s agent tooling: Ben Sully (who wrote the Grafana MCP server), and Ward Bekker and Dafydd Thomas from the team building gcx. We get into the two main ways an AI agent can reach the outside world — the Model Context Protocol (MCP) and good old command-line interfaces — and look at when each one makes sense, the tradeoffs around context windows, tool sprawl, token cost, and reliability. We walk through Grafana’s cloud and open-source MCP servers along with gcx, the Grafana Cloud CLI built for agents, do a live side-by-side demo of both creating an SLO, and talk about how we benchmark them against each other with o11y-bench.

Timestamps

00:00:00 — Introductions
00:02:49 — The last month in AI news
00:10:19 — What is Grafana Assistant?
00:12:30 — What does it mean for Assistant to use tools?
00:13:30 — What is MCP?
00:16:56 — What is a CLI in an AI context?
00:19:17 — What are skills?
00:24:37 — Can you use a custom MCP server with Grafana Assistant?
00:25:55 — Why team MCP?
00:28:22 — The context-window cost of MCP tool definitions
00:31:50 — Tool discoverability
00:39:25 — What is gcx, and why a CLI?
00:46:13 — Demo: creating an SLO with MCP vs gcx
00:52:58 — Benchmarking MCP vs gcx with o11y-bench
00:58:10 — How MCP and gcx relate to Grafana Assistant
01:02:35 — k6 MCP vs Grafana MCP
01:03:00 — When to use which (decision guide)
01:04:29 — Adding the OSS Grafana MCP to Assistant
01:05:41 — Turtles all the way down

Resources

MCP and CLI for agents

My Context Horizon piece: Unlearnings from building Grafana Assistant

My AI note-taking workflows

Fri, 29 May 2026 17:00:00 +0100

Just over a year ago, if you’d asked me how I felt about AI and note-taking, I would have told you I didn’t want AI anywhere near my notes. The world is different now, and so am I — I rely on AI in a lot of ways, and one of the easiest places to start is speech-to-text: using AI to convert my voice into something my PKM system can actually work with.

In this video, I walk through three tiers of AI note-taking workflows — free, paid, and (a little) overkill — and show what I’m using at each level.

The framing: note-taking vs. note-making

Nick Milo draws a useful distinction between note-taking and note-making:

Note-taking is collecting — recording information, paraphrasing what an author said, highlighting.
Note-making is connecting — active thinking, engaging with the information, linking it to other ideas, working out how it fits into your PKM system.

The best use of AI right now, for me, is to help with the note-taking part. I like to start with speech because speech used to be a form of input my PKM system couldn’t take very much of. Pre-AI, voice was lossy — you’d have to transcribe it yourself or lose the thought. Now: you speak, AI transcribes it, it lands in your notes, and the AI can keep working with it from there.

Tier 1 — Free

Use your phone’s recording app. iOS Voice Memos transcribes recordings, and the Notes app does too on newer Apple devices. Samsung’s Voice Recorder app does the same on Galaxy devices. For desktop, I demo Spokenly (Mac, Windows, iOS) — drop audio in, get text out.

The friction is manual: remember to record, remember to transcribe, remember to paste.
What you get: quick notes to yourself, captured instantly.
Best for: people who want to get started as easily as possible without paying for anything.

If you want to go local-first, I’d point you at NVIDIA Parakeet and the Qwen models for offline transcription.

Tier 2 — Paid

The second tier is paying for the convenience layer. These tools collapse the first two steps — you record, the transcript appears, and you usually get extras like AI summaries or the ability to ask questions of the transcript (sometimes live).

Tools: Voicenotes, Otter.ai, Letterly, Granola, Gemini on Google Meet, Zoom AI Companion.

For Granola specifically, I use the Granola Sync Obsidian plugin plus a QuickAdd macro (QuickAdd: Macro - Link Granola) to automatically link a transcript to the right note in my vault — in the video I link a Dungeon Crawler Carl recap to its Granola transcript. (The linking script and macro are on my Patreon.)

Tradeoffs:

Your notes live in their system, not yours. Shutdowns, pricing changes, and acquisitions all take your archive with them.
The friction reduction is real, though — if it means you actually capture instead of intending to capture, that’s worth a lot.

Best for: people who want to capture meetings, games, or lectures.

Tier 3 — Overkill

This is my setup. It’s a little ridiculous; I’m going to show it anyway.

Hardware: a Plaud AI recorder. Small, clips to my phone or sits in a pocket. Always running, or started deliberately for important conversations and walks. It can run entirely locally, or upload to Plaud’s servers.
Plugin: my fork of Leonard Sellem’s Plaud Sync plugin for Obsidian, installed via BRAT. The fork adds a couple of things I needed for my own daily workflow.
AI layer: Iris, my OpenClaw-powered AI agent, watches the synced files. Every morning she reads the previous day’s transcripts, plus my calendar and daily notes, and outputs a daily summary, extracted tasks, and flags anything I’ve said I wanted to remember.

I also use “Hey Iris” memos — voice notes I record into Plaud while I’m out, addressed to Iris, that she processes the next time she runs. “Hey Iris, remind me to follow up with X.” “Hey Iris, add this restaurant to my places-to-try list.” They land in my system without me opening anything.

Benefits: I don’t think about capture at all. I just live and talk, and the relevant pieces land in my notes.

Cons: the privacy of people around me. I’m careful about when this is on, and especially careful about explicit consent.

Why three tiers, not one

The slightly absurd thing is that I still use all three levels — in parallel, for different things:

Spokenly for dictating at my desk, sometimes straight into Obsidian, sometimes into a chat.
Granola for meeting notes, and increasingly for transcribing videos I’m watching.
Voicenotes for diary-style entries.
Plaud when it’s ethical to (especially when I’m travelling with my partner and I want a clean record of conversations and ideas we had on the move).

For me, voice is the lowest-friction capture method that exists. No stopping, no finding my phone, no opening an app, no typing. Just talk — and the system handles everything downstream.

That means the gap between thinking of something and writing it down is basically gone for me. Note-taking — the tedious part — gets offloaded. So when I’m in a meeting I can be present, knowing the tool is faithfully tracking what’s said. Whatever notes I do take are then what I think about what I’m hearing, which lets me start synthesising a lot faster.

If you want more details about any of these workflows, let me know in the comments. Arigatō gozaimashita! Mata kondo ne~

Resources

🎙️ The Mechanics of Knowledge Management podcast episode where Malik interviewed me on this stuff
🧠 My notes on OpenClaw — the agent runtime Iris runs on
🎤 My notes on Voicenotes
💛 My Patreon, where I share the QuickAdd linking scripts and macros

Grafana Campfire 🔥 — GrafanaCON 2026 recap

Thu, 28 May 2026 17:45:00 +0100

I co-hosted this month’s Grafana Campfire with Stephanie Hingtgen, where we recapped everything that came out of GrafanaCON 2026. Stephanie owned the Grafana 13 walkthrough — Git Sync GA, Dynamic Dashboards GA, Dashboards API v2 GA, Grafana Advisor GA, SCIM GA — what she described as the year Grafana stopped being beta in production. I covered the AI section (Grafana Assistant now in OSS and Enterprise, the three scaling challenges from “Scaling Grafana Assistant”, Investigations, Skills going GA), the three-teams-one-architecture push across Loki Thor, Mimir 3.0, and Pyroscope 2.0, the OTel + Alloy roadmap, alerting, k6 v2.0, and the Marketplace and Logline.

We also took questions from the Champions community on the pace of UI change, Mimir Alertmanager, and the agent-scale question — what happens when your AI agents are themselves the noisiest users of your observability stack? — and threw in a flavour break about the TamaGROTchi and the ocean digital twin demo.

Resources

Asimov's Zeroth Law of Robotics: Observability for AI (ExpoQA Madrid 2026)

Tue, 26 May 2026 10:00:00 +0200

This week I gave the opening keynote at ExpoQA Madrid 2026 in Madrid, Spain. I presented a talk called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously given at KubeCon EU 2025 in London, at Dutch Cloud Native Day 2025 in Utrecht, and at Newcrafts in Paris. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

This was a keynote slot at a testing-focused audience, which gave me the chance to lean more into the evals vs. observability framing — what evals can and can’t tell you about an AI system in production, and how distributed tracing fills the gap that benchmarks leave behind. The demo app (a Flask-based “play.py” instrumented with OpenTelemetry, flowing into Tempo for traces, Prometheus for metrics, and Loki for logs) is still on GitHub, and I’ve been refining it with each iteration.

Here’s the recording of the talk:

Resources

Demo app on GitHub

Zero to Dashboard with Grafana Assistant and the Infinity datasource plugin

Fri, 22 May 2026 17:00:00 +0100

What if you didn’t have to write a single query to get a useful dashboard?

In this short video, I go from nothing to a working dashboard in a few minutes — no queries — using Grafana Assistant and the Infinity datasource plugin. The data I’m visualising is the RAWG video game database API, so I end up with a dashboard that helps me decide what to play next.

Why Infinity is fun here

Most Grafana data sources are scoped to a specific kind of backend — Prometheus, Loki, a SQL database. Infinity is different: it lets you treat any REST/JSON/CSV/HTML endpoint as a data source. That means anything with an API can become a dashboard, which is exactly the right shape for goofing around with a games API on a Friday afternoon.

Pair Infinity with Assistant and you don’t even have to know what queries to write — Assistant configures the data source, asks the API the right questions, and builds the panels for you.

Timestamps

00:00 — Intro
00:35 — The RAWG video game database
01:01 — Setting up the Infinity datasource
05:10 — Customising the dashboard with Assistant
09:31 — Closing thoughts

Resources

Step-by-step blog post: using Assistant with the Infinity datasource
Grafana Assistant documentation
Infinity datasource plugin
RAWG video game database API
Get started with Grafana Cloud (free tier)
My notes on Grafana Assistant — everything I’m learning as I work on it

Do Androids Dream of Second Brains?

Mon, 18 May 2026 07:00:00 +0100

We often describe our personal knowledge management systems as “second brains” — but can we really say they think for themselves? In this video, I make the case for instrumenting your vault with an observability stack and layering AI on top, not because it’ll turn your notes into a brain, but because it might get us closer than anything else has.

This is the video version of the talk I gave at PKM Summit 2026 in Utrecht earlier this year, with the live demos cleaner and the dashboards working without the conference wifi gods getting in the way.

What I cover

🐑 Why “second brain” is the wrong framing

I take a small detour through Philip K. Dick’s Do Androids Dream of Electric Sheep? — the inspiration for Blade Runner — and the Voigt-Kampff test, which measures empathy through observed responses to staged scenarios. It’s a useful lens for asking what we actually mean when we say a knowledge system “thinks”: we’re really asking whether it can respond to us in a way that surprises us. Most second-brain setups can’t, yet.

📊 Observability for PKM

Observability is a concept I’ve borrowed wholesale from how we monitor software systems — the idea that you can understand a complex system by what it emits, not by reading its internals. Three steps: instrumentation, analysis, improvement. I walk through what each one looks like for a personal vault:

Instrumentation: a small script that emits metrics about my Obsidian vault — note counts, link density, growth over time, what folders are loud and which are quiet
Analysis: a Grafana dashboard sitting on top of Loki + Prometheus, surfacing things I genuinely didn’t know about my own vault
Improvement: the part where seeing the data actually changes how you work

There’s a live dashboard tour, and a section where I ask Grafana Assistant to edit the dashboard for me in natural language — including writing the PromQL queries I never want to write by hand.

👁️ Meet Iris

About halfway through the video I introduce Iris — the AI agent I built using OpenClaw that lives in my vault, reads my notes, and works alongside me. I walk through:

What “agentic AI” actually means in practice (vs. just chatting with an LLM)
The OpenClaw setup — Docker sandbox, Syncthing, the whole pipeline
Iris rewriting my observability map of content
Iris doing talk-topic analysis on my vault
Iris running benchmark research for a blog post
Iris OCRing my reMarkable handwriting

🌙 Dreaming

The last section is the one I had the most fun with. I argue that the thing most second-brain setups are missing isn’t more notes or better search — it’s synthesis. The capacity to take everything you’ve captured and surface patterns, contradictions, and ideas you hadn’t formed yet.

I introduce OpenClaw’s dreaming feature: a memory consolidation process inspired by how sleep stages work in the brain. Light sleep, REM, and deep sleep each do different work — surfacing recent material, finding unexpected connections, and consolidating long-term memory. I show how each phase actually behaves in my own system.

And there’s a Zhuangzi reference. (You’ll see.)

Resources

🎤 PKM Summit 2026 talk recap (with photos)
🎞 Slides
👁️ OpenClaw
📓 Obsidian
🔄 Syncthing
📊 Grafana

The Architecture of Forgetting — the essay version of why memory consolidation matters
Unlearnings from building Grafana Assistant

If you want to follow more of how Iris works and what I’m learning about AI + PKM, Context Horizon is where I write about it.

The Architecture of Forgetting

Thu, 07 May 2026 16:00:00 +0100

Originally posted here on Context Horizon.

In early April, a new version of everyone’s favourite lobster, OpenClaw, was shipped with an interesting new feature: dreaming. It’s a memory consolidation feature. But to understand why it matters, we need to back up.

Early AI chatbots, like the initial GPT wrappers, had zero memory. Every conversation started fresh. You asked a question, you got an answer, and you moved on: a glorified search engine.

Somewhere along the way we began to expect more and more of our chatbots. Models acquired context windows that became their de facto short-term memory. As the context window filled up, they remembered what had been said earlier in the conversation, making them better conversational partners. Still, it was ephemeral: close the tab, start fresh. And it had a hard ceiling: hit the limit, and the oldest things fall off. This felt like forgetting because it was. But it was also, accidentally, a form of curation. Finite space forced priority.

Then came the push for real persistence. RAG pipelines, vector databases, memory files, agent frameworks that write notes to themselves. Now our agents can remember that we prefer the Oxford comma, that we just started a new project at work, that our D&D character is named Zuhri. This felt like the solution. More memory = better agent.

That mostly works— until it doesn’t.

As it turns out, the problem with AI memory isn’t that agents forget too much. It’s that they forget too little.

Why remembering is hard

It turns out that having a persistent memory, even for an agent like OpenClaw that has that as one of its defining features, doesn’t solve all the memory problems.

I happen to have an OpenClaw agent that named itself Iris.

Iris does a lot of things for me, including parsing my meeting notes to identify next actions, curating my tasks and projects, and generating a daily briefing. She also creates a summary of the day’s work based on my meetings, conversations, and notes. One day she was giving me the daily summary, and here’s that conversation:

Iris: …Tonsil is finally healed after weeks of pain… Me: Wait what? Where did you get the tonsil stuff? Iris: From your memory file — it’s in the Apr 10 daily note context: “Tonsil finally healed — Nicole no longer in pain 🎉”. I’ve had that sitting in memory since the Apr 10 summary. Is that wrong? Was it something else that healed, or is the tonsil detail off? Me: I have no idea where you got that. I have never had issues with my tonsil.

Where did that come from? I’m still not sure, but I occasionally have meetings in cafés, and it’s possible that a meeting recording overheard the conversation of someone else in the café. That got faithfully transcribed, copied over into my notes, and written into Iris’s memory. Iris treated it as fact for days.

The problem isn’t that Iris forgot. It’s that she remembered the wrong thing… and didn’t know how to forget it.

Imagine this at scale: not one wrong fact, but dozens, quietly shaping every summary, every recommendation, every task Iris surfaces for me. The signal-to-noise ratio degrades until it doesn’t matter if Iris remembers mostly accurate memories— I won’t trust it.

So how do you build a memory system that knows what to keep? Researchers have been working on this longer than you might think.

Meet the residents of Smallville

Stanford researchers Park et al. released a charming study in 2023 called Generative Agents: Interactive Simulacra of Human Behavior in which they talk about how they created a simulated environment called Smallville, populated it with 25 generative agents, and then watched.

Source: Park et al. (2023)

The study was an attempt to simulate emergent social behavior, but the key insight I took from it isn’t the sim itself— it’s what they had to do to the inhabitants of Smallville to enable the rich social interactions they wanted to observe.

The researchers had to give the inhabitants a way to remember.

Above is a diagram of the memory architecture of every generative agent.

Observation: The agent perceives new information. The new information is saved into the memory stream, a database that serves as the record of everything an agent has experienced.
Retrieval: When the agent is looking for information, a retrieval function takes that information from the memory stream and passes it onto the agent.
Action: The agent, armed with retrieved information, can decide to act.
Reflection: Patterns are synthesized from the observations and written back to the memory stream.

To determine which memories to retrieve at any given point in time, the researchers scored memories according to three weighted factors:

Recency: when a memory was observed
Importance: how significant the memory was
Relevance: how closely related a memory is to the input memory (the agent’s current context)

The top-ranking memories that fit within the agent’s context window were selected to be retrieved and used with the prompt.

The researchers also introduced a second type of memory: reflection. Reflections were generated when a threshold of important scores was reached per day, which typically happened 2-3 times a day.

Crucially, reflection is a compression step — the agent doesn’t just accumulate observations, it periodically steps back and asks “what do these observations actually mean?” That reflection pass is where low-signal memories get subordinated to higher-order patterns. That’s the first instance of intelligent forgetting.

The Smallville paper was published in 2023. Three years later, the same insight shipped in a lobster.

Dreaming of electric lobsters

Every system that has solved the memory problem has solved it the same way: not by storing more, but by knowing what to throw away.

Your brain does it while you sleep by consolidating important experiences into long-term memory, letting the rest dissolve. Some programming languages do it with garbage collection, periodically scanning for unused objects and reclaiming that space. Databases do it with compaction: Loki, Cassandra, and other LSM-tree-inspired systems periodically merge and discard obsolete data so reads stay fast.

In all cases, the architecture of good memory is really the architecture of intelligent forgetting. OpenClaw’s dreaming feature is the same insight applied to agents.

Diagram of OpenClaw’s internal memory architecture. The default components (“Awake”) are in black, and the new components are in blue (“Dreaming”).

Note that without dreaming, there is no automatic promotion from the active context window to daily notes and then to long-term memory. When a new bit of information is mentioned, the agent has to decide whether it should be added to the daily note and/or the long-term memory file manually. Dreaming, by contrast, is an automated process.

It’s called “Dreaming” because the feature works similarly to the biological process, as if in three sleep phases:

Light phase: Creates a session corpus from session files and then succeeding sweeps tally how many times a chunk has been recalled in a session (a counter called lightHit).
REM phase: Spots patterns and themes from chunks and adds a reinforcement signal (remHits).
Deep phase: Ranks the candidates by several factors and promotes those that exceed a threshold score to long-term memory in MEMORY.md

Just like in the Smallville simulation, OpenClaw ranks information by several categories. Here’s the table of signals and weights from the OpenClaw docs at the time of writing:

Signal	Weight	Description
Frequency	0.24	How many short-term signals the entry accumulated
Relevance	0.30	Average retrieval quality for the entry
Query diversity	0.15	Distinct query/day contexts that surfaced it
Recency	0.15	Time-decayed freshness score
Consolidation	0.10	Multi-day recurrence strength
Conceptual richness	0.06	Concept-tag density from snippet/path

Only candidates that have high scores according to these signals graduate to be part of MEMORY.md.

How dreaming helps with forgetting

The new dreaming architecture doesn’t ever delete things, technically. The forgetting is more passive. However, it’s still an improvement in a few ways:

Low-signal chunks don’t get promoted: Anything that doesn’t cross the deep phase thresholds simply never reaches MEMORY.md. It exists, but it’s functionally invisible to future sessions.
Score competition: The staging index has finite attention, so chunks compete against each other for recall signal. A high-frequency chunk that keeps getting surfaced crowds out low-frequency ones. Relative obscurity is functional forgetting.
Recency decay on scoring: The deep phase ranking includes a recency signal. So something that was frequently recalled two weeks ago but hasn’t come up since will score lower than something with fewer total hits but recent activity. Time degrades relevance passively.

What’s notable: all three are selection mechanisms, not deletion mechanisms. The corpus files still exist. The staging index entries still exist. Nothing is removed… it’s just progressively less likely to be chosen. This is forgetting as diminishing probability of retrieval, not erasure.

That’s the garbage collection analogy again: the objects are still in memory, they’re just unreachable.

Transparency

Dreaming also adds a layer of transparency. Aside from the session corpus and staging files, it also produces some output about its decisions. All three dreaming stages create files per day that live in .openclaw/workspace/memory/dreaming/{phase}/YYYY-MM-DD.md.

Here’s an example of an entry in the light directory:

# Light Sleep
- Candidate: "Don't Call It Eval" blog post (Context Horizon): `Dont Call It Eval - Blog Post Outline.md` — full outline with 7 open questions; `Dont Call It Eval - Draft.md` — 2,550-word draft, 6 sections; `Dont Call It Eval - Opening Research.md` — Tier 1/2/3 shortlist of popular AI eval po
- confidence: 0.62
- evidence: memory/2026-04-19.md:13-15
- recalls: 0
- status: staged
- Candidate: "Don't Call It Eval" blog post (Context Horizon): **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since."
- confidence: 0.62
- evidence: memory/2026-04-19.md:17-17
- recalls: 0
- status: staged

This entry identified a blog draft I was working on.

REM sleep goes through these candidates and highlights when it spots themes:

# REM Sleep
### Reflections
- Theme: `assistant` kept surfacing across 532 memories.
- confidence: 1.00
- evidence: memory/2026-04-12.md:27-29, memory/.dreams/session-corpus/2026-04-17.txt:26-26, memory/.dreams/session-corpus/2026-04-17.txt:28-28
- note: reflection
### Possible Lasting Truths
- "Don't Call It Eval" blog post (Context Horizon): **Datadog Bits AI eval platform** (Apr 7, 2026) — "Bring the noise" = noise injection = production-like envs from 90s QA. SECTION HEADERS are gold ("When one improvement caused subtle regressions", "Why tool-level testing and live [confidence=0.58 evidence=memory/2026-04-19.md:30-33]
- "Don't Call It Eval" blog post (Context Horizon): **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since." [confidence=0.58 evidence=memory/2026-04-19.md:17-17]
- "Don't Call It Eval" blog post (Context Horizon): **Title:** "Don't Call It Eval" (Nicole's original; I picked it over "Congratulations, you've invented testing") [confidence=0.58 evidence=memory/2026-04-19.md:27-27]

In this case, REM sleep correctly identified that I’m currently working on the Assistant feature at work (which had already been promoted to long-term memory) and the blog post that the light sleep phase had identified.

Here’s the same day’s deep sleep file:

# Deep Sleep
- Ranked 10 candidate(s) for durable promotion.
- Promoted 3 candidate(s) into MEMORY.md.

And here’s the addition to MEMORY.md for the same day:

## Promoted From Short-Term Memory (2026-04-26)
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:17:17 -->
- **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since." [score=0.926 recalls=0 avg=0.620 source=memory/2026-04-19.md:17-17]
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:27:27 -->
- **Title:** "Don't Call It Eval" (Nicole's original; I picked it over "Congratulations, you've invented testing") [score=0.926 recalls=0 avg=0.620 source=memory/2026-04-19.md:27-27]
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:12:12 -->
- Three files in openclaw-output: [score=0.894 recalls=0 avg=0.620 source=memory/2026-04-19.md:12-12]

There’s also an entry in DREAMS.md for the entire sequence, which is more dreamlike and human-readable. The per-day Deep Sleep file is the structured internal audit trail; DREAMS.md is the rolling, more narrative log for humans.

This level of transparency and reflection is a welcome change when you compare it to what I had before, which was nothing. Iris had to manually decide on the spot what had to be written where, and those decisions were not logged anywhere.

The tonsil dilemma

So dreaming could have prevented the tonsil hallucination, right? Well… no.

In this case, the idea of my supposed tonsil pain had come from an external source: my meeting recordings. Iris had faithfully manually added that to the daily note for the day, then decided to call on it a few days later, unprompted, in conversation with me.

With dreaming, Iris still could have made those decisions. But in fact, even if she hadn’t, dreaming could have made it worse:

Light phase sweeps it and scores it well since it’s recent and distinctive.
Tonsil pain could be included in queries about health, such as the daily health tracker that is part of Iris’s tasks. REM phase sees a pattern and increments remHits.
Deep phase ranks it highly and it crosses the promotion threshold.
It gets written to MEMORY.md and gets loaded every session. And long-term memory is sticky. Only manual intervention (by me or by Iris) removes things from that file.

Even dreaming doesn’t solve everything. Dreaming adds a mechanism for forgetting. It doesn’t prevent OpenClaw from remembering the wrong things in the first place.

The residual layer

Every memory architecture we’ve looked at — biological sleep consolidation, garbage collection, LSM compaction, Smallville’s reflection pass, OpenClaw’s dreaming — has the same shape: automated compression plus a residual human layer. Your brain has you. GC has the programmer. Iris’s dreaming has me. The human in the loop isn’t a bug in the design; it’s load-bearing.

What’s worth keeping

Dreaming is a genuine step forward for OpenClaw’s memory architecture. The mechanism of passive forgetting isn’t perfect, but the automated promotion systems do add a necessary layer of periodic reflection.

But there are limitations. Dreaming can’t yet distinguish between a memory worth keeping and a hallucinated tonsil. It can only measure how often something gets recalled. The next frontier is active forgetting: automated demotion from MEMORY.md, weighted by signals like source provenance and trust. The provenance side of that work landed in v2026.4.29-beta.2 — memory now exposes evidence-kind drilldown and privacy/provenance reports, so you can at least see where a memory came from. But the demotion half — automatically down-ranking memories based on a low-trust source like an audio transcript — still hasn’t been built.

What remains, for now, is the human layer. The transparency of the dreaming process becomes a checkpoint for the human in the loop by surfacing promotion decisions. I still read the summaries. I still catch the errors. I still decide what goes into long-term memory and what gets quietly corrected. That’s not a gap in the system — it’s probably the right design. The most important memories should require a human to notice them twice.

Good memory systems are defined by what they don’t keep as much as by what they do keep.

Grafana k6 Community Call — Secrets Management & AI Script Authoring

Thu, 30 Apr 2026 16:30:00 +0100

The first k6 Community Call in over a year, and we packed it. I sat down with four of the people building k6 — Bukola Ayodele, Facundo Batista (Software Engineer), Vicente Ortega (Senior Backend Engineer), and Charlie Burnett (Product Manager) — to walk through two new pieces of work that have shipped recently, plus a look at what’s coming in k6 v2.

What we covered

🔐 Secrets management in k6 Cloud (Facundo)

A new way to store API tokens, passwords, and other sensitive values for your tests — centralized in Grafana’s secrets manager, encrypted at rest, and never visible in the UI or API once written. Your scripts reference secrets by name, and at runtime the values are injected and automatically redacted from logs.

Why this matters for load testers:

No more hardcoded credentials sitting in scripts or in Git
Easy rotation — change the secret value, every script that uses it picks up the new value, no script edits needed
Same script works across dev/staging/prod by just swapping which secrets are scoped to which environment
Works with k6 OSS locally too — your local runs can pull cloud-managed secrets when you want them

Facundo Batista demoed the full flow: creating a secret in the Grafana Cloud UI (name, description, labels — up to 10 per secret, all write-only), and using it in a script with one line. There’s also an API for programmatic management.

🤖 Grafana Assistant k6 script authoring mode (Vicente)

This is the AI-powered script generation that’s been in the works for a while — fine-tuned specifically for k6 best practices, not a generic “ask an LLM to write you a load test” workflow. The interesting bit is the context sources Assistant can pull from when generating a script:

A service name — Assistant analyzes the existing telemetry data to discover endpoints and shapes
A GitHub repository URL — pulls the code to understand the API
An OpenAPI specification — generates tests that exercise every endpoint with realistic request shapes
Existing observability data — metrics, logs, traces from your actual production traffic

Generated scripts come with proper structure, error handling, checks, and TODO comments where the model genuinely doesn’t know what to write (e.g. for auth tokens, request bodies, or anything that needs your secrets — by design, Assistant won’t fabricate those). You can also point it at an existing script and ask it to refactor or extend.

🚀 k6 v2 (Charlie)

Release candidate 1 is out, GA is very soon. The headline changes:

Extensions marketplace — community protocols and integrations
Enhanced Playwright browser support with compatible syntax
New assertions library
MCP / agent connectivity for AI-driven workflows
DNS, MQTT, and other network protocols

Migration advice from Charlie Burnett: pin your script versions, test your existing scripts against the RC before GA hits, and don’t rush.

🔮 What’s next on the AI roadmap (next 1–2 months)

Visual browser-based testing experience
Enhanced observability workflow automation
Improved user-journey creation across both personal and enterprise apps
Integration with autonomous monitoring systems

Resources

Thanks to Bukola, Facundo, Vicente, and Charlie for joining, and to everyone who showed up live.

High INT, Low WIS: What LLM Benchmarks Miss

Thu, 23 Apr 2026 09:00:00 +0100

Originally posted here on Context Horizon.

In the tabletop roleplaying game Dungeons & Dragons (D&D), the very first thing you do when you create a character is to choose ability scores. Ability scores are numbers that determine how good you are at the game’s main characteristics: Strength (STR), Dexterity (DEX), Constitution (CON), Intelligence (INT), Wisdom (WIS), and Charisma (CHA). 20 is the default maximum that a character can have in any ability, though some in-game events/items can increase that.

Here’s the thing, though: every D&D player knows that a 20 Intelligence doesn’t mean your wizard aces every check. Whether you have proficiency in the specific skill still matters. That 20 INT wizard, for all their raw cognitive horsepower, can absolutely still fail to identify a magical rune. The ability score is a starting point, not a verdict.

This is exactly what’s happening to LLM benchmarks. All the general LLM/AI model benchmarks are ability scores. They tell you something real about a model’s general capabilities, the same way a 20 INT tells you something real about a wizard. But as frontier models crowd toward the top of every leaderboard, we’re running into a problem D&D solved decades ago: the stat block isn’t the character. And it was never supposed to be.

A screenshot of the stats of my Level 10 bard. Low INT, but look at that CHA!

The stat block: A tour of the ability scores everyone’s maxing out

There are so many new models and model versions out there that it’s hard to keep up with what we’re supposed to be using for which tasks. LLM benchmarks have come to be relied on to give a somewhat objective assessment of each model’s capability. Here are a few big ones that exist.

General knowledge/reasoning

MMLU (Measuring Massive Multitask Language Understanding, 2020)

MMLU is the classic “general knowledge” INT check. It was released in September 2020 by Hendrycks et al. and contains 15,908 multiple-choice questions across 57 subjects (humanities, social science, STEM, other). However, there were a few issues with it:

a study in 2024 showed that 6.49% of those questions contained errors
The New York Times reported (2024) that data contamination was an issue — models could be trained on the MMLU questions (and answers) to inflate their MMLU scores
many frontier LLMs soon consistently achieved high scores

In response, a new benchmark called MMLU-Pro was created with 12,000 questions that each have 10 options. The increase in options theoretically required more reasoning capability than the original benchmark, but frontier models are already knocking at 90% on the MMLU-Pro.

Humanity’s Last Exam (HLE, 2025)

HLE was developed specifically to address the fact that LLMs “now achieve over 90% accuracy on popular benchmarks like MMLU” (from the HLE site). The paper by Phan et al explains the lengths they took to make it hard enough to be meaningful:

they solicited high-quality questions from nearly 1000 subject matter experts, even going as far as paying them for successful submissions (with a total prize pool of US $500,000)
they included graduate-level classics (one of the questions includes translating Palmyrene script)
they reviewed potential submissions using current LLMs and only proceeded with submissions if the LLMs are unable to answer correctly
experts then reviewed the submissions further

As a result, HLE scores are low — intentionally so:

While current LLMs achieve very low accuracy on HLE, recent history shows benchmarks are quickly saturated – with models dramatically progressing from near-zero to near-perfect performance in a short timeframe [13, 45]. Given the rapid pace of AI development, it is plausible that models could exceed 50% accuracy on HLE by the end of 2025. (Phan et al., 2025)

Software engineering

HumanEval (OpenAI, 2021)

In 2021, OpenAI released a paper and an evaluation harness for a benchmark consisting of 164 Python programming problems based on publicly available code on GitHub. It was developed to test coding in particular.

Eventually, though, experts found that the problem sets in HumanEval were insufficient and ambiguous (Liu et al., 2023). HumanEval+ was released to include more tests, but the next benchmark’s authors suggested that models might be overfitting to HumanEval+ as well, memorizing the dataset and answers rather than genuinely demonstrating reasoning ability.

LiveCodeBench (2024)

Jain et al. tried a different approach to HumanEval to avoid data contamination: it was released with a continuously-updated coding benchmark. Problems have release dates and models can only be evaluated using problems released after their training data cutoff dates.

LiveCodeBench represents a shift in benchmarks towards a more continuous, dynamic measure. However, the last submission to the benchmark was 9 months ago at the time of this writing, and the leaderboard has not been updated since then either, suggesting the benchmark is no longer being actively maintained.

SWE-bench (2024)

This benchmark was created by Jimenez et al. to include 2,294 actual GitHub issues from 12 Python projects. When it launched, it was a much harder benchmark than predecessors, with models scoring 2-4%. Today, models are scoring around 70-80% on the original benchmark. There were enough issues with the original’s dataset quality that variants were introduced, including OpenAI’s SWE-bench Verified. Today, the SWE-bench website shows these variants:

SWE-bench
SWE-bench Verified
SWE-bench Multilingual
SWE-bench Multimodal
SWE-bench Lite

Agents and tool use

This last category is a preview of where I think the field should go: benchmarks that measure what a model does rather than just what a model knows. Here are some of them:

GAIA (Mialon et al., 2023) — 466 real-world assistant tasks that combine web browsing, file reading, and multi-step reasoning. Easy for humans (92% success), hard for models (15% success for GPT-4), at least at launch.
Terminal-Bench — agents working in actual Linux terminals. Tasks include “build the Linux kernel from source” and “configure a git server plus a webserver on port 8080.” No mocks; if it doesn’t actually build, it fails.
WebArena (Zhou et al., 2023) — realistic simulated websites (Reddit, GitLab, e-commerce) where agents complete actual user goals like “filter the last six months of orders by refund status.”

These benchmarks are harder to saturate than general-knowledge ones, because passing means the thing actually has to work in the real world. A model can memorise a question; it can’t memorise a Linux kernel that isn’t there until it builds it.

The wrong answer: Build a harder general benchmark

By now, you probably see that benchmarks tend to go the same way: A benchmark is created, models get too good at it and perform too well to be useful, and then a variant benchmark is created to address the problem.

This is the saturation problem in a nutshell: a benchmark that once provided meaningful information to help distinguish models eventually fails to do so because all the models’ scores cluster at the top.

Saturation occurs for two reasons:

Contamination: Many benchmarks publish their problem sets online, and models train on data scraped from the web. The benchmark now measures memorization, not capability.
Overfitting: Model providers are incentivized to optimize specifically for gaming performance on benchmarks. Scores go up without corresponding capability improvements.

When you have a party of wizards with 20 INT, “What’s your INT score?” stops being an interesting question because any wizard’s Fireball will hit the same. Building harder benchmarks is like an arms race for ability scores, when what you actually want to know is whether your wizard can kill a dragon.

Harder benchmarks don’t tell us what we want to know: can this model do my specific job?

The right answer: skill checks with real DCs

The fix isn’t a harder ability check. It’s a different kind of check altogether. In D&D, straight ability checks are rare compared to skill checks. There are 18 official skills that break down the six ability scores into more granular activities. For example, your wizard is much more likely to roll one of these instead of a straight INT check:

Arcana
History
Investigation
Nature
Religion

All of these are still based on INT, but test narrow types of intelligence rather than general intelligence.

Specific, domain-grounded benchmarks are what the field actually needs. The evaluations that actually help practitioners share a structure: they test a specific skill in a specific context, and they’re judged by the people who have to live with the results.

MedQA tests whether a model can answer the kind of clinical question a physician might face. LegalBench tests whether a model can correctly identify enforceability clauses, classify contract language, and apply legal reasoning. FinanceBench tests whether a model can answer the kind of question a financial analyst would actually ask about a real 10-K filing. None of these are trying to rank models on general intelligence. They’re asking the only question that matters downstream: can this model do this specific job well enough?

This week, Grafana released o11y-bench, an open benchmark for observability-specific agent tasks. I’m biased — I work there — but it’s a useful illustration of the shape I’m describing. o11y-bench doesn’t ask whether a model is smart in general. It asks whether an agent can do the things observability engineers actually do: read production traces, correlate logs with metrics, figure out why the thing that’s supposed to be working isn’t. Its leaderboard shows models’ performance for each activity.

Does that mean that more specific benchmarks won’t get saturated? No. We’ve already seen this play out with SWE-bench. But they’ll get saturated more slowly and remain more meaningful measures along the way.

These are skill checks with DCs that matter to someone. They’re not trying to produce a single scalar ranking of “which model is smartest.” They’re asking whether the wizard recognizes the family crest belonging to a political faction or knows this specific cult’s practices, not whether the wizard is good at “wizarding” in the abstract.

Roll for initiative: What this means in practice

If you’re an engineer picking a model for production: stop picking by leaderboard. Find or build an eval that reflects your actual use case. A handful of representative examples, graded the way you’d grade them if you were doing the work yourself, will tell you more than a dozen public benchmarks.

If you’re a researcher thinking about what to build next: the interesting work is at the specific-and-grounded end of the spectrum, not the general-and-bigger end. Another harder general-knowledge benchmark will saturate in eighteen months. A well-designed domain benchmark, co-built with the people who actually do the work, can stay useful for years.

And if you’re anyone else watching this field from the outside: be skeptical of the headlines. When a new model scores 94% on a benchmark you’ve never heard of, ask what it measures and whether the thing it measures is the thing you actually care about. Usually the answer is “not really,” and that’s the most important thing a benchmark score won’t tell you.

The goal of a character sheet was never to produce one number that ranks the character. It was to give a DM enough information to run a meaningful encounter — and to give the player enough to know when to try, when to pass, and when to roll with advantage. LLM evaluation should work the same way. Not a ranking; a readiness check. Can this specific model clear this specific challenge? Roll for initiative.

All About AI Observability (The Context Window #03)

Wed, 22 Apr 2026 18:00:00 +0100

In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and the engineers behind the product, Alexander Sniffin and Jack Gordley, for a deep dive into AI Observability in Grafana Cloud. We unpack what it actually is (a new way to instrument your AI apps and see canonical data about them, sitting alongside your existing telemetry), what an evaluation is in this context (and the difference between online evals on live traffic and offline evals on dataset conversations), and we walk through real demos: setting up AI Observability and evaluators, the analytics view, instrumenting a local coding agent, and Jack’s system prompt analysis tool. We also talk about the things nobody else is talking about — like LLM-as-judge as a method, evaluators that catch the most bugs, and the cheating-LLM phenomenon.

Timestamps

00:00:00 — Introductions
00:02:00 — The last month in AI news
00:11:09 — What is AI Observability in Grafana Cloud?
00:19:20 — What is an evaluation?
00:21:05 — The origin of AI Observability
00:25:09 — Demo: Setting up AI Observability and evaluators
00:32:04 — What is LLM as judge?
00:38:43 — Demo: AI Observability Analytics
00:40:52 — AI O11y is based on OpenTelemetry
00:42:17 — Demo: Instrumenting a local coding agent
00:47:18 — Potential future agentic use cases
00:52:00 — Evaluators that catch the most bugs
01:02:23 — Demo: System prompt analysis
01:05:11 — Guess the prompt

Resources

News from the episode

AI Observability

Mentioned

OpenCode — the open-source coding agent Jack instruments
Anthropic: Emotion concepts and their function in a large language model
OpenClaw

How to use Grafana Assistant with self-hosted Grafana

Mon, 20 Apr 2026 09:00:00 +0100

In case you missed it, possibly the biggest news to come from GrafanaCON this year is that if you’re running self-hosted or open-source Grafana, you can now connect it through your Grafana Cloud account to Grafana Assistant, the purpose-built AI tool for all things Grafana.

In this short tutorial, I show you how to enable Assistant on a self-hosted Grafana instance and go from nothing to a working server-health dashboard with a single prompt.

A quick note on what “for open source” actually means

Assistant itself isn’t open source. But you can now use it with your self-hosted or open-source Grafana instance. The way it works is:

You still need a Grafana Cloud account (the free tier has generous inclusions for Assistant).
You link your self-hosted Grafana to that Cloud account.
Assistant then lights up inside your OSS instance.

Think of the Grafana Cloud account as the key that unlocks Assistant wherever your Grafana lives.

Prerequisites

Grafana OSS v13 or later
A Grafana Cloud account (pricing here)

Enabling Assistant

Go to Administration → Plugins, search for Assistant, and click Install.
Refresh the page, then open the Connection tab.
Click Start Connection. You’ll be prompted to log into Grafana Cloud.
Select the organization and stack you want, then click Authorize connection.
Close the tab. Your self-hosted Grafana is now linked to Assistant.

A first dashboard from nothing

Every Grafana instance ships with a TestData data source — perfect for a first run when you don’t yet have real data wired up. Enable it from Data Sources, then open Assistant (the new icon on the top right).

Switch Assistant’s mode from General to Dashboarding — this mode is tuned for creating and managing dashboards — and ask in plain English, for example:

Hi Assistant, could you create a server health dashboard for me using the TestData data source?

What comes back is a starting dashboard with CPU, memory, disk, and network panels already wired up. From there you can tweak panels, edit queries, or ask Assistant to iterate. You’re not starting from scratch — you’re starting from 80%.

What else Assistant can do

I only showed dashboarding in this video, but Assistant can also:

Answer questions about the data already in your Grafana
Run investigations to troubleshoot production incidents
Learn new skills
Manage alerts
Teach you Grafana

That list keeps growing, so stay tuned.

Resources

Grafana Assistant (product page)
Grafana Cloud pricing
How Grafana Labs handles your data
My notes on Grafana Assistant — everything I’m learning as I work on it
Unlearnings from building Grafana Assistant — my Context Horizon piece on what the team had to unlearn

Unlearnings from building Grafana Assistant

Thu, 09 Apr 2026 09:00:00 +0100

Originally posted here

Everyone’s talking about what you need to learn to build AI agents: Prompt engineering. Tool calling. Retrieval strategies. The content ecosystem is full of it. But while I was watching Context Window Episode 1, it struck me that Mat Ryer, Cyril Tovena, and Sven Großmann– some of the team behind Grafana Assistant– spent most of their time on something different: what they had to unlearn.

Grafana Assistant started as a weekend hackathon project. By the time it shipped, the team had thrown out a popular framework, flipped their assumptions about tool design, and discovered that some of the instincts you bring from conventional software development actively work against you when you’re building agents.

I pulled out the ten moments where received wisdom turned out to be wrong. If you’re building anything agentic right now, these are worth sitting with.

❌ LangChain will help us build faster

LangChain is an opensource framework for building applications that use LLMs. It provides structured components that can be used for rapid prototyping.

In the video, Tovena talks about how they started with LangChain during the initial hackathon because they thought it would help them build more quickly, but instead it became obvious fairly early on that the opposite was true. They didn’t end up using a lot of the features that LangChain provided and felt they needed more flexibility and speed.

Abstraction overhead

One possible reason for their troubles with LangChain is that LangChain has standard abstractions for prompts, tool use, and orchestration– things that sound good in theory. But sometimes abstractions lend themselves to creating a sort of black box in your stack, where you don’t know what’s happening inside it. It sounds like the Assistant team missed the oversight and control of what was happening.

❌ Use the same prompt for every model

They also talked about how for the sake of simplicity and consistency, they initially used the same prompt for every model. However, they found out that this actually led to worse results. They found more success in creating slightly different prompts for different models, playing to their strengths and weaknesses a bit more.

There is even an argument (Mizrahi et al.’s “State of What Art? A Call for Multi-Prompt LLM Evaluation) that AI benchmarks that involve sending multiple LLMs the same prompt is a brittle and unreliable way to assess LLM capabilities. Different models respond differently to prompt variations.

In practice, what this means is that switching models isn’t a drop-in replacement. Even if a new model scores better on benchmarks, your prompts were written with a specific model’s tendencies in mind. The unlearning here is letting go of the idea that a prompt is model-agnostic — it’s more like a conversation style, and different models have different conversational defaults.

❌ Treat errors as failures

Traditionally in software, you catch errors and hide them from the user: you log them, handle them, but treat that transaction as a failed one.

The Assistant team had to unlearn this habit when dealing with an LLM, because with agents, error messages are useful data. Feeding it back to the LLM often results in it learning from its own mistake and trying again. It’s basically letting the agent debug itself, in a manner that Ryer noted was remarkably similar to how we humans debug errors too.

❌ Return errors in structured JSON

On the subject of error messages, Großmann mentioned that at one point they were trying to format the errors in JSON or stringified JSON. This practice shows solid engineer intuition because usually, structuring information helps machines process it faster and more accurately. What they found here was the reverse: it was easier and yielded better results when the LLM was just passed the entire error in natural language.

❌ Build as many tools as possible

The team mentioned that at the start, they were inclined to create as many tools/skills for Assistant: one for everything that could be done in Grafana. This made sense until they saw LLMs struggle to determine which tool to use in which context.

To help LLMs identify the correct tool, they added tool descriptions and also experimented with combining multiple tools into fewer, parameterized, tools. These helped, although the team still observed that allowing users to create their own tools (with potentially very general tool descriptions) could potentially confuse LLMs.

❌ Bigger context window = better

Gemini 1.5 Pro announced it had a 2M context window back in 2024, and it might seem that other providers had lagged behind: OpenAI only announced support for 1M tokens in 2025 and Anthropic followed in 2026. However, the Assistant team pointed out that context wasn’t everything. They said that they noticed that bigger wasn’t necessarily better, also pointing out that the chat summarization feature on IDEs like Cursor could still sometimes lose important details when summarizing.

This finding seems to coincide with a 2023 Stanford study (Lost in the Middle: How Language Models Use Long Contexts) that found there was a performance degradation when models access information within a very large context. In the video, they talked about how their system prompts were often 40k tokens long (half for the prompt itself and then the other half for tools).

❌ Start with a multi-agent architecture

Tovena mentioned that creating a multi-agent architecture for Assistant yielded more mixed results than he expected. He said that passing context between agents is difficult, outputs were inconsistent, and there was a bit of a “telephone game” problem where agents didn’t always perfectly understand the intent or output of other agents.

In early 2026, Khatua et al. released a study called “CooperBench: Why Coding Agents Cannot Be Your Teammates Yet” that showed that AI agents aren’t great at collaboration. The study suggests, surprisingly, that this might be because AI agents lack human social habits that help us collaborate better, such as feeling obliged to do what we say and having a good understanding of what others’ plans are.

The Assistant team’s experience is also a perfect example of Gall’s Law, which states that it’s easier to start with simplicity and build complexity than the other way around.

❌ Plan around models as they are now

Software has always moved quickly, but it feels like AI models are moving faster. Stanford’s AI Index Report 2025 reported that models went from having a 50% chance to near-human performance in roughly 2 years (2021-2023). In fact, benchmarks keep getting retired as AI saturates them– meaning that benchmarks can’t reliably be used to distinguish between good and bad models because all the models are testing near the ceiling.

The rapid pace of development around AI requires us to not be so attached to our work: Großmann recalled an incident where someone wrote a joke prompt that spoke like a dog that shockingly did better than their other system prompts. Ryer also mentioned that at one point someone deleted all the system prompts and Assistant weirdly became more accurate.

The lesson is clear: plan for models to improve, and plan to redo a lot of your work when they do.

❌ We don’t need evals

The Assistant team said that they went from having no evals to employing several types of evals:

integration-style evals that exercise whole workflows
unit-style evals that feed inputs plus a system prompt and check the output
LLM-as-judge, where they pass conversations to an LLM to ask it to judge whether it answered the user’s questions.
user feedback in the form of thumbs during conversations with Assistant

Beyond evaluation, they talked about how important it was to be able to observe AI. At the very minimum, showing what Assistant is doing at all times (thinking out loud) is very important to them because it builds users’ trust in the thinking process and, eventually, in the result.

Evals aren’t a sign that you don’t trust your product. They’re how you build the confidence to keep shipping it.

❌ Assume people already know how to work with AI

When you ship an AI feature, it’s tempting to assume the hard work is done. You’ve built the thing. Users will figure it out.

What the Grafana Assistant team found was that this assumption doesn’t hold. Users came in with wildly different mental models of what an AI assistant could or should do — some treating it like a search box, others expecting it to read their minds, most not yet sure how to prompt it effectively.

Their insight was that this isn’t just a UX problem to solve on your end. It’s a shared responsibility: AI should try to help the user, but the user also needs to learn how to work with AI. And building an assistant that teaches people how to use it well (through good responses, gentle corrections, showing its work) is part of building the product.

There’s also a deeper point here specific to observability. Grafana Assistant isn’t just an AI feature; it’s a way of interacting with your own infrastructure. Every good interaction teaches you something about what questions to ask, what signals to look for, what it means to understand a system. In other words: learning to use AI well makes you a better engineer. The tool and the skill co-evolve.

This is the unlearning that sits furthest from the code. It’s not about prompts or architectures or frameworks. It’s about recognising that you’re not just shipping a feature– you’re introducing a new way of working. That takes time. And it’s worth designing for explicitly.

Summary

What strikes me most about this list isn’t any single item — it’s the pattern. Most of these unlearnings point in the same direction: less is more, and messiness is the point.

Less abstraction. Fewer tools. Smaller context. Simpler architecture. And rather than hiding the mess — the errors, the reasoning, the uncertainty — let it show. That’s where the trust gets built.

We’re all still early with this. The team that built one of the most thoughtful AI products I’ve seen up close still didn’t have evals when they launched. That’s not a criticism — it’s a reminder. You don’t need to have it all figured out before you ship. You just need to commit to keep unlearning.

Let There Be Dashboards: Introducing Grafana Assistant

Thu, 26 Mar 2026 19:30:00 +0100

I gave this talk at the Grafana & Friends Lisbon Meetup in Lisbon, Portugal.

Abstract

This is a live demo of how to go from zero to a working Grafana dashboard using Assistant. Starting with a blank slate, we’ll use natural language to explore data and generate meaningful visualizations in minutes. Along the way, you’ll see how AI can accelerate dashboard creation without sacrificing control or understanding.

Thanks to everyone who came out! It was great to connect with the Lisbon tech community.

Do Androids Dream of Second Brains?

Sat, 21 Mar 2026 11:15:00 +0100

I gave this talk at PKM Summit 2026 in Utrecht, the Netherlands.

Abstract

We often describe our PKM systems as “second brains”. But can we really say our systems think for themselves?

In Philip K. Dick’s “Do Androids Dream of Electric Sheep?”, the inspiration for the movie Bladerunner, the boundary between humans and machines is tested through observation and subtle signals. In this talk, we’ll explore a similar question for our knowledge systems: what happens when we start observing them?

By instrumenting an Obsidian vault with an observability stack and layering AI tools on top, we can begin to see patterns in how ideas form, connect, and evolve. The result isn’t quite a thinking system… but it may be something closer than we’ve had before.

Slides

nicole.to/androids

Observability for GenAI Applications (Grafana ❤️🔥 OpenTelemetry Community Call #4)

Wed, 21 Jan 2026 16:00:00 +0100

In this community call with my coworker Liudmila Molkova, we dove into the emerging field of observability for Generative AI applications. We explored how to instrument LLM-powered applications using OpenTelemetry, what metrics and traces matter most when monitoring AI workloads, and how to debug issues in complex AI pipelines. The discussion covered prompt tracing, token usage monitoring, latency considerations, and the unique challenges of observing non-deterministic AI systems.

Asimov's Zeroth Law of Robotics: Observability for AI (Newcrafts 2025)

Thu, 06 Nov 2025 12:23:19 +0100

Today I presented at Newcrafts in Paris, France. I gave a talk I’ve given before, called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU in London and at Dutch Cloud Native Day 2025 in Utrecht. I was happy to present it again, as it gave me the chance to further refine my demo application, which you can find on GitHub.

The main change I made was to use the “AI as judge” approach that I had previously only talked about in theory. In this version of the demo app, I use AI in both the game as well as in the testing of the game, showing how AI can be integrated into both the application and its observability/testing stack. It was really interesting to note that testing with AI ended up exposing more defects than my hardcoded test cases did, which goes to show the potential of AI-driven testing.

Asimov's Zeroth Law of Robotics (Dutch Cloud Native Day 2025)

Thu, 03 Jul 2025 15:01:24 +0200

Today I presented at the Dutch Cloud Native Day 2025 in Utrecht, the Netherlands. I gave a talk titled Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU earlier on in the year, in London. I was very happy to be able to present it again, because it gave me the opportunity to improve the demo app I used, which you can find on GitHub.

Asimov's Zeroth Law of Robotics: Observability for AI (KubeCon EU 2025)

Wed, 02 Apr 2025 13:46:42 +0200

Today I gave a talk at KubeCon EU 2025 in London called “Asmiov’s Zeroth Law of Robotics: Observability for AI”. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

AI Observability with Ishan Jain (Grafana Office Hours #29)

Fri, 08 Nov 2024 18:24:29 +0100

In this Grafana Office Hours, Ishan Jain talks about AI Observability with Grafana: what it entails, factors to consider when monitoring and observing LLMs, and how to do it all with Grafana. He is joined by Senior Developer Advocate Nicole van der Hoeven.

The Role of AI in Gender Equality with Nicola Martin (Adobo & Avocados #15)

Wed, 04 Oct 2023 17:26:14 +0200

In this episode of Adobo and Avocados, my friend Marie Cruz and I chat with Nicola Martin, Head of Quality Engineering at Adarga, to talk about the role of AI, technology, and education in gender equality.

Nicola is a Council and Committee Member for the British Computer Society and WomenTech Global Network. She is also a mentor, speaker, and panelist speaking on subjects such as AI and quality, software testing, and diversity in tech.

Tagging and Linking With AI: Napkin

Fri, 05 May 2023 00:58:39 +0800

In this video, I demonstrate my workflow for tagging and linking with AI using the Napkin.one app. Napkin is an app that uses a fork of GPT-3 to intelligently tag thoughts with keywords that improve over time. I like to use Napkin to tag mostly my highlights from Readwise.

Resources

Napkin
Readwise
The Lost Art of Taking Good Notes, my talk at EuroSTAR 2023

AI tools for PKM and Obsidian

Fri, 27 Jan 2023 16:39:00 +0100

Here are some AI tools for learning and taking notes that I’ve incorporated into my processes. AI is a controversial topic, but it’s important to remember it’s just like any other technology: it’s only as good as the people that wield it. In this video, I show some ways I use artificial intelligence to improve my learning and note-taking rather than to replace them.

Ai on Nicole van der Hoeven

The Defiant Jazz Incident

How it works

Why a murder mystery?

No spoilers

Try it

Asimov

The idea

What’s in it

The talks

Try it

Build Apps on Your Data with Grafana Pages (The Context Window #07)

What Grafana Pages is

Seeing it on Grafana Play

Building a Page from a prompt

Personal data Pages

Pages vs dashboards

Remixing, editing, and building from a dashboard

Sharing and access control

How Pages work: React, sandboxes, and iframes

Limitations and roadmap (no promises)

Resources

Related reading

More from The Context Window

Transcript

Intro, guests, and ObservabilityCON recap

What Grafana Pages is

Grafana Pages examples on Grafana Play

Build a Grafana Page from a prompt

Personal data Pages: sleep and flight tracking

Grafana Pages vs dashboards

Remix a Page or create one from a dashboard

Share Pages and control access

How Pages work: React, sandboxes, and iframes

Limitations, roadmap, and more Page examples

AI Tools We Use Every Day on the Grafana AI Team

Resources

What Does a Grafana AI Offsite Look Like?

Resources

Roundtable: How We Really Use AI in Testing (HUSTEF 2026)

Resources

Asimov's Zeroth Law of Robotics: Testing for AI (HUSTEF 2026)

Resources

Context Engineering Fundamentals (The Context Window #06)

What we cover

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana GCX

Transcript

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana MCP

Transcript

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Transcript

Resources

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Transcript

Observability and AI explained

Resources

AI Week Day 4: Evaluating (Grafana AI Week — Thursday)

Transcript

Investigations Arena: how we test Assistant investigations

Transcript

Resources

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Resources

Related reading

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Transcript

AI Week Day 2: Building (Grafana AI Week — Tuesday)

Transcript

AI Week Day 1: Planning (Grafana AI Week — Monday)

Transcript

My reMarkable Obsidian workflow

The setup: a lot of E Ink

Step 1 — Sync the PDF into Obsidian

Step 2 — OCR the handwriting into real text

The accessible alternative: Claude Cowork / Claude Desktop

Why this matters

Resources