Observability on Nicole van der Hoeven

Spice Runner

Wed, 05 Nov 2025 00:00:00 +0000

Spice Runner is a browser-based endless runner. Think of the “unable to connect” dinosaur game in Chrome, except you are a Fremen running across the sands of Arrakis, jumping sandworms and Harkonnen ornithopters. It’s the demo app for my talk The Spice Must Flow: The Fremen Guide to Sustainable Observability, and it exists to make one point: even a simple, throwaway game generates a surprising amount of telemetry, and that telemetry has real financial and environmental costs.

The Fremen live on a planet that is rich in spice and poor in water, and they thrive by wasting nothing. Our systems are rich in telemetry and poor in budget, and in carbon. Spice Runner is a small, playable version of that problem: a real app, with real players, running on real infrastructure that scales up and down with them while I watch what it costs.

What it demonstrates

Real user monitoring with Grafana Faro. Game sessions, game starts, scores, collisions, restarts, and game overs are all sent as custom events, so the dashboard shows how people actually play.
A leaderboard that’s instrumented end to end. A Go API backed by PostgreSQL and Redis serves a public top-10 leaderboard. It’s instrumented with OpenTelemetry for traces and custom metrics, so a score can be followed from the browser through the API to the database and the cache.
Metrics, logs, and traces on Kubernetes. Prometheus, Grafana Loki, and Grafana Tempo run in the cluster, with Grafana Alloy as the collector and Grafana for dashboards. Telemetry is also sent to Grafana Cloud.
Autoscaling that follows the players. KEDA scales the game’s pods on HTTP request rate, CPU, and memory, and the GKE Cluster Autoscaler adds and removes nodes behind them. Production settings keep at least one pod running and cap it at 20, with a separate demo profile that is far more eager for live demonstrations.
Energy monitoring with Kepler. Power and energy use are tracked per workload, so “what does this cost the planet?” is a query rather than a guess.
Load testing with k6. k6 drives load against the leaderboard API and, with a browser script, plays the game to keep the dashboard populated.

The dashboard

The live dashboard on Grafana Play shows real player data as it happens. The dashboard is also entirely AI-generated: it was built with Grafana Assistant.

Open the live dashboard

Architecture

The app has two layers. The first is the game and its leaderboard: a vanilla JavaScript game served by Nginx, with a Go leaderboard API backed by PostgreSQL and Redis. The second is the observability stack: Faro instruments the browser, and the Nginx exporter, Alloy, Loki, Tempo, and Prometheus cover the backend. Everything runs on Google Kubernetes Engine, with persistent volumes so telemetry survives pod restarts, and autoscalers at both the pod and node level.

The talk

The Spice Must Flow uses Dune as its framing. It covers the hidden environmental and financial costs of an observability stack, how hard it is to measure the cost of ephemeral resources on Kubernetes, how to use KEDA to right-size resources when you don’t know what the right size is, and the most effective thing you can do to cut cost: deleting the telemetry you don’t need.

I first gave it at Øredev 2025 in Malmö, Sweden. You can read the write-up and abstract and see the slides.

Run it yourself

Everything you need to deploy it on your own GKE cluster, including the Kubernetes manifests, the dashboards, and the load tests, is in the GitHub repository. Its README has a quick start.

The Defiant Jazz Incident

Fri, 30 Oct 2026 00:00:00 +0000

The Defiant Jazz Incident is a murder mystery you solve with observability tools. It’s set on the Sleep Floor at Kipsake, a company that rents your sleep: at night, its Dreamers sleep in pods while their sleepwalking selves work the floor. During the Cocoa Hour, the floor’s middle-of-the-night holiday party, the power goes out for five minutes. When it comes back, the night supervisor is dead in the elevator.

Kipsake recorded everything it could: badges, doors, chat, sleep rings, power, microphones, cameras, sleep bands, and personnel records. Your job is to work out who did it, and the only way to do that is through the telemetry.

How it works

The case file is a set of logs, metrics, and traces. A loader sends them to Grafana Loki, Prometheus or Mimir, and Grafana Tempo, either in Grafana Cloud or in a local Docker stack, and you investigate from there with whatever you like: Grafana itself, Grafana Assistant, an MCP server, gcx, or the AI agent of your choice. The data is deliberately messy. Some devices lose clock sync during the outage, some records arrive late, and not everyone on the Sleep Floor remembers what their body did.

The game itself comes in two forms. A terminal version is for demos and playtests: you watch the testimonies and choose what to investigate, while an AI that only sees the telemetry answers your questions. A web version is made to embed in a Grafana dashboard panel right next to the data.

Why a murder mystery?

Investigating an incident is investigating an incident, whether the victim is a production service or a night supervisor. You start with a vague report, a pile of signals that don’t quite agree, and a clock that can’t be fully trusted, and you have to build a timeline you can defend. A case with a known answer is a safe way to practice that, to compare how different tools and AI agents cope with it, and to see how good a question you have to ask before the data answers.

No spoilers

The repository keeps the solution in clearly marked files, and the README tells you which ones to avoid if you want to play. This page deliberately says nothing about who did it.

Try it

The repository has everything: the case brief, the telemetry, the loader, and both versions of the game.

obsidian-grafana

Mon, 13 Oct 2025 00:00:00 +0000

obsidian-grafana turns an Obsidian vault into time-series data and dashboards. I work at Grafana Labs, helping people monitor their systems with open source tools, and at some point I realized I could use the same tools to monitor my personal knowledge management system. It’s a nerdy and totally unnecessary setup, and I enjoy it a lot. I mention it in my post on how I use visuals in my PKM.

The screenshot above was made by running the repo’s own Grafana setup on a copy of the posts from this website.

How it works

A Python script reads the Markdown files in your vault and extracts metadata: word, line, and character counts, file size, frontmatter fields, tags, wikilinks, and timestamps. It runs every five minutes and has two paths:

A log path for history. Only the notes that changed since the last run are written, as structured JSON, to a log file. Grafana Alloy tails that file and forwards it to Loki, with tags and frontmatter fields as labels, so you can see how your notes change over time.
A metrics path for the current state. A metrics exporter exposes totals such as note count, word count, unique tags, and wikilinks, and Prometheus scrapes them.

Grafana ties the two together with pre-configured dashboards: an overview, a time series dashboard for individual notes, a relationships dashboard for wikilinks, a dashboard of frontmatter types, and a word cloud of your most used words. Everything runs in Docker Compose.

What it’s useful for

Understanding your writing habits and how your vault grows, checking how consistently your notes are tagged, finding the notes that everything links to, and keeping an eye on notes with a given frontmatter status or priority. The README also covers installing the Grafana Assistant plugin into the local Grafana.

Try it

The repository has a quick start. You’ll need Docker and Python 3, and a vault to point it at.

Asimov

Wed, 02 Apr 2025 00:00:00 +0000

Asimov is the companion project to my talk Asimov’s Zeroth Law of Robotics. It’s an AI app built to be observed and tested: The Silent Enterprise, a single-player Star Trek text adventure. You play Data, and Claude is the game master. Every model call, dice roll, and turn shows up as telemetry in a Grafana that you can run on your own machine or in Grafana Cloud, and a k6 test suite, some of it judged by Claude, checks whether the AI follows the rules.

The game master has some known flaws, kept on purpose. It sometimes narrates a rescue that didn’t happen, or “destroys” a drone that’s still standing. The point of the demo is catching those flaws with telemetry, tests, and online evaluators.

The idea

Isaac Asimov’s three Laws of Robotics govern how robots behave. The talk argues for a law that comes before all of them: a robot must be observable, and, in its newest version, testable. AI systems fail differently from traditional software. There’s no single correct output to assert against, a response can be fluent and completely wrong, and models drift. You can’t test what you can’t see, and you can’t trust what you can’t test.

What’s in it

The game. A Go program with a command-line interface and an HTTP API. The game engine owns the rules and the state. Claude interprets the player’s input, makes the game master’s own rolls through a tool, and narrates, so a narration that contradicts the saved state is a defect that can be caught.
Telemetry. Traces, metrics, and logs go over OpenTelemetry to a local Grafana stack (Tempo, Prometheus, and Loki, with the game’s dashboard already loaded) or to Grafana Cloud. Generations can also go to Grafana Cloud’s Agent Observability, where online evaluators judge a sample of them.
Tests. A k6 suite with code checks, LLM judges, dice-trajectory evals, and whole playthroughs. All of it can run for free against a mock game master and judge.
No account needed to start. You can play offline with no API key, and see the local monitoring stack with one command and Docker.

The screenshot above is the game’s local Grafana dashboard.

The talks

I’ve given the talk, in several versions, at KubeCon Europe 2025, Dutch Cloud Native Day 2025, Newcrafts 2025, ExpoQA Madrid 2026, and HUSTEF 2026. The write-ups are linked below. The earlier talks used a Python and Flask version of the app, which has since been rewritten in Go.

Try it

The repository has a quick start for playing offline, a Docker-only path, and release binaries for Windows, macOS, and Linux.

Build Apps on Your Data with Grafana Pages (The Context Window #07)

Thu, 05 Nov 2026 18:00:00 +0000

My favourite announcement out of ObservabilityCON was Grafana Pages — a way to build a live, interactive data app right from your Grafana instance, with just prompts and no code. In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by my co-host Tiffany Clifford and by Stephanie and Tom from Grafana’s AI department, who built Pages (it started as a hackathon project to give Grafana Assistant a proper Linux sandbox, and turned into something much bigger). We dig into what Pages actually is, build one live from a prompt, and get into the stuff I find genuinely delightful about it: a page is still real Grafana — same data sources, same RBAC, same production-ready panels — it’s just no longer you arranging things on a grid. We also cover personal data Pages (my sleep-vs-flight tracker, Stephanie’s StairMaster world tour), Pages vs dashboards, how it all works under the hood (React in a locked-down iframe, served from a micro-VM sandbox), and the current limitations and roadmap.

Fun fact dropped mid-stream: the ObservabilityCON keynote itself was a Grafana Page. So was this presentation.

What Grafana Pages is

As Tom put it, a Page is “all of Grafana, minus the panel grid.” It’s the same data sources, the same RBAC, the same underlying panels and visualization libraries — but instead of you arranging panels on a grid (or Assistant trying to squeeze your intent into what a dashboard can represent), you unlock Assistant and let it do the thing modern models are genuinely good at: writing code and building a website. You keep everything that makes Grafana production- and enterprise-ready, carefully worked out over nine years, and you add interactive, tailored, pretty ways to look at your data.

It builds on the Grafana-managed sandboxes announced in the same release — the same underlying layer of Linux compute for AI agents that also powers auto-remediation and investigations.

Seeing it on Grafana Play

The fastest way to understand Pages is to go poke at it on Grafana Play — no login, no Cloud account needed. Stephanie showed a few:

An observability-survey quiz game in comic-book style, pulling real data already on Play — built from one prompt (“make it higher-lower, comic-book style”).
A quarterly-review page, fully interactive with drill-downs by region and customer sentiment — also one prompt, with “make it look pretty” doing most of the design work.
A teammate’s page that pulls your GitHub contribution data and thanks you — and, because it uses the inline Assistant, renders the thank-you in the viewer’s native language (inferred from internal data).

The point Stephanie and Tom kept coming back to: Pages aren’t static. A page can fetch recent data every time you open it and tell you the most noteworthy events in the last 24 hours — so an on-call summary page genuinely updates itself.

Building a Page from a prompt

We built one live on Play against the demo incident/outage data: “Create a page to summarize what is noisy and needs attention from the on-call engineer — give us the top three things.” Assistant loops through alerts, shows the queries it’s running in a Sources tab (no hand-waving about what it’s doing), takes live screenshots of the rendered page as it edits so it actually knows how it looks, and comes back with dynamic top-three items. Making a page really is just that easy now.

Personal data Pages

This is where I got carried away. I hooked up my two smart-ring data sources (Oura and Ultrahuman — both have API endpoints) and, from one prompt, got a whole page summarizing years of sleep: best month, worst month, seasonal breakdowns, the slightly-depressing reveal that I average just under seven hours. I didn’t ask for the blue-purple theming. I also didn’t ask it to correlate my flights (TripIt + Flighty, ten years of data) with my sleep — it just did, because the flight data was there. A second prompt gave me an animated replay of every flight I’ve taken.

Stephanie’s was my favourite: a StairMaster world tour, ranking gyms she’s visited for work by how good the view is (so she doesn’t have to think about being on a StairMaster), complete with an out-of-order-machine mini-game and motivational quotes from ultrarunners. And a summit-conditions page pulling weather data for different mountains.

Pages vs dashboards

So when do you build a page instead of a dashboard? The honest answer from the team is: it’s early, and Pages already cover many dashboard use cases. Tom’s product-brain answer is “make it a page, period” — but if you have a well-understood dashboard that serves you well, there’s no urgent reason to change it, and dashboards still win where you need a dashboards-as-code story (Pages has no as-code path yet). For me, I reach for a page when I want to tell a story with data — a narrative, some annotation, something I’d present to my manager without drowning her in every metric. Pages is great at filtering down to the readable essentials without deleting anything.

A nice pro tip from Tom: if you ever present real data on a big screen, Pages is for you — you can ask for presentation style and still get real panels backed by real data, instead of screenshotting them out. We’re both using Pages for KubeCon talks.

Remixing, editing, and building from a dashboard

There are several ways to create a page:

Ask Assistant directly in chat.
Create new page with a starter example (service health, minimal, etc.) that pre-seeds the prompt with styling hints.
Remix an existing page — reuses it as a reference and changes style, data sources, or format; saved as its own new page.
Start from a dashboard — turn an existing dashboard into a page (handy when the dashboard is the ongoing stream and you want a “how was last quarter” narrative view).

Editing happens through Assistant too: highlight an element and it’s attached as context (it carries the DOM node / data-page-id), so you can ask “how did you get this data?” or request a change to that specific element. Templates are coming — define a house style once and have everyone in your org generate pages or presentations that match.

Sharing today works a lot like Google Docs/Slides: add specific people, generate a link, or drop it in a folder (folder view/edit permissions carry over). But viewers must be logged into the Grafana instance — there is no fully public “share with the world” option yet, and no share-by-email-domain yet either. Both are open feature requests; the team explicitly asked for feedback on whether people want them.

Crucially, data access still respects user permissions. If you can open a page but don’t have access to its data sources, the page opens but shows no data — exactly like dashboards, because the queries still go through your regular data sources with full authorization and RBAC.

How Pages work: React, sandboxes, and iframes

A dashboard is configuration — a (very large) JSON blob that Grafana lays out into the panel grid at runtime. A Page is a layer lower: what the agent produces is React source code for a website, which Grafana renders inside a heavily locked-down, isolated iframe, piping through only carefully chosen data.

But something has to safely compile that TypeScript/JSX and bundle it — running arbitrary model output is exactly the kind of unsafe operation you don’t want on your main infrastructure. So each conversation gets a thin Linux micro-VM. The hard engineering problem was making it fast: nobody waits two or three minutes for an EC2 to spin up before losing interest. The team got it to start in under 10 seconds, with each command finishing in under two.

Tom clarified that “sandbox” is overloaded here — there are two:

The Linux micro-VM used only while building or editing a page. Once built, the output is just HTML in an S3 bucket — no VM involved anymore.
The iframe isolation layer inside Grafana, so AI-generated website content renders safely without leaking your data. You control what hosts a page can call out to, per page (important — not every page should be as shareable as, say, my health data). MCP server connections are supported too, and any viewer gets asked for permission before anything runs on their behalf.

Limitations and roadmap (no promises)

Pages is in public preview across Grafana Cloud (not OSS yet, and it will always require Assistant to drive the prompt → page step). Known rough edges and wishes, with the usual “nothing here is a commitment”:

Multi-editing: today you don’t know if someone else is editing the same page, so you can clobber each other. A “someone else is editing” warning is coming before GA, with proper multiplayer later. (We hit this building the KubeCon deck.)
Public sharing and share-by-domain: open questions, driven by feedback.
Storage: pages and their remixes naturally form a graph of related code artifacts, so a Git-inspired storage model feels like the next natural step.
Higher environment fidelity, templates (rolling out now), and more examples.

If you want it for OSS — which I personally do — leave a comment so the team can champion it internally.

Resources

Grafana Play — try the public Pages (games included), no login required
Grafana Cloud — sign up free; Pages is in public preview
Grafana Community Slack — there’s a Grafana Pages channel for feedback

My notes on Grafana Assistant

Transcript

Intro, guests, and ObservabilityCON recap

Nicole: Hi everyone, welcome to another episode of The Context Window. Today we’re talking about the hottest thing — in my opinion — to come out of ObservabilityCON: Grafana Pages. It’s a way to build a live, interactive data app right from your Grafana instance, with just prompts and no code. I’m not the only one excited about it, so — Tiffany, do you want to introduce yourself, my awesome co-host?

Tiffany: Hi, I’m Tiffany, a developer advocate. Until literally a couple of days ago I was focused on Tempo, Pyroscope, and AI — and now just AI, so I get to dive into all this really cool stuff. This one’s new to me anyway.

Stephanie: Hello, I’m Stephanie. I work in the AI department with Tom. We did a hackathon together for Pages and have been working on getting it released for ObservabilityCON. That’s been fun. Tom?

Tom: Hi, I’m Tom, also in the AI department — I’ve been at Grafana seven years. For the last couple of months Stephanie and I were toying with the idea of giving Assistant a proper Linux sandbox and a proper way to execute code. We didn’t even set out to build Pages — then we realized we could just let it write React code, and that gives you so much better visualizations and tailored outcomes than dashboards.

Nicole: I’m Nicole van der Hoeven, a developer advocate mainly focused on AI right now. And I wanted to mention ObservabilityCON, which just passed — I wasn’t there in person, but a lot came out of it. The keynote on YouTube is really good.

Tom: Can I tell you a secret? The keynote is a Page. And, sneakily, so is this presentation. The three ObservabilityCON themes were autonomous operations — and look, there’s Grafana Pages right there — cloud economics, and context. We’ll focus on Pages.

What Grafana Pages is

Stephanie: We started by working on sandboxing for coding agents, then in a hackathon realized we could use those sandboxes for Grafana Pages. It lets you see your data in more interactive ways — you can prompt Assistant with “I want to look at my business revenue for the past quarter,” or weather on a mountain, and it creates unique interactive ways to look at data.

Tom: If I had to describe it, it’s all of Grafana minus the panel grid. Same data sources, same RBAC, same underlying panels, same visualization libraries. It’s just no longer you arranging them on a grid, or Assistant trying to fuse your thoughts into what a dashboard can represent — we unlock Assistant and let it write code and build a website. But you’re not giving up the things that make Grafana production- and enterprise-ready, carefully worked out over nine years.

Tiffany: And the sandboxes you mentioned — is that the Grafana-managed sandboxes that also got released?

Tom: Those were coined more towards auto-remediation and investigations, where we use sandboxes to look at your source code and check a hypothesis for why something’s broken. But it’s the very same underlying layer of Linux compute for AI agents that powers Pages too.

Grafana Pages examples on Grafana Play

Nicole: The best way to understand Pages is to dive in on Grafana Play — this playground for all things Grafana. No login, no Cloud account. You can just mess around and see what people have already created.

Stephanie: Here’s an observability survey — we run one every year — turned into a game. It’s pulling in real data that’s already on Play, with little animations, streaks, ten questions. Maybe you think reading the survey is dry; a game could help. And this was one prompt: “make me a higher-lower game, comic-book style.” I’m not a front-end engineer — I did not do this.

Here’s a quarterly-review example, also one prompt, all interactive — drill into what the customer is thinking, different regions, fake sales data. I can’t get over how pretty it is. There’s also a Sources tab so you can see the queries behind the data. A teammate made one that pulls your GitHub data and thanks you for your contributions.

Tom: That one uses the inline Assistant, so the page itself can make model generations — it shows the thank-you in the viewer’s native language, guessed from internal data.

Stephanie: The powerful part is it’s not static. You can make a page that summarizes recent data, and it’s not hard-coding the summary — every time you open it, it first fetches recent data and tells you the most noteworthy events in the last 24 hours. I made an on-call rotation page that surfaces the most urgent things every time I look.

Build a Grafana Page from a prompt

Tom: Since it’s so easy to prompt — can we build one from scratch on Play?

Stephanie: You go to Pages, create a new page, pick from starter examples that give Assistant some styling direction so you don’t need a huge prompt.

Nicole: Let’s say: “Create a page to summarize what is noisy and needs attention from the on-call engineer — give us a summary of the top three things.” It’s so cool that making a page is now just that easy.

Stephanie: It’s looping through alerts, figuring out the alert states, and showing its queries in the Sources tab — so it’s not hand-wavy; you can check it’s actually doing what you asked.

Tom: Depending on what you ask, it can take a bit longer. Let’s leave it and jump to some other things.

Personal data Pages: sleep and flight tracking

Nicole: I have two smart rings — Ultrahuman and Oura. I hooked up both data sources (they have API endpoints), and this whole page was one prompt: how much sleep I get each year, best and worst months, by season. I average just under seven hours. I didn’t ask for the blue-purple theming either.

Then it added a travel page — I realized it’s correlating my flights with my sleep, because I have flight data in here too (TripIt and Flighty, ten years). I didn’t ask it to do that.

Tiffany: I was literally looking into TripIt yesterday — and apparently they closed the API in late September to anyone not already using it. I’m so glad you’ve been using it.

Nicole: This replay — this one I did ask for, so two prompts — animates all my flights, new countries, busiest month. Pages has been useful for more than the usual outages.

Grafana Pages vs dashboards

Nicole: When would you build a page versus a dashboard? Are you all-Pages all the time now?

Tom: The product person in me says “make it a page, period.” Honestly we don’t fully know yet — Pages cover many dashboard use cases. If you have a well-understood dashboard that serves you well, there might be no immediate reason to change it. But in general Pages should replace most dashboard use cases, and if they fall short, that’s a reason to talk to us.

Stephanie: For dashboards, if you need a dashboards-as-code story — running the same dashboard across many clusters — we don’t have Pages-as-code right now, so dashboards still win there.

Tom: Pages will not come to OSS without Assistant — the way it works, you have to prompt Assistant to make it into a page. Assistant has a lot of smarts about correlating Grafana data; a vanilla model wouldn’t have the opinionated, docs-backed best practices.

Tom: Pages are very much a Grafana thing. Grafana has always stood for two things — that it’s pretty, and that it’s serious, as in real data, not made up. We tried hard to preserve both: lots of tuning so a small prompt gives a pretty page, and it still retains inspectability and real data rather than looking like LLM slop. If you just want to post something up, tools like Claude artifacts already exist.

Remix a Page or create one from a dashboard

Stephanie: The main way is prompting. There’s also “make your own,” where you prompt it much more specifically about the look. Templates are coming — define a style for your whole org’s presentations. And you can remix a page: it uses the page as a reference and changes style, data sources, or format depending on what you ask; saved as its own new page.

For editing, you highlight an element and it’s attached as context — the DOM node with its data-page-id — so you can ask how it got the data, or request a change to that specific element. You can also start a page from a dashboard.

Nicole: I tend to build a page when I want to tell a story with the data — some narrative, maybe annotation. My dashboards get cluttered; Pages lets me streamline to what’s readable without deleting anything. Great for explaining something to my manager.

Stephanie: To share, you can add specific people, say “anyone with this link,” or put it in a folder so folder permissions apply. There’s no fully public “share with the world” option today — that’s a feature request; let us know in the comments. And data access respects user permissions: if you can open a page but lack access to its data sources, it opens but shows no data, just like dashboards.

Tom: It’s best described as the Google Docs/Slides model — share with specific people or generate a link — but viewers must be logged into the Grafana instance. There’s no public option yet.

How Pages work: React, sandboxes, and iframes

Tom: A dashboard is configuration — a big JSON blob Grafana lays out into the panel grid at runtime. A Page is lower-level: what the agent produces is React source code for a website, which we render in a heavily locked-down, isolated iframe, piping through carefully chosen data.

Something has to safely compile that JSX and bundle it, which is unsafe to do with arbitrary model output — so each conversation gets a thin micro-VM. Making that scalable took most of the platform time: you don’t want to wait two or three minutes for an EC2. We got it to start in under 10 seconds, every command finishing in under two.

The term “sandbox” is overloaded. There are two: the Linux micro-VM used while building or editing a page — once built, it’s just HTML in an S3 bucket, no VM anymore — and the iframe isolation layer inside Grafana, so AI-generated content renders safely without leaking your data.

Stephanie: There’s a setting for what a page can call out to, and you can control it per page — some things, like my health data, shouldn’t be broadly available. MCP server connections are supported too, and viewers are asked for permission before anything runs on their behalf.

Stephanie: Data sources still use the user’s permissions — if you don’t have access, it won’t show data. It builds on battle-tested, enterprise-ready foundations, so authorization and RBAC carry forward.

Limitations, roadmap, and more Page examples

Nicole: Beyond sharing, what other limitations should people be aware of?

Stephanie: Multi-editing is difficult — right now you don’t know if someone else is editing a page, so you can overwrite their work. Until GA, just tell people if you’re editing. We’ll have a “someone else is editing” warning before GA and proper multi-edit later. Templates are rolling out now.

Tom: Pages and remixed pages form a graph of relationships and are code artifacts, so some Git-inspired storage system feels like the natural next step — but it needs careful thinking.

Nicole: I have an unnecessary setup where Alloy on my laptop sends metrics about my Obsidian notes into Loki and Prometheus — and I have a page for all of it: top topic this month, a GitHub-style contribution heat map, a graph view. I didn’t ask for any of the styling; I just said “here’s some data, build me a page.”

Stephanie: My favourite is my StairMaster world tour — I do mountaineering, so as I travel for work I find StairMasters with pretty views. One prompt ranked the gyms by view, added a mini-game for out-of-order machines, and pulled motivational quotes from ultrarunners. I also made one for summit weather conditions on different mountains.

Nicole: There’s a Grafana Play page on Singapore weather and haze that’s beautiful — it shows wind direction and air quality across regions, with a 72-hour replay. I have family in Singapore, so I care about the air quality. Amazing stuff.

If you want to try this, sign up for a free Grafana Cloud account — Pages is in public preview, nothing to contact anyone about — or just play with the ones on Grafana Play. And please give feedback: there’s a Grafana Pages channel in the community Slack and a Reddit post. Comment on the video too, especially if you want it for OSS so we can champion it internally.

Nicole: Thank you all for joining — this was a fun one. The next Context Window livestream is later this month with William Dumont, about Assistant Watchers. Stay tuned.

Tom: Thanks for building this amazing tool — it’s my favourite feature on Grafana now.

Asimov's Zeroth Law of Robotics: Testing for AI (HUSTEF 2026)

Wed, 07 Oct 2026 10:00:00 +0200

This week I gave a keynote at HUSTEF 2026 in Budapest, Hungary. The talk is called Asimov’s Zeroth Law of Robotics: Testing for AI — a testing-focused evolution of a talk I’ve been giving (and rewriting) for a while, previously as Observability for AI at KubeCon EU 2025 in London, Dutch Cloud Native Day 2025 in Utrecht, NewCrafts in Paris, and ExpoQA Madrid 2026. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be testable.

AI systems fail differently than traditional software. There’s no single correct output to assert against. A response can be fluent and completely wrong. Models drift silently over time. And yet we’re shipping AI into production — often without knowing what “passing” even looks like.

Observability is what makes testing possible. When we can see inside our AI systems — trace the chain of reasoning, measure what changed, surface what failed — we can start to write evaluations, define what “good” looks like, and catch regressions we didn’t know to look for.

In this talk, I explore what it means to test AI in the real world, drawing on lessons from building and evaluating an AI assistant for a large-scale observability platform. I’ll cover how evaluations work as the AI equivalent of a test suite, what to measure, how to catch hallucinations, and how to know when your model has regressed.

You cannot test what you cannot see — and you cannot trust what you cannot test.

The big shift in this version is right there in the Zeroth Law itself. In earlier iterations I argued that a robot must be observable. That’s still true — observability is the only way to see the reasoning behind a system and find out what actually happened. But I’ve come to think that pure knowledge isn’t the same as trust. You can know exactly what a system did and still not trust it. The only way to trust something is to put it through its paces: to know how it behaves in the situations you foresee, and even the ones you don’t. So the Zeroth Law has become a robot must be testable.

I framed the whole thing around Asimov’s short story Runaround (from I, Robot) and its robot, Speedy. Speedy’s smuggler owners sent him to collect selenium on Mercury, and he got stuck in a loop — circling the selenium pool, unable to resolve the conflict between obedience (the second law) and self-preservation (the third). His owners couldn’t predict his behaviour and couldn’t reason about it, so they had to do the only thing left: go out into the danger zone and fix him in production. Speedy wasn’t untestable because his owners were incompetent or lacked the tools — he was untestable because he was never built to be tested. That’s the trap I think a lot of AI systems are walking into right now.

The through-line for a testing audience: AI is rewriting how software gets built, but it doesn’t rewrite the discipline of trust. Trust that a system does what we say it does, and that when it breaks we know why and can fix it — that’s always been the job, and it isn’t going anywhere. There’s a whole generation of AI engineers shipping software they haven’t tested and can’t explain, and when someone says “the agent does this,” our job is still to ask how do you know? Testing isn’t just going to survive the age of AI — testing is the reason AI gets to survive, if we make it testable.

I demoed this with my TTRPG side project: an AI Dungeon Master, instrumented and put under test. It turns out that writing evals for something non-deterministic forces you to confront your real requirements. I thought I knew exactly when a dungeon master should call for a dice roll — I GM myself! — but trying to put it into words (“in this situation I’d roll, but in that one I wouldn’t, because I want to keep the story moving”) surfaced decisions I’d never had to make explicit. Those are things we’d normally discover much later, in production. Having to figure them out up front is a win.

The Q&A was genuinely great and ranged well beyond the slides:

“When has an AI been tested enough?” Same answer as for anything else — it’s a mix of risk and budget. If it were a life-critical robot like Speedy, you test every scenario the people relying on it need. For my little demo app, less so.
Testers vs. developers in the age of AI. I said I’d fire developers before testers — provocatively, but the point stands: anyone can write code now. What made a good developer good was never the coding; it was communicating with business and engineers, translating requirements, finding creative-but-safe solutions. Testers already think at that level and translate across teams, which makes them more important in the age of AI, not less.
Defining expected results for non-deterministic systems, multi-turn evals (I give the simulated user a personality — adversarial, cooperative, beginner — rather than a script), and the broader shift from imperative to declarative: we no longer instruct step by step, we declare the end state we want, the way we do with Kubernetes.
Convincing colleagues not to fear AI. I don’t, really. AI is here to stay; the question isn’t whether you approve of it, it’s whether you want to stay in this industry and help make it safe and trustworthy — or sit on the sidelines.

I also recommended the book Slow AI, which lands where I do: not an AI apologist, not a boycott, but a clear-eyed list of what AI is and isn’t good for — and the reminder to keep checking whether a given use of AI is making you more capable or less.

The demo app (the instrumented AI DM) lives on GitHub, and I keep refining it with each iteration of this talk.

I’ll update this post with the recording once HUSTEF publishes it.

Resources

Slides
Demo app on GitHub
Slow AI (book)
Previous versions of this talk: KubeCon EU 2025 · Dutch Cloud Native Day 2025 · NewCrafts 2025 · ExpoQA Madrid 2026

Context Engineering Fundamentals (The Context Window #06)

Fri, 02 Oct 2026 18:00:00 +0100

What actually counts as “context” for an AI agent — and how does it find the right things to put in a limited window? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I dig into context engineering fundamentals with Ben Sully and Ioanna Armouti. We treat the context window as an attention budget, walk through Assistant Search and semantic retrieval, embeddings, and how you actually evaluate whether the retrieved context was any good.

What we cover

What counts as context — and why the context window behaves more like an attention budget than just a size limit.
Finding the right context — Assistant Search, semantic retrieval, and embeddings: how an agent surfaces what it needs instead of being handed everything.
Evaluating retrieved context — because retrieval that looks relevant isn’t the same as retrieval that is.
The tool tax and tool deferral — every tool definition costs context, so when do you load it versus defer it?
Progressive disclosure and long-term memory — giving the agent what it needs, when it needs it.
Why removing 80% of a system prompt can change nothing — a nice illustration that more context is not automatically better context.

Resources

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Fri, 31 Jul 2026 09:00:00 +0100

The final day of AI Week is about collaborating — using AI right where your team already works, no context switching required. Grafana Assistant in Slack is now generally available: investigate issues, ask follow-up questions, and query your metrics and logs from the channel where the incident conversation is already happening. And that’s just one place Assistant now meets you — also announced: the Grafana mobile app is GA (with Assistant on mobile in public preview), Assistant Desktop in public preview, Microsoft Teams support in private preview, and context-aware Assistant Search in public preview. Add it all up and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search. That’s a wrap on five days of announcements.

Transcript

It’s the final day of Grafana Labs’ AI Week, and Friday is all about collaborating — using AI right where your team already works, no context switching required.

Today, Grafana Assistant in Slack is generally available. Investigate issues, ask follow-up questions, query your metrics and logs — all from the channel where your incident conversation is already happening.

And that’s just one place Assistant now meets you. Today we also announced: the Grafana mobile app is GA, with Assistant on mobile in public preview. Assistant Desktop is in public preview. Microsoft Teams support is in private preview. And Assistant Search — deeper, context-aware search — is in public preview too.

Add it all up, and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search.

That’s a wrap on five days of announcements — and I know it was a lot. So if you missed a day, or you just want the full picture of where AI at Grafana stands today, I made a video for exactly that: everything you can do with AI in Grafana Cloud, including everything we shipped this week. It’s linked below and pinned in the comments.

One last time: grafana.ai for all of it. Thanks for spending AI Week with me.

Observability and AI explained

Fri, 31 Jul 2026 09:00:00 +0100

“AI observability” gets thrown around a lot, and it can mean one of two very different things. So in this video I map out the whole landscape of AI-related work at Grafana across the entire software development lifecycle — and show how AI and observability come together in two directions: AI for observability (AI that helps you observe your systems) and observability for AI (observing the AI itself).

The framing is deliberate: AI in observability isn’t about replacing engineers. It’s about handing back the exhausting parts of the job — the 3am triage, the repetitive queries, the report nobody wanted to write — so you can spend your energy on the work only humans can do.

I walk through each phase of the lifecycle and where the tools fit:

Planning — clarity about what’s really going on, with Grafana Assistant: not just a chatbot, but an agent that knows your telemetry, writes your queries, and builds your dashboards.
Building — momentum without losing control: Assistant, the open-source GCX CLI, the Grafana MCP server, the specialized Loki/Tempo/k6 MCP servers, and the AI SDK for building your own agents in Go.
Operating — staying calm under load: Investigations for agentic incident triage, Assistant Automations and the Watcher Agent for staying proactive, plus Grafana Machine Learning (Metrics Forecasting, Outlier Detection) and Adaptive Telemetry.
Testing & evaluating — doubt is a form of care: k6 and agentic testing on the systems side, and Agent Observability plus the open-source o11y-bench benchmark on the “observing your AI” side.
Collaborating — bringing it all to where your team already works: Assistant in Slack (and coming to Teams), on mobile, on the desktop, and via API/MCP, plus Assistant Search across your Grafana artifacts.

If you’re new to the Grafana stack itself, start with my open source observability video for the non-AI foundations that all of this builds on.

Resources

My notes on Grafana Assistant
My notes on GCX
My notes on k6
o11ybench.ai — the open leaderboard for AI agents doing observability work

AI Week Day 4: Evaluating (Grafana AI Week — Thursday)

Thu, 30 Jul 2026 09:00:00 +0100

Day four of AI Week is all about evaluating — because building an agent is the easy part; trusting it is where things get hard. The headline: Grafana Agent Observability is now generally available — the tool Grafana built for itself while scaling Assistant, once it grew too complex to evaluate with unit tests. Instrument your agent with the SDK and your conversations, tool calls, traces, latency, token use, and cost all become observable in one place; add evaluators (LLM judges plus deterministic checks) to score quality on live traffic, feed low-scoring conversations into your test suite, and gate CI on experiments comparing a candidate against a baseline. And because evaluator metrics land in Prometheus, you can get paged when a bad prompt change tanks quality. The fun bonus: k6 agentic testing — describe a browser test in plain language and an agent embedded in k6 drives the browser, no script required (they proved it by getting it to beat Wordle).

Transcript

It’s day four of Grafana Labs’ AI Week, and Thursday is all about evaluating — because building an agent is the easy part. Trusting it is where things get hard.

The big news: Grafana Agent Observability is now generally available. This is the tool we built for ourselves while scaling Grafana Assistant, when it grew too complex to evaluate with unit tests. Now it’s available to everyone on Grafana Cloud.

Here’s how it works: instrument your agent with the SDK, and your conversations, tool calls, traces, latency, token use, and cost all become observable in one place. Then add evaluators — LLM judges plus deterministic checks — to score quality on live traffic. Low-scoring conversations feed your test suite, and experiments let you compare a candidate against a baseline before you ship — you can even make it a CI gate. And because evaluator metrics land in Prometheus, you can get paged if a bad prompt change tanks quality in production.

But my favorite part of today? k6 agentic testing. Describe a browser test in plain natural language, and an agent embedded in k6 drives the browser for you — no script at all. And to prove it, we gave it one job: beat Wordle. Go watch that video. I’ll wait.

Everything’s at grafana.ai — see you tomorrow for the finale: using AI where your team already works.

Investigations Arena: how we test Assistant investigations

Thu, 30 Jul 2026 09:00:00 +0100

How do you test an AI agent whose whole job is to investigate incidents — when every investigation is different and “correct” isn’t a single deterministic answer? In this interview I sit down with Will, a software engineer at Grafana who’s spent the last year focused on AI: the Grafana Assistant, investigations, and memory features.

The story here is how the team got from Investigations V1 to a much better V2. V1 hit a ceiling, and the demo environment they’d been testing against turned out to be the problem — the cases were too easy and too far from the messy reality of a production cluster. So they built an arena: think competitive bot programming, but the “game” is investigating real incidents in Grafana’s own production environment. Different AI agents, each with its own strategy and tools, all take a swing at the same incident, and an LLM-as-judge scores their answers against a human-verified root cause.

What’s interesting is that the score wasn’t even the most valuable output at first — the insights were. Watching which strategies worked, which tools returned so much noise that they poisoned the context, and which agents wandered off is what let the team optimize bit by bit into the V2 release. The arena also compares more than models: different harnesses running Sonnet vs. Opus, Claude Code with the Grafana MCP, even just hitting the public API directly with curl.

A couple of my favorite moments: Will explains the guardrails that stop agents from “cheating” (they can only look at data from within the incident’s time window, so they can’t peek at the fix commit that resolved it) — and how the models mostly play fair because they treat it like a game. And there’s a lovely example of emergent creativity: an agent reached for tools the team hadn’t even considered, they turned out to be genuinely useful for investigation, and those tools got promoted into the real harness. Will’s hope for the future is to open this up — an OliBench-style shared benchmark where the community can throw their own investigation agents into the arena and raise quality across the board.

Transcript

Will: Hey, my name is Will. I’m a software engineer at Grafana. I’ve been there for three years, and for the last year I’ve been focusing on AI — the Grafana Assistant, investigations, and memory features.

Nicole: And what’s the latest thing you’ve been working on?

Will: So we developed Investigations V1, and we saw that we hit a ceiling at some point. We came up with a different approach, but this time we needed a good evaluation environment — the demo environment wasn’t enough, because those cases were too easy and too far from the reality of a production environment. So we decided to build an arena.

It’s a little bit like the idea of competitive bot programming, where bots fight against each other on specific tasks or games. In this case, the topic was investigations in our production cluster — real incidents. We started building bots that had different strategies and different tools; these were AI agents. Comparing a diversity of strategies let us surface the best approaches, and from there we could iterate, bring in more test cases from more incidents, and reach higher and higher quality.

The idea is that the bots try to solve the same case, and we use an LLM as a judge to compare the different approaches. We have different criteria, because for each incident we have a root cause that’s been verified by a human. That gives us our ground truth, and the LLM-as-judge compares that ground truth against the answers the agents came up with.

The important part at first wasn’t even the score — it was the insights. This tool was better with this strategy; this one had noise in the results that poisoned the context; the results returned from this specific tool were just too much. So we managed to really optimize bit by bit and come up with Investigations V2, which we just released. It’s a massive improvement compared to our V1 approach.

Nicole: So is it comparing just different models for investigations, or is it also comparing things like Claude Code and the Grafana MCP versus Assistant?

Will: Both, actually. We have different versions of our harness for investigations that we run with Sonnet and with Opus, but we also use the Claude Code harness with the Grafana MCP — and even just using curl to directly hit our public API.

Nicole: Do the models ever cheat?

Will: So far it’s been quite good. We have guardrails. Because we investigate past incidents, we need to make sure the LLM only looks at a specific time window — only from when the incident started — otherwise it would be able to find the pull request that was used to solve the incident. But they’re pretty good at following the rules; they take it as a game a little bit. We even have an agent that checks afterward, with the judge, to make sure they followed every rule.

It’s funny, actually — we had a case using Claude Code where it started to use some tools we hadn’t thought about in the first place, and they turned out to be very useful for investigation. We’d never considered them, so we brought them into our investigation harness and it helped a lot. Giving the agents freedom and creativity in this kind of environment is really useful. That diversity sponsors new ideas that we can put into our own work.

Nicole: Cool. Do you have any plans to publish this and share it so that other people can do it too — kind of like OliBench?

Will: Yeah, I think, why not? I think we can go this way too. I’m sure it would be useful for other agents. And it would be very nice if we could compare agents from other people and other companies — maybe with different harnesses, maybe even different tools — because right now it’s mostly about our MCP, our CLI, and our investigation harness. But if the community comes up with their own investigation agents — and I think I’ve seen a few — it would be fun to put them in the arena and help not just our product, but any product improve in quality.

Resources

My notes on Grafana Assistant

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Thu, 30 Jul 2026 09:00:00 +0900

This week I spoke at KubeCon + CloudNativeCon Japan 2026 in Yokohama, Japan. I gave a talk called The Great Doubt: What Building an AI Agent Taught Us About Trust. Here’s the abstract:

When we started building an AI assistant for observability, we thought the hard part would be making it smart. We were wrong. The hard part was knowing when to trust it.

AI agents hallucinate, forget context, and confidently give wrong answers. Traditional testing doesn’t catch these failures. What’s needed is evaluation grounded in systematic doubt.

This talk shares lessons from shipping an AI agent to production: how to build a golden dataset of use cases you must get right, how to use LLM-as-judge when there’s no ground truth, and how to use OpenTelemetry traces to debug eval failures. You’ll also hear what’s still unsolved: evaluating multi-agent handoffs and closing the feedback loop between what users ask and what your evals cover.

Kyoto School philosopher Nishitani Keiji called this “The Great Doubt” (大疑): questioning every assumption until only what survives is real. For AI agents, that’s not philosophy. It’s the job.

The talk walks through the layers of doubt we worked through while building Grafana Assistant: doubting the agent itself, then doubting our tests, then the LLM-as-judge, then our own scores — and finally doubting whether the doubt was even justified. Each layer needed a different tool, from a golden dataset run in a fully synthetic observability environment, through online evals on live traffic, to per-conversation OpenTelemetry traces that link every eval score back to the exact LLM and tool calls that produced it.

You can find the slides at nicole.to/doubtslides, and you can watch the full talk below:

Resources

Grafana Assistant (generally available for Grafana OSS, Grafana Cloud, and Grafana Play):

Assistant Investigations (Grafana Cloud):

Docs

Agent Observability (Grafana Cloud):

Docs

o11y-bench (OSS):

Site

Investigations Arena:

Video

My notes on Grafana Assistant
My Context Horizon piece Unlearnings from building Grafana Assistant
The Way of Zen by Alan W. Watts
Zen Mind, Beginner’s Mind by Shunryu Suzuki
Religion and Nothingness, Volume 1 by Nishitani Keiji
Philosophers of Nothingness: An Essay on the Kyoto School by James W. Heisig

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Wed, 29 Jul 2026 09:00:00 +0100

Day three of AI Week is about operations and maintenance — because once code reaches production, the operational work is still surprisingly manual: someone’s watching dashboards, deciding whether an error spike is real, and writing up what happened, every single day. Two Grafana Assistant features went GA to take that off your plate. Assistant Automations handle recurring work — save a prompt and run it on a schedule (like a morning brief on your service’s health), each run keeping its own history and able to notify you in Slack. Assistant Investigations is for when something actually breaks: it digs across metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report you can follow along with in a Workspace. You’re offloading vigilance, not accountability — the evidence is always there to review, and you still make the call.

Transcript

It’s day three of AI Week, and Wednesday is all about operations and maintenance. Because here’s the thing: once your code reaches production, the operational work is still surprisingly manual. Someone’s watching dashboards, someone’s deciding whether that error spike is real, and someone’s writing up what happened — every single day.

Today, two features of Grafana Assistant are now generally available to take that burden off your plate: Investigations and Automations.

Assistant Automations handle the recurring work of operating a system. Save an Assistant prompt and run it on a schedule — like a morning brief that summarizes the operational health of your service over the last 24 hours. Every run keeps its own conversation and history, and it can notify you in Slack when it’s done or needs approval.

Assistant Investigations is for when something actually breaks. Investigations digs into issues across your metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report. You can follow along in a Workspace, see what’s been ruled out, and add guidance while it works.

You’re offloading vigilance, not accountability — the evidence and reasoning are always there to review, and you still make the call.

Everything’s at grafana.ai — see you tomorrow for day four: evaluating your agents.

AI Week Day 2: Building (Grafana AI Week — Tuesday)

Tue, 28 Jul 2026 09:00:00 +0100

Day two of AI Week is all about building — the AI tools that help you write software with confidence. Two big announcements went generally available: gcx (the Grafana CLI) and the Grafana MCP server. Both connect your coding agents — Claude Code, Cursor, and friends — to your Grafana telemetry. The MCP server gives agents opinionated, ready-made tools for the common cases (querying metrics and logs, reading and updating dashboards); gcx is the more flexible, less opinionated option, built for agents from the ground up with read-only guardrails and its own bundle of agent skills. Both work for OSS and Grafana Cloud. The point: when an agent writes your code, it shouldn’t be guessing — it should be looking at how your system actually behaves in production. That’s telemetry-driven development.

Transcript

It’s day two of Grafana Labs’ AI Week, and Tuesday is all about building — the AI tools that help you build software with confidence.

Today we have two big announcements: gcx and the Grafana MCP server are both now generally available.

Both of them connect your coding agents — think Claude Code or Cursor — to your Grafana telemetry. So what’s the difference?

The Grafana MCP server gives your agent opinionated, ready-made tools for the most common use cases: querying your metrics and logs, reading and updating dashboards, that kind of thing. You can run it yourself, or use the hosted MCP server with your Grafana Cloud stack — so yes, it works for both OSS and Grafana Cloud.

gcx is the Grafana CLI, and it’s the more flexible, less opinionated option. Agents can explore it and build their own workflows, and it was designed for agents from the ground up, with read-only guardrails so your agent can’t touch anything you didn’t approve. It works with Cloud stacks and self-managed instances, and it even ships with its own bundle of agent skills.

Why does this matter? Because when an agent writes your code, it shouldn’t be guessing — it should be looking at how your system actually behaves in production. That’s telemetry-driven development.

Head to grafana.ai for everything we’re shipping this week, and I’ll see you tomorrow for day three.

AI Week Day 1: Planning (Grafana AI Week — Monday)

Mon, 27 Jul 2026 09:00:00 +0100

It’s day one of Grafana Labs’ AI Week, and Monday is all about planning — getting the clarity you need on what’s going on in your system so you can figure out what to build. Kicking things off is Workspace for Grafana Assistant: a full-page view of Assistant so your conversations and generated graphs aren’t crammed into the little right-hand sidebar. It’s a much less cluttered place to run investigations and to be agent-first. On top of that, Assistant is now easier to try — enable it in Grafana 13 for OSS without linking a Grafana Cloud account, or try it with no signup or download at all on Grafana Play.

Transcript

Hi everyone! I’m Nicole van der Hoeven and I’m really excited to welcome you to AI Week. Today is Monday, which is planning day, where we give you the tools you need to get clarity on what even is going on in your system so that you can figure out what to build.

Now we’re kicking things off with a feature called Workspace for Grafana Assistant. Now if you’ve used Assistant before, you’ll know that it comes up on the little sidebar on the right side of Grafana. Well, that can start to get cramped if you have multiple conversations or if you’ve asked Assistant to generate multiple graphs for you. Well, we’ve heard you, and now we’re giving you a full page view of Assistant. So you’ll have all of your conversations on the left, and this is also an excellent place to run investigations from. You can also zoom in on any of those graphs, and it’s just a much less cluttered view if you want to be agent-first.

Speaking of Assistant, the second thing I want to tell you is that you know, Assistant was already available everywhere. But now, Assistant is available… in more places than everywhere? And also more easily. So Assistant was already available for OSS, but you had to link your Grafana Cloud account, which was not so handy if you wanted to try it out. But not anymore. Now if you download Grafana 13 and enable Assistant, then you’ll just be able to go and try it out and chat with Assistant without doing anything else. It’s still not free or open source, but it IS a lot easier to try it out.

And if you don’t want to download anything, that’s what Grafana Play is for. Grafana Assistant is now available to try out on Grafana Play without any signup or any download required. Try it out at play.grafana.org. I really love seeing it there. Tune in tomorrow where we’re going to talk about everything we’re announcing to help you start to build.

An Overview of AI Evaluation (The Context Window #05)

Fri, 26 Jun 2026 18:00:00 +0100

How do you know if an AI agent is actually any good — and whether the change you just made to it made it better or worse? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Yas Ekinci, an engineer on Grafana’s AI team who has spent the last several months focused almost entirely on evaluation. We talk about why trusting AI is so hard (non-deterministic models, oversized PRs, and the shift from writing code to reviewing it), the difference between online and offline evals, how we actually test Grafana Assistant, and o11y-bench, the open observability benchmark we announced at GrafanaCON. We recorded this one ahead of Yas’s two-month leave — so consider it a deep dive banked for the road.

The “final answer problem”

A recurring theme — and one close to my testing heart — is what I call the final answer problem. As a tester I’m used to looking at an outcome and asking “did this pass or fail?” But with AI, a plausible-looking answer isn’t enough. If the process was wrong, you can’t trust the outcome even when it looks right — and a plausible-but-wrong answer is the dangerous case, because nothing on the surface tells you it’s wrong. Yas’s example: a broken Tempo tool that returned no spans, while the agent still answered convincingly by guessing from metadata. Plausible ≠ correct.

This is also why the bottleneck has moved. The cost of writing code has collapsed, so the burden now sits on review. Developers, as I like to put it, are becoming shepherds of intent — less about typing the code, more about making sure the intent is clear and faithfully implemented.

What evals are, and online vs offline

An eval is simply a test of whether an AI agent behaves the way you want — from “who are you?” → “I’m Grafana Assistant” all the way up to multi-step querying and dashboarding. The point is to know with confidence whether a change made the agent better, which is what lets you improve it consistently over time.

There are two complementary flavours, and they map neatly onto testing concepts we already have:

Online evals — scoring a sample of real production conversations, usually with an LLM-as-a-judge. Great as a pulse check (it’ll catch a quality drop when a model provider silently swaps models), but it only tests the deployed variant and is hard to verify. Closest analogy: traditional observability / RUM.
Offline evals — a “golden set” of tasks run during development and CI against a controlled, reproducible environment. You can test every change, prompt, and implementation, and drill into specific skills (PromQL vs LogQL, dashboarding). Closest analogy: integration testing.

How we test Grafana Assistant

Assistant evals run a real agent-plus-tools harness against a real-ish environment (a Grafana with data sources, alerts, and dashboards), then grade the full transcript with “graders” or “verifiers.” The hard tension is that the environment has to be real and reproducible at the same time. Because Assistant is a front-end plugin that runs some tools in the browser, the team uses Playwright to replicate real browser behaviour — which is where I made my pitch for k6’s browser module (now approaching Playwright parity) as a dogfooding option. The internal eval tool is a CLI called LLM Spec, now on its third major iteration after the lessons from o11y-bench.

o11y-bench

o11y-bench is our public benchmark, announced at GrafanaCON, with six task categories: dashboarding, investigation, logs, metrics, and traces. A few things I think make it genuinely interesting:

It’s built on the Harbor framework (also used by terminal-bench), which separates harness from environment from tasks — so you can swap harnesses and compare, say, gcx + Claude Code against the Grafana MCP.
The leaderboard includes open models (Qwen 3.6, Kimi), not just the frontier labs.
It distinguishes pass@3 (got it right at least once in three runs — rough capability) from pass^3 (got it right all three times — reliability). The 10–20% gaps between the two are the whole reliability story in a single number.
It runs against a synthetic environment (Prometheus, Loki, Tempo in Docker) with generated data, so the correct answer is known in advance — and it uses fact-based rubrics, where the grader runs its own known-good queries against the environment and checks the agent’s answer against them. That’s how you catch the subtle, plausible-but-wrong failures.

The internal eval set is a superset that adds scenarios drawn from real (private) conversations and Assistant-specific behaviour — like Slack’s not-quite-markdown formatting — while o11y-bench stays generic and harness-independent, grading on outcome rather than method.

The evaluation loop

Yas’s blog post (the first engineering post on Grafana’s new community Unprompted blog) and our GrafanaCON talk both center on the evaluation loop: measure → make changes (new tools, prompt tweaks, bug fixes) → measure again. The crucial bit is that the loop can be driven by a coding agent — via an LLM Spec Improve skill — because you can’t unit-test natural-language “programming” the way you test code. There’s no clean relationship between the instructions you write and what the model does, so the only way to know the impact of a change is to run the loop.

Where it’s going

We closed on open questions and wishes: higher environment fidelity, more and harder tasks (very likely an o11y-bench 2.0), and getting more contributors — including PMs writing evals to capture desired product behaviour. I floated a k6-extension-style community model for evaluators, and using o11y-bench to support bring-your-own-model, so users can see exactly where a smaller or local model falls short instead of assuming Assistant is broken. Yas’s industry hope is the neat one: if labs tune their models against o11y-bench, observability and Grafana performance improve across the board — free fine-tuning out of the box.

Resources

Transcript

Nicole: Hey everyone. Welcome to another episode of The Context Window. This is a semi-regular podcast — we try to make it every month — and it’s all about the things going on with AI at Grafana. Usually this is live, and you might think looking at this that it’s live, but it’s actually prerecorded, because Yas has a busy schedule ahead and I wanted to get him on here before he left. So hi, I’m Nicole van der Hoeven, a senior developer advocate at Grafana Labs, and I’m assigned to the AI team these days. And one of the people on the AI team is — this person over here. How do you say your last name, actually?

Yas: Oh, it’s Ekinci.

Nicole: Ekinci. I was totally confused because I just found out you speak Dutch. Did not know that.

Yas: I’m based in Belgium, and I was like, “Nicole van der Hoeven” — okay, that sounds familiar enough to me.

Nicole: Yeah, it’s the name of my ex-husband, but it’s okay, we still like him. So tell us what you do on the AI team.

Yas: I’m an engineer on the AI team — I’m getting to five years at Grafana next week. When I joined, this was pre-AI; we had the machine learning team building predictive models, forecasting, that kind of stuff. As the AI wave came, we all got absorbed into this newer, larger AI department. It’s been quite varied — the assistant, the AI agent inside Grafana — but especially the latter half of this year I’ve been more and more focused specifically on evals: how do we figure out the quality and reliability of these things we’re deploying?

AI is a very useful tool. It has utility — it’s not like NFTs. We use it day to day. My job has changed drastically over the last two years. But you still notice it needs guidance and sometimes has reliability issues. The trend is that the guidance has come down — a year or two ago I’d steer an agent 50, 60, 70 times in a session; now it’s more like five or ten. But you still need to do it. If you just blindly vibe-code your way out of this, you’re not going to end up in a good place.

Nicole: It moves so quickly. Whenever someone tells me they tried AI and it didn’t do what they wanted, I ask, “Well, when was that?” Because if it was six months ago — try it again. In this episode we’ll talk all about evals, but first, some announcements. Assistant is now available for OSS and self-hosted Grafana, and I made a video showing how to do that. There’s also a new Medium blog for Grafana Labs called Unprompted — a community-focused blog — and Yas wrote the first engineering post, about building an evaluation loop. And we’ve started AI weekly videos: the AI team has a weekly meeting, Mat Ryer had the idea to publish excerpts, usually the demo part. Just be warned we demo things that may never make it to production — you’re getting it raw, warts and all.

Yas: I need to make sure I don’t wear my PJs anymore for the AI weekly.

Nicole: A bit of quick AI news: Warp is now open source. There are new models — GPT-5.5, Gemini 3.1 Ultra with a two-million-token context window, Anthropic’s Claude Mythos security preview. And there’s a great article by Matt Shumer called “Something Big Is Happening,” where he says, “I am no longer needed for the actual technical work of my job” — which hits hard — and advises leaning into what’s hardest to replace.

Yas: It’s still that guidance thing — you need a human in the loop to guide it. But if you ask me how much code I’ve manually written lately, the actual technical piece is very close to zero. We’ll talk about how that’s different for evals, because evals don’t work as well as coding does there.

Nicole: So why is it so hard to trust AI? Why don’t we just YOLO it and accept everything?

Yas: A mix of things. Everybody knows these models are non-deterministic and not perfect — these aren’t new problems; underneath it’s all machine learning, numbers and math and stats. Every time you talk to it you might get a slightly different response. The second part is it writes a lot of code. You can 10x your output — send ten agents at ten different features and launch all those PRs — but then someone needs to review them, and the PRs are typically big and over-engineered. One thing I always do at the end of a piece of work is ask it to find a simpler way and simplify. Bigger PRs plus so many changes makes it hard to keep a mental model of the whole thing. And interestingly, I’ll use AI to help review a PR — which gets you back to the same problem: if you use AI to review the AI, how much can you trust that?

Nicole: I like the phrase that developers are becoming shepherds of intent — we’re not writing the code so much as making sure the intent is clear and implemented. And this is the difference between process and outcome — what I call the final answer problem. I come from performance testing; I’m used to looking at the outcome and asking did it pass or fail. But now, even if the outcome looks right, if the process was wrong I don’t trust it. And the reverse too: right process, wrong outcome, something else is broken.

Yas: Tests are interesting here, because most of those tests are also AI-generated — and the AI doesn’t shy away from just changing the test to make its branch pass. For any branch I review, I still look at all the changes; I won’t blindly rubber-stamp. That’s probably the biggest bottleneck right now.

Nicole: One approach to trusting AI is human evaluation — thumbs up/thumbs down, or something like LMArena where you compare two answers blind. Useful, but not scalable; you can’t run it in CI.

Yas: On the eval side they’re all complementary. So, what is an eval? It’s testing whether the AI agent behaves like you want. A simple one: ask “who are you?” and expect “I’m Grafana Assistant.” There are much more complicated ones around querying and dashboarding. At its core, an eval is how you figure out if the agent is any good — and the reason we do it is so that when we make changes, we know if it’s better. Answer that with confidence and you have a path to improve consistently over time.

We split into online and offline. Online — we recently launched AI observability for agents — is about measuring how the deployed agent does against real production conversations from real users. Valuable, because you get a score based on real behaviour, but hard to verify; we typically use an LLM as a judge, so it’s not as robust. Offline is used more during development, in CI — much more controlled. You have a golden set of evals covering the behaviour you expect in production, and because you always test the same set, you can see whether changes make it better, and dig into PromQL vs LogQL, or dashboarding. Online is a great pulse check — it can catch quality dropping when Anthropic swaps a model — but it only tests the deployed variant. Offline can test all your changes, your implementation, and all your prompts.

Nicole: It took me a while to translate these, because the testing industry already has this — we call it pre-prod and production testing. Pre-prod is everything from unit to system integration testing in dev and staging; you can really hammer the thing. Production you have to be more careful — synthetic monitoring, real user monitoring — same idea with an AI flavour.

Yas: Definitely. Online is much more like traditional observability — you’re observing a system that happens to be powered by AI, turning natural language into a number you track over time and alert on. Offline is closer to an integration test: you give it a task, it runs the whole agent against a real environment, you get something out, and then separately you grade that outcome.

Nicole: We did an AI observability episode without you — you were at GrafanaCON — so I’ll link that. Let’s talk more about offline evals, since we haven’t covered those.

Yas: Offline is a bit of a beast. There’s a good Anthropic post called “Demystifying evals.” Back when we had simple single-turn LLM calls — incident summary was one of the first — the prompt was “here’s the context of the incident, summarize this,” and grading checked readability, completeness, appropriate length. Easy enough to set up. Agents get more complicated, because every time you run a task the agent can take very different trajectories — five steps one time, seven the next. The agent plus its tools is roughly the harness; it runs against an environment — a real Grafana with data sources, a certain alert and dashboard state — and keeps interacting using the tools until it’s done. Then we take the whole transcript and grade it with graders, sometimes called verifiers, to verify it actually got the task right.

Nicole: So how do you do it for Assistant, given how broad that environment is?

Yas: We have the public benchmark we released recently and our internal evals — same approach, different number of tasks and some different data sources. We want an environment as real as possible, but with the tension that we also need to control it so a benchmark is reproducible. If two people run it and get different scores, which one was right? We started, like everybody, with manual tests — listing data sources to confirm a tool is wired up, some CPU questions, showing logs — with a human as the judge. That’s fine at four or five scenarios; you can’t do it as a human at 500. Assistant is a front-end plugin — some tools run in your browser, which is good for auth and security but makes testing hard — so we use Playwright to replicate real behaviour and pull the telemetry out. The internal eval tool is a CLI we call LLM Spec, and it can run against cloud instances or a local controlled environment.

Nicole: Before the environment — did you consider using k6 instead of Playwright?

Yas: Interesting — yes and no. k6 is also a tool the assistant needs to use, but our goal isn’t to load test; it’s to hook into the plugin and get the messages and telemetry out, so we have a Playwright harness.

Nicole: There’s actually a browser module for k6 going towards feature parity with Playwright.

Yas: I did not know that. Might be interesting.

Nicole: Just dogfood all the things. I’d love to work with you on that. Okay, the environment.

Yas: We test against cloud instances — for incidents especially, it’s a lot of data. And we run LLM Spec against a local controlled environment, which is quite similar to the open benchmark we published.

Nicole: You’ve mentioned o11y-bench — the benchmark announced at the last GrafanaCON. That was fast.

Yas: o11y-bench has tasks across six categories — dashboarding, investigation, logs, metrics, traces. We wanted to bring our internal approaches outward, and it was a learning opportunity: we used a framework called Harbor, also used by the makers of terminal-bench. The nice thing is it separates harness from environment from tasks — so the default harness is a simple system prompt plus the Grafana MCP, but the folks working on GCX run Claude Code plus GCX against o11y-bench and compare GCX with MCP. There’s a leaderboard with a bunch of models, including open ones like Qwen 3.6 and Kimi.

Nicole: They’re getting good. Qwen 3.6 is awesome — not top five, but for an open-weight model, impressive.

Yas: And this is the interesting part: sort on pass@3 and it looks different from pass^3. pass@3 runs each task three times and passes if it gets it right one out of three — rough capability. pass^3 means you need three out of three on every task — reliably solving it, not just getting lucky once. You see 10–20% differences between the two scores.

Nicole: That’s awesome. And this is public, right?

Yas: Yep — the repo’s on GitHub with all the tasks. We run o11y-bench against a synthetic environment — a Prometheus, Loki, and Tempo in Docker — and we generate the data, so every run uses the same data and we know what the answer should be before we even run it. We also do fact-based rubrics: separately from what the agent does, the grading side runs its own known-good queries against the environment and checks whether the agent’s answer matches. That matters because these agents fail in subtle ways. We once made a change to the Tempo tool that broke the spans — the agent got no spans but still answered, fairly realistically, from metadata. Without testing for it you’d never catch it; the text looked plausible.

Nicole: That’s exactly process versus outcome. If the answer were negative two seconds it’s obviously wrong, but a plausible answer is the dangerous one.

Yas: Exactly, and that happens a lot.

Nicole: An early Assistant once told me it created a dashboard, and it didn’t look right — turned out it couldn’t find the data source so it just used dummy data.

Yas: Same thing happened to me locally — I broke all the tools, and it still came up with tool calls it couldn’t execute because of rough instructions in the system prompt.

Nicole: So what’s missing in o11y-bench that means you still need an internal one?

Yas: Two levels. First, some scenarios are based on real conversations or our own internal environments — anything involving private data can’t go public. Second, there are very Assistant-specific things we want to test — Slack, for instance, has its own version of markdown that looks like markdown but isn’t, and early Slack Assistant struggled to write it consistently, so we wrote tests for that. The internal set is a superset; the public set is still part of it. The public set is generic observability tasks — asking questions about your data, dashboarding, basic Grafana API checks — and because the harness is independent, we can test GCX, MCP, our assistant, or any other agent against it, grading on the outcome rather than how it was solved.

Nicole: We talked about your blog post — let’s define the evaluation loop.

Yas: At its core: measure the current state, see where it works and where it doesn’t, make changes — a new tool, tool instructions, the system prompt, bug fixes — then measure again, and again. The key is that loop can be driven by a coding agent, not us manually. If your evals are robust enough to pass a clean feedback signal back, the agent can keep improving until it hits 100%, and then you make harder tasks or increase coverage. At scale it’s nearly impossible for one human to understand the whole system and make targeted improvements, because an agent isn’t programmed as code — it’s programmed with natural language, and there’s no clear relationship between the instructions and what the model does. The only way to probe that is to run a bunch of these loops.

Nicole: Is the loop equally implemented for o11y-bench and LLM Spec?

Yas: It’s implemented internally. o11y-bench is really just the measurement part — we’ve used it for GCX and MCP, but more manually. In LLM Spec we have an LLM Spec Improve skill — you pick the agent, and it runs measure-improve-measure-improve.

Nicole: Where are you going next — no promises?

Yas: I’ve done a lot on the harnesses — the Playwright stuff, and there was an LLM Spec 1.0, then a 2.0 revamp for speed and robustness, then o11y-bench, and the learnings from that went into what I’d call LLM Spec 3.0. The two big challenges now are environment fidelity — more an issue for internal evals — and the number and quality of tasks: how do we cover more of Grafana? We weren’t sure o11y-bench would be a recurring thing, but there’s a lot of interest, so we’ll probably do an o11y-bench 2.0. And I want more people contributing — I was talking at the PM all-hands about getting product managers to write evals, because the tasks capture desired behaviour: when someone asks about my product, how should the agent ideally respond?

Nicole: What I’d love is something like the k6 extension model applied to evaluators. When Grafana acquired k6 — I was on k6 — k6 had extensions and modules, and we got a lot of cool community extensions. I think evaluators would be the same. After my Madrid talk, all the questions about the evaluator section were “can we add this, can we add that, why isn’t this in the template?” — and it could be.

Yas: It would be nice to do that across both o11y-bench and the internal ones — there are open questions on making them work together; internally we’re in Go, o11y-bench is on Harbor in Python, but the tasks and most verifiers are language-independent by design. A fourth thing is getting to a better golden set over time — we haven’t yet automated going from real AI-observability conversations to verifiable offline tasks. As the product changes, your tasks might not stay relevant — imagine k6 didn’t exist and now people ask Assistant about k6. Bridging online and offline in a more automated way would be the fourth big thing.

Nicole: I also have a wish for o11y-bench beyond being a benchmark — with no internal knowledge and no promises: it could be a step toward making Assistant bring-your-own-model. One barrier is that people try Assistant with a tiny local model and conclude it doesn’t work, when really you need certain models. A standardized way to measure that and show users where their model falls short — versus where the latest Opus sits on the leaderboard — lets them make an informed decision.

Yas: I’d go beyond Assistant — we’ve got GCX and MCP too. If you want to use your own models with them, the benchmark should reflect, for observability tasks, which are the best models and the best open models. It gets to your point: people can make a more informed decision.

Nicole: Where are we still missing things as an industry, not just as a company?

Yas: Just recently a new benchmark called DeepSWE was released, and it’s a bit controversial because it flips the ranking from the most popular software-engineering benchmark — where Opus was always leading, here GPT-5.5 leads by a margin. So there’s something to be said about the reliability of the benchmarks themselves, since labs test and sometimes tune their models against them. That might actually be good for o11y-bench — I’d want a lab to tune against it, so we get free fine-tuning out of the box: as newer models use o11y-bench to tune on observability tasks, they get better at using Grafana and solving observability problems.

Nicole: Thank you — we just hit time. It was really informative, and I’m glad we both found time; this got rescheduled for various reasons. A great overview of all the evals we’re doing.

Yas: Thanks — it’s nice to step out of the day-to-day and talk about this stuff.

Nicole: And if you’re watching, leave any eval questions in the comments and I’ll bug Yas to answer them. Thanks everyone for watching.

Yas: Bye.

CLI and MCP Basics for AI Agents (The Context Window #04)

Wed, 03 Jun 2026 18:00:00 +0100

MCP vs CLI: which one should your AI agent actually use? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and three of the engineers and PMs behind Grafana’s agent tooling: Ben Sully (who wrote the Grafana MCP server), and Ward Bekker and Dafydd Thomas from the team building gcx. We get into the two main ways an AI agent can reach the outside world — the Model Context Protocol (MCP) and good old command-line interfaces — and look at when each one makes sense, the tradeoffs around context windows, tool sprawl, token cost, and reliability. We walk through Grafana’s cloud and open-source MCP servers along with gcx, the Grafana Cloud CLI built for agents, do a live side-by-side demo of both creating an SLO, and talk about how we benchmark them against each other with o11y-bench.

Timestamps

00:00:00 — Introductions
00:02:49 — The last month in AI news
00:10:19 — What is Grafana Assistant?
00:12:30 — What does it mean for Assistant to use tools?
00:13:30 — What is MCP?
00:16:56 — What is a CLI in an AI context?
00:19:17 — What are skills?
00:24:37 — Can you use a custom MCP server with Grafana Assistant?
00:25:55 — Why team MCP?
00:28:22 — The context-window cost of MCP tool definitions
00:31:50 — Tool discoverability
00:39:25 — What is gcx, and why a CLI?
00:46:13 — Demo: creating an SLO with MCP vs gcx
00:52:58 — Benchmarking MCP vs gcx with o11y-bench
00:58:10 — How MCP and gcx relate to Grafana Assistant
01:02:35 — k6 MCP vs Grafana MCP
01:03:00 — When to use which (decision guide)
01:04:29 — Adding the OSS Grafana MCP to Assistant
01:05:41 — Turtles all the way down

Resources

MCP and CLI for agents

My Context Horizon piece: Unlearnings from building Grafana Assistant

Grafana Campfire 🔥 — GrafanaCON 2026 recap

Thu, 28 May 2026 17:45:00 +0100

I co-hosted this month’s Grafana Campfire with Stephanie Hingtgen, where we recapped everything that came out of GrafanaCON 2026. Stephanie owned the Grafana 13 walkthrough — Git Sync GA, Dynamic Dashboards GA, Dashboards API v2 GA, Grafana Advisor GA, SCIM GA — what she described as the year Grafana stopped being beta in production. I covered the AI section (Grafana Assistant now in OSS and Enterprise, the three scaling challenges from “Scaling Grafana Assistant”, Investigations, Skills going GA), the three-teams-one-architecture push across Loki Thor, Mimir 3.0, and Pyroscope 2.0, the OTel + Alloy roadmap, alerting, k6 v2.0, and the Marketplace and Logline.

We also took questions from the Champions community on the pace of UI change, Mimir Alertmanager, and the agent-scale question — what happens when your AI agents are themselves the noisiest users of your observability stack? — and threw in a flavour break about the TamaGROTchi and the ocean digital twin demo.

Resources

Asimov's Zeroth Law of Robotics: Observability for AI (ExpoQA Madrid 2026)

Tue, 26 May 2026 10:00:00 +0200

This week I gave the opening keynote at ExpoQA Madrid 2026 in Madrid, Spain. I presented a talk called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously given at KubeCon EU 2025 in London, at Dutch Cloud Native Day 2025 in Utrecht, and at Newcrafts in Paris. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

This was a keynote slot at a testing-focused audience, which gave me the chance to lean more into the evals vs. observability framing — what evals can and can’t tell you about an AI system in production, and how distributed tracing fills the gap that benchmarks leave behind. The demo app (a Flask-based “play.py” instrumented with OpenTelemetry, flowing into Tempo for traces, Prometheus for metrics, and Loki for logs) is still on GitHub, and I’ve been refining it with each iteration.

Here’s the recording of the talk:

Resources

Demo app on GitHub

Do Androids Dream of Second Brains?

Mon, 18 May 2026 07:00:00 +0100

We often describe our personal knowledge management systems as “second brains” — but can we really say they think for themselves? In this video, I make the case for instrumenting your vault with an observability stack and layering AI on top, not because it’ll turn your notes into a brain, but because it might get us closer than anything else has.

This is the video version of the talk I gave at PKM Summit 2026 in Utrecht earlier this year, with the live demos cleaner and the dashboards working without the conference wifi gods getting in the way.

What I cover

🐑 Why “second brain” is the wrong framing

I take a small detour through Philip K. Dick’s Do Androids Dream of Electric Sheep? — the inspiration for Blade Runner — and the Voigt-Kampff test, which measures empathy through observed responses to staged scenarios. It’s a useful lens for asking what we actually mean when we say a knowledge system “thinks”: we’re really asking whether it can respond to us in a way that surprises us. Most second-brain setups can’t, yet.

📊 Observability for PKM

Observability is a concept I’ve borrowed wholesale from how we monitor software systems — the idea that you can understand a complex system by what it emits, not by reading its internals. Three steps: instrumentation, analysis, improvement. I walk through what each one looks like for a personal vault:

Instrumentation: a small script that emits metrics about my Obsidian vault — note counts, link density, growth over time, what folders are loud and which are quiet
Analysis: a Grafana dashboard sitting on top of Loki + Prometheus, surfacing things I genuinely didn’t know about my own vault
Improvement: the part where seeing the data actually changes how you work

There’s a live dashboard tour, and a section where I ask Grafana Assistant to edit the dashboard for me in natural language — including writing the PromQL queries I never want to write by hand.

👁️ Meet Iris

About halfway through the video I introduce Iris — the AI agent I built using OpenClaw that lives in my vault, reads my notes, and works alongside me. I walk through:

What “agentic AI” actually means in practice (vs. just chatting with an LLM)
The OpenClaw setup — Docker sandbox, Syncthing, the whole pipeline
Iris rewriting my observability map of content
Iris doing talk-topic analysis on my vault
Iris running benchmark research for a blog post
Iris OCRing my reMarkable handwriting

🌙 Dreaming

The last section is the one I had the most fun with. I argue that the thing most second-brain setups are missing isn’t more notes or better search — it’s synthesis. The capacity to take everything you’ve captured and surface patterns, contradictions, and ideas you hadn’t formed yet.

I introduce OpenClaw’s dreaming feature: a memory consolidation process inspired by how sleep stages work in the brain. Light sleep, REM, and deep sleep each do different work — surfacing recent material, finding unexpected connections, and consolidating long-term memory. I show how each phase actually behaves in my own system.

And there’s a Zhuangzi reference. (You’ll see.)

Resources

🎤 PKM Summit 2026 talk recap (with photos)
🎞 Slides
👁️ OpenClaw
📓 Obsidian
🔄 Syncthing
📊 Grafana

The Architecture of Forgetting — the essay version of why memory consolidation matters
Unlearnings from building Grafana Assistant

If you want to follow more of how Iris works and what I’m learning about AI + PKM, Context Horizon is where I write about it.

High INT, Low WIS: What LLM Benchmarks Miss

Thu, 23 Apr 2026 09:00:00 +0100

Originally posted here on Context Horizon.

In the tabletop roleplaying game Dungeons & Dragons (D&D), the very first thing you do when you create a character is to choose ability scores. Ability scores are numbers that determine how good you are at the game’s main characteristics: Strength (STR), Dexterity (DEX), Constitution (CON), Intelligence (INT), Wisdom (WIS), and Charisma (CHA). 20 is the default maximum that a character can have in any ability, though some in-game events/items can increase that.

Here’s the thing, though: every D&D player knows that a 20 Intelligence doesn’t mean your wizard aces every check. Whether you have proficiency in the specific skill still matters. That 20 INT wizard, for all their raw cognitive horsepower, can absolutely still fail to identify a magical rune. The ability score is a starting point, not a verdict.

This is exactly what’s happening to LLM benchmarks. All the general LLM/AI model benchmarks are ability scores. They tell you something real about a model’s general capabilities, the same way a 20 INT tells you something real about a wizard. But as frontier models crowd toward the top of every leaderboard, we’re running into a problem D&D solved decades ago: the stat block isn’t the character. And it was never supposed to be.

A screenshot of the stats of my Level 10 bard. Low INT, but look at that CHA!

The stat block: A tour of the ability scores everyone’s maxing out

There are so many new models and model versions out there that it’s hard to keep up with what we’re supposed to be using for which tasks. LLM benchmarks have come to be relied on to give a somewhat objective assessment of each model’s capability. Here are a few big ones that exist.

General knowledge/reasoning

MMLU (Measuring Massive Multitask Language Understanding, 2020)

MMLU is the classic “general knowledge” INT check. It was released in September 2020 by Hendrycks et al. and contains 15,908 multiple-choice questions across 57 subjects (humanities, social science, STEM, other). However, there were a few issues with it:

a study in 2024 showed that 6.49% of those questions contained errors
The New York Times reported (2024) that data contamination was an issue — models could be trained on the MMLU questions (and answers) to inflate their MMLU scores
many frontier LLMs soon consistently achieved high scores

In response, a new benchmark called MMLU-Pro was created with 12,000 questions that each have 10 options. The increase in options theoretically required more reasoning capability than the original benchmark, but frontier models are already knocking at 90% on the MMLU-Pro.

Humanity’s Last Exam (HLE, 2025)

HLE was developed specifically to address the fact that LLMs “now achieve over 90% accuracy on popular benchmarks like MMLU” (from the HLE site). The paper by Phan et al explains the lengths they took to make it hard enough to be meaningful:

they solicited high-quality questions from nearly 1000 subject matter experts, even going as far as paying them for successful submissions (with a total prize pool of US $500,000)
they included graduate-level classics (one of the questions includes translating Palmyrene script)
they reviewed potential submissions using current LLMs and only proceeded with submissions if the LLMs are unable to answer correctly
experts then reviewed the submissions further

As a result, HLE scores are low — intentionally so:

While current LLMs achieve very low accuracy on HLE, recent history shows benchmarks are quickly saturated – with models dramatically progressing from near-zero to near-perfect performance in a short timeframe [13, 45]. Given the rapid pace of AI development, it is plausible that models could exceed 50% accuracy on HLE by the end of 2025. (Phan et al., 2025)

Software engineering

HumanEval (OpenAI, 2021)

In 2021, OpenAI released a paper and an evaluation harness for a benchmark consisting of 164 Python programming problems based on publicly available code on GitHub. It was developed to test coding in particular.

Eventually, though, experts found that the problem sets in HumanEval were insufficient and ambiguous (Liu et al., 2023). HumanEval+ was released to include more tests, but the next benchmark’s authors suggested that models might be overfitting to HumanEval+ as well, memorizing the dataset and answers rather than genuinely demonstrating reasoning ability.

LiveCodeBench (2024)

Jain et al. tried a different approach to HumanEval to avoid data contamination: it was released with a continuously-updated coding benchmark. Problems have release dates and models can only be evaluated using problems released after their training data cutoff dates.

LiveCodeBench represents a shift in benchmarks towards a more continuous, dynamic measure. However, the last submission to the benchmark was 9 months ago at the time of this writing, and the leaderboard has not been updated since then either, suggesting the benchmark is no longer being actively maintained.

SWE-bench (2024)

This benchmark was created by Jimenez et al. to include 2,294 actual GitHub issues from 12 Python projects. When it launched, it was a much harder benchmark than predecessors, with models scoring 2-4%. Today, models are scoring around 70-80% on the original benchmark. There were enough issues with the original’s dataset quality that variants were introduced, including OpenAI’s SWE-bench Verified. Today, the SWE-bench website shows these variants:

SWE-bench
SWE-bench Verified
SWE-bench Multilingual
SWE-bench Multimodal
SWE-bench Lite

Agents and tool use

This last category is a preview of where I think the field should go: benchmarks that measure what a model does rather than just what a model knows. Here are some of them:

GAIA (Mialon et al., 2023) — 466 real-world assistant tasks that combine web browsing, file reading, and multi-step reasoning. Easy for humans (92% success), hard for models (15% success for GPT-4), at least at launch.
Terminal-Bench — agents working in actual Linux terminals. Tasks include “build the Linux kernel from source” and “configure a git server plus a webserver on port 8080.” No mocks; if it doesn’t actually build, it fails.
WebArena (Zhou et al., 2023) — realistic simulated websites (Reddit, GitLab, e-commerce) where agents complete actual user goals like “filter the last six months of orders by refund status.”

These benchmarks are harder to saturate than general-knowledge ones, because passing means the thing actually has to work in the real world. A model can memorise a question; it can’t memorise a Linux kernel that isn’t there until it builds it.

The wrong answer: Build a harder general benchmark

By now, you probably see that benchmarks tend to go the same way: A benchmark is created, models get too good at it and perform too well to be useful, and then a variant benchmark is created to address the problem.

This is the saturation problem in a nutshell: a benchmark that once provided meaningful information to help distinguish models eventually fails to do so because all the models’ scores cluster at the top.

Saturation occurs for two reasons:

Contamination: Many benchmarks publish their problem sets online, and models train on data scraped from the web. The benchmark now measures memorization, not capability.
Overfitting: Model providers are incentivized to optimize specifically for gaming performance on benchmarks. Scores go up without corresponding capability improvements.

When you have a party of wizards with 20 INT, “What’s your INT score?” stops being an interesting question because any wizard’s Fireball will hit the same. Building harder benchmarks is like an arms race for ability scores, when what you actually want to know is whether your wizard can kill a dragon.

Harder benchmarks don’t tell us what we want to know: can this model do my specific job?

The right answer: skill checks with real DCs

The fix isn’t a harder ability check. It’s a different kind of check altogether. In D&D, straight ability checks are rare compared to skill checks. There are 18 official skills that break down the six ability scores into more granular activities. For example, your wizard is much more likely to roll one of these instead of a straight INT check:

Arcana
History
Investigation
Nature
Religion

All of these are still based on INT, but test narrow types of intelligence rather than general intelligence.

Specific, domain-grounded benchmarks are what the field actually needs. The evaluations that actually help practitioners share a structure: they test a specific skill in a specific context, and they’re judged by the people who have to live with the results.

MedQA tests whether a model can answer the kind of clinical question a physician might face. LegalBench tests whether a model can correctly identify enforceability clauses, classify contract language, and apply legal reasoning. FinanceBench tests whether a model can answer the kind of question a financial analyst would actually ask about a real 10-K filing. None of these are trying to rank models on general intelligence. They’re asking the only question that matters downstream: can this model do this specific job well enough?

This week, Grafana released o11y-bench, an open benchmark for observability-specific agent tasks. I’m biased — I work there — but it’s a useful illustration of the shape I’m describing. o11y-bench doesn’t ask whether a model is smart in general. It asks whether an agent can do the things observability engineers actually do: read production traces, correlate logs with metrics, figure out why the thing that’s supposed to be working isn’t. Its leaderboard shows models’ performance for each activity.

Does that mean that more specific benchmarks won’t get saturated? No. We’ve already seen this play out with SWE-bench. But they’ll get saturated more slowly and remain more meaningful measures along the way.

These are skill checks with DCs that matter to someone. They’re not trying to produce a single scalar ranking of “which model is smartest.” They’re asking whether the wizard recognizes the family crest belonging to a political faction or knows this specific cult’s practices, not whether the wizard is good at “wizarding” in the abstract.

Roll for initiative: What this means in practice

If you’re an engineer picking a model for production: stop picking by leaderboard. Find or build an eval that reflects your actual use case. A handful of representative examples, graded the way you’d grade them if you were doing the work yourself, will tell you more than a dozen public benchmarks.

If you’re a researcher thinking about what to build next: the interesting work is at the specific-and-grounded end of the spectrum, not the general-and-bigger end. Another harder general-knowledge benchmark will saturate in eighteen months. A well-designed domain benchmark, co-built with the people who actually do the work, can stay useful for years.

And if you’re anyone else watching this field from the outside: be skeptical of the headlines. When a new model scores 94% on a benchmark you’ve never heard of, ask what it measures and whether the thing it measures is the thing you actually care about. Usually the answer is “not really,” and that’s the most important thing a benchmark score won’t tell you.

The goal of a character sheet was never to produce one number that ranks the character. It was to give a DM enough information to run a meaningful encounter — and to give the player enough to know when to try, when to pass, and when to roll with advantage. LLM evaluation should work the same way. Not a ranking; a readiness check. Can this specific model clear this specific challenge? Roll for initiative.

All About AI Observability (The Context Window #03)

Wed, 22 Apr 2026 18:00:00 +0100

In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and the engineers behind the product, Alexander Sniffin and Jack Gordley, for a deep dive into AI Observability in Grafana Cloud. We unpack what it actually is (a new way to instrument your AI apps and see canonical data about them, sitting alongside your existing telemetry), what an evaluation is in this context (and the difference between online evals on live traffic and offline evals on dataset conversations), and we walk through real demos: setting up AI Observability and evaluators, the analytics view, instrumenting a local coding agent, and Jack’s system prompt analysis tool. We also talk about the things nobody else is talking about — like LLM-as-judge as a method, evaluators that catch the most bugs, and the cheating-LLM phenomenon.

Timestamps

00:00:00 — Introductions
00:02:00 — The last month in AI news
00:11:09 — What is AI Observability in Grafana Cloud?
00:19:20 — What is an evaluation?
00:21:05 — The origin of AI Observability
00:25:09 — Demo: Setting up AI Observability and evaluators
00:32:04 — What is LLM as judge?
00:38:43 — Demo: AI Observability Analytics
00:40:52 — AI O11y is based on OpenTelemetry
00:42:17 — Demo: Instrumenting a local coding agent
00:47:18 — Potential future agentic use cases
00:52:00 — Evaluators that catch the most bugs
01:02:23 — Demo: System prompt analysis
01:05:11 — Guess the prompt

Resources

News from the episode

AI Observability

Mentioned

OpenCode — the open-source coding agent Jack instruments
Anthropic: Emotion concepts and their function in a large language model
OpenClaw

Unlearnings from building Grafana Assistant

Thu, 09 Apr 2026 09:00:00 +0100

Originally posted here

Everyone’s talking about what you need to learn to build AI agents: Prompt engineering. Tool calling. Retrieval strategies. The content ecosystem is full of it. But while I was watching Context Window Episode 1, it struck me that Mat Ryer, Cyril Tovena, and Sven Großmann– some of the team behind Grafana Assistant– spent most of their time on something different: what they had to unlearn.

Grafana Assistant started as a weekend hackathon project. By the time it shipped, the team had thrown out a popular framework, flipped their assumptions about tool design, and discovered that some of the instincts you bring from conventional software development actively work against you when you’re building agents.

I pulled out the ten moments where received wisdom turned out to be wrong. If you’re building anything agentic right now, these are worth sitting with.

❌ LangChain will help us build faster

LangChain is an opensource framework for building applications that use LLMs. It provides structured components that can be used for rapid prototyping.

In the video, Tovena talks about how they started with LangChain during the initial hackathon because they thought it would help them build more quickly, but instead it became obvious fairly early on that the opposite was true. They didn’t end up using a lot of the features that LangChain provided and felt they needed more flexibility and speed.

Abstraction overhead

One possible reason for their troubles with LangChain is that LangChain has standard abstractions for prompts, tool use, and orchestration– things that sound good in theory. But sometimes abstractions lend themselves to creating a sort of black box in your stack, where you don’t know what’s happening inside it. It sounds like the Assistant team missed the oversight and control of what was happening.

❌ Use the same prompt for every model

They also talked about how for the sake of simplicity and consistency, they initially used the same prompt for every model. However, they found out that this actually led to worse results. They found more success in creating slightly different prompts for different models, playing to their strengths and weaknesses a bit more.

There is even an argument (Mizrahi et al.’s “State of What Art? A Call for Multi-Prompt LLM Evaluation) that AI benchmarks that involve sending multiple LLMs the same prompt is a brittle and unreliable way to assess LLM capabilities. Different models respond differently to prompt variations.

In practice, what this means is that switching models isn’t a drop-in replacement. Even if a new model scores better on benchmarks, your prompts were written with a specific model’s tendencies in mind. The unlearning here is letting go of the idea that a prompt is model-agnostic — it’s more like a conversation style, and different models have different conversational defaults.

❌ Treat errors as failures

Traditionally in software, you catch errors and hide them from the user: you log them, handle them, but treat that transaction as a failed one.

The Assistant team had to unlearn this habit when dealing with an LLM, because with agents, error messages are useful data. Feeding it back to the LLM often results in it learning from its own mistake and trying again. It’s basically letting the agent debug itself, in a manner that Ryer noted was remarkably similar to how we humans debug errors too.

❌ Return errors in structured JSON

On the subject of error messages, Großmann mentioned that at one point they were trying to format the errors in JSON or stringified JSON. This practice shows solid engineer intuition because usually, structuring information helps machines process it faster and more accurately. What they found here was the reverse: it was easier and yielded better results when the LLM was just passed the entire error in natural language.

❌ Build as many tools as possible

The team mentioned that at the start, they were inclined to create as many tools/skills for Assistant: one for everything that could be done in Grafana. This made sense until they saw LLMs struggle to determine which tool to use in which context.

To help LLMs identify the correct tool, they added tool descriptions and also experimented with combining multiple tools into fewer, parameterized, tools. These helped, although the team still observed that allowing users to create their own tools (with potentially very general tool descriptions) could potentially confuse LLMs.

❌ Bigger context window = better

Gemini 1.5 Pro announced it had a 2M context window back in 2024, and it might seem that other providers had lagged behind: OpenAI only announced support for 1M tokens in 2025 and Anthropic followed in 2026. However, the Assistant team pointed out that context wasn’t everything. They said that they noticed that bigger wasn’t necessarily better, also pointing out that the chat summarization feature on IDEs like Cursor could still sometimes lose important details when summarizing.

This finding seems to coincide with a 2023 Stanford study (Lost in the Middle: How Language Models Use Long Contexts) that found there was a performance degradation when models access information within a very large context. In the video, they talked about how their system prompts were often 40k tokens long (half for the prompt itself and then the other half for tools).

❌ Start with a multi-agent architecture

Tovena mentioned that creating a multi-agent architecture for Assistant yielded more mixed results than he expected. He said that passing context between agents is difficult, outputs were inconsistent, and there was a bit of a “telephone game” problem where agents didn’t always perfectly understand the intent or output of other agents.

In early 2026, Khatua et al. released a study called “CooperBench: Why Coding Agents Cannot Be Your Teammates Yet” that showed that AI agents aren’t great at collaboration. The study suggests, surprisingly, that this might be because AI agents lack human social habits that help us collaborate better, such as feeling obliged to do what we say and having a good understanding of what others’ plans are.

The Assistant team’s experience is also a perfect example of Gall’s Law, which states that it’s easier to start with simplicity and build complexity than the other way around.

❌ Plan around models as they are now

Software has always moved quickly, but it feels like AI models are moving faster. Stanford’s AI Index Report 2025 reported that models went from having a 50% chance to near-human performance in roughly 2 years (2021-2023). In fact, benchmarks keep getting retired as AI saturates them– meaning that benchmarks can’t reliably be used to distinguish between good and bad models because all the models are testing near the ceiling.

The rapid pace of development around AI requires us to not be so attached to our work: Großmann recalled an incident where someone wrote a joke prompt that spoke like a dog that shockingly did better than their other system prompts. Ryer also mentioned that at one point someone deleted all the system prompts and Assistant weirdly became more accurate.

The lesson is clear: plan for models to improve, and plan to redo a lot of your work when they do.

❌ We don’t need evals

The Assistant team said that they went from having no evals to employing several types of evals:

integration-style evals that exercise whole workflows
unit-style evals that feed inputs plus a system prompt and check the output
LLM-as-judge, where they pass conversations to an LLM to ask it to judge whether it answered the user’s questions.
user feedback in the form of thumbs during conversations with Assistant

Beyond evaluation, they talked about how important it was to be able to observe AI. At the very minimum, showing what Assistant is doing at all times (thinking out loud) is very important to them because it builds users’ trust in the thinking process and, eventually, in the result.

Evals aren’t a sign that you don’t trust your product. They’re how you build the confidence to keep shipping it.

❌ Assume people already know how to work with AI

When you ship an AI feature, it’s tempting to assume the hard work is done. You’ve built the thing. Users will figure it out.

What the Grafana Assistant team found was that this assumption doesn’t hold. Users came in with wildly different mental models of what an AI assistant could or should do — some treating it like a search box, others expecting it to read their minds, most not yet sure how to prompt it effectively.

Their insight was that this isn’t just a UX problem to solve on your end. It’s a shared responsibility: AI should try to help the user, but the user also needs to learn how to work with AI. And building an assistant that teaches people how to use it well (through good responses, gentle corrections, showing its work) is part of building the product.

There’s also a deeper point here specific to observability. Grafana Assistant isn’t just an AI feature; it’s a way of interacting with your own infrastructure. Every good interaction teaches you something about what questions to ask, what signals to look for, what it means to understand a system. In other words: learning to use AI well makes you a better engineer. The tool and the skill co-evolve.

This is the unlearning that sits furthest from the code. It’s not about prompts or architectures or frameworks. It’s about recognising that you’re not just shipping a feature– you’re introducing a new way of working. That takes time. And it’s worth designing for explicitly.

Summary

What strikes me most about this list isn’t any single item — it’s the pattern. Most of these unlearnings point in the same direction: less is more, and messiness is the point.

Less abstraction. Fewer tools. Smaller context. Simpler architecture. And rather than hiding the mess — the errors, the reasoning, the uncertainty — let it show. That’s where the trust gets built.

We’re all still early with this. The team that built one of the most thoughtful AI products I’ve seen up close still didn’t have evals when they launched. That’s not a criticism — it’s a reminder. You don’t need to have it all figured out before you ship. You just need to commit to keep unlearning.

Do Androids Dream of Second Brains?

Sat, 21 Mar 2026 11:15:00 +0100

I gave this talk at PKM Summit 2026 in Utrecht, the Netherlands.

Abstract

We often describe our PKM systems as “second brains”. But can we really say our systems think for themselves?

In Philip K. Dick’s “Do Androids Dream of Electric Sheep?”, the inspiration for the movie Bladerunner, the boundary between humans and machines is tested through observation and subtle signals. In this talk, we’ll explore a similar question for our knowledge systems: what happens when we start observing them?

By instrumenting an Obsidian vault with an observability stack and layering AI tools on top, we can begin to see patterns in how ideas form, connect, and evolve. The result isn’t quite a thinking system… but it may be something closer than we’ve had before.

Slides

nicole.to/androids

Asimov's Zeroth Law of Robotics: Observability for AI (Newcrafts 2025)

Thu, 06 Nov 2025 12:23:19 +0100

Today I presented at Newcrafts in Paris, France. I gave a talk I’ve given before, called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU in London and at Dutch Cloud Native Day 2025 in Utrecht. I was happy to present it again, as it gave me the chance to further refine my demo application, which you can find on GitHub.

The main change I made was to use the “AI as judge” approach that I had previously only talked about in theory. In this version of the demo app, I use AI in both the game as well as in the testing of the game, showing how AI can be integrated into both the application and its observability/testing stack. It was really interesting to note that testing with AI ended up exposing more defects than my hardcoded test cases did, which goes to show the potential of AI-driven testing.

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

Wed, 05 Nov 2025 03:07:38 +0100

Today I gave a talk at Øredev 2025 in Malmö, Sweden, titled +The Spice Must Flow: The Fremen Guide to Sustainable Observability"_. The presentation focused on how to implement sustainable observability practices using open-source tools like Grafana, Prometheus, Loki, Tempo, KEDA, and Kepler. Here’s the abstract:

Nobody wants to be wasteful. But how do you balance that with the need for enough data to fix things when they go wrong? After all, no matter how much it costs, there are SLOs to keep. The spice– the telemetry– must flow.

The Fremen are a people who live on the planet Arrakis, simultaneously rich in spice while poor in water. But the Fremen have managed to thrive, not just survive, on very little while continuing to produce spice. What would it take to apply this “desert planet thinking” to observability?

In this talk, you’ll learn about the hidden environmental and financial costs of your observability stack and how to reduce them. You’ll learn about the complications of measuring cost for ephemeral resources on Kubernetes, how to use KEDA to “right-size” resources when you don’t know what the right size is, and how to convince your teams to drop all but the most meaningful telemetry data. In the end, the spice must flow, but there must still be enough water to sustain all.

Here are some resources mentioned in the talk:

You can play the game Spice Runner here
The spice-runner repo containing everything you need to know about how to deploy it yourself
The slides
The Spice Runner dashboard for observing the application
More about Spice Runner: architecture, tech stack, and what it demonstrates

One Year Later: How I Use Visuals in My PKM (Sketch Your Mind Conference)

Mon, 13 Oct 2025 13:33:18 -0700

Today I gave a talk at my friend Zsolt Viczián’s Sketch Your Mind Conference about how I use visuals in my PKM, a follow-up to my video about Obsidian Excalidraw last year where I shared that I’d changed all my Obsidian templates to hybrid Obsidian Excalidraw notes:

In the talk, I shared how I’ve evolved my use of visuals in my PKM over the last year, including:

Referring to icon and logo libraries that dynamically compile all the atomic visuals I use in my notes
Adopting a visual Zettelkasten-like approach
Creating notes at conferences to track people I’ve met and talks I’ve attended
Using Obsidian Excalidraw to create compelling presentations in lieu of slides
Combining visuals with audio recordings to create literature notes of books I’m reading
Using a full observability pipeline to monitor and analyze the notes in my Obsidian vault

I was particularly excited about the last point. I work at Grafana Labs and help people monitor their computer systems using opensource software. I realized that I could use the same tools to monitor my PKM system, so I set up a pipeline where a Python script scrapes note metadata, and sends information to Loki and Prometheus via Alloy, and then I visualize it all in Grafana. It’s a pretty nerdy and totally unnecessary setup, but here’s the repo for anyone interested in replicating it.

All About Structured Metadata in Loki (Loki Community Call August 2025)

Tue, 12 Aug 2025 19:56:32 -0700

In this video, Jay Clifford and I talk to Senior Software Engineer Salva Corts Sánchez from the Loki team about structured metadata in Loki. We discuss what it is, why you might want to use it, and also how to query it. Structured metadata is especially useful if you’re instrumenting your system with OpenTelemetry, so we also touch on that.

How to Turn Logs Into Metrics with Grafana Loki (Loki Community Call July 2025)

Tue, 08 Jul 2025 17:23:23 +0200

In this Loki Community Call, we talk all about how to turn metrics into logs in Grafana Loki using metric queries. We cover the basics of metric queries, how to use them to create metrics from logs, and some advanced use cases like using metric queries to create alerts and dashboards.

Asimov's Zeroth Law of Robotics (Dutch Cloud Native Day 2025)

Thu, 03 Jul 2025 15:01:24 +0200

Today I presented at the Dutch Cloud Native Day 2025 in Utrecht, the Netherlands. I gave a talk titled Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU earlier on in the year, in London. I was very happy to be able to present it again, because it gave me the opportunity to improve the demo app I used, which you can find on GitHub.

The Road to Loki 4.0 (Loki Community Call June 2025)

Tue, 24 Jun 2025 13:56:45 +0200

In this video, my colleague Jay Clifford and I interview Ed Welch, Principal Engineer on the Loki team at Grafana Labs, about the upcoming Loki 4.0 release. We talk about some pretty big features that make this next release almost an entire rewrite of Loki, including:

A shift towards columnar storage, using a new storage format called “DataObject”
A new Kafka/WarpStream architecture that decouples the read and write paths
A new Loki Query Engine that decouples the query language from the rest of the system, allowing for more options in, for example, allowing for SQL queries against Loki in the future

How to Query Grafana Loki

Fri, 13 Jun 2025 15:21:49 +0200

In this video, I show you how to query Grafana Loki, the open source log aggregation system from Grafana Labs. I cover the basics of Loki’s query language, LogQL, and demonstrate how to use it to filter and search logs effectively.

State of the Observability Databases (Grafana Office Hours #30)

Thu, 15 May 2025 18:22:26 +0100

In this Grafana Office Hours, we talk about the state of observability databases (Grafana Loki, Mimir, Tempo, and Pyroscope) and where they’re going. We talk about current and upcoming architectural changes in all four, how we’re making them more performant, how compatible they are with OpenTelemetry, and what we’re working on next for each database. In this conversation are Dee Kitchen (VP of Engineering for Databases) and Senior Developer Advocates Jay Clifford and Nicole van der Hoeven.

Meta-monitoring Loki (Loki Community Call May 2025)

Mon, 12 May 2025 17:59:48 +0100

In this Loki Community Call, we talk about the need for meta-monitoring Loki: why Loki needs to be monitored, what to watch out for, and how to do it. We talk about different ways to get information from Loki that allow you to make it reliable, consistent, and performant, including a Helm chart to deploy a meta-monitoring stack on Kubernetes. We discuss the Loki mixin for Grafana and how to use it to visualize data about Loki. On the call are Jay Clifford, Nicole van der Hoeven, and Dylan Guedes from Grafana Labs.

All about OTel and Logging on Kubernetes with Loki (Loki Community Call April 2025)

Fri, 25 Apr 2025 17:57:58 +0100

In this pre-recorded Loki Community Call, we talk all about OTel and logging on Kubernetes with Cyril Tovena, Ward Bekker, Jay Clifford, and Nicole van der Hoeven at KubeCon EU 2025 in London. We discuss when why you should switch to OTel and why you shouldn’t, what OTLP is exactly, and best practices for ingesting data through an OTLP endpoint.

Getting Started with Logging and Grafana Loki (webinar)

Wed, 16 Apr 2025 08:18:04 +0200

In this webinar, I talk about the key benefits of Grafana Loki, the open source log aggregation system from Grafana Labs. I cover the basics of logging, how Loki works, and how to get started with it. This was a live webinar, but I recorded the session here as well:

Asimov's Zeroth Law of Robotics: Observability for AI (KubeCon EU 2025)

Wed, 02 Apr 2025 13:46:42 +0200

Today I gave a talk at KubeCon EU 2025 in London called “Asmiov’s Zeroth Law of Robotics: Observability for AI”. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

Best Practices for Recording and Alerting (Loki Community Call February 2025)

Wed, 26 Feb 2025 17:54:15 +0100

In this Loki Community Call, we talk about the best practices for recording and alerting rules with Loki Staff Engineer George Robinson and Senior Developer Advocates Jay Clifford and Nicole van der Hoeven. Ever wondered what’s the difference between the Ruler and Alertmanager? 🤔 Curious about the different modes the Ruler can run in when querying Loki? Want to avoid the most common pitfalls when setting up alerting and recording rules? We cover all of these, including optimizing rule performance to structure alerts effectively.

How to Run Loki at Scale on Kubernetes (Loki Community Call January 2025)

Thu, 23 Jan 2025 17:51:00 +0100

Happy New Year from the Loki Engineering team. To kick off 2025, Nicole and Jay will be joined by Poyzan Taneli from the Loki Engineering team to discuss how to run Loki at scale on Kubernetes. If you are currently running Loki in microservices mode or preparing to do so, we will be discussing best practices for scaling its components to meet the demands of production use cases.

How many Ingesters vs Queriers do I need?
What is the minimum cost of running Loki as a production cluster?
What is the average node size for running Loki? How do I monitor my Loki deployment?

AI Observability with Ishan Jain (Grafana Office Hours #29)

Fri, 08 Nov 2024 18:24:29 +0100

In this Grafana Office Hours, Ishan Jain talks about AI Observability with Grafana: what it entails, factors to consider when monitoring and observing LLMs, and how to do it all with Grafana. He is joined by Senior Developer Advocate Nicole van der Hoeven.

All About Logs Drilldown (Loki Community Call October 2024)

Thu, 03 Oct 2024 17:45:33 +0100

In this Community Call, Senior Software Engineer Trevor Whitney talks to us all about Drilldown Logs for Grafana Loki, an open-source app for visualizing logs from Loki in Grafana without needing to learn and write LogQL queries. He is joined by Senior Developer Advocates Nicole van der Hoeven and Jay Clifford.

All the Components of Loki Explained

Fri, 20 Sep 2024 18:09:35 +0100

Here are all the components of Loki explained so that you can figure out how many instances of each you need for a resilient Loki deployment. Senior Developer Advocates Jay Clifford and Nicole van der Hoeven talk you through what each component does and which ones are optional.

Bloom Filter Changes for Grafana Loki (Loki Community Call September 2024)

Thu, 05 Sep 2024 17:41:18 +0100

In this Community Call, Senior Software Engineer Christian Haudum talks to us about bloom filter changes for Grafana Loki, including the deprecation of the bloom compactor and a pivot towards creating bloom filters for structured metadata. Bloom filters are a probabilistic data structure that we’re using to improve query performance in Loki.

Community Calls are monthly meetings that are open to everyone interested in the development of Loki. They are an opportunity for software engineers working on Loki to discuss new features as well as for open-source users of Loki to ask questions. To participate in the next Community Call, subscribe to the calendar here.

Cartography of the Deep: Distributed Tracing for Testing (Romanian Testing Conference)

Thu, 13 Jun 2024 12:24:02 -0700

I presented a talk today at the Romanian Testing Conference 2024 in Cluj-Napoca, Romania! My talk was called “Cartography of the Deep: Distributed Tracing for Testing” and it covered how to tie in distributed tracing into tests to get a better understanding of your system under test.

Here are my slides!

Abstract

From the Scandinavian Kraken that drags entire ships to their early graves to the Philippine Bakunawa said to rise from the waters to swallow the moon, our existential fears about what lurks at the bottom of our own oceans has inspired us to create hundreds of folkloric monsters across cultures. We’ve charted the surface of our planet, but the vast majority of the ocean floor remains a mystery.

The mysterious depths of modern software architectures makes it difficult for us to test a system for performance and reliability. The easy solution? Unmanned underwater probes: send inputs to the system, and judge its state based on the outputs that return. But this method is often too limited to tell us what’s really going on when something goes wrong. What if there were a way to make our tests themselves more observable?

In this talk, you will learn how to use open-source tools to instrument performance tests, identify bottlenecks at the code level using distributed tracing, and map out a system using a type of software bathymetry. You’ll also learn about the Mariana Trench expeditions, the difficulties of operating in a high-pressure underwater environment, and how to come face to face with monsters of our mythologies.

How to Get Started With Loki

Fri, 10 May 2024 18:12:09 +0100

If you were looking for how to get started with Loki, here’s a video where Senior Developer Advocates Jay Clifford and Nicole van der Hoeven talk about what Loki is, how to ingest logs with Loki, how to deploy Loki, and finally how to visualize your logs with Grafana.

Doing It in Public: Observability for PKM

Fri, 05 Apr 2024 17:26:42 +0100

The quest of learning in public with Personal Knowledge Management is really about making what you learn observable and exposing your process, rather than your result, early and often. This is a rendition of a talk I gave at the PKM Summit in Utrecht, the Netherlands, on March 23rd, 2024, and it relates concepts borrowed from performance engineering and software development to PKM systems.

How to do continuous profiling right with Grafana Pyroscope and Ryan Perry, (Grafana Office Hours #26)

Fri, 19 Jan 2024 18:09:01 +0100

My friend Paul Balogh and I interview Ryan Perry, co-founder of Grafana Pyroscope, about how to do continuous profiling right. Ryan, and the rest of the Pyroscope team, joined Grafana Labs, where he is now a Director of Engineer. Ryan discusses the main concerns in continuous profiling and how to avoid those pitfalls. Pyroscope is an open-source project for aggregating continuous profiling data about your system’s resources.

Entendiendo los sintéticos (Café con Grafana #008)

Thu, 11 Jan 2024 16:00:25 +0100

En este episodio de Café con Grafana les traemos a Marcelo Magallón, experto en monitoreo sintético, quien viene precisamente a contarnos que es en sí un sintético. Él nos va a mostrar algunos ejemplos, casos de uso y mucha mas diversión!

Lee la documentación sobre el monitoreo de sintéticos en Grafana Cloud aquí.

How to Deploy Grafana Beyla on Kubernetes as a Sidecar Container

Tue, 09 Jan 2024 13:41:38 +0100

Principal Staff Engineer Nikola Grčevski demonstrates how to deploy Grafana Beyla on Kubernetes as a sidecar container. Grafana Beyla is an eBPF auto-instrumentation tool for application observability: you just have to tell it what to listen to, give it permissions, and deploy it– and you can see metrics and logs from that component or service without having to do any manual instrumentation.

Introduction to eBPF With Grafana Beyla (Grafana Office Hours #25)

Fri, 05 Jan 2024 18:03:52 +0100

Nikola Grčevski, Principal Software Engineer at Grafana Labs, gives us an introduction to eBPF with Grafana Beyla. We discuss what is eBPF, how you can use it to auto-instrument applications, and how to get started with Beyla. eBPF observability is all the rage because we all want automagical instrumentation– but does it live up to that promise? Nikola’s here to tell us what eBPF can and can’t do, and where he’d like to take Beyla next.

AI Microscopy with Grafana, Theia Scientific, and Volkov Labs (Grafana Office Hours #24)

Fri, 15 Dec 2023 17:42:58 +0100

What do you get when you combine AI microscopy with Grafana? Well, in this case, you get the Theiascope platform: an application for doing real-time analysis on microscopy images using a combination of Grafana, Prometheus, and PostgreSQL/Timescale. The Theiascope platform was a collaboration between Dr. Christopher Field, Co-Founder/President/ Principal Investigator at Theia Scientific and Mikhail Volkov, Founder/CEO of Volkov Labs.

Getting Started with OpenTelemetry (Grafana Office Hours #23)

Fri, 08 Dec 2023 17:00:23 +0100

Principal Engineer Juraci Paixão Kröhling talks to my colleague Paul Balogh and me about getting started with OpenTelemetry: what it is, why everyone’s talking about it, and how to instrument your systems and improve their reliability and observability. Juraci is a contributor to open source projects like Jaeger and OpenTelemetry.

Repasemos el G-stack (Café con Grafana #005)

Thu, 30 Nov 2023 13:57:52 +0100

En este episodio de Café con Grafana tenemos a Raul Marin, Solutions Engineer (Ingeniero de soluciones?) quien viene a darnos una repasada completisima de todo el stack de Grafana. TODO! Pues hay muchos que no conocen todas las herramientas, caracteristicas y posibilidades que Grafana Labs viene a ofrecer.

Dimensionando el Grafana Suite (Café con Grafana #004)

Thu, 23 Nov 2023 13:51:25 +0100

Almudena Vivanco viene a platicarnos de todos los experimentos y pruebas a las que someten a toda la suite de Grafana. Desde Mimir hasta k6 ella pone al limite todos los elementos constantemente. Además nos dará muchos tips de su experiencia para usar Grafana y mucho mas!

(ES) Cómo hacer pruebas de carga con Grafana k6 (Café con Grafana #003)

Wed, 08 Nov 2023 13:43:50 +0100

Daniel González Lopes nos viene a demostrar cómo hacer pruebas de carga con Grafana k6 desde zero, y cómo utilizar k6 para ejecutar otros tipos de pruebas. En este episodio para principiantes, hablaremos también sobre la relación entre pruebas y observabilidad.

Resources

Combining Frontend and Backend Performance With John Hill (Grafana Office Hours #18)

Fri, 03 Nov 2023 12:30:43 +0100

In this episode of Grafana Office Hours, my colleagues Marie Cruz, Paul Balogh, and I talk to John Hill, a Web UI Test Engineer and Grafana k6 champion, to talk about how the Grafana and k6 ecosystems can be used to ensure performance in mission-critical applications like NASA’s Open MCT.

(ES) Observando Bitácoras With Matías Chomicki (Café con Grafana #002)

Thu, 02 Nov 2023 12:25:53 +0100

Hoy hababa con mis colegas Leandro Melendez and Matías Chomicki sobre observabilidad de logs en Grafana: qué son, con qué se comen, qué tienen que ver con Loki, otras plataformas donde se pueden usar, y muchos detalles mas!

(TL) Ano ang relasyon ng testing at observabilidad?

Wed, 01 Nov 2023 12:19:46 +0100

Ano ang relasyon ng testing sa observabilidad? Sa video na ito, pinag-uusapan namin ng kaibigan kong si Marie Cruz kung ano ang testing, ano ang observabilidad, at bakit dapat nating gamitin pero para pabutihin ang ating mga aplikasyon.

Real User Monitoring With Grafana Faro (Grafana Office Hours #17)

Fri, 27 Oct 2023 07:00:19 +0200

My friend Marie Cruz and I chat with Software Engineers Kostas Pelelis and Marco Schaefer to discuss what Grafana Faro is, why it’s important to have a real user monitoring solution, and how to get started with it.

Useful links

(ES) Grafana Como Código (Café con Grafana #01)

Thu, 26 Oct 2023 06:55:37 +0200

En este episodio platicamos con Joan López de la Franca, Ingeniero de software senior en Grafana. El nos viene a platicar de las funcionalidades de Grafana como código, que es, como funciona, cuando conviene utilizarlo e incluso nos traerá algunos demos al respecto.

Emergent Load Testing - TestCon Europe 2023

Wed, 25 Oct 2023 20:05:36 +0300

I presented at TestCon Europe 2023 in Vilnius, Lithuania about emergent load testing, and how to use it to find the limits of your system. My talk was called Emergent load testing: Rules for Organized Chaos.

You can find the slides here!

What is Grafana Agent? (Grafana Office Hours #16)

Fri, 20 Oct 2023 12:55:11 +0200

What is Grafana Agent? Senior Software Engineer Paschalis Tsilias tells us about what telemetry collectors do, why they’re useful, what you’d have to do without them, and how to get started with Grafana Agent. He is joined by Developer Advocates Nicole van der Hoeven and Paul Balogh.

Useful links

Agentless Monitoring for Prometheus in Grafana Cloud (Grafana Office Hours #15)

Fri, 13 Oct 2023 10:56:15 +0200

Did you know you could do agentless monitoring for Prometheus in Grafana Cloud? Senior Software Engineer Matt Nolf talks to me and my coworker Paul Balogh about the new Metrics Endpoint integration and how you can use it to collect Prometheus metrics from publicly addressable hosts– WITHOUT using Grafana Agent.

Resources

Distributed tracing with Grafana Cloud k6 (Grafana Office Hours #14)

Sat, 07 Oct 2023 08:31:03 +0200

My friend Marie Cruz and I interview Senior Software Engineer Łukasz Gut about a new feature: Distributed Tracing with Grafana Cloud k6. He discusses what distributed tracing is, why it matters, and how it can help teams find reliability issues faster.

How to Deploy Grafana on Kubernetes (Grafana Office Hours #13)

Fri, 29 Sep 2023 17:30:49 +0200

My coworker Usman Ahmad and I talk about how to deploy Grafana on Kubernetes for beginners: what Kubernetes is, how it’s an evolution of distributed computing, what its relationship to Docker is, other things you might need to know to work with Kubernetes, and how you can deploy Grafana on your own Kubernetes clusters.

Documentation on how to deploy Grafana on Kubernetes here.

OpenTelemetry demo app with Grafana, Tempo, Loki, and Prometheus (Grafana Office Hours #06)

Fri, 28 Jul 2023 20:25:37 +0200

DevOps Engineer Blueswen Liu 劉義瑋 joins us to walk us through some OpenTelemetry demo apps he created, instrumented with Grafana, Loki, Prometheus, and Tempo.

Load testing and tracing with k6 and Tempo

Thu, 18 May 2023 22:41:42 +1000

At ObservabilityCON on the Road 2023 in Sydney, Australia, I talked about why you should use load testing and distributed tracing, how to do it with k6 and Tempo, and the benefits of being able to see beyond traditional black-box testing.

Resources

How to Avoid the Most Common Kubernetes Monitoring Mistakes

Tue, 16 May 2023 22:41:16 +1000

For ObservabilityCON on the Road Singapore 2023, I talked about some mistakes people make when monitoring applications within Kubernetes clusters. I also demonstrated Grafana’s Kubernetes Monitoring solution and how it can be a shortcut for engineers looking to get started quickly.

Resources

Slides here
Monitoring Kubernetes in general (notes)
The Grafana Kubernetes Monitoring feature (notes)

Grafana Cloud k6 (k6 Office Hours #83)

Fri, 24 Mar 2023 18:00:00 +0100

With the release of Grafana Cloud k6, you can get observability and performance testing in the same platform. Senior Product Manager Mark Meier and Frontend Lead Simon Legander join me to talk about what Grafana Cloud k6 is, how you can use it, and demonstrate what it can do to improve your system’s reliability.

Introducing Grafana Cloud k6

Tue, 21 Mar 2023 15:20:58 +0000

Grafana Cloud k6 is a feature available to all users of Grafana Cloud. It allows you to run load and performance tests right from the comfort of your Grafana Cloud instance, without needing to create a separate account on k6 Cloud. Grafana Cloud k6 helps you get started quickly with a GUI Test Builder or with test scripts written in JavaScript, and then visualize your test results in real-time, alongside your other Grafana dashboards that monitor your system infrastructure. With Grafana Cloud k6, you can get a holistic look at your system’s performance– on any environment.

Learn more about Grafana Cloud k6.

Observability Trends and Future Best Practices (k6 Office Hours #59)

Fri, 12 Aug 2022 16:19:13 +0100

What are some observability trends and best practices in 2022 and going forward? Technical Program Manager Daniel González Lopes joins Developer Advocates Nicole van der Hoeven and Paul Balogh to chat about trending concepts and new areas of exploration in the field of observability. We’re going to talk about distributed tracing, continuous profiling, eBPF monitoring, and more!

Why Is Everyone Talking About Chaos Engineering? (k6 Office Hours #56)

Fri, 15 Jul 2022 18:00:13 +0100

Why is everyone talking about chaos engineering anyway? How different is it from testing? What does it have to do with performance? In this k6 Office Hours, k6 Technical Program Manager Vince Huang joins me and my colleague, Paul Balogh, to talk about these topics and more.

How to migrate to Kubernetes, with Carlos Ruiz Lantero (k6 Office Hours #55)

Fri, 08 Jul 2022 17:00:57 +0100

k6 Cloud Backend developer Carlos Ruiz Lantero joins Developer Advocates Nicole van der Hoeven and Paul Balogh to discuss how to migrate to Kubernetes. Carlos has been working on migrating k6 Cloud services from ECS to EKS, and in this video, he shares his best practices for migration and things he wishes he’d done differently.

Performance Objective: Intro to Load Testing With k6 and Grafana

Tue, 23 Nov 2021 17:05:06 +0100

I spoke at the Performance Objective meetup to talk about how to use k6 and Grafana for beginners. I always enjoy talking to like-minded engineers, and I think sharing ideas in a meetup context is probably the best way to do it.

Basics of load testing with k6 and Grafana in 20 minutes

Tue, 12 Oct 2021 01:04:02 +0200

This was a video I pre-recorded in preparation for the first Grafana EMEA virtual meetup.

In mid-June (just three months ago), k6 joined the Grafana Labs family. Many people didn’t quite understand what the connection was between k6 and Grafana, but I thought it made a lot of sense.

Google’s Site Reliability Engineering book talks about 4 golden signals that we should measure in a system: latency, traffic, errors, and saturation. Load testing provides 3 of those (latency, traffic, and errors), but it doesn’t show saturation, and it doesn’t give us visibility into the system.

Grafana brings observability. k6 brings data. They work really well together, which is why we’ve been using each other’s products for a few years now, before there was ever any talk of an acquisition.

In this video, I talk about why k6 and Grafana are better together and also demonstrate:

how to run a simple script in k6 OSS
how to make sense of the results of a load test
test execution options: running locally, sending results to k6 Cloud, and running a test from k6 Cloud– all triggered using the CLI
two different integration options to bring k6 and Grafana together: the k6 Cloud Grafana data source plugin and Prometheus Remote Write

Testing With k6 + Grafana: Better Together (Grafana virtual meetup EMEA)

Tue, 05 Oct 2021 21:49:13 +0200

In this first EMEA session of the Grafana Labs virtual meetups, I talked about how to use k6 and Grafana together to get the full performance picture for a software application. Observability, using tools like Prometheus and Grafana, lets us see what’s going on inside the application, and load testing, using k6, helps us get those insights earlier on in software cycles, such as during pre-production.

Intro to Load Testing With k6 and Grafana (webinar)

Thu, 16 Sep 2021 22:11:16 +0200

Tonight I did a webinar with Grafana Labs– the first k6-related webinar since the acquisition. I was more than a little nervous to give it, because I knew I was introducing k6 to a lot of new users, but also to Grafana for the first time. I think it went pretty well! I was still quite relieved when it was over, however. :)

In the webinar, I talk about:

why it’s still a good idea to do load testing in pre-production
what k6 is
why k6 and Grafana together make sense
and how to integrate k6 and Grafana

There’s quite a bit of demo in it, and I also had the k6 CEO, Robin Gustafsson, join me for Q&A as a nice treat.

Check out the full webinar here– it’s already up on the Grafana site. You can also go through my slides here, if you’d like.

Schrödinger's Pokémon: k6 and Grafana edition

Wed, 15 Sep 2021 18:22:31 +0200

Earlier this year, I gave a presentation at New Relic’s FutureStack conference about chaos engineering with k6 and New Relic. A few months ago, k6 got acquired by Grafana Labs, and so I wanted to redo it, but with the Grafana stack this time.

Here’s the result, and I think I like this iteration even better:

I presented this talk at TestCon Europe 2021 today, although the audio sync issues were not great, so I recommend you watch the video above instead.

Slides are here.

How to do continuous performance testing, with Lee Barnes (k6 Office Hours #26)

Fri, 27 Aug 2021 09:34:13 +0200

Lee Barnes and I got connected because we are both speaking at TestCon Europe 2021 this year, and I thought it would be nice to have a chat with him before the conference to get a sneak peek at what he’s presenting about: continuous performance testing. Lee is the CEO of Utopia Solutions, and he’s been in the testing industry for decades, so I really appreciated his insights on where the industry is going.

Catch Lee’s talk here and mine here!

Monitoring your k6 load test: how to install Grafana and Prometheus on a Kubernetes cluster

Wed, 11 Aug 2021 18:55:49 +0200

Running a load test is half of the equation, and not having the right observability tools to see how your application responds to the load can sometimes flat-out invalidate a test.

Here’s how to install open-source tools Grafana and Prometheus on a Kubernetes cluster in DigitalOcean and how a performance user could use them for increased visibility during a load test.

A tester's overview of Grafana Labs open-source projects

Mon, 19 Jul 2021 18:23:33 +0200

Since the acquisition of k6 by Grafana Labs, the entire k6 team has been absorbed into the new, larger team that we’re suddenly a part of. Grafana Labs maintains a lot of projects, so I was very quickly overwhelmed with things to learn about all of them. Here’s my attempt to make sense of the Grafana Labs landscape, and perhaps it can help some of you as well:

Acquisition talk with CEOs Raj Dutt (Grafana) and Robin Gustafsson (k6) (k6 Office Hours #19)

Fri, 02 Jul 2021 21:04:13 +0200

This week on k6 Office Hours, Simme and I were joined by two CEOs: k6 CEO Robin Gustafsson and Grafana Labs CEO Raj Dutt. It was a little nerve-wracking to have two CEOs on the stream, but it did give us the opportunity to ask both of them pointed questions on Grafana Labs’s acquisition of k6, including any plans they have for how both companies could integrate better in the future.

What you need to know about Grafana Labs's acquisition of k6

Wed, 30 Jun 2021 18:29:24 +0200

Recently, during Grafanaconline 2021, Grafana Labs CEO Raj Dutt announced the acquisition of k6– the company I work for. What does this mean? Here’s a video I made to explain it all:

Simme and I have also discussed the acquisition twice on k6 Office Hours:

once with our CEO, Robin Gustafsson
and another time with both Robin and Raj, Grafana Labs CEO

TL;DW: We unanimously agree it’s a positive thing. I myself am cautiously optimistic about the future of both k6 and Grafana Labs.

k6 is part of Grafana Labs! AMA with CEO Robin Gustafsson (k6 Office Hours #17)

Fri, 18 Jun 2021 22:26:49 +0200

On this week’s Office Hours, Simme and I were joined by Robin Gustafsson, the CEO of k6, to make a big announcement: k6 has been acquired by Grafana Labs.

The folks at Grafana Labs have been fans of k6 for a while, and we’ve been fans (and users) of their work since almost the beginning of k6. While a surprise to many, Grafana and k6 together is a combination that makes sense.

Read Grafana’s press release here to hear more about this milestone.

Application Performance Monitoring using Netdata and k6, with Odysseas Lamtzidis (k6 Office Hours)

Fri, 28 May 2021 23:24:48 +0200

This week, we had Odysseas Lamtzidis on the show to talk about Netdata, which is an application performance monitoring tool that prides itself on having the easiest setup ever. Odysseas admitted that it might not have all the bells and whistles of other monitoring platforms, it Just Works. I see this as being very interesting especially for testers like me, who just want to have something set up but might not want to be bothered by the implementation.

Learning how to use k6 live with Nočnica Fee from New Relic (k6 Office Hours)

Fri, 14 May 2021 21:53:09 +0100

We were happy to welcome Nočnica Fee from New Relic to k6 Office Hours. She very bravely volunteered to learn how to use k6– live! Talk about pressure. But she handled it like a champ!

We also talked about why observability is both particularly important and particularly difficult in serverless architecture. There will definitely be a part two at some point!

How to get started with chaos engineering (k6 Office Hours)

Fri, 30 Apr 2021 21:53:09 +0100

Recently, I’ve been learning more about chaos engineering. In this week’s k6 Office Hours, I shared my experience as a tester getting started with Kubernetes, chaos engineering with xk6-chaos and k6, and observability with New Relic.

k6 Office Hours 05: Observability with Daniel González Lopes

Fri, 12 Mar 2021 21:53:09 +0100

Our first repeat guest, Daniel González Lopes, also from k6, joined us today to talk about how to use observability in conjunction with load testing. We talked about the differences between metrics, logs, and traces, and how to implemented distributed tracing.

k6 Office Hours 02: Observability with New Relic

Fri, 19 Feb 2021 23:24:58 +0100

For this week’s Office Hours, Simme and I invited Gary Spencer from New Relic to talk to us about why he’s recently started using k6, and why an observability platform like New Relic is the perfect companion for performance testing and observability.

Observability on Nicole van der Hoeven

Spice Runner

What it demonstrates

The dashboard

Architecture

The talk

Run it yourself

The Defiant Jazz Incident

How it works

Why a murder mystery?

No spoilers

Try it

obsidian-grafana

How it works

What it’s useful for

Try it

Asimov

The idea

What’s in it

The talks

Try it

Build Apps on Your Data with Grafana Pages (The Context Window #07)

What Grafana Pages is

Seeing it on Grafana Play

Building a Page from a prompt

Personal data Pages

Pages vs dashboards

Remixing, editing, and building from a dashboard

Sharing and access control

How Pages work: React, sandboxes, and iframes

Limitations and roadmap (no promises)

Resources

Related reading

More from The Context Window

Transcript

Intro, guests, and ObservabilityCON recap

What Grafana Pages is

Grafana Pages examples on Grafana Play

Build a Grafana Page from a prompt

Personal data Pages: sleep and flight tracking

Grafana Pages vs dashboards

Remix a Page or create one from a dashboard

Share Pages and control access

How Pages work: React, sandboxes, and iframes

Limitations, roadmap, and more Page examples

Asimov's Zeroth Law of Robotics: Testing for AI (HUSTEF 2026)

Resources

Context Engineering Fundamentals (The Context Window #06)

What we cover

Resources

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Transcript

Observability and AI explained

Resources

AI Week Day 4: Evaluating (Grafana AI Week — Thursday)

Transcript

Investigations Arena: how we test Assistant investigations

Transcript

Resources

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Resources

Related reading

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Transcript

AI Week Day 2: Building (Grafana AI Week — Tuesday)

Transcript

AI Week Day 1: Planning (Grafana AI Week — Monday)

Transcript

An Overview of AI Evaluation (The Context Window #05)

The “final answer problem”

What evals are, and online vs offline

How we test Grafana Assistant

o11y-bench

The evaluation loop

Where it’s going

Resources

Related reading

More from The Context Window

Transcript

CLI and MCP Basics for AI Agents (The Context Window #04)