Grafana Assistant on Nicole van der Hoeven

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Mon, 17 Aug 2026 09:00:00 +0100

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. They’re not competitors. Think of it as a brain and two hands. In this short, I start with the brain: Grafana Assistant.

Assistant isn’t just a chatbot — it’s the reasoning layer. It manages context and knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. Instead of clicking around, you tell Assistant what you want to do and it figures out the how.

Reach for the Assistant when the work is visual or exploratory:

Building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
A real incident that needs root-cause analysis across multiple data sources? Assistant. That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in, and I’ll cover the first of those (the MCP server) next.

Transcript

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. I’m going to talk a bit about each one so you know which one is best for your use case.

Spoiler: They’re not competitors. Think of it as a brain and two hands.

Let’s start with Grafana Assistant, which is the brain.

It’s not just a chatbot — it’s the reasoning layer. It manages context, it knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. So instead of you clicking around to do something, you can say what you want to do to Assistant and it figures out the how.

Reach for the Assistant when the work is visual or exploratory.

If you’re building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
You have a real incident where you need root-cause analysis across multiple data sources? Assistant.
- That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in… and I’ll show you those next.

Resources

Grafana Assistant (product page)
My notes on Grafana Assistant
My notes on GCX
Unlearnings from building Grafana Assistant — my Context Horizon piece

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Fri, 31 Jul 2026 09:00:00 +0100

The final day of AI Week is about collaborating — using AI right where your team already works, no context switching required. Grafana Assistant in Slack is now generally available: investigate issues, ask follow-up questions, and query your metrics and logs from the channel where the incident conversation is already happening. And that’s just one place Assistant now meets you — also announced: the Grafana mobile app is GA (with Assistant on mobile in public preview), Assistant Desktop in public preview, Microsoft Teams support in private preview, and context-aware Assistant Search in public preview. Add it all up and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search. That’s a wrap on five days of announcements.

Transcript

It’s the final day of Grafana Labs’ AI Week, and Friday is all about collaborating — using AI right where your team already works, no context switching required.

Today, Grafana Assistant in Slack is generally available. Investigate issues, ask follow-up questions, query your metrics and logs — all from the channel where your incident conversation is already happening.

And that’s just one place Assistant now meets you. Today we also announced: the Grafana mobile app is GA, with Assistant on mobile in public preview. Assistant Desktop is in public preview. Microsoft Teams support is in private preview. And Assistant Search — deeper, context-aware search — is in public preview too.

Add it all up, and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search.

That’s a wrap on five days of announcements — and I know it was a lot. So if you missed a day, or you just want the full picture of where AI at Grafana stands today, I made a video for exactly that: everything you can do with AI in Grafana Cloud, including everything we shipped this week. It’s linked below and pinned in the comments.

One last time: grafana.ai for all of it. Thanks for spending AI Week with me.

Observability and AI explained

Fri, 31 Jul 2026 09:00:00 +0100

“AI observability” gets thrown around a lot, and it can mean one of two very different things. So in this video I map out the whole landscape of AI-related work at Grafana across the entire software development lifecycle — and show how AI and observability come together in two directions: AI for observability (AI that helps you observe your systems) and observability for AI (observing the AI itself).

The framing is deliberate: AI in observability isn’t about replacing engineers. It’s about handing back the exhausting parts of the job — the 3am triage, the repetitive queries, the report nobody wanted to write — so you can spend your energy on the work only humans can do.

I walk through each phase of the lifecycle and where the tools fit:

Planning — clarity about what’s really going on, with Grafana Assistant: not just a chatbot, but an agent that knows your telemetry, writes your queries, and builds your dashboards.
Building — momentum without losing control: Assistant, the open-source GCX CLI, the Grafana MCP server, the specialized Loki/Tempo/k6 MCP servers, and the AI SDK for building your own agents in Go.
Operating — staying calm under load: Investigations for agentic incident triage, Assistant Automations and the Watcher Agent for staying proactive, plus Grafana Machine Learning (Metrics Forecasting, Outlier Detection) and Adaptive Telemetry.
Testing & evaluating — doubt is a form of care: k6 and agentic testing on the systems side, and Agent Observability plus the open-source o11y-bench benchmark on the “observing your AI” side.
Collaborating — bringing it all to where your team already works: Assistant in Slack (and coming to Teams), on mobile, on the desktop, and via API/MCP, plus Assistant Search across your Grafana artifacts.

If you’re new to the Grafana stack itself, start with my open source observability video for the non-AI foundations that all of this builds on.

Resources

My notes on Grafana Assistant
My notes on GCX
My notes on k6
o11ybench.ai — the open leaderboard for AI agents doing observability work

Investigations Arena: how we test Assistant investigations

Thu, 30 Jul 2026 09:00:00 +0100

How do you test an AI agent whose whole job is to investigate incidents — when every investigation is different and “correct” isn’t a single deterministic answer? In this interview I sit down with Will, a software engineer at Grafana who’s spent the last year focused on AI: the Grafana Assistant, investigations, and memory features.

The story here is how the team got from Investigations V1 to a much better V2. V1 hit a ceiling, and the demo environment they’d been testing against turned out to be the problem — the cases were too easy and too far from the messy reality of a production cluster. So they built an arena: think competitive bot programming, but the “game” is investigating real incidents in Grafana’s own production environment. Different AI agents, each with its own strategy and tools, all take a swing at the same incident, and an LLM-as-judge scores their answers against a human-verified root cause.

What’s interesting is that the score wasn’t even the most valuable output at first — the insights were. Watching which strategies worked, which tools returned so much noise that they poisoned the context, and which agents wandered off is what let the team optimize bit by bit into the V2 release. The arena also compares more than models: different harnesses running Sonnet vs. Opus, Claude Code with the Grafana MCP, even just hitting the public API directly with curl.

A couple of my favorite moments: Will explains the guardrails that stop agents from “cheating” (they can only look at data from within the incident’s time window, so they can’t peek at the fix commit that resolved it) — and how the models mostly play fair because they treat it like a game. And there’s a lovely example of emergent creativity: an agent reached for tools the team hadn’t even considered, they turned out to be genuinely useful for investigation, and those tools got promoted into the real harness. Will’s hope for the future is to open this up — an OliBench-style shared benchmark where the community can throw their own investigation agents into the arena and raise quality across the board.

Transcript

Will: Hey, my name is Will. I’m a software engineer at Grafana. I’ve been there for three years, and for the last year I’ve been focusing on AI — the Grafana Assistant, investigations, and memory features.

Nicole: And what’s the latest thing you’ve been working on?

Will: So we developed Investigations V1, and we saw that we hit a ceiling at some point. We came up with a different approach, but this time we needed a good evaluation environment — the demo environment wasn’t enough, because those cases were too easy and too far from the reality of a production environment. So we decided to build an arena.

It’s a little bit like the idea of competitive bot programming, where bots fight against each other on specific tasks or games. In this case, the topic was investigations in our production cluster — real incidents. We started building bots that had different strategies and different tools; these were AI agents. Comparing a diversity of strategies let us surface the best approaches, and from there we could iterate, bring in more test cases from more incidents, and reach higher and higher quality.

The idea is that the bots try to solve the same case, and we use an LLM as a judge to compare the different approaches. We have different criteria, because for each incident we have a root cause that’s been verified by a human. That gives us our ground truth, and the LLM-as-judge compares that ground truth against the answers the agents came up with.

The important part at first wasn’t even the score — it was the insights. This tool was better with this strategy; this one had noise in the results that poisoned the context; the results returned from this specific tool were just too much. So we managed to really optimize bit by bit and come up with Investigations V2, which we just released. It’s a massive improvement compared to our V1 approach.

Nicole: So is it comparing just different models for investigations, or is it also comparing things like Claude Code and the Grafana MCP versus Assistant?

Will: Both, actually. We have different versions of our harness for investigations that we run with Sonnet and with Opus, but we also use the Claude Code harness with the Grafana MCP — and even just using curl to directly hit our public API.

Nicole: Do the models ever cheat?

Will: So far it’s been quite good. We have guardrails. Because we investigate past incidents, we need to make sure the LLM only looks at a specific time window — only from when the incident started — otherwise it would be able to find the pull request that was used to solve the incident. But they’re pretty good at following the rules; they take it as a game a little bit. We even have an agent that checks afterward, with the judge, to make sure they followed every rule.

It’s funny, actually — we had a case using Claude Code where it started to use some tools we hadn’t thought about in the first place, and they turned out to be very useful for investigation. We’d never considered them, so we brought them into our investigation harness and it helped a lot. Giving the agents freedom and creativity in this kind of environment is really useful. That diversity sponsors new ideas that we can put into our own work.

Nicole: Cool. Do you have any plans to publish this and share it so that other people can do it too — kind of like OliBench?

Will: Yeah, I think, why not? I think we can go this way too. I’m sure it would be useful for other agents. And it would be very nice if we could compare agents from other people and other companies — maybe with different harnesses, maybe even different tools — because right now it’s mostly about our MCP, our CLI, and our investigation harness. But if the community comes up with their own investigation agents — and I think I’ve seen a few — it would be fun to put them in the arena and help not just our product, but any product improve in quality.

Resources

My notes on Grafana Assistant

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Wed, 29 Jul 2026 09:00:00 +0100

Day three of AI Week is about operations and maintenance — because once code reaches production, the operational work is still surprisingly manual: someone’s watching dashboards, deciding whether an error spike is real, and writing up what happened, every single day. Two Grafana Assistant features went GA to take that off your plate. Assistant Automations handle recurring work — save a prompt and run it on a schedule (like a morning brief on your service’s health), each run keeping its own history and able to notify you in Slack. Assistant Investigations is for when something actually breaks: it digs across metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report you can follow along with in a Workspace. You’re offloading vigilance, not accountability — the evidence is always there to review, and you still make the call.

Transcript

It’s day three of AI Week, and Wednesday is all about operations and maintenance. Because here’s the thing: once your code reaches production, the operational work is still surprisingly manual. Someone’s watching dashboards, someone’s deciding whether that error spike is real, and someone’s writing up what happened — every single day.

Today, two features of Grafana Assistant are now generally available to take that burden off your plate: Investigations and Automations.

Assistant Automations handle the recurring work of operating a system. Save an Assistant prompt and run it on a schedule — like a morning brief that summarizes the operational health of your service over the last 24 hours. Every run keeps its own conversation and history, and it can notify you in Slack when it’s done or needs approval.

Assistant Investigations is for when something actually breaks. Investigations digs into issues across your metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report. You can follow along in a Workspace, see what’s been ruled out, and add guidance while it works.

You’re offloading vigilance, not accountability — the evidence and reasoning are always there to review, and you still make the call.

Everything’s at grafana.ai — see you tomorrow for day four: evaluating your agents.

AI Week Day 1: Planning (Grafana AI Week — Monday)

Mon, 27 Jul 2026 09:00:00 +0100

It’s day one of Grafana Labs’ AI Week, and Monday is all about planning — getting the clarity you need on what’s going on in your system so you can figure out what to build. Kicking things off is Workspace for Grafana Assistant: a full-page view of Assistant so your conversations and generated graphs aren’t crammed into the little right-hand sidebar. It’s a much less cluttered place to run investigations and to be agent-first. On top of that, Assistant is now easier to try — enable it in Grafana 13 for OSS without linking a Grafana Cloud account, or try it with no signup or download at all on Grafana Play.

Transcript

Hi everyone! I’m Nicole van der Hoeven and I’m really excited to welcome you to AI Week. Today is Monday, which is planning day, where we give you the tools you need to get clarity on what even is going on in your system so that you can figure out what to build.

Now we’re kicking things off with a feature called Workspace for Grafana Assistant. Now if you’ve used Assistant before, you’ll know that it comes up on the little sidebar on the right side of Grafana. Well, that can start to get cramped if you have multiple conversations or if you’ve asked Assistant to generate multiple graphs for you. Well, we’ve heard you, and now we’re giving you a full page view of Assistant. So you’ll have all of your conversations on the left, and this is also an excellent place to run investigations from. You can also zoom in on any of those graphs, and it’s just a much less cluttered view if you want to be agent-first.

Speaking of Assistant, the second thing I want to tell you is that you know, Assistant was already available everywhere. But now, Assistant is available… in more places than everywhere? And also more easily. So Assistant was already available for OSS, but you had to link your Grafana Cloud account, which was not so handy if you wanted to try it out. But not anymore. Now if you download Grafana 13 and enable Assistant, then you’ll just be able to go and try it out and chat with Assistant without doing anything else. It’s still not free or open source, but it IS a lot easier to try it out.

And if you don’t want to download anything, that’s what Grafana Play is for. Grafana Assistant is now available to try out on Grafana Play without any signup or any download required. Try it out at play.grafana.org. I really love seeing it there. Tune in tomorrow where we’re going to talk about everything we’re announcing to help you start to build.

Zero to Dashboard with Grafana Assistant and the Infinity datasource plugin

Fri, 22 May 2026 17:00:00 +0100

What if you didn’t have to write a single query to get a useful dashboard?

In this short video, I go from nothing to a working dashboard in a few minutes — no queries — using Grafana Assistant and the Infinity datasource plugin. The data I’m visualising is the RAWG video game database API, so I end up with a dashboard that helps me decide what to play next.

Why Infinity is fun here

Most Grafana data sources are scoped to a specific kind of backend — Prometheus, Loki, a SQL database. Infinity is different: it lets you treat any REST/JSON/CSV/HTML endpoint as a data source. That means anything with an API can become a dashboard, which is exactly the right shape for goofing around with a games API on a Friday afternoon.

Pair Infinity with Assistant and you don’t even have to know what queries to write — Assistant configures the data source, asks the API the right questions, and builds the panels for you.

Timestamps

00:00 — Intro
00:35 — The RAWG video game database
01:01 — Setting up the Infinity datasource
05:10 — Customising the dashboard with Assistant
09:31 — Closing thoughts

Resources

Step-by-step blog post: using Assistant with the Infinity datasource
Grafana Assistant documentation
Infinity datasource plugin
RAWG video game database API
Get started with Grafana Cloud (free tier)
My notes on Grafana Assistant — everything I’m learning as I work on it

Do Androids Dream of Second Brains?

Mon, 18 May 2026 07:00:00 +0100

We often describe our personal knowledge management systems as “second brains” — but can we really say they think for themselves? In this video, I make the case for instrumenting your vault with an observability stack and layering AI on top, not because it’ll turn your notes into a brain, but because it might get us closer than anything else has.

This is the video version of the talk I gave at PKM Summit 2026 in Utrecht earlier this year, with the live demos cleaner and the dashboards working without the conference wifi gods getting in the way.

What I cover

🐑 Why “second brain” is the wrong framing

I take a small detour through Philip K. Dick’s Do Androids Dream of Electric Sheep? — the inspiration for Blade Runner — and the Voigt-Kampff test, which measures empathy through observed responses to staged scenarios. It’s a useful lens for asking what we actually mean when we say a knowledge system “thinks”: we’re really asking whether it can respond to us in a way that surprises us. Most second-brain setups can’t, yet.

📊 Observability for PKM

Observability is a concept I’ve borrowed wholesale from how we monitor software systems — the idea that you can understand a complex system by what it emits, not by reading its internals. Three steps: instrumentation, analysis, improvement. I walk through what each one looks like for a personal vault:

Instrumentation: a small script that emits metrics about my Obsidian vault — note counts, link density, growth over time, what folders are loud and which are quiet
Analysis: a Grafana dashboard sitting on top of Loki + Prometheus, surfacing things I genuinely didn’t know about my own vault
Improvement: the part where seeing the data actually changes how you work

There’s a live dashboard tour, and a section where I ask Grafana Assistant to edit the dashboard for me in natural language — including writing the PromQL queries I never want to write by hand.

👁️ Meet Iris

About halfway through the video I introduce Iris — the AI agent I built using OpenClaw that lives in my vault, reads my notes, and works alongside me. I walk through:

What “agentic AI” actually means in practice (vs. just chatting with an LLM)
The OpenClaw setup — Docker sandbox, Syncthing, the whole pipeline
Iris rewriting my observability map of content
Iris doing talk-topic analysis on my vault
Iris running benchmark research for a blog post
Iris OCRing my reMarkable handwriting

🌙 Dreaming

The last section is the one I had the most fun with. I argue that the thing most second-brain setups are missing isn’t more notes or better search — it’s synthesis. The capacity to take everything you’ve captured and surface patterns, contradictions, and ideas you hadn’t formed yet.

I introduce OpenClaw’s dreaming feature: a memory consolidation process inspired by how sleep stages work in the brain. Light sleep, REM, and deep sleep each do different work — surfacing recent material, finding unexpected connections, and consolidating long-term memory. I show how each phase actually behaves in my own system.

And there’s a Zhuangzi reference. (You’ll see.)

Resources

🎤 PKM Summit 2026 talk recap (with photos)
🎞 Slides
👁️ OpenClaw
📓 Obsidian
🔄 Syncthing
📊 Grafana

The Architecture of Forgetting — the essay version of why memory consolidation matters
Unlearnings from building Grafana Assistant

If you want to follow more of how Iris works and what I’m learning about AI + PKM, Context Horizon is where I write about it.

Grafana k6 Community Call — Secrets Management & AI Script Authoring

Thu, 30 Apr 2026 16:30:00 +0100

The first k6 Community Call in over a year, and we packed it. I sat down with four of the people building k6 — Bukola Ayodele, Facundo Batista (Software Engineer), Vicente Ortega (Senior Backend Engineer), and Charlie Burnett (Product Manager) — to walk through two new pieces of work that have shipped recently, plus a look at what’s coming in k6 v2.

What we covered

🔐 Secrets management in k6 Cloud (Facundo)

A new way to store API tokens, passwords, and other sensitive values for your tests — centralized in Grafana’s secrets manager, encrypted at rest, and never visible in the UI or API once written. Your scripts reference secrets by name, and at runtime the values are injected and automatically redacted from logs.

Why this matters for load testers:

No more hardcoded credentials sitting in scripts or in Git
Easy rotation — change the secret value, every script that uses it picks up the new value, no script edits needed
Same script works across dev/staging/prod by just swapping which secrets are scoped to which environment
Works with k6 OSS locally too — your local runs can pull cloud-managed secrets when you want them

Facundo Batista demoed the full flow: creating a secret in the Grafana Cloud UI (name, description, labels — up to 10 per secret, all write-only), and using it in a script with one line. There’s also an API for programmatic management.

🤖 Grafana Assistant k6 script authoring mode (Vicente)

This is the AI-powered script generation that’s been in the works for a while — fine-tuned specifically for k6 best practices, not a generic “ask an LLM to write you a load test” workflow. The interesting bit is the context sources Assistant can pull from when generating a script:

A service name — Assistant analyzes the existing telemetry data to discover endpoints and shapes
A GitHub repository URL — pulls the code to understand the API
An OpenAPI specification — generates tests that exercise every endpoint with realistic request shapes
Existing observability data — metrics, logs, traces from your actual production traffic

Generated scripts come with proper structure, error handling, checks, and TODO comments where the model genuinely doesn’t know what to write (e.g. for auth tokens, request bodies, or anything that needs your secrets — by design, Assistant won’t fabricate those). You can also point it at an existing script and ask it to refactor or extend.

🚀 k6 v2 (Charlie)

Release candidate 1 is out, GA is very soon. The headline changes:

Extensions marketplace — community protocols and integrations
Enhanced Playwright browser support with compatible syntax
New assertions library
MCP / agent connectivity for AI-driven workflows
DNS, MQTT, and other network protocols

Migration advice from Charlie Burnett: pin your script versions, test your existing scripts against the RC before GA hits, and don’t rush.

🔮 What’s next on the AI roadmap (next 1–2 months)

Visual browser-based testing experience
Enhanced observability workflow automation
Improved user-journey creation across both personal and enterprise apps
Integration with autonomous monitoring systems

Resources

Thanks to Bukola, Facundo, Vicente, and Charlie for joining, and to everyone who showed up live.

How to use Grafana Assistant with self-hosted Grafana

Mon, 20 Apr 2026 09:00:00 +0100

In case you missed it, possibly the biggest news to come from GrafanaCON this year is that if you’re running self-hosted or open-source Grafana, you can now connect it through your Grafana Cloud account to Grafana Assistant, the purpose-built AI tool for all things Grafana.

In this short tutorial, I show you how to enable Assistant on a self-hosted Grafana instance and go from nothing to a working server-health dashboard with a single prompt.

A quick note on what “for open source” actually means

Assistant itself isn’t open source. But you can now use it with your self-hosted or open-source Grafana instance. The way it works is:

You still need a Grafana Cloud account (the free tier has generous inclusions for Assistant).
You link your self-hosted Grafana to that Cloud account.
Assistant then lights up inside your OSS instance.

Think of the Grafana Cloud account as the key that unlocks Assistant wherever your Grafana lives.

Prerequisites

Grafana OSS v13 or later
A Grafana Cloud account (pricing here)

Enabling Assistant

Go to Administration → Plugins, search for Assistant, and click Install.
Refresh the page, then open the Connection tab.
Click Start Connection. You’ll be prompted to log into Grafana Cloud.
Select the organization and stack you want, then click Authorize connection.
Close the tab. Your self-hosted Grafana is now linked to Assistant.

A first dashboard from nothing

Every Grafana instance ships with a TestData data source — perfect for a first run when you don’t yet have real data wired up. Enable it from Data Sources, then open Assistant (the new icon on the top right).

Switch Assistant’s mode from General to Dashboarding — this mode is tuned for creating and managing dashboards — and ask in plain English, for example:

Hi Assistant, could you create a server health dashboard for me using the TestData data source?

What comes back is a starting dashboard with CPU, memory, disk, and network panels already wired up. From there you can tweak panels, edit queries, or ask Assistant to iterate. You’re not starting from scratch — you’re starting from 80%.

What else Assistant can do

I only showed dashboarding in this video, but Assistant can also:

Answer questions about the data already in your Grafana
Run investigations to troubleshoot production incidents
Learn new skills
Manage alerts
Teach you Grafana

That list keeps growing, so stay tuned.

Resources

Grafana Assistant (product page)
Grafana Cloud pricing
How Grafana Labs handles your data
My notes on Grafana Assistant — everything I’m learning as I work on it
Unlearnings from building Grafana Assistant — my Context Horizon piece on what the team had to unlearn

Unlearnings from building Grafana Assistant

Thu, 09 Apr 2026 09:00:00 +0100

Originally posted here

Everyone’s talking about what you need to learn to build AI agents: Prompt engineering. Tool calling. Retrieval strategies. The content ecosystem is full of it. But while I was watching Context Window Episode 1, it struck me that Mat Ryer, Cyril Tovena, and Sven Großmann– some of the team behind Grafana Assistant– spent most of their time on something different: what they had to unlearn.

Grafana Assistant started as a weekend hackathon project. By the time it shipped, the team had thrown out a popular framework, flipped their assumptions about tool design, and discovered that some of the instincts you bring from conventional software development actively work against you when you’re building agents.

I pulled out the ten moments where received wisdom turned out to be wrong. If you’re building anything agentic right now, these are worth sitting with.

❌ LangChain will help us build faster

LangChain is an opensource framework for building applications that use LLMs. It provides structured components that can be used for rapid prototyping.

In the video, Tovena talks about how they started with LangChain during the initial hackathon because they thought it would help them build more quickly, but instead it became obvious fairly early on that the opposite was true. They didn’t end up using a lot of the features that LangChain provided and felt they needed more flexibility and speed.

Abstraction overhead

One possible reason for their troubles with LangChain is that LangChain has standard abstractions for prompts, tool use, and orchestration– things that sound good in theory. But sometimes abstractions lend themselves to creating a sort of black box in your stack, where you don’t know what’s happening inside it. It sounds like the Assistant team missed the oversight and control of what was happening.

❌ Use the same prompt for every model

They also talked about how for the sake of simplicity and consistency, they initially used the same prompt for every model. However, they found out that this actually led to worse results. They found more success in creating slightly different prompts for different models, playing to their strengths and weaknesses a bit more.

There is even an argument (Mizrahi et al.’s “State of What Art? A Call for Multi-Prompt LLM Evaluation) that AI benchmarks that involve sending multiple LLMs the same prompt is a brittle and unreliable way to assess LLM capabilities. Different models respond differently to prompt variations.

In practice, what this means is that switching models isn’t a drop-in replacement. Even if a new model scores better on benchmarks, your prompts were written with a specific model’s tendencies in mind. The unlearning here is letting go of the idea that a prompt is model-agnostic — it’s more like a conversation style, and different models have different conversational defaults.

❌ Treat errors as failures

Traditionally in software, you catch errors and hide them from the user: you log them, handle them, but treat that transaction as a failed one.

The Assistant team had to unlearn this habit when dealing with an LLM, because with agents, error messages are useful data. Feeding it back to the LLM often results in it learning from its own mistake and trying again. It’s basically letting the agent debug itself, in a manner that Ryer noted was remarkably similar to how we humans debug errors too.

❌ Return errors in structured JSON

On the subject of error messages, Großmann mentioned that at one point they were trying to format the errors in JSON or stringified JSON. This practice shows solid engineer intuition because usually, structuring information helps machines process it faster and more accurately. What they found here was the reverse: it was easier and yielded better results when the LLM was just passed the entire error in natural language.

❌ Build as many tools as possible

The team mentioned that at the start, they were inclined to create as many tools/skills for Assistant: one for everything that could be done in Grafana. This made sense until they saw LLMs struggle to determine which tool to use in which context.

To help LLMs identify the correct tool, they added tool descriptions and also experimented with combining multiple tools into fewer, parameterized, tools. These helped, although the team still observed that allowing users to create their own tools (with potentially very general tool descriptions) could potentially confuse LLMs.

❌ Bigger context window = better

Gemini 1.5 Pro announced it had a 2M context window back in 2024, and it might seem that other providers had lagged behind: OpenAI only announced support for 1M tokens in 2025 and Anthropic followed in 2026. However, the Assistant team pointed out that context wasn’t everything. They said that they noticed that bigger wasn’t necessarily better, also pointing out that the chat summarization feature on IDEs like Cursor could still sometimes lose important details when summarizing.

This finding seems to coincide with a 2023 Stanford study (Lost in the Middle: How Language Models Use Long Contexts) that found there was a performance degradation when models access information within a very large context. In the video, they talked about how their system prompts were often 40k tokens long (half for the prompt itself and then the other half for tools).

❌ Start with a multi-agent architecture

Tovena mentioned that creating a multi-agent architecture for Assistant yielded more mixed results than he expected. He said that passing context between agents is difficult, outputs were inconsistent, and there was a bit of a “telephone game” problem where agents didn’t always perfectly understand the intent or output of other agents.

In early 2026, Khatua et al. released a study called “CooperBench: Why Coding Agents Cannot Be Your Teammates Yet” that showed that AI agents aren’t great at collaboration. The study suggests, surprisingly, that this might be because AI agents lack human social habits that help us collaborate better, such as feeling obliged to do what we say and having a good understanding of what others’ plans are.

The Assistant team’s experience is also a perfect example of Gall’s Law, which states that it’s easier to start with simplicity and build complexity than the other way around.

❌ Plan around models as they are now

Software has always moved quickly, but it feels like AI models are moving faster. Stanford’s AI Index Report 2025 reported that models went from having a 50% chance to near-human performance in roughly 2 years (2021-2023). In fact, benchmarks keep getting retired as AI saturates them– meaning that benchmarks can’t reliably be used to distinguish between good and bad models because all the models are testing near the ceiling.

The rapid pace of development around AI requires us to not be so attached to our work: Großmann recalled an incident where someone wrote a joke prompt that spoke like a dog that shockingly did better than their other system prompts. Ryer also mentioned that at one point someone deleted all the system prompts and Assistant weirdly became more accurate.

The lesson is clear: plan for models to improve, and plan to redo a lot of your work when they do.

❌ We don’t need evals

The Assistant team said that they went from having no evals to employing several types of evals:

integration-style evals that exercise whole workflows
unit-style evals that feed inputs plus a system prompt and check the output
LLM-as-judge, where they pass conversations to an LLM to ask it to judge whether it answered the user’s questions.
user feedback in the form of thumbs during conversations with Assistant

Beyond evaluation, they talked about how important it was to be able to observe AI. At the very minimum, showing what Assistant is doing at all times (thinking out loud) is very important to them because it builds users’ trust in the thinking process and, eventually, in the result.

Evals aren’t a sign that you don’t trust your product. They’re how you build the confidence to keep shipping it.

❌ Assume people already know how to work with AI

When you ship an AI feature, it’s tempting to assume the hard work is done. You’ve built the thing. Users will figure it out.

What the Grafana Assistant team found was that this assumption doesn’t hold. Users came in with wildly different mental models of what an AI assistant could or should do — some treating it like a search box, others expecting it to read their minds, most not yet sure how to prompt it effectively.

Their insight was that this isn’t just a UX problem to solve on your end. It’s a shared responsibility: AI should try to help the user, but the user also needs to learn how to work with AI. And building an assistant that teaches people how to use it well (through good responses, gentle corrections, showing its work) is part of building the product.

There’s also a deeper point here specific to observability. Grafana Assistant isn’t just an AI feature; it’s a way of interacting with your own infrastructure. Every good interaction teaches you something about what questions to ask, what signals to look for, what it means to understand a system. In other words: learning to use AI well makes you a better engineer. The tool and the skill co-evolve.

This is the unlearning that sits furthest from the code. It’s not about prompts or architectures or frameworks. It’s about recognising that you’re not just shipping a feature– you’re introducing a new way of working. That takes time. And it’s worth designing for explicitly.

Summary

What strikes me most about this list isn’t any single item — it’s the pattern. Most of these unlearnings point in the same direction: less is more, and messiness is the point.

Less abstraction. Fewer tools. Smaller context. Simpler architecture. And rather than hiding the mess — the errors, the reasoning, the uncertainty — let it show. That’s where the trust gets built.

We’re all still early with this. The team that built one of the most thoughtful AI products I’ve seen up close still didn’t have evals when they launched. That’s not a criticism — it’s a reminder. You don’t need to have it all figured out before you ship. You just need to commit to keep unlearning.

Let There Be Dashboards: Introducing Grafana Assistant

Thu, 26 Mar 2026 19:30:00 +0100

I gave this talk at the Grafana & Friends Lisbon Meetup in Lisbon, Portugal.

Abstract

This is a live demo of how to go from zero to a working Grafana dashboard using Assistant. Starting with a blank slate, we’ll use natural language to explore data and generate meaningful visualizations in minutes. Along the way, you’ll see how AI can accelerate dashboard creation without sacrificing control or understanding.

Thanks to everyone who came out! It was great to connect with the Lisbon tech community.

Grafana Assistant on Nicole van der Hoeven

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Transcript

Resources

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Transcript

Observability and AI explained

Resources

Investigations Arena: how we test Assistant investigations

Transcript

Resources

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Transcript

AI Week Day 1: Planning (Grafana AI Week — Monday)

Transcript

Zero to Dashboard with Grafana Assistant and the Infinity datasource plugin

Why Infinity is fun here

Timestamps

Resources

Do Androids Dream of Second Brains?

What I cover

🐑 Why “second brain” is the wrong framing

📊 Observability for PKM

👁️ Meet Iris

🌙 Dreaming

Resources

Related reading

Grafana k6 Community Call — Secrets Management & AI Script Authoring

What we covered

🔐 Secrets management in k6 Cloud (Facundo)

🤖 Grafana Assistant k6 script authoring mode (Vicente)

🚀 k6 v2 (Charlie)

🔮 What’s next on the AI roadmap (next 1–2 months)

Resources

Related Context Horizon pieces

How to use Grafana Assistant with self-hosted Grafana

A quick note on what “for open source” actually means

Prerequisites

Enabling Assistant

A first dashboard from nothing

What else Assistant can do

Resources

Unlearnings from building Grafana Assistant

❌ LangChain will help us build faster

Abstraction overhead

❌ Use the same prompt for every model

❌ Treat errors as failures

❌ Return errors in structured JSON

❌ Build as many tools as possible

❌ Bigger context window = better

❌ Start with a multi-agent architecture

❌ Plan around models as they are now

❌ We don’t need evals

❌ Assume people already know how to work with AI

Summary

Let There Be Dashboards: Introducing Grafana Assistant

Abstract