Grafana Labs on Nicole van der Hoeven

AI Tools We Use Every Day on the Grafana AI Team

Fri, 30 Oct 2026 09:00:00 +0100

What AI tools do the people building AI features actually reach for day to day? I asked around the Grafana AI team to find out which tools have genuinely stuck — the ones we open without thinking — versus the ones that didn’t survive contact with real work.

Resources

My notes on Grafana Assistant

What Does a Grafana AI Offsite Look Like?

Wed, 28 Oct 2026 09:00:00 +0100

A quick behind-the-scenes short: what actually happens when Grafana’s AI team gets together in person for an offsite.

Resources

My notes on Grafana Assistant

Roundtable: How We Really Use AI in Testing (HUSTEF 2026)

Thu, 08 Oct 2026 14:00:00 +0200

On the final day of HUSTEF 2026 in Budapest, I joined the conference’s roundtable on AI in testing — “AI Open Space Experiences: a roundtable with speakers on their experiences with AI testing.” It was five of us on stage: me plus Marta Firlej, Kristoffer Nordström, Callum Akehurst-Ryan, and Maryia Tuleika, with Marta moderating.

The format was refreshingly honest — “how we really use AI, with no bullshit.” Marta had run the same “AI open space” exercise with the Polish testing community, crowdsourcing the actual, un-hyped ways people use AI (information gathering, functionality discovery, generating tests and code, TDD, recruitment, presentation prep, even designing furniture and planning holidays), and then opened it up to the panel and the room. Here’s roughly how I answered her questions:

What do you use AI for? Mostly two things at work: analysing codebases and test code to understand what something actually does, and filling gaps on teams without a dedicated quality engineer — baking my testing knowledge into an AI the team can ask questions of. On the personal side, I’ve run some weird experiments: I’m a lifelong note-taker with decades of context (journals, therapy notes, eight years of WhatsApp history with my partner), so I’ve fed it all into a local, open-weight model and asked things like “what do I claim to believe, and where have I acted contrary to that — cite your evidence?” It’s a surprisingly honest mirror. I’m in a privileged spot with effectively unlimited AI access right now, and I know it won’t last, so I want to find out what’s safe and what isn’t while I can.

How do you stay up to date? Honestly, I don’t — not by chasing model releases or news. I keep up with how other people, especially in testing, are using AI, and I lean on my coworkers (we’re at the forefront of this at work, so stepping outside that bubble is a useful reality check). I do have a scheduled Claude Code task that scrapes my ~34 AI-related Slack channels into a daily digest of just the things I care about.

One piece of advice? Don’t trust AI too much, and don’t believe you’re as far behind as you think. Almost no one has really tried this stuff; it’s genuinely early, everyone’s learning, and your foundational testing skills transfer. Just pick it up, play, and if it’s not for you, move on.

Anything you’d never use AI for? Communicating with people as a proxy. I’ll use it to translate or to help me sound more fluent (I live across multiple languages — Portugal, Dutch, English), but I never want it to craft a whole message I didn’t think of. I don’t want to lose the discipline of actually talking to a human.

Ethical dilemmas? Yes — which is exactly why I keep my use mostly in the context of work, and try not to generate imagery or anything that’s obviously lifting from artists who deserve to be paid for it. There’s no hard line; it’s a sliding scale, and I have to keep questioning myself. (A Mastodon line stuck with me: the strange future where the humans do the menial tasks and the robots write poetry and paint. I try not to let AI stop me from making art.)

The best part of these panels is always the disagreement and the texture — five people who use AI daily and still land in very different places on trust, ethics, and what it’s actually for. Thanks to Marta for moderating and to Kristoffer, Callum, and Maryia for a genuinely good conversation.

Resources

HUSTEF 2026
My HUSTEF keynote: Asimov’s Zeroth Law of Robotics: Testing for AI

Asimov's Zeroth Law of Robotics: Testing for AI (HUSTEF 2026)

Wed, 07 Oct 2026 10:00:00 +0200

This week I gave a keynote at HUSTEF 2026 in Budapest, Hungary. The talk is called Asimov’s Zeroth Law of Robotics: Testing for AI — a testing-focused evolution of a talk I’ve been giving (and rewriting) for a while, previously as Observability for AI at KubeCon EU 2025 in London, Dutch Cloud Native Day 2025 in Utrecht, NewCrafts in Paris, and ExpoQA Madrid 2026. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be testable.

AI systems fail differently than traditional software. There’s no single correct output to assert against. A response can be fluent and completely wrong. Models drift silently over time. And yet we’re shipping AI into production — often without knowing what “passing” even looks like.

Observability is what makes testing possible. When we can see inside our AI systems — trace the chain of reasoning, measure what changed, surface what failed — we can start to write evaluations, define what “good” looks like, and catch regressions we didn’t know to look for.

In this talk, I explore what it means to test AI in the real world, drawing on lessons from building and evaluating an AI assistant for a large-scale observability platform. I’ll cover how evaluations work as the AI equivalent of a test suite, what to measure, how to catch hallucinations, and how to know when your model has regressed.

You cannot test what you cannot see — and you cannot trust what you cannot test.

The big shift in this version is right there in the Zeroth Law itself. In earlier iterations I argued that a robot must be observable. That’s still true — observability is the only way to see the reasoning behind a system and find out what actually happened. But I’ve come to think that pure knowledge isn’t the same as trust. You can know exactly what a system did and still not trust it. The only way to trust something is to put it through its paces: to know how it behaves in the situations you foresee, and even the ones you don’t. So the Zeroth Law has become a robot must be testable.

I framed the whole thing around Asimov’s short story Runaround (from I, Robot) and its robot, Speedy. Speedy’s smuggler owners sent him to collect selenium on Mercury, and he got stuck in a loop — circling the selenium pool, unable to resolve the conflict between obedience (the second law) and self-preservation (the third). His owners couldn’t predict his behaviour and couldn’t reason about it, so they had to do the only thing left: go out into the danger zone and fix him in production. Speedy wasn’t untestable because his owners were incompetent or lacked the tools — he was untestable because he was never built to be tested. That’s the trap I think a lot of AI systems are walking into right now.

The through-line for a testing audience: AI is rewriting how software gets built, but it doesn’t rewrite the discipline of trust. Trust that a system does what we say it does, and that when it breaks we know why and can fix it — that’s always been the job, and it isn’t going anywhere. There’s a whole generation of AI engineers shipping software they haven’t tested and can’t explain, and when someone says “the agent does this,” our job is still to ask how do you know? Testing isn’t just going to survive the age of AI — testing is the reason AI gets to survive, if we make it testable.

I demoed this with my TTRPG side project: an AI Dungeon Master, instrumented and put under test. It turns out that writing evals for something non-deterministic forces you to confront your real requirements. I thought I knew exactly when a dungeon master should call for a dice roll — I GM myself! — but trying to put it into words (“in this situation I’d roll, but in that one I wouldn’t, because I want to keep the story moving”) surfaced decisions I’d never had to make explicit. Those are things we’d normally discover much later, in production. Having to figure them out up front is a win.

The Q&A was genuinely great and ranged well beyond the slides:

“When has an AI been tested enough?” Same answer as for anything else — it’s a mix of risk and budget. If it were a life-critical robot like Speedy, you test every scenario the people relying on it need. For my little demo app, less so.
Testers vs. developers in the age of AI. I said I’d fire developers before testers — provocatively, but the point stands: anyone can write code now. What made a good developer good was never the coding; it was communicating with business and engineers, translating requirements, finding creative-but-safe solutions. Testers already think at that level and translate across teams, which makes them more important in the age of AI, not less.
Defining expected results for non-deterministic systems, multi-turn evals (I give the simulated user a personality — adversarial, cooperative, beginner — rather than a script), and the broader shift from imperative to declarative: we no longer instruct step by step, we declare the end state we want, the way we do with Kubernetes.
Convincing colleagues not to fear AI. I don’t, really. AI is here to stay; the question isn’t whether you approve of it, it’s whether you want to stay in this industry and help make it safe and trustworthy — or sit on the sidelines.

I also recommended the book Slow AI, which lands where I do: not an AI apologist, not a boycott, but a clear-eyed list of what AI is and isn’t good for — and the reminder to keep checking whether a given use of AI is making you more capable or less.

The demo app (the instrumented AI DM) lives on GitHub, and I keep refining it with each iteration of this talk.

I’ll update this post with the recording once HUSTEF publishes it.

Resources

Slides
Demo app on GitHub
Slow AI (book)
Previous versions of this talk: KubeCon EU 2025 · Dutch Cloud Native Day 2025 · NewCrafts 2025 · ExpoQA Madrid 2026

Context Engineering Fundamentals (The Context Window #06)

Fri, 02 Oct 2026 18:00:00 +0100

What actually counts as “context” for an AI agent — and how does it find the right things to put in a limited window? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I dig into context engineering fundamentals with Ben Sully and Ioanna Armouti. We treat the context window as an attention budget, walk through Assistant Search and semantic retrieval, embeddings, and how you actually evaluate whether the retrieved context was any good.

What we cover

What counts as context — and why the context window behaves more like an attention budget than just a size limit.
Finding the right context — Assistant Search, semantic retrieval, and embeddings: how an agent surfaces what it needs instead of being handed everything.
Evaluating retrieved context — because retrieval that looks relevant isn’t the same as retrieval that is.
The tool tax and tool deferral — every tool definition costs context, so when do you load it versus defer it?
Progressive disclosure and long-term memory — giving the agent what it needs, when it needs it.
Why removing 80% of a system prompt can change nothing — a nice illustration that more context is not automatically better context.

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana GCX

Fri, 21 Aug 2026 09:00:00 +0100

Three AI tools people confuse: Assistant, MCP, and GCX. Last time I talked about the easy hand — the MCP server. Now the power hand: GCX, a CLI built for AI agents.

If Assistant is the brain and Grafana MCP is the easy hand, then GCX is the power hand — a CLI built for an agent to drive. (Quick refresher: a CLI is a command line interface — no clicking, just typed commands.)

So when does GCX win?

When you’re using terminal coding agents — Claude Code, Cursor, Codex. It writes code, pulls the exact log or metric, and verifies against it — without leaving the editor.
When you need everything. GCX wraps the entire Grafana Cloud API — SLOs, Synthetics, IRM, k6, Fleet — the esoteric corners MCP skips. The full banana.
When you want automation and CI/CD. Dry-run mode, JSON output, and exit codes an agent can act on — built for GitOps.

Do you need Grafana Assistant for this? No — not even a subscription. Bring your own brain, same as MCP: metrics, logs, dashboards, and alerts run on nothing but a service account token. You only really pay for Assistant if you call gcx assistant directly — which taps its reasoning, billed separately.

So there’s your map: Assistant’s the brain, MCP’s the easy hand, GCX is the power hand. It’s not one versus another — they work really well together to give you the best choice in how to access your Grafana stack.

Transcript

I’m differentiating Assistant, Grafana MCP and GCX and in this one I’m talking about GCX. If Assistant is the brain and Grafana MCP is the easy hand, then GCX is the power hand — a CLI built for AI agents.

Quick refresher: a CLI is a command line interface — no clicking, just typed commands. GCX is a CLI built for an agent to drive.

So when does GCX win?

When you’re using terminal coding agents — Claude Code, Cursor, Codex. It writes code, pulls the exact log or metric, and verifies against it — without leaving the editor.
When you need everything. GCX wraps the entire Grafana Cloud API — SLOs, Synthetics, IRM, k6, Fleet — the esoteric corners MCP skips — the full banana.
When you want to do automation and CI/CD. Dry-run mode, JSON output, exit codes an agent can act on — built for GitOps.

Do you need Grafana Assistant for this? No — not even a subscription. Bring your own brain, same as MCP: metrics, logs, dashboards, alerts run on nothing but a service account token. You only really pay for Assistant if you call gcx assistant directly — which taps its reasoning, billed separately.

So there’s your map: Assistant’s the brain, MCP’s the easy hand, GCX is the power hand. It’s not one versus another. They actually work really well together to give you the best choice in how to access your Grafana stack.

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana MCP

Wed, 19 Aug 2026 09:00:00 +0100

Three AI tools people confuse: Assistant, MCP, and GCX. Last time I talked about the brain — Assistant. Now hand number one: the MCP server, the easy hand.

Grafana MCP — Model Context Protocol — lets any AI client reach into Grafana. The magic part: you bring your own brain.

Using MCP requires two pieces:

The client is your AI app — the brain you bring: claude.ai, ChatGPT, Cursor.
The server is the door into Grafana that exposes the tools.

There are two ways to get that server. The easiest is the Grafana Cloud hosted one — already running, auth built in, just connect and go. But you can also self-host the open-source one on your own infra.

So when do you reach for Grafana MCP?

When you’re using hosted chat tools with no terminal — claude.ai, ChatGPT, your phone. No shell means no CLI, so MCP is the only way in.
When you want to bolt Grafana onto a tool you already use — already in Cursor or Claude? Just add the server. No new workflow.
When you want the best discovery out of the box — the tools describe themselves up front, so even smaller or cheaper models just know what’s available. No trial-and-error.

One honest catch: those tool definitions can eat some context if you load everything — but modern hosts fix that with lazy tool search.

MCP is deliberately curated — it doesn’t expose every corner of Grafana. So what do you do when you need the esoteric stuff? That’s exactly what the power hand, GCX, is for — next up.

Transcript

Three AI tools people confuse: Assistant, MCP, GCX. Last time, I talked about the brain — Assistant. Now hand number one: the MCP server, the easy hand.

Grafana MCP — Model Context Protocol — lets any AI client reach into Grafana. The magic part: you bring that AI client - you bring your own brain.

Using MCP requires two pieces. The client is your AI app — the brain you bring: claude.ai, ChatGPT, Cursor. The server is the door into Grafana that exposes the tools.

Two ways to get that server: the easiest way is to use the Grafana Cloud hosted one — already running, auth built in, just connect and go. But you can also self-host the open-source one on your own infra.

So when do you reach for Grafana MCP?

When you’re using hosted chat tools with no terminal — claude.ai, ChatGPT, your phone. No shell means no CLI, so MCP is the only way in.
When you want to bolt Grafana onto a tool you already use — already in Cursor or Claude? Just add the server. No new workflow.
When you want the best discovery out of the box — the tools describe themselves up front, so even smaller or cheaper models just know what’s available. No trial-and-error.

One honest catch: those tool definitions can eat some context if you load everything — but modern hosts fix that with lazy tool search.

But MCP is deliberately curated — it doesn’t expose every corner of Grafana. So what do you do when you need the esoteric stuff? That’s exactly what the power hand is for. Next up is GCX.

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Mon, 17 Aug 2026 09:00:00 +0100

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. They’re not competitors. Think of it as a brain and two hands. In this short, I start with the brain: Grafana Assistant.

Assistant isn’t just a chatbot — it’s the reasoning layer. It manages context and knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. Instead of clicking around, you tell Assistant what you want to do and it figures out the how.

Reach for the Assistant when the work is visual or exploratory:

Building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
A real incident that needs root-cause analysis across multiple data sources? Assistant. That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in, and I’ll cover the first of those (the MCP server) next.

Transcript

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. I’m going to talk a bit about each one so you know which one is best for your use case.

Spoiler: They’re not competitors. Think of it as a brain and two hands.

Let’s start with Grafana Assistant, which is the brain.

It’s not just a chatbot — it’s the reasoning layer. It manages context, it knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. So instead of you clicking around to do something, you can say what you want to do to Assistant and it figures out the how.

Reach for the Assistant when the work is visual or exploratory.

If you’re building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
You have a real incident where you need root-cause analysis across multiple data sources? Assistant.
- That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in… and I’ll show you those next.

Resources

Grafana Assistant (product page)
My notes on Grafana Assistant
My notes on GCX
Unlearnings from building Grafana Assistant — my Context Horizon piece

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Fri, 31 Jul 2026 09:00:00 +0100

The final day of AI Week is about collaborating — using AI right where your team already works, no context switching required. Grafana Assistant in Slack is now generally available: investigate issues, ask follow-up questions, and query your metrics and logs from the channel where the incident conversation is already happening. And that’s just one place Assistant now meets you — also announced: the Grafana mobile app is GA (with Assistant on mobile in public preview), Assistant Desktop in public preview, Microsoft Teams support in private preview, and context-aware Assistant Search in public preview. Add it all up and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search. That’s a wrap on five days of announcements.

Transcript

It’s the final day of Grafana Labs’ AI Week, and Friday is all about collaborating — using AI right where your team already works, no context switching required.

Today, Grafana Assistant in Slack is generally available. Investigate issues, ask follow-up questions, query your metrics and logs — all from the channel where your incident conversation is already happening.

And that’s just one place Assistant now meets you. Today we also announced: the Grafana mobile app is GA, with Assistant on mobile in public preview. Assistant Desktop is in public preview. Microsoft Teams support is in private preview. And Assistant Search — deeper, context-aware search — is in public preview too.

Add it all up, and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search.

That’s a wrap on five days of announcements — and I know it was a lot. So if you missed a day, or you just want the full picture of where AI at Grafana stands today, I made a video for exactly that: everything you can do with AI in Grafana Cloud, including everything we shipped this week. It’s linked below and pinned in the comments.

One last time: grafana.ai for all of it. Thanks for spending AI Week with me.

Observability and AI explained

Fri, 31 Jul 2026 09:00:00 +0100

“AI observability” gets thrown around a lot, and it can mean one of two very different things. So in this video I map out the whole landscape of AI-related work at Grafana across the entire software development lifecycle — and show how AI and observability come together in two directions: AI for observability (AI that helps you observe your systems) and observability for AI (observing the AI itself).

The framing is deliberate: AI in observability isn’t about replacing engineers. It’s about handing back the exhausting parts of the job — the 3am triage, the repetitive queries, the report nobody wanted to write — so you can spend your energy on the work only humans can do.

I walk through each phase of the lifecycle and where the tools fit:

Planning — clarity about what’s really going on, with Grafana Assistant: not just a chatbot, but an agent that knows your telemetry, writes your queries, and builds your dashboards.
Building — momentum without losing control: Assistant, the open-source GCX CLI, the Grafana MCP server, the specialized Loki/Tempo/k6 MCP servers, and the AI SDK for building your own agents in Go.
Operating — staying calm under load: Investigations for agentic incident triage, Assistant Automations and the Watcher Agent for staying proactive, plus Grafana Machine Learning (Metrics Forecasting, Outlier Detection) and Adaptive Telemetry.
Testing & evaluating — doubt is a form of care: k6 and agentic testing on the systems side, and Agent Observability plus the open-source o11y-bench benchmark on the “observing your AI” side.
Collaborating — bringing it all to where your team already works: Assistant in Slack (and coming to Teams), on mobile, on the desktop, and via API/MCP, plus Assistant Search across your Grafana artifacts.

If you’re new to the Grafana stack itself, start with my open source observability video for the non-AI foundations that all of this builds on.

Resources

My notes on Grafana Assistant
My notes on GCX
My notes on k6
o11ybench.ai — the open leaderboard for AI agents doing observability work

AI Week Day 4: Evaluating (Grafana AI Week — Thursday)

Thu, 30 Jul 2026 09:00:00 +0100

Day four of AI Week is all about evaluating — because building an agent is the easy part; trusting it is where things get hard. The headline: Grafana Agent Observability is now generally available — the tool Grafana built for itself while scaling Assistant, once it grew too complex to evaluate with unit tests. Instrument your agent with the SDK and your conversations, tool calls, traces, latency, token use, and cost all become observable in one place; add evaluators (LLM judges plus deterministic checks) to score quality on live traffic, feed low-scoring conversations into your test suite, and gate CI on experiments comparing a candidate against a baseline. And because evaluator metrics land in Prometheus, you can get paged when a bad prompt change tanks quality. The fun bonus: k6 agentic testing — describe a browser test in plain language and an agent embedded in k6 drives the browser, no script required (they proved it by getting it to beat Wordle).

Transcript

It’s day four of Grafana Labs’ AI Week, and Thursday is all about evaluating — because building an agent is the easy part. Trusting it is where things get hard.

The big news: Grafana Agent Observability is now generally available. This is the tool we built for ourselves while scaling Grafana Assistant, when it grew too complex to evaluate with unit tests. Now it’s available to everyone on Grafana Cloud.

Here’s how it works: instrument your agent with the SDK, and your conversations, tool calls, traces, latency, token use, and cost all become observable in one place. Then add evaluators — LLM judges plus deterministic checks — to score quality on live traffic. Low-scoring conversations feed your test suite, and experiments let you compare a candidate against a baseline before you ship — you can even make it a CI gate. And because evaluator metrics land in Prometheus, you can get paged if a bad prompt change tanks quality in production.

But my favorite part of today? k6 agentic testing. Describe a browser test in plain natural language, and an agent embedded in k6 drives the browser for you — no script at all. And to prove it, we gave it one job: beat Wordle. Go watch that video. I’ll wait.

Everything’s at grafana.ai — see you tomorrow for the finale: using AI where your team already works.

Investigations Arena: how we test Assistant investigations

Thu, 30 Jul 2026 09:00:00 +0100

How do you test an AI agent whose whole job is to investigate incidents — when every investigation is different and “correct” isn’t a single deterministic answer? In this interview I sit down with Will, a software engineer at Grafana who’s spent the last year focused on AI: the Grafana Assistant, investigations, and memory features.

The story here is how the team got from Investigations V1 to a much better V2. V1 hit a ceiling, and the demo environment they’d been testing against turned out to be the problem — the cases were too easy and too far from the messy reality of a production cluster. So they built an arena: think competitive bot programming, but the “game” is investigating real incidents in Grafana’s own production environment. Different AI agents, each with its own strategy and tools, all take a swing at the same incident, and an LLM-as-judge scores their answers against a human-verified root cause.

What’s interesting is that the score wasn’t even the most valuable output at first — the insights were. Watching which strategies worked, which tools returned so much noise that they poisoned the context, and which agents wandered off is what let the team optimize bit by bit into the V2 release. The arena also compares more than models: different harnesses running Sonnet vs. Opus, Claude Code with the Grafana MCP, even just hitting the public API directly with curl.

A couple of my favorite moments: Will explains the guardrails that stop agents from “cheating” (they can only look at data from within the incident’s time window, so they can’t peek at the fix commit that resolved it) — and how the models mostly play fair because they treat it like a game. And there’s a lovely example of emergent creativity: an agent reached for tools the team hadn’t even considered, they turned out to be genuinely useful for investigation, and those tools got promoted into the real harness. Will’s hope for the future is to open this up — an OliBench-style shared benchmark where the community can throw their own investigation agents into the arena and raise quality across the board.

Transcript

Will: Hey, my name is Will. I’m a software engineer at Grafana. I’ve been there for three years, and for the last year I’ve been focusing on AI — the Grafana Assistant, investigations, and memory features.

Nicole: And what’s the latest thing you’ve been working on?

Will: So we developed Investigations V1, and we saw that we hit a ceiling at some point. We came up with a different approach, but this time we needed a good evaluation environment — the demo environment wasn’t enough, because those cases were too easy and too far from the reality of a production environment. So we decided to build an arena.

It’s a little bit like the idea of competitive bot programming, where bots fight against each other on specific tasks or games. In this case, the topic was investigations in our production cluster — real incidents. We started building bots that had different strategies and different tools; these were AI agents. Comparing a diversity of strategies let us surface the best approaches, and from there we could iterate, bring in more test cases from more incidents, and reach higher and higher quality.

The idea is that the bots try to solve the same case, and we use an LLM as a judge to compare the different approaches. We have different criteria, because for each incident we have a root cause that’s been verified by a human. That gives us our ground truth, and the LLM-as-judge compares that ground truth against the answers the agents came up with.

The important part at first wasn’t even the score — it was the insights. This tool was better with this strategy; this one had noise in the results that poisoned the context; the results returned from this specific tool were just too much. So we managed to really optimize bit by bit and come up with Investigations V2, which we just released. It’s a massive improvement compared to our V1 approach.

Nicole: So is it comparing just different models for investigations, or is it also comparing things like Claude Code and the Grafana MCP versus Assistant?

Will: Both, actually. We have different versions of our harness for investigations that we run with Sonnet and with Opus, but we also use the Claude Code harness with the Grafana MCP — and even just using curl to directly hit our public API.

Nicole: Do the models ever cheat?

Will: So far it’s been quite good. We have guardrails. Because we investigate past incidents, we need to make sure the LLM only looks at a specific time window — only from when the incident started — otherwise it would be able to find the pull request that was used to solve the incident. But they’re pretty good at following the rules; they take it as a game a little bit. We even have an agent that checks afterward, with the judge, to make sure they followed every rule.

It’s funny, actually — we had a case using Claude Code where it started to use some tools we hadn’t thought about in the first place, and they turned out to be very useful for investigation. We’d never considered them, so we brought them into our investigation harness and it helped a lot. Giving the agents freedom and creativity in this kind of environment is really useful. That diversity sponsors new ideas that we can put into our own work.

Nicole: Cool. Do you have any plans to publish this and share it so that other people can do it too — kind of like OliBench?

Will: Yeah, I think, why not? I think we can go this way too. I’m sure it would be useful for other agents. And it would be very nice if we could compare agents from other people and other companies — maybe with different harnesses, maybe even different tools — because right now it’s mostly about our MCP, our CLI, and our investigation harness. But if the community comes up with their own investigation agents — and I think I’ve seen a few — it would be fun to put them in the arena and help not just our product, but any product improve in quality.

Resources

My notes on Grafana Assistant

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Thu, 30 Jul 2026 09:00:00 +0900

This week I spoke at KubeCon + CloudNativeCon Japan 2026 in Yokohama, Japan. I gave a talk called The Great Doubt: What Building an AI Agent Taught Us About Trust. Here’s the abstract:

When we started building an AI assistant for observability, we thought the hard part would be making it smart. We were wrong. The hard part was knowing when to trust it.

AI agents hallucinate, forget context, and confidently give wrong answers. Traditional testing doesn’t catch these failures. What’s needed is evaluation grounded in systematic doubt.

This talk shares lessons from shipping an AI agent to production: how to build a golden dataset of use cases you must get right, how to use LLM-as-judge when there’s no ground truth, and how to use OpenTelemetry traces to debug eval failures. You’ll also hear what’s still unsolved: evaluating multi-agent handoffs and closing the feedback loop between what users ask and what your evals cover.

Kyoto School philosopher Nishitani Keiji called this “The Great Doubt” (大疑): questioning every assumption until only what survives is real. For AI agents, that’s not philosophy. It’s the job.

The talk walks through the layers of doubt we worked through while building Grafana Assistant: doubting the agent itself, then doubting our tests, then the LLM-as-judge, then our own scores — and finally doubting whether the doubt was even justified. Each layer needed a different tool, from a golden dataset run in a fully synthetic observability environment, through online evals on live traffic, to per-conversation OpenTelemetry traces that link every eval score back to the exact LLM and tool calls that produced it.

You can find the slides at nicole.to/doubtslides, and you can watch the full talk below:

Resources

Grafana Assistant (generally available for Grafana OSS, Grafana Cloud, and Grafana Play):

Assistant Investigations (Grafana Cloud):

Docs

Agent Observability (Grafana Cloud):

Docs

o11y-bench (OSS):

Site

Investigations Arena:

Video

My notes on Grafana Assistant
My Context Horizon piece Unlearnings from building Grafana Assistant
The Way of Zen by Alan W. Watts
Zen Mind, Beginner’s Mind by Shunryu Suzuki
Religion and Nothingness, Volume 1 by Nishitani Keiji
Philosophers of Nothingness: An Essay on the Kyoto School by James W. Heisig

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Wed, 29 Jul 2026 09:00:00 +0100

Day three of AI Week is about operations and maintenance — because once code reaches production, the operational work is still surprisingly manual: someone’s watching dashboards, deciding whether an error spike is real, and writing up what happened, every single day. Two Grafana Assistant features went GA to take that off your plate. Assistant Automations handle recurring work — save a prompt and run it on a schedule (like a morning brief on your service’s health), each run keeping its own history and able to notify you in Slack. Assistant Investigations is for when something actually breaks: it digs across metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report you can follow along with in a Workspace. You’re offloading vigilance, not accountability — the evidence is always there to review, and you still make the call.

Transcript

It’s day three of AI Week, and Wednesday is all about operations and maintenance. Because here’s the thing: once your code reaches production, the operational work is still surprisingly manual. Someone’s watching dashboards, someone’s deciding whether that error spike is real, and someone’s writing up what happened — every single day.

Today, two features of Grafana Assistant are now generally available to take that burden off your plate: Investigations and Automations.

Assistant Automations handle the recurring work of operating a system. Save an Assistant prompt and run it on a schedule — like a morning brief that summarizes the operational health of your service over the last 24 hours. Every run keeps its own conversation and history, and it can notify you in Slack when it’s done or needs approval.

Assistant Investigations is for when something actually breaks. Investigations digs into issues across your metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report. You can follow along in a Workspace, see what’s been ruled out, and add guidance while it works.

You’re offloading vigilance, not accountability — the evidence and reasoning are always there to review, and you still make the call.

Everything’s at grafana.ai — see you tomorrow for day four: evaluating your agents.

AI Week Day 2: Building (Grafana AI Week — Tuesday)

Tue, 28 Jul 2026 09:00:00 +0100

Day two of AI Week is all about building — the AI tools that help you write software with confidence. Two big announcements went generally available: gcx (the Grafana CLI) and the Grafana MCP server. Both connect your coding agents — Claude Code, Cursor, and friends — to your Grafana telemetry. The MCP server gives agents opinionated, ready-made tools for the common cases (querying metrics and logs, reading and updating dashboards); gcx is the more flexible, less opinionated option, built for agents from the ground up with read-only guardrails and its own bundle of agent skills. Both work for OSS and Grafana Cloud. The point: when an agent writes your code, it shouldn’t be guessing — it should be looking at how your system actually behaves in production. That’s telemetry-driven development.

Transcript

It’s day two of Grafana Labs’ AI Week, and Tuesday is all about building — the AI tools that help you build software with confidence.

Today we have two big announcements: gcx and the Grafana MCP server are both now generally available.

Both of them connect your coding agents — think Claude Code or Cursor — to your Grafana telemetry. So what’s the difference?

The Grafana MCP server gives your agent opinionated, ready-made tools for the most common use cases: querying your metrics and logs, reading and updating dashboards, that kind of thing. You can run it yourself, or use the hosted MCP server with your Grafana Cloud stack — so yes, it works for both OSS and Grafana Cloud.

gcx is the Grafana CLI, and it’s the more flexible, less opinionated option. Agents can explore it and build their own workflows, and it was designed for agents from the ground up, with read-only guardrails so your agent can’t touch anything you didn’t approve. It works with Cloud stacks and self-managed instances, and it even ships with its own bundle of agent skills.

Why does this matter? Because when an agent writes your code, it shouldn’t be guessing — it should be looking at how your system actually behaves in production. That’s telemetry-driven development.

Head to grafana.ai for everything we’re shipping this week, and I’ll see you tomorrow for day three.

AI Week Day 1: Planning (Grafana AI Week — Monday)

Mon, 27 Jul 2026 09:00:00 +0100

It’s day one of Grafana Labs’ AI Week, and Monday is all about planning — getting the clarity you need on what’s going on in your system so you can figure out what to build. Kicking things off is Workspace for Grafana Assistant: a full-page view of Assistant so your conversations and generated graphs aren’t crammed into the little right-hand sidebar. It’s a much less cluttered place to run investigations and to be agent-first. On top of that, Assistant is now easier to try — enable it in Grafana 13 for OSS without linking a Grafana Cloud account, or try it with no signup or download at all on Grafana Play.

Transcript

Hi everyone! I’m Nicole van der Hoeven and I’m really excited to welcome you to AI Week. Today is Monday, which is planning day, where we give you the tools you need to get clarity on what even is going on in your system so that you can figure out what to build.

Now we’re kicking things off with a feature called Workspace for Grafana Assistant. Now if you’ve used Assistant before, you’ll know that it comes up on the little sidebar on the right side of Grafana. Well, that can start to get cramped if you have multiple conversations or if you’ve asked Assistant to generate multiple graphs for you. Well, we’ve heard you, and now we’re giving you a full page view of Assistant. So you’ll have all of your conversations on the left, and this is also an excellent place to run investigations from. You can also zoom in on any of those graphs, and it’s just a much less cluttered view if you want to be agent-first.

Speaking of Assistant, the second thing I want to tell you is that you know, Assistant was already available everywhere. But now, Assistant is available… in more places than everywhere? And also more easily. So Assistant was already available for OSS, but you had to link your Grafana Cloud account, which was not so handy if you wanted to try it out. But not anymore. Now if you download Grafana 13 and enable Assistant, then you’ll just be able to go and try it out and chat with Assistant without doing anything else. It’s still not free or open source, but it IS a lot easier to try it out.

And if you don’t want to download anything, that’s what Grafana Play is for. Grafana Assistant is now available to try out on Grafana Play without any signup or any download required. Try it out at play.grafana.org. I really love seeing it there. Tune in tomorrow where we’re going to talk about everything we’re announcing to help you start to build.

An Overview of AI Evaluation (The Context Window #05)

Fri, 26 Jun 2026 18:00:00 +0100

How do you know if an AI agent is actually any good — and whether the change you just made to it made it better or worse? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Yas Ekinci, an engineer on Grafana’s AI team who has spent the last several months focused almost entirely on evaluation. We talk about why trusting AI is so hard (non-deterministic models, oversized PRs, and the shift from writing code to reviewing it), the difference between online and offline evals, how we actually test Grafana Assistant, and o11y-bench, the open observability benchmark we announced at GrafanaCON. We recorded this one ahead of Yas’s two-month leave — so consider it a deep dive banked for the road.

The “final answer problem”

A recurring theme — and one close to my testing heart — is what I call the final answer problem. As a tester I’m used to looking at an outcome and asking “did this pass or fail?” But with AI, a plausible-looking answer isn’t enough. If the process was wrong, you can’t trust the outcome even when it looks right — and a plausible-but-wrong answer is the dangerous case, because nothing on the surface tells you it’s wrong. Yas’s example: a broken Tempo tool that returned no spans, while the agent still answered convincingly by guessing from metadata. Plausible ≠ correct.

This is also why the bottleneck has moved. The cost of writing code has collapsed, so the burden now sits on review. Developers, as I like to put it, are becoming shepherds of intent — less about typing the code, more about making sure the intent is clear and faithfully implemented.

What evals are, and online vs offline

An eval is simply a test of whether an AI agent behaves the way you want — from “who are you?” → “I’m Grafana Assistant” all the way up to multi-step querying and dashboarding. The point is to know with confidence whether a change made the agent better, which is what lets you improve it consistently over time.

There are two complementary flavours, and they map neatly onto testing concepts we already have:

Online evals — scoring a sample of real production conversations, usually with an LLM-as-a-judge. Great as a pulse check (it’ll catch a quality drop when a model provider silently swaps models), but it only tests the deployed variant and is hard to verify. Closest analogy: traditional observability / RUM.
Offline evals — a “golden set” of tasks run during development and CI against a controlled, reproducible environment. You can test every change, prompt, and implementation, and drill into specific skills (PromQL vs LogQL, dashboarding). Closest analogy: integration testing.

How we test Grafana Assistant

Assistant evals run a real agent-plus-tools harness against a real-ish environment (a Grafana with data sources, alerts, and dashboards), then grade the full transcript with “graders” or “verifiers.” The hard tension is that the environment has to be real and reproducible at the same time. Because Assistant is a front-end plugin that runs some tools in the browser, the team uses Playwright to replicate real browser behaviour — which is where I made my pitch for k6’s browser module (now approaching Playwright parity) as a dogfooding option. The internal eval tool is a CLI called LLM Spec, now on its third major iteration after the lessons from o11y-bench.

o11y-bench

o11y-bench is our public benchmark, announced at GrafanaCON, with six task categories: dashboarding, investigation, logs, metrics, and traces. A few things I think make it genuinely interesting:

It’s built on the Harbor framework (also used by terminal-bench), which separates harness from environment from tasks — so you can swap harnesses and compare, say, gcx + Claude Code against the Grafana MCP.
The leaderboard includes open models (Qwen 3.6, Kimi), not just the frontier labs.
It distinguishes pass@3 (got it right at least once in three runs — rough capability) from pass^3 (got it right all three times — reliability). The 10–20% gaps between the two are the whole reliability story in a single number.
It runs against a synthetic environment (Prometheus, Loki, Tempo in Docker) with generated data, so the correct answer is known in advance — and it uses fact-based rubrics, where the grader runs its own known-good queries against the environment and checks the agent’s answer against them. That’s how you catch the subtle, plausible-but-wrong failures.

The internal eval set is a superset that adds scenarios drawn from real (private) conversations and Assistant-specific behaviour — like Slack’s not-quite-markdown formatting — while o11y-bench stays generic and harness-independent, grading on outcome rather than method.

The evaluation loop

Yas’s blog post (the first engineering post on Grafana’s new community Unprompted blog) and our GrafanaCON talk both center on the evaluation loop: measure → make changes (new tools, prompt tweaks, bug fixes) → measure again. The crucial bit is that the loop can be driven by a coding agent — via an LLM Spec Improve skill — because you can’t unit-test natural-language “programming” the way you test code. There’s no clean relationship between the instructions you write and what the model does, so the only way to know the impact of a change is to run the loop.

Where it’s going

We closed on open questions and wishes: higher environment fidelity, more and harder tasks (very likely an o11y-bench 2.0), and getting more contributors — including PMs writing evals to capture desired product behaviour. I floated a k6-extension-style community model for evaluators, and using o11y-bench to support bring-your-own-model, so users can see exactly where a smaller or local model falls short instead of assuming Assistant is broken. Yas’s industry hope is the neat one: if labs tune their models against o11y-bench, observability and Grafana performance improve across the board — free fine-tuning out of the box.

Resources

Transcript

Nicole: Hey everyone. Welcome to another episode of The Context Window. This is a semi-regular podcast — we try to make it every month — and it’s all about the things going on with AI at Grafana. Usually this is live, and you might think looking at this that it’s live, but it’s actually prerecorded, because Yas has a busy schedule ahead and I wanted to get him on here before he left. So hi, I’m Nicole van der Hoeven, a senior developer advocate at Grafana Labs, and I’m assigned to the AI team these days. And one of the people on the AI team is — this person over here. How do you say your last name, actually?

Yas: Oh, it’s Ekinci.

Nicole: Ekinci. I was totally confused because I just found out you speak Dutch. Did not know that.

Yas: I’m based in Belgium, and I was like, “Nicole van der Hoeven” — okay, that sounds familiar enough to me.

Nicole: Yeah, it’s the name of my ex-husband, but it’s okay, we still like him. So tell us what you do on the AI team.

Yas: I’m an engineer on the AI team — I’m getting to five years at Grafana next week. When I joined, this was pre-AI; we had the machine learning team building predictive models, forecasting, that kind of stuff. As the AI wave came, we all got absorbed into this newer, larger AI department. It’s been quite varied — the assistant, the AI agent inside Grafana — but especially the latter half of this year I’ve been more and more focused specifically on evals: how do we figure out the quality and reliability of these things we’re deploying?

AI is a very useful tool. It has utility — it’s not like NFTs. We use it day to day. My job has changed drastically over the last two years. But you still notice it needs guidance and sometimes has reliability issues. The trend is that the guidance has come down — a year or two ago I’d steer an agent 50, 60, 70 times in a session; now it’s more like five or ten. But you still need to do it. If you just blindly vibe-code your way out of this, you’re not going to end up in a good place.

Nicole: It moves so quickly. Whenever someone tells me they tried AI and it didn’t do what they wanted, I ask, “Well, when was that?” Because if it was six months ago — try it again. In this episode we’ll talk all about evals, but first, some announcements. Assistant is now available for OSS and self-hosted Grafana, and I made a video showing how to do that. There’s also a new Medium blog for Grafana Labs called Unprompted — a community-focused blog — and Yas wrote the first engineering post, about building an evaluation loop. And we’ve started AI weekly videos: the AI team has a weekly meeting, Mat Ryer had the idea to publish excerpts, usually the demo part. Just be warned we demo things that may never make it to production — you’re getting it raw, warts and all.

Yas: I need to make sure I don’t wear my PJs anymore for the AI weekly.

Nicole: A bit of quick AI news: Warp is now open source. There are new models — GPT-5.5, Gemini 3.1 Ultra with a two-million-token context window, Anthropic’s Claude Mythos security preview. And there’s a great article by Matt Shumer called “Something Big Is Happening,” where he says, “I am no longer needed for the actual technical work of my job” — which hits hard — and advises leaning into what’s hardest to replace.

Yas: It’s still that guidance thing — you need a human in the loop to guide it. But if you ask me how much code I’ve manually written lately, the actual technical piece is very close to zero. We’ll talk about how that’s different for evals, because evals don’t work as well as coding does there.

Nicole: So why is it so hard to trust AI? Why don’t we just YOLO it and accept everything?

Yas: A mix of things. Everybody knows these models are non-deterministic and not perfect — these aren’t new problems; underneath it’s all machine learning, numbers and math and stats. Every time you talk to it you might get a slightly different response. The second part is it writes a lot of code. You can 10x your output — send ten agents at ten different features and launch all those PRs — but then someone needs to review them, and the PRs are typically big and over-engineered. One thing I always do at the end of a piece of work is ask it to find a simpler way and simplify. Bigger PRs plus so many changes makes it hard to keep a mental model of the whole thing. And interestingly, I’ll use AI to help review a PR — which gets you back to the same problem: if you use AI to review the AI, how much can you trust that?

Nicole: I like the phrase that developers are becoming shepherds of intent — we’re not writing the code so much as making sure the intent is clear and implemented. And this is the difference between process and outcome — what I call the final answer problem. I come from performance testing; I’m used to looking at the outcome and asking did it pass or fail. But now, even if the outcome looks right, if the process was wrong I don’t trust it. And the reverse too: right process, wrong outcome, something else is broken.

Yas: Tests are interesting here, because most of those tests are also AI-generated — and the AI doesn’t shy away from just changing the test to make its branch pass. For any branch I review, I still look at all the changes; I won’t blindly rubber-stamp. That’s probably the biggest bottleneck right now.

Nicole: One approach to trusting AI is human evaluation — thumbs up/thumbs down, or something like LMArena where you compare two answers blind. Useful, but not scalable; you can’t run it in CI.

Yas: On the eval side they’re all complementary. So, what is an eval? It’s testing whether the AI agent behaves like you want. A simple one: ask “who are you?” and expect “I’m Grafana Assistant.” There are much more complicated ones around querying and dashboarding. At its core, an eval is how you figure out if the agent is any good — and the reason we do it is so that when we make changes, we know if it’s better. Answer that with confidence and you have a path to improve consistently over time.

We split into online and offline. Online — we recently launched AI observability for agents — is about measuring how the deployed agent does against real production conversations from real users. Valuable, because you get a score based on real behaviour, but hard to verify; we typically use an LLM as a judge, so it’s not as robust. Offline is used more during development, in CI — much more controlled. You have a golden set of evals covering the behaviour you expect in production, and because you always test the same set, you can see whether changes make it better, and dig into PromQL vs LogQL, or dashboarding. Online is a great pulse check — it can catch quality dropping when Anthropic swaps a model — but it only tests the deployed variant. Offline can test all your changes, your implementation, and all your prompts.

Nicole: It took me a while to translate these, because the testing industry already has this — we call it pre-prod and production testing. Pre-prod is everything from unit to system integration testing in dev and staging; you can really hammer the thing. Production you have to be more careful — synthetic monitoring, real user monitoring — same idea with an AI flavour.

Yas: Definitely. Online is much more like traditional observability — you’re observing a system that happens to be powered by AI, turning natural language into a number you track over time and alert on. Offline is closer to an integration test: you give it a task, it runs the whole agent against a real environment, you get something out, and then separately you grade that outcome.

Nicole: We did an AI observability episode without you — you were at GrafanaCON — so I’ll link that. Let’s talk more about offline evals, since we haven’t covered those.

Yas: Offline is a bit of a beast. There’s a good Anthropic post called “Demystifying evals.” Back when we had simple single-turn LLM calls — incident summary was one of the first — the prompt was “here’s the context of the incident, summarize this,” and grading checked readability, completeness, appropriate length. Easy enough to set up. Agents get more complicated, because every time you run a task the agent can take very different trajectories — five steps one time, seven the next. The agent plus its tools is roughly the harness; it runs against an environment — a real Grafana with data sources, a certain alert and dashboard state — and keeps interacting using the tools until it’s done. Then we take the whole transcript and grade it with graders, sometimes called verifiers, to verify it actually got the task right.

Nicole: So how do you do it for Assistant, given how broad that environment is?

Yas: We have the public benchmark we released recently and our internal evals — same approach, different number of tasks and some different data sources. We want an environment as real as possible, but with the tension that we also need to control it so a benchmark is reproducible. If two people run it and get different scores, which one was right? We started, like everybody, with manual tests — listing data sources to confirm a tool is wired up, some CPU questions, showing logs — with a human as the judge. That’s fine at four or five scenarios; you can’t do it as a human at 500. Assistant is a front-end plugin — some tools run in your browser, which is good for auth and security but makes testing hard — so we use Playwright to replicate real behaviour and pull the telemetry out. The internal eval tool is a CLI we call LLM Spec, and it can run against cloud instances or a local controlled environment.

Nicole: Before the environment — did you consider using k6 instead of Playwright?

Yas: Interesting — yes and no. k6 is also a tool the assistant needs to use, but our goal isn’t to load test; it’s to hook into the plugin and get the messages and telemetry out, so we have a Playwright harness.

Nicole: There’s actually a browser module for k6 going towards feature parity with Playwright.

Yas: I did not know that. Might be interesting.

Nicole: Just dogfood all the things. I’d love to work with you on that. Okay, the environment.

Yas: We test against cloud instances — for incidents especially, it’s a lot of data. And we run LLM Spec against a local controlled environment, which is quite similar to the open benchmark we published.

Nicole: You’ve mentioned o11y-bench — the benchmark announced at the last GrafanaCON. That was fast.

Yas: o11y-bench has tasks across six categories — dashboarding, investigation, logs, metrics, traces. We wanted to bring our internal approaches outward, and it was a learning opportunity: we used a framework called Harbor, also used by the makers of terminal-bench. The nice thing is it separates harness from environment from tasks — so the default harness is a simple system prompt plus the Grafana MCP, but the folks working on GCX run Claude Code plus GCX against o11y-bench and compare GCX with MCP. There’s a leaderboard with a bunch of models, including open ones like Qwen 3.6 and Kimi.

Nicole: They’re getting good. Qwen 3.6 is awesome — not top five, but for an open-weight model, impressive.

Yas: And this is the interesting part: sort on pass@3 and it looks different from pass^3. pass@3 runs each task three times and passes if it gets it right one out of three — rough capability. pass^3 means you need three out of three on every task — reliably solving it, not just getting lucky once. You see 10–20% differences between the two scores.

Nicole: That’s awesome. And this is public, right?

Yas: Yep — the repo’s on GitHub with all the tasks. We run o11y-bench against a synthetic environment — a Prometheus, Loki, and Tempo in Docker — and we generate the data, so every run uses the same data and we know what the answer should be before we even run it. We also do fact-based rubrics: separately from what the agent does, the grading side runs its own known-good queries against the environment and checks whether the agent’s answer matches. That matters because these agents fail in subtle ways. We once made a change to the Tempo tool that broke the spans — the agent got no spans but still answered, fairly realistically, from metadata. Without testing for it you’d never catch it; the text looked plausible.

Nicole: That’s exactly process versus outcome. If the answer were negative two seconds it’s obviously wrong, but a plausible answer is the dangerous one.

Yas: Exactly, and that happens a lot.

Nicole: An early Assistant once told me it created a dashboard, and it didn’t look right — turned out it couldn’t find the data source so it just used dummy data.

Yas: Same thing happened to me locally — I broke all the tools, and it still came up with tool calls it couldn’t execute because of rough instructions in the system prompt.

Nicole: So what’s missing in o11y-bench that means you still need an internal one?

Yas: Two levels. First, some scenarios are based on real conversations or our own internal environments — anything involving private data can’t go public. Second, there are very Assistant-specific things we want to test — Slack, for instance, has its own version of markdown that looks like markdown but isn’t, and early Slack Assistant struggled to write it consistently, so we wrote tests for that. The internal set is a superset; the public set is still part of it. The public set is generic observability tasks — asking questions about your data, dashboarding, basic Grafana API checks — and because the harness is independent, we can test GCX, MCP, our assistant, or any other agent against it, grading on the outcome rather than how it was solved.

Nicole: We talked about your blog post — let’s define the evaluation loop.

Yas: At its core: measure the current state, see where it works and where it doesn’t, make changes — a new tool, tool instructions, the system prompt, bug fixes — then measure again, and again. The key is that loop can be driven by a coding agent, not us manually. If your evals are robust enough to pass a clean feedback signal back, the agent can keep improving until it hits 100%, and then you make harder tasks or increase coverage. At scale it’s nearly impossible for one human to understand the whole system and make targeted improvements, because an agent isn’t programmed as code — it’s programmed with natural language, and there’s no clear relationship between the instructions and what the model does. The only way to probe that is to run a bunch of these loops.

Nicole: Is the loop equally implemented for o11y-bench and LLM Spec?

Yas: It’s implemented internally. o11y-bench is really just the measurement part — we’ve used it for GCX and MCP, but more manually. In LLM Spec we have an LLM Spec Improve skill — you pick the agent, and it runs measure-improve-measure-improve.

Nicole: Where are you going next — no promises?

Yas: I’ve done a lot on the harnesses — the Playwright stuff, and there was an LLM Spec 1.0, then a 2.0 revamp for speed and robustness, then o11y-bench, and the learnings from that went into what I’d call LLM Spec 3.0. The two big challenges now are environment fidelity — more an issue for internal evals — and the number and quality of tasks: how do we cover more of Grafana? We weren’t sure o11y-bench would be a recurring thing, but there’s a lot of interest, so we’ll probably do an o11y-bench 2.0. And I want more people contributing — I was talking at the PM all-hands about getting product managers to write evals, because the tasks capture desired behaviour: when someone asks about my product, how should the agent ideally respond?

Nicole: What I’d love is something like the k6 extension model applied to evaluators. When Grafana acquired k6 — I was on k6 — k6 had extensions and modules, and we got a lot of cool community extensions. I think evaluators would be the same. After my Madrid talk, all the questions about the evaluator section were “can we add this, can we add that, why isn’t this in the template?” — and it could be.

Yas: It would be nice to do that across both o11y-bench and the internal ones — there are open questions on making them work together; internally we’re in Go, o11y-bench is on Harbor in Python, but the tasks and most verifiers are language-independent by design. A fourth thing is getting to a better golden set over time — we haven’t yet automated going from real AI-observability conversations to verifiable offline tasks. As the product changes, your tasks might not stay relevant — imagine k6 didn’t exist and now people ask Assistant about k6. Bridging online and offline in a more automated way would be the fourth big thing.

Nicole: I also have a wish for o11y-bench beyond being a benchmark — with no internal knowledge and no promises: it could be a step toward making Assistant bring-your-own-model. One barrier is that people try Assistant with a tiny local model and conclude it doesn’t work, when really you need certain models. A standardized way to measure that and show users where their model falls short — versus where the latest Opus sits on the leaderboard — lets them make an informed decision.

Yas: I’d go beyond Assistant — we’ve got GCX and MCP too. If you want to use your own models with them, the benchmark should reflect, for observability tasks, which are the best models and the best open models. It gets to your point: people can make a more informed decision.

Nicole: Where are we still missing things as an industry, not just as a company?

Yas: Just recently a new benchmark called DeepSWE was released, and it’s a bit controversial because it flips the ranking from the most popular software-engineering benchmark — where Opus was always leading, here GPT-5.5 leads by a margin. So there’s something to be said about the reliability of the benchmarks themselves, since labs test and sometimes tune their models against them. That might actually be good for o11y-bench — I’d want a lab to tune against it, so we get free fine-tuning out of the box: as newer models use o11y-bench to tune on observability tasks, they get better at using Grafana and solving observability problems.

Nicole: Thank you — we just hit time. It was really informative, and I’m glad we both found time; this got rescheduled for various reasons. A great overview of all the evals we’re doing.

Yas: Thanks — it’s nice to step out of the day-to-day and talk about this stuff.

Nicole: And if you’re watching, leave any eval questions in the comments and I’ll bug Yas to answer them. Thanks everyone for watching.

Yas: Bye.

OpenTelemetry's Graduation: What It Took (Grafana ❤️🔥 OpenTelemetry Community Call #8)

Mon, 22 Jun 2026 16:00:00 +0100

OpenTelemetry graduated from the CNCF — so for this community call I gathered a panel of people who were in the room for it to talk about what graduation actually means, what it took to get there, and where the project goes next. We covered the roughly two-year road to graduation, why semantic conventions are so genuinely hard, the state of the signals (traces, metrics, logs — and the newer profiles), AI’s double-edged impact on the project, and the most practical ways for new contributors to get involved.

What we covered

🎓 What graduation actually took

Graduation from the CNCF took about two years, with OTel evaluated on traces, metrics, and logs across the major server-side languages. Profiling, mobile, and other newer signals were explicitly out of scope for the graduation criteria. One of the most concrete pieces of CNCF feedback: stop camping on 0.x versions for components that are actually production-ready. The fix is to mark maintained, stable packages as 1.0 (reserving 2.0 for future breaking changes) — which unblocks the enterprises that simply aren’t allowed to ship anything still labeled “beta.”

📚 Documentation that’s actually findable

A recurring theme was that the documentation bar isn’t just “does it exist” — it’s “can someone find it.” Collector component docs are spread across 100+ GitHub repos, which makes discovery painful. The stopgap is an auto-updated component list with metadata on the OpenTelemetry website that links out to the relevant READMEs; the longer-term answer is the Ecosystem Explorer project, which aims to unify instrumentation details, versioning, and component metadata in one place.

🏷️ Why semantic conventions are so hard

Semantic conventions define normalized attribute names so that the same event — say, an HTTP request — looks identical no matter the language or library that produced it. The naming debates are slow for real reasons: the database SIG took over a year to stabilize a single set of attributes. The process is propose names → check them against all the major databases → build a proof-of-concept in three or more languages → iterate. A name that’s obvious for one database may not even exist as a concept in another. (OTel itself is the product of merging OpenTracing, OpenCensus, and Elastic Common Schema — and yes, the name was chosen specifically because it’s boring and self-describing. Everything is named what it does.)

🧳 Signals, baggage, and stability

All three original signals — traces, metrics, and logs — are now stable, and that includes baggage: a context-propagation mechanism that carries a small “bag” of attributes (trace ID, span ID, and more) across network hops. It was originally built for tracing but designed to be reusable for other cross-cutting concerns like security and feature flags, and it sat largely unused while the rest of OTel was being finished. Profiles, meanwhile, hit alpha at KubeCon in March and are treated as a bonus on top of the graduation baseline rather than part of it.

🤖 AI’s impact on OpenTelemetry — the good and the painful

The good: the explosion of GenAI and agentic systems is creating real demand for OTel-style observability. The Gen AI SIG is growing fast to standardize telemetry for AI systems, and there’s a whole crop of AI-observability startups building on OpenTelemetry. Maintainers are also using AI coding tools to move faster — the Weaver tool is being developed to constrain the problem space so AI-generated instrumentation code is more accurate, with the goal of going from spec to implementation with coding tools.

The painful: a massive influx of low-quality, AI-generated PRs across all of open source. Some are genuinely malicious (bot accounts farming credibility for supply-chain attacks); others are just thoughtless. One proposed mitigation is to cap the number of open PRs per contributor without merge rights to something like three to five.

🚀 Post-graduation priorities and how to get involved

The post-graduation backlog (“postgrad app”) focuses on stability cleanup, easier deployment, and self-observability. The deployment north star is something close to apt-get install opentelemetry that auto-instruments everything — including outside Kubernetes. New stable YAML-based collector config lets operators manage all languages from a single file, and the Blueprint project pairs a specific challenge with OTel guidelines and concrete implementation steps.

If you want in, the panel was refreshingly practical about the fastest path to approver/maintainer status:

Review other people’s PRs in your subject area.
Ask maintainers “what can I take off your plate?” rather than only filing your own issues.
Quality over quantity — AI-heavy contributions you can’t explain won’t earn status.
Localization is one of the easiest entry points; around ten language teams are active and docs translation is always welcome.

We also went around the panel on how everyone got started — from debugging giant distributed systems, to a beginner-questions live stream, to Elastic Common Schema work, to copy-editing PRs as a former librarian. The throughline: pick a small, real thing, do it well, and document your learning journey as you go.

Resources

Thanks to the panel — Ted Young, Marylia Gutierrez, Imma Valls, Tiffany Hrabusa, and Tiffany Jernigan — for joining, and to everyone who showed up live with questions.

CLI and MCP Basics for AI Agents (The Context Window #04)

Wed, 03 Jun 2026 18:00:00 +0100

MCP vs CLI: which one should your AI agent actually use? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and three of the engineers and PMs behind Grafana’s agent tooling: Ben Sully (who wrote the Grafana MCP server), and Ward Bekker and Dafydd Thomas from the team building gcx. We get into the two main ways an AI agent can reach the outside world — the Model Context Protocol (MCP) and good old command-line interfaces — and look at when each one makes sense, the tradeoffs around context windows, tool sprawl, token cost, and reliability. We walk through Grafana’s cloud and open-source MCP servers along with gcx, the Grafana Cloud CLI built for agents, do a live side-by-side demo of both creating an SLO, and talk about how we benchmark them against each other with o11y-bench.

Timestamps

00:00:00 — Introductions
00:02:49 — The last month in AI news
00:10:19 — What is Grafana Assistant?
00:12:30 — What does it mean for Assistant to use tools?
00:13:30 — What is MCP?
00:16:56 — What is a CLI in an AI context?
00:19:17 — What are skills?
00:24:37 — Can you use a custom MCP server with Grafana Assistant?
00:25:55 — Why team MCP?
00:28:22 — The context-window cost of MCP tool definitions
00:31:50 — Tool discoverability
00:39:25 — What is gcx, and why a CLI?
00:46:13 — Demo: creating an SLO with MCP vs gcx
00:52:58 — Benchmarking MCP vs gcx with o11y-bench
00:58:10 — How MCP and gcx relate to Grafana Assistant
01:02:35 — k6 MCP vs Grafana MCP
01:03:00 — When to use which (decision guide)
01:04:29 — Adding the OSS Grafana MCP to Assistant
01:05:41 — Turtles all the way down

Resources

MCP and CLI for agents

My Context Horizon piece: Unlearnings from building Grafana Assistant

Grafana Campfire 🔥 — GrafanaCON 2026 recap

Thu, 28 May 2026 17:45:00 +0100

I co-hosted this month’s Grafana Campfire with Stephanie Hingtgen, where we recapped everything that came out of GrafanaCON 2026. Stephanie owned the Grafana 13 walkthrough — Git Sync GA, Dynamic Dashboards GA, Dashboards API v2 GA, Grafana Advisor GA, SCIM GA — what she described as the year Grafana stopped being beta in production. I covered the AI section (Grafana Assistant now in OSS and Enterprise, the three scaling challenges from “Scaling Grafana Assistant”, Investigations, Skills going GA), the three-teams-one-architecture push across Loki Thor, Mimir 3.0, and Pyroscope 2.0, the OTel + Alloy roadmap, alerting, k6 v2.0, and the Marketplace and Logline.

We also took questions from the Champions community on the pace of UI change, Mimir Alertmanager, and the agent-scale question — what happens when your AI agents are themselves the noisiest users of your observability stack? — and threw in a flavour break about the TamaGROTchi and the ocean digital twin demo.

Resources

Asimov's Zeroth Law of Robotics: Observability for AI (ExpoQA Madrid 2026)

Tue, 26 May 2026 10:00:00 +0200

This week I gave the opening keynote at ExpoQA Madrid 2026 in Madrid, Spain. I presented a talk called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously given at KubeCon EU 2025 in London, at Dutch Cloud Native Day 2025 in Utrecht, and at Newcrafts in Paris. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

This was a keynote slot at a testing-focused audience, which gave me the chance to lean more into the evals vs. observability framing — what evals can and can’t tell you about an AI system in production, and how distributed tracing fills the gap that benchmarks leave behind. The demo app (a Flask-based “play.py” instrumented with OpenTelemetry, flowing into Tempo for traces, Prometheus for metrics, and Loki for logs) is still on GitHub, and I’ve been refining it with each iteration.

Here’s the recording of the talk:

Resources

Demo app on GitHub

Zero to Dashboard with Grafana Assistant and the Infinity datasource plugin

Fri, 22 May 2026 17:00:00 +0100

What if you didn’t have to write a single query to get a useful dashboard?

In this short video, I go from nothing to a working dashboard in a few minutes — no queries — using Grafana Assistant and the Infinity datasource plugin. The data I’m visualising is the RAWG video game database API, so I end up with a dashboard that helps me decide what to play next.

Why Infinity is fun here

Most Grafana data sources are scoped to a specific kind of backend — Prometheus, Loki, a SQL database. Infinity is different: it lets you treat any REST/JSON/CSV/HTML endpoint as a data source. That means anything with an API can become a dashboard, which is exactly the right shape for goofing around with a games API on a Friday afternoon.

Pair Infinity with Assistant and you don’t even have to know what queries to write — Assistant configures the data source, asks the API the right questions, and builds the panels for you.

Timestamps

00:00 — Intro
00:35 — The RAWG video game database
01:01 — Setting up the Infinity datasource
05:10 — Customising the dashboard with Assistant
09:31 — Closing thoughts

Resources

Step-by-step blog post: using Assistant with the Infinity datasource
Grafana Assistant documentation
Infinity datasource plugin
RAWG video game database API
Get started with Grafana Cloud (free tier)
My notes on Grafana Assistant — everything I’m learning as I work on it

Continuous Profiling, the OpenTelemetry Way (Grafana ❤️🔥 OpenTelemetry Community Call #7)

Mon, 18 May 2026 05:00:00 +0100

In this community call, my coworker Tiffany Jernigan and I sat down with Christian Simon, who works on Grafana Pyroscope, to dig into profiling as the fourth signal in OpenTelemetry. Metrics, logs, and traces are the classic three pillars — but a significant portion of latency lives inside a span, in the actual code execution, where the other three can’t see. Profiling fills that gap by giving you continuous, code-level visibility into where time is being spent.

What we covered

🔥 Profiling as the fourth signal

Most observability conversations stop at logs, metrics, and traces. But when you see a slow span and ask “what was the code actually doing?” — none of the three pillars can answer. Christian made the case that profiling is the missing piece: continuous, low-overhead, code-level visibility, typically visualized as flame graphs that attribute latency to specific functions and execution paths.

📊 What flame graphs actually tell you

If you haven’t worked with flame graphs before: wide bars = lots of time spent in that function, and the stack reads bottom-up. Once you understand how to read them, they become surprisingly fast to interpret. We walked through what to look for and how to spot the patterns that matter.

🌐 The current state of profiling in OpenTelemetry

Profiling is the newest signal to enter the OpenTelemetry spec, and the conversation around it is still evolving. Christian gave a clear-eyed status update on where things are — what’s stable, what’s experimental, and what the path to GA looks like.

🚀 Running profiling in production

The most common pushback on continuous profiling is “what’s the overhead?” — and the honest answer is that modern continuous profilers typically add 1–3% CPU overhead, which makes them safe to run in production all the time. We talked through deployment patterns, including Kubernetes / containerized environments, and how to think about cost and storage.

🧵 Correlating profiles with traces and metrics

This is where profiling really earns its place. Trace ID → profile is the killer pattern: you see a slow span, you click through to the flame graph for that exact execution, and you immediately see which function ate the time. Christian showed how this correlation works in practice and how it changes incident response.

🤝 How to get involved

OpenTelemetry profiling is genuinely an area where the community has room to shape the future. If the conversation sparked something, there’s a working group, open issues, and a friendly path in.

Resources

Thanks to Christian for joining, to Tiffany for co-hosting, and to everyone who showed up live with questions.

Grafana k6 Community Call — Secrets Management & AI Script Authoring

Thu, 30 Apr 2026 16:30:00 +0100

The first k6 Community Call in over a year, and we packed it. I sat down with four of the people building k6 — Bukola Ayodele, Facundo Batista (Software Engineer), Vicente Ortega (Senior Backend Engineer), and Charlie Burnett (Product Manager) — to walk through two new pieces of work that have shipped recently, plus a look at what’s coming in k6 v2.

What we covered

🔐 Secrets management in k6 Cloud (Facundo)

A new way to store API tokens, passwords, and other sensitive values for your tests — centralized in Grafana’s secrets manager, encrypted at rest, and never visible in the UI or API once written. Your scripts reference secrets by name, and at runtime the values are injected and automatically redacted from logs.

Why this matters for load testers:

No more hardcoded credentials sitting in scripts or in Git
Easy rotation — change the secret value, every script that uses it picks up the new value, no script edits needed
Same script works across dev/staging/prod by just swapping which secrets are scoped to which environment
Works with k6 OSS locally too — your local runs can pull cloud-managed secrets when you want them

Facundo Batista demoed the full flow: creating a secret in the Grafana Cloud UI (name, description, labels — up to 10 per secret, all write-only), and using it in a script with one line. There’s also an API for programmatic management.

🤖 Grafana Assistant k6 script authoring mode (Vicente)

This is the AI-powered script generation that’s been in the works for a while — fine-tuned specifically for k6 best practices, not a generic “ask an LLM to write you a load test” workflow. The interesting bit is the context sources Assistant can pull from when generating a script:

A service name — Assistant analyzes the existing telemetry data to discover endpoints and shapes
A GitHub repository URL — pulls the code to understand the API
An OpenAPI specification — generates tests that exercise every endpoint with realistic request shapes
Existing observability data — metrics, logs, traces from your actual production traffic

Generated scripts come with proper structure, error handling, checks, and TODO comments where the model genuinely doesn’t know what to write (e.g. for auth tokens, request bodies, or anything that needs your secrets — by design, Assistant won’t fabricate those). You can also point it at an existing script and ask it to refactor or extend.

🚀 k6 v2 (Charlie)

Release candidate 1 is out, GA is very soon. The headline changes:

Extensions marketplace — community protocols and integrations
Enhanced Playwright browser support with compatible syntax
New assertions library
MCP / agent connectivity for AI-driven workflows
DNS, MQTT, and other network protocols

Migration advice from Charlie Burnett: pin your script versions, test your existing scripts against the RC before GA hits, and don’t rush.

🔮 What’s next on the AI roadmap (next 1–2 months)

Visual browser-based testing experience
Enhanced observability workflow automation
Improved user-journey creation across both personal and enterprise apps
Integration with autonomous monitoring systems

Resources

Thanks to Bukola, Facundo, Vicente, and Charlie for joining, and to everyone who showed up live.

High INT, Low WIS: What LLM Benchmarks Miss

Thu, 23 Apr 2026 09:00:00 +0100

Originally posted here on Context Horizon.

In the tabletop roleplaying game Dungeons & Dragons (D&D), the very first thing you do when you create a character is to choose ability scores. Ability scores are numbers that determine how good you are at the game’s main characteristics: Strength (STR), Dexterity (DEX), Constitution (CON), Intelligence (INT), Wisdom (WIS), and Charisma (CHA). 20 is the default maximum that a character can have in any ability, though some in-game events/items can increase that.

Here’s the thing, though: every D&D player knows that a 20 Intelligence doesn’t mean your wizard aces every check. Whether you have proficiency in the specific skill still matters. That 20 INT wizard, for all their raw cognitive horsepower, can absolutely still fail to identify a magical rune. The ability score is a starting point, not a verdict.

This is exactly what’s happening to LLM benchmarks. All the general LLM/AI model benchmarks are ability scores. They tell you something real about a model’s general capabilities, the same way a 20 INT tells you something real about a wizard. But as frontier models crowd toward the top of every leaderboard, we’re running into a problem D&D solved decades ago: the stat block isn’t the character. And it was never supposed to be.

A screenshot of the stats of my Level 10 bard. Low INT, but look at that CHA!

The stat block: A tour of the ability scores everyone’s maxing out

There are so many new models and model versions out there that it’s hard to keep up with what we’re supposed to be using for which tasks. LLM benchmarks have come to be relied on to give a somewhat objective assessment of each model’s capability. Here are a few big ones that exist.

General knowledge/reasoning

MMLU (Measuring Massive Multitask Language Understanding, 2020)

MMLU is the classic “general knowledge” INT check. It was released in September 2020 by Hendrycks et al. and contains 15,908 multiple-choice questions across 57 subjects (humanities, social science, STEM, other). However, there were a few issues with it:

a study in 2024 showed that 6.49% of those questions contained errors
The New York Times reported (2024) that data contamination was an issue — models could be trained on the MMLU questions (and answers) to inflate their MMLU scores
many frontier LLMs soon consistently achieved high scores

In response, a new benchmark called MMLU-Pro was created with 12,000 questions that each have 10 options. The increase in options theoretically required more reasoning capability than the original benchmark, but frontier models are already knocking at 90% on the MMLU-Pro.

Humanity’s Last Exam (HLE, 2025)

HLE was developed specifically to address the fact that LLMs “now achieve over 90% accuracy on popular benchmarks like MMLU” (from the HLE site). The paper by Phan et al explains the lengths they took to make it hard enough to be meaningful:

they solicited high-quality questions from nearly 1000 subject matter experts, even going as far as paying them for successful submissions (with a total prize pool of US $500,000)
they included graduate-level classics (one of the questions includes translating Palmyrene script)
they reviewed potential submissions using current LLMs and only proceeded with submissions if the LLMs are unable to answer correctly
experts then reviewed the submissions further

As a result, HLE scores are low — intentionally so:

While current LLMs achieve very low accuracy on HLE, recent history shows benchmarks are quickly saturated – with models dramatically progressing from near-zero to near-perfect performance in a short timeframe [13, 45]. Given the rapid pace of AI development, it is plausible that models could exceed 50% accuracy on HLE by the end of 2025. (Phan et al., 2025)

Software engineering

HumanEval (OpenAI, 2021)

In 2021, OpenAI released a paper and an evaluation harness for a benchmark consisting of 164 Python programming problems based on publicly available code on GitHub. It was developed to test coding in particular.

Eventually, though, experts found that the problem sets in HumanEval were insufficient and ambiguous (Liu et al., 2023). HumanEval+ was released to include more tests, but the next benchmark’s authors suggested that models might be overfitting to HumanEval+ as well, memorizing the dataset and answers rather than genuinely demonstrating reasoning ability.

LiveCodeBench (2024)

Jain et al. tried a different approach to HumanEval to avoid data contamination: it was released with a continuously-updated coding benchmark. Problems have release dates and models can only be evaluated using problems released after their training data cutoff dates.

LiveCodeBench represents a shift in benchmarks towards a more continuous, dynamic measure. However, the last submission to the benchmark was 9 months ago at the time of this writing, and the leaderboard has not been updated since then either, suggesting the benchmark is no longer being actively maintained.

SWE-bench (2024)

This benchmark was created by Jimenez et al. to include 2,294 actual GitHub issues from 12 Python projects. When it launched, it was a much harder benchmark than predecessors, with models scoring 2-4%. Today, models are scoring around 70-80% on the original benchmark. There were enough issues with the original’s dataset quality that variants were introduced, including OpenAI’s SWE-bench Verified. Today, the SWE-bench website shows these variants:

SWE-bench
SWE-bench Verified
SWE-bench Multilingual
SWE-bench Multimodal
SWE-bench Lite

Agents and tool use

This last category is a preview of where I think the field should go: benchmarks that measure what a model does rather than just what a model knows. Here are some of them:

GAIA (Mialon et al., 2023) — 466 real-world assistant tasks that combine web browsing, file reading, and multi-step reasoning. Easy for humans (92% success), hard for models (15% success for GPT-4), at least at launch.
Terminal-Bench — agents working in actual Linux terminals. Tasks include “build the Linux kernel from source” and “configure a git server plus a webserver on port 8080.” No mocks; if it doesn’t actually build, it fails.
WebArena (Zhou et al., 2023) — realistic simulated websites (Reddit, GitLab, e-commerce) where agents complete actual user goals like “filter the last six months of orders by refund status.”

These benchmarks are harder to saturate than general-knowledge ones, because passing means the thing actually has to work in the real world. A model can memorise a question; it can’t memorise a Linux kernel that isn’t there until it builds it.

The wrong answer: Build a harder general benchmark

By now, you probably see that benchmarks tend to go the same way: A benchmark is created, models get too good at it and perform too well to be useful, and then a variant benchmark is created to address the problem.

This is the saturation problem in a nutshell: a benchmark that once provided meaningful information to help distinguish models eventually fails to do so because all the models’ scores cluster at the top.

Saturation occurs for two reasons:

Contamination: Many benchmarks publish their problem sets online, and models train on data scraped from the web. The benchmark now measures memorization, not capability.
Overfitting: Model providers are incentivized to optimize specifically for gaming performance on benchmarks. Scores go up without corresponding capability improvements.

When you have a party of wizards with 20 INT, “What’s your INT score?” stops being an interesting question because any wizard’s Fireball will hit the same. Building harder benchmarks is like an arms race for ability scores, when what you actually want to know is whether your wizard can kill a dragon.

Harder benchmarks don’t tell us what we want to know: can this model do my specific job?

The right answer: skill checks with real DCs

The fix isn’t a harder ability check. It’s a different kind of check altogether. In D&D, straight ability checks are rare compared to skill checks. There are 18 official skills that break down the six ability scores into more granular activities. For example, your wizard is much more likely to roll one of these instead of a straight INT check:

Arcana
History
Investigation
Nature
Religion

All of these are still based on INT, but test narrow types of intelligence rather than general intelligence.

Specific, domain-grounded benchmarks are what the field actually needs. The evaluations that actually help practitioners share a structure: they test a specific skill in a specific context, and they’re judged by the people who have to live with the results.

MedQA tests whether a model can answer the kind of clinical question a physician might face. LegalBench tests whether a model can correctly identify enforceability clauses, classify contract language, and apply legal reasoning. FinanceBench tests whether a model can answer the kind of question a financial analyst would actually ask about a real 10-K filing. None of these are trying to rank models on general intelligence. They’re asking the only question that matters downstream: can this model do this specific job well enough?

This week, Grafana released o11y-bench, an open benchmark for observability-specific agent tasks. I’m biased — I work there — but it’s a useful illustration of the shape I’m describing. o11y-bench doesn’t ask whether a model is smart in general. It asks whether an agent can do the things observability engineers actually do: read production traces, correlate logs with metrics, figure out why the thing that’s supposed to be working isn’t. Its leaderboard shows models’ performance for each activity.

Does that mean that more specific benchmarks won’t get saturated? No. We’ve already seen this play out with SWE-bench. But they’ll get saturated more slowly and remain more meaningful measures along the way.

These are skill checks with DCs that matter to someone. They’re not trying to produce a single scalar ranking of “which model is smartest.” They’re asking whether the wizard recognizes the family crest belonging to a political faction or knows this specific cult’s practices, not whether the wizard is good at “wizarding” in the abstract.

Roll for initiative: What this means in practice

If you’re an engineer picking a model for production: stop picking by leaderboard. Find or build an eval that reflects your actual use case. A handful of representative examples, graded the way you’d grade them if you were doing the work yourself, will tell you more than a dozen public benchmarks.

If you’re a researcher thinking about what to build next: the interesting work is at the specific-and-grounded end of the spectrum, not the general-and-bigger end. Another harder general-knowledge benchmark will saturate in eighteen months. A well-designed domain benchmark, co-built with the people who actually do the work, can stay useful for years.

And if you’re anyone else watching this field from the outside: be skeptical of the headlines. When a new model scores 94% on a benchmark you’ve never heard of, ask what it measures and whether the thing it measures is the thing you actually care about. Usually the answer is “not really,” and that’s the most important thing a benchmark score won’t tell you.

The goal of a character sheet was never to produce one number that ranks the character. It was to give a DM enough information to run a meaningful encounter — and to give the player enough to know when to try, when to pass, and when to roll with advantage. LLM evaluation should work the same way. Not a ranking; a readiness check. Can this specific model clear this specific challenge? Roll for initiative.

All About AI Observability (The Context Window #03)

Wed, 22 Apr 2026 18:00:00 +0100

In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and the engineers behind the product, Alexander Sniffin and Jack Gordley, for a deep dive into AI Observability in Grafana Cloud. We unpack what it actually is (a new way to instrument your AI apps and see canonical data about them, sitting alongside your existing telemetry), what an evaluation is in this context (and the difference between online evals on live traffic and offline evals on dataset conversations), and we walk through real demos: setting up AI Observability and evaluators, the analytics view, instrumenting a local coding agent, and Jack’s system prompt analysis tool. We also talk about the things nobody else is talking about — like LLM-as-judge as a method, evaluators that catch the most bugs, and the cheating-LLM phenomenon.

Timestamps

00:00:00 — Introductions
00:02:00 — The last month in AI news
00:11:09 — What is AI Observability in Grafana Cloud?
00:19:20 — What is an evaluation?
00:21:05 — The origin of AI Observability
00:25:09 — Demo: Setting up AI Observability and evaluators
00:32:04 — What is LLM as judge?
00:38:43 — Demo: AI Observability Analytics
00:40:52 — AI O11y is based on OpenTelemetry
00:42:17 — Demo: Instrumenting a local coding agent
00:47:18 — Potential future agentic use cases
00:52:00 — Evaluators that catch the most bugs
01:02:23 — Demo: System prompt analysis
01:05:11 — Guess the prompt

Resources

News from the episode

AI Observability

Mentioned

OpenCode — the open-source coding agent Jack instruments
Anthropic: Emotion concepts and their function in a large language model
OpenClaw

How to use Grafana Assistant with self-hosted Grafana

Mon, 20 Apr 2026 09:00:00 +0100

In case you missed it, possibly the biggest news to come from GrafanaCON this year is that if you’re running self-hosted or open-source Grafana, you can now connect it through your Grafana Cloud account to Grafana Assistant, the purpose-built AI tool for all things Grafana.

In this short tutorial, I show you how to enable Assistant on a self-hosted Grafana instance and go from nothing to a working server-health dashboard with a single prompt.

A quick note on what “for open source” actually means

Assistant itself isn’t open source. But you can now use it with your self-hosted or open-source Grafana instance. The way it works is:

You still need a Grafana Cloud account (the free tier has generous inclusions for Assistant).
You link your self-hosted Grafana to that Cloud account.
Assistant then lights up inside your OSS instance.

Think of the Grafana Cloud account as the key that unlocks Assistant wherever your Grafana lives.

Prerequisites

Grafana OSS v13 or later
A Grafana Cloud account (pricing here)

Enabling Assistant

Go to Administration → Plugins, search for Assistant, and click Install.
Refresh the page, then open the Connection tab.
Click Start Connection. You’ll be prompted to log into Grafana Cloud.
Select the organization and stack you want, then click Authorize connection.
Close the tab. Your self-hosted Grafana is now linked to Assistant.

A first dashboard from nothing

Every Grafana instance ships with a TestData data source — perfect for a first run when you don’t yet have real data wired up. Enable it from Data Sources, then open Assistant (the new icon on the top right).

Switch Assistant’s mode from General to Dashboarding — this mode is tuned for creating and managing dashboards — and ask in plain English, for example:

Hi Assistant, could you create a server health dashboard for me using the TestData data source?

What comes back is a starting dashboard with CPU, memory, disk, and network panels already wired up. From there you can tweak panels, edit queries, or ask Assistant to iterate. You’re not starting from scratch — you’re starting from 80%.

What else Assistant can do

I only showed dashboarding in this video, but Assistant can also:

Answer questions about the data already in your Grafana
Run investigations to troubleshoot production incidents
Learn new skills
Manage alerts
Teach you Grafana

That list keeps growing, so stay tuned.

Resources

Grafana Assistant (product page)
Grafana Cloud pricing
How Grafana Labs handles your data
My notes on Grafana Assistant — everything I’m learning as I work on it
Unlearnings from building Grafana Assistant — my Context Horizon piece on what the team had to unlearn

Unlearnings from building Grafana Assistant

Thu, 09 Apr 2026 09:00:00 +0100

Originally posted here

Everyone’s talking about what you need to learn to build AI agents: Prompt engineering. Tool calling. Retrieval strategies. The content ecosystem is full of it. But while I was watching Context Window Episode 1, it struck me that Mat Ryer, Cyril Tovena, and Sven Großmann– some of the team behind Grafana Assistant– spent most of their time on something different: what they had to unlearn.

Grafana Assistant started as a weekend hackathon project. By the time it shipped, the team had thrown out a popular framework, flipped their assumptions about tool design, and discovered that some of the instincts you bring from conventional software development actively work against you when you’re building agents.

I pulled out the ten moments where received wisdom turned out to be wrong. If you’re building anything agentic right now, these are worth sitting with.

❌ LangChain will help us build faster

LangChain is an opensource framework for building applications that use LLMs. It provides structured components that can be used for rapid prototyping.

In the video, Tovena talks about how they started with LangChain during the initial hackathon because they thought it would help them build more quickly, but instead it became obvious fairly early on that the opposite was true. They didn’t end up using a lot of the features that LangChain provided and felt they needed more flexibility and speed.

Abstraction overhead

One possible reason for their troubles with LangChain is that LangChain has standard abstractions for prompts, tool use, and orchestration– things that sound good in theory. But sometimes abstractions lend themselves to creating a sort of black box in your stack, where you don’t know what’s happening inside it. It sounds like the Assistant team missed the oversight and control of what was happening.

❌ Use the same prompt for every model

They also talked about how for the sake of simplicity and consistency, they initially used the same prompt for every model. However, they found out that this actually led to worse results. They found more success in creating slightly different prompts for different models, playing to their strengths and weaknesses a bit more.

There is even an argument (Mizrahi et al.’s “State of What Art? A Call for Multi-Prompt LLM Evaluation) that AI benchmarks that involve sending multiple LLMs the same prompt is a brittle and unreliable way to assess LLM capabilities. Different models respond differently to prompt variations.

In practice, what this means is that switching models isn’t a drop-in replacement. Even if a new model scores better on benchmarks, your prompts were written with a specific model’s tendencies in mind. The unlearning here is letting go of the idea that a prompt is model-agnostic — it’s more like a conversation style, and different models have different conversational defaults.

❌ Treat errors as failures

Traditionally in software, you catch errors and hide them from the user: you log them, handle them, but treat that transaction as a failed one.

The Assistant team had to unlearn this habit when dealing with an LLM, because with agents, error messages are useful data. Feeding it back to the LLM often results in it learning from its own mistake and trying again. It’s basically letting the agent debug itself, in a manner that Ryer noted was remarkably similar to how we humans debug errors too.

❌ Return errors in structured JSON

On the subject of error messages, Großmann mentioned that at one point they were trying to format the errors in JSON or stringified JSON. This practice shows solid engineer intuition because usually, structuring information helps machines process it faster and more accurately. What they found here was the reverse: it was easier and yielded better results when the LLM was just passed the entire error in natural language.

❌ Build as many tools as possible

The team mentioned that at the start, they were inclined to create as many tools/skills for Assistant: one for everything that could be done in Grafana. This made sense until they saw LLMs struggle to determine which tool to use in which context.

To help LLMs identify the correct tool, they added tool descriptions and also experimented with combining multiple tools into fewer, parameterized, tools. These helped, although the team still observed that allowing users to create their own tools (with potentially very general tool descriptions) could potentially confuse LLMs.

❌ Bigger context window = better

Gemini 1.5 Pro announced it had a 2M context window back in 2024, and it might seem that other providers had lagged behind: OpenAI only announced support for 1M tokens in 2025 and Anthropic followed in 2026. However, the Assistant team pointed out that context wasn’t everything. They said that they noticed that bigger wasn’t necessarily better, also pointing out that the chat summarization feature on IDEs like Cursor could still sometimes lose important details when summarizing.

This finding seems to coincide with a 2023 Stanford study (Lost in the Middle: How Language Models Use Long Contexts) that found there was a performance degradation when models access information within a very large context. In the video, they talked about how their system prompts were often 40k tokens long (half for the prompt itself and then the other half for tools).

❌ Start with a multi-agent architecture

Tovena mentioned that creating a multi-agent architecture for Assistant yielded more mixed results than he expected. He said that passing context between agents is difficult, outputs were inconsistent, and there was a bit of a “telephone game” problem where agents didn’t always perfectly understand the intent or output of other agents.

In early 2026, Khatua et al. released a study called “CooperBench: Why Coding Agents Cannot Be Your Teammates Yet” that showed that AI agents aren’t great at collaboration. The study suggests, surprisingly, that this might be because AI agents lack human social habits that help us collaborate better, such as feeling obliged to do what we say and having a good understanding of what others’ plans are.

The Assistant team’s experience is also a perfect example of Gall’s Law, which states that it’s easier to start with simplicity and build complexity than the other way around.

❌ Plan around models as they are now

Software has always moved quickly, but it feels like AI models are moving faster. Stanford’s AI Index Report 2025 reported that models went from having a 50% chance to near-human performance in roughly 2 years (2021-2023). In fact, benchmarks keep getting retired as AI saturates them– meaning that benchmarks can’t reliably be used to distinguish between good and bad models because all the models are testing near the ceiling.

The rapid pace of development around AI requires us to not be so attached to our work: Großmann recalled an incident where someone wrote a joke prompt that spoke like a dog that shockingly did better than their other system prompts. Ryer also mentioned that at one point someone deleted all the system prompts and Assistant weirdly became more accurate.

The lesson is clear: plan for models to improve, and plan to redo a lot of your work when they do.

❌ We don’t need evals

The Assistant team said that they went from having no evals to employing several types of evals:

integration-style evals that exercise whole workflows
unit-style evals that feed inputs plus a system prompt and check the output
LLM-as-judge, where they pass conversations to an LLM to ask it to judge whether it answered the user’s questions.
user feedback in the form of thumbs during conversations with Assistant

Beyond evaluation, they talked about how important it was to be able to observe AI. At the very minimum, showing what Assistant is doing at all times (thinking out loud) is very important to them because it builds users’ trust in the thinking process and, eventually, in the result.

Evals aren’t a sign that you don’t trust your product. They’re how you build the confidence to keep shipping it.

❌ Assume people already know how to work with AI

When you ship an AI feature, it’s tempting to assume the hard work is done. You’ve built the thing. Users will figure it out.

What the Grafana Assistant team found was that this assumption doesn’t hold. Users came in with wildly different mental models of what an AI assistant could or should do — some treating it like a search box, others expecting it to read their minds, most not yet sure how to prompt it effectively.

Their insight was that this isn’t just a UX problem to solve on your end. It’s a shared responsibility: AI should try to help the user, but the user also needs to learn how to work with AI. And building an assistant that teaches people how to use it well (through good responses, gentle corrections, showing its work) is part of building the product.

There’s also a deeper point here specific to observability. Grafana Assistant isn’t just an AI feature; it’s a way of interacting with your own infrastructure. Every good interaction teaches you something about what questions to ask, what signals to look for, what it means to understand a system. In other words: learning to use AI well makes you a better engineer. The tool and the skill co-evolve.

This is the unlearning that sits furthest from the code. It’s not about prompts or architectures or frameworks. It’s about recognising that you’re not just shipping a feature– you’re introducing a new way of working. That takes time. And it’s worth designing for explicitly.

Summary

What strikes me most about this list isn’t any single item — it’s the pattern. Most of these unlearnings point in the same direction: less is more, and messiness is the point.

Less abstraction. Fewer tools. Smaller context. Simpler architecture. And rather than hiding the mess — the errors, the reasoning, the uncertainty — let it show. That’s where the trust gets built.

We’re all still early with this. The team that built one of the most thoughtful AI products I’ve seen up close still didn’t have evals when they launched. That’s not a criticism — it’s a reminder. You don’t need to have it all figured out before you ship. You just need to commit to keep unlearning.

Behind the Magic of Auto-Instrumentation (Grafana ❤️🔥 OpenTelemetry Community Call #6)

Fri, 27 Feb 2026 16:00:00 +0100

In this community call with my coworker Liudmila Molkova, we pulled back the curtain on OpenTelemetry’s auto-instrumentation capabilities. We explored how auto-instrumentation works under the hood, when to use it versus manual instrumentation, and the trade-offs involved. The discussion covered language-specific implementations, the Collector’s role in auto-instrumentation, and practical tips for getting started with zero-code observability.

Observability for GenAI Applications (Grafana ❤️🔥 OpenTelemetry Community Call #4)

Wed, 21 Jan 2026 16:00:00 +0100

In this community call with my coworker Liudmila Molkova, we dove into the emerging field of observability for Generative AI applications. We explored how to instrument LLM-powered applications using OpenTelemetry, what metrics and traces matter most when monitoring AI workloads, and how to debug issues in complex AI pipelines. The discussion covered prompt tracing, token usage monitoring, latency considerations, and the unique challenges of observing non-deterministic AI systems.

Logging Best Practices (Grafana ❤️🔥 OpenTelemetry Community Call #3)

Wed, 17 Dec 2025 16:00:00 +0100

In this community call with my coworker Liudmila Molkova, we explored logging best practices in the context of OpenTelemetry. We discussed how to structure logs for observability, the relationship between logs and other telemetry signals (traces and metrics), and practical patterns for making logs more useful in distributed systems. The conversation covered log levels, contextual information, correlation with traces, and common pitfalls to avoid when instrumenting applications with OpenTelemetry logging.

Lightweight Opensource APM with OTel Demo (Grafana ❤️‍🔥 OpenTelemetry Community Call #2)

Wed, 19 Nov 2025 14:48:59 +0100

This community call with my coworker Liudmila Molkova featured an interview with Cyrille Le Clerc from Grafana Labs, focusing on the OpenTelemetry demo as a practical example of a lightweight, open-source Application Performance Monitoring (APM) solution. Cyril explained the purpose and architecture of the OTel demo, highlighting its role as a realistic, complex microservices application that has become a de facto standard for observability vendors. The discussion explored the modern definition of APM, followed by a detailed walkthrough of a Grafana dashboard built on OTel data, covering RED metrics, alerting, trace correlation, and infrastructure monitoring. A significant portion of the conversation was dedicated to the role and challenges of integrating logs within an OpenTelemetry framework. The interview concluded with a look at the future of the OTel demo, including plans for AI observability and eBPF instrumentation.

Transcript

Introduction

Liudmila:
Okay. Hi everyone. Welcome to the second Grafana and OpenTelemetry community call. This time we’re going to talk about the lightweight open-source APM and we will use the OpenTelemetry demo as our playground. We have an expert, Cyrille, who works at Grafana. Cyrille, do you want to introduce yourCyrillef?

Cyrille:
Thank you very much, Liudmila. My name is Cyrille, and I am a product manager at Grafana Labs. I’ve been working on open-source products for 15 years, first on Jenkins for CI/CDs, then at Elastic on observability, and now at Grafana Labs, also on observability. In addition to my day job, I have been an open-source contributor for about 20 years, starting with Apache Java projects and then moving on to Jenkins integration with Maven and OpenTelemetry. Now, I focus more on the OpenTelemetry project, like the OpenTelemetry demo, and I contribute to OpenTelemetry Helm charts as well. At Grafana Labs, I am the product management lead in charge of OpenTelemetry and I cover other areas as well. [laughter]

Nicole:
That’s great. It’s great to have you here with us. Also, great for my internet to act up just a few minutes before this. Anyway, hi! I’m Nicole van der Hoeven, a developer advocate at Grafana. I work with Cyrille over there, and I’m really excited to be talking about OpenTelemetry. I’m the newbie of all of us here. [laughter] I have used OpenTelemetry in some demo apps I’ve built. Most recently, I used it for instrumenting an AI app. I’m excited to ask Cyrille all my questions because when he says he likes OpenTelemetry, what he really means is he’s the guy at Grafana who’s always banging on that OpenTelemetry drum. He’s like, “Tell all the things!” [laughter]

Liudmila:
It’s probably my turn. I’m Liudmila. I work on the OpenTelemetry project and I’m a technical committee member. To echo what Nicole said, nobody knows all of OpenTelemetry, and this call is where we will uncover areas that are new to each of us. Going forward, we’ll probably have more of these calls. So, with that, let’s talk about the OpenTelemetry demo. Cyrille, what is it? Why do we care?

What is the OpenTelemetry Demo?

Cyrille:
So, the OpenTelemetry demo was born to showcase OpenTelemetry. OpenTelemetry was created in the context of cloud-native architecture. This demo is a cloud-native application—a distributed microservices application that is instrumented with OpenTelemetry. It’s maintained by the OpenTelemetry community and showcases all dimensions of modern monitoring or observability, including all telemetry types: traces, metrics, logs, and more at every layer. We started with monitoring the application layer and are now extending to show the infrastructure layer.

We demo various technologies: Java, .NET, Node.js, and nearly all programming languages, including Rust. We also show how to instrument middleware and databases, such as Kafka, PostgreSQL, and Redis, all deployed on Docker Compose to make it easy to play with on your local desktop or deployed on Kubernetes to be more production-like.

This is a complex demo that shows everything, and what’s interesting is that it is realistic and currently used by 40 observability vendors to demonstrate their solutions with the OpenTelemetry demo. So, when you get familiar with the OpenTelemetry demo, it’s really easy to evaluate different vendors quickly. It comes “batteries included,” with open-source monitoring and observability technologies like Prometheus, Jaeger, OpenSearch, and Grafana to showcase how to implement an observability solution with OpenTelemetry.

Exploring the Application

Liudmila:
Thanks for the intro! So maybe we should take a quick look at what the application itCyrillef looks like to make it more concrete.

I’m connected to a Kubernetes cluster, and I’m just port forwarding. This is the telescope shop we have in the OpenTelemetry demo. We can go and buy things, and now we have something in the cart that we can place an order for. Oh, actually, this application might not work as expected. Well, that’s why we need observability, right? If things don’t work, we need to figure out what’s broken and how to fix it.

Before we move on, I just want to show that even though you only see the front-end web shop, it’s a relatively large application. It has quite a few services, databases, caches, a front-end proxy, and a log generator that consistently sends requests.

We’re super lucky to have all the telemetry Cyrille talked about. We have metrics in Prometheus, traces in Jaeger, and logs in OpenSearch, all visible in Grafana as a single pane of glass that you can refer to. With this, we can start looking at the telemetry itCyrillef. But I wanted to check with Cyrille or Nicole if you have any additional thoughts or anything we should cover here.

Nicole:
No, I think you presented it well. It’s a very complex topology because we want to demo everything.

Cyrille:
Yeah, it’s almost complex by design. You could probably have made this simpler, but it’s supposed to approximate real applications that aren’t simple. That’s the problem with some of these other demo apps; they are so simple to deploy that they work because they’re monolithic or something like that. This one is much more of a polyglot architecture, involving microservices that represent the real challenges that people face in the field.

Application Performance Monitoring (APM)

Liudmila:
One great thing about this demo is that sometimes it’s hard to understand how someone onboarded an application to OpenTelemetry. The demo has a lot of information on what services are used, and it demonstrates what’s been done to instrument this application with OpenTelemetry, which can help anyone with their journey in the OpenTelemetry space. Okay, wonderful! Let’s change gears and start looking at the APM dashboard.

Before we go there, let’s talk about what APM is for a second. Why do we care? What is it?

Cyrille:
APM is a very old concept—over 15 years old. We’ve had several generations of monitoring tools. APM stands for Application Performance Monitoring. I see APM as a monitoring and observability technology that helps you monitor your application and understand its health.

When I think about it, the first thing that comes to mind is understanding if my service is healthy or unhealthy. In a distributed architecture, I need to see my service in the context of the connections between services, typically represented with a service map. If I find something unhealthy, I need to drill down to see the health of a service. This means switching back and forth between the service metrics, which are more aggregated, and the traces, which give indications of how business transactions are executed.

I also need to access logs to verify how things really happen and if there are messages that help me understand the problem. This is how I understand my service. Additionally, my service is deployed on a runtime and middleware, so I need to be able to zoom lower to see how it’s behaving on the Kubernetes pod or Linux box.

APM allows me to switch back and forth between all layers, zooming in and out all the time across all telemetry types. This is essential for understanding health and troubleshooting. Another dimension is detecting anomalies, so I can wake people up at night when something goes wrong. This is where I create alerts and notifications.

In the past, with monoliths, we created many alerts, but now we adopt best practices with SLOs, observing workflows, and root cause analysis. This is my vision of APM and why I think it’s more important than ever.

Nicole:
When you talk about application performance management or monitoring, do you differentiate that from the wider field of observability? Some people say APM sounds outdated because it focuses on the application or just on performance, which is ambiguous.

Cyrille:
The monitoring landscape has evolved a lot. With cloud-native architecture, many things have changed, and we had to change our best practices. The best practices from 20 years ago in monoliths are not the same as today’s best practices for distributed cloud-native architectures.

There were disruptions in the past by innovators who focused primarily on traces, and we learned a lot from that. But now I think the landscape is getting more stable. We recognize the value in metrics, traces, and logs, and we’ve evolved our best practices to leverage the best of all three.

Some people prefer to call it observability. I know there are nuances, but this is my current perspective.

Liudmila:
It’s good to synchronize on terms. It sounds like we are on the same page.

Cyrille:
Now, with microservices architecture, we focus more on alerting symptoms rather than causes, centering more on the health of the application and the service provided to users than on the infrastructure layer. So, application monitoring or observability is becoming the entry point for people in charge of keeping the systems running.

Nicole:
Why don’t we take a look at the dashboard and go through it?

APM Dashboard Exploration

Cyrille:
We will cover different parts of the dashboard throughout the hour. Let’s start with the top row—the red metrics. You can call them duration, error, and request rate.

Nicole:
I believe the red metrics were created by Tom Wilkerson, who still works at Grafana. I think it was looCyrilley based on the Google SRE book, even though they didn’t explicitly say “red” there.

Cyrille:
When you think about any particular service, you would think about some well-known terminology, which might be service level indicators, or just your application. Service level indicators talk more about what’s important for the application, while red metrics focus more on the telemetry itCyrillef, like the raw data behind your SLI.

One thing that stands out is the alerts. It looks pretty awesome to see that something violates the threshold we’ve set—probably high latency. Let’s explore it. We see it’s been firing for 17 hours. That’s quite a duration.

Nicole:
That’s not so good. Let’s see the alert on the latency. The P95 is firing for a specific operation.

Cyrille:
We can look at this chart, which shows the latency around P99, which is like 18 milliseconds. Is it really a problem?

Nicole:
No, it was for the demo. It’s very situational; it depends on your services. The focus was to show practitioners best practices for creating alerts on HTTP endpoints. Typically, you have two families of alerts: latency alerts and error rate alerts.

Cyrille:
We decided to take P95 as a commonly adopted indicator. Some take P99, but P95 is often more relevant. Something that’s perhaps not noted here is that we take a time interval of 5 minutes. We wanted to provide this out of the box because it relates to the data collection interval, which is typically every minute for hotel setups and quite common in Prometheus.

Having a 5-minute window is quite good. These are best practices we worked on, identifying good labels to help practitioners succeed by easily routing alerts to the right people and having the right notification schemes.

Nicole:
Imagine if there was actually high latency. Now we can enable this view and look at the examples of operations with the worst performance—like the outliers you would see here. We use Jaeger as a store for telemetry, and if it were any other store that supports tracing, you would see something similar.

Cyrille:
This brings us to the Jaeger UI. I don’t want to spend a lot of time here, but it’s just another confirmation that nothing is visibly wrong with this operation. Things are just going a bit slower in the cloud, as usual.

Alert Analysis

Liudmila:
We talked about alerts. There are other interesting things in this picture. For example, the request rate is pretty normal. If there were a spike in usage, we would see it here. The error rate looks good—there are no errors. By looking at these three metrics, you can get a basic understanding of the health of your system and this specific service.

Cyrille:
This dashboard illustrates years of best practices that the industry has accumulated. We have layers of information. At the top, you have your service identifier, the last metrics received time, and the overview of the aggregated service red metrics.

This aggregated view across all operations of my service will help you decide if this service is healthy or not. If you have doubts or think something is going wrong, P95 is good at surfacing outliers.

Typically, you will scroll down to the HTTP or gRPC operations, where you have the details broken down by operation. It’s likely that not all business operations have problems; it could be just a subset of them.

Here you can see the outbound services. If you have a problem, this data will help you understand if it’s happening within your service or is it a downstream service. In microservices architecture, that’s always a question.

Nicole:
Are those dashboards powered exclusively by metrics?

Cyrille:
Everything is powered by metrics, specifically the OpenTelemetry semantic conventions. Initially, the OpenTelemetry demo derived metrics from spans, which was a great idea four years ago, but it had some limits. Now, OpenTelemetry has standardized some semantic conventions for HTTP server calls, client calls, database client calls, RPC calls, and messaging.

Nicole:
I noticed during my time with the demo that the coverage of metrics is uneven across programming languages. Java, for instance, has great instrumentation, while in other languages, it depends on the instrumentation.

Cyrille:
Yes, that’s true. It’s uncommon for OpenTelemetry instrumentations to emit logs these days, but let’s move on to that.

Logs and Observability

Nicole:
So, what’s the deal with logs? Why are they not as important?

Cyrille:
Logs are traditionally outputted in stdout, and we’ve lost Nicole. [laughter] The challenge is that traces and metrics are captured by SDKs within your app, while logs come from external log collectors, making correlation tough. When we discuss with practitioners and tell them that the OpenTelemetry SDK can directly export logs, they often respond that their log processes are mature and they can’t disrupt them just to trust the new kid on the block.

It takes time to change your log toolchain. And while OpenTelemetry SDKs have capabilities to directly export logs, it’s not easy to get people to change practices they’ve had for 30 years.

Nicole:
To demonstrate Cyrille’s point about correlation, the most basic way to correlate is by trace ID and span ID. When we capture distributed traces, we can also attribute logs to the specific operation.

Cyrille:
Yes, and when we look at a trace inside Grafana, we can see logs associated with it.

Nicole:
If we open the trace view, we can see the details for this operation. There are a bunch of attributes you can correlate to understand what happened.

Cyrille:
Logs provide the details. I think the negative sentiment around logs came from the idea that they don’t work as well in distributed applications as they did in monolithic applications. We had to evolve our practices to log the right way for distributed applications.

Nicole:
It’s user error because logs are the signals you can abuse the most. You can put anything in a log line, which causes issues. With metrics, you don’t have as much leeway, which makes it simpler to deal with.

Cyrille:
Yes, and we need to convince users that going into log analytics tools is much more efficient than tailing logs directly.

Nicole:
It is more complicated to analyze logs than traces because they aren’t always structured.

Cyrille:
Exactly. We have to help organizations better structure their logs for better value.

Questions and Community Engagement

Liudmila:
We have a question here. “We are planning to adopt the LGTM stack as an observability platform. Is there any blueprint documentation aligned with observability 2.0?”

Cyrille:
LGTM stands for Loki, Grafana, Tempo, and Mimir. It’s an integrated solution. Observability 2.0 is about unified telemetry—having everything correlated and not just reacting to errors. It’s about understanding business impacts when you decrease response times.

Nicole:
So, it’s more about understanding the overall system rather than just monitoring.

Cyrille:
Yes, exactly.

Liudmila:
How do I know which metrics and traces to prioritize first when instrumenting a distributed system with OpenTelemetry?

Cyrille:
We recommend starting with auto-instrumentation first. It provides metrics and traces at a technical level, and with HTTP, the endpoints are meaningful from a business standpoint. You start with auto-instrumentation and then add custom instrumentation on critical workflows.

Nicole:
What’s the current state of the OpenTelemetry demo? Can it monitor infrastructure as well?

Cyrille:
Yes, OpenTelemetry was born in the application layer, but infrastructure monitoring came a bit later. We’re working with the hotel demo folks to showcase better infrastructure monitoring done by OpenTelemetry.

Liudmila:
If people want to contribute or try out the hotel demo for themCyrilleves, where should they go?

Nicole:
I posted the link in the chat to the OpenTelemetry site.

Cyrille:
If you want to participate in the demo, we have community meetings, and I can share another link in the channel for more information.

Liudmila:
Great! Thank you both for being here and for everyone watching. I think that’s a great way to end this call.

Cyrille:
Thank you very much! It was a pleasure to be invited and an honor.

Nicole:
Thank you!

Asimov's Zeroth Law of Robotics: Observability for AI (Newcrafts 2025)

Thu, 06 Nov 2025 12:23:19 +0100

Today I presented at Newcrafts in Paris, France. I gave a talk I’ve given before, called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU in London and at Dutch Cloud Native Day 2025 in Utrecht. I was happy to present it again, as it gave me the chance to further refine my demo application, which you can find on GitHub.

The main change I made was to use the “AI as judge” approach that I had previously only talked about in theory. In this version of the demo app, I use AI in both the game as well as in the testing of the game, showing how AI can be integrated into both the application and its observability/testing stack. It was really interesting to note that testing with AI ended up exposing more defects than my hardcoded test cases did, which goes to show the potential of AI-driven testing.

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

Wed, 05 Nov 2025 03:07:38 +0100

Today I gave a talk at Øredev 2025 in Malmö, Sweden, titled +The Spice Must Flow: The Fremen Guide to Sustainable Observability"_. The presentation focused on how to implement sustainable observability practices using open-source tools like Grafana, Prometheus, Loki, Tempo, KEDA, and Kepler. Here’s the abstract:

Nobody wants to be wasteful. But how do you balance that with the need for enough data to fix things when they go wrong? After all, no matter how much it costs, there are SLOs to keep. The spice– the telemetry– must flow.

The Fremen are a people who live on the planet Arrakis, simultaneously rich in spice while poor in water. But the Fremen have managed to thrive, not just survive, on very little while continuing to produce spice. What would it take to apply this “desert planet thinking” to observability?

In this talk, you’ll learn about the hidden environmental and financial costs of your observability stack and how to reduce them. You’ll learn about the complications of measuring cost for ephemeral resources on Kubernetes, how to use KEDA to “right-size” resources when you don’t know what the right size is, and how to convince your teams to drop all but the most meaningful telemetry data. In the end, the spice must flow, but there must still be enough water to sustain all.

Here are some resources mentioned in the talk:

You can play the game Spice Runner here
The spice-runner repo containing everything you need to know about how to deploy it yourself
The slides
The Spice Runner dashboard for observing the application
More about Spice Runner: architecture, tech stack, and what it demonstrates

Deploying Loki on Kubernetes via Helm (Loki Community Call October 2025)

Wed, 29 Oct 2025 18:04:30 +0100

This Loki Community Call is about deploying Loki on Kubernetes via Helm charts. We talk about why you might want to use Helm to deploy on Kubernetes, best practices for deployment, and which Helm chart you should use! We are Jay Clifford and Nicole van der Hoeven, Developer Advocates at Grafana Labs, and we have invited Grafana Champion and Loki Helm Maintainer Jan-Otto Kröpke, Principal Cloud Architect at QualityOperations GmbH, to talk about the state of the Loki Helm Chart.

Declarative Configuration in OpenTelemetry SDKs (Grafana ❤️‍🔥 OpenTelemetry Community Call October 2025)

Wed, 22 Oct 2025 15:02:21 -0700

I’m very lucky to have the opportunity to work with Liudmila Molkova, who just joined as a Staff Developer Advocate at Grafana Labs. We decided we would start a new monthly livestream called the Grafana ❤️‍🔥 OpenTelemetry Community Call. In this inaugural episode, we interviewed Marylia Gutierrez about declarative configuration for OpenTelemetry SDKs. We talked about:

Pains in getting started with OpenTelemetry, particularly around instrumentation and configuration
What declarative configuration is, and how it improves the experience
Which languages declarative configuration is for
The future of declarative configuration in OpenTelemetry
Marylia’s candidacy for Governing Committee of OTel (vote for her here from October 27th-30th, 2025)

One Year Later: How I Use Visuals in My PKM (Sketch Your Mind Conference)

Mon, 13 Oct 2025 13:33:18 -0700

Today I gave a talk at my friend Zsolt Viczián’s Sketch Your Mind Conference about how I use visuals in my PKM, a follow-up to my video about Obsidian Excalidraw last year where I shared that I’d changed all my Obsidian templates to hybrid Obsidian Excalidraw notes:

In the talk, I shared how I’ve evolved my use of visuals in my PKM over the last year, including:

Referring to icon and logo libraries that dynamically compile all the atomic visuals I use in my notes
Adopting a visual Zettelkasten-like approach
Creating notes at conferences to track people I’ve met and talks I’ve attended
Using Obsidian Excalidraw to create compelling presentations in lieu of slides
Combining visuals with audio recordings to create literature notes of books I’m reading
Using a full observability pipeline to monitor and analyze the notes in my Obsidian vault

I was particularly excited about the last point. I work at Grafana Labs and help people monitor their computer systems using opensource software. I realized that I could use the same tools to monitor my PKM system, so I set up a pipeline where a Python script scrapes note metadata, and sends information to Loki and Prometheus via Alloy, and then I visualize it all in Grafana. It’s a pretty nerdy and totally unnecessary setup, but here’s the repo for anyone interested in replicating it.

How to Turn Logs Into Metrics with Grafana Loki (Loki Community Call July 2025)

Tue, 08 Jul 2025 17:23:23 +0200

In this Loki Community Call, we talk all about how to turn metrics into logs in Grafana Loki using metric queries. We cover the basics of metric queries, how to use them to create metrics from logs, and some advanced use cases like using metric queries to create alerts and dashboards.

Asimov's Zeroth Law of Robotics (Dutch Cloud Native Day 2025)

Thu, 03 Jul 2025 15:01:24 +0200

Today I presented at the Dutch Cloud Native Day 2025 in Utrecht, the Netherlands. I gave a talk titled Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU earlier on in the year, in London. I was very happy to be able to present it again, because it gave me the opportunity to improve the demo app I used, which you can find on GitHub.

The Road to Loki 4.0 (Loki Community Call June 2025)

Tue, 24 Jun 2025 13:56:45 +0200

In this video, my colleague Jay Clifford and I interview Ed Welch, Principal Engineer on the Loki team at Grafana Labs, about the upcoming Loki 4.0 release. We talk about some pretty big features that make this next release almost an entire rewrite of Loki, including:

A shift towards columnar storage, using a new storage format called “DataObject”
A new Kafka/WarpStream architecture that decouples the read and write paths
A new Loki Query Engine that decouples the query language from the rest of the system, allowing for more options in, for example, allowing for SQL queries against Loki in the future

How to Query Grafana Loki

Fri, 13 Jun 2025 15:21:49 +0200

In this video, I show you how to query Grafana Loki, the open source log aggregation system from Grafana Labs. I cover the basics of Loki’s query language, LogQL, and demonstrate how to use it to filter and search logs effectively.

State of the Observability Databases (Grafana Office Hours #30)

Thu, 15 May 2025 18:22:26 +0100

In this Grafana Office Hours, we talk about the state of observability databases (Grafana Loki, Mimir, Tempo, and Pyroscope) and where they’re going. We talk about current and upcoming architectural changes in all four, how we’re making them more performant, how compatible they are with OpenTelemetry, and what we’re working on next for each database. In this conversation are Dee Kitchen (VP of Engineering for Databases) and Senior Developer Advocates Jay Clifford and Nicole van der Hoeven.

Meta-monitoring Loki (Loki Community Call May 2025)

Mon, 12 May 2025 17:59:48 +0100

In this Loki Community Call, we talk about the need for meta-monitoring Loki: why Loki needs to be monitored, what to watch out for, and how to do it. We talk about different ways to get information from Loki that allow you to make it reliable, consistent, and performant, including a Helm chart to deploy a meta-monitoring stack on Kubernetes. We discuss the Loki mixin for Grafana and how to use it to visualize data about Loki. On the call are Jay Clifford, Nicole van der Hoeven, and Dylan Guedes from Grafana Labs.

All about OTel and Logging on Kubernetes with Loki (Loki Community Call April 2025)

Fri, 25 Apr 2025 17:57:58 +0100

In this pre-recorded Loki Community Call, we talk all about OTel and logging on Kubernetes with Cyril Tovena, Ward Bekker, Jay Clifford, and Nicole van der Hoeven at KubeCon EU 2025 in London. We discuss when why you should switch to OTel and why you shouldn’t, what OTLP is exactly, and best practices for ingesting data through an OTLP endpoint.

Asimov's Zeroth Law of Robotics: Observability for AI (KubeCon EU 2025)

Wed, 02 Apr 2025 13:46:42 +0200

Today I gave a talk at KubeCon EU 2025 in London called “Asmiov’s Zeroth Law of Robotics: Observability for AI”. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

Do's and Don'ts of Scaling Loki Cache (Loki Community Call March 2025)

Tue, 25 Mar 2025 17:56:00 +0100

In this Loki community call, Poyzan (Staff Engineer on the Loki team) returns this time with Paul Rogers (Staff Engineer on the Loki team) to talk all about Loki cache! This topic was heavily discussed in our “How to run Loki at scale on Kubernetes” call. Some of the topics we will cover:

Types of cache and cache tiering
Best practices for scaling your cache
Understanding failure scenarios with Loki cache and how to detect them

Best Practices for Recording and Alerting (Loki Community Call February 2025)

Wed, 26 Feb 2025 17:54:15 +0100

In this Loki Community Call, we talk about the best practices for recording and alerting rules with Loki Staff Engineer George Robinson and Senior Developer Advocates Jay Clifford and Nicole van der Hoeven. Ever wondered what’s the difference between the Ruler and Alertmanager? 🤔 Curious about the different modes the Ruler can run in when querying Loki? Want to avoid the most common pitfalls when setting up alerting and recording rules? We cover all of these, including optimizing rule performance to structure alerts effectively.

How to Run Loki at Scale on Kubernetes (Loki Community Call January 2025)

Thu, 23 Jan 2025 17:51:00 +0100

Happy New Year from the Loki Engineering team. To kick off 2025, Nicole and Jay will be joined by Poyzan Taneli from the Loki Engineering team to discuss how to run Loki at scale on Kubernetes. If you are currently running Loki in microservices mode or preparing to do so, we will be discussing best practices for scaling its components to meet the demands of production use cases.

How many Ingesters vs Queriers do I need?
What is the minimum cost of running Loki as a production cluster?
What is the average node size for running Loki? How do I monitor my Loki deployment?

Grafana Loki Query Best Practices with LogQL (Loki Community Call December 2024)

Thu, 05 Dec 2024 17:48:49 +0100

In this December’s Loki Community Call, Cyril Tovena, Senior Principal Engineer and LogQL guru walks us through a Grafana Loki query tutorial with LogQL, the Log Query Language used for Loki. He talks about the key “Dos and Don’ts” of LogQL, offering practical tips to help you write better queries, boost performance, and sidestep common mistakes.

Whether you’re tuning up your current setup or just diving into LogQL, Cyril’s got you covered. Bring your questions, connect with the Loki community, and pick up insights that’ll make your log-querying life easier. See you there!

Make sure you leave your questions here.

Cyril is joined by Senior Developer Advocates Jay Clifford and Nicole van der Hoeven.

AI Observability with Ishan Jain (Grafana Office Hours #29)

Fri, 08 Nov 2024 18:24:29 +0100

In this Grafana Office Hours, Ishan Jain talks about AI Observability with Grafana: what it entails, factors to consider when monitoring and observing LLMs, and how to do it all with Grafana. He is joined by Senior Developer Advocate Nicole van der Hoeven.

All About Logs Drilldown (Loki Community Call October 2024)

Thu, 03 Oct 2024 17:45:33 +0100

In this Community Call, Senior Software Engineer Trevor Whitney talks to us all about Drilldown Logs for Grafana Loki, an open-source app for visualizing logs from Loki in Grafana without needing to learn and write LogQL queries. He is joined by Senior Developer Advocates Nicole van der Hoeven and Jay Clifford.

All the Components of Loki Explained

Fri, 20 Sep 2024 18:09:35 +0100

Here are all the components of Loki explained so that you can figure out how many instances of each you need for a resilient Loki deployment. Senior Developer Advocates Jay Clifford and Nicole van der Hoeven talk you through what each component does and which ones are optional.

Bloom Filter Changes for Grafana Loki (Loki Community Call September 2024)

Thu, 05 Sep 2024 17:41:18 +0100

In this Community Call, Senior Software Engineer Christian Haudum talks to us about bloom filter changes for Grafana Loki, including the deprecation of the bloom compactor and a pivot towards creating bloom filters for structured metadata. Bloom filters are a probabilistic data structure that we’re using to improve query performance in Loki.

Community Calls are monthly meetings that are open to everyone interested in the development of Loki. They are an opportunity for software engineers working on Loki to discuss new features as well as for open-source users of Loki to ask questions. To participate in the next Community Call, subscribe to the calendar here.

API Monitoring with Traefik, Grafana, and OpenTelemetry (Grafana Office Hours #28)

Fri, 12 Jul 2024 18:27:03 +0100

Maytham Alfouadi (Solutions Architect) and Immánuel Fodor (Product Manager) from Traefik Labs give us a demonstration of how to do API monitoring with Traefik, Grafana, and OpenTelemetry. They talk about Traefik, an open-source reverse proxy, among other things, that is now included by default in k3s and Rancher. They are joined by Usman Ahmad and Nicole van der Hoeven, both Senior Developer Advocates at Grafana Labs.

Cartography of the Deep: Distributed Tracing for Testing (Romanian Testing Conference)

Thu, 13 Jun 2024 12:24:02 -0700

I presented a talk today at the Romanian Testing Conference 2024 in Cluj-Napoca, Romania! My talk was called “Cartography of the Deep: Distributed Tracing for Testing” and it covered how to tie in distributed tracing into tests to get a better understanding of your system under test.

Here are my slides!

Abstract

From the Scandinavian Kraken that drags entire ships to their early graves to the Philippine Bakunawa said to rise from the waters to swallow the moon, our existential fears about what lurks at the bottom of our own oceans has inspired us to create hundreds of folkloric monsters across cultures. We’ve charted the surface of our planet, but the vast majority of the ocean floor remains a mystery.

The mysterious depths of modern software architectures makes it difficult for us to test a system for performance and reliability. The easy solution? Unmanned underwater probes: send inputs to the system, and judge its state based on the outputs that return. But this method is often too limited to tell us what’s really going on when something goes wrong. What if there were a way to make our tests themselves more observable?

In this talk, you will learn how to use open-source tools to instrument performance tests, identify bottlenecks at the code level using distributed tracing, and map out a system using a type of software bathymetry. You’ll also learn about the Mariana Trench expeditions, the difficulties of operating in a high-pressure underwater environment, and how to come face to face with monsters of our mythologies.

Grafana Loki Design Basics with Ed Welch (Grafana Office Hours #27)

Fri, 07 Jun 2024 18:29:26 +0100

Ed Welch, Principal Engineer on the Loki team, is talking to us about how to get started with Grafana Loki for beginners. He tells us the story of why Loki was created in the first place, talks about the terminology of Loki (what are indices, streams, labels, metadata, tags, and chunks?), and shares candidly about what Loki is good at and what it’s not good at.

How to Get Started With Loki

Fri, 10 May 2024 18:12:09 +0100

If you were looking for how to get started with Loki, here’s a video where Senior Developer Advocates Jay Clifford and Nicole van der Hoeven talk about what Loki is, how to ingest logs with Loki, how to deploy Loki, and finally how to visualize your logs with Grafana.

How to do continuous profiling right with Grafana Pyroscope and Ryan Perry, (Grafana Office Hours #26)

Fri, 19 Jan 2024 18:09:01 +0100

My friend Paul Balogh and I interview Ryan Perry, co-founder of Grafana Pyroscope, about how to do continuous profiling right. Ryan, and the rest of the Pyroscope team, joined Grafana Labs, where he is now a Director of Engineer. Ryan discusses the main concerns in continuous profiling and how to avoid those pitfalls. Pyroscope is an open-source project for aggregating continuous profiling data about your system’s resources.

How to Deploy Grafana Beyla on Kubernetes as a Sidecar Container

Tue, 09 Jan 2024 13:41:38 +0100

Principal Staff Engineer Nikola Grčevski demonstrates how to deploy Grafana Beyla on Kubernetes as a sidecar container. Grafana Beyla is an eBPF auto-instrumentation tool for application observability: you just have to tell it what to listen to, give it permissions, and deploy it– and you can see metrics and logs from that component or service without having to do any manual instrumentation.

Introduction to eBPF With Grafana Beyla (Grafana Office Hours #25)

Fri, 05 Jan 2024 18:03:52 +0100

Nikola Grčevski, Principal Software Engineer at Grafana Labs, gives us an introduction to eBPF with Grafana Beyla. We discuss what is eBPF, how you can use it to auto-instrument applications, and how to get started with Beyla. eBPF observability is all the rage because we all want automagical instrumentation– but does it live up to that promise? Nikola’s here to tell us what eBPF can and can’t do, and where he’d like to take Beyla next.

AI Microscopy with Grafana, Theia Scientific, and Volkov Labs (Grafana Office Hours #24)

Fri, 15 Dec 2023 17:42:58 +0100

What do you get when you combine AI microscopy with Grafana? Well, in this case, you get the Theiascope platform: an application for doing real-time analysis on microscopy images using a combination of Grafana, Prometheus, and PostgreSQL/Timescale. The Theiascope platform was a collaboration between Dr. Christopher Field, Co-Founder/President/ Principal Investigator at Theia Scientific and Mikhail Volkov, Founder/CEO of Volkov Labs.

Getting Started with OpenTelemetry (Grafana Office Hours #23)

Fri, 08 Dec 2023 17:00:23 +0100

Principal Engineer Juraci Paixão Kröhling talks to my colleague Paul Balogh and me about getting started with OpenTelemetry: what it is, why everyone’s talking about it, and how to instrument your systems and improve their reliability and observability. Juraci is a contributor to open source projects like Jaeger and OpenTelemetry.

How to Get Started with Tempo (Grafana Office Hours #22)

Fri, 01 Dec 2023 11:55:55 +0100

Grafana Labs Principal Engineer and Tempo maintainer Joe Elliot talks about how to get started with Tempo, what distributed tracing is, and why TraceQL is cool. He is joined by Developer Advocates Nicole van der Hoeven and Paul Balogh.

Resources

Getting started with Tempo (video)
Grafana Tempo repo
Grafana Tempo docs
The Tempo category on the Grafana community forum
Video playlist for Grafana Tempo
Distributed tracing with Grafana Cloud k6 (Grafana Office Hours video)

How to Use Flow Mode for Grafana Agent (Grafana Office Hours #21)

Fri, 24 Nov 2023 11:47:00 +0100

Senior Software Engineer Matt Durham shows us how to use flow mode for Grafana Agent. Flow mode is a new and better way to install and configure Grafana Agent than the older “static mode”. Among other things, flow mode’s modularity makes it easier to build more complex workflows like traditional data pipelines and allows for more use cases than collecting and processing telemetry.

Resources

Grafana Agent repo
Grafana Agent docs
Grafana Agent flow mode
What is Grafana Agent? (previous Grafana Office Hours video)
Grafana Slack - there’s an #agent channel
Grafana Agent Community Call agenda
Modules for Grafana Agent

Realistic Tests with Testcontainers (Grafana Office Hours #20)

Fri, 17 Nov 2023 11:40:46 +0100

In this episode of Grafana Office Hours, we talk about realistic unit tests utilizing actual instances of databases and other dependencies. We’ll be joined by Manuel de la Peña from AtomicJar, and Pablo Chacin from Grafana to learn how we can use the Testcontainers open-source framework to create lightweight, throwaway dependencies for our tests.

Load Testing on Kubernetes with k6 Private Load Zones (Grafana Office Hours #19)

Fri, 10 Nov 2023 10:46:48 +0100

This week, we’re talking about how you can do load testing on Kubernetes with k6 Private Load Zones, a new feature on Grafana Cloud k6 that leverages the k6 Kubernetes operator to allow you to run distributed load tests against applications behind a firewall. Here to discuss this new feature are Senior Software Engineer Olha Yevtushenko, Product Manager Daniel González Lopes, Developer Advocate Paul Balogh, and Senior Developer Advocate Nicole van der Hoeven.

Combining Frontend and Backend Performance With John Hill (Grafana Office Hours #18)

Fri, 03 Nov 2023 12:30:43 +0100

In this episode of Grafana Office Hours, my colleagues Marie Cruz, Paul Balogh, and I talk to John Hill, a Web UI Test Engineer and Grafana k6 champion, to talk about how the Grafana and k6 ecosystems can be used to ensure performance in mission-critical applications like NASA’s Open MCT.

(TL) Ano ang relasyon ng testing at observabilidad?

Wed, 01 Nov 2023 12:19:46 +0100

Ano ang relasyon ng testing sa observabilidad? Sa video na ito, pinag-uusapan namin ng kaibigan kong si Marie Cruz kung ano ang testing, ano ang observabilidad, at bakit dapat nating gamitin pero para pabutihin ang ating mga aplikasyon.

K-SIX News v0.47

Mon, 30 Oct 2023 11:57:21 +0100

In this video, my colleagues (Leandro Melendez, Marie Cruz, and Paul Balogh) and I discuss the latest updates in k6 v0.47.

Here are some of the highlights:

We’ve created an official k6 Prometheus Grafana dashboard and another dashboard for Prometheus with native histograms.
New APIs have been added for Prometheus Remote Write output for k6 (authentication and HTTP headers).
We’ve increased the character limit for metric names from 63 to 128, aligned with the OpenTelemetry specification update.
k6 browser tests can be executed in Grafana Cloud by everyone.
We’ve improved the cookie API for k6 browser.
We’ve added support for page.on('console') for k6 browser.
There are now official Docker images that include a Chromium-based browser for running k6 browser tests within CI/CD pipelines.
Distributed tracing is in public preview for Grafana Cloud.
xk6-file is a new extension that lets you rename files within your k6 script.
xk6-gcp is a new extension that adds k6 support for messaging using Google PubSub.
xk6-dashboard is now an official Grafana extension that lets you generate HTML dashboards from k6 test results.
k6x is a new utility that automatically detects when you import k6 extensions and builds the custom k6 binary for you so that you don’t have to.
k6-operator now has a Helmchart deployment option.

Real User Monitoring With Grafana Faro (Grafana Office Hours #17)

Fri, 27 Oct 2023 07:00:19 +0200

My friend Marie Cruz and I chat with Software Engineers Kostas Pelelis and Marco Schaefer to discuss what Grafana Faro is, why it’s important to have a real user monitoring solution, and how to get started with it.

Useful links

(ES) Grafana Como Código (Café con Grafana #01)

Thu, 26 Oct 2023 06:55:37 +0200

En este episodio platicamos con Joan López de la Franca, Ingeniero de software senior en Grafana. El nos viene a platicar de las funcionalidades de Grafana como código, que es, como funciona, cuando conviene utilizarlo e incluso nos traerá algunos demos al respecto.

Emergent Load Testing - TestCon Europe 2023

Wed, 25 Oct 2023 20:05:36 +0300

I presented at TestCon Europe 2023 in Vilnius, Lithuania about emergent load testing, and how to use it to find the limits of your system. My talk was called Emergent load testing: Rules for Organized Chaos.

You can find the slides here!

What is Grafana Agent? (Grafana Office Hours #16)

Fri, 20 Oct 2023 12:55:11 +0200

What is Grafana Agent? Senior Software Engineer Paschalis Tsilias tells us about what telemetry collectors do, why they’re useful, what you’d have to do without them, and how to get started with Grafana Agent. He is joined by Developer Advocates Nicole van der Hoeven and Paul Balogh.

Useful links

(ES) Inicia (Café con Grafana #000)

Thu, 19 Oct 2023 12:37:00 +0200

Tenemos un nuevo show para toda la comunidad de habla en Español! En el vamos a estar invitando a Grafanaistas, Champions, Entusiastas y a miembros de la comunidad. Aquí vamos a compartir experiencias, conocimientos, perspectivas, diversion y mucho Español!

Agentless Monitoring for Prometheus in Grafana Cloud (Grafana Office Hours #15)

Fri, 13 Oct 2023 10:56:15 +0200

Did you know you could do agentless monitoring for Prometheus in Grafana Cloud? Senior Software Engineer Matt Nolf talks to me and my coworker Paul Balogh about the new Metrics Endpoint integration and how you can use it to collect Prometheus metrics from publicly addressable hosts– WITHOUT using Grafana Agent.

Resources

Distributed tracing with Grafana Cloud k6 (Grafana Office Hours #14)

Sat, 07 Oct 2023 08:31:03 +0200

My friend Marie Cruz and I interview Senior Software Engineer Łukasz Gut about a new feature: Distributed Tracing with Grafana Cloud k6. He discusses what distributed tracing is, why it matters, and how it can help teams find reliability issues faster.

How to Deploy Grafana on Kubernetes (Grafana Office Hours #13)

Fri, 29 Sep 2023 17:30:49 +0200

My coworker Usman Ahmad and I talk about how to deploy Grafana on Kubernetes for beginners: what Kubernetes is, how it’s an evolution of distributed computing, what its relationship to Docker is, other things you might need to know to work with Kubernetes, and how you can deploy Grafana on your own Kubernetes clusters.

Documentation on how to deploy Grafana on Kubernetes here.

Distributed tracing for testing with Grafana Tempo and Tracetest (Grafana Office Hours #05)

Fri, 14 Jul 2023 20:21:30 +0200

In this video, my colleague Paul Balogh and I welcomed Daniel Mangum, Lead Cloud Engineer from Golioth. We learn how Golioth uses Grafana to report data from varied sensors and monitors.

Monitor behind a Firewall With Private Data Source Connect (Grafana Office Hours #04)

Fri, 07 Jul 2023 21:59:57 +0100

How do you monitor behind a firewall? With Grafana Cloud Private Data Connect, you can create a secure tunnel to query data sources, even including those in VPCs on public cloud vendors. Senior Software Engineers Stephanie Hingtgen and Georges Chaudy talk to me about how it works.

Recording a Performance Test With Grafana Cloud k6 (k6 Office Hours #93)

Fri, 30 Jun 2023 23:33:12 +0100

Senior Frontend Engineer Edvinas Daugirdas demonstrates recording a performance test with Grafana Cloud k6 and some other new features that have recently come to Grafana Cloud. He is joined by Developer Advocates Marie Drake and Nicole van der Hoeven to talk about getting started with k6 for free.

Changes to Grafana Plugins in Grafana 10 (Grafana Office Hours #03)

Thu, 29 Jun 2023 23:29:35 +0100

Grafana 10 is fresh off the presses, and it includes some important changes to Grafana plugins that any plugin developer or user will want to know. Senior Product Manager David Harris chats to me about all the new features available to plugins (including the new Scenes framework) and some breaking changes to legacy plugins.

What Is Grafana Scenes? with Torkel Ödegaard (Grafana Office Hours #02)

Mon, 19 Jun 2023 23:03:43 +0100

In GrafanaCON 2023, the latest version of Grafana 10 was announced, and with it, Scenes. But what is Grafana Scenes? Torkel Ödegaard, creator of Grafana and co-founder of Grafana Labs, joined me to discuss what Scenes is and how you can use it to create dynamic dashboards like never before.

How to Avoid the Most Common Kubernetes Monitoring Mistakes

Tue, 16 May 2023 22:41:16 +1000

For ObservabilityCON on the Road Singapore 2023, I talked about some mistakes people make when monitoring applications within Kubernetes clusters. I also demonstrated Grafana’s Kubernetes Monitoring solution and how it can be a shortcut for engineers looking to get started quickly.

Resources

Slides here
Monitoring Kubernetes in general (notes)
The Grafana Kubernetes Monitoring feature (notes)

Frontend vs. backend: How to plan your performance testing strategy

Mon, 03 Apr 2023 13:20:53 +0100

Originally posted here.

There are many aspects of application performance, but they broadly fall into two categories: frontend performance and backend performance. As a tester, it’s important to know the differences between the two and how that impacts the way you approach your tests.

In this blog, I’ll provide a high-level overview of frontend performance testing and backend performance testing, including pros and cons of each one. I’ll also explain why you need both if you really want a holistic view of how your application is performing.

Why (and where) to use frontend performance testing

Frontend performance testing verifies user experience at the interface level. Basically, this entails everything a user sees when they access your web app, whether it’s on a laptop or other devices. This might involve automating user interactions with the site through the browser — for example, clicking on a button, filling out a form, or having scripts that load on the browser and run.

In frontend testing, you’re looking at how components of the UI respond.

The metrics we analyze for frontend performance tests are typically quite different from what we use for backend testing. Let’s say we want to know the time it takes for a page to render after a user clicks on a button that directs them there. This isn’t the same thing as measuring server response time; that total round trip takes a lot more into account — things like latency or the time it takes for a client-side script to run and display the requested information.

Advantages of frontend performance testing

The great thing about frontend performance testing is that it provides a more realistic measurement of your end users’ experiences. They don’t care what’s going on under the hood; they care about the overall experience — and that’s exactly what frontend performance testing measures. Unlike backend performance testing, frontend performance testing looks at everything from a user’s point of view: from the time they visit a webpage to the time they can interact with something on that page.

On top of that, most (about 80%) of the performance issues end users report are frontend issues. It doesn’t make much sense to skip this type of testing; it can be a critical step in finding ways to improve the overall user experience.

Disadvantages of frontend performance testing

The most obvious disadvantage of frontend testing is that it doesn’t look under the hood. If you only do frontend testing, you’re almost certain to miss something. Maybe a component isn’t working, which is creating issues for your users. But you wouldn’t know that because you’re only testing what’s visible from the browser. In other words, you might be in the situation where you know something in the application is wrong and responding too slowly, but you don’t know what.

Frontend performance testing can also be quite expensive compared to, say, protocol-based testing. That’s because you need enough resources (memory and CPU) to actually spin up a browser instance and interact with your application the way that a real user would. The same load generator that could run tens of thousands of backend virtual users may only be able to run 50 frontend virtual users before running out of resources.

Finally, only testing frontend performance can sometimes lead to pretty misleading results, especially as you scale. As load increases, the response time for the frontend remains fairly constant, but that’s not the case with the backend.

While the frontend component of response time remains more or less constant, the backend component of response time increases exponentially with the number of concurrent users.

This phenomenon occurs because most of the issues that frontend performance tests might identify, such as a slow-running client-side script, don’t tend to compound. It’ll be just as slow on each user’s computer regardless of how many users are running it. However, problems uncovered by backend performance testing are on the side of the application, not the users. If a database can’t keep up with 100 users sending queries to it, then 1,000 users sending it queries is likely to yield an even worse response time.

Why (and where) to use backend performance testing

As you might have guessed at this point, backend performance testing targets the infrastructure under the hood. This could be a database that contains customer information, a payment gateway, or an authentication service for customers that have already signed up. These are things that an end user doesn’t see and maybe isn’t ever even aware of, but they still have an effect on response times.

Backend testing addresses infrastructure users might not even be aware of, like different databases, gateways, and services that underpin an application.

Scripting against the backend is quite different from scripting against the frontend. Instead of scripting user interactions like clicking, you’re scripting certain requests that are being sent on the protocol level. So, for example, you could have an HTTP GET request for the page and also for its embedded resources.

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
 http.get('https://test.k6.io');
 sleep(1);
}

The sample Grafana k6 script above is an example of a simplified backend performance test. It makes a single HTTP request to retrieve the contents of https://test.k6.io, and then waits one second.

Backend performance includes things beyond just load times and latency. It includes scalability, availability, reliability, and similar attributes, so the metrics that we measure are different as well. We might look at server uptime or response times per component. We could also trace the path of a particular request through the application stack.

An illustration shows that performance testing can address scalability, elasticity, availability, latency, resiliency, and reliability. Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Advantages of backend performance testing

As we already discussed, frontend testing doesn’t cover everything, but it’s also more complicated than that. The 20% or so of performance bottlenecks that are backend-related can be the hardest to troubleshoot, identify, and fix. Even worse, a lot of backend performance issues can spiral out of control at higher levels of load and can be quite expensive to fix later on in a development cycle. The worst time to realize that you need to rethink your entire infrastructure is after it’s already failed in production. If you take care of backend performance testing earlier (commonly known as shifting left), it makes such pivots less costly.

Another advantage of backend testing is the ability to target different components of an application. You can choose which API endpoint to hit and how hard, and you can target specific user flows, so it doesn’t always have to be end-to-end testing.

And finally, backend performance tests tend to be less resource-intensive; they’re just protocol-level requests that can be easily ramped up.You don’t need to spin up a browser instance, or even parse through the Document Object Module (DOM) of a webpage or look for specific elements. It’s just a matter of sending these requests, and that’s something that you can increase the scale of pretty cheaply.

Disadvantages of backend performance testing

Backend performance testing won’t tell you anything about what’s happening on the frontend, and a lot of sites these days actually use scripts that have to run on the frontend. So with backend performance testing tools, you just download those scripts. But the response time they measure is the download time, which is not really realistic because those scripts are executed. A backend performance test would fetch the text of the script. A frontend performance test would run it — and that script might do something like perform a calculation. You need frontend performance testing to measure that properly.

Backend performance testing is also much more expansive than front end performance testing, especially with microservices-based applications. There are so many components to track that it can be difficult to determine where to look. You can quickly find yourself going down a rabbit hole of constant, never-ending performance tuning.

Frontend + backend testing for the win

So, as we’ve seen, frontend testing and backend testing both seem to have advantages and disadvantages. So which one should you use? Both, if at all possible! They look at different parts of application performance, so doing both is essential if you want a holistic view of user experience.

Software teams can use Grafana k6 to address frontend and backend performance testing.

If you’re operating at a smaller scale, use your own discretion about how to find the right mix for needs. But if you’re testing at a larger scale with thousands of users and you want to test both the frontend and backend aspects of application performance, I suggest you change your test strategy so the majority of the load is generated with backend performance testing scripts. That’s because of the resource intensiveness of frontend performance testing scripts, which we discussed earlier.

So, if you generate most of the load with the backend and then you have a handful of frontend performance testing scripts running in the background, you should still get a good idea of how both are performing. Remember, frontend application performance remains fairly constant throughout different levels of load, so you’ll still get a lot of useful insights on both types of performance without worrying too much about the scaling costs.

Frontend and backend performance both affect what your end users think of your application. If the underlying application servers respond immediately to all requests, but then a huge picture of a crocodile makes your homepage excruciatingly slow to load, then users aren’t going to like that.

On the other hand, if your homepage is really sleek and quick to load, but the application server hasn’t responded with the data that the browser needs to render, then users aren’t going to like that either.

A common problem when testing both frontend and backend performance is script duplication: because of the differences in the two approaches, you often have to use two separate tools — which means two test suites and potentially two scripting languages. While this is a valid approach, it’s more sustainable to look for a tool that can target both types of performance, such as k6.

Whichever tool(s) you end up using, testing both frontend and backend performance is going to give you the clearest view of how your application really performs.

The easiest way to get started with k6 is through Grafana Cloud k6. The free tier includes 500 virtual user hours/month for k6 testing and more. Sign up for free now!

Grafana Cloud k6 (k6 Office Hours #83)

Fri, 24 Mar 2023 18:00:00 +0100

With the release of Grafana Cloud k6, you can get observability and performance testing in the same platform. Senior Product Manager Mark Meier and Frontend Lead Simon Legander join me to talk about what Grafana Cloud k6 is, how you can use it, and demonstrate what it can do to improve your system’s reliability.

1 Year Ago We Were Acquired. Here's what happened next. (k6 Office Hours #58)

Fri, 29 Jul 2022 16:12:34 +0100

In June 2021, CEO Raj Dutt announced that k6 had become part of the Grafana Labs family, and the year that followed has been replete with changes in features, organizational structure, and overall culture within the k6 team. In this video, Nicole van der Hoeven and Paul Balogh reflect on everything that’s happened in the last year and how k6 has fared 1 year after the acquisition.

What Is Grafana Mimir? (k6 Office Hours #53)

Fri, 06 May 2022 17:00:28 +0200

Maintainer Marco Pracucci talks about the new Grafana project, Mimir, a next-generation time-series database for Prometheus. He joins Developer Advocates Nicole van der Hoeven and Paul Balogh to talk about what Mimir is, why it was needed, and how he helped scale it using k6.

Work week in the life of a Developer Advocate

Wed, 03 Nov 2021 00:31:46 +0000

Before I became a Developer Advocate, I didn’t really know what developer advocates did - so here’s my attempt at explaining what it is I do for a living!

Weekly breakdown

Here’s a list of my activities during this week:

Friday
- Filmed new ending for video #1
- Figured out integration options
- Did k6 Office Hours
- Prepared for Clubhouse talk
- Participated in customer demo
Saturday
- Did post-production for k6 Office Hours
- Talked at a Clubhouse meetup
Sunday
- Wrote (team profile) blog post #1
- Wrote talking points for Racket
- Started writing blog post #2
Monday
- Edited video #1
- Created thumbnail, description
- Scheduled video #1
Tuesday
- Created slides for presentation
- Played with Prometheus and Grafana for webinar
- Had a virtual coffee with colleagues
- Finished writing blog post #2
Wednesday
- Went to four meetings
- Caught up on emails, Slack, and social media (last night)
- Wrote blog post #3
Thursday
- Scripted and filmed videos #2 and #3
- Started editing video #2
- Talked to a potential guest on livestream

Grafana Labs on Nicole van der Hoeven

AI Tools We Use Every Day on the Grafana AI Team

Resources

What Does a Grafana AI Offsite Look Like?

Resources

Roundtable: How We Really Use AI in Testing (HUSTEF 2026)

Resources

Asimov's Zeroth Law of Robotics: Testing for AI (HUSTEF 2026)

Resources

Context Engineering Fundamentals (The Context Window #06)

What we cover

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana GCX

Transcript

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana MCP

Transcript

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Transcript

Resources

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Transcript

Observability and AI explained

Resources

AI Week Day 4: Evaluating (Grafana AI Week — Thursday)

Transcript

Investigations Arena: how we test Assistant investigations

Transcript

Resources

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Resources

Related reading

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Transcript

AI Week Day 2: Building (Grafana AI Week — Tuesday)

Transcript

AI Week Day 1: Planning (Grafana AI Week — Monday)

Transcript

An Overview of AI Evaluation (The Context Window #05)

The “final answer problem”

What evals are, and online vs offline

How we test Grafana Assistant

o11y-bench

The evaluation loop

Where it’s going

Resources

Related reading

More from The Context Window

Transcript

OpenTelemetry's Graduation: What It Took (Grafana ❤️🔥 OpenTelemetry Community Call #8)

What we covered

🎓 What graduation actually took

📚 Documentation that’s actually findable

🏷️ Why semantic conventions are so hard

🧳 Signals, baggage, and stability

🤖 AI’s impact on OpenTelemetry — the good and the painful

🚀 Post-graduation priorities and how to get involved

Resources

CLI and MCP Basics for AI Agents (The Context Window #04)

Timestamps

Resources

MCP and CLI for agents

Related reading

More from The Context Window

Grafana Campfire 🔥 — GrafanaCON 2026 recap

Resources

Asimov's Zeroth Law of Robotics: Observability for AI (ExpoQA Madrid 2026)

Resources

Zero to Dashboard with Grafana Assistant and the Infinity datasource plugin

Why Infinity is fun here

Timestamps

Resources

Continuous Profiling, the OpenTelemetry Way (Grafana ❤️🔥 OpenTelemetry Community Call #7)

What we covered

🔥 Profiling as the fourth signal

📊 What flame graphs actually tell you

🌐 The current state of profiling in OpenTelemetry

🚀 Running profiling in production

🧵 Correlating profiles with traces and metrics