Grafana on Nicole van der Hoeven

Spice Runner

Wed, 05 Nov 2025 00:00:00 +0000

Spice Runner is a browser-based endless runner. Think of the “unable to connect” dinosaur game in Chrome, except you are a Fremen running across the sands of Arrakis, jumping sandworms and Harkonnen ornithopters. It’s the demo app for my talk The Spice Must Flow: The Fremen Guide to Sustainable Observability, and it exists to make one point: even a simple, throwaway game generates a surprising amount of telemetry, and that telemetry has real financial and environmental costs.

The Fremen live on a planet that is rich in spice and poor in water, and they thrive by wasting nothing. Our systems are rich in telemetry and poor in budget, and in carbon. Spice Runner is a small, playable version of that problem: a real app, with real players, running on real infrastructure that scales up and down with them while I watch what it costs.

What it demonstrates

Real user monitoring with Grafana Faro. Game sessions, game starts, scores, collisions, restarts, and game overs are all sent as custom events, so the dashboard shows how people actually play.
A leaderboard that’s instrumented end to end. A Go API backed by PostgreSQL and Redis serves a public top-10 leaderboard. It’s instrumented with OpenTelemetry for traces and custom metrics, so a score can be followed from the browser through the API to the database and the cache.
Metrics, logs, and traces on Kubernetes. Prometheus, Grafana Loki, and Grafana Tempo run in the cluster, with Grafana Alloy as the collector and Grafana for dashboards. Telemetry is also sent to Grafana Cloud.
Autoscaling that follows the players. KEDA scales the game’s pods on HTTP request rate, CPU, and memory, and the GKE Cluster Autoscaler adds and removes nodes behind them. Production settings keep at least one pod running and cap it at 20, with a separate demo profile that is far more eager for live demonstrations.
Energy monitoring with Kepler. Power and energy use are tracked per workload, so “what does this cost the planet?” is a query rather than a guess.
Load testing with k6. k6 drives load against the leaderboard API and, with a browser script, plays the game to keep the dashboard populated.

The dashboard

The live dashboard on Grafana Play shows real player data as it happens. The dashboard is also entirely AI-generated: it was built with Grafana Assistant.

Open the live dashboard

Architecture

The app has two layers. The first is the game and its leaderboard: a vanilla JavaScript game served by Nginx, with a Go leaderboard API backed by PostgreSQL and Redis. The second is the observability stack: Faro instruments the browser, and the Nginx exporter, Alloy, Loki, Tempo, and Prometheus cover the backend. Everything runs on Google Kubernetes Engine, with persistent volumes so telemetry survives pod restarts, and autoscalers at both the pod and node level.

The talk

The Spice Must Flow uses Dune as its framing. It covers the hidden environmental and financial costs of an observability stack, how hard it is to measure the cost of ephemeral resources on Kubernetes, how to use KEDA to right-size resources when you don’t know what the right size is, and the most effective thing you can do to cut cost: deleting the telemetry you don’t need.

I first gave it at Øredev 2025 in Malmö, Sweden. You can read the write-up and abstract and see the slides.

Run it yourself

Everything you need to deploy it on your own GKE cluster, including the Kubernetes manifests, the dashboards, and the load tests, is in the GitHub repository. Its README has a quick start.

The Defiant Jazz Incident

Fri, 30 Oct 2026 00:00:00 +0000

The Defiant Jazz Incident is a murder mystery you solve with observability tools. It’s set on the Sleep Floor at Kipsake, a company that rents your sleep: at night, its Dreamers sleep in pods while their sleepwalking selves work the floor. During the Cocoa Hour, the floor’s middle-of-the-night holiday party, the power goes out for five minutes. When it comes back, the night supervisor is dead in the elevator.

Kipsake recorded everything it could: badges, doors, chat, sleep rings, power, microphones, cameras, sleep bands, and personnel records. Your job is to work out who did it, and the only way to do that is through the telemetry.

How it works

The case file is a set of logs, metrics, and traces. A loader sends them to Grafana Loki, Prometheus or Mimir, and Grafana Tempo, either in Grafana Cloud or in a local Docker stack, and you investigate from there with whatever you like: Grafana itself, Grafana Assistant, an MCP server, gcx, or the AI agent of your choice. The data is deliberately messy. Some devices lose clock sync during the outage, some records arrive late, and not everyone on the Sleep Floor remembers what their body did.

The game itself comes in two forms. A terminal version is for demos and playtests: you watch the testimonies and choose what to investigate, while an AI that only sees the telemetry answers your questions. A web version is made to embed in a Grafana dashboard panel right next to the data.

Why a murder mystery?

Investigating an incident is investigating an incident, whether the victim is a production service or a night supervisor. You start with a vague report, a pile of signals that don’t quite agree, and a clock that can’t be fully trusted, and you have to build a timeline you can defend. A case with a known answer is a safe way to practice that, to compare how different tools and AI agents cope with it, and to see how good a question you have to ask before the data answers.

No spoilers

The repository keeps the solution in clearly marked files, and the README tells you which ones to avoid if you want to play. This page deliberately says nothing about who did it.

Try it

The repository has everything: the case brief, the telemetry, the loader, and both versions of the game.

obsidian-grafana

Mon, 13 Oct 2025 00:00:00 +0000

obsidian-grafana turns an Obsidian vault into time-series data and dashboards. I work at Grafana Labs, helping people monitor their systems with open source tools, and at some point I realized I could use the same tools to monitor my personal knowledge management system. It’s a nerdy and totally unnecessary setup, and I enjoy it a lot. I mention it in my post on how I use visuals in my PKM.

The screenshot above was made by running the repo’s own Grafana setup on a copy of the posts from this website.

How it works

A Python script reads the Markdown files in your vault and extracts metadata: word, line, and character counts, file size, frontmatter fields, tags, wikilinks, and timestamps. It runs every five minutes and has two paths:

A log path for history. Only the notes that changed since the last run are written, as structured JSON, to a log file. Grafana Alloy tails that file and forwards it to Loki, with tags and frontmatter fields as labels, so you can see how your notes change over time.
A metrics path for the current state. A metrics exporter exposes totals such as note count, word count, unique tags, and wikilinks, and Prometheus scrapes them.

Grafana ties the two together with pre-configured dashboards: an overview, a time series dashboard for individual notes, a relationships dashboard for wikilinks, a dashboard of frontmatter types, and a word cloud of your most used words. Everything runs in Docker Compose.

What it’s useful for

Understanding your writing habits and how your vault grows, checking how consistently your notes are tagged, finding the notes that everything links to, and keeping an eye on notes with a given frontmatter status or priority. The README also covers installing the Grafana Assistant plugin into the local Grafana.

Try it

The repository has a quick start. You’ll need Docker and Python 3, and a vault to point it at.

Asimov

Wed, 02 Apr 2025 00:00:00 +0000

Asimov is the companion project to my talk Asimov’s Zeroth Law of Robotics. It’s an AI app built to be observed and tested: The Silent Enterprise, a single-player Star Trek text adventure. You play Data, and Claude is the game master. Every model call, dice roll, and turn shows up as telemetry in a Grafana that you can run on your own machine or in Grafana Cloud, and a k6 test suite, some of it judged by Claude, checks whether the AI follows the rules.

The game master has some known flaws, kept on purpose. It sometimes narrates a rescue that didn’t happen, or “destroys” a drone that’s still standing. The point of the demo is catching those flaws with telemetry, tests, and online evaluators.

The idea

Isaac Asimov’s three Laws of Robotics govern how robots behave. The talk argues for a law that comes before all of them: a robot must be observable, and, in its newest version, testable. AI systems fail differently from traditional software. There’s no single correct output to assert against, a response can be fluent and completely wrong, and models drift. You can’t test what you can’t see, and you can’t trust what you can’t test.

What’s in it

The game. A Go program with a command-line interface and an HTTP API. The game engine owns the rules and the state. Claude interprets the player’s input, makes the game master’s own rolls through a tool, and narrates, so a narration that contradicts the saved state is a defect that can be caught.
Telemetry. Traces, metrics, and logs go over OpenTelemetry to a local Grafana stack (Tempo, Prometheus, and Loki, with the game’s dashboard already loaded) or to Grafana Cloud. Generations can also go to Grafana Cloud’s Agent Observability, where online evaluators judge a sample of them.
Tests. A k6 suite with code checks, LLM judges, dice-trajectory evals, and whole playthroughs. All of it can run for free against a mock game master and judge.
No account needed to start. You can play offline with no API key, and see the local monitoring stack with one command and Docker.

The screenshot above is the game’s local Grafana dashboard.

The talks

I’ve given the talk, in several versions, at KubeCon Europe 2025, Dutch Cloud Native Day 2025, Newcrafts 2025, ExpoQA Madrid 2026, and HUSTEF 2026. The write-ups are linked below. The earlier talks used a Python and Flask version of the app, which has since been rewritten in Go.

Try it

The repository has a quick start for playing offline, a Docker-only path, and release binaries for Windows, macOS, and Linux.

When to use Grafana Assistant vs MCP vs GCX: Grafana GCX

Fri, 21 Aug 2026 09:00:00 +0100

Three AI tools people confuse: Assistant, MCP, and GCX. Last time I talked about the easy hand — the MCP server. Now the power hand: GCX, a CLI built for AI agents.

If Assistant is the brain and Grafana MCP is the easy hand, then GCX is the power hand — a CLI built for an agent to drive. (Quick refresher: a CLI is a command line interface — no clicking, just typed commands.)

So when does GCX win?

When you’re using terminal coding agents — Claude Code, Cursor, Codex. It writes code, pulls the exact log or metric, and verifies against it — without leaving the editor.
When you need everything. GCX wraps the entire Grafana Cloud API — SLOs, Synthetics, IRM, k6, Fleet — the esoteric corners MCP skips. The full banana.
When you want automation and CI/CD. Dry-run mode, JSON output, and exit codes an agent can act on — built for GitOps.

Do you need Grafana Assistant for this? No — not even a subscription. Bring your own brain, same as MCP: metrics, logs, dashboards, and alerts run on nothing but a service account token. You only really pay for Assistant if you call gcx assistant directly — which taps its reasoning, billed separately.

So there’s your map: Assistant’s the brain, MCP’s the easy hand, GCX is the power hand. It’s not one versus another — they work really well together to give you the best choice in how to access your Grafana stack.

Transcript

I’m differentiating Assistant, Grafana MCP and GCX and in this one I’m talking about GCX. If Assistant is the brain and Grafana MCP is the easy hand, then GCX is the power hand — a CLI built for AI agents.

Quick refresher: a CLI is a command line interface — no clicking, just typed commands. GCX is a CLI built for an agent to drive.

So when does GCX win?

When you’re using terminal coding agents — Claude Code, Cursor, Codex. It writes code, pulls the exact log or metric, and verifies against it — without leaving the editor.
When you need everything. GCX wraps the entire Grafana Cloud API — SLOs, Synthetics, IRM, k6, Fleet — the esoteric corners MCP skips — the full banana.
When you want to do automation and CI/CD. Dry-run mode, JSON output, exit codes an agent can act on — built for GitOps.

Do you need Grafana Assistant for this? No — not even a subscription. Bring your own brain, same as MCP: metrics, logs, dashboards, alerts run on nothing but a service account token. You only really pay for Assistant if you call gcx assistant directly — which taps its reasoning, billed separately.

So there’s your map: Assistant’s the brain, MCP’s the easy hand, GCX is the power hand. It’s not one versus another. They actually work really well together to give you the best choice in how to access your Grafana stack.

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana MCP

Wed, 19 Aug 2026 09:00:00 +0100

Three AI tools people confuse: Assistant, MCP, and GCX. Last time I talked about the brain — Assistant. Now hand number one: the MCP server, the easy hand.

Grafana MCP — Model Context Protocol — lets any AI client reach into Grafana. The magic part: you bring your own brain.

Using MCP requires two pieces:

The client is your AI app — the brain you bring: claude.ai, ChatGPT, Cursor.
The server is the door into Grafana that exposes the tools.

There are two ways to get that server. The easiest is the Grafana Cloud hosted one — already running, auth built in, just connect and go. But you can also self-host the open-source one on your own infra.

So when do you reach for Grafana MCP?

When you’re using hosted chat tools with no terminal — claude.ai, ChatGPT, your phone. No shell means no CLI, so MCP is the only way in.
When you want to bolt Grafana onto a tool you already use — already in Cursor or Claude? Just add the server. No new workflow.
When you want the best discovery out of the box — the tools describe themselves up front, so even smaller or cheaper models just know what’s available. No trial-and-error.

One honest catch: those tool definitions can eat some context if you load everything — but modern hosts fix that with lazy tool search.

MCP is deliberately curated — it doesn’t expose every corner of Grafana. So what do you do when you need the esoteric stuff? That’s exactly what the power hand, GCX, is for — next up.

Transcript

Three AI tools people confuse: Assistant, MCP, GCX. Last time, I talked about the brain — Assistant. Now hand number one: the MCP server, the easy hand.

Grafana MCP — Model Context Protocol — lets any AI client reach into Grafana. The magic part: you bring that AI client - you bring your own brain.

Using MCP requires two pieces. The client is your AI app — the brain you bring: claude.ai, ChatGPT, Cursor. The server is the door into Grafana that exposes the tools.

Two ways to get that server: the easiest way is to use the Grafana Cloud hosted one — already running, auth built in, just connect and go. But you can also self-host the open-source one on your own infra.

So when do you reach for Grafana MCP?

When you’re using hosted chat tools with no terminal — claude.ai, ChatGPT, your phone. No shell means no CLI, so MCP is the only way in.
When you want to bolt Grafana onto a tool you already use — already in Cursor or Claude? Just add the server. No new workflow.
When you want the best discovery out of the box — the tools describe themselves up front, so even smaller or cheaper models just know what’s available. No trial-and-error.

One honest catch: those tool definitions can eat some context if you load everything — but modern hosts fix that with lazy tool search.

But MCP is deliberately curated — it doesn’t expose every corner of Grafana. So what do you do when you need the esoteric stuff? That’s exactly what the power hand is for. Next up is GCX.

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Mon, 17 Aug 2026 09:00:00 +0100

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. They’re not competitors. Think of it as a brain and two hands. In this short, I start with the brain: Grafana Assistant.

Assistant isn’t just a chatbot — it’s the reasoning layer. It manages context and knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. Instead of clicking around, you tell Assistant what you want to do and it figures out the how.

Reach for the Assistant when the work is visual or exploratory:

Building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
A real incident that needs root-cause analysis across multiple data sources? Assistant. That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in, and I’ll cover the first of those (the MCP server) next.

Transcript

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. I’m going to talk a bit about each one so you know which one is best for your use case.

Spoiler: They’re not competitors. Think of it as a brain and two hands.

Let’s start with Grafana Assistant, which is the brain.

It’s not just a chatbot — it’s the reasoning layer. It manages context, it knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. So instead of you clicking around to do something, you can say what you want to do to Assistant and it figures out the how.

Reach for the Assistant when the work is visual or exploratory.

If you’re building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
You have a real incident where you need root-cause analysis across multiple data sources? Assistant.
- That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in… and I’ll show you those next.

Resources

Grafana Assistant (product page)
My notes on Grafana Assistant
My notes on GCX
Unlearnings from building Grafana Assistant — my Context Horizon piece

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Thu, 30 Jul 2026 09:00:00 +0900

This week I spoke at KubeCon + CloudNativeCon Japan 2026 in Yokohama, Japan. I gave a talk called The Great Doubt: What Building an AI Agent Taught Us About Trust. Here’s the abstract:

When we started building an AI assistant for observability, we thought the hard part would be making it smart. We were wrong. The hard part was knowing when to trust it.

AI agents hallucinate, forget context, and confidently give wrong answers. Traditional testing doesn’t catch these failures. What’s needed is evaluation grounded in systematic doubt.

This talk shares lessons from shipping an AI agent to production: how to build a golden dataset of use cases you must get right, how to use LLM-as-judge when there’s no ground truth, and how to use OpenTelemetry traces to debug eval failures. You’ll also hear what’s still unsolved: evaluating multi-agent handoffs and closing the feedback loop between what users ask and what your evals cover.

Kyoto School philosopher Nishitani Keiji called this “The Great Doubt” (大疑): questioning every assumption until only what survives is real. For AI agents, that’s not philosophy. It’s the job.

The talk walks through the layers of doubt we worked through while building Grafana Assistant: doubting the agent itself, then doubting our tests, then the LLM-as-judge, then our own scores — and finally doubting whether the doubt was even justified. Each layer needed a different tool, from a golden dataset run in a fully synthetic observability environment, through online evals on live traffic, to per-conversation OpenTelemetry traces that link every eval score back to the exact LLM and tool calls that produced it.

You can find the slides at nicole.to/doubtslides, and you can watch the full talk below:

Resources

Grafana Assistant (generally available for Grafana OSS, Grafana Cloud, and Grafana Play):

Assistant Investigations (Grafana Cloud):

Docs

Agent Observability (Grafana Cloud):

Docs

o11y-bench (OSS):

Site

Investigations Arena:

Video

My notes on Grafana Assistant
My Context Horizon piece Unlearnings from building Grafana Assistant
The Way of Zen by Alan W. Watts
Zen Mind, Beginner’s Mind by Shunryu Suzuki
Religion and Nothingness, Volume 1 by Nishitani Keiji
Philosophers of Nothingness: An Essay on the Kyoto School by James W. Heisig

Grafana Campfire 🔥 — GrafanaCON 2026 recap

Thu, 28 May 2026 17:45:00 +0100

I co-hosted this month’s Grafana Campfire with Stephanie Hingtgen, where we recapped everything that came out of GrafanaCON 2026. Stephanie owned the Grafana 13 walkthrough — Git Sync GA, Dynamic Dashboards GA, Dashboards API v2 GA, Grafana Advisor GA, SCIM GA — what she described as the year Grafana stopped being beta in production. I covered the AI section (Grafana Assistant now in OSS and Enterprise, the three scaling challenges from “Scaling Grafana Assistant”, Investigations, Skills going GA), the three-teams-one-architecture push across Loki Thor, Mimir 3.0, and Pyroscope 2.0, the OTel + Alloy roadmap, alerting, k6 v2.0, and the Marketplace and Logline.

We also took questions from the Champions community on the pace of UI change, Mimir Alertmanager, and the agent-scale question — what happens when your AI agents are themselves the noisiest users of your observability stack? — and threw in a flavour break about the TamaGROTchi and the ocean digital twin demo.

Resources

Asimov's Zeroth Law of Robotics: Observability for AI (ExpoQA Madrid 2026)

Tue, 26 May 2026 10:00:00 +0200

This week I gave the opening keynote at ExpoQA Madrid 2026 in Madrid, Spain. I presented a talk called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously given at KubeCon EU 2025 in London, at Dutch Cloud Native Day 2025 in Utrecht, and at Newcrafts in Paris. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

This was a keynote slot at a testing-focused audience, which gave me the chance to lean more into the evals vs. observability framing — what evals can and can’t tell you about an AI system in production, and how distributed tracing fills the gap that benchmarks leave behind. The demo app (a Flask-based “play.py” instrumented with OpenTelemetry, flowing into Tempo for traces, Prometheus for metrics, and Loki for logs) is still on GitHub, and I’ve been refining it with each iteration.

Here’s the recording of the talk:

Resources

Demo app on GitHub

Zero to Dashboard with Grafana Assistant and the Infinity datasource plugin

Fri, 22 May 2026 17:00:00 +0100

What if you didn’t have to write a single query to get a useful dashboard?

In this short video, I go from nothing to a working dashboard in a few minutes — no queries — using Grafana Assistant and the Infinity datasource plugin. The data I’m visualising is the RAWG video game database API, so I end up with a dashboard that helps me decide what to play next.

Why Infinity is fun here

Most Grafana data sources are scoped to a specific kind of backend — Prometheus, Loki, a SQL database. Infinity is different: it lets you treat any REST/JSON/CSV/HTML endpoint as a data source. That means anything with an API can become a dashboard, which is exactly the right shape for goofing around with a games API on a Friday afternoon.

Pair Infinity with Assistant and you don’t even have to know what queries to write — Assistant configures the data source, asks the API the right questions, and builds the panels for you.

Timestamps

00:00 — Intro
00:35 — The RAWG video game database
01:01 — Setting up the Infinity datasource
05:10 — Customising the dashboard with Assistant
09:31 — Closing thoughts

Resources

Step-by-step blog post: using Assistant with the Infinity datasource
Grafana Assistant documentation
Infinity datasource plugin
RAWG video game database API
Get started with Grafana Cloud (free tier)
My notes on Grafana Assistant — everything I’m learning as I work on it

How to use Grafana Assistant with self-hosted Grafana

Mon, 20 Apr 2026 09:00:00 +0100

In case you missed it, possibly the biggest news to come from GrafanaCON this year is that if you’re running self-hosted or open-source Grafana, you can now connect it through your Grafana Cloud account to Grafana Assistant, the purpose-built AI tool for all things Grafana.

In this short tutorial, I show you how to enable Assistant on a self-hosted Grafana instance and go from nothing to a working server-health dashboard with a single prompt.

A quick note on what “for open source” actually means

Assistant itself isn’t open source. But you can now use it with your self-hosted or open-source Grafana instance. The way it works is:

You still need a Grafana Cloud account (the free tier has generous inclusions for Assistant).
You link your self-hosted Grafana to that Cloud account.
Assistant then lights up inside your OSS instance.

Think of the Grafana Cloud account as the key that unlocks Assistant wherever your Grafana lives.

Prerequisites

Grafana OSS v13 or later
A Grafana Cloud account (pricing here)

Enabling Assistant

Go to Administration → Plugins, search for Assistant, and click Install.
Refresh the page, then open the Connection tab.
Click Start Connection. You’ll be prompted to log into Grafana Cloud.
Select the organization and stack you want, then click Authorize connection.
Close the tab. Your self-hosted Grafana is now linked to Assistant.

A first dashboard from nothing

Every Grafana instance ships with a TestData data source — perfect for a first run when you don’t yet have real data wired up. Enable it from Data Sources, then open Assistant (the new icon on the top right).

Switch Assistant’s mode from General to Dashboarding — this mode is tuned for creating and managing dashboards — and ask in plain English, for example:

Hi Assistant, could you create a server health dashboard for me using the TestData data source?

What comes back is a starting dashboard with CPU, memory, disk, and network panels already wired up. From there you can tweak panels, edit queries, or ask Assistant to iterate. You’re not starting from scratch — you’re starting from 80%.

What else Assistant can do

I only showed dashboarding in this video, but Assistant can also:

Answer questions about the data already in your Grafana
Run investigations to troubleshoot production incidents
Learn new skills
Manage alerts
Teach you Grafana

That list keeps growing, so stay tuned.

Resources

Grafana Assistant (product page)
Grafana Cloud pricing
How Grafana Labs handles your data
My notes on Grafana Assistant — everything I’m learning as I work on it
Unlearnings from building Grafana Assistant — my Context Horizon piece on what the team had to unlearn

Let There Be Dashboards: Introducing Grafana Assistant

Thu, 26 Mar 2026 19:30:00 +0100

I gave this talk at the Grafana & Friends Lisbon Meetup in Lisbon, Portugal.

Abstract

This is a live demo of how to go from zero to a working Grafana dashboard using Assistant. Starting with a blank slate, we’ll use natural language to explore data and generate meaningful visualizations in minutes. Along the way, you’ll see how AI can accelerate dashboard creation without sacrificing control or understanding.

Thanks to everyone who came out! It was great to connect with the Lisbon tech community.

Asimov's Zeroth Law of Robotics: Observability for AI (Newcrafts 2025)

Thu, 06 Nov 2025 12:23:19 +0100

Today I presented at Newcrafts in Paris, France. I gave a talk I’ve given before, called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU in London and at Dutch Cloud Native Day 2025 in Utrecht. I was happy to present it again, as it gave me the chance to further refine my demo application, which you can find on GitHub.

The main change I made was to use the “AI as judge” approach that I had previously only talked about in theory. In this version of the demo app, I use AI in both the game as well as in the testing of the game, showing how AI can be integrated into both the application and its observability/testing stack. It was really interesting to note that testing with AI ended up exposing more defects than my hardcoded test cases did, which goes to show the potential of AI-driven testing.

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

Wed, 05 Nov 2025 03:07:38 +0100

Today I gave a talk at Øredev 2025 in Malmö, Sweden, titled +The Spice Must Flow: The Fremen Guide to Sustainable Observability"_. The presentation focused on how to implement sustainable observability practices using open-source tools like Grafana, Prometheus, Loki, Tempo, KEDA, and Kepler. Here’s the abstract:

Nobody wants to be wasteful. But how do you balance that with the need for enough data to fix things when they go wrong? After all, no matter how much it costs, there are SLOs to keep. The spice– the telemetry– must flow.

The Fremen are a people who live on the planet Arrakis, simultaneously rich in spice while poor in water. But the Fremen have managed to thrive, not just survive, on very little while continuing to produce spice. What would it take to apply this “desert planet thinking” to observability?

In this talk, you’ll learn about the hidden environmental and financial costs of your observability stack and how to reduce them. You’ll learn about the complications of measuring cost for ephemeral resources on Kubernetes, how to use KEDA to “right-size” resources when you don’t know what the right size is, and how to convince your teams to drop all but the most meaningful telemetry data. In the end, the spice must flow, but there must still be enough water to sustain all.

Here are some resources mentioned in the talk:

You can play the game Spice Runner here
The spice-runner repo containing everything you need to know about how to deploy it yourself
The slides
The Spice Runner dashboard for observing the application
More about Spice Runner: architecture, tech stack, and what it demonstrates

One Year Later: How I Use Visuals in My PKM (Sketch Your Mind Conference)

Mon, 13 Oct 2025 13:33:18 -0700

Today I gave a talk at my friend Zsolt Viczián’s Sketch Your Mind Conference about how I use visuals in my PKM, a follow-up to my video about Obsidian Excalidraw last year where I shared that I’d changed all my Obsidian templates to hybrid Obsidian Excalidraw notes:

In the talk, I shared how I’ve evolved my use of visuals in my PKM over the last year, including:

Referring to icon and logo libraries that dynamically compile all the atomic visuals I use in my notes
Adopting a visual Zettelkasten-like approach
Creating notes at conferences to track people I’ve met and talks I’ve attended
Using Obsidian Excalidraw to create compelling presentations in lieu of slides
Combining visuals with audio recordings to create literature notes of books I’m reading
Using a full observability pipeline to monitor and analyze the notes in my Obsidian vault

I was particularly excited about the last point. I work at Grafana Labs and help people monitor their computer systems using opensource software. I realized that I could use the same tools to monitor my PKM system, so I set up a pipeline where a Python script scrapes note metadata, and sends information to Loki and Prometheus via Alloy, and then I visualize it all in Grafana. It’s a pretty nerdy and totally unnecessary setup, but here’s the repo for anyone interested in replicating it.

All About Structured Metadata in Loki (Loki Community Call August 2025)

Tue, 12 Aug 2025 19:56:32 -0700

In this video, Jay Clifford and I talk to Senior Software Engineer Salva Corts Sánchez from the Loki team about structured metadata in Loki. We discuss what it is, why you might want to use it, and also how to query it. Structured metadata is especially useful if you’re instrumenting your system with OpenTelemetry, so we also touch on that.

Asimov's Zeroth Law of Robotics (Dutch Cloud Native Day 2025)

Thu, 03 Jul 2025 15:01:24 +0200

Today I presented at the Dutch Cloud Native Day 2025 in Utrecht, the Netherlands. I gave a talk titled Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU earlier on in the year, in London. I was very happy to be able to present it again, because it gave me the opportunity to improve the demo app I used, which you can find on GitHub.

State of the Observability Databases (Grafana Office Hours #30)

Thu, 15 May 2025 18:22:26 +0100

In this Grafana Office Hours, we talk about the state of observability databases (Grafana Loki, Mimir, Tempo, and Pyroscope) and where they’re going. We talk about current and upcoming architectural changes in all four, how we’re making them more performant, how compatible they are with OpenTelemetry, and what we’re working on next for each database. In this conversation are Dee Kitchen (VP of Engineering for Databases) and Senior Developer Advocates Jay Clifford and Nicole van der Hoeven.

Getting Started with Logging and Grafana Loki (webinar)

Wed, 16 Apr 2025 08:18:04 +0200

In this webinar, I talk about the key benefits of Grafana Loki, the open source log aggregation system from Grafana Labs. I cover the basics of logging, how Loki works, and how to get started with it. This was a live webinar, but I recorded the session here as well:

Asimov's Zeroth Law of Robotics: Observability for AI (KubeCon EU 2025)

Wed, 02 Apr 2025 13:46:42 +0200

Today I gave a talk at KubeCon EU 2025 in London called “Asmiov’s Zeroth Law of Robotics: Observability for AI”. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

Grafana Loki Query Best Practices with LogQL (Loki Community Call December 2024)

Thu, 05 Dec 2024 17:48:49 +0100

In this December’s Loki Community Call, Cyril Tovena, Senior Principal Engineer and LogQL guru walks us through a Grafana Loki query tutorial with LogQL, the Log Query Language used for Loki. He talks about the key “Dos and Don’ts” of LogQL, offering practical tips to help you write better queries, boost performance, and sidestep common mistakes.

Whether you’re tuning up your current setup or just diving into LogQL, Cyril’s got you covered. Bring your questions, connect with the Loki community, and pick up insights that’ll make your log-querying life easier. See you there!

Make sure you leave your questions here.

Cyril is joined by Senior Developer Advocates Jay Clifford and Nicole van der Hoeven.

AI Observability with Ishan Jain (Grafana Office Hours #29)

Fri, 08 Nov 2024 18:24:29 +0100

In this Grafana Office Hours, Ishan Jain talks about AI Observability with Grafana: what it entails, factors to consider when monitoring and observing LLMs, and how to do it all with Grafana. He is joined by Senior Developer Advocate Nicole van der Hoeven.

All About Logs Drilldown (Loki Community Call October 2024)

Thu, 03 Oct 2024 17:45:33 +0100

In this Community Call, Senior Software Engineer Trevor Whitney talks to us all about Drilldown Logs for Grafana Loki, an open-source app for visualizing logs from Loki in Grafana without needing to learn and write LogQL queries. He is joined by Senior Developer Advocates Nicole van der Hoeven and Jay Clifford.

All the Components of Loki Explained

Fri, 20 Sep 2024 18:09:35 +0100

Here are all the components of Loki explained so that you can figure out how many instances of each you need for a resilient Loki deployment. Senior Developer Advocates Jay Clifford and Nicole van der Hoeven talk you through what each component does and which ones are optional.

Bloom Filter Changes for Grafana Loki (Loki Community Call September 2024)

Thu, 05 Sep 2024 17:41:18 +0100

In this Community Call, Senior Software Engineer Christian Haudum talks to us about bloom filter changes for Grafana Loki, including the deprecation of the bloom compactor and a pivot towards creating bloom filters for structured metadata. Bloom filters are a probabilistic data structure that we’re using to improve query performance in Loki.

Community Calls are monthly meetings that are open to everyone interested in the development of Loki. They are an opportunity for software engineers working on Loki to discuss new features as well as for open-source users of Loki to ask questions. To participate in the next Community Call, subscribe to the calendar here.

API Monitoring with Traefik, Grafana, and OpenTelemetry (Grafana Office Hours #28)

Fri, 12 Jul 2024 18:27:03 +0100

Maytham Alfouadi (Solutions Architect) and Immánuel Fodor (Product Manager) from Traefik Labs give us a demonstration of how to do API monitoring with Traefik, Grafana, and OpenTelemetry. They talk about Traefik, an open-source reverse proxy, among other things, that is now included by default in k3s and Rancher. They are joined by Usman Ahmad and Nicole van der Hoeven, both Senior Developer Advocates at Grafana Labs.

Cartography of the Deep: Distributed Tracing for Testing (Romanian Testing Conference)

Thu, 13 Jun 2024 12:24:02 -0700

I presented a talk today at the Romanian Testing Conference 2024 in Cluj-Napoca, Romania! My talk was called “Cartography of the Deep: Distributed Tracing for Testing” and it covered how to tie in distributed tracing into tests to get a better understanding of your system under test.

Here are my slides!

Abstract

From the Scandinavian Kraken that drags entire ships to their early graves to the Philippine Bakunawa said to rise from the waters to swallow the moon, our existential fears about what lurks at the bottom of our own oceans has inspired us to create hundreds of folkloric monsters across cultures. We’ve charted the surface of our planet, but the vast majority of the ocean floor remains a mystery.

The mysterious depths of modern software architectures makes it difficult for us to test a system for performance and reliability. The easy solution? Unmanned underwater probes: send inputs to the system, and judge its state based on the outputs that return. But this method is often too limited to tell us what’s really going on when something goes wrong. What if there were a way to make our tests themselves more observable?

In this talk, you will learn how to use open-source tools to instrument performance tests, identify bottlenecks at the code level using distributed tracing, and map out a system using a type of software bathymetry. You’ll also learn about the Mariana Trench expeditions, the difficulties of operating in a high-pressure underwater environment, and how to come face to face with monsters of our mythologies.

Grafana Loki Design Basics with Ed Welch (Grafana Office Hours #27)

Fri, 07 Jun 2024 18:29:26 +0100

Ed Welch, Principal Engineer on the Loki team, is talking to us about how to get started with Grafana Loki for beginners. He tells us the story of why Loki was created in the first place, talks about the terminology of Loki (what are indices, streams, labels, metadata, tags, and chunks?), and shares candidly about what Loki is good at and what it’s not good at.

How to Get Started With Loki

Fri, 10 May 2024 18:12:09 +0100

If you were looking for how to get started with Loki, here’s a video where Senior Developer Advocates Jay Clifford and Nicole van der Hoeven talk about what Loki is, how to ingest logs with Loki, how to deploy Loki, and finally how to visualize your logs with Grafana.

Entendiendo los sintéticos (Café con Grafana #008)

Thu, 11 Jan 2024 16:00:25 +0100

En este episodio de Café con Grafana les traemos a Marcelo Magallón, experto en monitoreo sintético, quien viene precisamente a contarnos que es en sí un sintético. Él nos va a mostrar algunos ejemplos, casos de uso y mucha mas diversión!

Lee la documentación sobre el monitoreo de sintéticos en Grafana Cloud aquí.

How to Deploy Grafana Beyla on Kubernetes as a Sidecar Container

Tue, 09 Jan 2024 13:41:38 +0100

Principal Staff Engineer Nikola Grčevski demonstrates how to deploy Grafana Beyla on Kubernetes as a sidecar container. Grafana Beyla is an eBPF auto-instrumentation tool for application observability: you just have to tell it what to listen to, give it permissions, and deploy it– and you can see metrics and logs from that component or service without having to do any manual instrumentation.

AI Microscopy with Grafana, Theia Scientific, and Volkov Labs (Grafana Office Hours #24)

Fri, 15 Dec 2023 17:42:58 +0100

What do you get when you combine AI microscopy with Grafana? Well, in this case, you get the Theiascope platform: an application for doing real-time analysis on microscopy images using a combination of Grafana, Prometheus, and PostgreSQL/Timescale. The Theiascope platform was a collaboration between Dr. Christopher Field, Co-Founder/President/ Principal Investigator at Theia Scientific and Mikhail Volkov, Founder/CEO of Volkov Labs.

Repasemos el G-stack (Café con Grafana #005)

Thu, 30 Nov 2023 13:57:52 +0100

En este episodio de Café con Grafana tenemos a Raul Marin, Solutions Engineer (Ingeniero de soluciones?) quien viene a darnos una repasada completisima de todo el stack de Grafana. TODO! Pues hay muchos que no conocen todas las herramientas, caracteristicas y posibilidades que Grafana Labs viene a ofrecer.

Dimensionando el Grafana Suite (Café con Grafana #004)

Thu, 23 Nov 2023 13:51:25 +0100

Almudena Vivanco viene a platicarnos de todos los experimentos y pruebas a las que someten a toda la suite de Grafana. Desde Mimir hasta k6 ella pone al limite todos los elementos constantemente. Además nos dará muchos tips de su experiencia para usar Grafana y mucho mas!

(ES) Cómo hacer pruebas de carga con Grafana k6 (Café con Grafana #003)

Wed, 08 Nov 2023 13:43:50 +0100

Daniel González Lopes nos viene a demostrar cómo hacer pruebas de carga con Grafana k6 desde zero, y cómo utilizar k6 para ejecutar otros tipos de pruebas. En este episodio para principiantes, hablaremos también sobre la relación entre pruebas y observabilidad.

Resources

Combining Frontend and Backend Performance With John Hill (Grafana Office Hours #18)

Fri, 03 Nov 2023 12:30:43 +0100

In this episode of Grafana Office Hours, my colleagues Marie Cruz, Paul Balogh, and I talk to John Hill, a Web UI Test Engineer and Grafana k6 champion, to talk about how the Grafana and k6 ecosystems can be used to ensure performance in mission-critical applications like NASA’s Open MCT.

(ES) Observando Bitácoras With Matías Chomicki (Café con Grafana #002)

Thu, 02 Nov 2023 12:25:53 +0100

Hoy hababa con mis colegas Leandro Melendez and Matías Chomicki sobre observabilidad de logs en Grafana: qué son, con qué se comen, qué tienen que ver con Loki, otras plataformas donde se pueden usar, y muchos detalles mas!

(ES) Grafana Como Código (Café con Grafana #01)

Thu, 26 Oct 2023 06:55:37 +0200

En este episodio platicamos con Joan López de la Franca, Ingeniero de software senior en Grafana. El nos viene a platicar de las funcionalidades de Grafana como código, que es, como funciona, cuando conviene utilizarlo e incluso nos traerá algunos demos al respecto.

(ES) Inicia (Café con Grafana #000)

Thu, 19 Oct 2023 12:37:00 +0200

Tenemos un nuevo show para toda la comunidad de habla en Español! En el vamos a estar invitando a Grafanaistas, Champions, Entusiastas y a miembros de la comunidad. Aquí vamos a compartir experiencias, conocimientos, perspectivas, diversion y mucho Español!

Modernizing government documents with Govable and Grafana (Grafana Office Hours #12)

Fri, 22 Sep 2023 21:01:39 +0200

Ari Hershowitz and Andrii Kovalov from Govable.ai talk about modernizing government documents with Govable and Grafana, and how they saved weeks of effort by using Grafana as a ready-made frontend for their clients. They are joined by Developer Advocates Nicole van der Hoeven and Paul Balogh from Grafana Labs.

Grafana k6 for WebSockets and infrastructure testing (Grafana Office Hours #11)

Fri, 08 Sep 2023 20:54:28 +0200

In this episode of Grafana Office Hours, Solution Architect Huzaifa Asif talks about how he has used Grafana k6 for WebSockets and infrastructure testing, and how k6 can be used for general reliability testing in addition to load testing. He is joined by Grafana Labs Developer Advocates Nicole van der Hoeven and Paul Balogh.

How to learn Grafana with Grafana Play (Grafana Office Hours #10)

Fri, 01 Sep 2023 20:50:02 +0200

If you were wondering how to learn Grafana, Grafana Play is probably the easiest way. Grafana Play is a collection of ready-made dashboards and apps that you can use without creating an account. Developer Advocates Matt Abrams, Paul Balogh, and Nicole van der Hoeven discuss how to take advantage of this awesome tool and what you can do with it.

Getting started with Grafana Loki (Grafana Office Hours #09)

Fri, 25 Aug 2023 20:41:32 +0200

Senior Principal Solutions Engineer Ward Bekker talks about getting started with Grafana Loki: what Loki is, why you need log aggregation, and how it fits into the rest of the Grafana stack. He is joined by Developer Advocates Paul Balogh and Nicole van der Hoeven to tell you everything you need to know about Loki.

Observing the Internet of Things (Grafana Office Hours #08)

Fri, 11 Aug 2023 20:31:53 +0200

My colleague Paul Balogh and I welcomed Daniel Mangum, Lead Cloud Engineer from Golioth. We learned how Golioth uses Grafana to report data from varied sensors and monitors.

How to get started with Grafana and Docker (Grafana Office Hours #07)

Thu, 03 Aug 2023 20:28:40 +0200

My colleague Usman Ahmad and I talk about how to use Docker to quickly get up and running with Grafana, how to add Grafana plugins, and how to troubleshoot common issues.

Grafana documentation on how to install Grafana on Docker here.

OpenTelemetry demo app with Grafana, Tempo, Loki, and Prometheus (Grafana Office Hours #06)

Fri, 28 Jul 2023 20:25:37 +0200

DevOps Engineer Blueswen Liu 劉義瑋 joins us to walk us through some OpenTelemetry demo apps he created, instrumented with Grafana, Loki, Prometheus, and Tempo.

What's New in Grafana 10? (Grafana Office Hours #01)

Fri, 16 Jun 2023 12:00:25 +0200

What’s new in Grafana 10? I interview Grafana Labs Product Director Mitch Seaman about all the new features in Grafana 10, including Scenes, correlations, some important plugin development changes, and improved security.

Grafana Cloud k6 (k6 Office Hours #83)

Fri, 24 Mar 2023 18:00:00 +0100

With the release of Grafana Cloud k6, you can get observability and performance testing in the same platform. Senior Product Manager Mark Meier and Frontend Lead Simon Legander join me to talk about what Grafana Cloud k6 is, how you can use it, and demonstrate what it can do to improve your system’s reliability.

Introducing Grafana Cloud k6

Tue, 21 Mar 2023 15:20:58 +0000

Grafana Cloud k6 is a feature available to all users of Grafana Cloud. It allows you to run load and performance tests right from the comfort of your Grafana Cloud instance, without needing to create a separate account on k6 Cloud. Grafana Cloud k6 helps you get started quickly with a GUI Test Builder or with test scripts written in JavaScript, and then visualize your test results in real-time, alongside your other Grafana dashboards that monitor your system infrastructure. With Grafana Cloud k6, you can get a holistic look at your system’s performance– on any environment.

Learn more about Grafana Cloud k6.

Distributed Tracing With Grafana k6 (k6 Office Hours #70)

Fri, 04 Nov 2022 14:03:04 +0100

Just announced: distributed tracing with Grafana k6 x Tempo! Daniel González Lopes joins us to talk about what distributed tracing is, and how traces using the Tempo backend could help you troubleshoot your performance issues and improve overall reliability. This new integration was just announced at ObservabilityCon 2022, and Daniel’s here to tell us all about it!

Resources

Observability Trends and Future Best Practices (k6 Office Hours #59)

Fri, 12 Aug 2022 16:19:13 +0100

What are some observability trends and best practices in 2022 and going forward? Technical Program Manager Daniel González Lopes joins Developer Advocates Nicole van der Hoeven and Paul Balogh to chat about trending concepts and new areas of exploration in the field of observability. We’re going to talk about distributed tracing, continuous profiling, eBPF monitoring, and more!

How I Learn in Obsidian - Keeping up with tech trends

Fri, 12 Aug 2022 16:17:11 +0100

Here’s how I learn in Obsidian by processing articles I’ve read into consolidated notes that help me understand technical concepts. I also show how I improve my learning by incorporating visual elements using Obsidian Excalidraw, Excalibrain, and Advanced Slides.

How to migrate to Kubernetes, with Carlos Ruiz Lantero (k6 Office Hours #55)

Fri, 08 Jul 2022 17:00:57 +0100

k6 Cloud Backend developer Carlos Ruiz Lantero joins Developer Advocates Nicole van der Hoeven and Paul Balogh to discuss how to migrate to Kubernetes. Carlos has been working on migrating k6 Cloud services from ECS to EKS, and in this video, he shares his best practices for migration and things he wishes he’d done differently.

Load testing results in the k6 Cloud App for Grafana, with Edgar Fisher (k6 Office Hours #49)

Fri, 08 Apr 2022 18:00:18 +0200

There are many ways to get k6 load testing results into Grafana, but our frontend software engineer, Edgar Fisher, talks to us about the easiest way of them all: the new k6 Cloud App for Grafana.

TestOps Ask Me Anything on Devops Toolkit

Thu, 07 Apr 2022 18:00:30 +0200

I joined my friend Viktor Farcic on his channel, DevOps Toolkit, to do an Ask Me Anything on the topic of TestOps.

We ended up talking a lot about k6.io, different Grafana Labs open-source projects, and what performance testing has to do with DevOps and Site Reliability Engineering.

How to Make Kubernetes Clusters Production Ready using Crossplane, with Viktor Farcic (k6 Office Hours #46)

Fri, 18 Mar 2022 17:30:45 +0100

In this video, Viktor Farcic, Paul Balogh and I talk about how to make Kubernetes clusters production-ready. Viktor is a Developer Advocate from Upbound, and he demonstrates a solution that ties together Crossplane, Grafana, Prometheus, and Loki to deploy and manage clusters in production.

Canonical Observability with Simon Aronsson and Michele Mancioppi (k6 Office Hours #38)

Fri, 21 Jan 2022 18:23:47 +0100

Today, my former colleague Simon Aronsson brought his new colleague, Michele Mancioppi, to k6 Office Hours to talk about observability, and how they do it at Canonical. It turned into a great conversation about trends in observability from the last decade, and how it relates to performance and reliability testing.

Transcript

Hello everyone, and welcome back to the first k6 Office Hours of the year. Happy 2022, I’m Nicole Vander Hoeven.
And I’m Simme, we’re back from the dead.
And I’m Michele.
Simme is back. Yes, so Simme unfortunately is only temporarily back, and he’s brought his new colleague. I mean, I’m kind of sad, but I’m happy to meet you Michele Manciopi.
Hello everyone.
Welcome. Simme why don’t you first tell everybody, how you betrayed us at k6.
Sure, I’m so sorry for that, was a couple of months ago now. I think I quit at k6 in October, beginning of October something like that, to join Canonical and Michele, where we are trying to build a really awesome observability stack to know little part built on top of Grafana products.
Yeah, it has been a very interesting three months. It’s actually a pretty refreshing to be able to use high quality software and compose it in ways that are, you know, they just work through, cool.
So we already have a few people in the peanut gallery, there’s Simme and Michele and our newest colleague, I think senior performer, all saying hi to you Simme. so you’re very much missed.
Congratulations to k6 for bringing Leandra.
Definitely, and by the way, Leandra will be on k6 Office Hours next week, he just accepted the invite. So when I’m letting him like relax, and do his bootcamp onboarding things, and his week of load testing this week. . Ooh Danielle is here as well. Michele, why don’t you talk about yourself, and what you do at Canonical?
Well, I am a product manager at Canonical, for observer BD team. I was brought in with the goal of making the entirety of the product line of Canonical observable. Canonical is chiefly known for Ubuntu, but we have like a meter other products and they all need to be observable because they all sit at different levels of critical pieces of the infrastructure. Before Canonical, I was the technical product manager at Instana, which has been what baby ham it’s a product that does very automated observability. It’s mostly proprietary, but it was a lot of fun trying to make observability the way it’s supposed to be. That is seamless. Very easy to roll out. Before that I was at SAP where I was the technical lead of the performance team for SAP cloud platform, where my job was mostly to figure out where things were slow and fix it. And then I realized that rather than helping one single developer at the time, would they be more productive in actually providing pretty decent tools to do that at scale so people could do it themselves, which then brought me down the path of figuring out it’s actually fun here to make an observability product than it is to using one. And then the CircleCI.
So Simme what was it about Canonical that won you over?
I actually think it was to no little part Michele won me over with his argumentation for–
Okay. I’m taking him off the stream sorry. No, no, No.
‘Cause I got you everyone, bye.
No. We’re still all friends.
I do really miss the k6 family for sure. But yeah, it was just such a cool and exciting opportunity to be able to make this huge leap in observability with Canonical, for such a vastly used project or suite of tools and products. So that’s mainly one of them.
I’m sorry. Can we stop to remark? That was the coolest cup I’ve seen in a long.
It is as if this is sponsored, this is a hydrate spark, it’s a smart water bottle. And it’s marketed like for people like me because it’s a water bottle, but it has Bluetooth and LED light. So, you know, obviously I had to buy.
Yeah, just out of curiosity, what are the key values of the Bluetooth support?
Oh, it tells me when I need to drink water. It’s very important stuff.
How do you monitor that A, the Bluetooth is working and how do you receive those notifications?
Well, it’s funny that you ask, there is an app for it. This is totally not sponsored. Okay. But there is an app for it. I was gonna say Simme that it seems like there was a slight change in your focus as well, because at k6, we’re more focused on the performance and reliability testing part of it. But definitely we’ve also dabbled in the observability space, especially with the acquisition of Grafana. So it seems like your new role is also more focused towards observability. Was that one of the considerations?
Sure it is. And then that was definitely one of the considerations during my, what is it now? 12 or something years in tech, I’ve been doing way more observability and monitoring work than I’ve been doing performance testing. And so that definitely felt like coming home in a sense to me.
And I wanna ask you both this because you’ve both been in observability for awhile. How do you think observability and performance testing go together? Do you see them as separate disciplines entirely, there’s clearly some sort of overlap, right? Should companies be doing both, one or the other?
I definitely think there is a lot of overlap. One thing that I know that you and I have been pushing forward when we’ve been discussing this previously on the stream is that performance testing is great, but it usually only allows you to do a black box perspective testing, right? You’re able to measure the performance of your system, given certain interfaces or end points that you are able to observe the performance off and pairing that with observability tools, like for instance, loci or tempo, or Prometheus’s allows you to really drill back into the stack and figure out what is causing those problems. So I definitely think that it makes sense to combine the both to be able to observe the whole system and not just the edges of it.
Do you agree with that Michele?
Yeah, very much, over go and say that actually, it’s very hard to draw a demarkation between the performance and of observability, especially since in the last few decades and the increasingly disproportionately larger role that internet and internet systems taking our lives. We have come to see, you know, slow as the new down, which means that not only the software needs to work but it needs to be fast. And if you observability is fundamentally the art of understanding, are there problems? I mean, so how bad was impacted and how badly, and that is both in terms of software, it is broken as software is luck. Many of the techniques that you would use in, for example, a QA system to check the scalability of your software is something that you can use at scale in production to make sure that you’re at least carrying that much load, as well as techniques for example, for performance testing, you often do a lot of black box monitoring. The techniques that you use to check if today in SIL on a mobile connection, your web store is reachable is very similar in terms of technologies and capabilities and as technical requirements. So the technology is there to use, to throw a lot of requests security to your software. So it’s fundamentally the performance tools are versatile like that.
Yeah, I totally agree with that. I have always taken the stance that it’s definitely better when you have both of them because observability is great. But then in pre-production environments, it can be a little tricky ‘cause you don’t have real traffic and that’s where performance testing can come in handy and performance testing is great, but you don’t always want to run it in production. And also you, it’s not always a good idea to be reliant on the results of the performance testing tool. You know, sometimes it makes a lot more sense to be already using the same observability stack that you use in production. And I also want to call out that performance testing is often thought of as load testing, but that’s really just one kind of performance that is being measured there. And so what about a related topic to observability as monitoring? I know Michele, you have a lot to say on that topic.
Oh my, can I rant?
Yes. Go ahead. I’m familiar with your blog posts.
Ooh influence. I have a –
I do research, you know.
I thought you came across it because, you know, super rabid spike. No, the, so I have been around the observability industry for awhile and I have a certain annoyance towards what the industry has done by rebranding perfectly good monitoring and application performance tools in terms of the hot new term up observability that you need to sell to sort of the software. The way I see it’s a much more formal, shall we say way. Observability is a property that you want your systems to have, monitoring is the way that we have to make most systems observable. However, something that I hope we will see in our lifetime, our software systems and such a level of quality and standardization in our profession of creating software systems. So that monitoring and observability becomes build team in the softwares we create. If you think about it all the for example, automatic distributed tracing and all the amazing innovations that have occurred over the past 15 years are effectively reckoning the lack of observability as a functional requirement of a piece of software. Monitoring is the art to collecting data.
Yeah, and I think that there’s also a connotation of monitoring being a bit more passive. Would you agree with that?
Not necessarily, no. At the end of it so there is, I really take the view that monitoring is a means to the end of observability. It is true that in the past few years, with new generations of tools, we have made leaps and bounds in both the capability of collecting useful telemetry with very little work in terms of toil to roll it out, without the proper tools are there and I used to work for one where you deploy an agent and it automatically discovers all around it. There is a lot of dark magic going on, but it works. And then it becomes very cheap to actually get to the holy grail of monitoring that is to have as little silos as possible. Telemetry wants to have friends. You want to see end to end systems how they relate to one another. Now, the moment that something that has changed recently in the industry is we have gotten better, not only at collecting telemetry, but analyzing it. When you compare monitoring tools that we started with right there the Nagios of the world. You’ll need it to know in advance what you were looking for. Now with broader collection of telemetry, for example, distributed tracing, a very powerful type of telemetry, very little underutilized in terms of the insights can provide you, you end up having a lot of data that is likely going to be useful, and you have much more freedom in how you analyze it, both in terms of discovering issues as they occur, as well as afterwards in terms of post-mortem.
I do have, however one, I agree with you completely Michele. I do however, think Nicole has a point in that if you look at the traditional monitoring tools that people used to use, like Nagios for instance, it was always this very weird situation where you put up SNMP traps or collected, you know, logs that were thrown at you in a very passive manner. But I would say that it turned monitoring itself has come to evolve into a state where it very much is an active practice in the same way as people would describe observability.
Oh. So you are making a very interesting distinction of rather than receiving as a data sync telemetry that was emitted, but something else that you go and get it through RTI, things like deploying an agent that does stuff. That’s pretty interesting.
Yeah. Or for instance, actually triggering a synthetic test that just prompts a common user flow and make sure that that still works. I mean, as compared to just admitting this passive telemetry data that just keeps rolling on, but maybe lacks the context as you put it on more advanced workflows where you’d correlate multiple sources of data or the multiple actions to derive your data that you want to.
It is true that in the past 15, 20 years, there has been an increase of active monitoring. I remember doing it back in the day with smoke tests that were of a crunch up every 15 minutes. I was so young. And now it’s that it’s kind of a baseline of observability. I mean, you need to want to catch everything, but if there is catastrophic failure, you will know hopefully before someone starts opening tickets or you’re upping your drops. Yeah.
Yeah. I think Simme it was right when he was saying, what I was talking about was like the more traditional approach that it’s just something that’s there in the background whereas these days, the way that it should be done is a lot more proactive.
And also they focus on actually gathering telemetry to be able to detect failure modes rather than probing for a certain thing. It used to be much more focused on a single point that you would probe for data. And–
It used to have simpler software too.
We sure did.
Yeah, that’s for sure. But I also wanted to say that there’s also been, there’s still some push based monitoring, but we’re also now seeing more poll based monitoring systems, more event driven ones. And I think this is all the evolution of monitoring and observability, right? Because these things did not exist not too long ago.
To be fair, the one should, it is pretty true. We have much more pool and pool length. This is something that a permit use has pretty much popularized with the concept of scraping, but there are entire domains where we deploy a software that is still very much push based. For example, IOT, for very basic reasons of network visibility, it’s very hard to go and reach out into someone else’s rooftop and go get the data. So there is still the mechanics of push buffer and pulled it for example, gave us in the primitive ecosystem, push gateway mechanism.
Grafana agent that actually sits in between you and a cloud service or something else, and allows you to not punch holes in your firewall, but rather reach out and deliver data to something.
Yeah, it’s kind of like you mentioned that our applications in general are just getting more and more complex, it reminds me the push the trend from monolith to microservices and then now some people are also going back. It’s never really gone, right. And it’s really difficult to say, no, this is the new way, nobody else is doing or should do the other way. It’s not really like that. A lot of things in tech are cyclical.
When I actually spoke about our systems are getting more complex. I was thinking exactly of micro services because we made a trade as an industry for very, very simple, very small components. And then pushing all the complexity that you cannot make disappear under the carpet of service meshes, many containers, things that need to be versioned independently remain compatible, which is actually one of the main drivers why in the observability industry, we moved from the very traditional, I don’t like the word pillars, but the very traditional telemetry types of logs and metrics with the occasional sprinkling of profiling to rather extensive use of distributed tracing because all of a sudden the problem was less often in the single simpler components and much more often in between.
But this is kind of a funny conversation to me ‘cause I don’t know if you remember, but 10 years ago we used to talk about microservices as well, but then we called it service-oriented architecture and it wasn’t the new cool thing. And then we decided that that was kind of cumbersome and powerful to work with. And we started to merge together our software, again, introduced monoliths.
But in a service record architecture, we tended to have a very expensive, very large piece of machinery called the enterprise service bus that actually was providing to some extent that kind of visibility that today we need to have through tracing.
And let’s also not discount the effect of the proliferation of cloud providers, because when you have your own data center and your audiences, maybe your internal company or something, maybe it is a good thing to go from one huge complex monolith to three simpler services. But when it’s not three, it’s like hundreds, maybe then you start needing like another layer to orchestrate and manage and all of the traffic between each of those components. It’s definitely not that clear cut to say which one is better.
I definitely think you’re bang on in that case, Nicole, ‘cause we’ve moved from maybe having a couple of software devs for a company that did some kind of HTML that website with some access database backend in the worst cases, right? We now have these huge teams of hundreds of developers that are all working on a compost system together. And having that as a monolith would in practice require us to either do these huge orchestrations of release trains that no one is able to do in a proficient fashion. Or we would have to have downtime all the time ‘cause we would need to redeploy everything every time we made a change. So in that sense, it really makes sense to split it up as we grow our teams and we grow our development efforts as well.
I have a rather, so coming from a very enterprise background I have a rather newest feel to be on the subject. And my take is that usually technological problems are not technological problems, there are people problems. One of the main motivators that was towards going to the extreme distribution of microservices. ‘Cause I tried to fight back against the Conway’s law. The fact that roughly the architecture of your system mirrored the architecture of your organization. So we said, look, we have all these teams, these pizza teams, it’s very hard to make them cooperate to create a well-behaved monolith. So let’s give each of them their independent infrastructure and talk over standardize protocols like most of the HTTP and some sort of RPC built on top of it. And that would provide for them a way to be productive independent of each other.
Exactly.
As an industry, we dramatically underestimated the cost and complexity that comes from the need of coordination and compatibility and understanding among these teams, because now you no longer have one code base that you can run tests against. You have to have multiple ones that may interact surprisingly at Atlanta. These for example, has led to introduction of contract based testing, where teams contribute to each other interface tests. And there’s also the saying, which is something that is a harsh reality that every observable behavior of an application with a large enough user base, will become a dependency for somebody somewhere.
I think this is the whole premise of Conway’s law, right? The idea that our architect, the software architecture or the infrastructure that we put into place in a lot of ways starts to evolve depending on how our team is structured. And I think it’s both a technological and a human problem as well. When you talk about coordination between teams or between services, it’s really the same thing.
Yeah. We have made actually strides in how easy it is to make software inter operate with each other. If you go back, we started, we were talking about SOA. SOA came with a curse called SOAP. It was incredibly complicated, mindbogglingly hard to make run at scale. Today, we have a bunch of, so we are using HTP for other things. Sometimes I would think too many, we have good stuff like GRPC, like arrow, a lot of different protocols that can be very efficiently used to either make a synchronous message passing happen or synchronous invocations. We have a lot of tools that in the past year and half, I think we’ve got the big two trigger happy in deploying a lot of new tools in production. And I’m saying this with a background of someone that fundamentally had to support every single piece of technology that made input or output, for example, on Java. It would surprise you how many different HTTP clients for Java are I think in production, I lost count. There were 50, it’s insane. And I’m, for example, you raised your mark right with a very, very beautiful LED and Bluetooth. And it’s super cool. I feel–
I was trying to share something that Simme was sharing there, but I can’t actually see it for some reason. Oh, there it is.
It’s okay.
What’s this Simme?
I love to say, I like to bring out this picture whenever we talk about why observability is needed. ‘Cause while it’s a lot larger than most of us operate with, and it’s still very striking that they think this is supposed to be Netflix’s is a microservice architecture at some point, whether it is or not, I don’t really know, but it’s supposed to be and trying to–
Can I be trashy? Can I be able to trashy?
Thank you.
Architecture has an element of deliberation that they do not believe there’s an actual out of the system has, this is an emergent system is a lot of architectures that then meshed into something.
Definitely.
It’s remarkable that it works so well.
Yeah. And one of the cool things there, I think is the fact that without observability where this big ball of yarn, it would be completely unfeasible to reason about anything this system is doing at scale. You wouldn’t be able to reason about the particular part, maybe that you are working on one of these small dots that we can’t even see because of all the lines. But to reason about what it does is as a whole would be completely unreasonable. But with tools like, for instance, distributed tracing, we’re suddenly able to follow end to end an action of a user or an agent within this system. That was all I wanted to think of that.
I love distributed systems, distributed tracing in distributed systems, because it actually gives you the means of answering the most important question or you’re faced with an issue, who is affected and how badly? When you go on an enterprise setting and our people are often databases service, your application completely blowing up, could be a drop in the bucket of their system. They will not probably have the central entry you have to have a very different take on priorities unless you have agreed upon a slice on the silos and the failure that you are fundamentally up streaming through your users may not be their concern, but the moment you have the shooting tracing, that goes from the deepest database to the device in the face of your user, yet you have a very clear perception of how bad is something that is the one most important piece of information that you need in front of your face when the pager goes off again at 3:00 AM, you need to decide whether it’s time to go on to Batman, or if it can wait until the third coffee tomorrow morning.
I think that’s the real success of the “Squid Game” right? Because that image that you had up Simme, and just imagine that times 114 million users that are potentially using many of those systems, it’s amazing that it works at all at that scale, especially.
Something that always gives me those thoughtful moments of, oh my God, what are we doing as an industry? Is that a potentially simple issues have disproportionate impact on popular services to the life of many, many people. It goes from lack of entertainment. You’re there you want to see the latest, “Witcher” episode, cool stuff. You’re disappointed if the thing is on top, multiply that by a medium people that cannot get there, there are fix, Hey, you have a significant amount of life around that has gotten worse, but it goes also to something that is maybe much more limited, but much more tragic. For example, when I was working at SAP, one of our customers was, I cannot say the country, but it was the first aid response service of one of the countries. We were scared plus that something so important that either the platform, because downtime would have consistent consequences to real people, something that we have seen recently with the move to the cloud and the disproportionate role that for example, U.S system one has on everybody’s life today. We have seen how the risk is getting more and more concentrated into fewer and fewer, very large, very complex, very reliable, but critical pieces of infrastructure, in ways that I do not believe as a society we have had yet. And that is where it really humbles me as an engineer to think, well, this has to work, right?
Yeah. We went through that in, I think in the last k6 Office Hours, I had somebody from oil funk, who is an emergency services provider in Austria, and it was cool, but also a little scary to find out that they are using k6 for, in part to also handle the traffic from the 112 emergency hotline. So it’s cool to see something that you’re working on, be used in such a critical industry, right. But at the same time, it’s also a little scary because it could actually be lives that are lost if you know, we don’t do our jobs well enough, it’s a little humbling to think of that.
I guess the closest resemblance we’ve had to these critical infrastructures that everyone relies on would it be like the transatlantic Telegraph cable? I mean, that has like in, I think it’s currently 19 cables or something like that, that make up the transatlantic telephone cable. But I really any catastrophic failure there would really isolate us in terms of cross continental communication.
But that is less likely to result in loss of life. Then, for example, your neighbor want service, not being able to follow you.
For sure.
I’m sorry for having completely spoiled the mood. I think that as an industry, we need to go in deserve the role of engineers. If we built bridges the way we push software to production, in some cases, I don’t think anybody would cross a river without crossing themselves first.
What’s interesting is in that spaghetti soup of microservices that you showed Simme, now we’re seeing that our observability stack is part of those. They are also contributing back to the system that we have to monitor. I don’t know if either of you saw the post-mortem by roadblocks.
It was classic. I had ops to them. It’s a, we have all been there.
Yes yes. Do you have to talk about it Michele?
I do, but I don’t think I would be able to be nearly as good as a Simme for this.
I just came to be honest. So I don’t have any detail commentary on that either.
It was the classic fallacy of oh jolly, oh golly. The same infrastructure that is powering my observability system was probably the system that went down.
All right. All right. Yeah, that really is something that you’ll see a lot. I mean, for instance, say that you have a deployment of the classic Grafana suite somewhere, and you have that running in the same data cluster maybe, or at least the same cloud provider as you have your critical workloads and to make it even worse, you could go down to having it in the same geographic region, or even on the same cluster nodes or the same hardware that you’re using for the rest of your critical infrastructure. When something goes south there, in terms of your critical infrastructure, it is likely that that was all will also, as I think, where to kick would roadblock, spill over to your monitoring system. And if that happens, how are you gonna be able to tell that you’ve been hit?
This is the classic blast radius question. Everything has failure modes, actually, several of them, and the more abstractions we deploy, the more failure modes we introduce, for example, Simme and I are building an observability system based on Kubernetes natives. Our guidance is if you deployed, deployed to a different cluster on different hardware, because you should never put the monitoring system on the same infrastructure, close to the infrastructure stuff we monitor.
It’s not only that. We also usually recommend that if you deploy our monitoring system, deploy another monitoring system that makes sure that your monitoring system is up and set them up in a, what did they call it? That dead mans trigger?
The dead man switch is the idea that, for example, do you know in the movies when the hero goes in front of him and says, if you kill me, you will be exposed in two days time. That’s the classic deadness which is something that happens unless someone periodically goes there and tells him not to. In this case, in the case of monitoring is unless your system, your monitoring system is telling you that it’s fine, assume tragedy. It never goes to the level of a specific use cases, but first of all, you need, let me take a step back. The best monitoring system is the one that tells you whether it’s something to look at, right? You actually do not want to have dashboards, however, colorful and beautiful in front of your face all the time, unless it is for Ferguson purposes, you actually want to be able to focus on something else and jump into the action when stuff is hitting the fan. In order to do that, you must have the guarantee that is the easy model of notification. You will receive it because silence, you want to know that it means that it’s good. You don’t want to have to doubt if there is something going on that my sister should tell me but doesn’t, which is why you set up fundamentally a health check thing that says I’m thinking here and all it’s fine. That is the diagnose switch.
I found it so interesting to read through the post-mortem and props to them too, by the way, for releasing something so detailed, they were not afraid to, you know, say what they’d done wrong and what they were doing to address it. I thought it was really well-written, but I want to read one part of it, because I thought it was so good. They said, “There was a circular dependency between our telemetry systems and console, which meant that when console was unhealthy, we lacked the telemetry data that would have made it easier for us to figure out what was wrong. We’ve removed the circular independency, our telemetry systems no longer depend on the systems they’re configured to monitor.” I thought that was a great takeaway from that whole incident. And because they’ve shared it in the post-mortem, they’re not the only ones who can learn from it.
Very generous of them to expose themselves that way and actually left the rest of the industry learn from their mistakes. And so hard work as you said, Michele, our love to the team at the roadblocks who probably need needed a bit of time out after that.
But there is also another component of a human component here. They did the right thing. They reestablished trust. They are telling us, we learned from our mistakes so that you can as well. In the scope of monitoring systems, trust is the ultimate currency because a monitoring system is literally a measurement system for the health of other systems. If you cannot trust your measurement system, what is the point of it?
And you usually only get one go with that as well. If you burn that trust, it’s gonna be very, very hard to build up again.
So this is something that touches me very personal because I used to work on a solution that was exquisitely powerful, but therefore also very complicated and complex. And the moment you have automation, someone somewhere sometimes is going to be broken by it. And reestablishing that trust was never easy and sometimes impossible depending on the scale of failure that you’re well-intended means of automatically monitoring something I should have brought to the customer. Something that we seldom say as an industry, when we try to sell a product to somebody else, is that everything has a cost. If you want to take away complexity from you to roll out very simple systems and set up that thing, reports stuff over there, and you want it to automatically, there’s a higher risk that the automation will cause an issue. Of course, that in many cases is a justified risk. You can roll out the monitoring gradually and the easier it is to roll out a monitoring system at scale, the more you reach the holy grail of monitoring, that is the network effect. The moment that you can start observing how issues in systems propagate, correlate, or are not correlated there is something very important.
So let’s talk a bit more about the characteristics of a good observability stack. We’ve already mentioned a few we’ve mentioned that, it should be independent as much as possible from the systems that they’re trying to monitor. We’ve also said that maybe we should also take into account the human side of things rather than just the technological. Maybe you should tailor the complexity of your observability stack to the size and experience and structure of your teams. What are some other things that you both could name in your ideal observability stack?
One thing that I really like to highlight here is the ease of use or the economics of the system that you are using. I used to do a analogy to how EpiPens work for people with allergies. If you’re eating a peanut and you’re allergic to peanuts, you don’t want to have to assemble the cure for that, right. Or something that alleviates or takes you out of immediate danger. You need it to be simple, just as take it out, push it in your leg, and it’s done. You don’t have to think about it. The same goes for monitoring or observability systems as well I would say. If you have to, when you’re awakened by a notification in the middle of the night, and maybe you’re even hung over or worse, and you are supposed to fix something that is critical, and you have to start thinking about what to do, then your observability system has in my eyes, at least failed you, you need to be able to get a notification that says this is wrong, here’s the data that proves that, click here to get a closer look and just be presented with all the information you need to be able to make a decent decision on how to proceed.
One thing recently on that note that I’ve come across is how potentially easy it is for a new Blake me to set up a whole observability stack or at least monitoring, but he could definitely expand upon it is EBPF, because we’ve been talking about distribution of systems and also of observability related services, but there’s also a trend of going lower and lower level, right? And EBPF monitoring is monitoring at the kernel level, recently with some people from the k6 team, Artem Charlie, Kevin, Danielle, Lopez, shout out to those guys. We did this hackathon project actually for EBPF monitoring. And the whole reason that I got into it was because the promise was you install this binary, you drop it into your cluster on your local machine. And then it just kind of exposes magically all of these things that I normally would have had to set up like node exporters for, you know, it was we’re living in a golden age of monitoring and observability I think. The things that are possible, like I can’t even keep up anymore, be talking about ergonomics. I mean, what is more ergonomic than not having to do much of anything.
You are talking about the day one, the fact that it is so easy to set up apart from monitoring system is exactly what you need to be able to monitor at scale. It removes the thought. EBPF is a fantastic technology. We are fundamentally given a virtual machine to run inside the kernel to execute logic, that allows us to do in a centralized place. Things that in the past were almost impossible. For example, the first thing I ever did with a team of vinyl in Stanhope with EBPF was actually to detect how a process that, that was impossible in user land. But with EBPF, it was actually pretty simple. We literally spent more time making sure that the techniques would work across all the operating systems that we need to support and to figure out how to find out that. EBPF the, I see two big potentials for that, for things that are not possible already today with sufficient tips. The first one is to give us a distributed tracing tool that largely independent from their own time. Now there are technology limitations there. It’s hard to find out how to trace context. I’m sure that someone somewhere out there is figuring it out as we speak, probably, that we reduce to some extent the toil that is necessary to support the federal fire hose of new technologies, libraries, and stuff like that is put in production every day. But the second one that excites me even more because the shoot the tracing is largely already possible is actually to have cheap enough in terms of overhead and consistent enough across run times production profiling. People look at distributed tracing as a way of figuring out what is slow. And it works remarkably well as long as the bottleneck is in between processes, the moment you want to drill down, why this requests into the insight, this process was taking too long. You see what you’re tracing is not so suited anymore. You end up looking at the leaf instead of the forest, because there is so many more requests going on through the system, that unless you have a lot of spans that are very, very detailed and fundamentally to trace threat level profiling, you are not going to which costs a huge amount of overhead. You are not going to have the information needed to find out, okay, good stock in the Dow library again. However, the moment that you have things, basically you have to find out where the CPU, memory or location, memory usage and wait time hotspots are in your application. You actually have the first places to work at to improve the overall system. It’s not gonna tell you why that request was going badly, but the moment you saw the hotspot, you are virtually guaranteed that many requests are going to get better. And then you can measure them again in terms of input output, observability tracing. Now people out there are going to say, Michele, what the hell are we talking about? Today we have a lot of amazing technologies to the production profile and they are true because for example, they are right because for example, if you look at the Java virtual machine with, for example, the Java flight recorder, you have excellent facilities with super low overhead to figure out what is your Java or cuddling or cradle program doing true. But they’re very specific to the particular run time and to a particular programming language. And when you start comparing what wait time, for example, looks like in Golang as respected over to machine with code routines, instead of, I dunno, loom threads, project loom, and all the rest of the stuff, you end up actually having telemetry that seems to say the same things, but in reality has a different semantic. And that is a challenge in our systems that have more and more grown polyglot. When we had the, our happy little monolith fault, it was one language. Now this year the systems are implementation of many different languages that behaves differently, and yet they are operated in some cases by roughly the same people, for example, necessarily teams, right? Having fluency in terms of performance across several languages is a significant challenge. Now, if we actually get through EBPF production profiling, for example, the folks behind parka and also the folks we have big pixie, there’s an amazing job there. If we manage to have something that will give us close enough, precise enough production, profiling information across whatever we may wanna throw into a container that is going to be a generational change in terms of making complex systems run better at scale.
That’s something that we’re interested in at k6 as well. I mean, I’m a performance Sesser so the first thing I think of as the network side of it, there are a lot of network metrics that could be gathered at that level. And the reality now is that without EBPF, you’re in this position where each new service has needs some other exporter that you need to install. And like there’s so many plugins to plugins and many different combinations of things, when really they might all be using HTTP. Like, why don’t we have something that just detects, maybe not, it won’t be everything right. It’s never EBPF has never going to completely be able to, or, well, I shouldn’t say never because you know, that could change very quickly, but I don’t see it as being the end all be all solution. But if it can provide some of the more common metrics, that’s already a step in the right direction. Talk about usability and ergonomics right?
And also provide a safe assessable baseline across a lot of different technologies working together.
Yeah. For sure. What other things do you think would you have in your ideal observability stack?
You asked me?
Either of you?
Go ahead Michele, I don’t have a lot of ideas on that.
Well, actually, if you go on the, okay, can I make us most shield for a link? If you go on google, WWW.ubuntu.com/whatisobservability/whatisobservability. There you see a kind of overview of the things that I find most meaningful, monitoring different types of systems. Second, I’m gonna give here that your own.
They can share my screen record.
I assume you want this one?
Awesome.
Yes. So this page is fundamentally a work of love of Simme and I, about the way we see observe a PVT. And the diagram that you see on screen is a, of course, simplified and very generic can your manage me if I lie, but it is a thirst overview of the kind of telemetry that you use at different levels in your infrastructure in order to monitor things. This is important for filamentary two reasons. One, is to provide guidance for people that may be new to observability in terms of which things to focus first, when you look at the telemetry types that are used on the right hand side, you will see that metrics and logs are of course universal. That’s what we started with. They are very good. Every software that deserves to run somewhere is capable of emitting clogs. Most of them can emit metrics and more and more, thanks to primitives that is really proud. There are the metrics sometimes serious thing to the forefront of enterprise systems. It didn’t used to be that. Then you start seeing that profiling gets, for example, the moment you move above the virtualization hardware, and you start talking about processes and applications, profiling them makes perfect sense.
Sorry Michele for cutting you off. But I think you might need to need to explain the icons.
That is correct.
That’s really. We don’t have that context.
Ah, they don’t explain themselves. Maybe then you use amino the icons.
Yes.
So the chart that you see on the left hand side is metrics, mentors, time series fundamental. That’s what we use to monitor things. There’s a lot. Then the table with the reds and greens and yellows followed by text that locks, the old, that bar chart is actually supposed to be flamed charts and that’s profiling showing hotspots. The strange thing with all those horizontal bars that is distributed tracing, that is the best way I can come up with to visualize spans. Spans is the basic data structure in distributed tracing. It describes an action taken by system to process a request. For example, you would have a span for the receiver of an HTTP request. The span is opened is created when your HTTP server is receiving the request, it’s going to be closed, but it simply is doing amazing things to get to the image in a screen–
Most of this you’ve probably seen in your browser, was what I was about to say, like something similar when you browse on webpage, you’d get this span alike view of how the different correlate.
It’s similar, but there is a key difference. One of the key concepts of distributed tracing as coming from dapper, which is fundamentally the progenitor of all modern distributed systems is the parent-child relation between spans, which is the closest you can get to cow’s reality. We are, for example, when your application is receiving an HTTP request and is issuing your query to Mongo to retrieve some data, you will have a parent’s span for the receiver of the HP request. And the child’s span that says, I spoke to Mongo, took me this longer. And this query, I got so many data back that is the parent-child relation. And it fundamentally, you would not have spoken to Mongo if not for the incoming request. Now things get Merck here when we talk about a synchronous messaging, because the beauty and simplicity of the trees of spans that you have in synchronous systems, kind of get lost with a messaging the moment that you have batch consumers. You’ll have many requests that result in a whole bunch of messaging being queued up in your messaging queue. And one application consuming messages originated in different traces all at once, fundamentally creating a merge, but I digress. I’m sorry. So we were talking about… I just get so excited about this future tree.
No, it’s great. I know that you said that you have a hard stop and I’d like to get you off a little bit early for your meeting Michele, but why don’t you talk about what Canonical has to do with observability at all?
Well, can I call does a lot of things, including a very beloved operating system together with an amazing community of volunteers, Ubuntu, we are responsible for a huge amount of software. We have offerings for Kubernetes, for OpenStack, for SAF, for a lot of systems that people use to run their applications and data centers. And it is the mission of Simme and I to make sure that those systems come with excellent monitoring capabilities, that you can monitor them with open source software, whether or not you want to use the monitoring system that Canonical provides you. At the end of the day, what we’re doing is in variable source fashion. We want to make sure that all the appliances that you can get from Canonical, the SAF, the OpenStack, the Kubernetes is, there are the cube flows and all the machine learning. They are, you can monitor them with best-in-class open source monitoring tools like videos, like getting logs out of SIS log, the security traces with open telemetry, where it makes sense. And that is the part of the work of Simme and I, that is about stepping up the game in terms of making software provided by Canonical observable in a way that you can use, for example, with your existing monitoring system, we really believe every single word we say, when we say the metric, once France, it’s all about having radio systems be monitored together. Now, if you do not happen to have a radio monitoring system, wherever monitoring the rest, we can give you one, but if you have a monitoring system and you’re comfortable with it, you by all rights should be able to monitor whatever piece of software you get from Canonical with it. And this is what to do.
That’s awesome. I wish we had more time to talk about this, but maybe we can have you both in some future episode, I feel like there’s so many things that I wanted to dig deeper into, but thank you both for coming.
Thank you so much.
I am so happy. Simme I’m so happy to have you back on again. I’ve always enjoyed doing these with you, so it’s a little bit weird that you’re now on as a guest, but I’m happy that you’re also bringing things that you’ve learned at Canonical back to us. And Michele welcome. I’d love to have you–
Thank you.
Back again for sure with or without Simme.
Anytime. But Simme is finer yeah.
All right. Thank you.
Sure. Sure. Thank you everybody for watching and I guess we’ll see you next week. Bye everyone.
Bye, have a wonderful day.

How Eurofunk uses k6 for load testing emergency services (k6 Office Hours #37)

Fri, 17 Dec 2021 17:17:07 +0100

In this k6 Office Hours, k6 CEO Robin Gustafsson joined me to talk to Daniel Knittl-Frank, a software engineer at Eurofunk, about how k6 is used in the emergency services industry. Daniel has an interesting use case for k6: he uses it to load test calls in Austria received from the 112 emergency services hotline!

What's new with k6 Cloud, with front-end lead Simon Legander (k6 Office Hours #36)

Fri, 03 Dec 2021 17:09:45 +0100

In this week’s k6 Office Hours, I was joined by my colleagues Mark Meier and Simon Legander, front-end lead at k6, to talk about the new features that have come to k6 Cloud this year so far. Simon also gives us a sneak peek of all the cool features coming on the horizon…

Performance Objective: Intro to Load Testing With k6 and Grafana

Tue, 23 Nov 2021 17:05:06 +0100

I spoke at the Performance Objective meetup to talk about how to use k6 and Grafana for beginners. I always enjoy talking to like-minded engineers, and I think sharing ideas in a meetup context is probably the best way to do it.

Everyone Can Contribute: Load Performance Testing With k6

Tue, 23 Nov 2021 16:57:26 +0100

This is a recording of a live meetup I spoke at, the Everybody Can Contribute Café hosted by Michael Friedrich, Senior Developer Evangelist at GitLab. This was a more casual conversation and show-and-tell about k6’s capabilities, and I really enjoyed it!

Browser testing and API load testing in the same script with k6 (k6 Office Hours #35)

Fri, 12 Nov 2021 00:28:02 +0000

Earlier this week, at ObservabilityCON 2021, we announced something new: k6 now supports browser-based testing with xk6-browser. Here’s a few of us from the k6 team talking about the announcement!

Resources mentioned

k6 load testing on Kubernetes using the k6 Operator, with Hans Knecht (k6 Office Hours #34)

Fri, 05 Nov 2021 00:23:15 +0000

I was really excited to have Hans Knecht on this week’s Office Hours. Hans works at Mission Lane, and is one of the biggest contributors to a k6 extension called k6 operator. k6 Operator lets you create a k6 custom resource object in Kubernetes.

Resources mentioned

Common performance testing problems and fears (k6 Office Hours #33)

Fri, 29 Oct 2021 00:20:27 +0000

For this week’s k6 Office Hours, we attempted to steer into Halloween by talking about some of the most common performance testing issues we’ve faced in our careers as performance testing consultants. For this one, I had Tom Miseur join me to commiserate and swap stories.

How to install and use the k6 Cloud Grafana data source plugin

Tue, 19 Oct 2021 00:02:05 +0000

In this video, I talk about the k6 Cloud Grafana data source plugin, which lets you use k6 Cloud as a data source for your Grafana visualizations.

Installing and using the k6 Cloud Grafana data source plugin

Installing the plugin for Grafana Cloud

Get your k6 Cloud API token on this page

Grafana Helm Charts repository

Conf42 podcast: in the k6 Universe

Thu, 14 Oct 2021 01:27:45 +0200

Miko Pawlikowski interviewed Simme Aronsson and I for the Conf42 podcast, and we talked about:

why I love to learn human languages, and what Esperanto sounds like
the Grafana acquisition of k6
what performance testing is, anyway
whether everyone needs to do performance testing, and my personal setup for this blog
chaos engineering and Schrödinger’s Pokémon
an overview of the Grafana Labs projects landscape
what k6 is, and how we would get started for free (k6 OSS)

Basics of load testing with k6 and Grafana in 20 minutes

Tue, 12 Oct 2021 01:04:02 +0200

This was a video I pre-recorded in preparation for the first Grafana EMEA virtual meetup.

In mid-June (just three months ago), k6 joined the Grafana Labs family. Many people didn’t quite understand what the connection was between k6 and Grafana, but I thought it made a lot of sense.

Google’s Site Reliability Engineering book talks about 4 golden signals that we should measure in a system: latency, traffic, errors, and saturation. Load testing provides 3 of those (latency, traffic, and errors), but it doesn’t show saturation, and it doesn’t give us visibility into the system.

Grafana brings observability. k6 brings data. They work really well together, which is why we’ve been using each other’s products for a few years now, before there was ever any talk of an acquisition.

In this video, I talk about why k6 and Grafana are better together and also demonstrate:

how to run a simple script in k6 OSS
how to make sense of the results of a load test
test execution options: running locally, sending results to k6 Cloud, and running a test from k6 Cloud– all triggered using the CLI
two different integration options to bring k6 and Grafana together: the k6 Cloud Grafana data source plugin and Prometheus Remote Write

Testing With k6 + Grafana: Better Together (Grafana virtual meetup EMEA)

Tue, 05 Oct 2021 21:49:13 +0200

In this first EMEA session of the Grafana Labs virtual meetups, I talked about how to use k6 and Grafana together to get the full performance picture for a software application. Observability, using tools like Prometheus and Grafana, lets us see what’s going on inside the application, and load testing, using k6, helps us get those insights earlier on in software cycles, such as during pre-production.

Intro to Load Testing With k6 and Grafana (webinar)

Thu, 16 Sep 2021 22:11:16 +0200

Tonight I did a webinar with Grafana Labs– the first k6-related webinar since the acquisition. I was more than a little nervous to give it, because I knew I was introducing k6 to a lot of new users, but also to Grafana for the first time. I think it went pretty well! I was still quite relieved when it was over, however. :)

In the webinar, I talk about:

why it’s still a good idea to do load testing in pre-production
what k6 is
why k6 and Grafana together make sense
and how to integrate k6 and Grafana

There’s quite a bit of demo in it, and I also had the k6 CEO, Robin Gustafsson, join me for Q&A as a nice treat.

Check out the full webinar here– it’s already up on the Grafana site. You can also go through my slides here, if you’d like.

Schrödinger's Pokémon: k6 and Grafana edition

Wed, 15 Sep 2021 18:22:31 +0200

Earlier this year, I gave a presentation at New Relic’s FutureStack conference about chaos engineering with k6 and New Relic. A few months ago, k6 got acquired by Grafana Labs, and so I wanted to redo it, but with the Grafana stack this time.

Here’s the result, and I think I like this iteration even better:

I presented this talk at TestCon Europe 2021 today, although the audio sync issues were not great, so I recommend you watch the video above instead.

Slides are here.

Meet the Grafana Labs team: Nicole Van Der Hoeven, Developer Advocate at k6

Fri, 13 Aug 2021 15:29:37 +0200

Originally posted here.

As Grafana Labs continues to grow, we’d like you to get to know the team members who are working on the cool stuff you’re using. Check out the latest of our Friday team profiles.

Meet Nicole!

Name: Nicole van der Hoeven

Current location/time zone

I currently live in Maastricht, in the Netherlands. I moved here with my husband four years ago from Melbourne, Australia.

What do you do at Grafana Labs?

I’m a Developer Advocate at k6, but I’m not actually a developer — I’m a performance tester! I guess there’s no such thing as a Performance Tester Advocate? My job is part engineering, part content creation: I write load testing scripts and code, but I also write blog posts, produce videos, and give presentations. I found out I was joining Grafana Labs a few months ago when k6 was acquired, but I was happy to join the team! I’ve met some fun and motivated people at Grafana Labs that I’m eager to collaborate with.

What open source projects do you contribute to?

I’m a budding developer and still learning the ins and outs of social coding, but some things I’ve contributed to recently are a chaos engineering extension for k6 (created by my colleague, the talented Simon Aronsson), an application called Poké API that I’ve added chaos experiments to, and a Python script for pushing my Kindle highlights to my note-taking app of choice, Obsidian.

What are your GitHub and Twitter handles?

GitHub: nicolevanderhoeven

Twitter: n_vanderhoeven

What do you like to do in your free time?

I play Dungeons & Dragons eight hours a week, take a ridiculous amount of notes on everything, and vlog about my (pre-COVID) travels.

A battle map in one of my Dungeons & Dragons games.

What’s your favorite new gadget or tech toy?

My new Rødecaster Pro, an audio mixer that has quickly become an essential part of my audio and video production process. It takes up a huge amount of precious real estate on my desk, but it is absolutely worth it!

My current desk setup, including the Rødecaster Pro peeking out from underneath the laptop in the upper right.

What’s the best part of your job?

I got into Developer Relations and advocacy in general because I realized that, as a performance testing consultant, I was being asked the same questions and solving the same problems. So I started to share my experience online, and I’ve found that significantly more efficient. Why explain something repeatedly to one person at a time, on the spot, when you could post a more considered explanation online and potentially help an unlimited amount of people? A large part of my job is simplifying technical concepts and making it accessible to beginners, and I love being able to help others by creating the content I looked for when I was a beginner, too.

Here’s one of my YouTube videos:

What’s the last thing you binge-watched?

I have binge-watched the entire two campaigns of Critical Role, a livestream of a Dungeons & Dragons game with a bunch of voice actors, which is over 1000 hours of watch time. Totally worth it!

What superpowers do you have?

I speak several languages: Tagalog (my native language), English, Dutch, Spanish, Esperanto, German, and French — in that order and to varying degrees. I am learning and have learned more, but don’t yet feel confident enough to claim to “speak” them! I choose travel destinations based on how interested I am in learning the local language.

What’s your Zoom virtual background and why?

I don’t use virtual backgrounds. The thing I like most about video is that it’s the most honest form of expression: writing, podcasting, and photography can all be edited far more easily. I like that my home office gives clues about who I am and what I like, and I would never want to hide that — mess and all!

Monitoring your k6 load test: how to install Grafana and Prometheus on a Kubernetes cluster

Wed, 11 Aug 2021 18:55:49 +0200

Running a load test is half of the equation, and not having the right observability tools to see how your application responds to the load can sometimes flat-out invalidate a test.

Here’s how to install open-source tools Grafana and Prometheus on a Kubernetes cluster in DigitalOcean and how a performance user could use them for increased visibility during a load test.

How we work at k6 - building software & working remotely with CTO Pawel Suwala (k6 Office Hours #23)

Fri, 06 Aug 2021 18:09:05 +0200

For this episode of k6 Office Hours, I got the k6 Chief Technical Officer, Pawel Suwala, to give a bit of a behind-the-scenes of how we work at k6 and what we’re working on next. We talked about the methodology we use for building k6, the different teams we have, and exciting new features that are coming soon!

A tester's overview of Grafana Labs open-source projects

Mon, 19 Jul 2021 18:23:33 +0200

Since the acquisition of k6 by Grafana Labs, the entire k6 team has been absorbed into the new, larger team that we’re suddenly a part of. Grafana Labs maintains a lot of projects, so I was very quickly overwhelmed with things to learn about all of them. Here’s my attempt to make sense of the Grafana Labs landscape, and perhaps it can help some of you as well:

Acquisition talk with CEOs Raj Dutt (Grafana) and Robin Gustafsson (k6) (k6 Office Hours #19)

Fri, 02 Jul 2021 21:04:13 +0200

This week on k6 Office Hours, Simme and I were joined by two CEOs: k6 CEO Robin Gustafsson and Grafana Labs CEO Raj Dutt. It was a little nerve-wracking to have two CEOs on the stream, but it did give us the opportunity to ask both of them pointed questions on Grafana Labs’s acquisition of k6, including any plans they have for how both companies could integrate better in the future.

What you need to know about Grafana Labs's acquisition of k6

Wed, 30 Jun 2021 18:29:24 +0200

Recently, during Grafanaconline 2021, Grafana Labs CEO Raj Dutt announced the acquisition of k6– the company I work for. What does this mean? Here’s a video I made to explain it all:

Simme and I have also discussed the acquisition twice on k6 Office Hours:

once with our CEO, Robin Gustafsson
and another time with both Robin and Raj, Grafana Labs CEO

TL;DW: We unanimously agree it’s a positive thing. I myself am cautiously optimistic about the future of both k6 and Grafana Labs.

k6 is part of Grafana Labs! AMA with CEO Robin Gustafsson (k6 Office Hours #17)

Fri, 18 Jun 2021 22:26:49 +0200

On this week’s Office Hours, Simme and I were joined by Robin Gustafsson, the CEO of k6, to make a big announcement: k6 has been acquired by Grafana Labs.

The folks at Grafana Labs have been fans of k6 for a while, and we’ve been fans (and users) of their work since almost the beginning of k6. While a surprise to many, Grafana and k6 together is a combination that makes sense.

Read Grafana’s press release here to hear more about this milestone.

Grafana on Nicole van der Hoeven

Spice Runner

What it demonstrates

The dashboard

Architecture

The talk

Run it yourself

The Defiant Jazz Incident

How it works

Why a murder mystery?

No spoilers

Try it

obsidian-grafana

How it works

What it’s useful for

Try it

Asimov

The idea

What’s in it

The talks

Try it

When to use Grafana Assistant vs MCP vs GCX: Grafana GCX

Transcript

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana MCP

Transcript

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Transcript

Resources

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Resources

Related reading

Grafana Campfire 🔥 — GrafanaCON 2026 recap

Resources

Asimov's Zeroth Law of Robotics: Observability for AI (ExpoQA Madrid 2026)

Resources

Zero to Dashboard with Grafana Assistant and the Infinity datasource plugin

Why Infinity is fun here

Timestamps

Resources

How to use Grafana Assistant with self-hosted Grafana

A quick note on what “for open source” actually means

Prerequisites

Enabling Assistant

A first dashboard from nothing

What else Assistant can do

Resources

Let There Be Dashboards: Introducing Grafana Assistant

Abstract

Asimov's Zeroth Law of Robotics: Observability for AI (Newcrafts 2025)

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

One Year Later: How I Use Visuals in My PKM (Sketch Your Mind Conference)

All About Structured Metadata in Loki (Loki Community Call August 2025)

Asimov's Zeroth Law of Robotics (Dutch Cloud Native Day 2025)

State of the Observability Databases (Grafana Office Hours #30)

Getting Started with Logging and Grafana Loki (webinar)

Asimov's Zeroth Law of Robotics: Observability for AI (KubeCon EU 2025)

Grafana Loki Query Best Practices with LogQL (Loki Community Call December 2024)

AI Observability with Ishan Jain (Grafana Office Hours #29)

All About Logs Drilldown (Loki Community Call October 2024)

All the Components of Loki Explained

Bloom Filter Changes for Grafana Loki (Loki Community Call September 2024)

API Monitoring with Traefik, Grafana, and OpenTelemetry (Grafana Office Hours #28)

Cartography of the Deep: Distributed Tracing for Testing (Romanian Testing Conference)

Abstract

Grafana Loki Design Basics with Ed Welch (Grafana Office Hours #27)

How to Get Started With Loki

Entendiendo los sintéticos (Café con Grafana #008)

How to Deploy Grafana Beyla on Kubernetes as a Sidecar Container

Links

AI Microscopy with Grafana, Theia Scientific, and Volkov Labs (Grafana Office Hours #24)

Repasemos el G-stack (Café con Grafana #005)

Dimensionando el Grafana Suite (Café con Grafana #004)

(ES) Cómo hacer pruebas de carga con Grafana k6 (Café con Grafana #003)

Resources

Combining Frontend and Backend Performance With John Hill (Grafana Office Hours #18)

(ES) Observando Bitácoras With Matías Chomicki (Café con Grafana #002)

(ES) Grafana Como Código (Café con Grafana #01)

(ES) Inicia (Café con Grafana #000)