Prometheus on Nicole van der Hoeven

The Defiant Jazz Incident

Fri, 30 Oct 2026 00:00:00 +0000

The Defiant Jazz Incident is a murder mystery you solve with observability tools. It’s set on the Sleep Floor at Kipsake, a company that rents your sleep: at night, its Dreamers sleep in pods while their sleepwalking selves work the floor. During the Cocoa Hour, the floor’s middle-of-the-night holiday party, the power goes out for five minutes. When it comes back, the night supervisor is dead in the elevator.

Kipsake recorded everything it could: badges, doors, chat, sleep rings, power, microphones, cameras, sleep bands, and personnel records. Your job is to work out who did it, and the only way to do that is through the telemetry.

How it works

The case file is a set of logs, metrics, and traces. A loader sends them to Grafana Loki, Prometheus or Mimir, and Grafana Tempo, either in Grafana Cloud or in a local Docker stack, and you investigate from there with whatever you like: Grafana itself, Grafana Assistant, an MCP server, gcx, or the AI agent of your choice. The data is deliberately messy. Some devices lose clock sync during the outage, some records arrive late, and not everyone on the Sleep Floor remembers what their body did.

The game itself comes in two forms. A terminal version is for demos and playtests: you watch the testimonies and choose what to investigate, while an AI that only sees the telemetry answers your questions. A web version is made to embed in a Grafana dashboard panel right next to the data.

Why a murder mystery?

Investigating an incident is investigating an incident, whether the victim is a production service or a night supervisor. You start with a vague report, a pile of signals that don’t quite agree, and a clock that can’t be fully trusted, and you have to build a timeline you can defend. A case with a known answer is a safe way to practice that, to compare how different tools and AI agents cope with it, and to see how good a question you have to ask before the data answers.

No spoilers

The repository keeps the solution in clearly marked files, and the README tells you which ones to avoid if you want to play. This page deliberately says nothing about who did it.

Try it

The repository has everything: the case brief, the telemetry, the loader, and both versions of the game.

obsidian-grafana

Mon, 13 Oct 2025 00:00:00 +0000

obsidian-grafana turns an Obsidian vault into time-series data and dashboards. I work at Grafana Labs, helping people monitor their systems with open source tools, and at some point I realized I could use the same tools to monitor my personal knowledge management system. It’s a nerdy and totally unnecessary setup, and I enjoy it a lot. I mention it in my post on how I use visuals in my PKM.

The screenshot above was made by running the repo’s own Grafana setup on a copy of the posts from this website.

How it works

A Python script reads the Markdown files in your vault and extracts metadata: word, line, and character counts, file size, frontmatter fields, tags, wikilinks, and timestamps. It runs every five minutes and has two paths:

A log path for history. Only the notes that changed since the last run are written, as structured JSON, to a log file. Grafana Alloy tails that file and forwards it to Loki, with tags and frontmatter fields as labels, so you can see how your notes change over time.
A metrics path for the current state. A metrics exporter exposes totals such as note count, word count, unique tags, and wikilinks, and Prometheus scrapes them.

Grafana ties the two together with pre-configured dashboards: an overview, a time series dashboard for individual notes, a relationships dashboard for wikilinks, a dashboard of frontmatter types, and a word cloud of your most used words. Everything runs in Docker Compose.

What it’s useful for

Understanding your writing habits and how your vault grows, checking how consistently your notes are tagged, finding the notes that everything links to, and keeping an eye on notes with a given frontmatter status or priority. The README also covers installing the Grafana Assistant plugin into the local Grafana.

Try it

The repository has a quick start. You’ll need Docker and Python 3, and a vault to point it at.

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

Wed, 05 Nov 2025 03:07:38 +0100

Today I gave a talk at Øredev 2025 in Malmö, Sweden, titled +The Spice Must Flow: The Fremen Guide to Sustainable Observability"_. The presentation focused on how to implement sustainable observability practices using open-source tools like Grafana, Prometheus, Loki, Tempo, KEDA, and Kepler. Here’s the abstract:

Nobody wants to be wasteful. But how do you balance that with the need for enough data to fix things when they go wrong? After all, no matter how much it costs, there are SLOs to keep. The spice– the telemetry– must flow.

The Fremen are a people who live on the planet Arrakis, simultaneously rich in spice while poor in water. But the Fremen have managed to thrive, not just survive, on very little while continuing to produce spice. What would it take to apply this “desert planet thinking” to observability?

In this talk, you’ll learn about the hidden environmental and financial costs of your observability stack and how to reduce them. You’ll learn about the complications of measuring cost for ephemeral resources on Kubernetes, how to use KEDA to “right-size” resources when you don’t know what the right size is, and how to convince your teams to drop all but the most meaningful telemetry data. In the end, the spice must flow, but there must still be enough water to sustain all.

Here are some resources mentioned in the talk:

You can play the game Spice Runner here
The spice-runner repo containing everything you need to know about how to deploy it yourself
The slides
The Spice Runner dashboard for observing the application
More about Spice Runner: architecture, tech stack, and what it demonstrates

One Year Later: How I Use Visuals in My PKM (Sketch Your Mind Conference)

Mon, 13 Oct 2025 13:33:18 -0700

Today I gave a talk at my friend Zsolt Viczián’s Sketch Your Mind Conference about how I use visuals in my PKM, a follow-up to my video about Obsidian Excalidraw last year where I shared that I’d changed all my Obsidian templates to hybrid Obsidian Excalidraw notes:

In the talk, I shared how I’ve evolved my use of visuals in my PKM over the last year, including:

Referring to icon and logo libraries that dynamically compile all the atomic visuals I use in my notes
Adopting a visual Zettelkasten-like approach
Creating notes at conferences to track people I’ve met and talks I’ve attended
Using Obsidian Excalidraw to create compelling presentations in lieu of slides
Combining visuals with audio recordings to create literature notes of books I’m reading
Using a full observability pipeline to monitor and analyze the notes in my Obsidian vault

I was particularly excited about the last point. I work at Grafana Labs and help people monitor their computer systems using opensource software. I realized that I could use the same tools to monitor my PKM system, so I set up a pipeline where a Python script scrapes note metadata, and sends information to Loki and Prometheus via Alloy, and then I visualize it all in Grafana. It’s a pretty nerdy and totally unnecessary setup, but here’s the repo for anyone interested in replicating it.

State of the Observability Databases (Grafana Office Hours #30)

Thu, 15 May 2025 18:22:26 +0100

In this Grafana Office Hours, we talk about the state of observability databases (Grafana Loki, Mimir, Tempo, and Pyroscope) and where they’re going. We talk about current and upcoming architectural changes in all four, how we’re making them more performant, how compatible they are with OpenTelemetry, and what we’re working on next for each database. In this conversation are Dee Kitchen (VP of Engineering for Databases) and Senior Developer Advocates Jay Clifford and Nicole van der Hoeven.

How to do continuous profiling right with Grafana Pyroscope and Ryan Perry, (Grafana Office Hours #26)

Fri, 19 Jan 2024 18:09:01 +0100

My friend Paul Balogh and I interview Ryan Perry, co-founder of Grafana Pyroscope, about how to do continuous profiling right. Ryan, and the rest of the Pyroscope team, joined Grafana Labs, where he is now a Director of Engineer. Ryan discusses the main concerns in continuous profiling and how to avoid those pitfalls. Pyroscope is an open-source project for aggregating continuous profiling data about your system’s resources.

Entendiendo los sintéticos (Café con Grafana #008)

Thu, 11 Jan 2024 16:00:25 +0100

En este episodio de Café con Grafana les traemos a Marcelo Magallón, experto en monitoreo sintético, quien viene precisamente a contarnos que es en sí un sintético. Él nos va a mostrar algunos ejemplos, casos de uso y mucha mas diversión!

Lee la documentación sobre el monitoreo de sintéticos en Grafana Cloud aquí.

AI Microscopy with Grafana, Theia Scientific, and Volkov Labs (Grafana Office Hours #24)

Fri, 15 Dec 2023 17:42:58 +0100

What do you get when you combine AI microscopy with Grafana? Well, in this case, you get the Theiascope platform: an application for doing real-time analysis on microscopy images using a combination of Grafana, Prometheus, and PostgreSQL/Timescale. The Theiascope platform was a collaboration between Dr. Christopher Field, Co-Founder/President/ Principal Investigator at Theia Scientific and Mikhail Volkov, Founder/CEO of Volkov Labs.

Repasemos el G-stack (Café con Grafana #005)

Thu, 30 Nov 2023 13:57:52 +0100

En este episodio de Café con Grafana tenemos a Raul Marin, Solutions Engineer (Ingeniero de soluciones?) quien viene a darnos una repasada completisima de todo el stack de Grafana. TODO! Pues hay muchos que no conocen todas las herramientas, caracteristicas y posibilidades que Grafana Labs viene a ofrecer.

Combining Frontend and Backend Performance With John Hill (Grafana Office Hours #18)

Fri, 03 Nov 2023 12:30:43 +0100

In this episode of Grafana Office Hours, my colleagues Marie Cruz, Paul Balogh, and I talk to John Hill, a Web UI Test Engineer and Grafana k6 champion, to talk about how the Grafana and k6 ecosystems can be used to ensure performance in mission-critical applications like NASA’s Open MCT.

What is Grafana Agent? (Grafana Office Hours #16)

Fri, 20 Oct 2023 12:55:11 +0200

What is Grafana Agent? Senior Software Engineer Paschalis Tsilias tells us about what telemetry collectors do, why they’re useful, what you’d have to do without them, and how to get started with Grafana Agent. He is joined by Developer Advocates Nicole van der Hoeven and Paul Balogh.

Useful links

(ES) Inicia (Café con Grafana #000)

Thu, 19 Oct 2023 12:37:00 +0200

Tenemos un nuevo show para toda la comunidad de habla en Español! En el vamos a estar invitando a Grafanaistas, Champions, Entusiastas y a miembros de la comunidad. Aquí vamos a compartir experiencias, conocimientos, perspectivas, diversion y mucho Español!

Agentless Monitoring for Prometheus in Grafana Cloud (Grafana Office Hours #15)

Fri, 13 Oct 2023 10:56:15 +0200

Did you know you could do agentless monitoring for Prometheus in Grafana Cloud? Senior Software Engineer Matt Nolf talks to me and my coworker Paul Balogh about the new Metrics Endpoint integration and how you can use it to collect Prometheus metrics from publicly addressable hosts– WITHOUT using Grafana Agent.

Resources

OpenTelemetry demo app with Grafana, Tempo, Loki, and Prometheus (Grafana Office Hours #06)

Fri, 28 Jul 2023 20:25:37 +0200

DevOps Engineer Blueswen Liu 劉義瑋 joins us to walk us through some OpenTelemetry demo apps he created, instrumented with Grafana, Loki, Prometheus, and Tempo.

How to Make Kubernetes Clusters Production Ready using Crossplane, with Viktor Farcic (k6 Office Hours #46)

Fri, 18 Mar 2022 17:30:45 +0100

In this video, Viktor Farcic, Paul Balogh and I talk about how to make Kubernetes clusters production-ready. Viktor is a Developer Advocate from Upbound, and he demonstrates a solution that ties together Crossplane, Grafana, Prometheus, and Loki to deploy and manage clusters in production.

Everyone Can Contribute: Load Performance Testing With k6

Tue, 23 Nov 2021 16:57:26 +0100

This is a recording of a live meetup I spoke at, the Everybody Can Contribute Café hosted by Michael Friedrich, Senior Developer Evangelist at GitLab. This was a more casual conversation and show-and-tell about k6’s capabilities, and I really enjoyed it!

Basics of load testing with k6 and Grafana in 20 minutes

Tue, 12 Oct 2021 01:04:02 +0200

This was a video I pre-recorded in preparation for the first Grafana EMEA virtual meetup.

In mid-June (just three months ago), k6 joined the Grafana Labs family. Many people didn’t quite understand what the connection was between k6 and Grafana, but I thought it made a lot of sense.

Google’s Site Reliability Engineering book talks about 4 golden signals that we should measure in a system: latency, traffic, errors, and saturation. Load testing provides 3 of those (latency, traffic, and errors), but it doesn’t show saturation, and it doesn’t give us visibility into the system.

Grafana brings observability. k6 brings data. They work really well together, which is why we’ve been using each other’s products for a few years now, before there was ever any talk of an acquisition.

In this video, I talk about why k6 and Grafana are better together and also demonstrate:

how to run a simple script in k6 OSS
how to make sense of the results of a load test
test execution options: running locally, sending results to k6 Cloud, and running a test from k6 Cloud– all triggered using the CLI
two different integration options to bring k6 and Grafana together: the k6 Cloud Grafana data source plugin and Prometheus Remote Write

Testing With k6 + Grafana: Better Together (Grafana virtual meetup EMEA)

Tue, 05 Oct 2021 21:49:13 +0200

In this first EMEA session of the Grafana Labs virtual meetups, I talked about how to use k6 and Grafana together to get the full performance picture for a software application. Observability, using tools like Prometheus and Grafana, lets us see what’s going on inside the application, and load testing, using k6, helps us get those insights earlier on in software cycles, such as during pre-production.

Intro to Load Testing With k6 and Grafana (webinar)

Thu, 16 Sep 2021 22:11:16 +0200

Tonight I did a webinar with Grafana Labs– the first k6-related webinar since the acquisition. I was more than a little nervous to give it, because I knew I was introducing k6 to a lot of new users, but also to Grafana for the first time. I think it went pretty well! I was still quite relieved when it was over, however. :)

In the webinar, I talk about:

why it’s still a good idea to do load testing in pre-production
what k6 is
why k6 and Grafana together make sense
and how to integrate k6 and Grafana

There’s quite a bit of demo in it, and I also had the k6 CEO, Robin Gustafsson, join me for Q&A as a nice treat.

Check out the full webinar here– it’s already up on the Grafana site. You can also go through my slides here, if you’d like.

Schrödinger's Pokémon: k6 and Grafana edition

Wed, 15 Sep 2021 18:22:31 +0200

Earlier this year, I gave a presentation at New Relic’s FutureStack conference about chaos engineering with k6 and New Relic. A few months ago, k6 got acquired by Grafana Labs, and so I wanted to redo it, but with the Grafana stack this time.

Here’s the result, and I think I like this iteration even better:

I presented this talk at TestCon Europe 2021 today, although the audio sync issues were not great, so I recommend you watch the video above instead.

Slides are here.

Monitoring your k6 load test: how to install Grafana and Prometheus on a Kubernetes cluster

Wed, 11 Aug 2021 18:55:49 +0200

Running a load test is half of the equation, and not having the right observability tools to see how your application responds to the load can sometimes flat-out invalidate a test.

Here’s how to install open-source tools Grafana and Prometheus on a Kubernetes cluster in DigitalOcean and how a performance user could use them for increased visibility during a load test.

Prometheus on Nicole van der Hoeven

The Defiant Jazz Incident

How it works

Why a murder mystery?

No spoilers

Try it

obsidian-grafana

How it works

What it’s useful for

Try it

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

One Year Later: How I Use Visuals in My PKM (Sketch Your Mind Conference)

State of the Observability Databases (Grafana Office Hours #30)

How to do continuous profiling right with Grafana Pyroscope and Ryan Perry, (Grafana Office Hours #26)

Links

Entendiendo los sintéticos (Café con Grafana #008)

AI Microscopy with Grafana, Theia Scientific, and Volkov Labs (Grafana Office Hours #24)

Repasemos el G-stack (Café con Grafana #005)

Combining Frontend and Backend Performance With John Hill (Grafana Office Hours #18)

What is Grafana Agent? (Grafana Office Hours #16)

Useful links

(ES) Inicia (Café con Grafana #000)

Agentless Monitoring for Prometheus in Grafana Cloud (Grafana Office Hours #15)

Resources

OpenTelemetry demo app with Grafana, Tempo, Loki, and Prometheus (Grafana Office Hours #06)

How to Make Kubernetes Clusters Production Ready using Crossplane, with Viktor Farcic (k6 Office Hours #46)

Everyone Can Contribute: Load Performance Testing With k6

Basics of load testing with k6 and Grafana in 20 minutes

Testing With k6 + Grafana: Better Together (Grafana virtual meetup EMEA)

Intro to Load Testing With k6 and Grafana (webinar)

Schrödinger's Pokémon: k6 and Grafana edition

Monitoring your k6 load test: how to install Grafana and Prometheus on a Kubernetes cluster