Tempo on Nicole van der Hoeven

The Defiant Jazz Incident

Fri, 30 Oct 2026 00:00:00 +0000

The Defiant Jazz Incident is a murder mystery you solve with observability tools. It’s set on the Sleep Floor at Kipsake, a company that rents your sleep: at night, its Dreamers sleep in pods while their sleepwalking selves work the floor. During the Cocoa Hour, the floor’s middle-of-the-night holiday party, the power goes out for five minutes. When it comes back, the night supervisor is dead in the elevator.

Kipsake recorded everything it could: badges, doors, chat, sleep rings, power, microphones, cameras, sleep bands, and personnel records. Your job is to work out who did it, and the only way to do that is through the telemetry.

How it works

The case file is a set of logs, metrics, and traces. A loader sends them to Grafana Loki, Prometheus or Mimir, and Grafana Tempo, either in Grafana Cloud or in a local Docker stack, and you investigate from there with whatever you like: Grafana itself, Grafana Assistant, an MCP server, gcx, or the AI agent of your choice. The data is deliberately messy. Some devices lose clock sync during the outage, some records arrive late, and not everyone on the Sleep Floor remembers what their body did.

The game itself comes in two forms. A terminal version is for demos and playtests: you watch the testimonies and choose what to investigate, while an AI that only sees the telemetry answers your questions. A web version is made to embed in a Grafana dashboard panel right next to the data.

Why a murder mystery?

Investigating an incident is investigating an incident, whether the victim is a production service or a night supervisor. You start with a vague report, a pile of signals that don’t quite agree, and a clock that can’t be fully trusted, and you have to build a timeline you can defend. A case with a known answer is a safe way to practice that, to compare how different tools and AI agents cope with it, and to see how good a question you have to ask before the data answers.

No spoilers

The repository keeps the solution in clearly marked files, and the README tells you which ones to avoid if you want to play. This page deliberately says nothing about who did it.

Try it

The repository has everything: the case brief, the telemetry, the loader, and both versions of the game.

Asimov's Zeroth Law of Robotics: Observability for AI (ExpoQA Madrid 2026)

Tue, 26 May 2026 10:00:00 +0200

This week I gave the opening keynote at ExpoQA Madrid 2026 in Madrid, Spain. I presented a talk called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously given at KubeCon EU 2025 in London, at Dutch Cloud Native Day 2025 in Utrecht, and at Newcrafts in Paris. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

This was a keynote slot at a testing-focused audience, which gave me the chance to lean more into the evals vs. observability framing — what evals can and can’t tell you about an AI system in production, and how distributed tracing fills the gap that benchmarks leave behind. The demo app (a Flask-based “play.py” instrumented with OpenTelemetry, flowing into Tempo for traces, Prometheus for metrics, and Loki for logs) is still on GitHub, and I’ve been refining it with each iteration.

Here’s the recording of the talk:

Resources

Demo app on GitHub

Asimov's Zeroth Law of Robotics: Observability for AI (Newcrafts 2025)

Thu, 06 Nov 2025 12:23:19 +0100

Today I presented at Newcrafts in Paris, France. I gave a talk I’ve given before, called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU in London and at Dutch Cloud Native Day 2025 in Utrecht. I was happy to present it again, as it gave me the chance to further refine my demo application, which you can find on GitHub.

The main change I made was to use the “AI as judge” approach that I had previously only talked about in theory. In this version of the demo app, I use AI in both the game as well as in the testing of the game, showing how AI can be integrated into both the application and its observability/testing stack. It was really interesting to note that testing with AI ended up exposing more defects than my hardcoded test cases did, which goes to show the potential of AI-driven testing.

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

Wed, 05 Nov 2025 03:07:38 +0100

Today I gave a talk at Øredev 2025 in Malmö, Sweden, titled +The Spice Must Flow: The Fremen Guide to Sustainable Observability"_. The presentation focused on how to implement sustainable observability practices using open-source tools like Grafana, Prometheus, Loki, Tempo, KEDA, and Kepler. Here’s the abstract:

Nobody wants to be wasteful. But how do you balance that with the need for enough data to fix things when they go wrong? After all, no matter how much it costs, there are SLOs to keep. The spice– the telemetry– must flow.

The Fremen are a people who live on the planet Arrakis, simultaneously rich in spice while poor in water. But the Fremen have managed to thrive, not just survive, on very little while continuing to produce spice. What would it take to apply this “desert planet thinking” to observability?

In this talk, you’ll learn about the hidden environmental and financial costs of your observability stack and how to reduce them. You’ll learn about the complications of measuring cost for ephemeral resources on Kubernetes, how to use KEDA to “right-size” resources when you don’t know what the right size is, and how to convince your teams to drop all but the most meaningful telemetry data. In the end, the spice must flow, but there must still be enough water to sustain all.

Here are some resources mentioned in the talk:

You can play the game Spice Runner here
The spice-runner repo containing everything you need to know about how to deploy it yourself
The slides
The Spice Runner dashboard for observing the application
More about Spice Runner: architecture, tech stack, and what it demonstrates

Asimov's Zeroth Law of Robotics (Dutch Cloud Native Day 2025)

Thu, 03 Jul 2025 15:01:24 +0200

Today I presented at the Dutch Cloud Native Day 2025 in Utrecht, the Netherlands. I gave a talk titled Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU earlier on in the year, in London. I was very happy to be able to present it again, because it gave me the opportunity to improve the demo app I used, which you can find on GitHub.

State of the Observability Databases (Grafana Office Hours #30)

Thu, 15 May 2025 18:22:26 +0100

In this Grafana Office Hours, we talk about the state of observability databases (Grafana Loki, Mimir, Tempo, and Pyroscope) and where they’re going. We talk about current and upcoming architectural changes in all four, how we’re making them more performant, how compatible they are with OpenTelemetry, and what we’re working on next for each database. In this conversation are Dee Kitchen (VP of Engineering for Databases) and Senior Developer Advocates Jay Clifford and Nicole van der Hoeven.

Asimov's Zeroth Law of Robotics: Observability for AI (KubeCon EU 2025)

Wed, 02 Apr 2025 13:46:42 +0200

Today I gave a talk at KubeCon EU 2025 in London called “Asmiov’s Zeroth Law of Robotics: Observability for AI”. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

Cartography of the Deep: Distributed Tracing for Testing (Romanian Testing Conference)

Thu, 13 Jun 2024 12:24:02 -0700

I presented a talk today at the Romanian Testing Conference 2024 in Cluj-Napoca, Romania! My talk was called “Cartography of the Deep: Distributed Tracing for Testing” and it covered how to tie in distributed tracing into tests to get a better understanding of your system under test.

Here are my slides!

Abstract

From the Scandinavian Kraken that drags entire ships to their early graves to the Philippine Bakunawa said to rise from the waters to swallow the moon, our existential fears about what lurks at the bottom of our own oceans has inspired us to create hundreds of folkloric monsters across cultures. We’ve charted the surface of our planet, but the vast majority of the ocean floor remains a mystery.

The mysterious depths of modern software architectures makes it difficult for us to test a system for performance and reliability. The easy solution? Unmanned underwater probes: send inputs to the system, and judge its state based on the outputs that return. But this method is often too limited to tell us what’s really going on when something goes wrong. What if there were a way to make our tests themselves more observable?

In this talk, you will learn how to use open-source tools to instrument performance tests, identify bottlenecks at the code level using distributed tracing, and map out a system using a type of software bathymetry. You’ll also learn about the Mariana Trench expeditions, the difficulties of operating in a high-pressure underwater environment, and how to come face to face with monsters of our mythologies.

How to Get Started with Tempo (Grafana Office Hours #22)

Fri, 01 Dec 2023 11:55:55 +0100

Grafana Labs Principal Engineer and Tempo maintainer Joe Elliot talks about how to get started with Tempo, what distributed tracing is, and why TraceQL is cool. He is joined by Developer Advocates Nicole van der Hoeven and Paul Balogh.

Resources

Getting started with Tempo (video)
Grafana Tempo repo
Grafana Tempo docs
The Tempo category on the Grafana community forum
Video playlist for Grafana Tempo
Distributed tracing with Grafana Cloud k6 (Grafana Office Hours video)

(ES) Inicia (Café con Grafana #000)

Thu, 19 Oct 2023 12:37:00 +0200

Tenemos un nuevo show para toda la comunidad de habla en Español! En el vamos a estar invitando a Grafanaistas, Champions, Entusiastas y a miembros de la comunidad. Aquí vamos a compartir experiencias, conocimientos, perspectivas, diversion y mucho Español!

Distributed tracing with Grafana Cloud k6 (Grafana Office Hours #14)

Sat, 07 Oct 2023 08:31:03 +0200

My friend Marie Cruz and I interview Senior Software Engineer Łukasz Gut about a new feature: Distributed Tracing with Grafana Cloud k6. He discusses what distributed tracing is, why it matters, and how it can help teams find reliability issues faster.

Load testing and tracing with k6 and Tempo

Thu, 18 May 2023 22:41:42 +1000

At ObservabilityCON on the Road 2023 in Sydney, Australia, I talked about why you should use load testing and distributed tracing, how to do it with k6 and Tempo, and the benefits of being able to see beyond traditional black-box testing.

Resources

Distributed Tracing With Grafana k6 (k6 Office Hours #70)

Fri, 04 Nov 2022 14:03:04 +0100

Just announced: distributed tracing with Grafana k6 x Tempo! Daniel González Lopes joins us to talk about what distributed tracing is, and how traces using the Tempo backend could help you troubleshoot your performance issues and improve overall reliability. This new integration was just announced at ObservabilityCon 2022, and Daniel’s here to tell us all about it!