Kubernetes on Nicole van der Hoeven

Spice Runner

Wed, 05 Nov 2025 00:00:00 +0000

Spice Runner is a browser-based endless runner. Think of the “unable to connect” dinosaur game in Chrome, except you are a Fremen running across the sands of Arrakis, jumping sandworms and Harkonnen ornithopters. It’s the demo app for my talk The Spice Must Flow: The Fremen Guide to Sustainable Observability, and it exists to make one point: even a simple, throwaway game generates a surprising amount of telemetry, and that telemetry has real financial and environmental costs.

The Fremen live on a planet that is rich in spice and poor in water, and they thrive by wasting nothing. Our systems are rich in telemetry and poor in budget, and in carbon. Spice Runner is a small, playable version of that problem: a real app, with real players, running on real infrastructure that scales up and down with them while I watch what it costs.

What it demonstrates

Real user monitoring with Grafana Faro. Game sessions, game starts, scores, collisions, restarts, and game overs are all sent as custom events, so the dashboard shows how people actually play.
A leaderboard that’s instrumented end to end. A Go API backed by PostgreSQL and Redis serves a public top-10 leaderboard. It’s instrumented with OpenTelemetry for traces and custom metrics, so a score can be followed from the browser through the API to the database and the cache.
Metrics, logs, and traces on Kubernetes. Prometheus, Grafana Loki, and Grafana Tempo run in the cluster, with Grafana Alloy as the collector and Grafana for dashboards. Telemetry is also sent to Grafana Cloud.
Autoscaling that follows the players. KEDA scales the game’s pods on HTTP request rate, CPU, and memory, and the GKE Cluster Autoscaler adds and removes nodes behind them. Production settings keep at least one pod running and cap it at 20, with a separate demo profile that is far more eager for live demonstrations.
Energy monitoring with Kepler. Power and energy use are tracked per workload, so “what does this cost the planet?” is a query rather than a guess.
Load testing with k6. k6 drives load against the leaderboard API and, with a browser script, plays the game to keep the dashboard populated.

The dashboard

The live dashboard on Grafana Play shows real player data as it happens. The dashboard is also entirely AI-generated: it was built with Grafana Assistant.

Open the live dashboard

Architecture

The app has two layers. The first is the game and its leaderboard: a vanilla JavaScript game served by Nginx, with a Go leaderboard API backed by PostgreSQL and Redis. The second is the observability stack: Faro instruments the browser, and the Nginx exporter, Alloy, Loki, Tempo, and Prometheus cover the backend. Everything runs on Google Kubernetes Engine, with persistent volumes so telemetry survives pod restarts, and autoscalers at both the pod and node level.

The talk

The Spice Must Flow uses Dune as its framing. It covers the hidden environmental and financial costs of an observability stack, how hard it is to measure the cost of ephemeral resources on Kubernetes, how to use KEDA to right-size resources when you don’t know what the right size is, and the most effective thing you can do to cut cost: deleting the telemetry you don’t need.

I first gave it at Øredev 2025 in Malmö, Sweden. You can read the write-up and abstract and see the slides.

Run it yourself

Everything you need to deploy it on your own GKE cluster, including the Kubernetes manifests, the dashboards, and the load tests, is in the GitHub repository. Its README has a quick start.

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Thu, 30 Jul 2026 09:00:00 +0900

This week I spoke at KubeCon + CloudNativeCon Japan 2026 in Yokohama, Japan. I gave a talk called The Great Doubt: What Building an AI Agent Taught Us About Trust. Here’s the abstract:

When we started building an AI assistant for observability, we thought the hard part would be making it smart. We were wrong. The hard part was knowing when to trust it.

AI agents hallucinate, forget context, and confidently give wrong answers. Traditional testing doesn’t catch these failures. What’s needed is evaluation grounded in systematic doubt.

This talk shares lessons from shipping an AI agent to production: how to build a golden dataset of use cases you must get right, how to use LLM-as-judge when there’s no ground truth, and how to use OpenTelemetry traces to debug eval failures. You’ll also hear what’s still unsolved: evaluating multi-agent handoffs and closing the feedback loop between what users ask and what your evals cover.

Kyoto School philosopher Nishitani Keiji called this “The Great Doubt” (大疑): questioning every assumption until only what survives is real. For AI agents, that’s not philosophy. It’s the job.

The talk walks through the layers of doubt we worked through while building Grafana Assistant: doubting the agent itself, then doubting our tests, then the LLM-as-judge, then our own scores — and finally doubting whether the doubt was even justified. Each layer needed a different tool, from a golden dataset run in a fully synthetic observability environment, through online evals on live traffic, to per-conversation OpenTelemetry traces that link every eval score back to the exact LLM and tool calls that produced it.

You can find the slides at nicole.to/doubtslides, and you can watch the full talk below:

Resources

Grafana Assistant (generally available for Grafana OSS, Grafana Cloud, and Grafana Play):

Assistant Investigations (Grafana Cloud):

Docs

Agent Observability (Grafana Cloud):

Docs

o11y-bench (OSS):

Site

Investigations Arena:

Video

My notes on Grafana Assistant
My Context Horizon piece Unlearnings from building Grafana Assistant
The Way of Zen by Alan W. Watts
Zen Mind, Beginner’s Mind by Shunryu Suzuki
Religion and Nothingness, Volume 1 by Nishitani Keiji
Philosophers of Nothingness: An Essay on the Kyoto School by James W. Heisig

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

Wed, 05 Nov 2025 03:07:38 +0100

Today I gave a talk at Øredev 2025 in Malmö, Sweden, titled +The Spice Must Flow: The Fremen Guide to Sustainable Observability"_. The presentation focused on how to implement sustainable observability practices using open-source tools like Grafana, Prometheus, Loki, Tempo, KEDA, and Kepler. Here’s the abstract:

Nobody wants to be wasteful. But how do you balance that with the need for enough data to fix things when they go wrong? After all, no matter how much it costs, there are SLOs to keep. The spice– the telemetry– must flow.

The Fremen are a people who live on the planet Arrakis, simultaneously rich in spice while poor in water. But the Fremen have managed to thrive, not just survive, on very little while continuing to produce spice. What would it take to apply this “desert planet thinking” to observability?

In this talk, you’ll learn about the hidden environmental and financial costs of your observability stack and how to reduce them. You’ll learn about the complications of measuring cost for ephemeral resources on Kubernetes, how to use KEDA to “right-size” resources when you don’t know what the right size is, and how to convince your teams to drop all but the most meaningful telemetry data. In the end, the spice must flow, but there must still be enough water to sustain all.

Here are some resources mentioned in the talk:

You can play the game Spice Runner here
The spice-runner repo containing everything you need to know about how to deploy it yourself
The slides
The Spice Runner dashboard for observing the application
More about Spice Runner: architecture, tech stack, and what it demonstrates

Deploying Loki on Kubernetes via Helm (Loki Community Call October 2025)

Wed, 29 Oct 2025 18:04:30 +0100

This Loki Community Call is about deploying Loki on Kubernetes via Helm charts. We talk about why you might want to use Helm to deploy on Kubernetes, best practices for deployment, and which Helm chart you should use! We are Jay Clifford and Nicole van der Hoeven, Developer Advocates at Grafana Labs, and we have invited Grafana Champion and Loki Helm Maintainer Jan-Otto Kröpke, Principal Cloud Architect at QualityOperations GmbH, to talk about the state of the Loki Helm Chart.

All about OTel and Logging on Kubernetes with Loki (Loki Community Call April 2025)

Fri, 25 Apr 2025 17:57:58 +0100

In this pre-recorded Loki Community Call, we talk all about OTel and logging on Kubernetes with Cyril Tovena, Ward Bekker, Jay Clifford, and Nicole van der Hoeven at KubeCon EU 2025 in London. We discuss when why you should switch to OTel and why you shouldn’t, what OTLP is exactly, and best practices for ingesting data through an OTLP endpoint.

How to Run Loki at Scale on Kubernetes (Loki Community Call January 2025)

Thu, 23 Jan 2025 17:51:00 +0100

Happy New Year from the Loki Engineering team. To kick off 2025, Nicole and Jay will be joined by Poyzan Taneli from the Loki Engineering team to discuss how to run Loki at scale on Kubernetes. If you are currently running Loki in microservices mode or preparing to do so, we will be discussing best practices for scaling its components to meet the demands of production use cases.

How many Ingesters vs Queriers do I need?
What is the minimum cost of running Loki as a production cluster?
What is the average node size for running Loki? How do I monitor my Loki deployment?

How to Deploy Grafana Beyla on Kubernetes as a Sidecar Container

Tue, 09 Jan 2024 13:41:38 +0100

Principal Staff Engineer Nikola Grčevski demonstrates how to deploy Grafana Beyla on Kubernetes as a sidecar container. Grafana Beyla is an eBPF auto-instrumentation tool for application observability: you just have to tell it what to listen to, give it permissions, and deploy it– and you can see metrics and logs from that component or service without having to do any manual instrumentation.

Load Testing on Kubernetes with k6 Private Load Zones (Grafana Office Hours #19)

Fri, 10 Nov 2023 10:46:48 +0100

This week, we’re talking about how you can do load testing on Kubernetes with k6 Private Load Zones, a new feature on Grafana Cloud k6 that leverages the k6 Kubernetes operator to allow you to run distributed load tests against applications behind a firewall. Here to discuss this new feature are Senior Software Engineer Olha Yevtushenko, Product Manager Daniel González Lopes, Developer Advocate Paul Balogh, and Senior Developer Advocate Nicole van der Hoeven.

How to Deploy Grafana on Kubernetes (Grafana Office Hours #13)

Fri, 29 Sep 2023 17:30:49 +0200

My coworker Usman Ahmad and I talk about how to deploy Grafana on Kubernetes for beginners: what Kubernetes is, how it’s an evolution of distributed computing, what its relationship to Docker is, other things you might need to know to work with Kubernetes, and how you can deploy Grafana on your own Kubernetes clusters.

Documentation on how to deploy Grafana on Kubernetes here.

Testing Kubernetes Networking With k6 (k6 Office Hours #91)

Fri, 16 Jun 2023 13:00:50 +0200

How would you go about testing Kubernetes networking? For his initial project with k6, our new Senior Software Engineer, Roberto Santalla, talks to us about how he used k6 to improve end-to-end network throughput and some insights he learned about Kubernetes networking along the way. I also invited my colleague, Leandro Melendez, to help me interview Roberto.

How to Avoid the Most Common Kubernetes Monitoring Mistakes

Tue, 16 May 2023 22:41:16 +1000

For ObservabilityCON on the Road Singapore 2023, I talked about some mistakes people make when monitoring applications within Kubernetes clusters. I also demonstrated Grafana’s Kubernetes Monitoring solution and how it can be a shortcut for engineers looking to get started quickly.

Resources

Slides here
Monitoring Kubernetes in general (notes)
The Grafana Kubernetes Monitoring feature (notes)

Emergent Load Testing - rules for organized chaos

Wed, 19 Apr 2023 00:21:06 +0200

Today I spoke at KubeCon EU 2023 about emergent load testing.

Resources

Here are the slides.

Below are some links to extensions and experimental modules that I mentioned:

Grafana
k6 site
k6 repo
k6 docs
k6 browser (experimental module)
Prometheus Remote Write (experimental module)
k6 tracing (experimental module)
xk6-kubernetes (extension)
k6 redis (experimental module)
xk6 disruptor (extension)

Abstract

When we write load testing scripts against our applications, we write them sequentially: A, then B, then C. But this doesn’t accurately reflect the organized chaos of a system in production, nor does it prepare the system for the unexpected.

Emergence is a phenomenon where parts of a whole independently develop properties not originally present in the whole. Emergence is what helps ant workers develop roles without leadership, prompts animals to evolve adaptive traits without forethought, and facilitates non-toxic communities without moderators.

The growing field of emergent software applies this swarm logic to the programs that we write. What would it take to write emergent load testing scripts? It turns out that there are a few ingredients for emergence: a large population size, opportunities to interact, feedback, and an element of control.

In this talk, Nicole van der Hoeven discusses how to bring these elements to load testing by writing a script in Grafana k6 that can independently decide what requests to make next, modify Kubernetes app pods, and disrupt services based on a continual feed of results during runtime– all without manual intervention. She shows how to wield this new breed of load testing to improve confidence in the complex systems we build.

Distributed Load Testing Using Kubernetes With k6

Sun, 18 Dec 2022 15:54:07 +0000

My colleague Paul Balogh and I demonstrate how to run distributed load testing using Kubernetes with k6 and k6 operator. k6 can already run up to 40,000 virtual users from a single load generator, but if you want to scale that up further, you can use the Kubernetes operator to create a k6 custom resource within your cluster.

Why Is Everyone Talking About Chaos Engineering? (k6 Office Hours #56)

Fri, 15 Jul 2022 18:00:13 +0100

Why is everyone talking about chaos engineering anyway? How different is it from testing? What does it have to do with performance? In this k6 Office Hours, k6 Technical Program Manager Vince Huang joins me and my colleague, Paul Balogh, to talk about these topics and more.

How to migrate to Kubernetes, with Carlos Ruiz Lantero (k6 Office Hours #55)

Fri, 08 Jul 2022 17:00:57 +0100

k6 Cloud Backend developer Carlos Ruiz Lantero joins Developer Advocates Nicole van der Hoeven and Paul Balogh to discuss how to migrate to Kubernetes. Carlos has been working on migrating k6 Cloud services from ECS to EKS, and in this video, he shares his best practices for migration and things he wishes he’d done differently.

How to Make Kubernetes Clusters Production Ready using Crossplane, with Viktor Farcic (k6 Office Hours #46)

Fri, 18 Mar 2022 17:30:45 +0100

In this video, Viktor Farcic, Paul Balogh and I talk about how to make Kubernetes clusters production-ready. Viktor is a Developer Advocate from Upbound, and he demonstrates a solution that ties together Crossplane, Grafana, Prometheus, and Loki to deploy and manage clusters in production.

Monitoring your k6 load test: how to install Grafana and Prometheus on a Kubernetes cluster

Wed, 11 Aug 2021 18:55:49 +0200

Running a load test is half of the equation, and not having the right observability tools to see how your application responds to the load can sometimes flat-out invalidate a test.

Here’s how to install open-source tools Grafana and Prometheus on a Kubernetes cluster in DigitalOcean and how a performance user could use them for increased visibility during a load test.

How to get started with chaos engineering (k6 Office Hours)

Fri, 30 Apr 2021 21:53:09 +0100

Recently, I’ve been learning more about chaos engineering. In this week’s k6 Office Hours, I shared my experience as a tester getting started with Kubernetes, chaos engineering with xk6-chaos and k6, and observability with New Relic.

Testing Kubernetes and infrastructure as code with Sid Palas (k6 Office Hours)

Fri, 09 Apr 2021 21:53:09 +0100

For this week’s k6 Office Hours, Simme and I had Sid Palas from DevOps Directive come on board to talk about what Kubernetes is, how it relates to Infrastructure as Code, and why either of those should matter to testers.

Kubernetes on Nicole van der Hoeven

Spice Runner

What it demonstrates

The dashboard

Architecture

The talk

Run it yourself

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Resources

Related reading

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

Deploying Loki on Kubernetes via Helm (Loki Community Call October 2025)

All about OTel and Logging on Kubernetes with Loki (Loki Community Call April 2025)

How to Run Loki at Scale on Kubernetes (Loki Community Call January 2025)

How to Deploy Grafana Beyla on Kubernetes as a Sidecar Container

Links

Load Testing on Kubernetes with k6 Private Load Zones (Grafana Office Hours #19)

Related links

How to Deploy Grafana on Kubernetes (Grafana Office Hours #13)

Testing Kubernetes Networking With k6 (k6 Office Hours #91)

How to Avoid the Most Common Kubernetes Monitoring Mistakes

Resources

Emergent Load Testing - rules for organized chaos

Resources

Abstract

Distributed Load Testing Using Kubernetes With k6

Why Is Everyone Talking About Chaos Engineering? (k6 Office Hours #56)

How to migrate to Kubernetes, with Carlos Ruiz Lantero (k6 Office Hours #55)

How to Make Kubernetes Clusters Production Ready using Crossplane, with Viktor Farcic (k6 Office Hours #46)

Monitoring your k6 load test: how to install Grafana and Prometheus on a Kubernetes cluster

How to get started with chaos engineering (k6 Office Hours)

Testing Kubernetes and infrastructure as code with Sid Palas (k6 Office Hours)