Faro on Nicole van der Hoeven

Spice Runner

Wed, 05 Nov 2025 00:00:00 +0000

Spice Runner is a browser-based endless runner. Think of the “unable to connect” dinosaur game in Chrome, except you are a Fremen running across the sands of Arrakis, jumping sandworms and Harkonnen ornithopters. It’s the demo app for my talk The Spice Must Flow: The Fremen Guide to Sustainable Observability, and it exists to make one point: even a simple, throwaway game generates a surprising amount of telemetry, and that telemetry has real financial and environmental costs.

The Fremen live on a planet that is rich in spice and poor in water, and they thrive by wasting nothing. Our systems are rich in telemetry and poor in budget, and in carbon. Spice Runner is a small, playable version of that problem: a real app, with real players, running on real infrastructure that scales up and down with them while I watch what it costs.

What it demonstrates

Real user monitoring with Grafana Faro. Game sessions, game starts, scores, collisions, restarts, and game overs are all sent as custom events, so the dashboard shows how people actually play.
A leaderboard that’s instrumented end to end. A Go API backed by PostgreSQL and Redis serves a public top-10 leaderboard. It’s instrumented with OpenTelemetry for traces and custom metrics, so a score can be followed from the browser through the API to the database and the cache.
Metrics, logs, and traces on Kubernetes. Prometheus, Grafana Loki, and Grafana Tempo run in the cluster, with Grafana Alloy as the collector and Grafana for dashboards. Telemetry is also sent to Grafana Cloud.
Autoscaling that follows the players. KEDA scales the game’s pods on HTTP request rate, CPU, and memory, and the GKE Cluster Autoscaler adds and removes nodes behind them. Production settings keep at least one pod running and cap it at 20, with a separate demo profile that is far more eager for live demonstrations.
Energy monitoring with Kepler. Power and energy use are tracked per workload, so “what does this cost the planet?” is a query rather than a guess.
Load testing with k6. k6 drives load against the leaderboard API and, with a browser script, plays the game to keep the dashboard populated.

The dashboard

The live dashboard on Grafana Play shows real player data as it happens. The dashboard is also entirely AI-generated: it was built with Grafana Assistant.

Open the live dashboard

Architecture

The app has two layers. The first is the game and its leaderboard: a vanilla JavaScript game served by Nginx, with a Go leaderboard API backed by PostgreSQL and Redis. The second is the observability stack: Faro instruments the browser, and the Nginx exporter, Alloy, Loki, Tempo, and Prometheus cover the backend. Everything runs on Google Kubernetes Engine, with persistent volumes so telemetry survives pod restarts, and autoscalers at both the pod and node level.

The talk

The Spice Must Flow uses Dune as its framing. It covers the hidden environmental and financial costs of an observability stack, how hard it is to measure the cost of ephemeral resources on Kubernetes, how to use KEDA to right-size resources when you don’t know what the right size is, and the most effective thing you can do to cut cost: deleting the telemetry you don’t need.

I first gave it at Øredev 2025 in Malmö, Sweden. You can read the write-up and abstract and see the slides.

Run it yourself

Everything you need to deploy it on your own GKE cluster, including the Kubernetes manifests, the dashboards, and the load tests, is in the GitHub repository. Its README has a quick start.

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

Wed, 05 Nov 2025 03:07:38 +0100

Today I gave a talk at Øredev 2025 in Malmö, Sweden, titled +The Spice Must Flow: The Fremen Guide to Sustainable Observability"_. The presentation focused on how to implement sustainable observability practices using open-source tools like Grafana, Prometheus, Loki, Tempo, KEDA, and Kepler. Here’s the abstract:

Nobody wants to be wasteful. But how do you balance that with the need for enough data to fix things when they go wrong? After all, no matter how much it costs, there are SLOs to keep. The spice– the telemetry– must flow.

The Fremen are a people who live on the planet Arrakis, simultaneously rich in spice while poor in water. But the Fremen have managed to thrive, not just survive, on very little while continuing to produce spice. What would it take to apply this “desert planet thinking” to observability?

In this talk, you’ll learn about the hidden environmental and financial costs of your observability stack and how to reduce them. You’ll learn about the complications of measuring cost for ephemeral resources on Kubernetes, how to use KEDA to “right-size” resources when you don’t know what the right size is, and how to convince your teams to drop all but the most meaningful telemetry data. In the end, the spice must flow, but there must still be enough water to sustain all.

Here are some resources mentioned in the talk:

You can play the game Spice Runner here
The spice-runner repo containing everything you need to know about how to deploy it yourself
The slides
The Spice Runner dashboard for observing the application
More about Spice Runner: architecture, tech stack, and what it demonstrates

Real User Monitoring With Grafana Faro (Grafana Office Hours #17)

Fri, 27 Oct 2023 07:00:19 +0200

My friend Marie Cruz and I chat with Software Engineers Kostas Pelelis and Marco Schaefer to discuss what Grafana Faro is, why it’s important to have a real user monitoring solution, and how to get started with it.