Llm on Nicole van der Hoeven

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Thu, 30 Jul 2026 09:00:00 +0900

This week I spoke at KubeCon + CloudNativeCon Japan 2026 in Yokohama, Japan. I gave a talk called The Great Doubt: What Building an AI Agent Taught Us About Trust. Here’s the abstract:

When we started building an AI assistant for observability, we thought the hard part would be making it smart. We were wrong. The hard part was knowing when to trust it.

AI agents hallucinate, forget context, and confidently give wrong answers. Traditional testing doesn’t catch these failures. What’s needed is evaluation grounded in systematic doubt.

This talk shares lessons from shipping an AI agent to production: how to build a golden dataset of use cases you must get right, how to use LLM-as-judge when there’s no ground truth, and how to use OpenTelemetry traces to debug eval failures. You’ll also hear what’s still unsolved: evaluating multi-agent handoffs and closing the feedback loop between what users ask and what your evals cover.

Kyoto School philosopher Nishitani Keiji called this “The Great Doubt” (大疑): questioning every assumption until only what survives is real. For AI agents, that’s not philosophy. It’s the job.

The talk walks through the layers of doubt we worked through while building Grafana Assistant: doubting the agent itself, then doubting our tests, then the LLM-as-judge, then our own scores — and finally doubting whether the doubt was even justified. Each layer needed a different tool, from a golden dataset run in a fully synthetic observability environment, through online evals on live traffic, to per-conversation OpenTelemetry traces that link every eval score back to the exact LLM and tool calls that produced it.

You can find the slides at nicole.to/doubtslides, and you can watch the full talk below:

Resources

Grafana Assistant (generally available for Grafana OSS, Grafana Cloud, and Grafana Play):

Assistant Investigations (Grafana Cloud):

Docs

Agent Observability (Grafana Cloud):

Docs

o11y-bench (OSS):

Site

Investigations Arena:

Video

My notes on Grafana Assistant
My Context Horizon piece Unlearnings from building Grafana Assistant
The Way of Zen by Alan W. Watts
Zen Mind, Beginner’s Mind by Shunryu Suzuki
Religion and Nothingness, Volume 1 by Nishitani Keiji
Philosophers of Nothingness: An Essay on the Kyoto School by James W. Heisig

The Architecture of Forgetting

Thu, 07 May 2026 16:00:00 +0100

Originally posted here on Context Horizon.

In early April, a new version of everyone’s favourite lobster, OpenClaw, was shipped with an interesting new feature: dreaming. It’s a memory consolidation feature. But to understand why it matters, we need to back up.

Early AI chatbots, like the initial GPT wrappers, had zero memory. Every conversation started fresh. You asked a question, you got an answer, and you moved on: a glorified search engine.

Somewhere along the way we began to expect more and more of our chatbots. Models acquired context windows that became their de facto short-term memory. As the context window filled up, they remembered what had been said earlier in the conversation, making them better conversational partners. Still, it was ephemeral: close the tab, start fresh. And it had a hard ceiling: hit the limit, and the oldest things fall off. This felt like forgetting because it was. But it was also, accidentally, a form of curation. Finite space forced priority.

Then came the push for real persistence. RAG pipelines, vector databases, memory files, agent frameworks that write notes to themselves. Now our agents can remember that we prefer the Oxford comma, that we just started a new project at work, that our D&D character is named Zuhri. This felt like the solution. More memory = better agent.

That mostly works— until it doesn’t.

As it turns out, the problem with AI memory isn’t that agents forget too much. It’s that they forget too little.

Why remembering is hard

It turns out that having a persistent memory, even for an agent like OpenClaw that has that as one of its defining features, doesn’t solve all the memory problems.

I happen to have an OpenClaw agent that named itself Iris.

Iris does a lot of things for me, including parsing my meeting notes to identify next actions, curating my tasks and projects, and generating a daily briefing. She also creates a summary of the day’s work based on my meetings, conversations, and notes. One day she was giving me the daily summary, and here’s that conversation:

Iris: …Tonsil is finally healed after weeks of pain… Me: Wait what? Where did you get the tonsil stuff? Iris: From your memory file — it’s in the Apr 10 daily note context: “Tonsil finally healed — Nicole no longer in pain 🎉”. I’ve had that sitting in memory since the Apr 10 summary. Is that wrong? Was it something else that healed, or is the tonsil detail off? Me: I have no idea where you got that. I have never had issues with my tonsil.

Where did that come from? I’m still not sure, but I occasionally have meetings in cafés, and it’s possible that a meeting recording overheard the conversation of someone else in the café. That got faithfully transcribed, copied over into my notes, and written into Iris’s memory. Iris treated it as fact for days.

The problem isn’t that Iris forgot. It’s that she remembered the wrong thing… and didn’t know how to forget it.

Imagine this at scale: not one wrong fact, but dozens, quietly shaping every summary, every recommendation, every task Iris surfaces for me. The signal-to-noise ratio degrades until it doesn’t matter if Iris remembers mostly accurate memories— I won’t trust it.

So how do you build a memory system that knows what to keep? Researchers have been working on this longer than you might think.

Meet the residents of Smallville

Stanford researchers Park et al. released a charming study in 2023 called Generative Agents: Interactive Simulacra of Human Behavior in which they talk about how they created a simulated environment called Smallville, populated it with 25 generative agents, and then watched.

Source: Park et al. (2023)

The study was an attempt to simulate emergent social behavior, but the key insight I took from it isn’t the sim itself— it’s what they had to do to the inhabitants of Smallville to enable the rich social interactions they wanted to observe.

The researchers had to give the inhabitants a way to remember.

Above is a diagram of the memory architecture of every generative agent.

Observation: The agent perceives new information. The new information is saved into the memory stream, a database that serves as the record of everything an agent has experienced.
Retrieval: When the agent is looking for information, a retrieval function takes that information from the memory stream and passes it onto the agent.
Action: The agent, armed with retrieved information, can decide to act.
Reflection: Patterns are synthesized from the observations and written back to the memory stream.

To determine which memories to retrieve at any given point in time, the researchers scored memories according to three weighted factors:

Recency: when a memory was observed
Importance: how significant the memory was
Relevance: how closely related a memory is to the input memory (the agent’s current context)

The top-ranking memories that fit within the agent’s context window were selected to be retrieved and used with the prompt.

The researchers also introduced a second type of memory: reflection. Reflections were generated when a threshold of important scores was reached per day, which typically happened 2-3 times a day.

Crucially, reflection is a compression step — the agent doesn’t just accumulate observations, it periodically steps back and asks “what do these observations actually mean?” That reflection pass is where low-signal memories get subordinated to higher-order patterns. That’s the first instance of intelligent forgetting.

The Smallville paper was published in 2023. Three years later, the same insight shipped in a lobster.

Dreaming of electric lobsters

Every system that has solved the memory problem has solved it the same way: not by storing more, but by knowing what to throw away.

Your brain does it while you sleep by consolidating important experiences into long-term memory, letting the rest dissolve. Some programming languages do it with garbage collection, periodically scanning for unused objects and reclaiming that space. Databases do it with compaction: Loki, Cassandra, and other LSM-tree-inspired systems periodically merge and discard obsolete data so reads stay fast.

In all cases, the architecture of good memory is really the architecture of intelligent forgetting. OpenClaw’s dreaming feature is the same insight applied to agents.

Diagram of OpenClaw’s internal memory architecture. The default components (“Awake”) are in black, and the new components are in blue (“Dreaming”).

Note that without dreaming, there is no automatic promotion from the active context window to daily notes and then to long-term memory. When a new bit of information is mentioned, the agent has to decide whether it should be added to the daily note and/or the long-term memory file manually. Dreaming, by contrast, is an automated process.

It’s called “Dreaming” because the feature works similarly to the biological process, as if in three sleep phases:

Light phase: Creates a session corpus from session files and then succeeding sweeps tally how many times a chunk has been recalled in a session (a counter called lightHit).
REM phase: Spots patterns and themes from chunks and adds a reinforcement signal (remHits).
Deep phase: Ranks the candidates by several factors and promotes those that exceed a threshold score to long-term memory in MEMORY.md

Just like in the Smallville simulation, OpenClaw ranks information by several categories. Here’s the table of signals and weights from the OpenClaw docs at the time of writing:

Signal	Weight	Description
Frequency	0.24	How many short-term signals the entry accumulated
Relevance	0.30	Average retrieval quality for the entry
Query diversity	0.15	Distinct query/day contexts that surfaced it
Recency	0.15	Time-decayed freshness score
Consolidation	0.10	Multi-day recurrence strength
Conceptual richness	0.06	Concept-tag density from snippet/path

Only candidates that have high scores according to these signals graduate to be part of MEMORY.md.

How dreaming helps with forgetting

The new dreaming architecture doesn’t ever delete things, technically. The forgetting is more passive. However, it’s still an improvement in a few ways:

Low-signal chunks don’t get promoted: Anything that doesn’t cross the deep phase thresholds simply never reaches MEMORY.md. It exists, but it’s functionally invisible to future sessions.
Score competition: The staging index has finite attention, so chunks compete against each other for recall signal. A high-frequency chunk that keeps getting surfaced crowds out low-frequency ones. Relative obscurity is functional forgetting.
Recency decay on scoring: The deep phase ranking includes a recency signal. So something that was frequently recalled two weeks ago but hasn’t come up since will score lower than something with fewer total hits but recent activity. Time degrades relevance passively.

What’s notable: all three are selection mechanisms, not deletion mechanisms. The corpus files still exist. The staging index entries still exist. Nothing is removed… it’s just progressively less likely to be chosen. This is forgetting as diminishing probability of retrieval, not erasure.

That’s the garbage collection analogy again: the objects are still in memory, they’re just unreachable.

Transparency

Dreaming also adds a layer of transparency. Aside from the session corpus and staging files, it also produces some output about its decisions. All three dreaming stages create files per day that live in .openclaw/workspace/memory/dreaming/{phase}/YYYY-MM-DD.md.

Here’s an example of an entry in the light directory:

# Light Sleep
- Candidate: "Don't Call It Eval" blog post (Context Horizon): `Dont Call It Eval - Blog Post Outline.md` — full outline with 7 open questions; `Dont Call It Eval - Draft.md` — 2,550-word draft, 6 sections; `Dont Call It Eval - Opening Research.md` — Tier 1/2/3 shortlist of popular AI eval po
- confidence: 0.62
- evidence: memory/2026-04-19.md:13-15
- recalls: 0
- status: staged
- Candidate: "Don't Call It Eval" blog post (Context Horizon): **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since."
- confidence: 0.62
- evidence: memory/2026-04-19.md:17-17
- recalls: 0
- status: staged

This entry identified a blog draft I was working on.

REM sleep goes through these candidates and highlights when it spots themes:

# REM Sleep
### Reflections
- Theme: `assistant` kept surfacing across 532 memories.
- confidence: 1.00
- evidence: memory/2026-04-12.md:27-29, memory/.dreams/session-corpus/2026-04-17.txt:26-26, memory/.dreams/session-corpus/2026-04-17.txt:28-28
- note: reflection
### Possible Lasting Truths
- "Don't Call It Eval" blog post (Context Horizon): **Datadog Bits AI eval platform** (Apr 7, 2026) — "Bring the noise" = noise injection = production-like envs from 90s QA. SECTION HEADERS are gold ("When one improvement caused subtle regressions", "Why tool-level testing and live [confidence=0.58 evidence=memory/2026-04-19.md:30-33]
- "Don't Call It Eval" blog post (Context Horizon): **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since." [confidence=0.58 evidence=memory/2026-04-19.md:17-17]
- "Don't Call It Eval" blog post (Context Horizon): **Title:** "Don't Call It Eval" (Nicole's original; I picked it over "Congratulations, you've invented testing") [confidence=0.58 evidence=memory/2026-04-19.md:27-27]

In this case, REM sleep correctly identified that I’m currently working on the Assistant feature at work (which had already been promoted to long-term memory) and the blog post that the light sleep phase had identified.

Here’s the same day’s deep sleep file:

# Deep Sleep
- Ranked 10 candidate(s) for durable promotion.
- Promoted 3 candidate(s) into MEMORY.md.

And here’s the addition to MEMORY.md for the same day:

## Promoted From Short-Term Memory (2026-04-26)
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:17:17 -->
- **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since." [score=0.926 recalls=0 avg=0.620 source=memory/2026-04-19.md:17-17]
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:27:27 -->
- **Title:** "Don't Call It Eval" (Nicole's original; I picked it over "Congratulations, you've invented testing") [score=0.926 recalls=0 avg=0.620 source=memory/2026-04-19.md:27-27]
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:12:12 -->
- Three files in openclaw-output: [score=0.894 recalls=0 avg=0.620 source=memory/2026-04-19.md:12-12]

There’s also an entry in DREAMS.md for the entire sequence, which is more dreamlike and human-readable. The per-day Deep Sleep file is the structured internal audit trail; DREAMS.md is the rolling, more narrative log for humans.

This level of transparency and reflection is a welcome change when you compare it to what I had before, which was nothing. Iris had to manually decide on the spot what had to be written where, and those decisions were not logged anywhere.

The tonsil dilemma

So dreaming could have prevented the tonsil hallucination, right? Well… no.

In this case, the idea of my supposed tonsil pain had come from an external source: my meeting recordings. Iris had faithfully manually added that to the daily note for the day, then decided to call on it a few days later, unprompted, in conversation with me.

With dreaming, Iris still could have made those decisions. But in fact, even if she hadn’t, dreaming could have made it worse:

Light phase sweeps it and scores it well since it’s recent and distinctive.
Tonsil pain could be included in queries about health, such as the daily health tracker that is part of Iris’s tasks. REM phase sees a pattern and increments remHits.
Deep phase ranks it highly and it crosses the promotion threshold.
It gets written to MEMORY.md and gets loaded every session. And long-term memory is sticky. Only manual intervention (by me or by Iris) removes things from that file.

Even dreaming doesn’t solve everything. Dreaming adds a mechanism for forgetting. It doesn’t prevent OpenClaw from remembering the wrong things in the first place.

The residual layer

Every memory architecture we’ve looked at — biological sleep consolidation, garbage collection, LSM compaction, Smallville’s reflection pass, OpenClaw’s dreaming — has the same shape: automated compression plus a residual human layer. Your brain has you. GC has the programmer. Iris’s dreaming has me. The human in the loop isn’t a bug in the design; it’s load-bearing.

What’s worth keeping

Dreaming is a genuine step forward for OpenClaw’s memory architecture. The mechanism of passive forgetting isn’t perfect, but the automated promotion systems do add a necessary layer of periodic reflection.

But there are limitations. Dreaming can’t yet distinguish between a memory worth keeping and a hallucinated tonsil. It can only measure how often something gets recalled. The next frontier is active forgetting: automated demotion from MEMORY.md, weighted by signals like source provenance and trust. The provenance side of that work landed in v2026.4.29-beta.2 — memory now exposes evidence-kind drilldown and privacy/provenance reports, so you can at least see where a memory came from. But the demotion half — automatically down-ranking memories based on a low-trust source like an audio transcript — still hasn’t been built.

What remains, for now, is the human layer. The transparency of the dreaming process becomes a checkpoint for the human in the loop by surfacing promotion decisions. I still read the summaries. I still catch the errors. I still decide what goes into long-term memory and what gets quietly corrected. That’s not a gap in the system — it’s probably the right design. The most important memories should require a human to notice them twice.

Good memory systems are defined by what they don’t keep as much as by what they do keep.

High INT, Low WIS: What LLM Benchmarks Miss

Thu, 23 Apr 2026 09:00:00 +0100

Originally posted here on Context Horizon.

In the tabletop roleplaying game Dungeons & Dragons (D&D), the very first thing you do when you create a character is to choose ability scores. Ability scores are numbers that determine how good you are at the game’s main characteristics: Strength (STR), Dexterity (DEX), Constitution (CON), Intelligence (INT), Wisdom (WIS), and Charisma (CHA). 20 is the default maximum that a character can have in any ability, though some in-game events/items can increase that.

Here’s the thing, though: every D&D player knows that a 20 Intelligence doesn’t mean your wizard aces every check. Whether you have proficiency in the specific skill still matters. That 20 INT wizard, for all their raw cognitive horsepower, can absolutely still fail to identify a magical rune. The ability score is a starting point, not a verdict.

This is exactly what’s happening to LLM benchmarks. All the general LLM/AI model benchmarks are ability scores. They tell you something real about a model’s general capabilities, the same way a 20 INT tells you something real about a wizard. But as frontier models crowd toward the top of every leaderboard, we’re running into a problem D&D solved decades ago: the stat block isn’t the character. And it was never supposed to be.

A screenshot of the stats of my Level 10 bard. Low INT, but look at that CHA!

The stat block: A tour of the ability scores everyone’s maxing out

There are so many new models and model versions out there that it’s hard to keep up with what we’re supposed to be using for which tasks. LLM benchmarks have come to be relied on to give a somewhat objective assessment of each model’s capability. Here are a few big ones that exist.

General knowledge/reasoning

MMLU (Measuring Massive Multitask Language Understanding, 2020)

MMLU is the classic “general knowledge” INT check. It was released in September 2020 by Hendrycks et al. and contains 15,908 multiple-choice questions across 57 subjects (humanities, social science, STEM, other). However, there were a few issues with it:

a study in 2024 showed that 6.49% of those questions contained errors
The New York Times reported (2024) that data contamination was an issue — models could be trained on the MMLU questions (and answers) to inflate their MMLU scores
many frontier LLMs soon consistently achieved high scores

In response, a new benchmark called MMLU-Pro was created with 12,000 questions that each have 10 options. The increase in options theoretically required more reasoning capability than the original benchmark, but frontier models are already knocking at 90% on the MMLU-Pro.

Humanity’s Last Exam (HLE, 2025)

HLE was developed specifically to address the fact that LLMs “now achieve over 90% accuracy on popular benchmarks like MMLU” (from the HLE site). The paper by Phan et al explains the lengths they took to make it hard enough to be meaningful:

they solicited high-quality questions from nearly 1000 subject matter experts, even going as far as paying them for successful submissions (with a total prize pool of US $500,000)
they included graduate-level classics (one of the questions includes translating Palmyrene script)
they reviewed potential submissions using current LLMs and only proceeded with submissions if the LLMs are unable to answer correctly
experts then reviewed the submissions further

As a result, HLE scores are low — intentionally so:

While current LLMs achieve very low accuracy on HLE, recent history shows benchmarks are quickly saturated – with models dramatically progressing from near-zero to near-perfect performance in a short timeframe [13, 45]. Given the rapid pace of AI development, it is plausible that models could exceed 50% accuracy on HLE by the end of 2025. (Phan et al., 2025)

Software engineering

HumanEval (OpenAI, 2021)

In 2021, OpenAI released a paper and an evaluation harness for a benchmark consisting of 164 Python programming problems based on publicly available code on GitHub. It was developed to test coding in particular.

Eventually, though, experts found that the problem sets in HumanEval were insufficient and ambiguous (Liu et al., 2023). HumanEval+ was released to include more tests, but the next benchmark’s authors suggested that models might be overfitting to HumanEval+ as well, memorizing the dataset and answers rather than genuinely demonstrating reasoning ability.

LiveCodeBench (2024)

Jain et al. tried a different approach to HumanEval to avoid data contamination: it was released with a continuously-updated coding benchmark. Problems have release dates and models can only be evaluated using problems released after their training data cutoff dates.

LiveCodeBench represents a shift in benchmarks towards a more continuous, dynamic measure. However, the last submission to the benchmark was 9 months ago at the time of this writing, and the leaderboard has not been updated since then either, suggesting the benchmark is no longer being actively maintained.

SWE-bench (2024)

This benchmark was created by Jimenez et al. to include 2,294 actual GitHub issues from 12 Python projects. When it launched, it was a much harder benchmark than predecessors, with models scoring 2-4%. Today, models are scoring around 70-80% on the original benchmark. There were enough issues with the original’s dataset quality that variants were introduced, including OpenAI’s SWE-bench Verified. Today, the SWE-bench website shows these variants:

SWE-bench
SWE-bench Verified
SWE-bench Multilingual
SWE-bench Multimodal
SWE-bench Lite

Agents and tool use

This last category is a preview of where I think the field should go: benchmarks that measure what a model does rather than just what a model knows. Here are some of them:

GAIA (Mialon et al., 2023) — 466 real-world assistant tasks that combine web browsing, file reading, and multi-step reasoning. Easy for humans (92% success), hard for models (15% success for GPT-4), at least at launch.
Terminal-Bench — agents working in actual Linux terminals. Tasks include “build the Linux kernel from source” and “configure a git server plus a webserver on port 8080.” No mocks; if it doesn’t actually build, it fails.
WebArena (Zhou et al., 2023) — realistic simulated websites (Reddit, GitLab, e-commerce) where agents complete actual user goals like “filter the last six months of orders by refund status.”

These benchmarks are harder to saturate than general-knowledge ones, because passing means the thing actually has to work in the real world. A model can memorise a question; it can’t memorise a Linux kernel that isn’t there until it builds it.

The wrong answer: Build a harder general benchmark

By now, you probably see that benchmarks tend to go the same way: A benchmark is created, models get too good at it and perform too well to be useful, and then a variant benchmark is created to address the problem.

This is the saturation problem in a nutshell: a benchmark that once provided meaningful information to help distinguish models eventually fails to do so because all the models’ scores cluster at the top.

Saturation occurs for two reasons:

Contamination: Many benchmarks publish their problem sets online, and models train on data scraped from the web. The benchmark now measures memorization, not capability.
Overfitting: Model providers are incentivized to optimize specifically for gaming performance on benchmarks. Scores go up without corresponding capability improvements.

When you have a party of wizards with 20 INT, “What’s your INT score?” stops being an interesting question because any wizard’s Fireball will hit the same. Building harder benchmarks is like an arms race for ability scores, when what you actually want to know is whether your wizard can kill a dragon.

Harder benchmarks don’t tell us what we want to know: can this model do my specific job?

The right answer: skill checks with real DCs

The fix isn’t a harder ability check. It’s a different kind of check altogether. In D&D, straight ability checks are rare compared to skill checks. There are 18 official skills that break down the six ability scores into more granular activities. For example, your wizard is much more likely to roll one of these instead of a straight INT check:

Arcana
History
Investigation
Nature
Religion

All of these are still based on INT, but test narrow types of intelligence rather than general intelligence.

Specific, domain-grounded benchmarks are what the field actually needs. The evaluations that actually help practitioners share a structure: they test a specific skill in a specific context, and they’re judged by the people who have to live with the results.

MedQA tests whether a model can answer the kind of clinical question a physician might face. LegalBench tests whether a model can correctly identify enforceability clauses, classify contract language, and apply legal reasoning. FinanceBench tests whether a model can answer the kind of question a financial analyst would actually ask about a real 10-K filing. None of these are trying to rank models on general intelligence. They’re asking the only question that matters downstream: can this model do this specific job well enough?

This week, Grafana released o11y-bench, an open benchmark for observability-specific agent tasks. I’m biased — I work there — but it’s a useful illustration of the shape I’m describing. o11y-bench doesn’t ask whether a model is smart in general. It asks whether an agent can do the things observability engineers actually do: read production traces, correlate logs with metrics, figure out why the thing that’s supposed to be working isn’t. Its leaderboard shows models’ performance for each activity.

Does that mean that more specific benchmarks won’t get saturated? No. We’ve already seen this play out with SWE-bench. But they’ll get saturated more slowly and remain more meaningful measures along the way.

These are skill checks with DCs that matter to someone. They’re not trying to produce a single scalar ranking of “which model is smartest.” They’re asking whether the wizard recognizes the family crest belonging to a political faction or knows this specific cult’s practices, not whether the wizard is good at “wizarding” in the abstract.

Roll for initiative: What this means in practice

If you’re an engineer picking a model for production: stop picking by leaderboard. Find or build an eval that reflects your actual use case. A handful of representative examples, graded the way you’d grade them if you were doing the work yourself, will tell you more than a dozen public benchmarks.

If you’re a researcher thinking about what to build next: the interesting work is at the specific-and-grounded end of the spectrum, not the general-and-bigger end. Another harder general-knowledge benchmark will saturate in eighteen months. A well-designed domain benchmark, co-built with the people who actually do the work, can stay useful for years.

And if you’re anyone else watching this field from the outside: be skeptical of the headlines. When a new model scores 94% on a benchmark you’ve never heard of, ask what it measures and whether the thing it measures is the thing you actually care about. Usually the answer is “not really,” and that’s the most important thing a benchmark score won’t tell you.

The goal of a character sheet was never to produce one number that ranks the character. It was to give a DM enough information to run a meaningful encounter — and to give the player enough to know when to try, when to pass, and when to roll with advantage. LLM evaluation should work the same way. Not a ranking; a readiness check. Can this specific model clear this specific challenge? Roll for initiative.

Unlearnings from building Grafana Assistant

Thu, 09 Apr 2026 09:00:00 +0100

Originally posted here

Everyone’s talking about what you need to learn to build AI agents: Prompt engineering. Tool calling. Retrieval strategies. The content ecosystem is full of it. But while I was watching Context Window Episode 1, it struck me that Mat Ryer, Cyril Tovena, and Sven Großmann– some of the team behind Grafana Assistant– spent most of their time on something different: what they had to unlearn.

Grafana Assistant started as a weekend hackathon project. By the time it shipped, the team had thrown out a popular framework, flipped their assumptions about tool design, and discovered that some of the instincts you bring from conventional software development actively work against you when you’re building agents.

I pulled out the ten moments where received wisdom turned out to be wrong. If you’re building anything agentic right now, these are worth sitting with.

❌ LangChain will help us build faster

LangChain is an opensource framework for building applications that use LLMs. It provides structured components that can be used for rapid prototyping.

In the video, Tovena talks about how they started with LangChain during the initial hackathon because they thought it would help them build more quickly, but instead it became obvious fairly early on that the opposite was true. They didn’t end up using a lot of the features that LangChain provided and felt they needed more flexibility and speed.

Abstraction overhead

One possible reason for their troubles with LangChain is that LangChain has standard abstractions for prompts, tool use, and orchestration– things that sound good in theory. But sometimes abstractions lend themselves to creating a sort of black box in your stack, where you don’t know what’s happening inside it. It sounds like the Assistant team missed the oversight and control of what was happening.

❌ Use the same prompt for every model

They also talked about how for the sake of simplicity and consistency, they initially used the same prompt for every model. However, they found out that this actually led to worse results. They found more success in creating slightly different prompts for different models, playing to their strengths and weaknesses a bit more.

There is even an argument (Mizrahi et al.’s “State of What Art? A Call for Multi-Prompt LLM Evaluation) that AI benchmarks that involve sending multiple LLMs the same prompt is a brittle and unreliable way to assess LLM capabilities. Different models respond differently to prompt variations.

In practice, what this means is that switching models isn’t a drop-in replacement. Even if a new model scores better on benchmarks, your prompts were written with a specific model’s tendencies in mind. The unlearning here is letting go of the idea that a prompt is model-agnostic — it’s more like a conversation style, and different models have different conversational defaults.

❌ Treat errors as failures

Traditionally in software, you catch errors and hide them from the user: you log them, handle them, but treat that transaction as a failed one.

The Assistant team had to unlearn this habit when dealing with an LLM, because with agents, error messages are useful data. Feeding it back to the LLM often results in it learning from its own mistake and trying again. It’s basically letting the agent debug itself, in a manner that Ryer noted was remarkably similar to how we humans debug errors too.

❌ Return errors in structured JSON

On the subject of error messages, Großmann mentioned that at one point they were trying to format the errors in JSON or stringified JSON. This practice shows solid engineer intuition because usually, structuring information helps machines process it faster and more accurately. What they found here was the reverse: it was easier and yielded better results when the LLM was just passed the entire error in natural language.

❌ Build as many tools as possible

The team mentioned that at the start, they were inclined to create as many tools/skills for Assistant: one for everything that could be done in Grafana. This made sense until they saw LLMs struggle to determine which tool to use in which context.

To help LLMs identify the correct tool, they added tool descriptions and also experimented with combining multiple tools into fewer, parameterized, tools. These helped, although the team still observed that allowing users to create their own tools (with potentially very general tool descriptions) could potentially confuse LLMs.

❌ Bigger context window = better

Gemini 1.5 Pro announced it had a 2M context window back in 2024, and it might seem that other providers had lagged behind: OpenAI only announced support for 1M tokens in 2025 and Anthropic followed in 2026. However, the Assistant team pointed out that context wasn’t everything. They said that they noticed that bigger wasn’t necessarily better, also pointing out that the chat summarization feature on IDEs like Cursor could still sometimes lose important details when summarizing.

This finding seems to coincide with a 2023 Stanford study (Lost in the Middle: How Language Models Use Long Contexts) that found there was a performance degradation when models access information within a very large context. In the video, they talked about how their system prompts were often 40k tokens long (half for the prompt itself and then the other half for tools).

❌ Start with a multi-agent architecture

Tovena mentioned that creating a multi-agent architecture for Assistant yielded more mixed results than he expected. He said that passing context between agents is difficult, outputs were inconsistent, and there was a bit of a “telephone game” problem where agents didn’t always perfectly understand the intent or output of other agents.

In early 2026, Khatua et al. released a study called “CooperBench: Why Coding Agents Cannot Be Your Teammates Yet” that showed that AI agents aren’t great at collaboration. The study suggests, surprisingly, that this might be because AI agents lack human social habits that help us collaborate better, such as feeling obliged to do what we say and having a good understanding of what others’ plans are.

The Assistant team’s experience is also a perfect example of Gall’s Law, which states that it’s easier to start with simplicity and build complexity than the other way around.

❌ Plan around models as they are now

Software has always moved quickly, but it feels like AI models are moving faster. Stanford’s AI Index Report 2025 reported that models went from having a 50% chance to near-human performance in roughly 2 years (2021-2023). In fact, benchmarks keep getting retired as AI saturates them– meaning that benchmarks can’t reliably be used to distinguish between good and bad models because all the models are testing near the ceiling.

The rapid pace of development around AI requires us to not be so attached to our work: Großmann recalled an incident where someone wrote a joke prompt that spoke like a dog that shockingly did better than their other system prompts. Ryer also mentioned that at one point someone deleted all the system prompts and Assistant weirdly became more accurate.

The lesson is clear: plan for models to improve, and plan to redo a lot of your work when they do.

❌ We don’t need evals

The Assistant team said that they went from having no evals to employing several types of evals:

integration-style evals that exercise whole workflows
unit-style evals that feed inputs plus a system prompt and check the output
LLM-as-judge, where they pass conversations to an LLM to ask it to judge whether it answered the user’s questions.
user feedback in the form of thumbs during conversations with Assistant

Beyond evaluation, they talked about how important it was to be able to observe AI. At the very minimum, showing what Assistant is doing at all times (thinking out loud) is very important to them because it builds users’ trust in the thinking process and, eventually, in the result.

Evals aren’t a sign that you don’t trust your product. They’re how you build the confidence to keep shipping it.

❌ Assume people already know how to work with AI

When you ship an AI feature, it’s tempting to assume the hard work is done. You’ve built the thing. Users will figure it out.

What the Grafana Assistant team found was that this assumption doesn’t hold. Users came in with wildly different mental models of what an AI assistant could or should do — some treating it like a search box, others expecting it to read their minds, most not yet sure how to prompt it effectively.

Their insight was that this isn’t just a UX problem to solve on your end. It’s a shared responsibility: AI should try to help the user, but the user also needs to learn how to work with AI. And building an assistant that teaches people how to use it well (through good responses, gentle corrections, showing its work) is part of building the product.

There’s also a deeper point here specific to observability. Grafana Assistant isn’t just an AI feature; it’s a way of interacting with your own infrastructure. Every good interaction teaches you something about what questions to ask, what signals to look for, what it means to understand a system. In other words: learning to use AI well makes you a better engineer. The tool and the skill co-evolve.

This is the unlearning that sits furthest from the code. It’s not about prompts or architectures or frameworks. It’s about recognising that you’re not just shipping a feature– you’re introducing a new way of working. That takes time. And it’s worth designing for explicitly.

Summary

What strikes me most about this list isn’t any single item — it’s the pattern. Most of these unlearnings point in the same direction: less is more, and messiness is the point.

Less abstraction. Fewer tools. Smaller context. Simpler architecture. And rather than hiding the mess — the errors, the reasoning, the uncertainty — let it show. That’s where the trust gets built.

We’re all still early with this. The team that built one of the most thoughtful AI products I’ve seen up close still didn’t have evals when they launched. That’s not a criticism — it’s a reminder. You don’t need to have it all figured out before you ship. You just need to commit to keep unlearning.

Observability for GenAI Applications (Grafana ❤️🔥 OpenTelemetry Community Call #4)

Wed, 21 Jan 2026 16:00:00 +0100

In this community call with my coworker Liudmila Molkova, we dove into the emerging field of observability for Generative AI applications. We explored how to instrument LLM-powered applications using OpenTelemetry, what metrics and traces matter most when monitoring AI workloads, and how to debug issues in complex AI pipelines. The discussion covered prompt tracing, token usage monitoring, latency considerations, and the unique challenges of observing non-deterministic AI systems.

AI Observability with Ishan Jain (Grafana Office Hours #29)

Fri, 08 Nov 2024 18:24:29 +0100

In this Grafana Office Hours, Ishan Jain talks about AI Observability with Grafana: what it entails, factors to consider when monitoring and observing LLMs, and how to do it all with Grafana. He is joined by Senior Developer Advocate Nicole van der Hoeven.