Context Horizon on Nicole van der Hoeven

The Architecture of Forgetting

Thu, 07 May 2026 16:00:00 +0100

Originally posted here on Context Horizon.

In early April, a new version of everyone’s favourite lobster, OpenClaw, was shipped with an interesting new feature: dreaming. It’s a memory consolidation feature. But to understand why it matters, we need to back up.

Early AI chatbots, like the initial GPT wrappers, had zero memory. Every conversation started fresh. You asked a question, you got an answer, and you moved on: a glorified search engine.

Somewhere along the way we began to expect more and more of our chatbots. Models acquired context windows that became their de facto short-term memory. As the context window filled up, they remembered what had been said earlier in the conversation, making them better conversational partners. Still, it was ephemeral: close the tab, start fresh. And it had a hard ceiling: hit the limit, and the oldest things fall off. This felt like forgetting because it was. But it was also, accidentally, a form of curation. Finite space forced priority.

Then came the push for real persistence. RAG pipelines, vector databases, memory files, agent frameworks that write notes to themselves. Now our agents can remember that we prefer the Oxford comma, that we just started a new project at work, that our D&D character is named Zuhri. This felt like the solution. More memory = better agent.

That mostly works— until it doesn’t.

As it turns out, the problem with AI memory isn’t that agents forget too much. It’s that they forget too little.

Why remembering is hard

It turns out that having a persistent memory, even for an agent like OpenClaw that has that as one of its defining features, doesn’t solve all the memory problems.

I happen to have an OpenClaw agent that named itself Iris.

Iris does a lot of things for me, including parsing my meeting notes to identify next actions, curating my tasks and projects, and generating a daily briefing. She also creates a summary of the day’s work based on my meetings, conversations, and notes. One day she was giving me the daily summary, and here’s that conversation:

Iris: …Tonsil is finally healed after weeks of pain… Me: Wait what? Where did you get the tonsil stuff? Iris: From your memory file — it’s in the Apr 10 daily note context: “Tonsil finally healed — Nicole no longer in pain 🎉”. I’ve had that sitting in memory since the Apr 10 summary. Is that wrong? Was it something else that healed, or is the tonsil detail off? Me: I have no idea where you got that. I have never had issues with my tonsil.

Where did that come from? I’m still not sure, but I occasionally have meetings in cafés, and it’s possible that a meeting recording overheard the conversation of someone else in the café. That got faithfully transcribed, copied over into my notes, and written into Iris’s memory. Iris treated it as fact for days.

The problem isn’t that Iris forgot. It’s that she remembered the wrong thing… and didn’t know how to forget it.

Imagine this at scale: not one wrong fact, but dozens, quietly shaping every summary, every recommendation, every task Iris surfaces for me. The signal-to-noise ratio degrades until it doesn’t matter if Iris remembers mostly accurate memories— I won’t trust it.

So how do you build a memory system that knows what to keep? Researchers have been working on this longer than you might think.

Meet the residents of Smallville

Stanford researchers Park et al. released a charming study in 2023 called Generative Agents: Interactive Simulacra of Human Behavior in which they talk about how they created a simulated environment called Smallville, populated it with 25 generative agents, and then watched.

Source: Park et al. (2023)

The study was an attempt to simulate emergent social behavior, but the key insight I took from it isn’t the sim itself— it’s what they had to do to the inhabitants of Smallville to enable the rich social interactions they wanted to observe.

The researchers had to give the inhabitants a way to remember.

Above is a diagram of the memory architecture of every generative agent.

Observation: The agent perceives new information. The new information is saved into the memory stream, a database that serves as the record of everything an agent has experienced.
Retrieval: When the agent is looking for information, a retrieval function takes that information from the memory stream and passes it onto the agent.
Action: The agent, armed with retrieved information, can decide to act.
Reflection: Patterns are synthesized from the observations and written back to the memory stream.

To determine which memories to retrieve at any given point in time, the researchers scored memories according to three weighted factors:

Recency: when a memory was observed
Importance: how significant the memory was
Relevance: how closely related a memory is to the input memory (the agent’s current context)

The top-ranking memories that fit within the agent’s context window were selected to be retrieved and used with the prompt.

The researchers also introduced a second type of memory: reflection. Reflections were generated when a threshold of important scores was reached per day, which typically happened 2-3 times a day.

Crucially, reflection is a compression step — the agent doesn’t just accumulate observations, it periodically steps back and asks “what do these observations actually mean?” That reflection pass is where low-signal memories get subordinated to higher-order patterns. That’s the first instance of intelligent forgetting.

The Smallville paper was published in 2023. Three years later, the same insight shipped in a lobster.

Dreaming of electric lobsters

Every system that has solved the memory problem has solved it the same way: not by storing more, but by knowing what to throw away.

Your brain does it while you sleep by consolidating important experiences into long-term memory, letting the rest dissolve. Some programming languages do it with garbage collection, periodically scanning for unused objects and reclaiming that space. Databases do it with compaction: Loki, Cassandra, and other LSM-tree-inspired systems periodically merge and discard obsolete data so reads stay fast.

In all cases, the architecture of good memory is really the architecture of intelligent forgetting. OpenClaw’s dreaming feature is the same insight applied to agents.

Diagram of OpenClaw’s internal memory architecture. The default components (“Awake”) are in black, and the new components are in blue (“Dreaming”).

Note that without dreaming, there is no automatic promotion from the active context window to daily notes and then to long-term memory. When a new bit of information is mentioned, the agent has to decide whether it should be added to the daily note and/or the long-term memory file manually. Dreaming, by contrast, is an automated process.

It’s called “Dreaming” because the feature works similarly to the biological process, as if in three sleep phases:

Light phase: Creates a session corpus from session files and then succeeding sweeps tally how many times a chunk has been recalled in a session (a counter called lightHit).
REM phase: Spots patterns and themes from chunks and adds a reinforcement signal (remHits).
Deep phase: Ranks the candidates by several factors and promotes those that exceed a threshold score to long-term memory in MEMORY.md

Just like in the Smallville simulation, OpenClaw ranks information by several categories. Here’s the table of signals and weights from the OpenClaw docs at the time of writing:

Signal	Weight	Description
Frequency	0.24	How many short-term signals the entry accumulated
Relevance	0.30	Average retrieval quality for the entry
Query diversity	0.15	Distinct query/day contexts that surfaced it
Recency	0.15	Time-decayed freshness score
Consolidation	0.10	Multi-day recurrence strength
Conceptual richness	0.06	Concept-tag density from snippet/path

Only candidates that have high scores according to these signals graduate to be part of MEMORY.md.

How dreaming helps with forgetting

The new dreaming architecture doesn’t ever delete things, technically. The forgetting is more passive. However, it’s still an improvement in a few ways:

Low-signal chunks don’t get promoted: Anything that doesn’t cross the deep phase thresholds simply never reaches MEMORY.md. It exists, but it’s functionally invisible to future sessions.
Score competition: The staging index has finite attention, so chunks compete against each other for recall signal. A high-frequency chunk that keeps getting surfaced crowds out low-frequency ones. Relative obscurity is functional forgetting.
Recency decay on scoring: The deep phase ranking includes a recency signal. So something that was frequently recalled two weeks ago but hasn’t come up since will score lower than something with fewer total hits but recent activity. Time degrades relevance passively.

What’s notable: all three are selection mechanisms, not deletion mechanisms. The corpus files still exist. The staging index entries still exist. Nothing is removed… it’s just progressively less likely to be chosen. This is forgetting as diminishing probability of retrieval, not erasure.

That’s the garbage collection analogy again: the objects are still in memory, they’re just unreachable.

Transparency

Dreaming also adds a layer of transparency. Aside from the session corpus and staging files, it also produces some output about its decisions. All three dreaming stages create files per day that live in .openclaw/workspace/memory/dreaming/{phase}/YYYY-MM-DD.md.

Here’s an example of an entry in the light directory:

# Light Sleep
- Candidate: "Don't Call It Eval" blog post (Context Horizon): `Dont Call It Eval - Blog Post Outline.md` — full outline with 7 open questions; `Dont Call It Eval - Draft.md` — 2,550-word draft, 6 sections; `Dont Call It Eval - Opening Research.md` — Tier 1/2/3 shortlist of popular AI eval po
- confidence: 0.62
- evidence: memory/2026-04-19.md:13-15
- recalls: 0
- status: staged
- Candidate: "Don't Call It Eval" blog post (Context Horizon): **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since."
- confidence: 0.62
- evidence: memory/2026-04-19.md:17-17
- recalls: 0
- status: staged

This entry identified a blog draft I was working on.

REM sleep goes through these candidates and highlights when it spots themes:

# REM Sleep
### Reflections
- Theme: `assistant` kept surfacing across 532 memories.
- confidence: 1.00
- evidence: memory/2026-04-12.md:27-29, memory/.dreams/session-corpus/2026-04-17.txt:26-26, memory/.dreams/session-corpus/2026-04-17.txt:28-28
- note: reflection
### Possible Lasting Truths
- "Don't Call It Eval" blog post (Context Horizon): **Datadog Bits AI eval platform** (Apr 7, 2026) — "Bring the noise" = noise injection = production-like envs from 90s QA. SECTION HEADERS are gold ("When one improvement caused subtle regressions", "Why tool-level testing and live [confidence=0.58 evidence=memory/2026-04-19.md:30-33]
- "Don't Call It Eval" blog post (Context Horizon): **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since." [confidence=0.58 evidence=memory/2026-04-19.md:17-17]
- "Don't Call It Eval" blog post (Context Horizon): **Title:** "Don't Call It Eval" (Nicole's original; I picked it over "Congratulations, you've invented testing") [confidence=0.58 evidence=memory/2026-04-19.md:27-27]

In this case, REM sleep correctly identified that I’m currently working on the Assistant feature at work (which had already been promoted to long-term memory) and the blog post that the light sleep phase had identified.

Here’s the same day’s deep sleep file:

# Deep Sleep
- Ranked 10 candidate(s) for durable promotion.
- Promoted 3 candidate(s) into MEMORY.md.

And here’s the addition to MEMORY.md for the same day:

## Promoted From Short-Term Memory (2026-04-26)
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:17:17 -->
- **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since." [score=0.926 recalls=0 avg=0.620 source=memory/2026-04-19.md:17-17]
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:27:27 -->
- **Title:** "Don't Call It Eval" (Nicole's original; I picked it over "Congratulations, you've invented testing") [score=0.926 recalls=0 avg=0.620 source=memory/2026-04-19.md:27-27]
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:12:12 -->
- Three files in openclaw-output: [score=0.894 recalls=0 avg=0.620 source=memory/2026-04-19.md:12-12]

There’s also an entry in DREAMS.md for the entire sequence, which is more dreamlike and human-readable. The per-day Deep Sleep file is the structured internal audit trail; DREAMS.md is the rolling, more narrative log for humans.

This level of transparency and reflection is a welcome change when you compare it to what I had before, which was nothing. Iris had to manually decide on the spot what had to be written where, and those decisions were not logged anywhere.

The tonsil dilemma

So dreaming could have prevented the tonsil hallucination, right? Well… no.

In this case, the idea of my supposed tonsil pain had come from an external source: my meeting recordings. Iris had faithfully manually added that to the daily note for the day, then decided to call on it a few days later, unprompted, in conversation with me.

With dreaming, Iris still could have made those decisions. But in fact, even if she hadn’t, dreaming could have made it worse:

Light phase sweeps it and scores it well since it’s recent and distinctive.
Tonsil pain could be included in queries about health, such as the daily health tracker that is part of Iris’s tasks. REM phase sees a pattern and increments remHits.
Deep phase ranks it highly and it crosses the promotion threshold.
It gets written to MEMORY.md and gets loaded every session. And long-term memory is sticky. Only manual intervention (by me or by Iris) removes things from that file.

Even dreaming doesn’t solve everything. Dreaming adds a mechanism for forgetting. It doesn’t prevent OpenClaw from remembering the wrong things in the first place.

The residual layer

Every memory architecture we’ve looked at — biological sleep consolidation, garbage collection, LSM compaction, Smallville’s reflection pass, OpenClaw’s dreaming — has the same shape: automated compression plus a residual human layer. Your brain has you. GC has the programmer. Iris’s dreaming has me. The human in the loop isn’t a bug in the design; it’s load-bearing.

What’s worth keeping

Dreaming is a genuine step forward for OpenClaw’s memory architecture. The mechanism of passive forgetting isn’t perfect, but the automated promotion systems do add a necessary layer of periodic reflection.

But there are limitations. Dreaming can’t yet distinguish between a memory worth keeping and a hallucinated tonsil. It can only measure how often something gets recalled. The next frontier is active forgetting: automated demotion from MEMORY.md, weighted by signals like source provenance and trust. The provenance side of that work landed in v2026.4.29-beta.2 — memory now exposes evidence-kind drilldown and privacy/provenance reports, so you can at least see where a memory came from. But the demotion half — automatically down-ranking memories based on a low-trust source like an audio transcript — still hasn’t been built.

What remains, for now, is the human layer. The transparency of the dreaming process becomes a checkpoint for the human in the loop by surfacing promotion decisions. I still read the summaries. I still catch the errors. I still decide what goes into long-term memory and what gets quietly corrected. That’s not a gap in the system — it’s probably the right design. The most important memories should require a human to notice them twice.

Good memory systems are defined by what they don’t keep as much as by what they do keep.

High INT, Low WIS: What LLM Benchmarks Miss

Thu, 23 Apr 2026 09:00:00 +0100

Originally posted here on Context Horizon.

In the tabletop roleplaying game Dungeons & Dragons (D&D), the very first thing you do when you create a character is to choose ability scores. Ability scores are numbers that determine how good you are at the game’s main characteristics: Strength (STR), Dexterity (DEX), Constitution (CON), Intelligence (INT), Wisdom (WIS), and Charisma (CHA). 20 is the default maximum that a character can have in any ability, though some in-game events/items can increase that.

Here’s the thing, though: every D&D player knows that a 20 Intelligence doesn’t mean your wizard aces every check. Whether you have proficiency in the specific skill still matters. That 20 INT wizard, for all their raw cognitive horsepower, can absolutely still fail to identify a magical rune. The ability score is a starting point, not a verdict.

This is exactly what’s happening to LLM benchmarks. All the general LLM/AI model benchmarks are ability scores. They tell you something real about a model’s general capabilities, the same way a 20 INT tells you something real about a wizard. But as frontier models crowd toward the top of every leaderboard, we’re running into a problem D&D solved decades ago: the stat block isn’t the character. And it was never supposed to be.

A screenshot of the stats of my Level 10 bard. Low INT, but look at that CHA!

The stat block: A tour of the ability scores everyone’s maxing out

There are so many new models and model versions out there that it’s hard to keep up with what we’re supposed to be using for which tasks. LLM benchmarks have come to be relied on to give a somewhat objective assessment of each model’s capability. Here are a few big ones that exist.

General knowledge/reasoning

MMLU (Measuring Massive Multitask Language Understanding, 2020)

MMLU is the classic “general knowledge” INT check. It was released in September 2020 by Hendrycks et al. and contains 15,908 multiple-choice questions across 57 subjects (humanities, social science, STEM, other). However, there were a few issues with it:

a study in 2024 showed that 6.49% of those questions contained errors
The New York Times reported (2024) that data contamination was an issue — models could be trained on the MMLU questions (and answers) to inflate their MMLU scores
many frontier LLMs soon consistently achieved high scores

In response, a new benchmark called MMLU-Pro was created with 12,000 questions that each have 10 options. The increase in options theoretically required more reasoning capability than the original benchmark, but frontier models are already knocking at 90% on the MMLU-Pro.

Humanity’s Last Exam (HLE, 2025)

HLE was developed specifically to address the fact that LLMs “now achieve over 90% accuracy on popular benchmarks like MMLU” (from the HLE site). The paper by Phan et al explains the lengths they took to make it hard enough to be meaningful:

they solicited high-quality questions from nearly 1000 subject matter experts, even going as far as paying them for successful submissions (with a total prize pool of US $500,000)
they included graduate-level classics (one of the questions includes translating Palmyrene script)
they reviewed potential submissions using current LLMs and only proceeded with submissions if the LLMs are unable to answer correctly
experts then reviewed the submissions further

As a result, HLE scores are low — intentionally so:

While current LLMs achieve very low accuracy on HLE, recent history shows benchmarks are quickly saturated – with models dramatically progressing from near-zero to near-perfect performance in a short timeframe [13, 45]. Given the rapid pace of AI development, it is plausible that models could exceed 50% accuracy on HLE by the end of 2025. (Phan et al., 2025)

Software engineering

HumanEval (OpenAI, 2021)

In 2021, OpenAI released a paper and an evaluation harness for a benchmark consisting of 164 Python programming problems based on publicly available code on GitHub. It was developed to test coding in particular.

Eventually, though, experts found that the problem sets in HumanEval were insufficient and ambiguous (Liu et al., 2023). HumanEval+ was released to include more tests, but the next benchmark’s authors suggested that models might be overfitting to HumanEval+ as well, memorizing the dataset and answers rather than genuinely demonstrating reasoning ability.

LiveCodeBench (2024)

Jain et al. tried a different approach to HumanEval to avoid data contamination: it was released with a continuously-updated coding benchmark. Problems have release dates and models can only be evaluated using problems released after their training data cutoff dates.

LiveCodeBench represents a shift in benchmarks towards a more continuous, dynamic measure. However, the last submission to the benchmark was 9 months ago at the time of this writing, and the leaderboard has not been updated since then either, suggesting the benchmark is no longer being actively maintained.

SWE-bench (2024)

This benchmark was created by Jimenez et al. to include 2,294 actual GitHub issues from 12 Python projects. When it launched, it was a much harder benchmark than predecessors, with models scoring 2-4%. Today, models are scoring around 70-80% on the original benchmark. There were enough issues with the original’s dataset quality that variants were introduced, including OpenAI’s SWE-bench Verified. Today, the SWE-bench website shows these variants:

SWE-bench
SWE-bench Verified
SWE-bench Multilingual
SWE-bench Multimodal
SWE-bench Lite

Agents and tool use

This last category is a preview of where I think the field should go: benchmarks that measure what a model does rather than just what a model knows. Here are some of them:

GAIA (Mialon et al., 2023) — 466 real-world assistant tasks that combine web browsing, file reading, and multi-step reasoning. Easy for humans (92% success), hard for models (15% success for GPT-4), at least at launch.
Terminal-Bench — agents working in actual Linux terminals. Tasks include “build the Linux kernel from source” and “configure a git server plus a webserver on port 8080.” No mocks; if it doesn’t actually build, it fails.
WebArena (Zhou et al., 2023) — realistic simulated websites (Reddit, GitLab, e-commerce) where agents complete actual user goals like “filter the last six months of orders by refund status.”

These benchmarks are harder to saturate than general-knowledge ones, because passing means the thing actually has to work in the real world. A model can memorise a question; it can’t memorise a Linux kernel that isn’t there until it builds it.

The wrong answer: Build a harder general benchmark

By now, you probably see that benchmarks tend to go the same way: A benchmark is created, models get too good at it and perform too well to be useful, and then a variant benchmark is created to address the problem.

This is the saturation problem in a nutshell: a benchmark that once provided meaningful information to help distinguish models eventually fails to do so because all the models’ scores cluster at the top.

Saturation occurs for two reasons:

Contamination: Many benchmarks publish their problem sets online, and models train on data scraped from the web. The benchmark now measures memorization, not capability.
Overfitting: Model providers are incentivized to optimize specifically for gaming performance on benchmarks. Scores go up without corresponding capability improvements.

When you have a party of wizards with 20 INT, “What’s your INT score?” stops being an interesting question because any wizard’s Fireball will hit the same. Building harder benchmarks is like an arms race for ability scores, when what you actually want to know is whether your wizard can kill a dragon.

Harder benchmarks don’t tell us what we want to know: can this model do my specific job?

The right answer: skill checks with real DCs

The fix isn’t a harder ability check. It’s a different kind of check altogether. In D&D, straight ability checks are rare compared to skill checks. There are 18 official skills that break down the six ability scores into more granular activities. For example, your wizard is much more likely to roll one of these instead of a straight INT check:

Arcana
History
Investigation
Nature
Religion

All of these are still based on INT, but test narrow types of intelligence rather than general intelligence.

Specific, domain-grounded benchmarks are what the field actually needs. The evaluations that actually help practitioners share a structure: they test a specific skill in a specific context, and they’re judged by the people who have to live with the results.

MedQA tests whether a model can answer the kind of clinical question a physician might face. LegalBench tests whether a model can correctly identify enforceability clauses, classify contract language, and apply legal reasoning. FinanceBench tests whether a model can answer the kind of question a financial analyst would actually ask about a real 10-K filing. None of these are trying to rank models on general intelligence. They’re asking the only question that matters downstream: can this model do this specific job well enough?

This week, Grafana released o11y-bench, an open benchmark for observability-specific agent tasks. I’m biased — I work there — but it’s a useful illustration of the shape I’m describing. o11y-bench doesn’t ask whether a model is smart in general. It asks whether an agent can do the things observability engineers actually do: read production traces, correlate logs with metrics, figure out why the thing that’s supposed to be working isn’t. Its leaderboard shows models’ performance for each activity.

Does that mean that more specific benchmarks won’t get saturated? No. We’ve already seen this play out with SWE-bench. But they’ll get saturated more slowly and remain more meaningful measures along the way.

These are skill checks with DCs that matter to someone. They’re not trying to produce a single scalar ranking of “which model is smartest.” They’re asking whether the wizard recognizes the family crest belonging to a political faction or knows this specific cult’s practices, not whether the wizard is good at “wizarding” in the abstract.

Roll for initiative: What this means in practice

If you’re an engineer picking a model for production: stop picking by leaderboard. Find or build an eval that reflects your actual use case. A handful of representative examples, graded the way you’d grade them if you were doing the work yourself, will tell you more than a dozen public benchmarks.

If you’re a researcher thinking about what to build next: the interesting work is at the specific-and-grounded end of the spectrum, not the general-and-bigger end. Another harder general-knowledge benchmark will saturate in eighteen months. A well-designed domain benchmark, co-built with the people who actually do the work, can stay useful for years.

And if you’re anyone else watching this field from the outside: be skeptical of the headlines. When a new model scores 94% on a benchmark you’ve never heard of, ask what it measures and whether the thing it measures is the thing you actually care about. Usually the answer is “not really,” and that’s the most important thing a benchmark score won’t tell you.

The goal of a character sheet was never to produce one number that ranks the character. It was to give a DM enough information to run a meaningful encounter — and to give the player enough to know when to try, when to pass, and when to roll with advantage. LLM evaluation should work the same way. Not a ranking; a readiness check. Can this specific model clear this specific challenge? Roll for initiative.