English on Nicole van der Hoeven

Roundtable: How We Really Use AI in Testing (HUSTEF 2026)

Thu, 08 Oct 2026 14:00:00 +0200

On the final day of HUSTEF 2026 in Budapest, I joined the conference’s roundtable on AI in testing — “AI Open Space Experiences: a roundtable with speakers on their experiences with AI testing.” It was five of us on stage: me plus Marta Firlej, Kristoffer Nordström, Callum Akehurst-Ryan, and Maryia Tuleika, with Marta moderating.

The format was refreshingly honest — “how we really use AI, with no bullshit.” Marta had run the same “AI open space” exercise with the Polish testing community, crowdsourcing the actual, un-hyped ways people use AI (information gathering, functionality discovery, generating tests and code, TDD, recruitment, presentation prep, even designing furniture and planning holidays), and then opened it up to the panel and the room. Here’s roughly how I answered her questions:

What do you use AI for? Mostly two things at work: analysing codebases and test code to understand what something actually does, and filling gaps on teams without a dedicated quality engineer — baking my testing knowledge into an AI the team can ask questions of. On the personal side, I’ve run some weird experiments: I’m a lifelong note-taker with decades of context (journals, therapy notes, eight years of WhatsApp history with my partner), so I’ve fed it all into a local, open-weight model and asked things like “what do I claim to believe, and where have I acted contrary to that — cite your evidence?” It’s a surprisingly honest mirror. I’m in a privileged spot with effectively unlimited AI access right now, and I know it won’t last, so I want to find out what’s safe and what isn’t while I can.

How do you stay up to date? Honestly, I don’t — not by chasing model releases or news. I keep up with how other people, especially in testing, are using AI, and I lean on my coworkers (we’re at the forefront of this at work, so stepping outside that bubble is a useful reality check). I do have a scheduled Claude Code task that scrapes my ~34 AI-related Slack channels into a daily digest of just the things I care about.

One piece of advice? Don’t trust AI too much, and don’t believe you’re as far behind as you think. Almost no one has really tried this stuff; it’s genuinely early, everyone’s learning, and your foundational testing skills transfer. Just pick it up, play, and if it’s not for you, move on.

Anything you’d never use AI for? Communicating with people as a proxy. I’ll use it to translate or to help me sound more fluent (I live across multiple languages — Portugal, Dutch, English), but I never want it to craft a whole message I didn’t think of. I don’t want to lose the discipline of actually talking to a human.

Ethical dilemmas? Yes — which is exactly why I keep my use mostly in the context of work, and try not to generate imagery or anything that’s obviously lifting from artists who deserve to be paid for it. There’s no hard line; it’s a sliding scale, and I have to keep questioning myself. (A Mastodon line stuck with me: the strange future where the humans do the menial tasks and the robots write poetry and paint. I try not to let AI stop me from making art.)

The best part of these panels is always the disagreement and the texture — five people who use AI daily and still land in very different places on trust, ethics, and what it’s actually for. Thanks to Marta for moderating and to Kristoffer, Callum, and Maryia for a genuinely good conversation.

Resources

HUSTEF 2026
My HUSTEF keynote: Asimov’s Zeroth Law of Robotics: Testing for AI

Asimov's Zeroth Law of Robotics: Testing for AI (HUSTEF 2026)

Wed, 07 Oct 2026 10:00:00 +0200

This week I gave a keynote at HUSTEF 2026 in Budapest, Hungary. The talk is called Asimov’s Zeroth Law of Robotics: Testing for AI — a testing-focused evolution of a talk I’ve been giving (and rewriting) for a while, previously as Observability for AI at KubeCon EU 2025 in London, Dutch Cloud Native Day 2025 in Utrecht, NewCrafts in Paris, and ExpoQA Madrid 2026. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be testable.

AI systems fail differently than traditional software. There’s no single correct output to assert against. A response can be fluent and completely wrong. Models drift silently over time. And yet we’re shipping AI into production — often without knowing what “passing” even looks like.

Observability is what makes testing possible. When we can see inside our AI systems — trace the chain of reasoning, measure what changed, surface what failed — we can start to write evaluations, define what “good” looks like, and catch regressions we didn’t know to look for.

In this talk, I explore what it means to test AI in the real world, drawing on lessons from building and evaluating an AI assistant for a large-scale observability platform. I’ll cover how evaluations work as the AI equivalent of a test suite, what to measure, how to catch hallucinations, and how to know when your model has regressed.

You cannot test what you cannot see — and you cannot trust what you cannot test.

The big shift in this version is right there in the Zeroth Law itself. In earlier iterations I argued that a robot must be observable. That’s still true — observability is the only way to see the reasoning behind a system and find out what actually happened. But I’ve come to think that pure knowledge isn’t the same as trust. You can know exactly what a system did and still not trust it. The only way to trust something is to put it through its paces: to know how it behaves in the situations you foresee, and even the ones you don’t. So the Zeroth Law has become a robot must be testable.

I framed the whole thing around Asimov’s short story Runaround (from I, Robot) and its robot, Speedy. Speedy’s smuggler owners sent him to collect selenium on Mercury, and he got stuck in a loop — circling the selenium pool, unable to resolve the conflict between obedience (the second law) and self-preservation (the third). His owners couldn’t predict his behaviour and couldn’t reason about it, so they had to do the only thing left: go out into the danger zone and fix him in production. Speedy wasn’t untestable because his owners were incompetent or lacked the tools — he was untestable because he was never built to be tested. That’s the trap I think a lot of AI systems are walking into right now.

The through-line for a testing audience: AI is rewriting how software gets built, but it doesn’t rewrite the discipline of trust. Trust that a system does what we say it does, and that when it breaks we know why and can fix it — that’s always been the job, and it isn’t going anywhere. There’s a whole generation of AI engineers shipping software they haven’t tested and can’t explain, and when someone says “the agent does this,” our job is still to ask how do you know? Testing isn’t just going to survive the age of AI — testing is the reason AI gets to survive, if we make it testable.

I demoed this with my TTRPG side project: an AI Dungeon Master, instrumented and put under test. It turns out that writing evals for something non-deterministic forces you to confront your real requirements. I thought I knew exactly when a dungeon master should call for a dice roll — I GM myself! — but trying to put it into words (“in this situation I’d roll, but in that one I wouldn’t, because I want to keep the story moving”) surfaced decisions I’d never had to make explicit. Those are things we’d normally discover much later, in production. Having to figure them out up front is a win.

The Q&A was genuinely great and ranged well beyond the slides:

“When has an AI been tested enough?” Same answer as for anything else — it’s a mix of risk and budget. If it were a life-critical robot like Speedy, you test every scenario the people relying on it need. For my little demo app, less so.
Testers vs. developers in the age of AI. I said I’d fire developers before testers — provocatively, but the point stands: anyone can write code now. What made a good developer good was never the coding; it was communicating with business and engineers, translating requirements, finding creative-but-safe solutions. Testers already think at that level and translate across teams, which makes them more important in the age of AI, not less.
Defining expected results for non-deterministic systems, multi-turn evals (I give the simulated user a personality — adversarial, cooperative, beginner — rather than a script), and the broader shift from imperative to declarative: we no longer instruct step by step, we declare the end state we want, the way we do with Kubernetes.
Convincing colleagues not to fear AI. I don’t, really. AI is here to stay; the question isn’t whether you approve of it, it’s whether you want to stay in this industry and help make it safe and trustworthy — or sit on the sidelines.

I also recommended the book Slow AI, which lands where I do: not an AI apologist, not a boycott, but a clear-eyed list of what AI is and isn’t good for — and the reminder to keep checking whether a given use of AI is making you more capable or less.

The demo app (the instrumented AI DM) lives on GitHub, and I keep refining it with each iteration of this talk.

I’ll update this post with the recording once HUSTEF publishes it.

Resources

Slides
Demo app on GitHub
Slow AI (book)
Previous versions of this talk: KubeCon EU 2025 · Dutch Cloud Native Day 2025 · NewCrafts 2025 · ExpoQA Madrid 2026

When to use Grafana Assistant vs MCP vs GCX: Grafana MCP

Wed, 19 Aug 2026 09:00:00 +0100

Three AI tools people confuse: Assistant, MCP, and GCX. Last time I talked about the brain — Assistant. Now hand number one: the MCP server, the easy hand.

Grafana MCP — Model Context Protocol — lets any AI client reach into Grafana. The magic part: you bring your own brain.

Using MCP requires two pieces:

The client is your AI app — the brain you bring: claude.ai, ChatGPT, Cursor.
The server is the door into Grafana that exposes the tools.

There are two ways to get that server. The easiest is the Grafana Cloud hosted one — already running, auth built in, just connect and go. But you can also self-host the open-source one on your own infra.

So when do you reach for Grafana MCP?

When you’re using hosted chat tools with no terminal — claude.ai, ChatGPT, your phone. No shell means no CLI, so MCP is the only way in.
When you want to bolt Grafana onto a tool you already use — already in Cursor or Claude? Just add the server. No new workflow.
When you want the best discovery out of the box — the tools describe themselves up front, so even smaller or cheaper models just know what’s available. No trial-and-error.

One honest catch: those tool definitions can eat some context if you load everything — but modern hosts fix that with lazy tool search.

MCP is deliberately curated — it doesn’t expose every corner of Grafana. So what do you do when you need the esoteric stuff? That’s exactly what the power hand, GCX, is for — next up.

Transcript

Three AI tools people confuse: Assistant, MCP, GCX. Last time, I talked about the brain — Assistant. Now hand number one: the MCP server, the easy hand.

Grafana MCP — Model Context Protocol — lets any AI client reach into Grafana. The magic part: you bring that AI client - you bring your own brain.

Using MCP requires two pieces. The client is your AI app — the brain you bring: claude.ai, ChatGPT, Cursor. The server is the door into Grafana that exposes the tools.

Two ways to get that server: the easiest way is to use the Grafana Cloud hosted one — already running, auth built in, just connect and go. But you can also self-host the open-source one on your own infra.

So when do you reach for Grafana MCP?

When you’re using hosted chat tools with no terminal — claude.ai, ChatGPT, your phone. No shell means no CLI, so MCP is the only way in.
When you want to bolt Grafana onto a tool you already use — already in Cursor or Claude? Just add the server. No new workflow.
When you want the best discovery out of the box — the tools describe themselves up front, so even smaller or cheaper models just know what’s available. No trial-and-error.

One honest catch: those tool definitions can eat some context if you load everything — but modern hosts fix that with lazy tool search.

But MCP is deliberately curated — it doesn’t expose every corner of Grafana. So what do you do when you need the esoteric stuff? That’s exactly what the power hand is for. Next up is GCX.

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Mon, 17 Aug 2026 09:00:00 +0100

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. They’re not competitors. Think of it as a brain and two hands. In this short, I start with the brain: Grafana Assistant.

Assistant isn’t just a chatbot — it’s the reasoning layer. It manages context and knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. Instead of clicking around, you tell Assistant what you want to do and it figures out the how.

Reach for the Assistant when the work is visual or exploratory:

Building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
A real incident that needs root-cause analysis across multiple data sources? Assistant. That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in, and I’ll cover the first of those (the MCP server) next.

Transcript

Grafana has three AI things people keep confusing — Assistant, MCP, and GCX. I’m going to talk a bit about each one so you know which one is best for your use case.

Spoiler: They’re not competitors. Think of it as a brain and two hands.

Let’s start with Grafana Assistant, which is the brain.

It’s not just a chatbot — it’s the reasoning layer. It manages context, it knows your Grafana: your dashboards, your data sources, a knowledge graph of your infrastructure. So instead of you clicking around to do something, you can say what you want to do to Assistant and it figures out the how.

Reach for the Assistant when the work is visual or exploratory.

If you’re building a dashboard? Assistant.
“Teach me how to do this” or “where is this in the UI”? Assistant.
You have a real incident where you need root-cause analysis across multiple data sources? Assistant.
- That’s its sweet spot — it forms a hypothesis, correlates your signals, and can even open a pull request with the fix.

Here’s the key thing though: Assistant is Grafana’s own brain — but it’s not the only brain that can drive Grafana. That’s where the two hands come in… and I’ll show you those next.

Resources

Grafana Assistant (product page)
My notes on Grafana Assistant
My notes on GCX
Unlearnings from building Grafana Assistant — my Context Horizon piece

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Fri, 31 Jul 2026 09:00:00 +0100

The final day of AI Week is about collaborating — using AI right where your team already works, no context switching required. Grafana Assistant in Slack is now generally available: investigate issues, ask follow-up questions, and query your metrics and logs from the channel where the incident conversation is already happening. And that’s just one place Assistant now meets you — also announced: the Grafana mobile app is GA (with Assistant on mobile in public preview), Assistant Desktop in public preview, Microsoft Teams support in private preview, and context-aware Assistant Search in public preview. Add it all up and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search. That’s a wrap on five days of announcements.

Transcript

It’s the final day of Grafana Labs’ AI Week, and Friday is all about collaborating — using AI right where your team already works, no context switching required.

Today, Grafana Assistant in Slack is generally available. Investigate issues, ask follow-up questions, query your metrics and logs — all from the channel where your incident conversation is already happening.

And that’s just one place Assistant now meets you. Today we also announced: the Grafana mobile app is GA, with Assistant on mobile in public preview. Assistant Desktop is in public preview. Microsoft Teams support is in private preview. And Assistant Search — deeper, context-aware search — is in public preview too.

Add it all up, and Assistant is now wherever you are: the Grafana UI, the API, your terminal, your coding agent, Slack, Teams, mobile, desktop, and search.

That’s a wrap on five days of announcements — and I know it was a lot. So if you missed a day, or you just want the full picture of where AI at Grafana stands today, I made a video for exactly that: everything you can do with AI in Grafana Cloud, including everything we shipped this week. It’s linked below and pinned in the comments.

One last time: grafana.ai for all of it. Thanks for spending AI Week with me.

Observability and AI explained

Fri, 31 Jul 2026 09:00:00 +0100

“AI observability” gets thrown around a lot, and it can mean one of two very different things. So in this video I map out the whole landscape of AI-related work at Grafana across the entire software development lifecycle — and show how AI and observability come together in two directions: AI for observability (AI that helps you observe your systems) and observability for AI (observing the AI itself).

The framing is deliberate: AI in observability isn’t about replacing engineers. It’s about handing back the exhausting parts of the job — the 3am triage, the repetitive queries, the report nobody wanted to write — so you can spend your energy on the work only humans can do.

I walk through each phase of the lifecycle and where the tools fit:

Planning — clarity about what’s really going on, with Grafana Assistant: not just a chatbot, but an agent that knows your telemetry, writes your queries, and builds your dashboards.
Building — momentum without losing control: Assistant, the open-source GCX CLI, the Grafana MCP server, the specialized Loki/Tempo/k6 MCP servers, and the AI SDK for building your own agents in Go.
Operating — staying calm under load: Investigations for agentic incident triage, Assistant Automations and the Watcher Agent for staying proactive, plus Grafana Machine Learning (Metrics Forecasting, Outlier Detection) and Adaptive Telemetry.
Testing & evaluating — doubt is a form of care: k6 and agentic testing on the systems side, and Agent Observability plus the open-source o11y-bench benchmark on the “observing your AI” side.
Collaborating — bringing it all to where your team already works: Assistant in Slack (and coming to Teams), on mobile, on the desktop, and via API/MCP, plus Assistant Search across your Grafana artifacts.

If you’re new to the Grafana stack itself, start with my open source observability video for the non-AI foundations that all of this builds on.

Resources

My notes on Grafana Assistant
My notes on GCX
My notes on k6
o11ybench.ai — the open leaderboard for AI agents doing observability work

AI Week Day 4: Evaluating (Grafana AI Week — Thursday)

Thu, 30 Jul 2026 09:00:00 +0100

Day four of AI Week is all about evaluating — because building an agent is the easy part; trusting it is where things get hard. The headline: Grafana Agent Observability is now generally available — the tool Grafana built for itself while scaling Assistant, once it grew too complex to evaluate with unit tests. Instrument your agent with the SDK and your conversations, tool calls, traces, latency, token use, and cost all become observable in one place; add evaluators (LLM judges plus deterministic checks) to score quality on live traffic, feed low-scoring conversations into your test suite, and gate CI on experiments comparing a candidate against a baseline. And because evaluator metrics land in Prometheus, you can get paged when a bad prompt change tanks quality. The fun bonus: k6 agentic testing — describe a browser test in plain language and an agent embedded in k6 drives the browser, no script required (they proved it by getting it to beat Wordle).

Transcript

It’s day four of Grafana Labs’ AI Week, and Thursday is all about evaluating — because building an agent is the easy part. Trusting it is where things get hard.

The big news: Grafana Agent Observability is now generally available. This is the tool we built for ourselves while scaling Grafana Assistant, when it grew too complex to evaluate with unit tests. Now it’s available to everyone on Grafana Cloud.

Here’s how it works: instrument your agent with the SDK, and your conversations, tool calls, traces, latency, token use, and cost all become observable in one place. Then add evaluators — LLM judges plus deterministic checks — to score quality on live traffic. Low-scoring conversations feed your test suite, and experiments let you compare a candidate against a baseline before you ship — you can even make it a CI gate. And because evaluator metrics land in Prometheus, you can get paged if a bad prompt change tanks quality in production.

But my favorite part of today? k6 agentic testing. Describe a browser test in plain natural language, and an agent embedded in k6 drives the browser for you — no script at all. And to prove it, we gave it one job: beat Wordle. Go watch that video. I’ll wait.

Everything’s at grafana.ai — see you tomorrow for the finale: using AI where your team already works.

Investigations Arena: how we test Assistant investigations

Thu, 30 Jul 2026 09:00:00 +0100

How do you test an AI agent whose whole job is to investigate incidents — when every investigation is different and “correct” isn’t a single deterministic answer? In this interview I sit down with Will, a software engineer at Grafana who’s spent the last year focused on AI: the Grafana Assistant, investigations, and memory features.

The story here is how the team got from Investigations V1 to a much better V2. V1 hit a ceiling, and the demo environment they’d been testing against turned out to be the problem — the cases were too easy and too far from the messy reality of a production cluster. So they built an arena: think competitive bot programming, but the “game” is investigating real incidents in Grafana’s own production environment. Different AI agents, each with its own strategy and tools, all take a swing at the same incident, and an LLM-as-judge scores their answers against a human-verified root cause.

What’s interesting is that the score wasn’t even the most valuable output at first — the insights were. Watching which strategies worked, which tools returned so much noise that they poisoned the context, and which agents wandered off is what let the team optimize bit by bit into the V2 release. The arena also compares more than models: different harnesses running Sonnet vs. Opus, Claude Code with the Grafana MCP, even just hitting the public API directly with curl.

A couple of my favorite moments: Will explains the guardrails that stop agents from “cheating” (they can only look at data from within the incident’s time window, so they can’t peek at the fix commit that resolved it) — and how the models mostly play fair because they treat it like a game. And there’s a lovely example of emergent creativity: an agent reached for tools the team hadn’t even considered, they turned out to be genuinely useful for investigation, and those tools got promoted into the real harness. Will’s hope for the future is to open this up — an OliBench-style shared benchmark where the community can throw their own investigation agents into the arena and raise quality across the board.

Transcript

Will: Hey, my name is Will. I’m a software engineer at Grafana. I’ve been there for three years, and for the last year I’ve been focusing on AI — the Grafana Assistant, investigations, and memory features.

Nicole: And what’s the latest thing you’ve been working on?

Will: So we developed Investigations V1, and we saw that we hit a ceiling at some point. We came up with a different approach, but this time we needed a good evaluation environment — the demo environment wasn’t enough, because those cases were too easy and too far from the reality of a production environment. So we decided to build an arena.

It’s a little bit like the idea of competitive bot programming, where bots fight against each other on specific tasks or games. In this case, the topic was investigations in our production cluster — real incidents. We started building bots that had different strategies and different tools; these were AI agents. Comparing a diversity of strategies let us surface the best approaches, and from there we could iterate, bring in more test cases from more incidents, and reach higher and higher quality.

The idea is that the bots try to solve the same case, and we use an LLM as a judge to compare the different approaches. We have different criteria, because for each incident we have a root cause that’s been verified by a human. That gives us our ground truth, and the LLM-as-judge compares that ground truth against the answers the agents came up with.

The important part at first wasn’t even the score — it was the insights. This tool was better with this strategy; this one had noise in the results that poisoned the context; the results returned from this specific tool were just too much. So we managed to really optimize bit by bit and come up with Investigations V2, which we just released. It’s a massive improvement compared to our V1 approach.

Nicole: So is it comparing just different models for investigations, or is it also comparing things like Claude Code and the Grafana MCP versus Assistant?

Will: Both, actually. We have different versions of our harness for investigations that we run with Sonnet and with Opus, but we also use the Claude Code harness with the Grafana MCP — and even just using curl to directly hit our public API.

Nicole: Do the models ever cheat?

Will: So far it’s been quite good. We have guardrails. Because we investigate past incidents, we need to make sure the LLM only looks at a specific time window — only from when the incident started — otherwise it would be able to find the pull request that was used to solve the incident. But they’re pretty good at following the rules; they take it as a game a little bit. We even have an agent that checks afterward, with the judge, to make sure they followed every rule.

It’s funny, actually — we had a case using Claude Code where it started to use some tools we hadn’t thought about in the first place, and they turned out to be very useful for investigation. We’d never considered them, so we brought them into our investigation harness and it helped a lot. Giving the agents freedom and creativity in this kind of environment is really useful. That diversity sponsors new ideas that we can put into our own work.

Nicole: Cool. Do you have any plans to publish this and share it so that other people can do it too — kind of like OliBench?

Will: Yeah, I think, why not? I think we can go this way too. I’m sure it would be useful for other agents. And it would be very nice if we could compare agents from other people and other companies — maybe with different harnesses, maybe even different tools — because right now it’s mostly about our MCP, our CLI, and our investigation harness. But if the community comes up with their own investigation agents — and I think I’ve seen a few — it would be fun to put them in the arena and help not just our product, but any product improve in quality.

Resources

My notes on Grafana Assistant

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Thu, 30 Jul 2026 09:00:00 +0900

This week I spoke at KubeCon + CloudNativeCon Japan 2026 in Yokohama, Japan. I gave a talk called The Great Doubt: What Building an AI Agent Taught Us About Trust. Here’s the abstract:

When we started building an AI assistant for observability, we thought the hard part would be making it smart. We were wrong. The hard part was knowing when to trust it.

AI agents hallucinate, forget context, and confidently give wrong answers. Traditional testing doesn’t catch these failures. What’s needed is evaluation grounded in systematic doubt.

This talk shares lessons from shipping an AI agent to production: how to build a golden dataset of use cases you must get right, how to use LLM-as-judge when there’s no ground truth, and how to use OpenTelemetry traces to debug eval failures. You’ll also hear what’s still unsolved: evaluating multi-agent handoffs and closing the feedback loop between what users ask and what your evals cover.

Kyoto School philosopher Nishitani Keiji called this “The Great Doubt” (大疑): questioning every assumption until only what survives is real. For AI agents, that’s not philosophy. It’s the job.

The talk walks through the layers of doubt we worked through while building Grafana Assistant: doubting the agent itself, then doubting our tests, then the LLM-as-judge, then our own scores — and finally doubting whether the doubt was even justified. Each layer needed a different tool, from a golden dataset run in a fully synthetic observability environment, through online evals on live traffic, to per-conversation OpenTelemetry traces that link every eval score back to the exact LLM and tool calls that produced it.

You can find the slides at nicole.to/doubtslides, and you can watch the full talk below:

Resources

Grafana Assistant (generally available for Grafana OSS, Grafana Cloud, and Grafana Play):

Assistant Investigations (Grafana Cloud):

Docs

Agent Observability (Grafana Cloud):

Docs

o11y-bench (OSS):

Site

Investigations Arena:

Video

My notes on Grafana Assistant
My Context Horizon piece Unlearnings from building Grafana Assistant
The Way of Zen by Alan W. Watts
Zen Mind, Beginner’s Mind by Shunryu Suzuki
Religion and Nothingness, Volume 1 by Nishitani Keiji
Philosophers of Nothingness: An Essay on the Kyoto School by James W. Heisig

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Wed, 29 Jul 2026 09:00:00 +0100

Day three of AI Week is about operations and maintenance — because once code reaches production, the operational work is still surprisingly manual: someone’s watching dashboards, deciding whether an error spike is real, and writing up what happened, every single day. Two Grafana Assistant features went GA to take that off your plate. Assistant Automations handle recurring work — save a prompt and run it on a schedule (like a morning brief on your service’s health), each run keeping its own history and able to notify you in Slack. Assistant Investigations is for when something actually breaks: it digs across metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report you can follow along with in a Workspace. You’re offloading vigilance, not accountability — the evidence is always there to review, and you still make the call.

Transcript

It’s day three of AI Week, and Wednesday is all about operations and maintenance. Because here’s the thing: once your code reaches production, the operational work is still surprisingly manual. Someone’s watching dashboards, someone’s deciding whether that error spike is real, and someone’s writing up what happened — every single day.

Today, two features of Grafana Assistant are now generally available to take that burden off your plate: Investigations and Automations.

Assistant Automations handle the recurring work of operating a system. Save an Assistant prompt and run it on a schedule — like a morning brief that summarizes the operational health of your service over the last 24 hours. Every run keeps its own conversation and history, and it can notify you in Slack when it’s done or needs approval.

Assistant Investigations is for when something actually breaks. Investigations digs into issues across your metrics, logs, traces, and profiles, develops and tests hypotheses, and hands you a structured report. You can follow along in a Workspace, see what’s been ruled out, and add guidance while it works.

You’re offloading vigilance, not accountability — the evidence and reasoning are always there to review, and you still make the call.

Everything’s at grafana.ai — see you tomorrow for day four: evaluating your agents.

AI Week Day 2: Building (Grafana AI Week — Tuesday)

Tue, 28 Jul 2026 09:00:00 +0100

Day two of AI Week is all about building — the AI tools that help you write software with confidence. Two big announcements went generally available: gcx (the Grafana CLI) and the Grafana MCP server. Both connect your coding agents — Claude Code, Cursor, and friends — to your Grafana telemetry. The MCP server gives agents opinionated, ready-made tools for the common cases (querying metrics and logs, reading and updating dashboards); gcx is the more flexible, less opinionated option, built for agents from the ground up with read-only guardrails and its own bundle of agent skills. Both work for OSS and Grafana Cloud. The point: when an agent writes your code, it shouldn’t be guessing — it should be looking at how your system actually behaves in production. That’s telemetry-driven development.

Transcript

It’s day two of Grafana Labs’ AI Week, and Tuesday is all about building — the AI tools that help you build software with confidence.

Today we have two big announcements: gcx and the Grafana MCP server are both now generally available.

Both of them connect your coding agents — think Claude Code or Cursor — to your Grafana telemetry. So what’s the difference?

The Grafana MCP server gives your agent opinionated, ready-made tools for the most common use cases: querying your metrics and logs, reading and updating dashboards, that kind of thing. You can run it yourself, or use the hosted MCP server with your Grafana Cloud stack — so yes, it works for both OSS and Grafana Cloud.

gcx is the Grafana CLI, and it’s the more flexible, less opinionated option. Agents can explore it and build their own workflows, and it was designed for agents from the ground up, with read-only guardrails so your agent can’t touch anything you didn’t approve. It works with Cloud stacks and self-managed instances, and it even ships with its own bundle of agent skills.

Why does this matter? Because when an agent writes your code, it shouldn’t be guessing — it should be looking at how your system actually behaves in production. That’s telemetry-driven development.

Head to grafana.ai for everything we’re shipping this week, and I’ll see you tomorrow for day three.

AI Week Day 1: Planning (Grafana AI Week — Monday)

Mon, 27 Jul 2026 09:00:00 +0100

It’s day one of Grafana Labs’ AI Week, and Monday is all about planning — getting the clarity you need on what’s going on in your system so you can figure out what to build. Kicking things off is Workspace for Grafana Assistant: a full-page view of Assistant so your conversations and generated graphs aren’t crammed into the little right-hand sidebar. It’s a much less cluttered place to run investigations and to be agent-first. On top of that, Assistant is now easier to try — enable it in Grafana 13 for OSS without linking a Grafana Cloud account, or try it with no signup or download at all on Grafana Play.

Transcript

Hi everyone! I’m Nicole van der Hoeven and I’m really excited to welcome you to AI Week. Today is Monday, which is planning day, where we give you the tools you need to get clarity on what even is going on in your system so that you can figure out what to build.

Now we’re kicking things off with a feature called Workspace for Grafana Assistant. Now if you’ve used Assistant before, you’ll know that it comes up on the little sidebar on the right side of Grafana. Well, that can start to get cramped if you have multiple conversations or if you’ve asked Assistant to generate multiple graphs for you. Well, we’ve heard you, and now we’re giving you a full page view of Assistant. So you’ll have all of your conversations on the left, and this is also an excellent place to run investigations from. You can also zoom in on any of those graphs, and it’s just a much less cluttered view if you want to be agent-first.

Speaking of Assistant, the second thing I want to tell you is that you know, Assistant was already available everywhere. But now, Assistant is available… in more places than everywhere? And also more easily. So Assistant was already available for OSS, but you had to link your Grafana Cloud account, which was not so handy if you wanted to try it out. But not anymore. Now if you download Grafana 13 and enable Assistant, then you’ll just be able to go and try it out and chat with Assistant without doing anything else. It’s still not free or open source, but it IS a lot easier to try it out.

And if you don’t want to download anything, that’s what Grafana Play is for. Grafana Assistant is now available to try out on Grafana Play without any signup or any download required. Try it out at play.grafana.org. I really love seeing it there. Tune in tomorrow where we’re going to talk about everything we’re announcing to help you start to build.

My reMarkable Obsidian workflow

Fri, 03 Jul 2026 17:00:00 +0100

I’m a big note-taker — but what most people don’t realise is that digital notes have only ever been a small part of that. I love analog notes, and lately I’ve loved meshing the analog with the digital using E Ink tablets. The problem with the reMarkable has always been the workflow: there hasn’t been an easy way to get everything back into Obsidian, which — as you know — is kind of important for me.

In this video I show how there’s finally a clean way to tie it all together: reMarkable → Obsidian as a PDF → searchable, OCR’d text.

The setup: a lot of E Ink

I’ve tried a lot of E Ink tablets and readers. My everyday carry is the reMarkable Paper Pro Move, and I keep a full-size reMarkable Paper Pro at my desk. I love them more than the Kobo or Kindle Scribe lines — but the deciding factor was always going to be whether I could get my handwriting into Obsidian without friction.

Step 1 — Sync the PDF into Obsidian

The piece that makes this work is a community plugin: reMarkable Sync by Tim Dommett.

In Obsidian, go to Community plugins → Browse, search for reMarkable, install and enable it.
The first time, you’ll register the plugin — it sends you to your reMarkable page, gives you a one-time code, and you paste that in and hit register.
In settings, choose a sync sub-folder. I put mine under plugins/reMarkable.
Set the sync interval — I have mine sync every 30 minutes, but you can go more frequently, manually, or trigger it on demand from the command palette (reMarkable Sync: Sync documents).

Once that’s done, anything I write on the reMarkable lands in my vault as a PDF — diagrams, cursive, print, colours and all. In the video I write out a page (neural nets, as explained by my partner Noel), trigger a manual sync, and watch it appear in Obsidian.

Step 2 — OCR the handwriting into real text

A PDF in Obsidian is a start, but it isn’t searchable or editable. That’s where OCR (optical character recognition) comes in — turning the handwritten image into text that behaves like any other note: searchable, linkable, modifiable. I keep the PDF too (I like preserving the colours and the handwriting, especially when someone else wrote on my tablet), but I want the durable text version as well.

I automate this with OpenClaw — my AI agent, Iris, runs a skill on a heartbeat that watches my reMarkable folder, notices new files as they drop in, and OCRs them automatically. It even fills in properties I like and, because it knows my context (and sometimes has the transcript too), it can describe diagrams more completely than I’d expect. It correctly read “Neural nets as explained by Noel” and “get diagrams from WhatsApp” — whether I wrote in print or cursive.

That setup is admittedly overkill, and not something I’d recommend to everyone. So in the video I also show the more accessible version:

The accessible alternative: Claude Cowork / Claude Desktop

If you’d like to automate OCR without a full agent setup, the Claude Desktop app (with Cowork) is a great option — especially if you already keep notes, since your vault is exactly the kind of context that makes AI immediately useful.

The prompt I use, roughly:

You are in my Obsidian vault, in the folder plugins/reMarkable. Look for the very latest PDF I’ve put in there and OCR it. If there’s a diagram, describe the diagram. If there’s text — handwritten cursive or print — transcribe it. Print all of your findings into a note in the plugins/Claude output folder, using the same file name as the original PDF.

It reads the latest PDF, describes the diagram (it got “neuron” and “perceptron” — words that were never actually written), and writes the note. Interestingly it did as well as my own setup, because under the hood we were both using the same model (4.8).

Then turn it into a skill. I tell Claude: “Please turn all of these instructions into a skill and call it OCR.” It writes the skill, preserving the paths, and I hit save. Now next time I have a reMarkable PDF to transcribe, I just run /OCR and name the file. If you want to go further, you can even ask it to make this a scheduled task.

(Tip: I use Spokenly to dictate these instructions to Claude rather than type them.)

Why this matters

This simple workflow has made it so I can actually use my reMarkables. I love typing notes on my laptop, but when I’m face-to-face with someone — in a talk, at a conference — a screen between me and the speaker feels distancing, and a little rude. I feel more present without it. And the little Paper Pro Move is far easier to carry than a laptop or iPad.

Now that I know everything ends up in Obsidian anyway, I get to pick whatever note-taking method I feel like in the moment — typing, reMarkable, or fully analog. That same OCR process (minus the reMarkable Sync plugin) also works for OCRing photos of pages from my paper traveller’s journal.

If you’d like to go deeper on how I use AI in my note-taking, check out my AI note-taking workflows video — I cover the different audio transcription tools I use there too.

Arigatō gozaimashita! Mata kondo ne~

Resources

🖊️ reMarkable Sync by Tim Dommett — the Obsidian community plugin that syncs reMarkable documents into your vault as PDFs
🧠 My notes on OpenClaw — the agent runtime Iris runs on
🎥 My AI note-taking workflows — the companion video on transcription and AI capture
🗣️ Spokenly — the dictation app I use to talk to Claude
💛 My Patreon

An Overview of AI Evaluation (The Context Window #05)

Fri, 26 Jun 2026 18:00:00 +0100

How do you know if an AI agent is actually any good — and whether the change you just made to it made it better or worse? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Yas Ekinci, an engineer on Grafana’s AI team who has spent the last several months focused almost entirely on evaluation. We talk about why trusting AI is so hard (non-deterministic models, oversized PRs, and the shift from writing code to reviewing it), the difference between online and offline evals, how we actually test Grafana Assistant, and o11y-bench, the open observability benchmark we announced at GrafanaCON. We recorded this one ahead of Yas’s two-month leave — so consider it a deep dive banked for the road.

The “final answer problem”

A recurring theme — and one close to my testing heart — is what I call the final answer problem. As a tester I’m used to looking at an outcome and asking “did this pass or fail?” But with AI, a plausible-looking answer isn’t enough. If the process was wrong, you can’t trust the outcome even when it looks right — and a plausible-but-wrong answer is the dangerous case, because nothing on the surface tells you it’s wrong. Yas’s example: a broken Tempo tool that returned no spans, while the agent still answered convincingly by guessing from metadata. Plausible ≠ correct.

This is also why the bottleneck has moved. The cost of writing code has collapsed, so the burden now sits on review. Developers, as I like to put it, are becoming shepherds of intent — less about typing the code, more about making sure the intent is clear and faithfully implemented.

What evals are, and online vs offline

An eval is simply a test of whether an AI agent behaves the way you want — from “who are you?” → “I’m Grafana Assistant” all the way up to multi-step querying and dashboarding. The point is to know with confidence whether a change made the agent better, which is what lets you improve it consistently over time.

There are two complementary flavours, and they map neatly onto testing concepts we already have:

Online evals — scoring a sample of real production conversations, usually with an LLM-as-a-judge. Great as a pulse check (it’ll catch a quality drop when a model provider silently swaps models), but it only tests the deployed variant and is hard to verify. Closest analogy: traditional observability / RUM.
Offline evals — a “golden set” of tasks run during development and CI against a controlled, reproducible environment. You can test every change, prompt, and implementation, and drill into specific skills (PromQL vs LogQL, dashboarding). Closest analogy: integration testing.

How we test Grafana Assistant

Assistant evals run a real agent-plus-tools harness against a real-ish environment (a Grafana with data sources, alerts, and dashboards), then grade the full transcript with “graders” or “verifiers.” The hard tension is that the environment has to be real and reproducible at the same time. Because Assistant is a front-end plugin that runs some tools in the browser, the team uses Playwright to replicate real browser behaviour — which is where I made my pitch for k6’s browser module (now approaching Playwright parity) as a dogfooding option. The internal eval tool is a CLI called LLM Spec, now on its third major iteration after the lessons from o11y-bench.

o11y-bench

o11y-bench is our public benchmark, announced at GrafanaCON, with six task categories: dashboarding, investigation, logs, metrics, and traces. A few things I think make it genuinely interesting:

It’s built on the Harbor framework (also used by terminal-bench), which separates harness from environment from tasks — so you can swap harnesses and compare, say, gcx + Claude Code against the Grafana MCP.
The leaderboard includes open models (Qwen 3.6, Kimi), not just the frontier labs.
It distinguishes pass@3 (got it right at least once in three runs — rough capability) from pass^3 (got it right all three times — reliability). The 10–20% gaps between the two are the whole reliability story in a single number.
It runs against a synthetic environment (Prometheus, Loki, Tempo in Docker) with generated data, so the correct answer is known in advance — and it uses fact-based rubrics, where the grader runs its own known-good queries against the environment and checks the agent’s answer against them. That’s how you catch the subtle, plausible-but-wrong failures.

The internal eval set is a superset that adds scenarios drawn from real (private) conversations and Assistant-specific behaviour — like Slack’s not-quite-markdown formatting — while o11y-bench stays generic and harness-independent, grading on outcome rather than method.

The evaluation loop

Yas’s blog post (the first engineering post on Grafana’s new community Unprompted blog) and our GrafanaCON talk both center on the evaluation loop: measure → make changes (new tools, prompt tweaks, bug fixes) → measure again. The crucial bit is that the loop can be driven by a coding agent — via an LLM Spec Improve skill — because you can’t unit-test natural-language “programming” the way you test code. There’s no clean relationship between the instructions you write and what the model does, so the only way to know the impact of a change is to run the loop.

Where it’s going

We closed on open questions and wishes: higher environment fidelity, more and harder tasks (very likely an o11y-bench 2.0), and getting more contributors — including PMs writing evals to capture desired product behaviour. I floated a k6-extension-style community model for evaluators, and using o11y-bench to support bring-your-own-model, so users can see exactly where a smaller or local model falls short instead of assuming Assistant is broken. Yas’s industry hope is the neat one: if labs tune their models against o11y-bench, observability and Grafana performance improve across the board — free fine-tuning out of the box.

Resources

Transcript

Nicole: Hey everyone. Welcome to another episode of The Context Window. This is a semi-regular podcast — we try to make it every month — and it’s all about the things going on with AI at Grafana. Usually this is live, and you might think looking at this that it’s live, but it’s actually prerecorded, because Yas has a busy schedule ahead and I wanted to get him on here before he left. So hi, I’m Nicole van der Hoeven, a senior developer advocate at Grafana Labs, and I’m assigned to the AI team these days. And one of the people on the AI team is — this person over here. How do you say your last name, actually?

Yas: Oh, it’s Ekinci.

Nicole: Ekinci. I was totally confused because I just found out you speak Dutch. Did not know that.

Yas: I’m based in Belgium, and I was like, “Nicole van der Hoeven” — okay, that sounds familiar enough to me.

Nicole: Yeah, it’s the name of my ex-husband, but it’s okay, we still like him. So tell us what you do on the AI team.

Yas: I’m an engineer on the AI team — I’m getting to five years at Grafana next week. When I joined, this was pre-AI; we had the machine learning team building predictive models, forecasting, that kind of stuff. As the AI wave came, we all got absorbed into this newer, larger AI department. It’s been quite varied — the assistant, the AI agent inside Grafana — but especially the latter half of this year I’ve been more and more focused specifically on evals: how do we figure out the quality and reliability of these things we’re deploying?

AI is a very useful tool. It has utility — it’s not like NFTs. We use it day to day. My job has changed drastically over the last two years. But you still notice it needs guidance and sometimes has reliability issues. The trend is that the guidance has come down — a year or two ago I’d steer an agent 50, 60, 70 times in a session; now it’s more like five or ten. But you still need to do it. If you just blindly vibe-code your way out of this, you’re not going to end up in a good place.

Nicole: It moves so quickly. Whenever someone tells me they tried AI and it didn’t do what they wanted, I ask, “Well, when was that?” Because if it was six months ago — try it again. In this episode we’ll talk all about evals, but first, some announcements. Assistant is now available for OSS and self-hosted Grafana, and I made a video showing how to do that. There’s also a new Medium blog for Grafana Labs called Unprompted — a community-focused blog — and Yas wrote the first engineering post, about building an evaluation loop. And we’ve started AI weekly videos: the AI team has a weekly meeting, Mat Ryer had the idea to publish excerpts, usually the demo part. Just be warned we demo things that may never make it to production — you’re getting it raw, warts and all.

Yas: I need to make sure I don’t wear my PJs anymore for the AI weekly.

Nicole: A bit of quick AI news: Warp is now open source. There are new models — GPT-5.5, Gemini 3.1 Ultra with a two-million-token context window, Anthropic’s Claude Mythos security preview. And there’s a great article by Matt Shumer called “Something Big Is Happening,” where he says, “I am no longer needed for the actual technical work of my job” — which hits hard — and advises leaning into what’s hardest to replace.

Yas: It’s still that guidance thing — you need a human in the loop to guide it. But if you ask me how much code I’ve manually written lately, the actual technical piece is very close to zero. We’ll talk about how that’s different for evals, because evals don’t work as well as coding does there.

Nicole: So why is it so hard to trust AI? Why don’t we just YOLO it and accept everything?

Yas: A mix of things. Everybody knows these models are non-deterministic and not perfect — these aren’t new problems; underneath it’s all machine learning, numbers and math and stats. Every time you talk to it you might get a slightly different response. The second part is it writes a lot of code. You can 10x your output — send ten agents at ten different features and launch all those PRs — but then someone needs to review them, and the PRs are typically big and over-engineered. One thing I always do at the end of a piece of work is ask it to find a simpler way and simplify. Bigger PRs plus so many changes makes it hard to keep a mental model of the whole thing. And interestingly, I’ll use AI to help review a PR — which gets you back to the same problem: if you use AI to review the AI, how much can you trust that?

Nicole: I like the phrase that developers are becoming shepherds of intent — we’re not writing the code so much as making sure the intent is clear and implemented. And this is the difference between process and outcome — what I call the final answer problem. I come from performance testing; I’m used to looking at the outcome and asking did it pass or fail. But now, even if the outcome looks right, if the process was wrong I don’t trust it. And the reverse too: right process, wrong outcome, something else is broken.

Yas: Tests are interesting here, because most of those tests are also AI-generated — and the AI doesn’t shy away from just changing the test to make its branch pass. For any branch I review, I still look at all the changes; I won’t blindly rubber-stamp. That’s probably the biggest bottleneck right now.

Nicole: One approach to trusting AI is human evaluation — thumbs up/thumbs down, or something like LMArena where you compare two answers blind. Useful, but not scalable; you can’t run it in CI.

Yas: On the eval side they’re all complementary. So, what is an eval? It’s testing whether the AI agent behaves like you want. A simple one: ask “who are you?” and expect “I’m Grafana Assistant.” There are much more complicated ones around querying and dashboarding. At its core, an eval is how you figure out if the agent is any good — and the reason we do it is so that when we make changes, we know if it’s better. Answer that with confidence and you have a path to improve consistently over time.

We split into online and offline. Online — we recently launched AI observability for agents — is about measuring how the deployed agent does against real production conversations from real users. Valuable, because you get a score based on real behaviour, but hard to verify; we typically use an LLM as a judge, so it’s not as robust. Offline is used more during development, in CI — much more controlled. You have a golden set of evals covering the behaviour you expect in production, and because you always test the same set, you can see whether changes make it better, and dig into PromQL vs LogQL, or dashboarding. Online is a great pulse check — it can catch quality dropping when Anthropic swaps a model — but it only tests the deployed variant. Offline can test all your changes, your implementation, and all your prompts.

Nicole: It took me a while to translate these, because the testing industry already has this — we call it pre-prod and production testing. Pre-prod is everything from unit to system integration testing in dev and staging; you can really hammer the thing. Production you have to be more careful — synthetic monitoring, real user monitoring — same idea with an AI flavour.

Yas: Definitely. Online is much more like traditional observability — you’re observing a system that happens to be powered by AI, turning natural language into a number you track over time and alert on. Offline is closer to an integration test: you give it a task, it runs the whole agent against a real environment, you get something out, and then separately you grade that outcome.

Nicole: We did an AI observability episode without you — you were at GrafanaCON — so I’ll link that. Let’s talk more about offline evals, since we haven’t covered those.

Yas: Offline is a bit of a beast. There’s a good Anthropic post called “Demystifying evals.” Back when we had simple single-turn LLM calls — incident summary was one of the first — the prompt was “here’s the context of the incident, summarize this,” and grading checked readability, completeness, appropriate length. Easy enough to set up. Agents get more complicated, because every time you run a task the agent can take very different trajectories — five steps one time, seven the next. The agent plus its tools is roughly the harness; it runs against an environment — a real Grafana with data sources, a certain alert and dashboard state — and keeps interacting using the tools until it’s done. Then we take the whole transcript and grade it with graders, sometimes called verifiers, to verify it actually got the task right.

Nicole: So how do you do it for Assistant, given how broad that environment is?

Yas: We have the public benchmark we released recently and our internal evals — same approach, different number of tasks and some different data sources. We want an environment as real as possible, but with the tension that we also need to control it so a benchmark is reproducible. If two people run it and get different scores, which one was right? We started, like everybody, with manual tests — listing data sources to confirm a tool is wired up, some CPU questions, showing logs — with a human as the judge. That’s fine at four or five scenarios; you can’t do it as a human at 500. Assistant is a front-end plugin — some tools run in your browser, which is good for auth and security but makes testing hard — so we use Playwright to replicate real behaviour and pull the telemetry out. The internal eval tool is a CLI we call LLM Spec, and it can run against cloud instances or a local controlled environment.

Nicole: Before the environment — did you consider using k6 instead of Playwright?

Yas: Interesting — yes and no. k6 is also a tool the assistant needs to use, but our goal isn’t to load test; it’s to hook into the plugin and get the messages and telemetry out, so we have a Playwright harness.

Nicole: There’s actually a browser module for k6 going towards feature parity with Playwright.

Yas: I did not know that. Might be interesting.

Nicole: Just dogfood all the things. I’d love to work with you on that. Okay, the environment.

Yas: We test against cloud instances — for incidents especially, it’s a lot of data. And we run LLM Spec against a local controlled environment, which is quite similar to the open benchmark we published.

Nicole: You’ve mentioned o11y-bench — the benchmark announced at the last GrafanaCON. That was fast.

Yas: o11y-bench has tasks across six categories — dashboarding, investigation, logs, metrics, traces. We wanted to bring our internal approaches outward, and it was a learning opportunity: we used a framework called Harbor, also used by the makers of terminal-bench. The nice thing is it separates harness from environment from tasks — so the default harness is a simple system prompt plus the Grafana MCP, but the folks working on GCX run Claude Code plus GCX against o11y-bench and compare GCX with MCP. There’s a leaderboard with a bunch of models, including open ones like Qwen 3.6 and Kimi.

Nicole: They’re getting good. Qwen 3.6 is awesome — not top five, but for an open-weight model, impressive.

Yas: And this is the interesting part: sort on pass@3 and it looks different from pass^3. pass@3 runs each task three times and passes if it gets it right one out of three — rough capability. pass^3 means you need three out of three on every task — reliably solving it, not just getting lucky once. You see 10–20% differences between the two scores.

Nicole: That’s awesome. And this is public, right?

Yas: Yep — the repo’s on GitHub with all the tasks. We run o11y-bench against a synthetic environment — a Prometheus, Loki, and Tempo in Docker — and we generate the data, so every run uses the same data and we know what the answer should be before we even run it. We also do fact-based rubrics: separately from what the agent does, the grading side runs its own known-good queries against the environment and checks whether the agent’s answer matches. That matters because these agents fail in subtle ways. We once made a change to the Tempo tool that broke the spans — the agent got no spans but still answered, fairly realistically, from metadata. Without testing for it you’d never catch it; the text looked plausible.

Nicole: That’s exactly process versus outcome. If the answer were negative two seconds it’s obviously wrong, but a plausible answer is the dangerous one.

Yas: Exactly, and that happens a lot.

Nicole: An early Assistant once told me it created a dashboard, and it didn’t look right — turned out it couldn’t find the data source so it just used dummy data.

Yas: Same thing happened to me locally — I broke all the tools, and it still came up with tool calls it couldn’t execute because of rough instructions in the system prompt.

Nicole: So what’s missing in o11y-bench that means you still need an internal one?

Yas: Two levels. First, some scenarios are based on real conversations or our own internal environments — anything involving private data can’t go public. Second, there are very Assistant-specific things we want to test — Slack, for instance, has its own version of markdown that looks like markdown but isn’t, and early Slack Assistant struggled to write it consistently, so we wrote tests for that. The internal set is a superset; the public set is still part of it. The public set is generic observability tasks — asking questions about your data, dashboarding, basic Grafana API checks — and because the harness is independent, we can test GCX, MCP, our assistant, or any other agent against it, grading on the outcome rather than how it was solved.

Nicole: We talked about your blog post — let’s define the evaluation loop.

Yas: At its core: measure the current state, see where it works and where it doesn’t, make changes — a new tool, tool instructions, the system prompt, bug fixes — then measure again, and again. The key is that loop can be driven by a coding agent, not us manually. If your evals are robust enough to pass a clean feedback signal back, the agent can keep improving until it hits 100%, and then you make harder tasks or increase coverage. At scale it’s nearly impossible for one human to understand the whole system and make targeted improvements, because an agent isn’t programmed as code — it’s programmed with natural language, and there’s no clear relationship between the instructions and what the model does. The only way to probe that is to run a bunch of these loops.

Nicole: Is the loop equally implemented for o11y-bench and LLM Spec?

Yas: It’s implemented internally. o11y-bench is really just the measurement part — we’ve used it for GCX and MCP, but more manually. In LLM Spec we have an LLM Spec Improve skill — you pick the agent, and it runs measure-improve-measure-improve.

Nicole: Where are you going next — no promises?

Yas: I’ve done a lot on the harnesses — the Playwright stuff, and there was an LLM Spec 1.0, then a 2.0 revamp for speed and robustness, then o11y-bench, and the learnings from that went into what I’d call LLM Spec 3.0. The two big challenges now are environment fidelity — more an issue for internal evals — and the number and quality of tasks: how do we cover more of Grafana? We weren’t sure o11y-bench would be a recurring thing, but there’s a lot of interest, so we’ll probably do an o11y-bench 2.0. And I want more people contributing — I was talking at the PM all-hands about getting product managers to write evals, because the tasks capture desired behaviour: when someone asks about my product, how should the agent ideally respond?

Nicole: What I’d love is something like the k6 extension model applied to evaluators. When Grafana acquired k6 — I was on k6 — k6 had extensions and modules, and we got a lot of cool community extensions. I think evaluators would be the same. After my Madrid talk, all the questions about the evaluator section were “can we add this, can we add that, why isn’t this in the template?” — and it could be.

Yas: It would be nice to do that across both o11y-bench and the internal ones — there are open questions on making them work together; internally we’re in Go, o11y-bench is on Harbor in Python, but the tasks and most verifiers are language-independent by design. A fourth thing is getting to a better golden set over time — we haven’t yet automated going from real AI-observability conversations to verifiable offline tasks. As the product changes, your tasks might not stay relevant — imagine k6 didn’t exist and now people ask Assistant about k6. Bridging online and offline in a more automated way would be the fourth big thing.

Nicole: I also have a wish for o11y-bench beyond being a benchmark — with no internal knowledge and no promises: it could be a step toward making Assistant bring-your-own-model. One barrier is that people try Assistant with a tiny local model and conclude it doesn’t work, when really you need certain models. A standardized way to measure that and show users where their model falls short — versus where the latest Opus sits on the leaderboard — lets them make an informed decision.

Yas: I’d go beyond Assistant — we’ve got GCX and MCP too. If you want to use your own models with them, the benchmark should reflect, for observability tasks, which are the best models and the best open models. It gets to your point: people can make a more informed decision.

Nicole: Where are we still missing things as an industry, not just as a company?

Yas: Just recently a new benchmark called DeepSWE was released, and it’s a bit controversial because it flips the ranking from the most popular software-engineering benchmark — where Opus was always leading, here GPT-5.5 leads by a margin. So there’s something to be said about the reliability of the benchmarks themselves, since labs test and sometimes tune their models against them. That might actually be good for o11y-bench — I’d want a lab to tune against it, so we get free fine-tuning out of the box: as newer models use o11y-bench to tune on observability tasks, they get better at using Grafana and solving observability problems.

Nicole: Thank you — we just hit time. It was really informative, and I’m glad we both found time; this got rescheduled for various reasons. A great overview of all the evals we’re doing.

Yas: Thanks — it’s nice to step out of the day-to-day and talk about this stuff.

Nicole: And if you’re watching, leave any eval questions in the comments and I’ll bug Yas to answer them. Thanks everyone for watching.

Yas: Bye.

OpenTelemetry's Graduation: What It Took (Grafana ❤️🔥 OpenTelemetry Community Call #8)

Mon, 22 Jun 2026 16:00:00 +0100

OpenTelemetry graduated from the CNCF — so for this community call I gathered a panel of people who were in the room for it to talk about what graduation actually means, what it took to get there, and where the project goes next. We covered the roughly two-year road to graduation, why semantic conventions are so genuinely hard, the state of the signals (traces, metrics, logs — and the newer profiles), AI’s double-edged impact on the project, and the most practical ways for new contributors to get involved.

What we covered

🎓 What graduation actually took

Graduation from the CNCF took about two years, with OTel evaluated on traces, metrics, and logs across the major server-side languages. Profiling, mobile, and other newer signals were explicitly out of scope for the graduation criteria. One of the most concrete pieces of CNCF feedback: stop camping on 0.x versions for components that are actually production-ready. The fix is to mark maintained, stable packages as 1.0 (reserving 2.0 for future breaking changes) — which unblocks the enterprises that simply aren’t allowed to ship anything still labeled “beta.”

📚 Documentation that’s actually findable

A recurring theme was that the documentation bar isn’t just “does it exist” — it’s “can someone find it.” Collector component docs are spread across 100+ GitHub repos, which makes discovery painful. The stopgap is an auto-updated component list with metadata on the OpenTelemetry website that links out to the relevant READMEs; the longer-term answer is the Ecosystem Explorer project, which aims to unify instrumentation details, versioning, and component metadata in one place.

🏷️ Why semantic conventions are so hard

Semantic conventions define normalized attribute names so that the same event — say, an HTTP request — looks identical no matter the language or library that produced it. The naming debates are slow for real reasons: the database SIG took over a year to stabilize a single set of attributes. The process is propose names → check them against all the major databases → build a proof-of-concept in three or more languages → iterate. A name that’s obvious for one database may not even exist as a concept in another. (OTel itself is the product of merging OpenTracing, OpenCensus, and Elastic Common Schema — and yes, the name was chosen specifically because it’s boring and self-describing. Everything is named what it does.)

🧳 Signals, baggage, and stability

All three original signals — traces, metrics, and logs — are now stable, and that includes baggage: a context-propagation mechanism that carries a small “bag” of attributes (trace ID, span ID, and more) across network hops. It was originally built for tracing but designed to be reusable for other cross-cutting concerns like security and feature flags, and it sat largely unused while the rest of OTel was being finished. Profiles, meanwhile, hit alpha at KubeCon in March and are treated as a bonus on top of the graduation baseline rather than part of it.

🤖 AI’s impact on OpenTelemetry — the good and the painful

The good: the explosion of GenAI and agentic systems is creating real demand for OTel-style observability. The Gen AI SIG is growing fast to standardize telemetry for AI systems, and there’s a whole crop of AI-observability startups building on OpenTelemetry. Maintainers are also using AI coding tools to move faster — the Weaver tool is being developed to constrain the problem space so AI-generated instrumentation code is more accurate, with the goal of going from spec to implementation with coding tools.

The painful: a massive influx of low-quality, AI-generated PRs across all of open source. Some are genuinely malicious (bot accounts farming credibility for supply-chain attacks); others are just thoughtless. One proposed mitigation is to cap the number of open PRs per contributor without merge rights to something like three to five.

🚀 Post-graduation priorities and how to get involved

The post-graduation backlog (“postgrad app”) focuses on stability cleanup, easier deployment, and self-observability. The deployment north star is something close to apt-get install opentelemetry that auto-instruments everything — including outside Kubernetes. New stable YAML-based collector config lets operators manage all languages from a single file, and the Blueprint project pairs a specific challenge with OTel guidelines and concrete implementation steps.

If you want in, the panel was refreshingly practical about the fastest path to approver/maintainer status:

Review other people’s PRs in your subject area.
Ask maintainers “what can I take off your plate?” rather than only filing your own issues.
Quality over quantity — AI-heavy contributions you can’t explain won’t earn status.
Localization is one of the easiest entry points; around ten language teams are active and docs translation is always welcome.

We also went around the panel on how everyone got started — from debugging giant distributed systems, to a beginner-questions live stream, to Elastic Common Schema work, to copy-editing PRs as a former librarian. The throughline: pick a small, real thing, do it well, and document your learning journey as you go.

Resources

Thanks to the panel — Ted Young, Marylia Gutierrez, Imma Valls, Tiffany Hrabusa, and Tiffany Jernigan — for joining, and to everyone who showed up live with questions.

CLI and MCP Basics for AI Agents (The Context Window #04)

Wed, 03 Jun 2026 18:00:00 +0100

MCP vs CLI: which one should your AI agent actually use? In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and three of the engineers and PMs behind Grafana’s agent tooling: Ben Sully (who wrote the Grafana MCP server), and Ward Bekker and Dafydd Thomas from the team building gcx. We get into the two main ways an AI agent can reach the outside world — the Model Context Protocol (MCP) and good old command-line interfaces — and look at when each one makes sense, the tradeoffs around context windows, tool sprawl, token cost, and reliability. We walk through Grafana’s cloud and open-source MCP servers along with gcx, the Grafana Cloud CLI built for agents, do a live side-by-side demo of both creating an SLO, and talk about how we benchmark them against each other with o11y-bench.

Timestamps

00:00:00 — Introductions
00:02:49 — The last month in AI news
00:10:19 — What is Grafana Assistant?
00:12:30 — What does it mean for Assistant to use tools?
00:13:30 — What is MCP?
00:16:56 — What is a CLI in an AI context?
00:19:17 — What are skills?
00:24:37 — Can you use a custom MCP server with Grafana Assistant?
00:25:55 — Why team MCP?
00:28:22 — The context-window cost of MCP tool definitions
00:31:50 — Tool discoverability
00:39:25 — What is gcx, and why a CLI?
00:46:13 — Demo: creating an SLO with MCP vs gcx
00:52:58 — Benchmarking MCP vs gcx with o11y-bench
00:58:10 — How MCP and gcx relate to Grafana Assistant
01:02:35 — k6 MCP vs Grafana MCP
01:03:00 — When to use which (decision guide)
01:04:29 — Adding the OSS Grafana MCP to Assistant
01:05:41 — Turtles all the way down

Resources

MCP and CLI for agents

My Context Horizon piece: Unlearnings from building Grafana Assistant

My AI note-taking workflows

Fri, 29 May 2026 17:00:00 +0100

Just over a year ago, if you’d asked me how I felt about AI and note-taking, I would have told you I didn’t want AI anywhere near my notes. The world is different now, and so am I — I rely on AI in a lot of ways, and one of the easiest places to start is speech-to-text: using AI to convert my voice into something my PKM system can actually work with.

In this video, I walk through three tiers of AI note-taking workflows — free, paid, and (a little) overkill — and show what I’m using at each level.

The framing: note-taking vs. note-making

Nick Milo draws a useful distinction between note-taking and note-making:

Note-taking is collecting — recording information, paraphrasing what an author said, highlighting.
Note-making is connecting — active thinking, engaging with the information, linking it to other ideas, working out how it fits into your PKM system.

The best use of AI right now, for me, is to help with the note-taking part. I like to start with speech because speech used to be a form of input my PKM system couldn’t take very much of. Pre-AI, voice was lossy — you’d have to transcribe it yourself or lose the thought. Now: you speak, AI transcribes it, it lands in your notes, and the AI can keep working with it from there.

Tier 1 — Free

Use your phone’s recording app. iOS Voice Memos transcribes recordings, and the Notes app does too on newer Apple devices. Samsung’s Voice Recorder app does the same on Galaxy devices. For desktop, I demo Spokenly (Mac, Windows, iOS) — drop audio in, get text out.

The friction is manual: remember to record, remember to transcribe, remember to paste.
What you get: quick notes to yourself, captured instantly.
Best for: people who want to get started as easily as possible without paying for anything.

If you want to go local-first, I’d point you at NVIDIA Parakeet and the Qwen models for offline transcription.

Tier 2 — Paid

The second tier is paying for the convenience layer. These tools collapse the first two steps — you record, the transcript appears, and you usually get extras like AI summaries or the ability to ask questions of the transcript (sometimes live).

Tools: Voicenotes, Otter.ai, Letterly, Granola, Gemini on Google Meet, Zoom AI Companion.

For Granola specifically, I use the Granola Sync Obsidian plugin plus a QuickAdd macro (QuickAdd: Macro - Link Granola) to automatically link a transcript to the right note in my vault — in the video I link a Dungeon Crawler Carl recap to its Granola transcript. (The linking script and macro are on my Patreon.)

Tradeoffs:

Your notes live in their system, not yours. Shutdowns, pricing changes, and acquisitions all take your archive with them.
The friction reduction is real, though — if it means you actually capture instead of intending to capture, that’s worth a lot.

Best for: people who want to capture meetings, games, or lectures.

Tier 3 — Overkill

This is my setup. It’s a little ridiculous; I’m going to show it anyway.

Hardware: a Plaud AI recorder. Small, clips to my phone or sits in a pocket. Always running, or started deliberately for important conversations and walks. It can run entirely locally, or upload to Plaud’s servers.
Plugin: my fork of Leonard Sellem’s Plaud Sync plugin for Obsidian, installed via BRAT. The fork adds a couple of things I needed for my own daily workflow.
AI layer: Iris, my OpenClaw-powered AI agent, watches the synced files. Every morning she reads the previous day’s transcripts, plus my calendar and daily notes, and outputs a daily summary, extracted tasks, and flags anything I’ve said I wanted to remember.

I also use “Hey Iris” memos — voice notes I record into Plaud while I’m out, addressed to Iris, that she processes the next time she runs. “Hey Iris, remind me to follow up with X.” “Hey Iris, add this restaurant to my places-to-try list.” They land in my system without me opening anything.

Benefits: I don’t think about capture at all. I just live and talk, and the relevant pieces land in my notes.

Cons: the privacy of people around me. I’m careful about when this is on, and especially careful about explicit consent.

Why three tiers, not one

The slightly absurd thing is that I still use all three levels — in parallel, for different things:

Spokenly for dictating at my desk, sometimes straight into Obsidian, sometimes into a chat.
Granola for meeting notes, and increasingly for transcribing videos I’m watching.
Voicenotes for diary-style entries.
Plaud when it’s ethical to (especially when I’m travelling with my partner and I want a clean record of conversations and ideas we had on the move).

For me, voice is the lowest-friction capture method that exists. No stopping, no finding my phone, no opening an app, no typing. Just talk — and the system handles everything downstream.

That means the gap between thinking of something and writing it down is basically gone for me. Note-taking — the tedious part — gets offloaded. So when I’m in a meeting I can be present, knowing the tool is faithfully tracking what’s said. Whatever notes I do take are then what I think about what I’m hearing, which lets me start synthesising a lot faster.

If you want more details about any of these workflows, let me know in the comments. Arigatō gozaimashita! Mata kondo ne~

Resources

🎙️ The Mechanics of Knowledge Management podcast episode where Malik interviewed me on this stuff
🧠 My notes on OpenClaw — the agent runtime Iris runs on
🎤 My notes on Voicenotes
💛 My Patreon, where I share the QuickAdd linking scripts and macros

Grafana Campfire 🔥 — GrafanaCON 2026 recap

Thu, 28 May 2026 17:45:00 +0100

I co-hosted this month’s Grafana Campfire with Stephanie Hingtgen, where we recapped everything that came out of GrafanaCON 2026. Stephanie owned the Grafana 13 walkthrough — Git Sync GA, Dynamic Dashboards GA, Dashboards API v2 GA, Grafana Advisor GA, SCIM GA — what she described as the year Grafana stopped being beta in production. I covered the AI section (Grafana Assistant now in OSS and Enterprise, the three scaling challenges from “Scaling Grafana Assistant”, Investigations, Skills going GA), the three-teams-one-architecture push across Loki Thor, Mimir 3.0, and Pyroscope 2.0, the OTel + Alloy roadmap, alerting, k6 v2.0, and the Marketplace and Logline.

We also took questions from the Champions community on the pace of UI change, Mimir Alertmanager, and the agent-scale question — what happens when your AI agents are themselves the noisiest users of your observability stack? — and threw in a flavour break about the TamaGROTchi and the ocean digital twin demo.

Resources

Asimov's Zeroth Law of Robotics: Observability for AI (ExpoQA Madrid 2026)

Tue, 26 May 2026 10:00:00 +0200

This week I gave the opening keynote at ExpoQA Madrid 2026 in Madrid, Spain. I presented a talk called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously given at KubeCon EU 2025 in London, at Dutch Cloud Native Day 2025 in Utrecht, and at Newcrafts in Paris. Here’s the abstract:

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

This was a keynote slot at a testing-focused audience, which gave me the chance to lean more into the evals vs. observability framing — what evals can and can’t tell you about an AI system in production, and how distributed tracing fills the gap that benchmarks leave behind. The demo app (a Flask-based “play.py” instrumented with OpenTelemetry, flowing into Tempo for traces, Prometheus for metrics, and Loki for logs) is still on GitHub, and I’ve been refining it with each iteration.

Here’s the recording of the talk:

Resources

Demo app on GitHub

Zero to Dashboard with Grafana Assistant and the Infinity datasource plugin

Fri, 22 May 2026 17:00:00 +0100

What if you didn’t have to write a single query to get a useful dashboard?

In this short video, I go from nothing to a working dashboard in a few minutes — no queries — using Grafana Assistant and the Infinity datasource plugin. The data I’m visualising is the RAWG video game database API, so I end up with a dashboard that helps me decide what to play next.

Why Infinity is fun here

Most Grafana data sources are scoped to a specific kind of backend — Prometheus, Loki, a SQL database. Infinity is different: it lets you treat any REST/JSON/CSV/HTML endpoint as a data source. That means anything with an API can become a dashboard, which is exactly the right shape for goofing around with a games API on a Friday afternoon.

Pair Infinity with Assistant and you don’t even have to know what queries to write — Assistant configures the data source, asks the API the right questions, and builds the panels for you.

Timestamps

00:00 — Intro
00:35 — The RAWG video game database
01:01 — Setting up the Infinity datasource
05:10 — Customising the dashboard with Assistant
09:31 — Closing thoughts

Resources

Step-by-step blog post: using Assistant with the Infinity datasource
Grafana Assistant documentation
Infinity datasource plugin
RAWG video game database API
Get started with Grafana Cloud (free tier)
My notes on Grafana Assistant — everything I’m learning as I work on it

Do Androids Dream of Second Brains?

Mon, 18 May 2026 07:00:00 +0100

We often describe our personal knowledge management systems as “second brains” — but can we really say they think for themselves? In this video, I make the case for instrumenting your vault with an observability stack and layering AI on top, not because it’ll turn your notes into a brain, but because it might get us closer than anything else has.

This is the video version of the talk I gave at PKM Summit 2026 in Utrecht earlier this year, with the live demos cleaner and the dashboards working without the conference wifi gods getting in the way.

What I cover

🐑 Why “second brain” is the wrong framing

I take a small detour through Philip K. Dick’s Do Androids Dream of Electric Sheep? — the inspiration for Blade Runner — and the Voigt-Kampff test, which measures empathy through observed responses to staged scenarios. It’s a useful lens for asking what we actually mean when we say a knowledge system “thinks”: we’re really asking whether it can respond to us in a way that surprises us. Most second-brain setups can’t, yet.

📊 Observability for PKM

Observability is a concept I’ve borrowed wholesale from how we monitor software systems — the idea that you can understand a complex system by what it emits, not by reading its internals. Three steps: instrumentation, analysis, improvement. I walk through what each one looks like for a personal vault:

Instrumentation: a small script that emits metrics about my Obsidian vault — note counts, link density, growth over time, what folders are loud and which are quiet
Analysis: a Grafana dashboard sitting on top of Loki + Prometheus, surfacing things I genuinely didn’t know about my own vault
Improvement: the part where seeing the data actually changes how you work

There’s a live dashboard tour, and a section where I ask Grafana Assistant to edit the dashboard for me in natural language — including writing the PromQL queries I never want to write by hand.

👁️ Meet Iris

About halfway through the video I introduce Iris — the AI agent I built using OpenClaw that lives in my vault, reads my notes, and works alongside me. I walk through:

What “agentic AI” actually means in practice (vs. just chatting with an LLM)
The OpenClaw setup — Docker sandbox, Syncthing, the whole pipeline
Iris rewriting my observability map of content
Iris doing talk-topic analysis on my vault
Iris running benchmark research for a blog post
Iris OCRing my reMarkable handwriting

🌙 Dreaming

The last section is the one I had the most fun with. I argue that the thing most second-brain setups are missing isn’t more notes or better search — it’s synthesis. The capacity to take everything you’ve captured and surface patterns, contradictions, and ideas you hadn’t formed yet.

I introduce OpenClaw’s dreaming feature: a memory consolidation process inspired by how sleep stages work in the brain. Light sleep, REM, and deep sleep each do different work — surfacing recent material, finding unexpected connections, and consolidating long-term memory. I show how each phase actually behaves in my own system.

And there’s a Zhuangzi reference. (You’ll see.)

Resources

🎤 PKM Summit 2026 talk recap (with photos)
🎞 Slides
👁️ OpenClaw
📓 Obsidian
🔄 Syncthing
📊 Grafana

The Architecture of Forgetting — the essay version of why memory consolidation matters
Unlearnings from building Grafana Assistant

If you want to follow more of how Iris works and what I’m learning about AI + PKM, Context Horizon is where I write about it.

Continuous Profiling, the OpenTelemetry Way (Grafana ❤️🔥 OpenTelemetry Community Call #7)

Mon, 18 May 2026 05:00:00 +0100

In this community call, my coworker Tiffany Jernigan and I sat down with Christian Simon, who works on Grafana Pyroscope, to dig into profiling as the fourth signal in OpenTelemetry. Metrics, logs, and traces are the classic three pillars — but a significant portion of latency lives inside a span, in the actual code execution, where the other three can’t see. Profiling fills that gap by giving you continuous, code-level visibility into where time is being spent.

What we covered

🔥 Profiling as the fourth signal

Most observability conversations stop at logs, metrics, and traces. But when you see a slow span and ask “what was the code actually doing?” — none of the three pillars can answer. Christian made the case that profiling is the missing piece: continuous, low-overhead, code-level visibility, typically visualized as flame graphs that attribute latency to specific functions and execution paths.

📊 What flame graphs actually tell you

If you haven’t worked with flame graphs before: wide bars = lots of time spent in that function, and the stack reads bottom-up. Once you understand how to read them, they become surprisingly fast to interpret. We walked through what to look for and how to spot the patterns that matter.

🌐 The current state of profiling in OpenTelemetry

Profiling is the newest signal to enter the OpenTelemetry spec, and the conversation around it is still evolving. Christian gave a clear-eyed status update on where things are — what’s stable, what’s experimental, and what the path to GA looks like.

🚀 Running profiling in production

The most common pushback on continuous profiling is “what’s the overhead?” — and the honest answer is that modern continuous profilers typically add 1–3% CPU overhead, which makes them safe to run in production all the time. We talked through deployment patterns, including Kubernetes / containerized environments, and how to think about cost and storage.

🧵 Correlating profiles with traces and metrics

This is where profiling really earns its place. Trace ID → profile is the killer pattern: you see a slow span, you click through to the flame graph for that exact execution, and you immediately see which function ate the time. Christian showed how this correlation works in practice and how it changes incident response.

🤝 How to get involved

OpenTelemetry profiling is genuinely an area where the community has room to shape the future. If the conversation sparked something, there’s a working group, open issues, and a friendly path in.

Resources

Thanks to Christian for joining, to Tiffany for co-hosting, and to everyone who showed up live with questions.

The Architecture of Forgetting

Thu, 07 May 2026 16:00:00 +0100

Originally posted here on Context Horizon.

In early April, a new version of everyone’s favourite lobster, OpenClaw, was shipped with an interesting new feature: dreaming. It’s a memory consolidation feature. But to understand why it matters, we need to back up.

Early AI chatbots, like the initial GPT wrappers, had zero memory. Every conversation started fresh. You asked a question, you got an answer, and you moved on: a glorified search engine.

Somewhere along the way we began to expect more and more of our chatbots. Models acquired context windows that became their de facto short-term memory. As the context window filled up, they remembered what had been said earlier in the conversation, making them better conversational partners. Still, it was ephemeral: close the tab, start fresh. And it had a hard ceiling: hit the limit, and the oldest things fall off. This felt like forgetting because it was. But it was also, accidentally, a form of curation. Finite space forced priority.

Then came the push for real persistence. RAG pipelines, vector databases, memory files, agent frameworks that write notes to themselves. Now our agents can remember that we prefer the Oxford comma, that we just started a new project at work, that our D&D character is named Zuhri. This felt like the solution. More memory = better agent.

That mostly works— until it doesn’t.

As it turns out, the problem with AI memory isn’t that agents forget too much. It’s that they forget too little.

Why remembering is hard

It turns out that having a persistent memory, even for an agent like OpenClaw that has that as one of its defining features, doesn’t solve all the memory problems.

I happen to have an OpenClaw agent that named itself Iris.

Iris does a lot of things for me, including parsing my meeting notes to identify next actions, curating my tasks and projects, and generating a daily briefing. She also creates a summary of the day’s work based on my meetings, conversations, and notes. One day she was giving me the daily summary, and here’s that conversation:

Iris: …Tonsil is finally healed after weeks of pain… Me: Wait what? Where did you get the tonsil stuff? Iris: From your memory file — it’s in the Apr 10 daily note context: “Tonsil finally healed — Nicole no longer in pain 🎉”. I’ve had that sitting in memory since the Apr 10 summary. Is that wrong? Was it something else that healed, or is the tonsil detail off? Me: I have no idea where you got that. I have never had issues with my tonsil.

Where did that come from? I’m still not sure, but I occasionally have meetings in cafés, and it’s possible that a meeting recording overheard the conversation of someone else in the café. That got faithfully transcribed, copied over into my notes, and written into Iris’s memory. Iris treated it as fact for days.

The problem isn’t that Iris forgot. It’s that she remembered the wrong thing… and didn’t know how to forget it.

Imagine this at scale: not one wrong fact, but dozens, quietly shaping every summary, every recommendation, every task Iris surfaces for me. The signal-to-noise ratio degrades until it doesn’t matter if Iris remembers mostly accurate memories— I won’t trust it.

So how do you build a memory system that knows what to keep? Researchers have been working on this longer than you might think.

Meet the residents of Smallville

Stanford researchers Park et al. released a charming study in 2023 called Generative Agents: Interactive Simulacra of Human Behavior in which they talk about how they created a simulated environment called Smallville, populated it with 25 generative agents, and then watched.

Source: Park et al. (2023)

The study was an attempt to simulate emergent social behavior, but the key insight I took from it isn’t the sim itself— it’s what they had to do to the inhabitants of Smallville to enable the rich social interactions they wanted to observe.

The researchers had to give the inhabitants a way to remember.

Above is a diagram of the memory architecture of every generative agent.

Observation: The agent perceives new information. The new information is saved into the memory stream, a database that serves as the record of everything an agent has experienced.
Retrieval: When the agent is looking for information, a retrieval function takes that information from the memory stream and passes it onto the agent.
Action: The agent, armed with retrieved information, can decide to act.
Reflection: Patterns are synthesized from the observations and written back to the memory stream.

To determine which memories to retrieve at any given point in time, the researchers scored memories according to three weighted factors:

Recency: when a memory was observed
Importance: how significant the memory was
Relevance: how closely related a memory is to the input memory (the agent’s current context)

The top-ranking memories that fit within the agent’s context window were selected to be retrieved and used with the prompt.

The researchers also introduced a second type of memory: reflection. Reflections were generated when a threshold of important scores was reached per day, which typically happened 2-3 times a day.

Crucially, reflection is a compression step — the agent doesn’t just accumulate observations, it periodically steps back and asks “what do these observations actually mean?” That reflection pass is where low-signal memories get subordinated to higher-order patterns. That’s the first instance of intelligent forgetting.

The Smallville paper was published in 2023. Three years later, the same insight shipped in a lobster.

Dreaming of electric lobsters

Every system that has solved the memory problem has solved it the same way: not by storing more, but by knowing what to throw away.

Your brain does it while you sleep by consolidating important experiences into long-term memory, letting the rest dissolve. Some programming languages do it with garbage collection, periodically scanning for unused objects and reclaiming that space. Databases do it with compaction: Loki, Cassandra, and other LSM-tree-inspired systems periodically merge and discard obsolete data so reads stay fast.

In all cases, the architecture of good memory is really the architecture of intelligent forgetting. OpenClaw’s dreaming feature is the same insight applied to agents.

Diagram of OpenClaw’s internal memory architecture. The default components (“Awake”) are in black, and the new components are in blue (“Dreaming”).

Note that without dreaming, there is no automatic promotion from the active context window to daily notes and then to long-term memory. When a new bit of information is mentioned, the agent has to decide whether it should be added to the daily note and/or the long-term memory file manually. Dreaming, by contrast, is an automated process.

It’s called “Dreaming” because the feature works similarly to the biological process, as if in three sleep phases:

Light phase: Creates a session corpus from session files and then succeeding sweeps tally how many times a chunk has been recalled in a session (a counter called lightHit).
REM phase: Spots patterns and themes from chunks and adds a reinforcement signal (remHits).
Deep phase: Ranks the candidates by several factors and promotes those that exceed a threshold score to long-term memory in MEMORY.md

Just like in the Smallville simulation, OpenClaw ranks information by several categories. Here’s the table of signals and weights from the OpenClaw docs at the time of writing:

Signal	Weight	Description
Frequency	0.24	How many short-term signals the entry accumulated
Relevance	0.30	Average retrieval quality for the entry
Query diversity	0.15	Distinct query/day contexts that surfaced it
Recency	0.15	Time-decayed freshness score
Consolidation	0.10	Multi-day recurrence strength
Conceptual richness	0.06	Concept-tag density from snippet/path

Only candidates that have high scores according to these signals graduate to be part of MEMORY.md.

How dreaming helps with forgetting

The new dreaming architecture doesn’t ever delete things, technically. The forgetting is more passive. However, it’s still an improvement in a few ways:

Low-signal chunks don’t get promoted: Anything that doesn’t cross the deep phase thresholds simply never reaches MEMORY.md. It exists, but it’s functionally invisible to future sessions.
Score competition: The staging index has finite attention, so chunks compete against each other for recall signal. A high-frequency chunk that keeps getting surfaced crowds out low-frequency ones. Relative obscurity is functional forgetting.
Recency decay on scoring: The deep phase ranking includes a recency signal. So something that was frequently recalled two weeks ago but hasn’t come up since will score lower than something with fewer total hits but recent activity. Time degrades relevance passively.

What’s notable: all three are selection mechanisms, not deletion mechanisms. The corpus files still exist. The staging index entries still exist. Nothing is removed… it’s just progressively less likely to be chosen. This is forgetting as diminishing probability of retrieval, not erasure.

That’s the garbage collection analogy again: the objects are still in memory, they’re just unreachable.

Transparency

Dreaming also adds a layer of transparency. Aside from the session corpus and staging files, it also produces some output about its decisions. All three dreaming stages create files per day that live in .openclaw/workspace/memory/dreaming/{phase}/YYYY-MM-DD.md.

Here’s an example of an entry in the light directory:

# Light Sleep
- Candidate: "Don't Call It Eval" blog post (Context Horizon): `Dont Call It Eval - Blog Post Outline.md` — full outline with 7 open questions; `Dont Call It Eval - Draft.md` — 2,550-word draft, 6 sections; `Dont Call It Eval - Opening Research.md` — Tier 1/2/3 shortlist of popular AI eval po
- confidence: 0.62
- evidence: memory/2026-04-19.md:13-15
- recalls: 0
- status: staged
- Candidate: "Don't Call It Eval" blog post (Context Horizon): **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since."
- confidence: 0.62
- evidence: memory/2026-04-19.md:17-17
- recalls: 0
- status: staged

This entry identified a blog draft I was working on.

REM sleep goes through these candidates and highlights when it spots themes:

# REM Sleep
### Reflections
- Theme: `assistant` kept surfacing across 532 memories.
- confidence: 1.00
- evidence: memory/2026-04-12.md:27-29, memory/.dreams/session-corpus/2026-04-17.txt:26-26, memory/.dreams/session-corpus/2026-04-17.txt:28-28
- note: reflection
### Possible Lasting Truths
- "Don't Call It Eval" blog post (Context Horizon): **Datadog Bits AI eval platform** (Apr 7, 2026) — "Bring the noise" = noise injection = production-like envs from 90s QA. SECTION HEADERS are gold ("When one improvement caused subtle regressions", "Why tool-level testing and live [confidence=0.58 evidence=memory/2026-04-19.md:30-33]
- "Don't Call It Eval" blog post (Context Horizon): **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since." [confidence=0.58 evidence=memory/2026-04-19.md:17-17]
- "Don't Call It Eval" blog post (Context Horizon): **Title:** "Don't Call It Eval" (Nicole's original; I picked it over "Congratulations, you've invented testing") [confidence=0.58 evidence=memory/2026-04-19.md:27-27]

In this case, REM sleep correctly identified that I’m currently working on the Assistant feature at work (which had already been promoted to long-term memory) and the blog post that the light sleep phase had identified.

Here’s the same day’s deep sleep file:

# Deep Sleep
- Ranked 10 candidate(s) for durable promotion.
- Promoted 3 candidate(s) into MEMORY.md.

And here’s the addition to MEMORY.md for the same day:

## Promoted From Short-Term Memory (2026-04-26)
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:17:17 -->
- **Thesis (locked):** "AI eval community has rediscovered software testing and given it a new name. The testing world has been busy since." [score=0.926 recalls=0 avg=0.620 source=memory/2026-04-19.md:17-17]
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:27:27 -->
- **Title:** "Don't Call It Eval" (Nicole's original; I picked it over "Congratulations, you've invented testing") [score=0.926 recalls=0 avg=0.620 source=memory/2026-04-19.md:27-27]
<!-- openclaw-memory-promotion:memory:memory/2026-04-19.md:12:12 -->
- Three files in openclaw-output: [score=0.894 recalls=0 avg=0.620 source=memory/2026-04-19.md:12-12]

There’s also an entry in DREAMS.md for the entire sequence, which is more dreamlike and human-readable. The per-day Deep Sleep file is the structured internal audit trail; DREAMS.md is the rolling, more narrative log for humans.

This level of transparency and reflection is a welcome change when you compare it to what I had before, which was nothing. Iris had to manually decide on the spot what had to be written where, and those decisions were not logged anywhere.

The tonsil dilemma

So dreaming could have prevented the tonsil hallucination, right? Well… no.

In this case, the idea of my supposed tonsil pain had come from an external source: my meeting recordings. Iris had faithfully manually added that to the daily note for the day, then decided to call on it a few days later, unprompted, in conversation with me.

With dreaming, Iris still could have made those decisions. But in fact, even if she hadn’t, dreaming could have made it worse:

Light phase sweeps it and scores it well since it’s recent and distinctive.
Tonsil pain could be included in queries about health, such as the daily health tracker that is part of Iris’s tasks. REM phase sees a pattern and increments remHits.
Deep phase ranks it highly and it crosses the promotion threshold.
It gets written to MEMORY.md and gets loaded every session. And long-term memory is sticky. Only manual intervention (by me or by Iris) removes things from that file.

Even dreaming doesn’t solve everything. Dreaming adds a mechanism for forgetting. It doesn’t prevent OpenClaw from remembering the wrong things in the first place.

The residual layer

Every memory architecture we’ve looked at — biological sleep consolidation, garbage collection, LSM compaction, Smallville’s reflection pass, OpenClaw’s dreaming — has the same shape: automated compression plus a residual human layer. Your brain has you. GC has the programmer. Iris’s dreaming has me. The human in the loop isn’t a bug in the design; it’s load-bearing.

What’s worth keeping

Dreaming is a genuine step forward for OpenClaw’s memory architecture. The mechanism of passive forgetting isn’t perfect, but the automated promotion systems do add a necessary layer of periodic reflection.

But there are limitations. Dreaming can’t yet distinguish between a memory worth keeping and a hallucinated tonsil. It can only measure how often something gets recalled. The next frontier is active forgetting: automated demotion from MEMORY.md, weighted by signals like source provenance and trust. The provenance side of that work landed in v2026.4.29-beta.2 — memory now exposes evidence-kind drilldown and privacy/provenance reports, so you can at least see where a memory came from. But the demotion half — automatically down-ranking memories based on a low-trust source like an audio transcript — still hasn’t been built.

What remains, for now, is the human layer. The transparency of the dreaming process becomes a checkpoint for the human in the loop by surfacing promotion decisions. I still read the summaries. I still catch the errors. I still decide what goes into long-term memory and what gets quietly corrected. That’s not a gap in the system — it’s probably the right design. The most important memories should require a human to notice them twice.

Good memory systems are defined by what they don’t keep as much as by what they do keep.

Grafana k6 Community Call — Secrets Management & AI Script Authoring

Thu, 30 Apr 2026 16:30:00 +0100

The first k6 Community Call in over a year, and we packed it. I sat down with four of the people building k6 — Bukola Ayodele, Facundo Batista (Software Engineer), Vicente Ortega (Senior Backend Engineer), and Charlie Burnett (Product Manager) — to walk through two new pieces of work that have shipped recently, plus a look at what’s coming in k6 v2.

What we covered

🔐 Secrets management in k6 Cloud (Facundo)

A new way to store API tokens, passwords, and other sensitive values for your tests — centralized in Grafana’s secrets manager, encrypted at rest, and never visible in the UI or API once written. Your scripts reference secrets by name, and at runtime the values are injected and automatically redacted from logs.

Why this matters for load testers:

No more hardcoded credentials sitting in scripts or in Git
Easy rotation — change the secret value, every script that uses it picks up the new value, no script edits needed
Same script works across dev/staging/prod by just swapping which secrets are scoped to which environment
Works with k6 OSS locally too — your local runs can pull cloud-managed secrets when you want them

Facundo Batista demoed the full flow: creating a secret in the Grafana Cloud UI (name, description, labels — up to 10 per secret, all write-only), and using it in a script with one line. There’s also an API for programmatic management.

🤖 Grafana Assistant k6 script authoring mode (Vicente)

This is the AI-powered script generation that’s been in the works for a while — fine-tuned specifically for k6 best practices, not a generic “ask an LLM to write you a load test” workflow. The interesting bit is the context sources Assistant can pull from when generating a script:

A service name — Assistant analyzes the existing telemetry data to discover endpoints and shapes
A GitHub repository URL — pulls the code to understand the API
An OpenAPI specification — generates tests that exercise every endpoint with realistic request shapes
Existing observability data — metrics, logs, traces from your actual production traffic

Generated scripts come with proper structure, error handling, checks, and TODO comments where the model genuinely doesn’t know what to write (e.g. for auth tokens, request bodies, or anything that needs your secrets — by design, Assistant won’t fabricate those). You can also point it at an existing script and ask it to refactor or extend.

🚀 k6 v2 (Charlie)

Release candidate 1 is out, GA is very soon. The headline changes:

Extensions marketplace — community protocols and integrations
Enhanced Playwright browser support with compatible syntax
New assertions library
MCP / agent connectivity for AI-driven workflows
DNS, MQTT, and other network protocols

Migration advice from Charlie Burnett: pin your script versions, test your existing scripts against the RC before GA hits, and don’t rush.

🔮 What’s next on the AI roadmap (next 1–2 months)

Visual browser-based testing experience
Enhanced observability workflow automation
Improved user-journey creation across both personal and enterprise apps
Integration with autonomous monitoring systems

Resources

Thanks to Bukola, Facundo, Vicente, and Charlie for joining, and to everyone who showed up live.

High INT, Low WIS: What LLM Benchmarks Miss

Thu, 23 Apr 2026 09:00:00 +0100

Originally posted here on Context Horizon.

In the tabletop roleplaying game Dungeons & Dragons (D&D), the very first thing you do when you create a character is to choose ability scores. Ability scores are numbers that determine how good you are at the game’s main characteristics: Strength (STR), Dexterity (DEX), Constitution (CON), Intelligence (INT), Wisdom (WIS), and Charisma (CHA). 20 is the default maximum that a character can have in any ability, though some in-game events/items can increase that.

Here’s the thing, though: every D&D player knows that a 20 Intelligence doesn’t mean your wizard aces every check. Whether you have proficiency in the specific skill still matters. That 20 INT wizard, for all their raw cognitive horsepower, can absolutely still fail to identify a magical rune. The ability score is a starting point, not a verdict.

This is exactly what’s happening to LLM benchmarks. All the general LLM/AI model benchmarks are ability scores. They tell you something real about a model’s general capabilities, the same way a 20 INT tells you something real about a wizard. But as frontier models crowd toward the top of every leaderboard, we’re running into a problem D&D solved decades ago: the stat block isn’t the character. And it was never supposed to be.

A screenshot of the stats of my Level 10 bard. Low INT, but look at that CHA!

The stat block: A tour of the ability scores everyone’s maxing out

There are so many new models and model versions out there that it’s hard to keep up with what we’re supposed to be using for which tasks. LLM benchmarks have come to be relied on to give a somewhat objective assessment of each model’s capability. Here are a few big ones that exist.

General knowledge/reasoning

MMLU (Measuring Massive Multitask Language Understanding, 2020)

MMLU is the classic “general knowledge” INT check. It was released in September 2020 by Hendrycks et al. and contains 15,908 multiple-choice questions across 57 subjects (humanities, social science, STEM, other). However, there were a few issues with it:

a study in 2024 showed that 6.49% of those questions contained errors
The New York Times reported (2024) that data contamination was an issue — models could be trained on the MMLU questions (and answers) to inflate their MMLU scores
many frontier LLMs soon consistently achieved high scores

In response, a new benchmark called MMLU-Pro was created with 12,000 questions that each have 10 options. The increase in options theoretically required more reasoning capability than the original benchmark, but frontier models are already knocking at 90% on the MMLU-Pro.

Humanity’s Last Exam (HLE, 2025)

HLE was developed specifically to address the fact that LLMs “now achieve over 90% accuracy on popular benchmarks like MMLU” (from the HLE site). The paper by Phan et al explains the lengths they took to make it hard enough to be meaningful:

they solicited high-quality questions from nearly 1000 subject matter experts, even going as far as paying them for successful submissions (with a total prize pool of US $500,000)
they included graduate-level classics (one of the questions includes translating Palmyrene script)
they reviewed potential submissions using current LLMs and only proceeded with submissions if the LLMs are unable to answer correctly
experts then reviewed the submissions further

As a result, HLE scores are low — intentionally so:

While current LLMs achieve very low accuracy on HLE, recent history shows benchmarks are quickly saturated – with models dramatically progressing from near-zero to near-perfect performance in a short timeframe [13, 45]. Given the rapid pace of AI development, it is plausible that models could exceed 50% accuracy on HLE by the end of 2025. (Phan et al., 2025)

Software engineering

HumanEval (OpenAI, 2021)

In 2021, OpenAI released a paper and an evaluation harness for a benchmark consisting of 164 Python programming problems based on publicly available code on GitHub. It was developed to test coding in particular.

Eventually, though, experts found that the problem sets in HumanEval were insufficient and ambiguous (Liu et al., 2023). HumanEval+ was released to include more tests, but the next benchmark’s authors suggested that models might be overfitting to HumanEval+ as well, memorizing the dataset and answers rather than genuinely demonstrating reasoning ability.

LiveCodeBench (2024)

Jain et al. tried a different approach to HumanEval to avoid data contamination: it was released with a continuously-updated coding benchmark. Problems have release dates and models can only be evaluated using problems released after their training data cutoff dates.

LiveCodeBench represents a shift in benchmarks towards a more continuous, dynamic measure. However, the last submission to the benchmark was 9 months ago at the time of this writing, and the leaderboard has not been updated since then either, suggesting the benchmark is no longer being actively maintained.

SWE-bench (2024)

This benchmark was created by Jimenez et al. to include 2,294 actual GitHub issues from 12 Python projects. When it launched, it was a much harder benchmark than predecessors, with models scoring 2-4%. Today, models are scoring around 70-80% on the original benchmark. There were enough issues with the original’s dataset quality that variants were introduced, including OpenAI’s SWE-bench Verified. Today, the SWE-bench website shows these variants:

SWE-bench
SWE-bench Verified
SWE-bench Multilingual
SWE-bench Multimodal
SWE-bench Lite

Agents and tool use

This last category is a preview of where I think the field should go: benchmarks that measure what a model does rather than just what a model knows. Here are some of them:

GAIA (Mialon et al., 2023) — 466 real-world assistant tasks that combine web browsing, file reading, and multi-step reasoning. Easy for humans (92% success), hard for models (15% success for GPT-4), at least at launch.
Terminal-Bench — agents working in actual Linux terminals. Tasks include “build the Linux kernel from source” and “configure a git server plus a webserver on port 8080.” No mocks; if it doesn’t actually build, it fails.
WebArena (Zhou et al., 2023) — realistic simulated websites (Reddit, GitLab, e-commerce) where agents complete actual user goals like “filter the last six months of orders by refund status.”

These benchmarks are harder to saturate than general-knowledge ones, because passing means the thing actually has to work in the real world. A model can memorise a question; it can’t memorise a Linux kernel that isn’t there until it builds it.

The wrong answer: Build a harder general benchmark

By now, you probably see that benchmarks tend to go the same way: A benchmark is created, models get too good at it and perform too well to be useful, and then a variant benchmark is created to address the problem.

This is the saturation problem in a nutshell: a benchmark that once provided meaningful information to help distinguish models eventually fails to do so because all the models’ scores cluster at the top.

Saturation occurs for two reasons:

Contamination: Many benchmarks publish their problem sets online, and models train on data scraped from the web. The benchmark now measures memorization, not capability.
Overfitting: Model providers are incentivized to optimize specifically for gaming performance on benchmarks. Scores go up without corresponding capability improvements.

When you have a party of wizards with 20 INT, “What’s your INT score?” stops being an interesting question because any wizard’s Fireball will hit the same. Building harder benchmarks is like an arms race for ability scores, when what you actually want to know is whether your wizard can kill a dragon.

Harder benchmarks don’t tell us what we want to know: can this model do my specific job?

The right answer: skill checks with real DCs

The fix isn’t a harder ability check. It’s a different kind of check altogether. In D&D, straight ability checks are rare compared to skill checks. There are 18 official skills that break down the six ability scores into more granular activities. For example, your wizard is much more likely to roll one of these instead of a straight INT check:

Arcana
History
Investigation
Nature
Religion

All of these are still based on INT, but test narrow types of intelligence rather than general intelligence.

Specific, domain-grounded benchmarks are what the field actually needs. The evaluations that actually help practitioners share a structure: they test a specific skill in a specific context, and they’re judged by the people who have to live with the results.

MedQA tests whether a model can answer the kind of clinical question a physician might face. LegalBench tests whether a model can correctly identify enforceability clauses, classify contract language, and apply legal reasoning. FinanceBench tests whether a model can answer the kind of question a financial analyst would actually ask about a real 10-K filing. None of these are trying to rank models on general intelligence. They’re asking the only question that matters downstream: can this model do this specific job well enough?

This week, Grafana released o11y-bench, an open benchmark for observability-specific agent tasks. I’m biased — I work there — but it’s a useful illustration of the shape I’m describing. o11y-bench doesn’t ask whether a model is smart in general. It asks whether an agent can do the things observability engineers actually do: read production traces, correlate logs with metrics, figure out why the thing that’s supposed to be working isn’t. Its leaderboard shows models’ performance for each activity.

Does that mean that more specific benchmarks won’t get saturated? No. We’ve already seen this play out with SWE-bench. But they’ll get saturated more slowly and remain more meaningful measures along the way.

These are skill checks with DCs that matter to someone. They’re not trying to produce a single scalar ranking of “which model is smartest.” They’re asking whether the wizard recognizes the family crest belonging to a political faction or knows this specific cult’s practices, not whether the wizard is good at “wizarding” in the abstract.

Roll for initiative: What this means in practice

If you’re an engineer picking a model for production: stop picking by leaderboard. Find or build an eval that reflects your actual use case. A handful of representative examples, graded the way you’d grade them if you were doing the work yourself, will tell you more than a dozen public benchmarks.

If you’re a researcher thinking about what to build next: the interesting work is at the specific-and-grounded end of the spectrum, not the general-and-bigger end. Another harder general-knowledge benchmark will saturate in eighteen months. A well-designed domain benchmark, co-built with the people who actually do the work, can stay useful for years.

And if you’re anyone else watching this field from the outside: be skeptical of the headlines. When a new model scores 94% on a benchmark you’ve never heard of, ask what it measures and whether the thing it measures is the thing you actually care about. Usually the answer is “not really,” and that’s the most important thing a benchmark score won’t tell you.

The goal of a character sheet was never to produce one number that ranks the character. It was to give a DM enough information to run a meaningful encounter — and to give the player enough to know when to try, when to pass, and when to roll with advantage. LLM evaluation should work the same way. Not a ranking; a readiness check. Can this specific model clear this specific challenge? Roll for initiative.

All About AI Observability (The Context Window #03)

Wed, 22 Apr 2026 18:00:00 +0100

In this episode of The Context Window — Grafana’s livestream series on AI in observability — I’m joined by Tiffany Jernigan and the engineers behind the product, Alexander Sniffin and Jack Gordley, for a deep dive into AI Observability in Grafana Cloud. We unpack what it actually is (a new way to instrument your AI apps and see canonical data about them, sitting alongside your existing telemetry), what an evaluation is in this context (and the difference between online evals on live traffic and offline evals on dataset conversations), and we walk through real demos: setting up AI Observability and evaluators, the analytics view, instrumenting a local coding agent, and Jack’s system prompt analysis tool. We also talk about the things nobody else is talking about — like LLM-as-judge as a method, evaluators that catch the most bugs, and the cheating-LLM phenomenon.

Timestamps

00:00:00 — Introductions
00:02:00 — The last month in AI news
00:11:09 — What is AI Observability in Grafana Cloud?
00:19:20 — What is an evaluation?
00:21:05 — The origin of AI Observability
00:25:09 — Demo: Setting up AI Observability and evaluators
00:32:04 — What is LLM as judge?
00:38:43 — Demo: AI Observability Analytics
00:40:52 — AI O11y is based on OpenTelemetry
00:42:17 — Demo: Instrumenting a local coding agent
00:47:18 — Potential future agentic use cases
00:52:00 — Evaluators that catch the most bugs
01:02:23 — Demo: System prompt analysis
01:05:11 — Guess the prompt

Resources

News from the episode

AI Observability

Mentioned

OpenCode — the open-source coding agent Jack instruments
Anthropic: Emotion concepts and their function in a large language model
OpenClaw

How to use Grafana Assistant with self-hosted Grafana

Mon, 20 Apr 2026 09:00:00 +0100

In case you missed it, possibly the biggest news to come from GrafanaCON this year is that if you’re running self-hosted or open-source Grafana, you can now connect it through your Grafana Cloud account to Grafana Assistant, the purpose-built AI tool for all things Grafana.

In this short tutorial, I show you how to enable Assistant on a self-hosted Grafana instance and go from nothing to a working server-health dashboard with a single prompt.

A quick note on what “for open source” actually means

Assistant itself isn’t open source. But you can now use it with your self-hosted or open-source Grafana instance. The way it works is:

You still need a Grafana Cloud account (the free tier has generous inclusions for Assistant).
You link your self-hosted Grafana to that Cloud account.
Assistant then lights up inside your OSS instance.

Think of the Grafana Cloud account as the key that unlocks Assistant wherever your Grafana lives.

Prerequisites

Grafana OSS v13 or later
A Grafana Cloud account (pricing here)

Enabling Assistant

Go to Administration → Plugins, search for Assistant, and click Install.
Refresh the page, then open the Connection tab.
Click Start Connection. You’ll be prompted to log into Grafana Cloud.
Select the organization and stack you want, then click Authorize connection.
Close the tab. Your self-hosted Grafana is now linked to Assistant.

A first dashboard from nothing

Every Grafana instance ships with a TestData data source — perfect for a first run when you don’t yet have real data wired up. Enable it from Data Sources, then open Assistant (the new icon on the top right).

Switch Assistant’s mode from General to Dashboarding — this mode is tuned for creating and managing dashboards — and ask in plain English, for example:

Hi Assistant, could you create a server health dashboard for me using the TestData data source?

What comes back is a starting dashboard with CPU, memory, disk, and network panels already wired up. From there you can tweak panels, edit queries, or ask Assistant to iterate. You’re not starting from scratch — you’re starting from 80%.

What else Assistant can do

I only showed dashboarding in this video, but Assistant can also:

Answer questions about the data already in your Grafana
Run investigations to troubleshoot production incidents
Learn new skills
Manage alerts
Teach you Grafana

That list keeps growing, so stay tuned.

Resources

Grafana Assistant (product page)
Grafana Cloud pricing
How Grafana Labs handles your data
My notes on Grafana Assistant — everything I’m learning as I work on it
Unlearnings from building Grafana Assistant — my Context Horizon piece on what the team had to unlearn

Unlearnings from building Grafana Assistant

Thu, 09 Apr 2026 09:00:00 +0100

Originally posted here

Everyone’s talking about what you need to learn to build AI agents: Prompt engineering. Tool calling. Retrieval strategies. The content ecosystem is full of it. But while I was watching Context Window Episode 1, it struck me that Mat Ryer, Cyril Tovena, and Sven Großmann– some of the team behind Grafana Assistant– spent most of their time on something different: what they had to unlearn.

Grafana Assistant started as a weekend hackathon project. By the time it shipped, the team had thrown out a popular framework, flipped their assumptions about tool design, and discovered that some of the instincts you bring from conventional software development actively work against you when you’re building agents.

I pulled out the ten moments where received wisdom turned out to be wrong. If you’re building anything agentic right now, these are worth sitting with.

❌ LangChain will help us build faster

LangChain is an opensource framework for building applications that use LLMs. It provides structured components that can be used for rapid prototyping.

In the video, Tovena talks about how they started with LangChain during the initial hackathon because they thought it would help them build more quickly, but instead it became obvious fairly early on that the opposite was true. They didn’t end up using a lot of the features that LangChain provided and felt they needed more flexibility and speed.

Abstraction overhead

One possible reason for their troubles with LangChain is that LangChain has standard abstractions for prompts, tool use, and orchestration– things that sound good in theory. But sometimes abstractions lend themselves to creating a sort of black box in your stack, where you don’t know what’s happening inside it. It sounds like the Assistant team missed the oversight and control of what was happening.

❌ Use the same prompt for every model

They also talked about how for the sake of simplicity and consistency, they initially used the same prompt for every model. However, they found out that this actually led to worse results. They found more success in creating slightly different prompts for different models, playing to their strengths and weaknesses a bit more.

There is even an argument (Mizrahi et al.’s “State of What Art? A Call for Multi-Prompt LLM Evaluation) that AI benchmarks that involve sending multiple LLMs the same prompt is a brittle and unreliable way to assess LLM capabilities. Different models respond differently to prompt variations.

In practice, what this means is that switching models isn’t a drop-in replacement. Even if a new model scores better on benchmarks, your prompts were written with a specific model’s tendencies in mind. The unlearning here is letting go of the idea that a prompt is model-agnostic — it’s more like a conversation style, and different models have different conversational defaults.

❌ Treat errors as failures

Traditionally in software, you catch errors and hide them from the user: you log them, handle them, but treat that transaction as a failed one.

The Assistant team had to unlearn this habit when dealing with an LLM, because with agents, error messages are useful data. Feeding it back to the LLM often results in it learning from its own mistake and trying again. It’s basically letting the agent debug itself, in a manner that Ryer noted was remarkably similar to how we humans debug errors too.

❌ Return errors in structured JSON

On the subject of error messages, Großmann mentioned that at one point they were trying to format the errors in JSON or stringified JSON. This practice shows solid engineer intuition because usually, structuring information helps machines process it faster and more accurately. What they found here was the reverse: it was easier and yielded better results when the LLM was just passed the entire error in natural language.

❌ Build as many tools as possible

The team mentioned that at the start, they were inclined to create as many tools/skills for Assistant: one for everything that could be done in Grafana. This made sense until they saw LLMs struggle to determine which tool to use in which context.

To help LLMs identify the correct tool, they added tool descriptions and also experimented with combining multiple tools into fewer, parameterized, tools. These helped, although the team still observed that allowing users to create their own tools (with potentially very general tool descriptions) could potentially confuse LLMs.

❌ Bigger context window = better

Gemini 1.5 Pro announced it had a 2M context window back in 2024, and it might seem that other providers had lagged behind: OpenAI only announced support for 1M tokens in 2025 and Anthropic followed in 2026. However, the Assistant team pointed out that context wasn’t everything. They said that they noticed that bigger wasn’t necessarily better, also pointing out that the chat summarization feature on IDEs like Cursor could still sometimes lose important details when summarizing.

This finding seems to coincide with a 2023 Stanford study (Lost in the Middle: How Language Models Use Long Contexts) that found there was a performance degradation when models access information within a very large context. In the video, they talked about how their system prompts were often 40k tokens long (half for the prompt itself and then the other half for tools).

❌ Start with a multi-agent architecture

Tovena mentioned that creating a multi-agent architecture for Assistant yielded more mixed results than he expected. He said that passing context between agents is difficult, outputs were inconsistent, and there was a bit of a “telephone game” problem where agents didn’t always perfectly understand the intent or output of other agents.

In early 2026, Khatua et al. released a study called “CooperBench: Why Coding Agents Cannot Be Your Teammates Yet” that showed that AI agents aren’t great at collaboration. The study suggests, surprisingly, that this might be because AI agents lack human social habits that help us collaborate better, such as feeling obliged to do what we say and having a good understanding of what others’ plans are.

The Assistant team’s experience is also a perfect example of Gall’s Law, which states that it’s easier to start with simplicity and build complexity than the other way around.

❌ Plan around models as they are now

Software has always moved quickly, but it feels like AI models are moving faster. Stanford’s AI Index Report 2025 reported that models went from having a 50% chance to near-human performance in roughly 2 years (2021-2023). In fact, benchmarks keep getting retired as AI saturates them– meaning that benchmarks can’t reliably be used to distinguish between good and bad models because all the models are testing near the ceiling.

The rapid pace of development around AI requires us to not be so attached to our work: Großmann recalled an incident where someone wrote a joke prompt that spoke like a dog that shockingly did better than their other system prompts. Ryer also mentioned that at one point someone deleted all the system prompts and Assistant weirdly became more accurate.

The lesson is clear: plan for models to improve, and plan to redo a lot of your work when they do.

❌ We don’t need evals

The Assistant team said that they went from having no evals to employing several types of evals:

integration-style evals that exercise whole workflows
unit-style evals that feed inputs plus a system prompt and check the output
LLM-as-judge, where they pass conversations to an LLM to ask it to judge whether it answered the user’s questions.
user feedback in the form of thumbs during conversations with Assistant

Beyond evaluation, they talked about how important it was to be able to observe AI. At the very minimum, showing what Assistant is doing at all times (thinking out loud) is very important to them because it builds users’ trust in the thinking process and, eventually, in the result.

Evals aren’t a sign that you don’t trust your product. They’re how you build the confidence to keep shipping it.

❌ Assume people already know how to work with AI

When you ship an AI feature, it’s tempting to assume the hard work is done. You’ve built the thing. Users will figure it out.

What the Grafana Assistant team found was that this assumption doesn’t hold. Users came in with wildly different mental models of what an AI assistant could or should do — some treating it like a search box, others expecting it to read their minds, most not yet sure how to prompt it effectively.

Their insight was that this isn’t just a UX problem to solve on your end. It’s a shared responsibility: AI should try to help the user, but the user also needs to learn how to work with AI. And building an assistant that teaches people how to use it well (through good responses, gentle corrections, showing its work) is part of building the product.

There’s also a deeper point here specific to observability. Grafana Assistant isn’t just an AI feature; it’s a way of interacting with your own infrastructure. Every good interaction teaches you something about what questions to ask, what signals to look for, what it means to understand a system. In other words: learning to use AI well makes you a better engineer. The tool and the skill co-evolve.

This is the unlearning that sits furthest from the code. It’s not about prompts or architectures or frameworks. It’s about recognising that you’re not just shipping a feature– you’re introducing a new way of working. That takes time. And it’s worth designing for explicitly.

Summary

What strikes me most about this list isn’t any single item — it’s the pattern. Most of these unlearnings point in the same direction: less is more, and messiness is the point.

Less abstraction. Fewer tools. Smaller context. Simpler architecture. And rather than hiding the mess — the errors, the reasoning, the uncertainty — let it show. That’s where the trust gets built.

We’re all still early with this. The team that built one of the most thoughtful AI products I’ve seen up close still didn’t have evals when they launched. That’s not a criticism — it’s a reminder. You don’t need to have it all figured out before you ship. You just need to commit to keep unlearning.

Let There Be Dashboards: Introducing Grafana Assistant

Thu, 26 Mar 2026 19:30:00 +0100

I gave this talk at the Grafana & Friends Lisbon Meetup in Lisbon, Portugal.

Abstract

This is a live demo of how to go from zero to a working Grafana dashboard using Assistant. Starting with a blank slate, we’ll use natural language to explore data and generate meaningful visualizations in minutes. Along the way, you’ll see how AI can accelerate dashboard creation without sacrificing control or understanding.

Thanks to everyone who came out! It was great to connect with the Lisbon tech community.

Do Androids Dream of Second Brains?

Sat, 21 Mar 2026 11:15:00 +0100

I gave this talk at PKM Summit 2026 in Utrecht, the Netherlands.

Abstract

We often describe our PKM systems as “second brains”. But can we really say our systems think for themselves?

In Philip K. Dick’s “Do Androids Dream of Electric Sheep?”, the inspiration for the movie Bladerunner, the boundary between humans and machines is tested through observation and subtle signals. In this talk, we’ll explore a similar question for our knowledge systems: what happens when we start observing them?

By instrumenting an Obsidian vault with an observability stack and layering AI tools on top, we can begin to see patterns in how ideas form, connect, and evolve. The result isn’t quite a thinking system… but it may be something closer than we’ve had before.

Slides

nicole.to/androids

Behind the Magic of Auto-Instrumentation (Grafana ❤️🔥 OpenTelemetry Community Call #6)

Fri, 27 Feb 2026 16:00:00 +0100

In this community call with my coworker Liudmila Molkova, we pulled back the curtain on OpenTelemetry’s auto-instrumentation capabilities. We explored how auto-instrumentation works under the hood, when to use it versus manual instrumentation, and the trade-offs involved. The discussion covered language-specific implementations, the Collector’s role in auto-instrumentation, and practical tips for getting started with zero-code observability.

Observability for GenAI Applications (Grafana ❤️🔥 OpenTelemetry Community Call #4)

Wed, 21 Jan 2026 16:00:00 +0100

In this community call with my coworker Liudmila Molkova, we dove into the emerging field of observability for Generative AI applications. We explored how to instrument LLM-powered applications using OpenTelemetry, what metrics and traces matter most when monitoring AI workloads, and how to debug issues in complex AI pipelines. The discussion covered prompt tracing, token usage monitoring, latency considerations, and the unique challenges of observing non-deterministic AI systems.

Logging Best Practices (Grafana ❤️🔥 OpenTelemetry Community Call #3)

Wed, 17 Dec 2025 16:00:00 +0100

In this community call with my coworker Liudmila Molkova, we explored logging best practices in the context of OpenTelemetry. We discussed how to structure logs for observability, the relationship between logs and other telemetry signals (traces and metrics), and practical patterns for making logs more useful in distributed systems. The conversation covered log levels, contextual information, correlation with traces, and common pitfalls to avoid when instrumenting applications with OpenTelemetry logging.

Lightweight Opensource APM with OTel Demo (Grafana ❤️‍🔥 OpenTelemetry Community Call #2)

Wed, 19 Nov 2025 14:48:59 +0100

This community call with my coworker Liudmila Molkova featured an interview with Cyrille Le Clerc from Grafana Labs, focusing on the OpenTelemetry demo as a practical example of a lightweight, open-source Application Performance Monitoring (APM) solution. Cyril explained the purpose and architecture of the OTel demo, highlighting its role as a realistic, complex microservices application that has become a de facto standard for observability vendors. The discussion explored the modern definition of APM, followed by a detailed walkthrough of a Grafana dashboard built on OTel data, covering RED metrics, alerting, trace correlation, and infrastructure monitoring. A significant portion of the conversation was dedicated to the role and challenges of integrating logs within an OpenTelemetry framework. The interview concluded with a look at the future of the OTel demo, including plans for AI observability and eBPF instrumentation.

Transcript

Introduction

Liudmila:
Okay. Hi everyone. Welcome to the second Grafana and OpenTelemetry community call. This time we’re going to talk about the lightweight open-source APM and we will use the OpenTelemetry demo as our playground. We have an expert, Cyrille, who works at Grafana. Cyrille, do you want to introduce yourCyrillef?

Cyrille:
Thank you very much, Liudmila. My name is Cyrille, and I am a product manager at Grafana Labs. I’ve been working on open-source products for 15 years, first on Jenkins for CI/CDs, then at Elastic on observability, and now at Grafana Labs, also on observability. In addition to my day job, I have been an open-source contributor for about 20 years, starting with Apache Java projects and then moving on to Jenkins integration with Maven and OpenTelemetry. Now, I focus more on the OpenTelemetry project, like the OpenTelemetry demo, and I contribute to OpenTelemetry Helm charts as well. At Grafana Labs, I am the product management lead in charge of OpenTelemetry and I cover other areas as well. [laughter]

Nicole:
That’s great. It’s great to have you here with us. Also, great for my internet to act up just a few minutes before this. Anyway, hi! I’m Nicole van der Hoeven, a developer advocate at Grafana. I work with Cyrille over there, and I’m really excited to be talking about OpenTelemetry. I’m the newbie of all of us here. [laughter] I have used OpenTelemetry in some demo apps I’ve built. Most recently, I used it for instrumenting an AI app. I’m excited to ask Cyrille all my questions because when he says he likes OpenTelemetry, what he really means is he’s the guy at Grafana who’s always banging on that OpenTelemetry drum. He’s like, “Tell all the things!” [laughter]

Liudmila:
It’s probably my turn. I’m Liudmila. I work on the OpenTelemetry project and I’m a technical committee member. To echo what Nicole said, nobody knows all of OpenTelemetry, and this call is where we will uncover areas that are new to each of us. Going forward, we’ll probably have more of these calls. So, with that, let’s talk about the OpenTelemetry demo. Cyrille, what is it? Why do we care?

What is the OpenTelemetry Demo?

Cyrille:
So, the OpenTelemetry demo was born to showcase OpenTelemetry. OpenTelemetry was created in the context of cloud-native architecture. This demo is a cloud-native application—a distributed microservices application that is instrumented with OpenTelemetry. It’s maintained by the OpenTelemetry community and showcases all dimensions of modern monitoring or observability, including all telemetry types: traces, metrics, logs, and more at every layer. We started with monitoring the application layer and are now extending to show the infrastructure layer.

We demo various technologies: Java, .NET, Node.js, and nearly all programming languages, including Rust. We also show how to instrument middleware and databases, such as Kafka, PostgreSQL, and Redis, all deployed on Docker Compose to make it easy to play with on your local desktop or deployed on Kubernetes to be more production-like.

This is a complex demo that shows everything, and what’s interesting is that it is realistic and currently used by 40 observability vendors to demonstrate their solutions with the OpenTelemetry demo. So, when you get familiar with the OpenTelemetry demo, it’s really easy to evaluate different vendors quickly. It comes “batteries included,” with open-source monitoring and observability technologies like Prometheus, Jaeger, OpenSearch, and Grafana to showcase how to implement an observability solution with OpenTelemetry.

Exploring the Application

Liudmila:
Thanks for the intro! So maybe we should take a quick look at what the application itCyrillef looks like to make it more concrete.

I’m connected to a Kubernetes cluster, and I’m just port forwarding. This is the telescope shop we have in the OpenTelemetry demo. We can go and buy things, and now we have something in the cart that we can place an order for. Oh, actually, this application might not work as expected. Well, that’s why we need observability, right? If things don’t work, we need to figure out what’s broken and how to fix it.

Before we move on, I just want to show that even though you only see the front-end web shop, it’s a relatively large application. It has quite a few services, databases, caches, a front-end proxy, and a log generator that consistently sends requests.

We’re super lucky to have all the telemetry Cyrille talked about. We have metrics in Prometheus, traces in Jaeger, and logs in OpenSearch, all visible in Grafana as a single pane of glass that you can refer to. With this, we can start looking at the telemetry itCyrillef. But I wanted to check with Cyrille or Nicole if you have any additional thoughts or anything we should cover here.

Nicole:
No, I think you presented it well. It’s a very complex topology because we want to demo everything.

Cyrille:
Yeah, it’s almost complex by design. You could probably have made this simpler, but it’s supposed to approximate real applications that aren’t simple. That’s the problem with some of these other demo apps; they are so simple to deploy that they work because they’re monolithic or something like that. This one is much more of a polyglot architecture, involving microservices that represent the real challenges that people face in the field.

Application Performance Monitoring (APM)

Liudmila:
One great thing about this demo is that sometimes it’s hard to understand how someone onboarded an application to OpenTelemetry. The demo has a lot of information on what services are used, and it demonstrates what’s been done to instrument this application with OpenTelemetry, which can help anyone with their journey in the OpenTelemetry space. Okay, wonderful! Let’s change gears and start looking at the APM dashboard.

Before we go there, let’s talk about what APM is for a second. Why do we care? What is it?

Cyrille:
APM is a very old concept—over 15 years old. We’ve had several generations of monitoring tools. APM stands for Application Performance Monitoring. I see APM as a monitoring and observability technology that helps you monitor your application and understand its health.

When I think about it, the first thing that comes to mind is understanding if my service is healthy or unhealthy. In a distributed architecture, I need to see my service in the context of the connections between services, typically represented with a service map. If I find something unhealthy, I need to drill down to see the health of a service. This means switching back and forth between the service metrics, which are more aggregated, and the traces, which give indications of how business transactions are executed.

I also need to access logs to verify how things really happen and if there are messages that help me understand the problem. This is how I understand my service. Additionally, my service is deployed on a runtime and middleware, so I need to be able to zoom lower to see how it’s behaving on the Kubernetes pod or Linux box.

APM allows me to switch back and forth between all layers, zooming in and out all the time across all telemetry types. This is essential for understanding health and troubleshooting. Another dimension is detecting anomalies, so I can wake people up at night when something goes wrong. This is where I create alerts and notifications.

In the past, with monoliths, we created many alerts, but now we adopt best practices with SLOs, observing workflows, and root cause analysis. This is my vision of APM and why I think it’s more important than ever.

Nicole:
When you talk about application performance management or monitoring, do you differentiate that from the wider field of observability? Some people say APM sounds outdated because it focuses on the application or just on performance, which is ambiguous.

Cyrille:
The monitoring landscape has evolved a lot. With cloud-native architecture, many things have changed, and we had to change our best practices. The best practices from 20 years ago in monoliths are not the same as today’s best practices for distributed cloud-native architectures.

There were disruptions in the past by innovators who focused primarily on traces, and we learned a lot from that. But now I think the landscape is getting more stable. We recognize the value in metrics, traces, and logs, and we’ve evolved our best practices to leverage the best of all three.

Some people prefer to call it observability. I know there are nuances, but this is my current perspective.

Liudmila:
It’s good to synchronize on terms. It sounds like we are on the same page.

Cyrille:
Now, with microservices architecture, we focus more on alerting symptoms rather than causes, centering more on the health of the application and the service provided to users than on the infrastructure layer. So, application monitoring or observability is becoming the entry point for people in charge of keeping the systems running.

Nicole:
Why don’t we take a look at the dashboard and go through it?

APM Dashboard Exploration

Cyrille:
We will cover different parts of the dashboard throughout the hour. Let’s start with the top row—the red metrics. You can call them duration, error, and request rate.

Nicole:
I believe the red metrics were created by Tom Wilkerson, who still works at Grafana. I think it was looCyrilley based on the Google SRE book, even though they didn’t explicitly say “red” there.

Cyrille:
When you think about any particular service, you would think about some well-known terminology, which might be service level indicators, or just your application. Service level indicators talk more about what’s important for the application, while red metrics focus more on the telemetry itCyrillef, like the raw data behind your SLI.

One thing that stands out is the alerts. It looks pretty awesome to see that something violates the threshold we’ve set—probably high latency. Let’s explore it. We see it’s been firing for 17 hours. That’s quite a duration.

Nicole:
That’s not so good. Let’s see the alert on the latency. The P95 is firing for a specific operation.

Cyrille:
We can look at this chart, which shows the latency around P99, which is like 18 milliseconds. Is it really a problem?

Nicole:
No, it was for the demo. It’s very situational; it depends on your services. The focus was to show practitioners best practices for creating alerts on HTTP endpoints. Typically, you have two families of alerts: latency alerts and error rate alerts.

Cyrille:
We decided to take P95 as a commonly adopted indicator. Some take P99, but P95 is often more relevant. Something that’s perhaps not noted here is that we take a time interval of 5 minutes. We wanted to provide this out of the box because it relates to the data collection interval, which is typically every minute for hotel setups and quite common in Prometheus.

Having a 5-minute window is quite good. These are best practices we worked on, identifying good labels to help practitioners succeed by easily routing alerts to the right people and having the right notification schemes.

Nicole:
Imagine if there was actually high latency. Now we can enable this view and look at the examples of operations with the worst performance—like the outliers you would see here. We use Jaeger as a store for telemetry, and if it were any other store that supports tracing, you would see something similar.

Cyrille:
This brings us to the Jaeger UI. I don’t want to spend a lot of time here, but it’s just another confirmation that nothing is visibly wrong with this operation. Things are just going a bit slower in the cloud, as usual.

Alert Analysis

Liudmila:
We talked about alerts. There are other interesting things in this picture. For example, the request rate is pretty normal. If there were a spike in usage, we would see it here. The error rate looks good—there are no errors. By looking at these three metrics, you can get a basic understanding of the health of your system and this specific service.

Cyrille:
This dashboard illustrates years of best practices that the industry has accumulated. We have layers of information. At the top, you have your service identifier, the last metrics received time, and the overview of the aggregated service red metrics.

This aggregated view across all operations of my service will help you decide if this service is healthy or not. If you have doubts or think something is going wrong, P95 is good at surfacing outliers.

Typically, you will scroll down to the HTTP or gRPC operations, where you have the details broken down by operation. It’s likely that not all business operations have problems; it could be just a subset of them.

Here you can see the outbound services. If you have a problem, this data will help you understand if it’s happening within your service or is it a downstream service. In microservices architecture, that’s always a question.

Nicole:
Are those dashboards powered exclusively by metrics?

Cyrille:
Everything is powered by metrics, specifically the OpenTelemetry semantic conventions. Initially, the OpenTelemetry demo derived metrics from spans, which was a great idea four years ago, but it had some limits. Now, OpenTelemetry has standardized some semantic conventions for HTTP server calls, client calls, database client calls, RPC calls, and messaging.

Nicole:
I noticed during my time with the demo that the coverage of metrics is uneven across programming languages. Java, for instance, has great instrumentation, while in other languages, it depends on the instrumentation.

Cyrille:
Yes, that’s true. It’s uncommon for OpenTelemetry instrumentations to emit logs these days, but let’s move on to that.

Logs and Observability

Nicole:
So, what’s the deal with logs? Why are they not as important?

Cyrille:
Logs are traditionally outputted in stdout, and we’ve lost Nicole. [laughter] The challenge is that traces and metrics are captured by SDKs within your app, while logs come from external log collectors, making correlation tough. When we discuss with practitioners and tell them that the OpenTelemetry SDK can directly export logs, they often respond that their log processes are mature and they can’t disrupt them just to trust the new kid on the block.

It takes time to change your log toolchain. And while OpenTelemetry SDKs have capabilities to directly export logs, it’s not easy to get people to change practices they’ve had for 30 years.

Nicole:
To demonstrate Cyrille’s point about correlation, the most basic way to correlate is by trace ID and span ID. When we capture distributed traces, we can also attribute logs to the specific operation.

Cyrille:
Yes, and when we look at a trace inside Grafana, we can see logs associated with it.

Nicole:
If we open the trace view, we can see the details for this operation. There are a bunch of attributes you can correlate to understand what happened.

Cyrille:
Logs provide the details. I think the negative sentiment around logs came from the idea that they don’t work as well in distributed applications as they did in monolithic applications. We had to evolve our practices to log the right way for distributed applications.

Nicole:
It’s user error because logs are the signals you can abuse the most. You can put anything in a log line, which causes issues. With metrics, you don’t have as much leeway, which makes it simpler to deal with.

Cyrille:
Yes, and we need to convince users that going into log analytics tools is much more efficient than tailing logs directly.

Nicole:
It is more complicated to analyze logs than traces because they aren’t always structured.

Cyrille:
Exactly. We have to help organizations better structure their logs for better value.

Questions and Community Engagement

Liudmila:
We have a question here. “We are planning to adopt the LGTM stack as an observability platform. Is there any blueprint documentation aligned with observability 2.0?”

Cyrille:
LGTM stands for Loki, Grafana, Tempo, and Mimir. It’s an integrated solution. Observability 2.0 is about unified telemetry—having everything correlated and not just reacting to errors. It’s about understanding business impacts when you decrease response times.

Nicole:
So, it’s more about understanding the overall system rather than just monitoring.

Cyrille:
Yes, exactly.

Liudmila:
How do I know which metrics and traces to prioritize first when instrumenting a distributed system with OpenTelemetry?

Cyrille:
We recommend starting with auto-instrumentation first. It provides metrics and traces at a technical level, and with HTTP, the endpoints are meaningful from a business standpoint. You start with auto-instrumentation and then add custom instrumentation on critical workflows.

Nicole:
What’s the current state of the OpenTelemetry demo? Can it monitor infrastructure as well?

Cyrille:
Yes, OpenTelemetry was born in the application layer, but infrastructure monitoring came a bit later. We’re working with the hotel demo folks to showcase better infrastructure monitoring done by OpenTelemetry.

Liudmila:
If people want to contribute or try out the hotel demo for themCyrilleves, where should they go?

Nicole:
I posted the link in the chat to the OpenTelemetry site.

Cyrille:
If you want to participate in the demo, we have community meetings, and I can share another link in the channel for more information.

Liudmila:
Great! Thank you both for being here and for everyone watching. I think that’s a great way to end this call.

Cyrille:
Thank you very much! It was a pleasure to be invited and an honor.

Nicole:
Thank you!

Asimov's Zeroth Law of Robotics: Observability for AI (Newcrafts 2025)

Thu, 06 Nov 2025 12:23:19 +0100

Today I presented at Newcrafts in Paris, France. I gave a talk I’ve given before, called Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU in London and at Dutch Cloud Native Day 2025 in Utrecht. I was happy to present it again, as it gave me the chance to further refine my demo application, which you can find on GitHub.

The main change I made was to use the “AI as judge” approach that I had previously only talked about in theory. In this version of the demo app, I use AI in both the game as well as in the testing of the game, showing how AI can be integrated into both the application and its observability/testing stack. It was really interesting to note that testing with AI ended up exposing more defects than my hardcoded test cases did, which goes to show the potential of AI-driven testing.

The Spice Must Flow: The Fremen Guide to Sustainable Observability (Øredev 2025)

Wed, 05 Nov 2025 03:07:38 +0100

Today I gave a talk at Øredev 2025 in Malmö, Sweden, titled +The Spice Must Flow: The Fremen Guide to Sustainable Observability"_. The presentation focused on how to implement sustainable observability practices using open-source tools like Grafana, Prometheus, Loki, Tempo, KEDA, and Kepler. Here’s the abstract:

Nobody wants to be wasteful. But how do you balance that with the need for enough data to fix things when they go wrong? After all, no matter how much it costs, there are SLOs to keep. The spice– the telemetry– must flow.

The Fremen are a people who live on the planet Arrakis, simultaneously rich in spice while poor in water. But the Fremen have managed to thrive, not just survive, on very little while continuing to produce spice. What would it take to apply this “desert planet thinking” to observability?

In this talk, you’ll learn about the hidden environmental and financial costs of your observability stack and how to reduce them. You’ll learn about the complications of measuring cost for ephemeral resources on Kubernetes, how to use KEDA to “right-size” resources when you don’t know what the right size is, and how to convince your teams to drop all but the most meaningful telemetry data. In the end, the spice must flow, but there must still be enough water to sustain all.

Here are some resources mentioned in the talk:

You can play the game Spice Runner here
The spice-runner repo containing everything you need to know about how to deploy it yourself
The slides
The Spice Runner dashboard for observing the application
More about Spice Runner: architecture, tech stack, and what it demonstrates

Deploying Loki on Kubernetes via Helm (Loki Community Call October 2025)

Wed, 29 Oct 2025 18:04:30 +0100

This Loki Community Call is about deploying Loki on Kubernetes via Helm charts. We talk about why you might want to use Helm to deploy on Kubernetes, best practices for deployment, and which Helm chart you should use! We are Jay Clifford and Nicole van der Hoeven, Developer Advocates at Grafana Labs, and we have invited Grafana Champion and Loki Helm Maintainer Jan-Otto Kröpke, Principal Cloud Architect at QualityOperations GmbH, to talk about the state of the Loki Helm Chart.

Declarative Configuration in OpenTelemetry SDKs (Grafana ❤️‍🔥 OpenTelemetry Community Call October 2025)

Wed, 22 Oct 2025 15:02:21 -0700

I’m very lucky to have the opportunity to work with Liudmila Molkova, who just joined as a Staff Developer Advocate at Grafana Labs. We decided we would start a new monthly livestream called the Grafana ❤️‍🔥 OpenTelemetry Community Call. In this inaugural episode, we interviewed Marylia Gutierrez about declarative configuration for OpenTelemetry SDKs. We talked about:

Pains in getting started with OpenTelemetry, particularly around instrumentation and configuration
What declarative configuration is, and how it improves the experience
Which languages declarative configuration is for
The future of declarative configuration in OpenTelemetry
Marylia’s candidacy for Governing Committee of OTel (vote for her here from October 27th-30th, 2025)

One Year Later: How I Use Visuals in My PKM (Sketch Your Mind Conference)

Mon, 13 Oct 2025 13:33:18 -0700

Today I gave a talk at my friend Zsolt Viczián’s Sketch Your Mind Conference about how I use visuals in my PKM, a follow-up to my video about Obsidian Excalidraw last year where I shared that I’d changed all my Obsidian templates to hybrid Obsidian Excalidraw notes:

In the talk, I shared how I’ve evolved my use of visuals in my PKM over the last year, including:

Referring to icon and logo libraries that dynamically compile all the atomic visuals I use in my notes
Adopting a visual Zettelkasten-like approach
Creating notes at conferences to track people I’ve met and talks I’ve attended
Using Obsidian Excalidraw to create compelling presentations in lieu of slides
Combining visuals with audio recordings to create literature notes of books I’m reading
Using a full observability pipeline to monitor and analyze the notes in my Obsidian vault

I was particularly excited about the last point. I work at Grafana Labs and help people monitor their computer systems using opensource software. I realized that I could use the same tools to monitor my PKM system, so I set up a pipeline where a Python script scrapes note metadata, and sends information to Loki and Prometheus via Alloy, and then I visualize it all in Grafana. It’s a pretty nerdy and totally unnecessary setup, but here’s the repo for anyone interested in replicating it.

All About Structured Metadata in Loki (Loki Community Call August 2025)

Tue, 12 Aug 2025 19:56:32 -0700

In this video, Jay Clifford and I talk to Senior Software Engineer Salva Corts Sánchez from the Loki team about structured metadata in Loki. We discuss what it is, why you might want to use it, and also how to query it. Structured metadata is especially useful if you’re instrumenting your system with OpenTelemetry, so we also touch on that.

How to Turn Logs Into Metrics with Grafana Loki (Loki Community Call July 2025)

Tue, 08 Jul 2025 17:23:23 +0200

In this Loki Community Call, we talk all about how to turn metrics into logs in Grafana Loki using metric queries. We cover the basics of metric queries, how to use them to create metrics from logs, and some advanced use cases like using metric queries to create alerts and dashboards.

Asimov's Zeroth Law of Robotics (Dutch Cloud Native Day 2025)

Thu, 03 Jul 2025 15:01:24 +0200

Today I presented at the Dutch Cloud Native Day 2025 in Utrecht, the Netherlands. I gave a talk titled Asimov’s Zeroth Law of Robotics: Observability for AI, which I had previously presented at KubeCon EU earlier on in the year, in London. I was very happy to be able to present it again, because it gave me the opportunity to improve the demo app I used, which you can find on GitHub.

A Real Zettelkasten Workflow in Obsidian, with Tris Oaten and Bob Doto

Wed, 25 Jun 2025 12:13:37 +0200

I recently had the pleasure of joining my friends Tris Oaten of No Boilerplate on YouTube and Bob Doto on my YouTube channel for a discussion on Zettelkasten, including demos of all three of our real Zettelkasten workflows in Obsidian. We talked extensively about the differences in how we maintain our Zettelkästen and plugins that we use to do so. We particularly talked a lot about folgezettel, and how the Obsidian Breadcrumbs plugin can be used to implement this concept in Obsidian.

If you’re interested in this topic, you might also like Bob’s book A System for Writing: How an Unconventional Approach to Note-Making Can Help You Capture Ideas, Think Wildly, and Write Constantly. I reviewed it and found it extremely useful, and Tris and I both use it as a reference for our Zettelkasten workflows.

The Road to Loki 4.0 (Loki Community Call June 2025)

Tue, 24 Jun 2025 13:56:45 +0200

In this video, my colleague Jay Clifford and I interview Ed Welch, Principal Engineer on the Loki team at Grafana Labs, about the upcoming Loki 4.0 release. We talk about some pretty big features that make this next release almost an entire rewrite of Loki, including:

A shift towards columnar storage, using a new storage format called “DataObject”
A new Kafka/WarpStream architecture that decouples the read and write paths
A new Loki Query Engine that decouples the query language from the rest of the system, allowing for more options in, for example, allowing for SQL queries against Loki in the future

How to Sketch Your Mind in Obsidian and more, with Zsolt Viczián

Wed, 18 Jun 2025 15:30:49 +0200

My friend Zsolt Viczián, a visual thinker and developer behind the plugins Obsidian Excalidraw and Obsidian ExcaliBrain, has written a book called “Sketch Your Mind: Nurture a Playful and Creative Brain”. The book is about how to create a system for visual thinking that helps you to think better, make connections, and make things with your ideas.

Get Zsolt’s book here!

How to Query Grafana Loki

Fri, 13 Jun 2025 15:21:49 +0200

In this video, I show you how to query Grafana Loki, the open source log aggregation system from Grafana Labs. I cover the basics of Loki’s query language, LogQL, and demonstrate how to use it to filter and search logs effectively.

State of the Observability Databases (Grafana Office Hours #30)

Thu, 15 May 2025 18:22:26 +0100

In this Grafana Office Hours, we talk about the state of observability databases (Grafana Loki, Mimir, Tempo, and Pyroscope) and where they’re going. We talk about current and upcoming architectural changes in all four, how we’re making them more performant, how compatible they are with OpenTelemetry, and what we’re working on next for each database. In this conversation are Dee Kitchen (VP of Engineering for Databases) and Senior Developer Advocates Jay Clifford and Nicole van der Hoeven.

Meta-monitoring Loki (Loki Community Call May 2025)

Mon, 12 May 2025 17:59:48 +0100

In this Loki Community Call, we talk about the need for meta-monitoring Loki: why Loki needs to be monitored, what to watch out for, and how to do it. We talk about different ways to get information from Loki that allow you to make it reliable, consistent, and performant, including a Helm chart to deploy a meta-monitoring stack on Kubernetes. We discuss the Loki mixin for Grafana and how to use it to visualize data about Loki. On the call are Jay Clifford, Nicole van der Hoeven, and Dylan Guedes from Grafana Labs.

Nicole van der Hoeven: A Polymath with Obsidian as a Modern Zibaldone

Sun, 27 Apr 2025 09:40:45 +0200

I joined Malik Alimoekhamedov on his podcast The Mechanics of Knowledge Management for a long conversation about Obsidian, PKM, and the Renaissance-era zibaldone tradition — the personal commonplace books that thinkers like Leonardo da Vinci kept as a hybrid of journal, reference, and idea workshop. We talked about how a modern Obsidian vault can play the same role for someone trying to think across domains, and about what changes when you treat your notes as the exhaust of learning rather than the destination.

🎙️ Listen on Substack

🎙️ Listen on Apple Podcasts

Thanks to Malik for the wide-ranging conversation.

Getting Started with Logging and Grafana Loki (webinar)

Wed, 16 Apr 2025 08:18:04 +0200

In this webinar, I talk about the key benefits of Grafana Loki, the open source log aggregation system from Grafana Labs. I cover the basics of logging, how Loki works, and how to get started with it. This was a live webinar, but I recorded the session here as well:

Do's and Don'ts of Scaling Loki Cache (Loki Community Call March 2025)

Tue, 25 Mar 2025 17:56:00 +0100

In this Loki community call, Poyzan (Staff Engineer on the Loki team) returns this time with Paul Rogers (Staff Engineer on the Loki team) to talk all about Loki cache! This topic was heavily discussed in our “How to run Loki at scale on Kubernetes” call. Some of the topics we will cover:

Types of cache and cache tiering
Best practices for scaling your cache
Understanding failure scenarios with Loki cache and how to detect them

Best Practices for Recording and Alerting (Loki Community Call February 2025)

Wed, 26 Feb 2025 17:54:15 +0100

In this Loki Community Call, we talk about the best practices for recording and alerting rules with Loki Staff Engineer George Robinson and Senior Developer Advocates Jay Clifford and Nicole van der Hoeven. Ever wondered what’s the difference between the Ruler and Alertmanager? 🤔 Curious about the different modes the Ruler can run in when querying Loki? Want to avoid the most common pitfalls when setting up alerting and recording rules? We cover all of these, including optimizing rule performance to structure alerts effectively.

How to Run Loki at Scale on Kubernetes (Loki Community Call January 2025)

Thu, 23 Jan 2025 17:51:00 +0100

Happy New Year from the Loki Engineering team. To kick off 2025, Nicole and Jay will be joined by Poyzan Taneli from the Loki Engineering team to discuss how to run Loki at scale on Kubernetes. If you are currently running Loki in microservices mode or preparing to do so, we will be discussing best practices for scaling its components to meet the demands of production use cases.

How many Ingesters vs Queriers do I need?
What is the minimum cost of running Loki as a production cluster?
What is the average node size for running Loki? How do I monitor my Loki deployment?

My Year in PKM: 2024 in Numbers

Fri, 10 Jan 2025 17:36:38 +0100

Rather than a full explanation of what in the world happened to me last year, here’s my year in PKM. I use my Personal Knowledge Management system for everything, so I thought I’d let it speak for me through some statistics I gathered from different tools I consider part of that system.

Grafana Loki Query Best Practices with LogQL (Loki Community Call December 2024)

Thu, 05 Dec 2024 17:48:49 +0100

In this December’s Loki Community Call, Cyril Tovena, Senior Principal Engineer and LogQL guru walks us through a Grafana Loki query tutorial with LogQL, the Log Query Language used for Loki. He talks about the key “Dos and Don’ts” of LogQL, offering practical tips to help you write better queries, boost performance, and sidestep common mistakes.

Whether you’re tuning up your current setup or just diving into LogQL, Cyril’s got you covered. Bring your questions, connect with the Loki community, and pick up insights that’ll make your log-querying life easier. See you there!

Make sure you leave your questions here.

Cyril is joined by Senior Developer Advocates Jay Clifford and Nicole van der Hoeven.

AI Observability with Ishan Jain (Grafana Office Hours #29)

Fri, 08 Nov 2024 18:24:29 +0100

In this Grafana Office Hours, Ishan Jain talks about AI Observability with Grafana: what it entails, factors to consider when monitoring and observing LLMs, and how to do it all with Grafana. He is joined by Senior Developer Advocate Nicole van der Hoeven.

All About Logs Drilldown (Loki Community Call October 2024)

Thu, 03 Oct 2024 17:45:33 +0100

In this Community Call, Senior Software Engineer Trevor Whitney talks to us all about Drilldown Logs for Grafana Loki, an open-source app for visualizing logs from Loki in Grafana without needing to learn and write LogQL queries. He is joined by Senior Developer Advocates Nicole van der Hoeven and Jay Clifford.

Bloom Filter Changes for Grafana Loki (Loki Community Call September 2024)

Thu, 05 Sep 2024 17:41:18 +0100

In this Community Call, Senior Software Engineer Christian Haudum talks to us about bloom filter changes for Grafana Loki, including the deprecation of the bloom compactor and a pivot towards creating bloom filters for structured metadata. Bloom filters are a probabilistic data structure that we’re using to improve query performance in Loki.

Community Calls are monthly meetings that are open to everyone interested in the development of Loki. They are an opportunity for software engineers working on Loki to discuss new features as well as for open-source users of Loki to ask questions. To participate in the next Community Call, subscribe to the calendar here.

API Monitoring with Traefik, Grafana, and OpenTelemetry (Grafana Office Hours #28)

Fri, 12 Jul 2024 18:27:03 +0100

Maytham Alfouadi (Solutions Architect) and Immánuel Fodor (Product Manager) from Traefik Labs give us a demonstration of how to do API monitoring with Traefik, Grafana, and OpenTelemetry. They talk about Traefik, an open-source reverse proxy, among other things, that is now included by default in k3s and Rancher. They are joined by Usman Ahmad and Nicole van der Hoeven, both Senior Developer Advocates at Grafana Labs.

Cartography of the Deep: Distributed Tracing for Testing (Romanian Testing Conference)

Thu, 13 Jun 2024 12:24:02 -0700

I presented a talk today at the Romanian Testing Conference 2024 in Cluj-Napoca, Romania! My talk was called “Cartography of the Deep: Distributed Tracing for Testing” and it covered how to tie in distributed tracing into tests to get a better understanding of your system under test.

Here are my slides!

Abstract

From the Scandinavian Kraken that drags entire ships to their early graves to the Philippine Bakunawa said to rise from the waters to swallow the moon, our existential fears about what lurks at the bottom of our own oceans has inspired us to create hundreds of folkloric monsters across cultures. We’ve charted the surface of our planet, but the vast majority of the ocean floor remains a mystery.

The mysterious depths of modern software architectures makes it difficult for us to test a system for performance and reliability. The easy solution? Unmanned underwater probes: send inputs to the system, and judge its state based on the outputs that return. But this method is often too limited to tell us what’s really going on when something goes wrong. What if there were a way to make our tests themselves more observable?

In this talk, you will learn how to use open-source tools to instrument performance tests, identify bottlenecks at the code level using distributed tracing, and map out a system using a type of software bathymetry. You’ll also learn about the Mariana Trench expeditions, the difficulties of operating in a high-pressure underwater environment, and how to come face to face with monsters of our mythologies.

Grafana Loki Design Basics with Ed Welch (Grafana Office Hours #27)

Fri, 07 Jun 2024 18:29:26 +0100

Ed Welch, Principal Engineer on the Loki team, is talking to us about how to get started with Grafana Loki for beginners. He tells us the story of why Loki was created in the first place, talks about the terminology of Loki (what are indices, streams, labels, metadata, tags, and chunks?), and shares candidly about what Loki is good at and what it’s not good at.

Connecting Visual Notes with Obsidian Excalidraw

Fri, 26 Apr 2024 17:34:58 +0100

Connecting visual notes isn’t as easy as connecting textual notes. Yet those connections are an essential part of taking notes that actually help you learn or remember things. In this video, I talk about ways to connect Excalidraw drawings and hybrid notes.

Visual Note Templates With Obsidian Excalidraw

Fri, 19 Apr 2024 17:31:57 +0100

How can you create visual note templates that encourage both visual and textual note-taking? The Obsidian Excalidraw community plugin bridges this gap in Obsidian by allowing the creation of hybrid notes: notes that are both textual and visual by default. Obsidian Excalidraw is created by Zsolt Viczián.

Doing It in Public: Observability for PKM

Fri, 05 Apr 2024 17:26:42 +0100

The quest of learning in public with Personal Knowledge Management is really about making what you learn observable and exposing your process, rather than your result, early and often. This is a rendition of a talk I gave at the PKM Summit in Utrecht, the Netherlands, on March 23rd, 2024, and it relates concepts borrowed from performance engineering and software development to PKM systems.

Real-time Collaboration in Obsidian with Peerdraft

Sat, 23 Mar 2024 17:24:07 +0100

Working with others in Obsidian hasn’t always been the easiest task. Obsidian Peerdraft is a new community plugin and accompanying web app that brings real-time collaboration to Obsidian. In this video, I tell you how to use Peerdraft, collaborate with a friend live while using it, and interview the developer of the app and plugin, Dominik Schlund.

How I'm Writing a Book in Public

Fri, 16 Feb 2024 17:21:16 +0100

I previously announced I’d be writing a book in public, but that didn’t happen. I explain what happened and why I’m writing another book instead. The new book is called Doing It in Public, and it’s about learning in public, my way… and, appropriately, I’m writing it in public!

How to get started in DevRel with Gertrude Westrin (Adobo & Avocados #29)

Wed, 31 Jan 2024 11:04:30 +0200

Gertrude Chilufya Westrin joined my friend Marie Cruz and me on Adobo & Avocados to talk about how she got started with a career in Developer Relations:

how and why she moved from Zambia to Sweden
how she got her first job in tech
the difference between Developer Relations and Developer Advocacy
how a developer advocate can help the community.

How to publish your notes for free with Quartz

Fri, 26 Jan 2024 14:00:47 +0100

Quartz just won Best Tool in the Obsidian Gems of the Year 2023 awards, and with good reason. It’s a free, open-source static site generator that works really well with Markdown-based note-taking apps– including Obsidian, which it was originally designed for.

Here’s what Quartz looks like when published.

Mental Health as a Superpower with Karen Todd (Adobo & Avocados #28)

Thu, 25 Jan 2024 15:12:44 +0100

My friend Marie Cruz and I interviewed Karen Todd about neurodivergence and neurodiversity, and how her ADHD can be a positive thing in her career as a tester.

How to do continuous profiling right with Grafana Pyroscope and Ryan Perry, (Grafana Office Hours #26)

Fri, 19 Jan 2024 18:09:01 +0100

My friend Paul Balogh and I interview Ryan Perry, co-founder of Grafana Pyroscope, about how to do continuous profiling right. Ryan, and the rest of the Pyroscope team, joined Grafana Labs, where he is now a Director of Engineer. Ryan discusses the main concerns in continuous profiling and how to avoid those pitfalls. Pyroscope is an open-source project for aggregating continuous profiling data about your system’s resources.

Learning new things in Obsidian: in defense of remembering

Fri, 19 Jan 2024 12:01:34 +0100

Here’s how I’m learning things in Obsidian in 2024, and the tools I use in my information processing pipeline to read, listen, and watch content and turn that into ideas and output. I also talk about why it’s important to remember what you’ve consumed, and the value of writing stuff down to help you do so.

Why People Leave Software Testing, with Maria Kedemo (Adobo & Avocados #27)

Thu, 18 Jan 2024 11:54:46 +0100

In this episode of Adobo and Avocados, Maria Kedemo talks to my friend Marie Cruz and me about why she left software testing– and why she came back.

My workflow for making videos with Obsidian

Fri, 12 Jan 2024 18:00:32 +0100

I’ve been making videos with Obsidian for about three years now, and I thought I’d share what I’ve learned. Here’s how I plan, outline, and edit my videos, as well as how I process them back into my notes.

This video was sponsored by Skillshare. The first 500 people to use this link will get a 1 month free trial of Skillshare. Signing up, even for just the free month, supports me and my work. Thanks!

The Power of Community, with Tara Walton (Adobo & Avocados #26)

Wed, 10 Jan 2024 13:48:34 +0100

What does a community builder do? In this episode of Adobo and Avocados, Tara Walton talk to my friend Marie Cruz and me about what community is, its benefits, and ways to misuse it.

How to Deploy Grafana Beyla on Kubernetes as a Sidecar Container

Tue, 09 Jan 2024 13:41:38 +0100

Principal Staff Engineer Nikola Grčevski demonstrates how to deploy Grafana Beyla on Kubernetes as a sidecar container. Grafana Beyla is an eBPF auto-instrumentation tool for application observability: you just have to tell it what to listen to, give it permissions, and deploy it– and you can see metrics and logs from that component or service without having to do any manual instrumentation.

Why Not to Make Resolutions in 2024

Fri, 05 Jan 2024 18:24:10 +0100

I’ve done a lot of thinking about why not to make resolutions in 2024. In 2023, I went a little overboard with OKRs (Objectives and Key Results) and that just sent me spiralling into overwork and burnout. That’s not something I want to see happen again this year, so instead, I’m doing a yearly theme: a one-word direction for the rest of the year that will act as a North Star rather than a strict set of rules. I’m hoping this will allow me to be productive, but also keep me happy.

Introduction to eBPF With Grafana Beyla (Grafana Office Hours #25)

Fri, 05 Jan 2024 18:03:52 +0100

Nikola Grčevski, Principal Software Engineer at Grafana Labs, gives us an introduction to eBPF with Grafana Beyla. We discuss what is eBPF, how you can use it to auto-instrument applications, and how to get started with Beyla. eBPF observability is all the rage because we all want automagical instrumentation– but does it live up to that promise? Nikola’s here to tell us what eBPF can and can’t do, and where he’d like to take Beyla next.

Links vs Tags vs Folders: knowledge gardening with Jorge Arango

Wed, 27 Dec 2023 19:38:49 +0100

If you’ve ever questioned the benefits of links vs tags vs folders, my guest today, Jorge Arango, has some opinions about which method is best for you when creating a knowledge garden in Obsidian. Information architecture is a field of design that makes it easier to find information, and I’m talking to Jorge today about how the same principles can be used to find things in your notes.

Jorge is an information architect, author, and educator. For the past 25 years, he has used architectural thinking to bring clarity and alignment to digital products and services. He’s the author of Living in Information: Responsible Design for Digital Places, co-author of Information Architecture: for the Web and Beyond, and host of The Informed Life podcast. Besides consulting, writing, and podcasting, Jorge also teaches in the graduate interaction design program at the California College of the Arts.

Get a 15% discount on Jorge’s book by buying it here and using the code NVDH, valid through January 26th, 2024.

Or, buy it on Amazon using this affiliate link.

Learning about Glue Work and Quiet Quitting (Adobo & Avocados #25)

Wed, 20 Dec 2023 19:36:04 +0100

What is glue work, and how is it linked to quiet quitting and quality engineering? In this episode of Adobo and Avocados, Vernon Richards talks to Marie Cruz and me about what glue work is, the dangers of quiet quitting, its implications with quality engineering, and what we can do about it.

DevOps Culture from a Tester's Perspective (Adobo & Avocados #24)

Fri, 15 Dec 2023 17:50:03 +0100

We’ve all heard of DevOps, but what is DevOps from a tester’s perspective? How can we embrace this culture? In this episode of Adobo and Avocados, the one and only Lisa Crispin talks to my friend Marie Cruz and me about DevOps culture from a testing mindset, what holistic testing is really about, and her journey to being one of the well-respected voices in the testing community.

Note-taking Templates I Use in Obsidian

Fri, 15 Dec 2023 17:41:03 +0100

Using note taking templates helps me reduce the unnecessary tedium of taking and making notes, and I use them liberally in Obsidian. Why should you use them? Well, in this video I’m sharing some of the actual templates that I use on a daily basis and some recommended practices for working with templates in general.

My Daily Tech for Productivity

Fri, 08 Dec 2023 17:05:50 +0100

Here’s a list of my daily tech for work and play– all the everyday tech gear I use in these categories: laptops, mobiles, tablets, ereaders, wearables, and audio. I like to get the best devices for my use cases, but I’m also willing to use multiple devices in each category if I have a need for it.

You can find all the gear I talked about in this video, along with other things I use, on this page.

Getting Started with OpenTelemetry (Grafana Office Hours #23)

Fri, 08 Dec 2023 17:00:23 +0100

Principal Engineer Juraci Paixão Kröhling talks to my colleague Paul Balogh and me about getting started with OpenTelemetry: what it is, why everyone’s talking about it, and how to instrument your systems and improve their reliability and observability. Juraci is a contributor to open source projects like Jaeger and OpenTelemetry.

How I Refactor Notes in Obsidian

Fri, 01 Dec 2023 12:03:58 +0100

It’s not enough for me to just take notes - they have to be useful, too. Here’s how I refactor notes in Obsidian using community and core plugins to inventory, structure, categorise, explain, and modularise my notes. Refactoring isn’t just for code. It’s for notes too! It’s only through gradually updating my notes over time that I can begin to reap the benefits of continuous note-taking.

Resources

Monitoring (notes)
Coding and writing converging (notes)
Quick Switcher (notes)
Outline (notes)
Search (notes)
Strange New Worlds (notes)
Readwise Official (video)
ExcaliBrain (video)
Obsidian Excalidraw (video)
Note Refactor (notes)

How to Get Started with Tempo (Grafana Office Hours #22)

Fri, 01 Dec 2023 11:55:55 +0100

Grafana Labs Principal Engineer and Tempo maintainer Joe Elliot talks about how to get started with Tempo, what distributed tracing is, and why TraceQL is cool. He is joined by Developer Advocates Nicole van der Hoeven and Paul Balogh.

Resources

Getting started with Tempo (video)
Grafana Tempo repo
Grafana Tempo docs
The Tempo category on the Grafana community forum
Video playlist for Grafana Tempo
Distributed tracing with Grafana Cloud k6 (Grafana Office Hours video)

Performance Testing With Nicole van der Hoeven (Quality Bits Podcast)

Tue, 28 Nov 2023 14:01:35 +0100

My friend Lina Zubyte invited me to her podcast Quality Bits to talk about performance testing. We talked about what performance testing is, why it is important, how to get started, and how to get better at it. We also talked about the foundation of any type of testing: empathy.

Check out the episode here.

Navigating through Unemployment with Renata Andrade (Adobo & Avocados #22)

Tue, 28 Nov 2023 13:16:15 +0100

With the rise of redundancies in the tech industry, how do you navigate through unemployment calmly? In this episode of Adobo and Avocados, Renata Andrade, a Quality Engineer Lead, talks to Marie Cruz and me about her experience navigating unemployment and the steps she followed to bounce back.

How to Use Flow Mode for Grafana Agent (Grafana Office Hours #21)

Fri, 24 Nov 2023 11:47:00 +0100

Senior Software Engineer Matt Durham shows us how to use flow mode for Grafana Agent. Flow mode is a new and better way to install and configure Grafana Agent than the older “static mode”. Among other things, flow mode’s modularity makes it easier to build more complex workflows like traditional data pipelines and allows for more use cases than collecting and processing telemetry.

Resources

Grafana Agent repo
Grafana Agent docs
Grafana Agent flow mode
What is Grafana Agent? (previous Grafana Office Hours video)
Grafana Slack - there’s an #agent channel
Grafana Agent Community Call agenda
Modules for Grafana Agent

Realistic Tests with Testcontainers (Grafana Office Hours #20)

Fri, 17 Nov 2023 11:40:46 +0100

In this episode of Grafana Office Hours, we talk about realistic unit tests utilizing actual instances of databases and other dependencies. We’ll be joined by Manuel de la Peña from AtomicJar, and Pablo Chacin from Grafana to learn how we can use the Testcontainers open-source framework to create lightweight, throwaway dependencies for our tests.

Minimalism in Software Testing with Lina Zubyte (Adobo & Avocados #21)

Wed, 15 Nov 2023 13:13:52 +0100

How can we apply minimalism in software testing? Lina Zubyte, an Independent Quality Consultant, talks to Marie Cruz and I about what really matters in testing, why empathy is at the root of the testing attitude, and how we can incorporate principles from minimalism to improve quality.

Load Testing on Kubernetes with k6 Private Load Zones (Grafana Office Hours #19)

Fri, 10 Nov 2023 10:46:48 +0100

This week, we’re talking about how you can do load testing on Kubernetes with k6 Private Load Zones, a new feature on Grafana Cloud k6 that leverages the k6 Kubernetes operator to allow you to run distributed load tests against applications behind a firewall. Here to discuss this new feature are Senior Software Engineer Olha Yevtushenko, Product Manager Daniel González Lopes, Developer Advocate Paul Balogh, and Senior Developer Advocate Nicole van der Hoeven.

Learning about experience design with Jenna Charlton (Adobo & Avocados #20)

Wed, 08 Nov 2023 13:01:12 +0100

In this episode of Adobo and Avocados, my friend Marie Cruz and I chat with fellow Developer Advocate Jenna Charlton, where they talk about their career journey, what can we learn from experience design, and the impact of different job titles to testers.

Creating Characters in TTRPGs (Beyond the Character Sheet #01)

Sun, 05 Nov 2023 13:33:17 +0100

In this first episode of Beyond the Character Sheet, Leah Ferguson and I talk about creating characters in TTRPGs: how to create a character you love, how it works in different systems, and the interplay between player personality and character personality.

Combining Frontend and Backend Performance With John Hill (Grafana Office Hours #18)

Fri, 03 Nov 2023 12:30:43 +0100

In this episode of Grafana Office Hours, my colleagues Marie Cruz, Paul Balogh, and I talk to John Hill, a Web UI Test Engineer and Grafana k6 champion, to talk about how the Grafana and k6 ecosystems can be used to ensure performance in mission-critical applications like NASA’s Open MCT.

Becoming an accessibility advocate with Erin Crise (Adobo & Avocados #19)

Wed, 01 Nov 2023 12:22:40 +0100

In this episode of Adobo and Avocados, Marie Cruz and I speak with Erin Crise about her tech career journey, what led to her accessibility advocacy, and why empathy drives innovation.

K-SIX News v0.47

Mon, 30 Oct 2023 11:57:21 +0100

In this video, my colleagues (Leandro Melendez, Marie Cruz, and Paul Balogh) and I discuss the latest updates in k6 v0.47.

Here are some of the highlights:

We’ve created an official k6 Prometheus Grafana dashboard and another dashboard for Prometheus with native histograms.
New APIs have been added for Prometheus Remote Write output for k6 (authentication and HTTP headers).
We’ve increased the character limit for metric names from 63 to 128, aligned with the OpenTelemetry specification update.
k6 browser tests can be executed in Grafana Cloud by everyone.
We’ve improved the cookie API for k6 browser.
We’ve added support for page.on('console') for k6 browser.
There are now official Docker images that include a Chromium-based browser for running k6 browser tests within CI/CD pipelines.
Distributed tracing is in public preview for Grafana Cloud.
xk6-file is a new extension that lets you rename files within your k6 script.
xk6-gcp is a new extension that adds k6 support for messaging using Google PubSub.
xk6-dashboard is now an official Grafana extension that lets you generate HTML dashboards from k6 test results.
k6x is a new utility that automatically detects when you import k6 extensions and builds the custom k6 binary for you so that you don’t have to.
k6-operator now has a Helmchart deployment option.

Real User Monitoring With Grafana Faro (Grafana Office Hours #17)

Fri, 27 Oct 2023 07:00:19 +0200

My friend Marie Cruz and I chat with Software Engineers Kostas Pelelis and Marco Schaefer to discuss what Grafana Faro is, why it’s important to have a real user monitoring solution, and how to get started with it.

Useful links

Emergent Load Testing - TestCon Europe 2023

Wed, 25 Oct 2023 20:05:36 +0300

I presented at TestCon Europe 2023 in Vilnius, Lithuania about emergent load testing, and how to use it to find the limits of your system. My talk was called Emergent load testing: Rules for Organized Chaos.

You can find the slides here!

Inspiring others and fuelling your passion with Jonah Andersson (Adobo & Avocados #18)

Tue, 24 Oct 2023 20:00:53 +0300

In this episode of Adobo & Avocados, my friend Marie Cruz and I chat with Jonah Andersson, a Software Developer, Cloud & DevOps Engineer, and the founder of Azure User Group Sweden. We spoke about her career journey and how she is fuelling her passion and inspiring others.

What is Grafana Agent? (Grafana Office Hours #16)

Fri, 20 Oct 2023 12:55:11 +0200

What is Grafana Agent? Senior Software Engineer Paschalis Tsilias tells us about what telemetry collectors do, why they’re useful, what you’d have to do without them, and how to get started with Grafana Agent. He is joined by Developer Advocates Nicole van der Hoeven and Paul Balogh.

Useful links

Having a coaching mindset with Kulas Angeles (Adobo & Avocados #17)

Wed, 18 Oct 2023 12:27:16 +0200

In this episode of Adobo and Avocados, my friend Marie Cruz and I chat with Kulas Angeles, a DevOps coach and Quality Engineering leader, about his career journey and how he transitioned from engineering to coaching mindset.

Agentless Monitoring for Prometheus in Grafana Cloud (Grafana Office Hours #15)

Fri, 13 Oct 2023 10:56:15 +0200

Did you know you could do agentless monitoring for Prometheus in Grafana Cloud? Senior Software Engineer Matt Nolf talks to me and my coworker Paul Balogh about the new Metrics Endpoint integration and how you can use it to collect Prometheus metrics from publicly addressable hosts– WITHOUT using Grafana Agent.

Resources

Investing in yourself with Nicola Lindgren (Adobo & Avocados #16)

Wed, 11 Oct 2023 11:52:37 +0200

My friend Marie Cruz and I talked with Nicola Lindgren about different ways we’ve all invested in ourselves and our career– and how you can do the same.

Distributed tracing with Grafana Cloud k6 (Grafana Office Hours #14)

Sat, 07 Oct 2023 08:31:03 +0200

My friend Marie Cruz and I interview Senior Software Engineer Łukasz Gut about a new feature: Distributed Tracing with Grafana Cloud k6. He discusses what distributed tracing is, why it matters, and how it can help teams find reliability issues faster.

The Role of AI in Gender Equality with Nicola Martin (Adobo & Avocados #15)

Wed, 04 Oct 2023 17:26:14 +0200

In this episode of Adobo and Avocados, my friend Marie Cruz and I chat with Nicola Martin, Head of Quality Engineering at Adarga, to talk about the role of AI, technology, and education in gender equality.

Nicola is a Council and Committee Member for the British Computer Society and WomenTech Global Network. She is also a mentor, speaker, and panelist speaking on subjects such as AI and quality, software testing, and diversity in tech.

How to Deploy Grafana on Kubernetes (Grafana Office Hours #13)

Fri, 29 Sep 2023 17:30:49 +0200

My coworker Usman Ahmad and I talk about how to deploy Grafana on Kubernetes for beginners: what Kubernetes is, how it’s an evolution of distributed computing, what its relationship to Docker is, other things you might need to know to work with Kubernetes, and how you can deploy Grafana on your own Kubernetes clusters.

Documentation on how to deploy Grafana on Kubernetes here.

Easiest way to share a note on Obsidian - Obsius Publish plugin

Fri, 29 Sep 2023 17:11:21 +0200

Obsius Publish is a new Obsidian plugin that lets you share a note on Obsidian for free. It’s best used as a complement to Obsidian Publish in that it particularly shines when wanting to share notes temporarily. Obsius Publish is a third-party community plugin that adds commands to Obsidian to let you push the contents of a note to Obsius and gives you a shareable link to make the note accessible over the internet.

Resources

Support the developer of Obsius Publish, Jon G Stødle
Thanks to my friend Andy Polaine for telling me about Obsius Publish
Obsidian for Everyone

Modernizing government documents with Govable and Grafana (Grafana Office Hours #12)

Fri, 22 Sep 2023 21:01:39 +0200

Ari Hershowitz and Andrii Kovalov from Govable.ai talk about modernizing government documents with Govable and Grafana, and how they saved weeks of effort by using Grafana as a ready-made frontend for their clients. They are joined by Developer Advocates Nicole van der Hoeven and Paul Balogh from Grafana Labs.

The benefits and dangers of public speaking (Adobo & Avocados #14)

Tue, 19 Sep 2023 21:40:43 +0200

In this episode of Adobo and Avocados, my friend Marie Cruz and I chat with Maaret Pyhäjärvi, a Development Manager and Testing Specialist who has shared countless resources about exploratory testing. We talk about her journey to tech, the benefits and dangers of public speaking, and her approach to testing.

Winning over imposter syndrome with Sophie Küster (Adobo & Avocados #13)

Tue, 12 Sep 2023 21:38:15 +0200

My friend Marie Cruz and I chat with Sophie Küster, who currently works as a Test Automation Engineer, all about imposter syndrome.

Celebrating 100 episodes of k6 Office Hours (k6 Office Hours #100)

Fri, 08 Sep 2023 20:56:43 +0200

In this 100th episode of k6 Office Hours, the Grafana Developer Advocates celebrate how it all started! Join us at this live fika with our favorite drinks and discuss our favorite moments and episodes.

Grafana k6 for WebSockets and infrastructure testing (Grafana Office Hours #11)

Fri, 08 Sep 2023 20:54:28 +0200

In this episode of Grafana Office Hours, Solution Architect Huzaifa Asif talks about how he has used Grafana k6 for WebSockets and infrastructure testing, and how k6 can be used for general reliability testing in addition to load testing. He is joined by Grafana Labs Developer Advocates Nicole van der Hoeven and Paul Balogh.

Setting boundaries, time management, and essentialism (Adobo & Avocados #12)

Wed, 06 Sep 2023 21:34:49 +0200

My friend Marie Cruz and I discuss setting boundaries and managing time. We also discuss what essentialism is and why less is better.

Get the Essentialism book here (affiliate link).

How to learn Grafana with Grafana Play (Grafana Office Hours #10)

Fri, 01 Sep 2023 20:50:02 +0200

If you were wondering how to learn Grafana, Grafana Play is probably the easiest way. Grafana Play is a collection of ready-made dashboards and apps that you can use without creating an account. Developer Advocates Matt Abrams, Paul Balogh, and Nicole van der Hoeven discuss how to take advantage of this awesome tool and what you can do with it.

Getting started with Grafana Loki (Grafana Office Hours #09)

Fri, 25 Aug 2023 20:41:32 +0200

Senior Principal Solutions Engineer Ward Bekker talks about getting started with Grafana Loki: what Loki is, why you need log aggregation, and how it fits into the rest of the Grafana stack. He is joined by Developer Advocates Paul Balogh and Nicole van der Hoeven to tell you everything you need to know about Loki.

What's new in k6 browser? (k6 Office Hours #98)

Fri, 11 Aug 2023 20:37:25 +0200

k6 browser adds browser-level APIs to automate browser actions and collect web performance metrics as part of your k6 test. It’s an experimental module, and there is a good reason why!

In this k6 Office Hours, Developer Advocates Marie Cruz and Nicole van der Hoeven are joined by Software Engineers Ankur Agarwal and Daniel Jimenez to discuss the breaking changes that are about to come to k6 browser! You wouldn’t want to miss this.

Observing the Internet of Things (Grafana Office Hours #08)

Fri, 11 Aug 2023 20:31:53 +0200

My colleague Paul Balogh and I welcomed Daniel Mangum, Lead Cloud Engineer from Golioth. We learned how Golioth uses Grafana to report data from varied sensors and monitors.

Navigating parenthood with Laveena Ramchandani (A&A #11)

Mon, 07 Aug 2023 21:30:33 +0200

My friend Marie Cruz and I speak to Laveena Ramchandani, a Digital and Data Science Test Manager at easyJet and also a Digital Star finalist, speaker, blogger, and coach. Laveena talks about her career journey and her journey to parenthood.

How to get started with Grafana and Docker (Grafana Office Hours #07)

Thu, 03 Aug 2023 20:28:40 +0200

My colleague Usman Ahmad and I talk about how to use Docker to quickly get up and running with Grafana, how to add Grafana plugins, and how to troubleshoot common issues.

Grafana documentation on how to install Grafana on Docker here.

Making a career and being true to yourself with Lena Pejgan Nyström (Adobo & Avocados #10)

Fri, 28 Jul 2023 21:25:02 +0200

My friend Marie Cruz and I speak to Lena Pejgan Nyström, the author and creator of Would Heu-risk it? cards and book about various topics!

Lena opens up about how she got started in tech and her career journey from being a developer turned tester and now an Engineering Manager as well as an international keynote speaker! We also discuss topics such as diversity in tech, dealing with burnout, and any struggles Lena experienced between making a career and being true to her personal beliefs.

OpenTelemetry demo app with Grafana, Tempo, Loki, and Prometheus (Grafana Office Hours #06)

Fri, 28 Jul 2023 20:25:37 +0200

DevOps Engineer Blueswen Liu 劉義瑋 joins us to walk us through some OpenTelemetry demo apps he created, instrumented with Grafana, Loki, Prometheus, and Tempo.

Obsidian properties in YAML frontmatter (v1.4)

Wed, 26 Jul 2023 20:05:55 +0200

If you’ve ever tried to use Obsidian properties in YAML frontmatter, you may have noticed some issues. Obsidian v1.4 brings a new UI for properties that may just solve some of those, including the difficulty in keeping frontmatter properties and keys consistent.

Resources

Sign up for early access to all Obsidian releases
Full changelog
How to use the Dataview plugin (video)
How to use the Templater plugin (video)

All the Obsidian Core Plugins (Obsidian Tour 2023)

Fri, 21 Jul 2023 20:02:59 +0200

Lost in all the Obsidian core plugins? Here’s a video that goes through every single one of them: which ones I’ve left enabled, which ones I’ve disabled, and which community plugins I use instead. This “Obsidian Tour” series is about showing MY personal setup of Obsidian, not the “best” one. I’m hoping that sharing how I use these plugins will help you decide whether or not they could fit in your workflows, too.

Resources

Why monoculture is problematic for quality with Callum Akehurst-Ryan (Adobo & Avocados #9)

Wed, 19 Jul 2023 21:16:32 +0200

My friend Marie Cruz and I speak to Callum-Akehurst Ryan about his experience as an LGBTQIA+ software tester, the importance of diversity and inclusion, and how it can improve testing and quality.

Resources

Marie Cruz and Callum Akehurst-Ryan’s joint presentation on Testing in Story Shaping

Distributed tracing for testing with Grafana Tempo and Tracetest (Grafana Office Hours #05)

Fri, 14 Jul 2023 20:21:30 +0200

In this video, my colleague Paul Balogh and I welcomed Daniel Mangum, Lead Cloud Engineer from Golioth. We learn how Golioth uses Grafana to report data from varied sensors and monitors.

Taking notes on podcasts with Snipd, Readwise, and Obsidian

Fri, 14 Jul 2023 15:55:33 +0200

In this video, I show you my favorite method for taking notes on podcasts with Snipd, Readwise, and Obsidian: three tools that work very well together. Snipd is a mobile podcast player app for iOS and Android that integrates with Readwise. With the Readwise Official plugin for Obsidian, that means my highlights and notes from podcasts end up automatically in my Obsidian vault, tagged appropriately.

RESOURCES

Snipd AI Podcast Notes wishlist
Snipd
Readwise
Sample AI podcast notes
@CalNewportMedia - Deep Questions - The Failure of Cybernetic Productivity
@CalNewportMedia - Deep Questions - Decoding Overhead
@masteringdungeons - Mike Shea on Organized Play vs Home Games
The Informed Life - Andy Polaine on Service Design

What does an overloaded system look like for humans? (Adobo & Avocados #08)

Wed, 12 Jul 2023 21:13:41 +0200

My friend Marie Cruz and I talk about burnout from a more technical point of view. We discuss how burnout is similar to an overloaded system (and how it isn’t), and we draw comparisons on how to make an overloaded system more resilient and recover from burnout.

When Productivity Gets Toxic, with Andy Polaine

Fri, 07 Jul 2023 22:03:46 +0100

We usually see productivity as a positive thing, but what happens when productivity gets toxic? According to my friend Andy Polaine, design leadership coach, burnout can occur when you’ve been doing more than you used to, struggling with the work, and feeling like you can’t say anything to anyone. In this video, Andy shares some tips for setting up boundaries to create mental space and defending your calendar against external pressures.

Check out Andy’s work:

Andy’s website
Reach out about getting coaching from Andy here
Andy’s podcast, Power of Ten
Andy’s newsletter, Doctor’s Note
Andy’s upcoming book, The Design Leadership Dip

Monitor behind a Firewall With Private Data Source Connect (Grafana Office Hours #04)

Fri, 07 Jul 2023 21:59:57 +0100

How do you monitor behind a firewall? With Grafana Cloud Private Data Connect, you can create a secure tunnel to query data sources, even including those in VPCs on public cloud vendors. Senior Software Engineers Stephanie Hingtgen and Georges Chaudy talk to me about how it works.

Quality Engineering, DevOps, and work-life balance (Adobo & Avocados #07)

Thu, 06 Jul 2023 21:55:46 +0100

Developer Advocates Marie Cruz and I talk to Katja Obring, an experienced Quality Engineering Coach, about her experience in tech and moving to freelance for better work-life balance. We also talk about her book, “Testing in DevOps: A Guide to Optimised Testing for Speedy and High Quality Delivery,” and how Testing fits in the overall DevOps landscape, the importance of automated testing, and the cultural changes crucial to DevOps.

Why Representation Matters (Adobo & Avocados #06)

Sat, 01 Jul 2023 15:08:51 +0100

My friend Marie Cruz and I talk to Parveen Khan about her experience in tech and the importance of diversity and inclusion. Parveen is a Senior QA Consultant at Thoughtworks, a quality advocate, and an international speaker.

Every Single Plugin I Use (Obsidian Tour 2023)

Fri, 30 Jun 2023 23:39:17 +0100

In this video, I talk about all the Obsidian plugins I currently use… all 46 of them. I demonstrate some of them, talk about why I didn’t keep others, and talk through my use cases for each one. These plugins aren’t necessarily the best ones out there for Obsidian - they’re just the ones that are useful enough for my use cases to have them installed on a daily basis.

Recording a Performance Test With Grafana Cloud k6 (k6 Office Hours #93)

Fri, 30 Jun 2023 23:33:12 +0100

Senior Frontend Engineer Edvinas Daugirdas demonstrates recording a performance test with Grafana Cloud k6 and some other new features that have recently come to Grafana Cloud. He is joined by Developer Advocates Marie Drake and Nicole van der Hoeven to talk about getting started with k6 for free.

Changes to Grafana Plugins in Grafana 10 (Grafana Office Hours #03)

Thu, 29 Jun 2023 23:29:35 +0100

Grafana 10 is fresh off the presses, and it includes some important changes to Grafana plugins that any plugin developer or user will want to know. Senior Product Manager David Harris chats to me about all the new features available to plugins (including the new Scenes framework) and some breaking changes to legacy plugins.

From Load Testing to Reliability Testing Tool (k6 Office Hours #92)

Fri, 23 Jun 2023 23:43:31 +0100

In this k6 Office Hours, Developer Advocates Nicole van der Hoeven and Marie Cruz talk about the evolution of k6 from a load testing tool to a reliability testing tool. They also demonstrate how to combine various modules and extensions with k6.

How I Structure My Obsidian Vault (Obsidian Tour 2023)

Fri, 23 Jun 2023 18:00:09 +0100

Here’s how I structure my Obsidian vault using tags, folders, links, metadata (used in conjunction with Dataview queries), and bookmarks. This is an Obsidian tour of my main vault, and we’re starting with organization. As with anything on this channel, don’t take what you see here as gospel. I’m just sharing my approach in case it sparks ideas for some of you.

Do Appearances Matter in Tech (Adobo & Avocados #05)

Fri, 23 Jun 2023 15:03:33 +0100

My fellow Developer Advocate Marie Cruz and I talk to Imma Valls, a community support engineer who works with us at Grafana Labs, about image biases that people might have, whether that’s piercings, tattoos, or hair dye!

What Is Grafana Scenes? with Torkel Ödegaard (Grafana Office Hours #02)

Mon, 19 Jun 2023 23:03:43 +0100

In GrafanaCON 2023, the latest version of Grafana 10 was announced, and with it, Scenes. But what is Grafana Scenes? Torkel Ödegaard, creator of Grafana and co-founder of Grafana Labs, joined me to discuss what Scenes is and how you can use it to create dynamic dashboards like never before.

Taking Notes for Work With Obsidian

Fri, 16 Jun 2023 19:00:33 +0200

In this talk, I show why I think taking notes for work with Obsidian has not just helped me learn faster but improved my career working in tech as a Developer Advocate and performance engineer. I take notes continuously as a way to keep my skills relevant. This is a rehearsal of a presentation I gave at EuroSTAR 2023.

Testing Kubernetes Networking With k6 (k6 Office Hours #91)

Fri, 16 Jun 2023 13:00:50 +0200

How would you go about testing Kubernetes networking? For his initial project with k6, our new Senior Software Engineer, Roberto Santalla, talks to us about how he used k6 to improve end-to-end network throughput and some insights he learned about Kubernetes networking along the way. I also invited my colleague, Leandro Melendez, to help me interview Roberto.

What's New in Grafana 10? (Grafana Office Hours #01)

Fri, 16 Jun 2023 12:00:25 +0200

What’s new in Grafana 10? I interview Grafana Labs Product Director Mitch Seaman about all the new features in Grafana 10, including Scenes, correlations, some important plugin development changes, and improved security.

The Lost Art of Taking Good Notes (EuroSTAR 2023)

Wed, 14 Jun 2023 08:00:21 +0200

Today I gave the talk “The Lost Art of Taking Good Notes” at EuroSTAR 2023, about how I use Obsidian for work and why creating a personal knowledge management system is so valuable in the career of a tester.

Here are the slides.

How to use k6 for site reliability engineering (k6 Office Hours #90)

Fri, 09 Jun 2023 18:32:32 +0200

In this k6 Office Hours, our k6 Champion, Sarah Zipkin, chats to the Grafana Labs Developer Advocates on how a Site Reliability Engineer uses k6 to test for reliability.

Text Expansion in Obsidian With Espanso

Fri, 09 Jun 2023 18:12:41 +0200

I’ve gotten a few questions asking me how I do text expansion in Obsidian, because there are a lot of text expander plugins out there. The truth, though, is that I don’t use any of them. Instead, I use a separate, free, open-source app called Espanso to define text triggers and replace them with certain phrases. One of the best things about Espanso is that it stores all configuration in plain text files.

10 tips for starting (or progressing) a career in tech (Adobo & Avocados #04)

Wed, 07 Jun 2023 08:00:25 +0200

Marie Cruz and I did another Adobo & Avocados livestream, this time about 10 tips (five from each of us) about starting a career in tech.

How Supabase does performance benchmarking using k6 (k6 Office Hours #89)

Fri, 26 May 2023 16:19:51 +0200

In this k6 Office Hours, we talk about how Supabase does performance benchmarking using k6 with Egor Romanov, SDET at Supabase; Théo Crevon, Senior Software Engineer at Grafana Labs; and Nicole van der Hoeven, Senior Developer Advocate at Grafana Labs.

Supabase is an open-source alternative to Firebase. Learn more about Supabase here.

My Remote Work Setup While Traveling for a Month

Fri, 19 May 2023 18:00:14 +0200

In this video, I go through everything that’s in my remote work setup while traveling for a month in 5 different countries– all in one backpack and one sling. I work remotely full-time, and in this trip I have three talks in one month, several meetups with friends and family, and livestreams to do. Here’s what I chose to bring with me to remain fully operational while also travelling.

For a list of everything I mentioned in this video, check out my Gear list.

Load testing and tracing with k6 and Tempo

Thu, 18 May 2023 22:41:42 +1000

At ObservabilityCON on the Road 2023 in Sydney, Australia, I talked about why you should use load testing and distributed tracing, how to do it with k6 and Tempo, and the benefits of being able to see beyond traditional black-box testing.

Resources

How to Avoid the Most Common Kubernetes Monitoring Mistakes

Tue, 16 May 2023 22:41:16 +1000

For ObservabilityCON on the Road Singapore 2023, I talked about some mistakes people make when monitoring applications within Kubernetes clusters. I also demonstrated Grafana’s Kubernetes Monitoring solution and how it can be a shortcut for engineers looking to get started quickly.

Resources

Slides here
Monitoring Kubernetes in general (notes)
The Grafana Kubernetes Monitoring feature (notes)

More Than Just Taking Notes: Learning Exhaust

Fri, 12 May 2023 21:18:45 +0800

For me, using Obsidian is about more than just taking notes– it’s about creating learning exhaust. Learning exhaust is the byproduct of real learning, and thinking of my notes as learning exhaust has helped me shift my focus from taking good notes to learning new things well.

Intersectionality in Tech (Adobo & Avocados #03)

Wed, 10 May 2023 21:35:00 +0800

Marie Cruz and I talk about what intersectionality is and how intersecting identities can affect how we relate to the world and our experiences.

Tagging and Linking With AI: Napkin

Fri, 05 May 2023 00:58:39 +0800

In this video, I demonstrate my workflow for tagging and linking with AI using the Napkin.one app. Napkin is an app that uses a fork of GPT-3 to intelligently tag thoughts with keywords that improve over time. I like to use Napkin to tag mostly my highlights from Readwise.

Resources

Napkin
Readwise
The Lost Art of Taking Good Notes, my talk at EuroSTAR 2023

Getting started in testing (Adobo & Avocados #02)

Wed, 03 May 2023 21:22:18 +0800

Marie Cruz and I interviewed Rafaela Azevedo about her experiences in getting started in testing, what it’s like to be a woman in tech, cultural differences between Brazil and the UK, and how she became a founder of a new Blockchain startup.

Emergent Note Taking

Fri, 28 Apr 2023 18:21:15 +0200

Emergent note taking is the process of turning individual notes into something more than the sum of their parts and combining them in different ways. In this video, I talk about how to take notes to be found and how to explore your vault in ways to help ideas emerge in Obsidian.

Resources Mentioned

Emergence, by Steven Johnson: book that prompted this idea (book)
Zsolt Viczián - Book on a Page about Emergence (video)
Visual atomic note taking with ExcaliBrain (video)

How to Get Core Web Vitals in k6 (k6 Office Hours #86)

Fri, 28 Apr 2023 16:11:52 +0200

In this k6 Office Hours, we discuss how to get Core Web Vitals for your web app using k6, and how it can help you test for user experience. We are Nicole van der Hoeven and Marie Cruz, and in this livestream, we’re talking about frontend performance testing with Grafana k6.

Obsidian Bookmarks (new in Obsidian v1.2)

Thu, 20 Apr 2023 10:37:51 +0200

Obsidian bookmarks have arrived in the latest Obsidian version, v1.2. Here’s what they are, what you can bookmark, and how to use them within your workflows. I also go over other, smaller Quality of Life improvements in this release.

Resources mentioned

Emergent Load Testing - rules for organized chaos

Wed, 19 Apr 2023 00:21:06 +0200

Today I spoke at KubeCon EU 2023 about emergent load testing.

Resources

Here are the slides.

Below are some links to extensions and experimental modules that I mentioned:

Grafana
k6 site
k6 repo
k6 docs
k6 browser (experimental module)
Prometheus Remote Write (experimental module)
k6 tracing (experimental module)
xk6-kubernetes (extension)
k6 redis (experimental module)
xk6 disruptor (extension)

Abstract

When we write load testing scripts against our applications, we write them sequentially: A, then B, then C. But this doesn’t accurately reflect the organized chaos of a system in production, nor does it prepare the system for the unexpected.

Emergence is a phenomenon where parts of a whole independently develop properties not originally present in the whole. Emergence is what helps ant workers develop roles without leadership, prompts animals to evolve adaptive traits without forethought, and facilitates non-toxic communities without moderators.

The growing field of emergent software applies this swarm logic to the programs that we write. What would it take to write emergent load testing scripts? It turns out that there are a few ingredients for emergence: a large population size, opportunities to interact, feedback, and an element of control.

In this talk, Nicole van der Hoeven discusses how to bring these elements to load testing by writing a script in Grafana k6 that can independently decide what requests to make next, modify Kubernetes app pods, and disrupt services based on a continual feed of results during runtime– all without manual intervention. She shows how to wield this new breed of load testing to improve confidence in the complex systems we build.

Organizing PDFs in Obsidian

Fri, 14 Apr 2023 08:01:11 +0200

There are many tools for organizing PDFs in Obsidian, but which one is the best? In this video, I go through several options, including DevonThink, Readwise, Skim, and Zotero. For my (non-academic) use case of organizing my TTRPG-related PDFs, I decided on Zotero– but maybe something else will be better for you!

Resources

Tools

Zotero plugins mentioned

Support MCDM on Patreon

Thanks

Thanks to the following people for their input and help in experimenting:

Andy Polaine
Leah Fergusson
@argentum#3920 on Discord
The lovely folks on my Patreon Discord server

Lazy GM Note Taking With Notion and Obsidian

Wed, 12 Apr 2023 12:36:50 +0200

How does Mike Shea (Sly Flourish) do lazy GM note taking with Notion? Mike shows me how he uses Notion to prep and run games, and I’m taking the opportunity to show him some of how I do the same in Obsidian. Mike is the author of several RPG-related books, including the Return of the Lazy Dungeon Master and his latest, The Lazy DM’s Companion. Mike has been playing RPGs for almost 40 years and is a forever DM. He also has a YouTube channel where he releases two videos weekly: one about news in the RPG community and one where he preps live for one of his games.

About Sly Flourish

Taking Notes on People in Obsidian

Fri, 07 Apr 2023 13:16:27 +0100

Taking notes on people in Obsidian sounds weird, but I think it just shows how much I care. If I take notes on things that matter, why shouldn’t I take notes on people that matter? Here are the templates I use to remember important things about significant people in my life.

k6 Javascript Ecosystem (k6 Office Hours #85)

Fri, 07 Apr 2023 13:00:42 +0100

Did you know that there is a k6 JavaScript ecosystem on top of the Go extension ecosystem? What’s the difference between the two? What does k6 actually support? What types of things can you do with JavaScript? In this k6 Office Hours, Software Engineer and k6 maintainer Mihail Stoykov joins me to talk all about the different options for k6 available through JavaScript libraries.

Frontend vs. backend: How to plan your performance testing strategy

Mon, 03 Apr 2023 13:20:53 +0100

Originally posted here.

There are many aspects of application performance, but they broadly fall into two categories: frontend performance and backend performance. As a tester, it’s important to know the differences between the two and how that impacts the way you approach your tests.

In this blog, I’ll provide a high-level overview of frontend performance testing and backend performance testing, including pros and cons of each one. I’ll also explain why you need both if you really want a holistic view of how your application is performing.

Why (and where) to use frontend performance testing

Frontend performance testing verifies user experience at the interface level. Basically, this entails everything a user sees when they access your web app, whether it’s on a laptop or other devices. This might involve automating user interactions with the site through the browser — for example, clicking on a button, filling out a form, or having scripts that load on the browser and run.

In frontend testing, you’re looking at how components of the UI respond.

The metrics we analyze for frontend performance tests are typically quite different from what we use for backend testing. Let’s say we want to know the time it takes for a page to render after a user clicks on a button that directs them there. This isn’t the same thing as measuring server response time; that total round trip takes a lot more into account — things like latency or the time it takes for a client-side script to run and display the requested information.

Advantages of frontend performance testing

The great thing about frontend performance testing is that it provides a more realistic measurement of your end users’ experiences. They don’t care what’s going on under the hood; they care about the overall experience — and that’s exactly what frontend performance testing measures. Unlike backend performance testing, frontend performance testing looks at everything from a user’s point of view: from the time they visit a webpage to the time they can interact with something on that page.

On top of that, most (about 80%) of the performance issues end users report are frontend issues. It doesn’t make much sense to skip this type of testing; it can be a critical step in finding ways to improve the overall user experience.

Disadvantages of frontend performance testing

The most obvious disadvantage of frontend testing is that it doesn’t look under the hood. If you only do frontend testing, you’re almost certain to miss something. Maybe a component isn’t working, which is creating issues for your users. But you wouldn’t know that because you’re only testing what’s visible from the browser. In other words, you might be in the situation where you know something in the application is wrong and responding too slowly, but you don’t know what.

Frontend performance testing can also be quite expensive compared to, say, protocol-based testing. That’s because you need enough resources (memory and CPU) to actually spin up a browser instance and interact with your application the way that a real user would. The same load generator that could run tens of thousands of backend virtual users may only be able to run 50 frontend virtual users before running out of resources.

Finally, only testing frontend performance can sometimes lead to pretty misleading results, especially as you scale. As load increases, the response time for the frontend remains fairly constant, but that’s not the case with the backend.

While the frontend component of response time remains more or less constant, the backend component of response time increases exponentially with the number of concurrent users.

This phenomenon occurs because most of the issues that frontend performance tests might identify, such as a slow-running client-side script, don’t tend to compound. It’ll be just as slow on each user’s computer regardless of how many users are running it. However, problems uncovered by backend performance testing are on the side of the application, not the users. If a database can’t keep up with 100 users sending queries to it, then 1,000 users sending it queries is likely to yield an even worse response time.

Why (and where) to use backend performance testing

As you might have guessed at this point, backend performance testing targets the infrastructure under the hood. This could be a database that contains customer information, a payment gateway, or an authentication service for customers that have already signed up. These are things that an end user doesn’t see and maybe isn’t ever even aware of, but they still have an effect on response times.

Backend testing addresses infrastructure users might not even be aware of, like different databases, gateways, and services that underpin an application.

Scripting against the backend is quite different from scripting against the frontend. Instead of scripting user interactions like clicking, you’re scripting certain requests that are being sent on the protocol level. So, for example, you could have an HTTP GET request for the page and also for its embedded resources.

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
 http.get('https://test.k6.io');
 sleep(1);
}

The sample Grafana k6 script above is an example of a simplified backend performance test. It makes a single HTTP request to retrieve the contents of https://test.k6.io, and then waits one second.

Backend performance includes things beyond just load times and latency. It includes scalability, availability, reliability, and similar attributes, so the metrics that we measure are different as well. We might look at server uptime or response times per component. We could also trace the path of a particular request through the application stack.

An illustration shows that performance testing can address scalability, elasticity, availability, latency, resiliency, and reliability. Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Advantages of backend performance testing

As we already discussed, frontend testing doesn’t cover everything, but it’s also more complicated than that. The 20% or so of performance bottlenecks that are backend-related can be the hardest to troubleshoot, identify, and fix. Even worse, a lot of backend performance issues can spiral out of control at higher levels of load and can be quite expensive to fix later on in a development cycle. The worst time to realize that you need to rethink your entire infrastructure is after it’s already failed in production. If you take care of backend performance testing earlier (commonly known as shifting left), it makes such pivots less costly.

Another advantage of backend testing is the ability to target different components of an application. You can choose which API endpoint to hit and how hard, and you can target specific user flows, so it doesn’t always have to be end-to-end testing.

And finally, backend performance tests tend to be less resource-intensive; they’re just protocol-level requests that can be easily ramped up.You don’t need to spin up a browser instance, or even parse through the Document Object Module (DOM) of a webpage or look for specific elements. It’s just a matter of sending these requests, and that’s something that you can increase the scale of pretty cheaply.

Disadvantages of backend performance testing

Backend performance testing won’t tell you anything about what’s happening on the frontend, and a lot of sites these days actually use scripts that have to run on the frontend. So with backend performance testing tools, you just download those scripts. But the response time they measure is the download time, which is not really realistic because those scripts are executed. A backend performance test would fetch the text of the script. A frontend performance test would run it — and that script might do something like perform a calculation. You need frontend performance testing to measure that properly.

Backend performance testing is also much more expansive than front end performance testing, especially with microservices-based applications. There are so many components to track that it can be difficult to determine where to look. You can quickly find yourself going down a rabbit hole of constant, never-ending performance tuning.

Frontend + backend testing for the win

So, as we’ve seen, frontend testing and backend testing both seem to have advantages and disadvantages. So which one should you use? Both, if at all possible! They look at different parts of application performance, so doing both is essential if you want a holistic view of user experience.

Software teams can use Grafana k6 to address frontend and backend performance testing.

If you’re operating at a smaller scale, use your own discretion about how to find the right mix for needs. But if you’re testing at a larger scale with thousands of users and you want to test both the frontend and backend aspects of application performance, I suggest you change your test strategy so the majority of the load is generated with backend performance testing scripts. That’s because of the resource intensiveness of frontend performance testing scripts, which we discussed earlier.

So, if you generate most of the load with the backend and then you have a handful of frontend performance testing scripts running in the background, you should still get a good idea of how both are performing. Remember, frontend application performance remains fairly constant throughout different levels of load, so you’ll still get a lot of useful insights on both types of performance without worrying too much about the scaling costs.

Frontend and backend performance both affect what your end users think of your application. If the underlying application servers respond immediately to all requests, but then a huge picture of a crocodile makes your homepage excruciatingly slow to load, then users aren’t going to like that.

On the other hand, if your homepage is really sleek and quick to load, but the application server hasn’t responded with the data that the browser needs to render, then users aren’t going to like that either.

A common problem when testing both frontend and backend performance is script duplication: because of the differences in the two approaches, you often have to use two separate tools — which means two test suites and potentially two scripting languages. While this is a valid approach, it’s more sustainable to look for a tool that can target both types of performance, such as k6.

Whichever tool(s) you end up using, testing both frontend and backend performance is going to give you the clearest view of how your application really performs.

The easiest way to get started with k6 is through Grafana Cloud k6. The free tier includes 500 virtual user hours/month for k6 testing and more. Sign up for free now!

Taking Notes for RPGs in Obsidian with Andy Polaine

Fri, 31 Mar 2023 14:30:21 +0100

My friend, Andy Polaine, joins me to discuss taking notes for RPGs in Obsidian. Andy is the Dungeon Master for one of my weekly games of D&D, and in this livestream, we’re comparing our notes (his as a DM, mine as a player) for the same campaign, a homebrew one in the world of Exandria (from Critical Role). We’re showing the differences between our styles of note-taking and playing games.

Key takeaways

Andy’s DM style is improv-heavy and inspired by the Lazy GM, Sly Flourish. His notes reflect that philosophy.
Andy primarily uses folders for organization due to concerns regarding futureproofness, whereas I use links and Dataview and have no sub-folders.
Andy has Dice Roller tables for random generation, but he also embeds existing websites as iframes within Obsidian.
When a situation calls for many moving parts, monsters, or factions that the party might encounter, Andy uses an Adversary Roster to keep track of where everyone is.
Andy uses Canvas to display multiple statblocks at a time and be able to zoom in individually when necessary. He said he initially had performance issues with this approach, but those seem to be resolved now.

Resources mentioned

DM Heroes
RPG Tinker
TetraCube Statblock generator
Sly Flourish Lazy GM method for session prep
Knife Theory

Below is a transcript of the video.

How we met

NICOLE: Hello, everyone. Welcome to the channel. I’m Nicole van der Hoeven. You probably already know me, but today, I’m also Scura, the level eight Pallid Elf Twilight Cleric/Stars Druid. And with me today is a someone who should be a familiar face now. I think this is your third time back. Andy Polaine.

ANDY: Not sure. Second time on the stream, I think maybe, I’m not sure.

NICOLE: Okay, okay. But we already did a TTRPG thing together.

ANDY: That’s right. So it is the third time in the Obsidian DM world.

NICOLE: So Andy happens to be my DM, my Dungeon Master for my D&D 5e game. And that actually happened because of that presentation we did on Obsidian, right?

ANDY: It is, yes. We’re chatting on the Obsidian channel, the TTRPG channel. And at some point, I kinda said, oh yeah, well actually I need another person. But you probably, ‘cause you had said something like, I play 50 games a week or whatever. And I said, “You probably don’t need another game.”

NICOLE: It was like four.

ANDY: And you went, “Well actually.” So yeah.

NICOLE: Yeah. So you slid into my DMs and you’re like, “Well, just in case you need a fifth.” And I’m like, “Actually, I don’t have a weekday one yet. So maybe, yeah.” So let’s talk about that, because you definitely knew when you asked me to join your game that I took notes. What was going on inside your head? Were you like, “Yeah, I want someone to take notes.” Or did you not really know how much I would take notice of everything that happens?

ANDY: No, I mean, I wanted another player and I wanted someone who was into the same stuff and I thought you would be interested and a great player and you have been. It’s great to have you in the game. I mean, leading up to that, actually, I guess, I had played D&D when I was a kid, teenager, and then I stopped and then COVID happened and like a lot of people, I sort of thought, “Oh yeah, D&D that’s a thing.” And it was actually to talk to my daughter about it and try and persuade her to play. And then the internet happened between when I stopped playing D&D and when I wanted to start again. And so I was looking for ways to note take and have everything. And I’ve tried a few different things, including Notion and I didn’t really like Notion’s… The main thing was no offline mode, which was a real big thing for me. But I’m also like many people who use Obsidian, I’ve been keeping my notes for my work stuff in as plain text markdown for years. And so I thought, “Ah, I’m gonna, I’m gonna switch to that.” So very quickly, I did one game in Notion and I started actually the campaign that you’re currently play in Notion. I think the first few, maybe five or six sessions, and then I switched to Obsidian.

NICOLE: Oh, I didn’t know that.

ANDY: With all the pain that exporting out of Notion with like these horrid, long hex or these database codes attached to all the files. I’ve still got some of that junk occasionally appears in my auto completes.

NICOLE: Yeah, I can imagine. I think by that point, I’d been using Obsidian for a while, but I had just started to use it a lot for TTRPGs ‘cause you know, I was gonna use it for like serious stuff like work. and that changed very quickly. I know. Hey, do you have dice with you?

ANDY: I do have dice I do have dice.

NICOLE: Let’s do rollies again.

ANDY: Okay, alright.

NICOLE: I got my-

ANDY: You better explain what rollies is for people who may not-

NICOLE: It’s just rolling a d20, seeing which one’s higher ‘cause I have a question from Patreon. And the person who gets higher is going to decide whether they answer first or the other person answers first.

ANDY: All right, okay. All right, you ready?

NICOLE: Okay, yeah.

ANDY: 16.

NICOLE: Oh no. Natural one. Okay, you decide.

ANDY: Okay, okay. I decide whether, well, I don’t know what the question is. So that would-

NICOLE: No, you gotta decide if you wanna go first or me.

ANDY: Well ladies first.

NICOLE: Oh, great. Okay, so the question is for me, what GM are you? And for you, what player am I? Andy is a very empathetic GM. He listens and he listens to not just the stuff from the character but also your stuff. And maybe this is something where we should talk about knife theory ‘cause Andy’s particularly good at remembering things that draw you into the story, your character and you. And I think he’s mentioned quite a few times that he likes slightly messing with us all, so.

ANDY: Yeah. So, okay, so what player are you? Well, you’re great. You are actually very empathetic player and you play a cleric, right?

NICOLE: That was my thing.

ANDY: Yeah, yeah. And so, no, what I used to say is that basically, Scura runs around healing everyone way too much from my liking as the DM. But Scura does a lot of, if you like a hit point machine, just of vending machine, like here you go, you have hit points and you have some hit points. And so you’re very engaged on that front. You’re very engaged in the role play. And thanks to Obsidian, Scura has got like a sort of photographic memory. So you have… And you, Nicole, use it I think to kind of mess with me as well sometimes too, which is, usually through sending-

NICOLE: No, I would never.

How I use Sending to mess with Andy

ANDY: Should we talk about sendings here?

NICOLE: Yeah, I could.

ANDY: I’m assuming here that everyone knows about how D&D works and stuff. So I don’t know if this is true, but you might all be Obsidian people just interested in this. For those of you don’t know, there’s a spell in D&D. It’s called sending where you can send 25 words to someone sort of telepathically over distances. So tell me what you do with that.

NICOLE: Well, I actually have something to show on that front.

ANDY: Okay. All right.

(recorded video plays)

NICOLE: I’m gonna play a little clip that I got permission to show. I’m going to send to, I’m not sure what her name is. Sabilis, Sybilis.

ANDY: I don’t who.

NICOLE: The drow priestess that wanted Wally to come with her to Xhorhas.

ANDY: Oh my God. Let me just check who that might be.

TYLER: Great pull. Great pull.

ANDY: That is like, I gotta choose this, this really random that you haven’t thought about for ages.

(back to livestream)

NICOLE: So that was a clip from our actual game where I pulled out a particularly unexpected sending. I don’t know why it was unexpected really, because my character had been try trying to chase down this thread for the other character. So I don’t know why I was unexpected, Andy.

ANDY: Well…. There’s so much that’s unexpected. So I would say there was a couple things actually about that group. So I mean, it may be interesting on the note front for people. So if there’s an important NPC, and I’ll show some stuff in a second, if there’s an important NPC, I will usually make a note for them. But sometimes as in this case, there’s like a group of NPCs. In this case there was a group of drow, they’re called the Kryn in Exandria. We play in the critical role Exandria Wildemount world. There was one of those times where I didn’t have much prep time and I thought, “Oh, I just need to come up with some names and stuff. And so I think I probably just pulled some stat blocks form somewhere just in case you had a fight with them. But actually, the names, I just had them in a little list and there’s a block and like a one line description. I think you might see some in a minute too. And so they didn’t have any notes made for them. And so when you said that, and I’m looking through my dashboard and stuff and I think I can’t think who that is at all. And the name didn’t stick with me at all, so yeah.

NICOLE: Yeah, you know what’s weird is I actually wasn’t there. I wasn’t there when they met her. I was going off of the recording, this is when I think I briefly came in, I dialed in from the airport. So I had very limited video to go on because not everything was recorded. And so I wasn’t actually sure, that’s why I didn’t know the name. But I can show you how I look at how I decide who to send to. So I’m using the new bookmarks feature. I actually have one for sending, in particular. These are all the sendings. There’s a tag now that I do. So like for example, I didn’t change any of my notes so I hope nothing comes up. But this is how I do it. So I have the sending tag, I have the name of the person and then what I actually sent. And that’s easy because then I can highlight it and I can see, “Hey look, it’s 25 words, which is the limit of the spell.” And then when I go into a particular NPC, I also later put down all of the sendings I’ve ever sent to them and their response back. So this particular NPC, I have apparently reluctantly sent him messages a few times. But I also have an agenda page. So I have some sendings planned already and I go down the list. And I also have, this is a Dataview query that it’s a Dataview JS query. And it’s all like random NPC. So there is an element of randomness ‘cause sometimes, these are the ones that I know I want to send to, but these are the ones that I’m like, “Huh, I wonder who Rolf is and what they’re doing today.”

ANDY: It’s annoying.

NICOLE: I have-

ANDY: It’s in a good way, it’s very funny.

NICOLE: Annoying in a good way. That’s me. I have a follow up list. These are the people that I like to talk to often that I’ve sent more than one sending to. They’re like important to my character or to the group or to the story. And I’m also using Supercharged Links by the way. That’s why all of these icons are there. So the rainbow is like a PC and the other one, which is like a vampire actually. It’s like an NPC. So that’s basically it. I have ideas of whom to send to. And they’re also based on the quests and questions that we’ve got. ‘Cause these are like, some of them are just for me, for my character and some of them are for the group. I don’t know. I find sometimes that I’m the one that’s like, other people in the group are like, “Hey, what are we doing here again?” I’m like, “Okay, this is what we’re doing.”

ANDY: Scura would definitely have glasses or something and be the one that goes, the one who’s the character who is most likely the brainy, one of the group who actually knows everything’s going on and pulls everyone else into line on that front.

NICOLE: Yeah, I actually rolled a journal for her in her starting items.

ANDY: That’s perfect. So that’s handy, A magical journal where you can interlink different pages.

NICOLE: I know. I actually purposely never play a character with low intelligence because of it. Because I don’t think it’s realistic. We have a question. Oh, by the way, there’s someone, Mariana Bergo, who said:

What a lovely collision of worlds. I follow Andy’s work on service design but had no idea he’s a D&D guy as well. That’s awesome.

ANDY: I have in mind to do some crossover stuff actually about this, about some of stuff I’ve learned from RPGs too and managing teams and design leadership and stuff and vice versa. So we’ll see if that emerges. So what’s the question?

NICOLE: Courtney Robertson says:

Does your group share notes? Curious if the group might use GitHub or some other means to sync?

Okay, so no. Well, I publish my notes and sometimes, I’ll post my notes especially if somebody was away that week or something. I’ll publish my notes. There’s another person who we got into Obsidian and he doesn’t publish it though. He will just often like export to PDF from Obsidian. And Andy of course, can’t share his notes with us.

Andy’s DM style

ANDY: Yeah, I think a couple of them actually, a couple of others too. So I think three out of the other four use Obsidian and what state their notes are in. So I have to say it’s incredibly handy for me. So as the DM, I really do make up quite a lot. I mean, I think people forget… It maybe just my style. So one of the things you would ask me, what am I like as a DM? I follow the Sly Flourish’s Lazy GM format. So I prep kind of… I think about it a lot. I caught myself this week actually thinking about some stuff like all week. And in fact, the session that we just had on Tuesday, there was a little bit of trolling I did to Nicole’s character that I-

NICOLE: A little.

ANDY: Had been thinking about the joke basically, and the setup of it and just crafting it all week. But then generally, I spend maybe an hour, two hours maybe. And in fact the one we’re about to have a look at, the session we’ll look at in a second, I spent probably a lot of time. Over a couple of weeks, I probably spent about four hours maybe prepping this, plus the map. But usually, I spend less than that. But then, I did that classic thing where I sometimes I prep very little and I’m terribly worried. And literally, so this group is an interesting group in that they’re violent pacifists. They’ll sort of avoid combat and then chop people in half. Once they get triggered, it goes violent. And they’re quite a suspicious group, so quite a suspicious party. So we’ve had sessions where it’s literally taken like half an hour in real time to decide to go through a doorway or something. And I think this isn’t unusual actually in other parties. So sometimes that happens and I think I prep all of this stuff and I think, “Oh God, I’m not prepped at all.” And literally we get through about 20% of it. And sometimes, we blast through it or they desire to do something else. And there of course, I have to improvise an awful lot. Should we have a look at some notes?

NICOLE: I take pleasure at that happening. I wanna ask you first, how many notes, yeah, how many notes do you think I have for this campaign?

ANDY: Oh, I don’t know. It’s gonna be 1,000. No, I don’t know, 800. I don’t know how many, like including other stuff or just the campaign itself?

NICOLE: Just the campaign. Not including pictures.

ANDY: 700.

NICOLE: Less actually.

ANDY: Oh, okay.

NICOLE: It is 486 notes.

ANDY: It’s still quite a lot.

NICOLE: It is but it’s 133,784 words.

ANDY: Wow. It’s telling me I’ve got, well, 1,915 files. But actually, the DM notes come up to 321. And here’s where all the Obsidian people fall off the 102 folders. So I might want to explain-

NICOLE: Oh God.

ANDY: -why that is. Oh God, you use folders? You heathen.

NICOLE: Heathen!

ANDY: So do you wanna have a look?

NICOLE: Do you know bookmarks can- Yeah, yeah. Do you wanna add it or do you want me to do that?

ANDY: No, I’ll do it. I’ll do the same. Bookmarks can what?

NICOLE: Bookmarks can actually on my version, bookmarks can bookmark a folder.

A tour of Andy’s DM vault

ANDY: I can do that too. I can do that too. Yeah, it does work.

I’m gonna show people around my vault a little bit. Not too much. And you’ll see I’ve got the episode title here and it says sanitized ‘cause there there was like two secrets that you’re not allowed to see.

DM reference resources

So I have a bunch of DM resources here. I have the raw reference here, the SRD plus other stuff I’ve added. I have this thing which I don’t really use anymore, which is called default tabs. So I used to use it quite a lot where I’d pull up some generators and cheat sheet and stuff. I don’t really use it anymore. And in fact, I think I’ve lost my DM screen, but I’ll put it back up there. I have a DM screen, which is set up as a list. I used to have it as a Kanban, but I prefer it in a list ‘cause I can just sneak it in in that left-hand column. So I have two keyboard hotkeys set up, which is to hide the two sides and I use them all the time both in work and also DMing. So this is quite handy.

And the other thing plugin I use a lot is the hover one. So I’m really quite often doing this thing where I’m just checking something here. And so, that’s how I’m doing that. So back to this.

Random tables

So I have a bunch of random raw tables. I don’t use these so much. The one I use a lot is the NPC. So I have a big random roller, which is like, pulls from a lot of stuff. It’s got a button at the top and it’s pulling from a bunch of lists. I think some of these will break, but it’s also got things like sort of encounter some of these encounter roll tables and stuff that I’ve got around in my vault. So if I just go, I’m going to… So I re-roll that. And I can quickly find an NPC and a name. Something’s a bit weird I think with the name there actually now it’s pulling, it should be first name, second name. And they’ve got some stuff here and I sometimes re-roll that bit. Always drinking over by under. Sometimes, that’s handy. And occasionally, if you’re wandering around, you will meet some of this stuff. Probably less so. I sometimes might use it when I’m writing bits of an encounter or I might trust, okay, travelers on the road do they meet someone or a roadside encounter. But I actually don’t use this as much as I thought I would.

What I probably use most are things like these things which are iframes. So there’s a website called RPG Heroes. I use this all the time. So when you go, “So do I see anyone to ask directions?” And I’m like, “Oh yeah.” And then I go, “Yes, there’s a guy with a long beard and a cloak and stuff.” ‘Cause I like the visual. And I quite often ignore the rest of it, but sometimes, it’s quite useful. This one is useful for stat blocks. It generates stat blocks as well. So RPG Tinker is good and there’s a bestiary version of it too. And sometimes, use Tetra Cube one, but I mostly use those or I have quick name lists as well.

So in game, that’s what I’m doing. All the other stuff is mostly for adventure writing if I’m doing stuff. And there’s more of that in there. Now I’m seeing amount of PDFs and stuff in here.

I have some charts and tables like price list I use quite often. We’ve just had a couple of shopping episodes. I have a price list encyclopedia here and when someone goes, “Oh, I wanna go and buy a crossbow.” And I’m like, “Crossbow. Oh yeah, yeah, well that’s gonna cost you.” Let’s just go through hand crossbow, that’s 75 gold. So I’m not spending ages searching around in stuff for this.

So that’s most of the stuff I’m using in game. And there’s obviously sort of monsters and stuff like that. I have them in folders partly because this existed before Obsidian and I like to have my stuff in folders on the desktop. So when I don’t have Obsidian handy or maybe one day might use something else whatever, the stuff is organized. I don’t want to become so dependent on say on Dataview that I can’t use all the stuff in my folders or tags for that matter. I do actually just started tagging stuff on macOS in Finder.

DM notes

My actual DM notes are in Wildemounts - Tuesday Wildemount DM notes. I didn’t change that ‘cause this is like ripple across so many links. So I’ve got a previous sessions note and I have a current one, and I make a folder. And the reason why I have it in folder instead of just a note is ‘cause sometimes, I have sub bits of a plot arc. And particularly, if I’m borrowing bits from an existing adventure like a published adventure, I’ll have them in here and the bits of that are actually in that session’s folder.

And I do it as separate notes because then, what I can do is if we don’t get through it all, I can just move the bits that we haven’t got through into the next problem. So that’s why I do that. So that’s my current session always.

I do have a previous session lister. So this is a Dataview query that is doing that. And I can go through an hour, in fact hour, and sometimes I have a few other bits and pieces. I have some scratch stuff so you’re not seeing this but these are sort of things I’m thinking of that might come up. They end up sort of in there. So I have adventures, I’ve got some different quests. I’ve got some extra lore stuff in here. I’ve got some specific images to this world and items and stuff. And then I have locations.

Location dashboard

In every location, so we are gonna have a look at a place named Shadycreek Run. Every location has its folder, there’s some hooks, plot hooks in there, there are locations in there, there are maps and there are NPCs. And that’s how I structure it. And then I usually have a dashboard. So this one here is the Shadycreek dashboard I’ve actually gotta open up here.

And that is a bunch of… That’s a map using the leaflet plugin. I have a bunch of Dataview things that are pulling stuff out that’s tagged as locations, NPCs and so forth. And back to my point that I don’t wanna become so dependent on Dataview that I can’t live without it, that’s what the folders are for, right?

Stuff in here is tagged. And so if I didn’t have Obsidian, if I didn’t have Dataview, I can still go and find all those NPCs and stuff ‘cause they’re in that folder and I’m not having to hunt around amongst locations and other stuff as well. So that’s why I do it. But the dashboard is super handy. It’s particularly handy for when they go, “I wanna go and go to the blacksmith.” And I’m like, “Hang on a second, I have that table.”

NICOLE: We haven’t gone to most of these!

ANDY: Yeah, yeah, we have. So then I’m like, “Oh yeah, blacksmith. Oh I want to go to a shop.” And I think with this one here, so I’ll often just gonna pull up there. I don’t know why that’s looping. It’s into like some ridiculous nested thing there.

So the Jezean Emporium for example, they went there. I pretty much, that’s very weird. That’s doing that. What have I got down going on there that is making that nest? I think that’s a weird artifact. I don’t know why, maybe it’s this. What this is here, this blank thing, I still haven’t come up with an image for the NPC who’s here. I’m pretty sure I decided that he was a taxidermist like, on the spot. So you went-

NICOLE: Really?

ANDY: Yeah, yeah. And then I decided he was this known taxidermist and I thought it was funny to have this taxidermist that was this little gnome. And he was in the middle of pretty disgustingly actually stuffing a worg and preserving a worg, which was this big creature. So I had this idea, it was like sort of getting inside. And he was-

NICOLE: I thought that was intentional.

ANDY: No, no. And then as I was doing it I was thinking, “Oh, this is quite good because one of the characters, this guy, Ionos Jagentoth, I had him collecting these sort of exotic monsters and stuff and keeping them in a dungeon menagerie. So he’s got this dungeon or he had this dungeon, we’ll come to that in a minute. That was full of these different creatures.

‘Cause I was looking through Kobold Press’s Tome of Beasts and I thought, “Wouldn’t it be really great to have just a whole load of monsters of different kinds in one place in a dungeon or in an area, they’re in cages and there’s a bunch of switches on the walls and the players don’t really know what the switches do.” And so I had a random roll table for the switches and I was rolling to see what they did. I can’t remember if I got you to roll. I think I’ve got you to roll. Which what the switch did. So sometimes, it like it opened one cage but there was a couple of switches that opened everything and it would’ve probably been-

NICOLE: Oh God we would’ve died.

ANDY: All the creatures come out. And I thought it was really nice and funny. I thought it was funny most of the time, I’m trying to think of stuff that was funny idea to have all these different creatures.

And they might start fighting each other and stuff as well ‘cause they were creatures that definitely probably would just like feed off of each other and things in the midst of this sort of chaos. Plus you had a fairly major boss character turn up at the same time. So that was the idea behind this. And anyway, so Kimbal Jezean who was the… And I am pretty sure the source for this by the way, was the very good Exandria Discord. It’s a very good community online.

NICOLE: Wait a minute, his potion prices are plus 10%, what?!

ANDY: Yeah, his prices, well because you are in– so Shadycreek Run, for those of you who don’t know, is like the Mos Eisley of Wildemount. It is a dodgy sort of outpost outside of the main empire. And so yeah, so there is a premium on the prices. right?

So I had some potions for sale here, which you may or may not have got. Burned down, thanks to Halkin. So one of our players, his character got petrified, he’s a wizard and so he rolled another character, which is a barbarian dragonborn. And who followed these sort earth goddess called Melora. Anyway, seeing these stuffed animals, he was incensed and basically paid someone to burn it down. There was a lot of that going on in this particular thing.

So anyway, the main thing that I wanna say about him is that a lot of the stuff about him being a taxidermist and well I just sort of thought, “Oh, that’d be funny.” And then I thought, “Oh maybe he’s delivering creatures, either live creatures or stuffed versions of it as trophies to the Jagentoths.” And I thought that might be a good connection.

And then I just came up with his character and his voice, which I’m not gonna do right now immediately.

NICOLE: But that was the best part about him.

ANDY: Let’s go like this. Oh hang, on a minute. I’ve just gotta go and get this slime off of my hands.

NICOLE: Oh it was perfect. Did you do that just on the spot?

ANDY: I did do that on the spot, yeah. And I regretted it ‘cause I-

NICOLE: Oh wow.

ANDY: After about five minutes of talking I was like. So that was it. So that’s how I then reuse something like that dashboard and the NPCs that are in it ‘cause when Nicole as Scura goes, “Oh I’m just gonna send sending to this person.” I’m like, “Yeah, who’s that again?”

Nicole’s note on Shadycreek Run

NICOLE: I wonder if I should show my Shadycreek dashboard.

ANDY: Well do you wanna show, yeah, maybe let’s switch ‘cause there’s more I can show you. ‘Cause I’d like to show you the Jagentoth compound.

NICOLE: Ooh, we’re switching.

ANDY: But you go to yours first and see where you got to.

NICOLE: Okay, so this is my note on Shadycreek Run. I also have the Leaflet thing. I think you gave us this. And then the places that we have been to are already marked on there. So when I click on them, it goes to the note. I do also have Dataview queries. These are the locations that aren’t shops. I have factions because a big part of Shadycreek Run is that it’s ruled by these families. And so I have notes on each of them like, here’s the Jagentoth family. People that they’ve killed, businesses. I even have like a little diagram of their family tree.

ANDY: It’s more than I have, I can tell you.

NICOLE: I think I asked you and you’re like, “Uh, yeah, Reese. Reese is his name.”

ANDY: Yeah, I don’t know what I thought. No, I think some of them are in the book.

NICOLE: Oh, okay. And then also I have Dataview query that lists everybody that’s on there. So every NPC I put their family if it’s relevant as a faction as well. And then some of these have skulls, that’s ‘cause we killed them.

ANDY: Yeah, quite a lot of them.

NICOLE: Yeah, so here are all my NPCs from Shadycreek Run. Some of them are dead. And I have a little description of them. I mean, sometimes it’s really like, Sybastien: doesn’t like Wally. Sometimes, it’s very vague. And then here are the shops in Shadycreek, ‘cause that’s like a separate thing. Kimbal said I might get calming tea here which I did actually. And then I have sessions. So I think there was one point when we were like, “Oh, when was that? What session was it that we were last here?” I’m like, “Oh it was on 37 or whatever.” But my numbering is different from Andy’s by the way, just because I came in like after 17 sessions. So I’m 17 sessions behind. So that’s my Shadycreek Run one.

ANDY: Next week with session 83. So it’s been going for two years. Actually just around this time, I think April, it was the session zero.

NICOLE: Yeah. So should we go into a particular session that you-

ANDY: Let’s do that. Yeah, let’s do that.

NICOLE: Okay, well… Let me ask first though.

Background for the session Andy chose to show

NICOLE: Why did you really choose this session and why was it to troll me?

ANDY: Okay. So I did not deliberately troll you. So there was a significant event with an NPC from Scura’s backstory in this episode. And I actually thought it had happened in the previous episode, session.

NICOLE: Still not convinced.

ANDY: And I chose this one because it was one that I had prepped sort of quite a lot for it. It’s got some, I think some maybe some interesting prep stuff in it in terms of the way I went about structuring it. And it was interesting on because there was some extra bits that I did, which I then have probably then carried on and we’ll probably continue in the future because it was quite complex. There were a lot of NPCs. So the session is, they go to this place they’re called Jagentoths, they go to their compound, it’s this house set in the forest surrounded by the perimeter fence. And there’s an NPC called, sorry, there’s an NPC called Floyd. And I’ll explain why I’m laughing in a second. His name’s Floyd Hemsworth.

And one of the other characters-

NICOLE: Crownsguard Douche.

ANDY: -Milla, had a dalliance with him. Crownsguard Douche, as you’ve christened him.

And there was reverse damseling. So basically they had to go and rescue him. He had been captured by this boss guy.

And so you’ve got the NPCs, a lot of NPCs cause there’s guards and there’s all the other characters that have been, they’re also in it. And this is a thing we talked about in session zero. There are some slaves in this that these are nasty people. So I really wanted to go, this is one of those things where I’m not going to have slavery, like the characters have the main party, have slaves, for fun and stuff. But these are bad people.

So bad that in fact the party decided to burn their whole place down and kill them all and rescue the slaves. So you’ve got the slaves, you’ve got the family, you’ve got all the guards, plus you’ve got this basement, this dungeon menagerie in the basement with all these different creatures.

So there’s quite a lot going on and it could have gone in a lot of different directions for me as a DM. So that’s one of the reasons why I prepped it. Fairly often, I don’t do that much prep.

Fairly often, I’ve got maybe three or four scenes, very bare bones sketched out what’s gonna happen. If there’s any encounter stuff, I’ll have a sort of encounter set up and that’s about it. And then kind of, I don’t know, I think I’m fairly good at winging it. I think quite often, I said I feel so unprepared and you guys don’t think so.

What was entertaining about this one is I was really prepared and I expected you to do a sort of heist thing and sneak in to the house, rescue, explore the house. I had all these kind of, I’m gonna show you. Let me put up my thing and I’ll show you.

Andy’s session prep

ANDY: So in the house. Here’s the session, So the session starts with, I have just have a reminder calendar links back to the Wildemount, I can never remember the names of the week and stuff. So we do a little bit of a thing. There’s an eclipse coming up. I don’t use this that much, but it just reminds me to, in fact, this is an old Owlbear link. I don’t even use that anymore. I use Owlbear-

NICOLE: Actually Andy.

ANDY: The new one.

NICOLE: I remember in the first session ‘cause I went back to the first session we played, that played with you and you weren’t using calendars, no one had asked about the time.

ANDY: No, that’s right. That’s one of the things that now, we do have a calendar actually. So now we have the Fantasy Calendar plugin. Thank you, Jeremy. And as you can see, our game moves pretty slowly. There’s like five sessions in the space that have happened in the game day.

NICOLE: I think we started in Quen’pillar though, didn’t we?

ANDY: Yeah, I think you did. But I don’t remember. I don’t think I started adding to the calendar until more recently. So in this, I’ve made myself a little note to remind about the X or fast forward cards session zero thing. Because of the slavery thing and because one of the guys is the main guy, the main bad is a torturer. I wanted to just remind people if this is something they don’t want, they can skip. So then there’s a recap and then I have a few DM notes. So these are basically things I mustn’t forget to open or I know that I’m going to wanna go back and have a look at. I usually list all the NPCs out and this is all done manually.

NICOLE: Who is Phespo?

ANDY: Yeah, who’s Phespo? That’s a very good point. Oh, that’s the Medusa. You didn’t know? Did you not know her name? Her name is Phespo.

NICOLE: No.

ANDY: So there was a character, there was a Medusa that they faced a couple of sessions before this. And so that’s why she’s there. Just in case I have to go back and remember some of that. And then the rescue’s on. So there’s the Jagentoth. I called them Jagentoth, it’s probably Jagentoth but I live in Germany and for me the Js– Jagen is to hunt

NICOLE: Yeah, it should be Jagentoth, yeah.

ANDY: And so I’ve got these two people, I’ve got one note here. That’s the scene basically. They’re gonna dine around 8 PM. So that’s what I had in mind. They’re in the dining room at this point in time. You didn’t know this, did you?

NICOLE: No, I very nearly poisoned them. I had the midnight tears, ugh.

ANDY: Yeah, let me go to the compound because in the compound, I think I’ve got a map or have I? Haven’t got a map wherever I put it. I haven’t got a map. Somewhere, I’ve got a map. So what I’ve done here on the left-hand side, you can see I’ve broken out all of the rooms in the compound. So I’m sure I’d got… I thought I’ve got.

NICOLE: Wow, this is a lot.

ANDY: This is a lot, yeah. And so I had built this room, I built this mansion in Dungeon Alchemist.

NICOLE: Dungeon Alchemist.

ANDY: And what’s it called? What’d you say? It’s called Dungeon Alchemist.

NICOLE: Dungeon Alchemist.

ANDY: Yeah, yeah. And I’d furnished it all and all of these rooms have some thought behind it. There’s a little tree and there’s a kitchen here. And here is-

NICOLE: We didn’t see it all.

ANDY: No you didn’t. So here is the bedroom, the bedroom has a door to this study. The study has a secret passage, a secret bookshelf door into this secret study with a safe. And then here at the end of this corridor is a secret passageway down into the dungeon. And over here is the slave hut. This is a secret passageway down into the dungeon too. And this was the guardhouse, which you never got to see inside. Because…

NICOLE: ‘Cause we just burned it.

ANDY: Because you decided to burn the place down. So they-

NICOLE: We used an immovable rod.

ANDY: Yeah, yeah. So they used an invisible rod to block the guardhouse door. They put sort of casks of gunpowder, which they had got illegally elsewhere. You blew up the latrine was the first thing. And then you basically ended up setting fire to the house. So that was quite an interesting thing for me in the sense that, okay, so all that stuff I prepared, they’re not gonna see ‘cause they just burnt it down. And all those secrets and all that loot and things like that, that were quite important.

NICOLE: Oh no. There was loot??

ANDY: They’re gonna see. There’s loads of secrets and stuff in there that I had to relocate.

NICOLE: Oh no.

ANDY: So then what happened was quite an interesting mechanic for me though, ‘cause they had actually started that in the previous session. I then thought, “Okay, what would be going on?”

So all the guards are obviously some people are trying to put out the fires. A lot of people are gonna trying to be rescuing stuff from the house. So what was happening is that all these people are kind of pulling stuff out of the house and they’re putting onto the lawn and things like that. It’s kind like people do when there’s a bushfire or something trying to rescue the stuff.

But meanwhile down in the dungeon menagerie. Let me open that. What you guys ended up doing was I had, so there were these… For reasons I won’t go into, all the slaves were tieflings. And there was one of them.

So I had some stuff in here, which is my sort of read aloud thing. Oh, I think I’ve got a read aloud callout here and I use that so I can just flip that up. And then what I do is I use embed, so transcludes. So I have all the different bits here and then I can transclude them. That’s why I find this quite useful to split them into separate notes. Because then I can adjust the structure and things of what I’m gonna be doing in that session. ‘Cause you spent like two or three sessions around this compound. And so I can adjust it, I can embed a note in the scene like I did here and I go, “Okay, so the slave house so this is gonna be this thing.” And then the dungeon menagerie and then I’ve got some stuff around here.

So I have the dungeon menagerie. I’ve got Ionos and Ilbane, who was the big bad in this. I pulled a stat block from LevelUp, actually, 5e. Because it was more interesting.

NICOLE: Is this just an image?

ANDY: This is just an image. Yeah, I think I’ve just clipped a stat block. Out of LevelUp. And sometimes, you use the statblocks plugin and sometimes, it’s just easier to grab an image.

NICOLE: Yeah, I get that.

ANDY: Yeah. And in this one for the dungeon menagerie, I actually like an idiot.

NICOLE: Ooh, ooh, looking at all the secrets.

ANDY: It wasn’t like an- And there’s a secrets and clues things at the end. And I think you know all of these but you know all of these. So I have secrets and clues. So when they found this stuff I go, “Okay, right. They found that.” These will been reset I think because I can’t remember why. Probably made a new sanitized version of it.

So then I have the encounter. So this is using the encounter plugin. So when this is going on, it fires up the encounter there and I work it that way. But the dungeon menagerie was a real issue ‘cause I thought, “Oh my god. And there’s so many people in this place, how am I gonna manage it?”

So I had for the first time ever did an adversary roster. So if anyone knows the Alexandrian, Justin Alexander, he recommends this. It was quite an old school D&D thing to do. And the adversary roster looks like this. It’s just a little markdown table, it’s got links to some people. And it was just so I can remember, these are all the different people around in the house. And I have an idea of where they are, but they can move around.

NICOLE: Did you plan that one of them had a birthday?

ANDY: So I had this idea. I wanted the guards. I didn’t want the guards to dominate everything that you were gonna be fighting multiple guards for ages. And you’d gone there, you’d scoped it out during the day and then you’d gone there in the evening.

So I thought would it be fun for the guards to be in the guard house getting drunk and being loud. And then, I think in the moment, I thought you hear that one of them’s singing “Happy Birthday.” And it was particularly poetic because that one then came out and got axed, I think by Halkin. It’s like, “You just killed the guy whose birthday it was.”

And I quite like messing with them that way. I remember there was one thug and you found a note. I think, I can’t remember if you were playing for this point.

NICOLE: Many, many ways you like to mess with us.

ANDY: So they killed two random thugs who were guarding a place. And then of course, when they searched the bodies, I had a thing where one of them had a note in their pocket from his estranged wife saying, “Please come home. Your daughter misses you. Please give up this way of life” and stuff. And they’re all like, “Oh no, we just killed this henchman, but they had a life and stuff.” So I apologize.

NICOLE: How cruel is that? What a sadistic, okay, I change my answer. You’re a sadistic DM.

ANDY: My job is to mess with you as well.

So anyway, adversary roster and then I had an encounter table. So using the encounter plugin such as separate, it’s in a separate thing as well, a separate transclude, there’s an encounter table so that depending on where you went, I could fire up a, a different set of encounters. So if you were gonna fight the house guards and the captain, I can fire up that one and so on.

And this was the dungeon menagerie. So I just had this set up like all the creatures are in here and then I would just skip them in the initiative if depending on who was there. You would’ve probably struggled if they’d all escaped in one go.

But as it happened, you set fire to the house and missed out quite a lot of, all that prep, all these bedrooms and everything that I created, you guys just completely missed out ‘cause they turned to ash.

NICOLE: There was an ettercap?

ANDY: Yeah, there was an ettercap in the dungeon menagerie. So we play, we use Owlbear Rodeo. So this was loaded into our Owlbear Rodeo and yeah, there was the one that was the ether cap was, there was one cage you might remember that was full of spiderwebs and cobwebs. That was the…

NICOLE: Cool.

ANDY: And then I guess the last thing you were, Floyd… So the joke was that this Domovoi was Floyd. Ah, so Floyd, he was just a Crownsguard. He was just a guard on a gate. And I thought it would be funny if there was like a really good looking guy and a young doofus who was as a comedic pair.

And then I said, “Oh, this guy’s, his name’s…” I thought of Pink Floyd or something, his name’s Floyd. And when you ask him directions and that’s how it started. And then I said, “He’s really good looking. He looks like Chris Hemsworth.” And so that’s how he became Floyd Hemsworth. And then-

NICOLE: Oh, I didn’t know that.

ANDY: You asked him to come with you. Yeah, the party asked him to come with you. He started flirting with one of the characters who then was flirting, well, she was flirting with him. And then he had all sorts of thoughts about, he became, then I had to build a whole backstory and family and stuff and connections for him. But it was one of the examples of a randomly invented NPC became actually quite important.

So anyway, the way the setup was he was in here and the Domovoi had made him invisible but had made himself look like Floyd and was in here. So the idea was that I was gonna lure a character into the cage, get the cage closed, there was gonna be a little cage fight inside here except what happened?

What happened, Scura? Tell us.

NICOLE: I foiled your plans.

ANDY: Milla. It was Milla that foiled the plans actually. Two of you, both of you, between the two of you.

NICOLE: I cast the see invisibility, yeah.

ANDY: So you cast see invisibility and Milla cast Detect Thoughts. And so basically between the two of them, they could work out who was in which cage. So that didn’t work. Anyway. So that was my setup for all of that. That’s probably as much as I need to carry on going on about, it’s been quite a long time, but it was quite complex. And then got foiled by the players.

Nicole’s session notes

NICOLE: Good. I can show you mine for the same one.

ANDY: Okay. And if anyone has any questions, please fire away.

NICOLE: Okay, so this is the same session. I always like to have like a Midjourney thing that I do just so that in the worldview, in the world note, it’s just nice to have cards and each session has like an image of what it looks like. Sometimes it’s not quite right, but it’s enough to remind me of the session.

And oh, that’s our video of it actually happening. Oh, so we have a player-only chat where we scheme against Andy.

ANDY: I didn’t know about this.

NICOLE: And plan to mess with him almost as much as he plans to mess with us. So there were, at the time, Tyler, who plays Wally, he was away. So sometimes, I put things like instructions for Wally and he had this weapon. He told me when to use it and what the abilities were. And then I like to use these little call outs sometimes to mark what I know and what I don’t know or what other players don’t know.

Sometimes, I put things in comments so that if somebody goes into my Obsidian published notes, one of the other players, they won’t immediately see some things there. So it’s the same thing, right?

So here I cast see invisibility as we were going down into the dungeon. And then I also use Initiative Tracker.

I mean, I’m a healer. So part of the reason I love being a healer is that I always love keeping track of where everyone is. So this is what it looks like. I happen to have taken a screenshot of the end of this session. So this is what it looks like. I have everyone’s HP, I’m going through this using the initiative tracker and I have these custom statuses. So I track things like, if our barbarian has used reckless this round, then I know that he’s gonna need a little bit more help ‘cause he is probably gonna get hit.

And if I have temporary hit points that I’m giving to people, I would track that here as well. Even things like if people have reacted or not, ‘cause that affects whether my party can leave the threat range. And what I noticed though, you use the initiative tracker, but you don’t use the log, do you?

ANDY: Yes, sometimes. I do turn it on and off. I do use the log but once I, something crashed or something and I couldn’t remember everyone’s, the initiative order. What I don’t do is I don’t really log everyone’s hit points. I let you guys decide. I let you keep track of your character. Obviously, I’m keeping track of your opponents, but I’m not keeping track of your hit points. Obviously, I’m seeing it.

NICOLE: That’s fair enough.

ANDY: We still have our character sheets on D&D Beyond, so I’m usually checking where you’re at on D&D Beyond in the phone app.

NICOLE: Yeah, so I do that mainly in Obsidian because I need to know who needs healing and I always turn the log on, and this log is like an optional thing for Initiative Tracker, which is the same one that Andy uses. So I have a bunch of them here. There’s a separate Folder for log and those are the combats. And then I embed that combat into the session note so that way, it gets updated here. And then while I’m going through the rounds, I also just type in some explanation because otherwise, it’s just things like, “Grom took three damage.” And it doesn’t say, Oh that was because of Divine Smite. So I also track it along the way. And it’s just also handy I think if Andy, your initiative tracker stuffs up or something or you lose it, I can say like, “No, we’re in round three” or whatever.

ANDY: Yeah, it is super handy. I think one of this is one of the differences between being the DM and being a player also is that as a DM, I’m just flat out like in a combat thing, but there’s like lots of creatures and people going on. I’m flat out concentrating on like everything. Whereas there’s a player, you’ve got the space between your turn and the next time it comes round to-

NICOLE: We’re much more immersed.

ANDY: Yeah, yeah. Well, you are much more immersed, but also you have time. You have time to write notes.

NICOLE: Yeah, true.

ANDY: I barely write any notes when I’m DMing by the way. I probably should. I know loads of people who also don’t, I occasionally will write some at the end of the session if like, “Oh, I must remember that that’s happened.” I like Scura gave her two daggers to one of the NPCs, so I wrote that down.

NICOLE: Everything. I gave everything.

ANDY: I forget everything. I’m very reliant on note-taking players.

NICOLE: Yeah, so mine was basically the combat after that. And here, I’m tracking whether somebody has used my Guiding Bolt advantage on attacks and things like that.

And at the time, I was also playing two players because one of the other, two characters, because one of the other players was gone. So this is like where I put the next moves for both of them. And then at the end, I had a little note to tell him because I don’t have access to his character sheet.

But I was thinking that maybe I could also show how I create a new session because this is the world page and I have this nice little add new session button and when I click it, look, everything’s filled in for me.

I guess I haven’t processed the last one yet, The Leaky Cucumber. But it creates- It creates the session already. So yeah, I use funny titles as well. So this is where it is at the end of the session, I would put like a summary here so that when I generate the next one that’s pulled in, so this is the recap.

So often, I do the recap. Sometimes I try to get one of the other players to do it, to encourage them as well. But it’s just nice to have everything filled in. And this is also pulling in the date. I set today’s date on Fantasy Calendar so that it always chooses today’s date. My character is a Twilight Cleric and a Stars Druid, so moons matter. So I do have that as well. Let’s just show, oh, by the way, our party’s name is Temporary White Circle, and now you didn’t notice, Andy. Do you see there?

ANDY: Oh yeah, I see the logo. Yeah, yeah, that’s what we’ve got on here, TWC.

NICOLE: Yeah. So there, you can also see the moons, Catha and Ruidus. Isn’t that cool?

ANDY: So you are way more organized in terms of the automation than I am. I generally start by duplicating the previous session’s note and because of the stuff in there, there’s some secrets and clues still that they haven’t discovered and I sort of delete stuff out and then go back in. So my metadata is all manually done. I should probably try and automate some of that.

NICOLE: Yeah, I can just give it to you.

ANDY: Yeah, but then all the rest of the stuff doesn’t work and I just don’t have the inclination to go back through every session note though and add it all back in, so.

NICOLE: Octavian Datcu says:

“Nicole, how do you color tags without any tags surrounding it?”

I’m not sure what you mean exactly. Let me share my screen.

ANDY: It could be the theme. Is it the?

NICOLE: You mean, does Octavian mean like the italics and stuff? Because yeah, this is the Anuppuccin theme where you can have the text decorations a different color. So that’s handy. For example, if I just put something here, it’s automatically green because it’s italic. And then this one would be automatically red ‘cause it’s bold. I love Anuppuccin.

How has me taking notes changed the way Andy GMs?

NICOLE: Andy. Do you think that it’s changed anything about the way that you GM that we take notes, and I take notes, so much?

ANDY: I don’t know because this was the first big campaign. I’d played some small stuff with my family. But it’s the first big campaign that I’ve done since I was a kid. And so, I guess you came in, I mean, couple of the other players did take notes, but probably, you take obviously more extreme notes.

What has it changed? I don’t think it changes much. I mean, it’s just really useful for me. ‘Cause I know that I’ve like, “Oh God, I didn’t make a note of what happened here or who’s got what,” and things like that especially when you’re dividing up loot and things like that. And I know that one of you has made a note, so I’ll often ask for your notes or go and have a look at them and that’s pretty useful.

It’s probably, I think, we’ve joking about the sending stuff before. I don’t know what it’s like for you as a player. I mean, I get the feeling that it sort of makes the world a bit more alive because NPCs, they’re not just like, “Oh yeah, we spoke to that person and we recorded that one and then we moved on and we’ve forgotten all about them.” It means that they sort of come back into the world a bit and I think that maybe makes the world feel more alive maybe to you as players.

NICOLE: Yeah, I mean part of it is me just wanting to mess with you, but also some of these things are actually for a greater purpose. Most of them are not completely random. I am actually trying to do a background check or whatever.

And yeah, I am very engaged in the world. And there have been a few times where I would’ve gone further if I didn’t think I was taking up too much time already. I would’ve, like, engaged with an NPC a bit more.

Taking notes on other PCs

NICOLE: I don’t know, maybe I’ll also show my notes on the PCs. So this is one of them. Yeah, I’m hoping that they won’t watch till this part.

So this is the one for Milla, there’s a picture of her, there’s a backstory that I’ve pieced together from things that she’s told me in the game. I do something called an OGAS framework. Occupation, goal, attitudes, stake. So I’m just… I’m thinking of what her goals are because I think my character is very supportive and she wants her fellow party to achieve their goals. So I have to know them first. So I take notes on everybody. So I also have ideals, bonds, and flaws, not the actual ones that she has. So I don’t see that, but just, I’m guessing. I’m guessing based on what she said. And then I also have a section for knives.

Andy, can you explain knife theory?

ANDY: Oh, we didn’t talk about that, yeah.

So knife theory is the thing, and there’s a Reddit post. So if you Google like Reddit knife theory D&D or something, you’ll find it.

So knife theory is this idea that what you do when you create a player, when you create a character as a player, you give a bunch of knives in there that the DM can use basically to stab you with later.

Which is, there are some things like, I did this thing and I’m trying to escape from it or so I have this adversary or I have this ally, or I have this particular weakness, or I have this thing in my past.

And the more knives there are, the more useful it is for me. The more I can as a DM and the more I can integrate you into the world or integrate the story with their backstories.

And a lot of stuff comes up fairly randomly as I talked about Floyd Hemsworth. And he is, like I said, he’s just a random NPC. And then it forced me, if I go here.

ANDY: Actually, for this session, the dungeon menagerie one, I put all the stat blocks on a canvas. And it was really crawling on the day of the session, so I stopped using it, but it would now actually be quite useful. So I could just quickly grab the stat block.

So here’s Floyd. I found a picture of Chris Hemsworth, I don’t know what you know, what this is from here. He is dressed up and he looked a bit like.

NICOLE: Well, so he really is just a basic guard.

ANDY: Yeah, well he’s a captain now. Well, he’s more than the captain now anyway. And super handsome Crownsguard.

NICOLE: Super handsome!

ANDY: And then there’s a backstory for him. I came up with a kind of, his father’s a general and all this stuff. And they ended up being quite a scene with me talking to myself, like him talking to his father, NPC to NPC about it. And so it was quite of, I was able to connect him to a couple of the other characters’ backstories in interesting ways.

So one of the characters, Wally, I thought, well, like I said, Wally’s character is actually, his father’s also in the military. So I thought, “Oh, this would be interesting that they know each other and stuff.” You have to be a little bit careful because you can also end up with a thing like of too many coincidences. But it helps and I’d really like the knife thing. And then, occasionally, there are actual player things that I know like things that I then just use to mess with them.

NICOLE: Yeah, I like that stuff. But I also keep notes on knives for the other players and that’s what I like– my character likes to do Zone of Truth, which is basically just us telling– me, me asking questions to the group-

ANDY: It’s spin the bottle.

NICOLE: -to the characters. Pretty much, yeah. And those questions come from some knives that I’ve identified, like things that they react to in a positive or negative way.

And I also actually have a section where. So here, the knives are things that the tiefling likes and doesn’t like. And then I always have a section of how I can help them for every PC. What are the things that my character, Scura, could do. And then I have this section, which is all of the questions that I ask in Zone of Truth. I also have stuff for playing the tiefling in case she’s away.

ANDY: The tiefling.

NICOLE: Yeah, I have them. Yeah, I guess she’s not THE tiefling anymore, but yeah. They’re my notes.

ANDY: I sometimes– I have a cheat sheet. I found a cheat sheet for Scura actually from a session ‘cause I think that was the one where you were away, you were like getting on a plane or something, so I had a little cheat sheet.

NICOLE: Yeah. Well we should probably end this ‘cause we’re over time. There’s so much that I would’ve wanted to ask you, but probably for my own edification.

ANDY: You can ask one more.

NICOLE: Maybe I’ll ask you offline. Okay. What should I ask? Oh, no.

ANDY: Roll dice.

Taking notes on the players

NICOLE: Okay, okay, I know. Because the kind of GMing that you do, it is very specific to us as the players. I don’t know, at least that’s what I feel. It’s our characters and it’s what we’ve given you in our backstory as knives. But the more that you know us, the more that you also know our knives as players. So do you take notes on us as well as our characters? Do you just remember that?

ANDY: Only up here. Yeah, I don’t write notes really about you as people. No, no, not as players. Some things come up during the week or something and I’ll think of something I thought that would be funny to mess with the player. I can do this to their character to mess with the player and I do that, but I don’t have notes. I mean, I do have notes from session zero about you and stuff like that. And that’s mostly around so I can remember what you said you did and didn’t want and things like that. But otherwise no. I don’t take notes. Not like you.

NICOLE: Wow. You just have space in your mind for all sorts of devious things then.

ANDY: Well, lots of improv.

NICOLE: Yeah, no, it’s really well done. It means a lot to me to have a safe space to play a game and to like truly try out things that I wouldn’t do in real life.

ANDY: That’s good. I’m glad that it feels that way. That’s the aim.

NICOLE: All right, let’s close this so that I can ask you the real questions afterwards. Thanks for coming on to the channel again for the second time.

ANDY: Thanks for having me.

NICOLE: Maybe we’ll have you on for a third. You’re always welcome.

ANDY: Okay. Take care.

NICOLE: Thank you, everyone for listening and I’ll see you next week. Bye.

ANDY: Bye.

How we got into tech (Adobo & Avocados #01)

Wed, 29 Mar 2023 14:54:43 +0100

My friend Marie Cruz and I have been wanting to talk about a few topics related to diversity in tech, and over the few years we’ve known each other, we’ve only added to that list. We finally decided to take a leap and start a new YouTube channel/video podcast just so we have a space to have those discussions. Here’s the first one, where we talk about how we got into tech.

Below is a transcript of our conversation.

Why “Adobo & Avocados”?

MARIE: All right, we’re live. So, hey everyone, my name is Marie Cruz and with me I have my colleague today.

NICOLE: Hi, I’m Nicole van der Hoeven, I’m her colleague. And I’m a little sick, but I’m here.

MARIE: So we, today we’re actually gonna be doing the first ever recording for the Adobo and Avocados podcast. So we thought about this name and initially like Nicole and I wanted to, you know, ever since before we wanted to launch a podcast where we can talk about, you know, our experiences working in tech and all other things. And then we thought about the name Adobo and Avocado. So you know, to anyone who doesn’t know, so we’re both from the Philippines, and Adobo is the national dish of the Philippines. So that’s why we thought we’ll add the name of Adobo in our podcast title. Avocados, ‘cause we’re both Developer Advocates. But yeah, I think, why don’t we discuss like, why was it called avocados in the first place? Do you have any idea?

NICOLE: Yeah, well, we don’t really know for sure. So I thought that it was something to do with the language of it. Like, you know, an advocate is usually like a, could be like a lawyer in common non-tech senses. And the translation for that is abogado in Spanish or Tagalog actually,

MARIE: Even in Tagalog.

NICOLE: our native language.

MARIE: Yeah.

NICOLE: So it’s like kind of similar to avocado. I don’t know, that’s kind of what I thought.

MARIE: Yeah, that’s actually an interesting thought. Like, ‘cause the abogado does sound very, you know, similar to avocado, but I actually read this one article where I think their project manager had a hard time saying Developer Advocate, they ended up saying developer avocado. So who knows what the origin is. But yeah, I think it’s a great, you know, you know, term in terms of like, you know, the whole play, like playful, you know, meaning behind the word Developer Advocate.

How we got into tech

MARIE: So yeah, I guess first things first, like you know, who we are? Like, why are we doing this podcast? How, you know, we got into tech? So we’ll talk first about, I guess a bit of some introductions about ourselves. So Nicole, why don’t you start, like how did you get started? You know, what’s been your, I guess, background so far? Did you have a traditional, you know, computer science background or did you do like a different, I guess course back when you were still studying?

Nicole’s background

NICOLE: I did not have a traditional background. I grew up in the Philippines. I moved away when I was 18. And I don’t know what your experience was, Marie, but mine was that in the Philippines there are still, I think we do a pretty good job at being equal between genders, but I think there’s still gender roles. And when I was growing up, I really loved both the maths and sciences and like literature and languages, but I was very heavily pushed towards the humanities, ‘cause that’s what girls do. Girls don’t go into tech.

MARIE: Mm-hmm.

NICOLE: And so I really struggled at uni and in college. I shifted majors six times. I had no idea what I wanted to do.

MARIE: Wow.

NICOLE: And I went through, and that was over two countries. I went to the US to study as well. None of them was computer science because my brother was a developer. And I don’t know, he’s a little too introverted for me. And I just had this idea that, if I go into computer science, I have to be a developer, and if I’m a developer, I won’t talk to anyone. I’m just going to be

MARIE: Yes.

NICOLE: with a machine. Like I had that in mind, which now I know is completely wrong,

MARIE: Yeah.

NICOLE: but that’s why I never went into it. So finally my parents were just like, we don’t care what you finish,

MARIE: Yeah.

NICOLE: we don’t care if it’s basket weaving, just pick a major and finish it.

MARIE: Yeah.

NICOLE: And that was economics,

MARIE: Oh.

NICOLE: which I have never used in a job.

MARIE: Yeah.

NICOLE: And then so-

MARIE: That’s really interesting. Yeah.

NICOLE: And then I, how did I get into IT? I was applying for different economics related ones, and then I met my now husband and he,

MARIE: Hi Rob.

NICOLE: he was a test manager. Yeah, hi Rob! He was a test manager and I said, well what’s a test manager? And then he told me about the world of testing.

MARIE: Yeah.

NICOLE: And I didn’t understand it at first, but I wanted to know more. And then I thought the way that he described it sounded so interesting, like the idea that,

MARIE: Mm-hmm.

NICOLE: that there is a job, there is a person whose job it is to be the one using applications or products first. And then trying to see what’s wrong with it for the purposes of improving it. I just, I really loved that idea and it was another branch of tech that I didn’t know existed.

MARIE: Yeah.

NICOLE: So I started to, I did an ISTQB certification just to like have something

MARIE: No background.

NICOLE: ‘cause I didn’t have any background. Yeah. And I started job hunting, and somehow managed to land a performance testing job under someone who became my mentor.

Marie’s background

MARIE: Hmm, nice. Yeah, it’s really relatable because like similar with myself, I fall into tech accidentally. Like it was never my intention, you know, okay, I’m gonna study computer science, although I did actually have a computer science degree.

NICOLE: Oh, wow.

MARIE: But the story, the story behind that, I don’t share it too much to other people ‘cause I think the story behind it is a bit funny. But my first experience with tech is actually updating my MySpace profile. Do you remember MySpace?

NICOLE: Yes.

MARIE: Yes so, before you can customize your profile with a bunch of HTML, CSS codes and like I think before I was just trying to, you know, create something even though it didn’t look very accessible. I remember I had lots of animations, just lots of like

NICOLE: Oh.

MARIE: striking fonts, background music, and it was just something that I was doing for fun. I did a mass communications course back home in the Philippines. So I was majoring in digital media. And then I had to go back here in the UK. So I was working first for about two years before I decided to go back to uni. And during that time when I was looking at, you know, courses catalog, ‘cause I wasn’t really sure what to pick. So I saw a course which basically has three different things. So it’s computer science with business management and financial accounting.

So I thought,

NICOLE: Oh wow, okay.

MARIE: I was like thinking in my head I think that sounds like a really cool name for a course, so I’m gonna go for it. So to me I just found the name like very interesting. And it means that-

NICOLE: So when did you

MARIE: Yeah?

NICOLE: move out of the Philippines?

MARIE: So I, so we went back 2008. So I was in the UK since 2004. Actually no, we went back 2006, and then I went back in the UK 2008. So 2006 I enrolled, you know, to study mass communications. I didn’t finish the course ‘cause I had to go back in the UK after two years because of like visa, like status. And then when I got back, I just put that into a pause, you know, did some work experience. So I was working as an office assistant for about two years and then my friend and I just decided, okay, I think we should go back to uni. So I actually applied as a mature student ‘cause I didn’t, you know, had any like actual qualifications ‘cause I never graduated from my degree in the Philippines. I never finished high school as well, ‘cause I kept going back and forth between the UK and the Philippines. So the last time I actually graduated was during middle school.

NICOLE: Oh, wow.

MARIE: but applying as a mature student, they just looked at my, you know, work credentials, like my work attitude. And I submitted some of my grades like back, you know, when I was still in the Philippines ‘cause like my first two years I had some, you know, grades from like the different subjects that I was doing. But yeah, that’s how I got into tech. I just found the name like really interesting and I thought before I could mix and match it with business and accounting. So even if the computer science bit didn’t work out, like I can still maybe utilize some of the things that I’ve learned on the business side and on the accounting side. And then, when I was about to look for graduate jobs, I got approached by a testing consultancy company. I didn’t, I wasn’t aware that there’s a graduate job for, you know, just doing testing ‘cause most of the graduate jobs that, you know, we were being offered is, you know, how to, you know, do like development, or like database like management roles. So initially I wasn’t really sure if I wanted to do the developer route. Like I enjoyed the coding, I was doing the coding bit, but like back then I wasn’t like fully 100% sure that it was the right path for me. So I found out that, yeah you can have a career in testing. Again this is all accidental ‘cause if that testing company didn’t approach me, I would’ve probably looked at like other roles. So it was like really great to hear that, yeah, there’s like a graduate jobs for just doing testing. And yeah, we were doing the ISTQB as well. So we were doing some training. And yeah, the rest is history. It’s funny because I didn’t intend to continue on a developer route, but then now I work as a Developer Advocate.

How did we become Developer Advocates without being developers?

NICOLE: Yeah, maybe we should talk about that a bit ‘cause I also don’t consider myself as a developer even though I can code.

MARIE: Mm-hmm. Yeah.

NICOLE: And yet somehow we’re both Developer Advocates. How did that happen?

MARIE: Yeah. It’s like there’s some sort of, I guess, interesting link between software testing and developer advocacy. I don’t know like if it’s the same for you, but like with me, I really enjoy the teaching part, the learning and public part as well. And when I was doing my role as a software tester, I felt that I was doing the advocacy, but maybe, you know, not really in a way that it’s very out there. So I was, you know, helping my colleagues, I was talking about tools that I’ve learned, you know, with them, I was showing them how to use it. And then while I was working as well, like I started doing the learning in public on the side. So I started with my personal blog and then getting involved, you know, with meetups to talk about, you know, the different tools that I’ve learned. So I guess for me, like being a software tester also, you know, encouraged me to, I guess be a Developer Advocate ‘cause I was just telling people what I’ve been doing as a software tester. I’ve been telling them what different tools I’ve been using, you know, different concepts that I’ve been learning as well on like a day-to-day. So is that the same for you?

NICOLE: For me it was more because I was a consultant,

MARIE: Mm-hmm.

NICOLE: which means I was going on different projects all the time. And when you’re really new in an industry, I think it’s really exciting and you’re learning a lot. And certainly every project has like its own framework and different languages and different people, which is interesting for a while.

MARIE: Yeah.

NICOLE: But I don’t know, after several years of it, I started to feel like, people are actually asking the same questions. You know, there’s always, there’s some things that I always had to explain. You know, I always had to, for example, fight against the fact that performance testing is often seen as just something where you run the test, okay, you check the box, you’ve done performance testing.

MARIE: Uh-huh.

NICOLE: There’s always a certain amount of education where I’m like, no, it’s not just about checking the box, it can actually provide real value, here’s how.

MARIE: Yeah.

NICOLE: And I just found myself repeating a lot of the same things. And so I actually started to create videos also because load testing results are kind of difficult to explain sometimes.

MARIE: Yeah.

NICOLE: And I would create this like test summary report that lots of people don’t read.

MARIE: Yeah.

NICOLE: And so, as kind of an alternative way to explain things, I started creating video summaries to go along with it. And then I just thought, well maybe I can, ‘cause I can’t publish those, because it’s very specific to that company.

MARIE: Yeah.

NICOLE: But then I, I thought maybe I can pull out, I can abstract away some things and then publish those videos where I’m not talking about a specific project. And I guess that’s kind of how it started. I started, I put it into text first, because that’s just easier than video.

MARIE: Mm-hmm.

NICOLE: And then eventually found myself into video anyway.

MARIE: Yeah.

NICOLE: So I guess for me, I got into Developer Advocacy because I was tired of, I was trying to automate a part of my life, the explanation part where I kept having to say the same things and I just wanted to be like, hey, read this blog post where I explain why you need think time in a much more coherent and considerate way.

MARIE: Yeah.

NICOLE: So that’s, that’s how it naturally happened. I think you and I are the same in that regard that we were already doing it before we got the title.

MARIE: Yeah. I totally relate to that. Like, especially when you said that like, we’re just trying to solve the same problems. I started, you know, my blogging journey just trying to communicate like, you know, questions that people have been asking me on a day-to-day basis. And rather than just, you know, speaking to individual people and repeating the same things like again and again, like just put everything into a post or like you said like a video, and then just share that, you know, like to the public, ‘cause you’re not only gonna be able to help like one person but you know, maybe if someone else stumbled upon your blog or your video, then like, you might help them as well. I think like one of the, like, one of the best reward that I experienced when doing this like learning in public and you know, sharing more content out there is if you just get a message from someone or like a comment and then they say that, oh, like, I’ve been stuck with the same issue, I found your blog post or your video and it really helped me. And just that feeling that you know, you’ve helped someone is like a really great feeling. ‘Cause at first I wanted the blog as well just so that I can have something that I can look back to in case I, you know, forget like what I’ve done on a previous project. Like I can document it on a blog post, but then like now it’s also helping like other people who are experiencing the same like issues at work.

NICOLE: Yeah, I definitely see that. At first I was doing these things for myself because you know, you don’t really know that you know a subject until you can teach it to some degree. And I really believe that. And I, for me there were so many things ‘cause as a consultant, you have to switch tools very quickly, you have to switch languages, you have to be a polyglot on a lot of different levels and it’s impossible to keep all that in your mind. So after every project I tried to like, distill my learnings from it, so that I,

MARIE: Yeah.

NICOLE: future me would come back to that framework, or tool, or whatever, and not be starting from zero. But yeah,

MARIE: Yeah.

NICOLE: the kind of other people finding it was an unexpected boon.

MARIE: Yeah. So let’s just move on a bit. So we’ve talked about, you know, how we transition from software testing to developer advocacy. But I guess in terms of like the process, like did you apply to a lot of, you know, companies? Was developer advocacy something that, you know, you were aware of already, or like you just found a job description and then you’re like, oh, this is what I do already. Okay, let me apply to that job. So what’s been like the process for you?

NICOLE: So I was, this is only, this is my first job as an actual, like the first job where I have Developer Advocate in my title.

MARIE: Yeah.

NICOLE: My previous job was with a, well at the time it was a startup, it was Flood.io, which is now acquired by Tricentis.

MARIE: Yeah.

NICOLE: And when I joined, I think I was like, I think the fifth person that they hired in the team.

MARIE: Oh, wow.

NICOLE: So it was very small. And it was, I really love that environment because you can do a lot of things, you can wear multiple hats. And that’s really where it started. On a typical day, I might be answering questions in support, or presenting at a conference or ducking into a sales call or like,

MARIE: Yeah.

NICOLE: there were so many things that I could do. I was like part of the rotation for things going down basically, for things exploding. And I had to learn a lot about infrastructure as well to be competent in that role. And I started doing a lot of advocacy type things, mainly from a support perspective

MARIE: Yeah.

NICOLE: because I got the same questions there as well. And I wanted to be able to focus on the crunchier topics.

MARIE: Yeah.

NICOLE: And you can’t do that unless you address some of the more basic or common questions. And then my manager at the time told me about Developer Advocacy

MARIE: Yeah.

NICOLE: and my immediate response was, but I’m not a developer.

MARIE: Yeah.

NICOLE: He said that that’s just the name for the whole industry that I think you would be really good at and that you’re already doing. And with his encouragement, this is Tim Koopmans, the one of the co-founders of Flood, with his encouragement, strong encouragement by the way, things like, he said, oh, I’ve got this conference in Moscow that I’m speaking at. Do you wanna come watch me?

MARIE: Right.

NICOLE: And I said yes. And he backed out so I had to do it, things like that. Because of things like that that he did, I started to understand what developer advocacy is

MARIE: Mm-hmm.

NICOLE: and when it was, when I felt the time to change directions, I specifically looked at k6 actually. I didn’t apply to any other jobs because I was pretty happy where I was.

MARIE: Mm-hmm.

NICOLE: But I really loved k6. I had already not used it in a project, but I’d already, I knew what the tool was and I used it, I tested it a bit. I love the site, I love the principles behind it. And I saw that they had a Developer Advocate role. So,

MARIE: Yeah.

NICOLE: It just kind of happened. How about you?

MARIE: Well the first time I heard about the term was from you, when you said that, hey we’ve got this role for, you know like, you know like, we’ve got a Developer Advocate role for one of the tools that, you know, we’re gonna launch. Like, you’ll be like the great fit. So I didn’t really send any CV. I don’t think I sent any CV or like, yeah, I think, you just like, ‘cause we just connected already like via this Automation Guild conference. So like we were already connected via Twitter. And I think this is where like one of the benefits of like learning in public really came into play, ‘cause I was just, you know, publishing constantly. I was, you know, talking at various meetups and conferences already. But I was just doing all of this like as part of my, you know, side projects. Like it wasn’t part of my day-to-day role on my previous job. So when you explain to me what a Developer Advocate is, I’m like, oh yeah, I’ve been doing that on the side. So that was my first actual, you know, experience of like hearing what a Developer Advocate, you know, does on a day-to-day. ‘Cause I had the same perception that I’m not a developer, how can I be a Developer Advocate? But I think because as like as part of our day-to-day jobs, like even if we’re not writing application code, we’re still writing, you know, test scripts. we’re creating test automation frameworks. So I think that’s really helped as well transitioning into this role. But yeah, like I never sent any CV. I just remembered like we were talking a lot like back and forth and then I met up with, you know, a bunch of our colleagues now. And yeah, the rest is like history again.

NICOLE: Yeah, I think that there are just some people that when you meet them, you click or you don’t.

MARIE: Mm-hmm.

NICOLE: And when that click happens, I’m just like, okay, I’m gonna write her name down. I think I told you about it like for two years,

MARIE: Yeah, I just joined-

NICOLE: we were talking, and I was like, hey, this could happen…

MARIE: Yeah.

NICOLE: Yeah, I don’t know when. I think you’d be perfect for it. And you’re like, okay,

MARIE: Yeah.

NICOLE: but it’s kind of early to be telling me, right?

MARIE: Yeah.

NICOLE: I was like, Yeah, yeah, but let’s just keep in touch. You know, and it worked out. I can’t believe it actually did.

MARIE: Yeah. And now we’re, doing.

NICOLE: We didn’t actually-

MARIE: Yeah, go on Nicole.

NICOLE: We haven’t actually defined what a Developer Advocate is.

What’s a Developer Advocate?

MARIE: Yeah, I guess like we can both, you know, share what our, you know, view is of a Developer Advocate. So to me when it comes to developers, I guess trying out, you know, new tools or learning about new things, they feel most comfortable talking to someone who’s actually used, you know, the product, or you know, the tool that they want to use. So, we are, you know, that person. So we’re helping developers, we’re helping testers when it comes to, I guess learning about best recommended practices, learning about different tools. So we’re kind of like a bridge between the community and like the company that we work for. There’s a lot of misconception that people think that we’re salespeople but we’re actually not, like it’s a completely different, you know, role. We are focused more on the community side, helping them grow, helping them, you know, I guess in some ways, like be some sort of champions as well, so they can also promote like, you know, the different tools, the different practices that we have been communicating to them. But yeah, what about you? Is that like sort of the same concept?

NICOLE: Yeah, I agree with everything you said, especially about being a bridge.

MARIE: Mm-hmm.

NICOLE: Because as a Developer Advocate you’re not quite a developer, that’s not our primary responsibility,

MARIE: Yeah.

NICOLE: but we’re also not like marketers or sales people. Although there’s an overlap there in both. I kind of think of it as anti-marketing. So a marketer is going to be talking about something that they themselves have never used, are not the target user for.

MARIE: Yeah.

NICOLE: And might not know much about in terms of the technical details.

MARIE: Yeah.

NICOLE: A Developer Advocate is someone who actually is the target user. So for those who don’t know, Marie and I work at Grafana Labs specifically on k6, which is a performance testing tool. But we have both used k6, we have both been testers and still are. And so there are things that we can suggest. Part of it is advocating internally.

MARIE: Mm-hmm.

NICOLE: We advocate for testers everywhere to the product team so that we say, like these are the features that we would be looking for as testers. Like we would say it this way or we wouldn’t really use that feature, we want this other one. But another part of it is advocating externally. So this is the part that people think of as marketing, but I think of it as anti-marketing, because we’re not going out there and saying, hey, you should use this product that we’ve never used before.

MARIE: Yeah.

NICOLE: Instead, we build cool things and demonstrate cool things that we have done, not some other developer.

MARIE: Yeah.

NICOLE: Like we do our demos,

MARIE: Yeah.

NICOLE: And then when people ask us, then we say, oh yeah, we use k6. So I don’t know, marketing definitely has its place, but for me I am so addicted to that authenticity and genuineness that comes from just talking about a tool that you actually love, which is,

MARIE: Yeah.

NICOLE: what k6 is for me.

MARIE: Yeah. I really like that because for me, an important part of, you know, this advocacy is, like communicating something that I truly believe in. Like I’m not going to advocate for, you know, a tool that I haven’t used. Like I would just feel like, you know, I’m a fraud if I do that. So if I know that, you know, there are situations where a certain tool or a certain practice, you know, would work better, then I would advocate for, you know, that as well. So it’s not just about, you know, for example, like at the moment we can’t do k6 for accessibility and I’m an accessibility advocate as well. So I would go and, you know, recommend like other tools and practices based on what like I’ve experienced, based on what I’ve done. And I think, like for me, that’s one of the biggest like advantage, ‘cause yeah, if I’ve been put on the spot and say, hey, you should talk about this to then promote it to other people, that would just totally go against my, like one of my beliefs.

NICOLE: I think I also really appreciate that this position lets us shine a light on people like us who maybe,

MARIE: Mm-hmm.

NICOLE: who don’t look like the average person in tech. I know it’s still, I’ve been lucky to work with teams of mainly men that haven’t treated me any differently because I’m a woman, but there is still like a sense of being Other.

MARIE: Yeah And I guess that-

NICOLE: And being in a position of advocacy means that we’re, we’re also able to appeal to other women who maybe didn’t, who didn’t have as good an experience as we did.

What can you expect from Adobo & Avocados?

MARIE: Yeah. And I guess that really leads to this next question. And you know, people who will watch this in the future might also ask like, what are the expectations like for this podcast? Like why are we doing this? So do you want to maybe start.

NICOLE: Why ARE we doing this Marie?

MARIE: Yeah. Oh, yeah. So do you wanna start like from your side, like what are the expectations that you know, that we can, yeah, share to people?

NICOLE: Sure. I think that you and I both do a lot of technical advocacy. We, you know, do demos, we write scripts, and we tell people how to do things in tech, very technical things. But, I think that there’s a whole other side of tech that doesn’t get talked about and that’s like, things like interpersonal dynamics or cultural differences. You and I are both Europeans that have to work in, in our situation, predominantly American space, and we’re also Asian. So balancing the two, you know, we are on different sides of the spectrum on that. Just also the general stress that comes with tech. It is not easy, not just as a Developer Advocate, but just to be in tech to keep up with changing trends.

MARIE: Mm-hmm.

NICOLE: I feel like there are so many emotional aspects and psychological aspects that don’t get talked about that I would really like to cover.

MARIE: Yeah.

NICOLE: Then also, I kind of alluded to it. I would like to make it a little bit more open, an open discussion for particularly underrepresented people

MARIE: Mm-hmm.

NICOLE: like ourselves. We identify with several minority groups.

MARIE: Yeah.

NICOLE: And maybe help them along a bit by telling them things that we had to find out the hard way early on in our career. How about you?

MARIE: Yeah. Yeah, I agree with all of those. And I guess it’s also to just have a safe space, you know, for everyone just to share their experiences in tech. Like you said, there’s this whole other side of tech that we don’t talk about that maybe other women are also uncomfortable to talk about, you know, with like, I guess other colleagues, especially if it’s just predominantly male. So we want to, you know, have a safe space for them, share, you know, I guess from their side, like their unique experiences as well. And maybe like we can all have, I mean maybe we also have those shared experiences that we don’t talk about, but then when someone says, oh yeah, I’ve experienced that too. So it’s just really finding, you know, shared experiences that we can all relate to. And like you said, just helping them, you know, or empower them to, I guess move forward in terms of like the next stage in their career. Like, you know, share what’s worked for us, share what hasn’t worked for us. ‘cause I think it’s all about like being authentic as well and not just talking about the good things, but we could also talk about the bad and the negative things as well.

NICOLE: I’m excited to see where we go from here.

MARIE: Yeah.

NICOLE: I think we have a lot of ideas about where we’re going to take this and I hope people will come along for the ride.

MARIE: Yeah. So, yep. I guess this is it for the first episode. We are just, you know, extremely excited to do this on a regular basis. So we’ll talk about different, you know, topics in every episode. And we might, you know, have like really awesome, like guests as well in future episodes. So yeah, I guess just stay tuned and yeah, look out for Adobo and Avocados.

MARIE: Thanks everyone. See you next time.

NICOLE: Bye-bye.

Taking Notes for Research with Bianca Pereira

Mon, 27 Mar 2023 08:56:48 +0100

Bianca Pereira joins me to talk about taking notes for academic research as well as general knowledge creation. Bianca is a researcher, a software developer, a feminist by ideal, and an educator by passion, and she comes with a lot of experience in working with PKM (Personal Knowledge Management) and Tools for Thought.

Below is the transcript of our conversation.

Introductions

NICOLE: Hey, everyone, I’m Nicole van der Hoeven, and on this channel, I like to talk about tech, travel and taking notes. My guest today ticks off a few of these boxes. Please welcome Bianca Pereira. (Portuguese) Good afternoon, Bianca, how are you?

BIANCA: (Portuguese) Good afternoon, Nicole, I’m doing well. (English) Thanks a lot for inviting. I’m really excited for our chat today. I really enjoyed chatting with you, and I want to see whoever is here with us today. So if I like to invite, if anyone wants to participate in our chat, maybe can just send us a message down there.

NICOLE: Yeah, leave a comment with questions that you want to ask Bianca or me, or both of us. But Bianca, (Portuguese) Could you introduce yourself, in English? (English) I practiced that so I needed to say it.

BIANCA: Oh, okay. That’s really good. Yeah, well, my name is Bianca Pereira. I’m a researcher and a knowledge management coach as well. So my focus, when you think about taking notes, managing papers, managing the literature, the source is really like, how can we do personal knowledge management for the purpose of research? So how can we create notes so that we can write research-driven outputs later on? Research-driven outputs can be like papers or they can be an essay or a blog post can be anything that is based on research. So it’s really like how can we do the whole process of personal knowledge management for the specific purpose of research? So that’s what I work on now. And I keep doing my own research as well in computer science, as we talked before.

NICOLE: Rodrigo said, “Good Portuguese.” (Portuguese) Thank you, Rodrigo. (English) I’m trying. I moved to Portugal. But maybe we should explain that connection, why I am randomly speaking to you in Portuguese. You’re from Brazil, right?

BIANCA: Yes, I’m from Brazil, from Rio de Janeiro. And I learned that Nicole, you’re speaking Portuguese because you’re also living in Portugal, right?

NICOLE: Yeah, of course, I have to learn that.

Research isn’t just for academics

NICOLE: I thought it was interesting when we were talking about this stream and I said, “I wanted to call it taking notes for academic research.” And you said, “Oh, well, it’s not just academic research.” You said that you wanted it to be broader, just research in general and also knowledge creation. Can you talk a bit more about the distinction between the two? ‘Cause when I think research, I automatically jump to academic research.

BIANCA: Yes. Yeah, many people, they’re like, “Oh, okay. Let’s write academic research.” I’m like, “No, please, no.” Because at least like from my perspective, I think research, it’s not just inside of academia. Everyone not only can do research, but many people do research in their jobs. Even like when you are going to create content, for example.

So for me, researcher is more than someone who went through a PhD or went through like formal academic research, but they’re all people who are using critical thinking for the purpose of creating new knowledge. For example, someone is in a company, the company is facing a problem and they need to solve a problem, which may be very specific to their company. So they need to do some research. They need to use some tools that we use for doing research to solve the problem to come up with innovation to solve something inside of the company.

So for me, from my perspective, that person is a researcher, even when it’s not an academic researcher, for example. So that’s why I like to think about research as the process of knowledge creation when you’re using critical thinking for that. So for me, I think like anyone and everyone could become a researcher or is already a researcher, even when they don’t call themselves researchers. So that’s why I was like, “Oh, let’s try to not…” Don’t say, “Oh, only who is in academia is a researcher, only who went through a PhD is a researcher.” I don’t think that’s the case. I think everyone can use the research for their work, sorry, in their daily lives.

NICOLE: That’s really cool. By that definition, I think I would classify as a researcher and I had never really thought of myself as one. It’s very meta, but I’m taking notes on this while we’re talking. And you said that one of the things that differentiates, I’m paraphrasing here. One of the things that differentiates research from like other forms of note taking is critical thinking. Do you also think that there is a sort of rigor that is applied? Like maybe when people think research, it means that you’re collating things from a lot of different sources? Or do you think that that is necessary for research?

BIANCA: I think, well, it depends on what you mean by rigor. If you’ve been on rigor, like, “Oh, more like in the academic sense, you need to write in a given way. You needs to like to have your argument very strong for lots of experiments.” I think there are multiple, it can be a little bit more lightweight let’s say, when I talk about research in general. But there is a level of rigor that is important, which is related to the critical thinking, which is really like, okay, and I’m just like, I just woke up in the morning and I thought about something. I say, “Ah, this is true, or this is the thing.” Because I think it is. I don’t think there is any rigor on that. It’s just like someone has their experience and they’re right about that.

But that’s not what I mean by research itself. It’s really like, try to create an argument. And it’s hard when I talk about argument, because when you think about argument, you think about two people fighting. That’s not what I mean by an argument. So an argument is really like, “I’m trying to justify why I believe on something.” Rather than just like, “I believe on something, this is the truth.” It’s like, “I believe on this and those are the reasons. This is the reasoning behind why I believe on that something.”

So this is the level of rigor that we need to have in any research. And sometimes to justify, we need to do full experiments and all those things. That’s when there’s really like professional research. We need to do experiments, we need to do like a lot of literature research to synthesize literature to justify why you believe on something. But it can be very simple, really like checking a few sources, checking from your experience as well, mixing those things to justify why you believe that something’s true of something is the case. So that is a level.

NICOLE: I like when you say that, when you specify that it’s an argument, but not between two people. And it kind of made me think about how when I write something and something that I’ve done research on, I like to use steel man arguments. And the idea, I mean you maybe know this, but just to explain to other people, there’s something called a straw man where sometimes in order to prove a point, people construct this very flimsy argument and then they show how it’s wrong. But I really love constructing a steel man which is a contradicting point of view that is actually valid in some ways. And that way, when you are able to convincingly show that that steel man is incorrect or inaccurate, then it kind of strengthens your own premise. And in that way, it’s like an argument not between two people, but kind of between conflicting ideas.

BIANCA: Yes, yes, yes.

NICOLE: I wanna shout out as well, there’s some people who are saying they use Obsidian for both academic and non-academic research. So Grisha is agreeing that it doesn’t matter if the purpose of the research is for academia. And even William who’s a retired academic, agrees with you about research as knowledge creation. And a note about another Brazilian talking about PKM and Obsidian. They’re a cybersecurity researcher.

BIANCA: Oh, that’s really cool.

Taking notes for topic research

NICOLE: There’s a question here from Jan.

Could taking notes help in topic research? Using Obsidian to write random notes. And I’m interested if I could somehow use it to get some ideas for my thesis engineering field.

This is a perfect question for you.

BIANCA: I would say you can have a whole system just to help your research. That’s the area that I like to talk about. So you can spend one hour, even more hours here just talking about that. But I suppose that you were saying to get idea, so it’s really the beginning of the process of research. So it’s really like, what do I research about? And I would say definitely, you can definitely use your notes for that. I don’t know.

You may agree with me, Nicole, that our research, our notes taking space, note making space is really to put our thoughts in there, to let our minds really flow into the page or in our case, to the digital page. So when it comes to identifying the top, the first thing that we actually need is like, what are our interests? What do we care about? What do you want to work on? What will be spending, if you are thinking about the research topic, what do you spend the next years work on? So it needs to be something that you have interested in learning more about. So I think our note making space is a perfect place to just like put things out. What are the things that I like? Let me write everything I know about them so that you can then decide, okay, which one I want to maybe pursue, so investigate a little bit more.

So I think that would be, and I would say, you ask, Jan, could you take notes? Could take notes help topic recency? Definitely. I think that would be one step that would definitely help in defining what is the topic for your research.

NICOLE: I didn’t actually plan on doing this, but I’m gonna share my screen because this is something I’ve thought about a lot as well. So I have this thing called lightning rods of thought. It’s actually from a D&D or an RPG person.

Sly Flourish is a YouTuber who has the idea of using lightning rods, which is adding things in a game that cause the players to use abilities that they already have. So it’s kind of like attracting what the abilities and interests that they already have. So I had this idea of lightning rods of thought. And this is how I try to find out what things I’m interested in. I mean, there are many ways, this is one.

So this is a Dataview query that I use and it says, it spits out all of the notes that I have by the number of outgoing links, and this is just the top 50. But I am trying to identify maps of content, for example, and it’s going by descending order on outgoing. And then there’s also incoming links. So you could look at it that way. How many notes do you have that are linking to certain notes? And it’s like, these are the things that I am interested in apparently. So it’s just useful for me to start here. This doesn’t mean that I stop here, of course. And I can also embed searches for different keywords. And sometimes, it’s useful to look at orphans, the things that have no links to them. And I’m not sure why. Maybe I should think about that. That’s just as an aside, I didn’t intend to show that, but I thought it was kind of relevant.

BIANCA: Yeah, I think this is really good. I do something similar, although I don’t have the queries anymore ‘cause I’m not actively using Obsidian, but I do a similar thing, which I call it generalizations. So you’re like, what are the things that are making everything else be glued to each other? Let’s say, what are the central topics that everything is linked to or who is linking to everything else? Usually take the ones that is everything links to that, those are the things that I have been working around them a lot. So those are the core for my system as well, so I’ll do something similar so that’s really nice.

NICOLE: Yeah, it sounds like the same thing. It’s like abstraction and finding the highest level of abstraction. And from there, you can start digging into it. So I guess to answer the question that Jan asked, yes, they could definitely help. Taking notes could help in topic research. And one way that you can try to find out what topics you’re interested in is by trying to find these clusters, like what Bianca said, the generalized, the most generalized topics that you tend to write notes around. There’s another question for you from Grisha.

Bianca, do you first take notes on paper and then put them in Obsidian?

BIANCA: No. I still love and in paper, I love it, but there is just no time. So for me everything is like, phone all the time or in the computer is direct and digital because there is just no time. Especially now with small kids, there’s just like not an option anymore.

NICOLE: I definitely get that it’s hard. I always have a real notebook and a fountain pen. But my usage of it has shifted to like journaling and brainstorming and not really like as you say, Bianca, sense making.

AI for research and note-taking

NICOLE: Nikhil asks, “If applicable, have you come up with a workflow regarding materials? PDFs, videos, etc. to use an Obsidian, ChatGPT or GPT3? I’m looking to use it for academic research.” Have you looked into that?

BIANCA: I use ChatGPT most for copy edit for editing. It’s really, really good. I was doing a try, I was okay if I want to do a literature review. A base one like a scoping review, which is like the very first type of, could I use that? I even published it on Twitter sometime ago and ChatGPT has some good things and some very bad things. Sometimes, it’s just retrieving information that is random. I know there are people who use for example to summarize the source. If you don’t want to read the whole source, you just want to know does it talk about it?

Some people use for that. I haven’t tried to use for that. I prefer to process my source in a different way. But those are, I think the best, best, best section on the writing process where it’s actually like, “Okay, I wrote it this way, can you edit a little bit and see if sounds a little bit better?” So I think for copy editing and ChatGPT is really good.

NICOLE: Yeah, I want to add that I also made a video on AI for PKM. I talk about things like using Napkin. Napkin is something that brings in your highlights from Readwise and auto and in a smart way uses GPT3 to tag those highlights. So since Nikhil was asking about different materials, like PDFs and videos, if Readwise processes those, then Napkin could see those and start to intelligently make links.

With Obsidian in particular, since he asked about that, there’s text generator and someone from Mastodon, Rick Raftis actually wrote this blog post that I found really interesting and it is about using Elicit and Consensus, which I don’t really know much about, but I thought that the blog post was interesting.

Obsidian Canvas vs. Scrintal

NICOLE: And so what else is there? (Reading a comment in Spanish)

Greetings from Chile. Últimamente he estado utilizando canva para hacer presentaciones para mis clases, de modo que cuando actualizo la nota citada en la presentación, se actualiza de inmediato visual

(English) Oh I’m sorry, I didn’t even highlight that. So just to translate here, for those who don’t speak Spanish. This person is saying Greetings from Chile and they’ve been using Canvas, which is an Obsidian feature to create presentations for their classes. And so that when one note that is cited in the presentation is changed, then that Canvas is automatically updated. That this is a really good use case for it. And maybe this is a good time to also talk about another alternative for Canvas, because Canvas is for Obsidian and in many ways, I think it’s Obsidian’s answer to Scrintal. What do you think about that?

BIANCA: Yeah, that’s true. Yeah, just for who doesn’t know, I was a very passionate, obviously the user until I found Scrintal. Which kind of strange. I still love Obsidian, but I don’t use it anymore.

So Scrintal is an app where you can visually organize your notes. So Obsidian has Canva so you can decide to, let’s say just start with Canva, it’s more like organized in those stuff. It’s too easy to create new notes in Canva, but in Scrintal, everything is visual. So you always like to create a note and then the note appears a card already in the background from what you’re creating. So it’s really to visually organize the notes.

And it started before Obsidian and then Obsidian appear with confidence, like features, features, features. So there and there is now also Logseq, which appears now with whiteboard. I didn’t check yet, but people are saying that it’s actually also really good.

So yeah, I became a Scrintal user at some time ago. Mostly because I was using the phone and the phone at the time, Obsidian didn’t have the mobile app. And for me, one of the main challenge that I had is like, okay, I’m writing something and then I write too many notes and just like start linking and connect to the note. And I jumped another note, jumped to another note, jumped another note. And I always found it hard to remember where did I come from. I was like, “Where did I come from? Where was I working again before I jump into this note?” And Scrintal gave me that automatic note. As I’m creating the notes, the notes are in the background. As soon as I close it, i have all the other notes there. They’re in that same visual space, the same context, let’s say. So that’s what made me change.

I noticed at some point, I was using less Obsidian I was getting kind of stuck in my way of writing notes, in my way of thinking. And then Scrintal was kind of like accelerating, I was like, okay. And that’s when I did the change and it was before Canva, before the mobile app was released as well.

NICOLE: We have a question here from Mickael Fortunato.

By the way, Bianca, are you only relying on Scrintal?

BIANCA: Yeah, nowadays for idea management, yes. Now, I moved with everything into Scrintal now.

How important is visual thinking in PKM?

NICOLE: So let’s talk about this a little bit because I was also an early user of Scrintal. I think I talked to the team before it was released and gave my feedback. Ultimately, I decided that it wasn’t for me. I actually really liked the visual part of it. I really, really liked it.

What I didn’t like about it was that it’s not plain text markdown files anymore. That view, I mean Canvas is the same. If you use Obsidian Canvas, that’s no longer a standard format, which is also why I hesitate to use it because I just really like plain text. But I do love the idea of adding, of bringing in visual thinking into note taking.

I think it’s very visceral because when we see a note or even like something that looks like a note on the screen, instinctively, we know how to use it because that is a metaphor for an actual piece of paper that we can interact with. What do you think about visual thinking? Do you think that it’s always necessary for research or is it just something, another avenue that might be useful?

BIANCA: I find it, personally, extremely important. And the main reason is when you talk about knowledge creation, we need to be able to get our ideas, to put ideas together. Sometimes a big number of ideas so that we can create something from them or take them and break them down.

So for example, I talk like about sense-making strategies, they’re usually like you put the ideas on a space which will be a board, let’s say. And how now we can connect them, create new things from them, generalize from them, see the context about them. In all those steps when you’re playing with ideas, then you can be creating new ideas. Because what I notice that sometimes, we just take notes about, let’s say, what you found, our source. we have quotes, quotes, quotes. But then what is the idea?

At the end, of the day we want to create knowledge. Where are our ideas? So even my whole note, note making is like each note is one idea, which some people they call the atomic notes, although I don’t know anymore because we have many discussions with people who take atomic notes and we have some disagreements. So basically like each node is an idea.

So when you take ideas and you put them into a board for example, they’ll start to organize them. Or where things that are similar, somehow they can be closer together. If they have another type of similarity, you can now add color to them. If you have available shapes, shapes can have meaning as well. So if all those things you can start rearranging the ideas, organizing, grouping them and seeing how can I find the relations between them and which new ideas come out just by putting those things together?

The visual organization of notes is very important for knowledge creation itself because then, you can see the ideas and reflect about them so they can create something from them. Which pure text usually is really hard to do with pure text. Or with just like 10 notes around the screen, but not in a visual organization.

NICOLE: So this is a related question from Jan:

When is creating a Canvas or something similar necessary? Is it not tedious to do every time?

BIANCA: Well, my take is that it’s important to see why you want to create that. Do you want that as a navigational board for example. If you want to have an overview of everything, your vaults for example. Can you have one board that gives you the entry point to all these spaces in your board, in your vault? That’s one.

Things that you can use for, or can use as a sense-making space, you can bring up some notes or cards that you want to make sense of or create something from that. But I would not say that it’s always necessary.

Sometimes, all we want is just like open a note and do a brain dump, for example. Don’t need a board for that. Or you just want to work on the content of one single note. Maybe explain a little bit more what their ideas about, so we don’t really need a board for that or Canvas for that.

So I think it depends a lot from what you want to do. What do you think, Nicole? What would be your use for Canva?

NICOLE: Yeah, I think, I don’t wanna say it’s necessary because I just hate imposing what I think on other people.

For me, some element of visualization is always necessary. Yeah. And sometimes, that is me drawing up like an atomic visual from the beginning and other times, I’m just all text and then I rearrange the text on Canvas or something or Excalidraw.

And then other times it’s all purely based on the relationships and I’m just visualizing those links in either the graph view or ExcaliBrain.

But I think our visual processing as humans is much more advanced than maybe other senses. Like common sense isn’t common, right? So it is something that we actively have to practice. And so having these visual shortcuts I think is really helpful.

So for me, it’s always necessary, but I don’t always go to the extreme for everything ‘cause yeah, that would be tedious in that case.

Note-taking in multiple languages

I want to jump into Scrintal because it’s been a while since I’ve last seen it, but I can’t resist this question from Deb Leg who says:

As someone who also juggles languages, I’m curious to know how your knowledge of several languages impacts your thoughts or notes process and which languages you use for research and knowledge creation?

BIANCA: That’s interesting. I just use English all the time. Because I figured I started my research journey even when I was in Brazil, everything was in English, all papers were in English and for computer science all technical language was in English, so everything was always English. So I decided to just have one language for my specific, I don’t even know how to call it more knowledge base vaults or anything, I like to call knowledge base.

But one thing that is interesting that I was discussing with one of the members of my community the other day is that ‘cause you’re saying, well, I write in English and in Japanese. And one of the things that we notice with languages is that there is translation and there is how can I actually express this idea in a different language? And sometimes, translation is not the way to doing that. Because I always say, “Well if you take notes, for example, if our notes represent an idea, the title of the note is the name for that. How do we call that idea?” And sometimes, how we call that idea in English is not just a translation from stock in Japanese or in Portuguese. It’s like how do we call that idea in one language and how do we call that idea into the other language?

So that’s one one of the interesting things because I think the main challenge now is really like if you write in two languages, when it’s time to search for your notes, which language would you use to search? So it’s really hard. It’s like, which word did I use? Did I translate? So one thing could be like, okay, I have a title in both languages, if that’s important idea, just have, how do I call that idea in two different languages. But it is really hard to find later if you’re not very disciplined on how we name our ideas or how we write.

NICOLE: For me, I can’t really control it all the time. I think my daily notes for instance, are in a jumble of languages, whatever I was thinking in at that time. So they are messy. But what I do is at the end of the day, I then process it and then I do translate most of it to English.

Some things are naturally in English for me. Like I study, so my native language is Tagalog, English is a second language and it is my best non-native language. It is also the language that I studied in. So when I talk about anything that is like mathematical or computer science related, because in programming, even the keywords are in English, so it makes sense for me to translate those into English.

But what I’ll do is I’ll have an alias in Obsidian with something that I might refer to it as in another language. So for example, performance testing is the main note for it in English. And I’ll put an alias that is “pruebas de carga” which is the same thing in Spanish because I might be searching for that.

And I also noticed that with people, the notes tend to be in whatever language I was speaking to them in. So if I’m speaking to someone in Dutch, then my notes will be in Dutch ‘cause that kind of makes sense.

Omar Ortiz Meraz is saying:

Usually, I reserve each language based on topic.

Yeah, I have a bunch of private notes around relationships and relationship dynamics that are in Tagalog because that’s how I think with relationships. And looks like Omar’s also saying English for technical things, yeah.

Filipino, Tagalog and English here too.

Hey, (Filipino) compatriot, how are you?

Demo of Scrintal

NICOLE: (English) Let’s get into Scrintal, Bianca, if you don’t mind showing us what it looks like now. It’s been a while. It’s been like months, maybe a year since I looked at it. Should I add it?

BIANCA: Yeah. Can you add? Can you see my cursor or?

NICOLE: Yes, we can.

BIANCA: Yeah, okay.

So Scrintal, the hard part is that you are faced with a blank screen and you’re like, okay, there is a world of possibilities but I have no idea how to start. So it’s really like you can just create a note, which is called card here in… I like to think like it’s similar to the physical world, let’s say. You have your desk, you can double click or create that to create a card where you can just write wherever you want and create a title. My idea.

And then they also have this slash comments to actually add things. A link to a card or a link to a board. You can link to from a card to a board. Upload things, emails, add YouTube videos, tweets, upload, and the normal formatting that we have.

And if I want to create a link to another idea, I could just here plus. And then there’s another idea. So let’s say they all just appear on the screen and if you just close everything, they are there in the board already linked to each other.

So basically that’s the basic idea of Scrintal. So you have cards, you can have a desk and you can have a board, which is exactly the same thing of a desk, but you can save a board and give a name to the board. The desk is just one, where boards, you can create as many as you want. So you can just click here. Or for example, if I want to create a board with those, I just right-click, create a board. And then I have this new board, which I can give it a name and add a link for example, inside. Inside of here. So this is a link becomes a link to board rather than linked to a card for example. And it has similarly like have backlinks. So it appears when you have links to boards. In which boards does this card appear? Because a card exists in the panel, the board. It can appear as in multiple boards, it can be no boards at all.

So that’s all those things appear there. And if you want to see the rest here, then you can see all the cards that you have, all the boards you can share, you can actually have multiple people work in the same board. The same set of cards. And here, you can have just a normal search that you can search to find the boards or the cards again. So the cards they exist independent of anything, you can add them to your desk, you can add them to a board. So I can go out here and here on my desk. For example, I clear my desk, my desk is empty. But then the board, you still have those two cards. So you can use a board as an MOC for example if you want. You can use a board to make sense of things and save. It’s pretty similar tool to combine that sense because you can also link Canva I think in Obsidian as well. So those are the main feature.

We can add color as well if you want. You can add tags. Yeah, those are the main things the Scrintal has at the moment.

NICOLE: So we have two questions here on what were also my objections. Rodrigo Alexandre Costa:

Scrintal’s cards or boards format is proprietary.

And a related one:

How can you export your notes from Scrintal?

BIANCA: Yeah, that’s one thing that I have been asking the team for many months, the exporting part. They are working now on a full export of all the cards. Currently, can export one by one in Markdown or by just all the PDF, but they’re working on a export all feature. So it should be here, I think in the next months.

In terms of proprietary, yeah, they just have their own format. They don’t share what’s the format, but then you can export in Markdown. So ideally, in the future you could have, let’s say you write your things in Obsidian, you upload into Scrintal, you change the things there, there download, put back into Obsidian. So you can ideally be able to use both for depending what he wants to do. But yeah, we are not there yet if anyone wants to use that way.

NICOLE: I wanna say that to be fair, Canvas is the same way. So the Obsidian team hasn’t quite figured it out either. That’s kind of why I’m hesitant to use it because it’s no longer Markdown. And technically though, the format is not proprietary, Rodrigo. I don’t know about Scrintal, but for Obsidian, it is in JSON. So it is a standard format, but it isn’t nice to work with. It isn’t really something that you can export to another app or something. I imagine that the Scrintal team are facing the same limitations.

BIANCA: Yeah, well to be fair with Obsidian as well, this is a very new area. Like just now, there is like Scrintal, Obsidian and Logseq pushing into that direction. So there are really not no open standards to deal with this type of data because even the different tools are trying to figure out what is it that you’re creating, what is it that we want to represent, how we can represent that information.

So I think it would be very good to like, if in let’s say five years time maybe, or hopefully before, there will be open standards for those and multiple apps who maybe even work together to try to create those open standards that they all support. So I say, fingers crossed, that will be the future. But yeah, so far, I’m not even sure if there is any open store that would cater for this type of use case. So let’s see where it will go.

NICOLE: I think the closest one would be Mermaid.js. which Obsidian does support, but you have to learn it. And it’s also not as flexible as Scrintal is for example. I want to ask about the board and desk concept though. It kind of sounds like their analogous to maybe folders and subfolders or is that completely wrong?

BIANCA: Well, first there is no subfolders. You cannot have a board inside of a board, for example, board is a separate thing. And they’re not totally, they are similar a folder in the sense of container. Like it can bring things inside of it, let’s say. You can put cards inside of a board, but the same card can’t exist in multiple boards. So they don’t have the limitations of a folder because in a folder, like you put on a folder and that’s it. You cannot really put in multiple, you can create shortcuts, but that’s another story. But then really like folders, is this folder or in that folder? Whereas boards is really like, okay, you can just put on a board, remove, put it back again, put in five different boards. So the boards are much more flexible in that sense.

Taking notes on source material

NICOLE: So there are a few questions about how you would actually use these boards for research. So for starters, someone said if you were taking or making notes from an article, would you make many atomic notes or one main note? And then my question is, would you also like create a single board for that article and have all of your notes on there? Or would you make a board per day? And then a follow up question is from Jan who is asking about the serendipity aspect of it. So what if you just want to be able to explore your notes and discover new ones?

BIANCA: Just try to remember everything. I remember this.

For the input one, as I said, like I work with idea note, let’s say, so each note is one idea. So when you’re reading a source, there are tons of ideas in the source that you want to capture and understand. And there are main ideas that spring out. Why are you reading the source? So I have like tons of notes that come out from reading one single source. So I have multiple notes from that. They all link back to the source, but I have multiple notes on that.

But in my case, I’m playing now with creating a board for that because someone asked me the same question, I’m like, “I don’t create a board for that.” So I was like, “I’ll try and see if there is any benefit of doing that.” I did not find any benefits so far, unless you want to make sense of all those ideas together. But then you can create a board later. But while I’m reading a paper, I don’t really use a board. I just like, there are many ideas that I start to, many notes, many cards that I start to create, create, create, create. But I don’t actually create a board for that specific input. I don’t see any specific utility unless I want to make sense of those ideas even in isolation from the source or get group of ideas or things like that. Yeah, so that’s that.

NICOLE: So Niky Dix seems to be familiar with your work. They say that you have a brilliant method for using a combination using OneNote and then making atomic notes. Can you explain what they mean?

BIANCA: Yeah, well Nik is from our community. Because there is a process. one thing is like for example, how I do it, I create many notes. And what I talk about this, I’m not sure if Niky is referring to that, is the idea when we have the bibliographic note where like I call the input note, it’s a note that represents the source itself.

So this my note that represents the source and I have all my ideas linking to that source. And the reason why I link to that note and not direct to the source then I am talking about is because then I can see, I can answer two questions. One is where this idea came from because it has a link to the source and the other is like, what are all the ideas that came out from this source? So just go into the note, the backlinks shows me all the ideas that are related to this source. So I think maybe that’s the what Nik’s talking about. Using the OneNote to represent the source.

So I don’t use a board to represent the source to say, ah, this is the source and all the ideas in the source. Instead, I have one note and all the notes that refer to that are linked back to that source. So the backlinks by now can take, what are all the ideas that came out from that source? I don’t know, Nik, is that what you’re referring to?

NICOLE: Another person is also asking, “So you have a base note for the source and connect all notes?” So if I understand what you’re saying, yes, you have a note that is for that source and then you either explicitly connect other notes that have, like, you create notes of ideas from that source and you either explicitly link them or rely on backlinks to connect them to the base note. Is that right?

BIANCA: Yeah.

On the usefulness of highlights

NICOLE: And is it like, do you use something Readwise to do that? Or do you just manually create a base note?

BIANCA: I manually create, and actually, I was teaching a few weeks ago and one of the outcomes was that many people say, “Okay, I’ll remove my Readwise subscription.” because I’m really like, the whole flow, I talk like, highlight is good, but use highlight as a way to say, this is my note. There are many shortcuts especially for research.

‘Cause in research, you want to create knowledge. So our note should not be just a highlight. It’s like how can we have our own understanding inside of our system? And Readwise sometimes like, okay, I highlighted this in Readwise I can just put later into my system. And what usually happens that when we take that approach, let’s say we highlight and see we process later, later never arrives. So we end up just like our notes are just highlights, but we never process our understanding of what are the ideas that came out from reading that, from try to understanding that So that was like, well maybe highlight is good for when you want to reread the source, but if it’s the process of understanding, really need to write down what is our understanding. And if we use a tool that just say highlight to see it later, later never arrives.

NICOLE: Yeah, I think that’s when the problem is when you treat something like Readwise or any read it later app as the end of the process, then that’s still a very superficial way of learning really. I kind of treat it as the beginning. And then I process it from there. Jan is saying:

Do you mean highlighting encourages resource hoarding rather than taking notes?

I definitely think it does. Do you agree?

BIANCA: Yeah, I agree. Like especially when it’s highlight only. If it’s highlight only, then I think that’s the worst of the worst. Because sometimes, okay, I highlight and then Readwise, you just create my notes. I just import from Readwise into my Obsidian for example, and I have my notes with my highlights. But then like, so what? Those are other people’s words, other people’s thoughts, something they try to communicate, but what did we understand for that? What do we try to do with that? I think it means a little bit in the personal knowledge management. So I totally agree with you.

The one-pass reading strategy

NICOLE: So Niky Dix is saying that he was referring to the one pass strategy where we use free writing from one source and then make idea notes using the pick a phrase and then create idea notes and reverse outline. Okay, we need to unpack this. Can you explain what he means?

BIANCA: Yeah, yeah, yeah. As a solution to this process, the highlight and the process later and later never arrives.

So when it comes the time to write from our research, there are no notes. Because we didn’t actually process those highlights with our understanding and build the knowledge that we want to write about.

So what I develop is what I call the one pass reading strategy. So that is when you finish reading, you have all your notes ready. Even if you stop in the middle, you are ready, have all your notes ready for you. So the process where you go, I develop a process step by step how we can do that. And that’s where all those words come from. We are reading something, we feel the urge to highlight something was either interesting, relevant, or puzzling. We stop immediately rather than just highlight and keep moving on, we stop and then start making sense of that.

So it’s really like, let me free write and just write my understanding of what I just read. Even try to explain what the person tried to tell me or try to explain how does it relate to my work or anything that comes to mind really. So just really like free writing that.

And then when you talk about idea notes, many people use the thing of atomic notes, I say, or idea notes as like, okay, what are the ideas in the highlighting? And that’s why I say don’t do that. The highlight is someone else ideas. You want to extract the atomic notes from what you wrote from your free wrote. Because those are your ideas, even if they’re just an explanation of what the person said. So those are the ones that become the atomic notes. And then you may want to cite to mention, okay, these thoughts actually are not mine. They came when reading this quote from the source. So that I can say, this is not mine, it came from this source.

But then that’s where I use and some methods that I teach on how to create those idea notes. Those are the pick a phrase and the reverse outline methods. So those are some methods that to help convert a free writing into idea notes. So that’s a little bit of what Nik was referring to.

NICOLE: Okay. So what’s so interesting is that you’re using different terminology, but I’ve kind of arrived at the same conclusion, at the same process.

I think like Tiago Forte uses the term progressive summarization.

And since you’re a developer, I like to think of it as like continuous note taking in the sense of CI/CD processes. For those who don’t know, in tech, we are now emphasizing a more agile way to work where that is more iterative. So it’s not like in the first pass, you get everything down and you’re done and you move on. It’s like you do, it kind of sounds like what you’re describing, Bianca, where the initial pass is like freeform. You’re not really separating things out yet. You’re just getting through it and then you go back and iterate over it, and then you start creating idea notes that are now in your words, not in the author’s words. And then you keep like continually processing it from there. So I’m seeing kind of a lot of similarities.

Do you think that it is significantly different from other methodologies like Zettelkasten or progressive summarization?

BIANCA: I cannot say that I understand completely progressive summarization or the Zettelkasten. What I understand, progressive summarization is that you stop too much on the source side of things. It’s like you try to highlight and summarize, is it too much on what the other process try to say, and it takes too much time to arrive on you, let’s say.

Whereas my case, let’s say like as soon as you write something that is important, your thoughts should go down, your thoughts should go there immediately.

But I agree with you on the agile process, actually even the people in the community, they heard me talking about agile a little bit because even I call agile note making. Because like yeah, let’s I compare to I guess the waterfall. So there’s a lot of like computer science things going on.

NICOLE: Oh, that’s awesome. I love that.

BIANCA: Yeah, so-

NICOLE: I’ve had a lot of the same thoughts.

BIANCA: Yeah, so it’s really like the process of really iterative. It’s really like put your thoughts out. Because one of the big problems that people have in note making or in writing, especially in research, is that we try to organize first rather than create the things first or express our thoughts first.

So it happens in some note making tools. They force you to think about that because okay, where would I put this note even when I don’t know what I write inside of the note yet. Whereas I see more like, first you express what you want to say. I don’t even know what I’m gonna say, then just write it all down. And then the process of organizing those notes so that I can find them later.

And the atomic note is really a process of organization. We are trying to organize our ideas in a way that we can grow them, using some sense making strategies or we want to find them later. I see that that’s why like the process really like. Write everything down first, then you express yourself. Now let’s start to organize as a second step.

Nicole showing her process

NICOLE: So I want to show quickly this something that’s in the process of me going through and creating and synthesizing my thoughts on it. This is an example of highlights from Readwise. And as you say, this is really like, it’s all source, there’s none of my own thoughts in here, but it’s like a prompt for me. It’s the beginning and then I remember what I thought was interesting about it, and then I create another note but that is about still what the author is saying, but now it’s in my words.

So I like this thesis, an antithesis, antithesis and synthesis kind of structure. It’s the Hegelian dialect and I like that in the first one I talk about what are the main part points of the book? What is the author trying to say? But in my own words. And then I also think about similar ideas that other authors I know or have read, wrote about or said about the situation. Then antithesis is like, what are the things that are wrong or what are the weak points of that book? And then how can they come together?

So obviously, I’m not done with this one and it’s still in progress, but I just wanted to show how it can be done like in Obsidian as well. I guess in Scrintal, this would be… I’m interested that the process is really very similar. It’s just, okay, the tool is different, but we’re actually doing the same thing.

Which tool is best?

BIANCA: Yeah. This one, I like to say like, because most people they ask, “Okay, which tool should I be using? Should I then move to Scrintal?” I was like, “Wait, what do you want to do?” Because it’s more about which type of thinking we are trying to support and then which methods we can use to put that way of thinking to practice. And then what is the tool that I want to use?

So many people say they started with Obsidian, they say, “Well that’s not for me. I want something else.” And they go to Scrintal. Some people start Scrintal, they say, “That’s not for me. I want something else. They go into Obsidian.”

So it’s really like, what do you want to do with the tool? What method supports that? And then, okay, which tool will support me in applying that method? So I like to think more on the mindset, what I call the mindset, and then the methods. Then choose, what’s then the best tool for my use case or for this specific use case?

I think I can maybe share my personal screen. I was saying that I was, I didn’t want to…

NICOLE: Totally up to you. You don’t have to if you don’t want to, but yeah, sure. But I really love what you’re saying about tools because people think like, “Oh, my channel’s about Obsidian so I must think that Obsidian is the best tool for everyone.” I don’t, I hate saying something is the best for everyone. I’m like the opposite of that. That is the opposite of inclusivity.

I always say try it out. a lot of these tools are free or have a free trial or a free tier. Use it all. And if that’s Notion or if that’s Evernote or Obsidian, like go for it. We all think in different ways. Why do we assume that one tool will placate all of us?

BIANCA: Exactly. I totally agree with that.

NICOLE: Yeah, if you’d like to share your screen, you can-

BIANCA: Yeah, I’ll try.

NICOLE: Just go ahead and do that.

Languages as thinking hats

NICOLE: And Nik was also saying:

Write first, organize later. Sounds like what Nicole did with her daily notes in multiple languages too.

Absolutely, that is exactly it. My daily note is kind of like Bianca’s one pass process, what is it called? One pass reading. And that’s because it’s just like a log. There are bullet points. I am not really thinking too much about what language I’m writing in, but then when they get processed, I mean, very few people are going to want to read a single article in multiple languages that might not coincide with their languages, right?

Mage Prometheus says:

Writing down your own words immediately helps identify where you don’t understand. I don’t like to read on if I don’t understand what I’ve just read.

Yeah, and I would also add to this, something I didn’t mention with the multiple languages question is I actually use my knowledge of multiple languages to test myself. And kind of like the question that was asked earlier about AI. One thing that I am actively using AI for is to provide me the opposing argument. Here’s what I think, tell me what’s wrong. And that is super cool because sometimes the AI, GPT3. or GPT4 says things and I’m like, I should have thought of that. That’s actually a pretty solid argument that has me rethinking how I should argue, how I should present my evidence.

And I do the same thing with languages because I find that for me at least, like certain languages are linked with certain attitudes. It’s kind of like the six hats, the thinking hats from Edward de Bono. That’s what languages are for me.

If I have a premise out there and I think I want to attack this logically, I might try English because that for me is the logical language. But if I want to think about like, well what does this mean in terms of people? I might do Filipino, Tagalog or Spanish. Then if it’s something that’s more like, about freedom or liberty, okay, let’s do the same thing in Dutch and let’s see what I come up with.

It’s actually interesting, the things that I come up with just from putting the same thoughts in two different languages. It’s like, why did I use that word in Dutch? But I didn’t use a direct translation in English. It’s like, okay, that’s kind of interesting.

BIANCA: Yeah, that’s why you see like it’s really about the idea itself, not the words. Because when you start to thinking what is the name for this idea? That’s where it’s like, oh, okay, what am I actually trying to say here? What’s the idea that I try to talk about? So I find that fascinating.

NICOLE: And also agglutinative languages like German, Dutch or even Esperanto, they just let you freeform, push two ideas together that maybe don’t belong. And I love that. We can’t just do that in other languages. So it’s super useful from that perspective. Just putting a name on a feeling or an idea is so powerful.

Sparking serendipity in research

NICOLE: I want to get to the question earlier about serendipity, ‘cause I think that’s really important. So we’ve talked about how you process something for research, you have a base note for the source, you create idea notes that you pull out in kind of like an atomic note process. And then in the base note are things that are just more free form, but still in your own words, if I understand correctly, or is that like a separate note?

BIANCA: I feel I can share now. I managed to share so.

NICOLE: Oh, okay, great, yeah.

BIANCA: Just look at it.

NICOLE: Yeah.

BIANCA: So that’s, for example, this will be what I call, it’s some people call biographic notes, literature note, it’s just an information, what it is, some videographic formation, what is the link. And for example, in here, I don’t have really anything. It’s just a summary. Very tiny one. ‘Cause i still write, see, I’m on page six. I read almost nothing from this book.

But then, those are the backlinks with the ideas that come out of it. There is sometimes information about the person or information about specific idea. And then I have some explanation here, which links to other things. So that’s really how I link. I explain the things and link back.

What I link, I link back to this note, so I can see here what are all the ideas that are related to it. So what I started doing, I started writing sometimes here just like free form, free writing. I just open here. There was something interesting, I just start writing here. And then I separated them as soon as I finished writing.

Before I started reading again, I separate them into their own notes or put into a note if already existed a note about that specific thing. And that’s how I go.

So I don’t keep my original free writing, I just like throw it away, except in some situations. Oh wow. Where it’s more like the decision but it’s more like I’m reporting, I’m writing a decision that I took or is more journaling, how I’m more about why am I moving my research in that direction? If it’s from a source, it cease to exist as one text. It’s actually existing multiple cards that’s breaking down into multiple cards about different ideas.

NICOLE: I gotta say, that makes me anxious just thinking about it. You delete it? I do the same thing, but I keep the original.

BIANCA: Yeah. I say, well if it makes you comfortable, keep it there. And maybe the daily note can come back the best. But for me, it’s just like… Because it’s still there, but it’s nothing in one single text. It’s in multiple place. Each idea is a different note.

NICOLE: So how would you stumble across? Like this note that you have, this sign or symbol. Is there any way that in Scrintal, you can stumble across that in a different context? Like not when you’re looking into this book?

BIANCA: Yes, and sometimes, it happens with me because when you’re creating idea notes, when I talk about ideas, ideas are not just things, for example, this the example here, I have let’s say sign or double articulation.

We could think them as things or concepts, but I can say maybe like signs that have, no something say, double articulation, for example. Let’s a statement. I’m saying that signs have this property. So in that case, I will say sign. And just by doing that, many times, I started writing and I just say, okay, I’ll explain this idea here.

This is a new idea from me. I don’t wanna talk about science or I don’t wanna talk about whatever the word articulation is. I want to talk about the relationship between them. And many times, when I’m doing that, I notice I may think that this doesn’t exist. I will write for the first time, but then as I’m writing I say, whoa, I have a note about that already. It happened many times. So when I start thinking about the relationships with between things, I start noticing, I have notes that I didn’t know that I had before, and they have text in them, they have already content in them. I was like, whoa, I’m really happy with my previous self for having created a note for me, let’s say.

But in terms of serendipity in general, I wouldn’t say that Scrintal is focus on serendipity. It’s not like Napkin. Napkin focuses on serendipity. You open a note and two other notes just appear beside it. And so the focus of is really serendipity. Obsidian, you still have some plug, let’s say, give me a random note that is that level of serendipity.

In Scrintal, is really up to you. It’s really like, just open my card and open a random card and start to read again. And what I find interesting is that’s very often, we take notes, but we don’t read them. I mean, we always go to read something else from the external word, but we don’t read what our previous self wrote, let’s say. Very often. So I think a little bit of serendipity comes just by going to our vaults or our systems and reading our notes again.

NICOLE: Yeah, there are a few ways that I like to do it. I think last week, I released a video just on Serendipity for Obsidian. But I think that the concepts are the same.

Although I do use Dice Roller for Obsidian to randomly select elements from tables or also notes. And you can say like, show me all the notes with this tag. Or you can do a bunch of different things with it.

So I like to use that, but I also like to do chronologically. Like that’s why I used daily notes. In every daily note, I have a link that says like, one year ago today, here’s the daily note. And then I see what I was looking at or what I created on that day. And sometimes, I find seasonal things, like if I’m working on something for Christmas, it’s like, hey, this time last year I was working on that too.

So that’s another way, by date, by topic, random or visualization, which I think Scrintal, I don’t know if Scrintal can, I guess it is good at letting you visualize different ideas, but is there a way to visualize all of the ideas? Like in all of the boards? I don’t really remember.

BIANCA: No. Not yet. I tried, not a very good idea. I tried to put all my notes into on board and not yet. It just like didn’t open, everything crashed. It was just like, okay, lowsy, and we stayed there. I hope that the team has some ideas for that so you can actually see everything. But at the moment, there is no such option.

How do we find the time to make notes?

NICOLE: So we’re already over time. So I just wanted to end on one question from random user:

How is there ever enough time to make notes on everything you’re interested in?

BIANCA: There is not enough time to make notes on everything you’re interested in, that would be the main answer. But I think for the notes that we can take, I think it’s worth taking the time to take and engage with them later on. So I feel that that would be my take.

NICOLE: That’s a really good answer. I totally agree, there’s never enough time. I relate so hard to this question because I want to take more notes on things than I actually have time to take notes on them. But just like anything else, prioritize.

You’re not gonna be graded on this. Maybe you are actually. But for most other things, if you’re just a personal researcher, you’re not going to be graded on the thoroughness with which you attacked every single main premise that the author says.

So feel free to pick the ones that are relevant to you for the work that you are doing. And I think that’s a much more sustainable way of doing things because you never have enough time.

Bianca, thank you so much for joining me. If people want to know more about you, as I’m sure they will, where would you suggest they go?

BIANCA: Well, as we are on YouTube. I’m starting my YouTube channel. I have some livestreams there, but I will start publishing more frequently. And can always comment there and start chatting through there. Otherwise, I’m also on Twitter. Much more active in Twitter. Can just DM me at any moment and we can chat from there.

NICOLE: Awesome. And I also have the links in the description below of this video on YouTube. You can see Bianca’s YouTube channel there, but also, I don’t know why you didn’t even mention it. But you also have now, it’s not a course anymore, it’s a community, right? It’s the Prolific Researcher Community. Can you just talk a little bit about that?

BIANCA: Yeah, the main idea that I changed from course to a community is that many times we take a course and we say, “Wow, I learned a lot.” But then it comes a moment to put everything to practice. And I was like, “Okay, I’m stuck, so how can I get help?”

So the idea of the community that you, you come, you have an onboarding course where you learn how to take notes, how to express your thoughts, how to create a idea notes. And then, later in the second follow up course that you can take immediately or a little bit later. Because when you come to the community, come for a few months. Now that I have my notes, how can I create new ideas from those notes and use and write my output base on my notes.

So there are the course, the onboarding course and the follow up course, and there are a few others coming up, but there is also events or co-working sessions, Q&As, hot seat coaching, many other things.

It’s like, okay, you heard from me, now it’s time to develop your own practice. So then as you develop your own practice, then you get feedback, then you get support from me and from the community. And that’s why I change from a course into more community model because I think the one important aspect, okay, it needs to be something that works for you. That’s why I say it needs to be for each person. And we are there to support each one to decide which tool I want to use, how do I want to implement that? What works for my specific case and what doesn’t work for my specific case?

So that’s the idea of the prolific researcher community. And we have also course about research skills specifically. They’re related to the process of idea management or note making, but we have other things related to research specifically as well.

NICOLE: All right. Thank you, Bianca. I’m so happy to finally have you on my channel. We’ve been talking about it for a while. You’re welcome back anytime.

BIANCA: Thanks a lot for invitation. I I really enjoyed talking to you and thanks for everyone also who join us today. And I hope to be back and I hope to have you my channel at some point as well.

NICOLE: Yeah, absolutely. I’m game, just tell me when and where.

For everybody else, thank you for sticking around. Later this Friday, I’m having another livestream. This time, we’re going to be talking about something less serious than the topic of today’s discussion ‘cause we’re talking about all play, we are talking about role playing games. And I am going to be with my friend and dungeon master, Andy Polaine. We’re going be comparing how he takes notes and how I take notes on the same sessions, on the same campaign. So that should be interesting as well. Thank you, everyone for joining us and I’ll see you on Friday. Bye.

BIANCA: Bye-Bye.

Informed Life Podcast Interview

Sun, 26 Mar 2023 20:14:56 +0100

Information architect, author, and educator Jorge Arango interviewed me on his podcast, The Informed Life, about how I apply the principles of CI/CD to taking notes, why I use Obsidian, and how I use it to learn in public and play roleplaying games.

Grafana Cloud k6 (k6 Office Hours #83)

Fri, 24 Mar 2023 18:00:00 +0100

With the release of Grafana Cloud k6, you can get observability and performance testing in the same platform. Senior Product Manager Mark Meier and Frontend Lead Simon Legander join me to talk about what Grafana Cloud k6 is, how you can use it, and demonstrate what it can do to improve your system’s reliability.

Introducing Grafana Cloud k6

Tue, 21 Mar 2023 15:20:58 +0000

Grafana Cloud k6 is a feature available to all users of Grafana Cloud. It allows you to run load and performance tests right from the comfort of your Grafana Cloud instance, without needing to create a separate account on k6 Cloud. Grafana Cloud k6 helps you get started quickly with a GUI Test Builder or with test scripts written in JavaScript, and then visualize your test results in real-time, alongside your other Grafana dashboards that monitor your system infrastructure. With Grafana Cloud k6, you can get a holistic look at your system’s performance– on any environment.

Learn more about Grafana Cloud k6.

Randomness in Obsidian With Dice Roller

Fri, 17 Mar 2023 17:00:25 +0000

The Obsidian Dice Roller plugin lets you use randomness in Obsidian to provoke serendipity in personal knowledge management. Dice Roller is a plugin that was created for use in tabletop roleplaying games, but you don’t have to be a gamer to be able to use it to spark creativity in your thinking for other topics.

Resources Mentioned

Experimental modules in k6 (k6 Office Hours #82)

Fri, 17 Mar 2023 16:00:35 +0000

Experimental modules in k6 were released in v0.40, and new modules for distributed tracing and browser testing have been added since. In this video, k6 Senior Software Engineer Théo Crevon talks to Developer Advocates Marie Cruz and Nicole van der Hoeven about what experimental modules are and how to use them.

The Future of Obsidian with CEO Stephan Ango and Andy Polaine

Sun, 12 Mar 2023 00:27:40 +0100

What is the future of Obsidian? The new Obsidian CEO, Stephan Ango (@kepano), talks about why he joined the Obsidian team and what he wants to do next. I’m Nicole van der Hoeven, and today I’m joined by my friend, Andy Polaine, to find out what’s on the horizon for our favourite note-taking app.

Key takeaways

10 notable points raised in the video were:

Stephan thinks the proliferation of tools for thought is a positive thing, in that every new tool that uses Markdown makes the standard more futureproof.
Obsidian’s three major pillars are extensibility, privacy, and longevity. Those are things Obsidian excels at.
The things Obsidian is bad at are the consequences of those three major pillars (such as fragmentation, difficulty in sharing, and being restricted to Markdown).
The Obsidian team consists of five full-time people, including founders Licat (Shida, dev) and Silver (Erica, product), Liam (dev of Calendar plugin), and Joethei (plugin reviews and dev ecosystem).
Upcoming features: better metadata format than YAML (think something like Dataview inline fields), better querying of tasks
Stephan wants to keep the Obsidian team small, and grow only when absolutely necessary.
It’s difficult to get exact usage figures for Obsidian, but going by downloads of the app bundle, there are about 1 million users.
The Obsidian team doesn’t want to bring on venture capital investors.
Better sharing in Obsidian is something they’re thinking about, but they need to prioritize use cases.
Artificial intelligence in Obsidian is likely to remain in the realm of community plugins for now.

Below is a transcript of the video.

Introductions: Nicole van der Hoeven, Stephan Ango, Andy Polaine

Nicole: Hello, everyone. I’m Nicole van der Hoeven, and today I’m joined by two special guests. And I’d like to introduce first, I guess, Stephan Ango, or you might know him better as Kepano.

Stephan: Hey, Nicole. Thank you for having me.

Nicole: You’re welcome.

Stephan: Kepano’s great. You can call me Kepano. I’m fine with that.

Nicole: So, Stephan is a designer and an engineer, and most recently, he was announced to be the CEO of Obsidian. So that’s going to be interesting. I actually know you more for the stuff that you’ve made for Obsidian, like Minimal, and also Hider and Web Clipper. But it looks like you’ve been in the industry for a while making, I’d say similar tools.

Stephan: Yeah. Well, I mean, up until now it wasn’t really in the consumer space. I was building startups in more in the B2B area, but I’ve been using a lot of these tools for a long time. So, as a user, I’m older than I look, I guess. I’ve been like, building, I think the first thing that I really got into when I was a teenager was Winamp Skins. And so like, this is like, ’90s, early 2000s, making my first website and things around that time. So that’s how, that was my first, I guess, experience, like, building things in the software world.

Nicole: Awesome. And also here to help me interview you basically, is Andy Polaine, who is a lot of things. Designer, educator, writer, design leadership coach. He’s also co-authored a book called, “Service Design: From Insight to Implementation.” And he has a podcast called, “Power of Ten.” But really, the most important thing is he’s my Dungeon Master. We play D&D every week, and he’s also a good friend. We actually met because of Obsidian. So, welcome to the channel.

Andy: Thanks. Thanks for having me on. I’m excited to be a co-host. Or co-interviewer. Co-interviewer.

Nicole: Yeah, it’s really weird for me to not take cues from you and roll dice when you tell me to, but, you know, we’ll see how that goes.

Andy: I’ve got the dice here just in case we need-

Nicole: Okay, well I’ve got mine too, so. Anytime you need me to roll a save, I’m ready.

Andy: Well, let’s just roll for initiative to see who interviews first, huh?

Nicole: Yeah. Okay, let’s do that. Rollies, what are you going to roll? Okay. Just a d20. I rolled a two, so.

Andy: I got an 11.

Nicole: You go first.

PKM history and getting started with Obsidian

Andy: Okay, all right. I will, so, really, the first question for you is, it’s kind of under a sort of batch of questions really, about why Obsidian, but how did you get started with Obsidian?

Stephan: First, can you give me any like, buffs or stats or anything as my character would?

Andy: Well, I think you get sort of plus one in design and coding, right? So, yeah.

Nicole: What class do you think Stephan would be, Andy?

Andy: Wizard, I reckon. Or an arcanist.

Nicole: Yeah, I was thinking like-

Andy: No, I reckon-

Nicole: Sorcerer?

Andy: Yeah, or a, you know, someone who kind of makes stuff as well.

Nicole: Artificer.

Andy: Artificer, yeah. That’s what I reckon he is.

Stephan: Back in the World of Warcraft days-

Andy: So high end.

Stephan: I played a lot of the paladin class. I like the hybrid class, but I think that I also in like, “Lord of the Rings” really love, you know, Gandalf and those types of guy, Radagast, you know. I’m definitely into those types of characters.

Andy: Someone who can produce something out of nothing. So. So. As you were.

Stephan: I forgot what your question was.

Andy: Yes, so we’re going to lose half the audience straightaway. So, you know, you were a customer, right? You were a user before you became CEO and pretty quickly I seem to remember seeing your, certainly your theme, I think, probably a couple of other things, seeing you in the Obsidian Discord. So, you know, what was your hook into Obsidian in the first place?

Stephan: We’re getting some comments that your mic is very crackly.

Andy: We are hearing it. This is weird, isn’t it?

Stephan: Yeah. Yeah, yeah.

Andy: I had it before.

Stephan: Some people are hearing it.

Andy: I don’t know what’s going on.

Stephan: But to answer your question, I had been using the, over the past like, 15 or so years, I’ve used a lot of different tools to kind of try to solve what for me was mostly a journaling question. So, I wanted to be able to just write kind of, my own diary notes in a tool that, over time I discovered that the Wiki type of format was what I really loved. I kind of fell in love with the wiki link kind of idea. And that idea’s been around, I’m not sure exactly who invented the notation. I think it was the Wikipedia people in the MediaWiki, the kind of like, double-bracket concept. But once I discovered that, I felt that I only wanted to use like, journaling with that. So the one that I used for a long time was TiddlyWiki, and TiddlyWiki has that same kind of bracket notation, but I found that my personal sensibilities as a user, I really wanted something that felt like, very simple and minimalistic. Like, as you may notice from the kind of like, type of themes and things that I’ve made for Obsidian. And when Obsidian came out, I found out about it almost right away, like, in, I guess April, 2020, you know, as right as the pandemic was kicking off. Like, I think a lot of people, I don’t know, around that time were staying at home, were exploring different things and I found out about it, I had spent probably a year trying to kind of customize TiddlyWiki to be a little bit more user friendly, and Obsidian, even in like, V1 or whatever, not V1, it was like, V0.001, was already better than what TiddlyWiki was doing for me in terms of, it had the quick switcher, it had the prompts, it had the file Nav in the way that I wanted. It didn’t quite look like what I wanted, but it had a lot of the kind of, nice user interface patterns that I was looking for. And so I pretty much started making, I took, I was looking back at my notes, kind of preparing for this conversation, and I pretty much went all in on Obsidian after using it just for a couple days. I imported everything, and started making the minimal theme like, immediately. But I was just making it for myself at first. I wasn’t intending to publish it, I just kind of made it to solve my own UI preferences.

Nicole: So I actually used TiddlyWiki quite a bit. The reason that I used it was a lot of, ‘cause I was a consultant at the time, and a lot of the tools that were out there required, like, they were basically on the cloud so it was a SaaS platform that I’d have to log into and not, and those sites weren’t always allowed by whatever company I happened to be working for, so that’s what I loved about TiddlyWiki, that it could just sit on like, a flash drive and it would work. And that’s part of the reason that I started using Obsidian too, is ‘cause it’s local.

Stephan: Yeah. Yeah, and that was something, you know, being able to use it on your computer when you’re offline, just having that control. I was, you know, using it on flights, and so that local first idea to me is, I don’t know, I just find it much more natural and it allows you to do things so much faster. And because I have experience, you know, I wouldn’t say that like, being an engineer is first and foremost what I do, but I do dabble in a lot of different things on the code side and using, I use Sublime Text as my main code editor, but having, being used to using like, a code editor, you just expect everything to be there for you, right away, super fast. Like, even a small delay feels like it gets in the way of my productivity in some cases. So, being in the cloud, it always feels a little slow. And I’ve used a lot of those tools in my previous, especially for work purposes like Confluence or Google Docs or those things. And the sluggishness I feel like, really gets in the way of being able to make progress on whatever you’re trying to do.

Andy: Yeah. I mean for me, it was the portability thing. I switched to Markdown for everything, I think, for my blog. Like really, really soon after Gruber kind of released it. And I used to use Notational Velocity, which I think it was probably one of the first ones I know of, of a kind of this idea of a database that just sits on top of a bunch of Markdown files. And then Brett Terpstra was putting together NvALT, and nvUltra, which he’s still working on. And you know, I’ve been pretty disloyal. I’ve used lots and lots of like, every and probably most people, every Markdown out there. And a lot of them are starting to do the kind of wiki thing too, but depending-

Stephan: And there’s so many of them and-

Andy: Yeah. Yeah.

Stephan: And I think that is such a great thing that now that the ecosystem of Markdown editing tools is so deep and broad, that I often think that Obsidian, to me what’s exciting about this whole kind of area is like, I’m very curious about how we can make digital information persist over a long period of time. Like, how will the ideas that we have right now continue to persist for hundreds of years in the future? We’ve had books for thousands of years, but we’ve only had digital files for, you know, 70 or 80 years. And I like the idea that something like, plain text could still be around for the next few hundred years. And so using something like Markdown is, like, the file in a way is way more important than the apps. Like, Obsidian will eventually become obsolete because technology changes so much. But hopefully, the plain text will continue to live forever.

Andy: Yeah. I mean a bit like you, I’m gonna ask you about Computer Show actually later, but a bit like you, I’ve been around for quite a long time and I thought, I’ve been in digital stuff for like, 30-something years and I got fed up, but one of the reasons I moved to doing everything, I do all that, still do my invoicing, you know, with Markdown and stuff because I had so many times where like, “Oh the app’s gone out of, you know, out of production,” or “I’ve got so much of my work still on like, CD-ROMs.” I keep an old laptop that’s still got a CD-ROM drive, just in order to be able to view that.

Stephan: Oh, really? You’ve got to transfer those.

Andy: It still runs, kinda like, oh, it’s a, no, but yeah, but some of it’s like, Director MacroMind, actually, Macromedia Director stuff, that that’s the only thing you can play on, so you sort of have to keep the machine for it, too. And I transferred a lot of other stuff, but I had the same thing and I started using Evernote once and then trying getting stuff out. It was horrendous. Had the same thing with Notions, of exporting out of Notion, and it kind of dumps all this other crap in there. And so, you know, yeah, I agree. I think the file format is the kind of winner.

Stephan: Yeah, the file is more important than that.

Dealing with competitors

Andy: I’m interested though more as CEO, that you’ve got, you know, there are a whole lot of, well, I kind of would perhaps a little bit rudely say Obsidian clones kind of popping up. It feels like, in the last, you know, even in the last year there’s been a lot of stuff coming up. So, you know, whether you perceive those, is that a challenge for Obsidian? Is Obsidian kind of, just take its own path that each of them have their kind of own flavors? Now is it a good thing, you see it as a good thing?

Stephan: Well, I see it as a good thing because I am still kind of like, a plain text maximalist. So I think that you have to have a diversity of clients in order for that to kind of last forever. Like, I still think the file is more important than the app. I do think that where it gets dangerous is if there’s too many app-specific syntaxes that start to emerge. I think that is kind of dangerous because it reduces interoperability. Like, I think that, you know, Markdown, for all its flaws, and I think a lot of people have frustrations with Markdown, it has kind of stood the test of time, at least for the past, however many, 12, 15 years it’s been around, I’m not exactly sure, but maybe more at this point, I’m getting old. And so, I think that that is one of the biggest challenges that we have with Obsidian is sometimes we wanna do something new. Like, recently we launched the Canvas feature and there isn’t a, you know, off-the-shelf kind of canvas type of bio format. So we, you know, created a new one that is, you know, open and the type specification for it is out there, but there’s no other client for that file format yet. Hopefully there will be, but I think that that diversity of clients is good because there’s a lot of people who have lots of different ideas about how thinking should work, and Obsidian is not particularly dogmatic, like, a big part of, you know, one of the main values that we always think about is extensibility. So how do we make it easy for individual users to customize Obsidian? And also, how do we enable developers to build on top of it? But there’s a, I think it’s very viable to create a very opinionated Markdown or tool, you know, editing tool that takes a different approach than Obsidian. And I’ve seen many people who enjoy using more than one, different one for different parts of their workflow. So yeah, I think it’s good. I think we’re living through a little bit of like, this Cambrian explosion moment because for some reason right now, in the past couple of years, like, people have realized this a really exciting problem to work on and I think we all learn a lot from each other, so, I see it as a good thing.

Nicole: So one of the things that you said, ‘cause I trolled through your blog while I was thinking of questions for this. You had a post about how great tools choose to be bad at some things. So, what do you think Obsidian has chosen to be bad at?

Stephan: Well, I think that design principles are only real design principles if you can take both sides of it and create something compelling. Like, for example, we talked about extensibility. That’s one of our, I would say there’s like, three major pillars to what Obsidian is trying to be. One of them is around privacy, one of them is around longevity of your notes, and one of them is around extensibility. So those three things are the things that we want to continue being really good at. But you can take the extensibility example as kind of one of our design principles. We wanna make Obsidian super extensible. But you could, I can see why some people would prefer an app that’s not at all extensible, that is, you know, completely designed by one small group of people who are trying to make a very cohesive experience. I think one of the big challenges with Obsidian is we’re always asking ourselves, “What should be a community plugin versus a core plugin?” We’re always asking ourselves, “How can we make community plugins feel more native to the app and more cohesive and intuitive to people? Do they make sense with the rest of the app? Because any developer can basically build anything inside of Obsidian. And so it, I think it is a common point of frustration for some new Obsidian users that they have to download a lot of plugins and some of them feel a little janky. So, you know, Obsidian has chosen to be bad at that, with the trade off that it allows a lot of freedom for people to customize the app. And you can take the same example with, you know, privacy and longevity. Those have their own trade offs as well. Andy, did you wanna say something?

Andy: I was thinking, you know, it reminded me of what you just said. It was just as you talked about the plugin thing. One of the tricks that Apple did with the iPhone, was to make the phone bit an app. And it sounds just like, a really banal thing, but up until then there were phones, smartphones, were kind of phones with sort of app bits in them, and they were really janky. And I think one of the things also, obviously not having a keyboard, having a touch screen, means that that thing where you get whatever app you’re in is completely the app. All right? So, you know, I’m in FaceTime, it’s completely FaceTime, I’m in Obsidian and it’s completely Obsidian, the whole phone turns into whatever app you’re in. And I feel that, you know, in a way that’s what the plugin mechanism can do for Obsidian. At its base it’s, you know, it’s a text editor that’s browsing as a viewer on a set of Markdown files. But you know, as soon as you start putting, particularly some of the more powerful plugins in it, it becomes something completely else. And I think, I find it really kind of powerful that you can switch that stuff on and off. I can see that the, you know, I was sharing my students the other day ‘cause they were talking about design research and how you can use Obsidian, and quite a lot of them use Notion. And I can completely see why they use Notion, right? ‘Cause it’s just, you know, zero learning curve for them. Until such time as they want to take that stuff with them, and then they’ll find out. And I think some of that stuff’s a bit like doing backups. You find out the hard way, right? But I kind of feel like that is a, it’s been a smart trade off to make with Obsidian, with the plugins. And I was just wondering at this point-

Stephan: But it has its downsides. I mean, I think the point, you know, on privacy, the way we think about that is that is having ownership over your files. They’re not in somebody else’s database. They’re on your computer, on your device. You know, if you choose to use the first-party sync, everything’s end-to-end encrypted, and so we have this kind of strong pillar around privacy, but it has the trade off that it makes certain things harder for us. For example, people ask us, would we ever see a web client for Obsidian? A like, online, you know, cloud-based Obsidian? And that’s a lot harder for us to do because we don’t have access to your files. We would have to compromise on that value of encrypting your data and kind of being able to see that information on our side in order to be able to present it back to you. And that makes things really tough. Unless, there’s some, we’ve thought about this question a lot, but it really is a very hard problem to solve. But for someone who, you know, for whatever scenario they might be in, either can’t access a local app because they, you know, there were computer restricts, you know, apps and they can’t download Obsidian or they’re on a school computer and they have to access it from the school computer, it does present a limitation. But I think strong principles force those kinds of trade-offs where you could take either side and make a compelling and legitimate app. We just, you know, have chosen our own particular set of trade off and that’s what that blog post was about. So. So there’s other things that Obsidian is bad at as well, like, relating to longevity and like, the choice of using Markdown, for example. It’s not the most intuitive to some people. I can come up with a long list of things that Obsidian is bad at, in favor of something that it wants to be really good at. And I think that’s a, making trade offs is our job.

Nicole: Well, one of the things that I hear a lot is that Obsidian is too developer focused, or some people think it’s not approachable. And I think it comes back to what we’ve been talking about. You do have to make a decision, you do sacrifice a little bit of usability in exchange for flexibility and modularity. My background is in performance engineering, so I care a lot about performance and I love this idea of making the most streamlined possible app, and then just leaving room for extensions so that other people can make their own decisions. I would rather have that, than have an app that has everything baked in and I have really no choice as to what to turn on or off because my use case is going to be different from everybody else’s.

Andy: That’s Microsoft Word, right?

Stephan: Yeah.

Andy: When you just have the sort of, bloat and they get those screenshots, sometimes, of every palette open and there’s about this much space, they actually edit some text.

Nicole: By the way, do you like my shirt?

Stephan: Oh yes. Classic.

Nicole: Where’s yours?

Andy: Yeah.

Nicole: How disappointing.

Stephan: I know, I haven’t, I’m such a minimalist-

Andy: It’s a print.

Stephan: I haven’t gotten my Obsidian merch yet, but, yeah, you’re looking good over there.

Andy: So, you know, you’ve been contributing to Obsidian, pretty much from the get go, and then you started working with them. What is it with them? It’s the, the Licat and Silver, basically-

Stephan: It’s the smart, yeah, the smart team.

Joining the Obsidian team

Andy: Two-person team. So, you know, how did that, I don’t know how much you can reveal, how did that come about and then, and what was the sort of transition? When was there this kind of moment of like, “Hey, how about becoming CEO?”

Stephan: Yeah, well, it was a very organic process because I, you know, was running a different company at the time when Obsidian first came out. And so, I had, you know, started building minimal theme, a bunch of different small plugins here and there, the Web Clipper. And I was a very vocal in the Discord and forum about some of the things that I thought would be, you know, improvements or bug fixes. And at the time, the Obsidian community was way smaller than it is today. And so, I was often interacting with Licat and Silver, the founders of Obsidian, through the Discord. And just kind of, just talking very organically about here are some things that could be improved, here are some, you know, bugs I found. Can you help me with this feature of Minimal that I am trying to, you know, build into my plugin? Can you give me some feedback on how to do this properly? And so, we started interacting, you know, talking about Obsidian for a couple of years. And, I’m not exactly sure at what point it started, and maybe it was around the time that, I’m trying to like, actually remember, but I think there was like, a Gems of the Year contest where Minimal won like, the Best Theme category, and I think I started DMing more with Silver and Licat around that time, and just kind of getting to know them just over chat in Discord. And they found out that I had been, you know, running some startups and asked me for some advice on a couple small things here and there. And so we just kind of kept that conversation going, and at some point last year, there was this kind of idea that Obsidian wanted to kinda reach the 1.0 release. And the 1.0 release, I mean I think it, you know, it’s funny that it, it was really a milestone moment, because we, that’s when I started becoming involved in the project and they asked me to basically create the new default theme. And so I was working with them part-time, kind of taking a lot of my experience on Minimal and bringing that into the core app. But also, they were asking me about some of the things that I felt were on my like, top priority list of small details that we felt were still kind of embarrassing or like, wanted to really get nailed down before we could really feel like we could call it a 1.0. And, that’s when I started working much more closely with them, as well as Liam, who is now a full-time part of the team. You might know Liam from the calendar plugin. He’s amazing. We have basically five people full-time. Joethei, you might have seen around the community who helps a lot with plugin reviews and the developer ecosystem, and then a few other folks who’ve been helping us part-time, like MG Meyers, who you might know from the Kanban plugin and style settings is working on, has been working on Canvas and kind of like, led the Canvas effort. And so, the more conversations kept going on with this project, I just realized it was so much fun and I love Obsidian so much. Like I just, I really want it to continue to be an important tool in my life, in a very selfish way, but also I think that Shida and Erica, Licat and Silver, are such mature like, entrepreneurs, they’re quite a bit younger than me. They make me feel old all the time. But it’s a great exchange of knowledge and yeah, so we started discussing this idea and they were on board. I think it’s a very, we have very complimentary skills. I’m really excited about, like, Licat’s one of the most incredible engineers that I’ve ever worked with. I’ve worked with a lot of like, amazing engineers throughout my life and I have like, so much respect for him. He’s just super, super impressive and continues to be like, they’re both gonna continue to be involved for the foreseeable future. And Erica has just an incredible mind for, you know, product and how to think about what users really care about. And so, and has been, I think, the person who’s, in a way, influenced why Obsidian has such an amazing community more than anybody else. Just kind of from the beginning, making that an important part of Obsidian, I think is a lot of credit to Erica. So, seeing, like, getting to know them just made me realize like, there’s nothing that I would rather do than help that community continue to grow, help make sure that Obsidian has the, kind of like, structure that it can last for a long time, kind of going back to that principle of longevity that matters at the app level. How do we actually make that viable at the company level so that the kind of, product and the company can be aligned? So yeah, that’s why I kind of believe that-

Nicole: I also just wanted to call out-

Stephan: Very organic.

Nicole There’s some people who are asking-

Stephan: Oh, where do we get the merchandise?

Nicole: Where do we get the merch? By the way, I bought this. And it is here.

Stephan: Yeah.

Nicole: Just so everybody can see, ‘cause there’s some awesome things on there.

Stephan: Yeah, I think it’s-

Nicole: I want I wanted to ask you-

Stephan: Yeah.

Building intentional tools for tool builders

Nicole: Yeah, it is. So I wanted to also ask you, earlier when I was introducing you, I said that you have worked in similar spaces, which I realize now is a bit odd because the companies that you had worked on before aren’t note-taking apps. But what I meant by that, is actually I’m seeing a theme in what you’ve worked on and it’s all about creating things that are minimal, intentional, well-made. And I think like, in your previous company, Lumi, you were talking about packaging, and making that a little bit more transparent and intentional as well. I know you’re also an industrial designer, or you claim not to be one anymore, but that’s also something that permeates your work there as well, doesn’t it?

Stephan: Yeah, I’m a generalist. I like to dabble in a lot of different things. And so, it has taken me in a lot of different directions, but I think the common theme is I’ve always been interested in making tools. So everything basically in my professional career has been a tool of some kind. So in the case of Lumi, it was a tool, it was actually two different tools. One was for factories, to help manufacturers kind of bring their capabilities online, and a tool that helped businesses who need to collaborate with manufacturers, find the right manufacturers and work with them. And so, I like building tools that kind of empower individuals to do something that they couldn’t do before. And so Obsidian definitely fits into that because I feel like, I mean, in my own personal life, it’s been so rewarding and exciting to use Obsidian. Like, I feel like, my thought process is clear and more reliable when I’m using Obsidian. And I feel like, that’s such an exciting superpower that I have that I want to help more people get that feeling, too. So yeah, like, throughout my life basically like, that theme just keeps coming back of like, making tools that help other people do whatever they’re trying to do.

Nicole: And it’s also helping other people make other things, right? ‘Cause like, you know, you’ve also worked on things like, that kind of pare down what is out there, because there’s an information overload that’s out there and I know you’ve made like, Tidy, which kind of streamlines what we even see on the page. What do you think, is there anything in Obsidian, ‘cause we’ve talked about things that it’s bad at. Are there any big features that you’d like to introduce to it, that don’t detract from, that don’t make it like this bloated bit of software but still are along the same lines of being intentional and helping people?

Stephan: Definitely. I think that the area we want to keep working on is the primitives. Like, what are the basic things that you can use that you can build on top of? So, we have a public roadmap that’s not always the most up to date, but one of the big things we’re working on right now is around metadata. So, a lot of, I’m a huge fan in my personal use, of DataView is one of my favorite plugins for Obsidian, and I use it for everything. And kind of asking ourselves like, “What can we do that would enable the user and developer ecosystem of Obsidian to leverage metadata even more?” So, you know, you can, there’s a couple different ways that you can add metadata to your notes with Obsidian. One of them is through YAML in the frontmatter, and one of them is through DataView has these like, custom fields. So that’s an area that, we want to, like, the author of DataView had to create this new notation with the double colon. You know, you put a key in a value and separate them with a double colon, and that notation was essentially invented by the author of DataView because it’s so painful to write YAML. And so, what can we do to clean that up a little bit and make metadata feel more like a first-class citizen from the user’s perspective, but also enable more developers to build on top of that. Yeah, API docs and developer portal. I’m looking, I can’t even remember what we have on here. Tasks. There’s a few other things that we have up our sleeve at the moment, that I’ll be excited to share in a little bit. Talking about the roadmap-

Nicole: A few questions, Stephan-

Stephan: Yeah, go ahead.

Nicole: You mentioned DataView. I know that he’s working on DataCore, right? Is that going to be something that’s going to be built into Obsidian or is it still going to be a community plugin? Honestly, I would never use Obsidian without DataView.

Stephan: You know, I haven’t really, I’m not exactly sure what the goals of DataCore are. I’m not super informed about what he’s trying to do with that. I think that our job is always to find the right balance between what feels like a core plugin, what feels like it should be part of the community. And so, I think that long term we do wanna have good ways for people to add metadata to their notes and perhaps view kind of an output of that in some way. I think that’s kind of a long-term project, but in the short term, the metadata improvements is really about making it a lot easier to add metadata to your notes and then hopefully that makes things like DataView easier to use. But maybe long term will consider some more ways that people can like, look and view their data, or the metadata from their notes. I do found that DataView, one challenge with it is it is pretty nerdy. Like, it does require, it has a syntax that’s a little bit, you know, like, SQL and I think that that’s not the most user friendly. So we’ve seen quite a few plugins pop up, like DB Folder is one that’s kind of interesting that makes it a little bit more WYSIWYG. And that’s one of the big benefits that we have as a community is that people can build plugins that optimize for different types of users. People who love writing SQL, they’re gonna love using DataView, but people who prefer something more WYSIWYG are gonna prefer a different type of UI. And we can be patient in our approach to what we actually wanna build into the core because we could just kind of watch what people like, what people make, and decide later down the road what we wanna build ourselves.

Growth of Obsidian as a team and company

Andy: Do you think there are, I mean are there any plans to then grow the team? Or is partly what makes it special, the fact that you are a very small team?

Stephan: I think that for now we wanna stay really small. We wanna be as small as we can be because, you know, one of the other things that’s really important that made me want to join Obsidian, is that from an early stage, Shida and Erica decided that they didn’t wanna bring on investors, like, venture capital investors. And I have, throughout my career, worked and built companies that have used every single funding model under the sun. Like, my first company was one of the very first Kickstarter campaigns ever, back in 2009. I’ve gone down the venture capital route, I’ve done bootstrapping, like, I’ve tried it all. And I think when it comes to the goals of Obsidian that we’ve been talking about, like, longevity in particular comes to mind, but also privacy and extensibility, I find that is not very compatible with the VC route. And, you know, if you just kind of survey the ecosystem, you’ll find that actually, almost all of the kind of big PKM apps, if you wanna call them that, have raised a lot of money, like, millions of dollars from, or hundreds of millions of dollars, even. And the problem with that is that, I think it’s misaligning the incentives in the long term for, you know, the user and how they want to kind of, store their notes. Like, basically it incentivizes certain behaviors that are kind of against our principles. And so for that reason, we’re trying to build very intentionally and also work through the extensibility part of our like, principle to enable the community to customize things more. Like, we don’t feel like it’s necessarily our job to make this like, huge app that has every single feature. We like seeing how the community kind of extends Obsidian in their own ways. So I would say that growing the team, it will be more by necessity than by a desire to like, try to do everything ourselves. We’re getting to a point now where, I would guess, we don’t actually even know how many Obsidian users there are because we’re so focused on privacy, we don’t have analytics, we just like, don’t know. But we can guess based on a few different like, things like, how many people downloaded the app bundle, that there’s probably close to a million users of Obsidian and only basically five core members of the team. And eventually, from a support standpoint are things, things might start to break and we might need to hire people to help on that front. But we’re trying to stay small if we can.

Andy: Yeah. I think you’ve been very diplomatic about the VC thing. I mean, in general it’s, you know, you’ve got, in my coaching practice, and I coach a lot of design leaders who are kind of suffering in startups or scale ups because of this constant pressure to scale, because this constant pressure for just more and you know, I want a 10 X on my, you know, investors want 10 X return on their investment in some stupid amount of time, things like that. And it definitely leads to that, you know, what Cory, Dr. Al calls sort of, an end certification, you know, this thing where then there’s a choke point and we have everyone’s data on our server and they have to use app and then you just start, it becomes a sort of process of trying to extract value. And I-

Stephan: Well, I’ve touched on the thing-

Andy: Oh, go on, go on.

Stephan: To defend VC, I will say there’s some businesses where, it kind of is necessary because especially, we’re in a privileged position, building software, because software has no cost to us. But there’s some companies, you know, that have to produce a physical product in the world-

Andy: Yeah, that’s true.

Stephan:: And they have to buy the materials and put in orders and so they need to raise money somehow, and VC is one of those paths. You can get loans, like, there’s different ways. So I think it, I’m not like, I just have a kind of, like, a more zen point of view on that where like, I understand why a lot of companies go that direction and it makes sense. I think in the case of what we’re trying to build specifically, it’s a bad fit. But it’s not necessarily a bad fit for everything.

Nicole: So-

Andy: You mentioned anything just now which is, oh sorry, Nicole. What were you gonna say?

Nicole: I was gonna say I’d like to go back to, and just quickly, to what Obsidian is bad at. And you know, Stephan, I love Obsidian. It is, I’m all for it. I am possibly the biggest fan. I think one thing that Obsidian is bad at that I really would love to see improved, is sharing. Because I think that is the thing that stops most people from using it, because, you know, using it on your own in an isolated way is one thing. It excels at that. But anytime that you have other people that you wanna do that with, as an example, Andy and I play D&D. Okay, this not a work thing, right? But it is a recreational use, and honestly, it takes up a lot of, my usage of Obsidian is role playing games. And in that group, there are four of us, three and a half, who use Obsidian. We can’t share our notes. We can’t have like, one repository for all of us to collaborate on. So we have like, individual notes, and yes, we could use something like Dropbox to share those notes, but then Andy for example, can’t, or wouldn’t want to share all of his notes with us. Maybe some of them he would. So I would love to see like, a little bit more permissions around sharing.

Stephan: Yeah, I think sharing is a big, like, it’s a challenge because there’s like, three or four different use cases nested under sharing that are quite different from one another. So, we have published, so we’re doing a lot of work right now, actually, on making some publish improvements. There’s a big batch of publish improvements that just went live like, a week or two ago, and that’s more about sharing to the external world, an entire vault, and we wanna keep improving that. Then there is sync, which is like, has this concept called Shared Vaults. I don’t know if you’ve tried that out, but Sync Shared Vaults is actually what we use internally on the Obsidian team. So everyone who’s like, you know, between the five of us, are working on projects, we keep our notes in one shared vault. Then there’s also the idea of, “Hey, I’m writing this note, I just wanna send a link to someone so that they can view it.” They don’t have Obsidian or they don’t know, you know, I don’t wanna like, put them through the process of having to, you know, learn everything about Obsidian just to read this one note. And that note might be, you know, semi-private, like, it’s like, something where it’s not like, a published use case where you want the world to see it, you just want this one person to see it. And then there’s a collaboration use case where like, Google Docs or Notion makes it really easy for a few people to be like, collaboratively editing something, even if they don’t, or have Obsidian, for example. So like, those are all kind of different problems to solve in different ways, and I do think that we want to find ways to solve those problems, figuring out what the right order, that’s another challenge about being a small team is like, what do we work on now versus later? But I think all of them are problems that we would like to solve, you know, over the kind of medium to long term for sure. We’re very aware of that.

Andy: It feels like there’s a bit of a thread here, though, as well, when you’ve talked about, you know, ‘cause obviously the sharing bit brings up the privacy issue that you talked about before.

Stephan: Correct. Yeah.

Andy: And it also links to the, is there a browser-based version of Obsidian? You know, because one of the ways of obviously, easily sharing and collaborating on a document would be to have kind of a browser-based editor, even if it was just for that one document. And it’s almost like, you need a kind of, instead of a separate vault, a sort of enclave in your existing vault that you can say, “Oh this the shareable bit and all the rest of it, you know, remains encrypted,” let’s say, I don’t know if the encryption thing is an issue, but you know, at some point once you share it, you have to not do that, right.

Nicole: So I think-

Andy: And that may be at the beginning of the browser-based version.

Nicole: I think my problem is that I am able to do those use cases, but I use separate tools for each of them. I would really love to streamline in this area. Like you were talking about, being able to share a link. I use Etherpad with that, but it’s not the best. It works, though. And then when you talk about using Sync, I also subscribe to Sync, but that also requires other people to have Sync. So for those people who don’t use Obsidian enough to warrant Sync, I use Dropbox. And then at work, because we have all of our internal docs on GitHub repos, I use GitHub to bring my Obsidian notes and share them. And that brings a lot of things but also brings like, comments and being able to roll back changes, and, you know, handle merge conflicts, and that’s not something that Sync does as well for now.

Stephan: I totally agree. Yeah, I think that those are things that we have to eventually solve. And I do, going back to what feels like it should be a core thing and what feels like it should be a community thing, I think that sharing is something that belongs in core, or at least core needs to have good sharing mechanisms for all of the different use cases. Publishing, collaboration, single note sharing, is just a matter of, how do we do that? In what order do we build those features? And can we build them in a way that is consistent with our principles?

Task management in Obsidian

Nicole: Can we talk about task management? ‘Cause that was something that was on the roadmap. So is there anything you can share with us on that front?

Stephan: Our goal around that one is fairly conservative, I would say. It’s not gonna be a major, you know, complete revamp to everything. It’s more about, how do we make tasks queriable in a better way, like, and also adding better date supports? So we’ve been thinking a lot about how do due dates work with tasks? So right now, there’s ways that you can do that with community plugins, but we wanna figure out, should there be a core way of dealing with due dates? Again, thinking about the API first and how that might, you know, enable developers to come up with some new experiences on top of that. But I would say that that project’s been a little bit on the back burner for us because, we had some other ideas that we wanted to prioritize first, like Canvas, and some of the things that we’re working on right now, like metadata and publish improvements, for example.

Nicole: It does occur to me that a lot of these things, like task management, could be helped if you share a log something like DataCorp or DataView. ‘Cause the functionality is there, you just need to make it look pretty, but you’re good at that.

Stephan: Yeah, I mean the challenge there is just again, like, you know, two, three people building things. We only have so much bandwidth and kind of finding that happy medium between, we wanna make sure that we’re, you know, releasing features at a good pace. But every time we add new capabilities, we also add, you know, add to the surface area of potential bugs of potential issues that might come up or edge cases with certain plugins, we increase the number of feature requests, like, when we add something like Canvas, then we get a whole bunch of new feature requests from people who are like, “Oh, Canvas is great, but it doesn’t do X, Y or Z.” So that, I don’t know that anyone has ever solved that problem. It’s just kind of a problem of building software in the world. But, that is probably the biggest challenge that we deal with as a team is like, how to find the right balance of what to build now, how comprehensive do we want this feature to be versus, you know, starting small and kind of building on top of it. But trail locking a plugin, we tend to, we’ve done it a couple times, and when we do, we tend to rewrite it from scratch pretty much. And so that’s a pretty big effort. Like, rewriting something like DataView or Data, I haven’t looked at the code for DataCore. It’s not a small task. It tends to be a pretty big thing.

Artificial intelligence in Obsidian

Nicole: Another question here by Imaginationscene and another one by another user called Remy, “What about AI? Or integrating something like ChatGPT?” Is that something that’s on your radar?

Stephan: I’ve been thinking a lot about this. I think that right now, so this kind of like, goes back to the benefit of the extensibility of Obsidian. I’m pretty sure that Obsidian was the first app that had ChatGPT or like GPT-3 integrated into it, because someone in the community built a plugin for that called Text Generator. And now I think there’s three or four plug-ins out there that connect to the Open AI API. So, the cool thing for Obsidian is that because it’s so easy to build plug-ins, people tend to go really like, we both have the ability to be very fast and very slow, so, the community can go and build a plugin that solves that use case. And I think that they’re quite good, actually. I’ve used Text Generator a bunch, but we can also be patient about what’s the right way of building it for Obsidian. And I think that there’s a lot of companies right now rushing into adding GPT to everything. My personal preference when it comes to this, is I would like for it to all be local first, completely private. I don’t wanna send, you know, data that I don’t, like, I wanna have complete control over any data that is being sent. You know, there’s this concept of training and inference in, we’re getting like, right into the weeds of how AI works, but I would like, someday, to be able to have maybe a plugin, whether it’s a core plugin or a community plugin, that can do some training on your vault and kind of learn from your vault and be able to do the inference locally on your computer, so that it completely aligns with the principles of privacy and local first that Obsidian is known for. The challenge with that right now is that that’s quite intensive on the graphics card, and it’s quite complicated to set up locally for users, and we’re still kind of waiting to see whether a company like Stability, that they made stable diffusion and is an open-source platform for like, image generation, there’s been talks about them releasing a language model that would be open source. So we’re kind of waiting for all of those things to come together before we make a decision. We would, I think we would prefer to build something that kind of, is aligned with our principles, but in the meantime, there’s a bunch of plug-ins that you can use that solve that use case and I think they do quite a good job with them, actually.

Nicole: Stephan, have you heard of Napkin?

Stephan: No, I don’t know Napkin. Tell me more.

Nicole: It’s, yeah, it’s an awesome service that I recently have been using. And it uses AI in a really interesting way, because I think there’s a lot of focus on using AI for like, generating blocks of text or outlines or images. But what Napkin does is it looks at your vault, well, it’s not a vault because it doesn’t do it for Obsidian, it looks at your Readwise highlights, and it uses AI to tag them intelligently. So, it has this interface where you can determine, you can click on like, a thought or a highlight, and it shows you intelligently linked other highlights that you already have, that you may not have put together. And sometimes they’re like, they’re not even using the same keyword or they’re not tagged the same way, but somehow it still figures it out. That kind of thing would be invaluable for Obsidian because there’s a lot to be said for manually linking things, but getting some help would be nice.

Stephan: I think that sounds awesome. I’m seeing a lot of comments here in the chat of like, people who are suggesting some different ideas and some plugins that I’m not even aware of that are already doing some more local AI stuff. I think that’s a, I think what’s amazing about the Obsidian community is we have people kind of, at every level of the spectrum of nerdiness. Like, we have like, people who are some of the most advanced engineers that I’ve ever talked to are developing their own custom plugins, and then we have people who are complete beginners and novices, not engineers, people who are just learning about note taking. We have a lot of young students in the community. And so, I like that we have the ability to have those people who are like, all the way at the deepest end of like, understanding all of those principles, able to start working on, you know, potential plugins and extensions that could do something like that. And that can be kind of a breeding ground where we can learn, what do people actually want out of that type of tool? But I do think that if we want to continue to stay true to these kind of principles of being local and private, there’s gonna be some constraints that we have to deal with that are actual like, performance constraints and technical constraints, that may mean Obsidian will be a little slower to get to those features than apps that are completely cloud-based where they can upload that to a server and do that, you know, somewhere else.

Andy: Hm. Yeah.

Stephan: So I think it is a very tricky challenge for us to try to solve that right away. I think we’re gonna tend to watch what plugin developers do, and then start to think about that in a year or two. ‘Cause right now there’s, I mean, literally every month there’s something new going on with AI and the open source packages that are going on. So, we’re maybe a little less cutting edge than the plugin ecosystem can be, if that makes sense, on the core side it.

Andy: It feels like you could get dependent on something very quickly there as well, and just kind get really kind of bogged down. And then, you know, a couple of people have said keep that as a plugin, it’s much better that way.

Accessibility in Obsidian

Andy: There was a question, it was from Daniel Gartmann further up yeah, I don’t use a screen reader, but I think, you know, something that would need to be much more sort of, core fundamentals as obviously accessibility. I don’t know, maybe Daniel can say how, I don’t know how accessible Obsidian is to use with screen read, but read is-

Stephan: We’ve done a lot of work in that area with, yeah, I would be curious to hear a little bit more detail from Daniel about what they’re looking for, but yeah, Obsidian is already compatible with screen readers. We have done a lot of work on that side, but if there’s things that are missing in that respect, I would say let us know in the community, on the forum or on Discord. We’re definitely, I agree that building that into core is a must. I think that’s super important. We’ve added a lot of new customization features over the past year, so I’m actually kind of curious to see if there are themes that get developed that are focused on different use cases. It’s much easier today to design themes than it was a year ago in my, one of my, that was one of my biggest contributions over the past year from a code standpoint, was rewriting the way that themes can be built. And so that means that you can very easily now create like, very high-contrast themes. Themes with different font sizes, different fonts that might help people who have, you know, who need help on the vision side. But then there’s other things like, ARIA labels and other kind of web technologies that we’re able to incorporate that make it easier with screen readers for sure.

Tags vs Links

Nicole: Okay, we have a question that is a little bit controversial. It’s from Germ Germain, whom I actually, by chance, met a few weeks ago in person. He says, “What is his opinion (Well, I’m gonna ask both of you) about tags versus links?”.

Stephan: In my personal use? Or, how do you wanna clarify that?

Nicole: Yeah, in your personal use. Which one do you use more?

Stephan: So I tend to use links more than tags, personally. So, I do have quite a few tags, but I tend to use links because I like to see everything in the, all the relationships in my backlinks, I guess, is how I would say it. And I like that links, for example, like, I’ll have a top-level categories for books that I’ve read. And so I just have a category called books. And I link to it. In my template, I still use tags. I don’t actually really know why. I just feel like I should have it there for future need.

Andy: Future proofing.

Stephan: Yeah, for future proofing. Like, if I ever create a note about a book, it has both a tag for book and a link to the category books. So that redundancy helps sometimes if I’m creating like a DataView list or something I have access to both of those different ways of filtering down. But I tend to prefer links. I think links have a lot, are more powerful and cross over more different things. I think tags, where I use them the most, is for kind of very narrow meta segmentation of a very specific thing, where there’s only like, a very narrow subset of notes that needs to be grouped in a way that a link wouldn’t make sense, or I don’t really wanna have a page for that concept. Am I making sense?

Andy: Mm hm. I’m still here.

Stephan: But also, it’s not-

Andy: I’m still a fan of the folders, you know.

Stephan: Oh, I hear you.

Nicole: No. Do you really?

Andy: Oh, half the people have gone, “Oh god!”

Stephan: Yeah. I’m very-

Nicole: What are you even doing here?

Stephan: In my personal organization-

Nicole: No, I’m joking, I’m joking.

Andy: I can explain.

Stephan: This is why I feel like Obsidian being not dogmatic about this is great, because people think in different ways. Like, I love that people can use folders or tags or links or all of them. Do whatever works for you. If it works for you, that’s great.

Andy: I think partly goes back to what you were saying right at the beginning, which is, you know, where the file format is the thing, right? And so I use several other things on that group of files that I have there. And whilst, you know, it’s great if you can search the tags or you can do whatever you want with the tags or links in Obsidian when you are trying to find a file, to kind of pull into something. I use Marked, Brett Terpstra’s Marked quite a lot for Output. It’s a massive pain in the ass. Well, I mean, I use search a lot anyway on the Finder, but, you know, I much prefer to have some organization that works across other things that aren’t gonna handle tags. So that’s why.

Stephan: Well, to be fair, in our team vault, we do use folders for our like, internal grouping. But I think that that use case is very different than my personal notes.

Andy: Yeah, yeah.

Nicole: We have an answer from Daniel, actually. He says he has issues both for Windows and iOS screen reader support.

Stephan: Okay, well let’s-

Nicole: “I love the potential that Obsidian has for me as a blind person who’s already using notes in a text-based format.”

Stephan: Jump into Discord and let’s talk about it.

Andy: I don’t know how accessible Discord is, by the way.

Stephan: That’s a great question.

Andy: Because we might be saying jump into something that’s really terrible for you to use on Discord. So, yeah, on the forum, I’m guessing the forum-

Nicole: The forum.

Andy: It runs on Discourse, doesn’t it, of the forum?

Stephan: Correct, yeah.

Andy: I would imagine it’s kinda more accessible.

Stephan: If you go to obsidian.md/community, you’ll see the links there and send us your feedback. Or if you wanna just email support@obsidian.md we’ll take a look. But the best is definitely through the community and we’ll definitely take a look, yeah.

Andy: Tell us who does that email go to? Of the three of you?

Stephan: Well, right now it’s like, a kind of a shared thing where you kind of look in there, but I think that most people who use the support inbox are for kind of like, our enterprise users. We prefer to use the community as the main place because there’s more transparency around that and like, other people can jump in and add their own kind of experiences with whatever the feature request or bug might be. So in general, that’s why we kind of direct people towards the community if they wanna provide some feature requests or bug fix requests.

Andy: I was kind of hoping it triggered like, a kind of a bat signal that projected us an Obsidian sort of into the sky, the Obsidian crystal into the sky, and you all kinda come running.

Stephan: Yeah, unfortunately, it’s just one of those other things that, as we scale up, we have to find good ways of parsing what the community is looking for and how we can fit it into our roadmap.

Message to the community

Nicole: Well, I don’t, I would love to have you stay, both of you, stay on for longer. You’re both welcome back, but I also wanna be mindful of your time. So thank you for coming on to talk about the future of Obsidian, Stephan, and for helping me interview him, Andy. Stephan, is there anything that you’d like to say to the community before we end?

Stephan: Wow. Thank you. I just, I’m so thankful. I think that the thing that I love about Obsidian is, so many smart people and kind and like, kind people are part of the community. I love, you know, watching your videos, Nicole. You do such a great job of kind of making Obsidian feel more accessible. And I think that, in a way, like, beyond what we’re working on in terms of features and so on, like, the fact that we have such a friendly and collaborative and helpful community, is one of my favorite things. And so, I’m extremely thankful for that. You know, continue to be, continue doing an amazing job, being nice and helpful. Like, I think we’re still at the very beginning of people adopting these tools. Like, anyone who’s watching this is probably in the like, top 0.01% of people who knows anything about these, whatever you wanna call ’em, tools for thought, how you like, some people have been calling it integrated thinking environment. Like, we’re creating a new generation of tools that help people unlock the way that they think. And anyone who’s watching this right now is kind of, pretty much on the cutting edge of that concept. And I’m really curious what happens if that idea becomes more pervasive in society. I’m very hopeful that it can help us solve a lot of big problems. And so, if you’re one of the, you know, nerds, I say that lovingly. Geeks, dorks, that are part of this conversation, I consider myself one of those, introducing people to Obsidian and being part of, you know, helping them learn how to kind of, make their thought process more reliable is just an amazing gift. Like, I just think that’s like, and you’re doing that every day with your videos, Nicole. And so, I wanna thank you for that. So that, it’s not a specific ask, but that’s just kind of the way that I feel and I hope we can continue that for a really long time.

Nicole: Awesome. Thank you. I’ll talk to you both in a second, but we’ll say goodbye to everybody else. Thank you for joining us on this weekend, and see you next week.

Andy: Thanks. Bye.

Nicole: Bye!

Stephan: Thank you.

Writing a book in Obsidian

Fri, 03 Mar 2023 16:51:30 +0100

I’m writing a book in Obsidian– about Obsidian! And I want to share it all with you. This is the beginning of that journey: what I’m starting with, my idea, and the plugins I’m using to get it done. This is all a work in progress, so if you have any suggestions about what I should do, I’d love to hear them!

You can follow along with my progress by signing up for my Patreon.

How to Do Exploratory Testing for Performance (k6 Office Hours #79)

Fri, 24 Feb 2023 23:01:26 +0000

Marie Cruz and Nicole van der Hoeven talk about what exploratory testing is, why you should do it, and how to do exploratory testing to improve application performance. They discuss some heuristics to add structure and intention to exploratory testing for both frontend and backend performance.

My new favorite Obsidian theme

Fri, 24 Feb 2023 16:49:38 +0100

I have a new favorite Obsidian theme, and it’s called AnuPpuccin! Anup is as pretty and feature-packed as it is hard to spell/say. Didn’t know themes could have features? Anup comes with custom callouts, checkboxes, speech bubbles, and lots of color options.

My setup in Obsidian for TTRPGs in 2023

Fri, 03 Feb 2023 16:41:58 +0100

I heavily use Obsidian for TTRPGs: playing them, running them, and learning new systems. Here’s an update of my entire TTRPG setup in Obsidian, including how I create new worlds, sessions, NPCs, places, maps, statblocks, a GM dashboard, and encounter tracking.

Integrating k6 and Tracetest (k6 Office Hours #77)

Fri, 03 Feb 2023 08:00:15 +0000

Tracetest enables trace-based testing with assertions at any point within a request transaction. Using the xk6-tracetest extension, your k6 tests can launch deep testing by unlocking the power of OpenTelemetry.

Join us as Developer Advocates Nicole and der Hoeven and Paul Balogh are joined by founder Ken Hamric and software engineer Oscar Reyes from Tracetest.

Check out Tracetest:

site
xk6-tracetest: repo for k6 extension
docs on how to use Tracetest with k6

Frontend vs Backend Performance Testing

Wed, 01 Feb 2023 22:59:18 +0000

Which should you do: frontend vs. backend performance testing? The short answer is both. Each has its pros and cons, but hybrid load testing combines them and gives you the most holistic view of performance, including frontend concerns like how quickly a page renders or perceived application performance as well as backend concerns like response times of individual components.

How to Organize Testing Scripts in k6 (k6 Office Hours #76)

Fri, 27 Jan 2023 17:00:10 +0000

In this video, we talk about how to organize testing scripts in k6 using features like tags, groups, and functions depending on what type of testing you want to do with your test script. Developer Advocates Paul Balogh, Marie Cruz, and Nicole van der Hoeven show us a few examples of how they would organize and modularize scripts in a test suite.

AI tools for PKM and Obsidian

Fri, 27 Jan 2023 16:39:00 +0100

Here are some AI tools for learning and taking notes that I’ve incorporated into my processes. AI is a controversial topic, but it’s important to remember it’s just like any other technology: it’s only as good as the people that wield it. In this video, I show some ways I use artificial intelligence to improve my learning and note-taking rather than to replace them.

Introduction to Performance Testing

Wed, 25 Jan 2023 22:28:43 +0000

In this introduction to performance testing, we talk about how performance is part of the functionality of an application, why you should do performance testing, and some myths about performance that discourage teams from testing for it and make applications more likely to fail and fail miserably.

Top 10 Core Plugins for Obsidian in 2023

Fri, 20 Jan 2023 16:37:32 +0100

In this video, I talk about my top 10 core plugins in Obsidian: the built-in options that improve your productivity without adding clutter and complexity to your notes. Unlike community plugins, core plugins are a lower security risk, yet provide a lot of extra value– but they’re also easy to overlook. Here are my favourite underappreciated core plugins.

Overview of Obsidian - Obsidian for Everyone course

Fri, 13 Jan 2023 16:32:30 +0100

Here’s an overview of Obsidian: what it is, why everyone’s using it, and how to get started with Obsidian for beginners with a brand new course called Obsidian for Everyone. I’m extremely proud of the course and I can’t wait to share it with you after working on it for what seems like forever. It contains everything you need to know to use Obsidian – but without installing any third-party community plugins.

Guide to Load Testing Websites

Thu, 12 Jan 2023 17:12:42 +0000

Original article here, written for k6.io

This doc explains some key concepts about load testing websites, including:

The difference between backend and frontend performance testing
When to choose between protocol-based, browser-based, or hybrid scripts
Recommended practices about how to load test websites

All load tests try to simulate real user traffic to prevent failures, improve reliability, and release new code with confidence. But your approach to load testing must adapt to the type of application you want to test.

In this guide, learn about strategies to test websites, including specific recommendations for complex scenario scripting and test execution.

Load testing approaches

When you approach the load test, first consider the following perspectives:

Backend vs. frontend performance
Protocol-based, browser-based, or hybrid load testing
Component testing vs. end-to-end testing

Backend vs. frontend performance

Performance has a significant influence on the user experience of a website.

For example, when users want to visualize or interact with some information, they expect the website to react quickly.

To measure performance, testers often look at response times, which are affected by two main factors:

Frontend performance
Backend performance

Frontend performance

Frontend performance testing verifies application performance on the interface level, measuring round-trip metrics that consider how and when page elements appear on the screen. It’s concerned with the end-user experience of an application, usually involving a browser.

Frontend performance testing excels at identifying issues on a micro level but does not expose issues in the underlying architecture of a system.

Because it primarily measures a single user’s experience of the system, frontend performance testing tends to be easier to run on a small scale. Frontend performance testing has metrics that are distinct from backend performance testing. Frontend performance tests for things like:

Whether the pages of the application are optimized to render quickly on a user’s screen
How long it takes a user to interact with the UI elements of the application.

Some concerns when doing this type of performance testing are its dependency on fully integrated environments and the cost of scaling. You can test frontend performance only once the application code and infrastructure have been integrated with a user interface. Tools to automate frontend testing are also inherently more resource-intensive, so they can be costly to run at scale and are not suitable for high load tests.

Backend performance

Backend performance testing targets the underlying application servers to see how they behave under production-like conditions. When it comes to websites, while frontend performance involves how assets included in the page are rendered, backend performance focuses on how those assets are processed by the application servers, served to users, and downloaded by browsers.

Backend testing is broader in scope than frontend performance testing. API testing can be used to target specific components or integrated components, meaning that application teams have more flexibility and higher chances of finding performance issues earlier. Backend testing is less resource-intensive than frontend performance testing and is thus more suitable for generating high load.

Which one should you test?

It depends! Ideally, both.

Frontend testing tools are executed on the client side and are limited in scope: they do not provide enough information about backend components for fine-tuning beyond the user interface.

This limitation can lead to false confidence in overall application performance when the amount of traffic against an application increases. While the frontend component of response time remains more or less constant, the backend component of response time increases exponentially with the number of concurrent users:

Testing only frontend performance ignores a large part of the application, one more susceptible to increased failures and performance bottlenecks at higher levels of load.

Testing only backend performance, on the other hand, ignores “the first mile” of user experience and breadth. Backend testing involves messaging at the protocol level rather than interacting with page elements the way a real user would. It verifies the foundation of an application rather than the highest layer of it that a user ultimately sees.

Testing both frontend and backend performance leads to the best overall performance and user experience for your application. Ignoring one or the other exposes you to performance bottlenecks that significantly decrease a user’s satisfaction with your application.

However, if your testing is smaller in scope, you can choose to focus on either frontend or backend with the goal of eventually building a test suite that encompasses both.

Component testing vs. end-to-end testing

When you’re testing a web app, you may wonder whether how you structure your test scripts. Consider the following two methods.

Component testing

One way you could load test a web app is to load test its components. Maybe you know there are issues with a specific functionality due to previous production issues, or maybe you’d like to target components that are business-critical to reduce risk exposure.

In these cases, the type of test you write may:

be protocol-based
call only those API endpoints that are relevant
have smaller or no sleep/think time
be focused on eventually stress testing a component or service or finding its breaking point

Testing in this way is more flexible. With protocol testing, you can specify the endpoints or servers you’d like to hit, and narrow your test’s target that way. You can test specific functionalities while skipping others that come chronologically before it in a standard user flow. You can more finely control the type of traffic that is generated. If you’re recreating a specific mix of requests, you can script those requests and reproduce them in a repeatable manner.

Doing component testing for load may not always require that your script behave like an end user. In fact, it may be necessary to artificially inflate traffic to more quickly reproduce issues, or to deflate traffic to reduce the noise in the logs. By the nature of this type of testing, scripts don’t contain the full flow of a request that you would expect to see in production, so realism is not a priority.

End-to-end testing

You could also do end-to-end testing against a web app. End-to-end testing seeks to replicate real user behaviour and track its effects across the entire stack. When doing end-to-end testing, you might:

do protocol-level, browser-level, or hybrid load testing
replicate the actions in a typical user flow
think about the performance of the entire workflow as well as how long a request took to be processed by each component
write scripts that access the application the same way users would, such as by browsing to the homepage before finding their way to other parts of the website

This type of load testing is broader in scope than component testing, but shallower in terms of depth. With end-to-end testing, you get a better idea of the full user experience of the application as a whole. However, it can also be more complex to troubleshoot, as you have more components to monitor and more places to look for issues that are found.

End-to-end testing is based on real user behavior, so it’s often important that end-to-end test scripts also be realistic.

Protocol-based, browser-based, or hybrid load testing

The decision of whether to test the frontend, backend, or both will also affect the type of load testing you should carry out and the kind of scripts you should write.

Protocol-based load testing

Protocol-based load testing verifies the backend performance of an application by simulating the requests underlying user actions. For websites, this commonly involves HTTP requests that bypass the user interface of your application and are sent directly to a server or application component.

For example, a protocol-based load testing script might request all the resources on a webpage from the application servers, but those resources are merely downloaded. The response times reported by a purely protocol-based script do not include frontend metrics such as the time taken for images to render on a browser. The load is generated by simulating many requests sent to application servers.

While protocol-based load testing may seem to lend itself better to component testing, you can also do end-to-end website testing with protocol-level scripts.

Sample protocol-based test script

The following is an example of a protocol-based load testing script in k6 that fetches the homepage, along with resources embedded into the page.

import http from 'k6/http';
import { sleep, check } from 'k6';

export function Homepage() {
 const params = {
 'sec-ch-ua': '"Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"',
 'accept-encoding': 'gzip, deflate, br',
 'accept-language': 'en-GB,en;q=0.9',
 };

 // 01. Go to the homepage
 let responses = http.batch([
 ['GET', 'https://mywebsite.com/', params],
 ['GET', 'https://mywebsite.com/style.min.css', params],
 ['GET', 'https://website.com/header.png', params],
 ['GET', 'https://website.com/polyfill.min.js', params],
 ]);
 check(responses, {
 'Homepage loaded': (r) => JSON.stringify(r).includes('Welcome to my site'),
 });

 sleep(4);

 // 02. View products
 responses = http.batch([
 ['GET', 'https://mywebsite.com/products', params],
 ['GET', 'https://mywebsite.com/style.css', params],
 ['GET', 'https://website.com/product1.jpg', params],
 ['GET', 'https://website.com/product2.jpg', params],
 ['GET', 'https://website.com/displaylist.js', params],
 ]);
 check(responses, {
 'Products loaded': (r) => JSON.stringify(r).includes('Add to Cart'),
 });

 sleep(1);
}

Recording browser traffic helps you prototype to test websites on the protocol level.

Browser-based load testing

Browser-based load testing verifies the frontend performance of an application by simulating real users using a browser to access your website.

For example, a browser-based load testing script might include instructions to navigate to a page, click on a button, and type out text on a form. Those user actions then trigger underlying requests on the protocol layer, but only user actions are scripted in browser-based testing.

Unlike protocol-based load testing, browser-based load testing scripts generate load by starting multiple instances of browsers and interacting with your application the way real users would. Testing at the browser level can also be the only option for testing Single-Page Applications where a lot of the application logic is executed by client-side scripts.

Scripting on the browser level usually requires the use of different tools from the ones used to test at the protocol level. However, k6 has an extension called xk6-browser that allows the creation of browser-based test scripts alongside protocol-based ones.

Sample browser-based test script

The following is an example of a browser-based load testing script in k6 using xk6-browser on a dummy website. Instead of making an HTTP request, the script views the homepage, then looks for and clicks on a link to the product page.

import { chromium } from 'k6/x/browser';
import { sleep } from 'k6';

export default function () {
 const browser = chromium.launch({ headless: false });
 const page = browser.newPage();

 // 01. Go to the homepage
 page
 .goto('https://mywebsite.com', { waitUntil: 'networkidle' })
 .then(() => {
 page.waitForSelector('p[class="woocommerce-result-count"]"]');
 page.screenshot({ path: 'screenshots/01_homepage.png' });

 sleep(4);

 // 02. View products
 const element = page.$(
 'a[class="woocommerce-LoopProduct-link woocommerce-loop-product__link"]'
 );
 element.click();
 page.waitForSelector('button[name="add-to-cart"]');
 page.screenshot({ path: 'screenshots/02_view-product.png' });

 sleep(1);
 })
 .finally(() => {
 page.close();
 browser.close();
 });
}

Tips for writing browser-level scripts

The following steps can help you get started with a browser-level test script.

Script user actions, not requests. Determine what a user might do for a particular task, and script interactions with elements on the browser level. For example, script what buttons the user clicks on.

Identify unique selectors. Once you have identified which page elements a user interacts with, use the Element Inspector for DevTools in your browser to find a unique, static, and simple way to identify each element. The script needs selectors to find the right element to interact with.

Use elements to verify responses. After every action, use locators to search for elements on the page that you would expect to find. This verification helps ensure that the script has reached the expected page.

Take screenshots for every action while debugging. One of the advantages of browser-based testing is the ability to take screenshots. After every user interaction the script simulates, use page.screenshot to save a visual image of what the script encountered for later troubleshooting.

Hybrid load testing

Hybrid load testing is a combination of protocol-based and browser-based load testing. While you can use two tools or two scripts to execute different types of load tests (one protocol-based and one browser-based), having both types of tests executed by the same script and the same tool is ideal. Aggregating results between different tools can be difficult at best and inconsistent at worst.

A best practice in hybrid load testing is to generate most of the load using the protocol-level test, and then to run a smaller number of the browser-level testing scripts. This approach:

reduces the number of load generators needed, since protocol-level testing requires fewer machines to generate the same load
measures backend and frontend performance in the same test execution
provides a single source of aggregated output at the end
reduces complexity in the creation and maintenance of scripts

Scripting considerations

When you script a test for a website, consider these recommendations.

Consider factors that affect script realism

Record your user journey. Using the browser recorder can facilitate initial test script creation by capturing all the embedded resources on webpages. Check out the Session Recording guide to learn more about how to auto-generate your load test from a user session.

Correlate data. Recordings often don’t take into account dynamic values that are generated anew each time a request is made. Go through the recorded requests and determine whether you need to extract values from previous responses and use parameters in subsequent requests. This practice ensures your VUs behave more like real users would.

Include or exclude static resources. Determine whether you should include or exclude static resources on pages such as images, JavaScript, etc. Consider including them if you want to measure overall user experience. Consider excluding them if you are using a Content Delivery Network (CDN) that is under a separate Service Level Agreement (SLA).

Exclude third-party requests. Don’t load test servers that you don’t own. Many applications make calls to third-party providers for authentication, social sharing, and marketing analytics. Disable these requests unless you have permission to include them in your tests.

Use concurrent requests. To mimic the way modern browsers download some requests in parallel, use batching.

Determine cache and cookie behaviour. k6 automatically resets cookies between iterations, but you can also change this behavior if maintaining cookies would be more realistic.

Use dynamic think time and pacing. Consider adding varying delays so you don’t artificially stagger a script with completely uniform delays.

Use test data. Real users typically don’t search for or submit the same data repeatedly. Consider adding a test data file for the script to iterate through.

Model test parameters and load profile after production. In k6, you can use test options to determine the exact shape and profile of your load test script. Select the appropriate executors for the job.

Create a reusable framework

Use tags and groups. Organizing requests by tagging and grouping them helps you consolidate like metrics and makes your test scripts more understandable by others.

Use scenarios. When combining protocol-based and browser-based tests, use scenarios to independently control their test parameters and executors.

Modularize scripts. Use modules to separate and organize functions for protocol-level testing and browser-level testing, and then use a test runner script to execute them. This approach means different scripts can be versioned and changed without affecting each other.

Integrate your tests into your CI pipeline. Adopting a “tests as code” approach enables you to tie your load tests more closely into your project’s existing CI/CD processes, helping you get the most value out of every test.

Test with thresholds in mind

Create thresholds for both types of testing. Some browser-level and protocol-level metrics cannot be combined because they don’t measure the same thing. Set thresholds for relevant metrics from both the browser-level script and the protocol-level one.

When possible, use hybrid load testing

Use protocol-based scripts to generate majority of the load. When writing the test scenario, use the protocol-based requests to simulate most of the traffic and use fewer VUs for the browser-based requests. Relying on the protocol-level traffic helps keep resource utilization on load generators down.

Execution considerations

When you run tests, consider the test environment and the load generator location.

Run your tests in the appropriate environment

Testing in pre-production environments and in production environments both add value.

Testing in pre-production environments (staging, test, system integration testing, user acceptance testing, and production replica environments) enables you to identify performance defects early, which can save a lot of time and effort (and reputation) later on. Running in test environments also means you can often afford to be more aggressive with your tests. However, it’s also more critical to get your load profiles right, and the test results you get may not necessarily apply to production.

Testing in production yields the most accurate results, but it’s also more risky. Often, testing production is the only feasible alternative. You can reduce the risk of impact to real customers while testing in production by using lower levels of load when running load tests during peak hours, schedule tests for off-peak hours, choosing load test types that are less risky, using techniques like synthetic monitoring that generate less traffic, using real user monitoring tools to get snapshots of user performance at load, and ensuring that your observability stack is working at peak efficiency.

Run tests where your customers are

The location of the load generator(s), where the traffic is coming from, can also have an impact on your test results. The question is: where are your end users located?

On-premises load testing can be ideal when testing early on in the development a website, or when there are machines that can be repurposed as load generators. However, testing entirely from within a corporate network may also yield false positives, in that response times reported are significantly lower than if the same application servers were accessed from across the country.

Load testing on the cloud is an essential part of the testing strategy for many public-facing websites. Using load generators on the cloud gives you access to test in different states and geographical countries, creating a mix of load generators proportional to your users’ locations. Cloud load generators are easier to provision and cheaper to maintain in the long run than on-premise ones. Load testing on the cloud can help you include the effects of network latency in your tests and yield results that are more realistic.

Recommendations

Here are some recommendations to help you plan, script, and execute your load tests for websites.

If you want to test the last-mile user experience of your website:

focus on frontend performance,
write browser-based test scripts,
and consider doing more realistic end-to-end tests of the user flow.

If you want to test the underlying infrastructure of your website:

focus on backend performance,
write protocol-based test scripts,
and consider starting with component testing and then gradually increasing the scope.

If your website is meant for internal or has limited access:

Use on-premise load generators located within the network that most of your users access the website from.

If your website is external and public-facing:

Use cloud load generators in the load zones where your users reside.

Test in pre-production environments when you can, but also consider testing in production in a limited capacity.

Load testing websites can be complex due to the number of viable testing approaches available, the scope for performance testing, and the potential effects of releasing undertested code. By following the recommendations we’ve put forward here, you can tailor your testing more closely to your objectives.

My Time Management Setup in Obsidian

Fri, 06 Jan 2023 16:09:54 +0100

My time management setup in Obsidian consists of a way to set goals, a daily note for rapid logging the present, and a review/retrospective system to evaluate past performance. In this video, I talk about all the plugins and apps I use to get all my responsibilities done– without me feeling like I’m overwhelmed.

Taking Notes From Youtube Videos in Obsidian

Fri, 30 Dec 2022 11:30:38 +0000

In this video, I talk about five different ways to take notes on YouTube videos in Obsidian, for those (like me) who love learning things on YouTube and want a way to capture new thoughts in Obsidian. I mention a few community plugins, a browser extension, and a separate service as well as my workflow for just using vanilla Obsidian, so you can take your pick!

Mastobot: a Mastodon Bot for Readwise and Obsidian

Sun, 25 Dec 2022 00:31:28 +0000

I’ve had a love affair with Readwise for a while, enough to write some code to extend it. I wrote a Python script to save Readwise highlights to my Obsidian vault, switched to the Readwise Official plugin when they released it, wrote another script sending Shortform highlights to Readwise, and then was happy when the two companies established an official integration thanks to me introducing them.

Maybe I should have learned to just be patient and wait for the awesome Readwise team to build integrations I want, but remembered I’m not that patient. This is about how I created a Mastodon bot to send Mastodon posts I like to my Readwise account, and, by extension, to my Obsidian vault.

You might say there’s a pattern here.

The problem

I’ve been dabbling in Mastodon for a while, but only recently got extremely involved with it when I decided to start and maintain a Mastodon server, pkm.social, for the personal knowledge management community.

Readwise already has an awesome Twitter integration that I really enjoyed. Basically, once you’ve set it up, sending any tweet to @readwise adds it to your Readwise account. This means that whenever someone says something actually meaningful or interesting on Twitter, I can make sure it gets sent to Readwise, and then from there to my notes in Obsidian.

I wanted the same setup for Mastodon.

The result

First, the end result: I managed to get Mastodon posts I send to my bot to be sent to my Readwise account. Here are some early posts in the “Tweets” section of my Readwise Library:

And here are the resulting notes in Obsidian, also automatically synced and formatted:

The solution

Next, here’s how to set it up.

Pre-requisites

You’ll need a few things to get started:

A Mastodon account (I use and co-host pkm.social for free)
A Readwise account (Here’s an affiliate link)
Familiarity with GitHub, the command line, Node.js (although I’m not a Node.js developer, so you don’t have to be, either)
The willingness/ability to leave a Node.js script running on a server somewhere

Setup

Clone my Mastobot repository.
In your Mastodon account settings, go to the Development section for your Mastodon server (e.g. https://pkm.social/settings/applications) and click New Application.
Fill in the application name (this can be anything you want), leave everything else the same, and then click Save Changes.
Click on the application you just created, and note your client key, client secret, and access token. You’ll need these later.
In this repository, copy the contents of .env.sample to a new file called .env.

It should look like this:

ACCESS_TOKEN=
CLIENT_SECRET=
CLIENT_KEY=
READWISE_TOKEN=
DOMAIN=

Fill out the fields in .env with the information you just obtained from your Mastodon account.
For the READWISE_TOKEN field, enter your Readwise access token.
For the DOMAIN field, enter your Mastodon server’s domain (e.g. https://pkm.social).
Run npm install to install the dependencies.
Run node bot.js to start the bot.
In Mastodon, find a post you like and reply to it with @accountname Your comment here. Within a few minutes, the original post you replied to will be sent to your Readwise account as a highlight and your comment as a note.

Limitations

This workflow is not without its limitations:

You need to keep the script running somewhere to be able to use it. I considered using a polling approach, but decided I wanted to experiment with Mastodon’t streaming API, which would also be more responsive, so I went with that.
Right now, because the bot takes environment variables about your particular setup, it’s not as versatile as the official Twitter implementation. As it is, it’s for one person’s use only.
It uses the “Tweets” category in Readwise, which is not ideal, but I can’t really do anything about that until they give Mastodon its own category. This isn’t really a big deal.
Right now, the bot doesn’t support images or other media embedded to posts.

I may address some of these issues, but for now, I’m just happy with the result. I fully expect that the Readwise team will eventually be able to flesh this out into a more robust integration, so I’m hesitant to spend too much more effort now that I’ve got it doing the minimum of what I wanted.

Resources

If you’re interested in doing this for yourself, here are some resources that might be useful:

Mastobot on GitHub
Mastodon API client for NodeJS
Mastodon official API
Great video tutorial series by The Coding Train on creating a Mastodon bot

Note: I’ve included affiliate links for Readwise in this post, in accordance with my Ethics Statement.

Visual Atomic Note Taking With Obsidian Excalidraw

Fri, 23 Dec 2022 16:35:57 +0000

Visual atomic note taking in Obsidian can be a great complement to taking notes in text form if you use tools like the Obsidian Excalidraw plugin. Obsidian Excalidraw lets you link and backlink with images, tag them for querying later, and even embed images AND parts of images. It’s an essential part of taking notes visually while not missing out on the benefits of plain text.

Part of this video was sponsored by Brilliant. Thanks, Brilliant!

Distributed Load Testing Using Kubernetes With k6

Sun, 18 Dec 2022 15:54:07 +0000

My colleague Paul Balogh and I demonstrate how to run distributed load testing using Kubernetes with k6 and k6 operator. k6 can already run up to 40,000 virtual users from a single load generator, but if you want to scale that up further, you can use the Kubernetes operator to create a k6 custom resource within your cluster.

Reading and Taking Notes on Kindle and Obsidian

Fri, 16 Dec 2022 16:28:29 +0000

Reading on a Kindle can be more convenient than reading paperbacks because you can import your Kindle highlights and notes into Obsidian or your note-taking app of choice. Here are some tips for making the most out of every book you read.

Obsidian Canvas Core Plugin (new in v1.0.0)

Fri, 09 Dec 2022 16:23:10 +0000

The new Obsidian Canvas core plugin was introduced recently in v1.1.0, currently available only to Insiders but soon available to everyone. Canvas is an optional plugin that gives Obsidian an infinitely scrolling space to arrange notes, media, and other cards (potentially including external websites). If you’ve heard of Scrintal, this core plugin brings some of the same visual thinking workflows to Obsidian, which I think is a very smart step.

My Tech Bag for Working Remotely

Fri, 02 Dec 2022 16:06:07 +0000

What’s in my tech bag? Here’s all the gear I bring with me for work and play, sometimes for months at a time, including my full digital nomad setup. This is also essentially my travel bag– just add a single packing cube for clothes and toiletries!

Making Notes Actually Useful in Obsidian

Fri, 25 Nov 2022 15:57:17 +0000

How do you go about making notes actually useful? Using notes well entails taking good notes in the first place, processing them, and then creating something new out of them. In this video, I talk about the plugins that I use in every stage of the process, and some guidelines for how I actually make a note on something I’ve learned into content that I (and others) can make use of.

This video was sponsored by Brilliant. Thanks, Brilliant! :)

Notion-like Content Calendar in Obsidian Projects

Fri, 18 Nov 2022 15:38:33 +0000

If you’re looking to create a content calendar in Obsidian just like in Notion, the new Obsidian Projects plugin presents new options for managing content using multiple views: Table, Board, Calendar, and Gallery to start with, but hopefully more in the future. Projects is a framework for creating new views for the same data and aims to play well with other plugins.

How Many Obsidian Vaults Should You Have?

Fri, 11 Nov 2022 14:08:39 +0100

The more notes you have, the more you might wonder if you should have multiple vaults in Obsidian. There is no definitive answer, but in this video, I talk about reasons you should have only one vault as well as good reasons to have several vaults, based on my own experience. Like with anything I say, take what’s useful and do what makes sense!

Notion Database Views in Obsidian (Live)

Mon, 07 Nov 2022 14:06:02 +0100

Have you ever wanted rotating Notion database views in Obsidian? Marcus Olsson’s Obsidian Projects plugin does exactly that, letting you create Table, Board, Calendar, and Gallery views for the same notes in Obsidian, just like in Notion! Join us as Marcus shows us how this awesome new plugin works and how it can be used to display the same data in different ways.

Resources

Distributed Tracing With Grafana k6 (k6 Office Hours #70)

Fri, 04 Nov 2022 14:03:04 +0100

Just announced: distributed tracing with Grafana k6 x Tempo! Daniel González Lopes joins us to talk about what distributed tracing is, and how traces using the Tempo backend could help you troubleshoot your performance issues and improve overall reliability. This new integration was just announced at ObservabilityCon 2022, and Daniel’s here to tell us all about it!

Resources

How to Organize Your Notes in Obsidian

Fri, 04 Nov 2022 13:57:35 +0100

Wondering how to organize your notes in Obsidian? In this video, I talk about some methods in Obsidian for imposing structure and organization, the LATCH method for adding data to your data, and which methods I would use in Obsidian for each part of LATCH (Location, Alphabet, Time, Category, and Hierarchy). This is not meant to be a definitive or prescriptive guide for how YOU should organize your notes. Instead, take this as a look into how I prefer to do it, in the hopes that you may find inspiration for how you’d like to do it as well.

Taking Better Notes From Books: Shortform-Readwise-Obsidian

Fri, 28 Oct 2022 16:00:34 +0200

I’ve been using Shortform for over a year. I’ve written some code for it and I’ve also written a blog post about it. It’s an essential part of my workflow for taking better notes from books, particularly in conjunction with Readwise and my current note-taking tool of choice, Obsidian. Here’s what you need to know!

What is Shortform?

Shortform is a service that creates book summaries out of popular books, but it’s also so much more than that! They’re like Netflix but for books, if you can imagine that. The summaries that Shortform gives you are not just your typical high school level summaries of a book report that we used to do when we were younger.

Sound familiar?

What makes Shortform different?

Here are things that differentiate Shortform from other book summary services.

Researched commentary

Shortform has really thoughtful and nuanced commentary that place it firmly head and shoulders above other services.

Here’s a screenshot of some Shortform commentary from the book Why We Sleep, taken on the Shortform mobile app.

The highlighted section is a Shortform note– additional commentary that the Shortform team added. In this case, they are fact-checking the author, citing sources that prove a statement the author made to be false.

When I was reading this book, I didn’t really think about questioning those claims. But Shortform thoroughly goes over every claim that Walker, the author, made in thsi book.

Reading with Shortform is like having somebody else read a book with you– a more critical and more well-informed someone who has the time to fact-check statements. It’s a bit like having your own personal book club.

Shortform vs. Blinkist

Another service in this vein that I’ve used is Blinkist. Let’s look at the same book in Shortform and Blinkist to figure out why I like Shortform so much better.

This is what the Blinkist summary for the book Indistractable looks like.

Blinkist is based on the idea that every book can be distilled into these blinks: very short summaries of every idea. For this book it has eight. Each of these blinks is a few paragraphs long. Here’s what the first looks like.

A blink is a straight summary. There’s no commentary. There’s no research or anything like that. It seems like the whole focus of this is to be able to distill exactly what the author has said and let other people do the work of trying to figure out for themselves whether something is true or not. There’s really not much more than just these blinks.

In contrast, if we look at the one on Shortform, this is the same book and it also has like a one page summary like Blinkist did.

But then it also has a full book guide, which goes through every single chapter or part and adds more detail into it.

As you can see, Shortform takes a different approach: instead of just summarizing the book, they’re truly trying to explain it in their own unique way, providing examples and making it more readable using bullet points.

Action-oriented exercises

Shortform also has these exercises based on the things that the author has said in the book. These exercises are peppered throughout and between these chapters, and they really force you to put things into practice right away as soon as you learn them, which is another element of learning.

Critical thinking: comparisons to other ideas

Shortform tries to examine ideas in the context of other books they’ve made summaries for. Here’s an example of a Shortform note where Shortform suggests the summary of the book How to Stop Worrying and Start Living.

Fans of the Zettelkasten method will be happy to see this, because Shortform often prompts me to consider the differences or similarities between ideas from different authors. It helps me see connections I might not have considered. Incidentally, I’ve also found many good books this way!

How to use Shortform with Obsidian

The other thing that really sells it for me is the integration. Because I don’t want that learning to stop in the Shortform app. You know me, and I want everything in Obsidian. And Shortform has an integration that Blinkist does not.

I actually tried to solve this problem myself a year ago, because no integration existed. I wrote a Python script that would take the Shortform highlights, because it has an API, and then call the Readwise API and kind of string them together. Now, that still works as far as I know, but that’s not what I recommend anymore. I talked to the Readwise founders and suggested to them that Shortform was a great company to partner with. Not too long after that, they came out with an awesome integration!

First, sign up to both Shortform and Readwise. They both have free trials, so you don’t have to commit to anything just yet.
Then, click on this link to connect your Readwise account to Shortform.
(optional) If you use Obsidian like I do, follow these instructions to install the Readwise Official plugin for Obsidian.

That’s pretty much it!

From now on, here’s how things will work:

You make highlights on books on Shortform.
Those highlights are sent to Readwise automatically.
Through the Readwise official plugin, those highlights get synced into your Obsidian vault.

The result: Shortform highlights in Obsidian

Here’s what my Shortform highlights look like in Obsidian. Note that after setting this up, all I had to do was make highlights on Shortform– it’s quite low-touch after that.

I tweaked the export settings in Readwise to set the format I want the note to have in Obsidian. You don’t have to do this, but if you want to, you can change those settings here.

My Shortform workflow for taking notes on books

This is pretty much how a lot of my book-buying and -reading goes:

I start by browsing books on Shortform and read the one-page summaries.
If I find one I like, I buy the book on Kindle.
I make highlights on the Kindle version. (Those are synced to Obsidian too!)
If I really like the book, I flip back to the Shortform detailed chapter summaries as I finish each chapter, because they provide more context. I make highlights and notes from Shortform here sometimes.
Sometimes I do the Shortform exercises too.
After I finish the book, I go back to the Shortform summary and make more highlights/notes.
After I’ve read my fill of both the Shortform summary and the book itself, I go back to Obsidian and write my own notes that combine what I’ve learned.

If that sounds like a lot, it is! But keep in mind that I don’t do this for every book. At every step of the way, Shortform helps me filter out books that I don’t want to read, or don’t want to finish.

Shortform definitely saves me money: it costs less than a book a month and it stops me from reading more than one book a month. From that perspective, it’s already worth it.

But more than that, the thoughtfulness of the summaries and the research that goes into those commentaries save me more than just money: they save me time. And that’s really what I keep coming back for.

Some of the links on this page are affiliate links. Click here to check out my ethics statement and see how I decide what to promote in this way.

All About Load Testing Metrics (k6 Office Hours #69)

Fri, 28 Oct 2022 15:00:07 +0200

Leandro Melendez and Nicole van der Hoeven discuss all about load testing metrics. What do they all mean? Which ones do you REALLY need? Understanding metrics and analysing test results is arguably the most important part of a load test– and also the most difficult! We’re answering all the questions you ever had about the many metrics that come with load and performance testing.

Obsidian for Non-coders

Fri, 21 Oct 2022 18:17:53 +0200

In this video, I talk about how to get started with Obsidian for non-coders, specifically tackling topics like what Markdown is and how to use it to write notes in Obsidian. I also talk about some Obsidian-specific syntax that help you do things like embed Obsidian notes, create footnotes, and create queries to find the right note.

In the video, I mention Shortform– follow that link to get 20% off an annual subscription!

Assertions in k6 (k6 Office Hours #68)

Fri, 21 Oct 2022 16:29:35 +0200

How do you do assertions in k6? CTO Pawel Suwala talks about different ways to implement BDD/TDD in k6, including an assertion library called k6chaijs. We also discuss the differences between checks, thresholds, and assertions.

End-to-end browser testing using xk6-browser (k6 Office Hours #51)
Browser-based testing (notes)
Browser testing and API load testing in the same script with k6 (k6 Office Hours #35)
In search of the best Pokémon: Browser automation and load testing in one script with k6 (presentation at Automation Guild 2022)
Real browser testing with xk6-browser

How to Take Notes for D&D in Obsidian, with Leah Ferguson

Sat, 15 Oct 2022 22:38:50 +0200

Obsidian moderator and my friend Leah Ferguson and I talk about how to take notes for D&D in Obsidian, including TTRPG plugins we use to prepare for, play, and process our D&D (and other TTRPG) sessions. We take our play seriously! :)

Load Testing Types and k6 Executors (k6 Office Hours #67)

Fri, 14 Oct 2022 18:36:55 +0200

What are the different load testing types, and how do they relate to the k6 executors? We go through every single k6 executor and how, when, and why you should use each one in this week’s episode of k6 Office Hours.

Tabs in Obsidian (new in v1.0)

Thu, 13 Oct 2022 22:28:48 +0200

Introducing: Tabs in Obsidian v1.0! Obsidian’s latest release brings tab functionality to Obsidian, further adding to its flexibility in terms of letting us build workspaces the way we want. We can now have tabs in Obsidian and use them the way we would browser tabs. This release also came with some improvements for themes and the overall Obsidian UI.

Real Use Case for Obsidian with Dataview

Fri, 07 Oct 2022 18:23:05 +0200

In this real use case for Obsidian, I use plugins like Dataview, Database Folder, QuickAdd, Templater, and Buttons to create a database in Obsidian to track Kickstarter projects I’ve backed. I show two approaches to creating a database and demonstrate a relatively new plugin, Database Folder, which shows a lot of promise.

All About Web Performance (k6 Office Hours #66)

Fri, 07 Oct 2022 17:25:24 +0200

My fellow Developer Advocate and tester Marie Cruz and I talk all about web performance and best practices for making websites faster. What’s the difference between performance testing a website vs. an API? What are some things you should keep in mind when testing a web app?

What's in my EDC Bag (2022)

Fri, 30 Sep 2022 12:47:32 +0200

What’s in my tech EDC bag? In this video, I show you the sling bag I bring around on a daily basis, and every bit of gear in it. This is my everyday carry for 2022, and everything I take with me to stay productive on the go.

Here’s everything I mentioned in the video above that I could find a link for:

Samsung Galaxy Active 2: my smartwatch
d20 spinner ring: a ring that acts like a 20-sided die for roleplaying games on the go
Samsung Galaxy Z Fold 3: my primary mobile phone
Aer Day Sling 3 X-Pac: my sling bag of choice
Nitecore Tini 2 USB C flashlight: tiny USB-C rechargeable flashlight
Nite-Ize locking S-biners: for a little extra security for zippers
Orbitkey holder: to hold my keys
Nano bag 2.0: compactable bag
Kable Card: all essential cables and adapters, a light, a mobile stand, and a memory card reader all in one card
Aurtech bluetooth keyboard: portable bluetooth keyboard
Samsung Galaxy S Pen Fold Edition: stylus for my mobile
Sony WF 1000 XM3: wireless noise-cancelling earbuds
Spigen Arc Powerstation 20W: small GaN charger
Secrid mini-wallet: my minimalist wallet
Dispel Dice Lilac Song: a full-sized d20 I carry around
Sprawlopolis: small cooperative card game
LIE: fun party game for 2-6 players and is the size of a pack of gum
Victorinox Jetsetter: small, travel-friendly multitool
Mighty Hanks: microfiber on one side, cotton on the other
Burts Bees: lip balm
Small red Sharpie: mini marker
Joby Gorillapod GripTight Smartphone Micro Tripod: tripod for my mobile
Samsung Galaxy Note 10+: secondary mobile
Topo designs micro pouch: small pouch
Tile Mate Pro: bluetooth tracker
Trezor Model T: hardware wallet and security key

Performance testing for beginners (k6 Office Hours #65)

Fri, 30 Sep 2022 12:45:08 +0200

My colleague Paul Balogh and I interview the k6 technical writer, Matt Dodson, who asks all the questions about load and performance testing that you were always afraid to ask.

The future of k6 (k6 Office Hours #64)

Fri, 23 Sep 2022 12:18:07 +0200

We’re talking about the future of k6 with CEO Robin Gustafsson and CTO Pawel Suwala. What’s next for k6? What new features are we working on? What’s the plan for further integrations with Grafana? What are the plans for the k6 team as we transition out of the honeymoon period after the Grafana acquisition?

How I Learn Languages in Obsidian

Fri, 23 Sep 2022 12:14:58 +0200

In this video, I demonstrate how I learn Swedish in 30 days using Obsidian and other apps. Learning foreign languages is never easy, but having a system in place definitely makes the process easier. The Obsidian Spaced Repetition plugin in particular helps create flashcards that I can use on my laptop or mobile devices.

Visit Brilliant (affiliate link) to get started learning STEM for free, and the first 200 people will get 20% off their annual premium subscription. Check out my Ethics page to see how I decide on products to promote.

Getting started with xk6-browser (k6 Office Hours #63)

Fri, 16 Sep 2022 12:13:33 +0200

Marie Drake, the newest Developer Advocate to join the k6 team, shares her experience in getting started with xk6-browser: how to install it, issues she encountered, and the state of browser testing with k6 as she sees it.

How to Use the Excalibrain Obsidian Plugin

Fri, 16 Sep 2022 12:11:37 +0200

In this video, I demonstrate how to use the ExcaliBrain Obsidian plugin and what it can be used for. ExcaliBrain is a very powerful plugin, and it can significantly improve your Obsidian workflows. I also mention go through installing and getting started with ExcaliBrain at four different levels of uses for it: letting it infer relationships, setting explicit relationships, creating custom relationships, and finally creating semantic relationships in ExcaliBrain.

Load Testing Apache Kafka using k6 (k6 Office Hours #62)

Fri, 09 Sep 2022 12:09:36 +0200

Senior Software Engineer Mostafa Moradian joins us to talk about load testing Apache Kafka using k6 and the extension he created, called xk6-kafka. He describes what Kafka is, the basics of Kafka, and why he thinks load testing it is important. Mostafa works on the backend team at k6/Grafana Labs.

I can finally tell you about Readwise Reader

Wed, 07 Sep 2022 18:01:12 +0200

Readwise Reader is the latest read-it-later app, except not just for books. Reader brings in your highlights from books, articles, tweets, PDFs, epubs, emails, and even newsletters! While it’s not yet officially launched, it’s now in public beta and you can sign up to be put on a wait list to try out this awesome product that’s been so essential to my workflows.

Note: I do have an affiliate link of Readwise, but this video was not sponsored. They know I’m making this video, because I had to reach out to get permission to post the screenshot of a private email, but they did not review this video and had no idea what I was going to say. :)

How to Take Notes from a book with Obsidian (The Culture Map)

Fri, 02 Sep 2022 17:58:52 +0200

I demonstrate how to take notes from a book (The Culture Map) using Obsidian. In the process of taking notes, I also create learning exhaust in the form of notes, an atomic essay to go on Twitter, a visual that summarises a concept, and a video presentation. Taking notes from books is more than just understanding what the author wanted to say; it should also involve consolidation and interrogation of the book’s ideas into our personal knowledge management systems.

Resilience and Chaos Testing with SteadyBit and k6 (k6 Office Hours #61)

Fri, 02 Sep 2022 16:56:14 +0200

What is the difference between resilience testing and chaos testing? SteadyBit CEO and co-founder Benjamin Wilms talks about how to integrate SteadyBit with k6 for both resilience and chaos, and why he prefers “chaos testing” to “chaos engineering”.

How I Study Using Obsidian

Fri, 26 Aug 2022 17:53:49 +0200

Visit https://brilliant.org/nicolevdh/ to get started learning STEM for free, and the first 200 people will get 20% off their annual premium subscription. I also talk about how I study in Obsidian: the courses, platforms, and apps I use– and how I make sense of it all using Obsidian. Studying isn’t just for students. as a professional, I am also constantly learning. To make sure I make the most of the costs of education, I’ve made a deal with myself to commit to processing everything in Obsidian.

This video was sponsored by Brilliant! (affiliate link)

Live - Visual PKM With Zsolt Viczián

Sat, 20 Aug 2022 17:48:41 +0200

What is a visual Personal Knowledge Management (PKM) and how can it help you make connections between your notes? Zsolt Vizcain, developer of the Obsidian Excalidraw and Excalibrain plugins, joins me to talk about how to use his plugins to make your PKM system more visual.

Zsolt makes videos too! Check his channel out: https://www.youtube.com/c/VisualPKM

What is browser testing, with Marie Drake (k6 Office Hours #60)

Fri, 19 Aug 2022 17:51:38 +0200

What is browser testing, and do we really need it? What does browser or automation testing have to do with load testing? Our new colleague, Marie Drake, joins us today to talk about why she joined k6 and why she is an advocate for browser testing.

How to Make Your Notes Visual in Obsidian

Thu, 18 Aug 2022 16:21:10 +0100

In this video, I demonstrate how to make your notes visual with visual PKM plugins like Excalidraw, Excalibrain, Advanced Slides, and Mind Map. Adding visual elements to your notes, whether through distinguishing text or adding images and video, helps make notes more meaningful to you, increasing understanding and retention.

Observability Trends and Future Best Practices (k6 Office Hours #59)

Fri, 12 Aug 2022 16:19:13 +0100

What are some observability trends and best practices in 2022 and going forward? Technical Program Manager Daniel González Lopes joins Developer Advocates Nicole van der Hoeven and Paul Balogh to chat about trending concepts and new areas of exploration in the field of observability. We’re going to talk about distributed tracing, continuous profiling, eBPF monitoring, and more!

How I Learn in Obsidian - Keeping up with tech trends

Fri, 12 Aug 2022 16:17:11 +0100

Here’s how I learn in Obsidian by processing articles I’ve read into consolidated notes that help me understand technical concepts. I also show how I improve my learning by incorporating visual elements using Obsidian Excalidraw, Excalibrain, and Advanced Slides.

Using the Obsidian QuickAdd Plugin

Fri, 05 Aug 2022 16:15:15 +0100

The Obsidian Quick Add plugin won #10 in the Obsidian Gems of the Year 2021, but how do you use it? There are four types of “choices” in the plugin: multi, template, capture, and macro. Learn how to use each one with examples for real use cases.

1 Year Ago We Were Acquired. Here's what happened next. (k6 Office Hours #58)

Fri, 29 Jul 2022 16:12:34 +0100

In June 2021, CEO Raj Dutt announced that k6 had become part of the Grafana Labs family, and the year that followed has been replete with changes in features, organizational structure, and overall culture within the k6 team. In this video, Nicole van der Hoeven and Paul Balogh reflect on everything that’s happened in the last year and how k6 has fared 1 year after the acquisition.

How to Do D&D Prep in Obsidian

Fri, 29 Jul 2022 15:57:47 +0100

In this video, I show you in detail how I do D&D prep in Obsidian as I prepare for my D&D 5e game. I also show my setup using plugins like Obsidian Templater, Callouts, Leaflet, QuickAdd, Buttons, Initiative Tracker, and more.

Why You Should Move to Another Country

Fri, 22 Jul 2022 19:00:33 +0100

If you’re wondering why I’ve lived in 5 countries, here’s a video to explain it. Living abroad is something I do intentionally, and with an eye on how it would affect me financially. 5 countries might sound extreme, but I’ve enjoyed every one! Every country has its taxation quirks, but doing the research will reveal a lot of opportunities to live the life you want while effectively earning more.

Workload Modeling for Performance Testing (k6 Office Hours #57)

Fri, 22 Jul 2022 17:00:05 +0100

What exactly is workload modeling for performance testing, and why is it so important? In this video, Leandro Melendez and I talk about the shape of your load profile, why increasing the number of users may not be enough, and how you can make your load tests more realistic.

Creating k6 Scripts With the Browser Recorder and IDE

Thu, 21 Jul 2022 14:33:05 +0100

In this video, I talk about creating k6 scripts with the browser recorder and by using your favorite IDE. Scripts in k6 can be created through the Test Builder on the k6 Cloud, but sometimes you just want to hit record instead of starting from scratch. Also in this video is a rundown on the basics of using the k6 CLI and some important commands for running tests locally and on the cloud– right from your command line.

Why Is Everyone Talking About Chaos Engineering? (k6 Office Hours #56)

Fri, 15 Jul 2022 18:00:13 +0100

Why is everyone talking about chaos engineering anyway? How different is it from testing? What does it have to do with performance? In this k6 Office Hours, k6 Technical Program Manager Vince Huang joins me and my colleague, Paul Balogh, to talk about these topics and more.

Introducing Popout Windows for Obsidian

Fri, 15 Jul 2022 16:00:52 +0100

Obsidian Desktop v0.15.6 has just released, and with it is a new feature: popout windows for Obsidian! This gives us the functionality to open multiple windows of the same Obsidian vault– great for anyone who’s been running out of visible space while working on their personal knowledge management system.

How to migrate to Kubernetes, with Carlos Ruiz Lantero (k6 Office Hours #55)

Fri, 08 Jul 2022 17:00:57 +0100

k6 Cloud Backend developer Carlos Ruiz Lantero joins Developer Advocates Nicole van der Hoeven and Paul Balogh to discuss how to migrate to Kubernetes. Carlos has been working on migrating k6 Cloud services from ECS to EKS, and in this video, he shares his best practices for migration and things he wishes he’d done differently.

June 2022 Patreon Q&A

Fri, 08 Jul 2022 16:00:39 +0100

So you’re using Obsidian Templater, but what do you do when your template changes? What is YNAB? How do you use daily notes in workspaces? How do you process emails? These questions were sourced from the awesome Patreons that support these videos. Thanks to you all!

Regular Reviews With Obsidian Periodic Notes

Fri, 01 Jul 2022 16:09:40 +0100

The Obsidian Periodic Notes plugin is the best way to do regular reviews. It allows you to create daily, weekly, monthly, quarterly, and yearly note templates to automate the process of doing personal retrospectives.

How to Use Maps in Obsidian - Leaflet and Kanban

Fri, 24 Jun 2022 17:00:54 +0100

I share how to use maps in Obsidian using the Obsidian Leaflet and Kanban plugins, specifically for planning a move to another country. Using a map and marking places on it is a great way to keep track of your exploration of a country when you’re not quite sure where to live yet!

How to create things with your notes

Fri, 17 Jun 2022 17:37:14 +0100

What do you do with all those notes? In this presentation, I talk about how to create things with your notes. I demonstrate my process of going from raw notes to this very presentation and share some of my strategies for creating output with little effort by learning in public and identifying topics to create content about using existing notes.

On May 19th, I did a talk at the Linking Your Thinking Conference about using your notes (or losing them), and about what we can do to make something useful of all our notes. This was my run through!

DevOps Paradox: Performance Testing With k6

Wed, 08 Jun 2022 17:40:22 +0100

Darin Pope and Viktor Farcic from the DevOps Paradox podcast and the DevOps Toolkit YouTube channel invited me onto their show to talk about performance testing with k6, and why it’s awesome to be able to write load tests in languages that aren’t vendor-specific!

Listen to the podcast here.

Visual Map of Content in Obsidian // Excalidraw plugin

Fri, 03 Jun 2022 15:30:37 +0200

Did you know you can create a visual map of content in Obsidian to consolidate your notes? The Excalidraw plugin by Zsolt Viczian can be used to create a dashboard in Obsidian, take sketchnotes instead of text notes, express your ideas visually, and many more. Maps of content are useful for bottom-up thinking, and having a visual representation of your notes can inspire you to consider other perspectives or communicate your ideas in a clearer way.

How to Be a Gish - API, browser, and chaos testing in one script (ExpoQA 2022)

Wed, 01 Jun 2022 14:06:49 +0200

For the ExpoQA 2022 conference in Madrid, Spain, I presented a talk about how to be a gish. In this talk, I covered the following topics:

what a gish is, and how it applies to the field of testing
how testing is a multidisciplinary field, and why generalists triumph over specialists in the face of uncertainty
how the tool we choose to do our testing informs the type of testing and the type of results we get
a hands-on example of how to do API load testing, browser automation, and chaos engineering in one script using a tool like k6

There was no recording of the conference talk, but here is a video of me going through my presentation beforehand:

You can find the slides here.

And here are some photos from the event!

Presentations as Code - How to use Obsidian Advanced Slides

Fri, 27 May 2022 17:00:03 +0200

The Obsidian Advanced Slides plugin is a great way to create presentations as code (or notes) from right inside your Obsidian vault. Using this plugin has helped me reduce the friction and the time between creating something and sharing it, whether that’s a presentation, a coding project, a video, or a blog post.

Use It or Lose It - Linking Your Thinking Conference

Thu, 19 May 2022 14:03:13 +0200

Today I spoke at the Linking Your Thinking conference about what I do with notes I take, and how I manage to use them to actually create stuff.

If you’d like to see the result for yourself, here are the slides!

How Real People Process Notes

Thu, 12 May 2022 17:00:49 +0200

There’s a glaring gap in most productivity systems: time. How do REAL people, with real jobs and obligations, find the time to process notes? Is Inbox Zero an achievable and sustainable goal? I think the answer is no. In this video, I talk about what to do instead, and what is a healthier way to look at productivity.

What Is Grafana Mimir? (k6 Office Hours #53)

Fri, 06 May 2022 17:00:28 +0200

Maintainer Marco Pracucci talks about the new Grafana project, Mimir, a next-generation time-series database for Prometheus. He joins Developer Advocates Nicole van der Hoeven and Paul Balogh to talk about what Mimir is, why it was needed, and how he helped scale it using k6.

Using the Obsidian Templater Plugin

Fri, 06 May 2022 16:00:22 +0200

What IS the Obsidian Templater plugin and more importantly, how do you use it? I outline the three main use cases for it, from the simplest to the most complex, to show off its capabilities and usefulness in a personal knowledge management system.

API load testing using k6, with Tom Miseur (k6 Office Hours #52)

Fri, 29 Apr 2022 17:00:26 +0200

In this second part of demonstrating the dogfooding we do at k6, I talk to Tom Miseur, Technical Specialist, who shows how he wrote API load testing scripts with k6 to test k6 Cloud– and what his results were after his testing.

Actually Getting Things Done With Obsidian - Checklist plugin

Fri, 29 Apr 2022 16:00:31 +0200

The Getting Things Done methodology is showing its age now, but there are still some parts of it that can be updated and used in the digital age. In this video, I talk about how to actually get things done with Obsidian using the Obsidian community plugin called Checklist.

End-to-end browser testing using xk6-browser, with Thomas Wikman (k6 Office Hours #51)

Fri, 22 Apr 2022 17:00:12 +0200

Have you ever wondered how to test a platform built for testing other products? Our frontend developer, Thomas Wikman, joins us on k6 Office Hours to talk about his experience being a user of one of our tools - xk6-browser - and how he uses it to test the entire k6 Cloud frontend.

Obsidian task management with Dataviewjs, Templates, Daily Notes

Wed, 20 Apr 2022 17:00:25 +0200

Ever wanted to use Obsidian for managing your tasks? In this video, I demonstrate one way to consolidate tasks from all your notes and show them on the date they’re due, using DataviewJS. Obsidian Dataview is a community plugin that can be used to show a task list.

How to Use the Obsidian Dataview Plugin

Wed, 13 Apr 2022 17:00:06 +0200

What even IS Obsidian Dataview? Dataview has quickly become one of the most popular plugins for Obsidian, but how do you actually use it? Whether you’re a non-programmer or an expert coder, here’s how to use the Obsidian Dataview plugin and some common use cases for it.

How to Run a Load Test With the k6 Cloud Test Builder

Tue, 12 Apr 2022 17:00:48 +0200

In this video, I demonstrate how to create and run a load test from scratch without coding, using k6 Cloud’s Test Builder. k6 Cloud is a load testing platform as a service, and you can run 50 free load tests on the cloud here.

The dev behind the Obsidian TTRPG plugins - Jeremy Valentine

Sat, 09 Apr 2022 19:00:57 +0200

Jeremy Valentine is the developer behind 13 Obsidian plugins, including 5e statblocks, Admonition, Dice Roller, Fantasy Calendar, Initiative Tracker, Leaflet, and more, and he joins me today to talk about all things Obsidian, plugin development, and maybe even what he’s working on next…

Load testing results in the k6 Cloud App for Grafana, with Edgar Fisher (k6 Office Hours #49)

Fri, 08 Apr 2022 18:00:18 +0200

There are many ways to get k6 load testing results into Grafana, but our frontend software engineer, Edgar Fisher, talks to us about the easiest way of them all: the new k6 Cloud App for Grafana.

TestOps Ask Me Anything on Devops Toolkit

Thu, 07 Apr 2022 18:00:30 +0200

I joined my friend Viktor Farcic on his channel, DevOps Toolkit, to do an Ask Me Anything on the topic of TestOps.

We ended up talking a lot about k6.io, different Grafana Labs open-source projects, and what performance testing has to do with DevOps and Site Reliability Engineering.

Notion vs Obsidian - Databases vs Dataview

Wed, 06 Apr 2022 17:00:26 +0200

Notion and Obsidian are similar in many ways, but they have a fundamental difference in philosophy and the way they structure notes. In this video, I talk about structure as well as Notion databases vs. Obsidian’s Dataview plugin, and tell you why I ultimately ended up with Obsidian.

Building Reliable Systems With k6 (k6 Office Hours #48)

Fri, 01 Apr 2022 10:02:29 +0200

In this week’s k6 Office Hours, my colleague Pandurovic, an engineering manager/backend lead at k6, joined Paul Balogh and me to talk about how k6 works on the backend, and how the backend team is working to make it even more reliable.

Working in Devrel Chat (with Gift Egwuenu)

Thu, 31 Mar 2022 09:58:46 +0200

My good friend Gift Egwuenu, a Developer Advocate at Cloudflare, invited me to a livestream on her channel to talk about what Developer Advocacy is, what I do at k6, and how we both got started in the Netherlands. Gift and I have a lot in common, so I was happy to be on her channel!

New in Obsidian: Obsidian Callouts (available now)

Wed, 30 Mar 2022 17:36:20 +0200

Transcript:

One thing I’ve learned from writing technical documentation in the last few years is that good writing is skimmable, at least in the digital age. So since I look at my notes in Obsidian the same way as I do a sort of personal documentation, I think that a lot of the same principles apply.

Today I wanna talk about an early access feature that the Obsidian team rolled out two weeks ago, and it’s called Callouts. So before I demonstrate this, I want to say that this is only available right now in the Insider Build version 0.14, but that build always gets pushed to the public, it’s just a little bit ahead of the public release. So think of this as a bit of a sneak preview. (Note: Obsidian callouts is out now!)

What about Obsidian Admonition?

Now, before Obsidian Callouts was released, I was using something called Obsidian Admonition. So if we go here, I can bring up my Obsidian Admonition page, and it looked something like this.

You would do three backticks. That’s kind of the code block syntax. And you would do something like add note. Maybe you’d have a title here, like reminder, and then you’d have your note here. And because I have Live Preview turned on, if I enter out of this, it’s going to look like this. So this is what Obsidian Admonition always looked like.

So in general, the idea here is that to make your writing more skimmable, you have these kind of, well, callout boxes that highlight sections of your text and make sure that they address something that is about the notes that you’re writing, kind of like meta notes, they’re notes about your notes, but because they’re visually distinguished from the rest of the text, it’s a lot easier for you or for your readers to easily figure out whether something is important to read or only in specific situations.

Admonition/callout example

And that happens a lot in technical documentation, so it’s no secret that I have used it a lot in sort of programming kind of notes. So if we go back to my vault here, let’s look at my note on g. g is a package manager for Go, the language.

And here’s a type of callout or admonition that I used. And this is kind of like this page tells you how to use, what g is and how to use it. But this note, this particular callout, is kind of like an aside. It’s like, hey, just in case you get this error, here’s how to troubleshoot it. And it’s really cool that it’s hidden by, it’s collapsed by default. And you can just click on that, and then it says, oh, if you get the error, g not found, hey, maybe restart your shell session after installation.

So that sort of callout really helps when you’re trying to quickly skim a page. So programming is certainly one use for it, and another one is tutorials in general. You know, I have this a lot for my Obsidian stuff in case I need to say that this only works for Mac or this only works for windows or something like that.

And I also use it a lot in my D&D games. As a Dungeon Master, I have different scenarios that I may not even need to go into depending on what my players decide to do. That is the perfect task for callouts or for admonitions.

Enter Obsidian Callouts

So I was using this admonitions format with a code block for, basically, since I learned about it. It is made by a TTRPG plugin developer called Jeremy Valentine, and he’s made a lot of very popular ones, like the Dice Roller and the Initiative Tracker and Leaflet, and it just goes on and on. But I’m digressing.

This plugin was apparently so good that even the Obsidian team realized that it really should be part of Obsidian, because a key part of Obsidian’s functionality is documentation.

So the big release is that they’ve now incorporated Admonitions-like functionality into core Obsidian, not a core plugin, so it’s not something that you even have to install or enable, it just ships by default with Obsidian, which is really awesome.

How to use Obsidian Callouts

So let’s look into how to use it. I’m going to go to my Obsidian Callouts page here. And the syntax is a little bit different from the admonitions one. Instead of the code blocks with the three backticks, this callouts functionality now uses blockquotes.

> [!info] Your title
> The contents of your note here

So you do a single blockquote like an arrow, and then you do brackets, and then an exclamation point. And then there are a bunch of different types, which we’ll get into, but the most common one, I guess, is note. Then you would put your title here, and then you can put the rest of the note here. So this is now doing it as a blockquote. And if I exit out of that, you’ll see that it has a nice icon of a kinda pencil or pen there, and it is visually distinguished from the rest of the note.

Types of Callouts

note

Now, let’s go into the types, ‘cause there are a few of them.

The first one I already showed you, this is a note. I put this within code blocks just to show you the syntax. And then this is what it looks like. Basically, this is the standard format of callouts, you just change whatever’s here. And then you always put the title outside of the brackets, and then the content of the note below it. So this is what a note looks like. That’s probably the most common.

abstract, summary, tldr

And then another one is abstract, summary, or TLDR.

Now, all of these are the same, so like this is a TLDR, so I can actually edit it here as well. So here in this case, I was using TLDR, but if I do abstract, it’s going to look, actually, it looks a little bit different now that I think about it I thought it was going to look the same, but it’s also dependent on the theme that you’re using.

Yeah, so these three are actually different, even though they’re semantically the same. Well, that’s interesting. I just literally found that out right now as well.

info, todo

And there’s info and todo. So this I believe is the todo. And if I go to info, it looks slightly different, just has a little eye here.

This is what I use for descriptions. What we call it in TTRPG is what we call box text, when players enter a new scene and there’s a part that you read out word for word, rather than improvising it. This is what I use for it.

tip, hint, important

Then you have tips, hint, tip, hint, or important. This one is a tip.

Let’s see what important looks like. All right. Yeah, all right. So there’s some highlighting going on there as well, and it’s marked by an asterisk.

success, check, done

There’s success, check, done. I particularly like, I think it was done. It’s kind of like the todo had a checkbox. This one actually has the check.

questions, help, faq

There’s questions, help, or faq.

warning, caution, attention

There’s warning, caution, or attention, which is what I use for errors or gotchas when you’re programming or something things that you might fall into that I want to call out, because I did fall into it.

fail, failure, missing

And there’s also fail, failure, and missing. So this is all bad stuff, essentially.

danger, error

There’s danger, error. It looks like this with a little lightning symbol.

bug

There’s bug, which I use quite a bit when I’m reporting bugs or documenting bugs, anyway. This example, which kind of looks like abstract, actually.

quote, cite

Quote, and cite, which is kind of a blockquote, except a little bit fancier.

Collapsing and expanding callouts

So as you can see, there are a lot of different types of callouts. And aside from all of those, you can also choose whether it’s collapsed or expanded, and that’s kind of the folding behavior that I showed you earlier. So let’s just do that now as well.

I’m going to say bug, and I’ll say bug detected, let’s say. And this time though, where just before the title, right outside of the parenthesis for the type that I’m using, I’m going to type a minus. And I’ll put, this is a bug. And I get out of that. This is what it looks like.

So it is very different from this one where it’s just expanded by default, that’s a default behavior, but if you add the minus there, that means that you’re saying that this must be minimized or collapsed by default. So then you can always click on that, and look, you see what was in the text there. You can also specify that it’s always expanded by default, but it already is always expanded by default, so you don’t really have to set that, unless you’ve changed the settings.

Benefits of Obsidian Callouts

Better documentation

So this is a really cool and easy way of having these different colors. I know a lot of people that have come from Notion are kind of missing some of that in Obsidian. Well, this is an easy way to do that.

Ability to modify callouts

And you can also change how it looks by modifying either the CSS of the theme or by selecting themes that just really use it and display it to its advantages.

This theme here that I’m using is, let me check that, because I’m sure that people will ask. This is one of the gems of the year, one of the themes that won, it’s called Primary. Yeah, it’s really cool, and it’s on the light setting. This is a test vault.

Callouts are native on Obsidian Publish

And one of the coolest things about this new callouts feature is that it now works natively on Obsidian Publish. So I use Obsidian Publish for my public Obsidian vault, that’s notes.nicolevanderhoeven.com. There’ll be a link in the description if you want to see what that looks like. It’s basically my way of learning in public. And I select different notes to publish on there for people to see, for anybody to see.

Now, for a long time, admonitions, because it’s a plugin, wouldn’t show up up there. And then there was a way to do it where you can change the CSS and then kind of do it that way, but now that callouts is in core Obsidian, you don’t even have to do that. It’s just going to work right out of the box. So all of your notes that have the Obsidian callout syntax on them will show up the same way in your Obsidian Publish site, which is really awesome.

Why did the format change from Admonition to callouts?

So you might think, this is really just a format change, right? From admonition to callout, and part of it is that. I asked about why they changed the format, and the answer was that having the blockquote format falls back to blockquotes in unsupported Markdown renderer.

New format works better even if callouts aren’t supported

So for example, if you’re reading this note in an ID and IDE that doesn’t support that syntax, it’s at least still just going to look like a blockquote, which does make more sense than it looking like a code block, because you might not have code there. This is not just for code, this is for any sort of like sticky note that you wanna put on your notes.

Markdown inside blockquotes gets parsed

And also because it just uses blockquotes, this means that all of the Markdown inside it also gets parsed, including links and embeds and even Dataview queries apparently.

And that also means that spellcheck can check it. Because if you put something in a code block, that’s kind of a signal for spellchecks not to have a look at the contents of that code block.

Does this completely replace Obsidian Admonition?

So, what happens with admonition? Because it’s been around for a while, it’s actually still up. And in fact, admonition-style code blocks will continue to work. They’re now supporting, Obsidian is now supporting both the callouts and admonition if you have admonition plugin still enabled, then you’re still going to be able to use both. So it’s not like it’s going away, but you should probably use the callout syntax going forward, just because that’s what everyone’s going to be developing based off of.

Admonition can convert between formats

So there is also another thing that admonition is going to do is it does conversion. So if you have existing code block admonitions, then those can be converted to callouts. Actually, I want to show where you can find that.

So if you go to settings here and look at admonition, there are a bunch of settings here, but let’s look at, yeah, the conversions. You convert MSDoc admonitions to callouts, and you can convert code block admonitions to callouts as well. Now, this will modify your notes, so just be a little bit careful with it. Save a backup or do a push to Git if you save it in a Git repo like I do. But that’s what admonition is going to be doing.

Admonition lets you change callout appearance

And in addition to that, it’s also shifted to being able to let you change the appearance of those callouts.

So it has a bunch of functionality like, do you wanna add drop shadows? And do you want it to be collapsible by default? Because I said that it’s expanded by default, but you could change that behavior and toggle it. And then, you could still just expand it on a case-by-case basis. But if you don’t specify anything, then it’ll just say that it should be collapsed.

There’s also the add copy button, which is actually really interesting because a lot of times that is something that I just want to copy and paste to like my players or something if I wanna paste a description or if it actually is a code block or some sort of troubleshooting thing, so I enable that. And you can set admonition colors.

You can also change the icons that are loaded there. So right now, admonition uses the Font Awesome icons. So I was showing you all of these different types of callouts, right? But those are the default ones. And admonition adds the Font Awesome ones as well. And you can even add your own types of icons. So you can load additional icons. There’s Octicons and RPG-Awesome, which sounds really interesting, actually.

And a cool effect of that is we need to do this in admonition syntax, so it’s code blocks. Let’s say, let’s just go with question. This is a question. A cool thing that you can do with admonition is you can do icon, and then choose an icon that isn’t already in there. So as an example, I happen to know that there is a Font Awesome icon that’s a cloud, so I’m just going to put cloud there and then I’m going to put, here’s the answer. If I go out there, look, now it looks like a little green cloud, so I can change everything about how it looks. This is still in the admonition syntax, though, but it functions the same way as callouts, just extended.

I really like this for being able to give instructions and then say, you know, this is Apple-only and not for Windows. So that’s really cool.

Should you keep using Obsidian Admonition?

So I guess the question is, and this was my question when I learned about these changes, is it still worthwhile to use admonitions because of all these quality-of-life improvements?

I think it is, because, don’t get me wrong, I’m very happy that callouts is part of Obsidian now, that just made sense for an app that is so focused on notes and note-taking and documentation. You really need these different ways to take notes on your notes.

Custom icons and admonition types

But I do think think that admonition is still worthwhile installing and using if you want to delve down into the specifics of this feature. So if you want different icons that make sense to you, but maybe not to other people, this is still the best way to do it.

User interface for inserting callouts OR admonitions

Another cool thing that I didn’t show, let me show that now, is maybe you don’t want to remember the syntax with either with the blockquotes or with the code blocks.

So with admonition turned on and enable, installed and enabled, you can bring up your command pane, which for me is Command + P, and I’m gonna type in admonition, and then you can just either a callout or an admonition and a bunch of other things as well, but I’m just going to insert a callout. And if you didn’t remember what the types were, they’re all here. And this is what they’re going to look like.

Now, as I mentioned, I think these, the ones that I grouped, are actually supposed to look the same without the admonition plugin, it’s just that either the admonition plugin or my theme is making them look a little bit different, but these are the standard ones.

So then you can choose which one you want. You can have the title there. You can choose whether or not it is collapsible. It even shows you a preview. So let’s insert that. And look, I didn’t have to type out any of that and I didn’t have to remember any of that syntax. So that’s just a nice little upgrade that you can only get with admonitions.

Obsidian callouts is available now

Note: Obsidian callouts is available for all users for free now!

Again, Obsidian callouts, at the time of this recording, is only available in Insider Build version 0.14, which means you can only get it right now, if you need to get it right now, you’d have to pay for the Catalyst plan, which is a one-off $25 fee. I’ve paid that just because I wanted to give back to the Obsidian developers for this amazing tool that they just give for free.

But if you’re willing to wait, this feature, like all features, will soon be publicly available, Insiders just get it a little bit earlier. And when that happens, I’ll pin a comment in the comments below for this video so that you know when it’s available.

And if you’d like to learn about other Obsidian plugins, I do have a video on the top 10 Obsidian plugins, so maybe check that out.

Thank you for watching!

How to Analyze Load Testing Results with k6 (k6 Office Hours #47

Fri, 25 Mar 2022 18:00:56 +0200

In this k6 Office Hours, Paul Balogh and I discuss the different ways to turn k6 metrics into graphs for analyzing performance testing results.

How to Process Notes in Obsidian // Readwise Official Obsidian plugin

Wed, 23 Mar 2022 16:00:28 +0100

How do you process notes in Obsidian? I demonstrate how to use the Readwise Official Obsidian plugin to go from raw content to notes in a Zettelkasten-like Obsidian vault. Sign up for Readwise using my link and get double the trial period.

Note: This post contains an affiliate link for Readwise. Clicking on that link will give you a two-month trial of Readwise, double the usual amount, and they give me a small percentage if you sign up for a plan. Check out My Ethics Statement to see my standards for promotions.

Obsidian Dataview, building a second brain, and more about me // Q&A Livestream

Sat, 19 Mar 2022 17:42:54 +0100

In celebration of reaching 1000 subscribers on my personal channel, I did a live Q&A where I demonstrated how to use Obsidian Dataview, my personal philosophy about building a second brain, and how I stay organized.

How to Make Kubernetes Clusters Production Ready using Crossplane, with Viktor Farcic (k6 Office Hours #46)

Fri, 18 Mar 2022 17:30:45 +0100

In this video, Viktor Farcic, Paul Balogh and I talk about how to make Kubernetes clusters production-ready. Viktor is a Developer Advocate from Upbound, and he demonstrates a solution that ties together Crossplane, Grafana, Prometheus, and Loki to deploy and manage clusters in production.

E-residency Benefits: Why and How to Start a Company in Estonia

Wed, 16 Mar 2022 17:09:33 +0100

In this video, I talk about why I got e-residency four years ago, and how I used it to start a company in Estonia. I also mention a service called Xolo that I use to manage my Estonian company and do my taxes.

Transcript

85,000 people from all over the world are Estonian e-residents. And in 2018, I became one of them. I set up my own Estonian company. And for the last four years I’ve been doing some freelance work through it with clients in the U.S. and the EU. In this video, I’m gonna talk about why I started an Estonia company, and if you’re interested, how you can do it too.

Tax benefits

My first reason was taxes. Now, wait, if you are one of those people that think that Estonia is some sort of tax haven, you’d be wrong. You’re not going to be paying no taxes but you may be paying lower taxes than what you’re used to, depending on where you live.

The Estonian tax system is also simple– well, simpler than most tax systems. Everything is in English. I personally don’t speak Estonian, but I haven’t had any difficulty finding out rulings, everything is online and accessible in English via the Estonian government’s site.

Control

The second reason I started my company there was control.

As a freelancer, you may not have as much control over your financial affairs as you would like. The government probably dictates to some extent what you can declare as expenses or how much you get to take as income. There are still rules around that for Estonian companies, but in general I’ve found the rules to be way more reasonable and lenient than I would otherwise have to contend with.

For example, if you’re a small business and you choose to reinvest the profit that you’ve made into your business, so for example, buying equipment that you need or paying for other sort of operating expenses, then you don’t have to pay taxes on that income.

Remote-friendly

My third reason is that Estonian companies are set up to be remote friendly.

That is a very big deal to me, because when I set up a company here in the Netherlands, I had to be here, I had to have a permanent address where I could send correspondence by snail mail and they would piecemeal like my password and little bits of information and other unique IDs that I would later need to lodge my tax returns and other paperwork. There was so much back and forth.

Estonia isn’t like that. Everything is completely online. You do have to still have a physical address for your company, but there’s a way around that which I will get to soon.

The Cool Factor

The fourth reason, of course, is, honestly, the cool factor.

I am very much a proponent of new technologies and being able to take advantage of things that are cool about other countries is something that I’m a fan of. So when Estonia set up this e-residency program I was pretty sure that I was going to be applying for e-residency, even if I didn’t go ahead with my company.

I think this is the way of the future. And I can’t wait to see how many more countries are going to do it, following in Estonia’s footsteps.

Should you manage your company yourself?

So that’s why I started a company, but usually it’s the logistics around actually starting it that are more difficult than the decision itself.

I am a kind of person who prefers to do everything on her own. Just to give you an idea, I do my taxes, my husband’s taxes and company taxes here in the Netherland in Dutch. I specifically got lessons on reading the tax code in the Netherlands in Dutch. I also do it for Australia because we still have a property there and some other things.

And so I’m telling you this to say, I am not one to shy away from reading tax laws of another country, getting up to speed, and doing it myself. I’ve found that that is often not just the most cost efficient way, but actually the best way to maximize it because nobody else cares about my finances more than me. That’s not the strategy that I took with Estonia. I actually have hired a company to do that for me which is quite unusual, and they’re called Xolo.

What does Xolo do?

Quick disclaimer here, this is not sponsored by Xolo or anybody else. I haven’t spoken to them. They don’t know I’m making this video. My communications with them have been related to my personal financial circumstances. So they in no way had anything to do with what I’m about to say.

The only reason that I’m using my referral link is that I think it’s actually pretty useful. So the deal is that I get 150 euro if you use the link but you get 265 euro, which is exactly how much it costs for you to start up a company. I mean, if you want to go and try Xolo without using my referral code, please feel free to– it’s xolo.io. But 265 euro is also a lot of money. So it’s up to you. I do have an ethics statement about this sort of thing. Basically I will never promote or even talk about a product or service positively unless I’ve been consistently using it.

Here’s an email from Xolo in 2018, Back when they were still called LeapIn, just to prove to you that I’ve actually been using them for four years and heartily recommend their service.

You can start up an Estonian company all on your own without really any help from any other company, but it will take a lot more work if you’re just trying to find the easiest way possible. I’ve found that this is it.

Xolo’s offerings

Xolo offers a few services. First, they help you with registering your company. So you pay the fee to the state, not to them. It’s 265 euro.

They provide a physical address that your company can use. That includes mail forwarding, if you happen to use that for any company-related or government-related communication.

They help you with applying for a VAT number if you need that.

And they also help you with opening up a bank account. Now, I think that there are ways so that you can just use something like PayPal or Wise for your bank account. I did those, but I also chose to have an actual bank account in Estonia. That is the only part in this process where you actually have to be in Tallinn, in Estonia. Personally, I was looking for an excuse to go to Tallinn, so I went there and set one up. It was pretty easy but you can certainly start a company without it.

The most important thing that Xolo do is that they lodge the taxes for you every year. They give you advice on your company structure. They keep track of your expenses.

And the best part about Xolo is that everything is done online. They have a really nice web app which is pretty slick and modern and user friendly, and that’s what I use to upload my receipts, or if I need something reimbursed or marked as an expense, or if I’ve paid for something with my own pocket, and need the company to reimburse me, then everything is just handled for me. I really think that this is one of the selling points because if I have to get on the phone and actually talk to someone and explain something, I’m out. I’d rather just do it on my own at that point. But in the four years that I’ve been with Xolo, I haven’t had to be on the phone with them once. Their entire staff corresponds by email promptly and really in a thorough, professional way.

Xolo’s pricing

So let’s talk pricing. Xolo costs either 49, 79 or 199 euros a month. And that includes all of those services.

The only thing that’s not included is the state fee. And that’s something that you pay to the Estonian government. That’s the 265 euro to actually get your company registered. You can check that out for yourself on the government site. This is something that you’ll have to pay even if you don’t go through Xolo.

Step-by-step process of starting a company

So if you wanna start a company in Estonia, here’s what you have to do.

First, click on the referral link before the end of March. So March 31st, 2022, you have to have clicked the link.

That doesn’t mean that you have to have paid by then. You just have to have clicked it and gone through the signup process to create a completely free account. At this point, there is no obligation that you’re going to continue.

Step 2: Apply for e-residency

Secondly, you have to apply for e-residency.

This is something that you’ll have to do directly with the Estonia government’s site, which is pretty easy. There’s a link below to an application form that you’re gonna need to fill out with your information and identification. And then you wait around for, usually it takes less than a month. For me, I think it took a couple of weeks.

And when it’s approved you’ll get an email notification asking you to go to your local embassy. So in my case, I applied while I was already in the Netherlands. So I went to the embassy or consulate of Estonia here in the Netherlands just to pick up this ID and get fingerprinted. The e-residency itself costs 120 euro, but it is a one-off fee and you never have to pay for it again.

Pay for Xolo Leap

And the last step is to pay for a Xolo leap plan before May 31st, 2022.

So you essentially get two full months to do your e-residency. And it shouldn’t really take that time. And if you used my referral link, once you sign up for the plan, then they’ll waive the 265 euro that you would need to pay to the state for your company’s registration.

And that’s pretty much it. After that, Xolo will tell you how to proceed and give you the options for banking and that, depending on your business’ unique requirements.

Conclusion

Please don’t take any of this as tax or financial advice because I’m not a tax professional and I’m not a financial advisor. I am simply sharing my experiences and referring you to experts who will certainly know better than me on Estonian taxation law.

Having a company in Estonia is an essential part of my strategy for maximizing my freedom and building the life that I want by moving aspects of my life to countries that are best suited for them. This practice is called flag theory. And if you wanna learn more about that, check out this video to see how I use it to improve my life. Thank you for watching. Or as they say in Estonian, I think, aitäh!

Note: This contains an affiliate link for Xolo. That means that if you use the link to sign up for Xolo, you get some money off your plan, and so do I! Read my Ethics Statement to find out how I decide whether or not to promote a product or service.

Real Browser Testing With xk6-browser

Wed, 16 Mar 2022 16:55:56 +0100

Transcript:

Without a doubt, my favorite k6 extension is xk6-browser. xk6-browser finally brings browser test automation to k6. So you don’t need to learn a new tool or figure out how to integrate it if you already have load tests. You can create JavaScript-based, browser-driven tests with k6. Here’s how.

Since xk6-browser is an extension, we’re gonna need to build a custom version of k6 that includes that extension. And the thing that builds that is called xk6. What I’m doing here is I’m instructing xk6 to create a build using version 0.35 of k6, that includes xk6-browser. And this is a link to the repository where xk6-browser is stored.

Writing a browser testing script

So now that we have a version of k6 that includes xk6-browser, the next step is to create the browser script. Now you can do that in the one script, but just for ease of debugging I’m gonna put it in another script for now. We’re not going to be using the HTTP requests anymore so I’m going to import launcher from k6/x/browser. And I’ll save that as browser.

I’m definitely going to be using sleep, so let me go ahead and import that now as well. We’ll also need a default function for this. We’re setting up an instance of the launcher that’s going to launch Chromium in particular, and we do want to see it so I’m going to say headless: false. If you set this to true, that means that you won’t actually see the web browser being driven by the script, which is handy to see during demos. So we’ll do that, and then we have to set up the context.

So browser, oops, new context. And then we set up a page. The first one is go to the homepage. The second one is to view a product. Third, add the product to the cart. Fourth is go to the cart and then last, I want to check out. So to do that, I’ve already defined the browser context in page. Now I need to do page.goto http://ecommerce.k6.io and I will also want to wait for waitUntil: networkidle.

Great. I wanted to add some sort of check to make sure that I’ve actually gotten to a place so, let’s open up DevTools here and I wanna make sure that some results are displayed. I don’t necessarily need, you know 1-12 of 17 but, I want this one. Okay, so I’ll also say page.waitForSelector and that’ll be this.

And then let’s do a screenshot, because we just for, just in case we wanna debug later we wanna make sure that we will be able to have a look at at this screenshot to see what exactly went wrong. And I’ll put that in screenshots and I’ll say 01_homepage. Then let’s do a sleep. And I always like to do the random one. Random, and let’s say it from zero to four. This time I’m doing the custom k6 version browser.js.

For browser-based tests, I find it much more useful to actually see what’s going on rather than to look at the output. Because as you can see there are all sorts of errors here and I believe that they’re actually application errors so they can be a little bit misleading. It looks like that happened but let’s just look at the screenshots. Screenshot, yeah. That, that loaded fine. So we can continue.

Next step is to view a product. Okay, let’s go back here. This time I want to click this thing. It doesn’t really matter which, which item it is. I’ll just choose the first one. Looks like this is the actual product link. So let element = page. And this is where we actually paste the selector which is what I got from DevTools.

And then we haven’t done anything yet with it. We want to actually click it. So element.click. After that though, I’m going to want to make sure that we get to the right page. So, let’s see what happens when we click it. What can we look for? I guess we can look for this add to cart button, cuz if the item didn’t show up then we wouldn’t be able to add it to the cart. So let’s grab that. Button, button name is add to cart. Copy that.

So then I’ll say page.waitForSelector, the selector with a button name there. Take another screenshot because why not? I’ll say 02_view-product and then another sleep. From here it was pretty much more of the same. Here’s me scripting adding the product to the cart, viewing the cart and then finally going to the checkout page. And here’s what happened when I played it all back. Here it is going to the homepage, and then clicking on the first product, and then adding that product to the cart, and then clicking view cart, and then proceeding to checkout.

The best part about this is that you don’t have to choose between the browser test and the protocol test. They’re both in k6 and written in JavaScript so you could include them in the same script. You can use stages to schedule when one starts and when one ends. You can change the load profile, add checks and thresholds. And you can also use all of the output options that are available for protocol tests for the browser tests as well.

The catch is that xk6-browser is really new. We’ve just announced it. It’s still in beta so you are likely to come across more than a few bugs. However, I’d still recommend that you give it a go. I’ve been playing around with it and I’ve found it more than serviceable for a lot of my use cases. Plus it’s just really cool to be able to test front-end performance along with back-end performance in the same testing script.

Thanks for watching. I will put a link to the xk6-browser repository below and happy testing.

Load Testing for DevOps with Victor Farcic (k6 Office Hours #45)

Fri, 11 Mar 2022 11:12:52 +0100

How can you use load testing for DevOps? What kind of performance testing can or should be included in CI/CD pipelines? Viktor Farcic from Upbound and DevOps Toolkit join Nicole van der Hoeven and Paul Balogh to discuss how he sees k6 fitting into the daily duties of a DevOps engineer.

Why I Chose Obsidian Over Roam Research (as a Roam investor)

Wed, 09 Mar 2022 10:03:41 +0100

I was a Roam Research power user, a Believer, and investor before I discovered Obsidian. Here’s why I still ended up choosing Obsidian over Roam as my personal knowledge management tool of choice. My goal with this video is to share with you my reasons for making the choice that I did, not to judge you or anyone else who disagrees. In the end, we all have to decide for ourselves what tool fits best in our lives.

Private Load Zones for Load Testing (k6 Office Hours #44)

Fri, 04 Mar 2022 10:01:57 +0100

Load testing on the cloud definitely has its advantages, but what if you need to load test behind a firewall? Here to talk about using private load zones in k6 are k6 CTO Pawel Suwala and k6 Backend Lead Samuel Regandell. Private load zones are a new feature of k6 Cloud that will allow you to run load tests against applications that are not publicly accessible and also use your own AWS account to generate the load.

Basics of Chaos Engineering for Testers

Wed, 02 Mar 2022 09:59:52 +0100

What is chaos engineering? And what does it mean for testers? In this video, I go through the basics of chaos engineering from the perspective of a software tester. How can we go about incorporating principles of chaos into our testing activities?

Postman for load testing using k6, with Tim Haselaars (k6 Office Hours #43)

Fri, 25 Feb 2022 18:00:11 +0100

Postman is a near-ubiquitous tool for anyone doing API testing, and many teams use collections to create and organize complicated test suites. However, there’s one thing that Postman isn’t good at: load testing. That’s where k6 comes in.

In this video, Tim Haselaars joined k6 Office Hours to talk to Paul Balogh and me about how to use Postman collections to run load tests using k6.

Transcript

Hello everyone and welcome to another k6 Office Hours. I am Nicole van der Hoeven and today I have two guests. One of them may already be familiar to you, Paul Balogh.
Hey everyone.
And our special guest is Tim Haselaars. Tim, would you like to introduce yourself?
Okay, hi, nice to meet you Nicole and Paul. I am Tim Haselaars. I’m living in Belgium. I’m a product manager, and I work at the marketing automation company called Selligent. I have a passion for APIs. I love music, I’m a DJ as well in my spare time.
Oh, nice.
And if I still have some spare time left, I like to contribute to a number of open source projects, which the Postman-to-k6 is one of the projects I’ve been working on together with the k6 team in the past. So that’s a bit of why I’m here today. So nice to meet you both.
Yeah, that’s a pretty big if though, if you have time, you’re a DJ.
It’s a hobby thing, but I’m really proud of it actually, because I’ve been playing a lot, and there’s one festival in Europe, Tomorrowland, I’ve been playing it each year for the past 10 years.
Oh, wow.
So it’s a small stage, but I’m really proud of it.
Nice.
Yeah.
Well, that is awesome. You should be proud of that. I think that kind of makes sense, because product management is such a multidisciplinary role as well. Like, it’s not all technical, and it’s not all people skills. It’s a mixture of both, and a lot of other things.
Yeah, that’s also what keeps it interesting, at least for me, what keeps it interesting, actually, so yeah.
So what do you typically do in your current job, and what does it have to do with k6?
Okay, in my current job, I’m a product manager, so we have a digital platform, marketing automation platform and I’m spearheading the API renewal program. So we had a number of APIs in our company which customers use to connect and exchange data with marketing data, and we’re renewing that, and performance is critical, because we have a number of really big customers in Europe, or , the Dutch luxury brand, and we talk about huge volume, so our APIs has to be scalable, and performant, and we use k6 to guarantee that we keep them up to standard, actually. So that’s what we do. And my role there is to make sure the teams, I help with the API design, and I check with the teams, and I document stuff together with the teams, we try to do that. So what I’m going to talk about is also my experience with k6 and the way we use it at the company a bit more.
Awesome.
Cool, so how did you first start, what made you start to use k6? Did you try other things before that?
Yes. In a previous job, a couple of years ago, I was working at a hosting company, and they were doing a, there was an anniversary of the national lottery, it’s like the lottery company, so it had,, they expected a lot of visitors to the website, and we wanted to make sure the hosting was up to par, and it was able to handle the volumes. And I tried to test it, performance test it, load test it, and I used Jmeter, and the experience was, I struggled a lot with it. And I started browsing for another solution, and then I hit, I found k6. text was really nice. You could really quickly start nice documentation, and that’s how I rolled into k6. And then I shifted jobs, and at the current job, I said we have to use k6, because it’s so much easier than using JMeter. You can script it, you can automate it, and that’s it, yeah. I used it once, and never looked back.
So nice, with the automation, did you put hook that in your CI/CD pipeline as well?
Yes we do. I can, I have a diagram. Like, I’m really proud of it. Maybe I’m a bit, how do you say, biased by it, but I’m really proud of what we’ve achieved with the automation aspect actually. And we came from a far away, a far place, but now, we have a nice automated flow, actually.
That’s awesome.
Which I hope I can showcase a bit later on in the talk.
Yeah.
Yeah, I’d love to see it. And do you use k6 Cloud or k6 OSS?
We had used Cloud in the past, I think right now, my team are mostly using the OSS version of it. I don’t know why. I have to check with them, actually, but I think we had a number of dashboards set up, and they were a bit detailed to what we need, or what I needed at least, and that’s why I think we stick with, for now we stick with the OSS version of k6.
Okay, awesome. So I guess people might be wondering, seeing the title of this video, what Postman has to do with either Selligent or k6.
Yeah, actually that’s a story I want to tell, how did we ended up with k6, and the whole story. So what typically happens if you start developing an API is you build your API, and then you want to test it, you want to play around with it. You want to experiment with it. And the first thing that you do as a non-technical user like, I take Postman. Postman is an application where you can model your request in a nice tool, and then you can just fire them. You don’t have to a curl script, or you don’t have to use any CI tools for making API calls. So that was the logical step. So we build an API, I want to experiment, play around with it, so we ended up using Postman for it. But maintaining Postman is not so easy when you go for automation flows. And then when you want to take a step further, if you want to do performance tests, that’s another a thing. And yeah, that’s why I think we went from what we did to Postman just the next, as a first phase. But sooner, the more end points we have, the more we wanted to test and the more we wanted to play around with it, the bigger everything became. So that’s why, yeah, we wanted to make it an automation level and make it a bit more streamlined so that the manual work would go down a lot.
Sounds like you have a fan out in the viewers.
Yeah.
Yeah. I like to make things visual, because it helps me visualize my thoughts and everything. So yeah.
I’ve used Postman before. I think it’s almost ubiquitous now for anyone who’s doing any sort of API testing. I think at some point people would’ve used Postman just because it is very easy to get started, but I do, I have also run across some of the issues that you describe when you’re scaling up, when you have a very complex test suite or you want to have one, it is, I think I personally still like the scripting approach of being able to have the full, you know, being able to have condition loops and that sort of thing. There’s still a lot more flexibility when, when you’re just dealing with pure code. But I know that I think it was, I mean, relatively recently when I started using Postman, it was just firing off requests, and it was like a single user kind of thing, but what are the features available for performance testing for Postman, you alluded to it, them, and you said that maybe they’re not, they weren’t exactly what you were looking for.
No, no. Maybe I should quickly show a bit how I–
Yeah, sure.
Work with it if that’s okay for you. Okay, I’m gonna showcase my, so well, just as a reference, I created fake endpoint and I was set up a Postman collection. So what you do typically in Postman is you have an API that you define. In our case, I created a k6 office hour CRM API, just plain and simple where you can see, yeah, you could see some context you potentially want to interview in the future, and you model them. So you have your endpoint, you just put them in. You say, it’s a get post, put, whatever you need, and then you can just fire it and it’s response. And so this is an actual API request happening behind the scenes. That’s really nice because as a non-technical user, you don’t have to have any technical expertise. You just enter the endpoint percent and you’re up and running. But it also has some nice features if you want to make complex things, like you said, Nicole, if you want to use variables, because you might want to experiment with your API endpoints in different environments, so you can use variables. That’s what they use with the curly brackets here. So it’s a base URL. So I can shift around so if I want to switch, I have potentially two environments. I have a production one and a dev one. I can easily switch between them and then the variables will also switch along. That’s a really nice thing to do with Postman. The next thing, what I really like about Postman is that you can use it to create cruds and you can create flows with it. So it’s really straightforward. So if you say, I want to create one item and then I want to use the item in my detailed call in my, so I have all leads, create a lead and I want to get the leads, so this is the end point, you could just define it as a variable. You could say, I want to, that this lead ID can be used to fetch the right lead ID. So if you create one, that’s the body, I do a post request. And what happens is you get a new ID back, which is unique because I just created a new contact in our CRM application. And you can use scripting in Postman to say, ah, there is a data response, and I’m gonna store that response in my lead ID variable. So it allows some scripting within a GUI which is really nice, and it also allows you to define tests. Like I want to check what’s the response. It should be 200. You can define a Postman test. They use a bit of assertion, kind of syntax for that. So what started out as something really straightforward, I wanted to just test and play around with it, experiment of the endpoints. It became much more. And we started building like a full suite out of Postman, in Postman itself, a test suite. And that’s really cool because as a non-technical user, you could just share this collection with anybody on the team and they could just run the same request and play around with it. You can even pass it on to partners or colleagues or customers, when they say, when they want to use your API, you can just say, ah, you can use a Postman collection to play around with it. And another nice thing about Postman is that you can run scenarios, like I said, and you can run them at scale. So right now I’m doing get leads, creating a lead. Then I’m gonna fetch the lead that I just created. So I created Elon Musk as a user. I can update that one as well. So I can put a body and touch and I can even delete it. So if I would run this in a cycle, I would test actually the whole flow of a typical crud operation. You can fetch one, create one, edit one, delete one, and then it’s back to normal again. And so lemme talk about, yeah, sorry.
[Nicole] Sorry, I just wanted to, sorry to interrupt, but would it be possible to try and zoom here in this one to see if you can make it just a little bit bigger?
[Tim] Okay, something like that?
[Nicole] Yes. Thank you. Sorry.
[Tim] No, no, no worries. So what you can do Postman, if we talk about test automation, you can just say, I want to run all these, these API requests in the sequence, like they’re organized here, and you can just run them. So if you run them, you see you put in some test, this one is a success, success, creation successful. And so actually testing if your API’s responding as you expected, but you cannot test it at scale or at high volume. The only thing you could do in Postman is if I press run, I could define a number of iterations. So if I would say, I want to do 10 iterations, I’m just gonna run them now, so you can see, you can see it just creates blocks. but while I’m talking, it took to run all these 10, five times 10 requests, it took me, let’s say one minute to run. So we cannot really test at scale if, that’s not the use case of Postman. The use case of Postman is to experiment with request, do some testing on some validation of the response and the inputs are okay and that’s about it. So I needed something more if you want to do that scale. So while I discovered k6, k6 has this nice scripting language, you could define–
[Nicole] Tim before you, sorry. Before you move on, we do have a question. Chaitra asks, can you share this collection please, just to get along with Postman.
[Tim] Okay.
[Nicole] Is there a way that you could share, make that public or something?
You can, I haven’t tried it, but I guess if I click share and I could, can I create a JSON link? Maybe could JSON link, update the link, maybe like that? I haven’t tried this actually before either so.
[Nicole] Awesome, we’re learning in public.
[Paul] Exactly. No pressure, no pressure.
[Nicole] Yeah.
[Tim] Oh, should I put it in the comments, or how would I do that?
[Nicole] Yeah, sure. That would be great.
[Paul] Yeah, otherwise I was gonna say we could probably put something together too for after, you know, the presentation if someone wants to access that, if there’s no issue with that.
[Nicole] Okay, I’ll put that in the comments.
[Tim] So I was asked as well from the k6 team to update our blog post about Postman and k6. So what I’m gonna do after this session is going to review the blog post that they have already on the website and then update it with maybe some content that I’m talking about in this show here.
[Nicole] Great.
[Nicole] And we also got a comment from Jim Munro who says Postman is definitely not designed for load testing.
[Tim] No, no.
[Nicole] And that’s what it seemed like. So it seemed like you could only do iterations. Is that right? You can’t set, you can’t have more threads or users.
[Tim] No, that’s it. So the only thing you can do is you can run a sequence and you can run it in a number of iterations, but you can, you can’t define how many users, you can’t define how long iteration, you can define sleep, but that’s difficult. So Postman is great for what it is. It’s great for testing some of your contracts of your API, is great for experimenting with your API, and really kickstart using your API, but if you want to do volume, to test the volume at scale, I wouldn’t recommend Postman at all. But that’s also a bit the problem, because you already spent a lot of time making a nice collection, preparing it, grooming it, updating all the time. If you then want to go to k6, you would have to rewrite everything. So if you, this is how k6 syntax looks so you can define your iterations. You know this better than me actually. And you can just use a get request. And you can script your request. You can also check it. So that’s the same as you would do in the Postman where you test your response. You could check if it’s a 200. But if you have a complex scripts, you can, they become really large to maintain and to keep them up to date. And that’s what I struggled with because it felt like I’m reinventing the wheel. So I made a diagram out of it just to explain it. So if you have an API and you have to update it, you would have to really update your collection and say, every time something changes an endpoint was added or a property was changed, you needed to update, go into the Postman collection, manually correct everything, extend it, and yeah, keep it up to date. But if you would go to k6, you have twice the work, actually. You have your API that’s changing. So you need to update your collection. You need to update your k6 scripts. So if you have five endpoints, that’s doable. But if you have 20 endpoints for one API, that’s a lot of work to keep them in sync, because this is a moving target, but it forces you to keep in sync to other moving targets because you want to be able to experiment and test them, contract, and you also want to test them at scale with the load. And I was looking really for a way, like how can I leverage this nicely prepared Postman collection and use for k6 scripts? And that’s what I, when I ended up with this scenario where you have your API, I update my post collection, I want to generate a k6 script. And I was looking around like, how can I do that? And I found on the k6 website, I found some blog items on, on how that they provided a Postman to k6 converter. And that really triggered me and I started playing around with it. And I really, really liked it. So it gives me all the flexibility of modeling everything in Postman, preparing the whole flow. So if I want to run it in sequence, I could do that. But then it also converted it into a k6 script where I had no maintenance on it, and that was really nice. And when I found the Postman k6 package, like a year ago or something, the purpose of that was to kickstart your k6 experience. And I said, but you can do much more with it. You could even fully automate it with some parameters because the first time I converted my collection, it had like 20 endpoints. I still needed to do add some minor things, I needed to add some k6 specific stuff to it, like how many iterations do I want to do, is there certain parameters I want to add that are not by default converted over. So I said, okay, what can I do to make it even better? And I don’t have to touch the k6 script anymore. So the first time that you converted it, I had to modify it still. So each time that I would do this kind of a behavior, the API changes, I need to update my Postman collection and I needed to update my k6 script. I want to, my ideal world would be, I don’t have to touch the k6 script at all. I just convert it and I was able to run it in one go. And I can show you how it works, because it’s really straightforward in the end.
Yeah.
If it’s okay, you can export your collection–
[Nicole] Yeah, sure.
[Tim] So if you do export, you can just say export and you download a JSON file and you can do the same thing with your own environment. So if you want to have your test environment and you want to also convert them you can because you can also export these from Postman. And what you end up with, I’m gonna close this one, you end up with two JSON files. So as you can see, it’s the k6 office hours. It has some syntax in it, like the test scripts are there, the endpoints are there, the way you connect to them. So all the request information is in this configuration, in the environment file. You can find all your variables that you have defined. So let me just now showcase it. Sorry, I’m gonna, this one. So if you run the Postman-to-k6 converter, it expects actually two things. You define an input, my Postman collection, and you define an output. So if I would run this, just closely watch what happens there, it does some things behind the scenes, and once it’s finished, you would get, actually, if things go well, let’s see if it’s there, you would get this. Yeah, that that’s the goal of CLI. It should be fast. You get the k6 script. I’m gonna go over that later on, and you get this library folder. The library folder, what it is is because the converter, Postman uses number of functions, some methods that are not mapping exactly towards k6. So once this package, it adds some additional functionality that Postman uses. Like it uses low dash and some encryption, stuff like that, and it just adds the libraries there. So if you would do this manually, you would also have to include the libraries, but it does it for you. It includes all the libraries. And it’s actually a wrap around the, let’s go for the SHIM quickly. If it’s not clear what I’m trying to explain, stop me, Nicole.
[Nicole] Yeah, but we can also answer this question from Carlos Ramirez. You can also integrate, well, not a question, a comment. You can also integrate your process using the Postman API key to request information about your collection and environment. And he also adds, oh, no, sorry, this is from Chaitra Nanjesh, this is interesting, because we use k6 in Tesco, says load test, and I think we can try this to see if it works for us. So people are already liking what you’re sharing Tim.
[Tim] Yeah, that’s the goal, That’s also why I liked it actually–
[Nicole] Do you also do what Carlos was suggesting?
[Tim] Yes, and I even go a step further, but that’s, I’m gonna come back to that later. I’m gonna showcase how we do the Full Monty in the end, actually. I’m building up.
[Nicole] Awesome. Sorry, people are excited.
[Tim] Yeah, I’m also excited, so I hope that I’m coming across properly.
[Nicole] You’re doing great, don’t worry.
[Tim] Okay. Just for reference, you have to think of this Postman-to-k6 thing as it’s wrapping Postman functions and k6 functions and combining them in one place. So it’s abstracting away the k6 part and it matches with the Postman stuff. So that’s what it does behind the scenes. So if you see, if you do an execute request, which is an, it’s a function actually, behind the scenes, it just does the HTTP dot request, which is a k6 function. So it wraps a lot of things together, and I think if I would go to execute test, you would also see it uses the k6 check function. Just for reference, this is the wrapping part. This is all not so relevant. You don’t have to care about it. What’s relevant is what’s being generated for you. So it says on top, it’s an autogenerated file here. It does some imports. It imports also k6 related items and it sets default options. By default, it says maximum redirects. And then it does some strange things which are not k6 related, but that’s typically what the package does. That’s the wrap that you see. It defines a collection with a base URL, and then you will see that my group is there, I have my request get all leads. It showcases the URL. It showcases that it does a test here. So everything that you had defined in your Postman collection, sorry… Get all tests and you could see the test here. It’s translated into a k6 kind of syntax and it’s ready to be used. And you see all the requests on here, all ready to be used. It even matches a response codes that you can match to a variable in Postman. You could do anything. The body is there that I inserted that you can create. So if I were now just say k6, sorry, k6 run k6 scripts. Oh, dunno know why this is happening? Give it a second to start up. There you go.
[Nicole] Cool.
[Tim] So I haven’t touched anything, I just took my Postman collection, converted it with the converter that we built a while ago together with k6 and then you can see the tests there. So I created a number of tests. Postman expects to, the response is expected to be 200. I get all the data. My average is my group duration because you also have a group there. So if you have 20 endpoints, you will test 20 endpoints, but I don’t need to write any k6. So I only do my maintenance for, my maintenance only happens in here. I only have to maintain my Postman collection and my k6 gets generated out of them. So that’s really nice thing here. Yeah, that’s the library that we’re using. No, that’s not, that’s not my library actually. I think it’s moved to–
[Paul] Uh-oh.
[Nicole] Hope this is the right one.
[Paul] Yeah, we gotta update our show notes.
[Nicole] Yeah, we do. Oh, with the API deck one. Sorry about that.
[Tim] No worries. I think I made a pool request a while ago, so.
[Nicole] Ah, I see. Okay, I will update the description.
[Tim] But I, just to continue.
[Nicole] Can you also post, Tim, the link?
[Tim] Yeah, I’ll post it. Of course.
[Nicole] Thanks.
[Tim] But it’s really cool. Maybe cool’s not the right word because the package supports a number of things so that you can even tweak the output even further. So by default, I just say, give me the Postman collection and generate a k6 and it would work. But if you want to automate this on a higher level, need to do work for automation, you wanted to tweak some things. Maybe I want to play around with some k6 parameters, or I wanted to do something else, or I want to exclude stuff, or I want to include my environment because, if you see here, there is no, there’s only my base url, but I have a lot of variables I want to use in my Postman collection that I already defined. I have login information. I have some Postman base URL. So you can take it even further because now you just get a script, but it’s just, the script is already addressed. So you can take it step further. So for example, if you would say, I want to increase the number of iterations, you could just say like this. You have this parameter, says iteration 10. I’m not going to convert it. There we go. And if you see my script now has options already set at 10. That’s nice. That’s handy because you want to automate this, of course. And you can even go, take a step further. I want to include my Postman environment variables. So I’m gonna, there’s also a parameter for that. So you can pass some environment variables. So you can even pass some global variables. This is typically Postman, but you can just take the ones that Postman provides, export them and use them. So if I now do this kind of thing, so this is my file with my Postman variables, like I showed. Let’s do this again. And here, you’ll see all my variables that were exposed in my environment are now available for my k6 script to be used in automation workflows. So that’s really nice, but I can even take it a step further. What if I have special k6 related parameters? I’m just gonna copy paste them here for the sake of the demo, but I might want to define my iterations or my maximum redirects because k6 offers a lot of parameter options. Like I want to use cookies. I want to use headers. The number of redirects. I want to set some text. I want to define a timeout. All these k6 related parameters, I want to make sure that my automation flow was able to process them because I want to run the script without touching it. So I want to be able to just say, take these parameters for this pipeline, these parameters are applicable for production. So you could switch environments. For example, what we do is we have different test scenarios defined. So if we go for a small test, we run it like this, we define stages and thresholds. Well, if we want to really do performance testing, we have different ones. So these options, we use them to pass them along to the pipeline. So I could do something again with the generation part.
[Nicole] Question from Sri. Is it one environment at a time only?
[Tim] Yes, because what you generate is what you inject the script. So it, you would then have to generate a second script, but that’s really easy to do because you would just say, I want to have, for example, just showcasing it here, I’m gonna take this script is for the second test stage. You just run it and then you get a second script next to it. So you don’t have to worry about it. You don’t think about one script. You just fork it as much as you need with different configuration options defined that you would use.
[Nicole] Yeah, and Sri, I wanted to also add here that with k6, you can have, you can test more than one environment at a time. What Tim is showing here is that for, because of the way that the that Postman treats these collections separately, every environment is separate. And so the converter also converts them to separate k6 scripts. However, once they’re in k6 script format, once it’s all JavaScript, you could use k6 executors and scenarios to run them simultaneously, if that’s what you’d prefer or anything else.
[Tim] So in this example, what I’m doing, I’m going to pass on my K6 parameters as a JSON file so that I can put them in a Git somewhere. And if I then just say, press go, you will see that it also just updates this option spot on the top. There we go. So you will see, I have defined it to be in this case five virtual users, 10 iterations, maximum redirects, and just adds it on top. So I can switch around in my pipeline without touching anything. I just prepare everything, and then it’s just a matter of defining which command I should to run and it just does it for me. So that’s really handy and it really makes automation a lot easier in that regard. And there’s one thing I really like. It’s a small thing, actually. Now it generates one big file. So everything is in this one big file, all the request, it’s doable if you have like me, for example, four requests or five requests. But if you have like 20 endpoint or 20 requests, it’s a huge file. And to handle that, you can just pass along one parameter. I’m not sure if it’s commonly used a lot, but I’m a big fan of it. It’s called a separate parameter. So what happens now? The options are there. The request configuration is there. But you just get this in your k6 script. I’m not sure which one, I’m gonna remove one because otherwise I might get confused. There we go. You get this, and this is actually, it splits all your requests in a separate folder like you would see them in Postman and each request would be separated from the rest. And that makes it really easy. If you want to test something specifically, if you would have one script, it would run everything. But now I can just comment out, I don’t want to do the create, I don’t want to do the update and delete. I only want to test to get all leads. You just say k6, k6 run k6 script. Give it a second to warm up. There you go. It will only test that request that I defined. So if your 20 endpoints, your k6 script has been generated for you will have 20 lines and all your requests will be nicely separated from each other. So you can even say, I want to tweak it a bit, or you have all flexibility then, while if you don’t use a separate parameter, it just generates one file with everything in it. Do I make sense still?
[Nicole] Yes you do. But there’s a comment here from one of our engineers who wrote that initial blog post that you may be updating, Mostafa said last time you wrote an article about it, it needed a few changes to make things work, so well done.
[Tim] Yeah. Yes, that was one of my first VRs that I did. I wanted to make separate flow over pretty much everything. And there’s one final thing I want to showcase for, with the k6 part, is now you have to really, like all these parameters, they become very long and difficult. You could also just define them as a JSON file and just pass them along. And now you can just define the one file, put the main Git, and then you could just say, I want, this is the output, my environment. You can even define that it uses a handle summary, or you can add request tagging on top of it. And if you do it like that, you get, you just have to do one parameter, one command, actually. So if I do it like this, you see, it just says, this is my input file. I’ve defined an options file, which I referred to. And if I then just do it, it would take all these parameters that I’ve defined and you would get, again, everything nicely generated for you. All the requests are there and it even added the handle summary thing on top of that, which is something–
[Nicole] Oh nice.
[Tim] So everything is there. And so this is the automation flow that’s really powerful because otherwise you would have to add, if you want to have a summary, you have to add a modified file. But now I just define one configuration file, I define my parameters, I define whatever I need. Just run it, and my script, I don’t need to touch it. I can just say, okay, k6 run k6 script. And you would see that it generates, yeah, it generates a test. It does, I think five or 10 iterations. There we go. And it doesn’t open anymore. It generates the JSON file. If I close my, quickly here, you would see a summary report is here. So it just, yeah, there is no manual work anymore to run k6, just keep your equipment collection up to date.
[Paul] And you’re helping folks out with your example here.
[Tim] Yeah, okay, great. Okay, that’s in summary, the things that I want to showcase about the converter. There are some nice things added since the last releases from the k6 team. So I added some, requested and changed some things, and I’m really happy with it. It works flawless at the moment in our organization actually. So I’m really happy with it.
[Nicole] Yeah, it seems really, I think I tried this initially, like when Mostafa was saying, there are a few things that you, you had to get done, but this seems way slicker, and it seems like you’ve really focused on making it automatic so that don’t, there is no manual intervention, which totally makes sense in a CI/CD pipeline.
[Tim] Yes, correct. And I wanted to put my effort on the Postman and the management of the Postman collection, because that was a tool where we were all working on and maintaining it. And I wanted to keep the workload on k6 as low as possible. And I think we succeeded it quite well on that regard. There are some things that are not available out of the box. Like you cannot do the send request. You cannot set next request. But what I did add was, what I’m also really proud of, the dynamic functions from Postman. You could just add any, pretty much any generated stuff. Let’s see if it shows, if it works like that.
[Nicole] There’s also some nice comments for you. There’s Gertjan De Wilde, who says, Go Tim. There’s Pepe Cano as well from k6 chiming in with his support. James is saying nice to see this getting the love it deserves, much better experience. And Charly is saying, Very useful tool, congratulations.
[Tim] Thank you.
[Nicole] Lots of love in the comments for you.
[Paul] Definitely.
[Tim] Okay. But honesty, we took it a even a level further, but maybe I should, maybe there are, if there are other questions we should talk about or not, I’m not sure.
[Nicole] I’m interested in this. So what were you saying, the dynamic part that you added parameters?
[Tim] Let’s see.
[Nicole] I mean, you also don’t have to demonstrate everything. Can we just like acknowledge the fact that Tim who doesn’t even work for k6 or Postman volunteered to do like a live demo about this. So that’s awesome. Don’t feel like you have to show us everything. You can just talk about it.
Yeah, Postman really provides this nice, what they call dynamic variables. And so it’s, this is a screenshot of it. So if you use curly brackets, it’s random, and then you can say, I want the full name, I want the bank account or anything that uses Faker behind the scenes, and you can make dummy data out of it. So that’s really handy if you want to really make, yeah, realistic data without having to use for loops or anything, just put in the variable. And that was not fully supported in Postman to k6. So we brought in that support as well to make sure that you could have the full functionality of the dynamic variables attached to that. So yeah, there are a number of things we did since the last release of it, and I think dynamic variables was in much demand at that moment. There you go. Yeah, and I’m really happy for Mostafa, I’m not sure how to pronounce it correctly, because I think he was one of the first ones who pointed out in his blog post about functionality. So thank you for pointing it out to me.
Yeah, now that I think about it, I should have asked Mostafa to be here too. Oh, I missed that opportunity. Sorry about that Mostafa. That’s so good about the dynamic variables, because that’s one of the things that you have to think about when you shift from just a purely automization mode, like automated testing mode to load testing is that you can’t, you then have to start thinking about the data that you’re sending too, because if you’re trying to get realistic response times it could trigger caching on the service side if you’re just requesting the same thing. So yeah, totally makes sense. k6 can already, you can already import faker with k6, so great, great job for putting that together.
Yeah, no, that’s very cool though, but yeah, it’s like, you can use Postman for your user, your UI, your, you know, your development environment and do all this stuff and then even goes through there and generates some random data. So, yeah, that’s cool. Very cool.
Oh, go on.
But maybe I’m gonna talk first before I want to show the final piece of what we did.
Well, just, just quickly, I just wanted to ask about the limitations of this approach that you see.
Not all functions of Postman are supported. So that’s the downside of it actually. But for the use is that that we are using it, we have everything covered. Like you have the, the dynamic variables, you have the environment variables, you can define k6 parameters. Yeah, there was nothing stopping us. I think if you want to do complex scenarios and you want to do if-else situations, there it’s not suited for. That’s the same problem you would have in Postman. It would be difficult to make these if-else conditions in Postman as well. So that’s, if you do simple tests, functional tests in Postman, you can really easily port them to k6 and then you have performance tests end points.
And in the spirit of the whole free and open software. I mean, do you have issues where like you’re looking for help? Is there any gaps that you’re still looking to fill?
There are some bugs I cannot reproduce at the moment, something with XML conversions, but right now it’s feels quite feature complete. I haven’t added that much, actually. The base was already there by the k6 team members who built the initial library. So I just added some sugar coating on top to make the automation flows much smoother and less manual work, or linked actually. But if people have great ideas, I think they should just make a PR and add it to the package.
Yeah, did you want to show us something else?
Yes, I want to show how we ran a step further even, because yeah, if you update your API and then you have to update your Postman collection, it then generates a nice k6. But we also use openAPI as a documentation tool where we document and design our API. So every change that you do on your API, you need to add them to your API, and then you, so you still have to maintain not one output, the Postman, but you also have to update your documentation. And I want to flip that around. I want to make the API documentation leading, generate the Postman collection out of it and generate out of the Postman collection, a k6 suite. So that as a product manager, I don’t have to maintain all these documents anymore. I just maintain one thing, and that’s my openAPI and my team or our team then only just need to develop the application itself. And it’s called spec driven approach. It’s nothing new, but I’m a really big fan of it and I’m a really, yeah, we introduced it a year ago and it solved so much and it brought so much quality on the company level, in our documentation, in our test suites. So that’s something I want to share as well, because it’s so valuable once you get how it works and if the right tools and properties, you can really do it. And this is actually where we are right now. I can also demo how it works. So we have, I’m managing an openAPI documentation. We use the formatting and we validate it if it’s valid. And then we use a generated one. And from the generated one, we generate documentation. So public documentation, stop lights, just like whatever you call it, we generate a Postman collection that we can share with the customers and we run contract tests, we run soak tests and we run smoke tests all based on this openAPI documentation. So, and again, there’s no no contact or no changes involved. We just, we managed the spec on its own and it’s really, really something powerful and useful. And I can quickly show how it would work actually if you’re interested.
[Nicole] Yes, but before you do, a wild Mostafa appears.
Hello.
How awesome is that? Let’s just say hello to Mostafa, who just joined last minute since he was already in the comments anyway.
Nice to meet you. And I’m grateful that you wrote the blog post so many, so many months or years ago, so thank you for that.
Yeah, yeah, I really felt the need that we needed this article, so that’s why I wrote it in the first place. And then, yeah, I found a lot of issues that you solved, and thank you, it was a great job you did, and well done.
Thank you. And thank you for putting out the 90% of it. I said I only had to put in the sugar on the top of it and the cherry on the top.
No, no, no, no, that’s not true, that’s not true.
Mostafa–
In the first place just to clarify.
Yeah, Mostafa while you’re here can you quickly introduce who you are and why we’ve randomly invited you onto the stream?
Of course. Okay, so I am a backend developer at k6. I’m almost three years with k6, like it was called Load Impact before being renamed completely to k6, and now Grafana. I’ve been a developer advocate when the first like idea came into place that we want to have a developer advocacy program. And yeah, I’ve been a developer advocate for a year and then came back to back it. So I worked with lots of these tools and tried to help others because like, there were no other resources available. Like there was this read-me on the project and that was it. No mention of it anywhere. Only the hidden knowledge of everyone inside the company of how to use this tool, but nobody knew without outside the company. So tried to learn it from others, specifically, Pepe, thanks for that, from here. And I tried to write that blog post and try to share the knowledge. So that’s basically it.
Great, so we also got a comment: Great with the diagram. Can you share this?
Of course I can. Yeah, I’ll do that.
All right, I guess that’s our cue to head back. Please continue, Tim.
Oh, I’m not gonna, I’m just gonna showcase it, and then I hope everybody will really appreciate and see the power that it brings actually. So, okay, so maybe start with the beginning. I’m gonna close my screens a bit to clean it up. So I have this openAPI spec that I, it’s just a copy of what we just described. So this is how it looks, this is documentation using Docly. You can see it’s the k6 office hours CRM application. You can see the list in there. You can see the create, the get. So it’s just the definition of what the API would look like. This is the documentation part of it. So I’m using now two tools. I’m gonna use Portman as another package that I built together with API Deck and make an API Deck where we can convert the openAPI to Postman, and from the Postman, you can generate a k6 one. And that’s really nice because you only need two commands, one source file, and you have two outputs out of the box. So that’s really powerful. I find it powerful at least. So I’m gonna take my commands correctly.
So If it’s okay, can you, I’m not, I’m not up to speed with Portman. So it creates a Postman collection already given openAPI, is something in openAPI specification?
Yes.
That’s awesome.
There was, Postman built a patch like Mostafa, k6 build a patch where you can convert from Postman to k6, and Postman build a Postman where you can convert from openAPI to Postman. And when I found out, I said, that’s amazing. You don’t have to maintain a Postman collection anymore, but it wasn’t, it was missing some functionality, and the major thing that I was missing, it was just creating the request than Postman, but not the tests. It was not handling the tests of it. It was not assigning variables. It was just taking the Postman request and converting them into openAPI requests, into Postman. So you still needed to maintain and modify them. And I said, it has to be, because an openAPI documentation or specification, it describes everything, the request type, the body, the properties, if you do it properly, you have all, everything in one go. And I said, why not use that to create a test suite, update your variables, do all the dynamic stuff you would expect it to come out of the box. And that’s why we created Portman. It started as a PR on that openAPI to Postman package, but then I met Nick from openAPI, from API Deck, sorry. And then we came together and we said, let’s collaborate, and let’s make a startup package out of it that you can plug in and use it to automate these kind of workflows. And it’s really, yeah, I’m biased again, but I really like it.
No, it sounds awesome.
A question here. May I ask a question here? So have you seen the openAPI generator package that I’ve written?
Yes, I saw that and that I had the same comments as I had initially. It generates a nice k6 script, but it it’s usually you still have to manually adapt a lot of things and correct it.
Yeah, but sorry that I interrupted you. There’s this latest PR point? I think Michael, if I’m currently remembering his name, so Michael added a feature to, to be able to add certain extension to the, to your Swagger or openAPI collection so that it generates everything for you. And if you add examples to your API output, to your responses, then it can extract those examples and put it in your script so you don’t need to modify anything. So you can create basically the same thing you did for Postman, you can do it for the openAPI. So the new revision, the new PR and the new version that is released by Ophelia, the maintainer of the openAPI project helps you create a complete test from start to finish with no modification.
Yeah, I think that’s great, but we do two types, three types of tests. We will do contract tests, and then we use Newman for it. And we want to use k6 because Newman, it can do schema validation, it can do a lot, I think a lot more than k6 can right now. I think it’s also the goal for k6 to do all these contract validation. It can, I guess, but I treat them differently. I think the Newman Postman has more contract testing capabilities, while I think k6 is perfect for performance and load testing at scale, where you only focus on what’s the response, is it 200 or is it not a 200, at least that’s how I use it actually.
That’s fair enough.
Yeah, what Portman does it converts openAPI to Postman. So I’m gonna showcase that quickly. So it just took the, this openAPI that’s here now, and then gives a while to spin it up. There we go. It’s converting it. And if things go well… Let me check something happening here. Here it is. So here’s my Postman collection. It’s just generated it. It does everything for you. So it creates, recreate, so there wasn’t a collection there. It just recreated for you. And I hope things, it’s not showcasing it here. That’s bad, too bad. So it just generates and if I would import in Postman, let’s quickly do that… Give me a second to find everything. There we go. So this is the k6 and it’s generated by Portman. There we go. If you see you have the same endpoints–
[Nicole] Oh, nice.
[Tim] So the first thing it’s great because you don’t have to recreate your Postman collection. You just, you just run it from your openAPI specification and you get, as a result, you get your requests pre-tailed already, also your post exercise is ready. So you can see here, it inserts a body, everything, I don’t have to do anything. It just takes your openAPI and converts it to Postman. Plus it adds a lot of tests already. You see, it does contract testing by design. So it includes automatically 200 contact applications. So it just injects your contract test in there. And that’s really nice because I only need to maintain one document, my openAPI specification, as a result, I get all my requests in Postman, and I get all my tests in Postman without writing any code. I just need to maintain it in the openAPI documentation.
Wow this is really awesome, Tim.
So keep in mind that this was the flow before you had to create your Postman collection and add the tests, and then you could generate k6. Now we turn it around. Now we generate the Postman and request, and then we can use that to generate, okay, six scripts. And so we only need to maintain this part in, yeah, or you know, that’s the only thing I need to maintain as a product manager and the team, of course, they built the API and the service and everything, but I get this and I can focus only on this. And this is easy to manage because it’s a YAML, you can play around, you can add endpoints, you can remove endpoints and just run it. A minute later, I have a Postman collection to play with it and I can run it at scale with k6, doing the volume tests. What else can I showcase quickly here? Let me see if I can. I just–
[Nicole] Some more nice comments here. Such an awesome flow, says Gertjan, and James says his mind is blown and, and I join him on that sentiment right now.
[Mostafa] Exactly. That’s a full test suite, like from start to finish.
Yeah, and like example, schema validation. It comes out of the box. You don’t have to add it. It just does it for you. And the richer you make a documentation, the richer the test suite will become. And it also, Portman, be extended, you can also do variations. So now it just validates if it’s a 200, but you can also have it generate 404s, and then it will remove items from it. If it’s a required property, it’ll create a variation out of it. And then you can test I’d the 404s are not present. And so you get two test suites in one go, actually.
Wow. That is, that’s so awesome. Maybe we can like, try and summarize this as we try to collectively recapture our blown minds. So I think that there are two different tools here, Postman and k6, which are really for different things. The Postman is really better for API testing, and as you mentioned, Tim ,contract testing, and k6 is better for load testing. And their features are pretty divergent after a certain point, but they do have a commonality, and it seems like that’s what you’ve really capitalized on. Mostafa and Tim both have worked on this. and you you’re trying to make, you’ve gotten us closer to this ideal of having just one set of test suites, and it doesn’t really matter whether it’s automation test suites or load test suites or, it can just be the one thing and you no longer have to maintain two separate ones. And that’s having like the amount of time that you need to set it up, but also running it. Because having to explain this to somebody else is also a huge deal when you’re not the only tester on your team.
Nope, exactly.
And the nice thing is as a product manager, I’m not technically, so but I can contribute to it. I can, I just need to maintain the openAPI specification and the whole team benefits from anything that I do. If we need to add a new endpoint, I can design it. Not even five minutes later, one minute later, I can just run it in a pipeline, see if it works, and that’s a bit, the cycle of spec driven development. You put your spec at the center, the spec at the center, and then you design it, simulate it, fill it, Then you build it, you test it. But the spec is a test driver here and then you can even run it on production and yeah, do the management aspect as well, because then you can monitor it as well based on your API specification. So that’s, as a product manager, I’m really proud that we pulled this off because that was one of the ambitions when we wanted a newer, or API program, like how can we use a manual workload, what can we automate, and yeah, I’m really happy because it’s all open source based. There is no other software involved. We just use k6. It has excellent dashboards to individualize reports. We use Postman to generate contract testing and manage it, play around with it. And we benefit from such large ecosystem , Swagger as documentation interface, where you can display or openAPI as well. So it’s almost full cycle or I don’t know what’s the term for it, but yeah, I really, I’m really happy with where we are right now. It’s a good place to be in an API landscape actually, especially from, yeah–
You’ve taken it a step further, right? This is no longer just about test suites. It’s also because you are building on the openAPI specifications, it’s all connected even from not just the development part, but the design part. And Pepe actually has a question about this. He says, can you show us more about the process that updates the APIs? What does it do?
How do you mean? This flow, this one?
Sorry, which one? Yes, I believe so.
The powerful thing about this can also be because if you have an openAPI specification, you can even generate clients or service side code from it so that in the end, your API service can also just be generated from an API specification. There is a really nice openAPI, there’s a really nice openAPI open source project, it’s called I think openAPI Generator, and then you can just run a C# library or an API for that. You just take your openAPI documentation and you run, you run it and you get as a result, you get C# code that you can run in production. And that’s also what we use in our company. You have some teams that just manage to build the openAPI specification and they use that to generate C# code and have APIs up and running in no time. I hope that this is the answering this question.
Yeah, Mostafa, Paul, anything you want to say to Tim.
Yeah, no questions but, yeah, that would come later, but I need to definitely pass this along, this video along to some people that I used to work with, because this will blow their minds as well. This is something that they definitely need.
Yeah.
Do everyone.
Great job, great job.
Yeah, great job, exactly. I’m literally speechless because I’ve worked with all these tools and I know all the–
Clearly not literally.
Well, not the ones that Tim wrote, the ones that we had before. But anyhow, this is a great addition to this mix of open source software that was out there and that we used, and this is basically end-to-end testing. So based on a YAML file, which is an openAPI specification, you can generate tests for your API. You can generate load tests. You can generate, I don’t know, like literally everything.
It’s also documentation. You get documentation, nice looking documentation out the box, a playground with Postman and you get the numbers to back it up with even graphs from k6. I really like your dashboarding. So if you have a customer and they they’re asking about SLA, we just pull up some graphs from k6 and everybody’s happy with that. So we, the full ecosystem has really matured a lot over the last couple of years.
Bravo.
Yeah, this is so freaking awesome. Thank you so much, Tim, for coming on. By the way, he’s not even working for k6. He’s just, and I think you said at one point, oh, I’m a product manager. I’m not technical. I beg to differ. This is pretty amazing things that you’ve put together. There’s a bunch of comments here too. James says he’s literally waiting for the livestream to end, so he can link to the video with a timestamp. Well, let’s just try and drag it on for him, shall we? Thanh says plus a thousand votes, and people are also saying thank you, and outstanding, almost the holy grail. So lots of fans in comments. Thank you so much, Tim, for joining us. I know we’re a bit over time. I’m kind of taken aback by how much value you just showed us. This is amazing, I think we need to have you back.
Yeah, I’m gonna write a blog post for k6 and one for API Deck where I’m going to get more into the details because yeah, it’s a show now, it’s a difficult to but I’m gonna try to write it up and then extend a bit on what, sorry.
Mostafa.
Mostafa, apologies, I’m really bad.
It’s fine, it’s fine, no worries.
Yeah, I think can be really powerful to share with the rest of the world that they can, yeah, maybe use some parts of it or not, but I benefit a lot from it. The whole team benefits a lot from it, actually. I can showcase, this is a pipeline we use, for example. So this is we do tests, then we run Postman and we do tests and if it’s 100%, it gets deployed. So it’s not just a gimmick, me showcasing it. This is really happening in our company. So we use it to build it. We use it to have nice dashboard with k6 attached to that. So yeah, I’m grateful that you put it out there for us to benefit from it.
Thank you.
That’s so great, thank you.
Yeah, I–
Go on, Mostafa..
Review the article that you want to write.
Okay, I’ll keep Pepe in the loop and then you’ll reach out.
Of course, of course.
Great, thank you. Thank you as well, Paul and Mostafa, for coming on, Mostafa, even though, sorry, I should have invited you even earlier. Thank you everybody for watching. Thanks everyone in the chat. Have a great weekend, everybody.
Absolutely.
Thank you for having me.
Yeah, thank you.
Yeah, all right, see you all.
Bye.

Top 10 Obsidian Plugins I Can't Do Without

Wed, 23 Feb 2022 17:00:08 +0100

In this video, I talk about my experience in getting started with Obsidian, and how I went from thinking it wouldn’t meet my needs at all to not being able to shut up about how awesome it is.

Transcript

The first time I tried to use Obsidian, I didn’t like it. It didn’t seem intuitive to use. I couldn’t quite get it to do what I wanted. And frankly, I just didn’t get all of the hype.

Today, I have over 10,000 notes over multiple vaults in Obsidian.

So what changed? Well, it turns out that there’s a pretty crucial element of the experience that I was missing the first time around and that’s plugins. Here are my top 10 community plugins for Obsidian so that you don’t make the same mistake. \

Obsidian already comes with some core plugins, some of which are enabled by default. Today, I’m talking about community plugins, which are extensions to Obsidian that are created by other Obsidian users. The security of your notes is still on you though so think about what you’re comfortable with before installing anything. \

Obsidian Calendar and Fantasy Calendar

These 10 plugins are arranged alphabetically and the first one is Calendar.

Calendar is the most popular Obsidian plugin and for a reason. I don’t know at this point why you would ever install Obsidian without the Calendar plugin because Obsidian doesn’t have a native implementation of a calendar. The Calendar plugin doesn’t just show the calendar. It also lets you create daily notes and see those daily notes over the entire month. And it also lets you create weekly reviews if you have that option enabled, which I do.

Recently, I’ve been using Calendar and this other plugin, Fantasy Calendar. Even though Fantasy Calendar definitely was made for use in a tabletop role playing game like Dungeons and Dragons, you don’t have use it as that. I’ve used Fantasy Calendar as my content calendar in Obsidian. It just provides a little bit more customization than the Calendar plugin can but I install and use both heavily.

Obsidian Dataview

Number two is Dataview.

Dataview has changed the way I use Obsidian. It’s changed the way I format my notes. It essentially allows you to define parameters and values in the YAML front matter or the metadata of your notes and then it lets you search for them. So it essentially lets you create your own database in Obsidian with custom fields for anything that you want.

You could use it for meetings or people or tools or books that you’ve read. The possibilities are endless. By the way, in addition to the very SQL-like query language that they have, you can also use JavaScript in Dataview. There’s just so much to do with this plugin. And I really haven’t even scratched the surface of it.

Obsidian Editor Syntax Highlight

Number three is so small but so useful that I had to include it in the list. It is Editor Syntax Highlight.

If you’ve ever copied over any code into Obsidian, you may find that it lacks a little bit in readability. Editors Syntax Highlight dramatically improves readability by changing the syntax highlighting depending on the programming language that you select. It isn’t going to replace VS Code anytime soon, but if you’re like me and you use it sort of as a dev log while you’re trying figure something out and you include your final solution or the code into your note, then I think that this is a must-have.

Obsidian Kanban

Number four is Kanban which is funny because I don’t even usually like the Kanban approach. I never understood Trello and I never liked Post-it Notes. I don’t use Kanban for task management.

I’ve discovered what it’s really good for is a content calendar. I create a card in Kanban for every bit of content that I’d like to work on, whether that’s code or a section in a workshop, a blog post, a video, or really basically anything. I really like being able to create my own columns and spicing it up by embedding photos within the card itself. It just makes the whole process a lot more visual.

Obsidian Outliner

Number five is Outliner. Before Obsidian, I was using Roam and it definitely shows. While I’ve moved on philosophically from Roam Research, I still love the very task-oriented bulletin format that it had.

Well, Outliner brings the same thing to Obsidian. So you can have your keyboard shortcuts for quickly moving something up and down a list or indenting and unindenting as well. It has made my transition from Roam Research significantly easier.

Obsidian Periodic Notes

Number six is Periodic Notes.

I think one of the most fundamental philosophical differences between Obsidian and tools like Evernote is that Obsidian doesn’t want you to just capture it all. Obsidian is about bringing those notes to the forefront, processing them, analyzing them, really learning them, and then changing them as your knowledge changes and Periodic Notes really helps with that.

It is particularly useful for OKR or objective setting because it provides a system for reviewing notes daily, weekly, monthly, quarterly, and yearly. You don’t have to use all of those, but I find that using some of them is really useful in helping me step back and think about what I actually want to accomplish.

Obsidian Publish

Number seven probably would be my number one if I were ranking these, but since it’s alphabetical, number seven is Obsidian Publish.

Obsidian Publish is a premium add-on to Obsidian that lets you instantly publish all of your notes or at least the notes that you select to be published to a certain domain. Now I have it going to a custom domain and yes, it’s not free. And yes, I do know that there are other ways that you could do this with markdown files. I have my personal blog using Hugo as a static generator for instance, but there’s just something about Publish and just it taking just a few clicks between writing something and putting it out there that really reduces the inertia for me to learn in public.

Learning in public and growing a digital garden that is constantly changing is really important to me and it has been essential to my professional and personal development. So Obsidian Publish has to be, hands-down, my favorite plugin for Obsidian.

Obsidian Readwise Official

Number eight is the Readwise Official plugin and this is a plugin that I’ve been wanting for a very long time. In fact, before the official plugin came out, I created my own Python script that I was running regularly just to be able to move my things from Readwise to Obsidian.

Readwise, if you don’t know, is a way to collect all of the things that you’re doing on the internet. It’s an easy way to collect eBooks, web articles, audiobooks, tweets, and even PDFs. And it all brings your highlights of those into your Obsidian vault.

It is a really essential part to my learning process because it ensures that everything I’ve learned from a variety of different sources ends up in my Obsidian vault for me to process and analyze.

Obsidian Sync

Number nine is Obsidian Sync and it is also a premium add-on, kind of like Obsidian Publish, but it’s also totally optional.

You don’t really need Obsidian Sync. I already use Dropbox, but unfortunately, one of the problems with Dropbox is that you can’t yet use it to sync between different devices. I use Obsidian on my laptop, on my iPad, and on my Samsung mobile. So I have three different devices that I need all of my notes on.

There are currently two options for this. It’s either iCloud or Obsidian Sync. I don’t like iCloud, and so, Obsidian Sync is my service of choice.

Obsidian Templater

Number 10 is the most recent plugin of this 10 and it is Templater.

Now don’t get it confused with Templates which is an Obsidian core plugin. TemplateR is a community plugin that’s like Templates on steroids. It extends templating functionality by letting you run JavaScript in your vault. This blows the functionality of Obsidian wide open as now I can run JavaScript in my vault to do certain things for me.

I was already a fan of the core plugin Templates, so having Templater and all of the customization options that are possible is just amazing. Definitely recommend it.

Conclusion

If you’ve made it this far and you’re wondering what the heck is Obsidian, well maybe take a step back and watch this video on how to get started with Obsidian. And if you want to know about my usage of any of the particular plugins that I mentioned, leave me a comment below so that I know that I should make a video on that next.

Thank you for watching. Happy note taking!

How to design a workload model in k6, with the k6 Developer Advocates (k6 Office Hours #42)

Fri, 18 Feb 2022 18:20:21 +0100

On this week’s Office Hours, Paul Balogh and Leandro Melendez join me to talk about how to design workload models in k6 load testing scripts using scenarios and executors.

Transcript

Hello everyone, and welcome to another k6 office hours. I am Nicole van der Hoeven, and for the first time, all three of us developer advocates are in one stream. Could you please introduce yourselves?
Oh, who’s going first?
I guess you are.
All right, age before beauty, as they say. Okay, well, hey, I’m Paul Balogh, the newest DevRel here at k6, nice to meet you.
And on my end Leandro Melendez, also known as a Senor Performo, very happy to contribute here and bringing, usually it’s knowledge, but this time I’m gonna be a little bit on the learning side, which is super exciting.
We learn together.
Oh yeah.
Yeah, I mean, I don’t think you can really have one without the other. I think it’s super valuable to also just ask questions.
And Nicole, I’m a little disappointed that you don’t have a fake mustache to support Leandro and I.
Yeah. That is not something that I just have lying around, unfortunately.
We may have to fix that. I was commenting the equivalent here in Mexico is the unibrow.
I could potentially draw something on my face, maybe, I could try that.
Is that the Frida Kahlo?
Oh yeah.
So, Leandro and Paul are relatively new to k6. I think you both have been here for less than a month.
Just a month here.
Oh, just a month now, that’s awesome.
Three.
But this is the first time, I’ve been here with you with each of you separately, but this is the first time that all three of us are here. It is pretty cool, welcome.
Exciting, hopefully it’s not overload for everybody watching.
Yeah. One thing that I wanted to get out of the way is that we have a very awesome community, we have several community champions, but one of them, NaveenKumar, just posted a couple of videos on how to use checks and thresholds in k6. So I’m going to put that in the comments, as well, because I thought that they were pretty awesome videos. NaveenKumar has a YouTube channel where he goes through a bunch of different tools, including k6, but also like JMeter and Gatling, and he always does a good job, so check those out. Oops, I actually didn’t post his videos, I think I posted the documentation for what we were talking about, so that’s a little premature. Sorry about that, let me put the actual links this time.
Well, this is a live podcast.
Yeah, obviously. Sorry, you get what you get. So if that hadn’t already tipped you off, what we actually want to about today are scenarios and executors in k6. You may have been confused by the thumbnail because it was like, I realized afterwards, that it was the three of us, our photos, and then the words executors in k6.
Here come the executors, the three of us.
I guess that’s our new name or something.
You know, if you invert the k6 logo, it almost looks like a guillotine, so maybe we are the executors.
Oh, okay.
I had not noticed that.
That’s a little morbid.
Oh, you know, hey, I do what I can.
You’re going like play Beatles songs backwards and all that type of stuff, man. I’m sure it was not on purpose.
It looks like we’ve lost access to LinkedIn for some reason, but, oh well, this has happened before. Thank you, Mark, for saying we’re killer performance engineers, love the pun. So the reason that I wanted to talk about this was that, I don’t know if you also have got the same, Leandro, Leandro and I were both in the Automation Guild 2022. We both had a talk there and there are a lot of common questions that come up with regards to load testing. And a lot of it has to do with the results, specifically when, I’m not sure if you can actually hear the wind in my background.
Is that you, oh wow.
Are you okay?
I’m okay, but apparently there’s a storm that’s in Northern Europe.
I was gonna say, hopefully you’ll still be in Kansas, right?
I know. It’s like 140 kilometer per hour winds.
Don’t end up in Kansas.
But can I be the straw man, at least, in this Oz thing?
Sure.
I always liked him.
I guess I’m Dorothy. Or the wizard.
I ask Toto, please.
I’ve got Toto around here somewhere, let her in.
So one question that I was getting a lot, or I saw other people talking about variations of it, was the situation where you might have already run your load test and seen all check marks and everything passed, and the response times were within the expected results, and yet when the application goes live, suddenly there’s all these errors, and performance issues, and maybe production incidents that you never saw in any of your tests. And the question is, how can that be when you’ve done your testing and you didn’t find any of that and you only see it in production?
Yes, of course I experienced that several times, but many things come to mind on, in what environment did you test? What happened in this environment? If you test in production, hopefully ‘cause we are modern performance engineers that are not afraid of testing in production. Settings, changes, what was the workload? I mean, I don’t know, the list is pretty long.
Yeah, there’s a lot that, that could, that’s why that question is hard to answer because there’s no one answer. It’s like, oh and that happens, always check this. And that’s also because it could be that there’s something different with the environment. Like maybe it’s you didn’t test the same version of the code even, or maybe you were testing in not a production-like environment in terms of size, or test data, or something like that. But another reason that I think is easier because we testers normally control it is on the side of the script. It could be that you weren’t testing the things that you actually wanted to test. And a big part of that, I think, is workload modeling.
That’s one that I also wanted to comment, ‘cause that has been a pain in different occasions. On one hand, watch model, ‘cause from what you’re saying, there were these projects where in the end we were told, hey, this is a workload, this is a utilization that we are expecting to have in production or that we actually have in production, we want you to test for this. And it was nothing like that. We were told, yeah, it’s a thousand clicks per day, but you wanna test an emergency center when there’s a tornado. How is that true, how is that relevant, or vice versa? For some reason, our scripts are just simulating a thousand clicks per day where it’s 10,000 billions, gazillions, what would’ve been the right thing. Of course your performance test is gonna be awesome if you underdid it. And the other side of the risk, everyone running and, like, the system is dying, what is happening? Oh, we’re pressing it too hard, sorry, our bad.
Yeah, that’s true, could go both ways.
But definitely you always have to worry about configuration drift anyway. Differences in your environments, unless you’re using something like a terraform or something, infrastructure is code where you can make sure that your environments are equal. So if you do your load testing and performance testing in a lower environment or isolated environment that it gets some of that, that it is truly configured the same.
Yeah, even when the environments are exactly identical, though, let’s just say that it’s possible to say that, yeah, there are still situations where if your load tests weren’t simulating realistic enough load, then you can’t really, you may not be able to apply those results to production either.
And one thing, as I said earlier, on one hand, the problem to getting those realistic numbers, at times, the team doesn’t even know, they have no idea what are you asking for. All they know is we want to be able to handle 10 users in the system. I’m kind of simplifying it, usually it’s 10 billion users or something like that, right. Okay, 10 billion users, what does that translate into in terms of utilization? And they go like utiliti-what? And that’s the first struggle, right. Supposing that you get the right numbers that you know exactly how much to stress each thing, ‘cause there there’s another differentiator, right. We get this action a thousand times, okay, but to get to that action at times or on some systems that are the same, you have to do others many more times, less times, and they are just telling you about this one. What is the proportion, how much to stress the other one that the other one does need not that many times? So that’s one discussion. If you get the numbers right then to landing it into the script and your scenarios, also, it’s a very interesting situation, headache-bringing situation at times.
I think part of the problem is that a lot of times, when we think about load testing, there are only a few parameters that we think about immediately. One them is the number of virtual users. Another one is the test duration. And sometimes you might get like a request per second, like you were saying, Leandro, but those three don’t really describe the shape of a load. It’s kind of like trying to encapsulate test results into an average figure, like when you get the geometric mean of something, it’s not a great way to express the distribution. You know, if you say that the average response time is three seconds, that doesn’t mean that you don’t have really high response times. Like you could still have a handful of requests that are at 15 seconds, but maybe the rest of them are a hundred milliseconds. It doesn’t really
There is context.
Accurately describe the whole context, the big picture of the load. And in the same way, things like the number of virtual users, and duration, and requests per second also don’t show the entire picture.
Yeah, wen you’re describing it that way, it makes me think when you are asked to describe someone, oh, has beautiful glasses. I mean, I was talking about Paul, sorry, Nicole.
Oh, sorry.
No, no, I like your glasses, too. But you need a full picture, right? There are so many elements that a scenario or a workload model includes, so many variations, details that, yeah, as you were saying, I need 10 million users in the system doing what, how often, how long, how many of them are doing which ones? And many of those things, you have to get into the weeds to truly understand and design ‘cause this is another one, you will want different workloads for different type of load scenarios. You should not, I was gonna say you cannot, but you can try, but it’s not recommended at all to try to kill too many birds with a single stone. You may be able to get two, but more than that gets complicated, gets crazy. So you have to create a work model for that specific risk situation, that scenario that you want to test for. Just one at a time gets challenging, getting all those components, all those items and things that you need to be careful putting together to model and simulate something that resembles realistically that risk scenario, that risk circumstance that you want to test for.
And here you go again with that killer mentality there, killing multiple birds with one stone. And now Mark’s making comments here that actually for killers.
Okay, okay, let’s say you may be looking for a needle into multiple haystacks, no killing.
There we go.
I was thinking about what you were saying about the hypothetical scenario of a thousand requests over 24 hours. There’s still so many ways to interpret that, right? Because it could be, one interpretation could be, take a thousand requests divided by 24 hours and that’s the amount of load that you want to test with a one-hour load test. It could be like that, but if it’s emergency services, there are certain times of the day when emergencies are more likely. It could be seasonal, as well, like the incidents of emergencies of ambulances that are sent out increases over the holidays for inexplicable reasons. That kind of thing does give more context than just a thousand over 24 hours. I mean, if you just look at a period of 24 hours, those thousand could have been simulated, could have happened in one hour for all you know.
Step in a little far from the ambulance example, don’t wanna go more on the killer topic. Let’s think of a restaurant, if they say, this Golden Arches restaurant, we sell a thousand burgers per day. Okay. Most probably you are selling 80% of that at lunchtime and the rest of the hours we don’t care about. So those different behaviors during that period of time is what we should focus on, or care for, or have better information. Well, in the morning you get more, what’s the name of the thing, hash Browns, no burgers. Even if you are the same restaurant, which one do you want to simulate? What is the utilization at different periods in time during the day? ‘Cause yeah, probably no one, not that many people get a milkshake in the morning, but most of them getting in the night. So some of those variations, you want to make sure that you are doing the right mix, not only how many at a single number per long period of time, what happens each hour of the day, which ones do you care for. As a restaurant, I would say I don’t wanna simulate a load test of, I don’t know, after lunchtime where everyone is gone and we are super quiet. I don’t care load test-wise, right. But lunch time, breakfast time, I don’t know, 7:00 AM in a gym, I want to test those type of things of what people are doing and focus on, get a good work load model for what is happening at those times, right.
So I thought, Mark has a couple comments here. I think also the number of vusers or threads compared to the target arrival rate or expected response time for me is the health of the workload generation. And he follows up with, also synchronizing or rendezvous vuser requests is underused in my opinion. Do you agree, Leandro.
Well, it’s in my personal experience, there have been a few times, at least on the render view or synchronizing part, there are not that many times where I had to leverage those functionalities. I think it kind of depends on the projects that you work on. One that comes to mind, and I really love this example for workload modeling in general, is I worked at a gaming company, Australia’s largest gaming company, and the project that I worked on was for the Melbourne Cup. So they sold tickets for the Melbourne Cup. And if you don’t know, and probably you don’t unless you’re Australian, but the Melbourne Cup is a horse race. It is like a two-minute horse race that is for some reason incredibly popular in Australia, in my state, in Victoria, ‘cause I lived in Melbourne, we actually got a public holiday for it. So, it’s that big. And for some companies, for a lot of these betting companies, the revenue from that one race could be up to 80% of the entire year’s revenue. So it is that big.
Just in that two minutes.
Yeah, and that’s the thing. Betting is open until the race starts. And yet when you look at the workload model for that, it really, the load spikes, the betting spikes just in the few seconds before the race. And that’s for a number of reasons, people might not have had the chance to bet earlier in the week because they were working or something. And if you have a public holiday, then you do have the time to just be there watching. And also there are, depending on the types of bet that you placed, there could be a difference in odds. So they’re trying to get the last possible information that they can before they have to make a decision about which horse to back. And that’s a good case for a rendezvous because there is a clear cutoff and sometimes you see the reverse, like for example, in a university portal or one of these shopping sales, where at midnight all the sales go live. And so you get people maybe refreshing that page all up to that point and then, bam, all of a sudden they start checking out things. And that would be, I think, a good time to use rendezvous.
No, and in a quick example, you got me thinking, as well, video gaming, when you are on these multiplayer games, most probably everyone watching this are familiar, campaigns, online campaigns, certainly there are so many things happening, there are people playing other things, but a campaign is gonna start at this moment, this schedule. I think there a rendezvous would be a very good use of those type of things. Sadly, in my experience, some of those situations were solved with a spike scenario run out here with an average day in production, so overlapping scenarios. But that’s interesting ‘cause, as well, it depends on how do you want to tackle that problem. We are going to have a normal day and suddenly something’s gonna happen. How do we simulate it? How do we model that workload? What tools do we use? That’s a very good point, and again, with performance testing and load testing, there’s no one size fit all or the way to do everything. No, it depends, and you can get two very similar loads, simulations, workloads with different approaches, with different things. And that’s one that I also wanted to mention, like when modeling, it’s something that when you wanna try to get, there are two elements that I like a lot and I struggle a lot of times, when you are modeling your workloads, you have to somewhat randomize the actions. A big thing that we used to leverage in the load runner days was randomized think time in between steps. But eventually you have to say, okay, it’s randomized, it’s happening, like real people is using the system, but in the end I want just this process, just this virtual user to iterate a thousand times in an hour. Oh, but I have random times in between, okay let’s, that modeling, as well, I can tell got so interesting. And even like mathematic, not too complex, but mathematic functions like, okay, over time we have this deviation, if everything goes in the higher random wait time, we may get up here, we may get down there, so on average we’ll be simulating this. It’s all so challenging.
Yeah, those delays are definitely part of the workload model. In k6, we call them sleep. In JMeter, they’re timers. LoadRunner is think time. There are a lot of different terms for it, but basically a delay is some sort of planned sleep in your script that is meant to simulate a real user maybe taking time to think about what they’re going to do next, or read the text on the screen, or type out something in a form. And it actually can have a pretty significant impact on your load tests. First of all, if you don’t have any think time at all, then you need to be careful that you’re not maxing out your CPU and memory utilization on your load generator because that situation where you’re just very quickly firing off requests may not be entirely realistic. And I think that if you have think time that’s always the same time, like this is what you were talking about, having dynamic ones, if you always use the same value for your think time, you might see the staggering in your results, but that’s due to how your test is set up and not at all to do with how your application is responding. And then if you use dynamic think time, like you said, it can make it harder to plan because it’s more difficult to know exactly how many iterations your test is going to complete. So there’s always trade-offs, right. There’s no one way that’s always the best way.
And it depends, as well, on what are you actually trying to simulate. As you mentioned, there are times where we put think times that want to simulate a real person using the system. There are some others where you just don’t want these events to happen at the same time. To give you an example would be like if your script was a soldier marching one, two, one, two, and you’re adding soldiers at the same pace, one, two, eventually they all will be doing the same step at the very same time. If you are simulating a military parade or something like that, that’s awesome. You have the test. On that time, you may want that to happen or, or work it out with some rendezvous continuously happening. But if not, if you just want to simulate a regular march or people passing in the subway, they are not like one, two, one, it would be very interesting to see, but that’s not how real people walk in the every day. There are situations for everything, and it’s challenging at times, as you say, not only wait in between steps, but waiting after you complete a cycle and you want to start another one. There, you may also want some randomization or not to be that static so that your generally load simulations do not have to be that robotic, even if they are done by robots or virtual users. So there are several trade-offs.
I have an extension to your metaphor, I think you’ll like, Leandro, what if instead, this is a way that your script might change depending on the business processes and the context of the industry. So what if instead of trying to simulate a military, what you’re actually trying to do is simulate Fremen running across the open sand, trying to not have patterns in their steps to avoid attracting Shai-Hulud, the great worm. If you don’t know the reference, I’m sorry. But in that case you do not want anything constant. You want something more dynamic, something that isn’t easily noticeable, that won’t skew the results as much. Oh, we have a question from ChipsAho. I don’t know if it’s just the Filipino thing, but there’s actually a good packet of cookies that’s called Chips Ahoy.
Ahoy, love them.
If you are faced with budget constraints, how would you prioritize the workload model to use? That’s really good. And they follow up with, as performance engineers, I know sometimes our curiosity tends to get the best of us, trying to see how the system behaves in different circumstances.
Well, answering to, oh yeah, ChipsAho, is from Ahoy, very good cookies, put ’em in the microwave eight seconds and you’ll love ’em. So it depends a lot, I would say, and your budget constraints, what are you testing would be my question. What risks does your application will face, or what are the most common reasons? Again, there’s no silver bullet or single test that will clear all your performance risks ever again that you may have under loads. What loads were you? What things may harm your system? Those are the ones that you need to focus and prioritize based on that risk. If you already tested one, you know that it’ll survive, next test, maybe go to the next one and down the ladder, the risk ladder. But first it depends on your system. What is it going to subject to? What are the processes? And the risks may be different a spring ago from today. You may be releasing a new process that is super critical, super dangerous, will be touching databases or doing important or even dangerous queries. This time, you may want to focus on that one. There’s no one test to rule them all, one workload scenario, so it depends.
I think there are a few things that I would think about first. What is, like Leandro said, what’s the biggest risk? Meaning if there’s something where if you don’t load test this component there might be security implications, test that. Anything that’s critical, like payment or something that might affect compliance regulations for whatever industry it is, meaning like how you handle individuals’ data, potentially identifiable data, that would be worth load testing. And also anything that’s very public, so maybe something like you’re about us page is nowhere near as public as your homepage. It can deal your company very significant damage, but hard to quantify damage, in terms of reputation, if your entire homepage is down, rather than if your about us page is down. So that could be a situation where something is not really mission critical or important to the business functionality, but it still would deal a lot of damage. So that’s a lot of reputation risk, I guess. A question here not about executors, but we can quickly talk about that. Do we get any performance HTML report in browser k6 load testing? Not yet ‘cause this person specified browser. So I don’t think that we have that yet for browser, but there are a few extensions. There’s the xk6 dashboard one by Ivan Skiba. And there’s also xk6, what is it called, xk6 HTML report or something like that. Go have a look at the session that I just had, the show that I just had with Paul last week, actually, where we talked about this very thing. They’re both extensions and they’re both awesome. They give like a, the Ivan Skiba one, I think, is the one that you did, Paul, right?
Yeah, and actually I was gonna show that a little bit later here.
Okay, all right, well let me know when you want to.
Stay tuned, yeah.
We might as well jump into that, ‘cause I definitely want to show some things about how to actually implement these workload models in k6. So one way that you can do that is by using scenarios. So I have, let’s just start with this one. So I have one script here, it is a protocol level script. And I’m just putting it in a separate JavaScript file just so this is a little cleaner to look at, so that we can focus on the scenario. In k6, we have something called scenarios that are ways to set things that run independently of each other. So you can have it be consecutive, or run one before the other, or you can have them run simultaneously. So this one just has, this example that I’ve got up now is just one scenario and it has an executor. So let’s talk a little bit about executors because this is something that’s specific to k6. So let’s look here. An executor is basically a way to change the way that the load looks. There are a bunch in k6 and I’m actually going to, I already, ‘cause I get this question a lot, I’m going to put this in the chat because I have this cheat sheet for how to go about thinking, deciding which one to use. So I’m going to post it in the chat and we, I think that the rule of thumb is if you want to just try one, you can, or you want a simple one, it’s probably ramping-vus that you want. And that’s because this is the default one and a lot of load testing tools where you set the number of virtual users and duration and then sometimes you might have stages. This is also what k6 defaults to. So let’s have a look. This is what that looks like. As you can see, I have ramping-vus set here, and this is a way to change the load depending on the number of virtual users. I also have these stages here. The first one is kind of like, I was trying to simulate a ramp up. So from when the test starts, this will start with zero virtual users. I’ve got that set here. And then it’ll gradually ramp up to 10 over five minutes. This is shorter because I think I was trying to, this was from a demo, so I was trying to make it something that we’d actually get results from, but this could be, you know, 30 minutes, if you wanted, depending on your test. And then this is kind of what we call steady state, where the number of virtual users doesn’t fluctuate. It’s just holding steady for this period of time, in this case 10 minutes. And this last one where it goes from 10 virtual users to zero is reflective of a ramp down. Paul, did you want to share your screen ‘cause I know you said you had some things in cloud to show us.
I did, yes. And then unfortunately I wasn’t able to share for my script, there’s something going on with my system here. But let me go ahead and share my browser, which is working. So yeah.
Awesome.
So this is a script that I had done and what I did was, as I was trying to go through and then figure out all these different executors, I wanted to see the pattern that each of these executors applies. So now here, let me, maybe it’s better if I start down here. So this is the actual script that I executed. So if you look here, you’ll see that in these scenarios and hopefully, let me see if I can.
[Nicole] Yeah, I was just going to ask.
[Paul] A little bit more legible, there we go. I’m not the only one that may have old eyes and just can’t see. So I loaded up the options so that I would have all these different scenarios. ‘Cause I wanted to see every one of these executors and I wanted to be able to see what kind of pattern that they applied. And originally you’ll see that some of these numbers are a little bit different, but these are all the very specific things that each one requires. So in this case, the shared iteration scenario, and again, I’m not gonna be the expert on what each of these different executors provides, but they’re in the docs. So yeah, we can go through there. I’m gonna be working on these docs, as well, in the next couple weeks and try to clean these up a little bit more, maybe make ’em a little bit more, I don’t know, state it a little bit differently or to help my understanding anyway. But yeah, so I loaded this up, so I have each one of the different executors set up. So here’s the ramping, so similar to what Nicole was talking about with the different durations. So here over a ten-second period, I had it ramp up to, I think, from zero to 10 vus. And then for 10 seconds I would have it keep going at a level 10, so it would be a flat top. And then the last 10 seconds would be ramping down to zero. All these other ones here, too. So, I put every one of these in there so that way then we could go ahead and compare them and, whoa, let me, all right. So each one of these is kind of denoted here and I guess I can’t, I wish I could isolate these and then what’s really, I guess, the most important here is the purple line and hopefully you’re not, no one’s really color blind or missing the color differences here. But yeah, so the purple line is the actual number of the request coming through. And then this blue line, or the aqua, is the actual response time for the service. We’ll just kind of focus on the purple line that’s what these executors are really affecting more so. And then the gray line, this shaded area is actually the virtual users. So you’ll see that here it started at zero, and then it ramped up for a period of time, and then it started kind of flattening, and then it would ramp down. Now this is for the accumulation of all the different executors. Unfortunately, and this might be something I might put in as an issue to possibly add or an enhancement is that each one of these lines here is a scenario. So this actually has the shape from that particular executor. So it’d be nice to show the vus in here, as well, but you can kind of infer it based on the activity, the request. So again, that’s the purple line again. And so here, you see this executor actually ran pretty quickly to get to the number of requests. I was trying to keep them about the same because I wanted to do, like, a progress bar race. So when I was running this locally, I could see the different areas in which was actually completing the same number of requests faster and all that. But yeah, so this was kind of my playing around and, again, this is on the cloud product. So this is an add feature. But now with talking about the HTML reporting, this is what I actually did here. So let me go ahead and I’m gonna go ahead and run this script again. Now unfortunately, you’re not gonna be able to see the actual execution as I run it locally, but let’s see.
[Nicole] Paul, we got a question about the k6 cloud results that you were showing. Could you create a shareable link so that people can also have a look at it?
[Paul] Well, let’s see. How can I do that?
[Nicole] You can go to the three dots in the upper right. Yep, and then share test results.
[Paul] All right.
[Nicole] We can just copy and paste that into the chat.
[Paul] Cool.
[Leandro] And then they are out into the world.
[Paul] Out into the world, yeah. Awesome, cool. See, today I learned again.
[Nicole] Yeah. Awesome. I’m so glad you did this, though, because I wanted to, but I didn’t get around to doing it. So I’m really glad you did.
[Paul] Yeah, and then I wanted to actually have them separate so I could kind of show each one’s pattern again. That’s the thing that I was really trying to get to, and I’ll probably put together a blog post, so keep an eye out for that, where I’ll kind of go through that, and describe, and show patterns that are created by the executors. And then that is as I learn more. Again, so this is, this dashboard that I’m showing here, this is actually an extension. So I did compile this into the k6 binary, custom with this output. And this is available here. This is this xk6 dashboard, and this is also included in our registry on the site, so you’ll be able to find this as well. Pretty easy to do this, the xk6, to create the extension and build it. But anyway, we can always give help. Anyway, I’m gonna go ahead in this other window, and I’m gonna go ahead and start this test running. So now as we see here, and I shouldn’t have to refresh, but let’s see, go ahead and refresh here. All right. So now this test is running in the background. And then we should start seeing the output. There we go. Now, there is a delay. So behind the scenes, this is actually using, it’s creating Prometheus-based metrics and then it’s serving that up in a little web server locally. So it’s really cool. So I haven’t been able to play around with it nearly as much as I need to, but anyway, but this is something, it’s a nice little tool to be able to use if you don’t want just the plain text display in your console. But yeah, so this test is actually already completed. Now, it’s actually still running in a way, though, because we have this externally controlled executor where now from another terminal I could sit there and I could say, okay, yeah, now I wanna ramp up the vus to 10 or that type of a thing, just via rest API. But unfortunately, due to technical difficulties, I can’t show you that, but I will definitely put that in the blog post when I start putting that together. Anyway, this is one of those cool little tools with this extension that you can kind of, I don’t know, really get some cool results. But yeah, that was the big thing. Here’s for the external access, this is the rest API where I could sit there and call that custom executor and say pause testing or even to ramp up the number of vus. Still figuring this out a little bit. It’s a little, I don’t know, it needs some play around with it a little bit to get a good understanding of it. But yeah, I wanted to show that, again, you can do this stuff, have the graphics there. Now, the HTML report that Nicole was talking about, that’s something where, yeah, you could save that off, and again, it doesn’t have the graphics, I don’t think, if I remember correctly, but it does have it where you could then save that off.
It’s not real time either. But it is more like a report.
Yeah, yeah. That’s it, let me go ahead and stop that test. Oh yeah. So again, I will put together something so that way then if anybody wants these details and obviously you can use that link, so it’s now shareable, so I will make sure to not delete it .
Great. I do want to share something, Paul, if you’re okay.
Yeah, absolutely.
Great. So I also wanted to talk about the fact that we can do it, you can put together a scenario using different executors, but another thing that you could do is have each scenario be a completely different test. One could be a load test. Another could be a chaos engineering test, for instance. Like this is one that I had where this get Pokemon is an HTTP request that just hits the API. And this is intended to be more of a load test, but then you could have a chaos scenario and you can say that at a certain point, you fire off this thing that then terminates one of the pods in Kubernetes, according to certain conditions. This is using the xk6-chaos extension. But in this way, you can see that these are not just load-testing scenarios. These are just testing scenarios and they’re different types of testing that you could use. Another type other than chaos is browser testing now that we’ve got xk6 browser. So now you can do things where you have one scenario that’s doing some sort of protocol level test and the other one is doing a browser level test. And because those two are fundamentally different, you may want to be able to control them separately. You probably won’t be wanting to generate majority of the load with a browser level test, for instance, you probably wanna do that with your protocol level test, but you might want to piggyback off of the protocol test and run a few, like a handful, of browser users just to be able to get some front-end metrics. So, I think it’s kind of cool that you can do that. Another thing that you can do with scenarios and executors, well, it’s something with workload modeling anyway for k6, this is a script that I’ve got for percentages. Now, I think the use case for this, I mean, there are a lot of use cases, but what I actually think this could be used for is an e-commerce scenario where a typical scenario is to have like 100% of the virtual users go into your homepage, and then maybe 50% of them go and browse through the product pages, and then of that 50%, maybe only half of those actually added to the cart. So that is also something that isn’t directly related to scenarios and executors, but it does have something to do with the workload model of your test. So in this example, I’m setting a random number between one and 100, and then depending on that, I have this combination, in this case, I’m setting different clients for it. So let’s say that there’s a really big client that we want to use 50% of the time, you could use that, and then pass that on to some request that you’re sending. And in this way, you’re already making something that’s more realistic than just get a client, just get details for a client. Well, which client is that because maybe that’s being cached? Maybe you do want it to be cached, you’d have to see what you want, what you’re trying to achieve with that particular test, but it all has an impact on your testing results.
Yeah, and if I may add for learning some of these functionalities, it makes, and I think I started a comment how the situations, how the ramp-ups work, how when we want to model these targets in the old days, as I said, getting these, okay, my user takes this amount of time of execution, it has this amount of random timers, and it’s a pain at times when your process starts to degrade a little bit of performance, and because of that, they don’t go as fast as you need them to, as you want to generate, because you did the calculations based on optimal response times, right. So some of these controls will be very useful depending on the scenario that you are wanting to focus and not to do all that type of crazy math that we had to do. I remember with the load runner days, it was like, okay, but if the response time starts to deviate a little bit, this we need to, there’s so much that on this situation with some of these options, k6 can mutate may be the wrong word, but adapt to what you’re trying to do, right, and keep doing the loads and doing those processes that you are aimed for. There are some others that you are like, if the system starts to go slow, the users will slow down and that’s right, that’s okay, that’s normal, but on others it’s like, no, I absolutely want that the users keep coming, keeps hearing the system, even if the system is slowing or many other things that can start to happen. These controls truly make our lives easier. in so many of those situations like crazy.
We have a question from SJ who says, is there a pacing config in k6 or maybe a throughput controller equivalent? So let’s talk about pacing, Pacing and load testing terms is related to think time in that it does impose some sort of timing on the requests, but while a think time is more like a direct sleep on a thread, pacing is more like the space between each iteration and there are different ways to achieve this. k6 doesn’t have something that is exactly pacing. But for instance, the way that JMeter would do this is by using the throughput controller, which is what I think SJ is alluding to. And so instead of specifying the time between each iteration, JMeter does it by allowing you to set a target request per second, some sort of throughput. And we do have that. Let me just share my screen again. So this is the executors that we have. And I think this constant arrival rate would be the best for that. Because if we look at the example here, you can set a rate here, a requests per second, that is your target. What k6 does is it sends a request or it does one iteration, sees how many requests were sent, and then adjusts the next one, waits a certain time so that before it does the next iteration to adjust to that. So if it already did, you know, 200 requests per second the first iteration, then maybe it’ll just do the same thing for the second iteration. And it will dynamically change throughout the test. So maybe at the beginning of the test, response times are faster because not all the users have ramped up yet. In that case, the pacing might be slower because there’s no delay on the server side. And so the script is easily hitting the 200 requests per second, but you’ll find that later on throughout the test, you may experience higher response times, and in that case, the pacing will be quicker. So that number will be lower so that it still reaches the 200 requests per second.
Yeah, it’s part of what I was mentioning that easiness of. JMeter, I remember before they didn’t have that plugin for the throughput, what was the name, I just forgot about it. At the very first times that we started to use
Throughput controller.
Throughput controller, thank you. We didn’t have those type of controllers, ways to managing them. And again, bringing the load runner days, I don’t remember many useful devices to go through this. And on this case, when you have a scenario that at times when we were designing this workflow, this scenario, we would know, okay, I know that per user, I want that in an hour these many iterations happen. So if you want that, you can use the per vu iterations executor that will help you to quickly configure and manage that, you do have the iterations per user, and maybe you have 10 users, 20 users, you don’t care about the number of users. Then you select the shared iterations. I just need this throughput to happen this number of times in this period of time, maybe next time I will have more virtual users, maybe next time I will have less. So you have more control on the end game, to call it in a way, that you’re looking for, rather than, and I cannot say this enough, it was a struggle to try to control, and simulate, and reach those goals that you were trying to get in your simulation, and there are so many reasons, as I said, the wait times in between steps were randomized before, as Nicole says, you have the pacing times in between each cycle, and it was so many, like three factors, main factors, I would say, wait time, pacing time, and processing time, each step would add up to that time that there are so many variables that were somewhat random that you could not control. So here you would have some assistance, some help on hitting those goals that you’re trying to simulate and having it realistic or on the goal that you’re trying to reach, right. It’s that difference, I don’t know if anyone has gym experience or has ever visited one of those things, but they will tell you, hey, I want to work leg today. Okay, do five repetitions three times. That’s if you wanna think of that, how long do you take and do each sit up to go, to give an example, that will be your think time, your wait time in between steps. Okay, you need some rest time before you do the next one. That’s the pacing. So for the ones that are not so familiar with those terms, just wanted to throw out that silly example.
It’s one of those things I’d like to, I don’t know, I keep thinking of playing around with, you know, if you look at say HTTP audit logs or something like that, they have a timestamp on there as far as when everything came in. Well, if you take that and use that as the basis to basically replay your activity, I don’t know, that’s one of the things that I keep thinking about, that would be cool to do, or to have, you know, all your activity going into Kafka, and then being able to repeatedly replay the exact same, perfect storm scenario without having the randomness injected in there. I don’t know, I’m probably going off on a tangent, but that immediately made me think of that, though. It’s like this is one of those things where it’s like I always kinda wanna play around with this idea.
I think that there’s a time for that. I think there’s a use case for it, but there are also some disadvantages, I think. that people should be aware of. If it’s information from a production environment, there are potentially security concerns there for using a user’s data to test or a user’s account to test. And what happens if that actually gets replayed on their account, which it could be, it, it might not be. Sometimes it doesn’t work, sometimes sending the same request, even if it’s exactly the same, won’t work. I think that that’s, I think, more experimental, I guess, when you don’t really know what you’re looking for and you’re trying to narrow it down, but I also think that once you’ve narrowed it down, it’s probably easier and cleaner to script it up so that you can consistently reproduce that scenario. And you can also use test data. Also, you’d be able to kind of break it down, you’d control all the variables, so you know with certainty that when there’s only one thing that changed and it was a negative result, then you know that that was the cause of it. But yeah, I think that there’s a valid reason to use that, as well.
Yeah, there’s definitely data sanitization that you gotta be concerned about unless we’re replaying activity on Leandro’s credit card, then it’s okay.
No, please don’t. Oops, there are 10,000 charges per hour in your credit card. What?
Load testing your credit card. Navin Kumar, whose videos I was just plugging at the start of this has a question.
Very good question.
Can we intentionally include sleep time in the response time? Now, Mihail, who’s one of our developers, if I say something wrong, tell me in the comments, but I believe that there isn’t a way, that there’s no k6 specific function to do this or option. What I’m thinking of is you could create a custom metric because you can create really a bunch of different metrics, counters, and timers, and gauges, and whatever in k6. So you could definitely do that and have that reported and you could set the timing manually, as well. Another thing that I’m thinking of is if I had to do this, I’d probably just use the xk6 browser extension, because that’s already going to include everything, the actual render time. ‘Cause now we’re getting, if you’re including the sleep time or the think time, this is more like getting into the front-end performance side of things and I think it might just be easier to do that.
That was gonna be kind of a question that I wanted to do, too, Naveen. Like why do you want to include the sleep time programmatically that you are generating in your script? I mean, as you say, Nicole, the front end is important to know those render times, those wasted moments in the response or back-and-forth communication, but what are you trying to use it for? It would be very interesting if you can share it. I’m curious what’s the use. Oh, okay, you were just being creative.
Yeah, I think that, yeah, probably if you want to just see front-end experience, I do think it would be more useful if in this case, if you’re including it in the response time, then maybe have a constant sleep, actually, because then it’s consistent because you don’t want the sleep to be influencing your results.
I did use something like that a long time ago, but precisely to model this workload to calculate. I generally recommend do a small-ish version of the load test that you want to do, let it run for a while so that you can massage those values of how much random time to sleep here, how much random pacing you have here, and how is the response time affected so that you can balance it out. That could be an interesting use, real-life example, but that’s a good question, as well.
I mean, injecting a network latency and things like that is always a nice thing for backend services. I mean, with microservices, because one single API call on the front end could be going through 10 different microservices ultimately. And to be able to test something where if halfway through those calls if you have a network hiccup where one of the services is not accessible, but it’s, anyway, but it’s all one of those things where it’s like this feels a little bit more on the chaos engineering aspect of things, you know, how is your application going to behave when something goes wrong? If all of a sudden your network grinds to a halt where it’s going really slow, is your application handling that scenario appropriately?
Okay, and on that note, we are actually a little bit over on time already, that went by really quickly. Thank you both for coming and for sharing your thoughts on these topics and also for demoing the k6 cloud stuff that you already had, Paul. Yeah, you are welcome back anytime and I hope we do this again. And for everybody,
I thought that was the last one.
Oh, well, sorry to disappoint you. You’re stuck now, like you’ve signed the contract, that was it. You didn’t read the fine print. Must come to office hours every Friday.
This would be the last time that you would want me on here.
Well, for those of you who are watching, thank you for watching. Thank you for all the awesome questions we got this week and have a good weekend. And if you are in a place where it’s very stormy, good luck, hope we all survive it.
Stay safe.
Bye everyone, have a good weekend.
Take care everyone, adios.

How to get started with Obsidian in 2022 - from scratch

Wed, 16 Feb 2022 17:00:28 +0100

In this video, I talk about how to ignore all the complicated methodologies for note-taking and just get started with the good old-fashioned art of taking notes - with Obsidian, of course!

Getting started with using k6 extensions, with Paul Balogh (k6 Office Hours #41)

Fri, 11 Feb 2022 18:00:24 +0100

In this week’s k6 Office Hours, I was joined by my new colleague Paul Balogh, to talk about his experience in joining the k6 team and his first impressions of the xk6 extension system.

Foreigners Try Latvian Food (Lucky Dippers)

Thu, 10 Feb 2022 20:04:46 +0100

Latvia was the last country we travelled to in 2018. We took the bus out of Tallinn and into Riga, Latvia. We unfortunately didn’t have much time there before we had to fly back to our home in the Netherlands, but we DID have enough time to sample some Latvian food!

In search of the best Pokémon: Browser automation and load testing in one script with k6 (Automation Guild 2022)

Thu, 10 Feb 2022 12:01:53 +0200

Being a tester is like being a pokémon trainer: we want to choose the right pokémon for the job. The problem with doing different types of testing is that we also end up with multiple scripts in different languages and testing tools. We end up choosing a squad of pokémon: one for unit testing, one for automation testing, one for load testing, and so on. Making sure they all work together is a task in itself!

But what if we could just use one? What if we only needed one Mega-evolved pokémon that could fulfill multiple roles? Is it really possible to write a browser automation script and then reuse it for load testing?

In this presentation, I talk about my search for the elusive one pokémon that does multiple jobs, the advantages of hybrid load testing, and how to pick pokémon that make our life as trainers and testers easier.

Reclaim.ai Review - Smart calendar blocking for the productive engineer

Wed, 09 Feb 2022 19:56:13 +0100

For the last few months, I’ve been trying out an app called Reclaim that has been changing my mind about the feasibility of calendar blocking for time management in general.

Here’s my (rave) review of the app!

Using Executors and Stages in k6 with Théo Crevon (k6 Office Hours #40)

Fri, 04 Feb 2022 20:01:12 +0100

Today, Théo Crevon, one of our newest Senior Software Engineers at k6, joined me and my colleague, Leandro Melendez, to talk about his experiences in joining k6, and why scheduling in a load testing tool is simultaneously difficult and important to get right.

Pizza and Antiquing in Tallinn Estonia (Lucky Dippers)

Thu, 03 Feb 2022 10:59:00 +0100

In December 2018, my husband and I went with a few friends to Tallinn, Estonia, where we spent some time going around the cool district of Kalamaja. We ended up doing a bit of antiquing, thanks to some local friends who were willing to show us around!

Creating a D&D DM's Brain Attic With Obsidian

Wed, 02 Feb 2022 10:46:56 +0100

I started using Obsidian for professional reasons. I use it daily as my main task management and personal knowledge management system for my main job as a software developer advocate at k6.io. However, a surprising use case of mine for Obsidian is playing Dungeons & Dragons, the tabletop roleplaying game.

Here’s a video of me walking through my setup for how to use Obsidian to create my own “DM’s Brain Attic”.

Other Resources

Realistic Calendar Blocking for the Agile Knowledge Worker

Tue, 01 Feb 2022 16:07:31 +0100

Calendar blocking is the only way to identify and avoid distraction.

In a world that tempts us with “a little bit of everything, all of the time,” resisting distraction is the most valuable skill. Yet distraction isn’t always easy to recognize, especially when it comes in the guise of other things we need to do.

Here’s how to use calendar blocking to gain control over your schedule.

Set realistic schedules.

Setting an overly ambitious schedule inevitably leads to disappointment.

The hard part is figuring out what’s infeasible, and that has to come with time. In the meantime, expect to change your schedule.

Use digital tools.

Calendar Tetris isn’t fun– but there are tools that can help.

If you expect to change your schedule, choose tools that make that easier. I use Reclaim.ai to change my calendar in response to changing priorities.

Do a weekly review.

Review how well you stuck to your schedule and adjust it accordingly.

A good schedule, like a good budget, changes over time. Set aside a time every week to assess the previous week and set priorities for the next week.

Relate your schedule to your core principles.

Your calendar should be the reflection of the values you hold.

But what are those values? Keep a list of the principles you live by (or want to live by). Look at your calendar as a way to keep yourself on track.

Reframe restriction as freedom.

When you don’t set your calendar, others set it for you.

Restriction isn’t following a schedule you set. Restriction is being subject to the passing whims of bugs and meetings that demand your attention.

Calendar blocking defends your time.

It frees you to do things you decide are important.

5 Things the Obsidian Templater Plugin Can Do That Templates Can't

Mon, 31 Jan 2022 22:46:52 +0100

The Templater plugin is what the core Templates plugin should have been.

The name is unfortunate, as it implies that the two plugins are similar. Turns out they’re only similar in that one is a limited version of the other.

Here’s what you can do with Templater– and not with Templates.

Add dynamic variables to your notes

You’re probably thinking, but Templates can already add dates.

Templater can do that, and more. You can use it to fetch values in YAML frontmatter, random quotes from the interwebs, and even clipboard contents.

Make your notes context-sensitive

Templater can create notes with different templates depending on context.

For example, you could set up Obsidian to create a new note using the Person template in the People folder but a Weekly Review note on Sunday nights.

Increment your notes automatically

Templater can automatically increment new instances of a series.

You can set it up to create a session log for your 100th D&D session, a review note for your 12th monthly review, or your 10th book review.

Interact with other plugins

Templater leverages all the best parts of Obsidian’s ecosystem.

You can read from and write to YAML frontmatter using Dataview, or run commands that create notes based on text input with Buttons.

Write and run your own scripts

Templater lets you write and call user scripts within an Obsidian note.

This is the ultimate in extensibility. The ability to run custom JavaScript means you’re limited only by your imagination (and skill).

Templater lets you turn Obsidian into exactly what you want.

And isn’t that what a PKM should do?

10 Mandatory Obsidian Plugins for Every Software Engineer

Sat, 29 Jan 2022 23:58:12 +0100

If you haven’t installed any Obsidian plugins, you’re missing out.

Many other mainstream PKM apps seem to have been built with the intention of hand-holding users through every step using a pretty interface. On the other hand, Obsidian is perfectly suited to power users who want to change the tool to meet their needs exactly. But you can only do that with plugins.

Here are 10 Obsidian plugins you should install if you work in tech:

Enable these Core Plugins.

Core plugins are shipped with Obsidian but are disabled by default.

Daily notes: Creates a new Markdown page according to a specified template that serves as a scratchpad and springboard for every new day.
Templates: Lets you define predetermined formats that can be inserted into notes. Some template examples: Daily/Weekly/Monthly/Yearly notes, meeting notes with agendas, person notes to create your own CRM.
Backlinks: When you have a note open, this shows you all the other notes that have referred to it, explicitly (via [[links]]) or implicitly (unlinked).
Quick switcher: Lets you find any note by hitting CMD+O and showing search results as you type. With this enabled, I never bother with folders.
Workspaces: Lets you save the exact arrangement of notes and sidebar options you have selected, and then load it quickly later. I have workspaces for writing, processing content, playing D&D, and making videos.

Install these Community Plugins.

Community plugins are created by users and open-sourced on GitHub.

Calendar: Adds a real-world calendar to your side panel that shows daily and weekly notes. Use Fantasy Calendar for even more customization.
Editor Syntax Highlight: Formats text in code blocks according to the programming language set. Drastically improves readability of any code.
Readwise Official: Regularly imports your highlights from ebooks, web articles, audiobooks, tweets, and PDFs into your Obsidian vault.
Templater: Extends templating functionality by letting you run JAVASCRIPT. In your vault! This blows the functionality of Obsidian wide open.
Dataview: Turns your Obsidian vault into a database using custom parameters in YAML frontmatter. There’s even a way to write queries in js!

Señor Performo's Week of Load Testing (k6 Office Hours #39)

Fri, 28 Jan 2022 23:27:47 +0100

This week for k6 Office Hours, I talked to one of the newest members on our team, Leandro Melendez, aka Señor Performo, about his experience in joining k6/Grafana Labs. We ended up talking about performance “deadly sins” and how the practice of performance engineering has changed in the last decade.

Old Town Walking Tour of Tallinn Estonia (Lucky Dippers)

Thu, 27 Jan 2022 23:25:13 +0100

Here’s a walking tour we went on in Tallinn, Estonia, in December 2018.

Why I'm Leaving the Netherlands

Wed, 26 Jan 2022 21:15:31 +0100

It’s been almost five years since I first moved to the Netherlands, and while I love the country and I’m grateful for all the experiences I had here, I’ve decided to move away. Here’s why.

Why All Roads Lead to Testing

Tue, 25 Jan 2022 23:59:03 +0100

Why all roads lead to testing

Testing is a skill every software engineer should develop.

It’s easy to dismiss testing as a responsibility only some carry, but it’s best thought of as a hat that anyone can put on. How well you learn to wear this hat affects your career whether or not you’re a tester. Here’s why.

Ideas are cheap.

In tech, having an idea isn’t what sells– the success is in the execution.

iPhones didn’t become popular because it was the first mobile with a touchscreen. The Nintendo Switch isn’t ubiquitous because of its revolutionary hardware. Behind every successful product is a lot of testing.

Testing isn’t just for testers.

Every good software engineer does testing.

Debugging is a form of testing. Questioning requirements is a form of testing. Testers don’t (and shouldn’t) have a monopoly on testing. The reality is that everyone is doing it, often without calling it “testing”.

Testing is about people more than it is about machines.

The fundamental goal of any app is to address a human need.

Testing is finding new ways to represent and understand this need at all stages of development. Testing is a people problem, not a technical problem.

Humans are irrational.

The biggest challenge in building apps is human irrationality.

We don’t always want what we need, we don’t always know what we need, and even when we’re building for others, we don’t always work towards what we need. Testing is an attempt to make sense of this irrationality.

Testing is an attitude.

Getting good at testing extends to aspects beyond work.

Testing is a discipline of questioning. It’s having curiosity as a default state and trying to meet people where they are. These are things every project, and every community, needs in spades.

Testing is rarely the destination.

But some time, some how, all roads lead to testing.

How to Actually Remember What You Learn, for Engineers with Long Reading Lists

Mon, 24 Jan 2022 21:47:59 +0100

Who says you need the terminology of the Zettelkasten process? Here’s how I’ve learned how to learn and how to apply new ideas to my daily work.

If you’re reading this, you probably make your living on what you’ve learned that other people haven’t. Unfortunately, if you don’t have a written system to keep track of new ideas, you’ll probably unlearn everything in a few years.

Most people think reading = learning.

Here’s why reading isn’t enough.

Our brains are best at linking concepts, not at remembering them.
Writing doesn’t help us learn; it IS how we learn.
You can’t learn to swim by reading about it.

If reading isn’t enough, then what should you do instead?

Step 1: Consume

The first step is to consume content, but selectively so.

Content could be a book, a video, a podcast, an article, or even a conversation. Take notes on the author’s main points.

Example: Shape Up is a methodology emphasising iterative cycles, “betting” on work to be done, and building fully integrated slices of work.

Step 2: Redefine

Next, go beyond facts and add your thoughts.

Think about the broader picture of what the author might be saying, and supplement it with your opinions. Example: Shape Up is an Agile methodology concerned with the limits we should apply to different stages of work.

Step 3: Contextualize

Think about what the content means to you, specifically– how does it stack with other things you’ve learned? Where does it fit into your interests? Accept that your context, and your notes, will change.

Example: Shape Up’s idea of fully integrated work supports the notion of context-switching (see: Continuous integration, Multithreading) but could be in contention with productivity concepts like deep work (David Allen), batching, and Productivity Momentum.

It’s your brain!

In the end, nobody can tell you the “right” way to learn.

My Workflow for Making Tech and Travel Videos Online

Sun, 23 Jan 2022 16:16:33 +0100

In the last 5 years, I’ve made ~220 videos across 4 YouTube channels. Here’s what I’ve learned so far about making videos.

Why make videos?

A 10-minute video takes me anywhere from 7 to 15 hours to produce. I wouldn’t be doing it if I didn’t enjoy it!

It can be the most genuine form of expression.
Videos were a big part of how I got into tech, and I want to pass that on.
I’d rather make a video once than explain the same thing to 100 people.

I don’t think my way is the best way, but maybe sharing my workflow will make it easier for you to get started, too!

Pre-production: Ideation

Ideas are cheap. The best idea is worthless if I can’t convey it well.

I start with the idea, then the title and thumbnail. I search YouTube to see what titles have worked for others. Then, I use @Canva for thumbnail creation and thumbsup.tv to test combinations.

Only then do I start to outline the video or research it. I use @obsdmd as my project management tool and content calendar.

Production: Filming and editing

I like to film with talking points rather than a full script. It saves me time, and it comes across as more natural. My main gear: Sony A6600 + Sigma 16mm f/1.4, Røde PodMic, Elgato keylight.

I edit everything myself in Final Cut Pro, using stock footage from @StoryblocksCo and sound effects/music from @epidemicsound.

Post-production: Publishing

I export and upload the video to YouTube, adding the title, thumbnail, description, cards, end screen… I have a checklist so I don’t forget these.

I send most videos to humans at @Rev for captioning. I then use the captions to create YouTube chapters. From there, it’s just about publishing to appropriate platforms.

Optimize for consistency and fun

When in doubt, I chuck technique out the window and do what will keep video creation fun for me!

How to Do What Kind of Testing When: Testing Types by Scope

Sat, 22 Jan 2022 16:42:49 +0100

Correctly identifying when you test could be just as important as what. Below are types of testing according to scope.

Static testing

Static testing occurs when requirements are written. Question requirements and spell out what each one means.

Concerns: Setting up a system to track and link requirements to test cases, negotiating business needs with technical feasibility, clarifying scope.

Unit testing

Unit testing occurs when coding begins. Practice Test-Driven Development by translating requirements into tests before writing the code.

Concerns: Treating test code like application code, getting developer buy-in, communicating technical implementation, performance, pair programming.

System testing

System testing occurs when a build is released. Employ black-box testing: step back from the code and assess the feature as a whole for consistency.

Concerns: Adherence to common design standards, equivalence partitioning, boundary value analysis, fighting scope creep, usability, creating test data.

System integration testing (SIT)

SIT occurs when integrating multiple components. Use white-box or grey-box testing to target the interactions and traffic between components.

Concerns: API testing, stubbing and mocking, identifying request paths throughout the system, performance benchmarking, observability and analysis.

Business acceptance testing (BAT)

BAT occurs when the code is assessed against overall business goals. Seek out SMEs who can help you verify expected outcomes.

Concerns: Adherence to business requirements and regulations, process testing, simplifying request flows, identifying potential customer pain points.

User acceptance testing (UAT)

User acceptance testing occurs before code is released to production. Map test cases to requirements and test real-world scenarios.

Concerns: Testing for superuser workflows, data cleansing, verifying real user behavior from historical logs, exploratory testing, end-to-end performance.

Regression Testing

Regression testing occurs between builds. Supplement a robust automation test suite with nuanced manual test cases that address known problem areas.

Concerns: Setting up a CI/CD pipeline, deciding what to automate, consolidating test and development tools, tracking failures over time.

Question everything

Remember: testing is an attitude.

Canonical Observability with Simon Aronsson and Michele Mancioppi (k6 Office Hours #38)

Fri, 21 Jan 2022 18:23:47 +0100

Today, my former colleague Simon Aronsson brought his new colleague, Michele Mancioppi, to k6 Office Hours to talk about observability, and how they do it at Canonical. It turned into a great conversation about trends in observability from the last decade, and how it relates to performance and reliability testing.

Transcript

Hello everyone, and welcome back to the first k6 Office Hours of the year. Happy 2022, I’m Nicole Vander Hoeven.
And I’m Simme, we’re back from the dead.
And I’m Michele.
Simme is back. Yes, so Simme unfortunately is only temporarily back, and he’s brought his new colleague. I mean, I’m kind of sad, but I’m happy to meet you Michele Manciopi.
Hello everyone.
Welcome. Simme why don’t you first tell everybody, how you betrayed us at k6.
Sure, I’m so sorry for that, was a couple of months ago now. I think I quit at k6 in October, beginning of October something like that, to join Canonical and Michele, where we are trying to build a really awesome observability stack to know little part built on top of Grafana products.
Yeah, it has been a very interesting three months. It’s actually a pretty refreshing to be able to use high quality software and compose it in ways that are, you know, they just work through, cool.
So we already have a few people in the peanut gallery, there’s Simme and Michele and our newest colleague, I think senior performer, all saying hi to you Simme. so you’re very much missed.
Congratulations to k6 for bringing Leandra.
Definitely, and by the way, Leandra will be on k6 Office Hours next week, he just accepted the invite. So when I’m letting him like relax, and do his bootcamp onboarding things, and his week of load testing this week. . Ooh Danielle is here as well. Michele, why don’t you talk about yourself, and what you do at Canonical?
Well, I am a product manager at Canonical, for observer BD team. I was brought in with the goal of making the entirety of the product line of Canonical observable. Canonical is chiefly known for Ubuntu, but we have like a meter other products and they all need to be observable because they all sit at different levels of critical pieces of the infrastructure. Before Canonical, I was the technical product manager at Instana, which has been what baby ham it’s a product that does very automated observability. It’s mostly proprietary, but it was a lot of fun trying to make observability the way it’s supposed to be. That is seamless. Very easy to roll out. Before that I was at SAP where I was the technical lead of the performance team for SAP cloud platform, where my job was mostly to figure out where things were slow and fix it. And then I realized that rather than helping one single developer at the time, would they be more productive in actually providing pretty decent tools to do that at scale so people could do it themselves, which then brought me down the path of figuring out it’s actually fun here to make an observability product than it is to using one. And then the CircleCI.
So Simme what was it about Canonical that won you over?
I actually think it was to no little part Michele won me over with his argumentation for–
Okay. I’m taking him off the stream sorry. No, no, No.
‘Cause I got you everyone, bye.
No. We’re still all friends.
I do really miss the k6 family for sure. But yeah, it was just such a cool and exciting opportunity to be able to make this huge leap in observability with Canonical, for such a vastly used project or suite of tools and products. So that’s mainly one of them.
I’m sorry. Can we stop to remark? That was the coolest cup I’ve seen in a long.
It is as if this is sponsored, this is a hydrate spark, it’s a smart water bottle. And it’s marketed like for people like me because it’s a water bottle, but it has Bluetooth and LED light. So, you know, obviously I had to buy.
Yeah, just out of curiosity, what are the key values of the Bluetooth support?
Oh, it tells me when I need to drink water. It’s very important stuff.
How do you monitor that A, the Bluetooth is working and how do you receive those notifications?
Well, it’s funny that you ask, there is an app for it. This is totally not sponsored. Okay. But there is an app for it. I was gonna say Simme that it seems like there was a slight change in your focus as well, because at k6, we’re more focused on the performance and reliability testing part of it. But definitely we’ve also dabbled in the observability space, especially with the acquisition of Grafana. So it seems like your new role is also more focused towards observability. Was that one of the considerations?
Sure it is. And then that was definitely one of the considerations during my, what is it now? 12 or something years in tech, I’ve been doing way more observability and monitoring work than I’ve been doing performance testing. And so that definitely felt like coming home in a sense to me.
And I wanna ask you both this because you’ve both been in observability for awhile. How do you think observability and performance testing go together? Do you see them as separate disciplines entirely, there’s clearly some sort of overlap, right? Should companies be doing both, one or the other?
I definitely think there is a lot of overlap. One thing that I know that you and I have been pushing forward when we’ve been discussing this previously on the stream is that performance testing is great, but it usually only allows you to do a black box perspective testing, right? You’re able to measure the performance of your system, given certain interfaces or end points that you are able to observe the performance off and pairing that with observability tools, like for instance, loci or tempo, or Prometheus’s allows you to really drill back into the stack and figure out what is causing those problems. So I definitely think that it makes sense to combine the both to be able to observe the whole system and not just the edges of it.
Do you agree with that Michele?
Yeah, very much, over go and say that actually, it’s very hard to draw a demarkation between the performance and of observability, especially since in the last few decades and the increasingly disproportionately larger role that internet and internet systems taking our lives. We have come to see, you know, slow as the new down, which means that not only the software needs to work but it needs to be fast. And if you observability is fundamentally the art of understanding, are there problems? I mean, so how bad was impacted and how badly, and that is both in terms of software, it is broken as software is luck. Many of the techniques that you would use in, for example, a QA system to check the scalability of your software is something that you can use at scale in production to make sure that you’re at least carrying that much load, as well as techniques for example, for performance testing, you often do a lot of black box monitoring. The techniques that you use to check if today in SIL on a mobile connection, your web store is reachable is very similar in terms of technologies and capabilities and as technical requirements. So the technology is there to use, to throw a lot of requests security to your software. So it’s fundamentally the performance tools are versatile like that.
Yeah, I totally agree with that. I have always taken the stance that it’s definitely better when you have both of them because observability is great. But then in pre-production environments, it can be a little tricky ‘cause you don’t have real traffic and that’s where performance testing can come in handy and performance testing is great, but you don’t always want to run it in production. And also you, it’s not always a good idea to be reliant on the results of the performance testing tool. You know, sometimes it makes a lot more sense to be already using the same observability stack that you use in production. And I also want to call out that performance testing is often thought of as load testing, but that’s really just one kind of performance that is being measured there. And so what about a related topic to observability as monitoring? I know Michele, you have a lot to say on that topic.
Oh my, can I rant?
Yes. Go ahead. I’m familiar with your blog posts.
Ooh influence. I have a –
I do research, you know.
I thought you came across it because, you know, super rabid spike. No, the, so I have been around the observability industry for awhile and I have a certain annoyance towards what the industry has done by rebranding perfectly good monitoring and application performance tools in terms of the hot new term up observability that you need to sell to sort of the software. The way I see it’s a much more formal, shall we say way. Observability is a property that you want your systems to have, monitoring is the way that we have to make most systems observable. However, something that I hope we will see in our lifetime, our software systems and such a level of quality and standardization in our profession of creating software systems. So that monitoring and observability becomes build team in the softwares we create. If you think about it all the for example, automatic distributed tracing and all the amazing innovations that have occurred over the past 15 years are effectively reckoning the lack of observability as a functional requirement of a piece of software. Monitoring is the art to collecting data.
Yeah, and I think that there’s also a connotation of monitoring being a bit more passive. Would you agree with that?
Not necessarily, no. At the end of it so there is, I really take the view that monitoring is a means to the end of observability. It is true that in the past few years, with new generations of tools, we have made leaps and bounds in both the capability of collecting useful telemetry with very little work in terms of toil to roll it out, without the proper tools are there and I used to work for one where you deploy an agent and it automatically discovers all around it. There is a lot of dark magic going on, but it works. And then it becomes very cheap to actually get to the holy grail of monitoring that is to have as little silos as possible. Telemetry wants to have friends. You want to see end to end systems how they relate to one another. Now, the moment that something that has changed recently in the industry is we have gotten better, not only at collecting telemetry, but analyzing it. When you compare monitoring tools that we started with right there the Nagios of the world. You’ll need it to know in advance what you were looking for. Now with broader collection of telemetry, for example, distributed tracing, a very powerful type of telemetry, very little underutilized in terms of the insights can provide you, you end up having a lot of data that is likely going to be useful, and you have much more freedom in how you analyze it, both in terms of discovering issues as they occur, as well as afterwards in terms of post-mortem.
I do have, however one, I agree with you completely Michele. I do however, think Nicole has a point in that if you look at the traditional monitoring tools that people used to use, like Nagios for instance, it was always this very weird situation where you put up SNMP traps or collected, you know, logs that were thrown at you in a very passive manner. But I would say that it turned monitoring itself has come to evolve into a state where it very much is an active practice in the same way as people would describe observability.
Oh. So you are making a very interesting distinction of rather than receiving as a data sync telemetry that was emitted, but something else that you go and get it through RTI, things like deploying an agent that does stuff. That’s pretty interesting.
Yeah. Or for instance, actually triggering a synthetic test that just prompts a common user flow and make sure that that still works. I mean, as compared to just admitting this passive telemetry data that just keeps rolling on, but maybe lacks the context as you put it on more advanced workflows where you’d correlate multiple sources of data or the multiple actions to derive your data that you want to.
It is true that in the past 15, 20 years, there has been an increase of active monitoring. I remember doing it back in the day with smoke tests that were of a crunch up every 15 minutes. I was so young. And now it’s that it’s kind of a baseline of observability. I mean, you need to want to catch everything, but if there is catastrophic failure, you will know hopefully before someone starts opening tickets or you’re upping your drops. Yeah.
Yeah. I think Simme it was right when he was saying, what I was talking about was like the more traditional approach that it’s just something that’s there in the background whereas these days, the way that it should be done is a lot more proactive.
And also they focus on actually gathering telemetry to be able to detect failure modes rather than probing for a certain thing. It used to be much more focused on a single point that you would probe for data. And–
It used to have simpler software too.
We sure did.
Yeah, that’s for sure. But I also wanted to say that there’s also been, there’s still some push based monitoring, but we’re also now seeing more poll based monitoring systems, more event driven ones. And I think this is all the evolution of monitoring and observability, right? Because these things did not exist not too long ago.
To be fair, the one should, it is pretty true. We have much more pool and pool length. This is something that a permit use has pretty much popularized with the concept of scraping, but there are entire domains where we deploy a software that is still very much push based. For example, IOT, for very basic reasons of network visibility, it’s very hard to go and reach out into someone else’s rooftop and go get the data. So there is still the mechanics of push buffer and pulled it for example, gave us in the primitive ecosystem, push gateway mechanism.
Grafana agent that actually sits in between you and a cloud service or something else, and allows you to not punch holes in your firewall, but rather reach out and deliver data to something.
Yeah, it’s kind of like you mentioned that our applications in general are just getting more and more complex, it reminds me the push the trend from monolith to microservices and then now some people are also going back. It’s never really gone, right. And it’s really difficult to say, no, this is the new way, nobody else is doing or should do the other way. It’s not really like that. A lot of things in tech are cyclical.
When I actually spoke about our systems are getting more complex. I was thinking exactly of micro services because we made a trade as an industry for very, very simple, very small components. And then pushing all the complexity that you cannot make disappear under the carpet of service meshes, many containers, things that need to be versioned independently remain compatible, which is actually one of the main drivers why in the observability industry, we moved from the very traditional, I don’t like the word pillars, but the very traditional telemetry types of logs and metrics with the occasional sprinkling of profiling to rather extensive use of distributed tracing because all of a sudden the problem was less often in the single simpler components and much more often in between.
But this is kind of a funny conversation to me ‘cause I don’t know if you remember, but 10 years ago we used to talk about microservices as well, but then we called it service-oriented architecture and it wasn’t the new cool thing. And then we decided that that was kind of cumbersome and powerful to work with. And we started to merge together our software, again, introduced monoliths.
But in a service record architecture, we tended to have a very expensive, very large piece of machinery called the enterprise service bus that actually was providing to some extent that kind of visibility that today we need to have through tracing.
And let’s also not discount the effect of the proliferation of cloud providers, because when you have your own data center and your audiences, maybe your internal company or something, maybe it is a good thing to go from one huge complex monolith to three simpler services. But when it’s not three, it’s like hundreds, maybe then you start needing like another layer to orchestrate and manage and all of the traffic between each of those components. It’s definitely not that clear cut to say which one is better.
I definitely think you’re bang on in that case, Nicole, ‘cause we’ve moved from maybe having a couple of software devs for a company that did some kind of HTML that website with some access database backend in the worst cases, right? We now have these huge teams of hundreds of developers that are all working on a compost system together. And having that as a monolith would in practice require us to either do these huge orchestrations of release trains that no one is able to do in a proficient fashion. Or we would have to have downtime all the time ‘cause we would need to redeploy everything every time we made a change. So in that sense, it really makes sense to split it up as we grow our teams and we grow our development efforts as well.
I have a rather, so coming from a very enterprise background I have a rather newest feel to be on the subject. And my take is that usually technological problems are not technological problems, there are people problems. One of the main motivators that was towards going to the extreme distribution of microservices. ‘Cause I tried to fight back against the Conway’s law. The fact that roughly the architecture of your system mirrored the architecture of your organization. So we said, look, we have all these teams, these pizza teams, it’s very hard to make them cooperate to create a well-behaved monolith. So let’s give each of them their independent infrastructure and talk over standardize protocols like most of the HTTP and some sort of RPC built on top of it. And that would provide for them a way to be productive independent of each other.
Exactly.
As an industry, we dramatically underestimated the cost and complexity that comes from the need of coordination and compatibility and understanding among these teams, because now you no longer have one code base that you can run tests against. You have to have multiple ones that may interact surprisingly at Atlanta. These for example, has led to introduction of contract based testing, where teams contribute to each other interface tests. And there’s also the saying, which is something that is a harsh reality that every observable behavior of an application with a large enough user base, will become a dependency for somebody somewhere.
I think this is the whole premise of Conway’s law, right? The idea that our architect, the software architecture or the infrastructure that we put into place in a lot of ways starts to evolve depending on how our team is structured. And I think it’s both a technological and a human problem as well. When you talk about coordination between teams or between services, it’s really the same thing.
Yeah. We have made actually strides in how easy it is to make software inter operate with each other. If you go back, we started, we were talking about SOA. SOA came with a curse called SOAP. It was incredibly complicated, mindbogglingly hard to make run at scale. Today, we have a bunch of, so we are using HTP for other things. Sometimes I would think too many, we have good stuff like GRPC, like arrow, a lot of different protocols that can be very efficiently used to either make a synchronous message passing happen or synchronous invocations. We have a lot of tools that in the past year and half, I think we’ve got the big two trigger happy in deploying a lot of new tools in production. And I’m saying this with a background of someone that fundamentally had to support every single piece of technology that made input or output, for example, on Java. It would surprise you how many different HTTP clients for Java are I think in production, I lost count. There were 50, it’s insane. And I’m, for example, you raised your mark right with a very, very beautiful LED and Bluetooth. And it’s super cool. I feel–
I was trying to share something that Simme was sharing there, but I can’t actually see it for some reason. Oh, there it is.
It’s okay.
What’s this Simme?
I love to say, I like to bring out this picture whenever we talk about why observability is needed. ‘Cause while it’s a lot larger than most of us operate with, and it’s still very striking that they think this is supposed to be Netflix’s is a microservice architecture at some point, whether it is or not, I don’t really know, but it’s supposed to be and trying to–
Can I be trashy? Can I be able to trashy?
Thank you.
Architecture has an element of deliberation that they do not believe there’s an actual out of the system has, this is an emergent system is a lot of architectures that then meshed into something.
Definitely.
It’s remarkable that it works so well.
Yeah. And one of the cool things there, I think is the fact that without observability where this big ball of yarn, it would be completely unfeasible to reason about anything this system is doing at scale. You wouldn’t be able to reason about the particular part, maybe that you are working on one of these small dots that we can’t even see because of all the lines. But to reason about what it does is as a whole would be completely unreasonable. But with tools like, for instance, distributed tracing, we’re suddenly able to follow end to end an action of a user or an agent within this system. That was all I wanted to think of that.
I love distributed systems, distributed tracing in distributed systems, because it actually gives you the means of answering the most important question or you’re faced with an issue, who is affected and how badly? When you go on an enterprise setting and our people are often databases service, your application completely blowing up, could be a drop in the bucket of their system. They will not probably have the central entry you have to have a very different take on priorities unless you have agreed upon a slice on the silos and the failure that you are fundamentally up streaming through your users may not be their concern, but the moment you have the shooting tracing, that goes from the deepest database to the device in the face of your user, yet you have a very clear perception of how bad is something that is the one most important piece of information that you need in front of your face when the pager goes off again at 3:00 AM, you need to decide whether it’s time to go on to Batman, or if it can wait until the third coffee tomorrow morning.
I think that’s the real success of the “Squid Game” right? Because that image that you had up Simme, and just imagine that times 114 million users that are potentially using many of those systems, it’s amazing that it works at all at that scale, especially.
Something that always gives me those thoughtful moments of, oh my God, what are we doing as an industry? Is that a potentially simple issues have disproportionate impact on popular services to the life of many, many people. It goes from lack of entertainment. You’re there you want to see the latest, “Witcher” episode, cool stuff. You’re disappointed if the thing is on top, multiply that by a medium people that cannot get there, there are fix, Hey, you have a significant amount of life around that has gotten worse, but it goes also to something that is maybe much more limited, but much more tragic. For example, when I was working at SAP, one of our customers was, I cannot say the country, but it was the first aid response service of one of the countries. We were scared plus that something so important that either the platform, because downtime would have consistent consequences to real people, something that we have seen recently with the move to the cloud and the disproportionate role that for example, U.S system one has on everybody’s life today. We have seen how the risk is getting more and more concentrated into fewer and fewer, very large, very complex, very reliable, but critical pieces of infrastructure, in ways that I do not believe as a society we have had yet. And that is where it really humbles me as an engineer to think, well, this has to work, right?
Yeah. We went through that in, I think in the last k6 Office Hours, I had somebody from oil funk, who is an emergency services provider in Austria, and it was cool, but also a little scary to find out that they are using k6 for, in part to also handle the traffic from the 112 emergency hotline. So it’s cool to see something that you’re working on, be used in such a critical industry, right. But at the same time, it’s also a little scary because it could actually be lives that are lost if you know, we don’t do our jobs well enough, it’s a little humbling to think of that.
I guess the closest resemblance we’ve had to these critical infrastructures that everyone relies on would it be like the transatlantic Telegraph cable? I mean, that has like in, I think it’s currently 19 cables or something like that, that make up the transatlantic telephone cable. But I really any catastrophic failure there would really isolate us in terms of cross continental communication.
But that is less likely to result in loss of life. Then, for example, your neighbor want service, not being able to follow you.
For sure.
I’m sorry for having completely spoiled the mood. I think that as an industry, we need to go in deserve the role of engineers. If we built bridges the way we push software to production, in some cases, I don’t think anybody would cross a river without crossing themselves first.
What’s interesting is in that spaghetti soup of microservices that you showed Simme, now we’re seeing that our observability stack is part of those. They are also contributing back to the system that we have to monitor. I don’t know if either of you saw the post-mortem by roadblocks.
It was classic. I had ops to them. It’s a, we have all been there.
Yes yes. Do you have to talk about it Michele?
I do, but I don’t think I would be able to be nearly as good as a Simme for this.
I just came to be honest. So I don’t have any detail commentary on that either.
It was the classic fallacy of oh jolly, oh golly. The same infrastructure that is powering my observability system was probably the system that went down.
All right. All right. Yeah, that really is something that you’ll see a lot. I mean, for instance, say that you have a deployment of the classic Grafana suite somewhere, and you have that running in the same data cluster maybe, or at least the same cloud provider as you have your critical workloads and to make it even worse, you could go down to having it in the same geographic region, or even on the same cluster nodes or the same hardware that you’re using for the rest of your critical infrastructure. When something goes south there, in terms of your critical infrastructure, it is likely that that was all will also, as I think, where to kick would roadblock, spill over to your monitoring system. And if that happens, how are you gonna be able to tell that you’ve been hit?
This is the classic blast radius question. Everything has failure modes, actually, several of them, and the more abstractions we deploy, the more failure modes we introduce, for example, Simme and I are building an observability system based on Kubernetes natives. Our guidance is if you deployed, deployed to a different cluster on different hardware, because you should never put the monitoring system on the same infrastructure, close to the infrastructure stuff we monitor.
It’s not only that. We also usually recommend that if you deploy our monitoring system, deploy another monitoring system that makes sure that your monitoring system is up and set them up in a, what did they call it? That dead mans trigger?
The dead man switch is the idea that, for example, do you know in the movies when the hero goes in front of him and says, if you kill me, you will be exposed in two days time. That’s the classic deadness which is something that happens unless someone periodically goes there and tells him not to. In this case, in the case of monitoring is unless your system, your monitoring system is telling you that it’s fine, assume tragedy. It never goes to the level of a specific use cases, but first of all, you need, let me take a step back. The best monitoring system is the one that tells you whether it’s something to look at, right? You actually do not want to have dashboards, however, colorful and beautiful in front of your face all the time, unless it is for Ferguson purposes, you actually want to be able to focus on something else and jump into the action when stuff is hitting the fan. In order to do that, you must have the guarantee that is the easy model of notification. You will receive it because silence, you want to know that it means that it’s good. You don’t want to have to doubt if there is something going on that my sister should tell me but doesn’t, which is why you set up fundamentally a health check thing that says I’m thinking here and all it’s fine. That is the diagnose switch.
I found it so interesting to read through the post-mortem and props to them too, by the way, for releasing something so detailed, they were not afraid to, you know, say what they’d done wrong and what they were doing to address it. I thought it was really well-written, but I want to read one part of it, because I thought it was so good. They said, “There was a circular dependency between our telemetry systems and console, which meant that when console was unhealthy, we lacked the telemetry data that would have made it easier for us to figure out what was wrong. We’ve removed the circular independency, our telemetry systems no longer depend on the systems they’re configured to monitor.” I thought that was a great takeaway from that whole incident. And because they’ve shared it in the post-mortem, they’re not the only ones who can learn from it.
Very generous of them to expose themselves that way and actually left the rest of the industry learn from their mistakes. And so hard work as you said, Michele, our love to the team at the roadblocks who probably need needed a bit of time out after that.
But there is also another component of a human component here. They did the right thing. They reestablished trust. They are telling us, we learned from our mistakes so that you can as well. In the scope of monitoring systems, trust is the ultimate currency because a monitoring system is literally a measurement system for the health of other systems. If you cannot trust your measurement system, what is the point of it?
And you usually only get one go with that as well. If you burn that trust, it’s gonna be very, very hard to build up again.
So this is something that touches me very personal because I used to work on a solution that was exquisitely powerful, but therefore also very complicated and complex. And the moment you have automation, someone somewhere sometimes is going to be broken by it. And reestablishing that trust was never easy and sometimes impossible depending on the scale of failure that you’re well-intended means of automatically monitoring something I should have brought to the customer. Something that we seldom say as an industry, when we try to sell a product to somebody else, is that everything has a cost. If you want to take away complexity from you to roll out very simple systems and set up that thing, reports stuff over there, and you want it to automatically, there’s a higher risk that the automation will cause an issue. Of course, that in many cases is a justified risk. You can roll out the monitoring gradually and the easier it is to roll out a monitoring system at scale, the more you reach the holy grail of monitoring, that is the network effect. The moment that you can start observing how issues in systems propagate, correlate, or are not correlated there is something very important.
So let’s talk a bit more about the characteristics of a good observability stack. We’ve already mentioned a few we’ve mentioned that, it should be independent as much as possible from the systems that they’re trying to monitor. We’ve also said that maybe we should also take into account the human side of things rather than just the technological. Maybe you should tailor the complexity of your observability stack to the size and experience and structure of your teams. What are some other things that you both could name in your ideal observability stack?
One thing that I really like to highlight here is the ease of use or the economics of the system that you are using. I used to do a analogy to how EpiPens work for people with allergies. If you’re eating a peanut and you’re allergic to peanuts, you don’t want to have to assemble the cure for that, right. Or something that alleviates or takes you out of immediate danger. You need it to be simple, just as take it out, push it in your leg, and it’s done. You don’t have to think about it. The same goes for monitoring or observability systems as well I would say. If you have to, when you’re awakened by a notification in the middle of the night, and maybe you’re even hung over or worse, and you are supposed to fix something that is critical, and you have to start thinking about what to do, then your observability system has in my eyes, at least failed you, you need to be able to get a notification that says this is wrong, here’s the data that proves that, click here to get a closer look and just be presented with all the information you need to be able to make a decent decision on how to proceed.
One thing recently on that note that I’ve come across is how potentially easy it is for a new Blake me to set up a whole observability stack or at least monitoring, but he could definitely expand upon it is EBPF, because we’ve been talking about distribution of systems and also of observability related services, but there’s also a trend of going lower and lower level, right? And EBPF monitoring is monitoring at the kernel level, recently with some people from the k6 team, Artem Charlie, Kevin, Danielle, Lopez, shout out to those guys. We did this hackathon project actually for EBPF monitoring. And the whole reason that I got into it was because the promise was you install this binary, you drop it into your cluster on your local machine. And then it just kind of exposes magically all of these things that I normally would have had to set up like node exporters for, you know, it was we’re living in a golden age of monitoring and observability I think. The things that are possible, like I can’t even keep up anymore, be talking about ergonomics. I mean, what is more ergonomic than not having to do much of anything.
You are talking about the day one, the fact that it is so easy to set up apart from monitoring system is exactly what you need to be able to monitor at scale. It removes the thought. EBPF is a fantastic technology. We are fundamentally given a virtual machine to run inside the kernel to execute logic, that allows us to do in a centralized place. Things that in the past were almost impossible. For example, the first thing I ever did with a team of vinyl in Stanhope with EBPF was actually to detect how a process that, that was impossible in user land. But with EBPF, it was actually pretty simple. We literally spent more time making sure that the techniques would work across all the operating systems that we need to support and to figure out how to find out that. EBPF the, I see two big potentials for that, for things that are not possible already today with sufficient tips. The first one is to give us a distributed tracing tool that largely independent from their own time. Now there are technology limitations there. It’s hard to find out how to trace context. I’m sure that someone somewhere out there is figuring it out as we speak, probably, that we reduce to some extent the toil that is necessary to support the federal fire hose of new technologies, libraries, and stuff like that is put in production every day. But the second one that excites me even more because the shoot the tracing is largely already possible is actually to have cheap enough in terms of overhead and consistent enough across run times production profiling. People look at distributed tracing as a way of figuring out what is slow. And it works remarkably well as long as the bottleneck is in between processes, the moment you want to drill down, why this requests into the insight, this process was taking too long. You see what you’re tracing is not so suited anymore. You end up looking at the leaf instead of the forest, because there is so many more requests going on through the system, that unless you have a lot of spans that are very, very detailed and fundamentally to trace threat level profiling, you are not going to which costs a huge amount of overhead. You are not going to have the information needed to find out, okay, good stock in the Dow library again. However, the moment that you have things, basically you have to find out where the CPU, memory or location, memory usage and wait time hotspots are in your application. You actually have the first places to work at to improve the overall system. It’s not gonna tell you why that request was going badly, but the moment you saw the hotspot, you are virtually guaranteed that many requests are going to get better. And then you can measure them again in terms of input output, observability tracing. Now people out there are going to say, Michele, what the hell are we talking about? Today we have a lot of amazing technologies to the production profile and they are true because for example, they are right because for example, if you look at the Java virtual machine with, for example, the Java flight recorder, you have excellent facilities with super low overhead to figure out what is your Java or cuddling or cradle program doing true. But they’re very specific to the particular run time and to a particular programming language. And when you start comparing what wait time, for example, looks like in Golang as respected over to machine with code routines, instead of, I dunno, loom threads, project loom, and all the rest of the stuff, you end up actually having telemetry that seems to say the same things, but in reality has a different semantic. And that is a challenge in our systems that have more and more grown polyglot. When we had the, our happy little monolith fault, it was one language. Now this year the systems are implementation of many different languages that behaves differently, and yet they are operated in some cases by roughly the same people, for example, necessarily teams, right? Having fluency in terms of performance across several languages is a significant challenge. Now, if we actually get through EBPF production profiling, for example, the folks behind parka and also the folks we have big pixie, there’s an amazing job there. If we manage to have something that will give us close enough, precise enough production, profiling information across whatever we may wanna throw into a container that is going to be a generational change in terms of making complex systems run better at scale.
That’s something that we’re interested in at k6 as well. I mean, I’m a performance Sesser so the first thing I think of as the network side of it, there are a lot of network metrics that could be gathered at that level. And the reality now is that without EBPF, you’re in this position where each new service has needs some other exporter that you need to install. And like there’s so many plugins to plugins and many different combinations of things, when really they might all be using HTTP. Like, why don’t we have something that just detects, maybe not, it won’t be everything right. It’s never EBPF has never going to completely be able to, or, well, I shouldn’t say never because you know, that could change very quickly, but I don’t see it as being the end all be all solution. But if it can provide some of the more common metrics, that’s already a step in the right direction. Talk about usability and ergonomics right?
And also provide a safe assessable baseline across a lot of different technologies working together.
Yeah. For sure. What other things do you think would you have in your ideal observability stack?
You asked me?
Either of you?
Go ahead Michele, I don’t have a lot of ideas on that.
Well, actually, if you go on the, okay, can I make us most shield for a link? If you go on google, WWW.ubuntu.com/whatisobservability/whatisobservability. There you see a kind of overview of the things that I find most meaningful, monitoring different types of systems. Second, I’m gonna give here that your own.
They can share my screen record.
I assume you want this one?
Awesome.
Yes. So this page is fundamentally a work of love of Simme and I, about the way we see observe a PVT. And the diagram that you see on screen is a, of course, simplified and very generic can your manage me if I lie, but it is a thirst overview of the kind of telemetry that you use at different levels in your infrastructure in order to monitor things. This is important for filamentary two reasons. One, is to provide guidance for people that may be new to observability in terms of which things to focus first, when you look at the telemetry types that are used on the right hand side, you will see that metrics and logs are of course universal. That’s what we started with. They are very good. Every software that deserves to run somewhere is capable of emitting clogs. Most of them can emit metrics and more and more, thanks to primitives that is really proud. There are the metrics sometimes serious thing to the forefront of enterprise systems. It didn’t used to be that. Then you start seeing that profiling gets, for example, the moment you move above the virtualization hardware, and you start talking about processes and applications, profiling them makes perfect sense.
Sorry Michele for cutting you off. But I think you might need to need to explain the icons.
That is correct.
That’s really. We don’t have that context.
Ah, they don’t explain themselves. Maybe then you use amino the icons.
Yes.
So the chart that you see on the left hand side is metrics, mentors, time series fundamental. That’s what we use to monitor things. There’s a lot. Then the table with the reds and greens and yellows followed by text that locks, the old, that bar chart is actually supposed to be flamed charts and that’s profiling showing hotspots. The strange thing with all those horizontal bars that is distributed tracing, that is the best way I can come up with to visualize spans. Spans is the basic data structure in distributed tracing. It describes an action taken by system to process a request. For example, you would have a span for the receiver of an HTTP request. The span is opened is created when your HTTP server is receiving the request, it’s going to be closed, but it simply is doing amazing things to get to the image in a screen–
Most of this you’ve probably seen in your browser, was what I was about to say, like something similar when you browse on webpage, you’d get this span alike view of how the different correlate.
It’s similar, but there is a key difference. One of the key concepts of distributed tracing as coming from dapper, which is fundamentally the progenitor of all modern distributed systems is the parent-child relation between spans, which is the closest you can get to cow’s reality. We are, for example, when your application is receiving an HTTP request and is issuing your query to Mongo to retrieve some data, you will have a parent’s span for the receiver of the HP request. And the child’s span that says, I spoke to Mongo, took me this longer. And this query, I got so many data back that is the parent-child relation. And it fundamentally, you would not have spoken to Mongo if not for the incoming request. Now things get Merck here when we talk about a synchronous messaging, because the beauty and simplicity of the trees of spans that you have in synchronous systems, kind of get lost with a messaging the moment that you have batch consumers. You’ll have many requests that result in a whole bunch of messaging being queued up in your messaging queue. And one application consuming messages originated in different traces all at once, fundamentally creating a merge, but I digress. I’m sorry. So we were talking about… I just get so excited about this future tree.
No, it’s great. I know that you said that you have a hard stop and I’d like to get you off a little bit early for your meeting Michele, but why don’t you talk about what Canonical has to do with observability at all?
Well, can I call does a lot of things, including a very beloved operating system together with an amazing community of volunteers, Ubuntu, we are responsible for a huge amount of software. We have offerings for Kubernetes, for OpenStack, for SAF, for a lot of systems that people use to run their applications and data centers. And it is the mission of Simme and I to make sure that those systems come with excellent monitoring capabilities, that you can monitor them with open source software, whether or not you want to use the monitoring system that Canonical provides you. At the end of the day, what we’re doing is in variable source fashion. We want to make sure that all the appliances that you can get from Canonical, the SAF, the OpenStack, the Kubernetes is, there are the cube flows and all the machine learning. They are, you can monitor them with best-in-class open source monitoring tools like videos, like getting logs out of SIS log, the security traces with open telemetry, where it makes sense. And that is the part of the work of Simme and I, that is about stepping up the game in terms of making software provided by Canonical observable in a way that you can use, for example, with your existing monitoring system, we really believe every single word we say, when we say the metric, once France, it’s all about having radio systems be monitored together. Now, if you do not happen to have a radio monitoring system, wherever monitoring the rest, we can give you one, but if you have a monitoring system and you’re comfortable with it, you by all rights should be able to monitor whatever piece of software you get from Canonical with it. And this is what to do.
That’s awesome. I wish we had more time to talk about this, but maybe we can have you both in some future episode, I feel like there’s so many things that I wanted to dig deeper into, but thank you both for coming.
Thank you so much.
I am so happy. Simme I’m so happy to have you back on again. I’ve always enjoyed doing these with you, so it’s a little bit weird that you’re now on as a guest, but I’m happy that you’re also bringing things that you’ve learned at Canonical back to us. And Michele welcome. I’d love to have you–
Thank you.
Back again for sure with or without Simme.
Anytime. But Simme is finer yeah.
All right. Thank you.
Sure. Sure. Thank you everybody for watching and I guess we’ll see you next week. Bye everyone.
Bye, have a wonderful day.

First Look at Estonia | Getting to Tallinn (Lucky Dippers)

Thu, 20 Jan 2022 20:36:59 +0100

In December 2018, my husband and I met up with some friends and went from Valkenburg and Maastricht in the Netherlands to Riga, Latvia; and then to Tallinn, Estonia. Three countries in one day!

An Overview of Software Testing

Wed, 19 Jan 2022 17:10:11 +0100

Transcript

NICOLE: What is the difference between good software and bad software? Is it features or user interface or marketing? In my totally biased opinion, what actually makes a difference is how well the software was tested. But software testing encompasses a lot of different areas and it doesn’t even always have to be someone with a word tester in their job title that does the testing. So whether you’re looking to get started in the industry, or just trying to cross skill, here’s an overview of the main types of software testing.

Is it just me, or does your desk get that clutter too after a week of work?

Anyway, software testing is the process of validating whether an application works as intended, but there’s a lot of vagueness in that statement. One of the broadest ways to categorize testing types though, is whether it’s functional or operational.

Functional testing is more about application features. Is there a form to capture user email addresses? Is there a button next to it that submits those addresses? Once the button is clicked, where do the user details go? Do they go to the database? Are they stored correctly? Those are the sort of ways in which functional testers think. Operational testing traditionally called non-functional testing, is a way of verifying software quality and aspects that extend beyond the features of the application.

Operational testing sometimes involves concerns that are more subjective or qualitative. So it can be a little bit more difficult to test, because those things only arise in certain conditions. I’m saying operational testing here, instead of non-functional testing, because I was talking to Paul Bruce who made an excellent video just on this topic. He talks about why the term nonfunctional requirement is really a bit of a misnomer.

PAUL: It relegates all these other things, security, performance, installation, deployment, accessibility. It relegates them to a nice to have, so it should be functional versus operational.

NICOLE: So I’m using his term operational testing instead.

Here are the different forms of operational or non-functional testing.

First there’s accessibility testing. It is involved with verifying that end users can consume or navigate content. Usually through assistive devices like screen readers. While most forms of testing begin with a statistically normal user in mind, accessibility testing seeks out outliers in terms of location, linguistic background and disability.

Usability testing is designing with user experience in mind, specifically concerning the placement of on-screen elements and the intuitiveness of the process flow of an application.

Localization testing is a way of making sure that applications are usable in the local context, in the sense of being colloquially appropriate for users in different countries and from different cultures.

Security testing verifies that a system has no flaws that could be exploited, to gain access to confidential information or features that would not otherwise be accessed by expected means.

So that was me talking from my home in the Netherlands, but this week I am in Portugal. The remote working life, am I right?

But anyway, performance testing is my favorite topic and my favorite type of testing because that’s the one that I’ve had the most experience with. Performance testing is more qualitative. So it’s more about how well the application works. And there are two main types of performance testing and that’s front-end performance, and back-end performance testing.

Front-end performance testing seeks to verify the user experience at the interface level. So that’s everything that the end user sees on their local machine.

There are different approaches to front-end performance testing, and the first of them is performance profiling. Performance profiling involves running your application through a series of automated checks. And there are a lot of sites for this, where you just put in your URL, if it’s a web app, and you get a score in the end. That tells you how performance your application is, the most popular example for this and also the easiest to get started with, is DevTools because that’s already in Chrome and Firefox based browsers. But you can also go to sites like Webpagetest, or GTMetrix, YSlow, which is a plugin, or you can use Google Lighthouse, which is now included with Chrome Dev Tools.

The next approach to front-end performance testing is called RUM or Real User Monitoring. So the difference is that with performance profiling, that’s often done manually, and as a one-off thing, like maybe at the beginning of a test or during, but Real User Monitoring is not usually done within testing environments. As the name suggests, these tools measure the experience of real users. So they have to be run in production. Examples of tools in this space are; New Relic’s RUM, Elastic has a RUM as well, there’s Catchpoint, SpeedCurve Uptrends and many more.

And the third approach to measuring front-end performance is by using some sort of browser-based automation tool. So typically this has been something like Selenium, but that’s falling out of favor in terms of more modern solutions like Playwright, Puppeteer, Cypress and so on.

So front-end performance is about everything that the user sees on their end, but back-end performance testing is about everything that they don’t see. It’s about the network. It’s about latency, the application servers, the way the infrastructure is set up and all of those things that are a little bit more difficult to look into, but can also have severe bottlenecks. Back-end performance testing seeks to verify aspects of an application such as scalability, elasticity, availability, reliability and responsiveness.

Scalability is the application’s ability to respond to changes in demand. This could mean scaling up, which means increasing the resources available to a server, but it could also mean scaling out, meaning increasing the number of servers that are available to handle the application load.

Elasticity is a related topic to scalability, except where scalability is all about scaling up or out. Elasticity also emphasize a scaling down or in, and that’s because having these services or servers available on the Cloud that are dynamically adjusting can also be quite expensive. So elasticity makes sure that the infrastructure that’s available or the resources available are always appropriate for the size of the demand. So when demand increases, elasticity also checks whether the application decreases it servers or resources in response to that.

Availability is how long the application stays up, despite changing circumstances in production. So this is typically measured by uptime. Availability can also be tested using disaster recovery procedures, and you’re not just testing the application itself, but also the processes around it. So are there enough people that are on call so that they would be able to come in and troubleshoot when there’s a production incident? All of those things are included in application availability.

Reliability checks to see how your application behaves in response to unexpected errors. Reliability basically takes it as a given that your application is going to fail somehow, sometime, maybe you don’t know exactly when that will be or how it will fail, but it’s going to happen. So a big part of reliability and site reliability engineering is trying to expose your application to those failures beforehand so that you can build confidence about how it will behave when those things actually happen in production.

And the last aspect to back-end performance testing is responsiveness. And this is usually equated with load testing. Load testing is more of a technique for testing though. And really load tests can be used to test not just responsiveness, but also reliability or availability or scalability, basically any of the “ilities” in this part of back-end performance testing.

Protocol-based load testing verifies the backend performance of an application, not by simulating real users accessing the application, but by simulating the underlying requests that go back and forth between the client or the user and the application servers. One of the benefits of protocol-based load testing is that it takes up fewer resources on the side of the load generator, and therefore is less expensive to run. Some popular protocol-based load testing tools are k6, JMeter, Gatling LoadRunner, NeoLoad, and Silk Performer.

Browser-based load testing verifies the application by simulating real users accessing it. So it also has the benefit of including these front-end performance metrics. Browser-based testing can also be called browser-driven or UI-based testing. Some popular tools for browser-based load testing are Playwright, Puppeteer, Flood Element and as of a few weeks ago, k6.

Then there’s hybrid load testing, which takes the best of protocol and browser-based load testing tools by combining them in one test. Now this can present some challenges, especially if you’re using different tools for the browser-based script, than you’re using for the protocol-based script. The ideal is to use your protocol based load testing tool, to simulate majority of the load to cut down on costs for the execution and scaling out of your load test. And then you run a handful of users using your browser based load testing tool. So you still get some metrics for front-end performance while that load is being applied by the protocol based tool.

Testing software can be really daunting because there are many aspects of an application that you can test. But that’s because there’s also many aspects of an application that can fail. And that’s why all roads lead to testing and why the success of an app hinges on how well it’s been tested.

If you’d like to zero in, on load testing in particular, check out this video that I made on what load testing is. Thanks for watching. And like they say, here in Portugal, Obrigada!

How to Use Executors in k6: Adapting Load Test Workload Models to Production

Tue, 18 Jan 2022 22:45:52 +0100

Executors in k6 are like traffic cops.

They define the shape and volume of traffic that your k6 load testing script follows during test execution. Executors can also drastically affect your results if you don’t get them right.

Here are all the executor options in k6 and when you should use each one:

k6 executors

shared-iterations: Fix the number of total iterations for the entire test. Use when you want to create precisely X rows of test data during a prep run.
per-vu-iterations: Fix the number of iterations each virtual user (VU) does. Use when you want each VU to use exactly X rows of test data.
constant-vus: Fix the number of VUs and duration, but allow the number of iterations to vary. Use for straightforward tests or shakeout tests.
ramping-vus: Set stages, each with a number of VUs and a duration, and let the number of iterations vary. Use for easiest comparison with other load testing tools and for custom load profiles.
constant-arrival-rate: Fix the number of requests per second (rps) and duration, and allow the number of VUs and iterations to vary. Use when you have a simple load profile with a request throughput requirement (rps).
ramping-arrival rate: Like constant-arrival-rate, but with stages. For each stage, fix the number of rps and duration. Use for spike tests, multiphase load profiles with distinct requirements, and Game Day simulations.
externally-controlled: Fix max VUs and time, and allow everything else to be changed at runtime. Use for disaster recovery, breakpoint, or stress tests where you want ultimate flexibility in adapting your test to real-time results.

Still unsure?

Use ramping-vus. It’s probably what you’re used to from other tools.

Ready to move up? Use ramping-arrival-rate. Rps is a better (but still not perfect) measure of throughput for comparison than just the number of VUs.

When in doubt, look at production.

Making sure your test looks and acts like real traffic yields the best results!

For more details, check out the k6 documentation

Testing tool cheat sheet for load testers who don't just care about numbers

Mon, 17 Jan 2022 16:30:07 +0100

It’s not enough to decide how much load you want to generate. You also have to know how you’re generating it.

Your approach affects what part of your application is tested, the results returned, and the budget you’ll need to run your test.

Here are three main categories of load testing tools:

Protocol-based load testing tools:

These tools simulate the underlying messages sent back and forth between an application and a client.

Choose a protocol-based tool if: you want to generate massive load, you want to load test before components are integrated, you want to test APIs, or you are budget-conscious.

Best in class tools: Gatling, JMeter, k6, Locust, Neoload

Browser-based load testing tools:

These tools automate real user interactions with an interface.

Choose a browser-based tool if: you want to measure end-to-end performance, you want to reuse your browser automation scripts, you want front-end performance metrics, or you have a higher budget.

Best in class tools: Cypress, Flood Element, k6, Playwright, Puppeteer

Hybrid load testing tools:

This approach is a mixture of the previous two. You can do this in a few ways:

Use a protocol tool to generate the majority of the load, and a browser tool for a handful of virtual users (best functionality, highest effort/cost)
Use a browser-based tool for all load (expensive, simplest)
Find a tool that can do both (ideal solution, few options)

Whatever you choose, compare like with like.

Happy testing!

How to use Dataview and Templater to run JavaScript in Obsidian for automating task management

Sun, 16 Jan 2022 16:30:22 +0100

TIL how to run JavaScript in @obsdmd. A friend of mine had an interesting use case: He had a long reading list from his studies, and wanted to use Obsidian to decide what he could read given a certain amount of time.

TL;DR: Here’s the script. And here’s the end result:

Requires: Obsidian. The Dataview and Templater plugins. Also requires a separate markdown page for every chapter or bite-sized section of the book or course, with the metadata time_required in the YAML frontmatter.

Why Dataview? Dataview can parse that frontmatter, grab the value for time_required, and return a list of pages and links. There is a Dataview Query Language, but dataviewjs is more extensible.

Why Templater? I initially wanted to write a script within a dataviewjs block, but:

Dataview results are only rendered in Preview mode (which I don’t spend much time in)
I hate testing JavaScript within Obsidian.

Instead, I created a folder scripts within my vault and selected it as the “user scripts folder” in Templater settings. Then, I created a script, returnStudyOptions.js. Here’s the gist for that.

In a new note, I typed <% tp.user.returnStudyOptions(tp, 60) %>. This line calls the script I created and also passes 60 minutes as the time available.

I selected “Templater: Replace templates in the active file” from the Command Pane, which runs the script and replaces the line with the script’s returned value, which looks something like this:

I also assigned a hotkey to for replacing the template, for ease of use. That’s it! I thought this was a fun way to learn how to use JavaScript in Obsidian. Thanks to @ealameda for the use case!

Building a DM's Brain Attic

Sat, 15 Jan 2022 16:45:22 +0100

Anyone other DMs have this problem? My TRPGs folder, containing paid third-party assets for D&D and other RPGs, is made up of 37,884 items (287 GB).

Here’s how I make sure I actually get value out of this embarrassment of riches: 👇

Mike from @SlyFlourish wrote an article applying Sherlock Holmes’s brain attic for lazy dungeon masters, and he called it the “[DM Brain Attic](https://slyflourish.com/brain_attic.html". The idea is to collect interesting techniques or ideas and file them away for later.

Sounds like a good idea, but for most of us, this attic is theoretical. We read books, we watch videos, and we read blog posts on monsters and settings and plots, and we trust that they’ll all be saved somewhere in our heads.

That’s not how it works. Our minds are good at making connections, not remembering CRs from monster statblocks. So how do we make this DM Brain Attic less theoretical and more practical?

Read and take notes. If you haven’t read everything you’ve bought, you’re buying too much. Read what you’ve got and write down the parts that are actually relevant to you and your game. Here’s what mine looks like for the first Arcadia (@helloMCDM) issue.

Combine like with like. This is the hard part, because it involves creating a central place for monsters, items, spells, etc. This is my attempt to do this for monsters. It creates one page per monster that includes its source. Here’s a monster example.

Here’s an example for items that includes SRD items and Vault of Magic Items (by @KoboldPress). Each one links to a page for the item with full details.

Resurface via search: You may have noticed the metadata in the previous page. The metadata gives me criteria to search monsters by. Here’s how I can query my monster database using the Dataview plugin for Obsidian.

And here’s the result of that query, returning monsters that fit my criteria and, importantly, which of my PDFs they come from.

Resurface via random table generators. Another way that I resurface content is by using random table generators using the Dice Roller plugin for Obsidian by Jeremy Valentine. Here’s my NPC page, compiled from multiple sources.

Each die icon can be clicked to generate a new “roll”. Here’s a gif for how that works, using stuff from Sly Flourish’s Uncovered Secrets, accessible to his Patreons. Each click “rolls” on a table I’ve defined in a separate note.

Is building a DM Brain Attic a lot of work? Yes.

But how likely is it that I’d use my 287 GB of third-party stuff while preparing for my game if I didn’t have this system? Not very.

At least this way, I’m using what I’ve actually bought and putting good content into a very real DM’s Brain Attic that I can actually scrounge around for good ideas in, before or during a game.

If you’d like to know more, check out this video I made about how I use Obsidian to run and play D&D online:

Six Performance Testing Hats

Fri, 14 Jan 2022 16:30:34 +0100

I was reading Six Thinking Hats by Edward de Bono when I realized that the premise could be applied to performance testing:

Each hat represents a different way of thinking, which can be useful when working through issues. Sometimes, wearing only one hat can cause us to fixate on incorrect assumptions, and “putting on another hat” can help shake things up and get to actually fixing what went wrong.

Financial Independence Isn't Just for Old People

Thu, 13 Jan 2022 16:43:46 +0100

5 years ago, at age 31, I became financially independent.

Financial independence (FI) means that if I wanted to retire tomorrow, I could. I’d have to make significant changes to my life to do it, but I’d have a plan, and enough resources at hand to get by.

Unfortunately, just talking about FI is uncommon at best and taboo at worst, even in an industry with higher-than-average salaries like tech.

You might be thinking that I received some sort of inheritance or unexpected windfall. The truth is that 10 years ago, my husband and I were in credit card debt and had no major assets. Now he’s retired, and I have a job I love.

The funny thing is, there’s really nothing magical about how we got here.

Here’s how you can do it too.

Step 1: Track your money.

You can’t fix something you aren’t monitoring.

To this day, my husband and I use (YNAB) to track every cent of our money, whether we’re saving it or spending it. When your financial situation is right there in black and white, it’s hard to ignore.

Step 2: Reduce your expenses.

Too many people focus on earning more, when the reality is that it’s far easier to adjust what you’re spending.

You can afford (almost) anything, but you can’t afford everything– so figure out what matters. Reducing your expenses is even better than getting a raise. Do regular audits of fixed expenses. If you have a partner, consider living on less than one income and banking the other entirely.

Step 3: Automate investments.

Set up a recurring monthly payment for a painful amount of money. Invest in low-cost ETFs like Vanguard’s and then forget about them. Resign yourself to living at your current lifestyle level (or lower), and don’t live it up when you get a raise.

And for those who want a goal post, as I did: Actually calculate the minimum amount you’d need to retire. It may be less than you think.

Money isn’t everything, but it can be. To someone who doesn’t have it, money can buy freedom.

How to Set Goals in Obsidian for 2022

Wed, 12 Jan 2022 16:58:39 +0100

How do you set goals in Obsidian?

New Year’s Resolutions are infamously left undone. So what should you do instead? OKRs.

Setting OKRs (Objective Key Results) instead of resolutions increases your chances of succeeding. Tracking your progress throughout the year also make sure you’re getting things done. In this video, I talk about what OKRs are and how I’ve set them up in Obsidian for the year ahead.

What Tabletop Games Teach Us About Application Perfomance

Wed, 12 Jan 2022 16:30:29 +0100

Tabletop games and application performance have more in common than you might think.

See, it’s all about optimization. Both involve juggling several variables: some that you can control, and some that you can’t.

Here’s what I’ve learned about application performance from playing tabletop games.

#1: Get more workers

In worker placement games, you should always spend your first worker to train more workers.

Similarly, one of the most basic ways to increase concurrency is multithreading: using more than one thread at a time. (See: Golang) This applies to testing tools as much as to systems. The 1 Thread: 1 Virtual User paradigm is flawed.

#2: Spend time on getting your engine going

Get your big combos in place before you tap to attack.

Things like setting up automation frameworks or CI/CD pipelines DO set you back in the beginning, but they are worth their mana cost in the long run.

#3: Draft with others in mind

The trick to drafting is knowing what you’re leaving for others as much as what you’re taking for yourself.

The short game is getting your team to do the testing. The long game is getting everyone in the company involved in testing.

#4: Hope for a nat 20, plan for a nat 1

Sometimes proficiency, expertise, and a luckstone just aren’t enough.

Instead of wondering whether an app will fail, take for granted that it will, and start planning for it. Keep that Action Surge in your back pocket.

#5: Sometimes, you just have to trade for sheep

The 2:1 port is valuable, but not always the best option.

Learn what you can, but accept that you can’t learn it all. Reach out to others with compatible skillsets and ask the “stupid” questions that nobody else wants to ask.

Testing is a team sport, and everyone gets a chance to play.

Happy gaming!

The Tester's Toolbox Is Flawed

Tue, 11 Jan 2022 16:30:49 +0100

Every tester has a toolbox.

In the toolbox rests a variety of tools for every conceivable purpose. Part of our job, as testers, is to choose the right one for the job. Sounds reasonable, right?

That kind of thinking could be dangerous.

“The right tool for every job” fails for multiple jobs.

You end up with multiple tools. One for accessibility testing. Another for API testing. Another for browser testing. Another for load testing. This is the modern tester’s dilemma: we spend more time working with our tools than with the application under test.

In the real world, including a new tool in your testing stack means a new framework, usually a new programming language, and sometimes even new colleagues with experience in those tools. And what about continuous testing? All those tools have to talk to each other at some point.

More tools = more time, effort, and money.

When choosing tools, prioritize multitools.

A multitool can be used for different purposes and prioritizes flexibility.

Here are a few ways to spot a multitool:

#1: It is a polymath. A multitool is good at many things and is limited in use case only by the imagination of those that use it.

#2: It uses what you know. A multitool is built on languages and frameworks you and your team already know, reducing ramp-up time.

#3 It plays well with others. A multitool doesn’t force you to use it to the exclusion of everything else. It integrates well with best-in-class tools.

When you optimize for individual jobs, you’re missing the big picture. You’re ultimately left holding the ends of multiple tools and creating a kraken of a test suite with all these tools cobbled together.

Instead, prioritize multitools.

Obsidian for the Uninitiated

Mon, 10 Jan 2022 23:07:06 +0100

Obsidian for the Uninitiated

You’ve installed Obsidian and created a new vault. Now what?

Unfortunately, it can be more than a little daunting to get started with. It doesn’t help that there are terms like Zettelkasten and Maps of Content and fleeting notes and digital garden that are all just trying to structure what we all learned to do in primary school.

Start by logging what you’re doing.

Having a rigid structure does wonders for curtailing creativity and plain fun. Instead, I recommend starting small.

Here are some small steps to take after installing Obsidian:

Step 1: Enable the Daily Notes plugin

The most game-changing feature of Obsidian is plugins.

Go into Settings > Core Plugins, and enable the Daily notes plugin.

Then, exit out of the dialog boxes and hit CMD P (Mac) or CTRL P (Windows) to bring up the command pane. Search for and select Daily notes: Open today’s daily note. That creates a new note in your vault with today’s date.

From now on, do this every day and log things you do during the day: things to do, appointments to keep, videos you watched.

Step 2: The next time you consume content, take notes.

Now, the next time you read or watch something interesting, type its title into Obsidian with brackets:[[Cool video I watched]]

It’s now a link. CMD+click on that link, or click the glasses icon on the top right to switch to Preview mode, and then click on the link. You should now have a new note. Write about what you thought of the video.

Step 3: Start identifying and fleshing out ideas.

Keep going with these steps and every now and then, start to identify cool ideas you come across. Create new notes from them just like you did in Step 2. Don’t worry about getting it all; just do it when and until you feel like it.

These three simple steps, over time, will be the foundations of your own Personal Knowlege Management system with Obsidian.

5 Ways to Use Obsidian as a Software Engineer

Sun, 09 Jan 2022 19:20:49 +0100

Yesterday, I wrote about why I think taking notes is the surprising foundation of a tester’s career.

But it’s not just testers. Every software engineer is a knowledge worker, and that means we get paid for what we know. Using a Personal Knowledge Management (PKM) system such as Obsidian.md helps us keep up with the constant barrage of new ideas in tech.

Here are 5 things I use Obsidian for.

#1: My very own Stack Overflow

If you code, you’ve used Stack Overflow.

What if you had your own Stack Overflow, one that contained cheat sheets of every language you’ve ever tried to hack something up in, including only commands relevant to your job? This is exactly what I’ve done for Python recently, and it’s helping me solidify my knowledge.

#2: Tools database

There’s a tool for everything.

To keep them straight, I have a page for every load testing tool I try, with tutorials for how to get started, an assessment of its features, and observations I had while attempting to learn it. Here’s what I have for the load testing tool k6.

Bonus: I also use the Dataview plugin to query tools that fit my criteria.

#3: An Agile workspace

If Agile is good enough for my professional work, it’s good enough for my personal work.

I use the Kanban plugin to keep track of projects, create weekly and monthly review/retro templates, and work iteratively using the Incremental Writing plugin.

#4: A public learning log

Learning in public is the fastest way to improve.

When I’m trying to figure something out, I document my attempts. With Obsidian Publish, I can quickly push this log out to my domain and share them with colleagues to get a quick review so that they can easily follow my train of thought and tell me where I went wrong.

#5: A digital garden of software ideas

Build libraries, not feeds.

I use Obsidian to save things I read/watch/listen to but also to save space for ideas I’ve learned to germinate. I publish notes before they’re fully fleshed out, with the goal of growing everything over time.

Today, I have 4,567 notes, all about things that matter to me.

Every engineer should have a PKM system.

Take Notes, Not Tools

Sat, 08 Jan 2022 17:00:44 +0100

When I first got into testing, I thought that testing was about learning different tools and frameworks.

I learned how to use Neoload, then LoadRunner, then JMeter, and then SeleniumSilkPerformerVisualStudioGatlingPuppeteerPlaywrightk6 in short succession. I took courses on JavaScript, on C, on Java. I ticked off tools like a grocery list.

It didn’t take me long to realize that I still wasn’t a good tester.

Taking notes, not tools, is what makes a good tester.

When you think about it, a lot of things a tester does revolves around taking good notes.

Requirements are just notes on what the business wants to achieve. Test scenarios are notes on situations that the team wants to simulate. Test cases are notes on how a user would use an application. Defects are notes on issues.

The quality of a tester’s notes dictates the quality of their work.

Listen to people, not tool tutorials

Take notes on what users do. Talk to the business. Pore through Google Analytics logs. Piece together a story of the kind of people who use your application, and how they use it.

Take notes on problem areas. Talk to customer support about the most common questions they get– they know more about pain points than anyone else in the company. Talk to the ops engineers who were on call during the last production incident. These problems are what you’ll need to test later.

Take notes on what you’ve learned. If you don’t like learning, you’re in the wrong field. Document everything, even things you think you’ll never forget, because one day, you WILL forget.

Taking notes, not tools, is the foundation every testing career is built on.

Top 10 Last Minute Tech Gifts (2021)

Mon, 20 Dec 2021 17:20:19 +0100

In this super last-minute video, I talked about 10 awesome bits of tech I’ve loved throughout this year, and why I think each of them would make great gifts for the techie in your life (or maybe even yourself).

Below are all the items I mentioned. I include affiliate links where possible, so I will get a miniscule commission if you buy something through my link. However, I wouldn’t be recommending these if I didn’t genuinely use and love them all!

Røde PodMic: https://amzn.to/3dKhflp
Røde PSA1 boom arm: https://amzn.to/3sdVSkK
Rødecaster Pro: https://amzn.to/3rYZnv9
Elgato Wave 3 USB condenser microphone and digital mixer: https://amzn.to/3GDo5oV
Keychron K6 mechanical keyboard: https://amzn.to/3s6cScz
Keychron K3 mechanical keyboard: https://amzn.to/3DL35Ln
Omnicharge 20c: https://amzn.to/31RP3KU
Elgato Keylight Air: https://amzn.to/3ynpjSp
Aputure Amaran MC bundle: https://amzn.to/3oOf2vh
You Need a Budget app (YNAB): https://app.youneedabudget.com/gifts
0Kindle Oasis: https://amzn.to/3IODEMr
Yubikey: https://amzn.to/3ETRvyH
Trezor Model T: https://shop.trezor.io/product/trezor-model-t
Kindle Oasis: https://amzn.to/3GIXas7
Leatherman Skeletool RX: https://amzn.to/3m0drks
Bose NC 700 headphones: https://amzn.to/3q3xD5W

How Eurofunk uses k6 for load testing emergency services (k6 Office Hours #37)

Fri, 17 Dec 2021 17:17:07 +0100

In this k6 Office Hours, k6 CEO Robin Gustafsson joined me to talk to Daniel Knittl-Frank, a software engineer at Eurofunk, about how k6 is used in the emergency services industry. Daniel has an interesting use case for k6: he uses it to load test calls in Austria received from the 112 emergency services hotline!

Load Tester Reacts to Pokemon Go Scales for 166m Users

Thu, 09 Dec 2021 16:13:59 +0100

Transcript

NICOLE: Last year, Pokemon GO, a mobile app developed by Niantic Labs, made $1.23 billion in revenue with 166 million players from all over the world. This level of success is unprecedented, especially for an app that was encouraging people to go outside at a time when many governments were encouraging them to stay inside. In tech, though, success comes at a cost. Here’s what Niantic has to do under the hood to keep this massively popular game running.

Hey everyone, I’m Nicole van der Hoeven. I’ve been helping companies through their load testing for over 10 years, and I like to talk about tech travel and taking notes. I’m also a Pokemon GO player. I’ve been playing it since the day it was released in July 2016.

In this video, I’m going to be reacting to a conversation between Priyanka Vergadia from Google and James Prompanya from Niantic Labs. Thank you to Joe Colantonio’s TestGuild News Show last month, for bringing it to my attention.

Let’s get started.

PRIYANKA: Yep. NICOLE: Okay. Everybody knows that you should be doing a curve ball to catch Pokemon the fastest way.

PRIYANKA: I am catching some Pokemons. Did you know that it was built on Google Cloud? Well, let’s talk–

NICOLE: GCP, okay.

PRIYANKA: At Niantic Labs to take a behind the scenes look at Pokemon Go’s architecture, and how it scales for large number of players. Hi James, welcome to the show.

JAMES: Pokemon Go is not your typical mobile game.

NICOLE: I love that he’s got Pokemon paraphernalia everywhere.

JAMES: Catching these little creatures around you called Pokemon that are appearing in a real world. Encourage you to go outside, explore and discover things using augmented reality. A big part of that, which wasn’t in–

NICOLE: See this is one of the things that make me really impressed with their implementation. They have majority of their users using mobile networks. That’s a really hard thing to account for, especially when you’re talking about a global audience that is playing in different parts of the world. So it’s really important that everything is front-loaded, and that they’re minimizing the amount of traffic, the amount of resources that each player has to download on their mobile network.

JAMES: Wasn’t included in the game when it first came out, was this big community aspect of it. We really liked this part of the game. So we made it a major part by hosting regular live events, such as Community Days, Raid Hours,Team GO Rocket Takeovers, all culminates in this big summer event that we call GO Fest.

PRIYANKA: So it’s–

NICOLE: So this is a really brave move as well because they have events that are timed. So they happen at the same time for people in that time zone. And earlier on in Pokemon Go’s history, I did experience some slowness and some congestion in the mobile networks. But I mean, that’s a really difficult thing for them to plan for. I’m interested to see how they solve that.

PRIYANKA: How does Pokemon Go back and actually scale to handle these peaks in traffic?

JAMES: So we use Google Cloud. (chuckles) We have lots of services that need scaling such as the Google Kubernetes Engine and Spanner. Our front-end service is hosted on GKE, and it’s pretty easy to scale the nodes there. Google Cloud provides us with all the tools we need to manage the Kubernetes cluster.

NICOLE: So it’s really cool to hear him talking about their stacks. So they’re using GCP, Google Cloud Platform. They’re on Kubernetes, so they’re using GKE, the Google Kubernetes Engine. And they use Google Cloud Spanner, which is a very highly available relational database.

JAMES: To give an idea of scale, we have about 5,000 spanner nodes handling the traffic and thousands of Kubernetes nodes running just for the Pokemon–

NICOLE: That is so interesting. 5,000 Spanner nodes. That’s just for the data. That makes sense because Niantic’s main reason for creating this game, or the way that they’re making money, I guess, is by getting the data from users about walking paths and …Basically, geo location or spatial data. So it’s super important for them to be able to store the data that they’re getting from their application usage.

JAMES: We also have the various microservices that host other parts of the–

NICOLE: So another thing that they’re doing to keep it scalable is by using microservices. See the traditional way is to use a monolith, so one large application with many sub-components. The problem with that though, is you can’t independently scale those components, you’d have to scale the entire application. And when that application is already really big and complex, that’s really inefficient because you’re scaling everything, even services that you may not need to scale. So it’s really smart that they’ve chosen for a microservices based architecture, so that each of those components can be scaled separately of each other.

JAMES: And if you compare ourselves to maybe World of Warcraft, you’ll see that other massively multiplayer online games, they split players into multiple realms, but for us, all of our players, they reside on a single realm. It was important for us that players can always interact with each other and share the same game, the experience, no matter where or when they’re playing.

PRIYANKA: Yeah–

NICOLE: Okay. So I happen to have also played WoW, World of Warcraft. My main ’toon was a disciplined priest. So when you sign up to WoW, you have to choose a realm, that’s basically a server or group of servers so that they don’t have to shoulder the entire load from all users. So that was WoW’s way to reduce the load. And he’s making the point that Pokemon Go can’t really do that to a certain extent. It’s not entirely correct to say that they don’t have that sort of splintering, because they do still splinter based on the time zone. You’re only seeing people in that location in your time zone.

Even events, they sometimes say 10:00 AM in your local time. So you’re not necessarily playing at the same time that people in Australia are playing. So there is some splintering, but it is less than what there would be for a game like WoW.

JAMES: When we’re looking at different providers, we saw that Spanner has consistent indexing that allows us to do more complex database schemas with these primary and secondary keys. Data Store was also non-relational with atomic and durable transactions. So we needed a relational database with full consistency, and made the choice to go to Spanner, which gave us these global ACID transactions.

NICOLE: So he’s talking about consistency, global consistency, being a big deal and a big factor for why they chose to go and use Google Cloud Spanner instead of Google Data Store. And the reason is that normally, when you have a bunch of different instances of database, it doesn’t matter so much that those requests are processed in exactly the order that they came in. Because usually those requests are being made by different people, and they’re trying to change different accounts. But with Pokemon Go, there’s the added complexity of wanting to keep the integrity of the chronological order. So if I go and attack a gym before you do, my, the process, the request that I created by attacking the gym, should be processed before yours is because it matters who gets to the gym first. So that’s an interesting twist.

PRIYANKA: Now let’s say I’m a player and I’m playing the game right now. I open my app to catch a Pokemon. What is happening behind the scenes? How does the request flow work?

JAMES: When the user catches a Pokemon, we receive that request through the Google Cloud Load Balancer. All of the static media is also stored on Google Cloud storage buckets. And it’s downloaded to the phone when you first start the app. We also have caching enabled at the load balancers. So it’s–

NICOLE: Already, they’ve mentioned they have a load balancer, so that that’s like a traffic controller that’s sitting above the servers and saying, “Okay, this request goes to this server.” And then maybe when that server is processing too many requests, that load balancer should be able to say, “All right, now let’s divert some of the traffic to another server that maybe isn’t as heavily utilized.”

He also mentioned that they employ caching. This is what I was saying earlier that because you want to minimize the amount of traffic and the bytes that are flowing on the network, you want to employ caching for maybe things that aren’t going to change so more static assets. Maybe you might want to change the Pokemon that’s in a gym, but the location of that gym is something that should and can be cached. He’s right now talking about caching on the server side, on the application side, and not on the mobile phone. But I wonder if they’re going to talk about that later as well.

JAMES: It’s all cached and served through Google Cloud CDN. So when the Poke–

NICOLE: So he also mentions using a CDN, so a Content Delivery Network, which is really good because that means that if I’m in the Netherlands and I make a request to their servers, my request isn’t then going to be sent to a server in the US, because it’s going to be sent to one of the servers in my country or nearby. That means that the latency between my request being sent, and that request getting to the server, and retrieving that resource and back, is going to be a lot lower. So it’s … This is a way to improve performance.

JAMES: The player catches the Pokemon, the GCLB sends the request to our GKE cluster. And we have a front-end service that sits behind Nginx reverse proxy. The request goes from the user’s phone through the reverse proxy, to one of these player, front-end services. We also have this thing called a Spatial Query Backend. It’s a cache by location.

So that is where we store the information that determines where a Pokemon is shown on the map, what gyms and Poke stops are around you, what time zone you’re in, or any other feature that is location-based.

PRIYANKA: Now, what happens when I hunt a Pokemon down and catch it?

JAMES: When you catch a Pokemon, it sends the request to the front end, to the Spanner database, where your player entity is stored and it store it there. For catching Pokemon, that actually doesn’t go to the Spatial Query Backend. If you were, instead, if you were battling in a gym or adding lures to the Poke stop, that information is stored on the Spatial Query Backend, and–

NICOLE: Okay. So it sounds like the Spatial Query Backend is only storing data about the world, the Pokemon Go world that everybody can see. So people don’t need to know whether another person has caught a particular Pokemon. They just need to know whether there’s a lure on a Poke stop, or whether the gym is Team Mystic and things like that that everybody else can see. So there’s this segmentation of some data that is specific to the user, and other data that all users need to see.

JAMES: We also use Pub/Sub to send the message to a Pub/Sub topic, which is then used for analytics pipeline.

NICOLE: He talks about a Pub/Sub model, which means that their architecture is event driven. So the traditional way that this is handled is by polling. So normally when an service is waiting for something to happen, it goes, and it sends a message and it checks. Basically like, “Hey, are we there yet? Are we there yet? Are we there yet?”

And then when it finally gets a response that “Yes, we are,” then it can do something with that data. Instead what they’re doing as a Pub/Sub. So instead it waits for events. There is no constant polling, again, reducing the amount of traffic that’s going back and forth. And it just specifies criteria for what it is actually waiting for. And when that happens, after the services that require that event are subscribed, then that event is published to all of them. And then they can then do whatever they want with that event.

PRIYANKA: So, how do you ensure that when two people who are in the same geographic regions, see the same Pokemon data, and keep that relatively in sync, especially during the events?

JAMES: So it’s actually pretty interesting. Everything in our servers are deterministic. So even if players are on different machines, they’re sensitive to different front-end services, if they’re in the same physical location, all the inputs would be the same and the same Pokemon would get returned to both users. There’s a lot of caching involved and as well as precise timing, especially when the settings are changed and they need to be in sync across all the servers.

So it’s very tricky, but it was very important for us that all the players feel like they’re part of this one shared world.

NICOLE: I’m not entirely sure what he means when he says that it’s deterministic.I’m going to guess, and I have no relationship with Google or with Niantic, but I’m thinking that what he means is that when you’re catching a Pokemon, there is no request that’s being sent to the servers to say, “Hey, did I catch this?” Instead, there’s probably some sort of algorithm. This Psyduck is caught 80% of the time and appears 2% of the time. So 2% of the Pokemons that you’re gonna see appear in this area, are going to be Psyducks. And that’s also reducing the load going back and forth, right? Because it’s just the equation or the algorithm that determines whether you catch a Pokemon or not.

JAMES: This game data is of interest to our data science team. They use this for marketing purposes, for verifications. For example, we might want to say that this player, or players caught x millions amount of Pokemon during our Saturday Community Day events. So, it’s important that we have that information in an easily accessible manner.

NICOLE: So there’s a lot of data here that they track. So I’ve got my distance walked, Pokemon caught, I’m at 18,878. Poke stops visited, total XP, start date. And then there are these medals as well. So, they look at things like, how far have you walked in total? So all of these things are data points that they store for every single user. No wonder they have such a focus on databases, right?

PRIYANKA: As the events grow, like you mentioned, in some of these events, the traffic grows to millions of users per second. You also said–

NICOLE: Per second!

PRIYANKA: …of data is generated. How does the system scale when it comes to data pipeline?

JAMES: Yeah. So when there’s increase of transactions, there is increase of load throughout the system with a lot of our data pipelines for things like BigQuery, one of that–

NICOLE: I’m not actually sure she was saying millions of users per second, but that’s a little bit ambiguous. Maybe she meant millions of requests per second. And the way that they’re trying to do that is they’re reducing the size of those requests. If she means there are millions of concurrent users at any given time, that doesn’t necessarily mean that those users are all doing something at the exact same time. So I would have wanted to see exactly how many requests, to hear how many requests per second they’re needing to deal with, or how many bytes per second are being sent over the wire. That would’ve been cool.

JAMES: …Just works. Google Cloud will handle the increase in traffic without any intervention.

PRIYANKA: With that much traffic, obviously health of the system is critical as well. How do you monitor the health of the system during these massive events?

JAMES: So we use multiple monitoring systems, including Prometheus, Grafana, and Google Cloud monitoring. On a personal level, I actually prefer it to Google Cloud monitoring because it’s very simple to use–

NICOLE: Whoo whoo! And they’re using Prometheus and Grafana … Oh, this is entirely unexpected, but I’m actually wearing a Grafana shirt. So it’s really cool that they’re using Prometheus and it really, it makes sense, because they’re on a Kubernetes cluster and Prometheus and Kubernetes just go really well together. By the way, I work for Grafana, for context.

PRIYANKA: What are some of the expansions and improvements you’re looking to do in the future?

JAMES: On the Google Cloud side of things, we are looking into things such as Agones and Google Game servers, and using that to see how we can make our game even better.

PRIYANKA: We wish you and the Niantic team best of luck for these enhancements. And this has obviously been super insightful for us, James. Thanks for joining me today and sharing your architecture with us.

JAMES: Yeah, thanks for having me. This was fun.

NICOLE: That was really good. So there were a few things that were mentioned there for how they’re scaling Pokemon Go, that I wrote down here.

One, they use an event driven architecture and also a microservices based architecture, both of which are way more performant than the traditional methods.And the microservices based architecture will also let them scale up independently as demand increases as well. And they’re also using Cloud Spanner instead of Google Data Store, prioritizing global consistency, as well as high availability.

They front-load a lot of the static assets that are used by the application. And they’re being downloaded when the app is initially downloaded or installed onto users’ mobile phones. That way they don’t have to keep serving the same data if it’s going to be used consistently. And when something changes, like a different lot of Pokemon, they release these patches that let you download a lot of those images so that when you catch those Pokemons, then you already have all that information on your phone. And on a related note, they’re employing a lot of caching. They talked about caching on the mobile app side, and on their server side.

Events are also staggered. So splintered realms sort of, because they’re happening on different time zones, depending on where you live. And not completely simultaneously, regardless of the time in the part of the world where you’re from.

They’re also using things like Prometheus, Grafana and Google Cloud Monitoring. They didn’t mention how they’re actually using multi-tenant versions of Prometheus like, maybe, I don’t know if they’re using Cortex or something like that, but that’s something that they should look into.

And they also mentioned potentially increasing the number of Google Game Servers.

I really appreciated that behind the scenes look. I wish more companies would do this, especially for apps that I personally love and use. One thing that I’m really curious about though, is the testing side of it. Because testing an application of this scale and complexity, is another thing entirely from observability. And I would have wanted to know how they’re doing their performance and load testing. And whether they’re doing things like chaos engineering maybe, or reliability testing to improve the availability and reliability of their application.

If you’re still watching this, then chances are high that you like Pokemon and load testing as much as I do, which is a bit of an odd combination. And in that case, check out this video, on how I run chaos experiments using k6, Prometheus and Grafana. Thank you for joining me as I watched this awesome video.

And as they say in the Netherlands, (Dutch) Until next time! Bye!

What's new with k6 Cloud, with front-end lead Simon Legander (k6 Office Hours #36)

Fri, 03 Dec 2021 17:09:45 +0100

In this week’s k6 Office Hours, I was joined by my colleagues Mark Meier and Simon Legander, front-end lead at k6, to talk about the new features that have come to k6 Cloud this year so far. Simon also gives us a sneak peek of all the cool features coming on the horizon…

Creating a Content Calendar in Obsidian

Thu, 25 Nov 2021 17:01:06 +0100

Despite the fact that I’ve long given up using Notion as my main note-taking tool, I’ve still continued to use it as a content calendar. Obsidian just hadn’t caught up with it in terms of functionality… until now.

In this video, I show my current Obsidian workflow for creating and maintaining a content calendar.

Performance Objective: Intro to Load Testing With k6 and Grafana

Tue, 23 Nov 2021 17:05:06 +0100

I spoke at the Performance Objective meetup to talk about how to use k6 and Grafana for beginners. I always enjoy talking to like-minded engineers, and I think sharing ideas in a meetup context is probably the best way to do it.

Everyone Can Contribute: Load Performance Testing With k6

Tue, 23 Nov 2021 16:57:26 +0100

This is a recording of a live meetup I spoke at, the Everybody Can Contribute Café hosted by Michael Friedrich, Senior Developer Evangelist at GitLab. This was a more casual conversation and show-and-tell about k6’s capabilities, and I really enjoyed it!

How I Use Obsidian for Work as a Software Developer Advocate

Wed, 17 Nov 2021 16:48:22 +0100

I’ve previously written about how I use Obsidian at work, but this video shows exactly how I do it, and all the plugins I use to make Obsidian as useful at work as it is for play.

How I use Obsidian to run and play D&D online

Sat, 13 Nov 2021 00:38:11 +0000

I’ve previously talked about how non-lazy DMs use Obsidian for D&D, and a few weeks ago, I participated in a TTRPG Obsidian Showcase with a few other Obsidian DMs and players. Here’s my part of that presentation:

Resources mentioned

This talk was part of the TTRPG Obsidian Showcase (video): https://www.youtube.com/watch?v=Ovqu_1aW3Sw
Non-lazy DMs use Obsidian for D&D (blog post): https://nicolevanderhoeven.com/blog/20210930-non-lazy-dms-use-obsidian-for-dnd/
How I Use Obsidian for D&D - Player’s Edition (blog post): https://nicolevanderhoeven.com/blog/20210809-dnd-obsidian-player/
Dice Roller: https://github.com/valentine195/obsidian-dice-roller
Initiative Tracker: https://github.com/valentine195/obsidian-initiative-tracker
Fantasy Calendar: https://github.com/valentine195/obsidian-fantasy-calendar
Gallery: https://github.com/Darakah/obsidian-gallery
Dataview: https://github.com/blacksmithgu/obsidian-dataview
Leaflet: https://github.com/valentine195/obsidian-leaflet-plugin
Admonition: https://github.com/valentine195/obsidian-admonition
Advanced Tables: https://github.com/tgrosinger/advanced-tables-obsidian
5e statblocks: https://github.com/valentine195/obsidian-5e-statblocks
How I use Obsidian for work as a software developer advocate (video): https://youtube.com/watch?v=D7e1ud_Dk24

Browser testing and API load testing in the same script with k6 (k6 Office Hours #35)

Fri, 12 Nov 2021 00:28:02 +0000

Earlier this week, at ObservabilityCON 2021, we announced something new: k6 now supports browser-based testing with xk6-browser. Here’s a few of us from the k6 team talking about the announcement!

Resources mentioned

k6 load testing on Kubernetes using the k6 Operator, with Hans Knecht (k6 Office Hours #34)

Fri, 05 Nov 2021 00:23:15 +0000

I was really excited to have Hans Knecht on this week’s Office Hours. Hans works at Mission Lane, and is one of the biggest contributors to a k6 extension called k6 operator. k6 Operator lets you create a k6 custom resource object in Kubernetes.

Resources mentioned

Work week in the life of a Developer Advocate

Wed, 03 Nov 2021 00:31:46 +0000

Before I became a Developer Advocate, I didn’t really know what developer advocates did - so here’s my attempt at explaining what it is I do for a living!

Weekly breakdown

Here’s a list of my activities during this week:

Friday
- Filmed new ending for video #1
- Figured out integration options
- Did k6 Office Hours
- Prepared for Clubhouse talk
- Participated in customer demo
Saturday
- Did post-production for k6 Office Hours
- Talked at a Clubhouse meetup
Sunday
- Wrote (team profile) blog post #1
- Wrote talking points for Racket
- Started writing blog post #2
Monday
- Edited video #1
- Created thumbnail, description
- Scheduled video #1
Tuesday
- Created slides for presentation
- Played with Prometheus and Grafana for webinar
- Had a virtual coffee with colleagues
- Finished writing blog post #2
Wednesday
- Went to four meetings
- Caught up on emails, Slack, and social media (last night)
- Wrote blog post #3
Thursday
- Scripted and filmed videos #2 and #3
- Started editing video #2
- Talked to a potential guest on livestream

Common performance testing problems and fears (k6 Office Hours #33)

Fri, 29 Oct 2021 00:20:27 +0000

For this week’s k6 Office Hours, we attempted to steer into Halloween by talking about some of the most common performance testing issues we’ve faced in our careers as performance testing consultants. For this one, I had Tom Miseur join me to commiserate and swap stories.

Site Reliability Engineering and performance testing with Stephen Townshend (k6 Office Hours #32)

Tue, 19 Oct 2021 00:09:50 +0000

In this week’s Office Hours, I wanted to talk about the intersection of Site Reliability Engineering and performance testing, so I decided to invite two people with different backgrounds from me to discuss it. Stephen Townshend is a performance engineer who recently changed jobs to become an SRE, and Daniel González Lopes is an SRE who had not had experience in performance testing before he joined k6. It turned out to be a great talk!

Resources mentioned

How to install and use the k6 Cloud Grafana data source plugin

Tue, 19 Oct 2021 00:02:05 +0000

In this video, I talk about the k6 Cloud Grafana data source plugin, which lets you use k6 Cloud as a data source for your Grafana visualizations.

Installing and using the k6 Cloud Grafana data source plugin

Installing the plugin for Grafana Cloud

Get your k6 Cloud API token on this page

Grafana Helm Charts repository

Load testing with k6 for beginners, with İnanç Gümüş and Viktor Mihajlović (k6 Office Hours #31)

Fri, 15 Oct 2021 01:52:48 +0200

Did you know that every new person to k6 does a Week of Testing? It’s a k6 tradition for new hires to spend their first week seeing firsthand what the experience of a new user of k6 is like. How quickly do they find the documentation? What issues do they get stuck on? Then, at the end of the week, the new employees do a presentation for the rest of the team to show their first impressions and findings.

In today’s Office Hours, İnanç Gümüş, one of the newest k6 team members, shares his presentation and his experience as a k6 beginner. Another colleague, Viktor Mihajlović, a Technical Customer Success Engineer, also shares some of the most commonly asked questions he’s heard on the job.

Conf42 podcast: in the k6 Universe

Thu, 14 Oct 2021 01:27:45 +0200

Miko Pawlikowski interviewed Simme Aronsson and I for the Conf42 podcast, and we talked about:

why I love to learn human languages, and what Esperanto sounds like
the Grafana acquisition of k6
what performance testing is, anyway
whether everyone needs to do performance testing, and my personal setup for this blog
chaos engineering and Schrödinger’s Pokémon
an overview of the Grafana Labs projects landscape
what k6 is, and how we would get started for free (k6 OSS)

How to make load testing scripts realistic

Wed, 13 Oct 2021 15:17:45 +0200

A good load testing script exercises the functions and components as specified in the requirements gathering phase while simulating real users as closely as possible. Here are a few ways to make scripts more realistic.

Timestamps so you can skip around!

0:00 Intro
1:22 Static resources
2:31 Cache and cookies
3:02 Dynamic think time
5:07 Concurrent requests
5:45 Load profile
6:19 Test data
7:01 Different user paths
7:55 Other forms of testing

Basics of load testing with k6 and Grafana in 20 minutes

Tue, 12 Oct 2021 01:04:02 +0200

This was a video I pre-recorded in preparation for the first Grafana EMEA virtual meetup.

In mid-June (just three months ago), k6 joined the Grafana Labs family. Many people didn’t quite understand what the connection was between k6 and Grafana, but I thought it made a lot of sense.

Google’s Site Reliability Engineering book talks about 4 golden signals that we should measure in a system: latency, traffic, errors, and saturation. Load testing provides 3 of those (latency, traffic, and errors), but it doesn’t show saturation, and it doesn’t give us visibility into the system.

Grafana brings observability. k6 brings data. They work really well together, which is why we’ve been using each other’s products for a few years now, before there was ever any talk of an acquisition.

In this video, I talk about why k6 and Grafana are better together and also demonstrate:

how to run a simple script in k6 OSS
how to make sense of the results of a load test
test execution options: running locally, sending results to k6 Cloud, and running a test from k6 Cloud– all triggered using the CLI
two different integration options to bring k6 and Grafana together: the k6 Cloud Grafana data source plugin and Prometheus Remote Write

TTRPG Obsidian Showcase

Sat, 09 Oct 2021 00:43:25 +0000

Here’s a presentation I participated in about a bunch of nerds who like both Obsidian and D&D. My part begins around 0:45:20, but I thought the other speakers’ parts were really fascinating as well!

SLOs in performance testing, with Leandro Melendez aka Señor Performo (k6 Office Hours #30)

Fri, 08 Oct 2021 22:05:22 +0200

In this Office Hours, I invited my colleague Tom Miseur and friend Leandro Melendez, aka Señor Performo, to talk all about different criteria in performance testing, including non-functional requirements, SLAs, SLOs, SLIs, thresholds, and checks. Criteria at different levels is how we determine whether testing, including performance testing, has been successful, so it’s important to understand what they are and incorporate them into testing scripts and processes.

Testing in Public: How to plan a load test (practical example) - Wyrmwood Gaming

Wed, 06 Oct 2021 11:21:41 +0200

When you’re learning a new skill, there’s only so much that studying can do. The problem is that there’s still a big gulf between studying something and doing it yourself. The solution is having a mentor to bridge the gap– someone who will let you shadow them while they’re executing tests.

I made this video thinking about the me from over a decade ago who would have wanted to get the chance to look over a more experienced tester’s shoulder. In this video, I talk about how considerations for planning load testing, using a hypothetical use case for a real company: Wyrmwood Gaming.

Testing With k6 + Grafana: Better Together (Grafana virtual meetup EMEA)

Tue, 05 Oct 2021 21:49:13 +0200

In this first EMEA session of the Grafana Labs virtual meetups, I talked about how to use k6 and Grafana together to get the full performance picture for a software application. Observability, using tools like Prometheus and Grafana, lets us see what’s going on inside the application, and load testing, using k6, helps us get those insights earlier on in software cycles, such as during pre-production.

How to use k6 Cloud - a k6 load testing tutorial with Bill Rainaud (k6 Office Hours #29)

Fri, 01 Oct 2021 13:58:03 +0200

Bill Rainaud is an inside account executive at k6, which means he talks to users and potential users of k6 every day. In this Office Hours, he explains how to get started with load testing using k6 Cloud, which is a SaaS platform for running and scaling up load tests on the cloud. We also answered a few frequently asked questions about k6’s capabilities and how to do correlation in k6.

Non-Lazy DMs use Obsidian for D&D

Thu, 30 Sep 2021 16:29:54 +0200

Last month, I wrote about how I use Obsidian for D&D as a player in a 4e campaign (I play a half-elf hybrid wizard/cleric). This is about my second weekly game, in which I am the Dungeon Master for a 5e campaign. I’m a relatively new DM (I’ve got 39 games under my belt at the time this writing, and Obsidian is at the center of my workflow for improving my DMing skills, getting inspiration, preparing for a game, running the game, and making sure my players are having fun.

In many ways, my setup is the Non-Lazy DM’s approach to preparing for D&D games: high-effort, high-touch, and high-return. But I only do it because I enjoy it!

Plugins I use

This is going to be a long article, so if you’re just after the plugins I use, here they are up front!

I go into detail for some, but not all of these, in the article below.

Additionally, feel free to skip around to different sections by clicking on the hamburger menu on the upper right-hand corner and selecting the session that interests you.

Outside the game

Although I use Obsidian while playing D&D, I use it most for D&D between games: fleshing out the world, getting inspiration for new adventures, or preparing for specific sessions.

World-building

Many DMs run published campaigns, but I’m not one of them. I don’t think there’s anything wrong with running published campaigns; I just personally enjoy world-building and being able to tailor the environment to what I know my specific players will like. However, I still do read campaign books, for D&D as well as for other systems, for inspiration. I am also a voracious fantasy reader, and consider everything fair game for “stealing” for my games.

One thing I’ve begun experimenting with is processing my fiction reads the way I do my nonfiction reads. What better place to look for inspiration about creating a fantasy world than other fantasy worlds? How could I use a Bene Gesserit-like faction in my campaign?

I recently reread the first Dune book in preparation for the movie coming out later this year, but while I was reading it, I remembered how much I love the world it’s set in. I read books on Kindle, either on my actual physical Kindle ereader or on the Kindle app on my mobile, and I took highlights while I was reading Dune. You can see my full highlights here, synced via Readwise. I highlight turns of phrases I like, character descriptions that sounded cool, or passages that are particularly revealing of the type of world the story of Dune is set in. Below is an example of a quote I highlighted:

Schools were started to train human talents.” “Bene Gesserit schools?” She nodded. “We have two chief survivors of those ancient schools: the Bene Gesserit and the Spacing Guild. The Guild, so we think, emphasizes almost pure mathematics. Bene Gesserit performs another function.” “Politics,” he said. (Location 248)

It really resonated with me as a good characterization of two important factions in the Dune universe.

Then, just as I do with nonfiction books, I create a separate note about the book itself, attempting to summarize the parts of it that might be interesting or useful to me in the future. Below is a screenshot of my page on the first book of Dune. I identify “Key truths” about the world, because I also like to use that terminology when creating secrets or themes for a fantasy world or city I’ve created.

Here are my full notes on Dune.

Then, I also think about tropes or patterns of character, plot, or setting present in the book. I’ve started collecting these in a a page called Forms in my vault. I think it’s useful to spot things that occur again and again in fantasy fiction, whether they’re overused or not. If they’re tropes that I’d rather not repeat, having a “formula” of the trope helps me either avoid it or consciously subvert it in my games. Sometimes, I just take a form wholesale and incorporate it as is.

Here’s an example of a page I created that was inspired by the Bene Gesserit– a short note about “Nuns who think and fight”.

I haven’t been doing it for very long, but it’s already come in useful when thinking of major events and players in my world.

Adventure creation

Adventures are smaller-scale and higher-resolution than campaigns or worlds, so I often need more specific notes than forms.

I read and highlight adventure blog posts and articles using Readwise’s recently announced Reader app, and then they, along with tweets, books, and PDFs, get sent to Readwise and then Obsidian using the Official Readwise plugin for Obsidian. I’ve written about this process in depth, but here’s what that looks like for Mike Shea’s Return of the Lazy Dungeon Master - affiliate link.

An advantage of using Reader (which is currently in a private beta, so I’m not allowed to share screenshots of it) is that I can even highlight text locked behind a paywall. For example, I have a D&D Beyond subscription and I’ve bought all the books available on their store, but most reading apps don’t know that, so it’s been hard to find one that will let me highlight text from one of the WotC sourcebooks. Not so with Reader. Here’s what my highlights from the first chapter of Volo’s Guide to Monsters looks like in my Obsidian vault after it’s synced from Reader:

Readwise is great because I can highlight something on my Kindle and worry about processing it later in Obsidian. I process incoming Obsidian notes daily from 9-10pm– that doesn’t mean I get through them all, just that I have a scheduled time to go through highlights and notes every day.

For D&D-specific notes, that also involves summarizing what I’ve learned into the notes I already have on how to be a dungeon master:

The referencing feature on Obsidian lets me confidently paraphrase without stealing. All I need to do is put a [^lazydm] where I want the superscript to be within the text, and then at the end of the note, I put something like:

[^lazydm]: Shea, M. (2018). _Return of the lazy dungeon master_. [[Return of the Lazy Dungeon Master|My highlights]].

That gets parsed as a footnote by Obsidian and lets me properly credit authors.

When I find good tools that help me flesh out adventures, I try to update my page on DM tools and inspiration, so they’re all in one place.

While creating adventures, I like to use Obsidian Admonition to call out specific parts of the text for me as I’m running the adventure, like this:

Random tables

I definitely have overarching plans about the wider plot in the world at large, but I also like to be surprised. Throwing random elements into games is one of the ways that I can keep myself on my toes and remind myself that I’m not the only one making the story.

There are many random tables in both official and unofficial D&D sourcebooks, but I only use those as starting points. For me, creating random tables that are tailored to my characters and to the things they care about is where it’s at!

I’ve experimented with creating my own random generators for various things, as well as using existing ones like Donjon, but it’s an Obsidian plugin, Jeremy Valentine’s Dice Roller, that I have settled on.

The name “Dice Roller” makes you think it’s only for rolling dice, but it’s much more than that. Its REAL value, in my opinion, is in how it lets me create tables for ANYTHING, and then randomly pull one row out of it. Here’s how it works.

First, you create a table. I had a session where my players discovered a place I called Shipwreck Beach, and had a lot of fun going through several of the ships that were there. Here’s a table of “encounters” I put together for that session.

### Ship encounters
| Roll | Encounter |
| ---- | --------------------------------------------------------------------- |
| 1 | Carnivorous Ship, ToB II p. 56 |
| 2 | Map Mimic: https://www.dndbeyond.com/monsters/1624929-map-mimic |
| 3 | Random curse: p.249 of GM's Book of Random Encounters - d100 |
| 4 | Clockwork weaving spider (1, ToB p. 66) |
| 5 | 1d6 Cobbleswarm (2, ToB p. 68) |
| 6 | a feral little girl wearing a periapt of proof against poison |
| 7 | A large gold coin bearing the crest of the Kryn Dynasty and the queen |
| 8 | Book about the Arms of the Betrayer Gods |
^ship

Note the ^ship identifier at the bottom there– it can be called whatever you want, but you’ll use that identifier when rolling on that table. I also included the Roll column, in case I decided on the spot to let players roll instead of doing it myself.

Next, in another note (or the same one, if you want), include this line:

Shipwreck encounter: `dice: [[Shipwreck Beach#^ship]]`

When you flip to Preview mode, you’ll see this in the place of the line above:

That’s it!! This has made a HUGE impact on my games, because it’s so easy to create or modify tables. I use random encounter tables sparingly during the game, but often use it before a game, to introduce something fresh that I hadn’t thought of.

Session prep

As you might have guessed, I mostly use Sly Flourish’s Lazy DM steps for encounter creation, saved as an Obsidian template:

---
type: session
world: Exandria
campaign: "The Slayers Five"
date: {{date}}
game_date: 836-03-
location: Rumblecusp
characters: [Boris, Lira, Ora, Jamedi, Flint]
tags: [TVZ]
---
## [[Session Summary]]:
^summary
---
## Housekeeping
## Recap
![[^summary]]
## Strong start
>
## Scenes
- [ ]
- [ ]
- [ ]
- [ ]
## Secrets and Clues
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
## [[Loot]]
- [ ]
---
## Log

Before each session, I’ll go through each of these sections.

I add metadata to every session for use with the Dataview plugin later, in case I want to do a search for all sessions that involved a particular NPC or that took place in a certain location, for example.

I fill out the Session Summary section after the game, but I put it at the top so I can immediately see what happened in the session in case I need to refresh my memory on it in the future. I also identify this section with a ^summary so that I can embed it in the next session, as a recap - more on that below.

In the Housekeeping section, I write down things I want to mention before we dive into the narrative of the session. Usually, these topics are scheduling- or rules-related. For example, if my players have just levelled up and I see that one of them has chosen a feat, I might make a note of it here to make sure that they know what it means. My players are pretty new, so I think it’s better to err on the side of more communication rather than less.

In the Recap section, I call the sumary from the previous session and embed it into the current session using something like ![[38_8360310-8360311 Seduction at the Moonlight Chateau#^summary]]. When in preview mode, Obsidian shows just that section labelled with ^summary from the previous game. I like to start the game by either reading out the summary myself, or asking one of the players to recount what they remember. That way we’re all on the same page when we begin.

In Strong start, I write a paragraph or two of prepared text that I use to kick off the story. This is usually the only text I’ll read out word for word during the game.

Scenes are a list of events that I THINK will happen. My players frequently surprise me, which I welcome, so I also list these with checkboxes so that I can keep track of where the story deviated from the plan.

Sly Flourish does recommend a separate section for Fantastic Locations, but I tend to prepare these as separate pages. For example, here are my notes on a possible location, The Charred Circus. I don’t include them within the session note itself, but I do make sure I have a few that characters could go to. Then, I link to all of them from the note I have for the city they’re in, like this one for Port Damali.

Secrets and Clues is where I put a list of 10 things that players MIGHT find out somehow. This is where I look when a player rolls a natural 20 on a Perception check. The idea is to describe what the secrets are, but not get too caught up in trying to plan when or how the players will find out those secrets. I try to include things that might be interesting for the players, as well as a few potential adventure hooks. Here’s a list that I made for a previous game. I have never used all 10 up in one game, but I like having more than I need.

In Loot, I put in any items I know the party will likely get to. If they’re in a boss fight, I usually save in this section all the things I’ve decided beforehand they’ll get. If I don’t have something specific like that in mind, I sometimes randomize a bunch of non-magical trinkets. The Loot page contains items I’ve picked out for specific characters.

Post-session processing

If your players are anything like mine, you’ll often find yourself with a bunch of unused material and a lot of stuff you still had to improvise on the spot, despite adequate preparation. That’s okay! I actually enjoy when my players do something unexpected. I encourage and reward creative play… but it does mean that I have to do a bit of a debrief after each session.

I try not to prepare things that may really be “wasted”. Things like locations, NPCs, or general plot hooks can usually be recycled. After games, I spend some time tying off loose ends, noting down decisions the party made and the reasoning behind those decisions, and tweaking connections between NPCs or places based on how the party reacted.

Fleshing out bits of the world, with each city or NPC getting a page
Tagging cities and NPCs with Location or NPC, and then within the city, doing a query for NPC and city name to get everyone living in that city.

I use metadata for every page in my campaign so that I can use the excellent Dataview plugin later to find whatever I want. You already saw my session template, but here’s an NPC template that I use:

---
type: NPC
faction: []
location: []
world: Exandria
campaign: "The Slayers Five"
date: [{{date}}]
description: ""
race: []
gender: []
class: []
---

I insert this template for every single NPC in my world, whether my players get to it or not. Because I use it so often, I’ve set up a hotkey for inserting templates in the Obsidian settings:

That way, all I have to do is hit CMD + ; and select the right template from the list, and the text above is inserted at the beginning of the note as frontmatter, with all the fields I need to fill out. The date is also automatically generated.

Using the Dataview plugin, I can then embed queries on location pages so that I can see NPCs in that place at a glance. For example, here’s the snippet I put under the People section in my Port Damali page:

\```dataview
table shop as Shop, faction as Faction, description as Description from "Exandria"
WHERE contains(type, "NPC") and contains(location,"Port Damali")
SORT file.name ASC
\```

(If you try this, remove the backticks \ in the snippet above– those are just to show you the syntax for a dataview query.)

That query, when rendered in Preview mode in Obsidian, looks like this:

Having the metadata for every NPC, monster, item, and place in my world is essential for creating queries like this. I’ve also found it useful to maintain a model of the metadata I use:

- `type`
- `NPC`
- `PC`
- `shop`
- `place`
- `item`
- `session` for D&D play sessions
- `quest`
- `race`
- `faction`
- `family`
- `monster`
- `condition`
- `deity`
- `other` for anything else
- `characters` (for sessions): characters involved in this session
- `deceased`
- `true`
- `false`
- `family`
- `faction`
- `race`
- `gender`
- `class`
- `description`
- `date`
- `game_date`
- `world`
- `Exandria`
- `Forgotten Realms`
- `Exandya`
- `campaign`
- `Order of the Wandering Mind`
- `The Slayers Five`
- `Die vergessene Schwester`
- `Temporary White Circle`
- `shop` (for proprietors)
- `shop_type` (for shops)
- `general` : general goods store
- `alchemy`: potions
- `magical items`: sells magical items
- `services`: magical spells, messaging, brothel
- `tavern`: food and drink
- `inn`: lodging
- `herbalism`
- `smithy`
- `location`: city of residence
- `origin`: city of origin

During the game

I keep a “World” page for each of my D&D campaigns, including a list of sessions:

During the game, I use the session page that I mentioned above, and start filling in the Log section. I tend to take way fewer notes as a DM than as a player, just because I’m busy doing other things, but I do always write down things that the players do that I might need to remember later.

One thing that comes up frequently in my games is the need to improvise encounters, NPCs, or locations. I’ve had great success in creating a Random Rolls page that collates many of these random tables in one note. If you click on that link, you unfortunately won’t be able to see the results rendered, because the Dice Roller plugin isn’t available for Obsidian Publish, but maybe you’ll get an idea of things I’m rolling on. Most of the tables referred to are from this Random Tables note, but every now and then I’ll include tables in location notes instead.

I always have this Random Rolls page up as part of my DM workspace, in case I need to quickly improvise something. Here’s what the NPC section looks like:

I can use any or all of those things that I want, or use them as inspiration for something else I make up on the spot. Either way, it means I’ll never be completely at a loss for the name of an NPC. ;)

Initiative Tracker

I use D&D Beyond for character sheets, but I use Jeremy Valentine’s initiative tracker to handle combat because there are a few critical features that D&D Beyond’s Encounter Tracker doesn’t yet support:

Status conditions
(Easy to tweak) homebrew monsters
Hiding monsters’ names from the players

The Initiative Tracker plugin supports all of these. When activated, it appears as a side panel in Obsidian:

I can easily see everyone’s HP and AC at a glance, and also apply special status conditions when necessary. My players don’t have to know whose turn it is– they just have to know when their turn is. I warn them verbally when they’re “on deck”, and they haven’t had an issue with that.

D&D Beyond does allow you to create homebrew monsters, but the only real benefit to doing so is to be able to use it in their Combat Tracker, so I’ve been using another plugin by Jeremy Valentine, the author of this one, to create monster statblocks

Statblocks

The 5e statblocks plugin lets me easily create homebrew creatures just by using this format:

\```statblock
name: string
size: string
type: string
subtype: string
alignment: string
ac: number
hp: number
hit_dice: string
speed: string
stats: [number, number, number, number, number, number]
fage_stats: [number, number, number, number, number, number, number, number, number]
saves:
- <ability-score>: number
skillsaves:
- <skill-name>: number
damage_vulnerabilities: string
damage_resistances: string
damage_immunities: string
condition_immunities: string
senses: string
languages: string
cr: number
spells:
- <description>
- <spell level>: <spell-list>
traits:
- [<trait-name>, <trait-description>]
- ...
actions:
- [<trait-name>, <trait-description>]
- ...
legendary_actions:
- [<legendary_actions-name>, <legendary_actions-description>]
- ...
reactions:
- [<reaction-name>, <reaction-description>]
- ...
\```

(Again, remove the backticks \ at the starting and ending lines of the snippet above if you copy and paste this into your vault.)

I usually start with a “real” monster from official or unofficial sourcebooks, then tweak it from there. For example, here’s a sample one that I tweaked from Tome of Beasts:

The best part about this plugin is that it integrates with the Initiative Tracker. When you create a statblock and save it, that monster then becomes available for use within the Initiative Tracker, as you can see from the screenshot above.

Publishing

I subscribe to Obsidian Publish, partly to give back to the fine Obsidian developers who’ve made a tool I use daily and partly because I’ve found that it’s reduced friction in publishing. I use it primarily for non-D&D-related things, but D&D is part of my life, so it’s there as well. You can find my notes on the D&D campaign I DM here.

Some of the plugins that I use unfortunately don’t render correctly when Published, but the developers are working on that.

I publish my campaign notes mainly because I love seeing other DM’s thought processes and systems, so I thought I’d return the favour by sharing mine as well. In case you were wondering, my players don’t know I publish these notes– or, at least, I hope they don’t!

Player-facing campaign vault

I used to drop handouts into a WhatsApp group with my players, but it became increasingly unwieldy to look for something. We switched to Slack, but that had the same problem of needing to scroll back through lots of photos before finding the right one. Photos aren’t very searchable in those apps.

A few months ago, I switched to Obsidian for handouts. I created a separate vault for the campaign and shared it with my players via Dropbox. I did have to talk them through the setup, but it’s worked out very well for us as a free alternative.

In the vault, I put things like:

Pictures and other handouts
Session videos and summaries
Links for everything we use as a group
House rules

Session page

I create a page for every session we have, which has a section for the video recording of that session (unlisted and just for us, sorry), a player-friendly summary, and a log. The log is where I dump photos I give them during the session. Here’s what that looks like:

I don’t put any other notes in the log; that’s the players’ jobs (if they want to).

I like to have a “Story Thus Far” page that embeds the video and summary sections of all the sessions, so any of us can read through it to remember what’s happened. It’s fun to see how far we’ve come as a group.

About our game

The About our game page has a list of links to all the tools we use, our schedule, and a few things about the campaign and the world.

Rules

I keep a house rules page that shows all the extra rules I’ve put into the game. I mostly play with all the WoTC-produced sourcebooks, but I’ve added some extra rules about things that my players are interested in, such as strongholds (for which I use Matt Colville’s Strongholds & Followers book) and smith’s tools.

I also have some other pages on rules that come up often, such as Ship statblocks (since the campaign is set on a coast) and suggested downtime activities.

Conclusion

I call my approach to game preparation the Non-Lazy DM’s approach as a tongue-in-cheek nod to Mike Shea’s content, but there’s truth to the moniker, too. My setup is extremely detail-oriented, time-consuming, and fun… at least for me! And that’s the key: I wouldn’t be doing any of this if I didn’t find it truly enjoyable to world-build to this level of detail.

If you’re a non-lazy DM too, thanks for sticking around! I hope this has given you some ideas for how you could use Obsidian to run your games and stimulate your imagination, too.

How to load test a GraphQL API built with Hasura using k6 (k6 Office Hours #28)

Fri, 17 Sep 2021 13:58:42 +0200

My co-host Simme and I had invited Adron Hall from Hasura to come talk to us about how to load test a GraphQL API, but he was unable to come due to unforeseen circumstances. Luckily, Tom Miseur, who is part of the Professional Services branch of k6, was able to step in on very little notice!

Tom has done a load testing engagement using k6 that involved Hasura, so he was able to tell us what it is and how it can be tested with k6. It turned out really well, thankfully!

Intro to Load Testing With k6 and Grafana (webinar)

Thu, 16 Sep 2021 22:11:16 +0200

Tonight I did a webinar with Grafana Labs– the first k6-related webinar since the acquisition. I was more than a little nervous to give it, because I knew I was introducing k6 to a lot of new users, but also to Grafana for the first time. I think it went pretty well! I was still quite relieved when it was over, however. :)

In the webinar, I talk about:

why it’s still a good idea to do load testing in pre-production
what k6 is
why k6 and Grafana together make sense
and how to integrate k6 and Grafana

There’s quite a bit of demo in it, and I also had the k6 CEO, Robin Gustafsson, join me for Q&A as a nice treat.

Check out the full webinar here– it’s already up on the Grafana site. You can also go through my slides here, if you’d like.

Schrödinger's Pokémon: k6 and Grafana edition

Wed, 15 Sep 2021 18:22:31 +0200

Earlier this year, I gave a presentation at New Relic’s FutureStack conference about chaos engineering with k6 and New Relic. A few months ago, k6 got acquired by Grafana Labs, and so I wanted to redo it, but with the Grafana stack this time.

Here’s the result, and I think I like this iteration even better:

I presented this talk at TestCon Europe 2021 today, although the audio sync issues were not great, so I recommend you watch the video above instead.

Slides are here.

How Gitlab uses k6 for load testing, with Grant Young (k6 Office Hours #27)

Fri, 10 Sep 2021 18:13:26 +0200

Grant Young, Software Engineer in Test at GitLab, joined k6 OSS maintainer Mihail Stoykov and me to talk about how GitLab uses k6. Apparently, they began using k6 internally some time ago, and because they like to eat their own dogfood and use GitLab to test GitLab, they also incorporated k6 into GitLab so that their users could reap the benefits, too.

Here’s the GitLab documentation about how to do load testing in GitLab, and here’s a k6 blog post on how to set it up!

How to do continuous performance testing, with Lee Barnes (k6 Office Hours #26)

Fri, 27 Aug 2021 09:34:13 +0200

Lee Barnes and I got connected because we are both speaking at TestCon Europe 2021 this year, and I thought it would be nice to have a chat with him before the conference to get a sneak peek at what he’s presenting about: continuous performance testing. Lee is the CEO of Utopia Solutions, and he’s been in the testing industry for decades, so I really appreciated his insights on where the industry is going.

Catch Lee’s talk here and mine here!

How to debug k6 load testing scripts with Tom Miseur (k6 Office Hours #25)

Fri, 20 Aug 2021 14:38:11 +0200

Tom Miseur joined k6 at about the same time I did late last year, and we immediately hit it off because we both come from a performance testing consulting background AND we both had worked for different competing testing tool vendors before moving to k6. I was excited to have him on k6 Office Hours to talk about a topic we both had a lot to say about: how to debug load testing scripts, and specifically k6 scripts. We got a chance to do a bit of a show-and-tell of tools that we use that we use, and they are all independent of k6.

I also got to demo a cool HTML reporting dashboard that Ben Coleman, a k6 user, created. While it’s not baked into the tool itself, it’s fairly straightforward to add it, requiring only two snippets of code that can be pasted onto any script.

How to import Shortform highlights to Obsidian via Readwise

Mon, 16 Aug 2021 21:40:16 +0200

I’ve always been dubious of those book summary apps. You know the ones– they tend to have obnoxious ads that promise to help you “read 5 books a day” or learn the world’s secrets in 15 minutes. Having tried a few of them, I’ve been disappointed with the quality of their summaries (or lack thereof). They attempt to distill the message of a 300+ page book into a handful of bullet points, which I’ve found neither accurate nor useful. Recently, though, I found myself happily subscribing to one, and here’s why.

What’s Shortform and why is it awesome?

Shortform really took me aback because it isn’t just a book summary. I should mention here that I’m not affiliated with Shortform and gain nothing by recommending them. I’m just a happy subscriber.

When I read their guide for the book Why We Sleep, I was pleasantly surprised to find that Shortform not only summarized it succinctly, but they also added intelligent counterarguments to the author’s points. In some cases, when the author made a false claim, there were Shortform notes that linked to studies that showed why the claim was false. In other cases, Shortform mentioned related concepts proposed in other books that strengthened the author’s viewpoint. I found myself looking up their guides of books I’d already read, just because I was curious to see if I had missed anything. And I had!

In short, Shortform is the only book summary reader I’ve found that goes beyond summarizing and actually adds value to the reading experience, even if you’ve read the book.

They also have a one-page summary of the book AND a summary per chapter, so information is presented at different levels of detail. In my time with it, my most frequent workflow with it has involved reading the one-page summary of a book, buying it if it was interestering enough, reading the book, and then going back to Shortform to go read the chapter summaries. And both the one-page and detailed chapter summaries, including Shortform notes, can be highlighted!

Naturally I wanted to take it further than that. I already sync my Kindle highlights to Obsidian automatically and incrementally process them in Obsidian, so I wrote a script to have my Shortform highlights going to Readwise as well.

My Shortform highlights for the book Mindset, in my Obsidian vault.

The screenshot above shows the final result: highlights I made in the Shortform web or mobile app that have been synced to a page in my Obsidian vault. Here’s how I did that.

Note: Since this article was published, Shortform and Readwise have created an integration, so the script I mention here is no longer necessary. Click here to see what I’d recommend doing today.

What you need

A Shortform account. (affiliate link)
A Readwise account. (This is an affiliate link. If you sign up with this link, you’ll get a two-month free trial instead of one, and I get a small percentage if you sign up for a plan.)
A tool that Readwise can export to, such as Obsidian, Roam Research, Notion, or Evernote. You can also use their CSV and Markdown exports.
A local installation of Python 3, which you can download here.

Obsidian is my Personal Knowledge Management tool of choice, and I use the Readwise Official community plugin to get Readwise to send highlights to Obsidian. However, using Obsidian is not necessary as my script connects only Shortform to Readwise.

The setup

Clone my Shortform-to-Readwise repository on GitHub or download it to your computer.

Enter your tokens

The file variables.py requires two tokens: one from your Shortform account and another from your Readwise account:

# Your authorization token for Shortform
authToken = ''

# Your Readwise API token
# Retrieve this token from https://readwise.io/access_token
readwiseToken = ''

Getting your Shortform token

This is a little hacky, because Shortform doesn’t expose an API token in their web app. So you’ll need to log in to your Shortform account and open up DevTools in your browser. In Chrome, go to Options > More Tools > Developer Tools (or, on Mac, hit CMD+OPTION+I). Click on the Network panel.

You should now see Shortform on one side, and DevTools on another in the same browser window.

Then, go to your Highlights page. Doing so will populate the Network panel with a bunch of requests that look like this:

DevTools open in Chrome, showing requests from Shortform.

Click on the request with the name ?sort=date, highlighted above. That’ll open another side panel that will show Headers. Scroll down to the section marked “Request Headers”, and look for where it says Authorization: Basic followed by a bunch of characters.

Authorization headers for Shortform, in DevTools.

I’ve removed my personal authorization token from the screenshot above, but it’s a long series of characters that comes after Authorization: Basic . Go ahead and right click on it and click Copy Value. Paste it into variables.py as the value for authToken, and then _remove Basic (including the space after it) so that you’re saving only everything that comes afterwards.

Getting your Readwise token

Thankfully, this one’s a little easier. Go to this page on Readwise after logging in. Click “Get Token”, click “Copy to clipboard”, and then paste it into variables.py.

Save variables.py.

Note: Don’t share either of these tokens with anyone else, as they will then be able to access your Shortform or Readwise accounts.

Run the script

Open up your terminal, and cd into the /shortform-to-readwise repository. Then, run:

python3 getHighlights.py

It should finish in a few seconds (depending on how many Shortform highlights you have), and then you can verify that it worked by going to your Readwise library.

Shortform exports showing up in your Readwise library.

You should see books that have the Shortform- prefix. If you do, then you’re done! If you’ve got your Readwise account set up to push your highlights elsewhere (such as Obsidian), then you’re all set!

In case you’re wondering why I added the Shortform- prefix, it’s because I often read the Shortform guide, and then read the book itself. Rather than end up in a sticky situation where the Shortform guide highlights might override the actual book highlights, I titled it differently so that I get two pages for the two sets of highlights.

What about running the script regularly?

The whole point of this, at least for me, was to set it up so that I’ve got highlights going to Obsidian automatically, without needing my intervention. What I did was to use crontab on macOS to run the script every 30 minutes, but you can set up your own schedule.

Give cron full system access

Cron needs to have the right permissions to run scripts on your behalf, so you may have to go to System Preferences > Security & Privacy > Privacy > Full Disk Access on your laptop and add /usr/bin/cron to the “allowed” list.

Find the path for your python3 installation

In your terminal, run:

which python3

This will give you a path. For me, it was /Library/Frameworks/Python.framework/Versions/3.9/bin/python3. Copy this.

Open crontab

Then, from your terminal, run the command env EDITOR=nano crontab -e to open up a list of cronjobs. If you don’t have any other jobs, this will be blank.

Set a schedule

Add a line to crontab for the schedule you want to create. This line has to be in the format

* * * * * command

where the five asterisks correspond to:

minute
hour
day of the month
month
day of the week

In my case, I wanted it to run every 30 minutes, so here’s what my schedule looks like:

*/30 * * * * /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/nic/python/shortform-to-readwise/gethighlights.py

Make sure you change /Users/nic/python/ to the path where you put the repository and your python3 path instead of /Library/Frameworks/Python.framework/Versions/3.9/bin/python3. Now exit out of it and save it (CTRL+X, and yes). The Python script should automatically run every 30 minutes.

That’s it!

I consume a lot of content daily, but without systems in place to capture insights from that content and process it, it all just gets forgotten eventually. That’s why I love services like Shortform and Readwise– they let me get what I’ve read/watched/heard out of my head and into Obsidian, where I can review them more critically and truly learn the important parts.

Both are paid services (which I paid for out of my own money), but I really think they’re worth it.

Note: This post contains affiliate links. Check out My Ethics Statement to see my standards for promotions.

Kubedim: Microservices testing using k6 for reliability, with Kelvin Zhang (k6 Office Hours #24)

Fri, 13 Aug 2021 15:50:46 +0200

Today, my colleague Daniel González Lopes joined me as a co-host on k6 Office Hours, and we interviewed Kelvin Zhang about his thesis at Imperial College London, using k6 to test microservices-based architectures. He created something called Kubedim, an application that intelligently applies non-uniform brownout theory to intentionally dim application requests for the purpose of improving overall availability.

This was a dense Office Hours to digest, but Kelvin really impressed us with his experiments and findings! It’s definitely an awesome use case for k6 and just a great one to watch for any site reliability engineer.

Meet the Grafana Labs team: Nicole Van Der Hoeven, Developer Advocate at k6

Fri, 13 Aug 2021 15:29:37 +0200

Originally posted here.

As Grafana Labs continues to grow, we’d like you to get to know the team members who are working on the cool stuff you’re using. Check out the latest of our Friday team profiles.

Meet Nicole!

Name: Nicole van der Hoeven

Current location/time zone

I currently live in Maastricht, in the Netherlands. I moved here with my husband four years ago from Melbourne, Australia.

What do you do at Grafana Labs?

I’m a Developer Advocate at k6, but I’m not actually a developer — I’m a performance tester! I guess there’s no such thing as a Performance Tester Advocate? My job is part engineering, part content creation: I write load testing scripts and code, but I also write blog posts, produce videos, and give presentations. I found out I was joining Grafana Labs a few months ago when k6 was acquired, but I was happy to join the team! I’ve met some fun and motivated people at Grafana Labs that I’m eager to collaborate with.

What open source projects do you contribute to?

I’m a budding developer and still learning the ins and outs of social coding, but some things I’ve contributed to recently are a chaos engineering extension for k6 (created by my colleague, the talented Simon Aronsson), an application called Poké API that I’ve added chaos experiments to, and a Python script for pushing my Kindle highlights to my note-taking app of choice, Obsidian.

What are your GitHub and Twitter handles?

GitHub: nicolevanderhoeven

Twitter: n_vanderhoeven

What do you like to do in your free time?

I play Dungeons & Dragons eight hours a week, take a ridiculous amount of notes on everything, and vlog about my (pre-COVID) travels.

A battle map in one of my Dungeons & Dragons games.

What’s your favorite new gadget or tech toy?

My new Rødecaster Pro, an audio mixer that has quickly become an essential part of my audio and video production process. It takes up a huge amount of precious real estate on my desk, but it is absolutely worth it!

My current desk setup, including the Rødecaster Pro peeking out from underneath the laptop in the upper right.

What’s the best part of your job?

I got into Developer Relations and advocacy in general because I realized that, as a performance testing consultant, I was being asked the same questions and solving the same problems. So I started to share my experience online, and I’ve found that significantly more efficient. Why explain something repeatedly to one person at a time, on the spot, when you could post a more considered explanation online and potentially help an unlimited amount of people? A large part of my job is simplifying technical concepts and making it accessible to beginners, and I love being able to help others by creating the content I looked for when I was a beginner, too.

Here’s one of my YouTube videos:

What’s the last thing you binge-watched?

I have binge-watched the entire two campaigns of Critical Role, a livestream of a Dungeons & Dragons game with a bunch of voice actors, which is over 1000 hours of watch time. Totally worth it!

What superpowers do you have?

I speak several languages: Tagalog (my native language), English, Dutch, Spanish, Esperanto, German, and French — in that order and to varying degrees. I am learning and have learned more, but don’t yet feel confident enough to claim to “speak” them! I choose travel destinations based on how interested I am in learning the local language.

What’s your Zoom virtual background and why?

I don’t use virtual backgrounds. The thing I like most about video is that it’s the most honest form of expression: writing, podcasting, and photography can all be edited far more easily. I like that my home office gives clues about who I am and what I like, and I would never want to hide that — mess and all!

Grotten Sint Pietersberg Maastricht (LuckyDippers)

Thu, 12 Aug 2021 12:21:09 +0200

Another travel vlog from safer times: in September 2018, my husband and I took a friend out to Grotten Sint Pietersberg, a fortress and cave system in Maastricht, the Netherlands. They had been used in the times before that area was not yet part of the city of Maastricht; they were in a small village called Sint Pietersberg that had gotten caught between several invading foreign armies and the people of Maastricht. We enjoyed the day out!

Monitoring your k6 load test: how to install Grafana and Prometheus on a Kubernetes cluster

Wed, 11 Aug 2021 18:55:49 +0200

Running a load test is half of the equation, and not having the right observability tools to see how your application responds to the load can sometimes flat-out invalidate a test.

Here’s how to install open-source tools Grafana and Prometheus on a Kubernetes cluster in DigitalOcean and how a performance user could use them for increased visibility during a load test.

The easiest way to sync Kindle highlights to Obsidian: Readwise to Obsidian Redux

Tue, 10 Aug 2021 19:50:40 +0200

Almost exactly six months ago, I wrote a blog post detailing my workflow for processing content. Specifically, I wrote about how I used a python script to take my highlights from Instapaper, Kindle, Twitter, and Airr and automatically put them into Obsidian.

That method worked, and you can read the original article in the link above. However, today, Readwise announced that they now have an official Obsidian community plugin, and I disabled my trusty old python script for good.

You see, although my old script worked, there were quirks– not the least of which was that making changes to it meant programming in Python. I’m not a developer, so while I made do, it was never a painless experience. I also ran into issues with time zones and invalid characters being used in titles. So I’m thrilled to have something significantly better from Readwise itself! Here’s a video from the Readwise team, or read on for instructions on how to get started.

From Readwise on YouTube

Install the Obsidian plugin.

In your Obsidian vault, go to Settings > Community plugins. You’ll need to turn Safe mode off for community plugins if you haven’t already.

Then, click Browse and search for Readwise Official.

Install it, and then enable it.

Connect your Readwise account to the Obsidian plugin

Head back to the Settings menu and scroll down to the “Plugin Options” section, and then click Readwise Official.

Click connect as highlighted in orange above, and a browser window will open, showing the Obsidian Export options. You’ll need to sign into your Readwise account if you haven’t already.

At this point, your Readwise account will have been connected to the Readwise Official plugin, but you won’t have triggered any syncs yet. If you don’t want to change any of the configuration settings, you can then skip ahead to triggering the sync on Obsidian, but let’s run through the options anyway.

Modifying configuration settings

On the Obsidian Exports page, you’ll be presented with a few options.

Group Files in Category Folders

By default, this will be enabled. When enabled, Readwise will put your highlights in different folders depending on its source. For example, Kindle highlights will appear in the folder Books. Disabling this option means all new notes will be created in the directory specified in the Obsidian plugin settings, which we’ll cover later. I left this enabled in case I read a book that has the same name as a blog post about the book, for example.

Use Custom Formatting

Custom formatting is disabled by default, and lets you choose how the final note looks. Here are the things you can change in this section

Page title
Page metadata
Highlights header
Highlight
YAML front matter
Sync notification

I did make some changes in this section just to tweak how I want information to be displayed.

Page metadata

I don’t see the need for Page metadata since this is what I use YAML front matter for, so I removed the matadata section and the document tags (I never use those either). So my Page metadata section looks like this:

{% if image_url -%}
![rw-book-cover]({{image_url}})
{% endif -%}
{% if url -%}
URL: {{url}}
{% endif -%}

Highlight

By default, highlights are entered in bullet-point form. That’s fine when I’m taking notes, but when I’m highlighting someone else’s words, I like to be very careful to use blockquotes. I frequently do block searches in Obsidian, and I do NOT want to accidentally pass off a quote as something I said.

To this end, I added a > before each highlight and two newlines (carriage returns) at the end to delineate each highlight. That way, I can quote just one highlight at a time.

> {{ highlight_text }}{% if highlight_location and highlight_location_url %} ([{{highlight_location}}]({{highlight_location_url}})){% elif highlight_location %} ({{highlight_location}}){% endif %}{% if highlight_note %}
- Note: {{ highlight_note }}{% endif %}

As you can see above, I also removed highlight tags, which I never use.

YAML front matter

Metadata has to be quite specific, so I added parameters I already use for other notes in this section. The TVZ is a tag I use that means “To Verzettelen”, meaning pages that I need to process further.

Title: {{full_title}}
Author: {{author}}
Tags: TVZ, readwise, {{category}}
date: {{date}}

Sync notification

As someone who has worked on a syncing plugin for Readwise to Obsidian, I know that things go wrong sometimes. Enabling this outputs a synchronization log to a note, which has come in handy MANY times in the past. Has that book been synced, or did something go wrong? So I copied what the Readwise team suggested:

- [[{{date|date('Y-m-d')}}]] {{time}} — Synced {{num_highlights}} highlight{{num_highlights|pluralize}} from {{num_books}} document{{num_books|pluralize}}.
{% for book in books %} - {{ book.num_highlights_added}} highlights from {{ book.title }}
{%endfor %}

Select Items to be Exported

In this section, you can choose which books/podcasts/other content in Readwise you want to export. It is disabled by default. If you haven’t synced Readwise to Obsidian before, and you just want to pull in everything, leave it disabled.

In my case, I have already synchronized everything using the plugin I mentioned, and didn’t want to create another few hundred duplicate notes in my vault. So I enabled it and selected a few more recent books.

A weird thing about this Export settings page is that there is no saving– it just automatically saves it. Just a note, because I was scrolling up and down that page looking for an indication that it had been saved before I realized it is done automatically.

Initiate Sync from Obsidian

Head back to Obsidian and to the Readwise Official plugin options page. You should see a “Initiate Sync” button where the “Connect” one was before.

That’s a good sign - that means your Readwise account has been connected, and your configuration settings from the previous step will be fetched.

Before you click the button, note that the Customize base folder option is set to Readwise. This is the directory that all your Readwise notes will be created. So if you enabled Group Files in Category Folders earlier, your Kindle highlights will be saved in Readwise/Books/booktitle.md. You can change the base folder if you prefer, but I just kept the default.

Time to click the button!

You’ll see a few messages popping up on the upper right corner of your screen, and you’ll also see a Synced! message appear next to the button you just clicked.

Verifying results

In your Obsidian vault, search for one of the items you synced. Here’s what my note for Pat Flynn’s Superfans looks like:

Great! It looks exactly how I want it to look.

Set up automatic sync

If you leave it there, the plugin will only sync when you tell it to, so you’d need to go into the plugin options each time. That’s not particularly useful for me, because I want it to just happen.

If you feel similarly, head back into Plugin Options one more time and select your desired frequency in the Configure resync frequency dropdown menu. I chose 1 hour, the minimum. I also enabled the automatic sync when Obsidian opens.

Conclusion

I love this new plugin! It was very much needed, because existing options for accomplishing the same thing were flawed in some way, mainly because they were community-driven– meaning the people who created these workarounds (myself included) were “just” users who had other jobs. I’m grateful that the Readwise team has acknowledged the popularity of Obsidian and taken on the maintenance burden. I’ve experimented with the plugin quite a bit and haven’t found any major issues so far. It is a seamless way to get the best of the content I’ve consumed from various sources into one single source of truth: Obsidian. I highly recommend it to anyone looking to streamline content processing and creation processes.

How I use Obsidian for D&D - Player's Edition

Mon, 09 Aug 2021 17:46:51 +0200

I’ve previously written about how I use Obsidian for work, and I’ve also talked about how I use Roam Research for D&D. Shortly after the latter post, however, I moved to Obsidian. That was six months ago, so I thought it was time for an update.

Before the game

World

My World page is the main jumping-off point for all my notes for a specific campaign, and each campaign gets one. It contains:

The main locations in the world
The player characters, and links to their individual pages with information about what my character knows or thinks of them
A link to Open Loops (Quests and Questions and Agenda, as discussed below)
The list of all the sessions I’ve played for the campaign
Campaign-specific house rules that the DM has introduced

Quests and Questions

I like to play high-intelligence characters, so I like to keep track of unanswered questions and other adventure hooks my DM has dropped that the party hasn’t quite gotten around to resolving yet. Some of them are critical quest or story arcs, and others are random tidbits of information that NPCs have mentioned. I organize them by location, and then by quest, so that it’s easy to follow a thread depending on where the party is or how much downtime we have. Here’s an example of that page:

Agenda

While the Quests and Questions page records unfinished business that my character is actively thinking about but may not quite know how to explore yet, my Agenda page is where I write down specific tasks. I loosely follow the Getting Things Done methodology and divide this page into Next Actions and Waiting For. I group tasks by person and/or by location.

I use the Obsidian Completed Area plugin so that hitting a CTRL + ENTER on a task automatically checks it and then moves that task to a separate Completed section at the bottom of the note. This helps me keep the agenda clean and relevant.

During a game

Workspace

One of the best parts about Obsidian is that it allows me to save and load custom workspaces. This is useful because I use Obsidian for work as well as play, so I can set up the two workspaces differently. I also DM one campaign and play in another, and I have a workspace for each of them because I need different information available to me. The workspace features lets me customize my dashboard. Here’s what my workspace looks like for the 4e game that I play in:

At the beginning of a game, I’ll create a new session page in the last pane, enter into it, and then stay in the workspace for the rest of the game. I have my character’s stats sheet handy (which I need to do manually since I haven’t found a D&D Beyond-like service for 4e) and all the powers and abilities I need to refer to.

Session template

I love Obsidian templates! I use templates for things like inserting an HP tracker with all my party’s names, their max HP, healing surges (4e); a defense table to keep track of a creature’s defenses (again, 4e); and campaign-specific session log formats.

At the beginning of the game, I’ll create a new page in the Session List, click into it, and insert a session log template (I have this hotkeyed to CMD+;). Here’s what a fresh one looks like, in Edit mode:

I add metadata to every session so that I can query it later using the excellent Obsidian Dataview plugin, though I do this more often for the campaign I DM rather than the one I play in.

We usually do a recap at the beginning of the game, so I jump into the Recap of the last session section and embed the summary from the previous session using ![](/assets/86_20210717 Day 16 Hunting the Ice Phase Spider#^summary), which pulls in the section marked with ^summary from the last session. That embeds just that portion of the last session in the current session, like this:

This helps me get back up to speed quickly instead of doing the usual “Where were we again?” that takes up time.

HP table

My character is a multiclassed healer, so it’s important for me to know who needs heals or other buffs. I keep track of this separately, and it helps out my DM as well because it’s easy to miss something in the excitement of a battle. I do include the HP table in my workspace, in Preview mode, but here’s what it looks like in Edit mode:

Tables in Markdown can be a pain, but the Advanced Tables plugin goes a long way towards making this easier by adding some nice features like tabbing to the next cell, adding commands for inserting and deleting a column or row, and just automatically formatting a table so that it’s not a jumble of lines. I also heavily use this plugin for my characters stats and skills, both of which I keep in tables.

Keeping track of fights

When a fight breaks out, I insert an intiative template, which is just a list of all the characters in the party:

Initiative!
- Ka'ira
- Maestro
- Xio
- Eraithe
- Keren

This makes it easier to add the initiative value rolled, as well as that of any other baddies we’re fighting. I use the Outliner plugin to be able to easily move people on the list up or down as needed without having to copy or paste. I have the shortcuts set to CMD+SHIFT+UP and CMD+SHIFT+DOWN so I don’t even need to use my mouse.

I also have a template for Defenses. D&D 4th Edition has four different kinds of defenses, and since my character’s powers target a few of them, I keep track of the defenses of each monster. Here’s what that template looks like, in Edit mode:

| Defense | Value |
| --------------------- | ----- |
| [[Reflex Defense]] | |
| [[Fortitude Defense]] | |
| [[Will Defense]] | |
| [[Armor Class]] | |

You’ll notice that I also link to the pages I have for each defense type, in case I need to remind myself which stat a defense is comprised of.

Character powers

Like many, I have migrated from playing games on a literal tabletop to doing it virtually. These days, I play 100% online, using a bunch of different tools. For the most part, though, I still prefer to roll actual dice. There’s just something satisfying about the feel of a die in my hand. Since I switched to Obsidian, though, I have looked into virtual dice. On the off chance I don’t have any dice (ha) or all my dice have betrayed me (much more likely), I use the Dice Roller plugin. It turns

`dice: 1d20+7`

into a random die roll. Here it is when incorporated into my character’s list of powers:

Sometimes, when I’m feeling superstitious about my dice, I’ll switch to using this dice roller “just to be safe”. It has other cool features too, which I’ll cover more in detail in the Dungeon Master’s edition of this article.

Search, linked, unlinked mentions

Frequently, something comes up in a game that I’m sure we’ve heard of before, but I can’t quite remember where or when. Actually, sometimes something comes up that I’m quite sure we’ve never heard of before, but it turns out we have. Luckily, Obsidian is quick to tell me that using the linked/unlinked mentions and the search features.

Here are the mentions in my note for the word “Netheril”:

Unlinked mentions show up too, in case the word came up somewhere and I didn’t think to specifically create a page on it.

On monster pages, mentions remind me when we fought them last, and any tips for fighting them. It can be handy when one of the other players has forgotten something that happened to them, too. If I want to remember how Ka’ira got her festering wound, I can type it into the Obsidian search bar:

and be able to confidently say that Ka’ira got her last festering wound on session 68, in the Blue Crystal Fortress that we now know to be the Twisted Fane, while fighting aberrations.

After a game

I think I might use Obsidian even more between games than during the game, because after the game is when I process everything my character learned.

Session summary

Immediately after the game is when I create a short session summary. I like to do this when the session is still fresh in my mind. Then, I add ^summary at the end of the paragraph to identify it and to be able to pull it into the next session’s recap. Block references and block embeds are some of my most frequently used Obsidian features.

Creating new pages

I then create new pages for any people, places, magical items, or events that we encountered. I like to use a standard frontmatter:

---
type:
location: [Neverwinter]
world: "Forgotten Realms"
campaign: "Order of the Wandering Mind"
date: 2021-06-23
---

Here’s an example of the page for an NPC:

If my DM has given us character art, I include it in the page.

And here’s an example of a page for a location:

For this page, I used a plugin called Obsidian Leaflet, which lets me add maps to my notes! In this case, my DM gave us this map of the structure as a guide, but during combats we didn’t use a map. To keep myself oriented between games, I added red pawprint pins where we had spotted or fought monsters, green thumbtacks to mark where we’d ended sessions, and green markers where we had spotted allies. I also like to add links to monsters or sessions on the pins/markers so that I can jump to those pages quickly if I need to.

History

To keep historical events straight in my head, especially since we tend to discover them out of sequence, I use the Obsidian Timelines plugin, which lets me create pages like this:

by using this code in the timeline page:

```timeline
fr
\```

(without the backtick) and this for every event that I want to include:

---
date: [2021-06-23]
tags: [timeline, fr]
---
<span
 class='ob-timelines'
 data-date='1443-00'
 data-title='Xionerys is born'
 data-class='orange'
 data-img = 'assets/xionerys.jpg'
 data-type='range'
 data-end='1443-00'>
 Xionerys is born in Calimshan.
</span>

Using this method, I’ve been able to call out discrepancies in what two different NPCs told us and figure out that one was lying– or at least misinformed.

Replace all

When I inevitably get a fantasy name wrong, having pages as flat .md files is great because I can do a find and replace on ALL mentions of a certain NPC name/place I’ve misspelled (those fantasy names!). I do this with my usual code/text editor, VSCode, just because I usually already have it up, but you can do it using terminal commands. If it’s just the name of a page, updating the spelling in Obsidian changes it everywhere else it’s linked to as well, which is useful.

Updating open loops

To finish processing the last session, I update Open Loops: I add new points or update existing points in Quests and Questions, and then add tasks to Agenda that I want to do in the next session. I find that it’s best to do it after a game because it’s a week, sometimes more, between games and there’s a high likelihood I’ll have forgotten what I want to do by then!

Publish

I also publish my D&D notes, just for transparency. My DM has joked that he looks in there sometimes when he’s forgotten something. Even if he doesn’t, I enjoy having my notes so easily accessible. I often navigate there myself when I have a spare moment, remembering old sessions and thinking about some loose threads. If you’re interested, here they are! Just be warned that the plugins won’t work in the web version, so sometimes you’ll come across bits of code that didn’t render. I also migrated these notes from Roam Research, and haven’t gone back and updated all the older notes.

Conclusion

Since I’ve switched to Obsidian, it’s been an indispensable part of my workflow for D&D. Without it, I’d be just another murder hobo with no notes. And no DM wants that!

Realistic load testing

Sat, 07 Aug 2021 22:15:13 +0200

This is a conversation I recorded from Clubhouse in the Performance Engineers group created by NaveenKumar Namachivayam today, August 7th, 2021. I started by introducing myself and then let everyone know that I would be recording the conversation. Enjoy!

How we work at k6 - building software & working remotely with CTO Pawel Suwala (k6 Office Hours #23)

Fri, 06 Aug 2021 18:09:05 +0200

For this episode of k6 Office Hours, I got the k6 Chief Technical Officer, Pawel Suwala, to give a bit of a behind-the-scenes of how we work at k6 and what we’re working on next. We talked about the methodology we use for building k6, the different teams we have, and exciting new features that are coming soon!

Balkan Bites free food tour with Bulgarian food | Sofia, Bulgaria (Lucky Dippers)

Fri, 06 Aug 2021 13:01:44 +0200

In September 2018, while my husband and I were in Sofia, Bulgaria, we decided to try some delicious Bulgarian cuisine by going on the Balkan Bites free food tour, which let us try Bulgarian relish, yoghurt, doughnuts, cheese, and desserts. Afterwards we went back to one of the restaurants we had gone to on the food tour to have a full meal, which ended up being very delicious. Along the way, I talked about how ilearned the Cyrillic alphabet, and why I thinks it’s worth learning even for a short trip like this one.

We really enjoyed the Balkan Bites food tour, so if you want to join as well, check them out.

And here’s the website for the restaurant we ate at, “Hadjidraganov’s House”.

Schrödinger's Pokémon: Observability for chaotic load testing (New Relic FutureStack)

Sun, 01 Aug 2021 18:54:27 +0200

This is a presentation I gave at New Relic FutureStack 2021.

Deploying an application to production is often like putting a pokémon in a box and hoping it manages to stay alive despite real-world chaos and potential outages. What can we, as testers, do to prepare applications for these situations before they go live? In this talk, I discuss chaos engineering concepts and demonstrates how to apply them by incorporating load tests with chaos experiments and setting up observability tools to watch it all happen. Because without observability, how can we tell whether Schrödinger’s pokémon is alive or dead?

Free walking tour in Sofia, Bulgaria (LuckyDippers)

Fri, 30 Jul 2021 16:00:48 +0200

We haven’t travelled since January 2020 for obvious reasons, but fortunately (?) I have a whole lot of footage taken from trips pre-pandemic that I haven’t yet turned into travel vlogs. In case you weren’t aware, I learned a lot of what I know now about creating videos from making travel vlogs for friends and family as my husband and I travelled around the world for months at a time.

Here’s a trip we took in September 2018. We had just finished off a trip to Greece with some friends from Australia, and on the way home, I decided to book a three-day stopover in Sofia, Bulgaria– a city I really knew next to nothing about. Here’s that first day!

A tester's overview of Grafana Labs open-source projects

Mon, 19 Jul 2021 18:23:33 +0200

Since the acquisition of k6 by Grafana Labs, the entire k6 team has been absorbed into the new, larger team that we’re suddenly a part of. Grafana Labs maintains a lot of projects, so I was very quickly overwhelmed with things to learn about all of them. Here’s my attempt to make sense of the Grafana Labs landscape, and perhaps it can help some of you as well:

Shopping at Bicester Village and welcome my niece to her new home | London, England (Lucky Dippers)

Fri, 16 Jul 2021 18:38:17 +0200

Here’s a fun little vlog that I’ve been meaning to make of the last trip my husband Rob and I took before the pandemic. In January 2020, we went to see my cousin and her daughter (my “niece”, whom I grew up with) in London. My niece, Steffi, had just arrived in London for some studies abroad (she normally lives in the Philippines), so Rob and I popped over to meet up with her and help her settle in.

Poor thing– not a month after Steffi got to London, news of the pandemic began to spread. Rob and I had intended to visit her again, but of course that didn’t happen.

How to run distributed load tests with k6 Operator (k6 Office Hours #21)

Fri, 16 Jul 2021 18:17:13 +0200

In this week’s Office Hours, Simme and I talked about the k6 Operator, which is a open-source project that Simme created for distributed load testing with k6.

k6 Cloud already offers a no-hassle distributed load testing experience, but for those who prefer to use their own infrastructure, k6 Operator allows the setup of a k6 test as a custom resource object that can be deployed along with a Kubernetes cluster.

How to add performance testing to CI/CD pipelines using k6 and GitHub Actions

Mon, 12 Jul 2021 22:26:20 +0200

Continuous testing means verifying and improving software quality at all stages, including development. It can sound like something only large companies do, but it’s an approach I use even for this personal blog that you’re reading this on.

Every new blog post or change I make to this site automatically triggers the execution of a small test. This test, when run over time, lets me see whether changes I’ve made have been detrimental to site performance. Here’s a video on my setup:

Gear I Use

Mon, 12 Jul 2021 21:18:55 +0200

Gear I Use

One of my favorite obsessions is watching or reading about the gear other people use. I’ve watched almost every “What’s in my tech bag” video on YouTube, it feels like, and I’m always prowling blog posts that detail what someone is using. In case there’s anybody out there that is as interested in these things as I am, here’s an overview of the gear that I use to make videos, write both blog posts and code, and generally mess around on the internet.

Edit: After a reader suggested I provide links to these items, I’ve included them below, and some are affiliate links. You are under no obligation to click these links, but if you do, I get a tiny percentage of the sale. Thanks!

Sections

There’s a lot here! I’ve broken it down into sections, so you can skip to the section you’re interested in.

Daily hardware
Home Office 1
Home Office 2
Everyday Carry
Remote Working Tech Bag
Remote Work Setup for a month

Daily hardware

(back to top)

Here are the main devices I use every day.

Computers

Macbook Air 15" (2023): 24 GB RAM, 2TB SSD, M2 chip - this is my main workhorse
Macbook Pro 13" (2020): 2,3 GHz Quad-Core Intel Core i7, 32 GB RAM - secondary laptop
Mac Mini (later version here): M1 chip, 16GB RAM, 1 TB SSD - desktop, always on

Mobiles

Samsung Galaxy Z Fold 5: 512GB - my main mobile, always with me
Apple iPhone 15: Pro Max - secondary mobile, always with me
Samsung Galaxy Note 10+: extra mobile

Tablets

Apple iPad Pro 11" (2020) tablet - still works great!

Readers

Amazon Kindle Scribe: my couch reader, for nonfiction and PDFs
Amazon Kindle Oasis: my bed reader, for fiction
Kobo Sage: for non-Amazon ebooks

Wearables

Oura: health and sleep tracking ring
Samsung Galaxy Watch 6: smartwatch
American Express Prada wearable (not available for purchase separately): for contactless payments

Audio

Apple AirPods Pro: second generation, for Apple products
Bose QuietComfort Earbuds II: for non-Apple products

Home Office 1 (the Netherlands)

(back to top)

Video production gear

Not on desk

Elgato green screen (long, black metal object on the left side of the photo, on the floor)
Canon 6D Mark II with a Canon 16-35mm f/2.8L (upper left, on tripod with teleprompter)
Manfrotto 290 tripod
TeleprompterPAD iLight Pro 10"
Neewer 5-in-1 reflective reflector set (gold, silver, white, black, and transparent)
Neewer two-way rotating aluminum light stand
Manfrotto universal smartphone clamp (mounted on light stand above desk)
Neewer Camera Monopod with rotating column for panoramic shooting
2 x Nanlite PavoTube 6C II tubes on the shelves in the background

On desk

Aputure AL-MC portable light (attached magnetically to the back of my external monitor)
Blue Yeti microphone - (pictured, but retired for the next item)
Røde PodMic microphone - my new main mic
Elgato Key Light
Canon M50 (between my laptop and monitor screens, used as my main webcam)
Elgato multi-mount (holding up the M50)
Blackmagic Atem Mini video switcher
Røde Wireless Go wireless microphone (secondary microphone when I need to move around)
Rødecaster Pro for crispy audio on calls and videos
Samsung Portable SSD T5, 1TB (always connected to my laptop) the drive I edit off
Macbook Pro 13" (2020), 2,3 GHz Quad-Core Intel Core i7, 32 GB RAM - main laptop

Not pictured but in frequent use

Sony A6600 with Sigma 16mm f1.4 lens (main shooter, used to take this shot)
Røde Wireless Go II two-person microphones (used as on-board camera mics)
SwitchPod tripod
Peak Design Travel tripod: Pricy, but the only full tripod I use these days. Excellent for travel!

Other tech

Synology DS920+ 4 Bay NAS with 4 x WD RED 4TB drives (just outside the photo)
Amazon Echo Dot (on the bookshelf)
Google Wifi wireless extender (just outside photo)
2 x Nintendo Switch (on bookshelf; one Animal Crossing edition)
Nintendo Lite Zacien and Zamazenta edition (on bookshelf)
Amazon Kindle Oasis (bookshelf)
Amazon Kindle Paperwhite (bookshelf)
Kobo Aura One (bookshelf)
Risa Electric Solid Tenor Ukulele, black (bookshelf)

Non-tech gear

Wyrmwood Gaming Cherry magnetic dice tower bundle including dice tower, personal dice tray, and dice vault
Kimi Rainbow Acrylic kalimba
Visconti Homo Sapiens Lava Midi, steel age fountain pen (the only fountain pen of many that I use regularly)

Non-video gear on my desk

I already listed the video-related gear on my desk in the first section above, but here’s everything else.

Samsung Galaxy S10 (secondary phone, primarily for timelapses and overhead videos)
Samsung Galaxy Wireless Buds, rose
Samsung Galaxy Note 10+
Apple Macbook Air 11" (2011, used as my presentation laptop)
Apple Magic Trackpad
Apple iPad Pro 11" (2020) tablet
Niz Plum Atom 68 electro-capacitive keyboard with Pikachu keycap
Logitech MX Master 3 the best mouse I’ve used!
Bose NC700 wireless noise-cancelling headphones
Oura health and sleep tracking ring

Home Office 2 (Portugal)

(back to top)

I live between two countries: the Netherlands and Portugal. Since I spend a few months in each place, that also means I need a place to work in each apartment! My home office in Portugal is larger but definitely less well-equipped. It’s a little embarrassing, but I thought it would be interesting to show a more realistic, in-progress setup.

Why yes, I do indeed have clothes hanging on a rack behind my desk to try and soak up sound.

Desk

Elgato Key Light Air x 2: I bought two of the smaller key lights than what I have in the Netherlands, because this room is larger and requires more light
Hidrate Spark Pro: bluetooth water bottle
Samsung 32" 4K UHD monitor: a bit overkill on the size front, but nice
Apple iPad Pro 11": This is from 2020, but still going strong
Elgato Stream Deck MK.2: 15 programmable keys for shortcuts and macros
Keychron K6: hotswappable, RGB mechanical keyboard
Logitech MX Anywhere 3: my travel mouse
Blackmagic Atem Mini: my video switcher
Røde Rodecaster Pro II: my audio mixer
Røde PSA1: the best mic arm
Røde Podmic: great XLR microphone
Samsung Galaxy Note 10+: my secondary phone
Baseus 17-in-1 USB-C Docking Station: for connecting my laptop to all the peripherals
Nintendo Switch Lite in coral: my travel gaming console
Fully monitor and laptop arm: arms to hold my monitor and laptop
Macbook Pro 13" (2020): same laptop that I bring around with me
Sony A6600 with Sigma 16mm f1.4 lens (main camera that I take with me from the Netherlands)
SwitchPod DSLR Tripod: best travel tabletop tripod

Free-standing

Fully Jarvis standing desk: bamboo-topped motorized desk
Neewer 77 inch tripod: basic tripod
Neewer Photo Studio Backdrop Support System: I hung an acoustic curtain and some clothes on this to help with echo for now

Not pictured

Samsung Galaxy Z Fold 3: used to take this photo

My everyday carry

(back to top)

What about what I carry every day?

Samsung Galaxy Active 2: my smartwatch
d20 spinner ring: a ring that acts like a 20-sided die for roleplaying games on the go
Samsung Galaxy Z Fold 3
Aer Day Sling 3 X-Pac: my sling bag of choice
Nitecore Tini 2 USB C flashlight: tiny USB-C rechargeable flashlight
Nite-Ize locking S-biners: for a little extra security for zippers
Orbitkey holder: to hold my keys
Nano bag 2.0: compactable bag
Kable Card: all essential cables and adapters, a light, a mobile stand, and a memory card reader all in one card
Aurtech bluetooth keyboard: portable bluetooth keyboard
Samsung Galaxy S Pen Fold Edition: stylus for my mobile
Sony WF 1000 XM3: wireless noise-cancelling earbuds
Spigen Arc Powerstation 20W: small GaN charger
Secrid mini-wallet: my minimalist wallet
Dispel Dice Lilac Song: a full-sized d20 I carry around
Sprawlopolis: small cooperative card game
LIE: fun party game for 2-6 players and is the size of a pack of gum
Victorinox Jetsetter: small, travel-friendly multitool
Mighty Hanks: microfiber on one side, cotton on the other
Burts Bees: lip balm
Small red Sharpie: mini marker
Joby Gorillapod GripTight Smartphone Micro Tripod: tripod for my mobile
Samsung Galaxy Note 10+: secondary mobile
Topo designs micro pouch: small pouch
Tile Mate Pro: bluetooth tracker
Trezor Model T: hardware wallet and security key

My Remote Working Tech Bag

(back to top)

If I need to go somewhere to work, my EDC bag won’t cut it! Here’s what I bring with me to work and play remotely:

Aer x Ministry of Supply Lunar Pack
Platypus soft bottle. Similar here.
PacSafe Retractasafe 250
iWalk 10,000 mAH USB C power bank
Critical Role Nott the Brave face mask (2 pack)
Apple iPad Pro 11"
Apple Pencil
The Crew: Mission Deep Sea (card game)
Kindle 2022
Popsockets PopGrip
Nintendo Switch Lite
Aer Day Sling 3 X-Pac
SwitchPod
Peak Design Tech Pouch
Keychron K3 travel pouch
Nuphy Air 60 keyboard
Roost laptop stand
Nedrelow Magic Sleeve
URWizards soft wool felt dice tray
Shure Aonic 215 wired sound isolating earbuds
Logitech MX Anywhere 3: travel mouse
2 x Lexon Mino X bluetooth speakers: awesome USB C-rechargeable bluetooth speakers that can be paired together
Logitech Spotlight Wireless Presentation Remote. Similar here.
2.5mm to 3.5mm audio adapter. Similar here.
Yubico Yubikey 5C security key
Yubico Yubikey 5C nano security key (always plugged in to laptop)
8Bitdo Zero 2 bluetooth Switch mini controller
Anker USB C cables
USB C adapters. Similar here.
Røde Wireless Go wireless microphone system
UGreen GaN 100W USB C charger
Hyper Drive USB hub. Updated version.

My Remote Work Setup for a month

(back to top)

Here’s what I bring with me when I need to be able to work while travelling continuously for a month. This setup lets me create videos, run livestreams, do programming work, and do presentations at conferences with minimal bulk and maximal functionality.

Platypus soft bottle. Similar here.
Samsung Galaxy Active 2: my smartwatch
Le Chivée USB C portable wireless charger for Samsung Galaxy Active 2 watch. Similar here.
Bose QuietComfort Earbuds II.
Apple iPad Pro 11".
Apple Pencil
Nedrelow Magic Sleeve
3 x Dispel Dice d20s.
Nuphy Air 60 keyboard
Logitech Spotlight Wireless Presentation Remote. Similar here.
Apple Magic Trackpad.
2 x Lexon Mino X bluetooth speakers: awesome USB C-rechargeable bluetooth speakers that can be paired together
Røde Wireless Go wireless microphone system
Roost laptop stand
Macbook Pro 13" (2020), 2,3 GHz Quad-Core Intel Core i7, 32 GB RAM - main laptop
Yubico Yubikey 5C nano security key (always plugged in to laptop)
Sony A6600 with Sigma 16mm f1.4 lens (main shooter)
Peak Design Travel tripod: Pricy, but the only full tripod I use these days. Excellent for travel!
Elgato Cam Link 4K.
USB C adapters. Similar here.
Peak Design Tech Pouch
Anker USB C cables
Lume Cube Panel Mini LED light.
Joby Gorillapod GripTight Smartphone Micro Tripod: tripod for Lume Cube
Samsung X5 2TB SSD. Better here - T7.
Samsung Galaxy Z Fold 3
(not pictured, but since added) Logitech POP Mouse: travel mouse, even though I hate that it requires an AA battery.

My WFH desk and office setup

Mon, 12 Jul 2021 21:18:55 +0200

One of my favorite obsessions is watching or reading about the gear other people use. I’ve watched almost every “What’s in my tech bag” video on YouTube, it feels like, and I’m always prowling blog posts that detail what someone is using. In case there’s anybody out there that is as interested in these things as I am, here’s an overview of what my home office looks like and the gear that I use to make videos, write both blog posts and code, and generally mess around on the internet.

I’m only mentioning larger items of note that I use frequently.

Video production gear

Not on desk

Elgato green screen (long, black metal object on the left side of the photo, on the floor)
Canon 6D Mark II with a Canon 16-35mm f/2.8L (upper left, on tripod with teleprompter)
Manfrotto 290 tripod
TeleprompterPAD iLight Pro 10"
Neewer 5-in-1 reflective reflector set (gold, silver, white, black, and transparent)
Neewer two-way rotating aluminum light stand
Manfrotto universal smartphone clamp (mounted on light stand above desk)
Neewer Camera Monopod with rotating column for panoramic shooting
2 x Nanlite PavoTube 6C II tubes on the shelves in the background

On desk

Aputure AL-MC portable light (attached magnetically to the back of my external monitor)
Blue Yeti microphone
Elgato Key Light
Canon M50 (between my laptop and monitor screens, used as my main webcam)
Elgato multi-mount (holding up the M50)
Blackmagic Atem Mini video switcher
Røde Wireless Go wireless microphone (main microphone, even for voice calls)
Samsung Portable SSD T5, 1TB (always connected to my laptop) the drive I edit off
Macbook Pro 13" (2020), 2,3 GHz Quad-Core Intel Core i7, 32 GB RAM - main laptop

Not pictured but in frequent use

Sony A6600 with Sigma 16mm f1.4 lens (main shooter, used to take this shot)
Røde Wireless Go II two-person microphones (used as on-board camera mics)
SwitchPod tripod

Other tech

Synology DS920+ 4 Bay NAS with 4 x WD RED 4TB drives (just outside the photo)
Amazon Echo Dot (on the bookshelf)
Google Wifi wireless extender (just outside photo)
2 x Nintendo Switch (on bookshelf; one Animal Crossing edition)
Nintendo Lite Zacien and Zamazenta edition (on bookshelf)
Amazon Kindle Oasis (bookshelf)
Amazon Kindle Paperwhite (bookshelf)
Kobo Aura One (bookshelf)
Risa Electric Solid Tenor Ukulele, black (bookshelf)

Non-tech gear

Wyrmwood Gaming Cherry magnetic dice tower bundle including dice tower, personal dice tray, and dice vault
Kimi Rainbow Acrylic kalimba
Visconti Homo Sapiens Lava Midi, steel age fountain pen (the only fountain pen of many that I use regularly)

Non-video gear on my desk

I already listed the video-related gear on my desk in the first section above, but here’s everything else.

Samsung Galaxy Note 8 (secondary phone, primarily for timelapses and overhead videos)
Samsung Galaxy Wireless Buds, rose
Samsung Galaxy Note 10+ (primary phone)
Apple Macbook Air 11" (2011, used as my presentation laptop)
Apple Magic Trackpad
Apple iPad Pro 11" (2020) tablet
Niz Plum Atom 68 electro-capacitive keyboard with Pikachu keycap
Logitech MX Master 3 the best mouse I’ve used!
Bose NC700 wireless noise-cancelling headphones
Oura health and sleep tracking ring

Load testing and Azure DevOps with José Luis Latorre Millas (k6 Office Hours #20)

Fri, 09 Jul 2021 21:08:36 +0200

José Luis Latorre Millas, Developer Community Lead and Software Architect at Swiss Life AG, joined Simme and me on k6 Office Hours to talk about how he personally uses k6 with Azure DevOps to add load testing to his team’s CI/CD pipeline.

I got to know José from the k6 Community Slack workspace when he mentioned that he would be presenting on k6 in a few meetups. I went to his livestream with CrossDvlUp (Spanish) and enjoyed his approach, so I knew I wanted to have him on. It’s always great to have real users of k6 (both of the OSS and the commercial Cloud option) talk about how they personally use k6 and even what they found lacking in the tool.

José’s multi-scenario template repo: https://github.com/SwissLife-OSS/K6-MultiScenario-template

Register to watch José’s talk at Azure Summit 2021, later this year: https://azuresummit.live/speaker/jose-luis-latorre-millas/

Acquisition talk with CEOs Raj Dutt (Grafana) and Robin Gustafsson (k6) (k6 Office Hours #19)

Fri, 02 Jul 2021 21:04:13 +0200

This week on k6 Office Hours, Simme and I were joined by two CEOs: k6 CEO Robin Gustafsson and Grafana Labs CEO Raj Dutt. It was a little nerve-wracking to have two CEOs on the stream, but it did give us the opportunity to ask both of them pointed questions on Grafana Labs’s acquisition of k6, including any plans they have for how both companies could integrate better in the future.

How to use k6 to benchmark hosting providers with Kevin Ohashi (k6 Office Hours #18)

Fri, 25 Jun 2021 20:57:42 +0200

Kevin Ohashi joined me and Floor Drees on k6 Office Hours to talk about how he used k6 to benchmark the performance of WordPress hosting providers. He talks about the difficulties he ran into while trying to keep tests repeatable and consistent, and I also make some suggestions for how he could improve his load testing script for future comparisons.

Of particular note for me in his results was how clearly the performance of the hosting providers varied depending on the location Kevin had selected for his load generators. One in particular showed a significant delay when a site was fetched from Sydney, indicating the possibility of a lack of a CDN server in that region. It made a great argument for load testing in the cloud, from different geographical regions.

WordPress Hosting Performance Benchmarks: https://wphostingbenchmarks.com/

Benchmarks methodology: https://wphostingbenchmarks.com/methodology/

k6 is part of Grafana Labs! AMA with CEO Robin Gustafsson (k6 Office Hours #17)

Fri, 18 Jun 2021 22:26:49 +0200

On this week’s Office Hours, Simme and I were joined by Robin Gustafsson, the CEO of k6, to make a big announcement: k6 has been acquired by Grafana Labs.

The folks at Grafana Labs have been fans of k6 for a while, and we’ve been fans (and users) of their work since almost the beginning of k6. While a surprise to many, Grafana and k6 together is a combination that makes sense.

Read Grafana’s press release here to hear more about this milestone.

Accessibility and open source support program with Floor Drees (k6 Office Hours #16)

Fri, 11 Jun 2021 22:38:52 +0200

On this week’s Office Hours, Simme and I were joined by Floor Drees, the newest member to join the k6 team. Every new person at k6 goes through a week of testing, and Floor decided to spend her week of testing doing an accessibility review on the k6 Cloud web app and documentation. We talk about how accessibiltiy is an often-overlooked form of operational testing, and Floor shares specific tools that testers can use to make applications more accessible.

We also discuss the k6 open source support program, which makes k6 free for open-source maintainers and regular contributors.

Resources mentioned

k6 Open Source Support Program Boia.org, a good tool for accessibility reviews AlexJS, for identifying inconsiderate language Paul Bruce’s rant on operational requirements instead of “non-functional” requirements Marie Drake on accessibility using Cypress Chrome extension to get dark mode everywhere Contributing Today - open source funding and sponsorship Contributing Today - on open source licensing

Convergence of Code and Speech

Wed, 09 Jun 2021 22:23:19 +0200

As part of the Women Tech Global 2021 conference, I presented a short talk called “Convergence of Code and Speech”, under the theme of “Future of Work”. In hindsight, the title was a little pretentious, but what are you going to do?

I haven’t gotten the recording of the live version yet, but I always like to do a recorded test run by myself, and here’s what that was like:

Programming code and human speech are often seen as being positioned at different ends of the spectrum. At one end, code is terse, strictly logical, and limited. On the other end, speech is unbounded and often verbose. However, the gap between the two is narrowing.

Code is moving towards speech. Good code these days is code that exceeds simply being functional; good code is concise, but expressive; good code is easily comprehensible; good code refers to human concepts rather than technical components.

Shrinking global borders and the internet are making collaborating on code easier than ever before, but contributors may come from different cultures and languages. Remote work has significantly increased the likelihood of a multilingual devleopment team, so code style guides have emerged that are similar to writing style guides. Artificial intelligence is learning to interpret intention, and many apps now employ fuzzy search algorithms. Our expectations for code have increased such that we now expect code to understand us even when we’re being imprecise or inaccurate. Modern programming languages like React and Python use hierarchical abstraction to hide complexity, moving towards code that prioritizes being understood over being thorough.

On the other side of the spectrum, speech is also moving towards coding. Multilingual teams also encourage simplicity in written communication, removing colloquialisms and cultural references so that a more global audience can understand us. Tools like Grammarly correct for simplicity and conciseness. Verbal speech, too, is becoming increasingly programmatic with the rise of “smart” speakers and personal assistants that enforce a syntax for voice commands. Advances in artificial intelligence and machine learning have enabled us to interact with machines in natural language, and machines to interact with us back. OpenAI’s GPT-3 can produce convincingly human text about a variety of topics with little prompting. Writing tools like Notion, Roam Research, and Obsidian are growing ecosystems that mix human language with programming keywords to enable backlinks or access other features.

What are the implications of this convergence of code and speech? Writers will need to learn to code, and coders will need to learn to write. The two will become synonymous, and equally essential especially for knowledge workers. Tools that enable both coding and writing will become more valuable, so it’s worth investing the time to learn them. Both code and speech have inherent linguistic or cultural biases, so using inclusive language will become even more important.

What are coding and speech converging into? Their intersection is simply expressing thought. She who is able to express thought, both programmatically and naturally, will be best poised to take advantage of this convergence.

Making k6 extensions using xk6, with Iván Szkiba and Mihail Stoykov (k6 Office Hours)

Sat, 05 Jun 2021 22:28:29 +0200

This week on k6 Office Hours, Simme and I were joined by Iván Szkiba and Mihail Stoykov to talk all about extensions. Iván, who is a System Architect at NNG LLC, is the most active k6 extension creator outside of the k6 team. Mihail is a software engineer at k6, and one third of the k6 OSS team.

We talked about the many extensions Iván has created for k6 so that he can incorporate k6 into his team’s daily testing activities, his considerations for creating extensions, and how he got started. Mihail also shares some inside information on why certain decisions were made about the way k6 is structured, and common mistakes he sees people make in their custom extensions.

Downloading files uploaded to Roam Research for use with Obsidian

Wed, 02 Jun 2021 00:27:59 +0200

If you’ve recently transitioned from Roam Research to Obsidian, you might run into a few issues, even after you’ve run the “Markdown format importer” core plugin. One of the most annoying ones that I ran into was that files uploaded to Roam (images, pdfs, etc.) are automatically uploaded to Google Firebase.

Roam assets are linked to in the Markdown files like this:

![](https://firebasestorage.googleapis.com/v0/b/firescript-471a1.appspot.com/o/imgs%2Fapp%2FMyNotes%2FxRFXx7l-ZK.png?alt=media&token=1e03b680-5472-33dc-9b06-a127301af37f)

Now, since Roam relies on security by obscurity on this front, the links to your assets will still be accessible in your Obsidian vault, and they’ll still render correctly. However, I still think it’s worth downloading the files for three reasons:

If you don’t intend to use Roam anymore and cancel your subscription, they will likely delete those assets at some point, leaving you with a lot of broken links in Obsidian.
Part of Obsidian’s appeal is that you own your assets. Not so if they’re on someone else’s server.
You can’t access Firebase files without an internet connection.

So, instead, you want to download the contents of that link and replace it in your notes with something like this:

![](assets/newfile.png)

That way, you’ll have full control over all your assets, and you won’t be dependent on those assets being stored on Firebase by Roam.

To fix this, I wrote a Python script that will:

Go through every file and every folder within a specified root vault directory
Look for links to Google Firebase
Download all assets from Firebase to a local folder (/assets by default), using the timestamp and index number as a filename
Replace the Firebase links in your vault with links to the newly downloaded files, accounting for PDFs, whose links are formatted differently in Roam as {{ pdf: ... }}

Before you run this script, I highly recommend backing up your vault or trying this out on a dummy vault.

After downloading and saving the script, run python3 downloadfirebase.py on your terminal.

This worked really well for me, and I can breathe a little more easily now that my vault is self-contained.

What Is Load Testing? - A real-world explanation

Tue, 01 Jun 2021 20:23:17 +0200

I’ve made about 60 videos on load testing and application performance to date, but I realized that I’ve never made one defining load testing, which seemed like a glaring mistake. So, here’s my attempt!

Rough transcript below - though I went off script a few times:

Load testing is a key part of making sure that an application performs well and reliably. But what IS load testing, anyway? In this video, I break it down for you in simple terms. And if you’re a family member or friend of mine trying to understand what it is I do for a living, thank you! This is for you too.

Software testing

It takes a whole team to build software, but some roles are more visible than others. For example, pretend you want to build a bridge. Who do you need to build it?

An architect to design the specifications of the bridge
A liaison or representative to the gemeente or local council to clarify what the specifications ARE
Builders who will physically build the bridge, who might need to hire subcontractors for various materials and services
Someone to double-check that what you’re building is up to code (testers)
Someone to manage it all, and maybe a few more levels of team leads
Traffic enforcers to detour traffic away from the bridge while it’s being built

So who is responsible for “building” the bridge? The answer is, everybody. Everybody has a part to play. Every part is important.

It’s the same thing with software. When we think about building software, we often think about developers, but that’s not the full picture. Just like building a bridge, it’s a team effort. But I want to zoom in on one role in particular, because it’s the role I’ve had most often: load tester.

But first, let’s go on a field trip.

Load testing on the field

Does this bridge look familar? This is the Sint Servaasbrug, and they SAY that it’s the bridge on the €10 note. It might be true, too, because it’s here in my current hometown of Maastricht, the Netherlands, where the European Union was founded in the Treaty of Maastricht. The name Maastricht means the crossing of the Maas, which is the name of this river here.

So let’s pretend the bridge hasn’t been built yet, and that we’re helping build it. Let’s put on our load tester hats. As a load tester, it’s our job to make sure that the bridge can support the weight it is intended to support.

How do we know how much weight it’s intended to support? Well, the council or the rest of the team helping us build the bridge might have some ideas on that, or some building standards that we can use. That’s a good place to start, and we should definitely discuss that with them, but as a load tester, it’s also our job to question those requirements. Let’s do some research.

This is the street leading up to the bridge. Remember, the bridge isn’t built yet, but maybe we can get an idea of the kind of traffic that the area gets.

Some things to watch out for or questions to answer:

How many people are walking?
How many are on bikes?
How many are on bakfiets bikes?
Are vehicles allowed on this street, and if so, how many of them drive through?
Do people tend to pass by in groups, or on their own?

Another thing we can do is check other bridges. This is the Hoogbrug, another bridge across the Maas river. It’s not going to be an exact match for the kind of traffic we might expect on OUR bridge, but it might give us some ideas. So we can scout this out the way we did the street near our bridge.

One difference we can see right away is that this bridge has stairs on both ends. Our bridge isn’t going to have that. The stairs do have the lip to make pushing a bike up it easier, but it’s not as easy as riding through, and any heavier vehicles are out. And people are probably less likely to go up stairs than they are to walk up an incline. So we already know that this bridge will probably experience less traffic than our bridge, but it’s still good to take notes on.

Okay, let’s head back home, write down our findings, and think about what we’ve learned.

Tying it back to software testing

What we learned was useful, but there are still a few things we don’t know:

It is 10 AM right now. But is that really the highest point of traffic during the day? To be really sure, we should really try to go back at different times. Maybe there’s an increased amount of traffic during lunch as people go to restaurants or back to their homes to eat. Maybe a horde of kids would go across our bridge after school to go home.
And we should think beyond that, too. Are weekends busier because more people are out? Or are weekdays busier because people are going to and from work?
Then there are some things that are seasonal. Maastricht happens to be in the part of the Netherlands that celebrates Carnaval, so sometimes that area looks like this. Carnaval involves a slow parade of people, bikes, and trucks through the bridge to the central town square, called the Vrijthof. That’s absolutely something we’ll have to plan for when testing our bridge.

We’ve been talking about the bridge this whole time, but as load testers, we go through the same process for software, too. What we’ve just done on our field trip is called a workload model. A workload model is describes the load profile for a given test scenario, and it involves determining what (the key transactions), how much (the load distribution among the transactions) and when (timing of the load) to test.

For a web application, for example, like a web site, we might try to look through analytics and monitoring tools to try to determine things like:

How many users access our site?
Which pages do they visit? Some pages are like pedestrians - they don’t have much comparative weight and aren’t complex to process. But other pages are like big old trucks, that involve many components of our application and are comparatively very intensive.
When do they visit the site? Is our site the sort of site people go to to work, or to avoid work? at weekends, or during the week?
Are there any seasonal events, like sales or holidays, that might drive MORE people to our site?
Where are the users from? Are they in the same country that our application servers are located in, or are they quite far away? What sort of internet speeds do they have? What percentage of them use computers to access our site, and what percentage of them use tablets or mobiles?

That was just a taste of the kinds of questions a load tester has to answer during the course of testing. There’s still a lot more to cover - for starters, we haven’t even talked about how to actually RUN the tests yet! I plan to make more videos going through typical activites in load testing in more detail, but for now, I hope that gave you a good idea of what load testing is. And Mom, I hope you know what I do for a living now. :D

Thanks for watching!

Interview on Performance Time Podcast

Mon, 31 May 2021 11:28:18 +0200

I first heard of Stephen Townshend because he happens to have gotten the same introduction to performance testing that I did– our shared mentor, Stijn Schephers. He has started a podcast called Performance Time, where he interviews performance engineers about a variety of topics. This week, it was me!

We talked about how I got started, one of my biggest failures as a performance tester, chaos engineering, what we can do to improve diversity in tech, and why I think note-taking is a key skill for any performance tester.

Application Performance Monitoring using Netdata and k6, with Odysseas Lamtzidis (k6 Office Hours)

Fri, 28 May 2021 23:24:48 +0200

This week, we had Odysseas Lamtzidis on the show to talk about Netdata, which is an application performance monitoring tool that prides itself on having the easiest setup ever. Odysseas admitted that it might not have all the bells and whistles of other monitoring platforms, it Just Works. I see this as being very interesting especially for testers like me, who just want to have something set up but might not want to be bothered by the implementation.

End-to-End Guide for Load Testing

Mon, 24 May 2021 10:48:55 +0200

Originally published on the Abstracta blog.

If you have a public-facing web application, the odds are you should probably be thinking about load testing. Any tester can (and should) add load testing to their repertoire.

Functional tests are often carried out with a single user’s experience of your application in mind, but what happens when hundreds or thousands of users are on your site at the same time? Load testing helps prepare an application for this eventuality by simulating that situation before the application goes live. That way, you have time to fix any issues you find, and there are no nasty surprises in production.

In this guide, I’m going to walk you through the basics of the load testing process, which fits in closely with traditional functional testing phases as well:

Planning: Why are we testing? What are we testing?
Writing test scripts: How are we testing?
Execution: Running test scripts against the application.
Analysis and reporting: What happened, and what do we need to fix?

Planning for load testing

During the planning phase, the team gets together and determines what load testing will involve, including the objectives, scope, resources needed, and test scenarios. The deliverable for this phase is a test plan, but that can be as formal or informal as the team requires.

Planning for any testing is a team activity, and you’ll want to get input from all the stakeholders – that’s anyone who is involved in or might be affected by the testing. The first thing you need to clarify is the reason for load testing.

Requirements

Why are we doing load testing? seems like a very simple question, but you may be surprised by the varying answers from your stakeholders. Their reasons may require different test scenarios, so it’s a good idea to agree on the collective objectives of this load testing cycle so that expectations are set accordingly. A requirement is a concise statement of how an application should behave in a given situation.

Phrasing objectives as nonfunctional requirements (NFRs) is a good way to formalize them. An NFR defines the desired state of the application and can includes performance thresholds that the team decides on. For example: Logging in as a registered user must take no more than 3 seconds (90th percentile response time) is a requirement based on response time. Good requirements are SMART (Specific, Measurable, Agreed Upon, Realistic, and Timely).

Here are some other things that should be decided on during this phase:

Scope. What are we testing? Deciding on a shared objective for load testing lets you narrow down what areas of the application need to be targeted. Make sure everyone is clear on what won’t be tested.

Photo by Daria Nepriakhina on Unsplash

Entry Criteria. What do you need to begin? This is the time to start thinking about resources that are required for testing, including the number of people who need to be involved, the time it takes to carry out the testing, and environment requirements (including test data). You don’t always need a separate testing environment to do load testing, but if you do, that should be discussed up front so that the DevOps team knows you can’t begin without an environment.

Workload Modeling. A workload model is a set of characteristics that describe the load you’re attempting to simulate. If you want your load testing to simulate registered users accessing your site, what does that entail? This might require you to look through your site analytics to determine which pages are visited most often and which application features are most commonly used, or arrive at an estimate if analytics are not available. Are most users visiting from their laptops at lunch on work days, or do they use your site from mobile devices after work? Are most of them located in the US or in Europe? You’ll want your scripts to match real load as much as possible, so take note of key characteristics that might have an effect on the traffic your application experiences.

Server monitoring. Which application components will be involved? You want to have a plan in place to have monitoring, alerts, and logs set up on every server that will be tested by your scripts. If your load testing tool shows performance degradation, you want to be able to have information to look through on the relevant servers to troubleshoot that issue. Keep in mind that any sort of monitoring adds some resource overhead, so decide as a team what information you absolutely need to track.

Test Scenarios. Which situations do you want to recreate? A test scenario objectively describes the situation to be simulated. Do you want to recreate the load profile of a typical day, or one from a very high-traffic period, such as during a sale? Do you want to recreate a production outage? You can use analytics data to come up with parameters for your each test scenario such as duration, number of concurrent users or requests, ramp up period (how quickly the application went from 0 users to your highest number of users), or how “spiky” the load needs to be (how the number of requests per second fluctuates throughout the test).

Everything discussed in this phase should be written in some sort of a load testing plan, which could be a formal, multi-page document, an epic with many user stories, or a bunch of Trello cards. Use the tools your team already uses. The format isn’t important; formalizing the outcome of the discussions is what counts. The test plan makes sure everyone has the same intent with regard to the testing.

Writing load testing scripts

Now it’s time to start writing scripts! You’ve already front-loaded a lot of the important decisions to the planning phase, so all that’s left to do in this step is to implement the plan in the form of load testing scripts.

Tool selection

There are many load testing tools available on the market, and the reality is that you don’t always get to be the one to choose what you use. Often, companies have already made agreements with tool vendors for licenses before you begin your load testing, and it would be financially infeasible to switch to another tool. However, if you’re lucky enough that you can choose the load testing tool, here are a few considerations for assessing different tools:

Resources and costs

How much does it cost? Does the pricing model allow for scaling usage up and down as necessary?
Should you build or buy an end-to-end load testing solution?
What technical skills or experience with tools does your team have? Who will be doing the testing?

Features

Does the load testing tool support the protocols and technologies used by the application?
How easy would it be for the team to learn this tool and/or scripting language?
Does the tool make it easy for multiple people to collaborate on the same script?
Does it integrate with other tools in the testing stack (for test management, test result analysis, server monitoring, team communication)?
Could you add this tool to your CI/CD pipeline?
Does the tool have an easy way of setting thresholds or SLOs for performance?
Are the test results displayed in usable reports, or does the tool integrate with existing data visualization tools you are using?

Support

If something goes wrong, does customer support cost extra?
Is there ample documentation available online for getting started without help?
Is there a community of users of the tool whose experiences you could learn from?

Scaling up load testing

How many virtual users or requests per second can a single load generator simulate using the tool?
How many load generators would you need if you used this tool?
Can the tool handle the amount of test data required by your tests?
Can you execute tests from different geographical locations, depending on your the location of your users?
Does the tool allow you to generate load from the cloud?

These questions are not exhaustive, but you can use them as a starting point for your comparisons of load testing tools.

For this guide, I’m going to outline a basic script in k6 for a few reasons:

It’s open-source, so anyone reading this can download the tool and the script and try it out for themselves;
It’s written in JavaScript, which makes it easy to learn for front-end developers or testers who have used tools like Cypress, Puppeteer, or Playwright.
It’s capable of generating tens of thousands of users on a single machine, so it’s cost-effective even with load ramped up.

However, the principles I discuss will be applicable to other tools. I highly recommend you conduct your own proof of concepts with several tools to come up with a tool that fits your particular situation, test scenarios, and company.

Installation and setup

If you’re on a Mac, the easiest way to install k6 is to use Brew. Run brew install k6 from your terminal, and you’ll be ready to go in a few seconds.

If you’re on a different operating system, check here for installation instructions that apply to you.

A basic k6 test script

A barebones k6 test script looks like this:

import http from 'k6/http';

export default function () {
 let response = http.get('https://test.k6.io');
}

This script sends an HTTP GET request for the URL https://test.k6.io, which is a simple test site. If you copy that code to your favorite text editor and save it as test.js, you can then run it with the command k6 run test.js.

You’ll get something like this in response:

A sample run of a simple local test in k6

The output above means that you’ve executed 1 iteration of your script with 1 virtual user from your local machine. The test run consisted of 1 HTTP request, which had a response time of 128.2 ms (http_req_duration).

It might be tempting to start scaling up the test at this point, increasing the number of virtual users and the duration of the script. But there’s one problem with the script as it is: it’s not realistic.

Making a test script realistic

The purpose of a load testing script is to simulate a real user who accesses your site. In most cases, that involves more than the simple GET request from the script above. Load testing scripts don’t have to mimic a real user exactly, but they do have to be close enough so as to apply the same amount, shape, and distribution of the load on your application server. If your load testing scripts aren’t realistic enough, you may get false positives during testing and discover too late that there is a performance issue in an application component that your scripts didn’t test.

Here are some ways to make your scripts more realistic:

Add static resources

If a page has static resources like images and scripts that are retrieved by a browser automatically, make sure your script retrieves those too, as they will have an impact on the response times you get. Consider excluding static resources fetched from third-party providers unless you’ve gotten their permission to run a load test on their servers.

Review your load testing tool’s default cache and cookie management settings and change them, if necessary, to match the scenario you want to test. First-time visitors to a site won’t have resources cached locally, but repeat visitors may be retrieving resources from the cache.

Add think time

Real users don’t usually fetch the same URL over and over again without stopping. They navigate to your site, then spend some time reading the contents of the homepage before they interact with the page again. Think time is the amount of time that a user spends “thinking”— that is, the delay between requests. Adding a think time that varies between 1 and 5 seconds, for example, prevents the application server from getting artificially bombarded by requests in a way that wouldn’t occur in production.

Using different user paths

A user path is the flow of how a user interacts with the application. What pages do they visit? What information do they type in? So far, you’re only testing one request, but perhaps your users may be performing other actions. The workload model that you put together in the Planning phase will come into play here. Other actions like logging in or paying for items in a cart exercise different parts of an application than browsing does, so the load testing script should reflect the most common user flows through your system. In k6, this is done by adding scenarios, but other tools may call them thread groups or execution groups.

Adding test data

In the same way that users don’t (typically) search for the same keyword over and over again and don’t all log into the same test account, a script shouldn’t use the same data repeatedly. This behavior could cause responses to be cached, leading to misleadingly quick response times, or increased error rates, in the case of multiple virtual users attempting to log into the same account. To prevent that, modify your script to take data from a CSV file instead, letting you add some natural variation.

Setting test parameters

Test parameters are the main characteristics of the test and include the number of users, ramp up, duration, and ramp down. They control the shape of the user load, which you can see when you plot the number of virtual users over time during the test:

The number of virtual users over time during a load test, as graphed in k6

Ramp up is a period of time during which virtual users are gradually added, usually at the start of the test, and ramp down is when they are gradually stopped. For most applications, ramps are more realistic than all users clicking a link at exactly the same time.

Setting failure criteria

Part of simulating real users is determining what constitutes failures in user experience. You can incorporate failure criteria into scripts by adding response verifications, checks, and thresholds (such as those for response time or error rate). Building these into the script is a good practice to get into, because it defines the limits of acceptable performance from the beginning.

The suggestions in this section are for general use, but there are many situations where taking the opposite approach would be more realistic. Spend some time thinking about what is “realistic” for your application and test scenario.

For your reference, here’s a k6 script that started from the basic script I showed earlier, but has been modified to be more realistic. That link includes a CSV file with login credentials.

Executing load tests

In the course of writing the test script, you probably would have already executed small verification tests, called shakeout tests. It’s a good idea to start with these shakeout tests on a new environment to make sure that both your test artifacts (your script and any other data files) and the environment are ready for full testing.

Running the test

If you’ve been following along with the k6 script so far, all you need to do to run a load test is to execute the following command:

k6 run test.js

This command will run the k6 script according to the test parameters set within it. The test will be executed locally. This is best used for shakeout tests, or if you prefer to run your load test on your own infrastructure.

Realism in test execution

One way can make executing load tests realistic is by matching the location of load generators to the physical location of your users. When you run a load test on machines in your data center, in the same network as your application servers, you will likely see response times that are significantly faster than your users would see. These false positives can look good on the report, but they don’t help you measure real user experience.

Running on the cloud is an easy way to make tests more realistic, especially if most of your end users are external to your organization.

If you’re following along with the k6 script, you can sign up for an account with k6 Cloud (the first 50 tests are free). Since k6 itself is open source, you can also run it on your own cloud infrastructure, but using k6 Cloud is the most convenient way to get started.

Once you have an account, copy your API token from here. Then, run this command in your terminal to give your local k6 installation permission to run on your account:

k6 login cloud --token <YOUR_K6_CLOUD_API_TOKEN>

Once you are successfully authenticated, you can continue to run tests locally using k6 run, but you now have the option of running on the cloud as well:

k6 cloud test.js

k6 will then begin to execute your load test, and you’ll see a screen like this:

Note that the execution mode is now cloud, to remind you where you’re running the test.

By default, your test will run on the k6 AWS account, in the Ashburn region. However, you can override this and specify other regions, including percentages of your virtual user to run on each, using cloud execution options.

Analyzing load testing results and reporting

Although I’m listing analysis as a different step in the process from execution, the two often (and should) overlap. The output from k6 in the screenshot above contains a link to the dashboard on k6 Cloud, so that you can see the results of the test in real time, as the test is running. Monitoring the test in real time lets you immediately spot issues as they arise, and, if necessary, stop the test to fix them. If the test is being executed by multiple load generators, having a consolidated dashboard is helpful in giving you an overview of the entire test.

![An example of the results after a load test in k6 Cloud](/assets/Finished k6 run.png) An example of the results after a load test in k6 Cloud

This is also a good time to monitor the health of the application servers and the load generator servers, which you set up monitoring tools for during the planning phase.

A load test is only useful when it’s providing information about the application. If errors occur in large quantities, or if the response time makes it clear that a certain component has failed to handle the load, don’t be afraid to stop the test. Your objective is to get the information you need to suggest performance improvements and solutions for any bottlenecks you find.

Often, you’ll run into issues you can’t solve or even identify during a test. That’s why it’s useful to explore the data that you have after a test. k6 lets you create shareable links to dashboards so that you can involve relevant engineering teams who may be able to help you understand what happened and why. Here’s an example of what that dashboard looks like.

Some projects require formal documentation of test execution by way of a Test Summary Report, but in more agile teams, it may be enough to share the test results dashboard and some supplementary information about what type of tests were run and what problems were encountered.

The most important part of load testing isn’t the execution itself; it’s what you do with the results. Communicating results clearly to stakeholders will make it more likely that performance issues are fixed.

What’s next

The phases I’ve outlined here are a good guide for anyone starting with load testing, but I always recommend that you take what works for your situation and leave the rest. In reality, the phases overlap with each other considerably, especially in more modern engineering cycles.

Another activity I’ve not covered here, but one that I recommend, is adding load testing to your CI/CD pipeline. This continuous testing approach involves automating load testing so that it occurs in regularly, in parallel with other development and testing activities. Continuous testing is a great way to get more value out of load testing, and it ensures that performance is front-of-mind for everyone on the application team.

Happy testing!

How I Use Obsidian at Work

Tue, 18 May 2021 22:33:18 +0200

Note-taking and personal knowledge management get talked about in academic circles, by both students and teachers. But there’s not much said about its use outside of the academe.

I’ll admit, due to my job description, I may be in a better position than most to take advantage of note-taking in a professional context. My background is in software performance testing. My job for several years has been to make sure that applications are prepared to withstand traffic from many users accessing it, usually by writing code to simulate that traffic before it’s released to real users. Currently, though, I’m a Developer Advocate at k6.io, which means a significant part of my job now is sharing with people what I’ve learned.

I consider myself a knowledge worker: a significant part of my job depends on how quickly I learn new technologies and how clearly I can impart what I’ve learned to others. I’d argue, though, that most people who work in a field as fast-moving as tech are knowledge workers too. It’s naive to think we won’t one day forget some skills we’ve learned, even if we were using them daily at one point. Yet nobody talks about note-taking.

Recently, my note-taking and personal knowledge management tool of choice has been Obsidian.md. I have no affiliation with the Obsidian team, and in fact I’m an investor of a competing product. Here are some ways I use Obsidian at work that might help you understand why Obsidian is my daily driver.

My setup

I have multiple Obsidian vaults, and each of them is a version-controlled Git repository, hosted on GitHub. I keep my main Obsidian vault in a Dropbox for extra version control and as a backup. Just for overkill, I also use Obsidian Sync as another backup and as the easiest way to get access to my Obsidian vault on my iPad Pro and my Android mobile.

I use Obsidian Publish to make some of my notes publicly available on a custom domain directly from my vault, as an experiment in learning in public that’s gone pretty well so far.

This setup isn’t free. All up, I’m giving Obsidian about US $16 a month. However, I want to point out that Obsidian itself is free for personal use, and if mobile access, cloud sync, and publishing your notes online aren’t important to you, you can really go a long way with using it for free.

Processing information

Remaining relevant in tech pretty much requires that you learn a lot, learn quickly, and enjoy the hell out of it. Luckily, Obsidian is pretty great at this.

Consume and collect

I consume a lot of content, and in different formats: articles, books, videos, presentations, podcasts are all great sources of free education. I initially capture my notes on them in different ways, but they all lead back to using Readwise, which I use as a collector and sorter of all these notes. Then, I import data from Readwise to Obsidian using a process I describe in more detail here.

For now, suffice it to say that all these notes and highlights end up in Obsidian for processing. Here’s an example of my notes during a presentation (called In the kitchen - a sprinkle of fire and chaos by Ana Medina, about chaos engineering).

When trying to learn something, many people stop at this phase. They’ve read the thing, they’ve highlighted passages from the thing, and now they think they’ve learned the thing. That hasn’t been my experience, because I forget many things I consume shallowly.

Process iteratively

Then, I process notes iteratively, first restating things into my own words and then creating my own notes about concepts I’ve learned. Each time I do that, I try to identify key concepts and create backlinks for them, which in Obsidian just means typing [[Chaos engineering]], with a topic between double brackets. If I click on that new link, it creates a page for it. Over time, I end up with a page like this one on Chaos Engineering that is a consolidation of what I’ve learned.

Interestingly, if I go to the Chaos Engineering page, I can also see all the instances I’ve linked to it from other notes:

What if I didn’t have the foresight to link to Chaos Engineering from an article I read? Obsidian also tracks unlinked mentions as well. I can see all the times I’ve used the words “chaos engineering” in other notes, even if I haven’t created formal links using [[Chaos engineering]].

Over time, things I’m interested in start to emerge this way, and topics grow deeper and more detailed, with branches of pages spinning off from it.

Situate and solidify

Then, I try to think about where this new piece of knowledge fits into the hierarchy of things I’m interested in. This is less about trying to classify the world and more about putting it somewhere on my personal map of interests so that I’m more likely to use that knowledge later. For me, chaos engineering fit into my page on Nonfunctional testing. Doing so helps me pit it against other things I’ve classified as forms of nonfunctional testing and think critically about whether it belongs there, or whether other things I already know contradict it in some way. Sometimes I’ll make notes about my reasoning for classification, such as this note on why I think chaos engineering is a testing discipline.

None of this is set in stone. I’m always changing notes around as I learn more things that change my mind. For example, I’m considering merging my nonfunctional testing and functional testing pages after a conversation where someone questioned whether it wasn’t a misnomer to call application performance “nonfunctional”. The point is to create a living, breathing second brain that helps me make connections between concepts and remember what I’ve learned.

Obsidian’s built-in graph view illustrates this point nicely:

This is my main Obsidian vault as of the time of this writing - all 5443 notes of it. The graph view plots every note and the connections between notes, so I’ve ended up with a slick visual representation of what I’ve been thinking and writing about. The graph view is actually functional, too– I can assign colors to nodes based on the directory they’re in, their tags, words they can contain, and a lot of other criteria. Because each node is sized according to how many links lead to it, this view makes it easy to see clusters in things I’ve thought or written about.

Publish

One of the advantages of Obsidian over something like Notion or Roam Research is how easy it is to publish notes from it.

Obsidian Publish (paid)

There’s the paid Publish service, which I’ve subscribed to for US $8 a month. After setting it up to publish on my own domain rather than the default at publish.obsidian.md, going from a note to a published post is as easy as clicking the Publish icon, selecting the notes to add, delete, or modify, and then selecting Publish.

Open folder as vault

If you don’t want to pay for Publish, Obsidian still saves all files locally in Markdown (.md) format. This is the same format that many static site generators take for content. This site that you’re reading this on is generated using Hugo, so everything written here is kept as a Markdown file before it’s turned into HTML. Since an “Obsidian vault” is just a folder with Markdown files, I can just open the directory containing the files for my site and write blog posts in Obsidian with no noticeable difference.

At work, we happen to store our main product documentation as Markdown files in a GitHub repo, which make it perfect for opening in Obsidian as well. I prefer to write documentation in Obsidian over VSCode for a few reasons:

I can install plugins, like Advanced Tables to make Markdown tables easier to deal with, create templates to standardize the format of notes you make often, see an outline of what you’re writing based on Markdown headings.

Outline core plugin for Obsidian

When I want to link to another page, it’s easier to just hit CMD + O to search through existing notes instead of finding it on GitHub.
I can choose an Obsidian theme that matches the theme of the published site. This “Gitsidian” theme has an Edit view in dark mode (my preference) but a rendered Preview mode that looks more like GitHub’s Markdown format, so I can see what it will actually look like for users.

When I’m done, I can still jump to my terminal and push my change to GitHub as normal.

Obsidian URIs

Another cool feature is the fact that every note in Obsidian has what’s called an Obsidian URI - it’s a link that opens up a note within a specified vault. The URI looks like this:

obsidian://open?vault=docs&file=src%2Fdata%2Fmarkdown%2Fdocs%2F03%20cloud%2F02%20Analyzing%20Results%2F02%20Performance%20Insights

Adding it and publishing that to internal documentation means I can switch easily from browsing on GitHub to modifying the same page in Obsidian on my local machine, which is a great little bonus.

Embedding published work

After publishing, I use iframes in Obsidian to update my notes with the final result. Here’s a video on YouTube about chaos engineering that my colleague Simme and I planned out collaboratively on Obsidian, embedded into Obsidian:

The video plays in Obsidian with all the same controls. The Obsidian desktop app is an Electron app, so it behaves just like a normal browser for playing back the video. To embed the video, I just used the Share > Embed code on YouTube and pasted it straight into Obsidian:

<iframe width="560" height="315" src="https://www.youtube.com/embed/gVwJZPo30rk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Meetings

I take meeting notes to help me focus on the meeting, and I create templates for recurring meetings. Templates let you define the format of a note. Once created, you can insert templates into notes with a keyboard shortcut (I set mine to CMD + ;). Here’s an example of what that looks like, with the template on the left and the inserted template on the right:

Template (left) and inserted template in a new note (right) in Obsidian

You may notice that the frontmatter includes {{date}} in the template, which gets automatically converted to today’s date when the template is used.

If anyone asks for a copy of my meeting notes, that’s no problem– any note can be exported to Obsidian by clicking on Options > Export to PDF.

Options for exporting a note to a PDF in Obsidian

I don’t want to actually publish any real meeting notes, so here’s what my note on the chaos engineering process looks like as a PDF, straight from Obsidian. And here’s what it looks like as a note.

Presentations

The easier way

Obsidian has a presentation mode! It’s basic, but it’s functional. You can create a presentation really easily by adding --- between paragraphs of text to delineate slides, right within a note. Here’s what that looks like:

Then you can click Options > Start presentation, and you get something like this:

A little plain, but given the minimal amount of effort to “convert” a note into a presentation, it’s pretty awesome.

The harder way

Sometimes the basic slides just don’t cut it, and it can be handy to be able to publish the slides in presentation format (since publishing them straight from Obsidian Publish would only publish the Markdown version). For those cases, I’ve figured out a way to do just that while still keeping my slides as Markdown files in my vault. Here’s a whole post on how to do that. This is definitely more complicated, but at the end of it, you get a nice page like this to send to attendees after a presentation. (Tip: Hit s to see the speaker notes while on those slides.)

Task management

I use a few community plugins to keep track of things I want to.

Kanban

The Kanban plugin turns Markdown task lists into full-on Kanban boards.

On the left is the markdown note, and the on the right is the rendered Kanban board. You can customize the titles of the lists that are there, and you can add due dates to each task.

Calendar

The Calendar plugin is Obsidian’s most popular plugin, and with good reason.

It lets me add a calendar on the sidebar:

I have the weekly notes option enabled, so clicking on the W column in the calendar takes me to a note that is automatically created for that week. I use it to keep track of weekly goals and to do a review at the end of the week.

Clicking on a date in the calendar takes me to the…

Daily Note

The Daily Notes core plugin comes with Obsidian, but you have to enable it. Once enabled, clicking on a day in the calendar creates/takes you to that day’s corresponding note. I use daily notes like a log or workbench for that day, with timestamped headings to help me remember later on what I did at what time. It’s a loose way to do time management as well.

Dataview

The Dataview plugin is my favorite recent discovery, and it’s changing the way I do frontmatter. This plugin brings something akin to SQL (albeit not as powerful) to Obsidian. It lets you query your notes based on some file-related metadata as well as parameters you set in your frontmatter.

For example, this query:

table file.ctime as Date
from "Meeting"
where file.ctime >= date(today) -dur(1 month)

when rendered (i.e., viewed in Preview mode in Obsidian), returns a list of all my notes under the subfolder Meeting in the last month:

This is huge! I’m still updating my notes with the appropriate frontmatter to make more use of this feature, but here are some use cases I can think of that would be handy at work:

Find all customers with tag annual that I haven’t spoken to in the last year.
Find a load testing tool with feature Slack_integration = true.
Find that woman I met at the Automation Test Guild conference by searching for job = tester and tag = AutomationTestGuild.
Return all notes tagged #plugin_ideas.

And that’s just the beginning!

Collaboration

Obsidian itself does not (yet) have collaboration features. This is a bummer, but given how quickly they’ve delivered on features that the community has clamored for (block references and mobile apps come immediately to mind), I’m hopeful that they’ll eventually improve in this area.

In the meantime, I’ve still found ways to work on the same vault with other people.

Remember how any folder of .md files can be opened as an Obsidian vault? Well, I’m technically collaborating with other people on that repo for work documentation, even if most people don’t use Obsidian. All I had to do was add the .obsidian directory to the .gitignore file, and now, nobody really even needs to know I’m using Obsidian to view/edit the documentation. They just see me pushing to the repository as if I had used a text editor like VSCode.

I’ve also had the experience of collaborating with someone who also used Obsidian. Since we were working closely together on the same document, we opted to put our shared vault on Dropbox. Dropbox’s main advantage over the Git solution is that Dropbox let us edit notes and see each other’s edits in real time. If you’re looking for an experience similar to Google Docs, this way got the closest to that. We did run into some quirks: if the Dropbox vault is also a Git repository, be aware that checking out a different branch will also change the contents of the directory for the other person.

Conclusion

Note-taking and personal management aren’t just for students and teachers; I think they’re particularly useful in tech, where knowledge work is highly prized. I’m talking about Obsidian because that’s my current tool of choice, but I think the tool you use is less important than the systems you put in place to help yourself learn and create.

If you’d prefer the video version, here’s a video I made showing this process:

Learning how to use k6 live with Nočnica Fee from New Relic (k6 Office Hours)

Fri, 14 May 2021 21:53:09 +0100

We were happy to welcome Nočnica Fee from New Relic to k6 Office Hours. She very bravely volunteered to learn how to use k6– live! Talk about pressure. But she handled it like a champ!

We also talked about why observability is both particularly important and particularly difficult in serverless architecture. There will definitely be a part two at some point!

Analyzing load testing results in k6 Cloud

Tue, 11 May 2021 21:53:09 +0100

Have you ever run a load test but been unable to decipher what it all means? In this video, I break it all down using k6 Cloud.

Performance testing tool feature requests (k6 Office Hours)

Fri, 07 May 2021 21:53:09 +0100

We managed to get two other folks from k6 to join our Office Hours: our CEO, Robin Gustafsson, and our Head of Customer Success, Mark Meier. We took advantage of the opportunity to discuss wishlist features for k6 and bug Robin about how likely each feature was to see the light of day!

Using the k6 Cloud interface

Tue, 04 May 2021 21:53:09 +0100

I’ve gotten the feedback that people would like to see more videos of the basics of using k6 and k6 Cloud, so here’s a video on getting situated with the k6 Cloud interface and navigation.

How to get started with chaos engineering (k6 Office Hours)

Fri, 30 Apr 2021 21:53:09 +0100

Recently, I’ve been learning more about chaos engineering. In this week’s k6 Office Hours, I shared my experience as a tester getting started with Kubernetes, chaos engineering with xk6-chaos and k6, and observability with New Relic.

Shift-left testing and creating a culture of quality with Simon Prior (k6 Office Hours)

Fri, 23 Apr 2021 21:53:09 +0100

Simon Prior from the Testing Peers podcast joined us on k6 Office Hours to talk to us about how he’s managed to foster a culture of quality at his job with EasyJet.

The future of load testing with k6, with k6 CEO Robin Gustafsson (k6 Office Hours)

Fri, 16 Apr 2021 21:53:09 +0100

This week, Simme and I convinced k6 CEO, Robin Gustafsson, to come onto k6 Office Hours to talk about the origins of k6 (spoiler: it began with a company called Gatorhole), things we’re working on right now, and what he sees as the future of the company.

Testing Kubernetes and infrastructure as code with Sid Palas (k6 Office Hours)

Fri, 09 Apr 2021 21:53:09 +0100

For this week’s k6 Office Hours, Simme and I had Sid Palas from DevOps Directive come on board to talk about what Kubernetes is, how it relates to Infrastructure as Code, and why either of those should matter to testers.

Using CircleCI and environment variables with k6 (k6 Office Hours)

Fri, 26 Mar 2021 21:53:09 +0100

Simme and I decided to take this week to answer questions on k6 Office Hours, and we ended up talking about CircleCI, environment variables, and how to use them with k6.

Performance Testing with k6 and Cypress (Ministry of Testing)

Tue, 23 Mar 2021 19:52:48 +0100

My friend Marie Drake and I teamed up to give a presentation about performance testing at a Ministry of Testing (Philippines) meetup, and we talked about using Cypress for front-end performance testing and k6 for back-end performance testing.

Performance testing vs. load testing with Señor Performo and Pepe Cano (k6 Office Hours)

Fri, 19 Mar 2021 21:53:09 +0100

What’s the difference between performance testing vs. load testing? This week, Simme and I were joined by Señor Performo, aka Leandro Melendez, and Pepe Cano, the Head of Marketing at k6, to talk about this common question.

Load Tests as Code: An introduction to k6

Thu, 18 Mar 2021 20:36:41 +0100

Today, I attended the Cambridge University Press QA Week 2021 to talk about load testing with k6. Here’s a recording of the presentation.

And here are the slides I used!

I had fun discussing the advantages of the load test as code approach and why k6 is a great way to implement it.

Testing Our Performance Test Tools

Tue, 16 Mar 2021 21:29:33 +0100

On this episode of the TestGuild performance podcast, I had a chat with Joe Colantonio about languages, board games, nontraditional backgrounds in tech, and why we should all test our performance test tools.

Lucky Dippers 103: Sleeping in the Isle of Skye | Portree, Scotland

Fri, 12 Mar 2021 21:57:56 +0100

This is a travel video from January 2020, when my husband Rob and I did a roadtrip in Scotland with our good friends, Nikki and Basti Filler. This is towards the tail end of that trip, and it involved some of the best scenery I’ve ever seen. Scotland is a gorgeous country that we thoroughly enjoyed exploring, despite its awful weather. As a bonus, this was also our first trip with our new niece, Cassie.

k6 Office Hours 05: Observability with Daniel González Lopes

Fri, 12 Mar 2021 21:53:09 +0100

Our first repeat guest, Daniel González Lopes, also from k6, joined us today to talk about how to use observability in conjunction with load testing. We talked about the differences between metrics, logs, and traces, and how to implemented distributed tracing.

k6 Office Hours 04: Using Cypress with Marie Drake

Fri, 05 Mar 2021 21:46:56 +0100

Marie Drake joined us today on k6 Office Hours to talk about front-end performance testing using a test tool called Cypress and how it can compliment a back-end performance testing strategy.

Presentation Slides as Code

Tue, 02 Mar 2021 19:07:31 +0100

I was working on a presentation for later this month when Marie Drake told me about slides.com. It’s basically a Powerpoint replacement (which, let’s be honest, needs replacing) in that it lets you create slides and it lets you present from it. It even goes beyond Powerpoint in that it also saves your slides publicly for people to view later.

Before she’d mentioned slides.com, I had been planning on creating a presentation on Obsidian, my note-taking tool of choice. The advantage of this approach is that presentations in Obsidian are just Markdown files with --- between lines of text to delineate slides.

The problem with using Obsidian on its own is that it’s severely lacking in features, and it’s also not shareable. You can, of course, publish the Markdown file, but people won’t be able to view them in slide format. After playing with slides.com, however, I also discovered it had its own weaknesses. It is completely UI-driven, so there’s no way to create slides just from a Markdown file. It’s also hosted by them, so there are limitations imposed, depending on which pricing tier you pay for.

So I got to Googling. And I think I found a pretty good setup for my needs.

The end result

I write the presentation in Obsidian, in Markdown. Here’s what that looks like in a sample presentation where I’m trying out a bunch of features.
I use Hugo to generate a static website and convert the Markdown files into javascript and html, using reveal.js and the reveal-hugo theme.
I push the static website to a remote repository on GitHub. GitHub Pages serves that site (for free!) on my custom subdomain.

Check out the final product here: https://slides.nicolevanderhoeven.com/sample

The setup

Prerequisites

An Obsidian vault (or you can actually just use any text editor)
Git installed and set up with credentials
A GitHub account
Hugo installation (brew install hugo)

Create a new Hugo site

Open up a terminal window and cd to a folder in your Obsidian vault. Then, create a new Hugo-generated site:

hugo new site slides

Replace slides with the name of your new site.

You should have a new directory, slides, in your Obsidian vault.

Add the reveal-hugo theme

cd into slides and clone the reveal-hugo repo. The documentation suggests using a submodule:

git submodule add git@github.com:dzello/reveal-hugo.git themes/reveal-hugo

Personally, I just cloned it because I know I’m probably going to be messing around with it substantially:

git clone git@github.com:dzello/reveal-hugo.git themes/reveal-hugo

You should have a /themes/reveal-hugo folder that contains the theme.

Configure reveal-hugo theme

Set up your new Hugo site to use the reveal-hugo theme by adding the following to /config.toml:

theme = "reveal-hugo"

[markup.goldmark.renderer]
unsafe = true

[outputFormats.Reveal]
baseName = "index"
mediaType = "text/html"
isHTML = true

The Reveal part is what lets you output text in slide format.

Create your slides

Go to /content and create a file there, called _index.md. The filename is important, as is its location (I learned that the hard way). This file is going to be the first page people see, so it should probably be a list of your presentation slides. For now, put in something like this:

+++
title = "My presentations"
outputs = ["Reveal"]
+++

Here are my presentation slides!

[sample presentation](/sample)

Then, create a new folder for your presentation, such as sample. In /content/sample, create a file called _index.md. Again , the filename is also important here (guess how I learned that).

This index file is going to be where you actually create your slides. Here’s the context of my /content/sample/_index.md:

Each slide is separated by ---, just like in Obsidian, which apparently uses the same converter that reveal-hugo does, reveal.js. Note that if you want to try out having an image as a background to one of the slides, you’ll need to add that to your /sample folder and change the filename in the slide in this part:

slide background-image="lucrezia-carnelos-AsM4T46tqcU-unsplash.jpg"

Generate your site

Now it’s time to generate your site! Wait - didn’t we already do that? Sort of that. That was creating the framework for the site, but now you need to get Hugo to convert those Markdown files you’ve just created into something publishable. So cd to the root directory slides and run this:

hugo

That command should generate a slides/public folder, where all the publishable code will go.

At this point, you can also check your work by starting a local server with hugo server and navigating to http://localhost:1313.

Push your site to a remote GitHub repo

cd into your public folder. This is important, as I initially started the repo in slides and it took me ages to realize that GitHub Pages requires an index.html in the root directory to function.

Initialize a Git repo:

git init

Stage and commit your changes:

git add .

git commit -m 'Initial commit'

Create a repository on your GitHub account called username.github.io where username is your GitHub username. (If you already have a repo called that, you can alternatively push the contents of public to a branch called gh-pages). The repo name matters.

Push your changes to your repo.

git push -u origin master

You should now have a repository on GitHub called username.github.io, containing everything in public.

Set up GitHub Pages

On GitHub, go to the repository settings and scroll down to the GitHub Pages section. Select the right branch (master in most cases) and click Save.

(Optional) Using a custom domain

If you want to use a custom domain, enter it in the repository settings from the previous step.

Then, go to your domain registrar and add a CNAME record pointing www to username.github.io. Here’s more information on that from GitHub.

It can take a few hours for DNS records to propagate, so after you set this up, take a break before you convince yourself you’ve set something up wrong. ;)

Check out your new site!

When you navigate to your new site, you should first see the contents of slides/content/_index.md, which has a link to your sample presentation. Clicking on that should reveal (ha) the slides one by one. You can navigate through them with the arrow keys, and you can also see speaker notes (if you put any in, which I did on the third slide if you copied my sample slides).

Other issues I faced

I ran into a few issues during this setup because:

I was already tracking changes in my Obsidian vault with Git
I already had a repo called nicolevanderhoeven.github.io, and I needed to create a branch called gh-pages to still use GitHub Pages.
My Obsidian repo on GitHub is a private one, but I wanted the slides to be public.

If you have a similar setup, maybe the following instructions will help.

Use worktrees, not subtrees or submodules

After attempts to make this setup work with submodules and subtrees, both of which were way too complicated for what I needed, I settled on using worktrees.

Add public to .gitignore. From slides:

echo "public/" >> .gitignore

Create a worktree for public, to be tracked on the branch gh-pages:

git worktree add public gh-pages

After you generate the static code via hugo:

cd public
git add .
git commit -m 'Commit message'
git push origin gh-pages

Your gh-pages branch should contain only the files in public.

GitHub Pages with a private repo

GitHub Pages is free, but if you want to be able to use it with a private repo, you can’t. Sorry!

You can look into using GitHub Actions to copy public from your private repo to a public one, or you could just spend the $4/month to remove the restriction. I opted to pay, because I love GitHub anyway.

Presentation as code

That’s it! I ran into a few snags while I was attempting to implement this, so I thought I’d document it here in case it helps anyone else. Despite the setbacks, I think it’s worth it– I can now whip up presentations in Markdown within Obsidian and have that published to a site I control for people to look through at their own leisure. This should speed up my presentation creation process considerably.

Theoretically.

If only I could stop procrastinating on creating this presentation by setting up a framework for creating presentations…

k6 Office Hours 03: Chaos Engineering with Daniel González Lopes

Fri, 26 Feb 2021 21:40:25 +0100

In this week’s Office Hours, Simme and I invited Daniel González Lopes, a DevOps engineer from k6, to join us to talk about chaos engineering and what it has to do with load testing. Simme also demonstrated how to use k6 to run simple chaos experiments.

k6 Office Hours 02: Observability with New Relic

Fri, 19 Feb 2021 23:24:58 +0100

For this week’s Office Hours, Simme and I invited Gary Spencer from New Relic to talk to us about why he’s recently started using k6, and why an observability platform like New Relic is the perfect companion for performance testing and observability.

The 503 Podcast

Thu, 11 Feb 2021 21:46:35 +0100

This must be the month for trying new things, because I jumped into livestreaming with the first ever k6 Office Hours just a week ago, and this week I’m jumping into podcasting with Simme Aronsson, who’s also doing the Office Hours with me. There’s definitely safety (at least psychological safety) in numbers, because doing it with someone else makes me feel a little more brave.

While the podcast is presented by k6, that’s really just a fancy term for saying that we’re both employed by k6 and we’re recording and producing the podcast on company time. Unlike the Office Hours, which is heavily k6-focused, the podcast will be about software performance and reliability in general. Simme and I happen to have different and complementary skillsets (he comes from a developer background, and I come from a testing background, and I think that makes for an interesting combination of perspectives.

Here’s the first episode! It should also be available on your audio podcasting tool of choice.

Readwise to Obsidian

Sat, 06 Feb 2021 09:55:36 +0100

August 10th, 2021: Readwise just released their own Obsidian plugin, Readwise Official, which is available now! I now recommend this method over the one outlined in this article.

I’m about 9 months into heavily using Roam Research as my primary note-taking, task management, capture-everything system, and for the most part, that’s gone very well. Obsidian is another excellent personal knowledge management (PKM) system, though, and I’ve been curious about it for almost as long as I’ve been using Roam. I’ve had it installed for a while, but I never really seriously attempted to make the switch. Until now. I’ve decided to use Obsidian as my primary PKM and zettelkasten for the next month, to see if it’s something I can switch to.

Why I’m considering switching from Roam to Obsidian is another post, probably one I’ll write at the end of this experiment, when my thoughts and opinions are more solidified. For now, I want to address one of the things I struggled with when switching over: Readwise integration.

Readwise is an app that’s all about importing notes and highlights you’ve made from all over the internet and exporting them all into a single format. Readwise supports far more import sources than I use, but here are the ones I’ve got hooked up to it:

Instapaper (for highlights and notes on online articles)
Amazon Kindle (for highlights on ebooks)
Twitter (for interesting tweets)
(Readwise) Web highlighter (for straight highlights off any webpage)
Airr (for quotes and notes from audio podcasts)

Readwise collates my activity in all of these input sources and automatically sends them to my Roam Research account. Here’s what that looks like when it gets pulled into my Daily Note:

Readwise also pulls metadata into Roam, depending on the source. Here’s what the page for the book “Ready Player Two” looks like in Roam:

Readwise integration is an essential feature for me in any PKM, because I can take notes wherever I’m consuming my content and just have it automagically appear in my notes, tagged and ready for processing. So it’s absolutely something I need to have in my Obsidian workflow.

That’s where things get tricky, because Readwise doesn’t have an Obsidian integration.

Source: Readwise Exports

The closest thing is the Markdown export, but that’s not automatic. You can manually trigger an export, and then copy those exports over into your Obsidian vault, and thne make sure they’re in the right format with the right tags. I knew I was never going to do that, so I found a way to automate it.

readwise2directory

I discovered that Readwise has an API, and briefly thought about building my own integration. Happily, I found this Python project on GitHub by nicrivard, who had already done that! Why rebuild the wheel? His solution already did what I needed:

It connects to Readwise via your Readwise access token
Checks for new highlights or updated ones
Exports the new/updated highlights to a local folder of your choosing (presumably your Obsidian vault)

It even pulls in highlight tags from Readwise, if you’ve created any (which I haven’t).

I forked the project here to add just a few minor touches:

Changed date format of imported notes
Added a section to output the import date to a comment block (an Obsidian Insider build feature) rather than YAML frontmatter, so that the date would actually be linked and backlinked.
Customized tags and filename for my use
Sort imports into folders within my Obsidian vault depending on source (Article/, Tweet/, and so on).

This works out pretty well. Here’s an example of an import from the podcast app Airr into Obsidian, in Edit mode.

It’s placed within my Podcast folder inside my value, is tagged appropriately, and contains links to the source. Switching to Preview mode shows an imported cover photo and even a playable recorder for the part of the podcast that I snipped on Airr:

So that works out well, except for one thing: firing off the script that does all this is still manual.

Scheduling the script

Luckily, it’s pretty easy to schedule a Python script to run at certain intervals. I’m on macOS, so I used a built-in utility called cron.

Give cron full system access

Cron needs to have the right permissions to run scripts on my behalf, so I had to go to System Preferences > Security & Privacy > Privacy > Full Disk Access on my laptop and add /usr/bin/cron to the “allowed” list.

Open crontab

Then, from my terminal, I ran the command env EDITOR=nano crontab -e to open up the list of cronjobs I had currently.

Set a schedule

I added a line to crontab for the schedule I wanted to create. This line has to be in the format

* * * * * command

where the five asterisks correspond to:

minute
hour
day of the month
month
day of the week

In my case, I wanted it to run every hour, so here’s what my schedule looks like:


0 * * * * cd /Users/nic/git/readwise2directory/ && /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/nic/git/readwise2directory/readwise-GET.py >> /Users/nic/git/readwise2directory/readwiseGET.log

Python issues

I had some trouble getting cron to work until I realized it was because I had multiple versions of Python installed. I ran a which python3 to find the path to the version I wanted to use and added that path to the command. I also had to cd to the project directory, for some reason.

While I was troubleshooting, I found it really useful to output to a log so I could see what had gone wrong.

Once I added the schedule as in the example above, though, it worked a treat!

Conclusion

This solution did require some setup at the beginning, but doesn’t need any maintenance once it’s going. I’m pretty happy with it as it means I can keep taking notes on various places from anywhere, and it’ll all be waiting for me in my Obsidian vault whenever I’m ready to sit down and process them into my system. Thanks to nicrivard for the repo that made it possible!

The first ever k6 Office Hours

Fri, 05 Feb 2021 21:47:16 +0100

The k6 virtual office is open!

The first time I heard about office hours was from Notion. Recently, Obsidian went in on this trend as well. The concept involves opening your virtual doors regularly to your community, to answer questions and share best practices.

A big part of being a Developer Advocate is talking to people, and even conferences these days are mostly pre-recorded. I kind of miss being able to interact with people. I miss the immediacy of live. So I talked Simme Aronsson into doing a k6 Office Hours with me.

We’re both new to livestreaming, so there’s a lot that we’ve learned and will be improving in future videos, but here’s our first attempt!

Comparing k6 and JMeter for load testing

Fri, 29 Jan 2021 23:01:18 +0100

Originally posted on k6.io.

Introduction

When I joined the k6 team in late November, the foremost question on my mind was “How does this compare to JMeter?” You see, I come from a performance testing consulting background, and in the last few years, JMeter has been far and away my tool of choice.

So what am I doing on the k6 site, talking about how much I love JMeter? Well, firstly, it turns out that the k6 team is pretty open and transparent, especially about potential improvements in k6. Secondly, I want to share my reasons for wanting to explore k6 as a load testing tool– the same reasons that ultimately convinced me to join the team that builds it.

What is JMeter?

JMeter is an open source load testing tool built entirely in Java by the Apache Foundation. It was first released in 1998, and it made waves because of its audacity in taking on more popular but proprietary load testing tools. JMeter took what other companies were charging a lot of money for and published it online for everyone to use– for free. Although scripts can also be extended using code, majority of the scripting in JMeter is done using the UI. At the time of this writing, the latest version of JMeter is 5.4.

What is k6?

k6 is an open source load testing tool released by LoadImpact in 2017. LoadImpact had previously consisted of the SaaS platform (now k6 Cloud) and professional services. k6 is written in Go, but scripts are written in Javascript. Taking the opposite tack to JMeter, k6’s main selling point is its code-based scripting and heavy prioritization of the developer experience. The latest version of k6 is 0.30.

Which is better?

This article isn’t about which tool is better, because as much as we might prefer a clear-cut answer, there’s no “best tool”. Everything depends on the situation your team is in: what, how, why, and when you’re testing.

The question that I want to answer here is: What situations does each tool excel in?

When does JMeter excel?

When you want to get started quickly

JMeter has a GUI

Many testing teams make decisions on load testing tools based on this fact alone. Sometimes, if you’re in a situation where people haven’t done load testing before and need to learn a new tool, a GUI-driven tool like JMeter is simply the most frictionless option. A tool with a GUI will be less daunting for many testers that are used to the mostly no-code UIs of the likes of Postman or SOAPUI.

I’ve always thought of JMeter as user-friendly, but a developer friend recently pointed out that the Test Plan screen that greets you when you first start JMeter doesn’t provide any indication of how to create an HTTP sampler. None of the icons add elements to the test plan. It takes a bit of exploring to realize that right clicking on the test plan, or going to Edit > Add, will show the “Add” menu that you need. He has a point there: UI is subjective to a certain degree. However, I’d argue that it’s definitely easier for non-developers to explore a UI than a bit of code.

k6 doesn’t have a GUI packaged within the tool, but the k6 Test Builder is available for free. It’s a way to create tests with a GUI interface, and despite its inclusion in k6 cloud, it doesn’t require any subscription to use. However, it isn’t as fully-featured as JMeter.

JMeter supports many protocols and features out of the box

Here’s the official list of the protocols that JMeter supports:

Web - HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, …)
SOAP / REST Webservices
FTP
Database via JDBC
LDAP
Message-oriented middleware (MOM) via JMS
Mail - SMTP(S), POP3(S) and IMAP(S)
Native commands or shell scripts
TCP
Java Objects

In contrast, k6 supports the following protocols:

Web - HTTP/1.1, HTTP/2 (Java, NodeJS, PHP, ASP.NET, …)
WebSockets
gRPC
SOAP / REST Webservices

The protocols above are those that are natively supported. However, both tools allow the development of third-party plugins, adding support for more protocols. The number of k6 community extensions, through xk6, is growing, but JMeter plugins still outnumber them. In many cases, the protocols your application uses may make the decision for you of which tool to use.

JMeter also has most of the features you’ll need for a basic load test, and you don’t have to code any of it up yourself. Its parent-child element structure means that you can modify a specific element, like an HTTP request, or modify all HTTP requests, with equal ease. Adding think time to all requests is as easy as adding a Uniform Random Timer to your thread group, and all requests within that group will inherit it. With code-based load testing tools, it can be a bit more difficult to find out what’s possible without a UI. Some of the gap, however, can be bridged by an IDE with good autocompletion.

When you want a mature community with tons of documentation

JMeter’s old, and sometimes that’s a good thing. It’s been around since 1998, so at this point, it’s had 22 years to improve and to build a following. JMeter has a presence on nearly every community networking site, and no matter how specific your use case for JMeter is, the chances are that it’s been done before. Searching “how to load test X with JMeter” is bound to yield thousands of hits and some videos showing you exactly how to do it.

JMeter’s Component Reference is proof of JMeter’s extensive and thorough documentation. Every element, function, and property is discussed in more detail than most people could ever want, and that’s just the “official” documentation that’s on the Apache site. There are thousands of JMeter books, tutorials, and courses by its passionate users.

The JMeter GitHub repository has 17,290 commits as of December 2020 - evidence of a community of developers that have committed some time and experience to JMeter over the last two decades.

In comparison, the k6 community is growing, but still small. There is an official and central community forum that allows users to share their experiences with k6, which is something missing from the JMeter community. The k6 repository does have more stars, but significantly fewer commits at 3,795. Much of the disparity, of course, is because k6 is very new - the open-source tool k6 was only released in 2017.

When you need a cost-effective way to do distributed load testing

One of JMeter’s absolute best features is that it gives you a framework for running distributed load tests with it. This is pretty special for a free and open-source tool.

Distributed testing means ramping up the amount of load you’re generating with your load tests, usually by increasing the number of virtual users and running multiple instances of your script on other load generators. JMeter accomplishes this by designating a controller node and letting you set up worker nodes. Each worker node needs to have a copy of jmeter-server, a utility included with every JMeter installation (in jmeter/bin).

Source: Apache JMeter

These worker nodes then run copies of your test plan and exert load against your application server. You can use on-premise or cloud machines as your worker nodes. A distributed execution like this takes a bit to set up, but it does work surprisingly well.

k6 doesn’t have a native distributed load testing mode. You can containerize it yourself and deploy it via Kubernetes, for instance, but it does require a more involved setup. The team is also working on the k6 Operator, which makes setup significantly easier, but it is experimental and comes with drawbacks such as not supporting threshold synchronization between instances. For now, the easiest way to scale up on k6 is by using the paid service k6 Cloud, which reduces setup time and complexity significantly.

It’s important to note that JMeter can be quite resource-hungry, and thus is more likely to require multiple load generators to generate load than k6. Depending on the resources of the load generator, JMeter can run about a thousand virtual users on average, and scaling up your test beyond that point will require a distributed execution setup. A single instance of k6, on the other hand, can run tens of thousands of virtual users given the same resources. More on that later.

When you want canned reports built into your load testing tool

JMeter’s many listeners allow you to decide exactly what format you want to view results in. Debugging? Use the View Results Tree. Wanting metrics for your load test? Use the Summary or Aggregate Report.

In addition, JMeter can generate HTML reports based on your load tests. They’re currently quite limited, but functional, and they have several default graphs to help you analyze your load test. They are not interactive, but they are pretty easy to generate and a good starting point for putting together a report template for your tests.

k6 does not have built-in reporting. Its modular nature lends itself better to integrating with many other data analytics tools, including Grafana, Datadog, New Relic, Amazon CloudWatch, and k6 Cloud– all of which are better analysis tools and can provide more insights than what JMeter’s reports can provide, but they do require you to set up another tool to analyze your tests. If you’re running low on time and can make do with its reports, JMeter does really well at providing that extra value out of the box.

When does k6 excel?

When you want to get started quickly

Yup, I put this one in both the JMeter and k6 sections. JMeter and k6 can each be easy to get started with, but for very different reasons.

Installation

k6 installs extremely quickly - on my first installation of it on macOS, it took 7 seconds via brew install k6. It doesn’t require nodeJS or any other dependencies, either.

JMeter itself is fairly easy to install, but before you install JMeter, you need to install Java. Especially if you’re using Windows, it’s a bit of a rite of passage for a new JMeter user to run into issues adding the environment variables needed for Java to function.

It can also be confusing to determine which version of Java is the right one. There’s JDK, JRE, and recently Oracle has added to the confusion by announcing that some versions of Java (but not all) are going to require a license.

Source: Oracle Java

While JMeter is still free via OpenJDK, I’ve spoken to a few testers who mistakenly thought they’d have to pay for JMeter. It’s a little disconcerting to see Oracle flexing its muscles here, especially if you’ve built a whole performance testing suite on Java through JMeter.

Plugins

k6 supports plugins, but they’re not necessary. For most use cases, k6’s built-in functionality should be more than sufficient. This is in sharp contrast to JMeter, where everything is a plugin … even the Plugin Manager!

Most JMeter users would argue that there is a standard set of plugins that you should download before you even begin to use it. A new JMeter user might not necessarily know where to find these and may have a less-than-satisfying experience out of the box.

Here are a few things you won’t have if you stick to a vanilla, no-plugin JMeter installation:

Thread groups: No custom load profile, stages, rendezvous
Controllers: No parallel controller to execute requests in parallel
Samplers: No HTTP/2 support
Test Data: No randomization of lines within a CSV file (sequential only)
Timers: No throughput shaping timer for setting throughput in stages

These functions are all available within k6 from the get-go.

Scripting

k6 scripts are written in javascript, making them simple to write and run. All you need is one installer (on Microsoft Windows) or a single command on your terminal and any text editor. For JMeter, you’ll need to install a few things locally before you can get started: Java, JMeter itself, and the standard set of plugins you’ll need. This can be confusing for new users. Who hasn’t struggled with environment variables at some point when learning to use JMeter, especially on Windows?

It also takes time to learn any new UI, whereas javascript is near-ubiquitous and it’s a good language to learn when you’re load testing web sites anyway.

When you want to maximize performance and efficiency

Test your test tools!

Why does all this matter?

When we’re talking about simulating many virtual users, slow performance = high cost. There is a cost to the number of load generators you use, whether it’s the cost to acquire and maintain them (on-prem) or the cost of a cloud service.

It makes sense to consider the performance limitations of your load testing tool before you use it to improve your own application’s performance.

k6 builds on Go’s stellar performance

k6 is written in Go, and Go is built with performance in mind. “Go is a compiled language and does not need to be interpreted, unlike Java or Python. There is no added layer of complexity.” It also has no external dependencies, which is a good thing. Fewer moving parts for a performance tester means fewer potential sources of a bottleneck.

The simplest and most common way for a load testing tool to run a virtual user is to assign one virtual user to one kernel or OS thread. However, the 1 Thread: 1 Virtual User paradigm has a serious flaw: when a virtual user is waiting for a response or executing a programmed sleep(), the thread is also blocked, and cannot process other work.

In k6, each virtual user is run on a goroutine, not a thread. What difference does that make? Well, goroutines can be controlled by something called the Go Scheduler, which acts like a traffic cop. It reuses idle threads and intelligently assigns work, by allowing “work stealing” and work hand-offs between threads. Does this sound familiar? This is the same principle that load balancers are built on: An external monitor that oversees the flow of work improves general performance. Go itself is intrinsically load-balanced in a way that many programming languages aren’t, which makes it the perfect foundation for a load testing tool.

Being able to leverage Go’s innate performance optimizations also translates into significantly less memory utilization. One thread running k6 doesn’t go above 100 kb, whereas a JVM thread like JMeter uses, for instance, uses the default of 1MB. That’s 1000% more than k6! Of course, Java allows a user to tweak an app’s memory utilization, so the difference is usually not quite as stark, but it’s still interesting to note that Go has a much lower starting point.

… creating [a] new thread in Java is not memory efficient. As every thread consumes approx 1MB of the memory heap size and eventually if you start spinning thousands of threads, they will put tremendous pressure on the heap and will cause shut down due to out of memory. Also, if you want to communicate between two or more threads, it’s very difficult. - Keval Patel

Performance benefits in practice

Fewer load generators needed

k6’s comparatively better performance means it needs fewer load generators to execute a given amount of load. Rafaela Azevedo made a comparison of the memory used by k6 and JMeter, and here are her results:

JMeter took up 760 MB of memory.

Source: Rafaela Azevedo

Meanwhile, k6 took up 256 MB of memory.

Source: Rafaela Azevedo

Rafaela’s findings are further affirmed by our own benchmarks.

Source: Ragnar Lönn, in our Load Testing Tool Comparison

k6’s lower memory footprint is the reason it can run more virtual users and generate more load than average. Whether you’re using load generators that are on-premise or in a cloud, you’ll pay less for provisioning costs when using k6. This cost saving makes k6 a great tool for budget-conscious teams.

The common misconception of many load testers is that distributed execution (ability to launch a load test on multiple machines) is required to generate large load. This is not the case with k6. - Rafaela Azevedo

k6 is different from many other load testing tools in the way it handles hardware resources. A single k6 process will efficiently use all CPU cores on a load generator machine. A single instance of k6 is often enough to generate load of 30.000-40.000 simultaneous users (VUs). This amount of VUs can generate upwards of 300,000 requests per second (RPS).

Unless you need more than 100,000-300,000 requests per second (6-12M requests per minute), a single instance of k6 will likely be sufficient for your needs. - k6 documentation

No out of memory errors

k6 is also good for when you’re short on time. Using JMeter means getting familiar with how to performance tune Java and how to fix the most common Java performance problems, because there are several. The one that I’ve personally run into the most while using JMeter is:

ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError:
Java heap space.

You’re not really a JMeter load tester until you’ve seen this error. :)

The cause for this error is usually that there was an insufficient amount of Heap memory allocated to JMeter. Yup– in addition to monitoring your load generator’s memory usage, you’ll also need to monitor the JVM’s memory usage. You can set the amount of heap memory allocated to JMeter by modifying the JMeter binary itself and changing this line:

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
: "${HEAP:="-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m"}"

Even JMeter official documentation suggests that you increase the Java Heap size beyond 1GB before you even run your first load test.

Increase the Java Heap size. By default JMeter runs with a heap of 1 GB, this might not be enough for your test and depends on your test plan and number of threads you want to run - Apache JMeter

No GUI means no resource overhead during load tests

GUIs typically add significantly more overhead to an application, which is one of the reasons why k6 doesn’t have one.

When you’re running a load test, the best practice is to run it from the command line anyway– because otherwise, it interferes with your results. Code-based load testing tools are performant from the beginning.

JMeter is peppered with warnings about the GUI’s effect on performance. Here’s one from the “JMeter documentation:

Don’t run load test using GUI mode ! - Apache JMeter

And here’s a message that occurs upon startup of JMeter:

Not having a GUI means there’s one less gotcha. Any load testing scripts you create in k6 are ready for prime-time execution whenever you’re ready.

When you want to do goal-oriented testing

The most important question a load tester can ask when starting a new load testing project is “Why”? Test plans should be made to directly address a team’s reasons for wanting to do load testing, and that should be reflected in the non-functional requirements. Non-functional requirements, in turn, should then give some guidance as to the bounds of acceptable performance. A common example for these bounds, or thresholds, is an average response time of less than 3 seconds for all transactions during the test. k6 does this natively with global thresholds worked into the script, and you can also create your own metric to use in a threshold:

import http from 'k6/http';
import { Rate } from 'k6/metrics';

const myFailRate = new Rate('failed requests');

export let options = {
 thresholds: {
 'failed requests': ['rate<0.1'], // threshold on a custom metric
 http_req_duration: ['p(95)<500'], // threshold on a standard metric
 },
};

export default function () {
 let res = http.get('https://test-api.k6.io/public/crocodiles/1/');
 myFailRate.add(res.status !== 200);
}

Source: k6 documentation

JMeter doesn’t natively support thresholds at the test-level. There are a few ways that I’ve been able to work around this:

Duration Assertion: You can add a Duration Assertion to any sampler in JMeter, which allows you set an acceptable response time for that request in milliseconds, beyond which the request (and the transaction that encompasses it) would be marked as failed. However, this is still just at the individual request level, and even putting the duration assertion element at the Test Plan level only applies the same threshold to each child request. It doesn’t constitute pass/fail criteria for the test as a whole.
Timers: For criteria related to throughput, you can use the Constant Throughput Timer or the Throughput Shaping Timer (plugin). This approach controls how many requests per second are spent. In my experience, it’s worth testing how these timers behave when using nested transaction controllers - it may take some rearranging to get right.
Performance Plugin: When integrating JMeter with Jenkins, you can use the Performance Plugin to set error thresholds and mark test runs (“builds”) as unstable or failed within Jenkins.
Custom Code: You can use the JSR223 Sampler to write a bit of Groovy or BeanShell code that collects and parses response metrics.

The problem is that even though these options make setting thresholds possible in JMeter, they’re still workarounds that don’t come with JMeter out of the box or don’t adequately address the creation of different types of thresholds (error, response time, throughput, CPU, memory). What I end up doing, more often than not, is exporting the raw data from JMeter and doing the analysis myself using another tool.

When you’re part of a team of scripters

k6 shines in situations where more than one person could be doing the scripting.

It strikes a middle ground that developers and testers can compromise on

Software quality should not be the sole responsibility of testers; quality should be baked into every activity when building software. One of the challenges in implementing this concept, however, is that there has traditionally been a segmentation in tools and languages used by team members with different functions. Developers use one tool; testers use another. It’s difficult to encourage developers to test and testers to code in this environment, and many testing tools only further encourage this divide. A developer can’t use a tool like JMeter for anything but testing, so the incentive to invest the effort in learning it is lower.

k6 bridges the gap by bringing testing concepts and features to an environment and language that developers are already spending their time in. Javascript is useful for more than just writing load testing scripts, and k6 works well with any IDE or text editor. A frictionless developer experience is central to the k6 philosophy.

It’s easier to collaborate on scripts

Once you’ve convinced both devs and testers to contribute to load testing scripts, how do you keep track of changes? How do you manage scripts that multiple people are working on at the same time? With tests as code, the easiest way is to just use versioning tools like Git, the way you probably are already using for your application code. Again, no new framework or processes to learn here. k6’s conciseness and portability really proves to be an advantage in collaboration situations. To illustrate this, let’s consider a simple test script that contains a few things:

A GET request to our test site, https://test.k6.io
A transaction that calls this request 01_Home
A check to fail non-HTTP 200 responses or responses that don’t have the text Collection of simple web-pages suitable for load testing
Variable think time
We’ll just run that as a single user, for a single iteration. Here’s what that script looks like in k6:

import http from 'k6/http';
import { sleep, check } from 'k6';

export default function () {
 let res = http.get('https://test.k6.io', {tags: { name: '01_Home' }});
 check(res, {
 'is status 200': (r) => r.status === 200,
 'text verification': (r) => r.body.includes("Collection of simple web-pages suitable for load testing")
 });
 sleep(Math.random() * 5);
}

You can track JMeter scripts on Git, too. Here’s what the same script looks like in JMeter:

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.3">
 <hashTree>
 <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
 <stringProp name="TestPlan.comments"></stringProp>
 <boolProp name="TestPlan.functional_mode">false</boolProp>
 <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
 <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
 <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
 <collectionProp name="Arguments.arguments"/>
 </elementProp>
 <stringProp name="TestPlan.user_define_classpath"></stringProp>
 </TestPlan>
 <hashTree>
 <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
 <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
 <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
 <boolProp name="LoopController.continue_forever">false</boolProp>
 <stringProp name="LoopController.loops">1</stringProp>
 </elementProp>
 <stringProp name="ThreadGroup.num_threads">1</stringProp>
 <stringProp name="ThreadGroup.ramp_time">1</stringProp>
 <boolProp name="ThreadGroup.scheduler">false</boolProp>
 <stringProp name="ThreadGroup.duration"></stringProp>
 <stringProp name="ThreadGroup.delay"></stringProp>
 <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
 </ThreadGroup>
 <hashTree>
 <TransactionController guiclass="TransactionControllerGui" testclass="TransactionController" testname="01_Home" enabled="true">
 <boolProp name="TransactionController.includeTimers">false</boolProp>
 <boolProp name="TransactionController.parent">true</boolProp>
 </TransactionController>
 <hashTree>
 <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
 <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
 <collectionProp name="Arguments.arguments"/>
 </elementProp>
 <stringProp name="HTTPSampler.domain">test.k6.io</stringProp>
 <stringProp name="HTTPSampler.port"></stringProp>
 <stringProp name="HTTPSampler.protocol">https</stringProp>
 <stringProp name="HTTPSampler.contentEncoding"></stringProp>
 <stringProp name="HTTPSampler.path"></stringProp>
 <stringProp name="HTTPSampler.method">GET</stringProp>
 <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
 <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
 <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
 <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
 <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
 <stringProp name="HTTPSampler.connect_timeout"></stringProp>
 <stringProp name="HTTPSampler.response_timeout"></stringProp>
 </HTTPSamplerProxy>
 <hashTree>
 <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
 <collectionProp name="Asserion.test_strings">
 <stringProp name="-745788246">Collection of simple web-pages suitable for load testing</stringProp>
 </collectionProp>
 <stringProp name="Assertion.custom_message"></stringProp>
 <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
 <boolProp name="Assertion.assume_success">false</boolProp>
 <intProp name="Assertion.test_type">16</intProp>
 </ResponseAssertion>
 <hashTree/>
 </hashTree>
 </hashTree>
 <UniformRandomTimer guiclass="UniformRandomTimerGui" testclass="UniformRandomTimer" testname="Uniform Random Timer" enabled="true">
 <stringProp name="ConstantTimer.delay">1000</stringProp>
 <stringProp name="RandomTimer.range">4000</stringProp>
 </UniformRandomTimer>
 <hashTree/>
 </hashTree>
 </hashTree>
 </hashTree>
</jmeterTestPlan>

Which one is easier to understand? If you had to modify the script to add a request, which format would you choose to do it in?

Scripting in XML makes merge conflicts more likely and breaking changes in automation pipelines way more difficult to pinpoint.

I’ve collaborated with others on JMeter scripts in the past, and we always opened up scripts in the GUI. It’s way easier to spot changes there - but that also created some new issues.

When you want something easy to maintain

Versioning for load testing

When ramping up load tests to more than one load generator, keeping track of the versions of necessary components ensures that one load generator’s traffic is the same as another’s. The value of reducing variables in scientific experiments holds true for load testing.

With k6, you just need to worry about the version of k6 and the script. No other dependencies are required. If you’re using plugins, you’d need to track those too, but as mentioned previously, plugins in k6 aren’t as necessary as they are in some other tools – and k6 is available as a Docker image to reduce those issues.

With something like JMeter, there are more levels to consider. There’s the version of JMeter itself, Java, versions of plugins, and then the script. Every new layer introduces more changes for incompatibilities. In previous projets I’ve worked on, someone would inevitably try out a new plugin and forget to discuss it with the team, and anyone who tried to open that script in their copy of JMeter would see this error:

The error means there’s a plugin that the script uses that JMeter doesn’t recognize, but depending on the plugin, it’s not always this easy to pinpoint which one (it was the Ultimate Thread Group in this case). You don’t want that happening in the middle of your load test because you forgot to add a plugin to some of your load generators!

It’s easier to link test code with application code

If your application code is in Javascript, k6 is a shoo-in as a load testing tool. You could build tests that are tightly integrated with application code by importing objects and having your tests interacting with them directly. This approach would reduce a lot of time spent in refactoring test scripts when objects are changed.

If you’re already using VS Code, k6 even has a VS Code plugin.

Test as code gives you control

Having your scripts be pure code reduces ambiguities in that you’re not limited by poor UI design or bugs in third-party plugins. k6 scripts are Typescript-typed, so most IDEs support autocompletion while you’re scripting - the code equivalent of helpful tooltips, except way more universal. k6 scripts give you the freedom to import and use JS libraries, so you can build on code others have already written. While k6 is not NodeJS, there are even some NodeJS packages that you can bundle for use with your scripts.

No external dependencies

k6 doesn’t require NodeJS or anything else to run, which also reduces the likelihood of bugs that might affect your scripts. For example, since JMeter runs on Java, it is also dependent on Java. When Java receives a new version, that often requires a JMeter update as well. JMeter versions understandably lag a bit behind Java versions as developers refactor code, but this gap can lead to potential security issues in a previous version of Java that may remain unpatched in the meantime.

With k6, development can proceed independently of any third-party component, so major issues can be addressed more easily.

Comparison table: JMeter vs k6

Feature	JMeter	k6
Written in	Java	Go
Scripting language	Limited: some Java (Groovy, Beanshell, etc)	Javascript
Protocols	Supports most protocols via plugins (native support for HTTP/1.1, SOAP, FTP, JDBC, LDAP, MOM via JMS, SMTP, POP3, IMAP, shell scripts, TCP, Java objects)	Supports fewer, modern protocols natively (HTTP/1.1, HTTP/2, WebSockets, gRPC)
External dependencies	Java	none
Resource utilization	Poor; One load generator can simulate a few thousand virtual users	Very good; One load generator can simulate tens of thousands of virtual users
Memory management	JVM heap memory must be set	Uses load generator memory natively
Threading model	1 Thread: 1 Virtual User; Slower performance, higher resource cost	1 Goroutine: 1 Virtual User; Faster performance, lower resource cost
Ease of scripting	GUI-driven, with code blocks	code-driven; VSCode plugin
Test-level thresholds	No, individual request only	Yes
Script format	XML	javascript
Collaboration	Difficult to work on simultaneously; tester-friendly; need the GUI app to edit	Developer-friendly, easy to version; Javascript format promotes collaboration
Maintenance	Verbose scripts; XML format is difficult to read	More concise scripts; JavaScript is easy to read
Community	Since 1998; Many third-party tutorials; Extensive documentation; No central community	Since 2017; Extensive documentation; Fewer third-party tutorials; Official community
Plugin support	Requires plugins for many features, but there are lots of plugins available	Most features are natively supported, but plugin support is new and availability is sparse
Native distributed load generation	Yes	No (Premium only)
Pre-generated reports	Default and custom HTML reports; logging via listeners	No built-in pre-generated reports; integration with analytics tools with third-party dashboards
Website	jmeter.apache.org	k6.io
Source Code	Link	Link

Conclusion

The best advice, when attempting to choose a load testing tool, is to do a proof of concept with the most promising candidates. Some features or bugs may prove more or less important when used in an actual testing cycle. However, don’t put too much weight on comparisons of the load test results from two different tools - each tool records metrics differently, and it’s more meaningful to compare results against previous runs with the same tool. When switching tools, reestablish a baseline in the new tool each time.

Here’s a summary that might help you decide between JMeter and k6.

Both tools do well at:

Generating protocol-level load on an application server by scripting complex user flows
Realistic scripting with the use of dynamic think time, test data generation and reuse, and customizable workload models
Documentation of features and consistency of releases

Neither tool supports:

Generating browser-level load and interacting with DOM elements, especially for SPA
Detailed results analysis (JMeter has pre-generated HTML reports and listeners, but they are far from ideal). Users should expect to integrate results with databases and data visualization software.

JMeter is best for:

traditional testing teams
those looking for a GUI-driven testing tool with tons of third-party tutorials and extensive protocol support
previous users of commercial tools like LoadRunner and NeoLoad

k6 is best for:

collaborative, cross-functional engineering teams where testing spans acrosss multiple roles
those looking for a simple and lightweight, yet fully-featured load testing tool
teams looking to integrate testing into existing development workflows and CI/CD pipelines

Load testing scripting tools are by no means the most important consideration for the success of load testing. Knowing why you’re testing, what the requirements are, and understanding and communicating results are all arguably more important. The right tool will enable you to address those concerns while providing as little friction as possible. There’s no unequivocal “best” tool; there’s only the right tool for your project and context.

Python for Everybody Specialization

Tue, 26 Jan 2021 23:01:18 +0100

I’m starting off the new year by learning something new. I am a big fan of non-traditional education, and Coursera is a platform I’ve used a few times before. You may not get courses that qualify as university credit, but I’ve found it to be a useful resource for learning. I might be biased (my university degree is in Economics, which I’ve never used), but in some ways, I actually prefer a MOOC (that’s a Massively Open Online Course) to a typical formal university course for a few reasons:

MOOCs are entirely online, which means it’s more accessible - I don’t need to physically be in the US to take a course at Harvard
MOOCs are usually self-paced - perfect for someone working full-time like me
MOOCs are free or cheap - no huge student loans here; MOOCs are democratizing education and making it available to more people
Good online education tends to be more practical. There is none of the intellectual posturing that I’ve seen in physical classrooms; there are no random classes you need to take to get a fancy piece of paper. I love being able to choose a topic and learn ONLY that– and get right down to business in the first lesson.
Formal education is often centered around getting students jobs, not helping workers do their jobs well. I already have a job, and I want to learn real, usable skills, not “marketable” ones.

My main complaint with MOOCs was that they tend to be short (in comparison to a formal course), but Coursera has addressed that somewhat with their specializations. A specialization is a series of courses, but it’s more than just disparate courses that are strung together; it’s a thoughtfully designed curriculum that builds up on knowledge from previous courses. Last year, I began learning Python.

Why Python? Others might have different reasons, but here were mine:

Python is one of the best languages to learn for data analysis (the other being R). I’ve always been interested in data visualization, and as a load tester, I’m frequently exposed to large quantities of data that I somehow need to make sense of.
Python is pretty good at retrieving data from websites, and I wanted to explore how good it is to use in load testing. I’m always eager to learn different ways to test, and I’ve wanted to try Locust for a while.
Python is increasing in popularity, and I wanted to see why.

So I decided to take the Python for Everybody Specialization from the University of Michigan (via Coursera) to get an introduction to this cool language. The specialization consists of five modules, each of which I had to take separately:

Programming for Everybody (Getting Started with Python)
Python Data Structure
Using Python to Access Web Data
Using Databases with Python
Capstone: Retrieving, Processing, and Visualizing Data with Python

The specialization was really fun, and the teacher, Charles Severance, is fantastic. I’d taken courses from him before, so I knew what to expect. He didn’t disappoint. He has a way of explaining daunting technical concepts in a way that’s easy to digest for beginners.

You can check out my certificate here. I’m pretty happy with my progress. I’m by no means a Python expert now, but I’m definitely eager to keep learning how to do more things with this new knowledge.

Week of Testing Wrap-up: My thoughts on k6 as a load testing tool

Thu, 17 Dec 2020 15:29:57 +0100

I decided to make a final wrap-up video for my “Week of Testing” with k6… ok, so it’s actually three weeks later now, but I wanted to explore k6 a little bit more before I gave my opinion on it.

And here’s the audio version:

Thank you, k6, for being awesome enough to agree to this video series before knowing what I’d say. Gotta love a company that welcomes transparency and feedback.

Taking Notes on a Video: my YiNote to Roam Research Workflow

Tue, 15 Dec 2020 22:00:34 +0100

Listen to the audio version of this blog post here:

I was working on an article yesterday comparing the performance of k6 with JMeter, and I had a chat to Mihail, one of the devs of k6. He mentioned that a large part of why k6 is so performant compared to other tools (particularly Java-based ones) is that it’s written in Go. He talked a bit about Goroutines, and when I told him I was a Go noob, he recommended this very excellent video by Kavya Joshi, which explains a bit more about the Go Scheduler and how it manages to achieve performance for Go apps that exceeds the typical 1 Virtual User: 1 kernel thread paradigm that many load testing tools (like JMeter) use.

Anyway, I got around to watching the video today, and because it was a video I knew I’d learn from, I took notes - and ended up improving part of the workflow (more on that later). Other people have commented on my notes before, so I thought I’d share how I process videos into my personal wiki, currently held in Roam Research.

Install YiNote Chrome extension

I use a free app called YiNote to take video notes in particular. I use their Chrome extension.

Take Notes

I was pretty dubious about using an app just to take notes for video specifically, but there were a few things that sold me:

When you click on the text field to take notes in the extension, YiNote automatically pauses the video so that you can type without missing what’s being said.
YiNote also automatically saves the timestamps associated with notes you create, so that you can click on it afterwards to jump to the specific section that you wrote a note on.
YiNote has an export to Markdown option - great compatibility with Roam or Obsidian
You can also export your notes to a PDF, which is great for sharing. It saves screenshots of the presentation and puts your notes next to the relevant screenshot.

Here’s what it looks like while taking notes:

Export YiNote notes

After watching the video, I click the Settings wheel from the Chrome extension panel.

I click the Export as Markdown button.

That lets me save a .md file to my local drive. I always save YiNote files in the same folder as the script below.

Run reformatter script

This is the part that I tweaked today. YiNote’s exported Markdown file is formatted in a way that doesn’t really play well with Roam. Here’s what it looks like, fresh from YiNote:

<!-- Generated by <a href="https://www.yinote.co/#installation">YiNote</a> -->

<!-- # [GopherCon 2018: Kavya Joshi - The Scheduler Saga](https://www.youtube.com/watch?v=YHRO5WQGh0k) -->

<!-- ## [1:19](https://www.youtube.com/watch?v=YHRO5WQGh0k&t=79) -->

The scheduler is what makes sure the Goroutines run concurrently.

<!-- ## [1:29](https://www.youtube.com/watch?v=YHRO5WQGh0k&t=89) -->

All the information is there, but importing that to Roam, with the comments, isn’t really useful. I want to see those timestamps as clickable links, and I want my notes to be nested underneath the corresponding timestamp so I know what belongs where.

So I dusted off my Python skills tonight and wrote up this little script to do just that. Warning: I am not a developer; I just occasionally code. So I’m sure that this isn’t the most concise way to do it (contact me and let me know how it could have been better if you have any ideas). But it’s functional. And, in the spirit of learning in public, here it is:

The script results in an output file that is better formatted for Roam.

Import resulting file to Roam

On Roam, I hit Options > Import Files, select the output file, hit OK, and that creates a new page in a way better format than the one directly from YiNote. After adding the date, author, and “Recommended By” fields, this is what it looks like in Roam:

That’s it!

I don’t do this for every video I watch, but I find this process useful for more technical videos that I really want to learn from. It’s really important to me to get all my notes from disparate sources into Roam, so that they then get processed like any other note into my Zettelkasten - but that’s a process for another day.

How I use Roam Research for D&D: Session Notes

Mon, 14 Dec 2020 20:13:13 +0100

In case you thought this site was all about performance and load testing, surprise! It’s not.

If you’ve talked to me personally for longer than ten minutes in the last year, there’s a pretty good chance I’ve mentioned Roam Research, or the note-taking and learning methodology called Zettelkasten. Sorrynotsorry. I’ve been obsessed since I discovered it.

To say Roam Research is a note-taking device is like saying Bitcoin is currency: yes, it IS that, but also so much more. To me, it’s my second brain. It’s a record of what I know and what I want to know, and sometimes even things I don’t know yet. The real reason that Roam and the Zettelkasten idea resonated with me so much is that I can use it a lot for my work. But I’ve discovered another use case that Roam is perfect for: Dungeons & Dragons.

This became longer than I thought it would be, so I’ll just talk about how I handle Session Notes in particular.

Starting a new session

Everything starts with sitting down to play D&D. I write a log of events and information that comes out while I’m playing. Here’s what a new session log looks like:

I preface pages in this campaign with D&D/ to make it a separate namespace. A namespace is like a folder, and I use it mainly because I have two D&D campaigns, both of which would have the page Campaign, Characters, Locations, etc, even though the two campaigns are on different worlds. To keep them separate, and to keep from getting mixed up (Was Gloomtoes an NPC in this campaign or that one?) I use different namespaces for each campaign.

I add the real-world date in the page name, and then an in-game date reference. The in-game date reference is useful for keeping track of time since I might go a couple weeks (okay, maybe one week; let’s not get too crazy) without playing this campaign, but no time in the in-game world has elapsed. Then, I end the page name with a descriptive title, which I always end up changing after the session. Sometimes I’ll title a page Exploring Castle Never, but then we end up getting carried away in another part of the city instead. That’s okay - when I change a page name, Roam then updates all references to that page that I’ve made from other pages.

I don’t type all this out manually - I use a TextExpander template so that to start a new session, all I have to do is create a page in Roam, type ;s.dnd and that is automatically replaced with prompts for a few fields.

Date: This is the real-world date of when I played this particular session.
Tags: When I first create session notes, this is prepopulated with #TVZ, which stands for To Verzettelen. It’s just my personal way of marking a note as one that needs to be processed.
Location: Here I note down the in-game places that my character visited this session. Usually I’ll just put city names, but I sometimes also split locations up into districts within a city or rooms within a dungeon.
Characters: I’ve found it helpful to keep track of which characters (players and NPCs) I’ve come across in this session.
Session Summary: This is a section I leave blank when the session is initially created.

Filling in the session log

The rest of the page, after Session Summary, is where I’ll spend most of the time writing. I just write whatever comes to mind and whatever I have time to write, in bullet form. I don’t worry about formatting or grammar or linking to other pages. I just type all my thoughts out in plain text.

HP

I like to play healers, which means I care a lot about how everyone in the party is doing so that I can figure out who best needs healing. {{[[table]]}} in Roam lets me create a custom table, and I have a ;hp snippet in TextExpander that recreates this table, with all the names of the people currently in my party, so that I just worry about filling in the values. Sometimes I still get it wrong, or miss a bit of damage that the DM calls out, but it’s enough to give me an idea of who needs heals.

I also write down initiative orders, the defenses/vulnerabilities/immunities of monsters we face, descriptions of NPCs we meet, suspicions or questions I might have… basically anything and everything that comes to mind. Session time is pretty much a brain dump, with no formatting.

Equipment

If I acquire a new item during a session, I’ll tag that line with #[[D&D/Carrying]]. This lets me stay on the Session Notes page while still maintaining an up-to-date record of what I’m carrying.

A tag in Roam is the same as a page, so when I click on that link, it takes me to the D&D/Carrying page, which conveniently also shows me all the references to that page from other pages.

Roam shows me not just the page, but the exact block where I made the reference to D&D/Carrying, so it’s easy to see at a glance what I have and where I got it.

Next Actions and Waiting For

I use David Allen’s GTD methodology in D&D too. When I want to remember to do something later, I tag it with #[[D&D/Agenda]] and #[[Next Actions]] or #[[Waiting For]]. This presents me with the same opportunities as D&D/Carrying, where I can see references to those pages later as well.

Processing the session

Sometime after the session, I’ll go and fill in the metadata from my template, including the session summary, which is just a paragraph or two describing what happened in the session at a high level. This is mainly to jog my memory in the next session, when I’ve already forgotten what we did in the previous one.

Here’s what it looks like, all filled in:

Adding links and fleshing out pages

I then go back through the notes themselves and add turn any people, places, or interesting objects that I mention into links. In Roam, that’s a matter of highlighting them and hitting [[. Then, I click on the page I’ve just created and try to add information that I’ve learned. This isn’t strictly necessary, since all mentions of that page will show up as a linked reference, but it’s an opportunity to consolidate my information on that subject and see how it fits into what else I’ve written there.

Here’s a page for an NPC:

Having these links means that when an NPC says “I’m from Leilon”, I can type Leilon into Roam:

… and say, “Oh, Leilon. I heard you had a dragon problem there recently. Did your family get out okay?” Boom! Instant advantage on a Diplomacy check.

Sometimes we’re in a dungeon with many rooms, and this is the point where I draw a quick map on my iPad Pro and Apple Pencil using Notability, and then I upload it right from my iPad to Roam.

This map is in my page for Castle Never:

The little blue “8” on the upper left of the map is versions. Information rarely comes all at once in D&D; it usually comes in trickles. As I learned more about the castle, I revised the map and added a new version to it, so that I can still see previous versions and what I thought the map was like when we first explored it.

At this stage, I also add any tags I forgot to add during the session.

Reviewing quests and questions

After I go through the session notes, I go to the page called D&D/Quests and Questions, where I keep a running list of all the open quest hooks and any unanswered questions I might have.

Not everything on this page is meant to be resolved, but I find it comforting to track everything anyway, in case I happen to find myself in a city with nothing in particular to do. I mark these with a checkbox ({{[[TODO]]}}) in Roam) and tick the box when one is resolved. Roam lets me filter out the resolved quests so I only see things that are outstanding.

This is where I get the most value out of taking notes. It’s at this point where I’ll add our progress towards quests, make connections between two seemingly disconnected quests, and situate what I’ve just learned in the session within the broader context of the world and the campaign. I also review and update items on the Next Actions and Waiting For.

This is the part that takes the longest, but it’s also what most prepares me for when I get to the table again next week.

Tying it all together

I use a D&D/Campaign page as my landing page for this campaign. It includes D&D/Quests and Questions as well as a list of all sessions to date, so that I always know what happened and when.

Why do I take notes for a game?

You might be wondering why I go to so much trouble. Why do I take complicated notes about something I do “just” for fun?

Taking notes helps me offload this all safely from my brain.
I take my fun as seriously as I take my work.

Week of Testing: Day 5 - The thing about learning in public

Sat, 28 Nov 2020 20:03:04 +0100

Yesterday I mentioned that I’d used the constant-vus executor to recreate the stepped load profile. Well, the thing about learning publicly is that it also means making mistakes publicly!

I was kindly informed by a few of my new colleagues that constant-vus was not the most optimal way to recreate the stepped load profile. Simme wrote up a little function so I wouldn’t have to repeat myself so much, and Pepe Cano and Pawel Suwala (the CTO of k6 himself) both told me nicely that using stages would have been the better solution.

So, let’s chalk that up to a TIL (Today I Learned) and record that snippet here for prosperity (found in the k6 documentation linked above):

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
 stages: [
 { duration: '30s', target: 20 },
 { duration: '1m30s', target: 10 },
 { duration: '20s', target: 0 },
 ],
};

export default function () {
 let res = http.get('https://httpbin.org/');
 check(res, { 'status was 200': (r) => r.status == 200 });
 sleep(1);
}

Week of Testing: Day 4 - Load profiles on k6

Fri, 27 Nov 2020 19:54:06 +0100

k6 has a cool feature where even if you start a load test locally via the CLI, you can still stream the results to k6 Cloud. It’s an interesting compromise between debugging a test locally (and therefore not paying for cloud infrastructure) and sharing results for load tests run on the cloud. It worked seamlessly, too.

I also wanted to see how to recreate a specific load profile that I commonly use - the stepped load profile. In JMeter, I typically do this with a plugin - the jp@gc Ultimate Thread Group is my thread group of choice. So, I created it in JMeter (I love that it shows a little image of it as well, while you’re building it) and then attempted to recreate it in k6.

I definitely had to look at the k6 documentation for this. I used the constant-vus executor for this, but k6 has several other executors to choose from, and it worked as well as I had hoped.

So far, my experience with k6 has been positive. Things are well-documented, and they just work out of the box. No plugins necessary.

Week of Testing: Day 3 - Running a load test locally using k6

Thu, 26 Nov 2020 19:31:33 +0100

I’ve had my fun playing around with k6 Cloud, the SaaS component. For Day 3, I thought it was time to jump into the heart of k6: k6 the open source load testing tool.

First impressions: logging was a little sparser than I expect, coming from having used load testing tools that just do this for me automatically. I think one of the first things I’ll do with k6 is to create my own little “load testing starting script” that I’ll always use, with all of the appropriate logging already set up.

Otherwise, I was really impressed with how quickly it installed. If you scrub over to the timestamp 1:46 in the video, you’ll hear that I was actually expecting to have to speed up the installation process while editing to save time, but the installation actually finished before I could say that I was going to do so. By my count that was 7 seconds between hitting enter in my terminal and the download finishing.

7 seconds.

I can’t help but compare that to JMeter. JMeter is honestly probably my go-to load testing right now– it’s not good for absolutely everything, but it’s the tool that I reach for when I “just want to test this one thing”. I’ve used it for years, and it’s the tool that I know the most. But one of the issues with JMeter is that there’s so much to do before it’s actually usable.

Before you even open JMeter, you need to install Java, and when I first started in load testing, I spent a long time wrestling with getting setting the Java environment variable in Windows before my computer would even recognize that Java was installed. Then you need to install JMeter– but any JMeter tester will tell you that there are some plugins that you just can’t do without, and that really you should always install. That’s why there are so many “best plugins in JMeter” lists (here’s my top 3)… because most of them are necessary, but you wouldn’t know that if you’re new to JMeter.

The fact that k6 starts up in 7 seconds is… shockingly good.

Week of Testing: Day 2 - Comparing tests against a baseline

Wed, 25 Nov 2020 19:21:53 +0100

Day 2 of my Week of Testing!

Yesterday I ran my first load test on k6, and actually set up the thresholds wrong. I still maintain that there are some UI improvements that could still be made in that area, and that feedback was well-received. Apparently I’m not the only one who has made the same mistake.

Today I learned that on k6, you can actually set one test to be a baseline. There’s so much I want to say on this topic - that’s probably a whole blog post there - but for now, suffice it to say that having a baseline test to compare succeeding tests to is a key to a successful round of load testing. I was happy to see that k6 thinks so too, because that functionality is built into the UI by default.

I also think that being able to schedule a test regularly is fantastic for practicing Continuous Testing. Of course, teams already using CI/CD pipelines would probably get the most benefit out of running load tests using the command line rather than through the UI, but it’s nice to know that scheduling is an option in the UI as well.

Week of Testing: Day 1 - Running a load test on k6

Tue, 24 Nov 2020 19:13:03 +0100

On my first real day of testing with k6.io, I wanted to jump right in and figure out how to run a load test on k6 Cloud. k6 has documentation on this, but I didn’t use it for this test, since the point of a week of testing is to record how easy the product is to use.

My verdict: I’m pleasantly surprised with how easily everything went. Is it just me, or was that AWS startup time exceptionally fast? I’m used to AWS taking at least a couple of minutes to provision a machine, but this seemed to start up almost instantaneously.

Week of Testing: Intro

Mon, 23 Nov 2020 20:45:31 +0100

Today was my first day at k6.io! I’m happy to announce that I’ve joined as their newest Developer Advocate for Performance Testing. k6 is a company I’ve had my eye on for a while, even (or maybe especially) when I was working for a competitor, Tricentis Flood. I always admired k6’s stance on open source, their involvement with the developer community, and the slick web app that brings load testing into the hands of many developers and testers.

As a load tester, I try to be tool-agnostic. I have to admit, though, that I do have a strong bias towards opensource tools. I joined Flood because of their focus on tools I already loved like JMeter, Gatling, and Element (based on Puppeteer). (And Selenium, but it’s a stretch to say I love Selenium.) But these are tools I’ve used for years, most of them from even before I joined Flood. What kind of load tester would I be if I could only really talk about scripting in a handful of tools?

So I thought it was time for a change, and k6 was the first and only company on my list for a couple reasons:

The free and open source tool k6, which recently crossed over to 10,000 stars on Github, is often touted as the developer’s load testing tool of choice.
k6’s cloud offering integrates seamlessly with the OSS tool, and it adds some killer features like performance insights that add real value to teams wanting to scale their load tests.
The k6 team is incredibly responsive to the needs of their community. Unlike most companies, where only marketing, sales, and customer support people talk to customers, k6 developers are active on GitHub, social media, and other blog sites. Everyone is passionate about what they’re building.
Roles in testing are changing, and the lines between tester and developer are blurring. Developers who don’t test are going to become just as obsolete as testers who don’t develop. I see k6’s popularity among the developer crowd as a good opportunity for me to sharpen my coding skills.

So I had to see for myself whether k6 was actually as good as it seemed.

Luckily, it turns out that k6 has a tradition of a “Week of Testing”, where every new member of the k6 team, regardless of role, spends their first week putting the product through its paces. Every part of the whole k6 solution (Cloud and OSS tool) is fair game, and the goal of the Week of Testing is to present feedback to the team at the end of it, while the new member’s eyes are fresh. When they told me about it, they said “We want to see you learn the tool on your own to see what parts are difficult to figure out.” To which I responded, “Challenge accepted!”

I decided to not only give a presentation at the end, but also to video the process for posterity’s sake… and, let’s face it, to impress my new managers and coworkers with my dazzling work ethic.

Step 1: Get the company’s name right.

In the first video, I share the differences between k6 vs. k6 Cloud vs. Load Impact!

How to Become a Load Tester

Wed, 07 Oct 2020 14:15:45 +0100

Originally posted here.

Load testing can be a pretty intimidating field to get into, but it’s easy to get started. Here are some concrete steps you can take to start learning how to load test from scratch.

In this video, Flood co-founder Tim Koopmans and I teamed up at Tricentis User Conference 2020 to talk about the essential ingredients for load testing.

Background required for load testing

Load testing can be quite technical. It involves not just the ability to use different programming languages to create load testing scripts but also familiarity with software engineering practices, DevOps, application infrastructure, site reliability, and networking. A good load tester needs to be able to understand the basics of how an application works and how different components interact with each other. It’s common for a load tester to work on applications employing a variety of technologies, so staying up to date with recent technological trends and switching seamlessly from one language to another, or from one platform to another, is of utmost importantance. This technical side of load testing might make you think that a load tester needs to have formal studies in computer science and several years of experience in software testing. The reality is that there are many load testers who stumbled into the field from wildly unexpected backgrounds. So you might think load testing looks like this:

‍ Photo by Christina @ wocintechchat.com on Unsplash

And sometimes it does. But it more often looks something like this:

‍ Photo by You X Ventures on Unsplash

You see, load testing is an interdisciplinary field. So much of the job involves talking to people. Sometimes the requirements for a project are not clear, and someone needs to talk to the both the business and the developers to make sure that nonfunctional requirements are clarified. A load tester is a detective, tracing a request as it goes from one application component to another, coordinating with different application teams. A load tester is a scientist, using the scientific method to design experiments that yield the results. A load tester is a statistician, making sense of a jumble of data and seeking correlations between variables. A load tester is a public speaker. Even when the actual load testing has been completed and the report has been sent, a load tester makes sure all stakeholders understand the report so that the results can be used to make meaningful changes to improve application performance. All these tasks require a mishmash of skills and competencies that can be difficult to find in just the field of computer science.

Let’s take our backgrounds as anecdotal examples.

Tim grew up and currently lives in a small country town in Australia, and he joined the military straight after school. He later developed a love of automating things, which led him first to use tools like perl to scrape websites and try to win bids with early versions of ebay, and then later to starting a performance engineering consulting career and co-founding Flood itself.

I, Nicole, grew up in Metro Manila in the Philippines (which has 14,000 more people per km2 than Mumbai does). I learned English as a foreign language in school, and I have a university degree in Economics. When I discovered load testing, I had a job in a completely different field– I was working for an airline and making sure engine components were properly maintained and repaired. My love for gadgets and technology took me down an 8-year rabbit hole involving doing performance consulting for clients across a variety of industries in Australia and the Netherlands. At Flood, I help customers around the world improve their application performance.

As you can see, Tim and I came from very different backgrounds and different parts of the world. Neither of us knew what load testing even was when we started working. Neither of us got formal instruction in load testing. But somehow, we still both ended up becoming load testers at Flood.

Reasons to become a load tester

So why become a load tester? Here are some practical reasons.

Load testing is cool

A little-known fact about load testing is that it can be really satisfying to be involved in big-name projects that end up going live.

You may have heard of Flood. But even if you haven’t, we’re sure you’re familiar with the results of load testing done on Flood. In the last few years, we have been helping Hotstar, one of the largest streaming services in the world, load test their systems in preparation for the Indian Premier League.

Hotstar uses an open-source tool that we support, Gatling, to create scripts to simulate users tuning in to watch cricket games. Then, they use Flood to create a “Game Day” simulation. They scale up those load tests and run 5 million users on thousands of machines in the cloud. We help them coordinate execution, spot problems quickly with real-time results from all those machines while the test is still running, and identify bottlenecks so that they don’t occur in production, when India’s playing.

As a load tester, you’ll definitely be working with brands you recognize and love.

Load testing can pay well

‍ Source: Payscale

I said this would be a practical list, right?

Here are 261,768 practical reasons. The average performance engineer’s salary is 71% more than that of an average software tester in India, according to Payscale.com. We all know that money isn’t everything, but this is a clear signal from the market about the importance of performance-related skills. Adding load testing to your CV as a tester makes you more attractive to employers. It’s a great way to differentiate yourself in an over-saturated market.

Load testing can literally take you places

‍ Source: Shamim from Flood

Then there are the side benefits of being in load testing. The photos above are real ones from the Flood team. That’s me on the upper left, exploring a castle in Belgrade, Serbia, and that’s Tim in the bottom right, taking a photo in San Francisco, USA.

We’re a fully distributed team on Flood, which means we can work from anywhere. Flood is a web app, so you don’t need to install anything– I’ve been known to start load tests from my iPad in a café. Load testing on the cloud frees you from being tied to running tests from the office. You can start load generators when you need them and shut them down when you don’t– all from a web browser. It’s a job that can come with a great work/life balance, if you find the right company.

Load testing can also be a great way to move to another country. It’s what’s allowed me to move to other countries– twice. In Australia, the average performance engineer’s salary is AU$108,053, or over 5.6 million rupee.

‍ Source: Payscale

So it’s clear that there are some pretty good reasons for you to become a load tester. I’m going to walk you through the practical steps to become a load tester. You may not be an expert by the end of this, but I hope I can convince you that you can get started in load testing whether your background looks like Tim’s or mine.

Load testing adds real value

And the trite, but absolutely true reason: Everyone fails.

As a load tester, you are a gatekeeper of quality. What you do directly affects revenue and customer experience.

Performance is an aspect of software that is often overlooked, but it’s only becoming more important in a world where everything is going online. People are being born who have never known what it’s like not to have the internet, and consumers in general expect applications to be responsive and stable. Would-be customers who can’t get an e-commerce site to load within 3 seconds may switch to a competitor’s site, and we all know there are many of them. Performance-related outages can be seriously damaging to a company’s reputation, and social networking amplifies customer dissatisfaction by effectively giving everyone a megaphone to complain into.

Software functionality is still important. However, if an online store goes live with no critical functional defects, but it fails when the 100th person logs into it, or it takes 10 seconds to load the homepage, can you really say that it was ready for production?

Load testing is a crucial part of ensuring software quality, and quality is the responsibility of everyone on a team, even if the word “performance” isn’t in your job title.

The essential ingredients of a load test

Load testing is like cooking a meal. This is one of Tim’s favorite dishes, aloo matar.

‍ Source: Cubes n Juliennes

You can approach cooking as a science. There are recipes with a list of ingredients, a precise measurement of how much you need for each, and a procedure for creating the dish.

But you can also look at cooking as an art. Cooking the same dish over and over again will give you an experience in working with those ingredients that a novice cook won’t have. There’s room in cooking for adding one ingredient, changing the quantity of another, or just trying something new.

A master chef knows how to use both the science and the art to create something delicious. But you don’t HAVE to be at that level in order to cook.

Anyone can cook. If you know how to follow a basic recipe, you may not create a masterpiece, but if you’re like me and you haven’t had lunch yet, you’ll still end up with something you’ll eat.

So here’s the recipe for load testing.

I prefer kadai paneer myself, but aloo matar requires, among other things, peas. And so does load testing.

P is Preparation. To prepare for a load test, you need to get your ingredients ready. You need to decide what it is you want to test, why you’re testing it, and how. You write load testing scripts that will simulate real users as closely as possible.

E is Execution. This is when you do the actual cooking. You run your load tests and scale them up according to your requirements.

A is Analysis. You figure out whether the dish you created is edible. You determine how your application servers responded during the load test, and decide if that’s acceptable.

Preparation – Execution – Analysis.

Preparing for a load test

“As far as the World Cup is concerned, it is a process. We don’t want to jump to the 50th floor straight away. We must start on the ground floor.” - Sachin Tendulkar, via ESPN

Load testing doesn’t start with writing a script. Load testing starts with building a foundation for later tests.

‍ Source: Inside Sport

Part of that foundation is asking WHY. Why do you want to do a load test? Has there been a performance-related issue before? Are you trying to determine whether an application is ready to go live, or is it already live, and you just want to improve its performance? What type of application do you want to test?

WHOM are you cooking for? I think you’d cook differently for yourself than you would for Sachin Tendulkar. What do your end users typically do on your site? Where do they come from?

These are questions that need to be answered before you think about scripting. Here are some resource for planning your load tests:

Start with Why: How to write requirements for API load testing How to write a test plan for load testing Selecting a tool for load testing Generating a load testing script without coding: Getting Started with Test Builder

Execution of a load test

So now we’re ready to start cooking. This is the Execution phase. You’ll need a machine to use as a load generator. Depending on the number of users you want to run with, you may need more than one machine. You might be tempted to run a load test on your laptop, or on a bunch of machines you have lying around, but that introduces new variables that may affect the results.

‍ Source: Ashwini Chaudhary on Unsplash

Like any good cook, you’ll also need to make sure you have all your ingredients handy. In load testing, that means making sure your scripts and test data files are ready to be executed on every load generator.

Both of these problems – provisioning a load generator and transferring files over to them – are solved pretty easily by running on the cloud using something like Flood.

Designing and executing your load test

Where should you run your load tests? A look at demand, hosted, and on-premise infrastructure

Analysis of Results

The last part is Analysis.

‍ Source: Fine Dining Lovers

This is a pretty crucial moment, because this is when you figure out whether what you cooked was delicious or not. Unfortunately, in load testing, it’s not always easy to tell how your test went. It’s like this photo of pakora (onion is my favorite kind). It’s kind of hard to say what expression that is in their faces. Is it disgust? Or is the pakora so yummy that they just can’t wait to eat more?

Load testing typically generates a LOT of data. You’d have to go copy every single file from every load generator you used. But can you imagine the hundreds of load generators that Hotstar used? Of course they didn’t manually copy those files. Instead, they used Flood.

On Flood, we show real-time graphs as you’re running the test so that you can tell immediately whether transactions are failing or whether a server has stopped responding.

After the test, you can also compare this data to previous tests to spot trends, or even create your own dashboards using an analytics engine.

Real-time metrics during your load test

Downloading results and logs after the test

Is that it?

You’ve prepared for a load test, you’ve executed a load test, and you’ve analysed the results from the load test. Congratulations, you’re a load tester! By following this recipe, you’re already doing better than many people, and you’re certainly doing a lot better than those who do not run any load tests at all.

The cool thing about load testing, though, is that this is just the beginning. There’s always room for improvement.

‍ Source: Pop Up City

Dabbawallas are people who deliver hot lunches, and while the concept is not new, the implementation in India is particularly interesting.

For these dabbawallas, having the meals prepared is not the end of the job. Once the food is ready, it needs to be packed into lunchboxes, picked up, separated according to destination, put on trains, and delivered. Later, the empty lunchboxes are picked up, and the system works in reverse.

It turns out that this is actually a pretty sophisticated system. In fact, it’s been estimated that only one tiffin every two months is not delivered. That’s equivalent to a Six-Sigma quality rating of 99.9999999% (that’s nine nines) or more.

Dabbawallas didn’t get that rating just by delivering quickly. They did it by being consistent and reliable. They did it by having a process.

Once you see the benefits of running a load test, you may also start to understand that the real value of load testing is not in a test that you run once whenever you think of it. Imagine having the benefits of a load test tied to your development process, so that it is automated.

As an advanced topic, adding your load testing to your team’s Continuous Integration/Continuous Development pipelines such as sets up a repeatable process that reaps benefits far into the future, not just whenever official load testing activities are carried out.

Continuous Integration with Flood: Integrating Travis CI, Circle CI, and Jenkins

Conclusion

So that’s the recipe for becoming a load tester.

Preparation – Execution – Analysis.

In Preparation, you determine why you’re testing and what tool best fits your needs. We showed you how to use Element, but there are many open source tools out there. Don’t get overwhelmed by scripting– a lot of times it can just mean replacing a URL with the one YOU want to test.

In Execution, you run a baseline test that will serve as a point of comparison for future testing, choose the number of users and the number of regions you want, and then you run your test. Running on the cloud is the easiest (and usually cheapest) way to get started.

In Analysis, you view the results in real-time, compare historical performance to see trends, and share reports to stakeholders so that your team can make changes to your application.

We also talked about the bonus topic of CI/CD and how you can incorporate load testing into your pipeline to get a continuous feedback during your development cycle.

Tim and I started in very different places and had different experiences before we discovered load testing. But eventually, we ended up on the same path, and that’s what we’ve outlined for you here. We hope that this convinces you that load testing doesn’t have to be complicated.

You absolutely don’t need to be an expert to run a load test. Even if you don’t think you’ll get all these steps right, a load test that is less than ideal is still better than no load testing at all.

In the end, anyone can be a load tester with the right attitude and the right tools.

Learning about load testing

High Performance Websites book by Steve Souders

Ministry of Testing/PerfBytes list of resources

PerfBytes podcast (EN and ES)

PerfGuild conferences by Joe Colantonio

Señor Performo’s YouTube Channel: Señor Performo ENG

Scott Moore’s Performance Tour on YouTube

Lifting the lid on performance testing

Thu, 24 Sep 2020 23:38:56 +0100

I joined other performance and testing engineers in this online panel at Testing Talks Online about load testing best practices.

From the Testing Talks Online site:

Performance testing is an essential part of the quality assurance function. However, across the industry the great majority of companies still struggle to master and execute a successful performance testing strategy that produces measurable results. In this panel discussion, Cameron Bradley will host some of the best performance engineers in the world. The panel will discuss how we as quality assurance experts and performance engineers can better understand performance testing, the value of performance testing and how world leaders are approaching and succeeding with performance testing as part of the delivery life cycle.

How to Write a Load Test Plan

Mon, 21 Sep 2020 16:33:23 +0100

Originally posted here.

People often start load testing by creating a script in their favorite load testing tool, but good load testing starts before that. Planning a load test may seem like a tedious exercise, but it is essential to ensuring the success of your testing.

A test plan should answer why, what, who, when, and how the testing will be carried out.

A good test plan, particularly for load testing, includes the following components:

Requirements
Scope
Entry criteria
Workload modeling
Server monitoring
Test doubltes
Test Scenarios

We’ll dive into each of these, but first, a note on the format of a test plan.

Test Plan Format

Test plans have been traditionally presented as a document, but there’s no hard-and-fast rule. Instead, the length of a test plan, and the level of detail it goes into, should be tailored for the complexity of the project and the backgrounds of the stakeholders.

Teams using Agile methodologies may not have a formal test plan at all, but I would argue that all teams should still have some sort of plan, whether it’s a Word document or a list of bullet points. Written plans clarify and communicate intent, and testing can be an expensive and fruitless exercise without a consensus on its intent.

Technical documentation can be difficult to digest and understand, and that’s dangerous when it comes to software development and testing, where it’s imperative that all stakeholders are on the same page about the state of an application.

Paul McLean, a performance engineer, has a creative solution for this. He creates a companion video for every technical report, giving stakeholders the option to read the written report for details but watch the video for a more high-level and nuanced explanation.

Here’s an example of a companion video he created for a test plan:

Requirements

Everything starts with requirements. Until requirements are identified, load testing can only ever be aimless and explanatory, and any performance bottlenecks spotted will be incidental.

Requirements inform every step of the load testing process. Why are we doing load testing? What exactly do we want to test? How will we know when a test has passed or failed? How will we know if application performance is good enough to go into production? What does “good enough” mean?

This is such an important part of the planning process that I felt it deserved its own blog post. See it here: Start with Why: How to Write Requirements for API Load Testing. Even though I specifically mention API load testing, the principles apply to load testing in general.

Scope

At Flood, we use a product development methodology called Shape Up to keep us on track as we build new features. Projects in Shape Up are called “shaped work”, and one of the basic properties of shaped work is that it is bounded.

…shaped work [also] indicates what not to do. It tells the team where to stop. There’s a specific appetite—the amount of time the team is allowed to spend on the project. Completing the project within that fixed amount of time requires limiting the scope and leaving specific things out.- Ryan Singer, Shape Up: Stop Running in Circles and Ship Work that Matters

Defining scope is setting boundaries. Business priorities need to be weighed against the resource limitations (number of people available to do the work and time available) in order for testing to deliver maximum value.

Some considerations for scope include:

Specific features or key transactions to be tested
Types of tests included (component test vs end-to-end test)
Test scenarios (peak load test vs disaster recovery)
Applications included in testing

As with most things in the planning phase, scope is something that can change during the test when unexpected circumstances arise or when priorities change. But it’s still a good practice to define the scope at the beginning, and update it as it changes.

It’s important, however, to prune with intention. By keeping our objectives in mind, we can keep our work in line with our intentions.

Entry criteria

Occasionally, you may be asked to load test an application that isn’t ready to be load tested. This happens more often than you think. While it’s true that Performance Testing is something that should be baked into all software development from the beginning, this doesn’t hold true for Load Testing. Load testing is only one activity that falls underneath the broader umbrella of performance testing.

Entry criteria are conditions that you need to be fulfilled before the testing actually begins. It’s a good idea to have these conditions communicated beforehand so that everyone is clear on what needs to be set up before you can do your job.

Functional testing: Does the application work?

Load testing cannot realistically be carried out until at least the core functionality has been tested and high-severity defects have been fixed. Depending on the kind of load test you want to execute, you may also want to specify that UAT has been executed, as there’s no point doing an end-to-end load test with 1000 users if it doesn’t work for one user.

Load Testing Environment: Is it production-like?

Nonfunctional testing has stricter requirements for an environment than does functional testing, and you may have to champion this cause. For load testing, it is not enough to have an application staging environment that is a virtual machine that is a quarter of the size of the production environment. It’s important to get as close to a production-like environment as possible in terms of capacity (memory, CPU), codebase (the actual build that will be deployed), and integrations with other environments or servers (if within test scope).

Load testing is not linear: a response time of 5 seconds on a server with half the capacity of the production server does not necessarily equate to a response time of 2.5 seconds in production.

This is also the time to think about your load injectors. Will they be on-premises, or in the cloud? A good entry criterion is the availability of the machines in the right network and with the right tools installed. If you’re using commercial tools, license provisioning should be a criterion. What sort of capacity will your load testing scripts require?

Support: Are there people available with expertise in key components?

Load testing is a team activity. When a load test involves multiple application teams, it’s important to request availability of key persons on those teams during the test. Often as load testers, we are seen as working independently, but the truth couldn’t be farther from that. Load testing is a team sport. We need support from:

business analysts who will be able to tell us how things are expected to work and what the current priorities are

developers whom we can consult when poorly performing code needs to be optimised
functional testers who can show us how the application works
DevOps engineers who can help us provision and monitor servers

and many more!

‍

Test Data: Do you have dummy data for your virtual users to interact with?

Once you have as close a copy of the production environment as possible, keep in mind that it’s still a clean copy, which may not be realistic. If there are databases in production, how much data do they contain? The application server may respond differently when your test database is empty compared to when it must contend with gigabytes of data in the production database.

Using production data for testing purposes can be dangerous, not to mention illegal, in certain circumstances. To avoid this, you can scrub sensitive data or generate your own, either by injecting records into a database or by writing an automation script to create data on the front-end. If you write a script, you may be able to reuse parts of it later for your load test.

Workload Modeling

A workload model is a schema describing the load profile for a given test scenario, and it involves determining what (the key transactions), how much (the load distribution among the transactions) and when (timing of the load) to test.

Workload modeling can be the most difficult part of the testing process because it involves finding out how load test scripting can best mimic what is actually happening in production. It can also be the most critical.

Here’s my account of working on a project in horse racing, where workload modeling was of particular importance: Workload Modeling - Preparing for Large Events like the Melbourne Cup.

‍ When you’re selling bets for the Melbourne Cup, every second matters.

Server monitoring

Executing a load test without monitoring server health is like flying blind. You’ll know when you land safely and you’ll know when you crash, but even if you do crash, you won’t know why—or how you can avoid it next time. Monitoring server health is the black box that will tell you what went wrong.

For load testing, you’ll want to monitor the application servers that you’re testing as well as the load generators that you’re using to run the load tests themselves. That’s right; if you’re not watching carefully, the machines you run the load tests on can be the bottlenecks in themselves, causing unnecessary failures in your load tests.

Joe Colantonio goes over the basics of resource utilization in terms of four main areas of concern:

CPU
Memory
Disk
Network

Here’s another guide I put together on the basic metrics as well as some tools you can use to set up your monitoring.

Regardless of which metrics you identify as key or which tools you use, you’ll want to make sure that they’re all set up to measure server health before you run your load tests.

Test doubles

Part of the strategic planning around load testing is deciding which components need to be tested in conjunction with your stated requirements. Applications can sometimes be complex enough that load testing end-to-end is not feasible due to the number of teams involved or the cost of duplicating application infrastructure.

Keeping it as simple as possible will lead to fewer dependencies, so it’s worth considering creating test doubles to isolate relevant components. Test doubles include stubs, mocks, and full virtual services.

‍

A test double is that part that replaces a complicated component that is not within scope. It’s a “dumber” version that responds to requests enough to allow you to go on with your load testing without actually requiring that component.

Test doubles allow us to abstract out other components we don’t want to test so that we can concentrate on the one that we do. The point of a test double is to remove the variables and noise introduced by other parts of the application so that we can focus on testing how one component responds.

If you are willing to put the time into creating a test double, you can drastically reduce the amount of resources you need to set up an environment and isolate components. Reducing variables in your test allows you to more quickly determine where performance bottlenecks lie.

Test Scenarios

A test scenario is the description of a contained situation or condition under which the application will be tested. A test scenario is usually based on several test cases and includes a plan for how these test cases will be executed.

Choosing your test scenarios means deciding which situation is most likely to yield the data that you require. Employing several different types of scenarios will give you a greater understanding of your application’s capabilities. You should feel free to create your own scenarios that are uniquely tailored to your requirements, but here are some common scenarios to start out with. Take the number of users and durations mentioned as guidelines and not rules.

Here’s what our Flooder amigo, Señor Performo, has to say about test scenarios:

Text version here

Conclusion

Creating a test plan can seem daunting, but the process of writing it serves as a prompt for discussions among your team.

Here’s a pro tip from Scott Moore: Once you come up with a template you’re happy with, save it and reuse it for future projects. Modify it over time based on feedback to see progressive improvement.

Building a set of reusable documents and other deliverables (“templates”) to use in the initial planning phase and final reporting will speed your initial startup time. I recommend working with or learning from others with experience to build out some of these templates.

‍When in doubt, remember that the format of a test plan is not as important as making sure it is meaningful and comprehensible for your team.

‍For more information about the load testing process, check out my book on API load testing.

Pros and Cons of API load testing

Tue, 18 Aug 2020 21:22:49 +0100

Originally posted here.

‍API load testing is one of the most popular ways to run a load test because of the number of open source and commercial load testing tools available on the market that support it. Creating a script to hit your application’s API endpoints is the traditional approach to load testing, but it’s not the only one.

Here are some things to consider when deciding whether or not API load testing is right for your application.

Disadvantages of API load testing

API load testing does not simulate real users interacting with elements of your webpage.

‌In contrast to a customer opening up a browser and filling out forms on your site, an API load testing script will consist only of the underlying requests to the server that are made by clicking on those on-screen elements. Exactly how the customer triggers those requests, and what buttons they’ve pressed, is irrelevant. It’s all about the raw requests.‌

The format of these requests will be dictated by the type of API your application uses, and it may take some technical knowledge or expertise to be able to formulate requests correctly. However, a RESTful API is one of the most common types, and it is considered to be relatively simple to use.

HTTPS requests for amazon.com. This is what an API load testing script would need to replicate.

It doesn’t give you an idea of how user-friendly your application is.

‌API load testing tools don’t give you feedback on how long your pictures took to render on your users’ browsers or whether that “SUBMIT” button is in an obvious spot.

‌It doesn’t measure front-end performance or how quickly pages render in different browsers.

‌While all the resources that the server returns can be downloaded by your tool of choice, there is no browser on which to run them.

‌API load testing doesn’t run client-side scripts.

‌It isn’t what you’re looking for if your application is a single-page web app that relies heavily on JavaScript or AJAX to dynamically populate and update the page. Your load testing tool will download the scripts, but not execute them.‌

‌This includes scripts like those required to trigger Google Analytics, which prompts many an engineer to question the results of an API load test due to the inability to see the traffic come in on Google Analytics. API load testing won’t help you with that. Running browser-level load tests using tools like Flood Element or Selenium may be more useful.

If you’ve decided that API load testing is not the best way to load test your application, you can look at browser-based load testing as an alternative.

Executing a browser-based load testing script using Flood Element. ‌

‌Advantages of API load testing

Now that we’ve talked about what API load testing isn’t good at, let’s talk about what it IS good at.

‌### It allows you to load test specific servers rather than the whole stack.

‌API load testing allows you to tailor your load testing by applying load only on particular servers. This is particularly useful in more complicated applications that involve several components and would require substantial effort to reproduce in a test environment. Using APIs, you can test only the functions you want to test.

It is well-supported. ‌

API load testing has been around for decades. There are lots of robust tools you can choose from, both commercial and open-source, and many of these tools have large communities and extensive documentation around how to script the most common cases. It’s a far cry from the browser-level testing space, which is relatively new and sparsely populated by comparison.

‍ The JMeter site contains a wealth of information, including a User’s Manual ‍

It’s not as resource-intensive as browser-level testing.

You can simulate more users with API load testing than with browser-level load testing. Since most interactions on the browser-level get translated to requests anyway, generating load this way is incredibly efficient, allowing you to hit your server with requests while bypassing the UI layer and the resource overhead that that entails.

For example, at Flood we’ve baselined some of the tools that we support to see how many users we can run on an AWS m5.xlarge instance:

Selenium: 5 users Flood Element: 20-40 users JMeter: 1000 users Gatling: 1000 users

‌You’ll see that the browser-level test tools, Selenium and Flood Element, can run significantly fewer users on the same sized node as can the protocol-level tools, JMeter and Gatling. This isn’t due to an inefficiency in the tools but rather to the fact that browser-level tools inherently need more CPU and memory to start separate instances of browsers per user and render pages graphically. You can run more users per node (or machine) by carrying out API load testing with protocol-level tools.

It’s cheaper.

‌The efficiency in resource utilization translates directly into cost savings because every node that you don’t need to execute your tests on is one fewer node that you’ll have to pay to provision (whether on premises or in the cloud).

‌For this reason, API load testing is one of the most cost-efficient ways you can get started with load testing, allowing you to scale up your load relatively cheaply while getting immediate results.

How to get started with API load testing

API load testing isn’t for every application, but depending on your test scenario, it may be the easiest way to test application performance.

If you’re new to non-functional testing or API load testing in particular and would like some step-by-step tutorials on how to get started, check out my free ebook, API Load Testing: A Beginner’s Guide. It takes you through the basics of API load testing, including planning your first load test, selecting an open source tool, scripting, executing a test on the cloud, analyzing results, and even integrating it into your CI/CD pipelines.

Performance testing the Dutch coronavirus hotline

Thu, 13 Aug 2020 22:18:04 +0100

Read the original Dutch version here. This English version was first posted here.

On June 1st, 2020, the National Institute of Health and Environment (RIVM) announced that everyone with certain symptoms could call the hotline 0800-1202 to arrange a free COVID-19 test. The promise: call today, get tested tomorrow, and receive the results the day after that. The reality, though, did not live up to expectations.

The opening day of the hotline was chaotic, and many Dutch people reported connection problems due to a system overload. Yet this could have been prevented with a better understanding of how human psychology can affect application performance.

The situation

According to Dr. Ann Vossen, chairperson of the task force charged with scaling up mass testing in the Netherlands, the country was ready to process 30,000 tests per day. This number corresponds to the national capacity of all the municipal health clinics (GGDs) in the Netherlands. Despite this claim, though, only 1,146 people were actually tested on the first day of testing (the day after the opening of the hotline). Perhaps more importantly, only 5,748 appointments were set on June 1st. The situation was no doubt exacerbated by the overwhelming 323,000 people who called the hotline on its first day, prompting even the telephone operators to admit that they’d had troubles logging into the appointment system.

Bron: From @ggdghornl, Twitter

[“We see that there are many calls going to 0800-1202. You can call for an appointment until 8pm tonight. Please try your call later to avoid long wait times.”]

On Twitter, users shared horror stories: one claimed to have waited on the line for 6 hours, only to be told that the system was down and no appointments could be made; another complained about getting disconnected after getting in touch with an operator. Yet another posted a video of calling the hotline and a recorded message saying “Al onze medewekers zijn op dit moment in gesprek. We zullen u zo snel mogelijk te woord staan. [All of our operators are currently on other calls. We will answer you as soon as possible].”

The second day was somewhat better: there were 11,000 appointments made, partly because of the lower amount of phone calls and partly because of some fixes of technical problems. In the first week, the number of appointments increased to 50,000 (about 7,142 per day), which was still significantly lower than expected.

There are some lessons that we can learn from this case study.

Source: Coronavirus test procedure from GGD Hollands Midden

English translation:

Do you have these (mild) symptoms?:

cough

blocked or runny nose

fever

loss of smell or taste

Step 1: Make an appointment

Call the national number 0800-1202

Have your BSN with you when you call

Make an appointment to get tested at one of our locations

Step 2: Test

Go to the selected location at the scheduled time

You will be tested at the test location

Step 3: Results and eventual start of source and contact research

You will be called with your results within 48 hours

If you are infected with the coronavirus, the GGD will call you for source and contact research *If you show more serious symptoms, or if you fall within a high-risk group, report to your doctor or to the emergency help line.

The effect of fear on performance

This coronavirus has already shown us a side of humanity that we’d perhaps rather not see: fights over toilet paper, chaos on the stock market due to panic, and sabotage of 5G cell towers based on a belief that they caused COVID-19. As much as we’d like to believe it, we’re not always the most rational beings. We all have the tendency to behave unpredictably, especially when we’re afraid.

It’s no wonder that the coronavirus testing line was so welcome. The fact that there were 323,000 callers on the first day is even more remarkable when we remember that majority of Dutch residents had been in self-isolation for months at that point, which lowered their chances of contracting the virus. In theory, the fear that drove these callers might have been somewhat irrational, but not unpredictable.

In English, we call this FOMO (Fear of Missing Out). We’ve already clearly seen how FOMO can drive users en masse to an application, with effects on its performance. In this case, people were afraid that coronavirus test shortages would mean there weren’t enough tests to go around, and they wanted to be first in line to get tested.

Another problem was that one in four callers just wanted general information about COVID-19, despite government pleas to keep the hotline free for coronavirus test appointments only. It didn’t help that the hotline opened on a Dutch national holiday, White Monday, and more people were home than usual for a Monday. The high number of unrelated calls was unexpected, but nevertheless factored into system overload.

What could we have done to improve this system?

Improvements to the corona hotline

Better estimation

As difficult as it is to predict FOMO’s effects, we can still try. We can make educated guesses based on related statistics.

Just one month before the coronavirus test hotline was opened, Jaap van Dissel, the head of RIVM, estimated that there were up to 700,000 Dutch residents infected by the coronavirus. Why, then, was it such a surprise that more than 300,000 of them had called? Furthermore, the RIVM had encouraged people with a broad list of symptoms (such as colds, coughs, or fever) to call, and it’s reasonable to assume that some people with conditions other than COVID-19 might also have called.

Even if we assume that 20% of the 700,000 people with the novel coronavirus did not have symptoms (according to WHO research), we still arrive at a predicted 140,000 calls, which is still 25 times what was the coronavirus testing system was able to process on the first day.

If we have an educated estimate, we can start testing.

‍

Injecting load

When a resident calls the hotline, the call is routed to one of several operators working from home. This is often done with a telephony server that forwards calls based on some type of round-robin system. We can execute load tests on this server to determine how many phone calls a server can process and route successfully, with minimal delay and without message queueing.

Then, the operator enters personal information from the caller into the appointment system, including their government identification number (BSN) and address. We can surmise that the system checks the identity to verify the validity of the BSN, and then sends some details to a database. The closest municipal clinic to the caller can then use the data to process and schedule an appointment. We can also inject load at this stage by simulating the calls that the appointment system creates with an API load testing tool like JMeter or Gatling.

Using different test scenarios

With such a public announcement of the hotline’s opening, a spike test before release could have yielded some useful information. In a spike test, we simulate a sharp increase in users over a short period of time. For example, we could have simulated an increase in users from 0 to 700,000 on the telephony system over 10 minutes to see how the application would have handled that.

Soak tests could also have been useful in this case. The hotline’s opening hours were from 8 am to 8 pm, so it was open for 12 hours in total. With soak tests, we can generate load on an application for an extended amount of time. Soak tests can reveal memory leaks or other bottlenecks in the processing of data that might occur after hours.

These are ways we could have tested the system as it is, but are there improvements that we could have made in the system to make it more resilient to reduce the FOMO factor?

Using automation to lessen FOMO risk

The system had some manual parts of the process that may have exacerbated load issues. Here’s how this system could have been automated to reduce bottlenecks.

Interactive Voice Response

Telephony servers commonly use Interactive Voice Response (IVR), an automated menu system for voice calls that uses either voice recognition or keypress tones to navigate a user through a series of choices. We’re all familiar with it: “For Dutch, press 1. For English, press 2…” Such a system might have helped reduce the overall load by quickly filtering out callers in the 25% who weren’t interested in creating an appointment. “If you’d like to book an appointment for a coronavirus test, press 1. Otherwise, please call the information hotline at….”

Here’s what an IVR system could look like:

From: Vanguard Networks ‍ Load testing IVR systems can be difficult, and it usually requires the use of highly specialized commercial tools. However, introducing a new tool may also increase the time required for load testing.

Building a web app

Another approach would be to implement a web app to verify identity and capture personal information quickly. In the Netherlands, we already have DigiD, an identity verification system that is heavily used for governmental services. Mijn Overheid, which is a central government portal accessible to every Dutch resident, already interfaces with DigiD as the sole method of logging in. Reusing these existing services could have saved a lot of the work on a web app— not to mention reducing the bottlenecks inherent in a more manual process like a telephone hotline.

‍ Source: Mijn Overheid ‍

Testing the system end-to-end

We’ve identified a few ways to load test various components of the coronavirus hotline, both in its current incarnation as well as in hypothetical improved versions. Isolating each component and running load tests at that level can help us resolve performance issues inherent in the component. However, there are still bottlenecks that are only revealed when the integrations between components is tested as well. That’s where end-to-end testing comes in.

For the coronavirus hotline, end-to-end testing means being able to run load tests on the entire process and seeing how data flows from one to the other: the call forwarding from the telephony server, the identity verification system, the appointment system, the database of personal information, the servers of every municipal clinic, and the email/text notifications with appointment details. As modern applications grow more complex, it can be tempting not to run end-to-end tests, but we do so at great risk of functional or nonfunctional issues.

An area of end-to-end testing that is often overlooked is the human factor, which can also be a bottleneck. Had hotline operators been trained to ask early on in the call whether the caller wanted to make an appointment, to filter out unrelated calls? Can the coronavirus testing and test result processing, heavily involving manual work from medical professionals, match the stated requirements (test within 24 hours, result within 48 hours) even when the digital components are performant? Some people waited for an hour in the cities of Eindhoven and Goes due to traffic congestion from cars near the test centers, leading to delays in the testing schedule.

While it can be difficult to load test these logistical systems with automated tools, they may have significant effects on the overall performance of an application and should be considered.

Conclusion

Often, performance issues can lead to a tangible loss in profits or a more intangible loss in reputation. In this case study, poor performance had a direct impact on people’s health. Potentially sick people were unable to make appointments to get tested, and later treated. Delayed tests could have led to an increased infection rate as people waited for a confirmed diagnosis. Hotline operators also admitted that slowness in the system led to them sometimes giving out test results against instructions to wait for trained medical professionals to do so.

‍

Human psychological factors can make a system more complex to load test. According to Andrea Evers (LUCM), a health psychological professor, “Door de uitbraak van corona zijn mensen in een langdurige stresssituatie beland. Onzekerheid, onvoorspelbaarheid en oncontroleerbaarheid maken dat ze de behoefte hebben om zaken juist wel te controleren of voorspelbaar te maken [The coronavirus outbreak has put people in a situation of chronic stress. The uncertainty, unpredictability, and helplessness of the situation cause a need to make things controllable or predictable].”

As performance testers, we can use statistics and a systematic attitude combined with knowledge of human factors in applications to try to predict the unpredictable.

Ask a Flooder 20: How can fear affect application performance?

Thu, 06 Aug 2020 22:43:28 +0100

Originally posted here.

In this video, I talk about the power of fear, and how it can end up affecting performance test results. ‍

Or listen to the audio version here:

Hi everyone, I’m Nicole van der Hoeven, back with another Ask a Flooder, and today I want to talk about the effect of fear on your performance outcomes.

The Dutch “coronalijn”

I currently live in Maastricht, in the Netherlands. At the beginning of this global pandemic, the only people getting tested actively for COVID-19 were either people who presented with serious symptoms of the disease, or health care professionals who were continually exposed to the disease and therefore at a higher risk of contracting it.

Last month, on June 1st, the Dutch government made testing available to everybody. They set up a telephone hotline that any Dutch resident could call to set an appointment to get tested for a COVID-19 test. A task force was dispatched to make sure that they could test up to 30.000 people per day. Yet on the first day of testing, only 1.146 people were tested. So what happened?

Well, it turned out that the bottleneck of the entire process wasn’t getting enough tests, registering the tests, or even processing the test results. The bottleneck actually occurred way before that, when people called up the hotline to get an appointment.

323.000 people called on the first day that the hotline opened. And not even all of them had called to make an appointment. Some of them were just calling for general information on the coronavirus despite the fact that the government made it clear that that’s not what the line was for, and that there were other lines dedicated to information. There were reports on Twitter of people waiting up to 6 hours on the line just to talk to someone who would make an appointment for them. There were calls that got dropped. People said that sometimes it happened in the middle of a call. Even telephone operators that the appointment system was down or slow and unusable for a lot of it.

As a result, only 5,500 people actually had appointments booked on a day where there were 323,000 that called. That’s 1.70%.

What does that have to do with performance?

As performance engineers, we like to believe that everything we do is reasoned. We design tests based on quantitative data. We look at historical trends and metrics to help us decide how to build a workload model. But the truth is that that’s often not enough.

If anything, this pandemic has shown us just how wrong our guesses can be, whether or not they’re informed by facts, especially when it has to do with human behavior. We’ve seen violent fights erupt over toilet paper. We’ve seen peaks and troughs on the stock market that didn’t have anything to do with the underlying companies. We’ve seen arson on 5G cell towers. Because human beings are irrational, and it’s really hard to capture that irrationality in our model for how users are going to behave.

We may never be able to accurate predict how irrationality is going to affect the load on our application in production. But that doesn’t mean that we shouldn’t try. There are still things that we can do to mitigate the risk. I think that any application that is public-facing should be assessed for exposure to the risk of fear or other human psychological factors.

How do we account for fear in our performance testing?

I actually wrote a whole blog post on this particular situation (the Dutch corona line), and how it might have been possible to predict (maybe not with 100% accuracy but with reasonable accuracy) how many people would have called the hotline.

For example, we can look at statistics like the of the Netherlands or how many people normally get flu-like symptoms or at least report them at this time of the year. These are statistics that are readily available from the Department of Health, and we could have used them to inform our decisions on the workload model for such an application. We may not have gotten it exactly, but we would have at least bin in the same ballpark.

If we have a good enough idea about the workload that our application would have to handle when it’s influenced by fear, we can start running load tests around that. Another good open-ended kind of test to run, especially for applications that are particularly susceptible to fear, is the stress test. In a stress test, you push your application to its limits and you find out what they are. You could, for instance, start with your peak work load, and then add a certain number of users every 30 minutes and see how much your application can actually handle. You can do the same with soak tests, where you’re exposing your application to the same amount of load for an extended period of time. Both of those kind of exploratory tests help you define the limits of your application and how it would respond if fear plays a role in the workload model in production.

If we overlook this critical human element in our test design, then we leave ourselves, and our applications, vulnerable to performance degradation and, perhaps, outages.

Till next time, happy flooding!

Read this blog post for more about the coronavirus hotline.

Ask a Flooder 18: How do I load test video streaming with JMeter?

Wed, 22 Jul 2020 22:49:15 +0100

Originally posted here.

In this video, I answer the question: “How do I load test video streaming with JMeter?”

Or listen to the audio version here:

HTTP Live Streaming (HLS) and performance

One of the most popular protocols for video streaming is HTTP Live Streaming, or HLS, and that’s what I’m going to show you today. An HLS video ends with the [file extension .m3u8‍](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8). According to the HLS protocol, when a video is streamed, the first request sent must be a manifest. A manifest is sometimes called a playlist, and it is a list of all the chunks or parts that make up the video. Then, each chunk is downloaded.

To assess the performance of a video stream, we still want the response time of the manifest, the chunks, and the whole video. However, there are many more metrics that are important in video streaming which aren’t captured by just the response time. For example, how do you determine whether a video was played smoothly or if it stopped to buffer? A video that keeps pausing might just drive a user away, and it’s difficult to tell whether that is happening just from the response time.

How to load test video streaming with JMeter

Downloading and installing the Ubik Video Streaming plugin

Luckily, there’s a plugin for that, created by a company called Ubik Ingénierie. The Ubik Load Pack is a set of enterprise-grade plugins for JMeter. One of those plugins is a Video Streaming Plugin, and it happens to be a really easy way to get started with video streaming. This is a paid plugin, but you can sign up for a free 11-day trial.

Once you do that, you’ll get an email from Ubik with a link to download the bundle. Go to lib/ext and then copy this .jar file into the same directory on your JMeter installation. Start up JMeter, and it’ll ask you for the license file. Browse to the license file in the bundle you downloaded. JMeter will restart.

Using the JMeter plugin in your load testing script

Now let’s start creating the Test Plan. Create a Thread group, and then right click on the thread group, go to Add > Sampler and then select the “ULP Video Streaming sampler”.

Click on the sampler and then paste in the URL to the HLS file that you’d like to test. This should end in a .m3u8. This plugin supports several protocols for video streaming, but we can just leave it at auto and it’ll automatically detect which one it is.

In Play Options, you can choose whether to play the entire video or just a part of it, or even whether you want to include the video subtitles in your test.

This Network Options panel is fantastic because it allows you to limit your network speed. Slow networks, like mobile networks, really affect user experience, so it’s really important to take this into account when running load tests. You might even use a parameter here and do something like throttle to 3G speeds for 80% of your traffic, and leave it on Auto 20% of the time. But I’ll just leave this blank for now.

Next, let’s add a View Results Tree listener. Then, let’s run the script once, just to see if it works.

As you can see, we can see the manifest retrieved first, and then all the chunks. We didn’t get any errors, so let’s run this as a load test. We can disable this listener because it just consumes extra resources on our load generator.

Ramping up your load test on Flood

To ramp this up into a load test, we’ll have to go into Thread Group and change these settings. Ideally, we want to be able to change these values from the Flood UI later so we don’t have to go into the script whenever we change them, so I’ll type in these Flood-specific properties. I’ll leave a link in the description about these properties. I’ll also set it to run Forever so that we can change the duration and the video will just keep playing to fill that duration.

To capture some of Ubik’s cool metrics, let’s add a Simple Data Writer listener here and write the results to a CSV file. On Flood, all flood results are stored on /data/flood/results, so we’ll add an output.csv file in that directory that we can download after the test.

So that’s our script! Now we’ll have to include the plugin and related files so that Flood can access them as well.

Let’s go back to the bundle we downloaded and go to the bin folder to look for user.properties. Let’s open that up, and we’ll change the license path to /data/flood/files, which is the directory that Flood saves files we upload to. To preserve this directory structure, we’ll compress just the lib and bin folders.

On Flood, we’ll start a new stream here and upload three things: the jmx script, the license, and the compressed archive of the lib and bin folders from the bundle.

Then we’ll go through the rest of the process here, setting the test parameters. Time to launch the test!

Analyzing video streaming load test results

Flood shows you results of your load test while it’s running as well as after the test. You can see metrics like concurrency, network throughput, transaction rate, and response time for the test. You can also see these metrics for each individual transaction, including the video or sampler as a whole, the manifest, and the chunks.

To see the CSV that our script generated, we can download the Archived Results from a flood.

In JMeter, click on Tools > Generate Report and fill in the fields, selecting the output.csv from the flood’s Archived Results, JMeter’s user.properties file, and then an output directory for the report.

Ubik’s plugin creates a few custom graphs in the HTML report:

Buffer fill in Millis
Lag ratio in percentage
Lag ratio without buffer fill in percentage
Lag time in Millis

Conclusion

Using Ubik’s Video Streaming plugin is an easy and thorough way to load test video streaming with JMeter, and I highly recommend it. I used an HLS video to demonstrate it, but this plugin supports several protocols:

HLS
Mpeg DASH
Smooth
Adobe HDS

For more information, read Ubik’s tutorial on how to use their plugin with Flood here.

Performance Testing FOMO

Wed, 22 Jul 2020 21:30:43 +0100

On this episode of the TestGuild performance podcast, Joe Colantonio interviews me about how human factors are often not taken into consideration when designing load tests– a mistake that can have serious consequences.

Here’s the plain audio version:

Ask a Flooder 17: How do I use unique test data on Element?

Thu, 18 Jun 2020 21:58:49 +0100

Originally posted on Tricentis Flood.

In this video, I talk about how to ensure that Flood Element uses unique test data, using the example of a customer wanting to script unique logins for his application.

Or listen to the audio version instead:

Transcript:

Hi everyone, Nicole van der Hoeven here again back with another Ask a Flooder, and this week’s question is a follow-up question from last week. That user had previously asked about test data in Flood Element. This week he’s asking: “How do I ensure that Flood Element uses unique test data?”

So this user is actually trying to script a login for an application. He has a list of usernames and corresponding passwords that he wants to use. He’s already set both of these up in a CSV file, and he’s also set up the Element script in the same way that I outline in my previous video. However, this user is seeing errors in the test due to the application not allowing a given user account to be used simultaneously in two or more devices. The problem is that the way that he’s set it up is that he has Element running on multiple nodes, and each node has the chance to pick the same user account.

So how do you make sure that it’s a unique 1:1 ratio for an instance of Element and a user account in the CSV file? If you haven’t already watched the previous week’s video, you should pause this, watch that, and then come back here, so it makes a little more sense because the script that I’m using is building on what we already did in the previous week.

Let’s talk about the differences between this script and the previous week’s.

First, I’m declaring a global variable here called globalBrowserID. The idea is to create a unique and globally identifiable string that I can attach to every account in the CSV file.

const globalBrowserID = ENV.FLOOD_GRID_INDEX + '_' + ENV.FLOOD_GRID_NODE_SEQUENCE_ID + '_' + ENV.BROWSER_ID.toString()

The global browser ID consists of three parts:- The first is a grid index, which is a number assigned to every grid, and it’s also globally unique. So if there are three grids being used in the test, then the grid indexes would be 0, 1, and 2.- The second one is a grid node sequence ID, and this is unique within the grid. The node sequence ID is a number that is assigned to every node in a single grid.- The last one is a browser ID. The browser ID is a number assigned to an instance of Element running on a single node.

These are all zero-based, so the very first instance of Element running on the very first node and on the very first grid in the entire test would be 0_0_0.

We’re going to have to modify our CSV file to make use of this browser ID. So whereas the user previously had username and password, I’ve added an ID column. From then, these will have to be sequential. So 0_0_0, 0_0_1... And 0_0_1 would be the second instance of Element running still on the first node, on the first grid. And you can make as many of these as you need.

id,username,password
0_0_0,user1,password1
0_0_1,user2,password2
0_0_2,user3,password3
1_0_0,user4,password4
1_0_1,user5,password5

I’ve also modified the interface UserData to describe the fields in the CSV file. So there’s id, username, and password, and I’ve defined them all as strings.

TestData.fromCSV<UserData>('users_multiplegrids.csv')
 .filter((line, index, browserID) => line.id === globalBrowserID)
 .circular()

This line is telling Element to take this data file with this filename and filter it based on the line, the index, and the browserID such that id is equal to the browser ID. So what happens here is that Element will take the global browser ID, which let’s say is 0_0_0 for the very first one. It’ll then take that 0_0_0 and look at the CSV file for anything in the ID field that has the value 0_0_0. And then it will continue to use the rest of that line.

I’ve also made the CSV file circular just because that’s always a good practice to get into.

And as always, I think it’s a great idea to debug this, especially when you’re dealing with multiple grids and multiple nodes on that grid.

step('Print to console', async (browser: Browser, data: UserData) => {
 console.log('global browser ID: ', globalBrowserID, '| user: ', `${data.username}`, ' | password: ', `${data.password}`)    
})

I’ve also added the username and password here, just in case we need to verify the state that that user is in.

So let’s run the test. Since we’re just running this locally, we’ll see that the first value indeed is 0_0_0, and the user we’re selecting is user1 and password1, which lines up with the username and password values corresponding to 0_0_0 in the CSV file.

I hope that helped, and until next time, happy flooding!

Download the full script and the data file to follow along.

Ask a Flooder 16: How do I use test data with Element?

Tue, 09 Jun 2020 22:18:41 +0100

Originally posted on Tricentis Flood.

In this video, I discuss how to make your load testing scripts more realistic by including test data. The load testing tool I use is Flood Element, but you can apply these principles to any tool. You’ll find a copy of the Element script I used below.

‍

And, if you prefer, here’s the audio version:

Transcript

Hi everyone, Nicole van der Hoeven here back with another Ask a Flooder, and this time the question is: How do I use test data with Flood Element?

Now let’s talk about why you might want to use test data. In load testing, it’s important to make your scripts behave as similarly to real users as possible so that the load that you generate is production-like. The problem is that if you send the same data over and over again to the server, that server is probably going to cache that response. The response being cached means that the server will return the response way faster than it normally would if it had been a unique request. So we can use test data, instead of static data hardcoded in your scripts, to make sure that this doesn’t happen.

Using test data can also help us test different responses. For example, let’s say we’re testing a loan application system, where a user inputs the application ID, the server looks it up, and returns the status of that loan application. Now, loan applications might be in different stages of the process. For instance, some might be saved but not submitted, or maybe they’ve been submitted but failed the identity verification check.

If we used the same application ID every time, then the server would always return the same status and the same details. Instead, let’s create a CSV file that we can use as test data. It might look something like this.‍

applicationId
A100001
B100002
C100003
D100004

We can have applicationId as the single header here, and we can populate this with a list of different application IDs in different stages of the process. That way, when we run the script, the server will have to respond authentically to every request without it being cached.

Now to create our script. This is an empty Element script. To start with, make sure you’re importing TestData here in the first line of your script.

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'

The first thing we’re going to need to do is to create an interface for the CSV file. Let’s call it AppData. Within this interface, we’ll need to define all the fields that we intend to use in our data file and what type of values we can expect to be in those fields. Looking at our CSV here, we’ve just got the application ID, so I’m just going to copy that, and it’s a mixture of characters and numbers, so I’m going to put that in as a string.

interface AppData {
 applicationId : string    
}

Now we need to tell Element how to find this test data. We’ll say that it’s from the CSV, using the interface we’ve just created. Now we’ll tell Element the filename. In this situation, the CSV file that I’m using is actually in the same directory, so I don’t need to specify a path here, but if you’re putting it somewhere else, then you’ll need to specify the folder and the path to it.

TestData.fromCSV<AppData>('applicationids.csv')

Now let’s use the data in a step. In real life, you might want to use an application ID to type it into a text field, but for demonstration purposes, I’m just going to show you how to output it to the console log. Printing to the console is also a good way to debug your tests, especially if you’re dealing with test data.

We need to let Element know that we want to use AppData. So now, let’s output this to the console. Since we’re just debugging, let’s output the application ID. To actually get the value from the data file, we’ll use this syntax:

step('Print to console', async (browser: Browser, data: AppData) => {
 console.log('application ID: ', `${data.applicationId}`) 
})

‍By default, Element reads the data files sequentially. The problem with that is we only have a handful of application IDs. So if we wanted to run this one script 100 times, we’re going to run out of data. What we can do here is add circular. That way, when the script reaches the very last value of the file, the next value it selects will be the first one, and it never runs out of test data.

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

Another thing you might be wondering is how to make it so that Element chooses a random value every time. To do that, we’ll also have to add shuffle:

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

‍Now let’s try to run the test.

element run test-data.ts

As you can see, it’s been finished. The application ID that was selected was indeed a random one because the first one would have started with A. So, it actually chose the last one randomly.

And that’s it for test data with Element. Now, this user actually had a follow-up question, which was about unique test data. I’m going to answer that in the next video. Until next time, happy Flooding!

Sample script

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'
import * as assert from 'assert'

export const settings: TestSettings = {    
 loopCount: -1,    
 description: 'Test Data',    
 screenshotOnFailure: true,    
 disableCache: true,    
 clearCache: true,    
 clearCookies: true,
}

/*** Test Data * Version: 1.0 */

export default () => {    
 interface AppData {        
 applicationId : string    }    
 TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()    
 .shuffle()    
 step('Print to console', async (browser: Browser, data: AppData) => {        
 console.log('application ID: ', `${data.applicationId}`)    
 })
}

Hybrid load testing: the best of testing using protocols and real browsers

Tue, 02 Jun 2020 22:21:11 +0100

At PerfGuild 2020, I spoke about hybrid load testing and how to mix two different approaches to load testing: protocol-based and browser-based.

Here’s the unlisted video from that presentation:

Load testing on the Google Cloud Platform

Thu, 28 May 2020 22:25:06 +0100

Originally posted here.

We’re excited to welcome another cloud provider to the Flood lineup, giving you even more options for running your load tests. You can now use Flood to run load tests on the Google Cloud Platform (GCP).

Many of you have requested an integration with Google Cloud, citing many of its strengths:

Full integrations with open-source tools like Influx and Redis
World-class security
High-performance global network

Seamless access to other apps in the Google ecosystem, such as Search and Maps

We think that adding support for GCP instances opens up a lot of opportunities for flooders wanting to take advantage of these benefits.

Using Flood with Google Cloud

You’ll notice a new Google Cloud section in the Integrations page while you’re logged into Flood:

You’ll need to add your GCP service account key to allow Flood to access your account on your behalf. Once you’ve done this, you’ll be able to start grids and floods right from the Flood interface you’re already using.

For more details on how to set up Flood to use your GCP account, check out our documentation or follow along with the video below:

Anyone with a Flood subscription, including a free one, can use Google Cloud instances at no extra cost from Flood; however, GCP usage may incur separate costs.

GCP is the third cloud provider that we support at Flood, after Amazon Web Services and Microsoft Azure.

Ask a Flooder 15: How do I install Element?

Tue, 12 May 2020 22:45:18 +0100

Originally posted here.

In this Flood Element tutorial, I demonstrate the steps required to install Flood Element, including dependencies it requires. Flood Element is an open-source browser-based load testing tool created by Tricentis Flood.

Learn more about Flood Element

Installation instructions for NodeJS

Installation instructions for Yarn

Installation instructions for Element

Video: Getting Started with Element

Guides: Using Element with Flood‍

Audio version here:

Transcript

Hi everyone, Nicole van der Hoeven here again, back with another Ask a Flooder, and today’s question is: “How do I install Element?”

Now, I already made a previous video on how to get started with Element, but I didn’t actually go into the details of how to install it, so that’s what I’m going to do in this video today. There are actually a few parts to installing Element, and all of these instructions are going to be on element.flood.io. If you go onto the site and click on Install, you’ll see all of the instructions depending on which operating system you’re on.

First, you need to install NodeJS. This is sort of like how, if you’re familiar with JMeter, you have to install Java before installing JMeter. NodeJS is what Element runs on. Secondly, you’ll need a package manager. You can use NPM or Yarn, but in this video, I’m going to take you through how to do it with Yarn. And thirdly, there’s installing Element itself.

First, download NodeJS. I’m going to leave links in the description to all of the sites that I visit so that you can go through all of the instructions later. On the Node site, download whichever installer suits your operating system. Go through the installation steps, and afterwards, check to make sure that NodeJS is properly installed on your machine. You can do that by going into the terminal and typing:

node -v

Just make sure that the version that’s shown here is the same version that you’ve downloaded.

Secondly, install the latest version of Yarn. On the Yarn site, you’re going to have to choose your operating system and the version. We recommend using the latest stable version. And then go through the instructions for your operating system. After you’ve done that, check your Yarn installation by typing:

yarn --version

And the last step is installing Element. On the Element site, you can find the exact code that you need to type into your terminal. It’s:

yarn global add @flood/element-cli

So you can just copy that into your terminal. As of the time of this recording, the latest version of Element is 1.2.3. Looks like that’s installed. To check the version of Element, type:

element --version

So that’s how to install Element. We’ve gotten a lot of questions recently about Element in particular, so I’m going to answering them in future videos. Hope this helped, and until next time, happy flooding!

Innovation in Isolation

Tue, 05 May 2020 22:59:57 +0100

Originally posted here.

We’ve always been a fully distributed team at Flood, but the current world crisis has still changed the way we work. Here are some of the things we’re doing to keep self-isolation from being too isolating— and maybe it’ll help your team too!

Within the Flood team, flooders-in-training (aka children) have made more surprise appearances in team meetings. Work has been punctuated with grocery runs to buy essentials before they run out. Some of us who were out of our home countries when all this happened have had to arrange emergency travel before international borders shut. It would be foolish to deny the impact on team morale as the entire world goes through this collective trauma, eyes glued to numbers that increase by the minute.

Not exactly the most productive work environment.

Yet there’s still work to be done (something we’re grateful for), and there are still many features we’d love to bring to Flood. So how do we keep everyone motivated to release and maintain software?

Encouraging banter

Work shouldn’t be all about work. Understanding everyone’s personal circumstances is vital to a team’s ability to work together professionally. We use Slack as our primary communication tool throughout the day, and we already had quite a few non-work channels on there (#anything, #hobbies, #gaming, #drones, and #travel to name a few). Now we have one for COVID-related chatter as well.

In this channel, we share news from our parts of the world, post photos of things that are sold out in our local supermarkets, talk about new hobbies we’ve taken up to stay sane, and discuss the struggles of co-working with spouses and children who are now also working or studying from home.

Having a dedicated channel gives people license to open up about what’s going on in their personal lives, and it helps the team understand where everyone’s headspace is at. We also use #eng-covid-19 for memes. A lot of memes. It’s easier to work through a tough situation if you can laugh about it together.

Over-communicating

Our co-founder and Product Manager, Ivan Vanderbyl, recently wrote a great article with some tips for keeping communication lines open. This is particularly important during this crisis, where schedules and priorities can change quickly.

Emojis - We use emojis as a non-verbal cue to convey tone, approval, agreement, and disagreement. We add emojis to messages we send and also use them as reactions on Slack to messages from other people. Even when you don’t have an answer or when a response is not required, emojis signal acknowledgment so that nobody feels like they’re shouting into the void. Over time, we’ve developed emojis that are specific to Flood culture, such as the avocado:

Source: Our Twitter account

‍We even have a fruit-based vocabulary of emojis that we can use in our Slack status messages to convey our general mood. Being in Carrot Mode, for instance, means that you’re trying to focus and would appreciate not being direct messaged or @ mentioned.

‍When working remotely, and especially when in self-isolation, everyone has their own cadence for working. It’s on each person to communicate that schedule so that others can respect it.

Text chat - We’re used to working asynchronously due to time zones, so text-based discussions are necessary. We try to over-communicate and “think out loud” before making decisions just in case we have to suddenly stop working due to illness or other pandemic-related reasons. Even something as simple as leaving a link to a document that we’re working on means that someone else will be able to find it if we’re unavailable.
Voice chat - We tend to use voice chat for quick discussions or updates about work-related things. It’s also great for those of us in Europe who are suffering from slower internet connection speeds at certain times of the day due to everyone being home and streaming Netflix.
Video calls - Video calls are for team retros, 1:1 calls with managers, and personal or sensitive conversations where body language is crucial. It’s also great for recreating that “hanging out at a pub” vibe. Speaking of which… welcome to the Flood pub!

The Flood Pub is open on Friday afternoons. It’s where we all hang out together with a beverage of our choice and talk about things that are entirely not work-related. We hop on a Zoom call together, Tim (our General Manager) fires up the “jukebox” (Spotify playlist), and we end the week by getting some facetime with each other—just as we would do (and have done) if we were together in person.

‍Meeting in this low-pressure context lets us talk about things we wouldn’t mention otherwise. It’s also a fun way to check in with each other after another week at work (and in quarantine).

Doing stuff

While communicating openly is productive, getting stuff done as a team boosts spirits. Here’s what we’ve done recently to kick our output into high gear.

BOINC

Like most of you, we’ve been cooped up in our houses, watching events unfold and feeling helpless as things happen that are beyond our control. BOINC gives us a chance to do something to contribute that IS in our control.

BOINC is a platform by the University of California Berkeley that lets you volunteer your spare computer (or mobile device!) processing power to science. From the official site, Boinc “downloads scientific jobs to your computers and runs them invisibly in the background.”

In our case, we wanted to help sequence proteins like the novel coronavirus using Rosetta@Home. BOINC is a perfect fit for Flood because we are in the unique position of starting up and managing thousands of virtual machines 24/7. Our Ops team has been hard at work automating the donation of our computing power to the Rosetta@Home project when these machines aren’t being used.

We’re proud to say our contribution is INcreasing over time as our engineers find new ways to streamline the process. We’re already in the top 5% of BOINC users in the world by contribution.

GIG: Great Ideas Gathering

Every year, the mothership (Tricentis) runs a three-day hackathon called GIG. Two weeks ago was the first-ever GIG where most participants, not just the Flood team, were working remotely.

The theory behind hackathons is that the daily grind can be exhausting after a while, and it can sometimes be beneficial to introduce a short burst of work to revitalize the team. Hackathons provide a framework for small groups to pitch, execute, and report on a single idea without committing too much time to it. Everyone gets a chance to step away from regular duties and work furiously together on something they might not have otherwise gotten to work on.

A hackathon where the team members are separated by distance and timezone is certainly another beast, but here are some lessons we learned about remote hackathons:

Come up with a good idea: This is easier said than done, but a good hackathon idea is a creative solution that delivers a lot of value. Here are the ideas we came up with:
- Floody - Use AI to improve onboarding by walking new users through the steps for running a flood (live on Flood now!)
- Flood Bot - Add smarter checks to detect patterns in load tests and help flooders understand their results (prototype)
- Flood Gate - Use AWS Fargate to reduce costs in provisioning, scaling, and managing virtual servers by about 30% (proof of concept)

When thinking up ideas for GIG, we didn’t commit to releasing them publicly. Even a project idea that doesn’t end up making it into the finished product is an occasional worthwhile detour for team productivity and morale.

Limit scope: GIG is a sprint, not a marathon. While my team (comprised of Jason Rizio and me) was planning out what we wanted to do, we quickly realized that there was a lot more that we wanted to do, but it just wasn’t feasible within the strict timeframe. In the end, we decided to go for an incremental piece of work that we could push to production before the deadline instead of completing just one step towards something abstract and unreleasable.The other Flood teams also needed to whittle down the scope to a smaller, completable chunk: a working prototype.
Clarify assignments: With members on different timezones, it was essential to have an initial meeting to plan out everything that needed to be done and by when. In this situation particularly, we found it useful to split apart the project into independent chunks, to minimize the need to collaborate in real-time.
Touch base regularly: When working independently, team members can quickly diverge off the main path. To avoid that, we met up at least daily by hopping on an audio call on Zoom to spend a few minutes updating each other on the work we’d done. Doing this gives others a chance to help out when one member encounters roadblocks.For anything that didn’t need to be real-time, we worked asynchronously on chat. Here’s a screenshot of the number of messages and files exchanged to our dedicated GIG Slack channel during the event:

‍Yup, 1,535 messages and 60 files in 3 days. That’s collaboration!

We generally prefer public rather than private channels, and we certainly stuck to that throughout GIG. When you’re working with more than one person, it’s counterproductive to have private conversations that can’t be referenced by other people.

We also learned that it’s useful to keep tabs on what the other teams are doing. After the event started, my team realized that another team was changing a page on the Flood app that we intended to use. Luckily, both teams were able to work around it.

It’s also only fair to let other teams chime in on important decisions within the project that might affect future work— like the name for your mascot.

Leave time for the video: It’s not just about doing the work; it’s about communicating what you’ve done in a way that others can appreciate. Here’s a video that my team (composed of Jason Rizio and me) put together for Floody and our improvements to the onboarding process:

Celebrate: After the work is done, don’t forget to pat each other on the back for the hard work! For us, this meant a round of tacos for everyone and a trip to the Flood pub. (Why do we have so many food-based rewards?)

It’s not easy

Keeping a team unified and working efficiently towards a common goal is always tricky, and working remotely, especially during a world crisis, doesn’t change that. Distributed team members need to put in more effort to consciously over-communicate, make room for each other’s unique circumstances, and do little projects that foster creativity when we’re all feeling a little lackluster. Incorporating these tips can create a culture of healthy productivity within a remote team.

‍

On-premise load testing advantages and disadvantages

Thu, 30 Apr 2020 23:06:59 +0100

Originally posted here.

Where you run your load test can affect the results that you get as well as your project’s bottom line. In general, there are two infrastructure options to consider for load testing your apps: on-premise and cloud. Which type of environment is best for load testing both web or mobile apps?

Differences between on-premise and cloud load testing

What is on-premise load testing?

On-premise load testing is running your performance or load testing scripts on load generators you physically control. Typically, these load generators are machines that are either in your office or data center. In on-premise load testing, you are responsible for all aspects of the maintenance of the load generators, including hardware upgrades and software installations such as testing tools, but that also comes with increased control and security.

What is cloud load testing?

In cloud load testing, you run your load tests on a service provider’s machines, effectively outsourcing the provisioning and maintenance of your load generators. Running “on the cloud” means you’re paying someone else to be responsible for these machines. In cloud load testing, you don’t “own” the machines your test is running on; it’s more like renting some time on someone’s data center whenever you need it. Cloud-based infrastructure isn’t necessarily less secure than on-premise infrastructure, but there are certainly more security considerations to keep in mind.

Pros and cons of on-premise load testing

On-premise load testing is no better or worse than cloud load testing; it just depends on what you need. There’s a meme that the cloud is just a fancy term for someone else’s computer— but that’s an oversimplification that glosses over nuances that might make a difference to your bottom line.

Here are some questions to ask yourself when deciding between load testing on the cloud or on-premise.

‍

Test infrastructure can make or break your load tests.

Cloud vs. on-premise load testing: how do you decide?

Why are you running a load test?

If your intention for running load tests is to test your web application’s load balancing, the source of the web traffic, and the location of your environment, may not be as important. If you intend to do stress testing, you may want the scalability of a cloud setup.

Who is the intended audience of your application?‍

Load testing is most accurate when it is as realistic to the production workload and user base as possible. Running a load test on-premises means that the traffic comes from one location, which could be ideal if your application is an internal one.

Do you have the resources in your team/company to maintain load generators?

All machines need to be provisioned, set up, and maintained to function correctly. Depending on how many machines you need, you may even need a team of trained professionals to monitor your environment.

What is your budget?

Both cloud and on-premise load testing have costs. Cloud load testing requires a smaller outlay of cash initially but may cost more to maintain as you continue to pay for the computing time. On-premise load testing requires more money upfront but may cost less to maintain in the long run.

How many virtual users do you want to run?

The number of virtual users you intend to test your application against correlates to the number of load generators you require. In general, the more machines you need, the more expensive it becomes (for both cloud and on-premise load testing), but if you only need a handful of users, perhaps you can find a machine or two on-premise that could do the trick.

Advantages of on-premise load testing

‍

Increased control

Since you own the machines you’re testing on, you can set your environment up with your exact specifications in mind. You’ll be able to make hardware upgrades or update versions of operating systems and load testing tools whenever you want, without needing to liaise with a third-party provider. This option can come in handy if you routinely use non-standard configurations or legacy software. If you’re also doing other types of non-functional testing, like security testing, it may be easier to change the configuration on on-premise machines.

Greater data security

The question of cloud vs. on-premises may be a business decision. Some industries or businesses with stringent compliance and regulatory requirements may have no choice but to run tests on-premise because owning the load generators also means controlling who has access to the data that’s on them. An on-premise environment lets you lock down access if necessary. With cloud load testing, you’d need to have security arrangements with the cloud provider that you choose so that any test data or identifying client information seen or accessed by your test scripts remains confidential.

Lower cost and lower barrier to entry in some situations

If you don’t intend to run too many users of your test script and you already have a spare machine or two at the office, on-premise load testing may be the easiest and most cost-effective way to get started. This way, you avoid having to request the budget for a new tool and needing to train your team on a new interface.

Fewer variations due to latency

Controlling your own load generators means controlling the network they run on. If your corporate network is generally stable, and your test scenarios don’t involve testing from different geographical locations, reported response times (and user experience) more closely reflect server processing time rather than latency and travel time.

Disadvantages of on-premise load testing

‍

Higher cost in some situations

Maintaining your own load generators comes with a higher up-front cost, which may be a showstopper for smaller projects or companies. The more concurrent users you require for your load test, the more machines you need, the more licenses you need for your testing tool (unless it’s open source), and the longer it takes to recover your investment.

Geographical limitations

Since all your machines are in one place, you can’t accurately simulate the user experience for customers who might access your web app from another location. You also can’t fully test the effect of CDNs and other tools that your application might employ to keep response times low. Web apps that can be accessed by anyone on the internet might require performance tests that take into account customers from different regions.

More setup

You need to do all the setup yourself for both hardware and software components. You also need a way to coordinate your test, so that load generators start and end when you want them to. You may want to divert some resources towards environment management.

Less flexibility

With on-premise load generators, you can’t add, remove, or scale up nodes on the fly according to your test— something cloud load testing excels at.

On-premise load testing best practices

The environment you’re testing in is especially important when you’re using your own infrastructure. Here are some best practices for a performance testing strategy that involves on-premise load generation.

Run baseline tests

Treat load testing like a scientific experiment. Before you begin changing variables to improve performance, spend some time figuring out your test configuration so that it’s repeatable. Pay attention to things like workloads, business processes, the number of users you’re running per load generator, test environment configuration values, script pacing and think time set in your testing tool, and test script transactions. Make sure that the resource utilization on your machines remains healthy during a test; otherwise, you’re load testing the load generators themselves rather than your application servers.

Then, run a few tests against your application servers without making any changes. This way, you create a “control group” or control test against which to compare future tests.

This process is called baselining your application performance.

‍ Flood’s Project View lets you track an application’s performance over time and compare recent tests to a baseline ‍

Consider using containers or virtual machine images

Along the lines of reducing variables, it’s especially important to use load generators that are as identical as you can make them. You can manually check that you’re installing the same software and tools on each one, or you can do as cloud providers do and automate this process.

It’s easy to miss software that are necessary for monitoring or the load testing tool you’re using. For example, JMeter and Gatling require certain versions of Java. Test automation or browser-level load testing tools might require web browsers.

You can use containers such as Docker to create a standalone package of everything you need for your test. Alternatively, if you’re using virtual machines, you can also create a full system image that you can use for every virtual machine.

The difference between the two is that a container lets you be more granular about what is included in the package that you then deploy to all your load generators. Virtual machines include hardware components that you can set, whereas containers are software-only, decreasing package sizes.

Using containers or images allows you to keep your load generators as identical as possible (in terms of hardware and software) so that you can assume similar results from each one.

‍‍ ‍

Don’t neglect monitoring

It’s easy to forget that your application servers aren’t the only ones that need monitoring. Your load generators need monitoring as well, so that they don’t end up being the bottleneck in your tests. Here’s a starter list of server metrics that you should consider monitoring.

You might also consider how you can integrate performance management tools you already use into your monitoring strategy.

Spend time on a framework for gathering data

One of the biggest challenges in on-premise load testing is figuring out how to get data from your tests. Each machine runs your test script independently of the others, so at the end of your test, you’re likely to end up with several results files. After you download these off every machine, how do you combine them into a single summary?

Manually collating results is an option, but it’s a tedious one. You can copy all the files into one directory and then concatenate them so that you end up with one giant one. Depending on the size of those results files and how long you ran the test, however, this may not be feasible. For larger files, you might consider aggregating the data and increasing the granularity, so that instead of seeing raw response time for every transaction, you only see the average response time for every 15 seconds. You can use a tool like Logstash for this.

It takes some time and technical expertise to set up a framework for results gathering that works for your purposes, especially if you want to go a step further and have real-time results.

Of course, you can also skip this part by just using the framework we’ve already set up on Flood. Tim Koopmans, our co-founder, created the first prototype of what would become Flood specifically because he got tired of having to set up this framework for every engagement.

How to run an on-premise load test with Flood Agent

‍ The easiest way to get started with an on-premise load test is by using Flood Agent. Flood Agent is an executable that you can install onto a machine that effectively turns it into a full-fledged load generator. Then, you can start a test from the Flood web interface and have your test script start running on your machines.

Agent takes care of a few typical problems we’ve discussed so far:

It uses containers to ensure that load generators are identically provisioned.
It automatically gathers CPU, memory, and network metrics on load generators.
It coordinates the execution of the load test so that you only need to start it from one central place.
It enables real-time monitoring of results while the test is running.
It aggregates and collates all the results for you into a dashboard.

We think Agent addresses pain points that we’ve seen over and over again in our own on-premise performance testing projects as well as in our customers’. Agent is a free download and works with all performance testing tools that Flood supports.

Conclusion

On-premise load testing is still a valid option despite the growth in popularity of cloud-based infrastructure, and it has its advantages and disadvantages. This type of load testing trades a little bit of flexibility for increased data security and control, which is a tradeoff that some companies may be willing to make.

On-premise load testing has traditionally included a lot of tedious setup and maintenance. Still, you can reduce maintenance effort by creating an internal load testing framework upfront that automates coordination, monitoring, and results collection. Using Flood Agent can mitigate time spent on this initial setup.

Ask a Flooder 13: How do you do mobile load testing?

Tue, 28 Apr 2020 23:21:08 +0100

Originally posted here.

How do you load test a native mobile app? I discuss the general approach to mobile load testing, what a device farm is, and why a device farm isn’t necessary for load testing. I also discuss the best mobile load testing tools to use.

Or listen to the audio version here:

Hi everyone, Nicole here again, back with another Ask a Flooder, and today’s question is:

How do you do mobile load testing?

Now, this question came from a customer who was under the misconception that in order to do mobile load testing, they would need to use what we call a device farm. A device farm is a collection of different kinds of mobile phones. You would ideally have the most popular makes of mobile phones (the hardware models) and the operating systems and system versions. You can imagine that this can very easily get a little out of hand to coordinate because of fragmentation. There are just so many devices out on the market, and then there’s the question of how you actually generate the load from that.

You would somehow need some sort of test automation script that was running and then coordinate os that they all run at the same time, and you would use that as a load test. As you can imagine, whether you use real devices or simulators, this kind of thing would get complicated and expensive very quickly. But the good news is that it’s unnecessary if it’s load testing that you want to do. Now I say that because, for functional testing or compatibility testing, you ARE going to want to test your application on a variety of different browsers. But with load testing, you’re really more concerned about the back end part of the equation. You want to generate load on your application servers. In reality, most mobile apps work by sending API requests to that server, usually by some sort of restful API. Luckily, API load testing has been around for a while, so we do have some industry-tested ways to run this.

What are the best tools for API load testing?

Essentially, you’re going to want to test protocol-level tools. That means that you don’t want to be using browser-level tools in which you’re using scripts to drive a full browser. All you really need is something that sends requests on the protocol-level. That’s great because it’s actually way more efficient.

You have two really good open-source options for this, JMeter and Gatling. I’ve already gone a little bit into the differences between these two in my previous video on how you choose a tool for load testing, which I will leave a link to.

And here are some other resources that will help you get started with mobile load testing. I also have another video about how to record network traffic from a native mobile app on JMeter, which sounds a little complicated, but I promise it’s not. In that video, I go over the steps with you, and I’m actually doing it in real-time with you as well. I show you the screen recording from my mobile and from JMeter on my laptop, so you can always pause and watch it at your own pace.

The second thing is that I already wrote a blog post, which is sort of a guide for mobile load testing, which goes into the steps into way more detail than I can here.

And lastly, I have written blog posts about two real-life examples of mobile load testing. The first one was the Iowa Caucus Reporter app. I ran through it from the workload modeling to the scripting and then the actual execution. More recently, I also wrote about the Robinhood app. Be sure to check out the links in the description to find out more.

Until next time, happy flooding!

Browser-based load testing in 2020

Thu, 23 Apr 2020 16:48:09 +0100

Originally posted here.

We first announced our open-source browser-based load testing tool based on Puppeteer, Flood Element, in February 2018. Despite our decision to make it an entirely separate tool from Flood, we’ve enjoyed using it so much as a team that several of us have written about it:

Tim Koopmans: Load testing for DevOps… with Browser Level Users (BLU)

Kevin Dunne: A new approach to load testing with browser level users

me: Prepare for viral videos with Flood Element

Lachie Cox: Why should you load test with real browsers?

our friend Eduardo Riol, as translated by Antonio Jimenez: Puppeteer vs. Selenium for test automation: is a new leader emerging?

Clearly, we’re huge fans of load testing a web application with real browsers. But does the wider software testing industry share that sentiment?

Browser-based load testing in 2020

Browser-level load testing is still pretty new, and there aren’t too many providers offering it right now. Luckily, we just so happen to have access to a database of load tests run on Flood since 2013. Here’s a bar chart visualizing data we’ve collected about the load testing tools flooders have used (we support Selenium, JMeter, Gatling, and Element).

Load tests executed on Flood over time, grouped by load testing tool

Browser-level load testing is increasing in popularity

In 2020, almost 35% of the load tests run on Flood are browser-based load tests.

While the total number of load tests run across all tools increased across the board, the percentage of tests that utilized Selenium and Element, as compared to JMeter and Gatling, has also increased from 2018. Both Selenium and Element are browser-based load testing tools.

By contrast, JMeter, a protocol-level load testing tool which previously enjoyed a 70% share of the traffic on Flood, now accounts for only a bit over 50% of usage. JMeter’s share of the protocol-level pie is higher than Gatling’s, but the percentage of usage for both tools appears to be contracting when compared to that of browser-based tools.

Element use is increasing faster than Selenium use

Despite the fact that Selenium has been around longer, 25% of all floods today are run using Element. This is a significant increase in just two years, and one that bodes well for the future. While we’re still fans of Selenium and still support it, one reason we see for the dominance of Element in today’s market is resource efficiency.

In our baseline tests, we’ve seen that Element can run from 20-50 users per Flood node (an AWS m5.xlarge instance), whereas Selenium can only run up to 5. This leads to some significant cost savings, as running the same number of users with Element requires fewer load generators than with Selenium.

Another reason might be the fact that Element uses Puppeteer in the background. Puppeteer, developed by Google, understandably shines when driving Google Chrome (and recently, Firefox). Puppeteer load testing is growing in popularity due to the wealth of metrics it has access to.

How browser-based load testing works

Traditional load testing is done on the protocol level by sending messages directly to the web server. These messages are sent without simulating the application interface, and the responses that are received from the server are not visualized, either. “Users” in protocol-level load testing scripts are actually threads, with each one sending a list of messages at predefined intervals, and we refer to these users as Protocol-Level Users (PLU). JMeter and Gatling use PLU to generate load.

The screenshot below shows what one request for a single action might look like in a JMeter test plan.

Example of an HTTP request with parameters on JMeter

In browser-based load testing, real browser instances are opened on load generators, and the load testing tool plays the part of a real user in navigating the web application using the GUI. We refer to the virtual users in browser-based load testing as Browser-Level Users (BLU).

Here’s an example of the same action in the JMeter example above, but this time in an Element test script:

step('Step 1', async browser => {
 //Click Start button
 let startButton = await browser.findElement('.btn-default')
 await startButton.click()
})

This version simply describes the button to be found and clicked, whereas the protocol-level version needs to be more precise in what exactly is passed to the server.

Both types of load testing tools can apply the same load on an application (as long as it supports both methods— see the next section). The main difference is in how the load is generated.

A protocol-level script will contain instructions on which messages to send, which protocols to use, which headers to include, and other information that a typical end-user would not see.

A browser-level script will contain instructions on how to interact with the application: which buttons to click, what to type in text fields, which links to follow, and other on-screen elements that an end-user would be familiar with. This leads us to one of the big advantages of browser-based load testing.

Pros and cons of browser-based load testing

Pros

Simple scripting— or no scripting at all

Protocol-level tools require in-depth knowledge about how an application works, what format it expects requests to be in, and how it responds. On the browser-level, you don’t need to know any of that. When scripting a login process, for instance, you just need to tell the script to type a username and password in the right fields and then click Log In. The script doesn’t need to know that the form submits an encrypted request using the Auth0 API (something you’d need to know for a protocol-level script).

In some cases, you might be able to get away with using something like our qTest Explorer plugin to record a business process, generating a script without any scripting at all. Even when you have to write or modify the script, though, it’s a lot less intimidating than most protocol-level scripts.

They’re also easier to troubleshoot. In addition to opening a real browser instance on your computer when you run a script locally, Element can also display this information on the console (shown here on iTerm):

‍ Screenshots upon execution on Element using iTerm

‍This allows you to understand what’s going on in the script and in the application, letting you detect bottlenecks faster.

Reduced test complexity

The streamlined requirements for scripting lead to a straightforward test. There’s no chance of getting request parameters or values wrong.

Ability to test entire stack from the user perspective

Unlike testing APIs, you don’t need to decide which server you want to hit or make sure you have the appropriate requests to hit all the servers. A browser-level test script interacts with page elements like a real user would, so all underlying application servers are tested. Pair this with cloud-based infrastructure on Flood, and you’ve got a compact package for testing your application.

Capable of testing any user behavior

Protocol-level load test tools struggle with some types of applications, such as single-page apps (SPAs), where a lot of the processing is done on the client side. Browser-level load testing tools shine here, because scripted interactions for these apps are no different for SPAs than other web apps.

Record network and user interaction times for front-end optimization

Browser-level tools show you response times from the user’s perspective: how long after a user clicks the log in button does the user see his or her account balance? This metric is a practical one that may differ wildly from the traditional response times that a protocol-level tool reports. Measuring real user experience is a great value add for your test.

Easier to maintain

Lower script complexity means that a browser-based test script is more resilient to changes in the codebase. A change in the underlying requests sent will not require a change in the test script, making it perfect for regression testing.

It’s also easier to pass on to other members of the team due to high readability. Here’s an example of how to select a value from a dropdown menu in Element:

await browser.selectByValue('#challenger_age','34')

It’s easy to see that the script is looking for an element with the ID “challenger_age” and selecting the value “34”. This step could correspond to dozens of requests that need to be scripted if testing on the protocol level, which makes those scripts harder to understand without a proper handover.

Cons

Browser-based load testing tools have a higher CPU and memory footprint than protocol-based tools. Test scripts in JMeter and Gatling send messages to and record responses from a server without really needing to parse responses. By contrast, browser-based tools parse the server response in order to access the Domain Object Model (DOM) elements on the page.

Using real browsers might be closer to the real user experience, but it also takes its toll on node resources. It is possible to run in headless mode, where browsers are started “invisibly” without actually graphically rendering them. However, even then, a browser-level tool can run fewer concurrent users on a node than a protocol-level tool.

Best tools and applications for browser-based load testing

While other commercial tools exist for this, we’re partial to open source tools at Flood.

Selenium

Selenium load testing is appealing because Selenium is already well-known for test automation and functional testing, and a lot of companies already have full test suites in Selenium. The possibility of reusing those resources and just running them at scale as load tests is, by far, the most compelling reason to use Selenium.

Puppeteer

Google’s just getting started with Puppeteer, and the amount of metrics it exposes is exciting: it seems like anything in Chrome Developer Tools can be accessed by Puppeteer. Imagine running a load test on ONE tool that gives you statistics about back-end performance (response times, the effect of bandwidth restrictions, resources returning errors) AND front-end performance (client-side script console logs, render times, time to first paint, etc). It’s hard to imagine a future where this ISN’T a valuable tool in the performance arsenal.

Under Puppeteer, of course, is Element— our load-flavored version of Puppeteer.

Is protocol-based load testing dead?

No. Testing on the protocol-level may have its drawbacks, but it’s popular for a reason. If you’re testing mobile apps or API endpoints, protocol-level testing tools allow you to focus on hitting what you want to test and only what you want to test. Sometimes you don’t want to test the whole stack. It really depends on your test scenarios.

Plus, protocol-level tools tend to be really resource efficient. We’ve been able to easily run about 1000 users per Flood node with JMeter and Gatling, easily outstripping browser-level tools. It makes sense, since JMeter and Gatling don’t need to start browser instances to generate load.

It’s significantly cheaper to generate very high loads with protocol-level tools. This efficiency makes protocol-level tools a good choice for high concurrency load testing and especially stress testing.

What are we doing to support BLU?

Clearly, our usage metrics show that browser-based load testing is on the rise, and we’re planning to ramp up our offerings in this area while also maintaining support for protocol-level tools. Here’s what you can look forward to from us:

New releases
- We just released version 1.2.3 of Element a few weeks ago, which addresses some bugs and improve stability for certain actions.
Dedicated Element team
- We’ve put together a team of developers who will be focusing on improving Element exclusively. Despite Element being an open source tool, we’re putting our own resources behind it to improve it faster.
Partnering with consulting services
- It can be difficult to get started on a new tool, and Element is no exception. While we provide a load testing platform to run Element scripts, some companies want help to create the scripts in the first place. To this end, we’ve partnered with expert consultants who have already been using Element with great success: Foulk Consulting and Planit Testing.

Conclusion

When choosing a performance testing tool, it’s important to consider the advantages and disadvantages of protocol-level and browser-level testing tools. We think both types of tools deserve to be in a solid performance and load testing strategy. They can also be used in tandem in a hybrid load testing approach, improving overall application performance and scalability.

In 2018, we predicted that testing web-based applications with real browsers would increase in popularity, so we’re thrilled to see our theory validated in 2020. Browser-based load testing is increasing in usage, and our open source tool, Element, is leading the charge. It’s exciting to be at the forefront of this revolution, and we intend to increase support for load testing with real browsers even more this year.

Ask a Flooder 12: Puppeteer vs Selenium: which one is better for load testing?

Tue, 21 Apr 2020 17:36:28 +0100

Originally posted here.

When it comes to software testing, the debate of Puppeteer vs Selenium is a hot one. Nicole tackles it here to determine which one is better for load testing in particular. Spoiler: The answer is still “it depends”.

Or catch the audio version here:

Hi everyone, Nicole here again, back for another Ask a Flooder, and today, the question is: “Puppeteer vs. Selenium: which one is better for load testing?”

We’re actually getting this quite a bit now as Puppeteer grows in popularity, but first, let’s discuss the similarities. They’re tools that you can use to drive browsers. The work by identifying and interacting with elements on a website in the same way that a user would. They are also both open-source, and both can be used for load testing. This is a great strategy because instead of writing tests for an automation suite and then tests for load testing, you can just write the one test suite and re-use it, so it saves a lot of resources in the end. To answer this question, I’m going to focus on load testing because that’s what my experience is in.

Difference between Selenium and Puppeteer

Pros and cons of Selenium

Now for the difference. Selenium supports more browsers than does Puppeteer. It supports Chrome, Firefox, IE, Safari, and Opera, so there are a lot more options to choose from depending on your application.

Selenium is also what I would call automation-first. It was built with automation in mind, which means it’s very good at that. It’s very robust, and if you have a complicated test scenario, Selenium is an excellent choice.

It’s also been around for much longer than Puppeteer, which is important because that means it’s had more time to build up a community. There are way more resources and tutorials for Selenium than for Puppeteer.

It’s also, unfortunately, very resource-intensive. That’s not something that you would typically care too much about from a test automation perspective, but when you’re running thousands of users on the cloud, it IS important to keep in mind that Selenium, just because of how it’s built and its complexity, can only run about five users per node. We’ve found that out at Flood, after baselining specifically an AWS m5.xlarge instance. So you can look at the specs of that and compare it with the machines that you’re wanting to run Selenium on, to see how that would equate for you.

Pros and cons of Puppeteer

Puppeteer is a NodeJS library developed by Google, which means it plays very well with Chrome. There are a whole lot of things that you have access to with Puppeteer, and if you’re using it with Chrome, pretty much anything in Chrome, including rendering times, you can see and access with a Puppeteer script. So that’s a huge advantage.

Unfortunately, that also means it supports fewer browsers. Obviously, it supports Chrome very well, and they recently came out with support for Firefox, although they do have plans to incorporate support for more browsers in the future.

One of Puppeteer’s advantages is that we at Flood actually created a tool on top of Puppeteer. It’s also open-source. It’s called Flood Element, and Element is built to be performance-first because obviously, that’s what we had in mind when we were building it. It takes everything that’s good about Puppeteer and makes it really easy to set parameters that you normally would expect to need for load testing, like users and ramp-up.

This also means that it’s way less resource-intensive. I said that with Selenium, you can run up to five users on an AWS m5.xlarge instance, and for the same instance type, you can run, from our experiments, anywhere from 30-50 Element scripts/users. That is a huge saving when you’re running a really large-scale test.

Should you choose Selenium or Puppeteer for load testing?

I would say that if you already have a large Selenium automation suite, if you have a complex test scenario, or if you aren’t running that many users for your load test, then Selenium is a great choice. If you ARE running thousands of users, though, and resource efficiency, and therefore cost efficiency, is really important to you, then choose Element.

My advice is always to try both. If you have the time, do a proof of concept with both of them. That way, your whole team can better understand what the pros and cons are for each one and decide which one is best going forward. Or maybe you don’t even decide on one. At Flood, we support both of them at the same price, so it depends on what you prefer.

Until next time, happy flooding!

Ask a Flooder 11: What are the best JMeter plugins?

Tue, 14 Apr 2020 19:47:53 +0100

Originally posted here.

In this video, I talk about how to install the JMeter Plugin Manager as well as other plugins, what the best JMeter plugins are, and why you need them. JMeter is an open-source load testing tool that is already robust, but there are some fantastic plugins that have been developed for it that you might want to use in your next load test.

Audio version here:

Hi everyone, Nicole here, back again for another Ask a Flooder, and I hope everybody is staying alert, not anxious, during these crazy times. In today’s Ask a Flooder, our customer asks: “What are the best JMeter plugins?”

What is a JMeter plugin?

So first, I want to talk about what a JMeter plugin is and how you can actually install one. Apache JMeter is a really great, robust load testing that is one of my personal favorites. There are still some things that could be done better or could be extended, and the great thing about being an open-source tool is that anybody, pretty much, can just jump in and develop these plugins for JMeter to extend functionality, so that it supports more protocols or to customize load profiles in a way that is easier. We already have a blog post on this, and I will leave a link to that article as well in the description because it goes into a lot more detail. However, I still want to give you my personal top 3 JMeter plugins.

How do you install a JMeter plugin?

You can install a plugin for JMeter in two ways.

One is to download it from JMeter Plugins, which I will give you a link to, and then you paste that into either the lib — it’s usually in the lib folder, but it could be in the ext (extensions) folder. Each plugin is going to differ as to what you need to copy into what folder, but that’s why I prefer the second method, which is the Plugin Manager.

The Plugin Manager is a plugin itself, so you will initially need to do that copy and pasting of the jar file. Once you have the JMeter Plugin Manager installed, downloading and installing a plugin is as simple as choosing which ones you want and then clicking install. JMeter restarts, and that’s pretty much it. So I do recommend that the Plugin Manager be the first plugin that you install. But that’s still not my top 3 favorite - that’s just an optional step.

What are the top 3 JMeter plugins?

Custom Thread Groups

Now for the actual plugins. My #1 favorite is the Custom Thread Groups. See, with JMeter, you get the standard default thread group, and all that does is it asks you how many users you want to run, how many loops, the duration, and the ramp-up. That’s it. So that’s good for a lot of uses, but it doesn’t really help you in cases where maybe you want to plan a spike test, and you want to make sure that all the users have fully ramped up and gotten to a certain point, and then all hit the server at once. Or maybe you want to do a step load. Previously I’ve discussed stress testing and how a step load can be useful for that. In that case, the Thread Group doesn’t quite cut it because it only ramps up a user, and then it continues at the same level of load for the duration of the test.

So, if you want any sort of variation in the middle (increasing users or decreasing), then you’re going to want something else.The Custom Thread Groups plugin has a lot of different types of thread groups. My favorite is the Ultimate One because you can probably do almost anything on that one. It also has the Stepping, Concurrency, Arrivals, and Free-Form Arrivals, so you have quite a few options.

Throughput Shaping Timer

Plugin #2 is the Throughput Shaping Timer. I love it because not a lot of load testing tools, even the proprietary ones, actually have this. This is really handy when you have a project where you’re given a requirement that isn’t in terms of the number of users; it’s in terms of hits per second, or requests per second, or something to do with the actual throughput of the test.

The problem with that is that the throughput tends to change, so you can’t actually take as a given that it’ll remain constant for the duration of the test. Things like response time or latency are going to affect that throughput. For instance, when you have one user or one thread that’s running, and it sends a request and receives the response back in one second, you can’t necessarily say that that’s going to be the same when there are a thousand threads doing the same thing. The server’s under more load, so maybe it’ll take five seconds instead of one second— but that’s already changed your throughput. Before, one thread was doing one request per second, and now it’s doing one request every five seconds.

So it can get a little complicated, and the Throughput Shaping Timer is one of my favorites because it promises to solve all that. The Throughput Shaping Timer dynamically changes the think time, so at the beginning of a test, for instance, when there are no response time issues and requests are being returned really quickly, it might increase the think time so that it slows down your load test. So you’re not going over that throughput that you wanted to maintain. And then later on in the test, when all the users are ramped up, then it’ll decrease the think time that it uses. So it’s great because you just have this consistent throughput throughout your test. It’s a really intelligent way to test.

I do want to say, though, that you do also want to play around with this a bit and make sure you understand how this works. One page could have several requests in it, and you’ll want to take that into consideration when you’re choosing what to set in that timer.

HTTP/2 Request Sampler

My #3 is the HTTP/2 Sampler. If you don’t know what HTTP/2 is, it’s an upgraded or next-generation protocol to the normal HTTP that we already use. The HTTP/2 protocol is supposed to be faster and more efficient, so more and more web apps are switching to it because of that.

The problem is that JMeter’s HTTP request sampler is only really for HTTP sites, so the HTTP/2 sampler is an upgraded version of the HTTP sampler. It’s really easy to use. The experience is pretty much going to be the same for you, but in the background, it uses the faster HTTP/2 protocol rather than the slower one.

My colleague, Jason Rizio, actually wrote a whole blog post about this, so if you want to learn more, I will put a link in the description.

I hope that answered your question, and until next time, happy flooding!

Introducing Flood Agent: standalone infrastructure for load testing

Tue, 07 Apr 2020 19:03:42 +0100

Originally posted here.

Working from home during this pandemic and wondering how to start load tests from the office? Today, we’re officially introducing Flood Agent, a way to turn your own physical or virtual machines into load generators that you can control from anywhere with an internet connection.

TL;DR:

Installing Flood Agent on a machine lets you start load tests on it remotely from Flood—even if it’s within a corporate VPN.
Flood Agent is a simple binary that you can install on Linux, macOS, and Windows machines.
Once you’ve installed Flood Agent, your machines show up on the Flood web app to use as load generators or grids.

Boldly going where no one has gone before

Load testing within your company’s VPN and trying to coordinate it from home is like being on a maintenance pod trying to contact your starship, but a Mazarite ship is broadcasting a jamming signal, so comms are down. That Mazarite ship is your company’s firewall, and it can make testing difficult.

‍

The easiest way around this is to provision load generators in the cloud. However, because your application servers are still behind a firewall, you’d need to convince IT security to whitelist the IP addresses of the load generators. Depending on how tight security policies are, this may not be an option even in this new world of remote working.

Flood Agent is a way to start load tests on Flood from the load generators you already have. It’s a cross-platform agent that you can install on your regular load generators, allowing you to control them on Flood.

‍‍

Flood Agent is kind of like the Borg Collective. Each machine that you install Flood Agent on is assimilated into the Borg. A hive mind (you, on the Flood site) controls all Borgs, coordinating test execution with robotic efficiency.

Check out this video to follow along as I walk you through the process of setting up Flood Agent, or you can keep reading for the text version.

‍

Installing and configuring Flood Agent

Flood Agent is a free binary, and it runs on Linux, Mac, or Windows machines. You can download the appropriate version for your operating system and then follow the installation instructions here for more information:

Getting started on Linux

Getting started on macOS

Getting started on Windows

After installing Flood Agent, you can configure it according to how you want to use it. Start Flood Agent in your terminal and use the configure functionality. Here’s what that command looks like on macOS, but you can follow the links above for other operating systems:

./flood-agent configure

Linking your Flood account

The command above yields a prompt for a token like this:

This token allows Flood to run load tests on the load generator that you’ve installed Flood Agent in. To get the token, go to the API Access page on your Flood account (you already have one, don’t you?) and click REVEAL token.

You should see a string of characters like this:

‍

Copy your token (anything after the = sign) and paste it into the Flood Agent terminal window.

Choosing load testing tools

Next, you can choose which tools you’d like to set up and which version of those tools Flood Agent uses to run your tests.

‍

Starting with JMeter, Flood Agent asks you which of the tools you’d like to configure. You can choose to set up your tools at the beginning so that you don’t have to set them up again, or you can go back to this configuration wizard at any point to add another tool. Either way, you need to provide the tools that you select in one of two ways: via a Docker image supplied by Flood, or via a local installation that you already have.

For this example, I chose to configure JMeter and to use a local installation. Local installations give you the most flexibility because you can use the version of the tool that you prefer, along with any special plugins. However, if you already have Docker installed or don’t want to manually install JMeter separately, you can also use our Docker image for JMeter.

Selecting a local installation prompts you to provide the location of that directory:

‍

For JMeter, this looks something like /Users/nvanderhoeven/jmeter/apache-jmeter-5.1.1.

After you enter this file path, Flood Agent checks to make sure the directory is valid. You can also set the configuration for other tools in this dialog.

After you finish the configuration, the wizard writes to a config.yaml file in your Flood Agent directory. You can also edit this configuration file directly in the future if you prefer.

Verifying Flood Agent configuration

To run a final check on whether Flood Agent has been installed correctly, type ./flood-agent check. You should see a summary of the configuration options you selected as well as some connectivity and validity checks, like this:

‍

These green check marks mean you’re good to go! If you’re running Flood Agent from within a corporate network, though, you may see some errors here.

Troubleshooting network connectivity within a corporate network

If your company already has a web proxy, check out our guide on running floods through this proxy.

Please ensure that the load generator that you’ve installed Flood Agent on has access to the following endpoints:

<https://drain.flood.io>
<https://beacon.flood.io>
<https://vault.flood.io>
<https://flood-archives.s3-accelerate.amazonaws.com>
<https://logs.us-east-1.amazonaws.com>
<https://sns>.*.amazonaws.com
<https://sqs>.*.amazonaws.com

You may need to request these from your IT department.

If you get stuck, please contact us at support@flood.io. We’d love to go over your specific situation and get you up and running.

Once Flood Agent returns a successful check, you can start up your own grid!

Turning your machine into a Flood grid

A grid, in Flood terminology, is a group of machines to be used for load generation. To make your machine a grid, run this command:

./flood-agent --grid askaflooder

where askaflooder is the name of your grid.

‍

Once you see those [agent] ready and awaiting next job messages, you’re ready to run your test!

Running a load test on your own infrastructure

Log into your account on Flood, if you aren’t already, and create a new stream like you normally would, uploading your load testing script and data files. When launching your test, though, select the option for Hosted infrastructure and then choose the grid with Flood Agent running on it.

‍

Go ahead and launch your test. Congratulations, you’re now set up to run a load test on your on-premise load generators!

Giving it a go

Flood Agent is a free download, so you’ll only need to pay when you run load tests on Flood. However, we also have a free tier— perfect for a proof-of-concept while you’re working from home.

If you have any questions about Flood Agent, check out our documentation or email us. We’d be happy to get you started!

Download Flood Agent for free here.

‍

Ask a Flooder 10: What is the difference between load testing and stress testing?

Mon, 06 Apr 2020 20:26:24 +0100

Originally posted here.

What’s the difference between load testing and stress testing? When should you choose to run a load test on your application, and when is a stress test more appropriate? Well, it’s all in the intention.

‍

Hi everyone, Nicole here again, back with another Ask a Flooder, and this time the question is:

What is the difference between load testing and stress testing?

Okay, again, this is a matter of semantics, but I think the difference is in the intention. Load testing and stress testing are both types of back-end performance testing, and I did also make a video on it that I’ll link in the description down below.

Load testing is about seeing how your server reacts to finite load

Now, with load testing, you usually have to start with a specific idea of the load profile that you want to test. You get to this number by looking through your Google Analytics or other analytics on your site to see what load you need to test at, and then the intention is to see how your server responds to that load. So the load is fixed: you know WHAT you want to test, and you just want to see how the server reacts to it. So it’s more finite in scope. In general, it’s also more about how your server reacts to normal or expected load, whether that’s now or in the future.

Stress testing explores your application’s upper limits

Stress testing, on the other hand, is a little bit more open-ended and a little bit more destructive. So instead of starting with how much load you want to put on [your server], in stress testing, what you generally do is you increase the load. You usually use a step load profile: so you start a certain number of users, you wait maybe 30 minutes, and then you increase it again. And you end up with this staircase kind of load profile. What you’re doing there is you just let it increase until your server fails because the intention of stress testing is to get your server to fail. And, once it fails, does it fail gracefully? Does it show an error message? These are really things that test the upper limits of your server, and that’s what stress testing really is.

They can be interchangeable

At Flood, we tend to use the word “load” testing to refer to both kinds of testing because both of them entail applying load on your application server. It’s not really that important to distinguish between one or another; what’s [important] is that your team uses the same definition so that when YOU say “load testing,” they know what you mean.

Until the next time, happy flooding!

Resources

Video on performance testing vs load testing

Real-life stress testing in action: Robinhood app failure

Ask a Flooder 09: What are the best tools for load testing?

Tue, 31 Mar 2020 20:35:36 +0100

Originally posted here.

One of our most commonly-asked questions is: “What are the best tools for load testing?” The answer will depend on a few things: your reason for load testing, the application you want to test, what kind of load testing you want to carry out, your team’s experience with scripting, and how many virtual users you want to run.

Note: In this video, I say that traditional load testing tools can’t simulate requests that are triggered by interactions on the client-side. This isn’t technically true: if you know what the requests are, you can build those requests manually on a protocol-level tool (like JMeter or Gatling). However, this takes more work compared to browser-level tools that just do this natively. ‍

Transcript:

Hi everyone, Nicole back again with another Ask a Flooder, and this time I’m tackling one of the questions we get asked A LOT, which is:

What are the best tools for load testing?

Why open-source load testing tools are awesome

Now, I want to preface this by saying that we really love open-source tools here at Flood, because 1) they’re really pretty full-fledged these days. You don’t have to make any compromises on features, and, in fact, some of these open-source tools have more features than some newer load testing tools just because they’ve been out longer. And secondly, they have large communities. Because they’re open-source, lots of people are using it, and lots of people are developing for it. So there are lots of plugins normally, and they support any kind of protocol that you can think of. And lastly, cost. It’s hard not to mention cost because it can be a really important factor depending on the budget of your project. And if you’re starting out as a load tester, I think that it might even be the best way to start with load testing, because you can take that knowledge with you and apply it to other projects and other companies regardless of the budget that they have.

Questions to ask yourself in choosing a load testing tool

Why are you doing load testing?

I always advocate starting with WHY. Why are you actually load testing? Is there a particular production defect that you noticed that you’re trying to solve? Is there already a problem, or are you just preparing to launch? These are things that are going to inform the type of tool that you use because you might want to make sure that that tool supports the protocols that you need.

What are you load testing?

Secondly, what type of app are you testing? Because if you have a single-page app, where your end-user is interacting with the page and client-side scripts are running to make requests to servers, then that’s something that most traditional load testing tools aren’t going to be able to do.

There are two general types of load testing tools: there’s browser-based, and there’s protocol-based. I’ll leave a link in the description talking about the differences between the two, but in general, if you want to go for browser-level testing, I would suggest Selenium and Element. Now, they’re both very good options. Selenium is probably the more robust in terms of automation of the two, but Element is better one in terms of efficiency because it was built with performance in mind. So Selenium’s automation-first and Element is really performance-first.

Now, for protocol-level tools, you can’t go wrong with either JMeter or Gatling.

What experience does your team have with scripting?

Thirdly, what’s the level of technical expertise that your team has? I think it’s important to be realistic about what your team can do, given the amount of time that they have. So if you have a team that has never done load testing and has never done any scripting before, then I would suggest using our Test Builder or something similar. On Flood, we have a Test Builder that is really just an easy interface that actually runs JMeter under the hood, but it just kind of simplifies JMeter and boils it down to what are the URLs that you want to test. That is a really easy way to get started, and it’s also a great [way to run a] proof of concept. If you have some automation testers on your team that are trying to do the load testing, well, maybe you should stick to tools that they already use. So it might be easier for them to get up and running with either Selenium or Element rather than learning a new tool. And if you have developers on your team, you might consider Ruby JMeter rather than the vanilla JMeter because Ruby JMeter is a lot easier to deal with— because you don’t have to deal with the XML files of a JMeter file (a normal JMX script). You can just plan out your load tests in Ruby. Or, Gatling is also a really good option because it allows you to write scripts in Scala. Scala is really really powerful, and you can create an entire framework with Gatling, which is also something to keep in mind.

How many virtual users do you want to run in your load tests?

And lastly, how many users do you want to simulate? Now protocol-level tools are going to really shine here because Element and Selenium both require a higher usage of resources on your load generators just because of the complexities involved in starting a browser instance, whereas protocol-level tools are just super efficient at sending these requests. So I would suggest that if you’re talking about thousands of users, I would use the protocol-level tools unless there’s a really good reason why that’s not going to give you the results that you’re looking for. Between JMeter and Gatling– well, they’re both excellent tools, but Gatling has been shown to particularly be good for really high levels of usage. But JMeter is also a good alternative, and you can run thousands of users of JMeter without any problem.

Just get started!

As always, I’d suggest just getting started. Pick whatever tool sounds like it might be the best for you, do a proof of concept on that one, and you can always try the other ones as well. On Flood, we don’t restrict you to just using one tool, so you can use one; you can use all four; you can run them at the same time if you want. It all depends on what you need for your testing.

Till next time, happy flooding!

Ask a Flooder 08: Why is concurrent users an ambiguous measure of throughput?

Thu, 26 Mar 2020 20:41:22 +0100

Originally posted here.

What is the meaning of concurrent users in performance testing? You might have seen concurrent users in JMeter or other tools, but what does it mean and how can you use the term properly?

‍

Transcript:

Hi everyone! Nicole here again for another Ask a Flooder, except this time it’s not really a question– it’s just something I want to talk about.

Source: The Flood Twitter account

So I’m going to put the tweet up here, but last month I actually tweeted about something that somebody else posted. Netflix actually reported some of their viewership figures, and one thing that they reported was that 76 million households watched their new show, The Witcher, which is a large amount of people. So I posted something about why concurrent users is actually an ambiguous way to measure throughput, and I want to expound a little bit more on that.

“Concurrent users” in performance testing - limitations

In load testing, the number of concurrent users is a pretty standard measure of throughput, and generally, it means the number of users that are accessing an application at any one point in time. And even on Flood, we use it because it’s a really good rule of thumb. When we want to get a feel for how big a test is, load testers generally ask, “How many users?” However, we should be aware, as load testers, that that doesn’t always give us the full picture.

What type of user is it?

For instance, the type of user that it is can really change the traffic. Sometimes mobile browsers will optimize requests so that they send fewer because they realize that mobile networks are slower.

What is the user doing?

Secondly, what is the user doing? Are they browsing on different tabs all the time? Or are they just refreshing the content on the same tab? Because if they’re just using the same page and fetching it over and over again, then caching could come into play, so that would create a different load on the system.

How long is the user doing it?

And thirdly, how long is the user doing it? Are we talking about a sustained load for an hour, or is it a shorter time period, like a spike test of five minutes where all the users just ram a system? There are things like the number of requests per second that you need to take into account, which is affected by the think time and the delays that you set in your script. Perhaps your script has a user that just sends a request, then waits for a minute, and then sends another request. That’s very different from a user just sending one request after another as fast as possible.

When does the user do it?

Lastly, it’s also important to take into account the seasonality of the request. When does the user do it? If you have a timesheeting application, for instance, then it might only get traffic on Fridays and a little bit on Monday, so you need to specify, when you report these numbers, whether it’s for the peak time period on Friday - you know, around 3 pm or something - or you’re reporting on figures from a Wednesday when there really wouldn’t be that many people accessing it.

In the case of Netflix, what they counted as a view was only a two-minute watch time. So if you’re one of those people who went into The Witcher and didn’t watch past two minutes, that still counted as a view. So the 76 million users is still not completely accurate, because not all of those people would have watched the entire thing.

Conclusion

Concurrent users is a good way in order to get a quick feel of the throughput that a test outputs, but it really needs to be examined in context along with things like requests per second and the network throughput (kbps) in order to get a really good understanding of how much load a test is generating.

Till next time, happy flooding!

Ask a Flooder 07: What is the difference between performance testing and load testing?

Tue, 24 Mar 2020 20:31:06 +0100

Originally posted here.

In a two-for-one Ask a Flooder, I answer two questions: What is the difference between performance testing and load testing and in which situations is performance testing a better option than load testing?

Transcript:

Hi again, it’s Nicole back for another Ask a Flooder. And this time, the question from one of our customers is:

What is the difference between performance testing and load testing?

Now, this is really a matter of semantics, and it kind of depends on whom you ask, but in general, performance is how quickly an application responds. So when you’re doing performance testing, you’re doing one of two things: there’s front-end performance, and then there’s back-end performance.

In front-end performance, you’re looking at things like how quickly does the application render for the customer on the client-side. It’s how fast something renders on the screen, so it’s how page resources can be rearranged or optimized so that the experience is a lot faster for the end customer. So the focus is on the client-side on the end user’s actual device.

On the back-end performance side, you’re looking at the server. How quickly does the server send a response to a request that’s sent by the browser?

So those are really two different things because back-end performance really looks at everything outside - everything beyond - the device of the customer. Both kinds of performance testing are important. A solid, holistic performance testing strategy will incorporate both front-end and back-end performance.

There are different types of back-end performance testing. You might have heard of load testing, stress testing, volume testing. At Flood, we kind of refer to all of those under the bracket of load testing. They do have some semantic differences between them, but in general, we talk about load testing to refer to simulating traffic against an application server to see how that server responds.

Now the type of the testing that you want to do will also affect the kind of performance testing tool that you choose. For instance, a tool like Flood Element can do both front-end and back-end testing, but a tool like JMeter is only going to be useful for the back-end performance testing. So you need to choose your test according to the testing type that you’d like to carry out.

This customer also had a follow-up question:

In which situations is performance testing a better option than load testing?

And one answer to this is when front-end performance is required. So if what you’re after is trying to speed up how quickly your application renders on a mobile device, for instance, you might be better off looking at tools like GTMetrix or WebPageTest in order to measure and improve that side of it.

I really want to stress here that neither front-end performance nor back-end performance is better than the other. If you’re asking me which type of testing you should do, my answer would be both. Till next time, happy flooding!

Ask a Flooder 06: How to record traffic from a mobile app with JMeter

Thu, 19 Mar 2020 20:50:44 +0100

Originally posted here.

How do you record the traffic from your native mobile app using Apache JMeter? In this video, I explain step-by-step how to begin load testing a native mobile app.

‍

You might want to be doing this because you’re load testing a mobile app but either you don’t have access to your developers right now so they can’t help you with how the requests are made or what the requests are, or maybe you just want to verify what you’ve already been told. Either way, I’m going to show you how using my Android device (it’s a Samsung Galaxy Note 10+) and a Macbook Pro 13". However, the same principles should apply to other mobile devices and laptops.

Step 1: Download and install the mobile app

The first thing you’ll want to do is download and install the app. Now, if the app has already been released, then that’s no problem. You just go to the Google Play Store and download it. However, if it hasn’t been released yet and you’ve been given an APK file for the app, then you’ll have to tweak your settings a little bit in order to get your phone to allow you to download it.

And I’m going to do this right with you. So go into Settings: pull down the little notification screen and hit the little gear icon, and you’ll see a bunch of settings there. Scroll down all the way to the bottom until you see About phone. Click that and then click Software Information. You’ll see the Build number halfway through the screen here– and I know this is weird, but click on it seven times. You have to draw your pattern if you have one set up, but you’ll see here that it says Developer Mode has been enabled. That’s all you need to do; from now on you can install any APK, but be careful, because if you don’t trust the source, you know, they could do some damage to your phone.

Step 2: Set up the HTTP(S) Test Script Recorder on JMeter

The second step is setting up an HTTP recorder on JMeter. So let’s head over to our laptop. Make sure you’re connected to a wifi network. Now, this is a completely new JMeter test plan here. So just right-click on Test Plan and then hover over Add, and then you’re going to go down to Non-Test Elements. Select the HTTP(S) Test Script Recorder. You’ll see here that the port it uses is 8888. We’ll need that for later. Now right-click on the Test Script Recorder, hit Add, and then Listener. We’ll add a View Results Tree to be able to capture the results that we need later. Now we’ll click on the Test Plan - right-click - and then we’re going to Add a Thread Group. Now let’s right-click on the Thread Group and go to Add > Logic Controller > Recording Controller. This is where the requests that we’re going to send are going to be recorded, and we’ll see both the requests and the responses in View Results Tree.

Step 3: Set up your mobile to use the HTTP(S) Test Script Recorder as a proxy

Now, the third step is to set up your mobile to use a proxy. So, back on the mobile, go back to Settings and hit Connections. And then click on Wifi. Now verify that the wifi network that you’re connected to on your mobile is the same one as the wifi network that you’re connected to on your laptop. If it’s the same, click the gear icon. And then hit Advanced all the way at the bottom. Now the Proxy settings are on the second option here so hit the down menu here and then hit Manual. Now, for the Proxy host name we’ll have to go back to the computer to see what our local IP address is.

So, in order to do that, you just have to go to System Preferences. I’m doing it from the Apple menu here and from System Preferences, click on Network. And once you’ve selected the wifi network, hit Advanced, and then TCP/IP. So what you’re looking for will be this IPv4 Address. Now you’re going to type that into the Host name field on your mobile. So let me just type that in.

You’ll note that this is actually a local IP address. It’s not your public IP address. Now for our proxy port, we’ll go and look at the laptop to see the Test Script Recorder Settings and it was 8888, so that’s what we’re going to put in here. All right. And then click on Save.

Step 4: Start recording

Step 4 is recording the traffic. Now, go back to your laptop on JMeter and click on the Test Script Recorder. Now we’re going to click on Start. It’ll come up with this popup box here and you can just click OK since we already trust the source. You’ll see this little popup here to confirm that we are recording traffic. Now let’s see if we’ve actually configured it correctly.

On our mobiles, let’s go back and open the app that you want to use. So I’m going to be using the IowaReporterApp. I just did an article on this on The New Stack (link in the description below) about how I would load test the Iowa Caucus Reporter App. So that’s what I’m going to be using for this tutorial as well. So I’m going to open that (“getting ready to caucus”) and I don’t have a valid precinct ID so I’m just going to put flood.io and I’m going to click Login. So it’s telling me that the connection isn’t private. I don’t actually expect this to work, because the app has been taken down and I don’t have valid login credentials, but I’m still just going to go forward. So, Proceed. You might have to do this as well if your certificates haven’t been set up correctly.

Okay, so now we’ve gotten this error here and it says, “Oops, something went wrong”. So it’s handy to know what it looks like on your mobile so that you can go into JMeter and see if that’s what you see as well.

So on JMeter let’s go to the View Results Tree and it looks like Google Play tried to do a few things beforehand but what we’re really looking for is this IDP caucus thing. So, these are the requests that my phone made and these are the responses. So this is where we got the certificate error because it says “certificate unknown”. Now this is the actual request we sent for login. You can see the request here - the full request - and also the full response. Now, just to check we’ll make sure we got the same response in the body here on JMeter that we saw on our mobile phones.

And here it is; it says “Oops!, something went wrong”. You can of course just search for this. So we’ll stop the recording now because we’ve confirmed that we’ve gotten what we wanted. So in the recording controller, we can see the requests that were recorded from our mobile app and the first one looks like it was one to Google so that’s not one we want. If we were doing this again, we could actually go into the Test Script Recorder, go into this Requests Filtering tab, and we could just filter out or exclude the domains that we don’t want to test, like Google. But this looks like the request that we were actually looking for. You’ll see the full request here with all the parameters and the values.

What’s next?

And that’s how you record traffic from a native mobile app on JMeter. From here, you can then play around with the request. Like we saw, there were a lot of dynamic values that were recorded in the request. We would probably want to correlate those and make sure that we’re doing those correctly. But this is a really great starting point for testing a mobile app.

Till the next Ask a Flooder, happy Flooding!

Ask a Flooder 05: How do I get started with Flood Element?

Tue, 17 Mar 2020 20:58:12 +0100

Originally posted here.

On this Ask a Flooder, I talk about getting started with Flood Element. Element is an open-source tool that is a good Selenium alternative for browser-based load testing. It runs Puppeteer under the hood to drive real browsers at scale and can be paired with Flood for cloud load testing.

‍

TRANSCRIPT:

If you’re sold on browser-based load testing and want to get started on Element, I think the easiest way to do that is by going to element.flood.io and clicking on Install on the upper left corner. You can follow those instructions to get started, and you can also click on the links on the left side if you get stuck.

Once Element is installed, open up the directory and look at the examples folder. There you’ll find a lot of sample scripts that we’ve already made up for you. The easiest way to get started is to just modify some of these slightly so that you’re testing your site. For example, you could just change the URL and step names to make it appropriate for what you’re testing.

Then, you can run it locally using the command element run the name of your test script, like test.ts, and then I also like to include the no-headless flag.

element run test.ts --no-headless

The no-headless flag is going to run a real browser, a Chromium browser, on your machine, so that you can see what the script is actually doing. This is a great way to debug and troubleshoot.

Once you’re ready to run your load test on the cloud, you can simply drag and drop your Element script into the Flood stream editor and then start your load test as usual.

Check out the links in the description below for more information about Element, how to get started, and how to run an Element script on Flood.

More about Element

Getting started with Element

Ask a Flooder 04: Why would your load testing tool report higher response times than you expect?

Wed, 11 Mar 2020 21:02:48 +0100

Originally posted here.

In this Ask a Flooder, I tackle a question we get all the time from users regarding results, particularly for cloud load testing. Sometimes you might notice that response times that are reported in by your load testing tool do not match up with the response time you see from your own browser. What causes this, and how can we fix it?

Transcript:

It’s always confusing when your load testing tool reports higher average response times than what you can see while you’re manually accessing your application on your browser. There are many things that might cause this, but one of the most common causes for it is that you’re overloading your node generators.

As a general rule of thumb, you’ll want to keep your CPU utilization at 80% or less throughout the duration of the test. It is normal for it to spike in the beginning, but it should ideally come to a more stable spot below 80%. If your node’s CPU utilization is going over 80% for most of the test, then what that means is that your node is struggling to get those requests out, which may lead to some inaccurate results.

You can confirm whether or not this is the cause of your high response times by checking the logs and checking the resource utilization on your nodes.If you’re running on Flood, you can do this while the flood is running by going over to Grids, selecting the grid, and then scrolling down on the panel on the right to see the CPU and memory as well as the network throughput.

There are a couple of things you can do to fix this. The first one is to increase any sort of delays or think times in your script — and, if you don’t already have any, you should add some. Think times make the script a lot more realistic and a lot closer to what real users would do. If these values are too low, then your load testing tool is just going to run the script as fast as possible, consuming more resources than necessary.

The second thing you can do is reduce the number of users you’re running per node. This will free up more resources per node and make it a lot easier to scale up your load tests more effectively.

Check out this article for more detailed information on resource over-utilization and other possible reasons for differences between response times in a test and on a browser.

FOMO and performance testing: Why Robinhood went down

Mon, 09 Mar 2020 20:29:40 +0100

Originally posted here.

Angry customers flocked to the internet when the Robinhood investment brokerage app went down last Monday and part of Tuesday, leaving them unable to trade stocks on a day where Wall Street reported historic gains. Could this have been prevented?

The answer is yes. Applications should always be assessed for risk and tested accordingly. Risk-based load testing helps unearth problems that would otherwise only surface in production. Testing applications end-to-end as a system instead of just as a collection of isolated components allows us to build software that is more resilient. The problem is that this type of testing can be difficult to carry out. In this article, I’ll go through the entire process.

What is Robinhood?

Robinhood is an app I’ve personally had my eye on since it opened a waiting list about five years ago for Australians wanting to see the American company come to our shores. It doesn’t look like it’s happening now, but if that waiting list still exists, I’m on it.

Robinhood is a brokerage app that takes inspiration from the legend it was named after to “give to the poor,” bring zero-commission trading to its customers. Instead of earning from commissions, Robinhood makes its money from a variety of other methods, including premium platform fees, interest on customers’ uninvested capital, and payments received in exchange for coursing customers’ orders through third-party market-makers. It’s an enticing premise, and one that the company has used in conjunction with mobile apps to appeal to millennials and technophiles.

Source: From Robinhood’s homepage ‍

What caused the Robinhood app to fail?

Unfortunately, its wild success with the tech-savvy crowd also meant that when it went down last Monday morning in the US, its customers took to the internet in droves to draw attention to the failure. In a blog post from Robinhood, the company’s co-founders, Baiju Bhatt and Vladimir Tenev, admitted that the outage was due to “stress on [their] infrastructure— which struggled with unprecedented load.”

The outages you have experienced over the last two days are not acceptable and we want to share an update on the current situation. Our team has spent the last two days evaluating and addressing this issue. We worked as quickly as possible to restore service, but it took us a while. Too long. - Baiju Bhatt and Bladimir Tenev

Bhatt and Tenev went on to explain that the load on their servers had caused a “thundering herd” effect. This effect describes a situation in which a backend server receives a large number of concurrent requests and, instead of different threads processing these requests simultaneously, all threads attempt to process the same request.

If you’ve ever played four-player Overcooked, it’s like having everyone sprint to a fire extinguisher to try to put out a fire from overcooked pasta, but instead, the constant button mashing means you just pass the extinguisher around and yell at each other while the fire rages on. Before you know it, the entire kitchen is on fire.

Source: From Twitter user @csac0425

So the technical cause of the outage was an infrastructure problem.

But I think the REAL cause was FOMO.

What is FOMO?

FOMO is the Fear Of Missing Out, and it’s a phenomenon that’s been particularly exacerbated by social media. Instead of only hearing news from people you interact with in the physical world, it just takes a few seconds to open up Twitter on your smartphone and see what thousands of people are talking about. Twitter will even helpfully tell you what’s “trending” in your country, and other social networks also employ algorithms to determine what messages to show you.

How does FOMO affect application performance?

We can see FOMO in how people have reacted to the coronavirus COVID-19. Despite pleas to leave masks for medical professionals at real risk, people have bought out the stock for these masks despite living in countries with little to no sign of the coronavirus. In Sydney, Australia, residents are stockpiling toilet paper (of all things) in large quantities, clearing out shelves from supermarkets, and getting into knife fights over toilet paper due to their panic at the thought of doing without.

This photo of an empty toilet paper aisle was taken by our Principal Engineer, Lachie Cox, in a supermarket in Sydney last week. ‍ FOMO of this magnitude ultimately also manifests itself in the stock market. The Dow Jones Industrial Average, which represents the performance of the top 30 largest companies in the US, dropped by 3,500 points last week amidst fears that the coronavirus would stall production in countries most affected by COVID-19, leading to lower returns across the board.

Then, an abrupt about-face. The Dow went from recording its biggest ever loss in a single day on Thursday to posting a record gain of 1,200 points on Monday.

Guess who gets to adjust to all this volatility.

Robinhood’s founders reported that the “highly volatile and historic market conditions; record volume; and record sign-ups” contributed to the infrastructure issues they faced. We can only imagine just how many users they saw on their systems during this period.

FOMO creates mass hysteria and panic, both of which cause customers to behave irrationally and likely very differently than previously recorded. So how do you create a workload model that accounts for this volatility? How do you tailor your load tests for FOMO? I’ll run you through the process, using Robinhood as an example.

Does your application require FOMO load testing?

All testing should be risk-based. Start with why: why should you test an application component? Why does your application require this kind of testing or that one?

In Robinhood’s case, there were a few indicators that pointed to the necessity of FOMO testing:

A customer base made up of a younger, tech-savvy generation. ‍This demographic is very likely to take to social media with complaints, making negative publicity a real risk.
Complexity. ‍Robinhood’s systems must receive customers’ orders, forward and place those orders to market makers, and display real-time market data. Robinhood customers can also purchase options on stocks as well as cryptocurrency, further increasing the complexity.
Financial component. ‍Anything that involves taking people’s money should have a high priority for any testing.

When it comes to your money, we know how important it is for you to have answers. - Baiju Bhatt and Bladimir Tenev, co-founders of Robinhood

Record-breaking growth. ‍Robinhood had 1 million users in 2016, 6 million in 2018, and 10 million last December, according to CNBC. Exponential growth brings some major growing pains.
Industry history of expensive failures. ‍Quartz points out that software or infrastructure failures have cost the financial industry millions in recent years.
Recent legal action. The regulatory body FINRA fined Robinhood US$1.25 million just last December for placing orders for customers without looking for the lowest price. While this isn’t a huge fine for a company that was valued at US$7.6 billion last July, any legal action should make a company tread more carefully.

These are solid reasons to warrant load testing Robinhood’s systems beyond just “expected” load.

Planning for FOMO

Once you’ve determined that your application is susceptible to FOMO, the next thing to do is plan how to structure your load tests to include its effect in your test runs.

What type of app is it?

Robinhood consists of three apps: two for mobile (iOS and Android) and one for the web. It is currently only available to US residents, which means that getting mobile apps downloaded is problematic as it would entail changing countries in the app stores to get access. Since I live in the Netherlands and I don’t want to do that, I’ll focus on the web app specifically when scripting the load test, and I’ll assume that the same underlying application servers service both web and mobile apps.

Workload modeling

Business processes to test

Given that the app went down on a day where markets were up, we can assume that most of the people on Robinhood’s apps were doing one of two things:

Signing up
Buying shares

We should ideally test both, but let’s focus on signups here because it’s easier to test without access to the actual app (which requires a US social security number).

Number of virtual users at peak load

Let’s try to come up with a number of users that we can use as Robinhood’s peak hourly load. This task would be easier if Robinhood shared their analytics, but we can still make some educated guesses with what has been made public.

Source: Image from Apptopia

According to Apptopia, Robinhood’s highest number of mobile installations was in 2018, when it saw 2 million downloads of its mobile app in a quarter. This period coincides with when Robinhood released cryptocurrency trading support, which looks like increased signups—in the third quarter of 2019, this figure was about 1.5 million. 2 million quarterly downloads translates to about 666,666 downloads per month. Let’s call that 700,000 and take this number of mobile downloads as a starting figure.

The stock market is only open on the weekdays, so we can divide 700,000 signups a month by 20 days, and we get 35,000 signups a day. The markets are also only open for about 7 hours a day, but the load probably isn’t evenly spread out across the 7 hours— I would expect that people would have more time during lunch to sign up for Robinhood. So let’s say that the majority of those signups would happen within three hours (from 11 am to 2 pm, for example). By dividing 35,000 signups per day by 3 hours, we get about 11,667 users per hour. Let’s round that up to 12,000.

How do we translate this to the number of virtual users we need to run? Let’s think about how long each user stays on the app. The signup process does require approval, so that’s a hard stop— new users won’t be able to sign up and then start trading immediately afterwards. This approval could take up to a day, or up to seven days if documents are required.

Looking at the application form itself, we can see it is relatively standard and requires only information that most people are likely to have handy (social security number and contact details). I timed how long it took me to go through the part of it I could access, and I estimated that 10 minutes would be sufficient time to go through all the tabs.

Source: Robinhood

We want to spread out users evenly across the hour— that is, we don’t want 12,000 users to sign up and leave, all within 10 minutes. We want equal portions of those users to be signing up throughout the entire hour. Dividing 12,000 by 6 (the number of 10-minute intervals in an hour), we get 2,000. 2,000 is the number of threads we need active at any one point of time. Each of those threads signs up one user for 10 minutes, then signs up 5 more users until the hour is up.

So we have our figure: we need 2,000 virtual users per hour to simulate 12,000 signups in an hour.

Test scenarios

In addition to running the standard battery of load tests, testing for FOMO requires exploring the upper limits of the application. Assuming the peak load testing goes well, we can use the 2,000 virtual users per hour figure from our peak load calculations as a starting point for these more destructive tests. To this end, here are some test scenarios I’d consider running if I were testing the Robinhood app:

Soak testing. ‍Soak testing means running typically a little less than the peak load profile, but over a longer amount of time. In Robinhood’s case, perhaps we could run 1,000 virtual users for 8 hours. We would expect that the response times reported during this test would be the same; otherwise, there is likely a performance bottleneck, such as a memory leak.
Stress testing. ‍Stress testing means increasing the number of users on an application at regular increments until the application crashes. We can start with 2,000 virtual users per hour, and then add 500 users every 30 minutes to see how the application handles it. This test helps determine how much room there is for growth.

Source: Stepped load profile for stress testing, generated in JMeter ‍

Resilience testing. ‍Resilience testing involves running the peak load test and then simulating an outage by turning off key components to see how the application behaves. If two servers share the load, for instance, we could turn one off and check to make sure that the user sessions on the terminated node are moved over onto the remaining node. This test shows whether the application recovers gracefully from an unforeseen event.

All these tests help us prepare for the unexpected, and they might have identified the “thundering herd” problem that contributed to Robinhood’s outage.

Scripting for FOMO

Tool selection

To test mobile as well as web apps, I would typically use a protocol-level load testing tool like JMeter to simulate the load. However, I’ve already done that. I also noticed that Robinhood makes use of some dynamic parameters, such as a csrf token:

‍

Source: Chrome Developer Tools on Robinhood ‍ We can usually find the value of this token in the response of the previous page. It is a security feature to allow Robinhood to verify that the same users make all requests in the same session. Tokens can be scripted around in a tool like JMeter, but it can be time-consuming. So for this article, I decided to show another tool that makes this problem go away. I used Flood Element, which is an open-source tool we created based on Puppeteer. One of its advantages is that it runs on the browser level. Instead of diving into HTTP requests and tokens, I just told Element which buttons to click.

I wanted a script that would do the following things:

Navigate to Robinhood’s home page.
Click on “Sign Up”.
Click on the “Get Your Free Stock” button.
Fill out the signup form.

Note that I stopped short of actually submitting the form because I don’t want to create dummy accounts on Robinhood’s database— I just want to show how this flow can be scripted with Element.

To help me identify the elements on the page that I wanted the script to interact with, I used Chrome’s built-in Developer Tools. Specifically, I used the Inspect Element button (in orange below) to click on a field and find where it was in the code.

‍

For instance, to find out how to identify the email field, I right-clicked on the code on the right, hovered over Copy, and then clicked Copy selector.

‍

This gave me the value #signup_email_field, which I then used in the script like this:

//Type Email address
await browser.type(By.id('signup_email_field'),'nicole@flood.io')

Here’s the script that I ended up with:

import { step, TestSettings, Until, By, MouseButtons, Device, Driver, ENV } from '@flood/element'
import * as assert from 'assert'
export const settings: TestSettings = {
 loopCount: -1,
 clearCache: true,
 disableCache: true,
 actionDelay: 8,
 stepDelay: 10,
 screenshotOnFailure: true,
 userAgent: 'flood-element-test',
}
export default () => {
 step('01_Home', async browser => {
 //Navigate to Robinhood homepage
 await browser.visit('<https://robinhood.com>')

 //Validate text
 let validation = By.visibleText('It’s Time to Do Money')
 await browser.wait(Until.elementIsVisible(validation))
 await browser.takeScreenshot()
 })

 step('02_Signup', async browser => {
 //Click "Sign Up"
 let signupBtn = await browser.findElement(By.xpath('//a[@href="<https://signup.robinhood.com>"]'))
 await signupBtn.click()

 //Validate text
 let validation = By.visibleText('Free Stock Waiting For You')
 await browser.wait(Until.elementIsVisible(validation))
 await browser.takeScreenshot()
 })

 step('03_GetFreeStock', async browser => {
 //Click Get Your Free Stock
 let freestockBtn = await browser.findElement(By.xpath('//a[@href="<https://signup.robinhood.com/?">]'))
 await freestockBtn.click()

 //Validate text
 let validation = By.visibleText('Make Your Money Move')
 await browser.wait(Until.elementIsVisible(validation))
 await browser.takeScreenshot()
 })

 step('04_FillForm', async browser => {
 //Type First name
 await browser.type(By.xpath('//input[@name="first_name"]'),'Nicole')

 //Type Last name
 await browser.type(By.xpath('//input[@name="last_name"]'),'van der Hoeven')

 //Type Email address
 await browser.type(By.id('signup_email_field'),'nicole@flood.io')

 //Type password
 await browser.type(By.xpath('//input[@name="password"]'),'demo123456')


 await browser.takeScreenshot()
 })
}

In every step, I also added a takeScreenshot() , which is one of my very favorite features of Element. It’s so useful to be able to save these screenshots and use them to determine exactly what’s happening in the script. Here’s what it looks like running my Element script locally. The automated browser is on the left, and my terminal (iTerm) is on the right, running Element.

‍

Running a FOMO test

Now we’ve got a script that runs locally. How do we run it at scale? Additionally, Robinhood won’t allow users outside the US to sign up. How do we make sure that our virtual users are allowed to sign up?

The answer is a load testing platform like Flood.

With Flood, running our script is a matter of uploading the script and selecting some options.

The Flood test design interface ‍ Flood also lets us choose which regions to generate load from. Below I’ve got it set up to run for an hour from four different US cities, with each one starting 500 users. This configuration gives us our 2,000 virtual user figure.

‍

If this peak load test goes well, we could then go on to run the other test scenarios that we described earlier. Each one puts pressure on application servers in different ways, identifying performance bottlenecks.

Results and reruns

Since Robinhood is a live application, I didn’t run the test— it’s never a good idea to load test a domain you don’t own when you’re talking about thousands of users. However, here’s a shareable link to another small Element test I ran previously.

Real-time dashboards like that one on Flood allow you to get a quick look into how the load test is running and to make any changes if necessary. When testing for FOMO, I think they’re invaluable. For example, during a stress test where users are added at regular intervals, a real-time dashboard would help you identify increases in response time as a result of the increased number of users. It would also let you monitor the test and stop it when necessary.

From the link above, click on the right arrow on the lone transaction.

That will take you to the Transaction Detail page. Remember those screenshots from Element? If you have any in your script, you’ll see them here, arranged according to time.

This lets you troubleshoot issues during FOMO testing and react quickly.

Conclusion

Even with sound load testing strategies in place, it’s challenging to plan for the massive spike in traffic that FOMO can bring, yet it’s necessary to do so. FOMO, as a phenomenon, is only going to increase in magnitude as social networks make it easier to spread fear, uncertainty, and doubt as well as information.

We may not be able to determine precisely what will break, or when, but what we can do is put systems into place to plan for it. We can assess an application’s susceptibility to FOMO and plan for end-to-end performance engineering of the system as a whole. We can start with the assumption that applications will fail and then determine the most likely candidates for that failure. We can routinely expose systems to traffic and circumstances that are extraordinary, and in doing so, we can improve our preparedness for something like FOMO.

FOMO is irrational and unpredictable— but that doesn’t mean it needs to be unexpected.

Load Testing Lessons from the Iowa Caucus

Fri, 06 Mar 2020 21:59:02 +0100

Originally posted here and here.

TL;DR? Here are the highlights in this article:

Shadow, Inc. created a mobile app for the Iowa Caucus in February 2020, and the app failed, causing reported results to be delayed. The app reported different vote totals due to a system integration bug where the output was in a different format than what it needed to be. Here are the steps to load testing a mobile app:

Test planning: Does the app require load testing? If yes, what is the underlying technology? How do we create a workload model?
Scripting: Which mobile load testing tool should we use? How do we record traffic from a mobile device?
Execution: What load test scenarios should we have? How can we scale up the load generated? Conclusion: While the Iowa Reported App failure was not due entirely to load testing, load testing could have helped make it more robust.

A company called Shadow, Inc. made headlines earlier this week when they developed an app to count and report votes during the Iowa Democratic Caucus in the US. This app, called IowaReporterApp, ended up failing on several fronts, resulting in a days-long delay in publishing the total votes, during which manual methods had to be employed.

Source: @ShadowIncHQ on Twitter

This delay led to significant criticism from the software testing industry as well as from the general public. Some developers raised questions about the code quality of the app. The app inaccurately reported voting totals that differed from the Democratic Party’s own manual reports, suggesting an issue with functional testing, particularly system integration testing. Shadow’s refusal to accept the Department of Homeland Security’s help with security testing also raised eyebrows. What was perhaps the most damning, though, was the question of whether such an app was necessary in the first place, when cheaper and more effective solutions existed.

What I’m most interested in, however, are the load issues that the app faced. Let’s rewind a little to before the caucus. How would I have load tested an app like the IowaReporterApp?

Planning a mobile load testing

One of the first things to figure out when planning a load test is whether a load test is even necessary.

Do you even need to load test your mobile app?

To answer that, I had to find out more about the app.

In the caucus, residents of Iowa who registered themselves as Democratic voters gathered to vote on whom they thought should be the party’s candidate for the presidential elections later this year.

There are a few things that I think make this app the perfect candidate for load testing:

It was highly publicized. The caucuses would always have been an event of international interest, but the hype surrounding electronic voting increased media attention on the caucus even further.
It was mission-critical. The precinct chairs would use the app as the primary device for capturing votes.
It needed to handle significant simultaneous load. There were over a thousand precinct caucuses organized, each with a precinct chair that would be using the app.

For these reasons, the answer is a resounding “Yes.”

What type of app is it?

The best way to learn about an app is to ask its developers. In this case, I opted to do a little research on my own by downloading the .apk file that Motherboard (Vice) made available and looking through the code myself.

The app is a mobile-only app for Android and iOS devices, written mostly using Facebook’s React Native package. The primary function of the app was to sent the vote totals to Shadow Inc.’s Google Cloud Functions server.

The technology involved is worth noting because it can help determine which load testing tool to use. Since the Iowa Reporter App, like most mobile apps, sends requests to a RESTful API, JMeter or Gatling, both well-tested, robust, and open-source protocol-level tools with large user bases, were perfect for the job. I decided to go with JMeter, my personal tool of choice.

How do we create a workload model?

Now let’s talk about the workload model, which is a plan for how the load we generate needs to look. The objective here is to simulate the load that matches the expected real-life load as closely as possible.

At this point, I would typically look at historical data to try to see the amount of load and the load profile. If I had metrics on the server traffic after previous releases of the app using something like Google Analytics or an APM tool, I would have used that data as a starting point for the workload model of this release. However, in this case, the app hadn’t been publicly released before, so I needed to make some educated guesses.

According to Wikipedia, there were a total of 1,777 Iowa precinct caucuses, of which 99 were global satellite caucuses set up for Iowa residents temporarily not able to attend the local caucuses.

In a Democratic caucus, the actual voting process generally takes about two hours. In the Iowa caucus, the precinct chairs would have used the app twice - once for each round of voting. Around the end of the two hours, each chair’s app would have sent the main payload, containing total votes for each candidate, to the backend server. All this is done simultaneously across all 1,777 precincts.

At a minimum, there are two main events here:

The login - Precinct chairs needed to log in to the app, which would presumably have happened around the same time across all precincts towards the beginning of the voting process, because the app also contained instructions on how to run a caucus.
The submission - This is the transmission of the total vote counts for each candidate, sent to Google Cloud Functions, and would also have happened around the same time.

It’s possible that the app also sent other requests between these two events, but since most of the app is inaccessible without a valid login, I’ll focus on the initial login request.

In a situation like this where I don’t have previous data to show how spread out those requests was, it’s prudent to err on the side of caution. Assuming the precinct chairs triggered those requests within 10 minutes of each other and rounding up the number of requests for some wiggle room, I estimated that there would have been 1,800 login requests within 10 minutes.

I decided to test this as a gradual ramp-up of users within 10 minutes.

Scripting for mobile

Okay, time to write the script. Since I didn’t have the luxury of being able to ask the developers for the format of the login request, I did the next best thing and tried to find out myself by recording the app traffic in action.

Recording mobile traffic on a laptop with JMeter

Here’s how I was able to see the request from the mobile app:

I downloaded the app to a spare Android mobile by using the APK link above.
I set up an HTTP(S) Test Script Recorder on JMeter on my laptop with the port 8888. I also made sure my laptop was connected to my wifi network.
I set up my mobile to use a proxy using my laptop’s local IP address on the network and port 8888.
Then, I went back to JMeter on my laptop and started the HTTP(S) Test Script Recorder.
Finally, I navigated to the app, typed in flood.io in the Precinct ID field (obviously not a real ID), and clicked the “Click here to log in” button.

Here’s the traffic as seen on JMeter’s View Results Tree listener afterward:

And here’s the recorded request within the recording controller:

Analyzing the request

It looks like at least the initial login request is a GET to an Auth0 service, with several parameters being passed in the body. It also has some authorization headers.

The parameter values also look like they might be dynamic, meaning they change from session to session. At this point, I would typically change some of these from being hardcoded values into dynamic ones so that the application receives a request in the right format, and I could look through Auth0’s API documentation to find out how the authentication message should be formulated.

Since I don’t have a valid login for testing purposes, I skipped this step and just used the raw request for demonstration purposes.

Preparing to run

Think time is a delay between requests meant to simulate the time it takes for a user to think before taking another action. Without think time, JMeter would fire off requests one after another, which often isn’t realistic and drains resources. I don’t want that, so I added a JMeter timer. There are plenty of timers available, but I settled on a Uniform Random Timer.

The CEO of Shadow told Motherboard that many precinct chairs were confused by the login process as they had to put in three strings of numbers in the right sequence. So, I decided to work that into the script with the timer. I set it up with a constant delay of 2 minutes and a random delay of 1 minute so that the delay would vary from 2 minutes to 3 minutes to simulate a precinct chair attempting to log in a few times.

Execution: Running your mobile load test

According to Motherboard, Shadow has already taken the app servers offline. However, it’s not a good idea to run a full load test against an endpoint that you don’t own, so I decided not to run a test.

Usually, though, I’d want to start with a shakeout test of 10 users or so, and then gradually increase the user load until the peak of 1,800 users. I also would have played around with changing the load profile, such as by increasing or decreasing the ramp-up or steady-state durations, experimenting with stress tests and soak tests, and adding traffic from another availability zone to simulate the 99 satellite precincts around the world. Running multiple load generators can quickly get out of hand if you want to use physical machines, so I would run load tests on the cloud with something like Flood. All of these little changes would have given valuable data that would help me gauge the app’s performance in production-like situations.

Conclusion

Building quality software is not a trivial task. There are many moving parts to account for, and critical errors can be introduced in every stage of an app’s development. While load issues were not the main culprit in the failure of the IowaReporterApp, a simple load test like the one I describe here, might have improved app stability and reliability. Even in cases where load testing doesn’t seem necessary, it can still uncover gross issues in configuration, usability, or security for an app developed on a deadline.

How AirTasker uses Ruby JMeter and Flood for their load testing

Mon, 24 Feb 2020 18:23:31 +0100

Originally posted here.

In the recent RubyConf 2020 held in Melbourne, Australia, Nancy Cai from AirTasker talked about how they built a performance testing framework using Ruby JMeter and Flood. Luckily, we had some pleasantly surprised Flooders in attendance to take notes during this fantastic talk!

AirTasker is a platform allowing the outsourcing of everyday tasks. When discussing the reasons that AirTasker decided they wanted to carry out performance and load testing, Nancy told a fascinating story about an overly successful marketing campaign—a good problem to have, but a problem nonetheless. The Sydney Cats and Dogs Home, an animal shelter, teamed up with AirTasker to pay one lucky person $500 to spend the day playing with the resident animals. The prospect of this dream job created an enormous surge in AirTasker’s traffic that caused alarm bells to go off on the underlying application servers, resulting in an end-user experience that was less than desired.

Nancy included this hilarious slide about some common feelings regarding application performance.

Steps for running a performance test, the AirTasker way

To avoid repeating that mistake, Nancy and her team set out to run performance testing by using the following simple steps:

Determine performance criteria.
Configure the test environment to be as production-like as possible.
Plan and design based on user behavior, including preparing test data.
Implement test design and set up the framework (which in AirTasker’s case involved Ruby JMeter as the performance testing tool and Flood as a load testing solution).
Run test.
Analyze and fine-tune results.

Setting up a load testing tool and framework

When it came to selecting the right tools for the job, AirTasker wanted tools that met the following criteria:

Ability to simulate production-like traffic easily
Widespread use in the industry
Ease of creation and maintenance of tests for everyone
Historical results aggregation
Visualization of performance metrics
Integration with CI and monitoring tools

Nancy’s team compared several load testing tools and found they enjoyed using the Apache JMeter tool, which is a popular open source load testing software. However, they quickly ran into an “out of memory” error after attempting to run about 400-800 concurrent users on a single machine. She and her team decided to use a distributed load testing platform to run the load they required to scale up more effectively.

Enter Flood.

Why AirTasker chose Flood

We also chose Flood.io […] because it comes in with a really good open-source gem called Ruby JMeter. It is a Ruby-based DSL for building JMeter test plans. - Nancy Cai, AirTasker

Nancy and her team enjoyed the benefits of being able to write load tests natively in Ruby and then using Flood’s Ruby JMeter gem, which is a Domain-Specific Language (DSL). Flood allowed them to turn their Ruby scripts into .jmx files, which is the native XML format that JMeter uses.

When they were ready to scale up, AirTasker ran the same JMeter script in Flood.

The real-time Flood dashboard allowed them to see that there was a problem on one of the API endpoints, called “Get pet reviews.” Further investigations revealed a performance issue due to a long-running query against a database server in the code, which AirTasker’s developers could fix.

Best practices

Nancy gives some great advice for anyone running a performance test, including:

asking the right questions as a load tester
running baseline tests
being aware of the difference between your test and production environments
keeping in mind the caching and autoscaling mechanisms that may drastically change the results
the differences between stress testing and other types of performance testing

We were so thrilled to hear Nancy’s story on how Flood helped AirTasker run their performance tests. Catch the entirety of Nancy’s talk below, and check out the resource links at the end of the post to follow Nancy’s story.‍

Flood’s Ruby JMeter gem

Nancy’s GitHub repo where you can find a copy of her test scripts

Follow Nancy on Twitter

Follow Nancy on GitHub

Mobile Load Testing on the Cloud

Fri, 21 Feb 2020 18:44:58 +0100

Originally posted here.

What is mobile load testing?

What is the difference between load testing web apps and mobile apps? In general, your goal is the same: to improve app performance by systematically exposing your app to the load you expect, whether it’s on the web or on a mobile device. However, while the purpose may be the same, there a few extra things to consider when it comes to mobile app performance, and it’s easy to make some key mistakes. In this article, I’ll give you a little bit of the theory behind mobile load testing and then show you practical steps to script and execute a load test.

Challenges in mobile load testing

If you have a web app that is mobile first, and your users typically access your site through a browser on their mobile devices as well as on their laptops, you have more choices for optimizing mobile performance. You can use browser-based load testing tools as well as protocol-level load testing tools to generate load.

If you have a native mobile app that you’d like to load test, though, browser-based load testing tools are a little more complicated. One of the most common approaches to mobile load testing is to use some type of device farm. A device farm is a collection of mobile devices and a way to orchestrate and control them so that they access an application simultaneously. Here’s why they’re not the best choice for load testing.

Device fragmentation

One of the biggest challenges is mobile device fragmentation. Not only are there a lot of mobile devices, but there are also many operating systems and versions. Even narrowing the pool to Android and ios leaves us with a large selection. A common mistake in mobile load testing is trying to account for these hundreds of combinations.

Logistics

Another challenge is the logistics of controlling physical devices with different operating systems, which is not trivial. Unless you have a thousand friends that you can rope into doing this at the same time, you’ll need to pay for some sort of service that can orchestrate this for you.

Cost

There’s also the fact that this process is resource-intensive. Even if your device farm consists of emulators on virtual machines rather than physical mobile devices, that still uses up a lot of CPU and memory, and you’d need to spin up dozens or hundreds of machines.

Luckily, this type of mobile load testing is not the only option available to us.

Better strategies for mobile load testing

A better option for back-end performance testing or load testing your mobile application is by using API load testing, which works on the protocol level. To understand this, let’s talk about what aspects of software we are analyzing when we test for “performance.”

We can generally break up performance testing into two types: front-end and back-end. Both types have the goal of making the app respond faster to end-users, but they do so by optimizing different parts of the application.

Front-end performance

Front-end performance is primarily concerned with the UI of the app. A front-end performance tester optimizes page resources (like CSS, images, JS) to render faster on the page. Since a mobile app runs on a user’s smartphone or another mobile device, its front-end performance would also be affected by things we can’t control, such as the operating system and version of the mobile or its physical configuration (CPU and memory). A front-end issue for a particular mobile device type might not necessarily affect performance on another. Device farms can be good at measuring front-end performance.

Back-end performance

In contrast, back-end performance focuses on the underlying mobile application servers. A back-end performance tester optimizes the things behind the interface, and it is affected by things like the size of the responses that a server sends to a user, server configuration, application infrastructure, and how downstream servers talk to each other. These factors may be invisible to a typical end-user. A performance bottleneck in the back-end is likely to affect all users of an app to some degree.

Load testing for back-end performance

Load testing is about determining how your mobile application behaves in response to traffic, so it targets the back-end components of the application, not the front-end.

Both front-end and back-end performance are necessary to get a holistic view of mobile performance. Still, if you want to load test your mobile app, you need to generate traffic on your application server, and not necessarily to drive the app on real devices. Understanding this opens up options for mobile load testing.

API load testing

Most native mobile apps work by using a RESTful API to send instructions to the application server. Luckily for us, there’s no magic in sending these types of requests from a smartphone— we can simulate this easily by using an API load testing tool to send these requests on a non-mobile device too.

Here are some advantages of using API load testing tools to test your mobile app instead of using a device farm:

It’s easier to set up and maintain: API load testing can be easier because you don’t have to account for changes in the user interface as long as the underlying API calls are the same.
It’s cost-effective: You can run about 1,000 users for an API test on a single machine (we used AWS m5.xlarge machines to arrive at this estimate). This translates into fewer resources that you’ll need to pay for whether you’re running a thousand users or [a million](whether you’re running a thousand users or a million), and, ultimately, a lower cost.
It lets you narrow your load testing scope: If you have a particular endpoint that has historically shown performance issues, you can use API load testing to target just that one endpoint. On a device farm, you would have to create a test automation script that might click a button and send dozens of requests, only one of which you actually wanted to test.

Tools for mobile load testing

You can use many tools for API load testing, but you can’t go wrong with robust open-source tools like JMeter and Gatling. Both tools are well-used tools in the performance industry, and they also have large user bases and ample documentation for helping anyone get started. In this article, I’ll continue with JMeter specifically.

How do you get started with writing your script? Ideally, your developers should be able to point you to the API documentation for your app or be able to describe the format of the request themselves. But what if you wanted to get started on your own?

How to record network traffic from a native app with Apache JMeter

Recently I wrote an article on The New Stack about how I would have load tested the IowaReporterApp, which was responsible for delays in reported vote totals for the 2020 Iowa Democratic Caucuses. To show how to load test that app, I created an actual load testing script for that article, without any input from the app developers. I chose to script using Apache JMeter, a popular open source API load testing tool, but Gatling (also open source) is another good option.

Here are the steps I took on my Android mobile, a Samsung Galaxy Note 10+. The same principles should apply to iOS devices and other Android devices.

1. Download and install the app.

Even if the app isn’t on the Google Play store, you can install an app using the .apk file. You may have to enable Developer Mode on your phone, which is not the same as jailbreaking. All this does is bypass the security feature that would otherwise not allow you to install apps that don’t come from the Play store.

To enable Developer Mode, go to Settings > About phone > Software information. Then, scroll down to the field Build number and tap it seven times. You should see a confirmation message that Developer mode is on.

Now download the .apk, click on it, and follow the prompts to install it.

2. Set up an HTTP(S) Test Script Recorder on JMeter

You’ll need to have JMeter set up already for this step.

Connect your laptop to a wi-fi network.

From a new test plan on JMeter, right-click on Test Plan and then click on Add > Non-Test Elements > HTTP(S) Test Script Recorder. Click on it and take note of the port it uses in Global Settings.

Next, right-click on the test recorder element you just created and click Add > Listener > View Results Tree. This element shows details of requests and responses when we run tests.

Then, right-click on Test Plan again and click Add > Threads > Thread Group.

Finally, right-click on Thread Group and click Add > Logic Controller > Recording Controller.

Your test plan should look something like this:

3. Set up your mobile to use a proxy.

On your mobile, connect to the same wireless network that your laptop is connected to.

Go back to your laptop and find out its local IP address. On a Mac, you can see this by clicking on the Apple logo > System Preferences > Network > Advanced > TCP/IP. The IPv4 Address is what you’re looking for, and it might look something like 192.168.86.125.

On your mobile, go to Settings > Connections > Wi-Fi. Under Current network, click on the settings wheel next to your wireless network. Click on Advanced.

Enter your computer’s local IP address in the Proxy host name field.

In the Proxy port field, enter 8888, or the port that was displayed in the HTTP(S) Test Script Recorder on JMeter.

Click Save.

4. Start recording traffic.

On JMeter, click on the HTTP(s) Test Script Recorder element and click Start.

On your mobile, open the app and log in or perform the action that you’d like to record.

As you use the app, you should be able to see the requests coming in on JMeter in the View Results Tree:

If you click on the Recording Controller, you’ll also see the exact format of the requests that the app made:

Scripting and troubleshooting your mobile load testing script

What you now have is a recording of the mobile traffic you need to simulate. If you stop here and try to run it, you may find that it fails. That’s because you haven’t done any scripting yet.

Here are a few things you’ll need to do to turn a recording into a script.

Add assertions

An assertion is a way to verify that the request is returning the response that you expect. JMeter only marks a request as failed when it returns an HTTP 4xx or HTTP 5xx response code. However, there are times when the HTTP response code returned is an HTTP 200 OK, but the page that was returned was still not the one you expected. Without assertions, JMeter would mark this request as passed, yielding false positives.

For instance, if you’re sending a login request, then a good assertion for the response to that request might be text like “Welcome, Nicole” or “Logged in.” To see the data that should be returned, you can either run through the action manually in your browser and use Developer Tools to inspect the response or click on the response tab in the View Results Tree listener in JMeter from your recording.

Assertions also allow you to pinpoint more accurately where in your script the error is.

Replace hardcoded values with dynamic values

The last screenshot above shows the parameters that the mobile application sends to the server to log in. If you examine the parameter values, you’ll notice that several of them are long strings. The script fails when you just send the same request again because some of these are dynamic. This means that parts of the request need to change before your device is allowed to access the application. You might need to add a timestamp, the right answer to a challenge, a session or client ID, or codes returned in previous requests. These dynamic values help the server determine whether the login request is a valid login attempt.

This is where help from a developer would come in handy because he or she can tell you how each parameter value needs to be formed.

Add think time

Think time is a delay between requests that simulates the pauses that a real user takes to read text on the page or otherwise interact with your application. Most performance testing tools do this. In JMeter, they are called timers, and there are several to choose from.

I like the Uniform Random Timer because it allows you to set a constant value for the delay as well as an additional random delay on top of it. I recommend having a variable delay because setting a completely constant delay makes the test too regular and causes unnaturally high load at the interval you set. Real users don’t coordinate with each other, so stagger your requests by using a variable component to your think time.

Debug your test script

Before you think about scaling up, run your test a few times with one user on your local machine. If you don’t get any errors, try other configurations such as running two sequential users and running two threads at the same time. This way, you can test any correlations you’ve had to make for those dynamic values.

It may be tempting to skip this step, but it’s necessary before scaling up your load test.

Set test parameters

Test parameters are settings like the number of threads, ramp-up, and duration, which allow you to shape the pattern of the load you want to generate according to your test scenarios and test cases.

Number of concurrent users or threads

This is the most obvious way to increase load. The more instances of your script running, the more requests are executed.

You can change this in JMeter by clicking on your Thread Group and changing the “Number of Threads (users)” field:

‍

Ramp-up

In production, load on servers very rarely goes from 0 to 1000 in one second. Even for cases with a very definitive start time (such as an item going on sale at a particular time), the load generally increases gradually. You can simulate this gradual increase by adding ramp-up times in your script. A ramp-up is the amount of time during which new users are added at staggered intervals.

Here’s an example of what that might look like:

Duration

Duration is how long the whole test lasts.

The simplest way to set these values in JMeter is to use the basic Thread Group. You’ll need to change the fields highlighted below:

To use the duration, you’ll have to select “Forever” next to Loop Count and “Scheduler” and type in the number of seconds in the Duration field. Otherwise, you can also run the script according to the number of loops, or iterations, that you want to run. However, since response times vary, using the Loop Count as a proxy for duration yields different results from test to test.

Running a shakeout test against your mobile application

When you’re ready to run on your load generators, resist the urge to jump straight to the peak load you want. This is a great way to incur unnecessary costs. Instead, run a small test with dozens of users. Running locally is for debugging issues in the script. The purpose of running a shakeout test is to spot errors elsewhere, such as in your test environment and your test data.

Baselining your mobile application performance

Finding a baseline is one of the primary goals you’ll have as you start executing load tests. To be able to assess how changes in the environment or code affect application performance, you’ll need a stable point of comparison. To accurately measure the effect of a variable (like a code change), it’s best to keep as much of the other circumstances surrounding the test as fixed as possible. The baseline is your unchanging test scenario that you can use as a point of comparison to assess future performance.

The application you’re scripting against, your tool of choice, and your script all affect your performance baseline, so do some experimenting to find out the setup that works best for your situation. This setup includes characteristics like:

the number of concurrent users
test duration
think time, pacing, and all other waits and delays
the script (including how requests are broken up into transactions)
user density

User Density

User density is the number of threads that you can run on a node or a load generator. Figuring out user density it essential because all load generators, even virtual ones, have finite resources. Each load generator also has a finite amount of load that it can generate, based on its CPU and memory utilization, among other things. Trying to generate too many users on a single node may result in the node itself being the bottleneck for your test.

To accurately assess your application’s performance, your load generators need to remain healthy during the test.

It can be helpful to have a number to start with. At Flood, we’ve found that we can reliably run 1000 users using JMeter or Gatling with one of our AWS m5.xlarge machines. For reference, an m5.xlarge machine has 4 VCPUs and 16 GB RAM.

If your machine is similarly specced, run a test on a single node with 1000 users. While the test runs, watch the CPU and memory utilization. If the test finishes without either of those consistently hitting over 80%, you’ll know that the node can handle that number of users.

Let me reiterate, though, that you should use the 1000 users figure as a starting point only. You can then figure out your number through trial and error. Increase the number of users past 1000 and watch the resource utilization again. If that still looks good, add some more users and rerun. When you get to a test where the resource utilization hovers above 80%, stop and fall back to the previous number of users. You’ll now have your number.

Another thing you might want to play with is the think time and pacing in your script. These waits tend to have a significant impact on resource utilization, so you can expect to be able to run more users per node if you increase your delays.

If you’re using JMeter, always run your tests in non-GUI mode. GUI mode is great for debugging, but is unnecessarily resource-intensive for real load tests. While you’re at it, disable any listeners you may have that you don’t need to capture results.

Figuring out the appropriate user density now prevents you from getting inaccurate test results. Running as many users as you can without overloading the load generators is also cost-effective, as you’ll be making sure that you provision only as many nodes as you need.

Scaling up your load test on the cloud

It’s relatively simple to run a load test on one machine, but if you want to run on two or more machines, things quickly get unwieldy. Here are some options for achieving this.

1. Upload your tool of choice and script to every machine and kick off each test separately.

Instead of running one large load test, think of this method as running several smaller ones at (about) the same time. This is relatively easy to set up, but the main disadvantage is that it’s tedious. You may also need licenses for your tools if you’re not using an open source one. You’ll want to ensure that each load generator is as similar as possible to the others in terms of operating system, tool version, and script version. Since each node kicks off a separate test, you’re not going to be able to see a real-time combined view of all the load tests, and after execution, you’ll need to retrieve results files individually and combine them.

2. Use your tool’s distributed testing mode.

JMeter has a feature that allows you to scale. This involves setting up agents on each load generator and using scripts to coordinate execution and results collection. This method requires a little bit more know-how and time to set up, but it’s a little more cohesive than the first method. Here are links on how to set this up: Remote Testing with JMeter

3. Use a load testing platform.

I’m admittedly biased, but if I were to go back to doing performance testing consulting work, I would choose Flood to run my tests. A platform like Flood that is built for distributed load testing is an excellent choice because it takes away all the setup considerations. This is the easiest option, especially for teams that are new to load testing or are perhaps less technical, because all the setup is done through a UI. Scaling out in this case just means uploading your script, choosing the number of nodes you want to run and in which region, and all the work is done for you in the background.

While you can run on-premises, I would consider running your test cloud-based infrastructure for greater flexibility and ease of scaling unless you already have a data center.

What happens next?

Running your test is just the beginning of mobile performance testing— there’s still metrics collection, results analysis, and resolving performance bottlenecks that you uncover. After all, it’s what you do with the test results that counts. Remember that performance and load testing, whether for web-based applications or mobile ones, are iterative. The real value comes from building the app performance testing principles discussed here into your release cycles, so that load testing becomes a habit rather than just a one-off exercise.

Load Testing Tutorial: Server Monitoring

Thu, 26 Sep 2019 19:30:10 +0100

Originally posted here.

It’s difficult to be a good load tester if you don’t like data, because a lot of the value of the load testing is in the analysis. Even if you successfully generate load against an application, it’s ineffective without enough data to determine how your application coped with the load. At Flood, we’re passionate about data because we know it drives decisions, so we’re always working to improve how we present data generated by our nodes. However, there’s another important source of data that in my experience is too often overlooked by new load testers: your own application servers.

Executing a load test without monitoring server health is like flying blind: you’ll know when you land safely and you’ll know when you crash, but even if you do crash, you won’t know why—or how you can avoid it next time. Monitoring server health is the black box that will tell you what went wrong.

What server metrics should you monitor?

There are a lot of metrics that you can monitor, and we always recommend that you choose what to measure based on your own specific circumstances, such as the purpose for your testing or any previous outages you’ve had in the past. If you’re new to load testing, though, here’s a list of metrics you might consider measuring:

Processor Time - how much the processor is being utilised
Processor Interrupt Time - how much time the processor is spending to handle interrupts
Processor Privileged Time - the time the processor spends handling overhead activities
Processor Queue Length - the number of threads that are waiting to be executed
Memory (Available Bytes) - unused memory available to process new requests
Memory Cache Bytes - the size of the memory cache, which is where copies of frequently accessed data are stored
Disk I/O - number of reads and writes to the disk during the test
Disk Idle Time - time that disks are not doing work
Disk Transfer/sec - average number of seconds that an I/O request takes to complete
Disk Write/sec - average number of seconds that a write request takes to complete
Network I/O - bytes sent and received

How do you determine which one to use?

If you’re still not sure where to start: At a minimum, you’ll need the CPU and memory utilisation (in bold in the list above) of every major component you want to test. These two metrics are vital and if either of these is consistently maxing out at (or close to) 100%, that’s a sign that the component is struggling with the number of requests.

As an aside, CPU and memory over-utilisation is a very common reason for less-than-ideal response times on the load generators, which is why we also show you those metrics on our nodes while you’re running the test.

How can you monitor these metrics?

How you get these metrics depends on your budget and your operating system. I’ll start with the free or lower cost methods and work my way up to enterprise solutions.

If your servers are running Windows, PerfMon is a good alternative. It’s built into Windows and the interface allows you to choose the counters that you want to measure and start recording.

There’s also a JMeter server agent that you can download, confusingly called “PerfMon” but not to be mistaken with Windows’ PerfMon, that you can install on your (Windows or Linux) server and will gather metrics. It’s normally used in conjunction with the PerfMon JMeter plugin.

Nagios Core is another good open-source way to not only collect metrics but also set up alerts when values exceed certain thresholds that you can set.

Moving from free and open-source options to more enterprise solutions, Dynatrace is a powerful tool that allows you to track not just server health but can actually be used to trace individual requests using a custom header.

AppDynamics is another fantastic tool that allows you to really drill down to specific SQL queries that take a long time to execute, for example, feeding you important information to give to your DBAs.

New Relic is another good choice for monitoring production as well as test environments, and we particularly love its thread profiling feature.

Other noteworthy tools are the Microsoft System Center Operations Manager, the Oracle Enterprise Manager, and BlueStripe FactFinder.

Flood offers out of the box integrations with Dynatrace, AppDynamics and New Relic. If you’re already using one of these tools, you can push the load metrics from Flood into the dashboards that you’re already using, allowing you to see both load metrics and server metrics in the same place. Here’s an example of a dashboard our own Antonio Jimenez created in New Relic which combined Flood-provided metrics with server-side metrics:

This view helps with spotting correlations or trends across the different sets of data, such as a response time graph that begins to increase at the same time that server memory utilization does.

Putting It All Together

Having both sides of the puzzle, the metrics from the application servers along with those from our load generators, gives you a more holistic look of your application performance as a whole, and increases the value you get from load testing. We’d encourage you to take advantage of the 5 free node hours we provide in our trial to give this integrated load testing solution a spin. You can head on over to Flood to sign up for a free trial which will include 5 free node hours and you can drop us a note to get help integrating Flood with your APM solution.

Start with Why: How to Write Requirements for API Load Testing

Wed, 19 Jun 2019 21:49:17 +0100

Originally published here.

Every good test begins with requirements.

You may think, “But I’m a tester! I’m not a business analyst!”, and I hear you. But as testers, especially on smaller teams or shorter projects, it’s still part of our job to make sure we know what our clients want. Otherwise, how do we know whether we’ve succeeded or failed?

At Flood, I often speak to project teams who ask for help interpreting reports. Most of those times, the problem is not that they don’t understand the metrics or what they measure. The problem is that they didn’t set pass or fail criteria in the first place, and so they don’t have anything to judge the results against. This is always a big warning sign.

Which performance testing metrics are important? Why are we testing, anyway?

Tim Koopmans, co-founder of Flood, coined the acronym SPEAR to describe the different aspects of a performance that should be considered in load testing. This is a great starting point when thinking about what we want our nonfunctional requirements to cover.

Scalability

Scalability is the application’s ability to cope with increasing demands by increasing the amount of server resources. This could mean scaling up (increasing the resources of the dedicated server) or scaling out (adding more nodes to shoulder the load). What happens when more users than expected sign up in response to a promotion on your site?

Performance

The most common performance metric is page response time, but there are other considerations to be made here, such as throughput (requests per minute) and the number of concurrent sessions that need to be supported. Things like the total size of the resources on the page, whether or not a CDN is being used, and what to cache are also worth discussing.

Elasticity

Elasticity is a relatively newer aspect to performance testing brought about by advances in the cloud that allow application infrastructure to adapt to changes in load. Unlike scalability, elasticity emphasises scaling down as much as it does scaling up. Testing that virtual machines scale up when load increases is important, but testing that virtual machines also scale down when load decreases can also help save on unnecessary costs.

Availability

To test for high availability, ask yourself what would happen when (not if) your application’s server fails. Is there another server that the load balancer will seamlessly send traffic towards? Does the throughput fluctuate wildly? If users are connected to one server that fails, is your application smart enough to make new connections to another server? Or will it simply serve up an error page that users won’t know what to do with? Disaster recovery is best tested when there’s no disaster imminent.

Reliability

Reliability encompasses a lot of scenarios, but they all have to do with whether or not your application returns expected responses. Does your error rate increase when you increase the duration of your load test? Are you adding verification steps to your load testing scripts to check whether or not the HTTP 200 response that the application returned is not actually an error page?

What should my application’s response time be?

Our clients frequently ask us what the industry standard is for response times, wanting to make sure their applications measure up. The answer, however, is more complicated than a single number.

Industry standards for response time are only useful when applications are very similar. Constantly changing technologies used in web development as well as innate differences in business processes, however, make it very difficult to extrapolate a single number that will apply to all, or even most, applications in a certain industry.

The home page of one e-commerce app, for instance, might be several seconds slower than that of their main competitor. However, that doesn’t take into account the fact that their app loads a video showcasing new products. Does that mean that the development team should remove the video in order to fall in line with their competition?

Well, maybe. But not necessarily. It’s a business decision that needs to be made after perhaps using focus groups to determine the impact of the video, forecasting changes in conversion rate due to it, and comparing its projected value to the effects of being slower than the competition. A/B experiments could be used to test these assumptions and gather quantifiable data to support the team decision.

These factors are often not considered in the search for one number to rule them all, which is why a fixation on that number can be detrimental. Instead, I encourage project teams to brainstorm and come up with their own numbers for all metrics that would be more appropriate for their application. Gathering comparative metrics from a competitor may be part of this process.

What is a good API load testing requirement?

A good requirement, just like any good goal, is SMART:

Specific

Vagueness in a requirement leads to vagueness in results interpretation.

Instead of: “The performance of the web application…”

Try: “The average response time of the Login transaction…”

Measurable

Make sure there is a quantifiable way to know whether requirements have been achieved.

Instead of: “Decrease user frustration.”

Try: “The error rate must be below 3% at peak load.”

Agreed Upon

Have the appropriate stakeholders been involved?

Instead of: “The system must be able to generate emails as soon as users register.”

Try: “The system must be able to generate a maximum of 100 emails an hour, after which emails are queued.”

A good example for this is a project I was involved in that aimed to increase the speed with which emails were generated. Unfortunately, the particular email being sent included a big change that many customers were expected to contact the support team about. The problem was that nobody had thought to include Support in the conversation. Once they heard about the expected end result, the support team quickly raised their concern that they would not be able to handle the expected volume of emails unless the emails were staggered. This could have been avoided if they had been brought into discussions from the very beginning, in the requirements gathering phase.

Realistic

Can we meet this requirement given the resources available?

Instead of: “All requests must be returned within 5 ms.”

Try: “90% of the Catalog page requests should be returned within 3 seconds.”

Timely

Especially for nonfunctional testing, consider adding a timeframe to requirements.

Instead of: “The digital code is sent by SMS upon successful client log in at peak load.”

Try: “The digital code will be sent by SMS no later than 5 minutes after successful client log in at peak load.”

Go forth and write great load testing requirements

Requirements represent a great opportunity to think things through and make sure everyone on the project team is on the same page about the goals for your load testing. Too often projects skip this phase, only to realise much later that the tests that were executed didn’t address a key stakeholder’s concerns.

In load testing, as in many things: when in doubt about what to do, start with why.

TestCon Moscow 2019: Dismantling Cognitive Biases

Wed, 03 Apr 2019 23:14:31 +0100

Originally posted here.

Developers and testers alike prefer to think of themselves as entirely rational beings, because both software development and software testing do require systematic rigor and logical reasoning. However, the reality is that the human element still comes into play when building and testing software.

In this year’s TestCon conference, held in Moscow, Russia, I delivered a talk on how cognitive biases affect web performance outcomes.

In the talk, I discuss:

two types of thinking: fast thinking and slow thinking (from Daniel Kahneman’s book Thinking, Fast and Slow)
how to influence the sentence duration that judges give perpetrators
what cognitive biases are and how they can affect performance testing results
a real-life production incident that occurred on Flood and how our team’s cognitive biases got in the way of its resolution
what to do to avoid cognitive biases when testing software

Here’s the video of my full talk:

Download the presentation in PDF format.

Performance Testing Interview Questions, Part 3: LoadRunner

Thu, 20 Dec 2018 22:54:24 +0100

Originally posted here.

Like it or not, LoadRunner is one of the most popular commercial load test tools, and with good reason. Its protocol support is extensive, and many a load tester has cut his/her teeth on LoadRunner. In this last part of our interview series, we’ll go over some questions you may have to answer when interviewing for a role using Mercury HP MicroFocus LoadRunner.

What sort of transactions would you put into vuser_init and vuser_end?

Both are used for transactions that would only need to run once, at the beginning and at the end of a script. They’re akin to JMeter’s setUp and tearDown thread groups. Some things you might put in vuser_init are user logins or navigation to get to the start of the business process you really want to test, and logouts and any data cleanup to reset the application for the next test are common things to put in vuser_end. Note that if you want each virtual user to iterate through a list of user accounts during the test, you won’t want to put logins and logouts in vuser_init and vuser_end.

When would you run a script in VuGen and when would you run it in Controller?

VuGen is short for Virtual User Generator, and it’s where the actual scripting is done. Running in VuGen is good for writing or debugging a script, where you can quickly see where a script fails, what the response is, and what you need to change. You can also step through the code line by line (F10) to more accurately see, for example, variable values changing throughout your script. It’s really only for shaking out scripts with a single user. Running in VuGen also only runs from your local machine. Once you’re ready to run a load test, you’d switch to Controller.

The Controller is like the command center in LoadRunner: from the Controller, you can actually ramp up the number of users and choose from different scenario configuration settings in order to tailor your workload model to the scenario you’re trying to test. Controller also runs the scripts headlessly; it won’t open up a separate instance of VuGen for every user, so it’s a lot more resource intensive. Controller will also allow you to configure remote load generators and run scripts on them.

How would you set up LoadRunner to run certain transactions a certain percentage of the time? For example, for every 100 users that browses to a site, you want 25 of them visiting each of the About, Catalog, Contact, and Reviews pages.

You can write the code to visit each page in separate Actions. Then, in Runtime Settings, you can modify the run logic, create an Action Block that has all four actions, and assign a percentage to them (25%).

Of course, you can also hard code this in C within the actions, if you’d prefer, but moving these values to Runtime Settings makes it easier to maintain and reuse by other testers.

How is LoadRunner pacing different from how it’s implemented in other tools?

Most test tools allow you to set a fixed or variable pacing that adds onto however long one iteration takes to complete.

LoadRunner has a pacing feature that allows you to set a specific time and then adds a variable amount of time to your test depending on response times for the included transactions. So if you set pacing to 5 minutes and one iteration takes 3 minutes to finish, LoadRunner will automatically wait 2 minutes before starting the next one. If it takes 6 minutes, it will not add any more wait time and proceed instead to the next iteration. This comes in handy because you can more accurately predict how many transactions the script will do in a certain amount of time.

Why would the traffic from a load test on LoadRunner not be seen in Google Analytics?

Google Analytics runs when the browser of a user downloads and executes a javascript file that then sends information to Google to increment the site statistics for the current visit. However, most traditional load test tools work on the protocol level; that is, they send requests directly to the application server and download resources from a page to simulate users. This means that while they download the Google Analytics script, they don’t execute it, which means the scripted visits aren’t counted.

In order to use Google Analytics to view statistics from a load test, you’ll need to use a browser-level load test tool. LoadRunner actually has this covered with their TruClient protocol, which interacts with elements on the page like a user would. It will also work for testing single-page applications that use AJAX to change the data displayed on a page instead of reloading new pages.

Of course, for open source alternatives to TruClient, we at Flood prefer Element or Selenium load testing, both of that will actually open the site in a browser and executed scripts like a real user.

Performance Testing Interview Questions, Part 2: JMeter

Thu, 20 Dec 2018 14:47:18 +0100

Originally posted here.

In this part, we’ll go over some questions that are commonly asked in technical interviews about the use of the most popular open-source load testing tool, Apache JMeter.

Using JMeter, you record a key business process. Upon playing it back, however, you notice an error on the first page. What could have gone wrong and how would you fix this?

While recorders have gone a long way towards making scripting easier, modern security measures like the implementation of viewstates have also made scripting more complicated to record. Some more complicated applications will pass dynamic values in responses that it expects to see in the succeeding request, otherwise the request is blocked. This is great for security, but it also means that you will need to manually extract these values in your script.

In JMeter, you would do this by adding a response extractor to your sample that will extract the dynamic string according to rules you set, and then save the string to a parameter. This parameter must then be used in the next request that you send.

While running your test plan, you get the error: java.lang.OutOfMemoryError: Java heap space. How do you fix it? The easy fix is to edit jmeter.bat and increase the heap memory (search for HEAP) and increase heap space allocated like this: HEAP="-Xms512m -Xmx2048m"

The part in bold increases the heap space to 2 GB.

However, it’s also worthwhile looking at your test plan and removing any obvious memory hogs. One of the things we see a lot is enabled listeners that aren’t even being used. Listeners are great while debugging, but should be disabled when running a load test. Writing too many lines to the log is another culprit here that should also be commented out when running important tests. And of course, you should always run JMeter in non-GUI mode during load tests!

After running a load test, the results show that 300 transactions were executed with a PASSED status and returning responses of HTTP 200. However, the DBA tells you that there were no records written to the database as a result of the transactions. How would you troubleshoot this?

I see this happening with clients all the time, so I think this makes a great technical interview question. This is also why I said earlier that you shouldn’t count on a test tool to tell you a test has passed. It’s possible for a transaction to have the status PASSED and return an HTTP 200 but still be hiding an error.

This is mainly due to poor scripting that does not include validations that check whether the script has gotten to the right page. For example, you could use a Response Assertion as a child of your home page sample with the words “Welcome, Nicole” to determine whether the user Nicole has logged in successfully.

This is a critical step, because without it, it’s possible for a server to return a nice error page, or an incorrect one, such as the home page, with an HTTP 200.

How do you use a data file to read user account details from a CSV in JMeter?

JMeter has a CSV Data Set Config Element built specifically for this purpose. You can simply add one to your test plan, browse to the CSV file that you’d like to use, and select the options for how to share that data among the threads in your JMeter script.

How would you go about testing a website that uses HTTP/2 on JMeter?

There is a JMeter plugin called HTTP/2 Sampler that accomplishes just this. My colleague Jason just wrote a blog post on this very topic, but basically the plugin adds new HTTP/2 versions to the standard HTTP request and request defaults samplers as well as the Simple Data Writer and View Results Tree listener. These elements will need to be used in order to test an application using HTTP/2.

What can you do to synchronise the execution of multiple JMeter instances on different load generators?

Apache JMeter has released this handy PDF file to describe the steps necessary to run JMeter in distributed mode. This involves installing Java and JMeter on all machines and setting them up in a master and slave configuration so that one machine controls the rest. On the master, you’ll need to add remote hosts to the jmeter.properties file. When everything is set up, you should be able to open up JMeter on the master and do a a remote start of your current test plan on any or all slaves that you choose.

The problem with this setup is the clunkiness of working with different machines and essentially having to set them up the same way. After the test, there’s the daunting task of logging back onto each machine, gathering the data, and transferring them back into a central repository. Then you’re faced with consolidating the data from each injector in order to see a picture of the whole test. You can see that this set up quickly goes from time-consuming to a nightmare, especially when potentially thousands of different machines are involved.

If only there were an easier way…

Performance Testing Interview Questions, Part 1: General

Thu, 20 Dec 2018 12:41:40 +0100

Originally published here.

Preparation is Key for the Perfect Interview

Performance testing interviews can be a nerve wracking experience. We all understand it’s necessary, but we dread it anyway. Each company has its own style of interview, so you may need to prepare differently for each opportunity. This interview series will cover some general sample questions and scenarios as well as my answers to them to help you nail that performance testing interview!

I’ve broken down the potential questions into the most popular categories, so you can prepare easily for your all-important interview:

What is Performance Testing? What is Load testing? What is Stress Testing?

Performance testing is the process of systematically determining, quantifying and improving the speed, reliability, and efficiency of an application with the aim of improving user experience. This includes both front-end and back-end performance testing.

Load testing is a specific type of back-end performance testing where automated scripts are written to send requests in an attempt to simulate the peak user load expected on the application servers.

Stress testing takes that one step further and applies load in excess of what is expected in order to determine the maximum capacity of the system under test.

What is the difference between front-end and back-end performance testing? Which one is more important?

Both front-end and back-end performance testing measure how fast an application responds, but they measure different components of that overall user response time.

Front-end performance is concerned with how quickly text, images, and other page elements are displayed on a user’s browser page. Back-end performance is concerned with how quickly these elements are processed by the site’s servers and sent to the user’s machine upon request. Front-end performance is the part of the iceberg above the water line, and back-end performance is everything underneath that you can’t see.

Both are important because they can both determine whether a user continues to use your application. Front-end performance tends to be easier to test and can provide some quick wins due to the large amount of optimisation tweaks that can be done without writing code. Back-end performance tends to be more difficult to test because it often uncovers problems with the underlying infrastructure and hardware that are of a more technical nature.

Why does performance testing matter?

Performance testing matters because application performance has a significant impact on user experience. A site that is unreachable or slow to load due to an inability to cope with unexpected load will cause users to browse to competitor’s sites and tarnish the brand’s reputation.

How do you know when a load test has passed?

Ideally, you would have discussed your nonfunctional requirements with key stakeholders before load testing begins. This means that you set your own pass criteria before you even run the tests. You would ideally have a list of specific transactions (selected based on criticality or complexity according to the business) whose response time needs to fall under a threshold you’ve predetermined. “Fast” is not specific enough– a number is better. Depending on what kind of tests you’re running (soak, stress, volume, etc) you may have other nonfunctional requirements about duration, resource utilisation on the server side, or specific outcomes to scenarios you’d like to test.

As a general rule, don’t rely on a load test tool to determine whether your load test has passed. Rely on it to report your results, but always compare the results to the requirements to determine successes or failures.

What would you advise to clients who say they can’t afford to performance test because they don’t have the resources to maintain several load generators on site?

This is the main reason that performance testing has for so long been considered a luxury that only big companies can afford. Luckily technology moves on, and in 2018 we’re at a point where everyone can load test. The big innovation here has been the cloud and the ability to spin up thousands of virtual machines with a few mouse clicks. Services like Amazon AWS, Microsoft Azure and Google Cloud make it so that every budding entrepreneur can “borrow” the computing hardware necessary to do cloud load testing with thousands of users and then give them back after the test, without the hassle and cost of maintaining them. I would advise the clients to look for a a cloud load testing solution that utilizes virtual machines on the cloud to run their tests affordably.

You run a load test against a server with 4GB RAM and the results show an average response time of 30 seconds for a particular request. The production server has been allocated 8GB RAM. What would you expect the average response time of the same request to be in production given the same load?

Trick question! While you may be tempted to answer that the response time would be halved to 15 seconds, reality is rarely that convenient. Response times are a factor of so much more than memory. Things like CPU utilisation, network throughput, latency, load balancing configuration, and just application logic are always going to influence load tests. You can’t assume a linear progression in response time just because you’ve upgraded one part of the hardware. This is why it’s important to load test against an environment that is as production-like as possible.

What is a percentile and why would you look at percentile response times when you already have average response times?

A percentile is a statistical measure that describes a value that a certain percentage of the sample either meets or falls under. For example, a 90th percentile response time of 5 seconds means that 90% of the responses took 5 seconds or less to be returned. It can be an important measure because they soften the impact that outliers have on more inclusive measures such as averages. A transaction with an average response time of 2.5 seconds may seem perfectly acceptable to the business, but when the 90th percentile response time is 20 seconds, this is a good reason to investigate further.

What are some trends in performance testing that you think will continue in 2019 and beyond?

Cloud is an easy answer, as the cloud brings some compelling benefits in terms of reduced cost and just ease of use. However, I already touched on that in a previous question, so I’ll talk about another trend: open source.

There’s a reason that open source is still around: it works. The main advantage of open source tools is not that they are free, although that is a big part of the appeal. The real advantage is that open source tools are community-based and community-led, which means features get built for it faster than sometimes commercial tools can keep up with, and they’re built by users of the tool themselves. Open source tools like JMeter, Gatling and Selenium have revolutionized the industry due to their impressive feature sets, built by a growing community that has developed plugins for everything you can think of. More and more, even big companies with big budgets choose to go open source simply because of the wealth of knowledge of these tools that are already available for free.

We’re huge proponents of open source tools at Flood, which is why we recently open-sourced our very own tool, Element. We will continue to support all things open source, because we firmly believe that the future is open source.

Tips and Tricks for Using Flood Element

Fri, 23 Nov 2018 23:07:24 +0100

Originally posted here.

In case you missed it, there’s a new performance test tool on the block, and it’s called Flood Element. It’s a scalable browser-based tool that allows you to write scripts in javascript that interact with web pages like a real user.

Using Browser Level Users is a newer approach to load testing that overcomes a lot of the common challenges we hear about traditional methods of testing:

Easier scripting that is akin to common functional tools like Selenium and easier to learn
More realistic results that are based on true browser performance rather than API responses
Ability to test against all components of your web app, including things like Javascript that is rendered via the browser

Given the above benefits, it’s a no brainer to check out Flood Element for your web load testing, especially if you have struggled with the existing tools on the market like JMeter or HP LoadRunner.

Pairing Element with Flood turns it into a pretty powerful load test tool. We have a great guide here that you can follow if you’d like to get started with it. As you can imagine, I’ve been using and testing Element for several months now and I thought I’d share some tips I’ve learned along the way.

Initialising your script

You can always start from scratch, but the quickest way to get started is to type

element init myfirstelementtest

from your terminal, filling in the project name you’d prefer.

You’ll then be asked to type in the title of your test as well as the URL you’d like to script against. After a minute, you’ll see that a new directory has been created:

Element will automatically create a file called test.ts. This file will contain the skeleton of a script, along with some sample code to help you find a button and then click on it. But before you open it, let’s move on to…

Choosing the right text editor

Scripting in Element is already pretty simple, but I’ve found two things that really help while scripting are syntax highlighting and code completion. Syntax highlighting will greatly improve your experience when first learning a new test tool like Element, and code completion will make your scripting lightning fast as you become more experienced in the process. My text editor of choice is Visual Studio Code, which has both of those features. It’s slick and clean, and it does the job.

Syntax highlighting is when the text editor intelligently changes the font colour of your code according to its role in the programming language you’re using. Here’s a screenshot of the test.ts file we generated earlier in VS Code to show you what I mean:

This makes it easier to make sense of the code at a glance: comments are in green, values and labels are in orange, etc.

Code completion is when you start to type something, and VS Code helpfully opens a context menu with suggestions for methods you can use.

I love this because it means I don’t have to remember the exact name of the method. It also suggests names of variables you’ve already defined and highlights code that doesn’t make sense. This will help to make your tests more maintainable and readable for others, which is a great benefit as you look to scale your testing out in the future.

Taking screenshots

One of the most powerful features of Element is its ability to take screenshots. I find it immensely useful when debugging because sometimes it’s just easier to see what’s going on visually. With protocol based tools, debugging can often be a much more involved and technical process.

There are two ways to take screenshots in Element:

Add a setting to automatically take a screenshot when an error is encountered. You can do this by setting screenshotOnFailure to true in TestSettings:

export const settings: TestSettings = {
device: Device.iPadLandscape,
userAgent: 'flood-chrome-test',
clearCache: true,
disableCache: true,
screenshotOnFailure: true,
}

Explicitly take a screenshot at a particular point in the script. You can do this by adding

await browser.takeScreenshot()

to your code.

Viewing Screenshots

Once you’ve taken screenshots within your tests, you will likely want to view them and know that they will be stored for future safekeeping. Whether you are running your test locally on have uploaded it to Flood to run with increased concurrency, Flood Element has you covered.

Locally Run Tests

Screenshots will be saved as jpg files in a timestamped folder corresponding to your run. It should look something like: ...myfirstelementtest/tmp/element-results/test/2018-11-20T135700.595Z/flood/screenshots/ and the screenshots will be uniquely named so that new screenshots, even for the same step, don’t overwrite older ones.

However, I rarely have to look up the screenshots in that folder because I prefer to see them in iTerm2 for macOS. iTerm is an alternative to the terminal that works particularly well with Element. When you take a screenshot, iTerm actually shows it in-line:

‍

Tests Run in Flood

Running an Element script on Flood is ideal when you need larger concurrency. Rather than accessing your screenshot locally, Flood will centralize the images into your account so the images remain even after the cloud load injectors are destroyed. You can get to the screenshot files by downloading Archived Results:

Or you can also choose to click on a step on the dashboard to see a film strip of your test:

Using Logs

You may need to check out the logs for more technical debugging, especially when the screenshots don’t tell the whole story. Whether you are running your test locally on have uploaded it to Flood to run with increased concurrency, Flood Element has you covered.

Locally Run Tests

You can print to the console by typing, for example:

console.log('orderValues = ' + orderValues)

This will print the value of the variable orderValues at that point in the script. You would see this in your terminal if you’re running Element locally.

Tests Run in Flood

If you’re running the script on Flood, you can either download the log (in the same Archived Results zipped file as mentioned earlier) or you can click on the Logs tab:

‍

Fun with Flags

Element comes with a few flags that give you more control over how the script is run locally. Here are a few of my favourites!

Headless Flag

When in doubt, run Element in non-headless mode to see the script actually opening the web app on Chrome and interacting with the page. This is only possible locally, but there’s nothing like actually seeing for yourself what’s happening in real time instead of relying on screenshots and logs after the fact. To enable this mode, add the flag when running your test:

element run myfirstelementtest.ts --no-headless

Watch Flag

Element will automatically close the browser window when it encounters an error or finishes the iteration. Adding --watch will leave the browser window open and then will monitor the script. As soon as the script is saved, it will automatically run it in the same window from the beginning. Simply add this flag like the above example:

element run myfirstelement.ts --watch

Dev Tools Flag

This opens a browser instance and runs the script with the Chrome Dev Tools open, allowing you to find locators for the next action you want to script. Simply add this flag like the first example:

--dev-tools

More Flags

For more flags, use:

element run --help

Try Element

You’ve just gotten a crash course on Flood Element. You’re all set! If you haven’t already, download Element here to start writing functional test scripts and reusing them as load test scripts on Flood. If you don’t have a Flood account, you can easily sign up for a free trial here.

We’re extremely proud to have been able to contribute to the open-source community and can’t wait to have you try this new addition to the Flood line!

Prepare for Viral Videos With Flood Element

Thu, 08 Nov 2018 23:24:31 +0100

Originally posted here.

One of the more interesting things to load test is video. You see, traditional load test tools are protocol-based, which means that their approach to load testing video is to capture and simulate the traffic that the browser generates under the covers, sending the same requests to the application. This approach can still be successful– in fact, one of our favourite clients here at Flood recently used Gatling to simulate 6 million concurrent users streaming video from their application. However, for some use cases, testing at the protocol level can’t simulate everything that happens in a real world use case.

The Challenge

If you’re a video creator, the ultimate goal is often a simple goal: creating a viral video. Unfortunately, preparing for and simulating this all-important use case is typically impossible with protocol based tools. The same feature that makes them so efficient – not using a browser to send the requests – is the same thing that prevents them from being used to simulate the needed browser level events. You see, things like view counts on a video are normally tracked by scripts running on the page. Traditional tools download these scripts along with other page resources, but the scripts themselves are not executed. Enter Flood Element.

A Solution: Flood Element

Flood Element represents a new way of looking at load testing. It is a browser-based tool, which means that it actually drives an instance of the browser to test an application. Rather than telling it what requests to send to simulate a user like you would do with a protocol-based tool, you tell Element which buttons to click or which elements on a page to interact with just as a real user would. Essentially Element IS a user, using your application the way it was meant to be used. This means that it also executes any scripts that would run on the browser side. It’s so realistic that you could even use Element to make your YouTube video viral* by scripting up users to view your content… hypothetically speaking.

Simulating Real User Views

Testing video with Flood Element is actually pretty easy, especially if you are familiar with open source testing tools like Selenium. If you haven’t yet begun using Element, go through the installation guide here. You can download the full script used for this tutorial here. In this example, we’re going to be using a YouTube video of the opening credits the classical cartoon Top Cat (the indisputable leader of the gang) from Warner Bros.

Step 1. Navigate to the video page

Let’s start with a basic script:

import { step, TestSettings, By, Until } from '@flood/element'
export const settings: TestSettings = {
 loopCount: -1,
 clearCache: true,
 disableCache: true,
 actionDelay: 8,
 stepDelay: 10,
 screenshotOnFailure: true,
 userAgent: 'flood-element-test',
}
export default () => {
 step('01_Home', async browser => {
 await browser.visit('https://www.youtube.com/watch?v=6fvhLlrBrPQI')

 await browser.takeScreenshot()
 })
}

This is the bones of a basic script in Flood Element. It consists of a single step that involves doing two things: visiting the starting URL, which in this case is the direct link to the YouTube video, and then taking a screenshot once loading the page. Taking screenshots is particularly useful when testing video because screenshots give you a way to check that your script is doing what you want it to do. It’s good to take periodic screenshots as well, to measure progress.

To help you validate that the script runs well locally before scaling to hundreds or thousands of users, you can use our Element CLI. If you save the above code as youtube.ts, open up your terminal in that directory and then type:

element run youtube.ts

Assuming you’ve got Element set up correctly, you should see something like this: (Note that screenshots are only available when using iTerm as your shell. They are also saved as files under .../tmp/element-results/youtube .) ‍

Oops! It looks like our script does navigate to the page, but it doesn’t play the video. We’ll need to click on that play button. Let’s add that to our script.

Step 2. Click the Play button

We’ll need to add some lines to click the play button after visiting the page but before we take the screenshot, so that we can verify later whether or not the play button was clicked.

step('02_ClickPlay', async browser => {
 //Click on Play button
 let playBtn = await browser.findElement(By.xpath('//button[@aria-label="Play"]'))
 await playBtn.click()

await browser.takeScreenshot()
 })

In this example I used Xpath, but you do have the option to select based on partial text, CSS selector, etc. A tool I like to use to get these is Chrome Developer Tools’ Elements tab, which allows you to get the selectors for a particular element.

Step 3. Take periodic screenshots

To make sure that the video is really playing, we’ll need to take a screenshot a few seconds after the initial screenshot in order to visually confirm that the video has progressed. Even better, we can get Element to take screenshots at regular intervals. So let’s add a third step:

step('03_Play', async browser => {
 //Take a screenshot every 5 seconds until video finishes
‍
 for (i = 0; i <= 60; i++) {
 await browser.takeScreenshot()
 await browser.wait(5)
 }
 })

To use i, we’ll also need to declare it earlier in the script:

import { step, TestSettings, By, Until } from '@flood/element'
‍
var i = 0

In this step, i will correspond to every 5 seconds of playtime for the video, so we’ll need to make sure that the limit of i that we set is sufficient time for the entire video to play. Every 5 seconds, Element will take a screenshot, which will help us verify that the video is progressing after the test.

Click here to download the script before moving on to the next step, unless you’ve been following along at home.

Step 4. Run it on Flood!

The final step is to upload it to Flood and run the test. Click here to see a step-by-step guide for how to do just that. You can choose how many users you’d like to run on Flood and across how many nodes. New users will be given 5 node hours which will allow them to run a test with up to 250 users for 1 hour. Each node will have a different IP address, so plan for that accordingly if your application takes that into account.

One of the things I love about running it this way is that if you click on the arrow to the right of the sole transaction, you can see screenshots taken along the way to prove that the video is being played:

Some Considerations for Load Testing Video with Flood Element

Taking screenshots and using logs are necessary for debugging.

Because the page elements while a video is streaming are static, we’re not going to be able to use those as clues as to where we are in our scenario. To compensate, it’s a good idea to take screenshots and use console.log() to determine progress, especially while debugging. Be careful not to go overboard on these when actually running the load test though, as they will consume system resources.

Videos may behave differently in headless mode.

While writing this tutorial, I realised that YouTube auto-plays videos when on non-headless mode (aka, when opening a video link manually on a browser or with the --no-headless flag in Element). When it’s executed in headless mode via Flood Element, it requires a click on the play button to begin. Again, use screenshots to aid you in spotting these differences.

Conclusion

Element’s browser-level scripting capabilities allow it to simulate users that interact with pages just like human users would. Element and Flood are a powerhouse combo, providing the flexibility to test with thousands of users across the world within a few minutes.

Element seeks to reduce the gap between a scripted user and a real user – and it’s so realistic that it’s hard to tell the difference. We encourage you to take advantage of our free trial offer and run some real world load tests today!

*We don’t advise or condone the use of Element to artificially inflate views on YouTube in any way which may violate its Terms of Service. The purpose of this article is to showcase Element and how realistically it recreates real users.

Manufactured Chaos: How Netflix Does Performance Testing

Wed, 28 Mar 2018 23:35:52 +0100

Originally posted here.

It’s always ideal to run performance tests against an environment that’s production-like. That way, you can ensure that your tests aren’t being influenced by factors that are not present in production - such as sub-optimal specifications, or the configurations of a test server. So, what does a good performance tester do? They compare their machine specifications and request production-sized boxes to run their tests against. But is that really enough?

Here’s the reality: despite our planning, we still sometimes see things in production that we don’t see, and can’t reproduce, in the test environment. It’s hard to predict every possible failure. Even if you do recognize the danger, it’s still difficult to recreate the perfect storm of circumstances that could potentially lead to its occurrence. The more complex your network architecture is, the more likely it is that these failures will occur. As a result, there’s something of a “don’t breathe or it might break” feeling that comes with deploying to production. So how do you test your application to get the maximum degree of confidence in its performance?

If you’re Netflix, your answer would be to lean into the chaos.

Chaos theory, in a nutshell, tells us that what looks random actually contains patterns that, while unpredictable, are recognizable and thus manageable. Far from representing disorder, chaos is our word for events that may be infinitely complex and interconnected in ways that aren’t immediately apparent. Chaos engineering takes this principle and applies it to software. Developers can code to reduce the occurrence of chaos as much as possible, but they can also code to make sure applications can withstand chaos if it does occur. As testers, one way we can test the latter is to manufacture chaos.

Netflix’s testing strategy puts a lot of emphasis on manufacturing chaos because their business model involves streaming gigabits of data to customers on demand, from anywhere in the world, at a consistently fast enough speed that people aren’t annoyed by buffering in the middle of a rousing speech by the Mother of Dragons. In 2015, Netflix accounted for 36% of traffic in North America. Netflix customers stream 125 million hours of video every day. Netflix used to serve the content from their own private data centers but moved to the cloud in 2008 after a database corruption disrupted their business for three days. Today, Netflix is 100% in the cloud, which provides tangible benefits such as architecture that can scale on demand thanks to their use of Amazon Web Services (AWS) - incidentally, the same CDN that we use here at Flood IO. However, even the cloud isn’t invulnerable to chaos. Enter the Simian Army.

Conjuring up the humorous image of a squadron of monkeys wreaking havoc in a data center, the Simian Army is a series of tools developed by Netflix that code chaos into their tests - on purpose. At Netflix, every build has to pass not just the standard suite of functional and nonfunctional tests, but also the chaotic-but-programmable tests of the Simian Army, all of which Netflix have made available as open-source tools on Github. Each monkey is a real-world manufactured-chaos test that the application under test must pass in order to be deployed into production with a high degree of confidence in its stability and continued performance.

Chaos Monkey selects a node or container within a node at random and terminates it unexpectedly, forcing Netflix engineers to adapt their code to deal with this behavior by quickly rerouting requests to backup nodes and containers.

Janitor Monkey detects unused resources (instances, volumes) in the cloud and terminates them. Ideally, all resources no longer used should be shut down, but in practice this doesn’t always happen. People forget to delete things they’re not using anymore. Applications are changed to use new volumes, and old ones are forgotten. While it’s a good idea to make sure everyone is aware of the need to clean up after themselves, Janitor Monkey assumes that things will fall through the cracks and automatically comes to the rescue. Janitor Monkey also sends a notification to the owner of the resource with enough time to mark the resource as an exception before deletion.

Conformity Monkey scans instances and determines whether they conform to a set of rules specified by the organization. If an instance is not configured the right way, Conformity Monkey sends the owner a notification. It can detect things like an instance being created without the appropriate security groups or tags.

Security Monkey monitors your entire public or private cloud to look for policy changes or incorrectly configured instances that could lead to serious security breaches. It even provides a UI so that you can see a history of changes to important policies and whom they were made by.

The Simian Army is a marked departure from traditional software testing techniques, which are very much procedural and based on predetermined flows. The assumption is that people behave logically, are all experts in their field, forget nothing, and always follow best practices. But is this actually realistic?

Netflix’s approach to testing is innovative in that they don’t assume that everyone involved knows exactly what they’re doing, from the developers to the infrastructure engineers to the testers. Neither do they assume that it is always possible to know the exact cause of failures in production. Sometimes bad stuff just hits the fan and nobody sees it coming. The very nature of our work in technology requires constant change, and it can be difficult to keep up with all the ways in which applications are becoming increasingly more integrated with each other. This will only become more apparent as the Internet of Things becomes more popular.

At Flood, we often see our customers targeting production environments, and there are some strong arguments for doing this. Testing later in the lifecycle means we can start to tackle production-sized problems around availability, scalability and reliability that would have otherwise been difficult to reproduce. Let’s face it. We all sometimes see monkeys in our data centers. Maybe it’s time we started planning for it.

References:

Completing the Netflix cloud migration What is chaos theory? Principles of chaos engineering The Netflix Simian Army Chaos engineering - surviving the failures in distributed systems Chaos engineering series - Part 1: chaos engineering deep dive What is Simian Army?

Why AirBNB Plus Could Make or Break AirBNB

Sat, 24 Feb 2018 23:24:00 +0100

Originally published on Medium.

“We’ve done the anywhere part,” Brian Chesky announced a few days ago, “but AirBNB is still, for some, an alternative way of travelling… until today.”

AirBNB Plus is a new tier of service that AirBNB have introduced for homes and hosts that go beyond the Superhost standard.

Here’s what an AirBNB Plus listing means:

The home has been verified by an AirBNB representative in person to have passed a 100-point checklist of quality. AirBNB has not yet publicly released the contents of the checklist.
The home comes with a list of extra amenities, such as a fully-stocked kitchen, wireless internet, and even paintings on the wall.
The host has received an average rating of at least 4.8 out of 5 stars.
You will be able to check in reliably at any time of the day. This could be due to an electronic keypad, safe lockbox or simply a host that’s willing to come out and let you in when you’re ready.
You automatically get premium support from a designated AirBNB Plus team in case anything goes wrong.
AirBNB has indicated that for now, only those who are already Superhosts will be eligible to request AirBNB Plus status.
According to the Verge, the host needs to pay $149 to apply to be considered for AirBNB Plus status.

AirBNB Plus listings are already available in several cities around the world: Austin, Barcelona, Cape Town, Chicago, Los Angeles, London, Melbourne, Milan, Rome, San Francisco, Shanghai, Sydney and Toronto, so you can go check them out yourself.

On the surface, this sounds like a great idea. I almost always travel with AirBNB, and when I’ve chosen to book a typical hotel, it’s always because I couldn’t find a Superhost listing to book. I’ve had bad experiences with AirBNB in the past, and have since decided to only book with Superhosts. However, even that is not a real guarantee of quality — in contrast to AirBNB Plus, becoming a Superhost has much less stringent rules.

Here’s what a Superhost listing means:

The host has received an average of at least a 4.8-star rating for all stays.
The host reponds to messages within 24 hours at least 90% of the time.
At least 10 guests have stayed at the property in the last year.
The host has never cancelled a reservation.

It’s almost entirely review-based: the requirement to maintain at least a 4.8-star rating is the most difficult hurdle. There are otherwise no quality requirements that the listing must maintain. In effect, Superhosts are community-selected, which is tricky because a string of teenagers used to hostels and leaving glowing reviews on a listing could make it appear more attractive than it actually is. I’ve stayed with Superhosts that had faulty showers, unstable internet and a kitchen that consisted of a sink and a kettle.

So I really like the idea of a higher class of accommodations that have been independently inspected. This is the first time they have ever offered anything like a guarantee for the quality of their things. Previously there was still a bit of a risk with booking an AirBNB, but this should dispel all doubt. AirBNB is doing what every company does after its initial growth spurt: it’s bunkering down, standardising its product… and, in effect, increasing some of its prices.

While price increases are not explicitly mentioned, it seems reasonable to expect that listings in this new category would fetch higher prices. After all, it costs more labour on AirBNB’s part to inspect these listings and it also costs the host at least $149, not to mention extra costs spent to meet the 100-point checklist. AirBNB Plus is no doubt a premium offering, catered towards a wealthier clientele.

Another concern is that the line between AirBNBs and hotels are blurring, and not necessarily in a good way. The original appeal of an AirBNB was that it was someone’s personal home, with all the quirks and defects that come with it. AirBNB’s overwhelming success then brought about a new class of landlords who buy properties with the specific intention to rent them via the platform. Already there are several boutique hotels that have been allowed to list accommodations as well.

The new Plus tier seems to narrow the gap between corporate hotel chains and spare rooms owned by real locals substantially. This feels less like a disruption of the hotel industry and more like a return to the status quo. I remain a huge fan of the brand, and I worry that AirBNB will lose that disruptive “plucky startup” charm — because they’re no longer a startup. They’re a major player, and if this announcement is anything to go by, they’re about to get even bigger.

Should I Upgrade to JMeter 4.0? 5 Reasons to Upgrade

Fri, 23 Feb 2018 23:43:25 +0100

Originally posted here.

Apache has just launched JMeter 4.0, a big step up from their previous release, 3.3, which was only released in September last year. Apache seems to be stepping up the frequency of it’s releases, and we’re not complaining. JMeter is an amazing open source load testing tool and the most popular one used at Tricentis Flood by far.

The big question is: Should you upgrade? Are the new features worth the leap? Will the new black UI make you a better load tester? While this choice should be made on a case-by-case basis and we certainly can’t pretend to give one-size-fits-all advice, here are some good arguments to do it:

#1: Java 9 support

JDK 9 was released back in September 2017, and if you were one of the people who upgraded to it when prompted, you would have noticed to your frustration that JMeter 3.3 did not play well with it, yielding the error:

Error: Java version -- 9.0.1 -- is too low to run JMeter. Needs a Java version greater than or equal to 1.8.0 errorlevel=3

Good news! This is now fixed. JMeter 4.0 runs smoothly on JDK 9. We’ll continue running on JDK 8 at Flood for consistency amongst all our open source load testing tools.

#2: New Boundary Extractor

Previously, my preferred way to extract dynamic values in JMeter was to use a regular expression extractor, but there’s no doubt that it can sometimes be tricky to get just the right text out of a response. Enter the Boundary Extractor.

The boundary extractor allows you to simply set the text immediately before and after the text you want to extract. This saves you wondering whether you should have used a + or a * in your regular expression, and is also easier to validate.

#3: New JSON assertion element

While XPath assertion was great, there was previously no counterpart for JSON. The JSON Assertion element fills that void nicely, using JsonPath to quickly assert the existence (or non-existence, with the Invert assertion option ticked) of a particular element.

#4: New throughput timer

We have a new throughput timer to play with, and it’s called Precise Throughput Timer.

Whereas the Constant Throughput Timer also had the disadvantage of constant pacing of requests, the Precise Throughput Timer is based on the Poisson Random Timer and randomly generates the delay between each request. The implementation seems to be well thought out, with the component reference section listing a lot of scenarios they’ve tested for that the previous Constant Throughput Timer might have struggled with. Throughput timers are one of my favourite JMeter features, and still rather unique in the industry.

#5: Test Plan Autosave

This one’s more of a usability tweak, but it’s a good one. JMeter now automatically saves your script when you run a test, saving you from those all-too-familiar “Wait… did I CTRL+S before I hit CTRL+R?” moments.

Click here to see the full release notes for version.

BONUS reason: Tricentis Flood support

You can now upload JMeter 4.0 scripts to flood.io and be up and running with your first test hosted on one of our grid nodes around the world. Our customer support team is always happy to help you make the switch to 4.0.

How I Became Financially Independent in 5 Years

Thu, 22 Feb 2018 11:40:32 +0000

Here’s my secret: I’m 32 years old and I am financially independent. And my husband and I did it all, from $0 to where we’re at now, in 5 years.

What is financial independence?

But let’s back up. Financial independence means I no longer need to work to support myself and my husband. This does not mean I’m filthy rich; this means only that if I stopped working right now, we would be able to continue living a simple life without needing to worry about paying the bills. If you actually do the calculations, that number is not as high as you might think.

Still, it’s a huge financial milestone that gives me a lot of freedom. Think about it: if you have enough money to cover my basic living expenses, why would you choose a job you don’t like? Why would you spend 8+ hours of my day, 5 days a week, with people whose company you don’t genuinely enjoy? It’s called financial independence because it’s the end of slavery — slavery to the rat race and to the whims of the corporate world. Financial independence gives you the freedom to design and pursue the life you actually want.

Where we started

When my husband Rob and I moved in together, we had a negative net worth. We had his three teenage children living with us and neither of us had a job. We lived in a small town that didn’t have many job opportunities in his industry and I, with my Australian permanent residency still in progress, was not allowed to work in Australia. Finally Rob found a job in Melbourne, and we moved the entire family there. We signed the lease on a ridiculously expensive rental house in order to get the kids in the right zone for the best high school we could afford. Despite Rob’s income, we struggled, borrowing from credit cards to finance expenses we just couldn’t seem to get on top of.

Eventually I got my permanent residency and a job. Even though my income could just barely cover the rent, that money made us feel rich. As the years went by and my salary increased, so did our expenses. We had lunch out nearly every day while we threw out spoiled groceries at home. We went to several concerts and theatres, some of them twice or thrice. We bought a car — on loan, of course. Almost our entire paychecks were immediately spent paying off the credit card bills from the last month; then we’d use the same credit cards to pay off this month’s expenses. In short, we were incredibly irresponsible. Still, thanks to dual higher-than-average IT incomes, we managed to convince a bank to give us a mortgage. Despite our incomes we could only afford a laughable 2% downpayment on the house — while we still owed money on credit cards.

Year 0 actually begins the day Rob’s current project ended. He was a contractor, so no project meant no job. Knowing we’d have to live tight on just my salary to meet the mortgage and our monthly living expenses, I started to search for budgeting tips online. What I found would eventually lead us down the road to financial independence. I took control of our finances and changed them around. Here’s what I did.

What we did

I consolidated all our accounts**.

We had several transaction accounts, savings account and credit cards, and the first step was to have fewer buckets for our money. Having fewer accounts made it easier to keep track of everything, especially in the beginning, and it also meant it was easier to calculate our (then negative) net worth and track changes to it.

I used the envelope system for budgeting to keep a lid on our daily expenses.

Specifically, I used an app called You Need a Budget (YNAB — and that’s an affiliate link), but you don’t need to buy it (although I personally think it’s more than worth it). I tracked every single cent then, and in fact I still do that today. Instead of having one account and one balance to look at, I created categories for every conceivable expense. Before I found YNAB, I just used an Excel spreadsheet. A big shift in habit was when I started to look at the category balance instead of the account balance. If we wanted to buy something that cost more than the budget for its category, we would look to see what other categories we could take the money out of. This helped us understand that every time we said yes to buying something, we were saying no to buying something else. Even if we paid using the credit card, that money still went out of the category balance so that we could at least make sure we weren’t adding to our credit card debt.

I automated transfers to pay back credit card debt and to our savings account.

I discussed with Rob an amount that we would save out of my salary every month for emergencies and I set up an automatic transfer from our normal transaction account to our separate savings account with another bank the day after payday. The automatic part is important — you don’t want to give yourself the chance to self-sabotage. This made sure that even if we went over budget, we would not touch our savings. I did the same for extra debt payments towards credit cards and our car loan until they were competely paid off. I also saved one month’s worth of household expenses as an emergency fund.

The waiting game

By the time Rob found a job again, we were debt free except for our mortgage. We suddenly had a lot of cash to spare, both from his salary and from paying off debts. As soon as that happened, rather than increasing our expenses again, I automated transfers to investment accounts instead. For us in Australia, this meant talking to our employers to salary sacrifice into our superannuation funds, for you, this could be contributing to your 401k. We started salary sacrificing as much as we could per year (at the time, that was $30,000 for me and $35,000 for Rob including employer contributions, but check the current limits with the ATO as they have changed). This was a huge deal because we never even saw the money as it was garnished from our wages. It also saved us heaps of tax (one year I calculated it to be about $8000 in savings for Rob alone). At this point we were saving 70% of our income.

When we got unexpected income increases or tax refunds, we put it towards investments. As our incomes grew, we maxed out our super contributions and started to look for something else to invest in. I invested our money into a low cost index fund, available from any stock broker. In our case, we used Vanguard’s VAS for Australian shares. Again, those payments were automated and transferred regardless of whether the stock market was doing well or not.

Then, we waited.

The result

Within five years we had nearly enough to meet our subsistence-level figure. Some surprise financial windfalls finally pushed us over the edge.

I’m sharing this story not to brag about my successes but to let people know that financial independence is not just possible but SIMPLE.

Let me break down the steps again for you:

Calculate your financial net worth.
Revisit and optimise all your accounts.
Track every single expense for at least a month.
Make a budget.
Automate transfers to improve your net worth: first towards your debt, then savings towards an emergency fund and then to an investment account.
Wait.

Now I’m not saying that everyone can do this in five years. My husband and I were lucky enough to have dual high incomes. But the point is not to do all of this in five years. The point is to start now.

English on Nicole van der Hoeven

Roundtable: How We Really Use AI in Testing (HUSTEF 2026)

Resources

Asimov's Zeroth Law of Robotics: Testing for AI (HUSTEF 2026)

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana MCP

Transcript

Resources

When to use Grafana Assistant vs MCP vs GCX: Grafana Assistant

Transcript

Resources

AI Week Day 5: Collaborating (Grafana AI Week — Friday)

Transcript

Observability and AI explained

Resources

AI Week Day 4: Evaluating (Grafana AI Week — Thursday)

Transcript

Investigations Arena: how we test Assistant investigations

Transcript

Resources

The Great Doubt: What Building an AI Agent Taught Us About Trust (KubeCon Japan 2026)

Resources

Related reading

AI Week Day 3: Operations & Maintenance (Grafana AI Week — Wednesday)

Transcript

AI Week Day 2: Building (Grafana AI Week — Tuesday)

Transcript

AI Week Day 1: Planning (Grafana AI Week — Monday)

Transcript

My reMarkable Obsidian workflow

The setup: a lot of E Ink

Step 1 — Sync the PDF into Obsidian

Step 2 — OCR the handwriting into real text

The accessible alternative: Claude Cowork / Claude Desktop

Why this matters

Resources

An Overview of AI Evaluation (The Context Window #05)

The “final answer problem”

What evals are, and online vs offline

How we test Grafana Assistant

o11y-bench

The evaluation loop

Where it’s going

Resources

Related reading

More from The Context Window

Transcript

OpenTelemetry's Graduation: What It Took (Grafana ❤️🔥 OpenTelemetry Community Call #8)

What we covered

🎓 What graduation actually took

📚 Documentation that’s actually findable

🏷️ Why semantic conventions are so hard

🧳 Signals, baggage, and stability

🤖 AI’s impact on OpenTelemetry — the good and the painful

🚀 Post-graduation priorities and how to get involved

Resources

CLI and MCP Basics for AI Agents (The Context Window #04)

Timestamps

Resources

MCP and CLI for agents

Related reading

More from The Context Window

My AI note-taking workflows

The framing: note-taking vs. note-making

Tier 1 — Free

Tier 2 — Paid

Tier 3 — Overkill

Why three tiers, not one

Resources

Grafana Campfire 🔥 — GrafanaCON 2026 recap

Resources

Asimov's Zeroth Law of Robotics: Observability for AI (ExpoQA Madrid 2026)

Resources

Zero to Dashboard with Grafana Assistant and the Infinity datasource plugin

Why Infinity is fun here

Timestamps

Resources

Do Androids Dream of Second Brains?

What I cover

🐑 Why “second brain” is the wrong framing