Performance on Nicole van der Hoeven

Entendiendo los sintéticos (Café con Grafana #008)

Thu, 11 Jan 2024 16:00:25 +0100

En este episodio de Café con Grafana les traemos a Marcelo Magallón, experto en monitoreo sintético, quien viene precisamente a contarnos que es en sí un sintético. Él nos va a mostrar algunos ejemplos, casos de uso y mucha mas diversión!

Lee la documentación sobre el monitoreo de sintéticos en Grafana Cloud aquí.

How to Get Started with Tempo (Grafana Office Hours #22)

Fri, 01 Dec 2023 11:55:55 +0100

Grafana Labs Principal Engineer and Tempo maintainer Joe Elliot talks about how to get started with Tempo, what distributed tracing is, and why TraceQL is cool. He is joined by Developer Advocates Nicole van der Hoeven and Paul Balogh.

Resources

Getting started with Tempo (video)
Grafana Tempo repo
Grafana Tempo docs
The Tempo category on the Grafana community forum
Video playlist for Grafana Tempo
Distributed tracing with Grafana Cloud k6 (Grafana Office Hours video)

Repasemos el G-stack (Café con Grafana #005)

Thu, 30 Nov 2023 13:57:52 +0100

En este episodio de Café con Grafana tenemos a Raul Marin, Solutions Engineer (Ingeniero de soluciones?) quien viene a darnos una repasada completisima de todo el stack de Grafana. TODO! Pues hay muchos que no conocen todas las herramientas, caracteristicas y posibilidades que Grafana Labs viene a ofrecer.

Performance Testing With Nicole van der Hoeven (Quality Bits Podcast)

Tue, 28 Nov 2023 14:01:35 +0100

My friend Lina Zubyte invited me to her podcast Quality Bits to talk about performance testing. We talked about what performance testing is, why it is important, how to get started, and how to get better at it. We also talked about the foundation of any type of testing: empathy.

Check out the episode here.

How to Use Flow Mode for Grafana Agent (Grafana Office Hours #21)

Fri, 24 Nov 2023 11:47:00 +0100

Senior Software Engineer Matt Durham shows us how to use flow mode for Grafana Agent. Flow mode is a new and better way to install and configure Grafana Agent than the older “static mode”. Among other things, flow mode’s modularity makes it easier to build more complex workflows like traditional data pipelines and allows for more use cases than collecting and processing telemetry.

Resources

Grafana Agent repo
Grafana Agent docs
Grafana Agent flow mode
What is Grafana Agent? (previous Grafana Office Hours video)
Grafana Slack - there’s an #agent channel
Grafana Agent Community Call agenda
Modules for Grafana Agent

Dimensionando el Grafana Suite (Café con Grafana #004)

Thu, 23 Nov 2023 13:51:25 +0100

Almudena Vivanco viene a platicarnos de todos los experimentos y pruebas a las que someten a toda la suite de Grafana. Desde Mimir hasta k6 ella pone al limite todos los elementos constantemente. Además nos dará muchos tips de su experiencia para usar Grafana y mucho mas!

Realistic Tests with Testcontainers (Grafana Office Hours #20)

Fri, 17 Nov 2023 11:40:46 +0100

In this episode of Grafana Office Hours, we talk about realistic unit tests utilizing actual instances of databases and other dependencies. We’ll be joined by Manuel de la Peña from AtomicJar, and Pablo Chacin from Grafana to learn how we can use the Testcontainers open-source framework to create lightweight, throwaway dependencies for our tests.

(ES) Cómo hacer pruebas de carga con Grafana k6 (Café con Grafana #003)

Wed, 08 Nov 2023 13:43:50 +0100

Daniel González Lopes nos viene a demostrar cómo hacer pruebas de carga con Grafana k6 desde zero, y cómo utilizar k6 para ejecutar otros tipos de pruebas. En este episodio para principiantes, hablaremos también sobre la relación entre pruebas y observabilidad.

Resources

(TL) Ano ang relasyon ng testing at observabilidad?

Wed, 01 Nov 2023 12:19:46 +0100

Ano ang relasyon ng testing sa observabilidad? Sa video na ito, pinag-uusapan namin ng kaibigan kong si Marie Cruz kung ano ang testing, ano ang observabilidad, at bakit dapat nating gamitin pero para pabutihin ang ating mga aplikasyon.

K-SIX News v0.47

Mon, 30 Oct 2023 11:57:21 +0100

In this video, my colleagues (Leandro Melendez, Marie Cruz, and Paul Balogh) and I discuss the latest updates in k6 v0.47.

Here are some of the highlights:

We’ve created an official k6 Prometheus Grafana dashboard and another dashboard for Prometheus with native histograms.
New APIs have been added for Prometheus Remote Write output for k6 (authentication and HTTP headers).
We’ve increased the character limit for metric names from 63 to 128, aligned with the OpenTelemetry specification update.
k6 browser tests can be executed in Grafana Cloud by everyone.
We’ve improved the cookie API for k6 browser.
We’ve added support for page.on('console') for k6 browser.
There are now official Docker images that include a Chromium-based browser for running k6 browser tests within CI/CD pipelines.
Distributed tracing is in public preview for Grafana Cloud.
xk6-file is a new extension that lets you rename files within your k6 script.
xk6-gcp is a new extension that adds k6 support for messaging using Google PubSub.
xk6-dashboard is now an official Grafana extension that lets you generate HTML dashboards from k6 test results.
k6x is a new utility that automatically detects when you import k6 extensions and builds the custom k6 binary for you so that you don’t have to.
k6-operator now has a Helmchart deployment option.

Real User Monitoring With Grafana Faro (Grafana Office Hours #17)

Fri, 27 Oct 2023 07:00:19 +0200

My friend Marie Cruz and I chat with Software Engineers Kostas Pelelis and Marco Schaefer to discuss what Grafana Faro is, why it’s important to have a real user monitoring solution, and how to get started with it.

Useful links

Recording a Performance Test With Grafana Cloud k6 (k6 Office Hours #93)

Fri, 30 Jun 2023 23:33:12 +0100

Senior Frontend Engineer Edvinas Daugirdas demonstrates recording a performance test with Grafana Cloud k6 and some other new features that have recently come to Grafana Cloud. He is joined by Developer Advocates Marie Drake and Nicole van der Hoeven to talk about getting started with k6 for free.

Testing Kubernetes Networking With k6 (k6 Office Hours #91)

Fri, 16 Jun 2023 13:00:50 +0200

How would you go about testing Kubernetes networking? For his initial project with k6, our new Senior Software Engineer, Roberto Santalla, talks to us about how he used k6 to improve end-to-end network throughput and some insights he learned about Kubernetes networking along the way. I also invited my colleague, Leandro Melendez, to help me interview Roberto.

How to use k6 for site reliability engineering (k6 Office Hours #90)

Fri, 09 Jun 2023 18:32:32 +0200

In this k6 Office Hours, our k6 Champion, Sarah Zipkin, chats to the Grafana Labs Developer Advocates on how a Site Reliability Engineer uses k6 to test for reliability.

How Supabase does performance benchmarking using k6 (k6 Office Hours #89)

Fri, 26 May 2023 16:19:51 +0200

In this k6 Office Hours, we talk about how Supabase does performance benchmarking using k6 with Egor Romanov, SDET at Supabase; Théo Crevon, Senior Software Engineer at Grafana Labs; and Nicole van der Hoeven, Senior Developer Advocate at Grafana Labs.

Supabase is an open-source alternative to Firebase. Learn more about Supabase here.

Intersectionality in Tech (Adobo & Avocados #03)

Wed, 10 May 2023 21:35:00 +0800

Marie Cruz and I talk about what intersectionality is and how intersecting identities can affect how we relate to the world and our experiences.

Getting started in testing (Adobo & Avocados #02)

Wed, 03 May 2023 21:22:18 +0800

Marie Cruz and I interviewed Rafaela Azevedo about her experiences in getting started in testing, what it’s like to be a woman in tech, cultural differences between Brazil and the UK, and how she became a founder of a new Blockchain startup.

How to Get Core Web Vitals in k6 (k6 Office Hours #86)

Fri, 28 Apr 2023 16:11:52 +0200

In this k6 Office Hours, we discuss how to get Core Web Vitals for your web app using k6, and how it can help you test for user experience. We are Nicole van der Hoeven and Marie Cruz, and in this livestream, we’re talking about frontend performance testing with Grafana k6.

Emergent Load Testing - rules for organized chaos

Wed, 19 Apr 2023 00:21:06 +0200

Today I spoke at KubeCon EU 2023 about emergent load testing.

Resources

Here are the slides.

Below are some links to extensions and experimental modules that I mentioned:

Grafana
k6 site
k6 repo
k6 docs
k6 browser (experimental module)
Prometheus Remote Write (experimental module)
k6 tracing (experimental module)
xk6-kubernetes (extension)
k6 redis (experimental module)
xk6 disruptor (extension)

Abstract

When we write load testing scripts against our applications, we write them sequentially: A, then B, then C. But this doesn’t accurately reflect the organized chaos of a system in production, nor does it prepare the system for the unexpected.

Emergence is a phenomenon where parts of a whole independently develop properties not originally present in the whole. Emergence is what helps ant workers develop roles without leadership, prompts animals to evolve adaptive traits without forethought, and facilitates non-toxic communities without moderators.

The growing field of emergent software applies this swarm logic to the programs that we write. What would it take to write emergent load testing scripts? It turns out that there are a few ingredients for emergence: a large population size, opportunities to interact, feedback, and an element of control.

In this talk, Nicole van der Hoeven discusses how to bring these elements to load testing by writing a script in Grafana k6 that can independently decide what requests to make next, modify Kubernetes app pods, and disrupt services based on a continual feed of results during runtime– all without manual intervention. She shows how to wield this new breed of load testing to improve confidence in the complex systems we build.

k6 Javascript Ecosystem (k6 Office Hours #85)

Fri, 07 Apr 2023 13:00:42 +0100

Did you know that there is a k6 JavaScript ecosystem on top of the Go extension ecosystem? What’s the difference between the two? What does k6 actually support? What types of things can you do with JavaScript? In this k6 Office Hours, Software Engineer and k6 maintainer Mihail Stoykov joins me to talk all about the different options for k6 available through JavaScript libraries.

Frontend vs. backend: How to plan your performance testing strategy

Mon, 03 Apr 2023 13:20:53 +0100

Originally posted here.

There are many aspects of application performance, but they broadly fall into two categories: frontend performance and backend performance. As a tester, it’s important to know the differences between the two and how that impacts the way you approach your tests.

In this blog, I’ll provide a high-level overview of frontend performance testing and backend performance testing, including pros and cons of each one. I’ll also explain why you need both if you really want a holistic view of how your application is performing.

Why (and where) to use frontend performance testing

Frontend performance testing verifies user experience at the interface level. Basically, this entails everything a user sees when they access your web app, whether it’s on a laptop or other devices. This might involve automating user interactions with the site through the browser — for example, clicking on a button, filling out a form, or having scripts that load on the browser and run.

In frontend testing, you’re looking at how components of the UI respond.

The metrics we analyze for frontend performance tests are typically quite different from what we use for backend testing. Let’s say we want to know the time it takes for a page to render after a user clicks on a button that directs them there. This isn’t the same thing as measuring server response time; that total round trip takes a lot more into account — things like latency or the time it takes for a client-side script to run and display the requested information.

Advantages of frontend performance testing

The great thing about frontend performance testing is that it provides a more realistic measurement of your end users’ experiences. They don’t care what’s going on under the hood; they care about the overall experience — and that’s exactly what frontend performance testing measures. Unlike backend performance testing, frontend performance testing looks at everything from a user’s point of view: from the time they visit a webpage to the time they can interact with something on that page.

On top of that, most (about 80%) of the performance issues end users report are frontend issues. It doesn’t make much sense to skip this type of testing; it can be a critical step in finding ways to improve the overall user experience.

Disadvantages of frontend performance testing

The most obvious disadvantage of frontend testing is that it doesn’t look under the hood. If you only do frontend testing, you’re almost certain to miss something. Maybe a component isn’t working, which is creating issues for your users. But you wouldn’t know that because you’re only testing what’s visible from the browser. In other words, you might be in the situation where you know something in the application is wrong and responding too slowly, but you don’t know what.

Frontend performance testing can also be quite expensive compared to, say, protocol-based testing. That’s because you need enough resources (memory and CPU) to actually spin up a browser instance and interact with your application the way that a real user would. The same load generator that could run tens of thousands of backend virtual users may only be able to run 50 frontend virtual users before running out of resources.

Finally, only testing frontend performance can sometimes lead to pretty misleading results, especially as you scale. As load increases, the response time for the frontend remains fairly constant, but that’s not the case with the backend.

While the frontend component of response time remains more or less constant, the backend component of response time increases exponentially with the number of concurrent users.

This phenomenon occurs because most of the issues that frontend performance tests might identify, such as a slow-running client-side script, don’t tend to compound. It’ll be just as slow on each user’s computer regardless of how many users are running it. However, problems uncovered by backend performance testing are on the side of the application, not the users. If a database can’t keep up with 100 users sending queries to it, then 1,000 users sending it queries is likely to yield an even worse response time.

Why (and where) to use backend performance testing

As you might have guessed at this point, backend performance testing targets the infrastructure under the hood. This could be a database that contains customer information, a payment gateway, or an authentication service for customers that have already signed up. These are things that an end user doesn’t see and maybe isn’t ever even aware of, but they still have an effect on response times.

Backend testing addresses infrastructure users might not even be aware of, like different databases, gateways, and services that underpin an application.

Scripting against the backend is quite different from scripting against the frontend. Instead of scripting user interactions like clicking, you’re scripting certain requests that are being sent on the protocol level. So, for example, you could have an HTTP GET request for the page and also for its embedded resources.

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
 http.get('https://test.k6.io');
 sleep(1);
}

The sample Grafana k6 script above is an example of a simplified backend performance test. It makes a single HTTP request to retrieve the contents of https://test.k6.io, and then waits one second.

Backend performance includes things beyond just load times and latency. It includes scalability, availability, reliability, and similar attributes, so the metrics that we measure are different as well. We might look at server uptime or response times per component. We could also trace the path of a particular request through the application stack.

An illustration shows that performance testing can address scalability, elasticity, availability, latency, resiliency, and reliability. Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Advantages of backend performance testing

As we already discussed, frontend testing doesn’t cover everything, but it’s also more complicated than that. The 20% or so of performance bottlenecks that are backend-related can be the hardest to troubleshoot, identify, and fix. Even worse, a lot of backend performance issues can spiral out of control at higher levels of load and can be quite expensive to fix later on in a development cycle. The worst time to realize that you need to rethink your entire infrastructure is after it’s already failed in production. If you take care of backend performance testing earlier (commonly known as shifting left), it makes such pivots less costly.

Another advantage of backend testing is the ability to target different components of an application. You can choose which API endpoint to hit and how hard, and you can target specific user flows, so it doesn’t always have to be end-to-end testing.

And finally, backend performance tests tend to be less resource-intensive; they’re just protocol-level requests that can be easily ramped up.You don’t need to spin up a browser instance, or even parse through the Document Object Module (DOM) of a webpage or look for specific elements. It’s just a matter of sending these requests, and that’s something that you can increase the scale of pretty cheaply.

Disadvantages of backend performance testing

Backend performance testing won’t tell you anything about what’s happening on the frontend, and a lot of sites these days actually use scripts that have to run on the frontend. So with backend performance testing tools, you just download those scripts. But the response time they measure is the download time, which is not really realistic because those scripts are executed. A backend performance test would fetch the text of the script. A frontend performance test would run it — and that script might do something like perform a calculation. You need frontend performance testing to measure that properly.

Backend performance testing is also much more expansive than front end performance testing, especially with microservices-based applications. There are so many components to track that it can be difficult to determine where to look. You can quickly find yourself going down a rabbit hole of constant, never-ending performance tuning.

Frontend + backend testing for the win

So, as we’ve seen, frontend testing and backend testing both seem to have advantages and disadvantages. So which one should you use? Both, if at all possible! They look at different parts of application performance, so doing both is essential if you want a holistic view of user experience.

Software teams can use Grafana k6 to address frontend and backend performance testing.

If you’re operating at a smaller scale, use your own discretion about how to find the right mix for needs. But if you’re testing at a larger scale with thousands of users and you want to test both the frontend and backend aspects of application performance, I suggest you change your test strategy so the majority of the load is generated with backend performance testing scripts. That’s because of the resource intensiveness of frontend performance testing scripts, which we discussed earlier.

So, if you generate most of the load with the backend and then you have a handful of frontend performance testing scripts running in the background, you should still get a good idea of how both are performing. Remember, frontend application performance remains fairly constant throughout different levels of load, so you’ll still get a lot of useful insights on both types of performance without worrying too much about the scaling costs.

Frontend and backend performance both affect what your end users think of your application. If the underlying application servers respond immediately to all requests, but then a huge picture of a crocodile makes your homepage excruciatingly slow to load, then users aren’t going to like that.

On the other hand, if your homepage is really sleek and quick to load, but the application server hasn’t responded with the data that the browser needs to render, then users aren’t going to like that either.

A common problem when testing both frontend and backend performance is script duplication: because of the differences in the two approaches, you often have to use two separate tools — which means two test suites and potentially two scripting languages. While this is a valid approach, it’s more sustainable to look for a tool that can target both types of performance, such as k6.

Whichever tool(s) you end up using, testing both frontend and backend performance is going to give you the clearest view of how your application really performs.

The easiest way to get started with k6 is through Grafana Cloud k6. The free tier includes 500 virtual user hours/month for k6 testing and more. Sign up for free now!

How we got into tech (Adobo & Avocados #01)

Wed, 29 Mar 2023 14:54:43 +0100

My friend Marie Cruz and I have been wanting to talk about a few topics related to diversity in tech, and over the few years we’ve known each other, we’ve only added to that list. We finally decided to take a leap and start a new YouTube channel/video podcast just so we have a space to have those discussions. Here’s the first one, where we talk about how we got into tech.

Below is a transcript of our conversation.

Why “Adobo & Avocados”?

MARIE: All right, we’re live. So, hey everyone, my name is Marie Cruz and with me I have my colleague today.

NICOLE: Hi, I’m Nicole van der Hoeven, I’m her colleague. And I’m a little sick, but I’m here.

MARIE: So we, today we’re actually gonna be doing the first ever recording for the Adobo and Avocados podcast. So we thought about this name and initially like Nicole and I wanted to, you know, ever since before we wanted to launch a podcast where we can talk about, you know, our experiences working in tech and all other things. And then we thought about the name Adobo and Avocado. So you know, to anyone who doesn’t know, so we’re both from the Philippines, and Adobo is the national dish of the Philippines. So that’s why we thought we’ll add the name of Adobo in our podcast title. Avocados, ‘cause we’re both Developer Advocates. But yeah, I think, why don’t we discuss like, why was it called avocados in the first place? Do you have any idea?

NICOLE: Yeah, well, we don’t really know for sure. So I thought that it was something to do with the language of it. Like, you know, an advocate is usually like a, could be like a lawyer in common non-tech senses. And the translation for that is abogado in Spanish or Tagalog actually,

MARIE: Even in Tagalog.

NICOLE: our native language.

MARIE: Yeah.

NICOLE: So it’s like kind of similar to avocado. I don’t know, that’s kind of what I thought.

MARIE: Yeah, that’s actually an interesting thought. Like, ‘cause the abogado does sound very, you know, similar to avocado, but I actually read this one article where I think their project manager had a hard time saying Developer Advocate, they ended up saying developer avocado. So who knows what the origin is. But yeah, I think it’s a great, you know, you know, term in terms of like, you know, the whole play, like playful, you know, meaning behind the word Developer Advocate.

How we got into tech

MARIE: So yeah, I guess first things first, like you know, who we are? Like, why are we doing this podcast? How, you know, we got into tech? So we’ll talk first about, I guess a bit of some introductions about ourselves. So Nicole, why don’t you start, like how did you get started? You know, what’s been your, I guess, background so far? Did you have a traditional, you know, computer science background or did you do like a different, I guess course back when you were still studying?

Nicole’s background

NICOLE: I did not have a traditional background. I grew up in the Philippines. I moved away when I was 18. And I don’t know what your experience was, Marie, but mine was that in the Philippines there are still, I think we do a pretty good job at being equal between genders, but I think there’s still gender roles. And when I was growing up, I really loved both the maths and sciences and like literature and languages, but I was very heavily pushed towards the humanities, ‘cause that’s what girls do. Girls don’t go into tech.

MARIE: Mm-hmm.

NICOLE: And so I really struggled at uni and in college. I shifted majors six times. I had no idea what I wanted to do.

MARIE: Wow.

NICOLE: And I went through, and that was over two countries. I went to the US to study as well. None of them was computer science because my brother was a developer. And I don’t know, he’s a little too introverted for me. And I just had this idea that, if I go into computer science, I have to be a developer, and if I’m a developer, I won’t talk to anyone. I’m just going to be

MARIE: Yes.

NICOLE: with a machine. Like I had that in mind, which now I know is completely wrong,

MARIE: Yeah.

NICOLE: but that’s why I never went into it. So finally my parents were just like, we don’t care what you finish,

MARIE: Yeah.

NICOLE: we don’t care if it’s basket weaving, just pick a major and finish it.

MARIE: Yeah.

NICOLE: And that was economics,

MARIE: Oh.

NICOLE: which I have never used in a job.

MARIE: Yeah.

NICOLE: And then so-

MARIE: That’s really interesting. Yeah.

NICOLE: And then I, how did I get into IT? I was applying for different economics related ones, and then I met my now husband and he,

MARIE: Hi Rob.

NICOLE: he was a test manager. Yeah, hi Rob! He was a test manager and I said, well what’s a test manager? And then he told me about the world of testing.

MARIE: Yeah.

NICOLE: And I didn’t understand it at first, but I wanted to know more. And then I thought the way that he described it sounded so interesting, like the idea that,

MARIE: Mm-hmm.

NICOLE: that there is a job, there is a person whose job it is to be the one using applications or products first. And then trying to see what’s wrong with it for the purposes of improving it. I just, I really loved that idea and it was another branch of tech that I didn’t know existed.

MARIE: Yeah.

NICOLE: So I started to, I did an ISTQB certification just to like have something

MARIE: No background.

NICOLE: ‘cause I didn’t have any background. Yeah. And I started job hunting, and somehow managed to land a performance testing job under someone who became my mentor.

Marie’s background

MARIE: Hmm, nice. Yeah, it’s really relatable because like similar with myself, I fall into tech accidentally. Like it was never my intention, you know, okay, I’m gonna study computer science, although I did actually have a computer science degree.

NICOLE: Oh, wow.

MARIE: But the story, the story behind that, I don’t share it too much to other people ‘cause I think the story behind it is a bit funny. But my first experience with tech is actually updating my MySpace profile. Do you remember MySpace?

NICOLE: Yes.

MARIE: Yes so, before you can customize your profile with a bunch of HTML, CSS codes and like I think before I was just trying to, you know, create something even though it didn’t look very accessible. I remember I had lots of animations, just lots of like

NICOLE: Oh.

MARIE: striking fonts, background music, and it was just something that I was doing for fun. I did a mass communications course back home in the Philippines. So I was majoring in digital media. And then I had to go back here in the UK. So I was working first for about two years before I decided to go back to uni. And during that time when I was looking at, you know, courses catalog, ‘cause I wasn’t really sure what to pick. So I saw a course which basically has three different things. So it’s computer science with business management and financial accounting.

So I thought,

NICOLE: Oh wow, okay.

MARIE: I was like thinking in my head I think that sounds like a really cool name for a course, so I’m gonna go for it. So to me I just found the name like very interesting. And it means that-

NICOLE: So when did you

MARIE: Yeah?

NICOLE: move out of the Philippines?

MARIE: So I, so we went back 2008. So I was in the UK since 2004. Actually no, we went back 2006, and then I went back in the UK 2008. So 2006 I enrolled, you know, to study mass communications. I didn’t finish the course ‘cause I had to go back in the UK after two years because of like visa, like status. And then when I got back, I just put that into a pause, you know, did some work experience. So I was working as an office assistant for about two years and then my friend and I just decided, okay, I think we should go back to uni. So I actually applied as a mature student ‘cause I didn’t, you know, had any like actual qualifications ‘cause I never graduated from my degree in the Philippines. I never finished high school as well, ‘cause I kept going back and forth between the UK and the Philippines. So the last time I actually graduated was during middle school.

NICOLE: Oh, wow.

MARIE: but applying as a mature student, they just looked at my, you know, work credentials, like my work attitude. And I submitted some of my grades like back, you know, when I was still in the Philippines ‘cause like my first two years I had some, you know, grades from like the different subjects that I was doing. But yeah, that’s how I got into tech. I just found the name like really interesting and I thought before I could mix and match it with business and accounting. So even if the computer science bit didn’t work out, like I can still maybe utilize some of the things that I’ve learned on the business side and on the accounting side. And then, when I was about to look for graduate jobs, I got approached by a testing consultancy company. I didn’t, I wasn’t aware that there’s a graduate job for, you know, just doing testing ‘cause most of the graduate jobs that, you know, we were being offered is, you know, how to, you know, do like development, or like database like management roles. So initially I wasn’t really sure if I wanted to do the developer route. Like I enjoyed the coding, I was doing the coding bit, but like back then I wasn’t like fully 100% sure that it was the right path for me. So I found out that, yeah you can have a career in testing. Again this is all accidental ‘cause if that testing company didn’t approach me, I would’ve probably looked at like other roles. So it was like really great to hear that, yeah, there’s like a graduate jobs for just doing testing. And yeah, we were doing the ISTQB as well. So we were doing some training. And yeah, the rest is history. It’s funny because I didn’t intend to continue on a developer route, but then now I work as a Developer Advocate.

How did we become Developer Advocates without being developers?

NICOLE: Yeah, maybe we should talk about that a bit ‘cause I also don’t consider myself as a developer even though I can code.

MARIE: Mm-hmm. Yeah.

NICOLE: And yet somehow we’re both Developer Advocates. How did that happen?

MARIE: Yeah. It’s like there’s some sort of, I guess, interesting link between software testing and developer advocacy. I don’t know like if it’s the same for you, but like with me, I really enjoy the teaching part, the learning and public part as well. And when I was doing my role as a software tester, I felt that I was doing the advocacy, but maybe, you know, not really in a way that it’s very out there. So I was, you know, helping my colleagues, I was talking about tools that I’ve learned, you know, with them, I was showing them how to use it. And then while I was working as well, like I started doing the learning in public on the side. So I started with my personal blog and then getting involved, you know, with meetups to talk about, you know, the different tools that I’ve learned. So I guess for me, like being a software tester also, you know, encouraged me to, I guess be a Developer Advocate ‘cause I was just telling people what I’ve been doing as a software tester. I’ve been telling them what different tools I’ve been using, you know, different concepts that I’ve been learning as well on like a day-to-day. So is that the same for you?

NICOLE: For me it was more because I was a consultant,

MARIE: Mm-hmm.

NICOLE: which means I was going on different projects all the time. And when you’re really new in an industry, I think it’s really exciting and you’re learning a lot. And certainly every project has like its own framework and different languages and different people, which is interesting for a while.

MARIE: Yeah.

NICOLE: But I don’t know, after several years of it, I started to feel like, people are actually asking the same questions. You know, there’s always, there’s some things that I always had to explain. You know, I always had to, for example, fight against the fact that performance testing is often seen as just something where you run the test, okay, you check the box, you’ve done performance testing.

MARIE: Uh-huh.

NICOLE: There’s always a certain amount of education where I’m like, no, it’s not just about checking the box, it can actually provide real value, here’s how.

MARIE: Yeah.

NICOLE: And I just found myself repeating a lot of the same things. And so I actually started to create videos also because load testing results are kind of difficult to explain sometimes.

MARIE: Yeah.

NICOLE: And I would create this like test summary report that lots of people don’t read.

MARIE: Yeah.

NICOLE: And so, as kind of an alternative way to explain things, I started creating video summaries to go along with it. And then I just thought, well maybe I can, ‘cause I can’t publish those, because it’s very specific to that company.

MARIE: Yeah.

NICOLE: But then I, I thought maybe I can pull out, I can abstract away some things and then publish those videos where I’m not talking about a specific project. And I guess that’s kind of how it started. I started, I put it into text first, because that’s just easier than video.

MARIE: Mm-hmm.

NICOLE: And then eventually found myself into video anyway.

MARIE: Yeah.

NICOLE: So I guess for me, I got into Developer Advocacy because I was tired of, I was trying to automate a part of my life, the explanation part where I kept having to say the same things and I just wanted to be like, hey, read this blog post where I explain why you need think time in a much more coherent and considerate way.

MARIE: Yeah.

NICOLE: So that’s, that’s how it naturally happened. I think you and I are the same in that regard that we were already doing it before we got the title.

MARIE: Yeah. I totally relate to that. Like, especially when you said that like, we’re just trying to solve the same problems. I started, you know, my blogging journey just trying to communicate like, you know, questions that people have been asking me on a day-to-day basis. And rather than just, you know, speaking to individual people and repeating the same things like again and again, like just put everything into a post or like you said like a video, and then just share that, you know, like to the public, ‘cause you’re not only gonna be able to help like one person but you know, maybe if someone else stumbled upon your blog or your video, then like, you might help them as well. I think like one of the, like, one of the best reward that I experienced when doing this like learning in public and you know, sharing more content out there is if you just get a message from someone or like a comment and then they say that, oh, like, I’ve been stuck with the same issue, I found your blog post or your video and it really helped me. And just that feeling that you know, you’ve helped someone is like a really great feeling. ‘Cause at first I wanted the blog as well just so that I can have something that I can look back to in case I, you know, forget like what I’ve done on a previous project. Like I can document it on a blog post, but then like now it’s also helping like other people who are experiencing the same like issues at work.

NICOLE: Yeah, I definitely see that. At first I was doing these things for myself because you know, you don’t really know that you know a subject until you can teach it to some degree. And I really believe that. And I, for me there were so many things ‘cause as a consultant, you have to switch tools very quickly, you have to switch languages, you have to be a polyglot on a lot of different levels and it’s impossible to keep all that in your mind. So after every project I tried to like, distill my learnings from it, so that I,

MARIE: Yeah.

NICOLE: future me would come back to that framework, or tool, or whatever, and not be starting from zero. But yeah,

MARIE: Yeah.

NICOLE: the kind of other people finding it was an unexpected boon.

MARIE: Yeah. So let’s just move on a bit. So we’ve talked about, you know, how we transition from software testing to developer advocacy. But I guess in terms of like the process, like did you apply to a lot of, you know, companies? Was developer advocacy something that, you know, you were aware of already, or like you just found a job description and then you’re like, oh, this is what I do already. Okay, let me apply to that job. So what’s been like the process for you?

NICOLE: So I was, this is only, this is my first job as an actual, like the first job where I have Developer Advocate in my title.

MARIE: Yeah.

NICOLE: My previous job was with a, well at the time it was a startup, it was Flood.io, which is now acquired by Tricentis.

MARIE: Yeah.

NICOLE: And when I joined, I think I was like, I think the fifth person that they hired in the team.

MARIE: Oh, wow.

NICOLE: So it was very small. And it was, I really love that environment because you can do a lot of things, you can wear multiple hats. And that’s really where it started. On a typical day, I might be answering questions in support, or presenting at a conference or ducking into a sales call or like,

MARIE: Yeah.

NICOLE: there were so many things that I could do. I was like part of the rotation for things going down basically, for things exploding. And I had to learn a lot about infrastructure as well to be competent in that role. And I started doing a lot of advocacy type things, mainly from a support perspective

MARIE: Yeah.

NICOLE: because I got the same questions there as well. And I wanted to be able to focus on the crunchier topics.

MARIE: Yeah.

NICOLE: And you can’t do that unless you address some of the more basic or common questions. And then my manager at the time told me about Developer Advocacy

MARIE: Yeah.

NICOLE: and my immediate response was, but I’m not a developer.

MARIE: Yeah.

NICOLE: He said that that’s just the name for the whole industry that I think you would be really good at and that you’re already doing. And with his encouragement, this is Tim Koopmans, the one of the co-founders of Flood, with his encouragement, strong encouragement by the way, things like, he said, oh, I’ve got this conference in Moscow that I’m speaking at. Do you wanna come watch me?

MARIE: Right.

NICOLE: And I said yes. And he backed out so I had to do it, things like that. Because of things like that that he did, I started to understand what developer advocacy is

MARIE: Mm-hmm.

NICOLE: and when it was, when I felt the time to change directions, I specifically looked at k6 actually. I didn’t apply to any other jobs because I was pretty happy where I was.

MARIE: Mm-hmm.

NICOLE: But I really loved k6. I had already not used it in a project, but I’d already, I knew what the tool was and I used it, I tested it a bit. I love the site, I love the principles behind it. And I saw that they had a Developer Advocate role. So,

MARIE: Yeah.

NICOLE: It just kind of happened. How about you?

MARIE: Well the first time I heard about the term was from you, when you said that, hey we’ve got this role for, you know like, you know like, we’ve got a Developer Advocate role for one of the tools that, you know, we’re gonna launch. Like, you’ll be like the great fit. So I didn’t really send any CV. I don’t think I sent any CV or like, yeah, I think, you just like, ‘cause we just connected already like via this Automation Guild conference. So like we were already connected via Twitter. And I think this is where like one of the benefits of like learning in public really came into play, ‘cause I was just, you know, publishing constantly. I was, you know, talking at various meetups and conferences already. But I was just doing all of this like as part of my, you know, side projects. Like it wasn’t part of my day-to-day role on my previous job. So when you explain to me what a Developer Advocate is, I’m like, oh yeah, I’ve been doing that on the side. So that was my first actual, you know, experience of like hearing what a Developer Advocate, you know, does on a day-to-day. ‘Cause I had the same perception that I’m not a developer, how can I be a Developer Advocate? But I think because as like as part of our day-to-day jobs, like even if we’re not writing application code, we’re still writing, you know, test scripts. we’re creating test automation frameworks. So I think that’s really helped as well transitioning into this role. But yeah, like I never sent any CV. I just remembered like we were talking a lot like back and forth and then I met up with, you know, a bunch of our colleagues now. And yeah, the rest is like history again.

NICOLE: Yeah, I think that there are just some people that when you meet them, you click or you don’t.

MARIE: Mm-hmm.

NICOLE: And when that click happens, I’m just like, okay, I’m gonna write her name down. I think I told you about it like for two years,

MARIE: Yeah, I just joined-

NICOLE: we were talking, and I was like, hey, this could happen…

MARIE: Yeah.

NICOLE: Yeah, I don’t know when. I think you’d be perfect for it. And you’re like, okay,

MARIE: Yeah.

NICOLE: but it’s kind of early to be telling me, right?

MARIE: Yeah.

NICOLE: I was like, Yeah, yeah, but let’s just keep in touch. You know, and it worked out. I can’t believe it actually did.

MARIE: Yeah. And now we’re, doing.

NICOLE: We didn’t actually-

MARIE: Yeah, go on Nicole.

NICOLE: We haven’t actually defined what a Developer Advocate is.

What’s a Developer Advocate?

MARIE: Yeah, I guess like we can both, you know, share what our, you know, view is of a Developer Advocate. So to me when it comes to developers, I guess trying out, you know, new tools or learning about new things, they feel most comfortable talking to someone who’s actually used, you know, the product, or you know, the tool that they want to use. So, we are, you know, that person. So we’re helping developers, we’re helping testers when it comes to, I guess learning about best recommended practices, learning about different tools. So we’re kind of like a bridge between the community and like the company that we work for. There’s a lot of misconception that people think that we’re salespeople but we’re actually not, like it’s a completely different, you know, role. We are focused more on the community side, helping them grow, helping them, you know, I guess in some ways, like be some sort of champions as well, so they can also promote like, you know, the different tools, the different practices that we have been communicating to them. But yeah, what about you? Is that like sort of the same concept?

NICOLE: Yeah, I agree with everything you said, especially about being a bridge.

MARIE: Mm-hmm.

NICOLE: Because as a Developer Advocate you’re not quite a developer, that’s not our primary responsibility,

MARIE: Yeah.

NICOLE: but we’re also not like marketers or sales people. Although there’s an overlap there in both. I kind of think of it as anti-marketing. So a marketer is going to be talking about something that they themselves have never used, are not the target user for.

MARIE: Yeah.

NICOLE: And might not know much about in terms of the technical details.

MARIE: Yeah.

NICOLE: A Developer Advocate is someone who actually is the target user. So for those who don’t know, Marie and I work at Grafana Labs specifically on k6, which is a performance testing tool. But we have both used k6, we have both been testers and still are. And so there are things that we can suggest. Part of it is advocating internally.

MARIE: Mm-hmm.

NICOLE: We advocate for testers everywhere to the product team so that we say, like these are the features that we would be looking for as testers. Like we would say it this way or we wouldn’t really use that feature, we want this other one. But another part of it is advocating externally. So this is the part that people think of as marketing, but I think of it as anti-marketing, because we’re not going out there and saying, hey, you should use this product that we’ve never used before.

MARIE: Yeah.

NICOLE: Instead, we build cool things and demonstrate cool things that we have done, not some other developer.

MARIE: Yeah.

NICOLE: Like we do our demos,

MARIE: Yeah.

NICOLE: And then when people ask us, then we say, oh yeah, we use k6. So I don’t know, marketing definitely has its place, but for me I am so addicted to that authenticity and genuineness that comes from just talking about a tool that you actually love, which is,

MARIE: Yeah.

NICOLE: what k6 is for me.

MARIE: Yeah. I really like that because for me, an important part of, you know, this advocacy is, like communicating something that I truly believe in. Like I’m not going to advocate for, you know, a tool that I haven’t used. Like I would just feel like, you know, I’m a fraud if I do that. So if I know that, you know, there are situations where a certain tool or a certain practice, you know, would work better, then I would advocate for, you know, that as well. So it’s not just about, you know, for example, like at the moment we can’t do k6 for accessibility and I’m an accessibility advocate as well. So I would go and, you know, recommend like other tools and practices based on what like I’ve experienced, based on what I’ve done. And I think, like for me, that’s one of the biggest like advantage, ‘cause yeah, if I’ve been put on the spot and say, hey, you should talk about this to then promote it to other people, that would just totally go against my, like one of my beliefs.

NICOLE: I think I also really appreciate that this position lets us shine a light on people like us who maybe,

MARIE: Mm-hmm.

NICOLE: who don’t look like the average person in tech. I know it’s still, I’ve been lucky to work with teams of mainly men that haven’t treated me any differently because I’m a woman, but there is still like a sense of being Other.

MARIE: Yeah And I guess that-

NICOLE: And being in a position of advocacy means that we’re, we’re also able to appeal to other women who maybe didn’t, who didn’t have as good an experience as we did.

What can you expect from Adobo & Avocados?

MARIE: Yeah. And I guess that really leads to this next question. And you know, people who will watch this in the future might also ask like, what are the expectations like for this podcast? Like why are we doing this? So do you want to maybe start.

NICOLE: Why ARE we doing this Marie?

MARIE: Yeah. Oh, yeah. So do you wanna start like from your side, like what are the expectations that you know, that we can, yeah, share to people?

NICOLE: Sure. I think that you and I both do a lot of technical advocacy. We, you know, do demos, we write scripts, and we tell people how to do things in tech, very technical things. But, I think that there’s a whole other side of tech that doesn’t get talked about and that’s like, things like interpersonal dynamics or cultural differences. You and I are both Europeans that have to work in, in our situation, predominantly American space, and we’re also Asian. So balancing the two, you know, we are on different sides of the spectrum on that. Just also the general stress that comes with tech. It is not easy, not just as a Developer Advocate, but just to be in tech to keep up with changing trends.

MARIE: Mm-hmm.

NICOLE: I feel like there are so many emotional aspects and psychological aspects that don’t get talked about that I would really like to cover.

MARIE: Yeah.

NICOLE: Then also, I kind of alluded to it. I would like to make it a little bit more open, an open discussion for particularly underrepresented people

MARIE: Mm-hmm.

NICOLE: like ourselves. We identify with several minority groups.

MARIE: Yeah.

NICOLE: And maybe help them along a bit by telling them things that we had to find out the hard way early on in our career. How about you?

MARIE: Yeah. Yeah, I agree with all of those. And I guess it’s also to just have a safe space, you know, for everyone just to share their experiences in tech. Like you said, there’s this whole other side of tech that we don’t talk about that maybe other women are also uncomfortable to talk about, you know, with like, I guess other colleagues, especially if it’s just predominantly male. So we want to, you know, have a safe space for them, share, you know, I guess from their side, like their unique experiences as well. And maybe like we can all have, I mean maybe we also have those shared experiences that we don’t talk about, but then when someone says, oh yeah, I’ve experienced that too. So it’s just really finding, you know, shared experiences that we can all relate to. And like you said, just helping them, you know, or empower them to, I guess move forward in terms of like the next stage in their career. Like, you know, share what’s worked for us, share what hasn’t worked for us. ‘cause I think it’s all about like being authentic as well and not just talking about the good things, but we could also talk about the bad and the negative things as well.

NICOLE: I’m excited to see where we go from here.

MARIE: Yeah.

NICOLE: I think we have a lot of ideas about where we’re going to take this and I hope people will come along for the ride.

MARIE: Yeah. So, yep. I guess this is it for the first episode. We are just, you know, extremely excited to do this on a regular basis. So we’ll talk about different, you know, topics in every episode. And we might, you know, have like really awesome, like guests as well in future episodes. So yeah, I guess just stay tuned and yeah, look out for Adobo and Avocados.

MARIE: Thanks everyone. See you next time.

NICOLE: Bye-bye.

Grafana Cloud k6 (k6 Office Hours #83)

Fri, 24 Mar 2023 18:00:00 +0100

With the release of Grafana Cloud k6, you can get observability and performance testing in the same platform. Senior Product Manager Mark Meier and Frontend Lead Simon Legander join me to talk about what Grafana Cloud k6 is, how you can use it, and demonstrate what it can do to improve your system’s reliability.

Introducing Grafana Cloud k6

Tue, 21 Mar 2023 15:20:58 +0000

Grafana Cloud k6 is a feature available to all users of Grafana Cloud. It allows you to run load and performance tests right from the comfort of your Grafana Cloud instance, without needing to create a separate account on k6 Cloud. Grafana Cloud k6 helps you get started quickly with a GUI Test Builder or with test scripts written in JavaScript, and then visualize your test results in real-time, alongside your other Grafana dashboards that monitor your system infrastructure. With Grafana Cloud k6, you can get a holistic look at your system’s performance– on any environment.

Learn more about Grafana Cloud k6.

Experimental modules in k6 (k6 Office Hours #82)

Fri, 17 Mar 2023 16:00:35 +0000

Experimental modules in k6 were released in v0.40, and new modules for distributed tracing and browser testing have been added since. In this video, k6 Senior Software Engineer Théo Crevon talks to Developer Advocates Marie Cruz and Nicole van der Hoeven about what experimental modules are and how to use them.

How to Do Exploratory Testing for Performance (k6 Office Hours #79)

Fri, 24 Feb 2023 23:01:26 +0000

Marie Cruz and Nicole van der Hoeven talk about what exploratory testing is, why you should do it, and how to do exploratory testing to improve application performance. They discuss some heuristics to add structure and intention to exploratory testing for both frontend and backend performance.

Integrating k6 and Tracetest (k6 Office Hours #77)

Fri, 03 Feb 2023 08:00:15 +0000

Tracetest enables trace-based testing with assertions at any point within a request transaction. Using the xk6-tracetest extension, your k6 tests can launch deep testing by unlocking the power of OpenTelemetry.

Join us as Developer Advocates Nicole and der Hoeven and Paul Balogh are joined by founder Ken Hamric and software engineer Oscar Reyes from Tracetest.

Check out Tracetest:

site
xk6-tracetest: repo for k6 extension
docs on how to use Tracetest with k6

Frontend vs Backend Performance Testing

Wed, 01 Feb 2023 22:59:18 +0000

Which should you do: frontend vs. backend performance testing? The short answer is both. Each has its pros and cons, but hybrid load testing combines them and gives you the most holistic view of performance, including frontend concerns like how quickly a page renders or perceived application performance as well as backend concerns like response times of individual components.

How to Organize Testing Scripts in k6 (k6 Office Hours #76)

Fri, 27 Jan 2023 17:00:10 +0000

In this video, we talk about how to organize testing scripts in k6 using features like tags, groups, and functions depending on what type of testing you want to do with your test script. Developer Advocates Paul Balogh, Marie Cruz, and Nicole van der Hoeven show us a few examples of how they would organize and modularize scripts in a test suite.

Introduction to Performance Testing

Wed, 25 Jan 2023 22:28:43 +0000

In this introduction to performance testing, we talk about how performance is part of the functionality of an application, why you should do performance testing, and some myths about performance that discourage teams from testing for it and make applications more likely to fail and fail miserably.

Guide to Load Testing Websites

Thu, 12 Jan 2023 17:12:42 +0000

Original article here, written for k6.io

This doc explains some key concepts about load testing websites, including:

The difference between backend and frontend performance testing
When to choose between protocol-based, browser-based, or hybrid scripts
Recommended practices about how to load test websites

All load tests try to simulate real user traffic to prevent failures, improve reliability, and release new code with confidence. But your approach to load testing must adapt to the type of application you want to test.

In this guide, learn about strategies to test websites, including specific recommendations for complex scenario scripting and test execution.

Load testing approaches

When you approach the load test, first consider the following perspectives:

Backend vs. frontend performance
Protocol-based, browser-based, or hybrid load testing
Component testing vs. end-to-end testing

Backend vs. frontend performance

Performance has a significant influence on the user experience of a website.

For example, when users want to visualize or interact with some information, they expect the website to react quickly.

To measure performance, testers often look at response times, which are affected by two main factors:

Frontend performance
Backend performance

Frontend performance

Frontend performance testing verifies application performance on the interface level, measuring round-trip metrics that consider how and when page elements appear on the screen. It’s concerned with the end-user experience of an application, usually involving a browser.

Frontend performance testing excels at identifying issues on a micro level but does not expose issues in the underlying architecture of a system.

Because it primarily measures a single user’s experience of the system, frontend performance testing tends to be easier to run on a small scale. Frontend performance testing has metrics that are distinct from backend performance testing. Frontend performance tests for things like:

Whether the pages of the application are optimized to render quickly on a user’s screen
How long it takes a user to interact with the UI elements of the application.

Some concerns when doing this type of performance testing are its dependency on fully integrated environments and the cost of scaling. You can test frontend performance only once the application code and infrastructure have been integrated with a user interface. Tools to automate frontend testing are also inherently more resource-intensive, so they can be costly to run at scale and are not suitable for high load tests.

Backend performance

Backend performance testing targets the underlying application servers to see how they behave under production-like conditions. When it comes to websites, while frontend performance involves how assets included in the page are rendered, backend performance focuses on how those assets are processed by the application servers, served to users, and downloaded by browsers.

Backend testing is broader in scope than frontend performance testing. API testing can be used to target specific components or integrated components, meaning that application teams have more flexibility and higher chances of finding performance issues earlier. Backend testing is less resource-intensive than frontend performance testing and is thus more suitable for generating high load.

Which one should you test?

It depends! Ideally, both.

Frontend testing tools are executed on the client side and are limited in scope: they do not provide enough information about backend components for fine-tuning beyond the user interface.

This limitation can lead to false confidence in overall application performance when the amount of traffic against an application increases. While the frontend component of response time remains more or less constant, the backend component of response time increases exponentially with the number of concurrent users:

Testing only frontend performance ignores a large part of the application, one more susceptible to increased failures and performance bottlenecks at higher levels of load.

Testing only backend performance, on the other hand, ignores “the first mile” of user experience and breadth. Backend testing involves messaging at the protocol level rather than interacting with page elements the way a real user would. It verifies the foundation of an application rather than the highest layer of it that a user ultimately sees.

Testing both frontend and backend performance leads to the best overall performance and user experience for your application. Ignoring one or the other exposes you to performance bottlenecks that significantly decrease a user’s satisfaction with your application.

However, if your testing is smaller in scope, you can choose to focus on either frontend or backend with the goal of eventually building a test suite that encompasses both.

Component testing vs. end-to-end testing

When you’re testing a web app, you may wonder whether how you structure your test scripts. Consider the following two methods.

Component testing

One way you could load test a web app is to load test its components. Maybe you know there are issues with a specific functionality due to previous production issues, or maybe you’d like to target components that are business-critical to reduce risk exposure.

In these cases, the type of test you write may:

be protocol-based
call only those API endpoints that are relevant
have smaller or no sleep/think time
be focused on eventually stress testing a component or service or finding its breaking point

Testing in this way is more flexible. With protocol testing, you can specify the endpoints or servers you’d like to hit, and narrow your test’s target that way. You can test specific functionalities while skipping others that come chronologically before it in a standard user flow. You can more finely control the type of traffic that is generated. If you’re recreating a specific mix of requests, you can script those requests and reproduce them in a repeatable manner.

Doing component testing for load may not always require that your script behave like an end user. In fact, it may be necessary to artificially inflate traffic to more quickly reproduce issues, or to deflate traffic to reduce the noise in the logs. By the nature of this type of testing, scripts don’t contain the full flow of a request that you would expect to see in production, so realism is not a priority.

End-to-end testing

You could also do end-to-end testing against a web app. End-to-end testing seeks to replicate real user behaviour and track its effects across the entire stack. When doing end-to-end testing, you might:

do protocol-level, browser-level, or hybrid load testing
replicate the actions in a typical user flow
think about the performance of the entire workflow as well as how long a request took to be processed by each component
write scripts that access the application the same way users would, such as by browsing to the homepage before finding their way to other parts of the website

This type of load testing is broader in scope than component testing, but shallower in terms of depth. With end-to-end testing, you get a better idea of the full user experience of the application as a whole. However, it can also be more complex to troubleshoot, as you have more components to monitor and more places to look for issues that are found.

End-to-end testing is based on real user behavior, so it’s often important that end-to-end test scripts also be realistic.

Protocol-based, browser-based, or hybrid load testing

The decision of whether to test the frontend, backend, or both will also affect the type of load testing you should carry out and the kind of scripts you should write.

Protocol-based load testing

Protocol-based load testing verifies the backend performance of an application by simulating the requests underlying user actions. For websites, this commonly involves HTTP requests that bypass the user interface of your application and are sent directly to a server or application component.

For example, a protocol-based load testing script might request all the resources on a webpage from the application servers, but those resources are merely downloaded. The response times reported by a purely protocol-based script do not include frontend metrics such as the time taken for images to render on a browser. The load is generated by simulating many requests sent to application servers.

While protocol-based load testing may seem to lend itself better to component testing, you can also do end-to-end website testing with protocol-level scripts.

Sample protocol-based test script

The following is an example of a protocol-based load testing script in k6 that fetches the homepage, along with resources embedded into the page.

import http from 'k6/http';
import { sleep, check } from 'k6';

export function Homepage() {
 const params = {
 'sec-ch-ua': '"Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"',
 'accept-encoding': 'gzip, deflate, br',
 'accept-language': 'en-GB,en;q=0.9',
 };

 // 01. Go to the homepage
 let responses = http.batch([
 ['GET', 'https://mywebsite.com/', params],
 ['GET', 'https://mywebsite.com/style.min.css', params],
 ['GET', 'https://website.com/header.png', params],
 ['GET', 'https://website.com/polyfill.min.js', params],
 ]);
 check(responses, {
 'Homepage loaded': (r) => JSON.stringify(r).includes('Welcome to my site'),
 });

 sleep(4);

 // 02. View products
 responses = http.batch([
 ['GET', 'https://mywebsite.com/products', params],
 ['GET', 'https://mywebsite.com/style.css', params],
 ['GET', 'https://website.com/product1.jpg', params],
 ['GET', 'https://website.com/product2.jpg', params],
 ['GET', 'https://website.com/displaylist.js', params],
 ]);
 check(responses, {
 'Products loaded': (r) => JSON.stringify(r).includes('Add to Cart'),
 });

 sleep(1);
}

Recording browser traffic helps you prototype to test websites on the protocol level.

Browser-based load testing

Browser-based load testing verifies the frontend performance of an application by simulating real users using a browser to access your website.

For example, a browser-based load testing script might include instructions to navigate to a page, click on a button, and type out text on a form. Those user actions then trigger underlying requests on the protocol layer, but only user actions are scripted in browser-based testing.

Unlike protocol-based load testing, browser-based load testing scripts generate load by starting multiple instances of browsers and interacting with your application the way real users would. Testing at the browser level can also be the only option for testing Single-Page Applications where a lot of the application logic is executed by client-side scripts.

Scripting on the browser level usually requires the use of different tools from the ones used to test at the protocol level. However, k6 has an extension called xk6-browser that allows the creation of browser-based test scripts alongside protocol-based ones.

Sample browser-based test script

The following is an example of a browser-based load testing script in k6 using xk6-browser on a dummy website. Instead of making an HTTP request, the script views the homepage, then looks for and clicks on a link to the product page.

import { chromium } from 'k6/x/browser';
import { sleep } from 'k6';

export default function () {
 const browser = chromium.launch({ headless: false });
 const page = browser.newPage();

 // 01. Go to the homepage
 page
 .goto('https://mywebsite.com', { waitUntil: 'networkidle' })
 .then(() => {
 page.waitForSelector('p[class="woocommerce-result-count"]"]');
 page.screenshot({ path: 'screenshots/01_homepage.png' });

 sleep(4);

 // 02. View products
 const element = page.$(
 'a[class="woocommerce-LoopProduct-link woocommerce-loop-product__link"]'
 );
 element.click();
 page.waitForSelector('button[name="add-to-cart"]');
 page.screenshot({ path: 'screenshots/02_view-product.png' });

 sleep(1);
 })
 .finally(() => {
 page.close();
 browser.close();
 });
}

Tips for writing browser-level scripts

The following steps can help you get started with a browser-level test script.

Script user actions, not requests. Determine what a user might do for a particular task, and script interactions with elements on the browser level. For example, script what buttons the user clicks on.

Identify unique selectors. Once you have identified which page elements a user interacts with, use the Element Inspector for DevTools in your browser to find a unique, static, and simple way to identify each element. The script needs selectors to find the right element to interact with.

Use elements to verify responses. After every action, use locators to search for elements on the page that you would expect to find. This verification helps ensure that the script has reached the expected page.

Take screenshots for every action while debugging. One of the advantages of browser-based testing is the ability to take screenshots. After every user interaction the script simulates, use page.screenshot to save a visual image of what the script encountered for later troubleshooting.

Hybrid load testing

Hybrid load testing is a combination of protocol-based and browser-based load testing. While you can use two tools or two scripts to execute different types of load tests (one protocol-based and one browser-based), having both types of tests executed by the same script and the same tool is ideal. Aggregating results between different tools can be difficult at best and inconsistent at worst.

A best practice in hybrid load testing is to generate most of the load using the protocol-level test, and then to run a smaller number of the browser-level testing scripts. This approach:

reduces the number of load generators needed, since protocol-level testing requires fewer machines to generate the same load
measures backend and frontend performance in the same test execution
provides a single source of aggregated output at the end
reduces complexity in the creation and maintenance of scripts

Scripting considerations

When you script a test for a website, consider these recommendations.

Consider factors that affect script realism

Record your user journey. Using the browser recorder can facilitate initial test script creation by capturing all the embedded resources on webpages. Check out the Session Recording guide to learn more about how to auto-generate your load test from a user session.

Correlate data. Recordings often don’t take into account dynamic values that are generated anew each time a request is made. Go through the recorded requests and determine whether you need to extract values from previous responses and use parameters in subsequent requests. This practice ensures your VUs behave more like real users would.

Include or exclude static resources. Determine whether you should include or exclude static resources on pages such as images, JavaScript, etc. Consider including them if you want to measure overall user experience. Consider excluding them if you are using a Content Delivery Network (CDN) that is under a separate Service Level Agreement (SLA).

Exclude third-party requests. Don’t load test servers that you don’t own. Many applications make calls to third-party providers for authentication, social sharing, and marketing analytics. Disable these requests unless you have permission to include them in your tests.

Use concurrent requests. To mimic the way modern browsers download some requests in parallel, use batching.

Determine cache and cookie behaviour. k6 automatically resets cookies between iterations, but you can also change this behavior if maintaining cookies would be more realistic.

Use dynamic think time and pacing. Consider adding varying delays so you don’t artificially stagger a script with completely uniform delays.

Use test data. Real users typically don’t search for or submit the same data repeatedly. Consider adding a test data file for the script to iterate through.

Model test parameters and load profile after production. In k6, you can use test options to determine the exact shape and profile of your load test script. Select the appropriate executors for the job.

Create a reusable framework

Use tags and groups. Organizing requests by tagging and grouping them helps you consolidate like metrics and makes your test scripts more understandable by others.

Use scenarios. When combining protocol-based and browser-based tests, use scenarios to independently control their test parameters and executors.

Modularize scripts. Use modules to separate and organize functions for protocol-level testing and browser-level testing, and then use a test runner script to execute them. This approach means different scripts can be versioned and changed without affecting each other.

Integrate your tests into your CI pipeline. Adopting a “tests as code” approach enables you to tie your load tests more closely into your project’s existing CI/CD processes, helping you get the most value out of every test.

Test with thresholds in mind

Create thresholds for both types of testing. Some browser-level and protocol-level metrics cannot be combined because they don’t measure the same thing. Set thresholds for relevant metrics from both the browser-level script and the protocol-level one.

When possible, use hybrid load testing

Use protocol-based scripts to generate majority of the load. When writing the test scenario, use the protocol-based requests to simulate most of the traffic and use fewer VUs for the browser-based requests. Relying on the protocol-level traffic helps keep resource utilization on load generators down.

Execution considerations

When you run tests, consider the test environment and the load generator location.

Run your tests in the appropriate environment

Testing in pre-production environments and in production environments both add value.

Testing in pre-production environments (staging, test, system integration testing, user acceptance testing, and production replica environments) enables you to identify performance defects early, which can save a lot of time and effort (and reputation) later on. Running in test environments also means you can often afford to be more aggressive with your tests. However, it’s also more critical to get your load profiles right, and the test results you get may not necessarily apply to production.

Testing in production yields the most accurate results, but it’s also more risky. Often, testing production is the only feasible alternative. You can reduce the risk of impact to real customers while testing in production by using lower levels of load when running load tests during peak hours, schedule tests for off-peak hours, choosing load test types that are less risky, using techniques like synthetic monitoring that generate less traffic, using real user monitoring tools to get snapshots of user performance at load, and ensuring that your observability stack is working at peak efficiency.

Run tests where your customers are

The location of the load generator(s), where the traffic is coming from, can also have an impact on your test results. The question is: where are your end users located?

On-premises load testing can be ideal when testing early on in the development a website, or when there are machines that can be repurposed as load generators. However, testing entirely from within a corporate network may also yield false positives, in that response times reported are significantly lower than if the same application servers were accessed from across the country.

Load testing on the cloud is an essential part of the testing strategy for many public-facing websites. Using load generators on the cloud gives you access to test in different states and geographical countries, creating a mix of load generators proportional to your users’ locations. Cloud load generators are easier to provision and cheaper to maintain in the long run than on-premise ones. Load testing on the cloud can help you include the effects of network latency in your tests and yield results that are more realistic.

Recommendations

Here are some recommendations to help you plan, script, and execute your load tests for websites.

If you want to test the last-mile user experience of your website:

focus on frontend performance,
write browser-based test scripts,
and consider doing more realistic end-to-end tests of the user flow.

If you want to test the underlying infrastructure of your website:

focus on backend performance,
write protocol-based test scripts,
and consider starting with component testing and then gradually increasing the scope.

If your website is meant for internal or has limited access:

Use on-premise load generators located within the network that most of your users access the website from.

If your website is external and public-facing:

Use cloud load generators in the load zones where your users reside.

Test in pre-production environments when you can, but also consider testing in production in a limited capacity.

Load testing websites can be complex due to the number of viable testing approaches available, the scope for performance testing, and the potential effects of releasing undertested code. By following the recommendations we’ve put forward here, you can tailor your testing more closely to your objectives.

Distributed Load Testing Using Kubernetes With k6

Sun, 18 Dec 2022 15:54:07 +0000

My colleague Paul Balogh and I demonstrate how to run distributed load testing using Kubernetes with k6 and k6 operator. k6 can already run up to 40,000 virtual users from a single load generator, but if you want to scale that up further, you can use the Kubernetes operator to create a k6 custom resource within your cluster.

Observability Trends and Future Best Practices (k6 Office Hours #59)

Fri, 12 Aug 2022 16:19:13 +0100

What are some observability trends and best practices in 2022 and going forward? Technical Program Manager Daniel González Lopes joins Developer Advocates Nicole van der Hoeven and Paul Balogh to chat about trending concepts and new areas of exploration in the field of observability. We’re going to talk about distributed tracing, continuous profiling, eBPF monitoring, and more!

How I Learn in Obsidian - Keeping up with tech trends

Fri, 12 Aug 2022 16:17:11 +0100

Here’s how I learn in Obsidian by processing articles I’ve read into consolidated notes that help me understand technical concepts. I also show how I improve my learning by incorporating visual elements using Obsidian Excalidraw, Excalibrain, and Advanced Slides.

Workload Modeling for Performance Testing (k6 Office Hours #57)

Fri, 22 Jul 2022 17:00:05 +0100

What exactly is workload modeling for performance testing, and why is it so important? In this video, Leandro Melendez and I talk about the shape of your load profile, why increasing the number of users may not be enough, and how you can make your load tests more realistic.

Creating k6 Scripts With the Browser Recorder and IDE

Thu, 21 Jul 2022 14:33:05 +0100

In this video, I talk about creating k6 scripts with the browser recorder and by using your favorite IDE. Scripts in k6 can be created through the Test Builder on the k6 Cloud, but sometimes you just want to hit record instead of starting from scratch. Also in this video is a rundown on the basics of using the k6 CLI and some important commands for running tests locally and on the cloud– right from your command line.

How to Be a Gish - API, browser, and chaos testing in one script (ExpoQA 2022)

Wed, 01 Jun 2022 14:06:49 +0200

For the ExpoQA 2022 conference in Madrid, Spain, I presented a talk about how to be a gish. In this talk, I covered the following topics:

what a gish is, and how it applies to the field of testing
how testing is a multidisciplinary field, and why generalists triumph over specialists in the face of uncertainty
how the tool we choose to do our testing informs the type of testing and the type of results we get
a hands-on example of how to do API load testing, browser automation, and chaos engineering in one script using a tool like k6

There was no recording of the conference talk, but here is a video of me going through my presentation beforehand:

You can find the slides here.

And here are some photos from the event!

What Is Grafana Mimir? (k6 Office Hours #53)

Fri, 06 May 2022 17:00:28 +0200

Maintainer Marco Pracucci talks about the new Grafana project, Mimir, a next-generation time-series database for Prometheus. He joins Developer Advocates Nicole van der Hoeven and Paul Balogh to talk about what Mimir is, why it was needed, and how he helped scale it using k6.

API load testing using k6, with Tom Miseur (k6 Office Hours #52)

Fri, 29 Apr 2022 17:00:26 +0200

In this second part of demonstrating the dogfooding we do at k6, I talk to Tom Miseur, Technical Specialist, who shows how he wrote API load testing scripts with k6 to test k6 Cloud– and what his results were after his testing.

End-to-end browser testing using xk6-browser, with Thomas Wikman (k6 Office Hours #51)

Fri, 22 Apr 2022 17:00:12 +0200

Have you ever wondered how to test a platform built for testing other products? Our frontend developer, Thomas Wikman, joins us on k6 Office Hours to talk about his experience being a user of one of our tools - xk6-browser - and how he uses it to test the entire k6 Cloud frontend.

Qué es k6

Thu, 21 Apr 2022 16:00:51 +0200

En este video hablo con mi colega de trabajo, Daniel Gonzáles Lopez, sobre qué es k6 y cómo usarlo para hacer pruebas de carga.

How to Run a Load Test With the k6 Cloud Test Builder

Tue, 12 Apr 2022 17:00:48 +0200

In this video, I demonstrate how to create and run a load test from scratch without coding, using k6 Cloud’s Test Builder. k6 Cloud is a load testing platform as a service, and you can run 50 free load tests on the cloud here.

Load testing results in the k6 Cloud App for Grafana, with Edgar Fisher (k6 Office Hours #49)

Fri, 08 Apr 2022 18:00:18 +0200

There are many ways to get k6 load testing results into Grafana, but our frontend software engineer, Edgar Fisher, talks to us about the easiest way of them all: the new k6 Cloud App for Grafana.

TestOps Ask Me Anything on Devops Toolkit

Thu, 07 Apr 2022 18:00:30 +0200

I joined my friend Viktor Farcic on his channel, DevOps Toolkit, to do an Ask Me Anything on the topic of TestOps.

We ended up talking a lot about k6.io, different Grafana Labs open-source projects, and what performance testing has to do with DevOps and Site Reliability Engineering.

Building Reliable Systems With k6 (k6 Office Hours #48)

Fri, 01 Apr 2022 10:02:29 +0200

In this week’s k6 Office Hours, my colleague Pandurovic, an engineering manager/backend lead at k6, joined Paul Balogh and me to talk about how k6 works on the backend, and how the backend team is working to make it even more reliable.

How to Analyze Load Testing Results with k6 (k6 Office Hours #47

Fri, 25 Mar 2022 18:00:56 +0200

In this k6 Office Hours, Paul Balogh and I discuss the different ways to turn k6 metrics into graphs for analyzing performance testing results.

How to Process Notes in Obsidian // Readwise Official Obsidian plugin

Wed, 23 Mar 2022 16:00:28 +0100

How do you process notes in Obsidian? I demonstrate how to use the Readwise Official Obsidian plugin to go from raw content to notes in a Zettelkasten-like Obsidian vault. Sign up for Readwise using my link and get double the trial period.

Note: This post contains an affiliate link for Readwise. Clicking on that link will give you a two-month trial of Readwise, double the usual amount, and they give me a small percentage if you sign up for a plan. Check out My Ethics Statement to see my standards for promotions.

Obsidian Dataview, building a second brain, and more about me // Q&A Livestream

Sat, 19 Mar 2022 17:42:54 +0100

In celebration of reaching 1000 subscribers on my personal channel, I did a live Q&A where I demonstrated how to use Obsidian Dataview, my personal philosophy about building a second brain, and how I stay organized.

How to Make Kubernetes Clusters Production Ready using Crossplane, with Viktor Farcic (k6 Office Hours #46)

Fri, 18 Mar 2022 17:30:45 +0100

In this video, Viktor Farcic, Paul Balogh and I talk about how to make Kubernetes clusters production-ready. Viktor is a Developer Advocate from Upbound, and he demonstrates a solution that ties together Crossplane, Grafana, Prometheus, and Loki to deploy and manage clusters in production.

Load Testing for DevOps with Victor Farcic (k6 Office Hours #45)

Fri, 11 Mar 2022 11:12:52 +0100

How can you use load testing for DevOps? What kind of performance testing can or should be included in CI/CD pipelines? Viktor Farcic from Upbound and DevOps Toolkit join Nicole van der Hoeven and Paul Balogh to discuss how he sees k6 fitting into the daily duties of a DevOps engineer.

Postman for load testing using k6, with Tim Haselaars (k6 Office Hours #43)

Fri, 25 Feb 2022 18:00:11 +0100

Postman is a near-ubiquitous tool for anyone doing API testing, and many teams use collections to create and organize complicated test suites. However, there’s one thing that Postman isn’t good at: load testing. That’s where k6 comes in.

In this video, Tim Haselaars joined k6 Office Hours to talk to Paul Balogh and me about how to use Postman collections to run load tests using k6.

Transcript

Hello everyone and welcome to another k6 Office Hours. I am Nicole van der Hoeven and today I have two guests. One of them may already be familiar to you, Paul Balogh.
Hey everyone.
And our special guest is Tim Haselaars. Tim, would you like to introduce yourself?
Okay, hi, nice to meet you Nicole and Paul. I am Tim Haselaars. I’m living in Belgium. I’m a product manager, and I work at the marketing automation company called Selligent. I have a passion for APIs. I love music, I’m a DJ as well in my spare time.
Oh, nice.
And if I still have some spare time left, I like to contribute to a number of open source projects, which the Postman-to-k6 is one of the projects I’ve been working on together with the k6 team in the past. So that’s a bit of why I’m here today. So nice to meet you both.
Yeah, that’s a pretty big if though, if you have time, you’re a DJ.
It’s a hobby thing, but I’m really proud of it actually, because I’ve been playing a lot, and there’s one festival in Europe, Tomorrowland, I’ve been playing it each year for the past 10 years.
Oh, wow.
So it’s a small stage, but I’m really proud of it.
Nice.
Yeah.
Well, that is awesome. You should be proud of that. I think that kind of makes sense, because product management is such a multidisciplinary role as well. Like, it’s not all technical, and it’s not all people skills. It’s a mixture of both, and a lot of other things.
Yeah, that’s also what keeps it interesting, at least for me, what keeps it interesting, actually, so yeah.
So what do you typically do in your current job, and what does it have to do with k6?
Okay, in my current job, I’m a product manager, so we have a digital platform, marketing automation platform and I’m spearheading the API renewal program. So we had a number of APIs in our company which customers use to connect and exchange data with marketing data, and we’re renewing that, and performance is critical, because we have a number of really big customers in Europe, or , the Dutch luxury brand, and we talk about huge volume, so our APIs has to be scalable, and performant, and we use k6 to guarantee that we keep them up to standard, actually. So that’s what we do. And my role there is to make sure the teams, I help with the API design, and I check with the teams, and I document stuff together with the teams, we try to do that. So what I’m going to talk about is also my experience with k6 and the way we use it at the company a bit more.
Awesome.
Cool, so how did you first start, what made you start to use k6? Did you try other things before that?
Yes. In a previous job, a couple of years ago, I was working at a hosting company, and they were doing a, there was an anniversary of the national lottery, it’s like the lottery company, so it had,, they expected a lot of visitors to the website, and we wanted to make sure the hosting was up to par, and it was able to handle the volumes. And I tried to test it, performance test it, load test it, and I used Jmeter, and the experience was, I struggled a lot with it. And I started browsing for another solution, and then I hit, I found k6. text was really nice. You could really quickly start nice documentation, and that’s how I rolled into k6. And then I shifted jobs, and at the current job, I said we have to use k6, because it’s so much easier than using JMeter. You can script it, you can automate it, and that’s it, yeah. I used it once, and never looked back.
So nice, with the automation, did you put hook that in your CI/CD pipeline as well?
Yes we do. I can, I have a diagram. Like, I’m really proud of it. Maybe I’m a bit, how do you say, biased by it, but I’m really proud of what we’ve achieved with the automation aspect actually. And we came from a far away, a far place, but now, we have a nice automated flow, actually.
That’s awesome.
Which I hope I can showcase a bit later on in the talk.
Yeah.
Yeah, I’d love to see it. And do you use k6 Cloud or k6 OSS?
We had used Cloud in the past, I think right now, my team are mostly using the OSS version of it. I don’t know why. I have to check with them, actually, but I think we had a number of dashboards set up, and they were a bit detailed to what we need, or what I needed at least, and that’s why I think we stick with, for now we stick with the OSS version of k6.
Okay, awesome. So I guess people might be wondering, seeing the title of this video, what Postman has to do with either Selligent or k6.
Yeah, actually that’s a story I want to tell, how did we ended up with k6, and the whole story. So what typically happens if you start developing an API is you build your API, and then you want to test it, you want to play around with it. You want to experiment with it. And the first thing that you do as a non-technical user like, I take Postman. Postman is an application where you can model your request in a nice tool, and then you can just fire them. You don’t have to a curl script, or you don’t have to use any CI tools for making API calls. So that was the logical step. So we build an API, I want to experiment, play around with it, so we ended up using Postman for it. But maintaining Postman is not so easy when you go for automation flows. And then when you want to take a step further, if you want to do performance tests, that’s another a thing. And yeah, that’s why I think we went from what we did to Postman just the next, as a first phase. But sooner, the more end points we have, the more we wanted to test and the more we wanted to play around with it, the bigger everything became. So that’s why, yeah, we wanted to make it an automation level and make it a bit more streamlined so that the manual work would go down a lot.
Sounds like you have a fan out in the viewers.
Yeah.
Yeah. I like to make things visual, because it helps me visualize my thoughts and everything. So yeah.
I’ve used Postman before. I think it’s almost ubiquitous now for anyone who’s doing any sort of API testing. I think at some point people would’ve used Postman just because it is very easy to get started, but I do, I have also run across some of the issues that you describe when you’re scaling up, when you have a very complex test suite or you want to have one, it is, I think I personally still like the scripting approach of being able to have the full, you know, being able to have condition loops and that sort of thing. There’s still a lot more flexibility when, when you’re just dealing with pure code. But I know that I think it was, I mean, relatively recently when I started using Postman, it was just firing off requests, and it was like a single user kind of thing, but what are the features available for performance testing for Postman, you alluded to it, them, and you said that maybe they’re not, they weren’t exactly what you were looking for.
No, no. Maybe I should quickly show a bit how I–
Yeah, sure.
Work with it if that’s okay for you. Okay, I’m gonna showcase my, so well, just as a reference, I created fake endpoint and I was set up a Postman collection. So what you do typically in Postman is you have an API that you define. In our case, I created a k6 office hour CRM API, just plain and simple where you can see, yeah, you could see some context you potentially want to interview in the future, and you model them. So you have your endpoint, you just put them in. You say, it’s a get post, put, whatever you need, and then you can just fire it and it’s response. And so this is an actual API request happening behind the scenes. That’s really nice because as a non-technical user, you don’t have to have any technical expertise. You just enter the endpoint percent and you’re up and running. But it also has some nice features if you want to make complex things, like you said, Nicole, if you want to use variables, because you might want to experiment with your API endpoints in different environments, so you can use variables. That’s what they use with the curly brackets here. So it’s a base URL. So I can shift around so if I want to switch, I have potentially two environments. I have a production one and a dev one. I can easily switch between them and then the variables will also switch along. That’s a really nice thing to do with Postman. The next thing, what I really like about Postman is that you can use it to create cruds and you can create flows with it. So it’s really straightforward. So if you say, I want to create one item and then I want to use the item in my detailed call in my, so I have all leads, create a lead and I want to get the leads, so this is the end point, you could just define it as a variable. You could say, I want to, that this lead ID can be used to fetch the right lead ID. So if you create one, that’s the body, I do a post request. And what happens is you get a new ID back, which is unique because I just created a new contact in our CRM application. And you can use scripting in Postman to say, ah, there is a data response, and I’m gonna store that response in my lead ID variable. So it allows some scripting within a GUI which is really nice, and it also allows you to define tests. Like I want to check what’s the response. It should be 200. You can define a Postman test. They use a bit of assertion, kind of syntax for that. So what started out as something really straightforward, I wanted to just test and play around with it, experiment of the endpoints. It became much more. And we started building like a full suite out of Postman, in Postman itself, a test suite. And that’s really cool because as a non-technical user, you could just share this collection with anybody on the team and they could just run the same request and play around with it. You can even pass it on to partners or colleagues or customers, when they say, when they want to use your API, you can just say, ah, you can use a Postman collection to play around with it. And another nice thing about Postman is that you can run scenarios, like I said, and you can run them at scale. So right now I’m doing get leads, creating a lead. Then I’m gonna fetch the lead that I just created. So I created Elon Musk as a user. I can update that one as well. So I can put a body and touch and I can even delete it. So if I would run this in a cycle, I would test actually the whole flow of a typical crud operation. You can fetch one, create one, edit one, delete one, and then it’s back to normal again. And so lemme talk about, yeah, sorry.
[Nicole] Sorry, I just wanted to, sorry to interrupt, but would it be possible to try and zoom here in this one to see if you can make it just a little bit bigger?
[Tim] Okay, something like that?
[Nicole] Yes. Thank you. Sorry.
[Tim] No, no, no worries. So what you can do Postman, if we talk about test automation, you can just say, I want to run all these, these API requests in the sequence, like they’re organized here, and you can just run them. So if you run them, you see you put in some test, this one is a success, success, creation successful. And so actually testing if your API’s responding as you expected, but you cannot test it at scale or at high volume. The only thing you could do in Postman is if I press run, I could define a number of iterations. So if I would say, I want to do 10 iterations, I’m just gonna run them now, so you can see, you can see it just creates blocks. but while I’m talking, it took to run all these 10, five times 10 requests, it took me, let’s say one minute to run. So we cannot really test at scale if, that’s not the use case of Postman. The use case of Postman is to experiment with request, do some testing on some validation of the response and the inputs are okay and that’s about it. So I needed something more if you want to do that scale. So while I discovered k6, k6 has this nice scripting language, you could define–
[Nicole] Tim before you, sorry. Before you move on, we do have a question. Chaitra asks, can you share this collection please, just to get along with Postman.
[Tim] Okay.
[Nicole] Is there a way that you could share, make that public or something?
You can, I haven’t tried it, but I guess if I click share and I could, can I create a JSON link? Maybe could JSON link, update the link, maybe like that? I haven’t tried this actually before either so.
[Nicole] Awesome, we’re learning in public.
[Paul] Exactly. No pressure, no pressure.
[Nicole] Yeah.
[Tim] Oh, should I put it in the comments, or how would I do that?
[Nicole] Yeah, sure. That would be great.
[Paul] Yeah, otherwise I was gonna say we could probably put something together too for after, you know, the presentation if someone wants to access that, if there’s no issue with that.
[Nicole] Okay, I’ll put that in the comments.
[Tim] So I was asked as well from the k6 team to update our blog post about Postman and k6. So what I’m gonna do after this session is going to review the blog post that they have already on the website and then update it with maybe some content that I’m talking about in this show here.
[Nicole] Great.
[Nicole] And we also got a comment from Jim Munro who says Postman is definitely not designed for load testing.
[Tim] No, no.
[Nicole] And that’s what it seemed like. So it seemed like you could only do iterations. Is that right? You can’t set, you can’t have more threads or users.
[Tim] No, that’s it. So the only thing you can do is you can run a sequence and you can run it in a number of iterations, but you can, you can’t define how many users, you can’t define how long iteration, you can define sleep, but that’s difficult. So Postman is great for what it is. It’s great for testing some of your contracts of your API, is great for experimenting with your API, and really kickstart using your API, but if you want to do volume, to test the volume at scale, I wouldn’t recommend Postman at all. But that’s also a bit the problem, because you already spent a lot of time making a nice collection, preparing it, grooming it, updating all the time. If you then want to go to k6, you would have to rewrite everything. So if you, this is how k6 syntax looks so you can define your iterations. You know this better than me actually. And you can just use a get request. And you can script your request. You can also check it. So that’s the same as you would do in the Postman where you test your response. You could check if it’s a 200. But if you have a complex scripts, you can, they become really large to maintain and to keep them up to date. And that’s what I struggled with because it felt like I’m reinventing the wheel. So I made a diagram out of it just to explain it. So if you have an API and you have to update it, you would have to really update your collection and say, every time something changes an endpoint was added or a property was changed, you needed to update, go into the Postman collection, manually correct everything, extend it, and yeah, keep it up to date. But if you would go to k6, you have twice the work, actually. You have your API that’s changing. So you need to update your collection. You need to update your k6 scripts. So if you have five endpoints, that’s doable. But if you have 20 endpoints for one API, that’s a lot of work to keep them in sync, because this is a moving target, but it forces you to keep in sync to other moving targets because you want to be able to experiment and test them, contract, and you also want to test them at scale with the load. And I was looking really for a way, like how can I leverage this nicely prepared Postman collection and use for k6 scripts? And that’s what I, when I ended up with this scenario where you have your API, I update my post collection, I want to generate a k6 script. And I was looking around like, how can I do that? And I found on the k6 website, I found some blog items on, on how that they provided a Postman to k6 converter. And that really triggered me and I started playing around with it. And I really, really liked it. So it gives me all the flexibility of modeling everything in Postman, preparing the whole flow. So if I want to run it in sequence, I could do that. But then it also converted it into a k6 script where I had no maintenance on it, and that was really nice. And when I found the Postman k6 package, like a year ago or something, the purpose of that was to kickstart your k6 experience. And I said, but you can do much more with it. You could even fully automate it with some parameters because the first time I converted my collection, it had like 20 endpoints. I still needed to do add some minor things, I needed to add some k6 specific stuff to it, like how many iterations do I want to do, is there certain parameters I want to add that are not by default converted over. So I said, okay, what can I do to make it even better? And I don’t have to touch the k6 script anymore. So the first time that you converted it, I had to modify it still. So each time that I would do this kind of a behavior, the API changes, I need to update my Postman collection and I needed to update my k6 script. I want to, my ideal world would be, I don’t have to touch the k6 script at all. I just convert it and I was able to run it in one go. And I can show you how it works, because it’s really straightforward in the end.
Yeah.
If it’s okay, you can export your collection–
[Nicole] Yeah, sure.
[Tim] So if you do export, you can just say export and you download a JSON file and you can do the same thing with your own environment. So if you want to have your test environment and you want to also convert them you can because you can also export these from Postman. And what you end up with, I’m gonna close this one, you end up with two JSON files. So as you can see, it’s the k6 office hours. It has some syntax in it, like the test scripts are there, the endpoints are there, the way you connect to them. So all the request information is in this configuration, in the environment file. You can find all your variables that you have defined. So let me just now showcase it. Sorry, I’m gonna, this one. So if you run the Postman-to-k6 converter, it expects actually two things. You define an input, my Postman collection, and you define an output. So if I would run this, just closely watch what happens there, it does some things behind the scenes, and once it’s finished, you would get, actually, if things go well, let’s see if it’s there, you would get this. Yeah, that that’s the goal of CLI. It should be fast. You get the k6 script. I’m gonna go over that later on, and you get this library folder. The library folder, what it is is because the converter, Postman uses number of functions, some methods that are not mapping exactly towards k6. So once this package, it adds some additional functionality that Postman uses. Like it uses low dash and some encryption, stuff like that, and it just adds the libraries there. So if you would do this manually, you would also have to include the libraries, but it does it for you. It includes all the libraries. And it’s actually a wrap around the, let’s go for the SHIM quickly. If it’s not clear what I’m trying to explain, stop me, Nicole.
[Nicole] Yeah, but we can also answer this question from Carlos Ramirez. You can also integrate, well, not a question, a comment. You can also integrate your process using the Postman API key to request information about your collection and environment. And he also adds, oh, no, sorry, this is from Chaitra Nanjesh, this is interesting, because we use k6 in Tesco, says load test, and I think we can try this to see if it works for us. So people are already liking what you’re sharing Tim.
[Tim] Yeah, that’s the goal, That’s also why I liked it actually–
[Nicole] Do you also do what Carlos was suggesting?
[Tim] Yes, and I even go a step further, but that’s, I’m gonna come back to that later. I’m gonna showcase how we do the Full Monty in the end, actually. I’m building up.
[Nicole] Awesome. Sorry, people are excited.
[Tim] Yeah, I’m also excited, so I hope that I’m coming across properly.
[Nicole] You’re doing great, don’t worry.
[Tim] Okay. Just for reference, you have to think of this Postman-to-k6 thing as it’s wrapping Postman functions and k6 functions and combining them in one place. So it’s abstracting away the k6 part and it matches with the Postman stuff. So that’s what it does behind the scenes. So if you see, if you do an execute request, which is an, it’s a function actually, behind the scenes, it just does the HTTP dot request, which is a k6 function. So it wraps a lot of things together, and I think if I would go to execute test, you would also see it uses the k6 check function. Just for reference, this is the wrapping part. This is all not so relevant. You don’t have to care about it. What’s relevant is what’s being generated for you. So it says on top, it’s an autogenerated file here. It does some imports. It imports also k6 related items and it sets default options. By default, it says maximum redirects. And then it does some strange things which are not k6 related, but that’s typically what the package does. That’s the wrap that you see. It defines a collection with a base URL, and then you will see that my group is there, I have my request get all leads. It showcases the URL. It showcases that it does a test here. So everything that you had defined in your Postman collection, sorry… Get all tests and you could see the test here. It’s translated into a k6 kind of syntax and it’s ready to be used. And you see all the requests on here, all ready to be used. It even matches a response codes that you can match to a variable in Postman. You could do anything. The body is there that I inserted that you can create. So if I were now just say k6, sorry, k6 run k6 scripts. Oh, dunno know why this is happening? Give it a second to start up. There you go.
[Nicole] Cool.
[Tim] So I haven’t touched anything, I just took my Postman collection, converted it with the converter that we built a while ago together with k6 and then you can see the tests there. So I created a number of tests. Postman expects to, the response is expected to be 200. I get all the data. My average is my group duration because you also have a group there. So if you have 20 endpoints, you will test 20 endpoints, but I don’t need to write any k6. So I only do my maintenance for, my maintenance only happens in here. I only have to maintain my Postman collection and my k6 gets generated out of them. So that’s really nice thing here. Yeah, that’s the library that we’re using. No, that’s not, that’s not my library actually. I think it’s moved to–
[Paul] Uh-oh.
[Nicole] Hope this is the right one.
[Paul] Yeah, we gotta update our show notes.
[Nicole] Yeah, we do. Oh, with the API deck one. Sorry about that.
[Tim] No worries. I think I made a pool request a while ago, so.
[Nicole] Ah, I see. Okay, I will update the description.
[Tim] But I, just to continue.
[Nicole] Can you also post, Tim, the link?
[Tim] Yeah, I’ll post it. Of course.
[Nicole] Thanks.
[Tim] But it’s really cool. Maybe cool’s not the right word because the package supports a number of things so that you can even tweak the output even further. So by default, I just say, give me the Postman collection and generate a k6 and it would work. But if you want to automate this on a higher level, need to do work for automation, you wanted to tweak some things. Maybe I want to play around with some k6 parameters, or I wanted to do something else, or I want to exclude stuff, or I want to include my environment because, if you see here, there is no, there’s only my base url, but I have a lot of variables I want to use in my Postman collection that I already defined. I have login information. I have some Postman base URL. So you can take it even further because now you just get a script, but it’s just, the script is already addressed. So you can take it step further. So for example, if you would say, I want to increase the number of iterations, you could just say like this. You have this parameter, says iteration 10. I’m not going to convert it. There we go. And if you see my script now has options already set at 10. That’s nice. That’s handy because you want to automate this, of course. And you can even go, take a step further. I want to include my Postman environment variables. So I’m gonna, there’s also a parameter for that. So you can pass some environment variables. So you can even pass some global variables. This is typically Postman, but you can just take the ones that Postman provides, export them and use them. So if I now do this kind of thing, so this is my file with my Postman variables, like I showed. Let’s do this again. And here, you’ll see all my variables that were exposed in my environment are now available for my k6 script to be used in automation workflows. So that’s really nice, but I can even take it a step further. What if I have special k6 related parameters? I’m just gonna copy paste them here for the sake of the demo, but I might want to define my iterations or my maximum redirects because k6 offers a lot of parameter options. Like I want to use cookies. I want to use headers. The number of redirects. I want to set some text. I want to define a timeout. All these k6 related parameters, I want to make sure that my automation flow was able to process them because I want to run the script without touching it. So I want to be able to just say, take these parameters for this pipeline, these parameters are applicable for production. So you could switch environments. For example, what we do is we have different test scenarios defined. So if we go for a small test, we run it like this, we define stages and thresholds. Well, if we want to really do performance testing, we have different ones. So these options, we use them to pass them along to the pipeline. So I could do something again with the generation part.
[Nicole] Question from Sri. Is it one environment at a time only?
[Tim] Yes, because what you generate is what you inject the script. So it, you would then have to generate a second script, but that’s really easy to do because you would just say, I want to have, for example, just showcasing it here, I’m gonna take this script is for the second test stage. You just run it and then you get a second script next to it. So you don’t have to worry about it. You don’t think about one script. You just fork it as much as you need with different configuration options defined that you would use.
[Nicole] Yeah, and Sri, I wanted to also add here that with k6, you can have, you can test more than one environment at a time. What Tim is showing here is that for, because of the way that the that Postman treats these collections separately, every environment is separate. And so the converter also converts them to separate k6 scripts. However, once they’re in k6 script format, once it’s all JavaScript, you could use k6 executors and scenarios to run them simultaneously, if that’s what you’d prefer or anything else.
[Tim] So in this example, what I’m doing, I’m going to pass on my K6 parameters as a JSON file so that I can put them in a Git somewhere. And if I then just say, press go, you will see that it also just updates this option spot on the top. There we go. So you will see, I have defined it to be in this case five virtual users, 10 iterations, maximum redirects, and just adds it on top. So I can switch around in my pipeline without touching anything. I just prepare everything, and then it’s just a matter of defining which command I should to run and it just does it for me. So that’s really handy and it really makes automation a lot easier in that regard. And there’s one thing I really like. It’s a small thing, actually. Now it generates one big file. So everything is in this one big file, all the request, it’s doable if you have like me, for example, four requests or five requests. But if you have like 20 endpoint or 20 requests, it’s a huge file. And to handle that, you can just pass along one parameter. I’m not sure if it’s commonly used a lot, but I’m a big fan of it. It’s called a separate parameter. So what happens now? The options are there. The request configuration is there. But you just get this in your k6 script. I’m not sure which one, I’m gonna remove one because otherwise I might get confused. There we go. You get this, and this is actually, it splits all your requests in a separate folder like you would see them in Postman and each request would be separated from the rest. And that makes it really easy. If you want to test something specifically, if you would have one script, it would run everything. But now I can just comment out, I don’t want to do the create, I don’t want to do the update and delete. I only want to test to get all leads. You just say k6, k6 run k6 script. Give it a second to warm up. There you go. It will only test that request that I defined. So if your 20 endpoints, your k6 script has been generated for you will have 20 lines and all your requests will be nicely separated from each other. So you can even say, I want to tweak it a bit, or you have all flexibility then, while if you don’t use a separate parameter, it just generates one file with everything in it. Do I make sense still?
[Nicole] Yes you do. But there’s a comment here from one of our engineers who wrote that initial blog post that you may be updating, Mostafa said last time you wrote an article about it, it needed a few changes to make things work, so well done.
[Tim] Yeah. Yes, that was one of my first VRs that I did. I wanted to make separate flow over pretty much everything. And there’s one final thing I want to showcase for, with the k6 part, is now you have to really, like all these parameters, they become very long and difficult. You could also just define them as a JSON file and just pass them along. And now you can just define the one file, put the main Git, and then you could just say, I want, this is the output, my environment. You can even define that it uses a handle summary, or you can add request tagging on top of it. And if you do it like that, you get, you just have to do one parameter, one command, actually. So if I do it like this, you see, it just says, this is my input file. I’ve defined an options file, which I referred to. And if I then just do it, it would take all these parameters that I’ve defined and you would get, again, everything nicely generated for you. All the requests are there and it even added the handle summary thing on top of that, which is something–
[Nicole] Oh nice.
[Tim] So everything is there. And so this is the automation flow that’s really powerful because otherwise you would have to add, if you want to have a summary, you have to add a modified file. But now I just define one configuration file, I define my parameters, I define whatever I need. Just run it, and my script, I don’t need to touch it. I can just say, okay, k6 run k6 script. And you would see that it generates, yeah, it generates a test. It does, I think five or 10 iterations. There we go. And it doesn’t open anymore. It generates the JSON file. If I close my, quickly here, you would see a summary report is here. So it just, yeah, there is no manual work anymore to run k6, just keep your equipment collection up to date.
[Paul] And you’re helping folks out with your example here.
[Tim] Yeah, okay, great. Okay, that’s in summary, the things that I want to showcase about the converter. There are some nice things added since the last releases from the k6 team. So I added some, requested and changed some things, and I’m really happy with it. It works flawless at the moment in our organization actually. So I’m really happy with it.
[Nicole] Yeah, it seems really, I think I tried this initially, like when Mostafa was saying, there are a few things that you, you had to get done, but this seems way slicker, and it seems like you’ve really focused on making it automatic so that don’t, there is no manual intervention, which totally makes sense in a CI/CD pipeline.
[Tim] Yes, correct. And I wanted to put my effort on the Postman and the management of the Postman collection, because that was a tool where we were all working on and maintaining it. And I wanted to keep the workload on k6 as low as possible. And I think we succeeded it quite well on that regard. There are some things that are not available out of the box. Like you cannot do the send request. You cannot set next request. But what I did add was, what I’m also really proud of, the dynamic functions from Postman. You could just add any, pretty much any generated stuff. Let’s see if it shows, if it works like that.
[Nicole] There’s also some nice comments for you. There’s Gertjan De Wilde, who says, Go Tim. There’s Pepe Cano as well from k6 chiming in with his support. James is saying nice to see this getting the love it deserves, much better experience. And Charly is saying, Very useful tool, congratulations.
[Tim] Thank you.
[Nicole] Lots of love in the comments for you.
[Paul] Definitely.
[Tim] Okay. But honesty, we took it a even a level further, but maybe I should, maybe there are, if there are other questions we should talk about or not, I’m not sure.
[Nicole] I’m interested in this. So what were you saying, the dynamic part that you added parameters?
[Tim] Let’s see.
[Nicole] I mean, you also don’t have to demonstrate everything. Can we just like acknowledge the fact that Tim who doesn’t even work for k6 or Postman volunteered to do like a live demo about this. So that’s awesome. Don’t feel like you have to show us everything. You can just talk about it.
Yeah, Postman really provides this nice, what they call dynamic variables. And so it’s, this is a screenshot of it. So if you use curly brackets, it’s random, and then you can say, I want the full name, I want the bank account or anything that uses Faker behind the scenes, and you can make dummy data out of it. So that’s really handy if you want to really make, yeah, realistic data without having to use for loops or anything, just put in the variable. And that was not fully supported in Postman to k6. So we brought in that support as well to make sure that you could have the full functionality of the dynamic variables attached to that. So yeah, there are a number of things we did since the last release of it, and I think dynamic variables was in much demand at that moment. There you go. Yeah, and I’m really happy for Mostafa, I’m not sure how to pronounce it correctly, because I think he was one of the first ones who pointed out in his blog post about functionality. So thank you for pointing it out to me.
Yeah, now that I think about it, I should have asked Mostafa to be here too. Oh, I missed that opportunity. Sorry about that Mostafa. That’s so good about the dynamic variables, because that’s one of the things that you have to think about when you shift from just a purely automization mode, like automated testing mode to load testing is that you can’t, you then have to start thinking about the data that you’re sending too, because if you’re trying to get realistic response times it could trigger caching on the service side if you’re just requesting the same thing. So yeah, totally makes sense. k6 can already, you can already import faker with k6, so great, great job for putting that together.
Yeah, no, that’s very cool though, but yeah, it’s like, you can use Postman for your user, your UI, your, you know, your development environment and do all this stuff and then even goes through there and generates some random data. So, yeah, that’s cool. Very cool.
Oh, go on.
But maybe I’m gonna talk first before I want to show the final piece of what we did.
Well, just, just quickly, I just wanted to ask about the limitations of this approach that you see.
Not all functions of Postman are supported. So that’s the downside of it actually. But for the use is that that we are using it, we have everything covered. Like you have the, the dynamic variables, you have the environment variables, you can define k6 parameters. Yeah, there was nothing stopping us. I think if you want to do complex scenarios and you want to do if-else situations, there it’s not suited for. That’s the same problem you would have in Postman. It would be difficult to make these if-else conditions in Postman as well. So that’s, if you do simple tests, functional tests in Postman, you can really easily port them to k6 and then you have performance tests end points.
And in the spirit of the whole free and open software. I mean, do you have issues where like you’re looking for help? Is there any gaps that you’re still looking to fill?
There are some bugs I cannot reproduce at the moment, something with XML conversions, but right now it’s feels quite feature complete. I haven’t added that much, actually. The base was already there by the k6 team members who built the initial library. So I just added some sugar coating on top to make the automation flows much smoother and less manual work, or linked actually. But if people have great ideas, I think they should just make a PR and add it to the package.
Yeah, did you want to show us something else?
Yes, I want to show how we ran a step further even, because yeah, if you update your API and then you have to update your Postman collection, it then generates a nice k6. But we also use openAPI as a documentation tool where we document and design our API. So every change that you do on your API, you need to add them to your API, and then you, so you still have to maintain not one output, the Postman, but you also have to update your documentation. And I want to flip that around. I want to make the API documentation leading, generate the Postman collection out of it and generate out of the Postman collection, a k6 suite. So that as a product manager, I don’t have to maintain all these documents anymore. I just maintain one thing, and that’s my openAPI and my team or our team then only just need to develop the application itself. And it’s called spec driven approach. It’s nothing new, but I’m a really big fan of it and I’m a really, yeah, we introduced it a year ago and it solved so much and it brought so much quality on the company level, in our documentation, in our test suites. So that’s something I want to share as well, because it’s so valuable once you get how it works and if the right tools and properties, you can really do it. And this is actually where we are right now. I can also demo how it works. So we have, I’m managing an openAPI documentation. We use the formatting and we validate it if it’s valid. And then we use a generated one. And from the generated one, we generate documentation. So public documentation, stop lights, just like whatever you call it, we generate a Postman collection that we can share with the customers and we run contract tests, we run soak tests and we run smoke tests all based on this openAPI documentation. So, and again, there’s no no contact or no changes involved. We just, we managed the spec on its own and it’s really, really something powerful and useful. And I can quickly show how it would work actually if you’re interested.
[Nicole] Yes, but before you do, a wild Mostafa appears.
Hello.
How awesome is that? Let’s just say hello to Mostafa, who just joined last minute since he was already in the comments anyway.
Nice to meet you. And I’m grateful that you wrote the blog post so many, so many months or years ago, so thank you for that.
Yeah, yeah, I really felt the need that we needed this article, so that’s why I wrote it in the first place. And then, yeah, I found a lot of issues that you solved, and thank you, it was a great job you did, and well done.
Thank you. And thank you for putting out the 90% of it. I said I only had to put in the sugar on the top of it and the cherry on the top.
No, no, no, no, that’s not true, that’s not true.
Mostafa–
In the first place just to clarify.
Yeah, Mostafa while you’re here can you quickly introduce who you are and why we’ve randomly invited you onto the stream?
Of course. Okay, so I am a backend developer at k6. I’m almost three years with k6, like it was called Load Impact before being renamed completely to k6, and now Grafana. I’ve been a developer advocate when the first like idea came into place that we want to have a developer advocacy program. And yeah, I’ve been a developer advocate for a year and then came back to back it. So I worked with lots of these tools and tried to help others because like, there were no other resources available. Like there was this read-me on the project and that was it. No mention of it anywhere. Only the hidden knowledge of everyone inside the company of how to use this tool, but nobody knew without outside the company. So tried to learn it from others, specifically, Pepe, thanks for that, from here. And I tried to write that blog post and try to share the knowledge. So that’s basically it.
Great, so we also got a comment: Great with the diagram. Can you share this?
Of course I can. Yeah, I’ll do that.
All right, I guess that’s our cue to head back. Please continue, Tim.
Oh, I’m not gonna, I’m just gonna showcase it, and then I hope everybody will really appreciate and see the power that it brings actually. So, okay, so maybe start with the beginning. I’m gonna close my screens a bit to clean it up. So I have this openAPI spec that I, it’s just a copy of what we just described. So this is how it looks, this is documentation using Docly. You can see it’s the k6 office hours CRM application. You can see the list in there. You can see the create, the get. So it’s just the definition of what the API would look like. This is the documentation part of it. So I’m using now two tools. I’m gonna use Portman as another package that I built together with API Deck and make an API Deck where we can convert the openAPI to Postman, and from the Postman, you can generate a k6 one. And that’s really nice because you only need two commands, one source file, and you have two outputs out of the box. So that’s really powerful. I find it powerful at least. So I’m gonna take my commands correctly.
So If it’s okay, can you, I’m not, I’m not up to speed with Portman. So it creates a Postman collection already given openAPI, is something in openAPI specification?
Yes.
That’s awesome.
There was, Postman built a patch like Mostafa, k6 build a patch where you can convert from Postman to k6, and Postman build a Postman where you can convert from openAPI to Postman. And when I found out, I said, that’s amazing. You don’t have to maintain a Postman collection anymore, but it wasn’t, it was missing some functionality, and the major thing that I was missing, it was just creating the request than Postman, but not the tests. It was not handling the tests of it. It was not assigning variables. It was just taking the Postman request and converting them into openAPI requests, into Postman. So you still needed to maintain and modify them. And I said, it has to be, because an openAPI documentation or specification, it describes everything, the request type, the body, the properties, if you do it properly, you have all, everything in one go. And I said, why not use that to create a test suite, update your variables, do all the dynamic stuff you would expect it to come out of the box. And that’s why we created Portman. It started as a PR on that openAPI to Postman package, but then I met Nick from openAPI, from API Deck, sorry. And then we came together and we said, let’s collaborate, and let’s make a startup package out of it that you can plug in and use it to automate these kind of workflows. And it’s really, yeah, I’m biased again, but I really like it.
No, it sounds awesome.
A question here. May I ask a question here? So have you seen the openAPI generator package that I’ve written?
Yes, I saw that and that I had the same comments as I had initially. It generates a nice k6 script, but it it’s usually you still have to manually adapt a lot of things and correct it.
Yeah, but sorry that I interrupted you. There’s this latest PR point? I think Michael, if I’m currently remembering his name, so Michael added a feature to, to be able to add certain extension to the, to your Swagger or openAPI collection so that it generates everything for you. And if you add examples to your API output, to your responses, then it can extract those examples and put it in your script so you don’t need to modify anything. So you can create basically the same thing you did for Postman, you can do it for the openAPI. So the new revision, the new PR and the new version that is released by Ophelia, the maintainer of the openAPI project helps you create a complete test from start to finish with no modification.
Yeah, I think that’s great, but we do two types, three types of tests. We will do contract tests, and then we use Newman for it. And we want to use k6 because Newman, it can do schema validation, it can do a lot, I think a lot more than k6 can right now. I think it’s also the goal for k6 to do all these contract validation. It can, I guess, but I treat them differently. I think the Newman Postman has more contract testing capabilities, while I think k6 is perfect for performance and load testing at scale, where you only focus on what’s the response, is it 200 or is it not a 200, at least that’s how I use it actually.
That’s fair enough.
Yeah, what Portman does it converts openAPI to Postman. So I’m gonna showcase that quickly. So it just took the, this openAPI that’s here now, and then gives a while to spin it up. There we go. It’s converting it. And if things go well… Let me check something happening here. Here it is. So here’s my Postman collection. It’s just generated it. It does everything for you. So it creates, recreate, so there wasn’t a collection there. It just recreated for you. And I hope things, it’s not showcasing it here. That’s bad, too bad. So it just generates and if I would import in Postman, let’s quickly do that… Give me a second to find everything. There we go. So this is the k6 and it’s generated by Portman. There we go. If you see you have the same endpoints–
[Nicole] Oh, nice.
[Tim] So the first thing it’s great because you don’t have to recreate your Postman collection. You just, you just run it from your openAPI specification and you get, as a result, you get your requests pre-tailed already, also your post exercise is ready. So you can see here, it inserts a body, everything, I don’t have to do anything. It just takes your openAPI and converts it to Postman. Plus it adds a lot of tests already. You see, it does contract testing by design. So it includes automatically 200 contact applications. So it just injects your contract test in there. And that’s really nice because I only need to maintain one document, my openAPI specification, as a result, I get all my requests in Postman, and I get all my tests in Postman without writing any code. I just need to maintain it in the openAPI documentation.
Wow this is really awesome, Tim.
So keep in mind that this was the flow before you had to create your Postman collection and add the tests, and then you could generate k6. Now we turn it around. Now we generate the Postman and request, and then we can use that to generate, okay, six scripts. And so we only need to maintain this part in, yeah, or you know, that’s the only thing I need to maintain as a product manager and the team, of course, they built the API and the service and everything, but I get this and I can focus only on this. And this is easy to manage because it’s a YAML, you can play around, you can add endpoints, you can remove endpoints and just run it. A minute later, I have a Postman collection to play with it and I can run it at scale with k6, doing the volume tests. What else can I showcase quickly here? Let me see if I can. I just–
[Nicole] Some more nice comments here. Such an awesome flow, says Gertjan, and James says his mind is blown and, and I join him on that sentiment right now.
[Mostafa] Exactly. That’s a full test suite, like from start to finish.
Yeah, and like example, schema validation. It comes out of the box. You don’t have to add it. It just does it for you. And the richer you make a documentation, the richer the test suite will become. And it also, Portman, be extended, you can also do variations. So now it just validates if it’s a 200, but you can also have it generate 404s, and then it will remove items from it. If it’s a required property, it’ll create a variation out of it. And then you can test I’d the 404s are not present. And so you get two test suites in one go, actually.
Wow. That is, that’s so awesome. Maybe we can like, try and summarize this as we try to collectively recapture our blown minds. So I think that there are two different tools here, Postman and k6, which are really for different things. The Postman is really better for API testing, and as you mentioned, Tim ,contract testing, and k6 is better for load testing. And their features are pretty divergent after a certain point, but they do have a commonality, and it seems like that’s what you’ve really capitalized on. Mostafa and Tim both have worked on this. and you you’re trying to make, you’ve gotten us closer to this ideal of having just one set of test suites, and it doesn’t really matter whether it’s automation test suites or load test suites or, it can just be the one thing and you no longer have to maintain two separate ones. And that’s having like the amount of time that you need to set it up, but also running it. Because having to explain this to somebody else is also a huge deal when you’re not the only tester on your team.
Nope, exactly.
And the nice thing is as a product manager, I’m not technically, so but I can contribute to it. I can, I just need to maintain the openAPI specification and the whole team benefits from anything that I do. If we need to add a new endpoint, I can design it. Not even five minutes later, one minute later, I can just run it in a pipeline, see if it works, and that’s a bit, the cycle of spec driven development. You put your spec at the center, the spec at the center, and then you design it, simulate it, fill it, Then you build it, you test it. But the spec is a test driver here and then you can even run it on production and yeah, do the management aspect as well, because then you can monitor it as well based on your API specification. So that’s, as a product manager, I’m really proud that we pulled this off because that was one of the ambitions when we wanted a newer, or API program, like how can we use a manual workload, what can we automate, and yeah, I’m really happy because it’s all open source based. There is no other software involved. We just use k6. It has excellent dashboards to individualize reports. We use Postman to generate contract testing and manage it, play around with it. And we benefit from such large ecosystem , Swagger as documentation interface, where you can display or openAPI as well. So it’s almost full cycle or I don’t know what’s the term for it, but yeah, I really, I’m really happy with where we are right now. It’s a good place to be in an API landscape actually, especially from, yeah–
You’ve taken it a step further, right? This is no longer just about test suites. It’s also because you are building on the openAPI specifications, it’s all connected even from not just the development part, but the design part. And Pepe actually has a question about this. He says, can you show us more about the process that updates the APIs? What does it do?
How do you mean? This flow, this one?
Sorry, which one? Yes, I believe so.
The powerful thing about this can also be because if you have an openAPI specification, you can even generate clients or service side code from it so that in the end, your API service can also just be generated from an API specification. There is a really nice openAPI, there’s a really nice openAPI open source project, it’s called I think openAPI Generator, and then you can just run a C# library or an API for that. You just take your openAPI documentation and you run, you run it and you get as a result, you get C# code that you can run in production. And that’s also what we use in our company. You have some teams that just manage to build the openAPI specification and they use that to generate C# code and have APIs up and running in no time. I hope that this is the answering this question.
Yeah, Mostafa, Paul, anything you want to say to Tim.
Yeah, no questions but, yeah, that would come later, but I need to definitely pass this along, this video along to some people that I used to work with, because this will blow their minds as well. This is something that they definitely need.
Yeah.
Do everyone.
Great job, great job.
Yeah, great job, exactly. I’m literally speechless because I’ve worked with all these tools and I know all the–
Clearly not literally.
Well, not the ones that Tim wrote, the ones that we had before. But anyhow, this is a great addition to this mix of open source software that was out there and that we used, and this is basically end-to-end testing. So based on a YAML file, which is an openAPI specification, you can generate tests for your API. You can generate load tests. You can generate, I don’t know, like literally everything.
It’s also documentation. You get documentation, nice looking documentation out the box, a playground with Postman and you get the numbers to back it up with even graphs from k6. I really like your dashboarding. So if you have a customer and they they’re asking about SLA, we just pull up some graphs from k6 and everybody’s happy with that. So we, the full ecosystem has really matured a lot over the last couple of years.
Bravo.
Yeah, this is so freaking awesome. Thank you so much, Tim, for coming on. By the way, he’s not even working for k6. He’s just, and I think you said at one point, oh, I’m a product manager. I’m not technical. I beg to differ. This is pretty amazing things that you’ve put together. There’s a bunch of comments here too. James says he’s literally waiting for the livestream to end, so he can link to the video with a timestamp. Well, let’s just try and drag it on for him, shall we? Thanh says plus a thousand votes, and people are also saying thank you, and outstanding, almost the holy grail. So lots of fans in comments. Thank you so much, Tim, for joining us. I know we’re a bit over time. I’m kind of taken aback by how much value you just showed us. This is amazing, I think we need to have you back.
Yeah, I’m gonna write a blog post for k6 and one for API Deck where I’m going to get more into the details because yeah, it’s a show now, it’s a difficult to but I’m gonna try to write it up and then extend a bit on what, sorry.
Mostafa.
Mostafa, apologies, I’m really bad.
It’s fine, it’s fine, no worries.
Yeah, I think can be really powerful to share with the rest of the world that they can, yeah, maybe use some parts of it or not, but I benefit a lot from it. The whole team benefits a lot from it, actually. I can showcase, this is a pipeline we use, for example. So this is we do tests, then we run Postman and we do tests and if it’s 100%, it gets deployed. So it’s not just a gimmick, me showcasing it. This is really happening in our company. So we use it to build it. We use it to have nice dashboard with k6 attached to that. So yeah, I’m grateful that you put it out there for us to benefit from it.
Thank you.
That’s so great, thank you.
Yeah, I–
Go on, Mostafa..
Review the article that you want to write.
Okay, I’ll keep Pepe in the loop and then you’ll reach out.
Of course, of course.
Great, thank you. Thank you as well, Paul and Mostafa, for coming on, Mostafa, even though, sorry, I should have invited you even earlier. Thank you everybody for watching. Thanks everyone in the chat. Have a great weekend, everybody.
Absolutely.
Thank you for having me.
Yeah, thank you.
Yeah, all right, see you all.
Bye.

How to design a workload model in k6, with the k6 Developer Advocates (k6 Office Hours #42)

Fri, 18 Feb 2022 18:20:21 +0100

On this week’s Office Hours, Paul Balogh and Leandro Melendez join me to talk about how to design workload models in k6 load testing scripts using scenarios and executors.

Transcript

Hello everyone, and welcome to another k6 office hours. I am Nicole van der Hoeven, and for the first time, all three of us developer advocates are in one stream. Could you please introduce yourselves?
Oh, who’s going first?
I guess you are.
All right, age before beauty, as they say. Okay, well, hey, I’m Paul Balogh, the newest DevRel here at k6, nice to meet you.
And on my end Leandro Melendez, also known as a Senor Performo, very happy to contribute here and bringing, usually it’s knowledge, but this time I’m gonna be a little bit on the learning side, which is super exciting.
We learn together.
Oh yeah.
Yeah, I mean, I don’t think you can really have one without the other. I think it’s super valuable to also just ask questions.
And Nicole, I’m a little disappointed that you don’t have a fake mustache to support Leandro and I.
Yeah. That is not something that I just have lying around, unfortunately.
We may have to fix that. I was commenting the equivalent here in Mexico is the unibrow.
I could potentially draw something on my face, maybe, I could try that.
Is that the Frida Kahlo?
Oh yeah.
So, Leandro and Paul are relatively new to k6. I think you both have been here for less than a month.
Just a month here.
Oh, just a month now, that’s awesome.
Three.
But this is the first time, I’ve been here with you with each of you separately, but this is the first time that all three of us are here. It is pretty cool, welcome.
Exciting, hopefully it’s not overload for everybody watching.
Yeah. One thing that I wanted to get out of the way is that we have a very awesome community, we have several community champions, but one of them, NaveenKumar, just posted a couple of videos on how to use checks and thresholds in k6. So I’m going to put that in the comments, as well, because I thought that they were pretty awesome videos. NaveenKumar has a YouTube channel where he goes through a bunch of different tools, including k6, but also like JMeter and Gatling, and he always does a good job, so check those out. Oops, I actually didn’t post his videos, I think I posted the documentation for what we were talking about, so that’s a little premature. Sorry about that, let me put the actual links this time.
Well, this is a live podcast.
Yeah, obviously. Sorry, you get what you get. So if that hadn’t already tipped you off, what we actually want to about today are scenarios and executors in k6. You may have been confused by the thumbnail because it was like, I realized afterwards, that it was the three of us, our photos, and then the words executors in k6.
Here come the executors, the three of us.
I guess that’s our new name or something.
You know, if you invert the k6 logo, it almost looks like a guillotine, so maybe we are the executors.
Oh, okay.
I had not noticed that.
That’s a little morbid.
Oh, you know, hey, I do what I can.
You’re going like play Beatles songs backwards and all that type of stuff, man. I’m sure it was not on purpose.
It looks like we’ve lost access to LinkedIn for some reason, but, oh well, this has happened before. Thank you, Mark, for saying we’re killer performance engineers, love the pun. So the reason that I wanted to talk about this was that, I don’t know if you also have got the same, Leandro, Leandro and I were both in the Automation Guild 2022. We both had a talk there and there are a lot of common questions that come up with regards to load testing. And a lot of it has to do with the results, specifically when, I’m not sure if you can actually hear the wind in my background.
Is that you, oh wow.
Are you okay?
I’m okay, but apparently there’s a storm that’s in Northern Europe.
I was gonna say, hopefully you’ll still be in Kansas, right?
I know. It’s like 140 kilometer per hour winds.
Don’t end up in Kansas.
But can I be the straw man, at least, in this Oz thing?
Sure.
I always liked him.
I guess I’m Dorothy. Or the wizard.
I ask Toto, please.
I’ve got Toto around here somewhere, let her in.
So one question that I was getting a lot, or I saw other people talking about variations of it, was the situation where you might have already run your load test and seen all check marks and everything passed, and the response times were within the expected results, and yet when the application goes live, suddenly there’s all these errors, and performance issues, and maybe production incidents that you never saw in any of your tests. And the question is, how can that be when you’ve done your testing and you didn’t find any of that and you only see it in production?
Yes, of course I experienced that several times, but many things come to mind on, in what environment did you test? What happened in this environment? If you test in production, hopefully ‘cause we are modern performance engineers that are not afraid of testing in production. Settings, changes, what was the workload? I mean, I don’t know, the list is pretty long.
Yeah, there’s a lot that, that could, that’s why that question is hard to answer because there’s no one answer. It’s like, oh and that happens, always check this. And that’s also because it could be that there’s something different with the environment. Like maybe it’s you didn’t test the same version of the code even, or maybe you were testing in not a production-like environment in terms of size, or test data, or something like that. But another reason that I think is easier because we testers normally control it is on the side of the script. It could be that you weren’t testing the things that you actually wanted to test. And a big part of that, I think, is workload modeling.
That’s one that I also wanted to comment, ‘cause that has been a pain in different occasions. On one hand, watch model, ‘cause from what you’re saying, there were these projects where in the end we were told, hey, this is a workload, this is a utilization that we are expecting to have in production or that we actually have in production, we want you to test for this. And it was nothing like that. We were told, yeah, it’s a thousand clicks per day, but you wanna test an emergency center when there’s a tornado. How is that true, how is that relevant, or vice versa? For some reason, our scripts are just simulating a thousand clicks per day where it’s 10,000 billions, gazillions, what would’ve been the right thing. Of course your performance test is gonna be awesome if you underdid it. And the other side of the risk, everyone running and, like, the system is dying, what is happening? Oh, we’re pressing it too hard, sorry, our bad.
Yeah, that’s true, could go both ways.
But definitely you always have to worry about configuration drift anyway. Differences in your environments, unless you’re using something like a terraform or something, infrastructure is code where you can make sure that your environments are equal. So if you do your load testing and performance testing in a lower environment or isolated environment that it gets some of that, that it is truly configured the same.
Yeah, even when the environments are exactly identical, though, let’s just say that it’s possible to say that, yeah, there are still situations where if your load tests weren’t simulating realistic enough load, then you can’t really, you may not be able to apply those results to production either.
And one thing, as I said earlier, on one hand, the problem to getting those realistic numbers, at times, the team doesn’t even know, they have no idea what are you asking for. All they know is we want to be able to handle 10 users in the system. I’m kind of simplifying it, usually it’s 10 billion users or something like that, right. Okay, 10 billion users, what does that translate into in terms of utilization? And they go like utiliti-what? And that’s the first struggle, right. Supposing that you get the right numbers that you know exactly how much to stress each thing, ‘cause there there’s another differentiator, right. We get this action a thousand times, okay, but to get to that action at times or on some systems that are the same, you have to do others many more times, less times, and they are just telling you about this one. What is the proportion, how much to stress the other one that the other one does need not that many times? So that’s one discussion. If you get the numbers right then to landing it into the script and your scenarios, also, it’s a very interesting situation, headache-bringing situation at times.
I think part of the problem is that a lot of times, when we think about load testing, there are only a few parameters that we think about immediately. One them is the number of virtual users. Another one is the test duration. And sometimes you might get like a request per second, like you were saying, Leandro, but those three don’t really describe the shape of a load. It’s kind of like trying to encapsulate test results into an average figure, like when you get the geometric mean of something, it’s not a great way to express the distribution. You know, if you say that the average response time is three seconds, that doesn’t mean that you don’t have really high response times. Like you could still have a handful of requests that are at 15 seconds, but maybe the rest of them are a hundred milliseconds. It doesn’t really
There is context.
Accurately describe the whole context, the big picture of the load. And in the same way, things like the number of virtual users, and duration, and requests per second also don’t show the entire picture.
Yeah, wen you’re describing it that way, it makes me think when you are asked to describe someone, oh, has beautiful glasses. I mean, I was talking about Paul, sorry, Nicole.
Oh, sorry.
No, no, I like your glasses, too. But you need a full picture, right? There are so many elements that a scenario or a workload model includes, so many variations, details that, yeah, as you were saying, I need 10 million users in the system doing what, how often, how long, how many of them are doing which ones? And many of those things, you have to get into the weeds to truly understand and design ‘cause this is another one, you will want different workloads for different type of load scenarios. You should not, I was gonna say you cannot, but you can try, but it’s not recommended at all to try to kill too many birds with a single stone. You may be able to get two, but more than that gets complicated, gets crazy. So you have to create a work model for that specific risk situation, that scenario that you want to test for. Just one at a time gets challenging, getting all those components, all those items and things that you need to be careful putting together to model and simulate something that resembles realistically that risk scenario, that risk circumstance that you want to test for.
And here you go again with that killer mentality there, killing multiple birds with one stone. And now Mark’s making comments here that actually for killers.
Okay, okay, let’s say you may be looking for a needle into multiple haystacks, no killing.
There we go.
I was thinking about what you were saying about the hypothetical scenario of a thousand requests over 24 hours. There’s still so many ways to interpret that, right? Because it could be, one interpretation could be, take a thousand requests divided by 24 hours and that’s the amount of load that you want to test with a one-hour load test. It could be like that, but if it’s emergency services, there are certain times of the day when emergencies are more likely. It could be seasonal, as well, like the incidents of emergencies of ambulances that are sent out increases over the holidays for inexplicable reasons. That kind of thing does give more context than just a thousand over 24 hours. I mean, if you just look at a period of 24 hours, those thousand could have been simulated, could have happened in one hour for all you know.
Step in a little far from the ambulance example, don’t wanna go more on the killer topic. Let’s think of a restaurant, if they say, this Golden Arches restaurant, we sell a thousand burgers per day. Okay. Most probably you are selling 80% of that at lunchtime and the rest of the hours we don’t care about. So those different behaviors during that period of time is what we should focus on, or care for, or have better information. Well, in the morning you get more, what’s the name of the thing, hash Browns, no burgers. Even if you are the same restaurant, which one do you want to simulate? What is the utilization at different periods in time during the day? ‘Cause yeah, probably no one, not that many people get a milkshake in the morning, but most of them getting in the night. So some of those variations, you want to make sure that you are doing the right mix, not only how many at a single number per long period of time, what happens each hour of the day, which ones do you care for. As a restaurant, I would say I don’t wanna simulate a load test of, I don’t know, after lunchtime where everyone is gone and we are super quiet. I don’t care load test-wise, right. But lunch time, breakfast time, I don’t know, 7:00 AM in a gym, I want to test those type of things of what people are doing and focus on, get a good work load model for what is happening at those times, right.
So I thought, Mark has a couple comments here. I think also the number of vusers or threads compared to the target arrival rate or expected response time for me is the health of the workload generation. And he follows up with, also synchronizing or rendezvous vuser requests is underused in my opinion. Do you agree, Leandro.
Well, it’s in my personal experience, there have been a few times, at least on the render view or synchronizing part, there are not that many times where I had to leverage those functionalities. I think it kind of depends on the projects that you work on. One that comes to mind, and I really love this example for workload modeling in general, is I worked at a gaming company, Australia’s largest gaming company, and the project that I worked on was for the Melbourne Cup. So they sold tickets for the Melbourne Cup. And if you don’t know, and probably you don’t unless you’re Australian, but the Melbourne Cup is a horse race. It is like a two-minute horse race that is for some reason incredibly popular in Australia, in my state, in Victoria, ‘cause I lived in Melbourne, we actually got a public holiday for it. So, it’s that big. And for some companies, for a lot of these betting companies, the revenue from that one race could be up to 80% of the entire year’s revenue. So it is that big.
Just in that two minutes.
Yeah, and that’s the thing. Betting is open until the race starts. And yet when you look at the workload model for that, it really, the load spikes, the betting spikes just in the few seconds before the race. And that’s for a number of reasons, people might not have had the chance to bet earlier in the week because they were working or something. And if you have a public holiday, then you do have the time to just be there watching. And also there are, depending on the types of bet that you placed, there could be a difference in odds. So they’re trying to get the last possible information that they can before they have to make a decision about which horse to back. And that’s a good case for a rendezvous because there is a clear cutoff and sometimes you see the reverse, like for example, in a university portal or one of these shopping sales, where at midnight all the sales go live. And so you get people maybe refreshing that page all up to that point and then, bam, all of a sudden they start checking out things. And that would be, I think, a good time to use rendezvous.
No, and in a quick example, you got me thinking, as well, video gaming, when you are on these multiplayer games, most probably everyone watching this are familiar, campaigns, online campaigns, certainly there are so many things happening, there are people playing other things, but a campaign is gonna start at this moment, this schedule. I think there a rendezvous would be a very good use of those type of things. Sadly, in my experience, some of those situations were solved with a spike scenario run out here with an average day in production, so overlapping scenarios. But that’s interesting ‘cause, as well, it depends on how do you want to tackle that problem. We are going to have a normal day and suddenly something’s gonna happen. How do we simulate it? How do we model that workload? What tools do we use? That’s a very good point, and again, with performance testing and load testing, there’s no one size fit all or the way to do everything. No, it depends, and you can get two very similar loads, simulations, workloads with different approaches, with different things. And that’s one that I also wanted to mention, like when modeling, it’s something that when you wanna try to get, there are two elements that I like a lot and I struggle a lot of times, when you are modeling your workloads, you have to somewhat randomize the actions. A big thing that we used to leverage in the load runner days was randomized think time in between steps. But eventually you have to say, okay, it’s randomized, it’s happening, like real people is using the system, but in the end I want just this process, just this virtual user to iterate a thousand times in an hour. Oh, but I have random times in between, okay let’s, that modeling, as well, I can tell got so interesting. And even like mathematic, not too complex, but mathematic functions like, okay, over time we have this deviation, if everything goes in the higher random wait time, we may get up here, we may get down there, so on average we’ll be simulating this. It’s all so challenging.
Yeah, those delays are definitely part of the workload model. In k6, we call them sleep. In JMeter, they’re timers. LoadRunner is think time. There are a lot of different terms for it, but basically a delay is some sort of planned sleep in your script that is meant to simulate a real user maybe taking time to think about what they’re going to do next, or read the text on the screen, or type out something in a form. And it actually can have a pretty significant impact on your load tests. First of all, if you don’t have any think time at all, then you need to be careful that you’re not maxing out your CPU and memory utilization on your load generator because that situation where you’re just very quickly firing off requests may not be entirely realistic. And I think that if you have think time that’s always the same time, like this is what you were talking about, having dynamic ones, if you always use the same value for your think time, you might see the staggering in your results, but that’s due to how your test is set up and not at all to do with how your application is responding. And then if you use dynamic think time, like you said, it can make it harder to plan because it’s more difficult to know exactly how many iterations your test is going to complete. So there’s always trade-offs, right. There’s no one way that’s always the best way.
And it depends, as well, on what are you actually trying to simulate. As you mentioned, there are times where we put think times that want to simulate a real person using the system. There are some others where you just don’t want these events to happen at the same time. To give you an example would be like if your script was a soldier marching one, two, one, two, and you’re adding soldiers at the same pace, one, two, eventually they all will be doing the same step at the very same time. If you are simulating a military parade or something like that, that’s awesome. You have the test. On that time, you may want that to happen or, or work it out with some rendezvous continuously happening. But if not, if you just want to simulate a regular march or people passing in the subway, they are not like one, two, one, it would be very interesting to see, but that’s not how real people walk in the every day. There are situations for everything, and it’s challenging at times, as you say, not only wait in between steps, but waiting after you complete a cycle and you want to start another one. There, you may also want some randomization or not to be that static so that your generally load simulations do not have to be that robotic, even if they are done by robots or virtual users. So there are several trade-offs.
I have an extension to your metaphor, I think you’ll like, Leandro, what if instead, this is a way that your script might change depending on the business processes and the context of the industry. So what if instead of trying to simulate a military, what you’re actually trying to do is simulate Fremen running across the open sand, trying to not have patterns in their steps to avoid attracting Shai-Hulud, the great worm. If you don’t know the reference, I’m sorry. But in that case you do not want anything constant. You want something more dynamic, something that isn’t easily noticeable, that won’t skew the results as much. Oh, we have a question from ChipsAho. I don’t know if it’s just the Filipino thing, but there’s actually a good packet of cookies that’s called Chips Ahoy.
Ahoy, love them.
If you are faced with budget constraints, how would you prioritize the workload model to use? That’s really good. And they follow up with, as performance engineers, I know sometimes our curiosity tends to get the best of us, trying to see how the system behaves in different circumstances.
Well, answering to, oh yeah, ChipsAho, is from Ahoy, very good cookies, put ’em in the microwave eight seconds and you’ll love ’em. So it depends a lot, I would say, and your budget constraints, what are you testing would be my question. What risks does your application will face, or what are the most common reasons? Again, there’s no silver bullet or single test that will clear all your performance risks ever again that you may have under loads. What loads were you? What things may harm your system? Those are the ones that you need to focus and prioritize based on that risk. If you already tested one, you know that it’ll survive, next test, maybe go to the next one and down the ladder, the risk ladder. But first it depends on your system. What is it going to subject to? What are the processes? And the risks may be different a spring ago from today. You may be releasing a new process that is super critical, super dangerous, will be touching databases or doing important or even dangerous queries. This time, you may want to focus on that one. There’s no one test to rule them all, one workload scenario, so it depends.
I think there are a few things that I would think about first. What is, like Leandro said, what’s the biggest risk? Meaning if there’s something where if you don’t load test this component there might be security implications, test that. Anything that’s critical, like payment or something that might affect compliance regulations for whatever industry it is, meaning like how you handle individuals’ data, potentially identifiable data, that would be worth load testing. And also anything that’s very public, so maybe something like you’re about us page is nowhere near as public as your homepage. It can deal your company very significant damage, but hard to quantify damage, in terms of reputation, if your entire homepage is down, rather than if your about us page is down. So that could be a situation where something is not really mission critical or important to the business functionality, but it still would deal a lot of damage. So that’s a lot of reputation risk, I guess. A question here not about executors, but we can quickly talk about that. Do we get any performance HTML report in browser k6 load testing? Not yet ‘cause this person specified browser. So I don’t think that we have that yet for browser, but there are a few extensions. There’s the xk6 dashboard one by Ivan Skiba. And there’s also xk6, what is it called, xk6 HTML report or something like that. Go have a look at the session that I just had, the show that I just had with Paul last week, actually, where we talked about this very thing. They’re both extensions and they’re both awesome. They give like a, the Ivan Skiba one, I think, is the one that you did, Paul, right?
Yeah, and actually I was gonna show that a little bit later here.
Okay, all right, well let me know when you want to.
Stay tuned, yeah.
We might as well jump into that, ‘cause I definitely want to show some things about how to actually implement these workload models in k6. So one way that you can do that is by using scenarios. So I have, let’s just start with this one. So I have one script here, it is a protocol level script. And I’m just putting it in a separate JavaScript file just so this is a little cleaner to look at, so that we can focus on the scenario. In k6, we have something called scenarios that are ways to set things that run independently of each other. So you can have it be consecutive, or run one before the other, or you can have them run simultaneously. So this one just has, this example that I’ve got up now is just one scenario and it has an executor. So let’s talk a little bit about executors because this is something that’s specific to k6. So let’s look here. An executor is basically a way to change the way that the load looks. There are a bunch in k6 and I’m actually going to, I already, ‘cause I get this question a lot, I’m going to put this in the chat because I have this cheat sheet for how to go about thinking, deciding which one to use. So I’m going to post it in the chat and we, I think that the rule of thumb is if you want to just try one, you can, or you want a simple one, it’s probably ramping-vus that you want. And that’s because this is the default one and a lot of load testing tools where you set the number of virtual users and duration and then sometimes you might have stages. This is also what k6 defaults to. So let’s have a look. This is what that looks like. As you can see, I have ramping-vus set here, and this is a way to change the load depending on the number of virtual users. I also have these stages here. The first one is kind of like, I was trying to simulate a ramp up. So from when the test starts, this will start with zero virtual users. I’ve got that set here. And then it’ll gradually ramp up to 10 over five minutes. This is shorter because I think I was trying to, this was from a demo, so I was trying to make it something that we’d actually get results from, but this could be, you know, 30 minutes, if you wanted, depending on your test. And then this is kind of what we call steady state, where the number of virtual users doesn’t fluctuate. It’s just holding steady for this period of time, in this case 10 minutes. And this last one where it goes from 10 virtual users to zero is reflective of a ramp down. Paul, did you want to share your screen ‘cause I know you said you had some things in cloud to show us.
I did, yes. And then unfortunately I wasn’t able to share for my script, there’s something going on with my system here. But let me go ahead and share my browser, which is working. So yeah.
Awesome.
So this is a script that I had done and what I did was, as I was trying to go through and then figure out all these different executors, I wanted to see the pattern that each of these executors applies. So now here, let me, maybe it’s better if I start down here. So this is the actual script that I executed. So if you look here, you’ll see that in these scenarios and hopefully, let me see if I can.
[Nicole] Yeah, I was just going to ask.
[Paul] A little bit more legible, there we go. I’m not the only one that may have old eyes and just can’t see. So I loaded up the options so that I would have all these different scenarios. ‘Cause I wanted to see every one of these executors and I wanted to be able to see what kind of pattern that they applied. And originally you’ll see that some of these numbers are a little bit different, but these are all the very specific things that each one requires. So in this case, the shared iteration scenario, and again, I’m not gonna be the expert on what each of these different executors provides, but they’re in the docs. So yeah, we can go through there. I’m gonna be working on these docs, as well, in the next couple weeks and try to clean these up a little bit more, maybe make ’em a little bit more, I don’t know, state it a little bit differently or to help my understanding anyway. But yeah, so I loaded this up, so I have each one of the different executors set up. So here’s the ramping, so similar to what Nicole was talking about with the different durations. So here over a ten-second period, I had it ramp up to, I think, from zero to 10 vus. And then for 10 seconds I would have it keep going at a level 10, so it would be a flat top. And then the last 10 seconds would be ramping down to zero. All these other ones here, too. So, I put every one of these in there so that way then we could go ahead and compare them and, whoa, let me, all right. So each one of these is kind of denoted here and I guess I can’t, I wish I could isolate these and then what’s really, I guess, the most important here is the purple line and hopefully you’re not, no one’s really color blind or missing the color differences here. But yeah, so the purple line is the actual number of the request coming through. And then this blue line, or the aqua, is the actual response time for the service. We’ll just kind of focus on the purple line that’s what these executors are really affecting more so. And then the gray line, this shaded area is actually the virtual users. So you’ll see that here it started at zero, and then it ramped up for a period of time, and then it started kind of flattening, and then it would ramp down. Now this is for the accumulation of all the different executors. Unfortunately, and this might be something I might put in as an issue to possibly add or an enhancement is that each one of these lines here is a scenario. So this actually has the shape from that particular executor. So it’d be nice to show the vus in here, as well, but you can kind of infer it based on the activity, the request. So again, that’s the purple line again. And so here, you see this executor actually ran pretty quickly to get to the number of requests. I was trying to keep them about the same because I wanted to do, like, a progress bar race. So when I was running this locally, I could see the different areas in which was actually completing the same number of requests faster and all that. But yeah, so this was kind of my playing around and, again, this is on the cloud product. So this is an add feature. But now with talking about the HTML reporting, this is what I actually did here. So let me go ahead and I’m gonna go ahead and run this script again. Now unfortunately, you’re not gonna be able to see the actual execution as I run it locally, but let’s see.
[Nicole] Paul, we got a question about the k6 cloud results that you were showing. Could you create a shareable link so that people can also have a look at it?
[Paul] Well, let’s see. How can I do that?
[Nicole] You can go to the three dots in the upper right. Yep, and then share test results.
[Paul] All right.
[Nicole] We can just copy and paste that into the chat.
[Paul] Cool.
[Leandro] And then they are out into the world.
[Paul] Out into the world, yeah. Awesome, cool. See, today I learned again.
[Nicole] Yeah. Awesome. I’m so glad you did this, though, because I wanted to, but I didn’t get around to doing it. So I’m really glad you did.
[Paul] Yeah, and then I wanted to actually have them separate so I could kind of show each one’s pattern again. That’s the thing that I was really trying to get to, and I’ll probably put together a blog post, so keep an eye out for that, where I’ll kind of go through that, and describe, and show patterns that are created by the executors. And then that is as I learn more. Again, so this is, this dashboard that I’m showing here, this is actually an extension. So I did compile this into the k6 binary, custom with this output. And this is available here. This is this xk6 dashboard, and this is also included in our registry on the site, so you’ll be able to find this as well. Pretty easy to do this, the xk6, to create the extension and build it. But anyway, we can always give help. Anyway, I’m gonna go ahead in this other window, and I’m gonna go ahead and start this test running. So now as we see here, and I shouldn’t have to refresh, but let’s see, go ahead and refresh here. All right. So now this test is running in the background. And then we should start seeing the output. There we go. Now, there is a delay. So behind the scenes, this is actually using, it’s creating Prometheus-based metrics and then it’s serving that up in a little web server locally. So it’s really cool. So I haven’t been able to play around with it nearly as much as I need to, but anyway, but this is something, it’s a nice little tool to be able to use if you don’t want just the plain text display in your console. But yeah, so this test is actually already completed. Now, it’s actually still running in a way, though, because we have this externally controlled executor where now from another terminal I could sit there and I could say, okay, yeah, now I wanna ramp up the vus to 10 or that type of a thing, just via rest API. But unfortunately, due to technical difficulties, I can’t show you that, but I will definitely put that in the blog post when I start putting that together. Anyway, this is one of those cool little tools with this extension that you can kind of, I don’t know, really get some cool results. But yeah, that was the big thing. Here’s for the external access, this is the rest API where I could sit there and call that custom executor and say pause testing or even to ramp up the number of vus. Still figuring this out a little bit. It’s a little, I don’t know, it needs some play around with it a little bit to get a good understanding of it. But yeah, I wanted to show that, again, you can do this stuff, have the graphics there. Now, the HTML report that Nicole was talking about, that’s something where, yeah, you could save that off, and again, it doesn’t have the graphics, I don’t think, if I remember correctly, but it does have it where you could then save that off.
It’s not real time either. But it is more like a report.
Yeah, yeah. That’s it, let me go ahead and stop that test. Oh yeah. So again, I will put together something so that way then if anybody wants these details and obviously you can use that link, so it’s now shareable, so I will make sure to not delete it .
Great. I do want to share something, Paul, if you’re okay.
Yeah, absolutely.
Great. So I also wanted to talk about the fact that we can do it, you can put together a scenario using different executors, but another thing that you could do is have each scenario be a completely different test. One could be a load test. Another could be a chaos engineering test, for instance. Like this is one that I had where this get Pokemon is an HTTP request that just hits the API. And this is intended to be more of a load test, but then you could have a chaos scenario and you can say that at a certain point, you fire off this thing that then terminates one of the pods in Kubernetes, according to certain conditions. This is using the xk6-chaos extension. But in this way, you can see that these are not just load-testing scenarios. These are just testing scenarios and they’re different types of testing that you could use. Another type other than chaos is browser testing now that we’ve got xk6 browser. So now you can do things where you have one scenario that’s doing some sort of protocol level test and the other one is doing a browser level test. And because those two are fundamentally different, you may want to be able to control them separately. You probably won’t be wanting to generate majority of the load with a browser level test, for instance, you probably wanna do that with your protocol level test, but you might want to piggyback off of the protocol test and run a few, like a handful, of browser users just to be able to get some front-end metrics. So, I think it’s kind of cool that you can do that. Another thing that you can do with scenarios and executors, well, it’s something with workload modeling anyway for k6, this is a script that I’ve got for percentages. Now, I think the use case for this, I mean, there are a lot of use cases, but what I actually think this could be used for is an e-commerce scenario where a typical scenario is to have like 100% of the virtual users go into your homepage, and then maybe 50% of them go and browse through the product pages, and then of that 50%, maybe only half of those actually added to the cart. So that is also something that isn’t directly related to scenarios and executors, but it does have something to do with the workload model of your test. So in this example, I’m setting a random number between one and 100, and then depending on that, I have this combination, in this case, I’m setting different clients for it. So let’s say that there’s a really big client that we want to use 50% of the time, you could use that, and then pass that on to some request that you’re sending. And in this way, you’re already making something that’s more realistic than just get a client, just get details for a client. Well, which client is that because maybe that’s being cached? Maybe you do want it to be cached, you’d have to see what you want, what you’re trying to achieve with that particular test, but it all has an impact on your testing results.
Yeah, and if I may add for learning some of these functionalities, it makes, and I think I started a comment how the situations, how the ramp-ups work, how when we want to model these targets in the old days, as I said, getting these, okay, my user takes this amount of time of execution, it has this amount of random timers, and it’s a pain at times when your process starts to degrade a little bit of performance, and because of that, they don’t go as fast as you need them to, as you want to generate, because you did the calculations based on optimal response times, right. So some of these controls will be very useful depending on the scenario that you are wanting to focus and not to do all that type of crazy math that we had to do. I remember with the load runner days, it was like, okay, but if the response time starts to deviate a little bit, this we need to, there’s so much that on this situation with some of these options, k6 can mutate may be the wrong word, but adapt to what you’re trying to do, right, and keep doing the loads and doing those processes that you are aimed for. There are some others that you are like, if the system starts to go slow, the users will slow down and that’s right, that’s okay, that’s normal, but on others it’s like, no, I absolutely want that the users keep coming, keeps hearing the system, even if the system is slowing or many other things that can start to happen. These controls truly make our lives easier. in so many of those situations like crazy.
We have a question from SJ who says, is there a pacing config in k6 or maybe a throughput controller equivalent? So let’s talk about pacing, Pacing and load testing terms is related to think time in that it does impose some sort of timing on the requests, but while a think time is more like a direct sleep on a thread, pacing is more like the space between each iteration and there are different ways to achieve this. k6 doesn’t have something that is exactly pacing. But for instance, the way that JMeter would do this is by using the throughput controller, which is what I think SJ is alluding to. And so instead of specifying the time between each iteration, JMeter does it by allowing you to set a target request per second, some sort of throughput. And we do have that. Let me just share my screen again. So this is the executors that we have. And I think this constant arrival rate would be the best for that. Because if we look at the example here, you can set a rate here, a requests per second, that is your target. What k6 does is it sends a request or it does one iteration, sees how many requests were sent, and then adjusts the next one, waits a certain time so that before it does the next iteration to adjust to that. So if it already did, you know, 200 requests per second the first iteration, then maybe it’ll just do the same thing for the second iteration. And it will dynamically change throughout the test. So maybe at the beginning of the test, response times are faster because not all the users have ramped up yet. In that case, the pacing might be slower because there’s no delay on the server side. And so the script is easily hitting the 200 requests per second, but you’ll find that later on throughout the test, you may experience higher response times, and in that case, the pacing will be quicker. So that number will be lower so that it still reaches the 200 requests per second.
Yeah, it’s part of what I was mentioning that easiness of. JMeter, I remember before they didn’t have that plugin for the throughput, what was the name, I just forgot about it. At the very first times that we started to use
Throughput controller.
Throughput controller, thank you. We didn’t have those type of controllers, ways to managing them. And again, bringing the load runner days, I don’t remember many useful devices to go through this. And on this case, when you have a scenario that at times when we were designing this workflow, this scenario, we would know, okay, I know that per user, I want that in an hour these many iterations happen. So if you want that, you can use the per vu iterations executor that will help you to quickly configure and manage that, you do have the iterations per user, and maybe you have 10 users, 20 users, you don’t care about the number of users. Then you select the shared iterations. I just need this throughput to happen this number of times in this period of time, maybe next time I will have more virtual users, maybe next time I will have less. So you have more control on the end game, to call it in a way, that you’re looking for, rather than, and I cannot say this enough, it was a struggle to try to control, and simulate, and reach those goals that you were trying to get in your simulation, and there are so many reasons, as I said, the wait times in between steps were randomized before, as Nicole says, you have the pacing times in between each cycle, and it was so many, like three factors, main factors, I would say, wait time, pacing time, and processing time, each step would add up to that time that there are so many variables that were somewhat random that you could not control. So here you would have some assistance, some help on hitting those goals that you’re trying to simulate and having it realistic or on the goal that you’re trying to reach, right. It’s that difference, I don’t know if anyone has gym experience or has ever visited one of those things, but they will tell you, hey, I want to work leg today. Okay, do five repetitions three times. That’s if you wanna think of that, how long do you take and do each sit up to go, to give an example, that will be your think time, your wait time in between steps. Okay, you need some rest time before you do the next one. That’s the pacing. So for the ones that are not so familiar with those terms, just wanted to throw out that silly example.
It’s one of those things I’d like to, I don’t know, I keep thinking of playing around with, you know, if you look at say HTTP audit logs or something like that, they have a timestamp on there as far as when everything came in. Well, if you take that and use that as the basis to basically replay your activity, I don’t know, that’s one of the things that I keep thinking about, that would be cool to do, or to have, you know, all your activity going into Kafka, and then being able to repeatedly replay the exact same, perfect storm scenario without having the randomness injected in there. I don’t know, I’m probably going off on a tangent, but that immediately made me think of that, though. It’s like this is one of those things where it’s like I always kinda wanna play around with this idea.
I think that there’s a time for that. I think there’s a use case for it, but there are also some disadvantages, I think. that people should be aware of. If it’s information from a production environment, there are potentially security concerns there for using a user’s data to test or a user’s account to test. And what happens if that actually gets replayed on their account, which it could be, it, it might not be. Sometimes it doesn’t work, sometimes sending the same request, even if it’s exactly the same, won’t work. I think that that’s, I think, more experimental, I guess, when you don’t really know what you’re looking for and you’re trying to narrow it down, but I also think that once you’ve narrowed it down, it’s probably easier and cleaner to script it up so that you can consistently reproduce that scenario. And you can also use test data. Also, you’d be able to kind of break it down, you’d control all the variables, so you know with certainty that when there’s only one thing that changed and it was a negative result, then you know that that was the cause of it. But yeah, I think that there’s a valid reason to use that, as well.
Yeah, there’s definitely data sanitization that you gotta be concerned about unless we’re replaying activity on Leandro’s credit card, then it’s okay.
No, please don’t. Oops, there are 10,000 charges per hour in your credit card. What?
Load testing your credit card. Navin Kumar, whose videos I was just plugging at the start of this has a question.
Very good question.
Can we intentionally include sleep time in the response time? Now, Mihail, who’s one of our developers, if I say something wrong, tell me in the comments, but I believe that there isn’t a way, that there’s no k6 specific function to do this or option. What I’m thinking of is you could create a custom metric because you can create really a bunch of different metrics, counters, and timers, and gauges, and whatever in k6. So you could definitely do that and have that reported and you could set the timing manually, as well. Another thing that I’m thinking of is if I had to do this, I’d probably just use the xk6 browser extension, because that’s already going to include everything, the actual render time. ‘Cause now we’re getting, if you’re including the sleep time or the think time, this is more like getting into the front-end performance side of things and I think it might just be easier to do that.
That was gonna be kind of a question that I wanted to do, too, Naveen. Like why do you want to include the sleep time programmatically that you are generating in your script? I mean, as you say, Nicole, the front end is important to know those render times, those wasted moments in the response or back-and-forth communication, but what are you trying to use it for? It would be very interesting if you can share it. I’m curious what’s the use. Oh, okay, you were just being creative.
Yeah, I think that, yeah, probably if you want to just see front-end experience, I do think it would be more useful if in this case, if you’re including it in the response time, then maybe have a constant sleep, actually, because then it’s consistent because you don’t want the sleep to be influencing your results.
I did use something like that a long time ago, but precisely to model this workload to calculate. I generally recommend do a small-ish version of the load test that you want to do, let it run for a while so that you can massage those values of how much random time to sleep here, how much random pacing you have here, and how is the response time affected so that you can balance it out. That could be an interesting use, real-life example, but that’s a good question, as well.
I mean, injecting a network latency and things like that is always a nice thing for backend services. I mean, with microservices, because one single API call on the front end could be going through 10 different microservices ultimately. And to be able to test something where if halfway through those calls if you have a network hiccup where one of the services is not accessible, but it’s, anyway, but it’s all one of those things where it’s like this feels a little bit more on the chaos engineering aspect of things, you know, how is your application going to behave when something goes wrong? If all of a sudden your network grinds to a halt where it’s going really slow, is your application handling that scenario appropriately?
Okay, and on that note, we are actually a little bit over on time already, that went by really quickly. Thank you both for coming and for sharing your thoughts on these topics and also for demoing the k6 cloud stuff that you already had, Paul. Yeah, you are welcome back anytime and I hope we do this again. And for everybody,
I thought that was the last one.
Oh, well, sorry to disappoint you. You’re stuck now, like you’ve signed the contract, that was it. You didn’t read the fine print. Must come to office hours every Friday.
This would be the last time that you would want me on here.
Well, for those of you who are watching, thank you for watching. Thank you for all the awesome questions we got this week and have a good weekend. And if you are in a place where it’s very stormy, good luck, hope we all survive it.
Stay safe.
Bye everyone, have a good weekend.
Take care everyone, adios.

Getting started with using k6 extensions, with Paul Balogh (k6 Office Hours #41)

Fri, 11 Feb 2022 18:00:24 +0100

In this week’s k6 Office Hours, I was joined by my new colleague Paul Balogh, to talk about his experience in joining the k6 team and his first impressions of the xk6 extension system.

Using Executors and Stages in k6 with Théo Crevon (k6 Office Hours #40)

Fri, 04 Feb 2022 20:01:12 +0100

Today, Théo Crevon, one of our newest Senior Software Engineers at k6, joined me and my colleague, Leandro Melendez, to talk about his experiences in joining k6, and why scheduling in a load testing tool is simultaneously difficult and important to get right.

Señor Performo's Week of Load Testing (k6 Office Hours #39)

Fri, 28 Jan 2022 23:27:47 +0100

This week for k6 Office Hours, I talked to one of the newest members on our team, Leandro Melendez, aka Señor Performo, about his experience in joining k6/Grafana Labs. We ended up talking about performance “deadly sins” and how the practice of performance engineering has changed in the last decade.

How to Do What Kind of Testing When: Testing Types by Scope

Sat, 22 Jan 2022 16:42:49 +0100

Correctly identifying when you test could be just as important as what. Below are types of testing according to scope.

Static testing

Static testing occurs when requirements are written. Question requirements and spell out what each one means.

Concerns: Setting up a system to track and link requirements to test cases, negotiating business needs with technical feasibility, clarifying scope.

Unit testing

Unit testing occurs when coding begins. Practice Test-Driven Development by translating requirements into tests before writing the code.

Concerns: Treating test code like application code, getting developer buy-in, communicating technical implementation, performance, pair programming.

System testing

System testing occurs when a build is released. Employ black-box testing: step back from the code and assess the feature as a whole for consistency.

Concerns: Adherence to common design standards, equivalence partitioning, boundary value analysis, fighting scope creep, usability, creating test data.

System integration testing (SIT)

SIT occurs when integrating multiple components. Use white-box or grey-box testing to target the interactions and traffic between components.

Concerns: API testing, stubbing and mocking, identifying request paths throughout the system, performance benchmarking, observability and analysis.

Business acceptance testing (BAT)

BAT occurs when the code is assessed against overall business goals. Seek out SMEs who can help you verify expected outcomes.

Concerns: Adherence to business requirements and regulations, process testing, simplifying request flows, identifying potential customer pain points.

User acceptance testing (UAT)

User acceptance testing occurs before code is released to production. Map test cases to requirements and test real-world scenarios.

Concerns: Testing for superuser workflows, data cleansing, verifying real user behavior from historical logs, exploratory testing, end-to-end performance.

Regression Testing

Regression testing occurs between builds. Supplement a robust automation test suite with nuanced manual test cases that address known problem areas.

Concerns: Setting up a CI/CD pipeline, deciding what to automate, consolidating test and development tools, tracking failures over time.

Question everything

Remember: testing is an attitude.

Canonical Observability with Simon Aronsson and Michele Mancioppi (k6 Office Hours #38)

Fri, 21 Jan 2022 18:23:47 +0100

Today, my former colleague Simon Aronsson brought his new colleague, Michele Mancioppi, to k6 Office Hours to talk about observability, and how they do it at Canonical. It turned into a great conversation about trends in observability from the last decade, and how it relates to performance and reliability testing.

Transcript

Hello everyone, and welcome back to the first k6 Office Hours of the year. Happy 2022, I’m Nicole Vander Hoeven.
And I’m Simme, we’re back from the dead.
And I’m Michele.
Simme is back. Yes, so Simme unfortunately is only temporarily back, and he’s brought his new colleague. I mean, I’m kind of sad, but I’m happy to meet you Michele Manciopi.
Hello everyone.
Welcome. Simme why don’t you first tell everybody, how you betrayed us at k6.
Sure, I’m so sorry for that, was a couple of months ago now. I think I quit at k6 in October, beginning of October something like that, to join Canonical and Michele, where we are trying to build a really awesome observability stack to know little part built on top of Grafana products.
Yeah, it has been a very interesting three months. It’s actually a pretty refreshing to be able to use high quality software and compose it in ways that are, you know, they just work through, cool.
So we already have a few people in the peanut gallery, there’s Simme and Michele and our newest colleague, I think senior performer, all saying hi to you Simme. so you’re very much missed.
Congratulations to k6 for bringing Leandra.
Definitely, and by the way, Leandra will be on k6 Office Hours next week, he just accepted the invite. So when I’m letting him like relax, and do his bootcamp onboarding things, and his week of load testing this week. . Ooh Danielle is here as well. Michele, why don’t you talk about yourself, and what you do at Canonical?
Well, I am a product manager at Canonical, for observer BD team. I was brought in with the goal of making the entirety of the product line of Canonical observable. Canonical is chiefly known for Ubuntu, but we have like a meter other products and they all need to be observable because they all sit at different levels of critical pieces of the infrastructure. Before Canonical, I was the technical product manager at Instana, which has been what baby ham it’s a product that does very automated observability. It’s mostly proprietary, but it was a lot of fun trying to make observability the way it’s supposed to be. That is seamless. Very easy to roll out. Before that I was at SAP where I was the technical lead of the performance team for SAP cloud platform, where my job was mostly to figure out where things were slow and fix it. And then I realized that rather than helping one single developer at the time, would they be more productive in actually providing pretty decent tools to do that at scale so people could do it themselves, which then brought me down the path of figuring out it’s actually fun here to make an observability product than it is to using one. And then the CircleCI.
So Simme what was it about Canonical that won you over?
I actually think it was to no little part Michele won me over with his argumentation for–
Okay. I’m taking him off the stream sorry. No, no, No.
‘Cause I got you everyone, bye.
No. We’re still all friends.
I do really miss the k6 family for sure. But yeah, it was just such a cool and exciting opportunity to be able to make this huge leap in observability with Canonical, for such a vastly used project or suite of tools and products. So that’s mainly one of them.
I’m sorry. Can we stop to remark? That was the coolest cup I’ve seen in a long.
It is as if this is sponsored, this is a hydrate spark, it’s a smart water bottle. And it’s marketed like for people like me because it’s a water bottle, but it has Bluetooth and LED light. So, you know, obviously I had to buy.
Yeah, just out of curiosity, what are the key values of the Bluetooth support?
Oh, it tells me when I need to drink water. It’s very important stuff.
How do you monitor that A, the Bluetooth is working and how do you receive those notifications?
Well, it’s funny that you ask, there is an app for it. This is totally not sponsored. Okay. But there is an app for it. I was gonna say Simme that it seems like there was a slight change in your focus as well, because at k6, we’re more focused on the performance and reliability testing part of it. But definitely we’ve also dabbled in the observability space, especially with the acquisition of Grafana. So it seems like your new role is also more focused towards observability. Was that one of the considerations?
Sure it is. And then that was definitely one of the considerations during my, what is it now? 12 or something years in tech, I’ve been doing way more observability and monitoring work than I’ve been doing performance testing. And so that definitely felt like coming home in a sense to me.
And I wanna ask you both this because you’ve both been in observability for awhile. How do you think observability and performance testing go together? Do you see them as separate disciplines entirely, there’s clearly some sort of overlap, right? Should companies be doing both, one or the other?
I definitely think there is a lot of overlap. One thing that I know that you and I have been pushing forward when we’ve been discussing this previously on the stream is that performance testing is great, but it usually only allows you to do a black box perspective testing, right? You’re able to measure the performance of your system, given certain interfaces or end points that you are able to observe the performance off and pairing that with observability tools, like for instance, loci or tempo, or Prometheus’s allows you to really drill back into the stack and figure out what is causing those problems. So I definitely think that it makes sense to combine the both to be able to observe the whole system and not just the edges of it.
Do you agree with that Michele?
Yeah, very much, over go and say that actually, it’s very hard to draw a demarkation between the performance and of observability, especially since in the last few decades and the increasingly disproportionately larger role that internet and internet systems taking our lives. We have come to see, you know, slow as the new down, which means that not only the software needs to work but it needs to be fast. And if you observability is fundamentally the art of understanding, are there problems? I mean, so how bad was impacted and how badly, and that is both in terms of software, it is broken as software is luck. Many of the techniques that you would use in, for example, a QA system to check the scalability of your software is something that you can use at scale in production to make sure that you’re at least carrying that much load, as well as techniques for example, for performance testing, you often do a lot of black box monitoring. The techniques that you use to check if today in SIL on a mobile connection, your web store is reachable is very similar in terms of technologies and capabilities and as technical requirements. So the technology is there to use, to throw a lot of requests security to your software. So it’s fundamentally the performance tools are versatile like that.
Yeah, I totally agree with that. I have always taken the stance that it’s definitely better when you have both of them because observability is great. But then in pre-production environments, it can be a little tricky ‘cause you don’t have real traffic and that’s where performance testing can come in handy and performance testing is great, but you don’t always want to run it in production. And also you, it’s not always a good idea to be reliant on the results of the performance testing tool. You know, sometimes it makes a lot more sense to be already using the same observability stack that you use in production. And I also want to call out that performance testing is often thought of as load testing, but that’s really just one kind of performance that is being measured there. And so what about a related topic to observability as monitoring? I know Michele, you have a lot to say on that topic.
Oh my, can I rant?
Yes. Go ahead. I’m familiar with your blog posts.
Ooh influence. I have a –
I do research, you know.
I thought you came across it because, you know, super rabid spike. No, the, so I have been around the observability industry for awhile and I have a certain annoyance towards what the industry has done by rebranding perfectly good monitoring and application performance tools in terms of the hot new term up observability that you need to sell to sort of the software. The way I see it’s a much more formal, shall we say way. Observability is a property that you want your systems to have, monitoring is the way that we have to make most systems observable. However, something that I hope we will see in our lifetime, our software systems and such a level of quality and standardization in our profession of creating software systems. So that monitoring and observability becomes build team in the softwares we create. If you think about it all the for example, automatic distributed tracing and all the amazing innovations that have occurred over the past 15 years are effectively reckoning the lack of observability as a functional requirement of a piece of software. Monitoring is the art to collecting data.
Yeah, and I think that there’s also a connotation of monitoring being a bit more passive. Would you agree with that?
Not necessarily, no. At the end of it so there is, I really take the view that monitoring is a means to the end of observability. It is true that in the past few years, with new generations of tools, we have made leaps and bounds in both the capability of collecting useful telemetry with very little work in terms of toil to roll it out, without the proper tools are there and I used to work for one where you deploy an agent and it automatically discovers all around it. There is a lot of dark magic going on, but it works. And then it becomes very cheap to actually get to the holy grail of monitoring that is to have as little silos as possible. Telemetry wants to have friends. You want to see end to end systems how they relate to one another. Now, the moment that something that has changed recently in the industry is we have gotten better, not only at collecting telemetry, but analyzing it. When you compare monitoring tools that we started with right there the Nagios of the world. You’ll need it to know in advance what you were looking for. Now with broader collection of telemetry, for example, distributed tracing, a very powerful type of telemetry, very little underutilized in terms of the insights can provide you, you end up having a lot of data that is likely going to be useful, and you have much more freedom in how you analyze it, both in terms of discovering issues as they occur, as well as afterwards in terms of post-mortem.
I do have, however one, I agree with you completely Michele. I do however, think Nicole has a point in that if you look at the traditional monitoring tools that people used to use, like Nagios for instance, it was always this very weird situation where you put up SNMP traps or collected, you know, logs that were thrown at you in a very passive manner. But I would say that it turned monitoring itself has come to evolve into a state where it very much is an active practice in the same way as people would describe observability.
Oh. So you are making a very interesting distinction of rather than receiving as a data sync telemetry that was emitted, but something else that you go and get it through RTI, things like deploying an agent that does stuff. That’s pretty interesting.
Yeah. Or for instance, actually triggering a synthetic test that just prompts a common user flow and make sure that that still works. I mean, as compared to just admitting this passive telemetry data that just keeps rolling on, but maybe lacks the context as you put it on more advanced workflows where you’d correlate multiple sources of data or the multiple actions to derive your data that you want to.
It is true that in the past 15, 20 years, there has been an increase of active monitoring. I remember doing it back in the day with smoke tests that were of a crunch up every 15 minutes. I was so young. And now it’s that it’s kind of a baseline of observability. I mean, you need to want to catch everything, but if there is catastrophic failure, you will know hopefully before someone starts opening tickets or you’re upping your drops. Yeah.
Yeah. I think Simme it was right when he was saying, what I was talking about was like the more traditional approach that it’s just something that’s there in the background whereas these days, the way that it should be done is a lot more proactive.
And also they focus on actually gathering telemetry to be able to detect failure modes rather than probing for a certain thing. It used to be much more focused on a single point that you would probe for data. And–
It used to have simpler software too.
We sure did.
Yeah, that’s for sure. But I also wanted to say that there’s also been, there’s still some push based monitoring, but we’re also now seeing more poll based monitoring systems, more event driven ones. And I think this is all the evolution of monitoring and observability, right? Because these things did not exist not too long ago.
To be fair, the one should, it is pretty true. We have much more pool and pool length. This is something that a permit use has pretty much popularized with the concept of scraping, but there are entire domains where we deploy a software that is still very much push based. For example, IOT, for very basic reasons of network visibility, it’s very hard to go and reach out into someone else’s rooftop and go get the data. So there is still the mechanics of push buffer and pulled it for example, gave us in the primitive ecosystem, push gateway mechanism.
Grafana agent that actually sits in between you and a cloud service or something else, and allows you to not punch holes in your firewall, but rather reach out and deliver data to something.
Yeah, it’s kind of like you mentioned that our applications in general are just getting more and more complex, it reminds me the push the trend from monolith to microservices and then now some people are also going back. It’s never really gone, right. And it’s really difficult to say, no, this is the new way, nobody else is doing or should do the other way. It’s not really like that. A lot of things in tech are cyclical.
When I actually spoke about our systems are getting more complex. I was thinking exactly of micro services because we made a trade as an industry for very, very simple, very small components. And then pushing all the complexity that you cannot make disappear under the carpet of service meshes, many containers, things that need to be versioned independently remain compatible, which is actually one of the main drivers why in the observability industry, we moved from the very traditional, I don’t like the word pillars, but the very traditional telemetry types of logs and metrics with the occasional sprinkling of profiling to rather extensive use of distributed tracing because all of a sudden the problem was less often in the single simpler components and much more often in between.
But this is kind of a funny conversation to me ‘cause I don’t know if you remember, but 10 years ago we used to talk about microservices as well, but then we called it service-oriented architecture and it wasn’t the new cool thing. And then we decided that that was kind of cumbersome and powerful to work with. And we started to merge together our software, again, introduced monoliths.
But in a service record architecture, we tended to have a very expensive, very large piece of machinery called the enterprise service bus that actually was providing to some extent that kind of visibility that today we need to have through tracing.
And let’s also not discount the effect of the proliferation of cloud providers, because when you have your own data center and your audiences, maybe your internal company or something, maybe it is a good thing to go from one huge complex monolith to three simpler services. But when it’s not three, it’s like hundreds, maybe then you start needing like another layer to orchestrate and manage and all of the traffic between each of those components. It’s definitely not that clear cut to say which one is better.
I definitely think you’re bang on in that case, Nicole, ‘cause we’ve moved from maybe having a couple of software devs for a company that did some kind of HTML that website with some access database backend in the worst cases, right? We now have these huge teams of hundreds of developers that are all working on a compost system together. And having that as a monolith would in practice require us to either do these huge orchestrations of release trains that no one is able to do in a proficient fashion. Or we would have to have downtime all the time ‘cause we would need to redeploy everything every time we made a change. So in that sense, it really makes sense to split it up as we grow our teams and we grow our development efforts as well.
I have a rather, so coming from a very enterprise background I have a rather newest feel to be on the subject. And my take is that usually technological problems are not technological problems, there are people problems. One of the main motivators that was towards going to the extreme distribution of microservices. ‘Cause I tried to fight back against the Conway’s law. The fact that roughly the architecture of your system mirrored the architecture of your organization. So we said, look, we have all these teams, these pizza teams, it’s very hard to make them cooperate to create a well-behaved monolith. So let’s give each of them their independent infrastructure and talk over standardize protocols like most of the HTTP and some sort of RPC built on top of it. And that would provide for them a way to be productive independent of each other.
Exactly.
As an industry, we dramatically underestimated the cost and complexity that comes from the need of coordination and compatibility and understanding among these teams, because now you no longer have one code base that you can run tests against. You have to have multiple ones that may interact surprisingly at Atlanta. These for example, has led to introduction of contract based testing, where teams contribute to each other interface tests. And there’s also the saying, which is something that is a harsh reality that every observable behavior of an application with a large enough user base, will become a dependency for somebody somewhere.
I think this is the whole premise of Conway’s law, right? The idea that our architect, the software architecture or the infrastructure that we put into place in a lot of ways starts to evolve depending on how our team is structured. And I think it’s both a technological and a human problem as well. When you talk about coordination between teams or between services, it’s really the same thing.
Yeah. We have made actually strides in how easy it is to make software inter operate with each other. If you go back, we started, we were talking about SOA. SOA came with a curse called SOAP. It was incredibly complicated, mindbogglingly hard to make run at scale. Today, we have a bunch of, so we are using HTP for other things. Sometimes I would think too many, we have good stuff like GRPC, like arrow, a lot of different protocols that can be very efficiently used to either make a synchronous message passing happen or synchronous invocations. We have a lot of tools that in the past year and half, I think we’ve got the big two trigger happy in deploying a lot of new tools in production. And I’m saying this with a background of someone that fundamentally had to support every single piece of technology that made input or output, for example, on Java. It would surprise you how many different HTTP clients for Java are I think in production, I lost count. There were 50, it’s insane. And I’m, for example, you raised your mark right with a very, very beautiful LED and Bluetooth. And it’s super cool. I feel–
I was trying to share something that Simme was sharing there, but I can’t actually see it for some reason. Oh, there it is.
It’s okay.
What’s this Simme?
I love to say, I like to bring out this picture whenever we talk about why observability is needed. ‘Cause while it’s a lot larger than most of us operate with, and it’s still very striking that they think this is supposed to be Netflix’s is a microservice architecture at some point, whether it is or not, I don’t really know, but it’s supposed to be and trying to–
Can I be trashy? Can I be able to trashy?
Thank you.
Architecture has an element of deliberation that they do not believe there’s an actual out of the system has, this is an emergent system is a lot of architectures that then meshed into something.
Definitely.
It’s remarkable that it works so well.
Yeah. And one of the cool things there, I think is the fact that without observability where this big ball of yarn, it would be completely unfeasible to reason about anything this system is doing at scale. You wouldn’t be able to reason about the particular part, maybe that you are working on one of these small dots that we can’t even see because of all the lines. But to reason about what it does is as a whole would be completely unreasonable. But with tools like, for instance, distributed tracing, we’re suddenly able to follow end to end an action of a user or an agent within this system. That was all I wanted to think of that.
I love distributed systems, distributed tracing in distributed systems, because it actually gives you the means of answering the most important question or you’re faced with an issue, who is affected and how badly? When you go on an enterprise setting and our people are often databases service, your application completely blowing up, could be a drop in the bucket of their system. They will not probably have the central entry you have to have a very different take on priorities unless you have agreed upon a slice on the silos and the failure that you are fundamentally up streaming through your users may not be their concern, but the moment you have the shooting tracing, that goes from the deepest database to the device in the face of your user, yet you have a very clear perception of how bad is something that is the one most important piece of information that you need in front of your face when the pager goes off again at 3:00 AM, you need to decide whether it’s time to go on to Batman, or if it can wait until the third coffee tomorrow morning.
I think that’s the real success of the “Squid Game” right? Because that image that you had up Simme, and just imagine that times 114 million users that are potentially using many of those systems, it’s amazing that it works at all at that scale, especially.
Something that always gives me those thoughtful moments of, oh my God, what are we doing as an industry? Is that a potentially simple issues have disproportionate impact on popular services to the life of many, many people. It goes from lack of entertainment. You’re there you want to see the latest, “Witcher” episode, cool stuff. You’re disappointed if the thing is on top, multiply that by a medium people that cannot get there, there are fix, Hey, you have a significant amount of life around that has gotten worse, but it goes also to something that is maybe much more limited, but much more tragic. For example, when I was working at SAP, one of our customers was, I cannot say the country, but it was the first aid response service of one of the countries. We were scared plus that something so important that either the platform, because downtime would have consistent consequences to real people, something that we have seen recently with the move to the cloud and the disproportionate role that for example, U.S system one has on everybody’s life today. We have seen how the risk is getting more and more concentrated into fewer and fewer, very large, very complex, very reliable, but critical pieces of infrastructure, in ways that I do not believe as a society we have had yet. And that is where it really humbles me as an engineer to think, well, this has to work, right?
Yeah. We went through that in, I think in the last k6 Office Hours, I had somebody from oil funk, who is an emergency services provider in Austria, and it was cool, but also a little scary to find out that they are using k6 for, in part to also handle the traffic from the 112 emergency hotline. So it’s cool to see something that you’re working on, be used in such a critical industry, right. But at the same time, it’s also a little scary because it could actually be lives that are lost if you know, we don’t do our jobs well enough, it’s a little humbling to think of that.
I guess the closest resemblance we’ve had to these critical infrastructures that everyone relies on would it be like the transatlantic Telegraph cable? I mean, that has like in, I think it’s currently 19 cables or something like that, that make up the transatlantic telephone cable. But I really any catastrophic failure there would really isolate us in terms of cross continental communication.
But that is less likely to result in loss of life. Then, for example, your neighbor want service, not being able to follow you.
For sure.
I’m sorry for having completely spoiled the mood. I think that as an industry, we need to go in deserve the role of engineers. If we built bridges the way we push software to production, in some cases, I don’t think anybody would cross a river without crossing themselves first.
What’s interesting is in that spaghetti soup of microservices that you showed Simme, now we’re seeing that our observability stack is part of those. They are also contributing back to the system that we have to monitor. I don’t know if either of you saw the post-mortem by roadblocks.
It was classic. I had ops to them. It’s a, we have all been there.
Yes yes. Do you have to talk about it Michele?
I do, but I don’t think I would be able to be nearly as good as a Simme for this.
I just came to be honest. So I don’t have any detail commentary on that either.
It was the classic fallacy of oh jolly, oh golly. The same infrastructure that is powering my observability system was probably the system that went down.
All right. All right. Yeah, that really is something that you’ll see a lot. I mean, for instance, say that you have a deployment of the classic Grafana suite somewhere, and you have that running in the same data cluster maybe, or at least the same cloud provider as you have your critical workloads and to make it even worse, you could go down to having it in the same geographic region, or even on the same cluster nodes or the same hardware that you’re using for the rest of your critical infrastructure. When something goes south there, in terms of your critical infrastructure, it is likely that that was all will also, as I think, where to kick would roadblock, spill over to your monitoring system. And if that happens, how are you gonna be able to tell that you’ve been hit?
This is the classic blast radius question. Everything has failure modes, actually, several of them, and the more abstractions we deploy, the more failure modes we introduce, for example, Simme and I are building an observability system based on Kubernetes natives. Our guidance is if you deployed, deployed to a different cluster on different hardware, because you should never put the monitoring system on the same infrastructure, close to the infrastructure stuff we monitor.
It’s not only that. We also usually recommend that if you deploy our monitoring system, deploy another monitoring system that makes sure that your monitoring system is up and set them up in a, what did they call it? That dead mans trigger?
The dead man switch is the idea that, for example, do you know in the movies when the hero goes in front of him and says, if you kill me, you will be exposed in two days time. That’s the classic deadness which is something that happens unless someone periodically goes there and tells him not to. In this case, in the case of monitoring is unless your system, your monitoring system is telling you that it’s fine, assume tragedy. It never goes to the level of a specific use cases, but first of all, you need, let me take a step back. The best monitoring system is the one that tells you whether it’s something to look at, right? You actually do not want to have dashboards, however, colorful and beautiful in front of your face all the time, unless it is for Ferguson purposes, you actually want to be able to focus on something else and jump into the action when stuff is hitting the fan. In order to do that, you must have the guarantee that is the easy model of notification. You will receive it because silence, you want to know that it means that it’s good. You don’t want to have to doubt if there is something going on that my sister should tell me but doesn’t, which is why you set up fundamentally a health check thing that says I’m thinking here and all it’s fine. That is the diagnose switch.
I found it so interesting to read through the post-mortem and props to them too, by the way, for releasing something so detailed, they were not afraid to, you know, say what they’d done wrong and what they were doing to address it. I thought it was really well-written, but I want to read one part of it, because I thought it was so good. They said, “There was a circular dependency between our telemetry systems and console, which meant that when console was unhealthy, we lacked the telemetry data that would have made it easier for us to figure out what was wrong. We’ve removed the circular independency, our telemetry systems no longer depend on the systems they’re configured to monitor.” I thought that was a great takeaway from that whole incident. And because they’ve shared it in the post-mortem, they’re not the only ones who can learn from it.
Very generous of them to expose themselves that way and actually left the rest of the industry learn from their mistakes. And so hard work as you said, Michele, our love to the team at the roadblocks who probably need needed a bit of time out after that.
But there is also another component of a human component here. They did the right thing. They reestablished trust. They are telling us, we learned from our mistakes so that you can as well. In the scope of monitoring systems, trust is the ultimate currency because a monitoring system is literally a measurement system for the health of other systems. If you cannot trust your measurement system, what is the point of it?
And you usually only get one go with that as well. If you burn that trust, it’s gonna be very, very hard to build up again.
So this is something that touches me very personal because I used to work on a solution that was exquisitely powerful, but therefore also very complicated and complex. And the moment you have automation, someone somewhere sometimes is going to be broken by it. And reestablishing that trust was never easy and sometimes impossible depending on the scale of failure that you’re well-intended means of automatically monitoring something I should have brought to the customer. Something that we seldom say as an industry, when we try to sell a product to somebody else, is that everything has a cost. If you want to take away complexity from you to roll out very simple systems and set up that thing, reports stuff over there, and you want it to automatically, there’s a higher risk that the automation will cause an issue. Of course, that in many cases is a justified risk. You can roll out the monitoring gradually and the easier it is to roll out a monitoring system at scale, the more you reach the holy grail of monitoring, that is the network effect. The moment that you can start observing how issues in systems propagate, correlate, or are not correlated there is something very important.
So let’s talk a bit more about the characteristics of a good observability stack. We’ve already mentioned a few we’ve mentioned that, it should be independent as much as possible from the systems that they’re trying to monitor. We’ve also said that maybe we should also take into account the human side of things rather than just the technological. Maybe you should tailor the complexity of your observability stack to the size and experience and structure of your teams. What are some other things that you both could name in your ideal observability stack?
One thing that I really like to highlight here is the ease of use or the economics of the system that you are using. I used to do a analogy to how EpiPens work for people with allergies. If you’re eating a peanut and you’re allergic to peanuts, you don’t want to have to assemble the cure for that, right. Or something that alleviates or takes you out of immediate danger. You need it to be simple, just as take it out, push it in your leg, and it’s done. You don’t have to think about it. The same goes for monitoring or observability systems as well I would say. If you have to, when you’re awakened by a notification in the middle of the night, and maybe you’re even hung over or worse, and you are supposed to fix something that is critical, and you have to start thinking about what to do, then your observability system has in my eyes, at least failed you, you need to be able to get a notification that says this is wrong, here’s the data that proves that, click here to get a closer look and just be presented with all the information you need to be able to make a decent decision on how to proceed.
One thing recently on that note that I’ve come across is how potentially easy it is for a new Blake me to set up a whole observability stack or at least monitoring, but he could definitely expand upon it is EBPF, because we’ve been talking about distribution of systems and also of observability related services, but there’s also a trend of going lower and lower level, right? And EBPF monitoring is monitoring at the kernel level, recently with some people from the k6 team, Artem Charlie, Kevin, Danielle, Lopez, shout out to those guys. We did this hackathon project actually for EBPF monitoring. And the whole reason that I got into it was because the promise was you install this binary, you drop it into your cluster on your local machine. And then it just kind of exposes magically all of these things that I normally would have had to set up like node exporters for, you know, it was we’re living in a golden age of monitoring and observability I think. The things that are possible, like I can’t even keep up anymore, be talking about ergonomics. I mean, what is more ergonomic than not having to do much of anything.
You are talking about the day one, the fact that it is so easy to set up apart from monitoring system is exactly what you need to be able to monitor at scale. It removes the thought. EBPF is a fantastic technology. We are fundamentally given a virtual machine to run inside the kernel to execute logic, that allows us to do in a centralized place. Things that in the past were almost impossible. For example, the first thing I ever did with a team of vinyl in Stanhope with EBPF was actually to detect how a process that, that was impossible in user land. But with EBPF, it was actually pretty simple. We literally spent more time making sure that the techniques would work across all the operating systems that we need to support and to figure out how to find out that. EBPF the, I see two big potentials for that, for things that are not possible already today with sufficient tips. The first one is to give us a distributed tracing tool that largely independent from their own time. Now there are technology limitations there. It’s hard to find out how to trace context. I’m sure that someone somewhere out there is figuring it out as we speak, probably, that we reduce to some extent the toil that is necessary to support the federal fire hose of new technologies, libraries, and stuff like that is put in production every day. But the second one that excites me even more because the shoot the tracing is largely already possible is actually to have cheap enough in terms of overhead and consistent enough across run times production profiling. People look at distributed tracing as a way of figuring out what is slow. And it works remarkably well as long as the bottleneck is in between processes, the moment you want to drill down, why this requests into the insight, this process was taking too long. You see what you’re tracing is not so suited anymore. You end up looking at the leaf instead of the forest, because there is so many more requests going on through the system, that unless you have a lot of spans that are very, very detailed and fundamentally to trace threat level profiling, you are not going to which costs a huge amount of overhead. You are not going to have the information needed to find out, okay, good stock in the Dow library again. However, the moment that you have things, basically you have to find out where the CPU, memory or location, memory usage and wait time hotspots are in your application. You actually have the first places to work at to improve the overall system. It’s not gonna tell you why that request was going badly, but the moment you saw the hotspot, you are virtually guaranteed that many requests are going to get better. And then you can measure them again in terms of input output, observability tracing. Now people out there are going to say, Michele, what the hell are we talking about? Today we have a lot of amazing technologies to the production profile and they are true because for example, they are right because for example, if you look at the Java virtual machine with, for example, the Java flight recorder, you have excellent facilities with super low overhead to figure out what is your Java or cuddling or cradle program doing true. But they’re very specific to the particular run time and to a particular programming language. And when you start comparing what wait time, for example, looks like in Golang as respected over to machine with code routines, instead of, I dunno, loom threads, project loom, and all the rest of the stuff, you end up actually having telemetry that seems to say the same things, but in reality has a different semantic. And that is a challenge in our systems that have more and more grown polyglot. When we had the, our happy little monolith fault, it was one language. Now this year the systems are implementation of many different languages that behaves differently, and yet they are operated in some cases by roughly the same people, for example, necessarily teams, right? Having fluency in terms of performance across several languages is a significant challenge. Now, if we actually get through EBPF production profiling, for example, the folks behind parka and also the folks we have big pixie, there’s an amazing job there. If we manage to have something that will give us close enough, precise enough production, profiling information across whatever we may wanna throw into a container that is going to be a generational change in terms of making complex systems run better at scale.
That’s something that we’re interested in at k6 as well. I mean, I’m a performance Sesser so the first thing I think of as the network side of it, there are a lot of network metrics that could be gathered at that level. And the reality now is that without EBPF, you’re in this position where each new service has needs some other exporter that you need to install. And like there’s so many plugins to plugins and many different combinations of things, when really they might all be using HTTP. Like, why don’t we have something that just detects, maybe not, it won’t be everything right. It’s never EBPF has never going to completely be able to, or, well, I shouldn’t say never because you know, that could change very quickly, but I don’t see it as being the end all be all solution. But if it can provide some of the more common metrics, that’s already a step in the right direction. Talk about usability and ergonomics right?
And also provide a safe assessable baseline across a lot of different technologies working together.
Yeah. For sure. What other things do you think would you have in your ideal observability stack?
You asked me?
Either of you?
Go ahead Michele, I don’t have a lot of ideas on that.
Well, actually, if you go on the, okay, can I make us most shield for a link? If you go on google, WWW.ubuntu.com/whatisobservability/whatisobservability. There you see a kind of overview of the things that I find most meaningful, monitoring different types of systems. Second, I’m gonna give here that your own.
They can share my screen record.
I assume you want this one?
Awesome.
Yes. So this page is fundamentally a work of love of Simme and I, about the way we see observe a PVT. And the diagram that you see on screen is a, of course, simplified and very generic can your manage me if I lie, but it is a thirst overview of the kind of telemetry that you use at different levels in your infrastructure in order to monitor things. This is important for filamentary two reasons. One, is to provide guidance for people that may be new to observability in terms of which things to focus first, when you look at the telemetry types that are used on the right hand side, you will see that metrics and logs are of course universal. That’s what we started with. They are very good. Every software that deserves to run somewhere is capable of emitting clogs. Most of them can emit metrics and more and more, thanks to primitives that is really proud. There are the metrics sometimes serious thing to the forefront of enterprise systems. It didn’t used to be that. Then you start seeing that profiling gets, for example, the moment you move above the virtualization hardware, and you start talking about processes and applications, profiling them makes perfect sense.
Sorry Michele for cutting you off. But I think you might need to need to explain the icons.
That is correct.
That’s really. We don’t have that context.
Ah, they don’t explain themselves. Maybe then you use amino the icons.
Yes.
So the chart that you see on the left hand side is metrics, mentors, time series fundamental. That’s what we use to monitor things. There’s a lot. Then the table with the reds and greens and yellows followed by text that locks, the old, that bar chart is actually supposed to be flamed charts and that’s profiling showing hotspots. The strange thing with all those horizontal bars that is distributed tracing, that is the best way I can come up with to visualize spans. Spans is the basic data structure in distributed tracing. It describes an action taken by system to process a request. For example, you would have a span for the receiver of an HTTP request. The span is opened is created when your HTTP server is receiving the request, it’s going to be closed, but it simply is doing amazing things to get to the image in a screen–
Most of this you’ve probably seen in your browser, was what I was about to say, like something similar when you browse on webpage, you’d get this span alike view of how the different correlate.
It’s similar, but there is a key difference. One of the key concepts of distributed tracing as coming from dapper, which is fundamentally the progenitor of all modern distributed systems is the parent-child relation between spans, which is the closest you can get to cow’s reality. We are, for example, when your application is receiving an HTTP request and is issuing your query to Mongo to retrieve some data, you will have a parent’s span for the receiver of the HP request. And the child’s span that says, I spoke to Mongo, took me this longer. And this query, I got so many data back that is the parent-child relation. And it fundamentally, you would not have spoken to Mongo if not for the incoming request. Now things get Merck here when we talk about a synchronous messaging, because the beauty and simplicity of the trees of spans that you have in synchronous systems, kind of get lost with a messaging the moment that you have batch consumers. You’ll have many requests that result in a whole bunch of messaging being queued up in your messaging queue. And one application consuming messages originated in different traces all at once, fundamentally creating a merge, but I digress. I’m sorry. So we were talking about… I just get so excited about this future tree.
No, it’s great. I know that you said that you have a hard stop and I’d like to get you off a little bit early for your meeting Michele, but why don’t you talk about what Canonical has to do with observability at all?
Well, can I call does a lot of things, including a very beloved operating system together with an amazing community of volunteers, Ubuntu, we are responsible for a huge amount of software. We have offerings for Kubernetes, for OpenStack, for SAF, for a lot of systems that people use to run their applications and data centers. And it is the mission of Simme and I to make sure that those systems come with excellent monitoring capabilities, that you can monitor them with open source software, whether or not you want to use the monitoring system that Canonical provides you. At the end of the day, what we’re doing is in variable source fashion. We want to make sure that all the appliances that you can get from Canonical, the SAF, the OpenStack, the Kubernetes is, there are the cube flows and all the machine learning. They are, you can monitor them with best-in-class open source monitoring tools like videos, like getting logs out of SIS log, the security traces with open telemetry, where it makes sense. And that is the part of the work of Simme and I, that is about stepping up the game in terms of making software provided by Canonical observable in a way that you can use, for example, with your existing monitoring system, we really believe every single word we say, when we say the metric, once France, it’s all about having radio systems be monitored together. Now, if you do not happen to have a radio monitoring system, wherever monitoring the rest, we can give you one, but if you have a monitoring system and you’re comfortable with it, you by all rights should be able to monitor whatever piece of software you get from Canonical with it. And this is what to do.
That’s awesome. I wish we had more time to talk about this, but maybe we can have you both in some future episode, I feel like there’s so many things that I wanted to dig deeper into, but thank you both for coming.
Thank you so much.
I am so happy. Simme I’m so happy to have you back on again. I’ve always enjoyed doing these with you, so it’s a little bit weird that you’re now on as a guest, but I’m happy that you’re also bringing things that you’ve learned at Canonical back to us. And Michele welcome. I’d love to have you–
Thank you.
Back again for sure with or without Simme.
Anytime. But Simme is finer yeah.
All right. Thank you.
Sure. Sure. Thank you everybody for watching and I guess we’ll see you next week. Bye everyone.
Bye, have a wonderful day.

An Overview of Software Testing

Wed, 19 Jan 2022 17:10:11 +0100

Transcript

NICOLE: What is the difference between good software and bad software? Is it features or user interface or marketing? In my totally biased opinion, what actually makes a difference is how well the software was tested. But software testing encompasses a lot of different areas and it doesn’t even always have to be someone with a word tester in their job title that does the testing. So whether you’re looking to get started in the industry, or just trying to cross skill, here’s an overview of the main types of software testing.

Is it just me, or does your desk get that clutter too after a week of work?

Anyway, software testing is the process of validating whether an application works as intended, but there’s a lot of vagueness in that statement. One of the broadest ways to categorize testing types though, is whether it’s functional or operational.

Functional testing is more about application features. Is there a form to capture user email addresses? Is there a button next to it that submits those addresses? Once the button is clicked, where do the user details go? Do they go to the database? Are they stored correctly? Those are the sort of ways in which functional testers think. Operational testing traditionally called non-functional testing, is a way of verifying software quality and aspects that extend beyond the features of the application.

Operational testing sometimes involves concerns that are more subjective or qualitative. So it can be a little bit more difficult to test, because those things only arise in certain conditions. I’m saying operational testing here, instead of non-functional testing, because I was talking to Paul Bruce who made an excellent video just on this topic. He talks about why the term nonfunctional requirement is really a bit of a misnomer.

PAUL: It relegates all these other things, security, performance, installation, deployment, accessibility. It relegates them to a nice to have, so it should be functional versus operational.

NICOLE: So I’m using his term operational testing instead.

Here are the different forms of operational or non-functional testing.

First there’s accessibility testing. It is involved with verifying that end users can consume or navigate content. Usually through assistive devices like screen readers. While most forms of testing begin with a statistically normal user in mind, accessibility testing seeks out outliers in terms of location, linguistic background and disability.

Usability testing is designing with user experience in mind, specifically concerning the placement of on-screen elements and the intuitiveness of the process flow of an application.

Localization testing is a way of making sure that applications are usable in the local context, in the sense of being colloquially appropriate for users in different countries and from different cultures.

Security testing verifies that a system has no flaws that could be exploited, to gain access to confidential information or features that would not otherwise be accessed by expected means.

So that was me talking from my home in the Netherlands, but this week I am in Portugal. The remote working life, am I right?

But anyway, performance testing is my favorite topic and my favorite type of testing because that’s the one that I’ve had the most experience with. Performance testing is more qualitative. So it’s more about how well the application works. And there are two main types of performance testing and that’s front-end performance, and back-end performance testing.

Front-end performance testing seeks to verify the user experience at the interface level. So that’s everything that the end user sees on their local machine.

There are different approaches to front-end performance testing, and the first of them is performance profiling. Performance profiling involves running your application through a series of automated checks. And there are a lot of sites for this, where you just put in your URL, if it’s a web app, and you get a score in the end. That tells you how performance your application is, the most popular example for this and also the easiest to get started with, is DevTools because that’s already in Chrome and Firefox based browsers. But you can also go to sites like Webpagetest, or GTMetrix, YSlow, which is a plugin, or you can use Google Lighthouse, which is now included with Chrome Dev Tools.

The next approach to front-end performance testing is called RUM or Real User Monitoring. So the difference is that with performance profiling, that’s often done manually, and as a one-off thing, like maybe at the beginning of a test or during, but Real User Monitoring is not usually done within testing environments. As the name suggests, these tools measure the experience of real users. So they have to be run in production. Examples of tools in this space are; New Relic’s RUM, Elastic has a RUM as well, there’s Catchpoint, SpeedCurve Uptrends and many more.

And the third approach to measuring front-end performance is by using some sort of browser-based automation tool. So typically this has been something like Selenium, but that’s falling out of favor in terms of more modern solutions like Playwright, Puppeteer, Cypress and so on.

So front-end performance is about everything that the user sees on their end, but back-end performance testing is about everything that they don’t see. It’s about the network. It’s about latency, the application servers, the way the infrastructure is set up and all of those things that are a little bit more difficult to look into, but can also have severe bottlenecks. Back-end performance testing seeks to verify aspects of an application such as scalability, elasticity, availability, reliability and responsiveness.

Scalability is the application’s ability to respond to changes in demand. This could mean scaling up, which means increasing the resources available to a server, but it could also mean scaling out, meaning increasing the number of servers that are available to handle the application load.

Elasticity is a related topic to scalability, except where scalability is all about scaling up or out. Elasticity also emphasize a scaling down or in, and that’s because having these services or servers available on the Cloud that are dynamically adjusting can also be quite expensive. So elasticity makes sure that the infrastructure that’s available or the resources available are always appropriate for the size of the demand. So when demand increases, elasticity also checks whether the application decreases it servers or resources in response to that.

Availability is how long the application stays up, despite changing circumstances in production. So this is typically measured by uptime. Availability can also be tested using disaster recovery procedures, and you’re not just testing the application itself, but also the processes around it. So are there enough people that are on call so that they would be able to come in and troubleshoot when there’s a production incident? All of those things are included in application availability.

Reliability checks to see how your application behaves in response to unexpected errors. Reliability basically takes it as a given that your application is going to fail somehow, sometime, maybe you don’t know exactly when that will be or how it will fail, but it’s going to happen. So a big part of reliability and site reliability engineering is trying to expose your application to those failures beforehand so that you can build confidence about how it will behave when those things actually happen in production.

And the last aspect to back-end performance testing is responsiveness. And this is usually equated with load testing. Load testing is more of a technique for testing though. And really load tests can be used to test not just responsiveness, but also reliability or availability or scalability, basically any of the “ilities” in this part of back-end performance testing.

Protocol-based load testing verifies the backend performance of an application, not by simulating real users accessing the application, but by simulating the underlying requests that go back and forth between the client or the user and the application servers. One of the benefits of protocol-based load testing is that it takes up fewer resources on the side of the load generator, and therefore is less expensive to run. Some popular protocol-based load testing tools are k6, JMeter, Gatling LoadRunner, NeoLoad, and Silk Performer.

Browser-based load testing verifies the application by simulating real users accessing it. So it also has the benefit of including these front-end performance metrics. Browser-based testing can also be called browser-driven or UI-based testing. Some popular tools for browser-based load testing are Playwright, Puppeteer, Flood Element and as of a few weeks ago, k6.

Then there’s hybrid load testing, which takes the best of protocol and browser-based load testing tools by combining them in one test. Now this can present some challenges, especially if you’re using different tools for the browser-based script, than you’re using for the protocol-based script. The ideal is to use your protocol based load testing tool, to simulate majority of the load to cut down on costs for the execution and scaling out of your load test. And then you run a handful of users using your browser based load testing tool. So you still get some metrics for front-end performance while that load is being applied by the protocol based tool.

Testing software can be really daunting because there are many aspects of an application that you can test. But that’s because there’s also many aspects of an application that can fail. And that’s why all roads lead to testing and why the success of an app hinges on how well it’s been tested.

If you’d like to zero in, on load testing in particular, check out this video that I made on what load testing is. Thanks for watching. And like they say, here in Portugal, Obrigada!

How to Use Executors in k6: Adapting Load Test Workload Models to Production

Tue, 18 Jan 2022 22:45:52 +0100

Executors in k6 are like traffic cops.

They define the shape and volume of traffic that your k6 load testing script follows during test execution. Executors can also drastically affect your results if you don’t get them right.

Here are all the executor options in k6 and when you should use each one:

k6 executors

shared-iterations: Fix the number of total iterations for the entire test. Use when you want to create precisely X rows of test data during a prep run.
per-vu-iterations: Fix the number of iterations each virtual user (VU) does. Use when you want each VU to use exactly X rows of test data.
constant-vus: Fix the number of VUs and duration, but allow the number of iterations to vary. Use for straightforward tests or shakeout tests.
ramping-vus: Set stages, each with a number of VUs and a duration, and let the number of iterations vary. Use for easiest comparison with other load testing tools and for custom load profiles.
constant-arrival-rate: Fix the number of requests per second (rps) and duration, and allow the number of VUs and iterations to vary. Use when you have a simple load profile with a request throughput requirement (rps).
ramping-arrival rate: Like constant-arrival-rate, but with stages. For each stage, fix the number of rps and duration. Use for spike tests, multiphase load profiles with distinct requirements, and Game Day simulations.
externally-controlled: Fix max VUs and time, and allow everything else to be changed at runtime. Use for disaster recovery, breakpoint, or stress tests where you want ultimate flexibility in adapting your test to real-time results.

Still unsure?

Use ramping-vus. It’s probably what you’re used to from other tools.

Ready to move up? Use ramping-arrival-rate. Rps is a better (but still not perfect) measure of throughput for comparison than just the number of VUs.

When in doubt, look at production.

Making sure your test looks and acts like real traffic yields the best results!

For more details, check out the k6 documentation

Testing tool cheat sheet for load testers who don't just care about numbers

Mon, 17 Jan 2022 16:30:07 +0100

It’s not enough to decide how much load you want to generate. You also have to know how you’re generating it.

Your approach affects what part of your application is tested, the results returned, and the budget you’ll need to run your test.

Here are three main categories of load testing tools:

Protocol-based load testing tools:

These tools simulate the underlying messages sent back and forth between an application and a client.

Choose a protocol-based tool if: you want to generate massive load, you want to load test before components are integrated, you want to test APIs, or you are budget-conscious.

Best in class tools: Gatling, JMeter, k6, Locust, Neoload

Browser-based load testing tools:

These tools automate real user interactions with an interface.

Choose a browser-based tool if: you want to measure end-to-end performance, you want to reuse your browser automation scripts, you want front-end performance metrics, or you have a higher budget.

Best in class tools: Cypress, Flood Element, k6, Playwright, Puppeteer

Hybrid load testing tools:

This approach is a mixture of the previous two. You can do this in a few ways:

Use a protocol tool to generate the majority of the load, and a browser tool for a handful of virtual users (best functionality, highest effort/cost)
Use a browser-based tool for all load (expensive, simplest)
Find a tool that can do both (ideal solution, few options)

Whatever you choose, compare like with like.

Happy testing!

Six Performance Testing Hats

Fri, 14 Jan 2022 16:30:34 +0100

I was reading Six Thinking Hats by Edward de Bono when I realized that the premise could be applied to performance testing:

Each hat represents a different way of thinking, which can be useful when working through issues. Sometimes, wearing only one hat can cause us to fixate on incorrect assumptions, and “putting on another hat” can help shake things up and get to actually fixing what went wrong.

What Tabletop Games Teach Us About Application Perfomance

Wed, 12 Jan 2022 16:30:29 +0100

Tabletop games and application performance have more in common than you might think.

See, it’s all about optimization. Both involve juggling several variables: some that you can control, and some that you can’t.

Here’s what I’ve learned about application performance from playing tabletop games.

#1: Get more workers

In worker placement games, you should always spend your first worker to train more workers.

Similarly, one of the most basic ways to increase concurrency is multithreading: using more than one thread at a time. (See: Golang) This applies to testing tools as much as to systems. The 1 Thread: 1 Virtual User paradigm is flawed.

#2: Spend time on getting your engine going

Get your big combos in place before you tap to attack.

Things like setting up automation frameworks or CI/CD pipelines DO set you back in the beginning, but they are worth their mana cost in the long run.

#3: Draft with others in mind

The trick to drafting is knowing what you’re leaving for others as much as what you’re taking for yourself.

The short game is getting your team to do the testing. The long game is getting everyone in the company involved in testing.

#4: Hope for a nat 20, plan for a nat 1

Sometimes proficiency, expertise, and a luckstone just aren’t enough.

Instead of wondering whether an app will fail, take for granted that it will, and start planning for it. Keep that Action Surge in your back pocket.

#5: Sometimes, you just have to trade for sheep

The 2:1 port is valuable, but not always the best option.

Learn what you can, but accept that you can’t learn it all. Reach out to others with compatible skillsets and ask the “stupid” questions that nobody else wants to ask.

Testing is a team sport, and everyone gets a chance to play.

Happy gaming!

5 Ways to Use Obsidian as a Software Engineer

Sun, 09 Jan 2022 19:20:49 +0100

Yesterday, I wrote about why I think taking notes is the surprising foundation of a tester’s career.

But it’s not just testers. Every software engineer is a knowledge worker, and that means we get paid for what we know. Using a Personal Knowledge Management (PKM) system such as Obsidian.md helps us keep up with the constant barrage of new ideas in tech.

Here are 5 things I use Obsidian for.

#1: My very own Stack Overflow

If you code, you’ve used Stack Overflow.

What if you had your own Stack Overflow, one that contained cheat sheets of every language you’ve ever tried to hack something up in, including only commands relevant to your job? This is exactly what I’ve done for Python recently, and it’s helping me solidify my knowledge.

#2: Tools database

There’s a tool for everything.

To keep them straight, I have a page for every load testing tool I try, with tutorials for how to get started, an assessment of its features, and observations I had while attempting to learn it. Here’s what I have for the load testing tool k6.

Bonus: I also use the Dataview plugin to query tools that fit my criteria.

#3: An Agile workspace

If Agile is good enough for my professional work, it’s good enough for my personal work.

I use the Kanban plugin to keep track of projects, create weekly and monthly review/retro templates, and work iteratively using the Incremental Writing plugin.

#4: A public learning log

Learning in public is the fastest way to improve.

When I’m trying to figure something out, I document my attempts. With Obsidian Publish, I can quickly push this log out to my domain and share them with colleagues to get a quick review so that they can easily follow my train of thought and tell me where I went wrong.

#5: A digital garden of software ideas

Build libraries, not feeds.

I use Obsidian to save things I read/watch/listen to but also to save space for ideas I’ve learned to germinate. I publish notes before they’re fully fleshed out, with the goal of growing everything over time.

Today, I have 4,567 notes, all about things that matter to me.

Every engineer should have a PKM system.

Take Notes, Not Tools

Sat, 08 Jan 2022 17:00:44 +0100

When I first got into testing, I thought that testing was about learning different tools and frameworks.

I learned how to use Neoload, then LoadRunner, then JMeter, and then SeleniumSilkPerformerVisualStudioGatlingPuppeteerPlaywrightk6 in short succession. I took courses on JavaScript, on C, on Java. I ticked off tools like a grocery list.

It didn’t take me long to realize that I still wasn’t a good tester.

Taking notes, not tools, is what makes a good tester.

When you think about it, a lot of things a tester does revolves around taking good notes.

Requirements are just notes on what the business wants to achieve. Test scenarios are notes on situations that the team wants to simulate. Test cases are notes on how a user would use an application. Defects are notes on issues.

The quality of a tester’s notes dictates the quality of their work.

Listen to people, not tool tutorials

Take notes on what users do. Talk to the business. Pore through Google Analytics logs. Piece together a story of the kind of people who use your application, and how they use it.

Take notes on problem areas. Talk to customer support about the most common questions they get– they know more about pain points than anyone else in the company. Talk to the ops engineers who were on call during the last production incident. These problems are what you’ll need to test later.

Take notes on what you’ve learned. If you don’t like learning, you’re in the wrong field. Document everything, even things you think you’ll never forget, because one day, you WILL forget.

Taking notes, not tools, is the foundation every testing career is built on.

How Eurofunk uses k6 for load testing emergency services (k6 Office Hours #37)

Fri, 17 Dec 2021 17:17:07 +0100

In this k6 Office Hours, k6 CEO Robin Gustafsson joined me to talk to Daniel Knittl-Frank, a software engineer at Eurofunk, about how k6 is used in the emergency services industry. Daniel has an interesting use case for k6: he uses it to load test calls in Austria received from the 112 emergency services hotline!

Load Tester Reacts to Pokemon Go Scales for 166m Users

Thu, 09 Dec 2021 16:13:59 +0100

Transcript

NICOLE: Last year, Pokemon GO, a mobile app developed by Niantic Labs, made $1.23 billion in revenue with 166 million players from all over the world. This level of success is unprecedented, especially for an app that was encouraging people to go outside at a time when many governments were encouraging them to stay inside. In tech, though, success comes at a cost. Here’s what Niantic has to do under the hood to keep this massively popular game running.

Hey everyone, I’m Nicole van der Hoeven. I’ve been helping companies through their load testing for over 10 years, and I like to talk about tech travel and taking notes. I’m also a Pokemon GO player. I’ve been playing it since the day it was released in July 2016.

In this video, I’m going to be reacting to a conversation between Priyanka Vergadia from Google and James Prompanya from Niantic Labs. Thank you to Joe Colantonio’s TestGuild News Show last month, for bringing it to my attention.

Let’s get started.

PRIYANKA: Yep. NICOLE: Okay. Everybody knows that you should be doing a curve ball to catch Pokemon the fastest way.

PRIYANKA: I am catching some Pokemons. Did you know that it was built on Google Cloud? Well, let’s talk–

NICOLE: GCP, okay.

PRIYANKA: At Niantic Labs to take a behind the scenes look at Pokemon Go’s architecture, and how it scales for large number of players. Hi James, welcome to the show.

JAMES: Pokemon Go is not your typical mobile game.

NICOLE: I love that he’s got Pokemon paraphernalia everywhere.

JAMES: Catching these little creatures around you called Pokemon that are appearing in a real world. Encourage you to go outside, explore and discover things using augmented reality. A big part of that, which wasn’t in–

NICOLE: See this is one of the things that make me really impressed with their implementation. They have majority of their users using mobile networks. That’s a really hard thing to account for, especially when you’re talking about a global audience that is playing in different parts of the world. So it’s really important that everything is front-loaded, and that they’re minimizing the amount of traffic, the amount of resources that each player has to download on their mobile network.

JAMES: Wasn’t included in the game when it first came out, was this big community aspect of it. We really liked this part of the game. So we made it a major part by hosting regular live events, such as Community Days, Raid Hours,Team GO Rocket Takeovers, all culminates in this big summer event that we call GO Fest.

PRIYANKA: So it’s–

NICOLE: So this is a really brave move as well because they have events that are timed. So they happen at the same time for people in that time zone. And earlier on in Pokemon Go’s history, I did experience some slowness and some congestion in the mobile networks. But I mean, that’s a really difficult thing for them to plan for. I’m interested to see how they solve that.

PRIYANKA: How does Pokemon Go back and actually scale to handle these peaks in traffic?

JAMES: So we use Google Cloud. (chuckles) We have lots of services that need scaling such as the Google Kubernetes Engine and Spanner. Our front-end service is hosted on GKE, and it’s pretty easy to scale the nodes there. Google Cloud provides us with all the tools we need to manage the Kubernetes cluster.

NICOLE: So it’s really cool to hear him talking about their stacks. So they’re using GCP, Google Cloud Platform. They’re on Kubernetes, so they’re using GKE, the Google Kubernetes Engine. And they use Google Cloud Spanner, which is a very highly available relational database.

JAMES: To give an idea of scale, we have about 5,000 spanner nodes handling the traffic and thousands of Kubernetes nodes running just for the Pokemon–

NICOLE: That is so interesting. 5,000 Spanner nodes. That’s just for the data. That makes sense because Niantic’s main reason for creating this game, or the way that they’re making money, I guess, is by getting the data from users about walking paths and …Basically, geo location or spatial data. So it’s super important for them to be able to store the data that they’re getting from their application usage.

JAMES: We also have the various microservices that host other parts of the–

NICOLE: So another thing that they’re doing to keep it scalable is by using microservices. See the traditional way is to use a monolith, so one large application with many sub-components. The problem with that though, is you can’t independently scale those components, you’d have to scale the entire application. And when that application is already really big and complex, that’s really inefficient because you’re scaling everything, even services that you may not need to scale. So it’s really smart that they’ve chosen for a microservices based architecture, so that each of those components can be scaled separately of each other.

JAMES: And if you compare ourselves to maybe World of Warcraft, you’ll see that other massively multiplayer online games, they split players into multiple realms, but for us, all of our players, they reside on a single realm. It was important for us that players can always interact with each other and share the same game, the experience, no matter where or when they’re playing.

PRIYANKA: Yeah–

NICOLE: Okay. So I happen to have also played WoW, World of Warcraft. My main ’toon was a disciplined priest. So when you sign up to WoW, you have to choose a realm, that’s basically a server or group of servers so that they don’t have to shoulder the entire load from all users. So that was WoW’s way to reduce the load. And he’s making the point that Pokemon Go can’t really do that to a certain extent. It’s not entirely correct to say that they don’t have that sort of splintering, because they do still splinter based on the time zone. You’re only seeing people in that location in your time zone.

Even events, they sometimes say 10:00 AM in your local time. So you’re not necessarily playing at the same time that people in Australia are playing. So there is some splintering, but it is less than what there would be for a game like WoW.

JAMES: When we’re looking at different providers, we saw that Spanner has consistent indexing that allows us to do more complex database schemas with these primary and secondary keys. Data Store was also non-relational with atomic and durable transactions. So we needed a relational database with full consistency, and made the choice to go to Spanner, which gave us these global ACID transactions.

NICOLE: So he’s talking about consistency, global consistency, being a big deal and a big factor for why they chose to go and use Google Cloud Spanner instead of Google Data Store. And the reason is that normally, when you have a bunch of different instances of database, it doesn’t matter so much that those requests are processed in exactly the order that they came in. Because usually those requests are being made by different people, and they’re trying to change different accounts. But with Pokemon Go, there’s the added complexity of wanting to keep the integrity of the chronological order. So if I go and attack a gym before you do, my, the process, the request that I created by attacking the gym, should be processed before yours is because it matters who gets to the gym first. So that’s an interesting twist.

PRIYANKA: Now let’s say I’m a player and I’m playing the game right now. I open my app to catch a Pokemon. What is happening behind the scenes? How does the request flow work?

JAMES: When the user catches a Pokemon, we receive that request through the Google Cloud Load Balancer. All of the static media is also stored on Google Cloud storage buckets. And it’s downloaded to the phone when you first start the app. We also have caching enabled at the load balancers. So it’s–

NICOLE: Already, they’ve mentioned they have a load balancer, so that that’s like a traffic controller that’s sitting above the servers and saying, “Okay, this request goes to this server.” And then maybe when that server is processing too many requests, that load balancer should be able to say, “All right, now let’s divert some of the traffic to another server that maybe isn’t as heavily utilized.”

He also mentioned that they employ caching. This is what I was saying earlier that because you want to minimize the amount of traffic and the bytes that are flowing on the network, you want to employ caching for maybe things that aren’t going to change so more static assets. Maybe you might want to change the Pokemon that’s in a gym, but the location of that gym is something that should and can be cached. He’s right now talking about caching on the server side, on the application side, and not on the mobile phone. But I wonder if they’re going to talk about that later as well.

JAMES: It’s all cached and served through Google Cloud CDN. So when the Poke–

NICOLE: So he also mentions using a CDN, so a Content Delivery Network, which is really good because that means that if I’m in the Netherlands and I make a request to their servers, my request isn’t then going to be sent to a server in the US, because it’s going to be sent to one of the servers in my country or nearby. That means that the latency between my request being sent, and that request getting to the server, and retrieving that resource and back, is going to be a lot lower. So it’s … This is a way to improve performance.

JAMES: The player catches the Pokemon, the GCLB sends the request to our GKE cluster. And we have a front-end service that sits behind Nginx reverse proxy. The request goes from the user’s phone through the reverse proxy, to one of these player, front-end services. We also have this thing called a Spatial Query Backend. It’s a cache by location.

So that is where we store the information that determines where a Pokemon is shown on the map, what gyms and Poke stops are around you, what time zone you’re in, or any other feature that is location-based.

PRIYANKA: Now, what happens when I hunt a Pokemon down and catch it?

JAMES: When you catch a Pokemon, it sends the request to the front end, to the Spanner database, where your player entity is stored and it store it there. For catching Pokemon, that actually doesn’t go to the Spatial Query Backend. If you were, instead, if you were battling in a gym or adding lures to the Poke stop, that information is stored on the Spatial Query Backend, and–

NICOLE: Okay. So it sounds like the Spatial Query Backend is only storing data about the world, the Pokemon Go world that everybody can see. So people don’t need to know whether another person has caught a particular Pokemon. They just need to know whether there’s a lure on a Poke stop, or whether the gym is Team Mystic and things like that that everybody else can see. So there’s this segmentation of some data that is specific to the user, and other data that all users need to see.

JAMES: We also use Pub/Sub to send the message to a Pub/Sub topic, which is then used for analytics pipeline.

NICOLE: He talks about a Pub/Sub model, which means that their architecture is event driven. So the traditional way that this is handled is by polling. So normally when an service is waiting for something to happen, it goes, and it sends a message and it checks. Basically like, “Hey, are we there yet? Are we there yet? Are we there yet?”

And then when it finally gets a response that “Yes, we are,” then it can do something with that data. Instead what they’re doing as a Pub/Sub. So instead it waits for events. There is no constant polling, again, reducing the amount of traffic that’s going back and forth. And it just specifies criteria for what it is actually waiting for. And when that happens, after the services that require that event are subscribed, then that event is published to all of them. And then they can then do whatever they want with that event.

PRIYANKA: So, how do you ensure that when two people who are in the same geographic regions, see the same Pokemon data, and keep that relatively in sync, especially during the events?

JAMES: So it’s actually pretty interesting. Everything in our servers are deterministic. So even if players are on different machines, they’re sensitive to different front-end services, if they’re in the same physical location, all the inputs would be the same and the same Pokemon would get returned to both users. There’s a lot of caching involved and as well as precise timing, especially when the settings are changed and they need to be in sync across all the servers.

So it’s very tricky, but it was very important for us that all the players feel like they’re part of this one shared world.

NICOLE: I’m not entirely sure what he means when he says that it’s deterministic.I’m going to guess, and I have no relationship with Google or with Niantic, but I’m thinking that what he means is that when you’re catching a Pokemon, there is no request that’s being sent to the servers to say, “Hey, did I catch this?” Instead, there’s probably some sort of algorithm. This Psyduck is caught 80% of the time and appears 2% of the time. So 2% of the Pokemons that you’re gonna see appear in this area, are going to be Psyducks. And that’s also reducing the load going back and forth, right? Because it’s just the equation or the algorithm that determines whether you catch a Pokemon or not.

JAMES: This game data is of interest to our data science team. They use this for marketing purposes, for verifications. For example, we might want to say that this player, or players caught x millions amount of Pokemon during our Saturday Community Day events. So, it’s important that we have that information in an easily accessible manner.

NICOLE: So there’s a lot of data here that they track. So I’ve got my distance walked, Pokemon caught, I’m at 18,878. Poke stops visited, total XP, start date. And then there are these medals as well. So, they look at things like, how far have you walked in total? So all of these things are data points that they store for every single user. No wonder they have such a focus on databases, right?

PRIYANKA: As the events grow, like you mentioned, in some of these events, the traffic grows to millions of users per second. You also said–

NICOLE: Per second!

PRIYANKA: …of data is generated. How does the system scale when it comes to data pipeline?

JAMES: Yeah. So when there’s increase of transactions, there is increase of load throughout the system with a lot of our data pipelines for things like BigQuery, one of that–

NICOLE: I’m not actually sure she was saying millions of users per second, but that’s a little bit ambiguous. Maybe she meant millions of requests per second. And the way that they’re trying to do that is they’re reducing the size of those requests. If she means there are millions of concurrent users at any given time, that doesn’t necessarily mean that those users are all doing something at the exact same time. So I would have wanted to see exactly how many requests, to hear how many requests per second they’re needing to deal with, or how many bytes per second are being sent over the wire. That would’ve been cool.

JAMES: …Just works. Google Cloud will handle the increase in traffic without any intervention.

PRIYANKA: With that much traffic, obviously health of the system is critical as well. How do you monitor the health of the system during these massive events?

JAMES: So we use multiple monitoring systems, including Prometheus, Grafana, and Google Cloud monitoring. On a personal level, I actually prefer it to Google Cloud monitoring because it’s very simple to use–

NICOLE: Whoo whoo! And they’re using Prometheus and Grafana … Oh, this is entirely unexpected, but I’m actually wearing a Grafana shirt. So it’s really cool that they’re using Prometheus and it really, it makes sense, because they’re on a Kubernetes cluster and Prometheus and Kubernetes just go really well together. By the way, I work for Grafana, for context.

PRIYANKA: What are some of the expansions and improvements you’re looking to do in the future?

JAMES: On the Google Cloud side of things, we are looking into things such as Agones and Google Game servers, and using that to see how we can make our game even better.

PRIYANKA: We wish you and the Niantic team best of luck for these enhancements. And this has obviously been super insightful for us, James. Thanks for joining me today and sharing your architecture with us.

JAMES: Yeah, thanks for having me. This was fun.

NICOLE: That was really good. So there were a few things that were mentioned there for how they’re scaling Pokemon Go, that I wrote down here.

One, they use an event driven architecture and also a microservices based architecture, both of which are way more performant than the traditional methods.And the microservices based architecture will also let them scale up independently as demand increases as well. And they’re also using Cloud Spanner instead of Google Data Store, prioritizing global consistency, as well as high availability.

They front-load a lot of the static assets that are used by the application. And they’re being downloaded when the app is initially downloaded or installed onto users’ mobile phones. That way they don’t have to keep serving the same data if it’s going to be used consistently. And when something changes, like a different lot of Pokemon, they release these patches that let you download a lot of those images so that when you catch those Pokemons, then you already have all that information on your phone. And on a related note, they’re employing a lot of caching. They talked about caching on the mobile app side, and on their server side.

Events are also staggered. So splintered realms sort of, because they’re happening on different time zones, depending on where you live. And not completely simultaneously, regardless of the time in the part of the world where you’re from.

They’re also using things like Prometheus, Grafana and Google Cloud Monitoring. They didn’t mention how they’re actually using multi-tenant versions of Prometheus like, maybe, I don’t know if they’re using Cortex or something like that, but that’s something that they should look into.

And they also mentioned potentially increasing the number of Google Game Servers.

I really appreciated that behind the scenes look. I wish more companies would do this, especially for apps that I personally love and use. One thing that I’m really curious about though, is the testing side of it. Because testing an application of this scale and complexity, is another thing entirely from observability. And I would have wanted to know how they’re doing their performance and load testing. And whether they’re doing things like chaos engineering maybe, or reliability testing to improve the availability and reliability of their application.

If you’re still watching this, then chances are high that you like Pokemon and load testing as much as I do, which is a bit of an odd combination. And in that case, check out this video, on how I run chaos experiments using k6, Prometheus and Grafana. Thank you for joining me as I watched this awesome video.

And as they say in the Netherlands, (Dutch) Until next time! Bye!

What's new with k6 Cloud, with front-end lead Simon Legander (k6 Office Hours #36)

Fri, 03 Dec 2021 17:09:45 +0100

In this week’s k6 Office Hours, I was joined by my colleagues Mark Meier and Simon Legander, front-end lead at k6, to talk about the new features that have come to k6 Cloud this year so far. Simon also gives us a sneak peek of all the cool features coming on the horizon…

Performance Objective: Intro to Load Testing With k6 and Grafana

Tue, 23 Nov 2021 17:05:06 +0100

I spoke at the Performance Objective meetup to talk about how to use k6 and Grafana for beginners. I always enjoy talking to like-minded engineers, and I think sharing ideas in a meetup context is probably the best way to do it.

Everyone Can Contribute: Load Performance Testing With k6

Tue, 23 Nov 2021 16:57:26 +0100

This is a recording of a live meetup I spoke at, the Everybody Can Contribute Café hosted by Michael Friedrich, Senior Developer Evangelist at GitLab. This was a more casual conversation and show-and-tell about k6’s capabilities, and I really enjoyed it!

Browser testing and API load testing in the same script with k6 (k6 Office Hours #35)

Fri, 12 Nov 2021 00:28:02 +0000

Earlier this week, at ObservabilityCON 2021, we announced something new: k6 now supports browser-based testing with xk6-browser. Here’s a few of us from the k6 team talking about the announcement!

Resources mentioned

k6 load testing on Kubernetes using the k6 Operator, with Hans Knecht (k6 Office Hours #34)

Fri, 05 Nov 2021 00:23:15 +0000

I was really excited to have Hans Knecht on this week’s Office Hours. Hans works at Mission Lane, and is one of the biggest contributors to a k6 extension called k6 operator. k6 Operator lets you create a k6 custom resource object in Kubernetes.

Resources mentioned

Work week in the life of a Developer Advocate

Wed, 03 Nov 2021 00:31:46 +0000

Before I became a Developer Advocate, I didn’t really know what developer advocates did - so here’s my attempt at explaining what it is I do for a living!

Weekly breakdown

Here’s a list of my activities during this week:

Friday
- Filmed new ending for video #1
- Figured out integration options
- Did k6 Office Hours
- Prepared for Clubhouse talk
- Participated in customer demo
Saturday
- Did post-production for k6 Office Hours
- Talked at a Clubhouse meetup
Sunday
- Wrote (team profile) blog post #1
- Wrote talking points for Racket
- Started writing blog post #2
Monday
- Edited video #1
- Created thumbnail, description
- Scheduled video #1
Tuesday
- Created slides for presentation
- Played with Prometheus and Grafana for webinar
- Had a virtual coffee with colleagues
- Finished writing blog post #2
Wednesday
- Went to four meetings
- Caught up on emails, Slack, and social media (last night)
- Wrote blog post #3
Thursday
- Scripted and filmed videos #2 and #3
- Started editing video #2
- Talked to a potential guest on livestream

Common performance testing problems and fears (k6 Office Hours #33)

Fri, 29 Oct 2021 00:20:27 +0000

For this week’s k6 Office Hours, we attempted to steer into Halloween by talking about some of the most common performance testing issues we’ve faced in our careers as performance testing consultants. For this one, I had Tom Miseur join me to commiserate and swap stories.

Site Reliability Engineering and performance testing with Stephen Townshend (k6 Office Hours #32)

Tue, 19 Oct 2021 00:09:50 +0000

In this week’s Office Hours, I wanted to talk about the intersection of Site Reliability Engineering and performance testing, so I decided to invite two people with different backgrounds from me to discuss it. Stephen Townshend is a performance engineer who recently changed jobs to become an SRE, and Daniel González Lopes is an SRE who had not had experience in performance testing before he joined k6. It turned out to be a great talk!

Resources mentioned

How to install and use the k6 Cloud Grafana data source plugin

Tue, 19 Oct 2021 00:02:05 +0000

In this video, I talk about the k6 Cloud Grafana data source plugin, which lets you use k6 Cloud as a data source for your Grafana visualizations.

Installing and using the k6 Cloud Grafana data source plugin

Installing the plugin for Grafana Cloud

Get your k6 Cloud API token on this page

Grafana Helm Charts repository

Load testing with k6 for beginners, with İnanç Gümüş and Viktor Mihajlović (k6 Office Hours #31)

Fri, 15 Oct 2021 01:52:48 +0200

Did you know that every new person to k6 does a Week of Testing? It’s a k6 tradition for new hires to spend their first week seeing firsthand what the experience of a new user of k6 is like. How quickly do they find the documentation? What issues do they get stuck on? Then, at the end of the week, the new employees do a presentation for the rest of the team to show their first impressions and findings.

In today’s Office Hours, İnanç Gümüş, one of the newest k6 team members, shares his presentation and his experience as a k6 beginner. Another colleague, Viktor Mihajlović, a Technical Customer Success Engineer, also shares some of the most commonly asked questions he’s heard on the job.

Conf42 podcast: in the k6 Universe

Thu, 14 Oct 2021 01:27:45 +0200

Miko Pawlikowski interviewed Simme Aronsson and I for the Conf42 podcast, and we talked about:

why I love to learn human languages, and what Esperanto sounds like
the Grafana acquisition of k6
what performance testing is, anyway
whether everyone needs to do performance testing, and my personal setup for this blog
chaos engineering and Schrödinger’s Pokémon
an overview of the Grafana Labs projects landscape
what k6 is, and how we would get started for free (k6 OSS)

How to make load testing scripts realistic

Wed, 13 Oct 2021 15:17:45 +0200

A good load testing script exercises the functions and components as specified in the requirements gathering phase while simulating real users as closely as possible. Here are a few ways to make scripts more realistic.

Timestamps so you can skip around!

0:00 Intro
1:22 Static resources
2:31 Cache and cookies
3:02 Dynamic think time
5:07 Concurrent requests
5:45 Load profile
6:19 Test data
7:01 Different user paths
7:55 Other forms of testing

Basics of load testing with k6 and Grafana in 20 minutes

Tue, 12 Oct 2021 01:04:02 +0200

This was a video I pre-recorded in preparation for the first Grafana EMEA virtual meetup.

In mid-June (just three months ago), k6 joined the Grafana Labs family. Many people didn’t quite understand what the connection was between k6 and Grafana, but I thought it made a lot of sense.

Google’s Site Reliability Engineering book talks about 4 golden signals that we should measure in a system: latency, traffic, errors, and saturation. Load testing provides 3 of those (latency, traffic, and errors), but it doesn’t show saturation, and it doesn’t give us visibility into the system.

Grafana brings observability. k6 brings data. They work really well together, which is why we’ve been using each other’s products for a few years now, before there was ever any talk of an acquisition.

In this video, I talk about why k6 and Grafana are better together and also demonstrate:

how to run a simple script in k6 OSS
how to make sense of the results of a load test
test execution options: running locally, sending results to k6 Cloud, and running a test from k6 Cloud– all triggered using the CLI
two different integration options to bring k6 and Grafana together: the k6 Cloud Grafana data source plugin and Prometheus Remote Write

TTRPG Obsidian Showcase

Sat, 09 Oct 2021 00:43:25 +0000

Here’s a presentation I participated in about a bunch of nerds who like both Obsidian and D&D. My part begins around 0:45:20, but I thought the other speakers’ parts were really fascinating as well!

SLOs in performance testing, with Leandro Melendez aka Señor Performo (k6 Office Hours #30)

Fri, 08 Oct 2021 22:05:22 +0200

In this Office Hours, I invited my colleague Tom Miseur and friend Leandro Melendez, aka Señor Performo, to talk all about different criteria in performance testing, including non-functional requirements, SLAs, SLOs, SLIs, thresholds, and checks. Criteria at different levels is how we determine whether testing, including performance testing, has been successful, so it’s important to understand what they are and incorporate them into testing scripts and processes.

Testing in Public: How to plan a load test (practical example) - Wyrmwood Gaming

Wed, 06 Oct 2021 11:21:41 +0200

When you’re learning a new skill, there’s only so much that studying can do. The problem is that there’s still a big gulf between studying something and doing it yourself. The solution is having a mentor to bridge the gap– someone who will let you shadow them while they’re executing tests.

I made this video thinking about the me from over a decade ago who would have wanted to get the chance to look over a more experienced tester’s shoulder. In this video, I talk about how considerations for planning load testing, using a hypothetical use case for a real company: Wyrmwood Gaming.

Testing With k6 + Grafana: Better Together (Grafana virtual meetup EMEA)

Tue, 05 Oct 2021 21:49:13 +0200

In this first EMEA session of the Grafana Labs virtual meetups, I talked about how to use k6 and Grafana together to get the full performance picture for a software application. Observability, using tools like Prometheus and Grafana, lets us see what’s going on inside the application, and load testing, using k6, helps us get those insights earlier on in software cycles, such as during pre-production.

How to use k6 Cloud - a k6 load testing tutorial with Bill Rainaud (k6 Office Hours #29)

Fri, 01 Oct 2021 13:58:03 +0200

Bill Rainaud is an inside account executive at k6, which means he talks to users and potential users of k6 every day. In this Office Hours, he explains how to get started with load testing using k6 Cloud, which is a SaaS platform for running and scaling up load tests on the cloud. We also answered a few frequently asked questions about k6’s capabilities and how to do correlation in k6.

How to load test a GraphQL API built with Hasura using k6 (k6 Office Hours #28)

Fri, 17 Sep 2021 13:58:42 +0200

My co-host Simme and I had invited Adron Hall from Hasura to come talk to us about how to load test a GraphQL API, but he was unable to come due to unforeseen circumstances. Luckily, Tom Miseur, who is part of the Professional Services branch of k6, was able to step in on very little notice!

Tom has done a load testing engagement using k6 that involved Hasura, so he was able to tell us what it is and how it can be tested with k6. It turned out really well, thankfully!

Intro to Load Testing With k6 and Grafana (webinar)

Thu, 16 Sep 2021 22:11:16 +0200

Tonight I did a webinar with Grafana Labs– the first k6-related webinar since the acquisition. I was more than a little nervous to give it, because I knew I was introducing k6 to a lot of new users, but also to Grafana for the first time. I think it went pretty well! I was still quite relieved when it was over, however. :)

In the webinar, I talk about:

why it’s still a good idea to do load testing in pre-production
what k6 is
why k6 and Grafana together make sense
and how to integrate k6 and Grafana

There’s quite a bit of demo in it, and I also had the k6 CEO, Robin Gustafsson, join me for Q&A as a nice treat.

Check out the full webinar here– it’s already up on the Grafana site. You can also go through my slides here, if you’d like.

Schrödinger's Pokémon: k6 and Grafana edition

Wed, 15 Sep 2021 18:22:31 +0200

Earlier this year, I gave a presentation at New Relic’s FutureStack conference about chaos engineering with k6 and New Relic. A few months ago, k6 got acquired by Grafana Labs, and so I wanted to redo it, but with the Grafana stack this time.

Here’s the result, and I think I like this iteration even better:

I presented this talk at TestCon Europe 2021 today, although the audio sync issues were not great, so I recommend you watch the video above instead.

Slides are here.

How Gitlab uses k6 for load testing, with Grant Young (k6 Office Hours #27)

Fri, 10 Sep 2021 18:13:26 +0200

Grant Young, Software Engineer in Test at GitLab, joined k6 OSS maintainer Mihail Stoykov and me to talk about how GitLab uses k6. Apparently, they began using k6 internally some time ago, and because they like to eat their own dogfood and use GitLab to test GitLab, they also incorporated k6 into GitLab so that their users could reap the benefits, too.

Here’s the GitLab documentation about how to do load testing in GitLab, and here’s a k6 blog post on how to set it up!

How to do continuous performance testing, with Lee Barnes (k6 Office Hours #26)

Fri, 27 Aug 2021 09:34:13 +0200

Lee Barnes and I got connected because we are both speaking at TestCon Europe 2021 this year, and I thought it would be nice to have a chat with him before the conference to get a sneak peek at what he’s presenting about: continuous performance testing. Lee is the CEO of Utopia Solutions, and he’s been in the testing industry for decades, so I really appreciated his insights on where the industry is going.

Catch Lee’s talk here and mine here!

How to debug k6 load testing scripts with Tom Miseur (k6 Office Hours #25)

Fri, 20 Aug 2021 14:38:11 +0200

Tom Miseur joined k6 at about the same time I did late last year, and we immediately hit it off because we both come from a performance testing consulting background AND we both had worked for different competing testing tool vendors before moving to k6. I was excited to have him on k6 Office Hours to talk about a topic we both had a lot to say about: how to debug load testing scripts, and specifically k6 scripts. We got a chance to do a bit of a show-and-tell of tools that we use that we use, and they are all independent of k6.

I also got to demo a cool HTML reporting dashboard that Ben Coleman, a k6 user, created. While it’s not baked into the tool itself, it’s fairly straightforward to add it, requiring only two snippets of code that can be pasted onto any script.

Kubedim: Microservices testing using k6 for reliability, with Kelvin Zhang (k6 Office Hours #24)

Fri, 13 Aug 2021 15:50:46 +0200

Today, my colleague Daniel González Lopes joined me as a co-host on k6 Office Hours, and we interviewed Kelvin Zhang about his thesis at Imperial College London, using k6 to test microservices-based architectures. He created something called Kubedim, an application that intelligently applies non-uniform brownout theory to intentionally dim application requests for the purpose of improving overall availability.

This was a dense Office Hours to digest, but Kelvin really impressed us with his experiments and findings! It’s definitely an awesome use case for k6 and just a great one to watch for any site reliability engineer.

Monitoring your k6 load test: how to install Grafana and Prometheus on a Kubernetes cluster

Wed, 11 Aug 2021 18:55:49 +0200

Running a load test is half of the equation, and not having the right observability tools to see how your application responds to the load can sometimes flat-out invalidate a test.

Here’s how to install open-source tools Grafana and Prometheus on a Kubernetes cluster in DigitalOcean and how a performance user could use them for increased visibility during a load test.

Realistic load testing

Sat, 07 Aug 2021 22:15:13 +0200

This is a conversation I recorded from Clubhouse in the Performance Engineers group created by NaveenKumar Namachivayam today, August 7th, 2021. I started by introducing myself and then let everyone know that I would be recording the conversation. Enjoy!

How we work at k6 - building software & working remotely with CTO Pawel Suwala (k6 Office Hours #23)

Fri, 06 Aug 2021 18:09:05 +0200

For this episode of k6 Office Hours, I got the k6 Chief Technical Officer, Pawel Suwala, to give a bit of a behind-the-scenes of how we work at k6 and what we’re working on next. We talked about the methodology we use for building k6, the different teams we have, and exciting new features that are coming soon!

Schrödinger's Pokémon: Observability for chaotic load testing (New Relic FutureStack)

Sun, 01 Aug 2021 18:54:27 +0200

This is a presentation I gave at New Relic FutureStack 2021.

Deploying an application to production is often like putting a pokémon in a box and hoping it manages to stay alive despite real-world chaos and potential outages. What can we, as testers, do to prepare applications for these situations before they go live? In this talk, I discuss chaos engineering concepts and demonstrates how to apply them by incorporating load tests with chaos experiments and setting up observability tools to watch it all happen. Because without observability, how can we tell whether Schrödinger’s pokémon is alive or dead?

A tester's overview of Grafana Labs open-source projects

Mon, 19 Jul 2021 18:23:33 +0200

Since the acquisition of k6 by Grafana Labs, the entire k6 team has been absorbed into the new, larger team that we’re suddenly a part of. Grafana Labs maintains a lot of projects, so I was very quickly overwhelmed with things to learn about all of them. Here’s my attempt to make sense of the Grafana Labs landscape, and perhaps it can help some of you as well:

How to run distributed load tests with k6 Operator (k6 Office Hours #21)

Fri, 16 Jul 2021 18:17:13 +0200

In this week’s Office Hours, Simme and I talked about the k6 Operator, which is a open-source project that Simme created for distributed load testing with k6.

k6 Cloud already offers a no-hassle distributed load testing experience, but for those who prefer to use their own infrastructure, k6 Operator allows the setup of a k6 test as a custom resource object that can be deployed along with a Kubernetes cluster.

How to add performance testing to CI/CD pipelines using k6 and GitHub Actions

Mon, 12 Jul 2021 22:26:20 +0200

Continuous testing means verifying and improving software quality at all stages, including development. It can sound like something only large companies do, but it’s an approach I use even for this personal blog that you’re reading this on.

Every new blog post or change I make to this site automatically triggers the execution of a small test. This test, when run over time, lets me see whether changes I’ve made have been detrimental to site performance. Here’s a video on my setup:

Load testing and Azure DevOps with José Luis Latorre Millas (k6 Office Hours #20)

Fri, 09 Jul 2021 21:08:36 +0200

José Luis Latorre Millas, Developer Community Lead and Software Architect at Swiss Life AG, joined Simme and me on k6 Office Hours to talk about how he personally uses k6 with Azure DevOps to add load testing to his team’s CI/CD pipeline.

I got to know José from the k6 Community Slack workspace when he mentioned that he would be presenting on k6 in a few meetups. I went to his livestream with CrossDvlUp (Spanish) and enjoyed his approach, so I knew I wanted to have him on. It’s always great to have real users of k6 (both of the OSS and the commercial Cloud option) talk about how they personally use k6 and even what they found lacking in the tool.

José’s multi-scenario template repo: https://github.com/SwissLife-OSS/K6-MultiScenario-template

Register to watch José’s talk at Azure Summit 2021, later this year: https://azuresummit.live/speaker/jose-luis-latorre-millas/

Acquisition talk with CEOs Raj Dutt (Grafana) and Robin Gustafsson (k6) (k6 Office Hours #19)

Fri, 02 Jul 2021 21:04:13 +0200

This week on k6 Office Hours, Simme and I were joined by two CEOs: k6 CEO Robin Gustafsson and Grafana Labs CEO Raj Dutt. It was a little nerve-wracking to have two CEOs on the stream, but it did give us the opportunity to ask both of them pointed questions on Grafana Labs’s acquisition of k6, including any plans they have for how both companies could integrate better in the future.

What you need to know about Grafana Labs's acquisition of k6

Wed, 30 Jun 2021 18:29:24 +0200

Recently, during Grafanaconline 2021, Grafana Labs CEO Raj Dutt announced the acquisition of k6– the company I work for. What does this mean? Here’s a video I made to explain it all:

Simme and I have also discussed the acquisition twice on k6 Office Hours:

once with our CEO, Robin Gustafsson
and another time with both Robin and Raj, Grafana Labs CEO

TL;DW: We unanimously agree it’s a positive thing. I myself am cautiously optimistic about the future of both k6 and Grafana Labs.

How to use k6 to benchmark hosting providers with Kevin Ohashi (k6 Office Hours #18)

Fri, 25 Jun 2021 20:57:42 +0200

Kevin Ohashi joined me and Floor Drees on k6 Office Hours to talk about how he used k6 to benchmark the performance of WordPress hosting providers. He talks about the difficulties he ran into while trying to keep tests repeatable and consistent, and I also make some suggestions for how he could improve his load testing script for future comparisons.

Of particular note for me in his results was how clearly the performance of the hosting providers varied depending on the location Kevin had selected for his load generators. One in particular showed a significant delay when a site was fetched from Sydney, indicating the possibility of a lack of a CDN server in that region. It made a great argument for load testing in the cloud, from different geographical regions.

WordPress Hosting Performance Benchmarks: https://wphostingbenchmarks.com/

Benchmarks methodology: https://wphostingbenchmarks.com/methodology/

k6 is part of Grafana Labs! AMA with CEO Robin Gustafsson (k6 Office Hours #17)

Fri, 18 Jun 2021 22:26:49 +0200

On this week’s Office Hours, Simme and I were joined by Robin Gustafsson, the CEO of k6, to make a big announcement: k6 has been acquired by Grafana Labs.

The folks at Grafana Labs have been fans of k6 for a while, and we’ve been fans (and users) of their work since almost the beginning of k6. While a surprise to many, Grafana and k6 together is a combination that makes sense.

Read Grafana’s press release here to hear more about this milestone.

Accessibility and open source support program with Floor Drees (k6 Office Hours #16)

Fri, 11 Jun 2021 22:38:52 +0200

On this week’s Office Hours, Simme and I were joined by Floor Drees, the newest member to join the k6 team. Every new person at k6 goes through a week of testing, and Floor decided to spend her week of testing doing an accessibility review on the k6 Cloud web app and documentation. We talk about how accessibiltiy is an often-overlooked form of operational testing, and Floor shares specific tools that testers can use to make applications more accessible.

We also discuss the k6 open source support program, which makes k6 free for open-source maintainers and regular contributors.

Resources mentioned

k6 Open Source Support Program Boia.org, a good tool for accessibility reviews AlexJS, for identifying inconsiderate language Paul Bruce’s rant on operational requirements instead of “non-functional” requirements Marie Drake on accessibility using Cypress Chrome extension to get dark mode everywhere Contributing Today - open source funding and sponsorship Contributing Today - on open source licensing

Making k6 extensions using xk6, with Iván Szkiba and Mihail Stoykov (k6 Office Hours)

Sat, 05 Jun 2021 22:28:29 +0200

This week on k6 Office Hours, Simme and I were joined by Iván Szkiba and Mihail Stoykov to talk all about extensions. Iván, who is a System Architect at NNG LLC, is the most active k6 extension creator outside of the k6 team. Mihail is a software engineer at k6, and one third of the k6 OSS team.

We talked about the many extensions Iván has created for k6 so that he can incorporate k6 into his team’s daily testing activities, his considerations for creating extensions, and how he got started. Mihail also shares some inside information on why certain decisions were made about the way k6 is structured, and common mistakes he sees people make in their custom extensions.

What Is Load Testing? - A real-world explanation

Tue, 01 Jun 2021 20:23:17 +0200

I’ve made about 60 videos on load testing and application performance to date, but I realized that I’ve never made one defining load testing, which seemed like a glaring mistake. So, here’s my attempt!

Rough transcript below - though I went off script a few times:

Load testing is a key part of making sure that an application performs well and reliably. But what IS load testing, anyway? In this video, I break it down for you in simple terms. And if you’re a family member or friend of mine trying to understand what it is I do for a living, thank you! This is for you too.

Software testing

It takes a whole team to build software, but some roles are more visible than others. For example, pretend you want to build a bridge. Who do you need to build it?

An architect to design the specifications of the bridge
A liaison or representative to the gemeente or local council to clarify what the specifications ARE
Builders who will physically build the bridge, who might need to hire subcontractors for various materials and services
Someone to double-check that what you’re building is up to code (testers)
Someone to manage it all, and maybe a few more levels of team leads
Traffic enforcers to detour traffic away from the bridge while it’s being built

So who is responsible for “building” the bridge? The answer is, everybody. Everybody has a part to play. Every part is important.

It’s the same thing with software. When we think about building software, we often think about developers, but that’s not the full picture. Just like building a bridge, it’s a team effort. But I want to zoom in on one role in particular, because it’s the role I’ve had most often: load tester.

But first, let’s go on a field trip.

Load testing on the field

Does this bridge look familar? This is the Sint Servaasbrug, and they SAY that it’s the bridge on the €10 note. It might be true, too, because it’s here in my current hometown of Maastricht, the Netherlands, where the European Union was founded in the Treaty of Maastricht. The name Maastricht means the crossing of the Maas, which is the name of this river here.

So let’s pretend the bridge hasn’t been built yet, and that we’re helping build it. Let’s put on our load tester hats. As a load tester, it’s our job to make sure that the bridge can support the weight it is intended to support.

How do we know how much weight it’s intended to support? Well, the council or the rest of the team helping us build the bridge might have some ideas on that, or some building standards that we can use. That’s a good place to start, and we should definitely discuss that with them, but as a load tester, it’s also our job to question those requirements. Let’s do some research.

This is the street leading up to the bridge. Remember, the bridge isn’t built yet, but maybe we can get an idea of the kind of traffic that the area gets.

Some things to watch out for or questions to answer:

How many people are walking?
How many are on bikes?
How many are on bakfiets bikes?
Are vehicles allowed on this street, and if so, how many of them drive through?
Do people tend to pass by in groups, or on their own?

Another thing we can do is check other bridges. This is the Hoogbrug, another bridge across the Maas river. It’s not going to be an exact match for the kind of traffic we might expect on OUR bridge, but it might give us some ideas. So we can scout this out the way we did the street near our bridge.

One difference we can see right away is that this bridge has stairs on both ends. Our bridge isn’t going to have that. The stairs do have the lip to make pushing a bike up it easier, but it’s not as easy as riding through, and any heavier vehicles are out. And people are probably less likely to go up stairs than they are to walk up an incline. So we already know that this bridge will probably experience less traffic than our bridge, but it’s still good to take notes on.

Okay, let’s head back home, write down our findings, and think about what we’ve learned.

Tying it back to software testing

What we learned was useful, but there are still a few things we don’t know:

It is 10 AM right now. But is that really the highest point of traffic during the day? To be really sure, we should really try to go back at different times. Maybe there’s an increased amount of traffic during lunch as people go to restaurants or back to their homes to eat. Maybe a horde of kids would go across our bridge after school to go home.
And we should think beyond that, too. Are weekends busier because more people are out? Or are weekdays busier because people are going to and from work?
Then there are some things that are seasonal. Maastricht happens to be in the part of the Netherlands that celebrates Carnaval, so sometimes that area looks like this. Carnaval involves a slow parade of people, bikes, and trucks through the bridge to the central town square, called the Vrijthof. That’s absolutely something we’ll have to plan for when testing our bridge.

We’ve been talking about the bridge this whole time, but as load testers, we go through the same process for software, too. What we’ve just done on our field trip is called a workload model. A workload model is describes the load profile for a given test scenario, and it involves determining what (the key transactions), how much (the load distribution among the transactions) and when (timing of the load) to test.

For a web application, for example, like a web site, we might try to look through analytics and monitoring tools to try to determine things like:

How many users access our site?
Which pages do they visit? Some pages are like pedestrians - they don’t have much comparative weight and aren’t complex to process. But other pages are like big old trucks, that involve many components of our application and are comparatively very intensive.
When do they visit the site? Is our site the sort of site people go to to work, or to avoid work? at weekends, or during the week?
Are there any seasonal events, like sales or holidays, that might drive MORE people to our site?
Where are the users from? Are they in the same country that our application servers are located in, or are they quite far away? What sort of internet speeds do they have? What percentage of them use computers to access our site, and what percentage of them use tablets or mobiles?

That was just a taste of the kinds of questions a load tester has to answer during the course of testing. There’s still a lot more to cover - for starters, we haven’t even talked about how to actually RUN the tests yet! I plan to make more videos going through typical activites in load testing in more detail, but for now, I hope that gave you a good idea of what load testing is. And Mom, I hope you know what I do for a living now. :D

Thanks for watching!

Interview on Performance Time Podcast

Mon, 31 May 2021 11:28:18 +0200

I first heard of Stephen Townshend because he happens to have gotten the same introduction to performance testing that I did– our shared mentor, Stijn Schephers. He has started a podcast called Performance Time, where he interviews performance engineers about a variety of topics. This week, it was me!

We talked about how I got started, one of my biggest failures as a performance tester, chaos engineering, what we can do to improve diversity in tech, and why I think note-taking is a key skill for any performance tester.

Application Performance Monitoring using Netdata and k6, with Odysseas Lamtzidis (k6 Office Hours)

Fri, 28 May 2021 23:24:48 +0200

This week, we had Odysseas Lamtzidis on the show to talk about Netdata, which is an application performance monitoring tool that prides itself on having the easiest setup ever. Odysseas admitted that it might not have all the bells and whistles of other monitoring platforms, it Just Works. I see this as being very interesting especially for testers like me, who just want to have something set up but might not want to be bothered by the implementation.

End-to-End Guide for Load Testing

Mon, 24 May 2021 10:48:55 +0200

Originally published on the Abstracta blog.

If you have a public-facing web application, the odds are you should probably be thinking about load testing. Any tester can (and should) add load testing to their repertoire.

Functional tests are often carried out with a single user’s experience of your application in mind, but what happens when hundreds or thousands of users are on your site at the same time? Load testing helps prepare an application for this eventuality by simulating that situation before the application goes live. That way, you have time to fix any issues you find, and there are no nasty surprises in production.

In this guide, I’m going to walk you through the basics of the load testing process, which fits in closely with traditional functional testing phases as well:

Planning: Why are we testing? What are we testing?
Writing test scripts: How are we testing?
Execution: Running test scripts against the application.
Analysis and reporting: What happened, and what do we need to fix?

Planning for load testing

During the planning phase, the team gets together and determines what load testing will involve, including the objectives, scope, resources needed, and test scenarios. The deliverable for this phase is a test plan, but that can be as formal or informal as the team requires.

Planning for any testing is a team activity, and you’ll want to get input from all the stakeholders – that’s anyone who is involved in or might be affected by the testing. The first thing you need to clarify is the reason for load testing.

Requirements

Why are we doing load testing? seems like a very simple question, but you may be surprised by the varying answers from your stakeholders. Their reasons may require different test scenarios, so it’s a good idea to agree on the collective objectives of this load testing cycle so that expectations are set accordingly. A requirement is a concise statement of how an application should behave in a given situation.

Phrasing objectives as nonfunctional requirements (NFRs) is a good way to formalize them. An NFR defines the desired state of the application and can includes performance thresholds that the team decides on. For example: Logging in as a registered user must take no more than 3 seconds (90th percentile response time) is a requirement based on response time. Good requirements are SMART (Specific, Measurable, Agreed Upon, Realistic, and Timely).

Here are some other things that should be decided on during this phase:

Scope. What are we testing? Deciding on a shared objective for load testing lets you narrow down what areas of the application need to be targeted. Make sure everyone is clear on what won’t be tested.

Photo by Daria Nepriakhina on Unsplash

Entry Criteria. What do you need to begin? This is the time to start thinking about resources that are required for testing, including the number of people who need to be involved, the time it takes to carry out the testing, and environment requirements (including test data). You don’t always need a separate testing environment to do load testing, but if you do, that should be discussed up front so that the DevOps team knows you can’t begin without an environment.

Workload Modeling. A workload model is a set of characteristics that describe the load you’re attempting to simulate. If you want your load testing to simulate registered users accessing your site, what does that entail? This might require you to look through your site analytics to determine which pages are visited most often and which application features are most commonly used, or arrive at an estimate if analytics are not available. Are most users visiting from their laptops at lunch on work days, or do they use your site from mobile devices after work? Are most of them located in the US or in Europe? You’ll want your scripts to match real load as much as possible, so take note of key characteristics that might have an effect on the traffic your application experiences.

Server monitoring. Which application components will be involved? You want to have a plan in place to have monitoring, alerts, and logs set up on every server that will be tested by your scripts. If your load testing tool shows performance degradation, you want to be able to have information to look through on the relevant servers to troubleshoot that issue. Keep in mind that any sort of monitoring adds some resource overhead, so decide as a team what information you absolutely need to track.

Test Scenarios. Which situations do you want to recreate? A test scenario objectively describes the situation to be simulated. Do you want to recreate the load profile of a typical day, or one from a very high-traffic period, such as during a sale? Do you want to recreate a production outage? You can use analytics data to come up with parameters for your each test scenario such as duration, number of concurrent users or requests, ramp up period (how quickly the application went from 0 users to your highest number of users), or how “spiky” the load needs to be (how the number of requests per second fluctuates throughout the test).

Everything discussed in this phase should be written in some sort of a load testing plan, which could be a formal, multi-page document, an epic with many user stories, or a bunch of Trello cards. Use the tools your team already uses. The format isn’t important; formalizing the outcome of the discussions is what counts. The test plan makes sure everyone has the same intent with regard to the testing.

Writing load testing scripts

Now it’s time to start writing scripts! You’ve already front-loaded a lot of the important decisions to the planning phase, so all that’s left to do in this step is to implement the plan in the form of load testing scripts.

Tool selection

There are many load testing tools available on the market, and the reality is that you don’t always get to be the one to choose what you use. Often, companies have already made agreements with tool vendors for licenses before you begin your load testing, and it would be financially infeasible to switch to another tool. However, if you’re lucky enough that you can choose the load testing tool, here are a few considerations for assessing different tools:

Resources and costs

How much does it cost? Does the pricing model allow for scaling usage up and down as necessary?
Should you build or buy an end-to-end load testing solution?
What technical skills or experience with tools does your team have? Who will be doing the testing?

Features

Does the load testing tool support the protocols and technologies used by the application?
How easy would it be for the team to learn this tool and/or scripting language?
Does the tool make it easy for multiple people to collaborate on the same script?
Does it integrate with other tools in the testing stack (for test management, test result analysis, server monitoring, team communication)?
Could you add this tool to your CI/CD pipeline?
Does the tool have an easy way of setting thresholds or SLOs for performance?
Are the test results displayed in usable reports, or does the tool integrate with existing data visualization tools you are using?

Support

If something goes wrong, does customer support cost extra?
Is there ample documentation available online for getting started without help?
Is there a community of users of the tool whose experiences you could learn from?

Scaling up load testing

How many virtual users or requests per second can a single load generator simulate using the tool?
How many load generators would you need if you used this tool?
Can the tool handle the amount of test data required by your tests?
Can you execute tests from different geographical locations, depending on your the location of your users?
Does the tool allow you to generate load from the cloud?

These questions are not exhaustive, but you can use them as a starting point for your comparisons of load testing tools.

For this guide, I’m going to outline a basic script in k6 for a few reasons:

It’s open-source, so anyone reading this can download the tool and the script and try it out for themselves;
It’s written in JavaScript, which makes it easy to learn for front-end developers or testers who have used tools like Cypress, Puppeteer, or Playwright.
It’s capable of generating tens of thousands of users on a single machine, so it’s cost-effective even with load ramped up.

However, the principles I discuss will be applicable to other tools. I highly recommend you conduct your own proof of concepts with several tools to come up with a tool that fits your particular situation, test scenarios, and company.

Installation and setup

If you’re on a Mac, the easiest way to install k6 is to use Brew. Run brew install k6 from your terminal, and you’ll be ready to go in a few seconds.

If you’re on a different operating system, check here for installation instructions that apply to you.

A basic k6 test script

A barebones k6 test script looks like this:

import http from 'k6/http';

export default function () {
 let response = http.get('https://test.k6.io');
}

This script sends an HTTP GET request for the URL https://test.k6.io, which is a simple test site. If you copy that code to your favorite text editor and save it as test.js, you can then run it with the command k6 run test.js.

You’ll get something like this in response:

A sample run of a simple local test in k6

The output above means that you’ve executed 1 iteration of your script with 1 virtual user from your local machine. The test run consisted of 1 HTTP request, which had a response time of 128.2 ms (http_req_duration).

It might be tempting to start scaling up the test at this point, increasing the number of virtual users and the duration of the script. But there’s one problem with the script as it is: it’s not realistic.

Making a test script realistic

The purpose of a load testing script is to simulate a real user who accesses your site. In most cases, that involves more than the simple GET request from the script above. Load testing scripts don’t have to mimic a real user exactly, but they do have to be close enough so as to apply the same amount, shape, and distribution of the load on your application server. If your load testing scripts aren’t realistic enough, you may get false positives during testing and discover too late that there is a performance issue in an application component that your scripts didn’t test.

Here are some ways to make your scripts more realistic:

Add static resources

If a page has static resources like images and scripts that are retrieved by a browser automatically, make sure your script retrieves those too, as they will have an impact on the response times you get. Consider excluding static resources fetched from third-party providers unless you’ve gotten their permission to run a load test on their servers.

Review your load testing tool’s default cache and cookie management settings and change them, if necessary, to match the scenario you want to test. First-time visitors to a site won’t have resources cached locally, but repeat visitors may be retrieving resources from the cache.

Add think time

Real users don’t usually fetch the same URL over and over again without stopping. They navigate to your site, then spend some time reading the contents of the homepage before they interact with the page again. Think time is the amount of time that a user spends “thinking”— that is, the delay between requests. Adding a think time that varies between 1 and 5 seconds, for example, prevents the application server from getting artificially bombarded by requests in a way that wouldn’t occur in production.

Using different user paths

A user path is the flow of how a user interacts with the application. What pages do they visit? What information do they type in? So far, you’re only testing one request, but perhaps your users may be performing other actions. The workload model that you put together in the Planning phase will come into play here. Other actions like logging in or paying for items in a cart exercise different parts of an application than browsing does, so the load testing script should reflect the most common user flows through your system. In k6, this is done by adding scenarios, but other tools may call them thread groups or execution groups.

Adding test data

In the same way that users don’t (typically) search for the same keyword over and over again and don’t all log into the same test account, a script shouldn’t use the same data repeatedly. This behavior could cause responses to be cached, leading to misleadingly quick response times, or increased error rates, in the case of multiple virtual users attempting to log into the same account. To prevent that, modify your script to take data from a CSV file instead, letting you add some natural variation.

Setting test parameters

Test parameters are the main characteristics of the test and include the number of users, ramp up, duration, and ramp down. They control the shape of the user load, which you can see when you plot the number of virtual users over time during the test:

The number of virtual users over time during a load test, as graphed in k6

Ramp up is a period of time during which virtual users are gradually added, usually at the start of the test, and ramp down is when they are gradually stopped. For most applications, ramps are more realistic than all users clicking a link at exactly the same time.

Setting failure criteria

Part of simulating real users is determining what constitutes failures in user experience. You can incorporate failure criteria into scripts by adding response verifications, checks, and thresholds (such as those for response time or error rate). Building these into the script is a good practice to get into, because it defines the limits of acceptable performance from the beginning.

The suggestions in this section are for general use, but there are many situations where taking the opposite approach would be more realistic. Spend some time thinking about what is “realistic” for your application and test scenario.

For your reference, here’s a k6 script that started from the basic script I showed earlier, but has been modified to be more realistic. That link includes a CSV file with login credentials.

Executing load tests

In the course of writing the test script, you probably would have already executed small verification tests, called shakeout tests. It’s a good idea to start with these shakeout tests on a new environment to make sure that both your test artifacts (your script and any other data files) and the environment are ready for full testing.

Running the test

If you’ve been following along with the k6 script so far, all you need to do to run a load test is to execute the following command:

k6 run test.js

This command will run the k6 script according to the test parameters set within it. The test will be executed locally. This is best used for shakeout tests, or if you prefer to run your load test on your own infrastructure.

Realism in test execution

One way can make executing load tests realistic is by matching the location of load generators to the physical location of your users. When you run a load test on machines in your data center, in the same network as your application servers, you will likely see response times that are significantly faster than your users would see. These false positives can look good on the report, but they don’t help you measure real user experience.

Running on the cloud is an easy way to make tests more realistic, especially if most of your end users are external to your organization.

If you’re following along with the k6 script, you can sign up for an account with k6 Cloud (the first 50 tests are free). Since k6 itself is open source, you can also run it on your own cloud infrastructure, but using k6 Cloud is the most convenient way to get started.

Once you have an account, copy your API token from here. Then, run this command in your terminal to give your local k6 installation permission to run on your account:

k6 login cloud --token <YOUR_K6_CLOUD_API_TOKEN>

Once you are successfully authenticated, you can continue to run tests locally using k6 run, but you now have the option of running on the cloud as well:

k6 cloud test.js

k6 will then begin to execute your load test, and you’ll see a screen like this:

Note that the execution mode is now cloud, to remind you where you’re running the test.

By default, your test will run on the k6 AWS account, in the Ashburn region. However, you can override this and specify other regions, including percentages of your virtual user to run on each, using cloud execution options.

Analyzing load testing results and reporting

Although I’m listing analysis as a different step in the process from execution, the two often (and should) overlap. The output from k6 in the screenshot above contains a link to the dashboard on k6 Cloud, so that you can see the results of the test in real time, as the test is running. Monitoring the test in real time lets you immediately spot issues as they arise, and, if necessary, stop the test to fix them. If the test is being executed by multiple load generators, having a consolidated dashboard is helpful in giving you an overview of the entire test.

![An example of the results after a load test in k6 Cloud](/assets/Finished k6 run.png) An example of the results after a load test in k6 Cloud

This is also a good time to monitor the health of the application servers and the load generator servers, which you set up monitoring tools for during the planning phase.

A load test is only useful when it’s providing information about the application. If errors occur in large quantities, or if the response time makes it clear that a certain component has failed to handle the load, don’t be afraid to stop the test. Your objective is to get the information you need to suggest performance improvements and solutions for any bottlenecks you find.

Often, you’ll run into issues you can’t solve or even identify during a test. That’s why it’s useful to explore the data that you have after a test. k6 lets you create shareable links to dashboards so that you can involve relevant engineering teams who may be able to help you understand what happened and why. Here’s an example of what that dashboard looks like.

Some projects require formal documentation of test execution by way of a Test Summary Report, but in more agile teams, it may be enough to share the test results dashboard and some supplementary information about what type of tests were run and what problems were encountered.

The most important part of load testing isn’t the execution itself; it’s what you do with the results. Communicating results clearly to stakeholders will make it more likely that performance issues are fixed.

What’s next

The phases I’ve outlined here are a good guide for anyone starting with load testing, but I always recommend that you take what works for your situation and leave the rest. In reality, the phases overlap with each other considerably, especially in more modern engineering cycles.

Another activity I’ve not covered here, but one that I recommend, is adding load testing to your CI/CD pipeline. This continuous testing approach involves automating load testing so that it occurs in regularly, in parallel with other development and testing activities. Continuous testing is a great way to get more value out of load testing, and it ensures that performance is front-of-mind for everyone on the application team.

Happy testing!

Learning how to use k6 live with Nočnica Fee from New Relic (k6 Office Hours)

Fri, 14 May 2021 21:53:09 +0100

We were happy to welcome Nočnica Fee from New Relic to k6 Office Hours. She very bravely volunteered to learn how to use k6– live! Talk about pressure. But she handled it like a champ!

We also talked about why observability is both particularly important and particularly difficult in serverless architecture. There will definitely be a part two at some point!

Analyzing load testing results in k6 Cloud

Tue, 11 May 2021 21:53:09 +0100

Have you ever run a load test but been unable to decipher what it all means? In this video, I break it all down using k6 Cloud.

Performance testing tool feature requests (k6 Office Hours)

Fri, 07 May 2021 21:53:09 +0100

We managed to get two other folks from k6 to join our Office Hours: our CEO, Robin Gustafsson, and our Head of Customer Success, Mark Meier. We took advantage of the opportunity to discuss wishlist features for k6 and bug Robin about how likely each feature was to see the light of day!

Using the k6 Cloud interface

Tue, 04 May 2021 21:53:09 +0100

I’ve gotten the feedback that people would like to see more videos of the basics of using k6 and k6 Cloud, so here’s a video on getting situated with the k6 Cloud interface and navigation.

How to get started with chaos engineering (k6 Office Hours)

Fri, 30 Apr 2021 21:53:09 +0100

Recently, I’ve been learning more about chaos engineering. In this week’s k6 Office Hours, I shared my experience as a tester getting started with Kubernetes, chaos engineering with xk6-chaos and k6, and observability with New Relic.

Shift-left testing and creating a culture of quality with Simon Prior (k6 Office Hours)

Fri, 23 Apr 2021 21:53:09 +0100

Simon Prior from the Testing Peers podcast joined us on k6 Office Hours to talk to us about how he’s managed to foster a culture of quality at his job with EasyJet.

The future of load testing with k6, with k6 CEO Robin Gustafsson (k6 Office Hours)

Fri, 16 Apr 2021 21:53:09 +0100

This week, Simme and I convinced k6 CEO, Robin Gustafsson, to come onto k6 Office Hours to talk about the origins of k6 (spoiler: it began with a company called Gatorhole), things we’re working on right now, and what he sees as the future of the company.

Testing Kubernetes and infrastructure as code with Sid Palas (k6 Office Hours)

Fri, 09 Apr 2021 21:53:09 +0100

For this week’s k6 Office Hours, Simme and I had Sid Palas from DevOps Directive come on board to talk about what Kubernetes is, how it relates to Infrastructure as Code, and why either of those should matter to testers.

Using CircleCI and environment variables with k6 (k6 Office Hours)

Fri, 26 Mar 2021 21:53:09 +0100

Simme and I decided to take this week to answer questions on k6 Office Hours, and we ended up talking about CircleCI, environment variables, and how to use them with k6.

Performance Testing with k6 and Cypress (Ministry of Testing)

Tue, 23 Mar 2021 19:52:48 +0100

My friend Marie Drake and I teamed up to give a presentation about performance testing at a Ministry of Testing (Philippines) meetup, and we talked about using Cypress for front-end performance testing and k6 for back-end performance testing.

Performance testing vs. load testing with Señor Performo and Pepe Cano (k6 Office Hours)

Fri, 19 Mar 2021 21:53:09 +0100

What’s the difference between performance testing vs. load testing? This week, Simme and I were joined by Señor Performo, aka Leandro Melendez, and Pepe Cano, the Head of Marketing at k6, to talk about this common question.

Load Tests as Code: An introduction to k6

Thu, 18 Mar 2021 20:36:41 +0100

Today, I attended the Cambridge University Press QA Week 2021 to talk about load testing with k6. Here’s a recording of the presentation.

And here are the slides I used!

I had fun discussing the advantages of the load test as code approach and why k6 is a great way to implement it.

Testing Our Performance Test Tools

Tue, 16 Mar 2021 21:29:33 +0100

On this episode of the TestGuild performance podcast, I had a chat with Joe Colantonio about languages, board games, nontraditional backgrounds in tech, and why we should all test our performance test tools.

k6 Office Hours 05: Observability with Daniel González Lopes

Fri, 12 Mar 2021 21:53:09 +0100

Our first repeat guest, Daniel González Lopes, also from k6, joined us today to talk about how to use observability in conjunction with load testing. We talked about the differences between metrics, logs, and traces, and how to implemented distributed tracing.

k6 Office Hours 04: Using Cypress with Marie Drake

Fri, 05 Mar 2021 21:46:56 +0100

Marie Drake joined us today on k6 Office Hours to talk about front-end performance testing using a test tool called Cypress and how it can compliment a back-end performance testing strategy.

k6 Office Hours 03: Chaos Engineering with Daniel González Lopes

Fri, 26 Feb 2021 21:40:25 +0100

In this week’s Office Hours, Simme and I invited Daniel González Lopes, a DevOps engineer from k6, to join us to talk about chaos engineering and what it has to do with load testing. Simme also demonstrated how to use k6 to run simple chaos experiments.

k6 Office Hours 02: Observability with New Relic

Fri, 19 Feb 2021 23:24:58 +0100

For this week’s Office Hours, Simme and I invited Gary Spencer from New Relic to talk to us about why he’s recently started using k6, and why an observability platform like New Relic is the perfect companion for performance testing and observability.

The 503 Podcast

Thu, 11 Feb 2021 21:46:35 +0100

This must be the month for trying new things, because I jumped into livestreaming with the first ever k6 Office Hours just a week ago, and this week I’m jumping into podcasting with Simme Aronsson, who’s also doing the Office Hours with me. There’s definitely safety (at least psychological safety) in numbers, because doing it with someone else makes me feel a little more brave.

While the podcast is presented by k6, that’s really just a fancy term for saying that we’re both employed by k6 and we’re recording and producing the podcast on company time. Unlike the Office Hours, which is heavily k6-focused, the podcast will be about software performance and reliability in general. Simme and I happen to have different and complementary skillsets (he comes from a developer background, and I come from a testing background, and I think that makes for an interesting combination of perspectives.

Here’s the first episode! It should also be available on your audio podcasting tool of choice.

The first ever k6 Office Hours

Fri, 05 Feb 2021 21:47:16 +0100

The k6 virtual office is open!

The first time I heard about office hours was from Notion. Recently, Obsidian went in on this trend as well. The concept involves opening your virtual doors regularly to your community, to answer questions and share best practices.

A big part of being a Developer Advocate is talking to people, and even conferences these days are mostly pre-recorded. I kind of miss being able to interact with people. I miss the immediacy of live. So I talked Simme Aronsson into doing a k6 Office Hours with me.

We’re both new to livestreaming, so there’s a lot that we’ve learned and will be improving in future videos, but here’s our first attempt!

Week of Testing Wrap-up: My thoughts on k6 as a load testing tool

Thu, 17 Dec 2020 15:29:57 +0100

I decided to make a final wrap-up video for my “Week of Testing” with k6… ok, so it’s actually three weeks later now, but I wanted to explore k6 a little bit more before I gave my opinion on it.

And here’s the audio version:

Thank you, k6, for being awesome enough to agree to this video series before knowing what I’d say. Gotta love a company that welcomes transparency and feedback.

Cuándo JMeter no alcanza: Pruebas de carga simulando navegadores

Wed, 16 Dec 2020 18:18:23 +0100

Aquí está el video de mi presentación en JMeter Scripting Day, un evento organizado por JMeter en Español el 31 de Oktubre, 2020.

Y la versión audio:

Introducción

Hola a todos! Yo soy Nicole van der Hoeven, y soy performance tester. JMeter es una de mis herramientas favoritas porque es gratis, simple, y por muchos casos hace exactamente lo que quiero y no más. Sin embargo, como performers es parte de nuestro trabajo elegir la herramienta adecuada para cada proyecto, y hay que admitir que hay situaciones en que JMeter simplemente no es suficiente.

Amazon - UI + Dev Tools

Esto es el sitio web de Amazon.com. Tengo abierta la pantalla de Chrome Developer Tools, que me muestra las solicitudes (o requests) que componen la página. Cuándo vamos a la página, el navegador (Chrome en este caso) envia estas solicitudes para nosotros, pero eso occure en el fondo. Un usuario normal no las ve.

JMeter - Creando un script

Lo que JMeter hace es grabar estas solicitudes y replicarlas. Aqui vemos una ejecución de un usuario virtual de un script en JMeter. Cada solicitud tiene la respuesta y el tiempo de respuesta. Pero eso no nos da una idea completa del rendimiento de la aplicación. ¿Por qué?

Por que JMeter, como otras herramientas de nivel protocolo, tiene unas limitaciónes. La primera es que si hay javascript o otros script que deben estar ejecutado por el navegador, JMeter no los ejecuta. JMeter solo los descarga. El tiempo de respuesta mide la descarga, no la duración de la ejecución. Entonces, siempre habrá una diferencia entre los tiempos de respuesta en una prueba de JMeter y los tiempos de respuesta en práctica.

A veces esta diferencia no es tan importante. Pero hay más y más aplicaciones hoy en día que se componen principalmente de estos javascript. Por esas aplicaciones, simplemente no es posible hacer un script en JMeter porque la mayoría de la lógica de la aplicación occure en el navegador, no en el servidor.

Flood Challenge

La segunda razón es que el aumento de la seguridad de aplicaciones conduce a solicitudes más complejas que incluyen parámetros dinámicos. Estos parámetros hacen que sea más difícil para un intermediario lanzar un ataque, pero también hacen que sea más difícil para nosotros scripters, porque tenemos que extraer información desde la última respuesta y ponerla en la próxima solicitud. Es posible, pero es la pesadilla de cada performance tester.

Podemos verlo en la aplicación de muestra Flood Challenge. Ya no trabajo para Flood, pero todavía es un buen ejemplo.

Correlación en JMeter - Flood challenge script

Esto es mi script en JMeter, y vemos que hay muchos parámetros que he tenido que correlacionar. La solicitud para simular hacer clic en el botón “Start” necesita los parámetros authenticity token, step ID, y step number, que son dinámicos. El contenido de los parámetros cambian cada vez que usamos esta aplicación.

La experiencia del usuario

La tercera razón es que JMeter tampoco mide la verdadera experiencia del usuario. Solo mide la duración de la comunicación entre el navegador y el servidor, pero no mide cosas como la representación de elementos en la página web, ni el efecto de un script que dura mucho tiempo para terminar.

Eso es importante porque estas cosas afectan significativamente la experiencia del usuario. De hecho, Steve Souders, un conocido el la industría de performance, dice que hasta 90% del tiempo de respuesta que experimenta el usuario está en la interfaz. Con este numero podemos estar ciertos que una prueba con una herramienta de nivel protocolo no va a poder encapsular el rendimiento final de una aplicación.

Developer Tools - Lighthouse

Entonces, ¿qué podemos hacer?

Lo más fácil es intentar de medir la interfaz durante una prueba de cargas. Mientras ejecutamos una prueba con JMeter, podemos ir manualmente a la aplicación web en Chrome para usar Developer Tools. Podemos usar la pestaña Lighthouse, que nos da más métricos del perspectivo del usuario y sugerencias para mejorar el rendimiento.

WebPageTest

También uso WebPageTest a veces. Eso tiene más opciones para ejecutar la prueba automáticamente varias veces desde un lugar que puedo elegir. Aún podemos elegir un dispositivo para simular la experiencia de un usuario en un iPad, por ejemplo.

Herramientas alternativas para pruebas de carga

Tambien podemos elegir herramientas alternativas para ejecutar nuestras pruebas de carga. Lo que tienen en común es que fueron diseñadas para pruebas de automatización, y utilizan un navegador real.

Selenium es probablemente la más popular de esas. Selenium WebDriver en particular. Es compatible con Ruby, Java, Python, C# y Javascript. El problema con Selenium es que puede ser muy pesada. En una machina probablamente podemos ejecutar cinco usuarios de Selenium, más o menos. Eso es importante si estamos tratando de simular miles de usuarios.

Hay muchas herramientas basadas en Javascript que también podemos usar. Hay varias framework que se utilizan para adaptar Javascript y nodeJS.

Puppeteer fue desarollado por Google. Tricentis Flood también ha diseñado Element que utiliza Puppeteer, pero con un enfoque en el rendimiento.
Playwright es una creación de Microsoft.
Jest fue desarrollado por Facebook
Mocha es otro framework que es la base para Cypress.

Todos esas herramientas son open source y gratis. También hay alternativas propietarias, pero prefiero las soluciones open source porque en la mayoría de casos hacen la misma cosa y se actualizan con más frecuencia.

Element script

Esto es un ejemplo de la misma acción, hacer clic en el botón Start en la aplicación Flood Challenge.

Otra ventaja de simulando navegadores en vez de simulando las solicitudes es que es más simple crear un script. No tenemos que saber cada parámetro. Solo tenemos que describir lo que queremos que cada usuario virtual haga.

Podemos ejecutar scripts usando estas herramientas al mismo tiempo que un script de JMeter o Gatling o lo que sea, y también tenemos la opción de reemplazar los script de protocolo con estas.

Conclusión

Para resumir, JMeter y otras herramientas de nivel protocolo tiene algunas limitaciones.

No ejecutan scripts del lado del navegador
Pueden ser más complicado correlacionar parámetros dinámicos
No miden la experiencia final de un usuario

Para solucionar estos problemas, podemos complementar pruebas de carga tradicionales, de nivel protocolo, como JMeter, con herramientas simulando navegadores. De este modo podremos tener una comprensión más completa del rendimiento total de la aplicación.

Bueno, se acabó el tiempo, entonces los dejo aquí. Gracias por escuchar, y me disculpo por mi español. Gracias a todos!

Week of Testing: Day 5 - The thing about learning in public

Sat, 28 Nov 2020 20:03:04 +0100

Yesterday I mentioned that I’d used the constant-vus executor to recreate the stepped load profile. Well, the thing about learning publicly is that it also means making mistakes publicly!

I was kindly informed by a few of my new colleagues that constant-vus was not the most optimal way to recreate the stepped load profile. Simme wrote up a little function so I wouldn’t have to repeat myself so much, and Pepe Cano and Pawel Suwala (the CTO of k6 himself) both told me nicely that using stages would have been the better solution.

So, let’s chalk that up to a TIL (Today I Learned) and record that snippet here for prosperity (found in the k6 documentation linked above):

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
 stages: [
 { duration: '30s', target: 20 },
 { duration: '1m30s', target: 10 },
 { duration: '20s', target: 0 },
 ],
};

export default function () {
 let res = http.get('https://httpbin.org/');
 check(res, { 'status was 200': (r) => r.status == 200 });
 sleep(1);
}

Week of Testing: Day 4 - Load profiles on k6

Fri, 27 Nov 2020 19:54:06 +0100

k6 has a cool feature where even if you start a load test locally via the CLI, you can still stream the results to k6 Cloud. It’s an interesting compromise between debugging a test locally (and therefore not paying for cloud infrastructure) and sharing results for load tests run on the cloud. It worked seamlessly, too.

I also wanted to see how to recreate a specific load profile that I commonly use - the stepped load profile. In JMeter, I typically do this with a plugin - the jp@gc Ultimate Thread Group is my thread group of choice. So, I created it in JMeter (I love that it shows a little image of it as well, while you’re building it) and then attempted to recreate it in k6.

I definitely had to look at the k6 documentation for this. I used the constant-vus executor for this, but k6 has several other executors to choose from, and it worked as well as I had hoped.

So far, my experience with k6 has been positive. Things are well-documented, and they just work out of the box. No plugins necessary.

Week of Testing: Day 3 - Running a load test locally using k6

Thu, 26 Nov 2020 19:31:33 +0100

I’ve had my fun playing around with k6 Cloud, the SaaS component. For Day 3, I thought it was time to jump into the heart of k6: k6 the open source load testing tool.

First impressions: logging was a little sparser than I expect, coming from having used load testing tools that just do this for me automatically. I think one of the first things I’ll do with k6 is to create my own little “load testing starting script” that I’ll always use, with all of the appropriate logging already set up.

Otherwise, I was really impressed with how quickly it installed. If you scrub over to the timestamp 1:46 in the video, you’ll hear that I was actually expecting to have to speed up the installation process while editing to save time, but the installation actually finished before I could say that I was going to do so. By my count that was 7 seconds between hitting enter in my terminal and the download finishing.

7 seconds.

I can’t help but compare that to JMeter. JMeter is honestly probably my go-to load testing right now– it’s not good for absolutely everything, but it’s the tool that I reach for when I “just want to test this one thing”. I’ve used it for years, and it’s the tool that I know the most. But one of the issues with JMeter is that there’s so much to do before it’s actually usable.

Before you even open JMeter, you need to install Java, and when I first started in load testing, I spent a long time wrestling with getting setting the Java environment variable in Windows before my computer would even recognize that Java was installed. Then you need to install JMeter– but any JMeter tester will tell you that there are some plugins that you just can’t do without, and that really you should always install. That’s why there are so many “best plugins in JMeter” lists (here’s my top 3)… because most of them are necessary, but you wouldn’t know that if you’re new to JMeter.

The fact that k6 starts up in 7 seconds is… shockingly good.

Week of Testing: Day 2 - Comparing tests against a baseline

Wed, 25 Nov 2020 19:21:53 +0100

Day 2 of my Week of Testing!

Yesterday I ran my first load test on k6, and actually set up the thresholds wrong. I still maintain that there are some UI improvements that could still be made in that area, and that feedback was well-received. Apparently I’m not the only one who has made the same mistake.

Today I learned that on k6, you can actually set one test to be a baseline. There’s so much I want to say on this topic - that’s probably a whole blog post there - but for now, suffice it to say that having a baseline test to compare succeeding tests to is a key to a successful round of load testing. I was happy to see that k6 thinks so too, because that functionality is built into the UI by default.

I also think that being able to schedule a test regularly is fantastic for practicing Continuous Testing. Of course, teams already using CI/CD pipelines would probably get the most benefit out of running load tests using the command line rather than through the UI, but it’s nice to know that scheduling is an option in the UI as well.

Week of Testing: Day 1 - Running a load test on k6

Tue, 24 Nov 2020 19:13:03 +0100

On my first real day of testing with k6.io, I wanted to jump right in and figure out how to run a load test on k6 Cloud. k6 has documentation on this, but I didn’t use it for this test, since the point of a week of testing is to record how easy the product is to use.

My verdict: I’m pleasantly surprised with how easily everything went. Is it just me, or was that AWS startup time exceptionally fast? I’m used to AWS taking at least a couple of minutes to provision a machine, but this seemed to start up almost instantaneously.

Week of Testing: Intro

Mon, 23 Nov 2020 20:45:31 +0100

Today was my first day at k6.io! I’m happy to announce that I’ve joined as their newest Developer Advocate for Performance Testing. k6 is a company I’ve had my eye on for a while, even (or maybe especially) when I was working for a competitor, Tricentis Flood. I always admired k6’s stance on open source, their involvement with the developer community, and the slick web app that brings load testing into the hands of many developers and testers.

As a load tester, I try to be tool-agnostic. I have to admit, though, that I do have a strong bias towards opensource tools. I joined Flood because of their focus on tools I already loved like JMeter, Gatling, and Element (based on Puppeteer). (And Selenium, but it’s a stretch to say I love Selenium.) But these are tools I’ve used for years, most of them from even before I joined Flood. What kind of load tester would I be if I could only really talk about scripting in a handful of tools?

So I thought it was time for a change, and k6 was the first and only company on my list for a couple reasons:

The free and open source tool k6, which recently crossed over to 10,000 stars on Github, is often touted as the developer’s load testing tool of choice.
k6’s cloud offering integrates seamlessly with the OSS tool, and it adds some killer features like performance insights that add real value to teams wanting to scale their load tests.
The k6 team is incredibly responsive to the needs of their community. Unlike most companies, where only marketing, sales, and customer support people talk to customers, k6 developers are active on GitHub, social media, and other blog sites. Everyone is passionate about what they’re building.
Roles in testing are changing, and the lines between tester and developer are blurring. Developers who don’t test are going to become just as obsolete as testers who don’t develop. I see k6’s popularity among the developer crowd as a good opportunity for me to sharpen my coding skills.

So I had to see for myself whether k6 was actually as good as it seemed.

Luckily, it turns out that k6 has a tradition of a “Week of Testing”, where every new member of the k6 team, regardless of role, spends their first week putting the product through its paces. Every part of the whole k6 solution (Cloud and OSS tool) is fair game, and the goal of the Week of Testing is to present feedback to the team at the end of it, while the new member’s eyes are fresh. When they told me about it, they said “We want to see you learn the tool on your own to see what parts are difficult to figure out.” To which I responded, “Challenge accepted!”

I decided to not only give a presentation at the end, but also to video the process for posterity’s sake… and, let’s face it, to impress my new managers and coworkers with my dazzling work ethic.

Step 1: Get the company’s name right.

In the first video, I share the differences between k6 vs. k6 Cloud vs. Load Impact!

How to Become a Load Tester

Wed, 07 Oct 2020 14:15:45 +0100

Originally posted here.

Load testing can be a pretty intimidating field to get into, but it’s easy to get started. Here are some concrete steps you can take to start learning how to load test from scratch.

In this video, Flood co-founder Tim Koopmans and I teamed up at Tricentis User Conference 2020 to talk about the essential ingredients for load testing.

Background required for load testing

Load testing can be quite technical. It involves not just the ability to use different programming languages to create load testing scripts but also familiarity with software engineering practices, DevOps, application infrastructure, site reliability, and networking. A good load tester needs to be able to understand the basics of how an application works and how different components interact with each other. It’s common for a load tester to work on applications employing a variety of technologies, so staying up to date with recent technological trends and switching seamlessly from one language to another, or from one platform to another, is of utmost importantance. This technical side of load testing might make you think that a load tester needs to have formal studies in computer science and several years of experience in software testing. The reality is that there are many load testers who stumbled into the field from wildly unexpected backgrounds. So you might think load testing looks like this:

‍ Photo by Christina @ wocintechchat.com on Unsplash

And sometimes it does. But it more often looks something like this:

‍ Photo by You X Ventures on Unsplash

You see, load testing is an interdisciplinary field. So much of the job involves talking to people. Sometimes the requirements for a project are not clear, and someone needs to talk to the both the business and the developers to make sure that nonfunctional requirements are clarified. A load tester is a detective, tracing a request as it goes from one application component to another, coordinating with different application teams. A load tester is a scientist, using the scientific method to design experiments that yield the results. A load tester is a statistician, making sense of a jumble of data and seeking correlations between variables. A load tester is a public speaker. Even when the actual load testing has been completed and the report has been sent, a load tester makes sure all stakeholders understand the report so that the results can be used to make meaningful changes to improve application performance. All these tasks require a mishmash of skills and competencies that can be difficult to find in just the field of computer science.

Let’s take our backgrounds as anecdotal examples.

Tim grew up and currently lives in a small country town in Australia, and he joined the military straight after school. He later developed a love of automating things, which led him first to use tools like perl to scrape websites and try to win bids with early versions of ebay, and then later to starting a performance engineering consulting career and co-founding Flood itself.

I, Nicole, grew up in Metro Manila in the Philippines (which has 14,000 more people per km2 than Mumbai does). I learned English as a foreign language in school, and I have a university degree in Economics. When I discovered load testing, I had a job in a completely different field– I was working for an airline and making sure engine components were properly maintained and repaired. My love for gadgets and technology took me down an 8-year rabbit hole involving doing performance consulting for clients across a variety of industries in Australia and the Netherlands. At Flood, I help customers around the world improve their application performance.

As you can see, Tim and I came from very different backgrounds and different parts of the world. Neither of us knew what load testing even was when we started working. Neither of us got formal instruction in load testing. But somehow, we still both ended up becoming load testers at Flood.

Reasons to become a load tester

So why become a load tester? Here are some practical reasons.

Load testing is cool

A little-known fact about load testing is that it can be really satisfying to be involved in big-name projects that end up going live.

You may have heard of Flood. But even if you haven’t, we’re sure you’re familiar with the results of load testing done on Flood. In the last few years, we have been helping Hotstar, one of the largest streaming services in the world, load test their systems in preparation for the Indian Premier League.

Hotstar uses an open-source tool that we support, Gatling, to create scripts to simulate users tuning in to watch cricket games. Then, they use Flood to create a “Game Day” simulation. They scale up those load tests and run 5 million users on thousands of machines in the cloud. We help them coordinate execution, spot problems quickly with real-time results from all those machines while the test is still running, and identify bottlenecks so that they don’t occur in production, when India’s playing.

As a load tester, you’ll definitely be working with brands you recognize and love.

Load testing can pay well

‍ Source: Payscale

I said this would be a practical list, right?

Here are 261,768 practical reasons. The average performance engineer’s salary is 71% more than that of an average software tester in India, according to Payscale.com. We all know that money isn’t everything, but this is a clear signal from the market about the importance of performance-related skills. Adding load testing to your CV as a tester makes you more attractive to employers. It’s a great way to differentiate yourself in an over-saturated market.

Load testing can literally take you places

‍ Source: Shamim from Flood

Then there are the side benefits of being in load testing. The photos above are real ones from the Flood team. That’s me on the upper left, exploring a castle in Belgrade, Serbia, and that’s Tim in the bottom right, taking a photo in San Francisco, USA.

We’re a fully distributed team on Flood, which means we can work from anywhere. Flood is a web app, so you don’t need to install anything– I’ve been known to start load tests from my iPad in a café. Load testing on the cloud frees you from being tied to running tests from the office. You can start load generators when you need them and shut them down when you don’t– all from a web browser. It’s a job that can come with a great work/life balance, if you find the right company.

Load testing can also be a great way to move to another country. It’s what’s allowed me to move to other countries– twice. In Australia, the average performance engineer’s salary is AU$108,053, or over 5.6 million rupee.

‍ Source: Payscale

So it’s clear that there are some pretty good reasons for you to become a load tester. I’m going to walk you through the practical steps to become a load tester. You may not be an expert by the end of this, but I hope I can convince you that you can get started in load testing whether your background looks like Tim’s or mine.

Load testing adds real value

And the trite, but absolutely true reason: Everyone fails.

As a load tester, you are a gatekeeper of quality. What you do directly affects revenue and customer experience.

Performance is an aspect of software that is often overlooked, but it’s only becoming more important in a world where everything is going online. People are being born who have never known what it’s like not to have the internet, and consumers in general expect applications to be responsive and stable. Would-be customers who can’t get an e-commerce site to load within 3 seconds may switch to a competitor’s site, and we all know there are many of them. Performance-related outages can be seriously damaging to a company’s reputation, and social networking amplifies customer dissatisfaction by effectively giving everyone a megaphone to complain into.

Software functionality is still important. However, if an online store goes live with no critical functional defects, but it fails when the 100th person logs into it, or it takes 10 seconds to load the homepage, can you really say that it was ready for production?

Load testing is a crucial part of ensuring software quality, and quality is the responsibility of everyone on a team, even if the word “performance” isn’t in your job title.

The essential ingredients of a load test

Load testing is like cooking a meal. This is one of Tim’s favorite dishes, aloo matar.

‍ Source: Cubes n Juliennes

You can approach cooking as a science. There are recipes with a list of ingredients, a precise measurement of how much you need for each, and a procedure for creating the dish.

But you can also look at cooking as an art. Cooking the same dish over and over again will give you an experience in working with those ingredients that a novice cook won’t have. There’s room in cooking for adding one ingredient, changing the quantity of another, or just trying something new.

A master chef knows how to use both the science and the art to create something delicious. But you don’t HAVE to be at that level in order to cook.

Anyone can cook. If you know how to follow a basic recipe, you may not create a masterpiece, but if you’re like me and you haven’t had lunch yet, you’ll still end up with something you’ll eat.

So here’s the recipe for load testing.

I prefer kadai paneer myself, but aloo matar requires, among other things, peas. And so does load testing.

P is Preparation. To prepare for a load test, you need to get your ingredients ready. You need to decide what it is you want to test, why you’re testing it, and how. You write load testing scripts that will simulate real users as closely as possible.

E is Execution. This is when you do the actual cooking. You run your load tests and scale them up according to your requirements.

A is Analysis. You figure out whether the dish you created is edible. You determine how your application servers responded during the load test, and decide if that’s acceptable.

Preparation – Execution – Analysis.

Preparing for a load test

“As far as the World Cup is concerned, it is a process. We don’t want to jump to the 50th floor straight away. We must start on the ground floor.” - Sachin Tendulkar, via ESPN

Load testing doesn’t start with writing a script. Load testing starts with building a foundation for later tests.

‍ Source: Inside Sport

Part of that foundation is asking WHY. Why do you want to do a load test? Has there been a performance-related issue before? Are you trying to determine whether an application is ready to go live, or is it already live, and you just want to improve its performance? What type of application do you want to test?

WHOM are you cooking for? I think you’d cook differently for yourself than you would for Sachin Tendulkar. What do your end users typically do on your site? Where do they come from?

These are questions that need to be answered before you think about scripting. Here are some resource for planning your load tests:

Start with Why: How to write requirements for API load testing How to write a test plan for load testing Selecting a tool for load testing Generating a load testing script without coding: Getting Started with Test Builder

Execution of a load test

So now we’re ready to start cooking. This is the Execution phase. You’ll need a machine to use as a load generator. Depending on the number of users you want to run with, you may need more than one machine. You might be tempted to run a load test on your laptop, or on a bunch of machines you have lying around, but that introduces new variables that may affect the results.

‍ Source: Ashwini Chaudhary on Unsplash

Like any good cook, you’ll also need to make sure you have all your ingredients handy. In load testing, that means making sure your scripts and test data files are ready to be executed on every load generator.

Both of these problems – provisioning a load generator and transferring files over to them – are solved pretty easily by running on the cloud using something like Flood.

Designing and executing your load test

Where should you run your load tests? A look at demand, hosted, and on-premise infrastructure

Analysis of Results

The last part is Analysis.

‍ Source: Fine Dining Lovers

This is a pretty crucial moment, because this is when you figure out whether what you cooked was delicious or not. Unfortunately, in load testing, it’s not always easy to tell how your test went. It’s like this photo of pakora (onion is my favorite kind). It’s kind of hard to say what expression that is in their faces. Is it disgust? Or is the pakora so yummy that they just can’t wait to eat more?

Load testing typically generates a LOT of data. You’d have to go copy every single file from every load generator you used. But can you imagine the hundreds of load generators that Hotstar used? Of course they didn’t manually copy those files. Instead, they used Flood.

On Flood, we show real-time graphs as you’re running the test so that you can tell immediately whether transactions are failing or whether a server has stopped responding.

After the test, you can also compare this data to previous tests to spot trends, or even create your own dashboards using an analytics engine.

Real-time metrics during your load test

Downloading results and logs after the test

Is that it?

You’ve prepared for a load test, you’ve executed a load test, and you’ve analysed the results from the load test. Congratulations, you’re a load tester! By following this recipe, you’re already doing better than many people, and you’re certainly doing a lot better than those who do not run any load tests at all.

The cool thing about load testing, though, is that this is just the beginning. There’s always room for improvement.

‍ Source: Pop Up City

Dabbawallas are people who deliver hot lunches, and while the concept is not new, the implementation in India is particularly interesting.

For these dabbawallas, having the meals prepared is not the end of the job. Once the food is ready, it needs to be packed into lunchboxes, picked up, separated according to destination, put on trains, and delivered. Later, the empty lunchboxes are picked up, and the system works in reverse.

It turns out that this is actually a pretty sophisticated system. In fact, it’s been estimated that only one tiffin every two months is not delivered. That’s equivalent to a Six-Sigma quality rating of 99.9999999% (that’s nine nines) or more.

Dabbawallas didn’t get that rating just by delivering quickly. They did it by being consistent and reliable. They did it by having a process.

Once you see the benefits of running a load test, you may also start to understand that the real value of load testing is not in a test that you run once whenever you think of it. Imagine having the benefits of a load test tied to your development process, so that it is automated.

As an advanced topic, adding your load testing to your team’s Continuous Integration/Continuous Development pipelines such as sets up a repeatable process that reaps benefits far into the future, not just whenever official load testing activities are carried out.

Continuous Integration with Flood: Integrating Travis CI, Circle CI, and Jenkins

Conclusion

So that’s the recipe for becoming a load tester.

Preparation – Execution – Analysis.

In Preparation, you determine why you’re testing and what tool best fits your needs. We showed you how to use Element, but there are many open source tools out there. Don’t get overwhelmed by scripting– a lot of times it can just mean replacing a URL with the one YOU want to test.

In Execution, you run a baseline test that will serve as a point of comparison for future testing, choose the number of users and the number of regions you want, and then you run your test. Running on the cloud is the easiest (and usually cheapest) way to get started.

In Analysis, you view the results in real-time, compare historical performance to see trends, and share reports to stakeholders so that your team can make changes to your application.

We also talked about the bonus topic of CI/CD and how you can incorporate load testing into your pipeline to get a continuous feedback during your development cycle.

Tim and I started in very different places and had different experiences before we discovered load testing. But eventually, we ended up on the same path, and that’s what we’ve outlined for you here. We hope that this convinces you that load testing doesn’t have to be complicated.

You absolutely don’t need to be an expert to run a load test. Even if you don’t think you’ll get all these steps right, a load test that is less than ideal is still better than no load testing at all.

In the end, anyone can be a load tester with the right attitude and the right tools.

Learning about load testing

High Performance Websites book by Steve Souders

Ministry of Testing/PerfBytes list of resources

PerfBytes podcast (EN and ES)

PerfGuild conferences by Joe Colantonio

Señor Performo’s YouTube Channel: Señor Performo ENG

Scott Moore’s Performance Tour on YouTube

Lifting the lid on performance testing

Thu, 24 Sep 2020 23:38:56 +0100

I joined other performance and testing engineers in this online panel at Testing Talks Online about load testing best practices.

From the Testing Talks Online site:

Performance testing is an essential part of the quality assurance function. However, across the industry the great majority of companies still struggle to master and execute a successful performance testing strategy that produces measurable results. In this panel discussion, Cameron Bradley will host some of the best performance engineers in the world. The panel will discuss how we as quality assurance experts and performance engineers can better understand performance testing, the value of performance testing and how world leaders are approaching and succeeding with performance testing as part of the delivery life cycle.

How to Write a Load Test Plan

Mon, 21 Sep 2020 16:33:23 +0100

Originally posted here.

People often start load testing by creating a script in their favorite load testing tool, but good load testing starts before that. Planning a load test may seem like a tedious exercise, but it is essential to ensuring the success of your testing.

A test plan should answer why, what, who, when, and how the testing will be carried out.

A good test plan, particularly for load testing, includes the following components:

Requirements
Scope
Entry criteria
Workload modeling
Server monitoring
Test doubltes
Test Scenarios

We’ll dive into each of these, but first, a note on the format of a test plan.

Test Plan Format

Test plans have been traditionally presented as a document, but there’s no hard-and-fast rule. Instead, the length of a test plan, and the level of detail it goes into, should be tailored for the complexity of the project and the backgrounds of the stakeholders.

Teams using Agile methodologies may not have a formal test plan at all, but I would argue that all teams should still have some sort of plan, whether it’s a Word document or a list of bullet points. Written plans clarify and communicate intent, and testing can be an expensive and fruitless exercise without a consensus on its intent.

Technical documentation can be difficult to digest and understand, and that’s dangerous when it comes to software development and testing, where it’s imperative that all stakeholders are on the same page about the state of an application.

Paul McLean, a performance engineer, has a creative solution for this. He creates a companion video for every technical report, giving stakeholders the option to read the written report for details but watch the video for a more high-level and nuanced explanation.

Here’s an example of a companion video he created for a test plan:

Requirements

Everything starts with requirements. Until requirements are identified, load testing can only ever be aimless and explanatory, and any performance bottlenecks spotted will be incidental.

Requirements inform every step of the load testing process. Why are we doing load testing? What exactly do we want to test? How will we know when a test has passed or failed? How will we know if application performance is good enough to go into production? What does “good enough” mean?

This is such an important part of the planning process that I felt it deserved its own blog post. See it here: Start with Why: How to Write Requirements for API Load Testing. Even though I specifically mention API load testing, the principles apply to load testing in general.

Scope

At Flood, we use a product development methodology called Shape Up to keep us on track as we build new features. Projects in Shape Up are called “shaped work”, and one of the basic properties of shaped work is that it is bounded.

…shaped work [also] indicates what not to do. It tells the team where to stop. There’s a specific appetite—the amount of time the team is allowed to spend on the project. Completing the project within that fixed amount of time requires limiting the scope and leaving specific things out.- Ryan Singer, Shape Up: Stop Running in Circles and Ship Work that Matters

Defining scope is setting boundaries. Business priorities need to be weighed against the resource limitations (number of people available to do the work and time available) in order for testing to deliver maximum value.

Some considerations for scope include:

Specific features or key transactions to be tested
Types of tests included (component test vs end-to-end test)
Test scenarios (peak load test vs disaster recovery)
Applications included in testing

As with most things in the planning phase, scope is something that can change during the test when unexpected circumstances arise or when priorities change. But it’s still a good practice to define the scope at the beginning, and update it as it changes.

It’s important, however, to prune with intention. By keeping our objectives in mind, we can keep our work in line with our intentions.

Entry criteria

Occasionally, you may be asked to load test an application that isn’t ready to be load tested. This happens more often than you think. While it’s true that Performance Testing is something that should be baked into all software development from the beginning, this doesn’t hold true for Load Testing. Load testing is only one activity that falls underneath the broader umbrella of performance testing.

Entry criteria are conditions that you need to be fulfilled before the testing actually begins. It’s a good idea to have these conditions communicated beforehand so that everyone is clear on what needs to be set up before you can do your job.

Functional testing: Does the application work?

Load testing cannot realistically be carried out until at least the core functionality has been tested and high-severity defects have been fixed. Depending on the kind of load test you want to execute, you may also want to specify that UAT has been executed, as there’s no point doing an end-to-end load test with 1000 users if it doesn’t work for one user.

Load Testing Environment: Is it production-like?

Nonfunctional testing has stricter requirements for an environment than does functional testing, and you may have to champion this cause. For load testing, it is not enough to have an application staging environment that is a virtual machine that is a quarter of the size of the production environment. It’s important to get as close to a production-like environment as possible in terms of capacity (memory, CPU), codebase (the actual build that will be deployed), and integrations with other environments or servers (if within test scope).

Load testing is not linear: a response time of 5 seconds on a server with half the capacity of the production server does not necessarily equate to a response time of 2.5 seconds in production.

This is also the time to think about your load injectors. Will they be on-premises, or in the cloud? A good entry criterion is the availability of the machines in the right network and with the right tools installed. If you’re using commercial tools, license provisioning should be a criterion. What sort of capacity will your load testing scripts require?

Support: Are there people available with expertise in key components?

Load testing is a team activity. When a load test involves multiple application teams, it’s important to request availability of key persons on those teams during the test. Often as load testers, we are seen as working independently, but the truth couldn’t be farther from that. Load testing is a team sport. We need support from:

business analysts who will be able to tell us how things are expected to work and what the current priorities are

developers whom we can consult when poorly performing code needs to be optimised
functional testers who can show us how the application works
DevOps engineers who can help us provision and monitor servers

and many more!

‍

Test Data: Do you have dummy data for your virtual users to interact with?

Once you have as close a copy of the production environment as possible, keep in mind that it’s still a clean copy, which may not be realistic. If there are databases in production, how much data do they contain? The application server may respond differently when your test database is empty compared to when it must contend with gigabytes of data in the production database.

Using production data for testing purposes can be dangerous, not to mention illegal, in certain circumstances. To avoid this, you can scrub sensitive data or generate your own, either by injecting records into a database or by writing an automation script to create data on the front-end. If you write a script, you may be able to reuse parts of it later for your load test.

Workload Modeling

A workload model is a schema describing the load profile for a given test scenario, and it involves determining what (the key transactions), how much (the load distribution among the transactions) and when (timing of the load) to test.

Workload modeling can be the most difficult part of the testing process because it involves finding out how load test scripting can best mimic what is actually happening in production. It can also be the most critical.

Here’s my account of working on a project in horse racing, where workload modeling was of particular importance: Workload Modeling - Preparing for Large Events like the Melbourne Cup.

‍ When you’re selling bets for the Melbourne Cup, every second matters.

Server monitoring

Executing a load test without monitoring server health is like flying blind. You’ll know when you land safely and you’ll know when you crash, but even if you do crash, you won’t know why—or how you can avoid it next time. Monitoring server health is the black box that will tell you what went wrong.

For load testing, you’ll want to monitor the application servers that you’re testing as well as the load generators that you’re using to run the load tests themselves. That’s right; if you’re not watching carefully, the machines you run the load tests on can be the bottlenecks in themselves, causing unnecessary failures in your load tests.

Joe Colantonio goes over the basics of resource utilization in terms of four main areas of concern:

CPU
Memory
Disk
Network

Here’s another guide I put together on the basic metrics as well as some tools you can use to set up your monitoring.

Regardless of which metrics you identify as key or which tools you use, you’ll want to make sure that they’re all set up to measure server health before you run your load tests.

Test doubles

Part of the strategic planning around load testing is deciding which components need to be tested in conjunction with your stated requirements. Applications can sometimes be complex enough that load testing end-to-end is not feasible due to the number of teams involved or the cost of duplicating application infrastructure.

Keeping it as simple as possible will lead to fewer dependencies, so it’s worth considering creating test doubles to isolate relevant components. Test doubles include stubs, mocks, and full virtual services.

‍

A test double is that part that replaces a complicated component that is not within scope. It’s a “dumber” version that responds to requests enough to allow you to go on with your load testing without actually requiring that component.

Test doubles allow us to abstract out other components we don’t want to test so that we can concentrate on the one that we do. The point of a test double is to remove the variables and noise introduced by other parts of the application so that we can focus on testing how one component responds.

If you are willing to put the time into creating a test double, you can drastically reduce the amount of resources you need to set up an environment and isolate components. Reducing variables in your test allows you to more quickly determine where performance bottlenecks lie.

Test Scenarios

A test scenario is the description of a contained situation or condition under which the application will be tested. A test scenario is usually based on several test cases and includes a plan for how these test cases will be executed.

Choosing your test scenarios means deciding which situation is most likely to yield the data that you require. Employing several different types of scenarios will give you a greater understanding of your application’s capabilities. You should feel free to create your own scenarios that are uniquely tailored to your requirements, but here are some common scenarios to start out with. Take the number of users and durations mentioned as guidelines and not rules.

Here’s what our Flooder amigo, Señor Performo, has to say about test scenarios:

Text version here

Conclusion

Creating a test plan can seem daunting, but the process of writing it serves as a prompt for discussions among your team.

Here’s a pro tip from Scott Moore: Once you come up with a template you’re happy with, save it and reuse it for future projects. Modify it over time based on feedback to see progressive improvement.

Building a set of reusable documents and other deliverables (“templates”) to use in the initial planning phase and final reporting will speed your initial startup time. I recommend working with or learning from others with experience to build out some of these templates.

‍When in doubt, remember that the format of a test plan is not as important as making sure it is meaningful and comprehensible for your team.

‍For more information about the load testing process, check out my book on API load testing.

Pros and Cons of API load testing

Tue, 18 Aug 2020 21:22:49 +0100

Originally posted here.

‍API load testing is one of the most popular ways to run a load test because of the number of open source and commercial load testing tools available on the market that support it. Creating a script to hit your application’s API endpoints is the traditional approach to load testing, but it’s not the only one.

Here are some things to consider when deciding whether or not API load testing is right for your application.

Disadvantages of API load testing

API load testing does not simulate real users interacting with elements of your webpage.

‌In contrast to a customer opening up a browser and filling out forms on your site, an API load testing script will consist only of the underlying requests to the server that are made by clicking on those on-screen elements. Exactly how the customer triggers those requests, and what buttons they’ve pressed, is irrelevant. It’s all about the raw requests.‌

The format of these requests will be dictated by the type of API your application uses, and it may take some technical knowledge or expertise to be able to formulate requests correctly. However, a RESTful API is one of the most common types, and it is considered to be relatively simple to use.

HTTPS requests for amazon.com. This is what an API load testing script would need to replicate.

It doesn’t give you an idea of how user-friendly your application is.

‌API load testing tools don’t give you feedback on how long your pictures took to render on your users’ browsers or whether that “SUBMIT” button is in an obvious spot.

‌It doesn’t measure front-end performance or how quickly pages render in different browsers.

‌While all the resources that the server returns can be downloaded by your tool of choice, there is no browser on which to run them.

‌API load testing doesn’t run client-side scripts.

‌It isn’t what you’re looking for if your application is a single-page web app that relies heavily on JavaScript or AJAX to dynamically populate and update the page. Your load testing tool will download the scripts, but not execute them.‌

‌This includes scripts like those required to trigger Google Analytics, which prompts many an engineer to question the results of an API load test due to the inability to see the traffic come in on Google Analytics. API load testing won’t help you with that. Running browser-level load tests using tools like Flood Element or Selenium may be more useful.

If you’ve decided that API load testing is not the best way to load test your application, you can look at browser-based load testing as an alternative.

Executing a browser-based load testing script using Flood Element. ‌

‌Advantages of API load testing

Now that we’ve talked about what API load testing isn’t good at, let’s talk about what it IS good at.

‌### It allows you to load test specific servers rather than the whole stack.

‌API load testing allows you to tailor your load testing by applying load only on particular servers. This is particularly useful in more complicated applications that involve several components and would require substantial effort to reproduce in a test environment. Using APIs, you can test only the functions you want to test.

It is well-supported. ‌

API load testing has been around for decades. There are lots of robust tools you can choose from, both commercial and open-source, and many of these tools have large communities and extensive documentation around how to script the most common cases. It’s a far cry from the browser-level testing space, which is relatively new and sparsely populated by comparison.

‍ The JMeter site contains a wealth of information, including a User’s Manual ‍

It’s not as resource-intensive as browser-level testing.

You can simulate more users with API load testing than with browser-level load testing. Since most interactions on the browser-level get translated to requests anyway, generating load this way is incredibly efficient, allowing you to hit your server with requests while bypassing the UI layer and the resource overhead that that entails.

For example, at Flood we’ve baselined some of the tools that we support to see how many users we can run on an AWS m5.xlarge instance:

Selenium: 5 users Flood Element: 20-40 users JMeter: 1000 users Gatling: 1000 users

‌You’ll see that the browser-level test tools, Selenium and Flood Element, can run significantly fewer users on the same sized node as can the protocol-level tools, JMeter and Gatling. This isn’t due to an inefficiency in the tools but rather to the fact that browser-level tools inherently need more CPU and memory to start separate instances of browsers per user and render pages graphically. You can run more users per node (or machine) by carrying out API load testing with protocol-level tools.

It’s cheaper.

‌The efficiency in resource utilization translates directly into cost savings because every node that you don’t need to execute your tests on is one fewer node that you’ll have to pay to provision (whether on premises or in the cloud).

‌For this reason, API load testing is one of the most cost-efficient ways you can get started with load testing, allowing you to scale up your load relatively cheaply while getting immediate results.

How to get started with API load testing

API load testing isn’t for every application, but depending on your test scenario, it may be the easiest way to test application performance.

If you’re new to non-functional testing or API load testing in particular and would like some step-by-step tutorials on how to get started, check out my free ebook, API Load Testing: A Beginner’s Guide. It takes you through the basics of API load testing, including planning your first load test, selecting an open source tool, scripting, executing a test on the cloud, analyzing results, and even integrating it into your CI/CD pipelines.

Performance testing the Dutch coronavirus hotline

Thu, 13 Aug 2020 22:18:04 +0100

Read the original Dutch version here. This English version was first posted here.

On June 1st, 2020, the National Institute of Health and Environment (RIVM) announced that everyone with certain symptoms could call the hotline 0800-1202 to arrange a free COVID-19 test. The promise: call today, get tested tomorrow, and receive the results the day after that. The reality, though, did not live up to expectations.

The opening day of the hotline was chaotic, and many Dutch people reported connection problems due to a system overload. Yet this could have been prevented with a better understanding of how human psychology can affect application performance.

The situation

According to Dr. Ann Vossen, chairperson of the task force charged with scaling up mass testing in the Netherlands, the country was ready to process 30,000 tests per day. This number corresponds to the national capacity of all the municipal health clinics (GGDs) in the Netherlands. Despite this claim, though, only 1,146 people were actually tested on the first day of testing (the day after the opening of the hotline). Perhaps more importantly, only 5,748 appointments were set on June 1st. The situation was no doubt exacerbated by the overwhelming 323,000 people who called the hotline on its first day, prompting even the telephone operators to admit that they’d had troubles logging into the appointment system.

Bron: From @ggdghornl, Twitter

[“We see that there are many calls going to 0800-1202. You can call for an appointment until 8pm tonight. Please try your call later to avoid long wait times.”]

On Twitter, users shared horror stories: one claimed to have waited on the line for 6 hours, only to be told that the system was down and no appointments could be made; another complained about getting disconnected after getting in touch with an operator. Yet another posted a video of calling the hotline and a recorded message saying “Al onze medewekers zijn op dit moment in gesprek. We zullen u zo snel mogelijk te woord staan. [All of our operators are currently on other calls. We will answer you as soon as possible].”

The second day was somewhat better: there were 11,000 appointments made, partly because of the lower amount of phone calls and partly because of some fixes of technical problems. In the first week, the number of appointments increased to 50,000 (about 7,142 per day), which was still significantly lower than expected.

There are some lessons that we can learn from this case study.

Source: Coronavirus test procedure from GGD Hollands Midden

English translation:

Do you have these (mild) symptoms?:

cough

blocked or runny nose

fever

loss of smell or taste

Step 1: Make an appointment

Call the national number 0800-1202

Have your BSN with you when you call

Make an appointment to get tested at one of our locations

Step 2: Test

Go to the selected location at the scheduled time

You will be tested at the test location

Step 3: Results and eventual start of source and contact research

You will be called with your results within 48 hours

If you are infected with the coronavirus, the GGD will call you for source and contact research *If you show more serious symptoms, or if you fall within a high-risk group, report to your doctor or to the emergency help line.

The effect of fear on performance

This coronavirus has already shown us a side of humanity that we’d perhaps rather not see: fights over toilet paper, chaos on the stock market due to panic, and sabotage of 5G cell towers based on a belief that they caused COVID-19. As much as we’d like to believe it, we’re not always the most rational beings. We all have the tendency to behave unpredictably, especially when we’re afraid.

It’s no wonder that the coronavirus testing line was so welcome. The fact that there were 323,000 callers on the first day is even more remarkable when we remember that majority of Dutch residents had been in self-isolation for months at that point, which lowered their chances of contracting the virus. In theory, the fear that drove these callers might have been somewhat irrational, but not unpredictable.

In English, we call this FOMO (Fear of Missing Out). We’ve already clearly seen how FOMO can drive users en masse to an application, with effects on its performance. In this case, people were afraid that coronavirus test shortages would mean there weren’t enough tests to go around, and they wanted to be first in line to get tested.

Another problem was that one in four callers just wanted general information about COVID-19, despite government pleas to keep the hotline free for coronavirus test appointments only. It didn’t help that the hotline opened on a Dutch national holiday, White Monday, and more people were home than usual for a Monday. The high number of unrelated calls was unexpected, but nevertheless factored into system overload.

What could we have done to improve this system?

Improvements to the corona hotline

Better estimation

As difficult as it is to predict FOMO’s effects, we can still try. We can make educated guesses based on related statistics.

Just one month before the coronavirus test hotline was opened, Jaap van Dissel, the head of RIVM, estimated that there were up to 700,000 Dutch residents infected by the coronavirus. Why, then, was it such a surprise that more than 300,000 of them had called? Furthermore, the RIVM had encouraged people with a broad list of symptoms (such as colds, coughs, or fever) to call, and it’s reasonable to assume that some people with conditions other than COVID-19 might also have called.

Even if we assume that 20% of the 700,000 people with the novel coronavirus did not have symptoms (according to WHO research), we still arrive at a predicted 140,000 calls, which is still 25 times what was the coronavirus testing system was able to process on the first day.

If we have an educated estimate, we can start testing.

‍

Injecting load

When a resident calls the hotline, the call is routed to one of several operators working from home. This is often done with a telephony server that forwards calls based on some type of round-robin system. We can execute load tests on this server to determine how many phone calls a server can process and route successfully, with minimal delay and without message queueing.

Then, the operator enters personal information from the caller into the appointment system, including their government identification number (BSN) and address. We can surmise that the system checks the identity to verify the validity of the BSN, and then sends some details to a database. The closest municipal clinic to the caller can then use the data to process and schedule an appointment. We can also inject load at this stage by simulating the calls that the appointment system creates with an API load testing tool like JMeter or Gatling.

Using different test scenarios

With such a public announcement of the hotline’s opening, a spike test before release could have yielded some useful information. In a spike test, we simulate a sharp increase in users over a short period of time. For example, we could have simulated an increase in users from 0 to 700,000 on the telephony system over 10 minutes to see how the application would have handled that.

Soak tests could also have been useful in this case. The hotline’s opening hours were from 8 am to 8 pm, so it was open for 12 hours in total. With soak tests, we can generate load on an application for an extended amount of time. Soak tests can reveal memory leaks or other bottlenecks in the processing of data that might occur after hours.

These are ways we could have tested the system as it is, but are there improvements that we could have made in the system to make it more resilient to reduce the FOMO factor?

Using automation to lessen FOMO risk

The system had some manual parts of the process that may have exacerbated load issues. Here’s how this system could have been automated to reduce bottlenecks.

Interactive Voice Response

Telephony servers commonly use Interactive Voice Response (IVR), an automated menu system for voice calls that uses either voice recognition or keypress tones to navigate a user through a series of choices. We’re all familiar with it: “For Dutch, press 1. For English, press 2…” Such a system might have helped reduce the overall load by quickly filtering out callers in the 25% who weren’t interested in creating an appointment. “If you’d like to book an appointment for a coronavirus test, press 1. Otherwise, please call the information hotline at….”

Here’s what an IVR system could look like:

From: Vanguard Networks ‍ Load testing IVR systems can be difficult, and it usually requires the use of highly specialized commercial tools. However, introducing a new tool may also increase the time required for load testing.

Building a web app

Another approach would be to implement a web app to verify identity and capture personal information quickly. In the Netherlands, we already have DigiD, an identity verification system that is heavily used for governmental services. Mijn Overheid, which is a central government portal accessible to every Dutch resident, already interfaces with DigiD as the sole method of logging in. Reusing these existing services could have saved a lot of the work on a web app— not to mention reducing the bottlenecks inherent in a more manual process like a telephone hotline.

‍ Source: Mijn Overheid ‍

Testing the system end-to-end

We’ve identified a few ways to load test various components of the coronavirus hotline, both in its current incarnation as well as in hypothetical improved versions. Isolating each component and running load tests at that level can help us resolve performance issues inherent in the component. However, there are still bottlenecks that are only revealed when the integrations between components is tested as well. That’s where end-to-end testing comes in.

For the coronavirus hotline, end-to-end testing means being able to run load tests on the entire process and seeing how data flows from one to the other: the call forwarding from the telephony server, the identity verification system, the appointment system, the database of personal information, the servers of every municipal clinic, and the email/text notifications with appointment details. As modern applications grow more complex, it can be tempting not to run end-to-end tests, but we do so at great risk of functional or nonfunctional issues.

An area of end-to-end testing that is often overlooked is the human factor, which can also be a bottleneck. Had hotline operators been trained to ask early on in the call whether the caller wanted to make an appointment, to filter out unrelated calls? Can the coronavirus testing and test result processing, heavily involving manual work from medical professionals, match the stated requirements (test within 24 hours, result within 48 hours) even when the digital components are performant? Some people waited for an hour in the cities of Eindhoven and Goes due to traffic congestion from cars near the test centers, leading to delays in the testing schedule.

While it can be difficult to load test these logistical systems with automated tools, they may have significant effects on the overall performance of an application and should be considered.

Conclusion

Often, performance issues can lead to a tangible loss in profits or a more intangible loss in reputation. In this case study, poor performance had a direct impact on people’s health. Potentially sick people were unable to make appointments to get tested, and later treated. Delayed tests could have led to an increased infection rate as people waited for a confirmed diagnosis. Hotline operators also admitted that slowness in the system led to them sometimes giving out test results against instructions to wait for trained medical professionals to do so.

‍

Human psychological factors can make a system more complex to load test. According to Andrea Evers (LUCM), a health psychological professor, “Door de uitbraak van corona zijn mensen in een langdurige stresssituatie beland. Onzekerheid, onvoorspelbaarheid en oncontroleerbaarheid maken dat ze de behoefte hebben om zaken juist wel te controleren of voorspelbaar te maken [The coronavirus outbreak has put people in a situation of chronic stress. The uncertainty, unpredictability, and helplessness of the situation cause a need to make things controllable or predictable].”

As performance testers, we can use statistics and a systematic attitude combined with knowledge of human factors in applications to try to predict the unpredictable.

Ask a Flooder 20: How can fear affect application performance?

Thu, 06 Aug 2020 22:43:28 +0100

Originally posted here.

In this video, I talk about the power of fear, and how it can end up affecting performance test results. ‍

Or listen to the audio version here:

Hi everyone, I’m Nicole van der Hoeven, back with another Ask a Flooder, and today I want to talk about the effect of fear on your performance outcomes.

The Dutch “coronalijn”

I currently live in Maastricht, in the Netherlands. At the beginning of this global pandemic, the only people getting tested actively for COVID-19 were either people who presented with serious symptoms of the disease, or health care professionals who were continually exposed to the disease and therefore at a higher risk of contracting it.

Last month, on June 1st, the Dutch government made testing available to everybody. They set up a telephone hotline that any Dutch resident could call to set an appointment to get tested for a COVID-19 test. A task force was dispatched to make sure that they could test up to 30.000 people per day. Yet on the first day of testing, only 1.146 people were tested. So what happened?

Well, it turned out that the bottleneck of the entire process wasn’t getting enough tests, registering the tests, or even processing the test results. The bottleneck actually occurred way before that, when people called up the hotline to get an appointment.

323.000 people called on the first day that the hotline opened. And not even all of them had called to make an appointment. Some of them were just calling for general information on the coronavirus despite the fact that the government made it clear that that’s not what the line was for, and that there were other lines dedicated to information. There were reports on Twitter of people waiting up to 6 hours on the line just to talk to someone who would make an appointment for them. There were calls that got dropped. People said that sometimes it happened in the middle of a call. Even telephone operators that the appointment system was down or slow and unusable for a lot of it.

As a result, only 5,500 people actually had appointments booked on a day where there were 323,000 that called. That’s 1.70%.

What does that have to do with performance?

As performance engineers, we like to believe that everything we do is reasoned. We design tests based on quantitative data. We look at historical trends and metrics to help us decide how to build a workload model. But the truth is that that’s often not enough.

If anything, this pandemic has shown us just how wrong our guesses can be, whether or not they’re informed by facts, especially when it has to do with human behavior. We’ve seen violent fights erupt over toilet paper. We’ve seen peaks and troughs on the stock market that didn’t have anything to do with the underlying companies. We’ve seen arson on 5G cell towers. Because human beings are irrational, and it’s really hard to capture that irrationality in our model for how users are going to behave.

We may never be able to accurate predict how irrationality is going to affect the load on our application in production. But that doesn’t mean that we shouldn’t try. There are still things that we can do to mitigate the risk. I think that any application that is public-facing should be assessed for exposure to the risk of fear or other human psychological factors.

How do we account for fear in our performance testing?

I actually wrote a whole blog post on this particular situation (the Dutch corona line), and how it might have been possible to predict (maybe not with 100% accuracy but with reasonable accuracy) how many people would have called the hotline.

For example, we can look at statistics like the of the Netherlands or how many people normally get flu-like symptoms or at least report them at this time of the year. These are statistics that are readily available from the Department of Health, and we could have used them to inform our decisions on the workload model for such an application. We may not have gotten it exactly, but we would have at least bin in the same ballpark.

If we have a good enough idea about the workload that our application would have to handle when it’s influenced by fear, we can start running load tests around that. Another good open-ended kind of test to run, especially for applications that are particularly susceptible to fear, is the stress test. In a stress test, you push your application to its limits and you find out what they are. You could, for instance, start with your peak work load, and then add a certain number of users every 30 minutes and see how much your application can actually handle. You can do the same with soak tests, where you’re exposing your application to the same amount of load for an extended period of time. Both of those kind of exploratory tests help you define the limits of your application and how it would respond if fear plays a role in the workload model in production.

If we overlook this critical human element in our test design, then we leave ourselves, and our applications, vulnerable to performance degradation and, perhaps, outages.

Till next time, happy flooding!

Read this blog post for more about the coronavirus hotline.

Ask a Flooder 18: How do I load test video streaming with JMeter?

Wed, 22 Jul 2020 22:49:15 +0100

Originally posted here.

In this video, I answer the question: “How do I load test video streaming with JMeter?”

Or listen to the audio version here:

HTTP Live Streaming (HLS) and performance

One of the most popular protocols for video streaming is HTTP Live Streaming, or HLS, and that’s what I’m going to show you today. An HLS video ends with the [file extension .m3u8‍](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8). According to the HLS protocol, when a video is streamed, the first request sent must be a manifest. A manifest is sometimes called a playlist, and it is a list of all the chunks or parts that make up the video. Then, each chunk is downloaded.

To assess the performance of a video stream, we still want the response time of the manifest, the chunks, and the whole video. However, there are many more metrics that are important in video streaming which aren’t captured by just the response time. For example, how do you determine whether a video was played smoothly or if it stopped to buffer? A video that keeps pausing might just drive a user away, and it’s difficult to tell whether that is happening just from the response time.

How to load test video streaming with JMeter

Downloading and installing the Ubik Video Streaming plugin

Luckily, there’s a plugin for that, created by a company called Ubik Ingénierie. The Ubik Load Pack is a set of enterprise-grade plugins for JMeter. One of those plugins is a Video Streaming Plugin, and it happens to be a really easy way to get started with video streaming. This is a paid plugin, but you can sign up for a free 11-day trial.

Once you do that, you’ll get an email from Ubik with a link to download the bundle. Go to lib/ext and then copy this .jar file into the same directory on your JMeter installation. Start up JMeter, and it’ll ask you for the license file. Browse to the license file in the bundle you downloaded. JMeter will restart.

Using the JMeter plugin in your load testing script

Now let’s start creating the Test Plan. Create a Thread group, and then right click on the thread group, go to Add > Sampler and then select the “ULP Video Streaming sampler”.

Click on the sampler and then paste in the URL to the HLS file that you’d like to test. This should end in a .m3u8. This plugin supports several protocols for video streaming, but we can just leave it at auto and it’ll automatically detect which one it is.

In Play Options, you can choose whether to play the entire video or just a part of it, or even whether you want to include the video subtitles in your test.

This Network Options panel is fantastic because it allows you to limit your network speed. Slow networks, like mobile networks, really affect user experience, so it’s really important to take this into account when running load tests. You might even use a parameter here and do something like throttle to 3G speeds for 80% of your traffic, and leave it on Auto 20% of the time. But I’ll just leave this blank for now.

Next, let’s add a View Results Tree listener. Then, let’s run the script once, just to see if it works.

As you can see, we can see the manifest retrieved first, and then all the chunks. We didn’t get any errors, so let’s run this as a load test. We can disable this listener because it just consumes extra resources on our load generator.

Ramping up your load test on Flood

To ramp this up into a load test, we’ll have to go into Thread Group and change these settings. Ideally, we want to be able to change these values from the Flood UI later so we don’t have to go into the script whenever we change them, so I’ll type in these Flood-specific properties. I’ll leave a link in the description about these properties. I’ll also set it to run Forever so that we can change the duration and the video will just keep playing to fill that duration.

To capture some of Ubik’s cool metrics, let’s add a Simple Data Writer listener here and write the results to a CSV file. On Flood, all flood results are stored on /data/flood/results, so we’ll add an output.csv file in that directory that we can download after the test.

So that’s our script! Now we’ll have to include the plugin and related files so that Flood can access them as well.

Let’s go back to the bundle we downloaded and go to the bin folder to look for user.properties. Let’s open that up, and we’ll change the license path to /data/flood/files, which is the directory that Flood saves files we upload to. To preserve this directory structure, we’ll compress just the lib and bin folders.

On Flood, we’ll start a new stream here and upload three things: the jmx script, the license, and the compressed archive of the lib and bin folders from the bundle.

Then we’ll go through the rest of the process here, setting the test parameters. Time to launch the test!

Analyzing video streaming load test results

Flood shows you results of your load test while it’s running as well as after the test. You can see metrics like concurrency, network throughput, transaction rate, and response time for the test. You can also see these metrics for each individual transaction, including the video or sampler as a whole, the manifest, and the chunks.

To see the CSV that our script generated, we can download the Archived Results from a flood.

In JMeter, click on Tools > Generate Report and fill in the fields, selecting the output.csv from the flood’s Archived Results, JMeter’s user.properties file, and then an output directory for the report.

Ubik’s plugin creates a few custom graphs in the HTML report:

Buffer fill in Millis
Lag ratio in percentage
Lag ratio without buffer fill in percentage
Lag time in Millis

Conclusion

Using Ubik’s Video Streaming plugin is an easy and thorough way to load test video streaming with JMeter, and I highly recommend it. I used an HLS video to demonstrate it, but this plugin supports several protocols:

HLS
Mpeg DASH
Smooth
Adobe HDS

For more information, read Ubik’s tutorial on how to use their plugin with Flood here.

Performance Testing FOMO

Wed, 22 Jul 2020 21:30:43 +0100

On this episode of the TestGuild performance podcast, Joe Colantonio interviews me about how human factors are often not taken into consideration when designing load tests– a mistake that can have serious consequences.

Here’s the plain audio version:

Performanceverbeteringen voor de coronalijn: de kracht van angst

Sat, 04 Jul 2020 21:35:43 +0100

Originally published on Medium. English version here.

Op 1 juni heeft de landelijke GGD Ghor aangekondigd dat iedereen met klachten een coronatest zou kunnen aanvragen door de afsprakenlijn (0800–1202) te bellen. De belofte: vandaag het telefoongesprek, morgen de coronatest en over twee dagen de uitslag. Echter was het in werkelijkheid niet zo makkelijk. Op de eerste dag was er chaos en veel Nederlanders meldden verbindingsproblemen vanwege drukte op de overbelaste systemen.

Dit had voorkomen kunnen worden door een beter begrip te hebben over de manier waarop de menselijke psychologie de performance van een systeem kan beïnvloeden.

Bron: @ggdghornl, Twitter

De situatie

“Voor die 30.000 tests per dag zijn we nu eindelijk klaar”, zei arts Ann Vossen op 31 mei. Dit aantal verwijst naar de landelijke capaciteit van alle GGD’s. In plaats daarvan werden volgens het AD maar 1146 mensen getest en, misschien belangrijker, een kleine 5478 afspraken gemaakt. De situatie werd erger gemaakt door de 323.000 mensen die de afspraaklijn op de eerste dag hebben gebeld. Sommige medewerkers konden zelfs niet in de systemen inloggen.

Op Twitter vertelden gebruikers enkele horrorverhalen: één beweerde 6 uur lang te hebben gewacht, alleen om een melding te horen dat het systeem geen afspraken kon aanmaken; een andere klaagde over verbroken verbindingen, zelfs toen een medewerker al had geantwoord, en nog een andere een video had gepost van een gesprek dat beëindigd was zonder melding. Ik heb zelf ook de coronalijn gebeld, en nieuwe problemen gevonden.

Gelukkig was de tweede dag beter: er waren 11 duizend afspraken gemaakt, mede vanwege het lagere aantal telefoontjes en mede vanwege verholpen technische problemen. In de eerste week nam het aantal geteste mensen toe naar 50.000 (gemiddeld 7142 per dag), wat toch nog veel lager zat dan verwacht. Er zijn wat lessen die we kunnen trekken uit dit verhaal.

Bron: GGD Hollands Midden

Het gevolgen van angst op performance

Dit virus heeft ons al een kant van de mensheid getoond die we liever niet willen zien: vechten over wc-papier, chaos op de beurs, en sabotage van zendmasten. Hoewel we rationeel willen zijn, is het onmogelijk te zeggen dat dat altijd het geval is. In werkelijkheid hebben we allemaal de neiging gekke dingen te doen. Vooral als we angst hebben.

Geen wonder dan dat de afspraaklijn zo welkom was. 323.000 telefoontjes werden er op de eerste dag verwerkt, wat bijzonder is als we bedenken dat het grootste deel van Nederland in zelfisolatie was, en dus maar een kleine kans had om het coronavirus opgelopen te hebben. In theorie lijkt dit gedrag irrationeel, maar niet onvoorspelbaar.

In het Engels heet het FOMO: Fear Of Missing Out. We hebben al duidelijk gezien hoe FOMO veel meer gebruikers naar een applicatie kan dwingen, met als gevolg bijwerkingen op de performance. Mensen hadden angst dat er niet genoeg coronatesten waren en wilden de eersten in de rij zijn voordat de voorraad helemaal op was.

Ook was er het probleem dat één op de vier telefoontjes niet over een afspraak ging, maar over algemene informatie over COVID-19. Dit werd verergerd door het feit dat de eerste dag van de coronalijn ook Tweede Pinksterdag was. Het teveel aan telefoontjes over andere zaken dan de coronatest was een bijkomende factor in de systeemoverbelasting.

Wat hadden we kunnen doen om dit systeem te verbeteren?

Verbeteringen

Beter inschatten

Hoewel het moeilijk is om de gevolgen van FOMO precies te kennen, kunnen we het wel inschatten. Volgens RIVM-hoofd Jaap van Dissel, waren er begin mei tot 700.000 mensen in Nederland besmet met het coronavirus, een paar weken voor de coronalijn werd opgestart. Waarom was het dan een verrassing dat meer dan 300.000 mensen hadden gebeld? Ook zijn de cijfers van niet-COVID-gerelateerde klachten daar nog niet bij inbegrepen.

Zelfs als we inschatten dat 20% van die 700.000 mensen geen klachten had, komen we toch op een voorspelling van 140.000 telefoontjes uit, wat nog 25 keer meer is dan het werkelijke aantal gemaakte afspraken op de eerste dag.

Zodra wij aantallen hebben, kunnen we testen.

Uit mijn eigen ervaring, toen ik het coronanummer belde, blijkt dat er een telefonieserver was die de oproepen heeft doorgeschakeld naar verschillende medewerkers (die thuis werkten). In deze stap konden loadtesten worden uitgevoerd om zeker te weten hoeveel gesprekken deze server aankon, en dat het snel genoeg was om geen of weinig message queueing te hebben.

Medewerkers hebben dan vermoedelijk gegevens direct in een afsprakensysteem ingevoerd tijdens de oproep, waarop de gegevens waarschijnlijk naar een backend server en database gingen. Ook hier kunnen we testen via een load testing tool op protocol niveau, zoals JMeter.

Met zo’n publieke startdatum zou ik spike testen hebben gevoerd met tot 700.000 gebruikers (misschien nog meer, als ik wat tijd had). In een spike test simuleren we een snelle toename van gebruikers op een kort tijdstip, bijvoorbeeld 1 tot 700.000 gebruikers in 10 minuten. Ook zouden soak testen heel nuttig zijn geweest, vooral omdat de coronalijn van 08.00 tot 20.00 was geopend, dus een stroom van 12 uur. In een soak test voeren we een constante aantal gebruikers voor een langere periode op.

Automatisering gebruiken

Het maken van een afspraak voor de coronatest had veel stappen. Eerst moesten mensen hun BSN en andere details aan de medewerkers geven. Dan moesten medewerkers vragen naar hun klachten. Alleen met de belangrijkste klachten die passen bij het coronavirus konden mensen verdergaan. Uiteindelijk moesten medewerkers adressen controleren en het dichtstbijzijnde testcentrum vinden om de afspraak te maken.

Telefoniesystemen gebruiken soms Interactive Voice Response (IVR) technologie, een systeem van menu’s en keuzen. We hebben dit allemaal gebruikt: “Voor het Nederlands, toets 1. Voor het Engels, toets 2…” Zo’n systeem had het aantal niet afspraak-gerelateerde gesprekken kunnen verlagen met 25% (één op vier klanten belden niet voor een afspraak), en op een geautomatiseerde manier.

Hier is een voorbeeld van een IVR-systeem:

Bron: Vanguard Networks

Op het niveau van het IVR-systeem kan het wel moeilijk zijn om load te simuleren. Empirix Hammer and Cyara zijn twee zeer gespecialiseerde tools die dit kunnen doen.

Nog een optie voor de implementatie van een afsprakensysteem is een web app, met verplichte velden voor BSN, adressen, enz, en met een verbinding met DigiD, een identiteitsverificatiesysteem. Mijn Overheid bestaat al, en ook met DigiD integratie. Er had een soort van beveiligde formulier kunnen worden gebruikt om klantengegevens snel op te nemen.

Bron: Mijn Overheid

End-to-end testing uitvoeren

Vaak spreken we over de performance van één applicatieserver, maar in praktijk zijn er meerdere componenten. Als best practice is het goed om de performance van elk onderdeel van het hele systeem te testen, en dan ook de hele keten.

Voor een afsprakenlijn zou het redelijk zijn geweest om het hele proces te testen: het oproepdoorschakelingssysteem, het afspraaksysteem, de database met gegevens en datums, de servers van elke GGD in het land, en het sturen van email en SMS-berichten met afspraak datums. Het is belangrijk om te weten of alle componenten snel samen kunnen werken.

Ook is er de performance van niet-computer onderdelen. De menselijke component kan ook een bottleneck zijn. Wisten de medewerkers hoe snel ze moesten vragen of een klant een afspraak wilde maken? Kunnen de testen worden uitgevoerd en de uitslagen worden gegeven, zelfs met een snelstromend afspraaksysteem? Het teveel aan afspraken toegewezen aan GGD’s kan ook files veroorzaken (zoals in Eindhoven en Goes).

Conclusie

In dit geval had slechte performance een invloed niet op de winst maar op de gezondheid. Zieke personen die klachten hadden, konden niet allemaal een afspraak maken om getest te worden. Hoeveel andere mensen hadden ze inmiddels besmet? Slechte performance heeft medewerkers ook gedwongen om soms uitslagen te geven “in verdekte termen”, wat tot een privacyschending kan leiden.

De menselijke component kan een systeem lastiger maken om te testen. Aldus Andrea Evers (LUMC), hoogleraar gezondheidspsychologie:

Door de uitbraak van corona zijn mensen in een langdurige stresssituatie beland. Onzekerheid, onvoorspelbaarheid en oncontroleerbaarheid maken dat ze de behoefte hebben om zaken juist wel te controleren of voorspelbaar te maken.

Als performancetesters kunnen we statistieken en een systematische aanpak combineren met kennis van de menselijke factoren om te plannen voor het onvoorspelbare.

Entrevista con Señor Performo en PerfBytes Español

Mon, 22 Jun 2020 21:56:02 +0100

Aquí está una entrevista que he hecho con Leando Melendez (Señor Performo) en su podcast, PerfBytes Español. Hablamos de performance testing, mis idiomas, y mis consejos para mujeres que quieren entrar en el mundo de tecnología.

Listen to "Entrevista Nicole van der Hoeven" on Spreaker.

Ask a Flooder 17: How do I use unique test data on Element?

Thu, 18 Jun 2020 21:58:49 +0100

Originally posted on Tricentis Flood.

In this video, I talk about how to ensure that Flood Element uses unique test data, using the example of a customer wanting to script unique logins for his application.

Or listen to the audio version instead:

Transcript:

Hi everyone, Nicole van der Hoeven here again back with another Ask a Flooder, and this week’s question is a follow-up question from last week. That user had previously asked about test data in Flood Element. This week he’s asking: “How do I ensure that Flood Element uses unique test data?”

So this user is actually trying to script a login for an application. He has a list of usernames and corresponding passwords that he wants to use. He’s already set both of these up in a CSV file, and he’s also set up the Element script in the same way that I outline in my previous video. However, this user is seeing errors in the test due to the application not allowing a given user account to be used simultaneously in two or more devices. The problem is that the way that he’s set it up is that he has Element running on multiple nodes, and each node has the chance to pick the same user account.

So how do you make sure that it’s a unique 1:1 ratio for an instance of Element and a user account in the CSV file? If you haven’t already watched the previous week’s video, you should pause this, watch that, and then come back here, so it makes a little more sense because the script that I’m using is building on what we already did in the previous week.

Let’s talk about the differences between this script and the previous week’s.

First, I’m declaring a global variable here called globalBrowserID. The idea is to create a unique and globally identifiable string that I can attach to every account in the CSV file.

const globalBrowserID = ENV.FLOOD_GRID_INDEX + '_' + ENV.FLOOD_GRID_NODE_SEQUENCE_ID + '_' + ENV.BROWSER_ID.toString()

The global browser ID consists of three parts:- The first is a grid index, which is a number assigned to every grid, and it’s also globally unique. So if there are three grids being used in the test, then the grid indexes would be 0, 1, and 2.- The second one is a grid node sequence ID, and this is unique within the grid. The node sequence ID is a number that is assigned to every node in a single grid.- The last one is a browser ID. The browser ID is a number assigned to an instance of Element running on a single node.

These are all zero-based, so the very first instance of Element running on the very first node and on the very first grid in the entire test would be 0_0_0.

We’re going to have to modify our CSV file to make use of this browser ID. So whereas the user previously had username and password, I’ve added an ID column. From then, these will have to be sequential. So 0_0_0, 0_0_1... And 0_0_1 would be the second instance of Element running still on the first node, on the first grid. And you can make as many of these as you need.

id,username,password
0_0_0,user1,password1
0_0_1,user2,password2
0_0_2,user3,password3
1_0_0,user4,password4
1_0_1,user5,password5

I’ve also modified the interface UserData to describe the fields in the CSV file. So there’s id, username, and password, and I’ve defined them all as strings.

TestData.fromCSV<UserData>('users_multiplegrids.csv')
 .filter((line, index, browserID) => line.id === globalBrowserID)
 .circular()

This line is telling Element to take this data file with this filename and filter it based on the line, the index, and the browserID such that id is equal to the browser ID. So what happens here is that Element will take the global browser ID, which let’s say is 0_0_0 for the very first one. It’ll then take that 0_0_0 and look at the CSV file for anything in the ID field that has the value 0_0_0. And then it will continue to use the rest of that line.

I’ve also made the CSV file circular just because that’s always a good practice to get into.

And as always, I think it’s a great idea to debug this, especially when you’re dealing with multiple grids and multiple nodes on that grid.

step('Print to console', async (browser: Browser, data: UserData) => {
 console.log('global browser ID: ', globalBrowserID, '| user: ', `${data.username}`, ' | password: ', `${data.password}`)    
})

I’ve also added the username and password here, just in case we need to verify the state that that user is in.

So let’s run the test. Since we’re just running this locally, we’ll see that the first value indeed is 0_0_0, and the user we’re selecting is user1 and password1, which lines up with the username and password values corresponding to 0_0_0 in the CSV file.

I hope that helped, and until next time, happy flooding!

Download the full script and the data file to follow along.

Ask a Flooder 16: How do I use test data with Element?

Tue, 09 Jun 2020 22:18:41 +0100

Originally posted on Tricentis Flood.

In this video, I discuss how to make your load testing scripts more realistic by including test data. The load testing tool I use is Flood Element, but you can apply these principles to any tool. You’ll find a copy of the Element script I used below.

‍

And, if you prefer, here’s the audio version:

Transcript

Hi everyone, Nicole van der Hoeven here back with another Ask a Flooder, and this time the question is: How do I use test data with Flood Element?

Now let’s talk about why you might want to use test data. In load testing, it’s important to make your scripts behave as similarly to real users as possible so that the load that you generate is production-like. The problem is that if you send the same data over and over again to the server, that server is probably going to cache that response. The response being cached means that the server will return the response way faster than it normally would if it had been a unique request. So we can use test data, instead of static data hardcoded in your scripts, to make sure that this doesn’t happen.

Using test data can also help us test different responses. For example, let’s say we’re testing a loan application system, where a user inputs the application ID, the server looks it up, and returns the status of that loan application. Now, loan applications might be in different stages of the process. For instance, some might be saved but not submitted, or maybe they’ve been submitted but failed the identity verification check.

If we used the same application ID every time, then the server would always return the same status and the same details. Instead, let’s create a CSV file that we can use as test data. It might look something like this.‍

applicationId
A100001
B100002
C100003
D100004

We can have applicationId as the single header here, and we can populate this with a list of different application IDs in different stages of the process. That way, when we run the script, the server will have to respond authentically to every request without it being cached.

Now to create our script. This is an empty Element script. To start with, make sure you’re importing TestData here in the first line of your script.

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'

The first thing we’re going to need to do is to create an interface for the CSV file. Let’s call it AppData. Within this interface, we’ll need to define all the fields that we intend to use in our data file and what type of values we can expect to be in those fields. Looking at our CSV here, we’ve just got the application ID, so I’m just going to copy that, and it’s a mixture of characters and numbers, so I’m going to put that in as a string.

interface AppData {
 applicationId : string    
}

Now we need to tell Element how to find this test data. We’ll say that it’s from the CSV, using the interface we’ve just created. Now we’ll tell Element the filename. In this situation, the CSV file that I’m using is actually in the same directory, so I don’t need to specify a path here, but if you’re putting it somewhere else, then you’ll need to specify the folder and the path to it.

TestData.fromCSV<AppData>('applicationids.csv')

Now let’s use the data in a step. In real life, you might want to use an application ID to type it into a text field, but for demonstration purposes, I’m just going to show you how to output it to the console log. Printing to the console is also a good way to debug your tests, especially if you’re dealing with test data.

We need to let Element know that we want to use AppData. So now, let’s output this to the console. Since we’re just debugging, let’s output the application ID. To actually get the value from the data file, we’ll use this syntax:

step('Print to console', async (browser: Browser, data: AppData) => {
 console.log('application ID: ', `${data.applicationId}`) 
})

‍By default, Element reads the data files sequentially. The problem with that is we only have a handful of application IDs. So if we wanted to run this one script 100 times, we’re going to run out of data. What we can do here is add circular. That way, when the script reaches the very last value of the file, the next value it selects will be the first one, and it never runs out of test data.

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

Another thing you might be wondering is how to make it so that Element chooses a random value every time. To do that, we’ll also have to add shuffle:

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

‍Now let’s try to run the test.

element run test-data.ts

As you can see, it’s been finished. The application ID that was selected was indeed a random one because the first one would have started with A. So, it actually chose the last one randomly.

And that’s it for test data with Element. Now, this user actually had a follow-up question, which was about unique test data. I’m going to answer that in the next video. Until next time, happy Flooding!

Sample script

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'
import * as assert from 'assert'

export const settings: TestSettings = {    
 loopCount: -1,    
 description: 'Test Data',    
 screenshotOnFailure: true,    
 disableCache: true,    
 clearCache: true,    
 clearCookies: true,
}

/*** Test Data * Version: 1.0 */

export default () => {    
 interface AppData {        
 applicationId : string    }    
 TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()    
 .shuffle()    
 step('Print to console', async (browser: Browser, data: AppData) => {        
 console.log('application ID: ', `${data.applicationId}`)    
 })
}

Hybrid load testing: the best of testing using protocols and real browsers

Tue, 02 Jun 2020 22:21:11 +0100

At PerfGuild 2020, I spoke about hybrid load testing and how to mix two different approaches to load testing: protocol-based and browser-based.

Here’s the unlisted video from that presentation:

Load testing on the Google Cloud Platform

Thu, 28 May 2020 22:25:06 +0100

Originally posted here.

We’re excited to welcome another cloud provider to the Flood lineup, giving you even more options for running your load tests. You can now use Flood to run load tests on the Google Cloud Platform (GCP).

Many of you have requested an integration with Google Cloud, citing many of its strengths:

Full integrations with open-source tools like Influx and Redis
World-class security
High-performance global network

Seamless access to other apps in the Google ecosystem, such as Search and Maps

We think that adding support for GCP instances opens up a lot of opportunities for flooders wanting to take advantage of these benefits.

Using Flood with Google Cloud

You’ll notice a new Google Cloud section in the Integrations page while you’re logged into Flood:

You’ll need to add your GCP service account key to allow Flood to access your account on your behalf. Once you’ve done this, you’ll be able to start grids and floods right from the Flood interface you’re already using.

For more details on how to set up Flood to use your GCP account, check out our documentation or follow along with the video below:

Anyone with a Flood subscription, including a free one, can use Google Cloud instances at no extra cost from Flood; however, GCP usage may incur separate costs.

GCP is the third cloud provider that we support at Flood, after Amazon Web Services and Microsoft Azure.

Ask a Flooder 15: How do I install Element?

Tue, 12 May 2020 22:45:18 +0100

Originally posted here.

In this Flood Element tutorial, I demonstrate the steps required to install Flood Element, including dependencies it requires. Flood Element is an open-source browser-based load testing tool created by Tricentis Flood.

Learn more about Flood Element

Installation instructions for NodeJS

Installation instructions for Yarn

Installation instructions for Element

Video: Getting Started with Element

Guides: Using Element with Flood‍

Audio version here:

Transcript

Hi everyone, Nicole van der Hoeven here again, back with another Ask a Flooder, and today’s question is: “How do I install Element?”

Now, I already made a previous video on how to get started with Element, but I didn’t actually go into the details of how to install it, so that’s what I’m going to do in this video today. There are actually a few parts to installing Element, and all of these instructions are going to be on element.flood.io. If you go onto the site and click on Install, you’ll see all of the instructions depending on which operating system you’re on.

First, you need to install NodeJS. This is sort of like how, if you’re familiar with JMeter, you have to install Java before installing JMeter. NodeJS is what Element runs on. Secondly, you’ll need a package manager. You can use NPM or Yarn, but in this video, I’m going to take you through how to do it with Yarn. And thirdly, there’s installing Element itself.

First, download NodeJS. I’m going to leave links in the description to all of the sites that I visit so that you can go through all of the instructions later. On the Node site, download whichever installer suits your operating system. Go through the installation steps, and afterwards, check to make sure that NodeJS is properly installed on your machine. You can do that by going into the terminal and typing:

node -v

Just make sure that the version that’s shown here is the same version that you’ve downloaded.

Secondly, install the latest version of Yarn. On the Yarn site, you’re going to have to choose your operating system and the version. We recommend using the latest stable version. And then go through the instructions for your operating system. After you’ve done that, check your Yarn installation by typing:

yarn --version

And the last step is installing Element. On the Element site, you can find the exact code that you need to type into your terminal. It’s:

yarn global add @flood/element-cli

So you can just copy that into your terminal. As of the time of this recording, the latest version of Element is 1.2.3. Looks like that’s installed. To check the version of Element, type:

element --version

So that’s how to install Element. We’ve gotten a lot of questions recently about Element in particular, so I’m going to answering them in future videos. Hope this helped, and until next time, happy flooding!

On-premise load testing advantages and disadvantages

Thu, 30 Apr 2020 23:06:59 +0100

Originally posted here.

Where you run your load test can affect the results that you get as well as your project’s bottom line. In general, there are two infrastructure options to consider for load testing your apps: on-premise and cloud. Which type of environment is best for load testing both web or mobile apps?

Differences between on-premise and cloud load testing

What is on-premise load testing?

On-premise load testing is running your performance or load testing scripts on load generators you physically control. Typically, these load generators are machines that are either in your office or data center. In on-premise load testing, you are responsible for all aspects of the maintenance of the load generators, including hardware upgrades and software installations such as testing tools, but that also comes with increased control and security.

What is cloud load testing?

In cloud load testing, you run your load tests on a service provider’s machines, effectively outsourcing the provisioning and maintenance of your load generators. Running “on the cloud” means you’re paying someone else to be responsible for these machines. In cloud load testing, you don’t “own” the machines your test is running on; it’s more like renting some time on someone’s data center whenever you need it. Cloud-based infrastructure isn’t necessarily less secure than on-premise infrastructure, but there are certainly more security considerations to keep in mind.

Pros and cons of on-premise load testing

On-premise load testing is no better or worse than cloud load testing; it just depends on what you need. There’s a meme that the cloud is just a fancy term for someone else’s computer— but that’s an oversimplification that glosses over nuances that might make a difference to your bottom line.

Here are some questions to ask yourself when deciding between load testing on the cloud or on-premise.

‍

Test infrastructure can make or break your load tests.

Cloud vs. on-premise load testing: how do you decide?

Why are you running a load test?

If your intention for running load tests is to test your web application’s load balancing, the source of the web traffic, and the location of your environment, may not be as important. If you intend to do stress testing, you may want the scalability of a cloud setup.

Who is the intended audience of your application?‍

Load testing is most accurate when it is as realistic to the production workload and user base as possible. Running a load test on-premises means that the traffic comes from one location, which could be ideal if your application is an internal one.

Do you have the resources in your team/company to maintain load generators?

All machines need to be provisioned, set up, and maintained to function correctly. Depending on how many machines you need, you may even need a team of trained professionals to monitor your environment.

What is your budget?

Both cloud and on-premise load testing have costs. Cloud load testing requires a smaller outlay of cash initially but may cost more to maintain as you continue to pay for the computing time. On-premise load testing requires more money upfront but may cost less to maintain in the long run.

How many virtual users do you want to run?

The number of virtual users you intend to test your application against correlates to the number of load generators you require. In general, the more machines you need, the more expensive it becomes (for both cloud and on-premise load testing), but if you only need a handful of users, perhaps you can find a machine or two on-premise that could do the trick.

Advantages of on-premise load testing

‍

Increased control

Since you own the machines you’re testing on, you can set your environment up with your exact specifications in mind. You’ll be able to make hardware upgrades or update versions of operating systems and load testing tools whenever you want, without needing to liaise with a third-party provider. This option can come in handy if you routinely use non-standard configurations or legacy software. If you’re also doing other types of non-functional testing, like security testing, it may be easier to change the configuration on on-premise machines.

Greater data security

The question of cloud vs. on-premises may be a business decision. Some industries or businesses with stringent compliance and regulatory requirements may have no choice but to run tests on-premise because owning the load generators also means controlling who has access to the data that’s on them. An on-premise environment lets you lock down access if necessary. With cloud load testing, you’d need to have security arrangements with the cloud provider that you choose so that any test data or identifying client information seen or accessed by your test scripts remains confidential.

Lower cost and lower barrier to entry in some situations

If you don’t intend to run too many users of your test script and you already have a spare machine or two at the office, on-premise load testing may be the easiest and most cost-effective way to get started. This way, you avoid having to request the budget for a new tool and needing to train your team on a new interface.

Fewer variations due to latency

Controlling your own load generators means controlling the network they run on. If your corporate network is generally stable, and your test scenarios don’t involve testing from different geographical locations, reported response times (and user experience) more closely reflect server processing time rather than latency and travel time.

Disadvantages of on-premise load testing

‍

Higher cost in some situations

Maintaining your own load generators comes with a higher up-front cost, which may be a showstopper for smaller projects or companies. The more concurrent users you require for your load test, the more machines you need, the more licenses you need for your testing tool (unless it’s open source), and the longer it takes to recover your investment.

Geographical limitations

Since all your machines are in one place, you can’t accurately simulate the user experience for customers who might access your web app from another location. You also can’t fully test the effect of CDNs and other tools that your application might employ to keep response times low. Web apps that can be accessed by anyone on the internet might require performance tests that take into account customers from different regions.

More setup

You need to do all the setup yourself for both hardware and software components. You also need a way to coordinate your test, so that load generators start and end when you want them to. You may want to divert some resources towards environment management.

Less flexibility

With on-premise load generators, you can’t add, remove, or scale up nodes on the fly according to your test— something cloud load testing excels at.

On-premise load testing best practices

The environment you’re testing in is especially important when you’re using your own infrastructure. Here are some best practices for a performance testing strategy that involves on-premise load generation.

Run baseline tests

Treat load testing like a scientific experiment. Before you begin changing variables to improve performance, spend some time figuring out your test configuration so that it’s repeatable. Pay attention to things like workloads, business processes, the number of users you’re running per load generator, test environment configuration values, script pacing and think time set in your testing tool, and test script transactions. Make sure that the resource utilization on your machines remains healthy during a test; otherwise, you’re load testing the load generators themselves rather than your application servers.

Then, run a few tests against your application servers without making any changes. This way, you create a “control group” or control test against which to compare future tests.

This process is called baselining your application performance.

‍ Flood’s Project View lets you track an application’s performance over time and compare recent tests to a baseline ‍

Consider using containers or virtual machine images

Along the lines of reducing variables, it’s especially important to use load generators that are as identical as you can make them. You can manually check that you’re installing the same software and tools on each one, or you can do as cloud providers do and automate this process.

It’s easy to miss software that are necessary for monitoring or the load testing tool you’re using. For example, JMeter and Gatling require certain versions of Java. Test automation or browser-level load testing tools might require web browsers.

You can use containers such as Docker to create a standalone package of everything you need for your test. Alternatively, if you’re using virtual machines, you can also create a full system image that you can use for every virtual machine.

The difference between the two is that a container lets you be more granular about what is included in the package that you then deploy to all your load generators. Virtual machines include hardware components that you can set, whereas containers are software-only, decreasing package sizes.

Using containers or images allows you to keep your load generators as identical as possible (in terms of hardware and software) so that you can assume similar results from each one.

‍‍ ‍

Don’t neglect monitoring

It’s easy to forget that your application servers aren’t the only ones that need monitoring. Your load generators need monitoring as well, so that they don’t end up being the bottleneck in your tests. Here’s a starter list of server metrics that you should consider monitoring.

You might also consider how you can integrate performance management tools you already use into your monitoring strategy.

Spend time on a framework for gathering data

One of the biggest challenges in on-premise load testing is figuring out how to get data from your tests. Each machine runs your test script independently of the others, so at the end of your test, you’re likely to end up with several results files. After you download these off every machine, how do you combine them into a single summary?

Manually collating results is an option, but it’s a tedious one. You can copy all the files into one directory and then concatenate them so that you end up with one giant one. Depending on the size of those results files and how long you ran the test, however, this may not be feasible. For larger files, you might consider aggregating the data and increasing the granularity, so that instead of seeing raw response time for every transaction, you only see the average response time for every 15 seconds. You can use a tool like Logstash for this.

It takes some time and technical expertise to set up a framework for results gathering that works for your purposes, especially if you want to go a step further and have real-time results.

Of course, you can also skip this part by just using the framework we’ve already set up on Flood. Tim Koopmans, our co-founder, created the first prototype of what would become Flood specifically because he got tired of having to set up this framework for every engagement.

How to run an on-premise load test with Flood Agent

‍ The easiest way to get started with an on-premise load test is by using Flood Agent. Flood Agent is an executable that you can install onto a machine that effectively turns it into a full-fledged load generator. Then, you can start a test from the Flood web interface and have your test script start running on your machines.

Agent takes care of a few typical problems we’ve discussed so far:

It uses containers to ensure that load generators are identically provisioned.
It automatically gathers CPU, memory, and network metrics on load generators.
It coordinates the execution of the load test so that you only need to start it from one central place.
It enables real-time monitoring of results while the test is running.
It aggregates and collates all the results for you into a dashboard.

We think Agent addresses pain points that we’ve seen over and over again in our own on-premise performance testing projects as well as in our customers’. Agent is a free download and works with all performance testing tools that Flood supports.

Conclusion

On-premise load testing is still a valid option despite the growth in popularity of cloud-based infrastructure, and it has its advantages and disadvantages. This type of load testing trades a little bit of flexibility for increased data security and control, which is a tradeoff that some companies may be willing to make.

On-premise load testing has traditionally included a lot of tedious setup and maintenance. Still, you can reduce maintenance effort by creating an internal load testing framework upfront that automates coordination, monitoring, and results collection. Using Flood Agent can mitigate time spent on this initial setup.

Ask a Flooder 13: How do you do mobile load testing?

Tue, 28 Apr 2020 23:21:08 +0100

Originally posted here.

How do you load test a native mobile app? I discuss the general approach to mobile load testing, what a device farm is, and why a device farm isn’t necessary for load testing. I also discuss the best mobile load testing tools to use.

Or listen to the audio version here:

Hi everyone, Nicole here again, back with another Ask a Flooder, and today’s question is:

How do you do mobile load testing?

Now, this question came from a customer who was under the misconception that in order to do mobile load testing, they would need to use what we call a device farm. A device farm is a collection of different kinds of mobile phones. You would ideally have the most popular makes of mobile phones (the hardware models) and the operating systems and system versions. You can imagine that this can very easily get a little out of hand to coordinate because of fragmentation. There are just so many devices out on the market, and then there’s the question of how you actually generate the load from that.

You would somehow need some sort of test automation script that was running and then coordinate os that they all run at the same time, and you would use that as a load test. As you can imagine, whether you use real devices or simulators, this kind of thing would get complicated and expensive very quickly. But the good news is that it’s unnecessary if it’s load testing that you want to do. Now I say that because, for functional testing or compatibility testing, you ARE going to want to test your application on a variety of different browsers. But with load testing, you’re really more concerned about the back end part of the equation. You want to generate load on your application servers. In reality, most mobile apps work by sending API requests to that server, usually by some sort of restful API. Luckily, API load testing has been around for a while, so we do have some industry-tested ways to run this.

What are the best tools for API load testing?

Essentially, you’re going to want to test protocol-level tools. That means that you don’t want to be using browser-level tools in which you’re using scripts to drive a full browser. All you really need is something that sends requests on the protocol-level. That’s great because it’s actually way more efficient.

You have two really good open-source options for this, JMeter and Gatling. I’ve already gone a little bit into the differences between these two in my previous video on how you choose a tool for load testing, which I will leave a link to.

And here are some other resources that will help you get started with mobile load testing. I also have another video about how to record network traffic from a native mobile app on JMeter, which sounds a little complicated, but I promise it’s not. In that video, I go over the steps with you, and I’m actually doing it in real-time with you as well. I show you the screen recording from my mobile and from JMeter on my laptop, so you can always pause and watch it at your own pace.

The second thing is that I already wrote a blog post, which is sort of a guide for mobile load testing, which goes into the steps into way more detail than I can here.

And lastly, I have written blog posts about two real-life examples of mobile load testing. The first one was the Iowa Caucus Reporter app. I ran through it from the workload modeling to the scripting and then the actual execution. More recently, I also wrote about the Robinhood app. Be sure to check out the links in the description to find out more.

Until next time, happy flooding!

Browser-based load testing in 2020

Thu, 23 Apr 2020 16:48:09 +0100

Originally posted here.

We first announced our open-source browser-based load testing tool based on Puppeteer, Flood Element, in February 2018. Despite our decision to make it an entirely separate tool from Flood, we’ve enjoyed using it so much as a team that several of us have written about it:

Tim Koopmans: Load testing for DevOps… with Browser Level Users (BLU)

Kevin Dunne: A new approach to load testing with browser level users

me: Prepare for viral videos with Flood Element

Lachie Cox: Why should you load test with real browsers?

our friend Eduardo Riol, as translated by Antonio Jimenez: Puppeteer vs. Selenium for test automation: is a new leader emerging?

Clearly, we’re huge fans of load testing a web application with real browsers. But does the wider software testing industry share that sentiment?

Browser-based load testing in 2020

Browser-level load testing is still pretty new, and there aren’t too many providers offering it right now. Luckily, we just so happen to have access to a database of load tests run on Flood since 2013. Here’s a bar chart visualizing data we’ve collected about the load testing tools flooders have used (we support Selenium, JMeter, Gatling, and Element).

Load tests executed on Flood over time, grouped by load testing tool

Browser-level load testing is increasing in popularity

In 2020, almost 35% of the load tests run on Flood are browser-based load tests.

While the total number of load tests run across all tools increased across the board, the percentage of tests that utilized Selenium and Element, as compared to JMeter and Gatling, has also increased from 2018. Both Selenium and Element are browser-based load testing tools.

By contrast, JMeter, a protocol-level load testing tool which previously enjoyed a 70% share of the traffic on Flood, now accounts for only a bit over 50% of usage. JMeter’s share of the protocol-level pie is higher than Gatling’s, but the percentage of usage for both tools appears to be contracting when compared to that of browser-based tools.

Element use is increasing faster than Selenium use

Despite the fact that Selenium has been around longer, 25% of all floods today are run using Element. This is a significant increase in just two years, and one that bodes well for the future. While we’re still fans of Selenium and still support it, one reason we see for the dominance of Element in today’s market is resource efficiency.

In our baseline tests, we’ve seen that Element can run from 20-50 users per Flood node (an AWS m5.xlarge instance), whereas Selenium can only run up to 5. This leads to some significant cost savings, as running the same number of users with Element requires fewer load generators than with Selenium.

Another reason might be the fact that Element uses Puppeteer in the background. Puppeteer, developed by Google, understandably shines when driving Google Chrome (and recently, Firefox). Puppeteer load testing is growing in popularity due to the wealth of metrics it has access to.

How browser-based load testing works

Traditional load testing is done on the protocol level by sending messages directly to the web server. These messages are sent without simulating the application interface, and the responses that are received from the server are not visualized, either. “Users” in protocol-level load testing scripts are actually threads, with each one sending a list of messages at predefined intervals, and we refer to these users as Protocol-Level Users (PLU). JMeter and Gatling use PLU to generate load.

The screenshot below shows what one request for a single action might look like in a JMeter test plan.

Example of an HTTP request with parameters on JMeter

In browser-based load testing, real browser instances are opened on load generators, and the load testing tool plays the part of a real user in navigating the web application using the GUI. We refer to the virtual users in browser-based load testing as Browser-Level Users (BLU).

Here’s an example of the same action in the JMeter example above, but this time in an Element test script:

step('Step 1', async browser => {
 //Click Start button
 let startButton = await browser.findElement('.btn-default')
 await startButton.click()
})

This version simply describes the button to be found and clicked, whereas the protocol-level version needs to be more precise in what exactly is passed to the server.

Both types of load testing tools can apply the same load on an application (as long as it supports both methods— see the next section). The main difference is in how the load is generated.

A protocol-level script will contain instructions on which messages to send, which protocols to use, which headers to include, and other information that a typical end-user would not see.

A browser-level script will contain instructions on how to interact with the application: which buttons to click, what to type in text fields, which links to follow, and other on-screen elements that an end-user would be familiar with. This leads us to one of the big advantages of browser-based load testing.

Pros and cons of browser-based load testing

Pros

Simple scripting— or no scripting at all

Protocol-level tools require in-depth knowledge about how an application works, what format it expects requests to be in, and how it responds. On the browser-level, you don’t need to know any of that. When scripting a login process, for instance, you just need to tell the script to type a username and password in the right fields and then click Log In. The script doesn’t need to know that the form submits an encrypted request using the Auth0 API (something you’d need to know for a protocol-level script).

In some cases, you might be able to get away with using something like our qTest Explorer plugin to record a business process, generating a script without any scripting at all. Even when you have to write or modify the script, though, it’s a lot less intimidating than most protocol-level scripts.

They’re also easier to troubleshoot. In addition to opening a real browser instance on your computer when you run a script locally, Element can also display this information on the console (shown here on iTerm):

‍ Screenshots upon execution on Element using iTerm

‍This allows you to understand what’s going on in the script and in the application, letting you detect bottlenecks faster.

Reduced test complexity

The streamlined requirements for scripting lead to a straightforward test. There’s no chance of getting request parameters or values wrong.

Ability to test entire stack from the user perspective

Unlike testing APIs, you don’t need to decide which server you want to hit or make sure you have the appropriate requests to hit all the servers. A browser-level test script interacts with page elements like a real user would, so all underlying application servers are tested. Pair this with cloud-based infrastructure on Flood, and you’ve got a compact package for testing your application.

Capable of testing any user behavior

Protocol-level load test tools struggle with some types of applications, such as single-page apps (SPAs), where a lot of the processing is done on the client side. Browser-level load testing tools shine here, because scripted interactions for these apps are no different for SPAs than other web apps.

Record network and user interaction times for front-end optimization

Browser-level tools show you response times from the user’s perspective: how long after a user clicks the log in button does the user see his or her account balance? This metric is a practical one that may differ wildly from the traditional response times that a protocol-level tool reports. Measuring real user experience is a great value add for your test.

Easier to maintain

Lower script complexity means that a browser-based test script is more resilient to changes in the codebase. A change in the underlying requests sent will not require a change in the test script, making it perfect for regression testing.

It’s also easier to pass on to other members of the team due to high readability. Here’s an example of how to select a value from a dropdown menu in Element:

await browser.selectByValue('#challenger_age','34')

It’s easy to see that the script is looking for an element with the ID “challenger_age” and selecting the value “34”. This step could correspond to dozens of requests that need to be scripted if testing on the protocol level, which makes those scripts harder to understand without a proper handover.

Cons

Browser-based load testing tools have a higher CPU and memory footprint than protocol-based tools. Test scripts in JMeter and Gatling send messages to and record responses from a server without really needing to parse responses. By contrast, browser-based tools parse the server response in order to access the Domain Object Model (DOM) elements on the page.

Using real browsers might be closer to the real user experience, but it also takes its toll on node resources. It is possible to run in headless mode, where browsers are started “invisibly” without actually graphically rendering them. However, even then, a browser-level tool can run fewer concurrent users on a node than a protocol-level tool.

Best tools and applications for browser-based load testing

While other commercial tools exist for this, we’re partial to open source tools at Flood.

Selenium

Selenium load testing is appealing because Selenium is already well-known for test automation and functional testing, and a lot of companies already have full test suites in Selenium. The possibility of reusing those resources and just running them at scale as load tests is, by far, the most compelling reason to use Selenium.

Puppeteer

Google’s just getting started with Puppeteer, and the amount of metrics it exposes is exciting: it seems like anything in Chrome Developer Tools can be accessed by Puppeteer. Imagine running a load test on ONE tool that gives you statistics about back-end performance (response times, the effect of bandwidth restrictions, resources returning errors) AND front-end performance (client-side script console logs, render times, time to first paint, etc). It’s hard to imagine a future where this ISN’T a valuable tool in the performance arsenal.

Under Puppeteer, of course, is Element— our load-flavored version of Puppeteer.

Is protocol-based load testing dead?

No. Testing on the protocol-level may have its drawbacks, but it’s popular for a reason. If you’re testing mobile apps or API endpoints, protocol-level testing tools allow you to focus on hitting what you want to test and only what you want to test. Sometimes you don’t want to test the whole stack. It really depends on your test scenarios.

Plus, protocol-level tools tend to be really resource efficient. We’ve been able to easily run about 1000 users per Flood node with JMeter and Gatling, easily outstripping browser-level tools. It makes sense, since JMeter and Gatling don’t need to start browser instances to generate load.

It’s significantly cheaper to generate very high loads with protocol-level tools. This efficiency makes protocol-level tools a good choice for high concurrency load testing and especially stress testing.

What are we doing to support BLU?

Clearly, our usage metrics show that browser-based load testing is on the rise, and we’re planning to ramp up our offerings in this area while also maintaining support for protocol-level tools. Here’s what you can look forward to from us:

New releases
- We just released version 1.2.3 of Element a few weeks ago, which addresses some bugs and improve stability for certain actions.
Dedicated Element team
- We’ve put together a team of developers who will be focusing on improving Element exclusively. Despite Element being an open source tool, we’re putting our own resources behind it to improve it faster.
Partnering with consulting services
- It can be difficult to get started on a new tool, and Element is no exception. While we provide a load testing platform to run Element scripts, some companies want help to create the scripts in the first place. To this end, we’ve partnered with expert consultants who have already been using Element with great success: Foulk Consulting and Planit Testing.

Conclusion

When choosing a performance testing tool, it’s important to consider the advantages and disadvantages of protocol-level and browser-level testing tools. We think both types of tools deserve to be in a solid performance and load testing strategy. They can also be used in tandem in a hybrid load testing approach, improving overall application performance and scalability.

In 2018, we predicted that testing web-based applications with real browsers would increase in popularity, so we’re thrilled to see our theory validated in 2020. Browser-based load testing is increasing in usage, and our open source tool, Element, is leading the charge. It’s exciting to be at the forefront of this revolution, and we intend to increase support for load testing with real browsers even more this year.

Ask a Flooder 12: Puppeteer vs Selenium: which one is better for load testing?

Tue, 21 Apr 2020 17:36:28 +0100

Originally posted here.

When it comes to software testing, the debate of Puppeteer vs Selenium is a hot one. Nicole tackles it here to determine which one is better for load testing in particular. Spoiler: The answer is still “it depends”.

Or catch the audio version here:

Hi everyone, Nicole here again, back for another Ask a Flooder, and today, the question is: “Puppeteer vs. Selenium: which one is better for load testing?”

We’re actually getting this quite a bit now as Puppeteer grows in popularity, but first, let’s discuss the similarities. They’re tools that you can use to drive browsers. The work by identifying and interacting with elements on a website in the same way that a user would. They are also both open-source, and both can be used for load testing. This is a great strategy because instead of writing tests for an automation suite and then tests for load testing, you can just write the one test suite and re-use it, so it saves a lot of resources in the end. To answer this question, I’m going to focus on load testing because that’s what my experience is in.

Difference between Selenium and Puppeteer

Pros and cons of Selenium

Now for the difference. Selenium supports more browsers than does Puppeteer. It supports Chrome, Firefox, IE, Safari, and Opera, so there are a lot more options to choose from depending on your application.

Selenium is also what I would call automation-first. It was built with automation in mind, which means it’s very good at that. It’s very robust, and if you have a complicated test scenario, Selenium is an excellent choice.

It’s also been around for much longer than Puppeteer, which is important because that means it’s had more time to build up a community. There are way more resources and tutorials for Selenium than for Puppeteer.

It’s also, unfortunately, very resource-intensive. That’s not something that you would typically care too much about from a test automation perspective, but when you’re running thousands of users on the cloud, it IS important to keep in mind that Selenium, just because of how it’s built and its complexity, can only run about five users per node. We’ve found that out at Flood, after baselining specifically an AWS m5.xlarge instance. So you can look at the specs of that and compare it with the machines that you’re wanting to run Selenium on, to see how that would equate for you.

Pros and cons of Puppeteer

Puppeteer is a NodeJS library developed by Google, which means it plays very well with Chrome. There are a whole lot of things that you have access to with Puppeteer, and if you’re using it with Chrome, pretty much anything in Chrome, including rendering times, you can see and access with a Puppeteer script. So that’s a huge advantage.

Unfortunately, that also means it supports fewer browsers. Obviously, it supports Chrome very well, and they recently came out with support for Firefox, although they do have plans to incorporate support for more browsers in the future.

One of Puppeteer’s advantages is that we at Flood actually created a tool on top of Puppeteer. It’s also open-source. It’s called Flood Element, and Element is built to be performance-first because obviously, that’s what we had in mind when we were building it. It takes everything that’s good about Puppeteer and makes it really easy to set parameters that you normally would expect to need for load testing, like users and ramp-up.

This also means that it’s way less resource-intensive. I said that with Selenium, you can run up to five users on an AWS m5.xlarge instance, and for the same instance type, you can run, from our experiments, anywhere from 30-50 Element scripts/users. That is a huge saving when you’re running a really large-scale test.

Should you choose Selenium or Puppeteer for load testing?

I would say that if you already have a large Selenium automation suite, if you have a complex test scenario, or if you aren’t running that many users for your load test, then Selenium is a great choice. If you ARE running thousands of users, though, and resource efficiency, and therefore cost efficiency, is really important to you, then choose Element.

My advice is always to try both. If you have the time, do a proof of concept with both of them. That way, your whole team can better understand what the pros and cons are for each one and decide which one is best going forward. Or maybe you don’t even decide on one. At Flood, we support both of them at the same price, so it depends on what you prefer.

Until next time, happy flooding!

Ask a Flooder 11: What are the best JMeter plugins?

Tue, 14 Apr 2020 19:47:53 +0100

Originally posted here.

In this video, I talk about how to install the JMeter Plugin Manager as well as other plugins, what the best JMeter plugins are, and why you need them. JMeter is an open-source load testing tool that is already robust, but there are some fantastic plugins that have been developed for it that you might want to use in your next load test.

Audio version here:

Hi everyone, Nicole here, back again for another Ask a Flooder, and I hope everybody is staying alert, not anxious, during these crazy times. In today’s Ask a Flooder, our customer asks: “What are the best JMeter plugins?”

What is a JMeter plugin?

So first, I want to talk about what a JMeter plugin is and how you can actually install one. Apache JMeter is a really great, robust load testing that is one of my personal favorites. There are still some things that could be done better or could be extended, and the great thing about being an open-source tool is that anybody, pretty much, can just jump in and develop these plugins for JMeter to extend functionality, so that it supports more protocols or to customize load profiles in a way that is easier. We already have a blog post on this, and I will leave a link to that article as well in the description because it goes into a lot more detail. However, I still want to give you my personal top 3 JMeter plugins.

How do you install a JMeter plugin?

You can install a plugin for JMeter in two ways.

One is to download it from JMeter Plugins, which I will give you a link to, and then you paste that into either the lib — it’s usually in the lib folder, but it could be in the ext (extensions) folder. Each plugin is going to differ as to what you need to copy into what folder, but that’s why I prefer the second method, which is the Plugin Manager.

The Plugin Manager is a plugin itself, so you will initially need to do that copy and pasting of the jar file. Once you have the JMeter Plugin Manager installed, downloading and installing a plugin is as simple as choosing which ones you want and then clicking install. JMeter restarts, and that’s pretty much it. So I do recommend that the Plugin Manager be the first plugin that you install. But that’s still not my top 3 favorite - that’s just an optional step.

What are the top 3 JMeter plugins?

Custom Thread Groups

Now for the actual plugins. My #1 favorite is the Custom Thread Groups. See, with JMeter, you get the standard default thread group, and all that does is it asks you how many users you want to run, how many loops, the duration, and the ramp-up. That’s it. So that’s good for a lot of uses, but it doesn’t really help you in cases where maybe you want to plan a spike test, and you want to make sure that all the users have fully ramped up and gotten to a certain point, and then all hit the server at once. Or maybe you want to do a step load. Previously I’ve discussed stress testing and how a step load can be useful for that. In that case, the Thread Group doesn’t quite cut it because it only ramps up a user, and then it continues at the same level of load for the duration of the test.

So, if you want any sort of variation in the middle (increasing users or decreasing), then you’re going to want something else.The Custom Thread Groups plugin has a lot of different types of thread groups. My favorite is the Ultimate One because you can probably do almost anything on that one. It also has the Stepping, Concurrency, Arrivals, and Free-Form Arrivals, so you have quite a few options.

Throughput Shaping Timer

Plugin #2 is the Throughput Shaping Timer. I love it because not a lot of load testing tools, even the proprietary ones, actually have this. This is really handy when you have a project where you’re given a requirement that isn’t in terms of the number of users; it’s in terms of hits per second, or requests per second, or something to do with the actual throughput of the test.

The problem with that is that the throughput tends to change, so you can’t actually take as a given that it’ll remain constant for the duration of the test. Things like response time or latency are going to affect that throughput. For instance, when you have one user or one thread that’s running, and it sends a request and receives the response back in one second, you can’t necessarily say that that’s going to be the same when there are a thousand threads doing the same thing. The server’s under more load, so maybe it’ll take five seconds instead of one second— but that’s already changed your throughput. Before, one thread was doing one request per second, and now it’s doing one request every five seconds.

So it can get a little complicated, and the Throughput Shaping Timer is one of my favorites because it promises to solve all that. The Throughput Shaping Timer dynamically changes the think time, so at the beginning of a test, for instance, when there are no response time issues and requests are being returned really quickly, it might increase the think time so that it slows down your load test. So you’re not going over that throughput that you wanted to maintain. And then later on in the test, when all the users are ramped up, then it’ll decrease the think time that it uses. So it’s great because you just have this consistent throughput throughout your test. It’s a really intelligent way to test.

I do want to say, though, that you do also want to play around with this a bit and make sure you understand how this works. One page could have several requests in it, and you’ll want to take that into consideration when you’re choosing what to set in that timer.

HTTP/2 Request Sampler

My #3 is the HTTP/2 Sampler. If you don’t know what HTTP/2 is, it’s an upgraded or next-generation protocol to the normal HTTP that we already use. The HTTP/2 protocol is supposed to be faster and more efficient, so more and more web apps are switching to it because of that.

The problem is that JMeter’s HTTP request sampler is only really for HTTP sites, so the HTTP/2 sampler is an upgraded version of the HTTP sampler. It’s really easy to use. The experience is pretty much going to be the same for you, but in the background, it uses the faster HTTP/2 protocol rather than the slower one.

My colleague, Jason Rizio, actually wrote a whole blog post about this, so if you want to learn more, I will put a link in the description.

I hope that answered your question, and until next time, happy flooding!

Introducing Flood Agent: standalone infrastructure for load testing

Tue, 07 Apr 2020 19:03:42 +0100

Originally posted here.

Working from home during this pandemic and wondering how to start load tests from the office? Today, we’re officially introducing Flood Agent, a way to turn your own physical or virtual machines into load generators that you can control from anywhere with an internet connection.

TL;DR:

Installing Flood Agent on a machine lets you start load tests on it remotely from Flood—even if it’s within a corporate VPN.
Flood Agent is a simple binary that you can install on Linux, macOS, and Windows machines.
Once you’ve installed Flood Agent, your machines show up on the Flood web app to use as load generators or grids.

Boldly going where no one has gone before

Load testing within your company’s VPN and trying to coordinate it from home is like being on a maintenance pod trying to contact your starship, but a Mazarite ship is broadcasting a jamming signal, so comms are down. That Mazarite ship is your company’s firewall, and it can make testing difficult.

‍

The easiest way around this is to provision load generators in the cloud. However, because your application servers are still behind a firewall, you’d need to convince IT security to whitelist the IP addresses of the load generators. Depending on how tight security policies are, this may not be an option even in this new world of remote working.

Flood Agent is a way to start load tests on Flood from the load generators you already have. It’s a cross-platform agent that you can install on your regular load generators, allowing you to control them on Flood.

‍‍

Flood Agent is kind of like the Borg Collective. Each machine that you install Flood Agent on is assimilated into the Borg. A hive mind (you, on the Flood site) controls all Borgs, coordinating test execution with robotic efficiency.

Check out this video to follow along as I walk you through the process of setting up Flood Agent, or you can keep reading for the text version.

‍

Installing and configuring Flood Agent

Flood Agent is a free binary, and it runs on Linux, Mac, or Windows machines. You can download the appropriate version for your operating system and then follow the installation instructions here for more information:

Getting started on Linux

Getting started on macOS

Getting started on Windows

After installing Flood Agent, you can configure it according to how you want to use it. Start Flood Agent in your terminal and use the configure functionality. Here’s what that command looks like on macOS, but you can follow the links above for other operating systems:

./flood-agent configure

Linking your Flood account

The command above yields a prompt for a token like this:

This token allows Flood to run load tests on the load generator that you’ve installed Flood Agent in. To get the token, go to the API Access page on your Flood account (you already have one, don’t you?) and click REVEAL token.

You should see a string of characters like this:

‍

Copy your token (anything after the = sign) and paste it into the Flood Agent terminal window.

Choosing load testing tools

Next, you can choose which tools you’d like to set up and which version of those tools Flood Agent uses to run your tests.

‍

Starting with JMeter, Flood Agent asks you which of the tools you’d like to configure. You can choose to set up your tools at the beginning so that you don’t have to set them up again, or you can go back to this configuration wizard at any point to add another tool. Either way, you need to provide the tools that you select in one of two ways: via a Docker image supplied by Flood, or via a local installation that you already have.

For this example, I chose to configure JMeter and to use a local installation. Local installations give you the most flexibility because you can use the version of the tool that you prefer, along with any special plugins. However, if you already have Docker installed or don’t want to manually install JMeter separately, you can also use our Docker image for JMeter.

Selecting a local installation prompts you to provide the location of that directory:

‍

For JMeter, this looks something like /Users/nvanderhoeven/jmeter/apache-jmeter-5.1.1.

After you enter this file path, Flood Agent checks to make sure the directory is valid. You can also set the configuration for other tools in this dialog.

After you finish the configuration, the wizard writes to a config.yaml file in your Flood Agent directory. You can also edit this configuration file directly in the future if you prefer.

Verifying Flood Agent configuration

To run a final check on whether Flood Agent has been installed correctly, type ./flood-agent check. You should see a summary of the configuration options you selected as well as some connectivity and validity checks, like this:

‍

These green check marks mean you’re good to go! If you’re running Flood Agent from within a corporate network, though, you may see some errors here.

Troubleshooting network connectivity within a corporate network

If your company already has a web proxy, check out our guide on running floods through this proxy.

Please ensure that the load generator that you’ve installed Flood Agent on has access to the following endpoints:

<https://drain.flood.io>
<https://beacon.flood.io>
<https://vault.flood.io>
<https://flood-archives.s3-accelerate.amazonaws.com>
<https://logs.us-east-1.amazonaws.com>
<https://sns>.*.amazonaws.com
<https://sqs>.*.amazonaws.com

You may need to request these from your IT department.

If you get stuck, please contact us at support@flood.io. We’d love to go over your specific situation and get you up and running.

Once Flood Agent returns a successful check, you can start up your own grid!

Turning your machine into a Flood grid

A grid, in Flood terminology, is a group of machines to be used for load generation. To make your machine a grid, run this command:

./flood-agent --grid askaflooder

where askaflooder is the name of your grid.

‍

Once you see those [agent] ready and awaiting next job messages, you’re ready to run your test!

Running a load test on your own infrastructure

Log into your account on Flood, if you aren’t already, and create a new stream like you normally would, uploading your load testing script and data files. When launching your test, though, select the option for Hosted infrastructure and then choose the grid with Flood Agent running on it.

‍

Go ahead and launch your test. Congratulations, you’re now set up to run a load test on your on-premise load generators!

Giving it a go

Flood Agent is a free download, so you’ll only need to pay when you run load tests on Flood. However, we also have a free tier— perfect for a proof-of-concept while you’re working from home.

If you have any questions about Flood Agent, check out our documentation or email us. We’d be happy to get you started!

Download Flood Agent for free here.

‍

Ask a Flooder 10: What is the difference between load testing and stress testing?

Mon, 06 Apr 2020 20:26:24 +0100

Originally posted here.

What’s the difference between load testing and stress testing? When should you choose to run a load test on your application, and when is a stress test more appropriate? Well, it’s all in the intention.

‍

Hi everyone, Nicole here again, back with another Ask a Flooder, and this time the question is:

What is the difference between load testing and stress testing?

Okay, again, this is a matter of semantics, but I think the difference is in the intention. Load testing and stress testing are both types of back-end performance testing, and I did also make a video on it that I’ll link in the description down below.

Load testing is about seeing how your server reacts to finite load

Now, with load testing, you usually have to start with a specific idea of the load profile that you want to test. You get to this number by looking through your Google Analytics or other analytics on your site to see what load you need to test at, and then the intention is to see how your server responds to that load. So the load is fixed: you know WHAT you want to test, and you just want to see how the server reacts to it. So it’s more finite in scope. In general, it’s also more about how your server reacts to normal or expected load, whether that’s now or in the future.

Stress testing explores your application’s upper limits

Stress testing, on the other hand, is a little bit more open-ended and a little bit more destructive. So instead of starting with how much load you want to put on [your server], in stress testing, what you generally do is you increase the load. You usually use a step load profile: so you start a certain number of users, you wait maybe 30 minutes, and then you increase it again. And you end up with this staircase kind of load profile. What you’re doing there is you just let it increase until your server fails because the intention of stress testing is to get your server to fail. And, once it fails, does it fail gracefully? Does it show an error message? These are really things that test the upper limits of your server, and that’s what stress testing really is.

They can be interchangeable

At Flood, we tend to use the word “load” testing to refer to both kinds of testing because both of them entail applying load on your application server. It’s not really that important to distinguish between one or another; what’s [important] is that your team uses the same definition so that when YOU say “load testing,” they know what you mean.

Until the next time, happy flooding!

Resources

Video on performance testing vs load testing

Real-life stress testing in action: Robinhood app failure

Ask a Flooder 09: What are the best tools for load testing?

Tue, 31 Mar 2020 20:35:36 +0100

Originally posted here.

One of our most commonly-asked questions is: “What are the best tools for load testing?” The answer will depend on a few things: your reason for load testing, the application you want to test, what kind of load testing you want to carry out, your team’s experience with scripting, and how many virtual users you want to run.

Note: In this video, I say that traditional load testing tools can’t simulate requests that are triggered by interactions on the client-side. This isn’t technically true: if you know what the requests are, you can build those requests manually on a protocol-level tool (like JMeter or Gatling). However, this takes more work compared to browser-level tools that just do this natively. ‍

Transcript:

Hi everyone, Nicole back again with another Ask a Flooder, and this time I’m tackling one of the questions we get asked A LOT, which is:

What are the best tools for load testing?

Why open-source load testing tools are awesome

Now, I want to preface this by saying that we really love open-source tools here at Flood, because 1) they’re really pretty full-fledged these days. You don’t have to make any compromises on features, and, in fact, some of these open-source tools have more features than some newer load testing tools just because they’ve been out longer. And secondly, they have large communities. Because they’re open-source, lots of people are using it, and lots of people are developing for it. So there are lots of plugins normally, and they support any kind of protocol that you can think of. And lastly, cost. It’s hard not to mention cost because it can be a really important factor depending on the budget of your project. And if you’re starting out as a load tester, I think that it might even be the best way to start with load testing, because you can take that knowledge with you and apply it to other projects and other companies regardless of the budget that they have.

Questions to ask yourself in choosing a load testing tool

Why are you doing load testing?

I always advocate starting with WHY. Why are you actually load testing? Is there a particular production defect that you noticed that you’re trying to solve? Is there already a problem, or are you just preparing to launch? These are things that are going to inform the type of tool that you use because you might want to make sure that that tool supports the protocols that you need.

What are you load testing?

Secondly, what type of app are you testing? Because if you have a single-page app, where your end-user is interacting with the page and client-side scripts are running to make requests to servers, then that’s something that most traditional load testing tools aren’t going to be able to do.

There are two general types of load testing tools: there’s browser-based, and there’s protocol-based. I’ll leave a link in the description talking about the differences between the two, but in general, if you want to go for browser-level testing, I would suggest Selenium and Element. Now, they’re both very good options. Selenium is probably the more robust in terms of automation of the two, but Element is better one in terms of efficiency because it was built with performance in mind. So Selenium’s automation-first and Element is really performance-first.

Now, for protocol-level tools, you can’t go wrong with either JMeter or Gatling.

What experience does your team have with scripting?

Thirdly, what’s the level of technical expertise that your team has? I think it’s important to be realistic about what your team can do, given the amount of time that they have. So if you have a team that has never done load testing and has never done any scripting before, then I would suggest using our Test Builder or something similar. On Flood, we have a Test Builder that is really just an easy interface that actually runs JMeter under the hood, but it just kind of simplifies JMeter and boils it down to what are the URLs that you want to test. That is a really easy way to get started, and it’s also a great [way to run a] proof of concept. If you have some automation testers on your team that are trying to do the load testing, well, maybe you should stick to tools that they already use. So it might be easier for them to get up and running with either Selenium or Element rather than learning a new tool. And if you have developers on your team, you might consider Ruby JMeter rather than the vanilla JMeter because Ruby JMeter is a lot easier to deal with— because you don’t have to deal with the XML files of a JMeter file (a normal JMX script). You can just plan out your load tests in Ruby. Or, Gatling is also a really good option because it allows you to write scripts in Scala. Scala is really really powerful, and you can create an entire framework with Gatling, which is also something to keep in mind.

How many virtual users do you want to run in your load tests?

And lastly, how many users do you want to simulate? Now protocol-level tools are going to really shine here because Element and Selenium both require a higher usage of resources on your load generators just because of the complexities involved in starting a browser instance, whereas protocol-level tools are just super efficient at sending these requests. So I would suggest that if you’re talking about thousands of users, I would use the protocol-level tools unless there’s a really good reason why that’s not going to give you the results that you’re looking for. Between JMeter and Gatling– well, they’re both excellent tools, but Gatling has been shown to particularly be good for really high levels of usage. But JMeter is also a good alternative, and you can run thousands of users of JMeter without any problem.

Just get started!

As always, I’d suggest just getting started. Pick whatever tool sounds like it might be the best for you, do a proof of concept on that one, and you can always try the other ones as well. On Flood, we don’t restrict you to just using one tool, so you can use one; you can use all four; you can run them at the same time if you want. It all depends on what you need for your testing.

Till next time, happy flooding!

Ask a Flooder 08: Why is concurrent users an ambiguous measure of throughput?

Thu, 26 Mar 2020 20:41:22 +0100

Originally posted here.

What is the meaning of concurrent users in performance testing? You might have seen concurrent users in JMeter or other tools, but what does it mean and how can you use the term properly?

‍

Transcript:

Hi everyone! Nicole here again for another Ask a Flooder, except this time it’s not really a question– it’s just something I want to talk about.

Source: The Flood Twitter account

So I’m going to put the tweet up here, but last month I actually tweeted about something that somebody else posted. Netflix actually reported some of their viewership figures, and one thing that they reported was that 76 million households watched their new show, The Witcher, which is a large amount of people. So I posted something about why concurrent users is actually an ambiguous way to measure throughput, and I want to expound a little bit more on that.

“Concurrent users” in performance testing - limitations

In load testing, the number of concurrent users is a pretty standard measure of throughput, and generally, it means the number of users that are accessing an application at any one point in time. And even on Flood, we use it because it’s a really good rule of thumb. When we want to get a feel for how big a test is, load testers generally ask, “How many users?” However, we should be aware, as load testers, that that doesn’t always give us the full picture.

What type of user is it?

For instance, the type of user that it is can really change the traffic. Sometimes mobile browsers will optimize requests so that they send fewer because they realize that mobile networks are slower.

What is the user doing?

Secondly, what is the user doing? Are they browsing on different tabs all the time? Or are they just refreshing the content on the same tab? Because if they’re just using the same page and fetching it over and over again, then caching could come into play, so that would create a different load on the system.

How long is the user doing it?

And thirdly, how long is the user doing it? Are we talking about a sustained load for an hour, or is it a shorter time period, like a spike test of five minutes where all the users just ram a system? There are things like the number of requests per second that you need to take into account, which is affected by the think time and the delays that you set in your script. Perhaps your script has a user that just sends a request, then waits for a minute, and then sends another request. That’s very different from a user just sending one request after another as fast as possible.

When does the user do it?

Lastly, it’s also important to take into account the seasonality of the request. When does the user do it? If you have a timesheeting application, for instance, then it might only get traffic on Fridays and a little bit on Monday, so you need to specify, when you report these numbers, whether it’s for the peak time period on Friday - you know, around 3 pm or something - or you’re reporting on figures from a Wednesday when there really wouldn’t be that many people accessing it.

In the case of Netflix, what they counted as a view was only a two-minute watch time. So if you’re one of those people who went into The Witcher and didn’t watch past two minutes, that still counted as a view. So the 76 million users is still not completely accurate, because not all of those people would have watched the entire thing.

Conclusion

Concurrent users is a good way in order to get a quick feel of the throughput that a test outputs, but it really needs to be examined in context along with things like requests per second and the network throughput (kbps) in order to get a really good understanding of how much load a test is generating.

Till next time, happy flooding!

Ask a Flooder 07: What is the difference between performance testing and load testing?

Tue, 24 Mar 2020 20:31:06 +0100

Originally posted here.

In a two-for-one Ask a Flooder, I answer two questions: What is the difference between performance testing and load testing and in which situations is performance testing a better option than load testing?

Transcript:

Hi again, it’s Nicole back for another Ask a Flooder. And this time, the question from one of our customers is:

What is the difference between performance testing and load testing?

Now, this is really a matter of semantics, and it kind of depends on whom you ask, but in general, performance is how quickly an application responds. So when you’re doing performance testing, you’re doing one of two things: there’s front-end performance, and then there’s back-end performance.

In front-end performance, you’re looking at things like how quickly does the application render for the customer on the client-side. It’s how fast something renders on the screen, so it’s how page resources can be rearranged or optimized so that the experience is a lot faster for the end customer. So the focus is on the client-side on the end user’s actual device.

On the back-end performance side, you’re looking at the server. How quickly does the server send a response to a request that’s sent by the browser?

So those are really two different things because back-end performance really looks at everything outside - everything beyond - the device of the customer. Both kinds of performance testing are important. A solid, holistic performance testing strategy will incorporate both front-end and back-end performance.

There are different types of back-end performance testing. You might have heard of load testing, stress testing, volume testing. At Flood, we kind of refer to all of those under the bracket of load testing. They do have some semantic differences between them, but in general, we talk about load testing to refer to simulating traffic against an application server to see how that server responds.

Now the type of the testing that you want to do will also affect the kind of performance testing tool that you choose. For instance, a tool like Flood Element can do both front-end and back-end testing, but a tool like JMeter is only going to be useful for the back-end performance testing. So you need to choose your test according to the testing type that you’d like to carry out.

This customer also had a follow-up question:

In which situations is performance testing a better option than load testing?

And one answer to this is when front-end performance is required. So if what you’re after is trying to speed up how quickly your application renders on a mobile device, for instance, you might be better off looking at tools like GTMetrix or WebPageTest in order to measure and improve that side of it.

I really want to stress here that neither front-end performance nor back-end performance is better than the other. If you’re asking me which type of testing you should do, my answer would be both. Till next time, happy flooding!

Ask a Flooder 06: How to record traffic from a mobile app with JMeter

Thu, 19 Mar 2020 20:50:44 +0100

Originally posted here.

How do you record the traffic from your native mobile app using Apache JMeter? In this video, I explain step-by-step how to begin load testing a native mobile app.

‍

You might want to be doing this because you’re load testing a mobile app but either you don’t have access to your developers right now so they can’t help you with how the requests are made or what the requests are, or maybe you just want to verify what you’ve already been told. Either way, I’m going to show you how using my Android device (it’s a Samsung Galaxy Note 10+) and a Macbook Pro 13". However, the same principles should apply to other mobile devices and laptops.

Step 1: Download and install the mobile app

The first thing you’ll want to do is download and install the app. Now, if the app has already been released, then that’s no problem. You just go to the Google Play Store and download it. However, if it hasn’t been released yet and you’ve been given an APK file for the app, then you’ll have to tweak your settings a little bit in order to get your phone to allow you to download it.

And I’m going to do this right with you. So go into Settings: pull down the little notification screen and hit the little gear icon, and you’ll see a bunch of settings there. Scroll down all the way to the bottom until you see About phone. Click that and then click Software Information. You’ll see the Build number halfway through the screen here– and I know this is weird, but click on it seven times. You have to draw your pattern if you have one set up, but you’ll see here that it says Developer Mode has been enabled. That’s all you need to do; from now on you can install any APK, but be careful, because if you don’t trust the source, you know, they could do some damage to your phone.

Step 2: Set up the HTTP(S) Test Script Recorder on JMeter

The second step is setting up an HTTP recorder on JMeter. So let’s head over to our laptop. Make sure you’re connected to a wifi network. Now, this is a completely new JMeter test plan here. So just right-click on Test Plan and then hover over Add, and then you’re going to go down to Non-Test Elements. Select the HTTP(S) Test Script Recorder. You’ll see here that the port it uses is 8888. We’ll need that for later. Now right-click on the Test Script Recorder, hit Add, and then Listener. We’ll add a View Results Tree to be able to capture the results that we need later. Now we’ll click on the Test Plan - right-click - and then we’re going to Add a Thread Group. Now let’s right-click on the Thread Group and go to Add > Logic Controller > Recording Controller. This is where the requests that we’re going to send are going to be recorded, and we’ll see both the requests and the responses in View Results Tree.

Step 3: Set up your mobile to use the HTTP(S) Test Script Recorder as a proxy

Now, the third step is to set up your mobile to use a proxy. So, back on the mobile, go back to Settings and hit Connections. And then click on Wifi. Now verify that the wifi network that you’re connected to on your mobile is the same one as the wifi network that you’re connected to on your laptop. If it’s the same, click the gear icon. And then hit Advanced all the way at the bottom. Now the Proxy settings are on the second option here so hit the down menu here and then hit Manual. Now, for the Proxy host name we’ll have to go back to the computer to see what our local IP address is.

So, in order to do that, you just have to go to System Preferences. I’m doing it from the Apple menu here and from System Preferences, click on Network. And once you’ve selected the wifi network, hit Advanced, and then TCP/IP. So what you’re looking for will be this IPv4 Address. Now you’re going to type that into the Host name field on your mobile. So let me just type that in.

You’ll note that this is actually a local IP address. It’s not your public IP address. Now for our proxy port, we’ll go and look at the laptop to see the Test Script Recorder Settings and it was 8888, so that’s what we’re going to put in here. All right. And then click on Save.

Step 4: Start recording

Step 4 is recording the traffic. Now, go back to your laptop on JMeter and click on the Test Script Recorder. Now we’re going to click on Start. It’ll come up with this popup box here and you can just click OK since we already trust the source. You’ll see this little popup here to confirm that we are recording traffic. Now let’s see if we’ve actually configured it correctly.

On our mobiles, let’s go back and open the app that you want to use. So I’m going to be using the IowaReporterApp. I just did an article on this on The New Stack (link in the description below) about how I would load test the Iowa Caucus Reporter App. So that’s what I’m going to be using for this tutorial as well. So I’m going to open that (“getting ready to caucus”) and I don’t have a valid precinct ID so I’m just going to put flood.io and I’m going to click Login. So it’s telling me that the connection isn’t private. I don’t actually expect this to work, because the app has been taken down and I don’t have valid login credentials, but I’m still just going to go forward. So, Proceed. You might have to do this as well if your certificates haven’t been set up correctly.

Okay, so now we’ve gotten this error here and it says, “Oops, something went wrong”. So it’s handy to know what it looks like on your mobile so that you can go into JMeter and see if that’s what you see as well.

So on JMeter let’s go to the View Results Tree and it looks like Google Play tried to do a few things beforehand but what we’re really looking for is this IDP caucus thing. So, these are the requests that my phone made and these are the responses. So this is where we got the certificate error because it says “certificate unknown”. Now this is the actual request we sent for login. You can see the request here - the full request - and also the full response. Now, just to check we’ll make sure we got the same response in the body here on JMeter that we saw on our mobile phones.

And here it is; it says “Oops!, something went wrong”. You can of course just search for this. So we’ll stop the recording now because we’ve confirmed that we’ve gotten what we wanted. So in the recording controller, we can see the requests that were recorded from our mobile app and the first one looks like it was one to Google so that’s not one we want. If we were doing this again, we could actually go into the Test Script Recorder, go into this Requests Filtering tab, and we could just filter out or exclude the domains that we don’t want to test, like Google. But this looks like the request that we were actually looking for. You’ll see the full request here with all the parameters and the values.

What’s next?

And that’s how you record traffic from a native mobile app on JMeter. From here, you can then play around with the request. Like we saw, there were a lot of dynamic values that were recorded in the request. We would probably want to correlate those and make sure that we’re doing those correctly. But this is a really great starting point for testing a mobile app.

Till the next Ask a Flooder, happy Flooding!

Ask a Flooder 05: How do I get started with Flood Element?

Tue, 17 Mar 2020 20:58:12 +0100

Originally posted here.

On this Ask a Flooder, I talk about getting started with Flood Element. Element is an open-source tool that is a good Selenium alternative for browser-based load testing. It runs Puppeteer under the hood to drive real browsers at scale and can be paired with Flood for cloud load testing.

‍

TRANSCRIPT:

If you’re sold on browser-based load testing and want to get started on Element, I think the easiest way to do that is by going to element.flood.io and clicking on Install on the upper left corner. You can follow those instructions to get started, and you can also click on the links on the left side if you get stuck.

Once Element is installed, open up the directory and look at the examples folder. There you’ll find a lot of sample scripts that we’ve already made up for you. The easiest way to get started is to just modify some of these slightly so that you’re testing your site. For example, you could just change the URL and step names to make it appropriate for what you’re testing.

Then, you can run it locally using the command element run the name of your test script, like test.ts, and then I also like to include the no-headless flag.

element run test.ts --no-headless

The no-headless flag is going to run a real browser, a Chromium browser, on your machine, so that you can see what the script is actually doing. This is a great way to debug and troubleshoot.

Once you’re ready to run your load test on the cloud, you can simply drag and drop your Element script into the Flood stream editor and then start your load test as usual.

Check out the links in the description below for more information about Element, how to get started, and how to run an Element script on Flood.

More about Element

Getting started with Element

Ask a Flooder 04: Why would your load testing tool report higher response times than you expect?

Wed, 11 Mar 2020 21:02:48 +0100

Originally posted here.

In this Ask a Flooder, I tackle a question we get all the time from users regarding results, particularly for cloud load testing. Sometimes you might notice that response times that are reported in by your load testing tool do not match up with the response time you see from your own browser. What causes this, and how can we fix it?

Transcript:

It’s always confusing when your load testing tool reports higher average response times than what you can see while you’re manually accessing your application on your browser. There are many things that might cause this, but one of the most common causes for it is that you’re overloading your node generators.

As a general rule of thumb, you’ll want to keep your CPU utilization at 80% or less throughout the duration of the test. It is normal for it to spike in the beginning, but it should ideally come to a more stable spot below 80%. If your node’s CPU utilization is going over 80% for most of the test, then what that means is that your node is struggling to get those requests out, which may lead to some inaccurate results.

You can confirm whether or not this is the cause of your high response times by checking the logs and checking the resource utilization on your nodes.If you’re running on Flood, you can do this while the flood is running by going over to Grids, selecting the grid, and then scrolling down on the panel on the right to see the CPU and memory as well as the network throughput.

There are a couple of things you can do to fix this. The first one is to increase any sort of delays or think times in your script — and, if you don’t already have any, you should add some. Think times make the script a lot more realistic and a lot closer to what real users would do. If these values are too low, then your load testing tool is just going to run the script as fast as possible, consuming more resources than necessary.

The second thing you can do is reduce the number of users you’re running per node. This will free up more resources per node and make it a lot easier to scale up your load tests more effectively.

Check out this article for more detailed information on resource over-utilization and other possible reasons for differences between response times in a test and on a browser.

FOMO and performance testing: Why Robinhood went down

Mon, 09 Mar 2020 20:29:40 +0100

Originally posted here.

Angry customers flocked to the internet when the Robinhood investment brokerage app went down last Monday and part of Tuesday, leaving them unable to trade stocks on a day where Wall Street reported historic gains. Could this have been prevented?

The answer is yes. Applications should always be assessed for risk and tested accordingly. Risk-based load testing helps unearth problems that would otherwise only surface in production. Testing applications end-to-end as a system instead of just as a collection of isolated components allows us to build software that is more resilient. The problem is that this type of testing can be difficult to carry out. In this article, I’ll go through the entire process.

What is Robinhood?

Robinhood is an app I’ve personally had my eye on since it opened a waiting list about five years ago for Australians wanting to see the American company come to our shores. It doesn’t look like it’s happening now, but if that waiting list still exists, I’m on it.

Robinhood is a brokerage app that takes inspiration from the legend it was named after to “give to the poor,” bring zero-commission trading to its customers. Instead of earning from commissions, Robinhood makes its money from a variety of other methods, including premium platform fees, interest on customers’ uninvested capital, and payments received in exchange for coursing customers’ orders through third-party market-makers. It’s an enticing premise, and one that the company has used in conjunction with mobile apps to appeal to millennials and technophiles.

Source: From Robinhood’s homepage ‍

What caused the Robinhood app to fail?

Unfortunately, its wild success with the tech-savvy crowd also meant that when it went down last Monday morning in the US, its customers took to the internet in droves to draw attention to the failure. In a blog post from Robinhood, the company’s co-founders, Baiju Bhatt and Vladimir Tenev, admitted that the outage was due to “stress on [their] infrastructure— which struggled with unprecedented load.”

The outages you have experienced over the last two days are not acceptable and we want to share an update on the current situation. Our team has spent the last two days evaluating and addressing this issue. We worked as quickly as possible to restore service, but it took us a while. Too long. - Baiju Bhatt and Bladimir Tenev

Bhatt and Tenev went on to explain that the load on their servers had caused a “thundering herd” effect. This effect describes a situation in which a backend server receives a large number of concurrent requests and, instead of different threads processing these requests simultaneously, all threads attempt to process the same request.

If you’ve ever played four-player Overcooked, it’s like having everyone sprint to a fire extinguisher to try to put out a fire from overcooked pasta, but instead, the constant button mashing means you just pass the extinguisher around and yell at each other while the fire rages on. Before you know it, the entire kitchen is on fire.

Source: From Twitter user @csac0425

So the technical cause of the outage was an infrastructure problem.

But I think the REAL cause was FOMO.

What is FOMO?

FOMO is the Fear Of Missing Out, and it’s a phenomenon that’s been particularly exacerbated by social media. Instead of only hearing news from people you interact with in the physical world, it just takes a few seconds to open up Twitter on your smartphone and see what thousands of people are talking about. Twitter will even helpfully tell you what’s “trending” in your country, and other social networks also employ algorithms to determine what messages to show you.

How does FOMO affect application performance?

We can see FOMO in how people have reacted to the coronavirus COVID-19. Despite pleas to leave masks for medical professionals at real risk, people have bought out the stock for these masks despite living in countries with little to no sign of the coronavirus. In Sydney, Australia, residents are stockpiling toilet paper (of all things) in large quantities, clearing out shelves from supermarkets, and getting into knife fights over toilet paper due to their panic at the thought of doing without.

This photo of an empty toilet paper aisle was taken by our Principal Engineer, Lachie Cox, in a supermarket in Sydney last week. ‍ FOMO of this magnitude ultimately also manifests itself in the stock market. The Dow Jones Industrial Average, which represents the performance of the top 30 largest companies in the US, dropped by 3,500 points last week amidst fears that the coronavirus would stall production in countries most affected by COVID-19, leading to lower returns across the board.

Then, an abrupt about-face. The Dow went from recording its biggest ever loss in a single day on Thursday to posting a record gain of 1,200 points on Monday.

Guess who gets to adjust to all this volatility.

Robinhood’s founders reported that the “highly volatile and historic market conditions; record volume; and record sign-ups” contributed to the infrastructure issues they faced. We can only imagine just how many users they saw on their systems during this period.

FOMO creates mass hysteria and panic, both of which cause customers to behave irrationally and likely very differently than previously recorded. So how do you create a workload model that accounts for this volatility? How do you tailor your load tests for FOMO? I’ll run you through the process, using Robinhood as an example.

Does your application require FOMO load testing?

All testing should be risk-based. Start with why: why should you test an application component? Why does your application require this kind of testing or that one?

In Robinhood’s case, there were a few indicators that pointed to the necessity of FOMO testing:

A customer base made up of a younger, tech-savvy generation. ‍This demographic is very likely to take to social media with complaints, making negative publicity a real risk.
Complexity. ‍Robinhood’s systems must receive customers’ orders, forward and place those orders to market makers, and display real-time market data. Robinhood customers can also purchase options on stocks as well as cryptocurrency, further increasing the complexity.
Financial component. ‍Anything that involves taking people’s money should have a high priority for any testing.

When it comes to your money, we know how important it is for you to have answers. - Baiju Bhatt and Bladimir Tenev, co-founders of Robinhood

Record-breaking growth. ‍Robinhood had 1 million users in 2016, 6 million in 2018, and 10 million last December, according to CNBC. Exponential growth brings some major growing pains.
Industry history of expensive failures. ‍Quartz points out that software or infrastructure failures have cost the financial industry millions in recent years.
Recent legal action. The regulatory body FINRA fined Robinhood US$1.25 million just last December for placing orders for customers without looking for the lowest price. While this isn’t a huge fine for a company that was valued at US$7.6 billion last July, any legal action should make a company tread more carefully.

These are solid reasons to warrant load testing Robinhood’s systems beyond just “expected” load.

Planning for FOMO

Once you’ve determined that your application is susceptible to FOMO, the next thing to do is plan how to structure your load tests to include its effect in your test runs.

What type of app is it?

Robinhood consists of three apps: two for mobile (iOS and Android) and one for the web. It is currently only available to US residents, which means that getting mobile apps downloaded is problematic as it would entail changing countries in the app stores to get access. Since I live in the Netherlands and I don’t want to do that, I’ll focus on the web app specifically when scripting the load test, and I’ll assume that the same underlying application servers service both web and mobile apps.

Workload modeling

Business processes to test

Given that the app went down on a day where markets were up, we can assume that most of the people on Robinhood’s apps were doing one of two things:

Signing up
Buying shares

We should ideally test both, but let’s focus on signups here because it’s easier to test without access to the actual app (which requires a US social security number).

Number of virtual users at peak load

Let’s try to come up with a number of users that we can use as Robinhood’s peak hourly load. This task would be easier if Robinhood shared their analytics, but we can still make some educated guesses with what has been made public.

Source: Image from Apptopia

According to Apptopia, Robinhood’s highest number of mobile installations was in 2018, when it saw 2 million downloads of its mobile app in a quarter. This period coincides with when Robinhood released cryptocurrency trading support, which looks like increased signups—in the third quarter of 2019, this figure was about 1.5 million. 2 million quarterly downloads translates to about 666,666 downloads per month. Let’s call that 700,000 and take this number of mobile downloads as a starting figure.

The stock market is only open on the weekdays, so we can divide 700,000 signups a month by 20 days, and we get 35,000 signups a day. The markets are also only open for about 7 hours a day, but the load probably isn’t evenly spread out across the 7 hours— I would expect that people would have more time during lunch to sign up for Robinhood. So let’s say that the majority of those signups would happen within three hours (from 11 am to 2 pm, for example). By dividing 35,000 signups per day by 3 hours, we get about 11,667 users per hour. Let’s round that up to 12,000.

How do we translate this to the number of virtual users we need to run? Let’s think about how long each user stays on the app. The signup process does require approval, so that’s a hard stop— new users won’t be able to sign up and then start trading immediately afterwards. This approval could take up to a day, or up to seven days if documents are required.

Looking at the application form itself, we can see it is relatively standard and requires only information that most people are likely to have handy (social security number and contact details). I timed how long it took me to go through the part of it I could access, and I estimated that 10 minutes would be sufficient time to go through all the tabs.

Source: Robinhood

We want to spread out users evenly across the hour— that is, we don’t want 12,000 users to sign up and leave, all within 10 minutes. We want equal portions of those users to be signing up throughout the entire hour. Dividing 12,000 by 6 (the number of 10-minute intervals in an hour), we get 2,000. 2,000 is the number of threads we need active at any one point of time. Each of those threads signs up one user for 10 minutes, then signs up 5 more users until the hour is up.

So we have our figure: we need 2,000 virtual users per hour to simulate 12,000 signups in an hour.

Test scenarios

In addition to running the standard battery of load tests, testing for FOMO requires exploring the upper limits of the application. Assuming the peak load testing goes well, we can use the 2,000 virtual users per hour figure from our peak load calculations as a starting point for these more destructive tests. To this end, here are some test scenarios I’d consider running if I were testing the Robinhood app:

Soak testing. ‍Soak testing means running typically a little less than the peak load profile, but over a longer amount of time. In Robinhood’s case, perhaps we could run 1,000 virtual users for 8 hours. We would expect that the response times reported during this test would be the same; otherwise, there is likely a performance bottleneck, such as a memory leak.
Stress testing. ‍Stress testing means increasing the number of users on an application at regular increments until the application crashes. We can start with 2,000 virtual users per hour, and then add 500 users every 30 minutes to see how the application handles it. This test helps determine how much room there is for growth.

Source: Stepped load profile for stress testing, generated in JMeter ‍

Resilience testing. ‍Resilience testing involves running the peak load test and then simulating an outage by turning off key components to see how the application behaves. If two servers share the load, for instance, we could turn one off and check to make sure that the user sessions on the terminated node are moved over onto the remaining node. This test shows whether the application recovers gracefully from an unforeseen event.

All these tests help us prepare for the unexpected, and they might have identified the “thundering herd” problem that contributed to Robinhood’s outage.

Scripting for FOMO

Tool selection

To test mobile as well as web apps, I would typically use a protocol-level load testing tool like JMeter to simulate the load. However, I’ve already done that. I also noticed that Robinhood makes use of some dynamic parameters, such as a csrf token:

‍

Source: Chrome Developer Tools on Robinhood ‍ We can usually find the value of this token in the response of the previous page. It is a security feature to allow Robinhood to verify that the same users make all requests in the same session. Tokens can be scripted around in a tool like JMeter, but it can be time-consuming. So for this article, I decided to show another tool that makes this problem go away. I used Flood Element, which is an open-source tool we created based on Puppeteer. One of its advantages is that it runs on the browser level. Instead of diving into HTTP requests and tokens, I just told Element which buttons to click.

I wanted a script that would do the following things:

Navigate to Robinhood’s home page.
Click on “Sign Up”.
Click on the “Get Your Free Stock” button.
Fill out the signup form.

Note that I stopped short of actually submitting the form because I don’t want to create dummy accounts on Robinhood’s database— I just want to show how this flow can be scripted with Element.

To help me identify the elements on the page that I wanted the script to interact with, I used Chrome’s built-in Developer Tools. Specifically, I used the Inspect Element button (in orange below) to click on a field and find where it was in the code.

‍

For instance, to find out how to identify the email field, I right-clicked on the code on the right, hovered over Copy, and then clicked Copy selector.

‍

This gave me the value #signup_email_field, which I then used in the script like this:

//Type Email address
await browser.type(By.id('signup_email_field'),'nicole@flood.io')

Here’s the script that I ended up with:

import { step, TestSettings, Until, By, MouseButtons, Device, Driver, ENV } from '@flood/element'
import * as assert from 'assert'
export const settings: TestSettings = {
 loopCount: -1,
 clearCache: true,
 disableCache: true,
 actionDelay: 8,
 stepDelay: 10,
 screenshotOnFailure: true,
 userAgent: 'flood-element-test',
}
export default () => {
 step('01_Home', async browser => {
 //Navigate to Robinhood homepage
 await browser.visit('<https://robinhood.com>')

 //Validate text
 let validation = By.visibleText('It’s Time to Do Money')
 await browser.wait(Until.elementIsVisible(validation))
 await browser.takeScreenshot()
 })

 step('02_Signup', async browser => {
 //Click "Sign Up"
 let signupBtn = await browser.findElement(By.xpath('//a[@href="<https://signup.robinhood.com>"]'))
 await signupBtn.click()

 //Validate text
 let validation = By.visibleText('Free Stock Waiting For You')
 await browser.wait(Until.elementIsVisible(validation))
 await browser.takeScreenshot()
 })

 step('03_GetFreeStock', async browser => {
 //Click Get Your Free Stock
 let freestockBtn = await browser.findElement(By.xpath('//a[@href="<https://signup.robinhood.com/?">]'))
 await freestockBtn.click()

 //Validate text
 let validation = By.visibleText('Make Your Money Move')
 await browser.wait(Until.elementIsVisible(validation))
 await browser.takeScreenshot()
 })

 step('04_FillForm', async browser => {
 //Type First name
 await browser.type(By.xpath('//input[@name="first_name"]'),'Nicole')

 //Type Last name
 await browser.type(By.xpath('//input[@name="last_name"]'),'van der Hoeven')

 //Type Email address
 await browser.type(By.id('signup_email_field'),'nicole@flood.io')

 //Type password
 await browser.type(By.xpath('//input[@name="password"]'),'demo123456')


 await browser.takeScreenshot()
 })
}

In every step, I also added a takeScreenshot() , which is one of my very favorite features of Element. It’s so useful to be able to save these screenshots and use them to determine exactly what’s happening in the script. Here’s what it looks like running my Element script locally. The automated browser is on the left, and my terminal (iTerm) is on the right, running Element.

‍

Running a FOMO test

Now we’ve got a script that runs locally. How do we run it at scale? Additionally, Robinhood won’t allow users outside the US to sign up. How do we make sure that our virtual users are allowed to sign up?

The answer is a load testing platform like Flood.

With Flood, running our script is a matter of uploading the script and selecting some options.

The Flood test design interface ‍ Flood also lets us choose which regions to generate load from. Below I’ve got it set up to run for an hour from four different US cities, with each one starting 500 users. This configuration gives us our 2,000 virtual user figure.

‍

If this peak load test goes well, we could then go on to run the other test scenarios that we described earlier. Each one puts pressure on application servers in different ways, identifying performance bottlenecks.

Results and reruns

Since Robinhood is a live application, I didn’t run the test— it’s never a good idea to load test a domain you don’t own when you’re talking about thousands of users. However, here’s a shareable link to another small Element test I ran previously.

Real-time dashboards like that one on Flood allow you to get a quick look into how the load test is running and to make any changes if necessary. When testing for FOMO, I think they’re invaluable. For example, during a stress test where users are added at regular intervals, a real-time dashboard would help you identify increases in response time as a result of the increased number of users. It would also let you monitor the test and stop it when necessary.

From the link above, click on the right arrow on the lone transaction.

That will take you to the Transaction Detail page. Remember those screenshots from Element? If you have any in your script, you’ll see them here, arranged according to time.

This lets you troubleshoot issues during FOMO testing and react quickly.

Conclusion

Even with sound load testing strategies in place, it’s challenging to plan for the massive spike in traffic that FOMO can bring, yet it’s necessary to do so. FOMO, as a phenomenon, is only going to increase in magnitude as social networks make it easier to spread fear, uncertainty, and doubt as well as information.

We may not be able to determine precisely what will break, or when, but what we can do is put systems into place to plan for it. We can assess an application’s susceptibility to FOMO and plan for end-to-end performance engineering of the system as a whole. We can start with the assumption that applications will fail and then determine the most likely candidates for that failure. We can routinely expose systems to traffic and circumstances that are extraordinary, and in doing so, we can improve our preparedness for something like FOMO.

FOMO is irrational and unpredictable— but that doesn’t mean it needs to be unexpected.

Load Testing Lessons from the Iowa Caucus

Fri, 06 Mar 2020 21:59:02 +0100

Originally posted here and here.

TL;DR? Here are the highlights in this article:

Shadow, Inc. created a mobile app for the Iowa Caucus in February 2020, and the app failed, causing reported results to be delayed. The app reported different vote totals due to a system integration bug where the output was in a different format than what it needed to be. Here are the steps to load testing a mobile app:

Test planning: Does the app require load testing? If yes, what is the underlying technology? How do we create a workload model?
Scripting: Which mobile load testing tool should we use? How do we record traffic from a mobile device?
Execution: What load test scenarios should we have? How can we scale up the load generated? Conclusion: While the Iowa Reported App failure was not due entirely to load testing, load testing could have helped make it more robust.

A company called Shadow, Inc. made headlines earlier this week when they developed an app to count and report votes during the Iowa Democratic Caucus in the US. This app, called IowaReporterApp, ended up failing on several fronts, resulting in a days-long delay in publishing the total votes, during which manual methods had to be employed.

Source: @ShadowIncHQ on Twitter

This delay led to significant criticism from the software testing industry as well as from the general public. Some developers raised questions about the code quality of the app. The app inaccurately reported voting totals that differed from the Democratic Party’s own manual reports, suggesting an issue with functional testing, particularly system integration testing. Shadow’s refusal to accept the Department of Homeland Security’s help with security testing also raised eyebrows. What was perhaps the most damning, though, was the question of whether such an app was necessary in the first place, when cheaper and more effective solutions existed.

What I’m most interested in, however, are the load issues that the app faced. Let’s rewind a little to before the caucus. How would I have load tested an app like the IowaReporterApp?

Planning a mobile load testing

One of the first things to figure out when planning a load test is whether a load test is even necessary.

Do you even need to load test your mobile app?

To answer that, I had to find out more about the app.

In the caucus, residents of Iowa who registered themselves as Democratic voters gathered to vote on whom they thought should be the party’s candidate for the presidential elections later this year.

There are a few things that I think make this app the perfect candidate for load testing:

It was highly publicized. The caucuses would always have been an event of international interest, but the hype surrounding electronic voting increased media attention on the caucus even further.
It was mission-critical. The precinct chairs would use the app as the primary device for capturing votes.
It needed to handle significant simultaneous load. There were over a thousand precinct caucuses organized, each with a precinct chair that would be using the app.

For these reasons, the answer is a resounding “Yes.”

What type of app is it?

The best way to learn about an app is to ask its developers. In this case, I opted to do a little research on my own by downloading the .apk file that Motherboard (Vice) made available and looking through the code myself.

The app is a mobile-only app for Android and iOS devices, written mostly using Facebook’s React Native package. The primary function of the app was to sent the vote totals to Shadow Inc.’s Google Cloud Functions server.

The technology involved is worth noting because it can help determine which load testing tool to use. Since the Iowa Reporter App, like most mobile apps, sends requests to a RESTful API, JMeter or Gatling, both well-tested, robust, and open-source protocol-level tools with large user bases, were perfect for the job. I decided to go with JMeter, my personal tool of choice.

How do we create a workload model?

Now let’s talk about the workload model, which is a plan for how the load we generate needs to look. The objective here is to simulate the load that matches the expected real-life load as closely as possible.

At this point, I would typically look at historical data to try to see the amount of load and the load profile. If I had metrics on the server traffic after previous releases of the app using something like Google Analytics or an APM tool, I would have used that data as a starting point for the workload model of this release. However, in this case, the app hadn’t been publicly released before, so I needed to make some educated guesses.

According to Wikipedia, there were a total of 1,777 Iowa precinct caucuses, of which 99 were global satellite caucuses set up for Iowa residents temporarily not able to attend the local caucuses.

In a Democratic caucus, the actual voting process generally takes about two hours. In the Iowa caucus, the precinct chairs would have used the app twice - once for each round of voting. Around the end of the two hours, each chair’s app would have sent the main payload, containing total votes for each candidate, to the backend server. All this is done simultaneously across all 1,777 precincts.

At a minimum, there are two main events here:

The login - Precinct chairs needed to log in to the app, which would presumably have happened around the same time across all precincts towards the beginning of the voting process, because the app also contained instructions on how to run a caucus.
The submission - This is the transmission of the total vote counts for each candidate, sent to Google Cloud Functions, and would also have happened around the same time.

It’s possible that the app also sent other requests between these two events, but since most of the app is inaccessible without a valid login, I’ll focus on the initial login request.

In a situation like this where I don’t have previous data to show how spread out those requests was, it’s prudent to err on the side of caution. Assuming the precinct chairs triggered those requests within 10 minutes of each other and rounding up the number of requests for some wiggle room, I estimated that there would have been 1,800 login requests within 10 minutes.

I decided to test this as a gradual ramp-up of users within 10 minutes.

Scripting for mobile

Okay, time to write the script. Since I didn’t have the luxury of being able to ask the developers for the format of the login request, I did the next best thing and tried to find out myself by recording the app traffic in action.

Recording mobile traffic on a laptop with JMeter

Here’s how I was able to see the request from the mobile app:

I downloaded the app to a spare Android mobile by using the APK link above.
I set up an HTTP(S) Test Script Recorder on JMeter on my laptop with the port 8888. I also made sure my laptop was connected to my wifi network.
I set up my mobile to use a proxy using my laptop’s local IP address on the network and port 8888.
Then, I went back to JMeter on my laptop and started the HTTP(S) Test Script Recorder.
Finally, I navigated to the app, typed in flood.io in the Precinct ID field (obviously not a real ID), and clicked the “Click here to log in” button.

Here’s the traffic as seen on JMeter’s View Results Tree listener afterward:

And here’s the recorded request within the recording controller:

Analyzing the request

It looks like at least the initial login request is a GET to an Auth0 service, with several parameters being passed in the body. It also has some authorization headers.

The parameter values also look like they might be dynamic, meaning they change from session to session. At this point, I would typically change some of these from being hardcoded values into dynamic ones so that the application receives a request in the right format, and I could look through Auth0’s API documentation to find out how the authentication message should be formulated.

Since I don’t have a valid login for testing purposes, I skipped this step and just used the raw request for demonstration purposes.

Preparing to run

Think time is a delay between requests meant to simulate the time it takes for a user to think before taking another action. Without think time, JMeter would fire off requests one after another, which often isn’t realistic and drains resources. I don’t want that, so I added a JMeter timer. There are plenty of timers available, but I settled on a Uniform Random Timer.

The CEO of Shadow told Motherboard that many precinct chairs were confused by the login process as they had to put in three strings of numbers in the right sequence. So, I decided to work that into the script with the timer. I set it up with a constant delay of 2 minutes and a random delay of 1 minute so that the delay would vary from 2 minutes to 3 minutes to simulate a precinct chair attempting to log in a few times.

Execution: Running your mobile load test

According to Motherboard, Shadow has already taken the app servers offline. However, it’s not a good idea to run a full load test against an endpoint that you don’t own, so I decided not to run a test.

Usually, though, I’d want to start with a shakeout test of 10 users or so, and then gradually increase the user load until the peak of 1,800 users. I also would have played around with changing the load profile, such as by increasing or decreasing the ramp-up or steady-state durations, experimenting with stress tests and soak tests, and adding traffic from another availability zone to simulate the 99 satellite precincts around the world. Running multiple load generators can quickly get out of hand if you want to use physical machines, so I would run load tests on the cloud with something like Flood. All of these little changes would have given valuable data that would help me gauge the app’s performance in production-like situations.

Conclusion

Building quality software is not a trivial task. There are many moving parts to account for, and critical errors can be introduced in every stage of an app’s development. While load issues were not the main culprit in the failure of the IowaReporterApp, a simple load test like the one I describe here, might have improved app stability and reliability. Even in cases where load testing doesn’t seem necessary, it can still uncover gross issues in configuration, usability, or security for an app developed on a deadline.

How AirTasker uses Ruby JMeter and Flood for their load testing

Mon, 24 Feb 2020 18:23:31 +0100

Originally posted here.

In the recent RubyConf 2020 held in Melbourne, Australia, Nancy Cai from AirTasker talked about how they built a performance testing framework using Ruby JMeter and Flood. Luckily, we had some pleasantly surprised Flooders in attendance to take notes during this fantastic talk!

AirTasker is a platform allowing the outsourcing of everyday tasks. When discussing the reasons that AirTasker decided they wanted to carry out performance and load testing, Nancy told a fascinating story about an overly successful marketing campaign—a good problem to have, but a problem nonetheless. The Sydney Cats and Dogs Home, an animal shelter, teamed up with AirTasker to pay one lucky person $500 to spend the day playing with the resident animals. The prospect of this dream job created an enormous surge in AirTasker’s traffic that caused alarm bells to go off on the underlying application servers, resulting in an end-user experience that was less than desired.

Nancy included this hilarious slide about some common feelings regarding application performance.

Steps for running a performance test, the AirTasker way

To avoid repeating that mistake, Nancy and her team set out to run performance testing by using the following simple steps:

Determine performance criteria.
Configure the test environment to be as production-like as possible.
Plan and design based on user behavior, including preparing test data.
Implement test design and set up the framework (which in AirTasker’s case involved Ruby JMeter as the performance testing tool and Flood as a load testing solution).
Run test.
Analyze and fine-tune results.

Setting up a load testing tool and framework

When it came to selecting the right tools for the job, AirTasker wanted tools that met the following criteria:

Ability to simulate production-like traffic easily
Widespread use in the industry
Ease of creation and maintenance of tests for everyone
Historical results aggregation
Visualization of performance metrics
Integration with CI and monitoring tools

Nancy’s team compared several load testing tools and found they enjoyed using the Apache JMeter tool, which is a popular open source load testing software. However, they quickly ran into an “out of memory” error after attempting to run about 400-800 concurrent users on a single machine. She and her team decided to use a distributed load testing platform to run the load they required to scale up more effectively.

Enter Flood.

Why AirTasker chose Flood

We also chose Flood.io […] because it comes in with a really good open-source gem called Ruby JMeter. It is a Ruby-based DSL for building JMeter test plans. - Nancy Cai, AirTasker

Nancy and her team enjoyed the benefits of being able to write load tests natively in Ruby and then using Flood’s Ruby JMeter gem, which is a Domain-Specific Language (DSL). Flood allowed them to turn their Ruby scripts into .jmx files, which is the native XML format that JMeter uses.

When they were ready to scale up, AirTasker ran the same JMeter script in Flood.

The real-time Flood dashboard allowed them to see that there was a problem on one of the API endpoints, called “Get pet reviews.” Further investigations revealed a performance issue due to a long-running query against a database server in the code, which AirTasker’s developers could fix.

Best practices

Nancy gives some great advice for anyone running a performance test, including:

asking the right questions as a load tester
running baseline tests
being aware of the difference between your test and production environments
keeping in mind the caching and autoscaling mechanisms that may drastically change the results
the differences between stress testing and other types of performance testing

We were so thrilled to hear Nancy’s story on how Flood helped AirTasker run their performance tests. Catch the entirety of Nancy’s talk below, and check out the resource links at the end of the post to follow Nancy’s story.‍

Flood’s Ruby JMeter gem

Nancy’s GitHub repo where you can find a copy of her test scripts

Follow Nancy on Twitter

Follow Nancy on GitHub

Mobile Load Testing on the Cloud

Fri, 21 Feb 2020 18:44:58 +0100

Originally posted here.

What is mobile load testing?

What is the difference between load testing web apps and mobile apps? In general, your goal is the same: to improve app performance by systematically exposing your app to the load you expect, whether it’s on the web or on a mobile device. However, while the purpose may be the same, there a few extra things to consider when it comes to mobile app performance, and it’s easy to make some key mistakes. In this article, I’ll give you a little bit of the theory behind mobile load testing and then show you practical steps to script and execute a load test.

Challenges in mobile load testing

If you have a web app that is mobile first, and your users typically access your site through a browser on their mobile devices as well as on their laptops, you have more choices for optimizing mobile performance. You can use browser-based load testing tools as well as protocol-level load testing tools to generate load.

If you have a native mobile app that you’d like to load test, though, browser-based load testing tools are a little more complicated. One of the most common approaches to mobile load testing is to use some type of device farm. A device farm is a collection of mobile devices and a way to orchestrate and control them so that they access an application simultaneously. Here’s why they’re not the best choice for load testing.

Device fragmentation

One of the biggest challenges is mobile device fragmentation. Not only are there a lot of mobile devices, but there are also many operating systems and versions. Even narrowing the pool to Android and ios leaves us with a large selection. A common mistake in mobile load testing is trying to account for these hundreds of combinations.

Logistics

Another challenge is the logistics of controlling physical devices with different operating systems, which is not trivial. Unless you have a thousand friends that you can rope into doing this at the same time, you’ll need to pay for some sort of service that can orchestrate this for you.

Cost

There’s also the fact that this process is resource-intensive. Even if your device farm consists of emulators on virtual machines rather than physical mobile devices, that still uses up a lot of CPU and memory, and you’d need to spin up dozens or hundreds of machines.

Luckily, this type of mobile load testing is not the only option available to us.

Better strategies for mobile load testing

A better option for back-end performance testing or load testing your mobile application is by using API load testing, which works on the protocol level. To understand this, let’s talk about what aspects of software we are analyzing when we test for “performance.”

We can generally break up performance testing into two types: front-end and back-end. Both types have the goal of making the app respond faster to end-users, but they do so by optimizing different parts of the application.

Front-end performance

Front-end performance is primarily concerned with the UI of the app. A front-end performance tester optimizes page resources (like CSS, images, JS) to render faster on the page. Since a mobile app runs on a user’s smartphone or another mobile device, its front-end performance would also be affected by things we can’t control, such as the operating system and version of the mobile or its physical configuration (CPU and memory). A front-end issue for a particular mobile device type might not necessarily affect performance on another. Device farms can be good at measuring front-end performance.

Back-end performance

In contrast, back-end performance focuses on the underlying mobile application servers. A back-end performance tester optimizes the things behind the interface, and it is affected by things like the size of the responses that a server sends to a user, server configuration, application infrastructure, and how downstream servers talk to each other. These factors may be invisible to a typical end-user. A performance bottleneck in the back-end is likely to affect all users of an app to some degree.

Load testing for back-end performance

Load testing is about determining how your mobile application behaves in response to traffic, so it targets the back-end components of the application, not the front-end.

Both front-end and back-end performance are necessary to get a holistic view of mobile performance. Still, if you want to load test your mobile app, you need to generate traffic on your application server, and not necessarily to drive the app on real devices. Understanding this opens up options for mobile load testing.

API load testing

Most native mobile apps work by using a RESTful API to send instructions to the application server. Luckily for us, there’s no magic in sending these types of requests from a smartphone— we can simulate this easily by using an API load testing tool to send these requests on a non-mobile device too.

Here are some advantages of using API load testing tools to test your mobile app instead of using a device farm:

It’s easier to set up and maintain: API load testing can be easier because you don’t have to account for changes in the user interface as long as the underlying API calls are the same.
It’s cost-effective: You can run about 1,000 users for an API test on a single machine (we used AWS m5.xlarge machines to arrive at this estimate). This translates into fewer resources that you’ll need to pay for whether you’re running a thousand users or [a million](whether you’re running a thousand users or a million), and, ultimately, a lower cost.
It lets you narrow your load testing scope: If you have a particular endpoint that has historically shown performance issues, you can use API load testing to target just that one endpoint. On a device farm, you would have to create a test automation script that might click a button and send dozens of requests, only one of which you actually wanted to test.

Tools for mobile load testing

You can use many tools for API load testing, but you can’t go wrong with robust open-source tools like JMeter and Gatling. Both tools are well-used tools in the performance industry, and they also have large user bases and ample documentation for helping anyone get started. In this article, I’ll continue with JMeter specifically.

How do you get started with writing your script? Ideally, your developers should be able to point you to the API documentation for your app or be able to describe the format of the request themselves. But what if you wanted to get started on your own?

How to record network traffic from a native app with Apache JMeter

Recently I wrote an article on The New Stack about how I would have load tested the IowaReporterApp, which was responsible for delays in reported vote totals for the 2020 Iowa Democratic Caucuses. To show how to load test that app, I created an actual load testing script for that article, without any input from the app developers. I chose to script using Apache JMeter, a popular open source API load testing tool, but Gatling (also open source) is another good option.

Here are the steps I took on my Android mobile, a Samsung Galaxy Note 10+. The same principles should apply to iOS devices and other Android devices.

1. Download and install the app.

Even if the app isn’t on the Google Play store, you can install an app using the .apk file. You may have to enable Developer Mode on your phone, which is not the same as jailbreaking. All this does is bypass the security feature that would otherwise not allow you to install apps that don’t come from the Play store.

To enable Developer Mode, go to Settings > About phone > Software information. Then, scroll down to the field Build number and tap it seven times. You should see a confirmation message that Developer mode is on.

Now download the .apk, click on it, and follow the prompts to install it.

2. Set up an HTTP(S) Test Script Recorder on JMeter

You’ll need to have JMeter set up already for this step.

Connect your laptop to a wi-fi network.

From a new test plan on JMeter, right-click on Test Plan and then click on Add > Non-Test Elements > HTTP(S) Test Script Recorder. Click on it and take note of the port it uses in Global Settings.

Next, right-click on the test recorder element you just created and click Add > Listener > View Results Tree. This element shows details of requests and responses when we run tests.

Then, right-click on Test Plan again and click Add > Threads > Thread Group.

Finally, right-click on Thread Group and click Add > Logic Controller > Recording Controller.

Your test plan should look something like this:

3. Set up your mobile to use a proxy.

On your mobile, connect to the same wireless network that your laptop is connected to.

Go back to your laptop and find out its local IP address. On a Mac, you can see this by clicking on the Apple logo > System Preferences > Network > Advanced > TCP/IP. The IPv4 Address is what you’re looking for, and it might look something like 192.168.86.125.

On your mobile, go to Settings > Connections > Wi-Fi. Under Current network, click on the settings wheel next to your wireless network. Click on Advanced.

Enter your computer’s local IP address in the Proxy host name field.

In the Proxy port field, enter 8888, or the port that was displayed in the HTTP(S) Test Script Recorder on JMeter.

Click Save.

4. Start recording traffic.

On JMeter, click on the HTTP(s) Test Script Recorder element and click Start.

On your mobile, open the app and log in or perform the action that you’d like to record.

As you use the app, you should be able to see the requests coming in on JMeter in the View Results Tree:

If you click on the Recording Controller, you’ll also see the exact format of the requests that the app made:

Scripting and troubleshooting your mobile load testing script

What you now have is a recording of the mobile traffic you need to simulate. If you stop here and try to run it, you may find that it fails. That’s because you haven’t done any scripting yet.

Here are a few things you’ll need to do to turn a recording into a script.

Add assertions

An assertion is a way to verify that the request is returning the response that you expect. JMeter only marks a request as failed when it returns an HTTP 4xx or HTTP 5xx response code. However, there are times when the HTTP response code returned is an HTTP 200 OK, but the page that was returned was still not the one you expected. Without assertions, JMeter would mark this request as passed, yielding false positives.

For instance, if you’re sending a login request, then a good assertion for the response to that request might be text like “Welcome, Nicole” or “Logged in.” To see the data that should be returned, you can either run through the action manually in your browser and use Developer Tools to inspect the response or click on the response tab in the View Results Tree listener in JMeter from your recording.

Assertions also allow you to pinpoint more accurately where in your script the error is.

Replace hardcoded values with dynamic values

The last screenshot above shows the parameters that the mobile application sends to the server to log in. If you examine the parameter values, you’ll notice that several of them are long strings. The script fails when you just send the same request again because some of these are dynamic. This means that parts of the request need to change before your device is allowed to access the application. You might need to add a timestamp, the right answer to a challenge, a session or client ID, or codes returned in previous requests. These dynamic values help the server determine whether the login request is a valid login attempt.

This is where help from a developer would come in handy because he or she can tell you how each parameter value needs to be formed.

Add think time

Think time is a delay between requests that simulates the pauses that a real user takes to read text on the page or otherwise interact with your application. Most performance testing tools do this. In JMeter, they are called timers, and there are several to choose from.

I like the Uniform Random Timer because it allows you to set a constant value for the delay as well as an additional random delay on top of it. I recommend having a variable delay because setting a completely constant delay makes the test too regular and causes unnaturally high load at the interval you set. Real users don’t coordinate with each other, so stagger your requests by using a variable component to your think time.

Debug your test script

Before you think about scaling up, run your test a few times with one user on your local machine. If you don’t get any errors, try other configurations such as running two sequential users and running two threads at the same time. This way, you can test any correlations you’ve had to make for those dynamic values.

It may be tempting to skip this step, but it’s necessary before scaling up your load test.

Set test parameters

Test parameters are settings like the number of threads, ramp-up, and duration, which allow you to shape the pattern of the load you want to generate according to your test scenarios and test cases.

Number of concurrent users or threads

This is the most obvious way to increase load. The more instances of your script running, the more requests are executed.

You can change this in JMeter by clicking on your Thread Group and changing the “Number of Threads (users)” field:

‍

Ramp-up

In production, load on servers very rarely goes from 0 to 1000 in one second. Even for cases with a very definitive start time (such as an item going on sale at a particular time), the load generally increases gradually. You can simulate this gradual increase by adding ramp-up times in your script. A ramp-up is the amount of time during which new users are added at staggered intervals.

Here’s an example of what that might look like:

Duration

Duration is how long the whole test lasts.

The simplest way to set these values in JMeter is to use the basic Thread Group. You’ll need to change the fields highlighted below:

To use the duration, you’ll have to select “Forever” next to Loop Count and “Scheduler” and type in the number of seconds in the Duration field. Otherwise, you can also run the script according to the number of loops, or iterations, that you want to run. However, since response times vary, using the Loop Count as a proxy for duration yields different results from test to test.

Running a shakeout test against your mobile application

When you’re ready to run on your load generators, resist the urge to jump straight to the peak load you want. This is a great way to incur unnecessary costs. Instead, run a small test with dozens of users. Running locally is for debugging issues in the script. The purpose of running a shakeout test is to spot errors elsewhere, such as in your test environment and your test data.

Baselining your mobile application performance

Finding a baseline is one of the primary goals you’ll have as you start executing load tests. To be able to assess how changes in the environment or code affect application performance, you’ll need a stable point of comparison. To accurately measure the effect of a variable (like a code change), it’s best to keep as much of the other circumstances surrounding the test as fixed as possible. The baseline is your unchanging test scenario that you can use as a point of comparison to assess future performance.

The application you’re scripting against, your tool of choice, and your script all affect your performance baseline, so do some experimenting to find out the setup that works best for your situation. This setup includes characteristics like:

the number of concurrent users
test duration
think time, pacing, and all other waits and delays
the script (including how requests are broken up into transactions)
user density

User Density

User density is the number of threads that you can run on a node or a load generator. Figuring out user density it essential because all load generators, even virtual ones, have finite resources. Each load generator also has a finite amount of load that it can generate, based on its CPU and memory utilization, among other things. Trying to generate too many users on a single node may result in the node itself being the bottleneck for your test.

To accurately assess your application’s performance, your load generators need to remain healthy during the test.

It can be helpful to have a number to start with. At Flood, we’ve found that we can reliably run 1000 users using JMeter or Gatling with one of our AWS m5.xlarge machines. For reference, an m5.xlarge machine has 4 VCPUs and 16 GB RAM.

If your machine is similarly specced, run a test on a single node with 1000 users. While the test runs, watch the CPU and memory utilization. If the test finishes without either of those consistently hitting over 80%, you’ll know that the node can handle that number of users.

Let me reiterate, though, that you should use the 1000 users figure as a starting point only. You can then figure out your number through trial and error. Increase the number of users past 1000 and watch the resource utilization again. If that still looks good, add some more users and rerun. When you get to a test where the resource utilization hovers above 80%, stop and fall back to the previous number of users. You’ll now have your number.

Another thing you might want to play with is the think time and pacing in your script. These waits tend to have a significant impact on resource utilization, so you can expect to be able to run more users per node if you increase your delays.

If you’re using JMeter, always run your tests in non-GUI mode. GUI mode is great for debugging, but is unnecessarily resource-intensive for real load tests. While you’re at it, disable any listeners you may have that you don’t need to capture results.

Figuring out the appropriate user density now prevents you from getting inaccurate test results. Running as many users as you can without overloading the load generators is also cost-effective, as you’ll be making sure that you provision only as many nodes as you need.

Scaling up your load test on the cloud

It’s relatively simple to run a load test on one machine, but if you want to run on two or more machines, things quickly get unwieldy. Here are some options for achieving this.

1. Upload your tool of choice and script to every machine and kick off each test separately.

Instead of running one large load test, think of this method as running several smaller ones at (about) the same time. This is relatively easy to set up, but the main disadvantage is that it’s tedious. You may also need licenses for your tools if you’re not using an open source one. You’ll want to ensure that each load generator is as similar as possible to the others in terms of operating system, tool version, and script version. Since each node kicks off a separate test, you’re not going to be able to see a real-time combined view of all the load tests, and after execution, you’ll need to retrieve results files individually and combine them.

2. Use your tool’s distributed testing mode.

JMeter has a feature that allows you to scale. This involves setting up agents on each load generator and using scripts to coordinate execution and results collection. This method requires a little bit more know-how and time to set up, but it’s a little more cohesive than the first method. Here are links on how to set this up: Remote Testing with JMeter

3. Use a load testing platform.

I’m admittedly biased, but if I were to go back to doing performance testing consulting work, I would choose Flood to run my tests. A platform like Flood that is built for distributed load testing is an excellent choice because it takes away all the setup considerations. This is the easiest option, especially for teams that are new to load testing or are perhaps less technical, because all the setup is done through a UI. Scaling out in this case just means uploading your script, choosing the number of nodes you want to run and in which region, and all the work is done for you in the background.

While you can run on-premises, I would consider running your test cloud-based infrastructure for greater flexibility and ease of scaling unless you already have a data center.

What happens next?

Running your test is just the beginning of mobile performance testing— there’s still metrics collection, results analysis, and resolving performance bottlenecks that you uncover. After all, it’s what you do with the test results that counts. Remember that performance and load testing, whether for web-based applications or mobile ones, are iterative. The real value comes from building the app performance testing principles discussed here into your release cycles, so that load testing becomes a habit rather than just a one-off exercise.

Load Testing Tutorial: Server Monitoring

Thu, 26 Sep 2019 19:30:10 +0100

Originally posted here.

It’s difficult to be a good load tester if you don’t like data, because a lot of the value of the load testing is in the analysis. Even if you successfully generate load against an application, it’s ineffective without enough data to determine how your application coped with the load. At Flood, we’re passionate about data because we know it drives decisions, so we’re always working to improve how we present data generated by our nodes. However, there’s another important source of data that in my experience is too often overlooked by new load testers: your own application servers.

Executing a load test without monitoring server health is like flying blind: you’ll know when you land safely and you’ll know when you crash, but even if you do crash, you won’t know why—or how you can avoid it next time. Monitoring server health is the black box that will tell you what went wrong.

What server metrics should you monitor?

There are a lot of metrics that you can monitor, and we always recommend that you choose what to measure based on your own specific circumstances, such as the purpose for your testing or any previous outages you’ve had in the past. If you’re new to load testing, though, here’s a list of metrics you might consider measuring:

Processor Time - how much the processor is being utilised
Processor Interrupt Time - how much time the processor is spending to handle interrupts
Processor Privileged Time - the time the processor spends handling overhead activities
Processor Queue Length - the number of threads that are waiting to be executed
Memory (Available Bytes) - unused memory available to process new requests
Memory Cache Bytes - the size of the memory cache, which is where copies of frequently accessed data are stored
Disk I/O - number of reads and writes to the disk during the test
Disk Idle Time - time that disks are not doing work
Disk Transfer/sec - average number of seconds that an I/O request takes to complete
Disk Write/sec - average number of seconds that a write request takes to complete
Network I/O - bytes sent and received

How do you determine which one to use?

If you’re still not sure where to start: At a minimum, you’ll need the CPU and memory utilisation (in bold in the list above) of every major component you want to test. These two metrics are vital and if either of these is consistently maxing out at (or close to) 100%, that’s a sign that the component is struggling with the number of requests.

As an aside, CPU and memory over-utilisation is a very common reason for less-than-ideal response times on the load generators, which is why we also show you those metrics on our nodes while you’re running the test.

How can you monitor these metrics?

How you get these metrics depends on your budget and your operating system. I’ll start with the free or lower cost methods and work my way up to enterprise solutions.

If your servers are running Windows, PerfMon is a good alternative. It’s built into Windows and the interface allows you to choose the counters that you want to measure and start recording.

There’s also a JMeter server agent that you can download, confusingly called “PerfMon” but not to be mistaken with Windows’ PerfMon, that you can install on your (Windows or Linux) server and will gather metrics. It’s normally used in conjunction with the PerfMon JMeter plugin.

Nagios Core is another good open-source way to not only collect metrics but also set up alerts when values exceed certain thresholds that you can set.

Moving from free and open-source options to more enterprise solutions, Dynatrace is a powerful tool that allows you to track not just server health but can actually be used to trace individual requests using a custom header.

AppDynamics is another fantastic tool that allows you to really drill down to specific SQL queries that take a long time to execute, for example, feeding you important information to give to your DBAs.

New Relic is another good choice for monitoring production as well as test environments, and we particularly love its thread profiling feature.

Other noteworthy tools are the Microsoft System Center Operations Manager, the Oracle Enterprise Manager, and BlueStripe FactFinder.

Flood offers out of the box integrations with Dynatrace, AppDynamics and New Relic. If you’re already using one of these tools, you can push the load metrics from Flood into the dashboards that you’re already using, allowing you to see both load metrics and server metrics in the same place. Here’s an example of a dashboard our own Antonio Jimenez created in New Relic which combined Flood-provided metrics with server-side metrics:

This view helps with spotting correlations or trends across the different sets of data, such as a response time graph that begins to increase at the same time that server memory utilization does.

Putting It All Together

Having both sides of the puzzle, the metrics from the application servers along with those from our load generators, gives you a more holistic look of your application performance as a whole, and increases the value you get from load testing. We’d encourage you to take advantage of the 5 free node hours we provide in our trial to give this integrated load testing solution a spin. You can head on over to Flood to sign up for a free trial which will include 5 free node hours and you can drop us a note to get help integrating Flood with your APM solution.

Start with Why: How to Write Requirements for API Load Testing

Wed, 19 Jun 2019 21:49:17 +0100

Originally published here.

Every good test begins with requirements.

You may think, “But I’m a tester! I’m not a business analyst!”, and I hear you. But as testers, especially on smaller teams or shorter projects, it’s still part of our job to make sure we know what our clients want. Otherwise, how do we know whether we’ve succeeded or failed?

At Flood, I often speak to project teams who ask for help interpreting reports. Most of those times, the problem is not that they don’t understand the metrics or what they measure. The problem is that they didn’t set pass or fail criteria in the first place, and so they don’t have anything to judge the results against. This is always a big warning sign.

Which performance testing metrics are important? Why are we testing, anyway?

Tim Koopmans, co-founder of Flood, coined the acronym SPEAR to describe the different aspects of a performance that should be considered in load testing. This is a great starting point when thinking about what we want our nonfunctional requirements to cover.

Scalability

Scalability is the application’s ability to cope with increasing demands by increasing the amount of server resources. This could mean scaling up (increasing the resources of the dedicated server) or scaling out (adding more nodes to shoulder the load). What happens when more users than expected sign up in response to a promotion on your site?

Performance

The most common performance metric is page response time, but there are other considerations to be made here, such as throughput (requests per minute) and the number of concurrent sessions that need to be supported. Things like the total size of the resources on the page, whether or not a CDN is being used, and what to cache are also worth discussing.

Elasticity

Elasticity is a relatively newer aspect to performance testing brought about by advances in the cloud that allow application infrastructure to adapt to changes in load. Unlike scalability, elasticity emphasises scaling down as much as it does scaling up. Testing that virtual machines scale up when load increases is important, but testing that virtual machines also scale down when load decreases can also help save on unnecessary costs.

Availability

To test for high availability, ask yourself what would happen when (not if) your application’s server fails. Is there another server that the load balancer will seamlessly send traffic towards? Does the throughput fluctuate wildly? If users are connected to one server that fails, is your application smart enough to make new connections to another server? Or will it simply serve up an error page that users won’t know what to do with? Disaster recovery is best tested when there’s no disaster imminent.

Reliability

Reliability encompasses a lot of scenarios, but they all have to do with whether or not your application returns expected responses. Does your error rate increase when you increase the duration of your load test? Are you adding verification steps to your load testing scripts to check whether or not the HTTP 200 response that the application returned is not actually an error page?

What should my application’s response time be?

Our clients frequently ask us what the industry standard is for response times, wanting to make sure their applications measure up. The answer, however, is more complicated than a single number.

Industry standards for response time are only useful when applications are very similar. Constantly changing technologies used in web development as well as innate differences in business processes, however, make it very difficult to extrapolate a single number that will apply to all, or even most, applications in a certain industry.

The home page of one e-commerce app, for instance, might be several seconds slower than that of their main competitor. However, that doesn’t take into account the fact that their app loads a video showcasing new products. Does that mean that the development team should remove the video in order to fall in line with their competition?

Well, maybe. But not necessarily. It’s a business decision that needs to be made after perhaps using focus groups to determine the impact of the video, forecasting changes in conversion rate due to it, and comparing its projected value to the effects of being slower than the competition. A/B experiments could be used to test these assumptions and gather quantifiable data to support the team decision.

These factors are often not considered in the search for one number to rule them all, which is why a fixation on that number can be detrimental. Instead, I encourage project teams to brainstorm and come up with their own numbers for all metrics that would be more appropriate for their application. Gathering comparative metrics from a competitor may be part of this process.

What is a good API load testing requirement?

A good requirement, just like any good goal, is SMART:

Specific

Vagueness in a requirement leads to vagueness in results interpretation.

Instead of: “The performance of the web application…”

Try: “The average response time of the Login transaction…”

Measurable

Make sure there is a quantifiable way to know whether requirements have been achieved.

Instead of: “Decrease user frustration.”

Try: “The error rate must be below 3% at peak load.”

Agreed Upon

Have the appropriate stakeholders been involved?

Instead of: “The system must be able to generate emails as soon as users register.”

Try: “The system must be able to generate a maximum of 100 emails an hour, after which emails are queued.”

A good example for this is a project I was involved in that aimed to increase the speed with which emails were generated. Unfortunately, the particular email being sent included a big change that many customers were expected to contact the support team about. The problem was that nobody had thought to include Support in the conversation. Once they heard about the expected end result, the support team quickly raised their concern that they would not be able to handle the expected volume of emails unless the emails were staggered. This could have been avoided if they had been brought into discussions from the very beginning, in the requirements gathering phase.

Realistic

Can we meet this requirement given the resources available?

Instead of: “All requests must be returned within 5 ms.”

Try: “90% of the Catalog page requests should be returned within 3 seconds.”

Timely

Especially for nonfunctional testing, consider adding a timeframe to requirements.

Instead of: “The digital code is sent by SMS upon successful client log in at peak load.”

Try: “The digital code will be sent by SMS no later than 5 minutes after successful client log in at peak load.”

Go forth and write great load testing requirements

Requirements represent a great opportunity to think things through and make sure everyone on the project team is on the same page about the goals for your load testing. Too often projects skip this phase, only to realise much later that the tests that were executed didn’t address a key stakeholder’s concerns.

In load testing, as in many things: when in doubt about what to do, start with why.

TestCon Moscow 2019: Dismantling Cognitive Biases

Wed, 03 Apr 2019 23:14:31 +0100

Originally posted here.

Developers and testers alike prefer to think of themselves as entirely rational beings, because both software development and software testing do require systematic rigor and logical reasoning. However, the reality is that the human element still comes into play when building and testing software.

In this year’s TestCon conference, held in Moscow, Russia, I delivered a talk on how cognitive biases affect web performance outcomes.

In the talk, I discuss:

two types of thinking: fast thinking and slow thinking (from Daniel Kahneman’s book Thinking, Fast and Slow)
how to influence the sentence duration that judges give perpetrators
what cognitive biases are and how they can affect performance testing results
a real-life production incident that occurred on Flood and how our team’s cognitive biases got in the way of its resolution
what to do to avoid cognitive biases when testing software

Here’s the video of my full talk:

Download the presentation in PDF format.

Performance Testing Interview Questions, Part 3: LoadRunner

Thu, 20 Dec 2018 22:54:24 +0100

Originally posted here.

Like it or not, LoadRunner is one of the most popular commercial load test tools, and with good reason. Its protocol support is extensive, and many a load tester has cut his/her teeth on LoadRunner. In this last part of our interview series, we’ll go over some questions you may have to answer when interviewing for a role using Mercury HP MicroFocus LoadRunner.

What sort of transactions would you put into vuser_init and vuser_end?

Both are used for transactions that would only need to run once, at the beginning and at the end of a script. They’re akin to JMeter’s setUp and tearDown thread groups. Some things you might put in vuser_init are user logins or navigation to get to the start of the business process you really want to test, and logouts and any data cleanup to reset the application for the next test are common things to put in vuser_end. Note that if you want each virtual user to iterate through a list of user accounts during the test, you won’t want to put logins and logouts in vuser_init and vuser_end.

When would you run a script in VuGen and when would you run it in Controller?

VuGen is short for Virtual User Generator, and it’s where the actual scripting is done. Running in VuGen is good for writing or debugging a script, where you can quickly see where a script fails, what the response is, and what you need to change. You can also step through the code line by line (F10) to more accurately see, for example, variable values changing throughout your script. It’s really only for shaking out scripts with a single user. Running in VuGen also only runs from your local machine. Once you’re ready to run a load test, you’d switch to Controller.

The Controller is like the command center in LoadRunner: from the Controller, you can actually ramp up the number of users and choose from different scenario configuration settings in order to tailor your workload model to the scenario you’re trying to test. Controller also runs the scripts headlessly; it won’t open up a separate instance of VuGen for every user, so it’s a lot more resource intensive. Controller will also allow you to configure remote load generators and run scripts on them.

How would you set up LoadRunner to run certain transactions a certain percentage of the time? For example, for every 100 users that browses to a site, you want 25 of them visiting each of the About, Catalog, Contact, and Reviews pages.

You can write the code to visit each page in separate Actions. Then, in Runtime Settings, you can modify the run logic, create an Action Block that has all four actions, and assign a percentage to them (25%).

Of course, you can also hard code this in C within the actions, if you’d prefer, but moving these values to Runtime Settings makes it easier to maintain and reuse by other testers.

How is LoadRunner pacing different from how it’s implemented in other tools?

Most test tools allow you to set a fixed or variable pacing that adds onto however long one iteration takes to complete.

LoadRunner has a pacing feature that allows you to set a specific time and then adds a variable amount of time to your test depending on response times for the included transactions. So if you set pacing to 5 minutes and one iteration takes 3 minutes to finish, LoadRunner will automatically wait 2 minutes before starting the next one. If it takes 6 minutes, it will not add any more wait time and proceed instead to the next iteration. This comes in handy because you can more accurately predict how many transactions the script will do in a certain amount of time.

Why would the traffic from a load test on LoadRunner not be seen in Google Analytics?

Google Analytics runs when the browser of a user downloads and executes a javascript file that then sends information to Google to increment the site statistics for the current visit. However, most traditional load test tools work on the protocol level; that is, they send requests directly to the application server and download resources from a page to simulate users. This means that while they download the Google Analytics script, they don’t execute it, which means the scripted visits aren’t counted.

In order to use Google Analytics to view statistics from a load test, you’ll need to use a browser-level load test tool. LoadRunner actually has this covered with their TruClient protocol, which interacts with elements on the page like a user would. It will also work for testing single-page applications that use AJAX to change the data displayed on a page instead of reloading new pages.

Of course, for open source alternatives to TruClient, we at Flood prefer Element or Selenium load testing, both of that will actually open the site in a browser and executed scripts like a real user.

Performance Testing Interview Questions, Part 2: JMeter

Thu, 20 Dec 2018 14:47:18 +0100

Originally posted here.

In this part, we’ll go over some questions that are commonly asked in technical interviews about the use of the most popular open-source load testing tool, Apache JMeter.

Using JMeter, you record a key business process. Upon playing it back, however, you notice an error on the first page. What could have gone wrong and how would you fix this?

While recorders have gone a long way towards making scripting easier, modern security measures like the implementation of viewstates have also made scripting more complicated to record. Some more complicated applications will pass dynamic values in responses that it expects to see in the succeeding request, otherwise the request is blocked. This is great for security, but it also means that you will need to manually extract these values in your script.

In JMeter, you would do this by adding a response extractor to your sample that will extract the dynamic string according to rules you set, and then save the string to a parameter. This parameter must then be used in the next request that you send.

While running your test plan, you get the error: java.lang.OutOfMemoryError: Java heap space. How do you fix it? The easy fix is to edit jmeter.bat and increase the heap memory (search for HEAP) and increase heap space allocated like this: HEAP="-Xms512m -Xmx2048m"

The part in bold increases the heap space to 2 GB.

However, it’s also worthwhile looking at your test plan and removing any obvious memory hogs. One of the things we see a lot is enabled listeners that aren’t even being used. Listeners are great while debugging, but should be disabled when running a load test. Writing too many lines to the log is another culprit here that should also be commented out when running important tests. And of course, you should always run JMeter in non-GUI mode during load tests!

After running a load test, the results show that 300 transactions were executed with a PASSED status and returning responses of HTTP 200. However, the DBA tells you that there were no records written to the database as a result of the transactions. How would you troubleshoot this?

I see this happening with clients all the time, so I think this makes a great technical interview question. This is also why I said earlier that you shouldn’t count on a test tool to tell you a test has passed. It’s possible for a transaction to have the status PASSED and return an HTTP 200 but still be hiding an error.

This is mainly due to poor scripting that does not include validations that check whether the script has gotten to the right page. For example, you could use a Response Assertion as a child of your home page sample with the words “Welcome, Nicole” to determine whether the user Nicole has logged in successfully.

This is a critical step, because without it, it’s possible for a server to return a nice error page, or an incorrect one, such as the home page, with an HTTP 200.

How do you use a data file to read user account details from a CSV in JMeter?

JMeter has a CSV Data Set Config Element built specifically for this purpose. You can simply add one to your test plan, browse to the CSV file that you’d like to use, and select the options for how to share that data among the threads in your JMeter script.

How would you go about testing a website that uses HTTP/2 on JMeter?

There is a JMeter plugin called HTTP/2 Sampler that accomplishes just this. My colleague Jason just wrote a blog post on this very topic, but basically the plugin adds new HTTP/2 versions to the standard HTTP request and request defaults samplers as well as the Simple Data Writer and View Results Tree listener. These elements will need to be used in order to test an application using HTTP/2.

What can you do to synchronise the execution of multiple JMeter instances on different load generators?

Apache JMeter has released this handy PDF file to describe the steps necessary to run JMeter in distributed mode. This involves installing Java and JMeter on all machines and setting them up in a master and slave configuration so that one machine controls the rest. On the master, you’ll need to add remote hosts to the jmeter.properties file. When everything is set up, you should be able to open up JMeter on the master and do a a remote start of your current test plan on any or all slaves that you choose.

The problem with this setup is the clunkiness of working with different machines and essentially having to set them up the same way. After the test, there’s the daunting task of logging back onto each machine, gathering the data, and transferring them back into a central repository. Then you’re faced with consolidating the data from each injector in order to see a picture of the whole test. You can see that this set up quickly goes from time-consuming to a nightmare, especially when potentially thousands of different machines are involved.

If only there were an easier way…

Performance Testing Interview Questions, Part 1: General

Thu, 20 Dec 2018 12:41:40 +0100

Originally published here.

Preparation is Key for the Perfect Interview

Performance testing interviews can be a nerve wracking experience. We all understand it’s necessary, but we dread it anyway. Each company has its own style of interview, so you may need to prepare differently for each opportunity. This interview series will cover some general sample questions and scenarios as well as my answers to them to help you nail that performance testing interview!

I’ve broken down the potential questions into the most popular categories, so you can prepare easily for your all-important interview:

What is Performance Testing? What is Load testing? What is Stress Testing?

Performance testing is the process of systematically determining, quantifying and improving the speed, reliability, and efficiency of an application with the aim of improving user experience. This includes both front-end and back-end performance testing.

Load testing is a specific type of back-end performance testing where automated scripts are written to send requests in an attempt to simulate the peak user load expected on the application servers.

Stress testing takes that one step further and applies load in excess of what is expected in order to determine the maximum capacity of the system under test.

What is the difference between front-end and back-end performance testing? Which one is more important?

Both front-end and back-end performance testing measure how fast an application responds, but they measure different components of that overall user response time.

Front-end performance is concerned with how quickly text, images, and other page elements are displayed on a user’s browser page. Back-end performance is concerned with how quickly these elements are processed by the site’s servers and sent to the user’s machine upon request. Front-end performance is the part of the iceberg above the water line, and back-end performance is everything underneath that you can’t see.

Both are important because they can both determine whether a user continues to use your application. Front-end performance tends to be easier to test and can provide some quick wins due to the large amount of optimisation tweaks that can be done without writing code. Back-end performance tends to be more difficult to test because it often uncovers problems with the underlying infrastructure and hardware that are of a more technical nature.

Why does performance testing matter?

Performance testing matters because application performance has a significant impact on user experience. A site that is unreachable or slow to load due to an inability to cope with unexpected load will cause users to browse to competitor’s sites and tarnish the brand’s reputation.

How do you know when a load test has passed?

Ideally, you would have discussed your nonfunctional requirements with key stakeholders before load testing begins. This means that you set your own pass criteria before you even run the tests. You would ideally have a list of specific transactions (selected based on criticality or complexity according to the business) whose response time needs to fall under a threshold you’ve predetermined. “Fast” is not specific enough– a number is better. Depending on what kind of tests you’re running (soak, stress, volume, etc) you may have other nonfunctional requirements about duration, resource utilisation on the server side, or specific outcomes to scenarios you’d like to test.

As a general rule, don’t rely on a load test tool to determine whether your load test has passed. Rely on it to report your results, but always compare the results to the requirements to determine successes or failures.

What would you advise to clients who say they can’t afford to performance test because they don’t have the resources to maintain several load generators on site?

This is the main reason that performance testing has for so long been considered a luxury that only big companies can afford. Luckily technology moves on, and in 2018 we’re at a point where everyone can load test. The big innovation here has been the cloud and the ability to spin up thousands of virtual machines with a few mouse clicks. Services like Amazon AWS, Microsoft Azure and Google Cloud make it so that every budding entrepreneur can “borrow” the computing hardware necessary to do cloud load testing with thousands of users and then give them back after the test, without the hassle and cost of maintaining them. I would advise the clients to look for a a cloud load testing solution that utilizes virtual machines on the cloud to run their tests affordably.

You run a load test against a server with 4GB RAM and the results show an average response time of 30 seconds for a particular request. The production server has been allocated 8GB RAM. What would you expect the average response time of the same request to be in production given the same load?

Trick question! While you may be tempted to answer that the response time would be halved to 15 seconds, reality is rarely that convenient. Response times are a factor of so much more than memory. Things like CPU utilisation, network throughput, latency, load balancing configuration, and just application logic are always going to influence load tests. You can’t assume a linear progression in response time just because you’ve upgraded one part of the hardware. This is why it’s important to load test against an environment that is as production-like as possible.

What is a percentile and why would you look at percentile response times when you already have average response times?

A percentile is a statistical measure that describes a value that a certain percentage of the sample either meets or falls under. For example, a 90th percentile response time of 5 seconds means that 90% of the responses took 5 seconds or less to be returned. It can be an important measure because they soften the impact that outliers have on more inclusive measures such as averages. A transaction with an average response time of 2.5 seconds may seem perfectly acceptable to the business, but when the 90th percentile response time is 20 seconds, this is a good reason to investigate further.

What are some trends in performance testing that you think will continue in 2019 and beyond?

Cloud is an easy answer, as the cloud brings some compelling benefits in terms of reduced cost and just ease of use. However, I already touched on that in a previous question, so I’ll talk about another trend: open source.

There’s a reason that open source is still around: it works. The main advantage of open source tools is not that they are free, although that is a big part of the appeal. The real advantage is that open source tools are community-based and community-led, which means features get built for it faster than sometimes commercial tools can keep up with, and they’re built by users of the tool themselves. Open source tools like JMeter, Gatling and Selenium have revolutionized the industry due to their impressive feature sets, built by a growing community that has developed plugins for everything you can think of. More and more, even big companies with big budgets choose to go open source simply because of the wealth of knowledge of these tools that are already available for free.

We’re huge proponents of open source tools at Flood, which is why we recently open-sourced our very own tool, Element. We will continue to support all things open source, because we firmly believe that the future is open source.

Should I Upgrade to JMeter 4.0? 5 Reasons to Upgrade

Fri, 23 Feb 2018 23:43:25 +0100

Originally posted here.

Apache has just launched JMeter 4.0, a big step up from their previous release, 3.3, which was only released in September last year. Apache seems to be stepping up the frequency of it’s releases, and we’re not complaining. JMeter is an amazing open source load testing tool and the most popular one used at Tricentis Flood by far.

The big question is: Should you upgrade? Are the new features worth the leap? Will the new black UI make you a better load tester? While this choice should be made on a case-by-case basis and we certainly can’t pretend to give one-size-fits-all advice, here are some good arguments to do it:

#1: Java 9 support

JDK 9 was released back in September 2017, and if you were one of the people who upgraded to it when prompted, you would have noticed to your frustration that JMeter 3.3 did not play well with it, yielding the error:

Error: Java version -- 9.0.1 -- is too low to run JMeter. Needs a Java version greater than or equal to 1.8.0 errorlevel=3

Good news! This is now fixed. JMeter 4.0 runs smoothly on JDK 9. We’ll continue running on JDK 8 at Flood for consistency amongst all our open source load testing tools.

#2: New Boundary Extractor

Previously, my preferred way to extract dynamic values in JMeter was to use a regular expression extractor, but there’s no doubt that it can sometimes be tricky to get just the right text out of a response. Enter the Boundary Extractor.

The boundary extractor allows you to simply set the text immediately before and after the text you want to extract. This saves you wondering whether you should have used a + or a * in your regular expression, and is also easier to validate.

#3: New JSON assertion element

While XPath assertion was great, there was previously no counterpart for JSON. The JSON Assertion element fills that void nicely, using JsonPath to quickly assert the existence (or non-existence, with the Invert assertion option ticked) of a particular element.

#4: New throughput timer

We have a new throughput timer to play with, and it’s called Precise Throughput Timer.

Whereas the Constant Throughput Timer also had the disadvantage of constant pacing of requests, the Precise Throughput Timer is based on the Poisson Random Timer and randomly generates the delay between each request. The implementation seems to be well thought out, with the component reference section listing a lot of scenarios they’ve tested for that the previous Constant Throughput Timer might have struggled with. Throughput timers are one of my favourite JMeter features, and still rather unique in the industry.

#5: Test Plan Autosave

This one’s more of a usability tweak, but it’s a good one. JMeter now automatically saves your script when you run a test, saving you from those all-too-familiar “Wait… did I CTRL+S before I hit CTRL+R?” moments.

Click here to see the full release notes for version.

BONUS reason: Tricentis Flood support

You can now upload JMeter 4.0 scripts to flood.io and be up and running with your first test hosted on one of our grid nodes around the world. Our customer support team is always happy to help you make the switch to 4.0.

Performance on Nicole van der Hoeven

Entendiendo los sintéticos (Café con Grafana #008)

How to Get Started with Tempo (Grafana Office Hours #22)

Resources

Repasemos el G-stack (Café con Grafana #005)

Performance Testing With Nicole van der Hoeven (Quality Bits Podcast)

How to Use Flow Mode for Grafana Agent (Grafana Office Hours #21)

Resources

Dimensionando el Grafana Suite (Café con Grafana #004)

Realistic Tests with Testcontainers (Grafana Office Hours #20)

Related links

(ES) Cómo hacer pruebas de carga con Grafana k6 (Café con Grafana #003)

Resources

(TL) Ano ang relasyon ng testing at observabilidad?

K-SIX News v0.47

Real User Monitoring With Grafana Faro (Grafana Office Hours #17)

Useful links

Recording a Performance Test With Grafana Cloud k6 (k6 Office Hours #93)

Testing Kubernetes Networking With k6 (k6 Office Hours #91)

How to use k6 for site reliability engineering (k6 Office Hours #90)

How Supabase does performance benchmarking using k6 (k6 Office Hours #89)

Intersectionality in Tech (Adobo & Avocados #03)

Getting started in testing (Adobo & Avocados #02)

How to Get Core Web Vitals in k6 (k6 Office Hours #86)

Emergent Load Testing - rules for organized chaos

Resources

Abstract

k6 Javascript Ecosystem (k6 Office Hours #85)

Frontend vs. backend: How to plan your performance testing strategy

Why (and where) to use frontend performance testing

Advantages of frontend performance testing

Disadvantages of frontend performance testing

Why (and where) to use backend performance testing

Advantages of backend performance testing

Disadvantages of backend performance testing

Frontend + backend testing for the win

How we got into tech (Adobo & Avocados #01)

Why “Adobo & Avocados”?

How we got into tech

Nicole’s background

Marie’s background

How did we become Developer Advocates without being developers?

What’s a Developer Advocate?

What can you expect from Adobo & Avocados?

Grafana Cloud k6 (k6 Office Hours #83)

Introducing Grafana Cloud k6

Experimental modules in k6 (k6 Office Hours #82)

How to Do Exploratory Testing for Performance (k6 Office Hours #79)

Integrating k6 and Tracetest (k6 Office Hours #77)

Frontend vs Backend Performance Testing

How to Organize Testing Scripts in k6 (k6 Office Hours #76)

Introduction to Performance Testing

Guide to Load Testing Websites

Load testing approaches

Backend vs. frontend performance

Frontend performance

Backend performance

Which one should you test?

Component testing vs. end-to-end testing

Component testing

End-to-end testing

Protocol-based, browser-based, or hybrid load testing

Protocol-based load testing

Sample protocol-based test script

Browser-based load testing

Sample browser-based test script

Tips for writing browser-level scripts

Hybrid load testing

Scripting considerations

Consider factors that affect script realism

Create a reusable framework

Test with thresholds in mind

When possible, use hybrid load testing

Execution considerations

Run your tests in the appropriate environment

Run tests where your customers are

Recommendations

Distributed Load Testing Using Kubernetes With k6

Observability Trends and Future Best Practices (k6 Office Hours #59)

How I Learn in Obsidian - Keeping up with tech trends