Load Testing on Nicole van der Hoeven

Entendiendo los sintéticos (Café con Grafana #008)

Thu, 11 Jan 2024 16:00:25 +0100

En este episodio de Café con Grafana les traemos a Marcelo Magallón, experto en monitoreo sintético, quien viene precisamente a contarnos que es en sí un sintético. Él nos va a mostrar algunos ejemplos, casos de uso y mucha mas diversión!

Lee la documentación sobre el monitoreo de sintéticos en Grafana Cloud aquí.

Repasemos el G-stack (Café con Grafana #005)

Thu, 30 Nov 2023 13:57:52 +0100

En este episodio de Café con Grafana tenemos a Raul Marin, Solutions Engineer (Ingeniero de soluciones?) quien viene a darnos una repasada completisima de todo el stack de Grafana. TODO! Pues hay muchos que no conocen todas las herramientas, caracteristicas y posibilidades que Grafana Labs viene a ofrecer.

Performance Testing With Nicole van der Hoeven (Quality Bits Podcast)

Tue, 28 Nov 2023 14:01:35 +0100

My friend Lina Zubyte invited me to her podcast Quality Bits to talk about performance testing. We talked about what performance testing is, why it is important, how to get started, and how to get better at it. We also talked about the foundation of any type of testing: empathy.

Check out the episode here.

Dimensionando el Grafana Suite (Café con Grafana #004)

Thu, 23 Nov 2023 13:51:25 +0100

Almudena Vivanco viene a platicarnos de todos los experimentos y pruebas a las que someten a toda la suite de Grafana. Desde Mimir hasta k6 ella pone al limite todos los elementos constantemente. Además nos dará muchos tips de su experiencia para usar Grafana y mucho mas!

Realistic Tests with Testcontainers (Grafana Office Hours #20)

Fri, 17 Nov 2023 11:40:46 +0100

In this episode of Grafana Office Hours, we talk about realistic unit tests utilizing actual instances of databases and other dependencies. We’ll be joined by Manuel de la Peña from AtomicJar, and Pablo Chacin from Grafana to learn how we can use the Testcontainers open-source framework to create lightweight, throwaway dependencies for our tests.

Load Testing on Kubernetes with k6 Private Load Zones (Grafana Office Hours #19)

Fri, 10 Nov 2023 10:46:48 +0100

This week, we’re talking about how you can do load testing on Kubernetes with k6 Private Load Zones, a new feature on Grafana Cloud k6 that leverages the k6 Kubernetes operator to allow you to run distributed load tests against applications behind a firewall. Here to discuss this new feature are Senior Software Engineer Olha Yevtushenko, Product Manager Daniel González Lopes, Developer Advocate Paul Balogh, and Senior Developer Advocate Nicole van der Hoeven.

(ES) Cómo hacer pruebas de carga con Grafana k6 (Café con Grafana #003)

Wed, 08 Nov 2023 13:43:50 +0100

Daniel González Lopes nos viene a demostrar cómo hacer pruebas de carga con Grafana k6 desde zero, y cómo utilizar k6 para ejecutar otros tipos de pruebas. En este episodio para principiantes, hablaremos también sobre la relación entre pruebas y observabilidad.

Resources

From Load Testing to Reliability Testing Tool (k6 Office Hours #92)

Fri, 23 Jun 2023 23:43:31 +0100

In this k6 Office Hours, Developer Advocates Nicole van der Hoeven and Marie Cruz talk about the evolution of k6 from a load testing tool to a reliability testing tool. They also demonstrate how to combine various modules and extensions with k6.

Emergent Load Testing - rules for organized chaos

Wed, 19 Apr 2023 00:21:06 +0200

Today I spoke at KubeCon EU 2023 about emergent load testing.

Resources

Here are the slides.

Below are some links to extensions and experimental modules that I mentioned:

Grafana
k6 site
k6 repo
k6 docs
k6 browser (experimental module)
Prometheus Remote Write (experimental module)
k6 tracing (experimental module)
xk6-kubernetes (extension)
k6 redis (experimental module)
xk6 disruptor (extension)

Abstract

When we write load testing scripts against our applications, we write them sequentially: A, then B, then C. But this doesn’t accurately reflect the organized chaos of a system in production, nor does it prepare the system for the unexpected.

Emergence is a phenomenon where parts of a whole independently develop properties not originally present in the whole. Emergence is what helps ant workers develop roles without leadership, prompts animals to evolve adaptive traits without forethought, and facilitates non-toxic communities without moderators.

The growing field of emergent software applies this swarm logic to the programs that we write. What would it take to write emergent load testing scripts? It turns out that there are a few ingredients for emergence: a large population size, opportunities to interact, feedback, and an element of control.

In this talk, Nicole van der Hoeven discusses how to bring these elements to load testing by writing a script in Grafana k6 that can independently decide what requests to make next, modify Kubernetes app pods, and disrupt services based on a continual feed of results during runtime– all without manual intervention. She shows how to wield this new breed of load testing to improve confidence in the complex systems we build.

Frontend vs. backend: How to plan your performance testing strategy

Mon, 03 Apr 2023 13:20:53 +0100

Originally posted here.

There are many aspects of application performance, but they broadly fall into two categories: frontend performance and backend performance. As a tester, it’s important to know the differences between the two and how that impacts the way you approach your tests.

In this blog, I’ll provide a high-level overview of frontend performance testing and backend performance testing, including pros and cons of each one. I’ll also explain why you need both if you really want a holistic view of how your application is performing.

Why (and where) to use frontend performance testing

Frontend performance testing verifies user experience at the interface level. Basically, this entails everything a user sees when they access your web app, whether it’s on a laptop or other devices. This might involve automating user interactions with the site through the browser — for example, clicking on a button, filling out a form, or having scripts that load on the browser and run.

In frontend testing, you’re looking at how components of the UI respond.

The metrics we analyze for frontend performance tests are typically quite different from what we use for backend testing. Let’s say we want to know the time it takes for a page to render after a user clicks on a button that directs them there. This isn’t the same thing as measuring server response time; that total round trip takes a lot more into account — things like latency or the time it takes for a client-side script to run and display the requested information.

Advantages of frontend performance testing

The great thing about frontend performance testing is that it provides a more realistic measurement of your end users’ experiences. They don’t care what’s going on under the hood; they care about the overall experience — and that’s exactly what frontend performance testing measures. Unlike backend performance testing, frontend performance testing looks at everything from a user’s point of view: from the time they visit a webpage to the time they can interact with something on that page.

On top of that, most (about 80%) of the performance issues end users report are frontend issues. It doesn’t make much sense to skip this type of testing; it can be a critical step in finding ways to improve the overall user experience.

Disadvantages of frontend performance testing

The most obvious disadvantage of frontend testing is that it doesn’t look under the hood. If you only do frontend testing, you’re almost certain to miss something. Maybe a component isn’t working, which is creating issues for your users. But you wouldn’t know that because you’re only testing what’s visible from the browser. In other words, you might be in the situation where you know something in the application is wrong and responding too slowly, but you don’t know what.

Frontend performance testing can also be quite expensive compared to, say, protocol-based testing. That’s because you need enough resources (memory and CPU) to actually spin up a browser instance and interact with your application the way that a real user would. The same load generator that could run tens of thousands of backend virtual users may only be able to run 50 frontend virtual users before running out of resources.

Finally, only testing frontend performance can sometimes lead to pretty misleading results, especially as you scale. As load increases, the response time for the frontend remains fairly constant, but that’s not the case with the backend.

While the frontend component of response time remains more or less constant, the backend component of response time increases exponentially with the number of concurrent users.

This phenomenon occurs because most of the issues that frontend performance tests might identify, such as a slow-running client-side script, don’t tend to compound. It’ll be just as slow on each user’s computer regardless of how many users are running it. However, problems uncovered by backend performance testing are on the side of the application, not the users. If a database can’t keep up with 100 users sending queries to it, then 1,000 users sending it queries is likely to yield an even worse response time.

Why (and where) to use backend performance testing

As you might have guessed at this point, backend performance testing targets the infrastructure under the hood. This could be a database that contains customer information, a payment gateway, or an authentication service for customers that have already signed up. These are things that an end user doesn’t see and maybe isn’t ever even aware of, but they still have an effect on response times.

Backend testing addresses infrastructure users might not even be aware of, like different databases, gateways, and services that underpin an application.

Scripting against the backend is quite different from scripting against the frontend. Instead of scripting user interactions like clicking, you’re scripting certain requests that are being sent on the protocol level. So, for example, you could have an HTTP GET request for the page and also for its embedded resources.

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
 http.get('https://test.k6.io');
 sleep(1);
}

The sample Grafana k6 script above is an example of a simplified backend performance test. It makes a single HTTP request to retrieve the contents of https://test.k6.io, and then waits one second.

Backend performance includes things beyond just load times and latency. It includes scalability, availability, reliability, and similar attributes, so the metrics that we measure are different as well. We might look at server uptime or response times per component. We could also trace the path of a particular request through the application stack.

An illustration shows that performance testing can address scalability, elasticity, availability, latency, resiliency, and reliability. Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Advantages of backend performance testing

As we already discussed, frontend testing doesn’t cover everything, but it’s also more complicated than that. The 20% or so of performance bottlenecks that are backend-related can be the hardest to troubleshoot, identify, and fix. Even worse, a lot of backend performance issues can spiral out of control at higher levels of load and can be quite expensive to fix later on in a development cycle. The worst time to realize that you need to rethink your entire infrastructure is after it’s already failed in production. If you take care of backend performance testing earlier (commonly known as shifting left), it makes such pivots less costly.

Another advantage of backend testing is the ability to target different components of an application. You can choose which API endpoint to hit and how hard, and you can target specific user flows, so it doesn’t always have to be end-to-end testing.

And finally, backend performance tests tend to be less resource-intensive; they’re just protocol-level requests that can be easily ramped up.You don’t need to spin up a browser instance, or even parse through the Document Object Module (DOM) of a webpage or look for specific elements. It’s just a matter of sending these requests, and that’s something that you can increase the scale of pretty cheaply.

Disadvantages of backend performance testing

Backend performance testing won’t tell you anything about what’s happening on the frontend, and a lot of sites these days actually use scripts that have to run on the frontend. So with backend performance testing tools, you just download those scripts. But the response time they measure is the download time, which is not really realistic because those scripts are executed. A backend performance test would fetch the text of the script. A frontend performance test would run it — and that script might do something like perform a calculation. You need frontend performance testing to measure that properly.

Backend performance testing is also much more expansive than front end performance testing, especially with microservices-based applications. There are so many components to track that it can be difficult to determine where to look. You can quickly find yourself going down a rabbit hole of constant, never-ending performance tuning.

Frontend + backend testing for the win

So, as we’ve seen, frontend testing and backend testing both seem to have advantages and disadvantages. So which one should you use? Both, if at all possible! They look at different parts of application performance, so doing both is essential if you want a holistic view of user experience.

Software teams can use Grafana k6 to address frontend and backend performance testing.

If you’re operating at a smaller scale, use your own discretion about how to find the right mix for needs. But if you’re testing at a larger scale with thousands of users and you want to test both the frontend and backend aspects of application performance, I suggest you change your test strategy so the majority of the load is generated with backend performance testing scripts. That’s because of the resource intensiveness of frontend performance testing scripts, which we discussed earlier.

So, if you generate most of the load with the backend and then you have a handful of frontend performance testing scripts running in the background, you should still get a good idea of how both are performing. Remember, frontend application performance remains fairly constant throughout different levels of load, so you’ll still get a lot of useful insights on both types of performance without worrying too much about the scaling costs.

Frontend and backend performance both affect what your end users think of your application. If the underlying application servers respond immediately to all requests, but then a huge picture of a crocodile makes your homepage excruciatingly slow to load, then users aren’t going to like that.

On the other hand, if your homepage is really sleek and quick to load, but the application server hasn’t responded with the data that the browser needs to render, then users aren’t going to like that either.

A common problem when testing both frontend and backend performance is script duplication: because of the differences in the two approaches, you often have to use two separate tools — which means two test suites and potentially two scripting languages. While this is a valid approach, it’s more sustainable to look for a tool that can target both types of performance, such as k6.

Whichever tool(s) you end up using, testing both frontend and backend performance is going to give you the clearest view of how your application really performs.

The easiest way to get started with k6 is through Grafana Cloud k6. The free tier includes 500 virtual user hours/month for k6 testing and more. Sign up for free now!

Guide to Load Testing Websites

Thu, 12 Jan 2023 17:12:42 +0000

Original article here, written for k6.io

This doc explains some key concepts about load testing websites, including:

The difference between backend and frontend performance testing
When to choose between protocol-based, browser-based, or hybrid scripts
Recommended practices about how to load test websites

All load tests try to simulate real user traffic to prevent failures, improve reliability, and release new code with confidence. But your approach to load testing must adapt to the type of application you want to test.

In this guide, learn about strategies to test websites, including specific recommendations for complex scenario scripting and test execution.

Load testing approaches

When you approach the load test, first consider the following perspectives:

Backend vs. frontend performance
Protocol-based, browser-based, or hybrid load testing
Component testing vs. end-to-end testing

Backend vs. frontend performance

Performance has a significant influence on the user experience of a website.

For example, when users want to visualize or interact with some information, they expect the website to react quickly.

To measure performance, testers often look at response times, which are affected by two main factors:

Frontend performance
Backend performance

Frontend performance

Frontend performance testing verifies application performance on the interface level, measuring round-trip metrics that consider how and when page elements appear on the screen. It’s concerned with the end-user experience of an application, usually involving a browser.

Frontend performance testing excels at identifying issues on a micro level but does not expose issues in the underlying architecture of a system.

Because it primarily measures a single user’s experience of the system, frontend performance testing tends to be easier to run on a small scale. Frontend performance testing has metrics that are distinct from backend performance testing. Frontend performance tests for things like:

Whether the pages of the application are optimized to render quickly on a user’s screen
How long it takes a user to interact with the UI elements of the application.

Some concerns when doing this type of performance testing are its dependency on fully integrated environments and the cost of scaling. You can test frontend performance only once the application code and infrastructure have been integrated with a user interface. Tools to automate frontend testing are also inherently more resource-intensive, so they can be costly to run at scale and are not suitable for high load tests.

Backend performance

Backend performance testing targets the underlying application servers to see how they behave under production-like conditions. When it comes to websites, while frontend performance involves how assets included in the page are rendered, backend performance focuses on how those assets are processed by the application servers, served to users, and downloaded by browsers.

Backend testing is broader in scope than frontend performance testing. API testing can be used to target specific components or integrated components, meaning that application teams have more flexibility and higher chances of finding performance issues earlier. Backend testing is less resource-intensive than frontend performance testing and is thus more suitable for generating high load.

Which one should you test?

It depends! Ideally, both.

Frontend testing tools are executed on the client side and are limited in scope: they do not provide enough information about backend components for fine-tuning beyond the user interface.

This limitation can lead to false confidence in overall application performance when the amount of traffic against an application increases. While the frontend component of response time remains more or less constant, the backend component of response time increases exponentially with the number of concurrent users:

Testing only frontend performance ignores a large part of the application, one more susceptible to increased failures and performance bottlenecks at higher levels of load.

Testing only backend performance, on the other hand, ignores “the first mile” of user experience and breadth. Backend testing involves messaging at the protocol level rather than interacting with page elements the way a real user would. It verifies the foundation of an application rather than the highest layer of it that a user ultimately sees.

Testing both frontend and backend performance leads to the best overall performance and user experience for your application. Ignoring one or the other exposes you to performance bottlenecks that significantly decrease a user’s satisfaction with your application.

However, if your testing is smaller in scope, you can choose to focus on either frontend or backend with the goal of eventually building a test suite that encompasses both.

Component testing vs. end-to-end testing

When you’re testing a web app, you may wonder whether how you structure your test scripts. Consider the following two methods.

Component testing

One way you could load test a web app is to load test its components. Maybe you know there are issues with a specific functionality due to previous production issues, or maybe you’d like to target components that are business-critical to reduce risk exposure.

In these cases, the type of test you write may:

be protocol-based
call only those API endpoints that are relevant
have smaller or no sleep/think time
be focused on eventually stress testing a component or service or finding its breaking point

Testing in this way is more flexible. With protocol testing, you can specify the endpoints or servers you’d like to hit, and narrow your test’s target that way. You can test specific functionalities while skipping others that come chronologically before it in a standard user flow. You can more finely control the type of traffic that is generated. If you’re recreating a specific mix of requests, you can script those requests and reproduce them in a repeatable manner.

Doing component testing for load may not always require that your script behave like an end user. In fact, it may be necessary to artificially inflate traffic to more quickly reproduce issues, or to deflate traffic to reduce the noise in the logs. By the nature of this type of testing, scripts don’t contain the full flow of a request that you would expect to see in production, so realism is not a priority.

End-to-end testing

You could also do end-to-end testing against a web app. End-to-end testing seeks to replicate real user behaviour and track its effects across the entire stack. When doing end-to-end testing, you might:

do protocol-level, browser-level, or hybrid load testing
replicate the actions in a typical user flow
think about the performance of the entire workflow as well as how long a request took to be processed by each component
write scripts that access the application the same way users would, such as by browsing to the homepage before finding their way to other parts of the website

This type of load testing is broader in scope than component testing, but shallower in terms of depth. With end-to-end testing, you get a better idea of the full user experience of the application as a whole. However, it can also be more complex to troubleshoot, as you have more components to monitor and more places to look for issues that are found.

End-to-end testing is based on real user behavior, so it’s often important that end-to-end test scripts also be realistic.

Protocol-based, browser-based, or hybrid load testing

The decision of whether to test the frontend, backend, or both will also affect the type of load testing you should carry out and the kind of scripts you should write.

Protocol-based load testing

Protocol-based load testing verifies the backend performance of an application by simulating the requests underlying user actions. For websites, this commonly involves HTTP requests that bypass the user interface of your application and are sent directly to a server or application component.

For example, a protocol-based load testing script might request all the resources on a webpage from the application servers, but those resources are merely downloaded. The response times reported by a purely protocol-based script do not include frontend metrics such as the time taken for images to render on a browser. The load is generated by simulating many requests sent to application servers.

While protocol-based load testing may seem to lend itself better to component testing, you can also do end-to-end website testing with protocol-level scripts.

Sample protocol-based test script

The following is an example of a protocol-based load testing script in k6 that fetches the homepage, along with resources embedded into the page.

import http from 'k6/http';
import { sleep, check } from 'k6';

export function Homepage() {
 const params = {
 'sec-ch-ua': '"Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"',
 'accept-encoding': 'gzip, deflate, br',
 'accept-language': 'en-GB,en;q=0.9',
 };

 // 01. Go to the homepage
 let responses = http.batch([
 ['GET', 'https://mywebsite.com/', params],
 ['GET', 'https://mywebsite.com/style.min.css', params],
 ['GET', 'https://website.com/header.png', params],
 ['GET', 'https://website.com/polyfill.min.js', params],
 ]);
 check(responses, {
 'Homepage loaded': (r) => JSON.stringify(r).includes('Welcome to my site'),
 });

 sleep(4);

 // 02. View products
 responses = http.batch([
 ['GET', 'https://mywebsite.com/products', params],
 ['GET', 'https://mywebsite.com/style.css', params],
 ['GET', 'https://website.com/product1.jpg', params],
 ['GET', 'https://website.com/product2.jpg', params],
 ['GET', 'https://website.com/displaylist.js', params],
 ]);
 check(responses, {
 'Products loaded': (r) => JSON.stringify(r).includes('Add to Cart'),
 });

 sleep(1);
}

Recording browser traffic helps you prototype to test websites on the protocol level.

Browser-based load testing

Browser-based load testing verifies the frontend performance of an application by simulating real users using a browser to access your website.

For example, a browser-based load testing script might include instructions to navigate to a page, click on a button, and type out text on a form. Those user actions then trigger underlying requests on the protocol layer, but only user actions are scripted in browser-based testing.

Unlike protocol-based load testing, browser-based load testing scripts generate load by starting multiple instances of browsers and interacting with your application the way real users would. Testing at the browser level can also be the only option for testing Single-Page Applications where a lot of the application logic is executed by client-side scripts.

Scripting on the browser level usually requires the use of different tools from the ones used to test at the protocol level. However, k6 has an extension called xk6-browser that allows the creation of browser-based test scripts alongside protocol-based ones.

Sample browser-based test script

The following is an example of a browser-based load testing script in k6 using xk6-browser on a dummy website. Instead of making an HTTP request, the script views the homepage, then looks for and clicks on a link to the product page.

import { chromium } from 'k6/x/browser';
import { sleep } from 'k6';

export default function () {
 const browser = chromium.launch({ headless: false });
 const page = browser.newPage();

 // 01. Go to the homepage
 page
 .goto('https://mywebsite.com', { waitUntil: 'networkidle' })
 .then(() => {
 page.waitForSelector('p[class="woocommerce-result-count"]"]');
 page.screenshot({ path: 'screenshots/01_homepage.png' });

 sleep(4);

 // 02. View products
 const element = page.$(
 'a[class="woocommerce-LoopProduct-link woocommerce-loop-product__link"]'
 );
 element.click();
 page.waitForSelector('button[name="add-to-cart"]');
 page.screenshot({ path: 'screenshots/02_view-product.png' });

 sleep(1);
 })
 .finally(() => {
 page.close();
 browser.close();
 });
}

Tips for writing browser-level scripts

The following steps can help you get started with a browser-level test script.

Script user actions, not requests. Determine what a user might do for a particular task, and script interactions with elements on the browser level. For example, script what buttons the user clicks on.

Identify unique selectors. Once you have identified which page elements a user interacts with, use the Element Inspector for DevTools in your browser to find a unique, static, and simple way to identify each element. The script needs selectors to find the right element to interact with.

Use elements to verify responses. After every action, use locators to search for elements on the page that you would expect to find. This verification helps ensure that the script has reached the expected page.

Take screenshots for every action while debugging. One of the advantages of browser-based testing is the ability to take screenshots. After every user interaction the script simulates, use page.screenshot to save a visual image of what the script encountered for later troubleshooting.

Hybrid load testing

Hybrid load testing is a combination of protocol-based and browser-based load testing. While you can use two tools or two scripts to execute different types of load tests (one protocol-based and one browser-based), having both types of tests executed by the same script and the same tool is ideal. Aggregating results between different tools can be difficult at best and inconsistent at worst.

A best practice in hybrid load testing is to generate most of the load using the protocol-level test, and then to run a smaller number of the browser-level testing scripts. This approach:

reduces the number of load generators needed, since protocol-level testing requires fewer machines to generate the same load
measures backend and frontend performance in the same test execution
provides a single source of aggregated output at the end
reduces complexity in the creation and maintenance of scripts

Scripting considerations

When you script a test for a website, consider these recommendations.

Consider factors that affect script realism

Record your user journey. Using the browser recorder can facilitate initial test script creation by capturing all the embedded resources on webpages. Check out the Session Recording guide to learn more about how to auto-generate your load test from a user session.

Correlate data. Recordings often don’t take into account dynamic values that are generated anew each time a request is made. Go through the recorded requests and determine whether you need to extract values from previous responses and use parameters in subsequent requests. This practice ensures your VUs behave more like real users would.

Include or exclude static resources. Determine whether you should include or exclude static resources on pages such as images, JavaScript, etc. Consider including them if you want to measure overall user experience. Consider excluding them if you are using a Content Delivery Network (CDN) that is under a separate Service Level Agreement (SLA).

Exclude third-party requests. Don’t load test servers that you don’t own. Many applications make calls to third-party providers for authentication, social sharing, and marketing analytics. Disable these requests unless you have permission to include them in your tests.

Use concurrent requests. To mimic the way modern browsers download some requests in parallel, use batching.

Determine cache and cookie behaviour. k6 automatically resets cookies between iterations, but you can also change this behavior if maintaining cookies would be more realistic.

Use dynamic think time and pacing. Consider adding varying delays so you don’t artificially stagger a script with completely uniform delays.

Use test data. Real users typically don’t search for or submit the same data repeatedly. Consider adding a test data file for the script to iterate through.

Model test parameters and load profile after production. In k6, you can use test options to determine the exact shape and profile of your load test script. Select the appropriate executors for the job.

Create a reusable framework

Use tags and groups. Organizing requests by tagging and grouping them helps you consolidate like metrics and makes your test scripts more understandable by others.

Use scenarios. When combining protocol-based and browser-based tests, use scenarios to independently control their test parameters and executors.

Modularize scripts. Use modules to separate and organize functions for protocol-level testing and browser-level testing, and then use a test runner script to execute them. This approach means different scripts can be versioned and changed without affecting each other.

Integrate your tests into your CI pipeline. Adopting a “tests as code” approach enables you to tie your load tests more closely into your project’s existing CI/CD processes, helping you get the most value out of every test.

Test with thresholds in mind

Create thresholds for both types of testing. Some browser-level and protocol-level metrics cannot be combined because they don’t measure the same thing. Set thresholds for relevant metrics from both the browser-level script and the protocol-level one.

When possible, use hybrid load testing

Use protocol-based scripts to generate majority of the load. When writing the test scenario, use the protocol-based requests to simulate most of the traffic and use fewer VUs for the browser-based requests. Relying on the protocol-level traffic helps keep resource utilization on load generators down.

Execution considerations

When you run tests, consider the test environment and the load generator location.

Run your tests in the appropriate environment

Testing in pre-production environments and in production environments both add value.

Testing in pre-production environments (staging, test, system integration testing, user acceptance testing, and production replica environments) enables you to identify performance defects early, which can save a lot of time and effort (and reputation) later on. Running in test environments also means you can often afford to be more aggressive with your tests. However, it’s also more critical to get your load profiles right, and the test results you get may not necessarily apply to production.

Testing in production yields the most accurate results, but it’s also more risky. Often, testing production is the only feasible alternative. You can reduce the risk of impact to real customers while testing in production by using lower levels of load when running load tests during peak hours, schedule tests for off-peak hours, choosing load test types that are less risky, using techniques like synthetic monitoring that generate less traffic, using real user monitoring tools to get snapshots of user performance at load, and ensuring that your observability stack is working at peak efficiency.

Run tests where your customers are

The location of the load generator(s), where the traffic is coming from, can also have an impact on your test results. The question is: where are your end users located?

On-premises load testing can be ideal when testing early on in the development a website, or when there are machines that can be repurposed as load generators. However, testing entirely from within a corporate network may also yield false positives, in that response times reported are significantly lower than if the same application servers were accessed from across the country.

Load testing on the cloud is an essential part of the testing strategy for many public-facing websites. Using load generators on the cloud gives you access to test in different states and geographical countries, creating a mix of load generators proportional to your users’ locations. Cloud load generators are easier to provision and cheaper to maintain in the long run than on-premise ones. Load testing on the cloud can help you include the effects of network latency in your tests and yield results that are more realistic.

Recommendations

Here are some recommendations to help you plan, script, and execute your load tests for websites.

If you want to test the last-mile user experience of your website:

focus on frontend performance,
write browser-based test scripts,
and consider doing more realistic end-to-end tests of the user flow.

If you want to test the underlying infrastructure of your website:

focus on backend performance,
write protocol-based test scripts,
and consider starting with component testing and then gradually increasing the scope.

If your website is meant for internal or has limited access:

Use on-premise load generators located within the network that most of your users access the website from.

If your website is external and public-facing:

Use cloud load generators in the load zones where your users reside.

Test in pre-production environments when you can, but also consider testing in production in a limited capacity.

Load testing websites can be complex due to the number of viable testing approaches available, the scope for performance testing, and the potential effects of releasing undertested code. By following the recommendations we’ve put forward here, you can tailor your testing more closely to your objectives.

Distributed Tracing With Grafana k6 (k6 Office Hours #70)

Fri, 04 Nov 2022 14:03:04 +0100

Just announced: distributed tracing with Grafana k6 x Tempo! Daniel González Lopes joins us to talk about what distributed tracing is, and how traces using the Tempo backend could help you troubleshoot your performance issues and improve overall reliability. This new integration was just announced at ObservabilityCon 2022, and Daniel’s here to tell us all about it!

Resources

All About Load Testing Metrics (k6 Office Hours #69)

Fri, 28 Oct 2022 15:00:07 +0200

Leandro Melendez and Nicole van der Hoeven discuss all about load testing metrics. What do they all mean? Which ones do you REALLY need? Understanding metrics and analysing test results is arguably the most important part of a load test– and also the most difficult! We’re answering all the questions you ever had about the many metrics that come with load and performance testing.

Assertions in k6 (k6 Office Hours #68)

Fri, 21 Oct 2022 16:29:35 +0200

How do you do assertions in k6? CTO Pawel Suwala talks about different ways to implement BDD/TDD in k6, including an assertion library called k6chaijs. We also discuss the differences between checks, thresholds, and assertions.

End-to-end browser testing using xk6-browser (k6 Office Hours #51)
Browser-based testing (notes)
Browser testing and API load testing in the same script with k6 (k6 Office Hours #35)
In search of the best Pokémon: Browser automation and load testing in one script with k6 (presentation at Automation Guild 2022)
Real browser testing with xk6-browser

Load Testing Types and k6 Executors (k6 Office Hours #67)

Fri, 14 Oct 2022 18:36:55 +0200

What are the different load testing types, and how do they relate to the k6 executors? We go through every single k6 executor and how, when, and why you should use each one in this week’s episode of k6 Office Hours.

All About Web Performance (k6 Office Hours #66)

Fri, 07 Oct 2022 17:25:24 +0200

My fellow Developer Advocate and tester Marie Cruz and I talk all about web performance and best practices for making websites faster. What’s the difference between performance testing a website vs. an API? What are some things you should keep in mind when testing a web app?

Performance testing for beginners (k6 Office Hours #65)

Fri, 30 Sep 2022 12:45:08 +0200

My colleague Paul Balogh and I interview the k6 technical writer, Matt Dodson, who asks all the questions about load and performance testing that you were always afraid to ask.

The future of k6 (k6 Office Hours #64)

Fri, 23 Sep 2022 12:18:07 +0200

We’re talking about the future of k6 with CEO Robin Gustafsson and CTO Pawel Suwala. What’s next for k6? What new features are we working on? What’s the plan for further integrations with Grafana? What are the plans for the k6 team as we transition out of the honeymoon period after the Grafana acquisition?

Getting started with xk6-browser (k6 Office Hours #63)

Fri, 16 Sep 2022 12:13:33 +0200

Marie Drake, the newest Developer Advocate to join the k6 team, shares her experience in getting started with xk6-browser: how to install it, issues she encountered, and the state of browser testing with k6 as she sees it.

Load Testing Apache Kafka using k6 (k6 Office Hours #62)

Fri, 09 Sep 2022 12:09:36 +0200

Senior Software Engineer Mostafa Moradian joins us to talk about load testing Apache Kafka using k6 and the extension he created, called xk6-kafka. He describes what Kafka is, the basics of Kafka, and why he thinks load testing it is important. Mostafa works on the backend team at k6/Grafana Labs.

Resilience and Chaos Testing with SteadyBit and k6 (k6 Office Hours #61)

Fri, 02 Sep 2022 16:56:14 +0200

What is the difference between resilience testing and chaos testing? SteadyBit CEO and co-founder Benjamin Wilms talks about how to integrate SteadyBit with k6 for both resilience and chaos, and why he prefers “chaos testing” to “chaos engineering”.

What Is Grafana Mimir? (k6 Office Hours #53)

Fri, 06 May 2022 17:00:28 +0200

Maintainer Marco Pracucci talks about the new Grafana project, Mimir, a next-generation time-series database for Prometheus. He joins Developer Advocates Nicole van der Hoeven and Paul Balogh to talk about what Mimir is, why it was needed, and how he helped scale it using k6.

API load testing using k6, with Tom Miseur (k6 Office Hours #52)

Fri, 29 Apr 2022 17:00:26 +0200

In this second part of demonstrating the dogfooding we do at k6, I talk to Tom Miseur, Technical Specialist, who shows how he wrote API load testing scripts with k6 to test k6 Cloud– and what his results were after his testing.

End-to-end browser testing using xk6-browser, with Thomas Wikman (k6 Office Hours #51)

Fri, 22 Apr 2022 17:00:12 +0200

Have you ever wondered how to test a platform built for testing other products? Our frontend developer, Thomas Wikman, joins us on k6 Office Hours to talk about his experience being a user of one of our tools - xk6-browser - and how he uses it to test the entire k6 Cloud frontend.

Load testing results in the k6 Cloud App for Grafana, with Edgar Fisher (k6 Office Hours #49)

Fri, 08 Apr 2022 18:00:18 +0200

There are many ways to get k6 load testing results into Grafana, but our frontend software engineer, Edgar Fisher, talks to us about the easiest way of them all: the new k6 Cloud App for Grafana.

Building Reliable Systems With k6 (k6 Office Hours #48)

Fri, 01 Apr 2022 10:02:29 +0200

In this week’s k6 Office Hours, my colleague Pandurovic, an engineering manager/backend lead at k6, joined Paul Balogh and me to talk about how k6 works on the backend, and how the backend team is working to make it even more reliable.

How to Analyze Load Testing Results with k6 (k6 Office Hours #47

Fri, 25 Mar 2022 18:00:56 +0200

In this k6 Office Hours, Paul Balogh and I discuss the different ways to turn k6 metrics into graphs for analyzing performance testing results.

Load Testing for DevOps with Victor Farcic (k6 Office Hours #45)

Fri, 11 Mar 2022 11:12:52 +0100

How can you use load testing for DevOps? What kind of performance testing can or should be included in CI/CD pipelines? Viktor Farcic from Upbound and DevOps Toolkit join Nicole van der Hoeven and Paul Balogh to discuss how he sees k6 fitting into the daily duties of a DevOps engineer.

Private Load Zones for Load Testing (k6 Office Hours #44)

Fri, 04 Mar 2022 10:01:57 +0100

Load testing on the cloud definitely has its advantages, but what if you need to load test behind a firewall? Here to talk about using private load zones in k6 are k6 CTO Pawel Suwala and k6 Backend Lead Samuel Regandell. Private load zones are a new feature of k6 Cloud that will allow you to run load tests against applications that are not publicly accessible and also use your own AWS account to generate the load.

In search of the best Pokémon: Browser automation and load testing in one script with k6 (Automation Guild 2022)

Thu, 10 Feb 2022 12:01:53 +0200

Being a tester is like being a pokémon trainer: we want to choose the right pokémon for the job. The problem with doing different types of testing is that we also end up with multiple scripts in different languages and testing tools. We end up choosing a squad of pokémon: one for unit testing, one for automation testing, one for load testing, and so on. Making sure they all work together is a task in itself!

But what if we could just use one? What if we only needed one Mega-evolved pokémon that could fulfill multiple roles? Is it really possible to write a browser automation script and then reuse it for load testing?

In this presentation, I talk about my search for the elusive one pokémon that does multiple jobs, the advantages of hybrid load testing, and how to pick pokémon that make our life as trainers and testers easier.

Load Testing on Nicole van der Hoeven

Entendiendo los sintéticos (Café con Grafana #008)

Repasemos el G-stack (Café con Grafana #005)

Performance Testing With Nicole van der Hoeven (Quality Bits Podcast)

Dimensionando el Grafana Suite (Café con Grafana #004)

Realistic Tests with Testcontainers (Grafana Office Hours #20)

Related links

Load Testing on Kubernetes with k6 Private Load Zones (Grafana Office Hours #19)

Related links

(ES) Cómo hacer pruebas de carga con Grafana k6 (Café con Grafana #003)

Resources

From Load Testing to Reliability Testing Tool (k6 Office Hours #92)

Emergent Load Testing - rules for organized chaos

Resources

Abstract

Frontend vs. backend: How to plan your performance testing strategy

Why (and where) to use frontend performance testing

Advantages of frontend performance testing

Disadvantages of frontend performance testing

Why (and where) to use backend performance testing

Advantages of backend performance testing

Disadvantages of backend performance testing

Frontend + backend testing for the win

Guide to Load Testing Websites

Load testing approaches

Backend vs. frontend performance

Frontend performance

Backend performance

Which one should you test?

Component testing vs. end-to-end testing

Component testing

End-to-end testing

Protocol-based, browser-based, or hybrid load testing

Protocol-based load testing

Sample protocol-based test script

Browser-based load testing

Sample browser-based test script

Tips for writing browser-level scripts

Hybrid load testing

Scripting considerations

Consider factors that affect script realism

Create a reusable framework

Test with thresholds in mind

When possible, use hybrid load testing

Execution considerations

Run your tests in the appropriate environment

Run tests where your customers are

Recommendations

Distributed Tracing With Grafana k6 (k6 Office Hours #70)

Resources

All About Load Testing Metrics (k6 Office Hours #69)

Assertions in k6 (k6 Office Hours #68)

Related articles

Load Testing Types and k6 Executors (k6 Office Hours #67)

All About Web Performance (k6 Office Hours #66)

Performance testing for beginners (k6 Office Hours #65)

The future of k6 (k6 Office Hours #64)

Getting started with xk6-browser (k6 Office Hours #63)

Load Testing Apache Kafka using k6 (k6 Office Hours #62)

Resilience and Chaos Testing with SteadyBit and k6 (k6 Office Hours #61)

What Is Grafana Mimir? (k6 Office Hours #53)

API load testing using k6, with Tom Miseur (k6 Office Hours #52)

End-to-end browser testing using xk6-browser, with Thomas Wikman (k6 Office Hours #51)

Load testing results in the k6 Cloud App for Grafana, with Edgar Fisher (k6 Office Hours #49)

Building Reliable Systems With k6 (k6 Office Hours #48)

How to Analyze Load Testing Results with k6 (k6 Office Hours #47

Load Testing for DevOps with Victor Farcic (k6 Office Hours #45)

Private Load Zones for Load Testing (k6 Office Hours #44)

In search of the best Pokémon: Browser automation and load testing in one script with k6 (Automation Guild 2022)