Ask a Flooder on Nicole van der Hoeven

Ask a Flooder 20: How can fear affect application performance?

Thu, 06 Aug 2020 22:43:28 +0100

Originally posted here.

In this video, I talk about the power of fear, and how it can end up affecting performance test results. ‍

Or listen to the audio version here:

Hi everyone, I’m Nicole van der Hoeven, back with another Ask a Flooder, and today I want to talk about the effect of fear on your performance outcomes.

The Dutch “coronalijn”

I currently live in Maastricht, in the Netherlands. At the beginning of this global pandemic, the only people getting tested actively for COVID-19 were either people who presented with serious symptoms of the disease, or health care professionals who were continually exposed to the disease and therefore at a higher risk of contracting it.

Last month, on June 1st, the Dutch government made testing available to everybody. They set up a telephone hotline that any Dutch resident could call to set an appointment to get tested for a COVID-19 test. A task force was dispatched to make sure that they could test up to 30.000 people per day. Yet on the first day of testing, only 1.146 people were tested. So what happened?

Well, it turned out that the bottleneck of the entire process wasn’t getting enough tests, registering the tests, or even processing the test results. The bottleneck actually occurred way before that, when people called up the hotline to get an appointment.

323.000 people called on the first day that the hotline opened. And not even all of them had called to make an appointment. Some of them were just calling for general information on the coronavirus despite the fact that the government made it clear that that’s not what the line was for, and that there were other lines dedicated to information. There were reports on Twitter of people waiting up to 6 hours on the line just to talk to someone who would make an appointment for them. There were calls that got dropped. People said that sometimes it happened in the middle of a call. Even telephone operators that the appointment system was down or slow and unusable for a lot of it.

As a result, only 5,500 people actually had appointments booked on a day where there were 323,000 that called. That’s 1.70%.

What does that have to do with performance?

As performance engineers, we like to believe that everything we do is reasoned. We design tests based on quantitative data. We look at historical trends and metrics to help us decide how to build a workload model. But the truth is that that’s often not enough.

If anything, this pandemic has shown us just how wrong our guesses can be, whether or not they’re informed by facts, especially when it has to do with human behavior. We’ve seen violent fights erupt over toilet paper. We’ve seen peaks and troughs on the stock market that didn’t have anything to do with the underlying companies. We’ve seen arson on 5G cell towers. Because human beings are irrational, and it’s really hard to capture that irrationality in our model for how users are going to behave.

We may never be able to accurate predict how irrationality is going to affect the load on our application in production. But that doesn’t mean that we shouldn’t try. There are still things that we can do to mitigate the risk. I think that any application that is public-facing should be assessed for exposure to the risk of fear or other human psychological factors.

How do we account for fear in our performance testing?

I actually wrote a whole blog post on this particular situation (the Dutch corona line), and how it might have been possible to predict (maybe not with 100% accuracy but with reasonable accuracy) how many people would have called the hotline.

For example, we can look at statistics like the of the Netherlands or how many people normally get flu-like symptoms or at least report them at this time of the year. These are statistics that are readily available from the Department of Health, and we could have used them to inform our decisions on the workload model for such an application. We may not have gotten it exactly, but we would have at least bin in the same ballpark.

If we have a good enough idea about the workload that our application would have to handle when it’s influenced by fear, we can start running load tests around that. Another good open-ended kind of test to run, especially for applications that are particularly susceptible to fear, is the stress test. In a stress test, you push your application to its limits and you find out what they are. You could, for instance, start with your peak work load, and then add a certain number of users every 30 minutes and see how much your application can actually handle. You can do the same with soak tests, where you’re exposing your application to the same amount of load for an extended period of time. Both of those kind of exploratory tests help you define the limits of your application and how it would respond if fear plays a role in the workload model in production.

If we overlook this critical human element in our test design, then we leave ourselves, and our applications, vulnerable to performance degradation and, perhaps, outages.

Till next time, happy flooding!

Read this blog post for more about the coronavirus hotline.

Ask a Flooder 18: How do I load test video streaming with JMeter?

Wed, 22 Jul 2020 22:49:15 +0100

Originally posted here.

In this video, I answer the question: “How do I load test video streaming with JMeter?”

Or listen to the audio version here:

HTTP Live Streaming (HLS) and performance

One of the most popular protocols for video streaming is HTTP Live Streaming, or HLS, and that’s what I’m going to show you today. An HLS video ends with the [file extension .m3u8‍](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8). According to the HLS protocol, when a video is streamed, the first request sent must be a manifest. A manifest is sometimes called a playlist, and it is a list of all the chunks or parts that make up the video. Then, each chunk is downloaded.

To assess the performance of a video stream, we still want the response time of the manifest, the chunks, and the whole video. However, there are many more metrics that are important in video streaming which aren’t captured by just the response time. For example, how do you determine whether a video was played smoothly or if it stopped to buffer? A video that keeps pausing might just drive a user away, and it’s difficult to tell whether that is happening just from the response time.

How to load test video streaming with JMeter

Downloading and installing the Ubik Video Streaming plugin

Luckily, there’s a plugin for that, created by a company called Ubik Ingénierie. The Ubik Load Pack is a set of enterprise-grade plugins for JMeter. One of those plugins is a Video Streaming Plugin, and it happens to be a really easy way to get started with video streaming. This is a paid plugin, but you can sign up for a free 11-day trial.

Once you do that, you’ll get an email from Ubik with a link to download the bundle. Go to lib/ext and then copy this .jar file into the same directory on your JMeter installation. Start up JMeter, and it’ll ask you for the license file. Browse to the license file in the bundle you downloaded. JMeter will restart.

Using the JMeter plugin in your load testing script

Now let’s start creating the Test Plan. Create a Thread group, and then right click on the thread group, go to Add > Sampler and then select the “ULP Video Streaming sampler”.

Click on the sampler and then paste in the URL to the HLS file that you’d like to test. This should end in a .m3u8. This plugin supports several protocols for video streaming, but we can just leave it at auto and it’ll automatically detect which one it is.

In Play Options, you can choose whether to play the entire video or just a part of it, or even whether you want to include the video subtitles in your test.

This Network Options panel is fantastic because it allows you to limit your network speed. Slow networks, like mobile networks, really affect user experience, so it’s really important to take this into account when running load tests. You might even use a parameter here and do something like throttle to 3G speeds for 80% of your traffic, and leave it on Auto 20% of the time. But I’ll just leave this blank for now.

Next, let’s add a View Results Tree listener. Then, let’s run the script once, just to see if it works.

As you can see, we can see the manifest retrieved first, and then all the chunks. We didn’t get any errors, so let’s run this as a load test. We can disable this listener because it just consumes extra resources on our load generator.

Ramping up your load test on Flood

To ramp this up into a load test, we’ll have to go into Thread Group and change these settings. Ideally, we want to be able to change these values from the Flood UI later so we don’t have to go into the script whenever we change them, so I’ll type in these Flood-specific properties. I’ll leave a link in the description about these properties. I’ll also set it to run Forever so that we can change the duration and the video will just keep playing to fill that duration.

To capture some of Ubik’s cool metrics, let’s add a Simple Data Writer listener here and write the results to a CSV file. On Flood, all flood results are stored on /data/flood/results, so we’ll add an output.csv file in that directory that we can download after the test.

So that’s our script! Now we’ll have to include the plugin and related files so that Flood can access them as well.

Let’s go back to the bundle we downloaded and go to the bin folder to look for user.properties. Let’s open that up, and we’ll change the license path to /data/flood/files, which is the directory that Flood saves files we upload to. To preserve this directory structure, we’ll compress just the lib and bin folders.

On Flood, we’ll start a new stream here and upload three things: the jmx script, the license, and the compressed archive of the lib and bin folders from the bundle.

Then we’ll go through the rest of the process here, setting the test parameters. Time to launch the test!

Analyzing video streaming load test results

Flood shows you results of your load test while it’s running as well as after the test. You can see metrics like concurrency, network throughput, transaction rate, and response time for the test. You can also see these metrics for each individual transaction, including the video or sampler as a whole, the manifest, and the chunks.

To see the CSV that our script generated, we can download the Archived Results from a flood.

In JMeter, click on Tools > Generate Report and fill in the fields, selecting the output.csv from the flood’s Archived Results, JMeter’s user.properties file, and then an output directory for the report.

Ubik’s plugin creates a few custom graphs in the HTML report:

Buffer fill in Millis
Lag ratio in percentage
Lag ratio without buffer fill in percentage
Lag time in Millis

Conclusion

Using Ubik’s Video Streaming plugin is an easy and thorough way to load test video streaming with JMeter, and I highly recommend it. I used an HLS video to demonstrate it, but this plugin supports several protocols:

HLS
Mpeg DASH
Smooth
Adobe HDS

For more information, read Ubik’s tutorial on how to use their plugin with Flood here.

Ask a Flooder 17: How do I use unique test data on Element?

Thu, 18 Jun 2020 21:58:49 +0100

Originally posted on Tricentis Flood.

In this video, I talk about how to ensure that Flood Element uses unique test data, using the example of a customer wanting to script unique logins for his application.

Or listen to the audio version instead:

Transcript:

Hi everyone, Nicole van der Hoeven here again back with another Ask a Flooder, and this week’s question is a follow-up question from last week. That user had previously asked about test data in Flood Element. This week he’s asking: “How do I ensure that Flood Element uses unique test data?”

So this user is actually trying to script a login for an application. He has a list of usernames and corresponding passwords that he wants to use. He’s already set both of these up in a CSV file, and he’s also set up the Element script in the same way that I outline in my previous video. However, this user is seeing errors in the test due to the application not allowing a given user account to be used simultaneously in two or more devices. The problem is that the way that he’s set it up is that he has Element running on multiple nodes, and each node has the chance to pick the same user account.

So how do you make sure that it’s a unique 1:1 ratio for an instance of Element and a user account in the CSV file? If you haven’t already watched the previous week’s video, you should pause this, watch that, and then come back here, so it makes a little more sense because the script that I’m using is building on what we already did in the previous week.

Let’s talk about the differences between this script and the previous week’s.

First, I’m declaring a global variable here called globalBrowserID. The idea is to create a unique and globally identifiable string that I can attach to every account in the CSV file.

const globalBrowserID = ENV.FLOOD_GRID_INDEX + '_' + ENV.FLOOD_GRID_NODE_SEQUENCE_ID + '_' + ENV.BROWSER_ID.toString()

The global browser ID consists of three parts:- The first is a grid index, which is a number assigned to every grid, and it’s also globally unique. So if there are three grids being used in the test, then the grid indexes would be 0, 1, and 2.- The second one is a grid node sequence ID, and this is unique within the grid. The node sequence ID is a number that is assigned to every node in a single grid.- The last one is a browser ID. The browser ID is a number assigned to an instance of Element running on a single node.

These are all zero-based, so the very first instance of Element running on the very first node and on the very first grid in the entire test would be 0_0_0.

We’re going to have to modify our CSV file to make use of this browser ID. So whereas the user previously had username and password, I’ve added an ID column. From then, these will have to be sequential. So 0_0_0, 0_0_1... And 0_0_1 would be the second instance of Element running still on the first node, on the first grid. And you can make as many of these as you need.

id,username,password
0_0_0,user1,password1
0_0_1,user2,password2
0_0_2,user3,password3
1_0_0,user4,password4
1_0_1,user5,password5

I’ve also modified the interface UserData to describe the fields in the CSV file. So there’s id, username, and password, and I’ve defined them all as strings.

TestData.fromCSV<UserData>('users_multiplegrids.csv')
 .filter((line, index, browserID) => line.id === globalBrowserID)
 .circular()

This line is telling Element to take this data file with this filename and filter it based on the line, the index, and the browserID such that id is equal to the browser ID. So what happens here is that Element will take the global browser ID, which let’s say is 0_0_0 for the very first one. It’ll then take that 0_0_0 and look at the CSV file for anything in the ID field that has the value 0_0_0. And then it will continue to use the rest of that line.

I’ve also made the CSV file circular just because that’s always a good practice to get into.

And as always, I think it’s a great idea to debug this, especially when you’re dealing with multiple grids and multiple nodes on that grid.

step('Print to console', async (browser: Browser, data: UserData) => {
 console.log('global browser ID: ', globalBrowserID, '| user: ', `${data.username}`, ' | password: ', `${data.password}`)    
})

I’ve also added the username and password here, just in case we need to verify the state that that user is in.

So let’s run the test. Since we’re just running this locally, we’ll see that the first value indeed is 0_0_0, and the user we’re selecting is user1 and password1, which lines up with the username and password values corresponding to 0_0_0 in the CSV file.

I hope that helped, and until next time, happy flooding!

Download the full script and the data file to follow along.

Ask a Flooder 16: How do I use test data with Element?

Tue, 09 Jun 2020 22:18:41 +0100

Originally posted on Tricentis Flood.

In this video, I discuss how to make your load testing scripts more realistic by including test data. The load testing tool I use is Flood Element, but you can apply these principles to any tool. You’ll find a copy of the Element script I used below.

‍

And, if you prefer, here’s the audio version:

Transcript

Hi everyone, Nicole van der Hoeven here back with another Ask a Flooder, and this time the question is: How do I use test data with Flood Element?

Now let’s talk about why you might want to use test data. In load testing, it’s important to make your scripts behave as similarly to real users as possible so that the load that you generate is production-like. The problem is that if you send the same data over and over again to the server, that server is probably going to cache that response. The response being cached means that the server will return the response way faster than it normally would if it had been a unique request. So we can use test data, instead of static data hardcoded in your scripts, to make sure that this doesn’t happen.

Using test data can also help us test different responses. For example, let’s say we’re testing a loan application system, where a user inputs the application ID, the server looks it up, and returns the status of that loan application. Now, loan applications might be in different stages of the process. For instance, some might be saved but not submitted, or maybe they’ve been submitted but failed the identity verification check.

If we used the same application ID every time, then the server would always return the same status and the same details. Instead, let’s create a CSV file that we can use as test data. It might look something like this.‍

applicationId
A100001
B100002
C100003
D100004

We can have applicationId as the single header here, and we can populate this with a list of different application IDs in different stages of the process. That way, when we run the script, the server will have to respond authentically to every request without it being cached.

Now to create our script. This is an empty Element script. To start with, make sure you’re importing TestData here in the first line of your script.

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'

The first thing we’re going to need to do is to create an interface for the CSV file. Let’s call it AppData. Within this interface, we’ll need to define all the fields that we intend to use in our data file and what type of values we can expect to be in those fields. Looking at our CSV here, we’ve just got the application ID, so I’m just going to copy that, and it’s a mixture of characters and numbers, so I’m going to put that in as a string.

interface AppData {
 applicationId : string    
}

Now we need to tell Element how to find this test data. We’ll say that it’s from the CSV, using the interface we’ve just created. Now we’ll tell Element the filename. In this situation, the CSV file that I’m using is actually in the same directory, so I don’t need to specify a path here, but if you’re putting it somewhere else, then you’ll need to specify the folder and the path to it.

TestData.fromCSV<AppData>('applicationids.csv')

Now let’s use the data in a step. In real life, you might want to use an application ID to type it into a text field, but for demonstration purposes, I’m just going to show you how to output it to the console log. Printing to the console is also a good way to debug your tests, especially if you’re dealing with test data.

We need to let Element know that we want to use AppData. So now, let’s output this to the console. Since we’re just debugging, let’s output the application ID. To actually get the value from the data file, we’ll use this syntax:

step('Print to console', async (browser: Browser, data: AppData) => {
 console.log('application ID: ', `${data.applicationId}`) 
})

‍By default, Element reads the data files sequentially. The problem with that is we only have a handful of application IDs. So if we wanted to run this one script 100 times, we’re going to run out of data. What we can do here is add circular. That way, when the script reaches the very last value of the file, the next value it selects will be the first one, and it never runs out of test data.

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

Another thing you might be wondering is how to make it so that Element chooses a random value every time. To do that, we’ll also have to add shuffle:

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

‍Now let’s try to run the test.

element run test-data.ts

As you can see, it’s been finished. The application ID that was selected was indeed a random one because the first one would have started with A. So, it actually chose the last one randomly.

And that’s it for test data with Element. Now, this user actually had a follow-up question, which was about unique test data. I’m going to answer that in the next video. Until next time, happy Flooding!

Sample script

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'
import * as assert from 'assert'

export const settings: TestSettings = {    
 loopCount: -1,    
 description: 'Test Data',    
 screenshotOnFailure: true,    
 disableCache: true,    
 clearCache: true,    
 clearCookies: true,
}

/*** Test Data * Version: 1.0 */

export default () => {    
 interface AppData {        
 applicationId : string    }    
 TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()    
 .shuffle()    
 step('Print to console', async (browser: Browser, data: AppData) => {        
 console.log('application ID: ', `${data.applicationId}`)    
 })
}

Ask a Flooder 15: How do I install Element?

Tue, 12 May 2020 22:45:18 +0100

Originally posted here.

In this Flood Element tutorial, I demonstrate the steps required to install Flood Element, including dependencies it requires. Flood Element is an open-source browser-based load testing tool created by Tricentis Flood.

Learn more about Flood Element

Installation instructions for NodeJS

Installation instructions for Yarn

Installation instructions for Element

Video: Getting Started with Element

Guides: Using Element with Flood‍

Audio version here:

Transcript

Hi everyone, Nicole van der Hoeven here again, back with another Ask a Flooder, and today’s question is: “How do I install Element?”

Now, I already made a previous video on how to get started with Element, but I didn’t actually go into the details of how to install it, so that’s what I’m going to do in this video today. There are actually a few parts to installing Element, and all of these instructions are going to be on element.flood.io. If you go onto the site and click on Install, you’ll see all of the instructions depending on which operating system you’re on.

First, you need to install NodeJS. This is sort of like how, if you’re familiar with JMeter, you have to install Java before installing JMeter. NodeJS is what Element runs on. Secondly, you’ll need a package manager. You can use NPM or Yarn, but in this video, I’m going to take you through how to do it with Yarn. And thirdly, there’s installing Element itself.

First, download NodeJS. I’m going to leave links in the description to all of the sites that I visit so that you can go through all of the instructions later. On the Node site, download whichever installer suits your operating system. Go through the installation steps, and afterwards, check to make sure that NodeJS is properly installed on your machine. You can do that by going into the terminal and typing:

node -v

Just make sure that the version that’s shown here is the same version that you’ve downloaded.

Secondly, install the latest version of Yarn. On the Yarn site, you’re going to have to choose your operating system and the version. We recommend using the latest stable version. And then go through the instructions for your operating system. After you’ve done that, check your Yarn installation by typing:

yarn --version

And the last step is installing Element. On the Element site, you can find the exact code that you need to type into your terminal. It’s:

yarn global add @flood/element-cli

So you can just copy that into your terminal. As of the time of this recording, the latest version of Element is 1.2.3. Looks like that’s installed. To check the version of Element, type:

element --version

So that’s how to install Element. We’ve gotten a lot of questions recently about Element in particular, so I’m going to answering them in future videos. Hope this helped, and until next time, happy flooding!

Ask a Flooder 13: How do you do mobile load testing?

Tue, 28 Apr 2020 23:21:08 +0100

Originally posted here.

How do you load test a native mobile app? I discuss the general approach to mobile load testing, what a device farm is, and why a device farm isn’t necessary for load testing. I also discuss the best mobile load testing tools to use.

Or listen to the audio version here:

Hi everyone, Nicole here again, back with another Ask a Flooder, and today’s question is:

How do you do mobile load testing?

Now, this question came from a customer who was under the misconception that in order to do mobile load testing, they would need to use what we call a device farm. A device farm is a collection of different kinds of mobile phones. You would ideally have the most popular makes of mobile phones (the hardware models) and the operating systems and system versions. You can imagine that this can very easily get a little out of hand to coordinate because of fragmentation. There are just so many devices out on the market, and then there’s the question of how you actually generate the load from that.

You would somehow need some sort of test automation script that was running and then coordinate os that they all run at the same time, and you would use that as a load test. As you can imagine, whether you use real devices or simulators, this kind of thing would get complicated and expensive very quickly. But the good news is that it’s unnecessary if it’s load testing that you want to do. Now I say that because, for functional testing or compatibility testing, you ARE going to want to test your application on a variety of different browsers. But with load testing, you’re really more concerned about the back end part of the equation. You want to generate load on your application servers. In reality, most mobile apps work by sending API requests to that server, usually by some sort of restful API. Luckily, API load testing has been around for a while, so we do have some industry-tested ways to run this.

What are the best tools for API load testing?

Essentially, you’re going to want to test protocol-level tools. That means that you don’t want to be using browser-level tools in which you’re using scripts to drive a full browser. All you really need is something that sends requests on the protocol-level. That’s great because it’s actually way more efficient.

You have two really good open-source options for this, JMeter and Gatling. I’ve already gone a little bit into the differences between these two in my previous video on how you choose a tool for load testing, which I will leave a link to.

And here are some other resources that will help you get started with mobile load testing. I also have another video about how to record network traffic from a native mobile app on JMeter, which sounds a little complicated, but I promise it’s not. In that video, I go over the steps with you, and I’m actually doing it in real-time with you as well. I show you the screen recording from my mobile and from JMeter on my laptop, so you can always pause and watch it at your own pace.

The second thing is that I already wrote a blog post, which is sort of a guide for mobile load testing, which goes into the steps into way more detail than I can here.

And lastly, I have written blog posts about two real-life examples of mobile load testing. The first one was the Iowa Caucus Reporter app. I ran through it from the workload modeling to the scripting and then the actual execution. More recently, I also wrote about the Robinhood app. Be sure to check out the links in the description to find out more.

Until next time, happy flooding!

Ask a Flooder 12: Puppeteer vs Selenium: which one is better for load testing?

Tue, 21 Apr 2020 17:36:28 +0100

Originally posted here.

When it comes to software testing, the debate of Puppeteer vs Selenium is a hot one. Nicole tackles it here to determine which one is better for load testing in particular. Spoiler: The answer is still “it depends”.

Or catch the audio version here:

Hi everyone, Nicole here again, back for another Ask a Flooder, and today, the question is: “Puppeteer vs. Selenium: which one is better for load testing?”

We’re actually getting this quite a bit now as Puppeteer grows in popularity, but first, let’s discuss the similarities. They’re tools that you can use to drive browsers. The work by identifying and interacting with elements on a website in the same way that a user would. They are also both open-source, and both can be used for load testing. This is a great strategy because instead of writing tests for an automation suite and then tests for load testing, you can just write the one test suite and re-use it, so it saves a lot of resources in the end. To answer this question, I’m going to focus on load testing because that’s what my experience is in.

Difference between Selenium and Puppeteer

Pros and cons of Selenium

Now for the difference. Selenium supports more browsers than does Puppeteer. It supports Chrome, Firefox, IE, Safari, and Opera, so there are a lot more options to choose from depending on your application.

Selenium is also what I would call automation-first. It was built with automation in mind, which means it’s very good at that. It’s very robust, and if you have a complicated test scenario, Selenium is an excellent choice.

It’s also been around for much longer than Puppeteer, which is important because that means it’s had more time to build up a community. There are way more resources and tutorials for Selenium than for Puppeteer.

It’s also, unfortunately, very resource-intensive. That’s not something that you would typically care too much about from a test automation perspective, but when you’re running thousands of users on the cloud, it IS important to keep in mind that Selenium, just because of how it’s built and its complexity, can only run about five users per node. We’ve found that out at Flood, after baselining specifically an AWS m5.xlarge instance. So you can look at the specs of that and compare it with the machines that you’re wanting to run Selenium on, to see how that would equate for you.

Pros and cons of Puppeteer

Puppeteer is a NodeJS library developed by Google, which means it plays very well with Chrome. There are a whole lot of things that you have access to with Puppeteer, and if you’re using it with Chrome, pretty much anything in Chrome, including rendering times, you can see and access with a Puppeteer script. So that’s a huge advantage.

Unfortunately, that also means it supports fewer browsers. Obviously, it supports Chrome very well, and they recently came out with support for Firefox, although they do have plans to incorporate support for more browsers in the future.

One of Puppeteer’s advantages is that we at Flood actually created a tool on top of Puppeteer. It’s also open-source. It’s called Flood Element, and Element is built to be performance-first because obviously, that’s what we had in mind when we were building it. It takes everything that’s good about Puppeteer and makes it really easy to set parameters that you normally would expect to need for load testing, like users and ramp-up.

This also means that it’s way less resource-intensive. I said that with Selenium, you can run up to five users on an AWS m5.xlarge instance, and for the same instance type, you can run, from our experiments, anywhere from 30-50 Element scripts/users. That is a huge saving when you’re running a really large-scale test.

Should you choose Selenium or Puppeteer for load testing?

I would say that if you already have a large Selenium automation suite, if you have a complex test scenario, or if you aren’t running that many users for your load test, then Selenium is a great choice. If you ARE running thousands of users, though, and resource efficiency, and therefore cost efficiency, is really important to you, then choose Element.

My advice is always to try both. If you have the time, do a proof of concept with both of them. That way, your whole team can better understand what the pros and cons are for each one and decide which one is best going forward. Or maybe you don’t even decide on one. At Flood, we support both of them at the same price, so it depends on what you prefer.

Until next time, happy flooding!

Ask a Flooder 11: What are the best JMeter plugins?

Tue, 14 Apr 2020 19:47:53 +0100

Originally posted here.

In this video, I talk about how to install the JMeter Plugin Manager as well as other plugins, what the best JMeter plugins are, and why you need them. JMeter is an open-source load testing tool that is already robust, but there are some fantastic plugins that have been developed for it that you might want to use in your next load test.

Audio version here:

Hi everyone, Nicole here, back again for another Ask a Flooder, and I hope everybody is staying alert, not anxious, during these crazy times. In today’s Ask a Flooder, our customer asks: “What are the best JMeter plugins?”

What is a JMeter plugin?

So first, I want to talk about what a JMeter plugin is and how you can actually install one. Apache JMeter is a really great, robust load testing that is one of my personal favorites. There are still some things that could be done better or could be extended, and the great thing about being an open-source tool is that anybody, pretty much, can just jump in and develop these plugins for JMeter to extend functionality, so that it supports more protocols or to customize load profiles in a way that is easier. We already have a blog post on this, and I will leave a link to that article as well in the description because it goes into a lot more detail. However, I still want to give you my personal top 3 JMeter plugins.

How do you install a JMeter plugin?

You can install a plugin for JMeter in two ways.

One is to download it from JMeter Plugins, which I will give you a link to, and then you paste that into either the lib — it’s usually in the lib folder, but it could be in the ext (extensions) folder. Each plugin is going to differ as to what you need to copy into what folder, but that’s why I prefer the second method, which is the Plugin Manager.

The Plugin Manager is a plugin itself, so you will initially need to do that copy and pasting of the jar file. Once you have the JMeter Plugin Manager installed, downloading and installing a plugin is as simple as choosing which ones you want and then clicking install. JMeter restarts, and that’s pretty much it. So I do recommend that the Plugin Manager be the first plugin that you install. But that’s still not my top 3 favorite - that’s just an optional step.

What are the top 3 JMeter plugins?

Custom Thread Groups

Now for the actual plugins. My #1 favorite is the Custom Thread Groups. See, with JMeter, you get the standard default thread group, and all that does is it asks you how many users you want to run, how many loops, the duration, and the ramp-up. That’s it. So that’s good for a lot of uses, but it doesn’t really help you in cases where maybe you want to plan a spike test, and you want to make sure that all the users have fully ramped up and gotten to a certain point, and then all hit the server at once. Or maybe you want to do a step load. Previously I’ve discussed stress testing and how a step load can be useful for that. In that case, the Thread Group doesn’t quite cut it because it only ramps up a user, and then it continues at the same level of load for the duration of the test.

So, if you want any sort of variation in the middle (increasing users or decreasing), then you’re going to want something else.The Custom Thread Groups plugin has a lot of different types of thread groups. My favorite is the Ultimate One because you can probably do almost anything on that one. It also has the Stepping, Concurrency, Arrivals, and Free-Form Arrivals, so you have quite a few options.

Throughput Shaping Timer

Plugin #2 is the Throughput Shaping Timer. I love it because not a lot of load testing tools, even the proprietary ones, actually have this. This is really handy when you have a project where you’re given a requirement that isn’t in terms of the number of users; it’s in terms of hits per second, or requests per second, or something to do with the actual throughput of the test.

The problem with that is that the throughput tends to change, so you can’t actually take as a given that it’ll remain constant for the duration of the test. Things like response time or latency are going to affect that throughput. For instance, when you have one user or one thread that’s running, and it sends a request and receives the response back in one second, you can’t necessarily say that that’s going to be the same when there are a thousand threads doing the same thing. The server’s under more load, so maybe it’ll take five seconds instead of one second— but that’s already changed your throughput. Before, one thread was doing one request per second, and now it’s doing one request every five seconds.

So it can get a little complicated, and the Throughput Shaping Timer is one of my favorites because it promises to solve all that. The Throughput Shaping Timer dynamically changes the think time, so at the beginning of a test, for instance, when there are no response time issues and requests are being returned really quickly, it might increase the think time so that it slows down your load test. So you’re not going over that throughput that you wanted to maintain. And then later on in the test, when all the users are ramped up, then it’ll decrease the think time that it uses. So it’s great because you just have this consistent throughput throughout your test. It’s a really intelligent way to test.

I do want to say, though, that you do also want to play around with this a bit and make sure you understand how this works. One page could have several requests in it, and you’ll want to take that into consideration when you’re choosing what to set in that timer.

HTTP/2 Request Sampler

My #3 is the HTTP/2 Sampler. If you don’t know what HTTP/2 is, it’s an upgraded or next-generation protocol to the normal HTTP that we already use. The HTTP/2 protocol is supposed to be faster and more efficient, so more and more web apps are switching to it because of that.

The problem is that JMeter’s HTTP request sampler is only really for HTTP sites, so the HTTP/2 sampler is an upgraded version of the HTTP sampler. It’s really easy to use. The experience is pretty much going to be the same for you, but in the background, it uses the faster HTTP/2 protocol rather than the slower one.

My colleague, Jason Rizio, actually wrote a whole blog post about this, so if you want to learn more, I will put a link in the description.

I hope that answered your question, and until next time, happy flooding!

Ask a Flooder 10: What is the difference between load testing and stress testing?

Mon, 06 Apr 2020 20:26:24 +0100

Originally posted here.

What’s the difference between load testing and stress testing? When should you choose to run a load test on your application, and when is a stress test more appropriate? Well, it’s all in the intention.

‍

Hi everyone, Nicole here again, back with another Ask a Flooder, and this time the question is:

What is the difference between load testing and stress testing?

Okay, again, this is a matter of semantics, but I think the difference is in the intention. Load testing and stress testing are both types of back-end performance testing, and I did also make a video on it that I’ll link in the description down below.

Load testing is about seeing how your server reacts to finite load

Now, with load testing, you usually have to start with a specific idea of the load profile that you want to test. You get to this number by looking through your Google Analytics or other analytics on your site to see what load you need to test at, and then the intention is to see how your server responds to that load. So the load is fixed: you know WHAT you want to test, and you just want to see how the server reacts to it. So it’s more finite in scope. In general, it’s also more about how your server reacts to normal or expected load, whether that’s now or in the future.

Stress testing explores your application’s upper limits

Stress testing, on the other hand, is a little bit more open-ended and a little bit more destructive. So instead of starting with how much load you want to put on [your server], in stress testing, what you generally do is you increase the load. You usually use a step load profile: so you start a certain number of users, you wait maybe 30 minutes, and then you increase it again. And you end up with this staircase kind of load profile. What you’re doing there is you just let it increase until your server fails because the intention of stress testing is to get your server to fail. And, once it fails, does it fail gracefully? Does it show an error message? These are really things that test the upper limits of your server, and that’s what stress testing really is.

They can be interchangeable

At Flood, we tend to use the word “load” testing to refer to both kinds of testing because both of them entail applying load on your application server. It’s not really that important to distinguish between one or another; what’s [important] is that your team uses the same definition so that when YOU say “load testing,” they know what you mean.

Until the next time, happy flooding!

Resources

Video on performance testing vs load testing

Real-life stress testing in action: Robinhood app failure

Ask a Flooder 09: What are the best tools for load testing?

Tue, 31 Mar 2020 20:35:36 +0100

Originally posted here.

One of our most commonly-asked questions is: “What are the best tools for load testing?” The answer will depend on a few things: your reason for load testing, the application you want to test, what kind of load testing you want to carry out, your team’s experience with scripting, and how many virtual users you want to run.

Note: In this video, I say that traditional load testing tools can’t simulate requests that are triggered by interactions on the client-side. This isn’t technically true: if you know what the requests are, you can build those requests manually on a protocol-level tool (like JMeter or Gatling). However, this takes more work compared to browser-level tools that just do this natively. ‍

Transcript:

Hi everyone, Nicole back again with another Ask a Flooder, and this time I’m tackling one of the questions we get asked A LOT, which is:

What are the best tools for load testing?

Why open-source load testing tools are awesome

Now, I want to preface this by saying that we really love open-source tools here at Flood, because 1) they’re really pretty full-fledged these days. You don’t have to make any compromises on features, and, in fact, some of these open-source tools have more features than some newer load testing tools just because they’ve been out longer. And secondly, they have large communities. Because they’re open-source, lots of people are using it, and lots of people are developing for it. So there are lots of plugins normally, and they support any kind of protocol that you can think of. And lastly, cost. It’s hard not to mention cost because it can be a really important factor depending on the budget of your project. And if you’re starting out as a load tester, I think that it might even be the best way to start with load testing, because you can take that knowledge with you and apply it to other projects and other companies regardless of the budget that they have.

Questions to ask yourself in choosing a load testing tool

Why are you doing load testing?

I always advocate starting with WHY. Why are you actually load testing? Is there a particular production defect that you noticed that you’re trying to solve? Is there already a problem, or are you just preparing to launch? These are things that are going to inform the type of tool that you use because you might want to make sure that that tool supports the protocols that you need.

What are you load testing?

Secondly, what type of app are you testing? Because if you have a single-page app, where your end-user is interacting with the page and client-side scripts are running to make requests to servers, then that’s something that most traditional load testing tools aren’t going to be able to do.

There are two general types of load testing tools: there’s browser-based, and there’s protocol-based. I’ll leave a link in the description talking about the differences between the two, but in general, if you want to go for browser-level testing, I would suggest Selenium and Element. Now, they’re both very good options. Selenium is probably the more robust in terms of automation of the two, but Element is better one in terms of efficiency because it was built with performance in mind. So Selenium’s automation-first and Element is really performance-first.

Now, for protocol-level tools, you can’t go wrong with either JMeter or Gatling.

What experience does your team have with scripting?

Thirdly, what’s the level of technical expertise that your team has? I think it’s important to be realistic about what your team can do, given the amount of time that they have. So if you have a team that has never done load testing and has never done any scripting before, then I would suggest using our Test Builder or something similar. On Flood, we have a Test Builder that is really just an easy interface that actually runs JMeter under the hood, but it just kind of simplifies JMeter and boils it down to what are the URLs that you want to test. That is a really easy way to get started, and it’s also a great [way to run a] proof of concept. If you have some automation testers on your team that are trying to do the load testing, well, maybe you should stick to tools that they already use. So it might be easier for them to get up and running with either Selenium or Element rather than learning a new tool. And if you have developers on your team, you might consider Ruby JMeter rather than the vanilla JMeter because Ruby JMeter is a lot easier to deal with— because you don’t have to deal with the XML files of a JMeter file (a normal JMX script). You can just plan out your load tests in Ruby. Or, Gatling is also a really good option because it allows you to write scripts in Scala. Scala is really really powerful, and you can create an entire framework with Gatling, which is also something to keep in mind.

How many virtual users do you want to run in your load tests?

And lastly, how many users do you want to simulate? Now protocol-level tools are going to really shine here because Element and Selenium both require a higher usage of resources on your load generators just because of the complexities involved in starting a browser instance, whereas protocol-level tools are just super efficient at sending these requests. So I would suggest that if you’re talking about thousands of users, I would use the protocol-level tools unless there’s a really good reason why that’s not going to give you the results that you’re looking for. Between JMeter and Gatling– well, they’re both excellent tools, but Gatling has been shown to particularly be good for really high levels of usage. But JMeter is also a good alternative, and you can run thousands of users of JMeter without any problem.

Just get started!

As always, I’d suggest just getting started. Pick whatever tool sounds like it might be the best for you, do a proof of concept on that one, and you can always try the other ones as well. On Flood, we don’t restrict you to just using one tool, so you can use one; you can use all four; you can run them at the same time if you want. It all depends on what you need for your testing.

Till next time, happy flooding!

Ask a Flooder 08: Why is concurrent users an ambiguous measure of throughput?

Thu, 26 Mar 2020 20:41:22 +0100

Originally posted here.

What is the meaning of concurrent users in performance testing? You might have seen concurrent users in JMeter or other tools, but what does it mean and how can you use the term properly?

‍

Transcript:

Hi everyone! Nicole here again for another Ask a Flooder, except this time it’s not really a question– it’s just something I want to talk about.

Source: The Flood Twitter account

So I’m going to put the tweet up here, but last month I actually tweeted about something that somebody else posted. Netflix actually reported some of their viewership figures, and one thing that they reported was that 76 million households watched their new show, The Witcher, which is a large amount of people. So I posted something about why concurrent users is actually an ambiguous way to measure throughput, and I want to expound a little bit more on that.

“Concurrent users” in performance testing - limitations

In load testing, the number of concurrent users is a pretty standard measure of throughput, and generally, it means the number of users that are accessing an application at any one point in time. And even on Flood, we use it because it’s a really good rule of thumb. When we want to get a feel for how big a test is, load testers generally ask, “How many users?” However, we should be aware, as load testers, that that doesn’t always give us the full picture.

What type of user is it?

For instance, the type of user that it is can really change the traffic. Sometimes mobile browsers will optimize requests so that they send fewer because they realize that mobile networks are slower.

What is the user doing?

Secondly, what is the user doing? Are they browsing on different tabs all the time? Or are they just refreshing the content on the same tab? Because if they’re just using the same page and fetching it over and over again, then caching could come into play, so that would create a different load on the system.

How long is the user doing it?

And thirdly, how long is the user doing it? Are we talking about a sustained load for an hour, or is it a shorter time period, like a spike test of five minutes where all the users just ram a system? There are things like the number of requests per second that you need to take into account, which is affected by the think time and the delays that you set in your script. Perhaps your script has a user that just sends a request, then waits for a minute, and then sends another request. That’s very different from a user just sending one request after another as fast as possible.

When does the user do it?

Lastly, it’s also important to take into account the seasonality of the request. When does the user do it? If you have a timesheeting application, for instance, then it might only get traffic on Fridays and a little bit on Monday, so you need to specify, when you report these numbers, whether it’s for the peak time period on Friday - you know, around 3 pm or something - or you’re reporting on figures from a Wednesday when there really wouldn’t be that many people accessing it.

In the case of Netflix, what they counted as a view was only a two-minute watch time. So if you’re one of those people who went into The Witcher and didn’t watch past two minutes, that still counted as a view. So the 76 million users is still not completely accurate, because not all of those people would have watched the entire thing.

Conclusion

Concurrent users is a good way in order to get a quick feel of the throughput that a test outputs, but it really needs to be examined in context along with things like requests per second and the network throughput (kbps) in order to get a really good understanding of how much load a test is generating.

Till next time, happy flooding!

Ask a Flooder 07: What is the difference between performance testing and load testing?

Tue, 24 Mar 2020 20:31:06 +0100

Originally posted here.

In a two-for-one Ask a Flooder, I answer two questions: What is the difference between performance testing and load testing and in which situations is performance testing a better option than load testing?

Transcript:

Hi again, it’s Nicole back for another Ask a Flooder. And this time, the question from one of our customers is:

What is the difference between performance testing and load testing?

Now, this is really a matter of semantics, and it kind of depends on whom you ask, but in general, performance is how quickly an application responds. So when you’re doing performance testing, you’re doing one of two things: there’s front-end performance, and then there’s back-end performance.

In front-end performance, you’re looking at things like how quickly does the application render for the customer on the client-side. It’s how fast something renders on the screen, so it’s how page resources can be rearranged or optimized so that the experience is a lot faster for the end customer. So the focus is on the client-side on the end user’s actual device.

On the back-end performance side, you’re looking at the server. How quickly does the server send a response to a request that’s sent by the browser?

So those are really two different things because back-end performance really looks at everything outside - everything beyond - the device of the customer. Both kinds of performance testing are important. A solid, holistic performance testing strategy will incorporate both front-end and back-end performance.

There are different types of back-end performance testing. You might have heard of load testing, stress testing, volume testing. At Flood, we kind of refer to all of those under the bracket of load testing. They do have some semantic differences between them, but in general, we talk about load testing to refer to simulating traffic against an application server to see how that server responds.

Now the type of the testing that you want to do will also affect the kind of performance testing tool that you choose. For instance, a tool like Flood Element can do both front-end and back-end testing, but a tool like JMeter is only going to be useful for the back-end performance testing. So you need to choose your test according to the testing type that you’d like to carry out.

This customer also had a follow-up question:

In which situations is performance testing a better option than load testing?

And one answer to this is when front-end performance is required. So if what you’re after is trying to speed up how quickly your application renders on a mobile device, for instance, you might be better off looking at tools like GTMetrix or WebPageTest in order to measure and improve that side of it.

I really want to stress here that neither front-end performance nor back-end performance is better than the other. If you’re asking me which type of testing you should do, my answer would be both. Till next time, happy flooding!

Ask a Flooder 06: How to record traffic from a mobile app with JMeter

Thu, 19 Mar 2020 20:50:44 +0100

Originally posted here.

How do you record the traffic from your native mobile app using Apache JMeter? In this video, I explain step-by-step how to begin load testing a native mobile app.

‍

You might want to be doing this because you’re load testing a mobile app but either you don’t have access to your developers right now so they can’t help you with how the requests are made or what the requests are, or maybe you just want to verify what you’ve already been told. Either way, I’m going to show you how using my Android device (it’s a Samsung Galaxy Note 10+) and a Macbook Pro 13". However, the same principles should apply to other mobile devices and laptops.

Step 1: Download and install the mobile app

The first thing you’ll want to do is download and install the app. Now, if the app has already been released, then that’s no problem. You just go to the Google Play Store and download it. However, if it hasn’t been released yet and you’ve been given an APK file for the app, then you’ll have to tweak your settings a little bit in order to get your phone to allow you to download it.

And I’m going to do this right with you. So go into Settings: pull down the little notification screen and hit the little gear icon, and you’ll see a bunch of settings there. Scroll down all the way to the bottom until you see About phone. Click that and then click Software Information. You’ll see the Build number halfway through the screen here– and I know this is weird, but click on it seven times. You have to draw your pattern if you have one set up, but you’ll see here that it says Developer Mode has been enabled. That’s all you need to do; from now on you can install any APK, but be careful, because if you don’t trust the source, you know, they could do some damage to your phone.

Step 2: Set up the HTTP(S) Test Script Recorder on JMeter

The second step is setting up an HTTP recorder on JMeter. So let’s head over to our laptop. Make sure you’re connected to a wifi network. Now, this is a completely new JMeter test plan here. So just right-click on Test Plan and then hover over Add, and then you’re going to go down to Non-Test Elements. Select the HTTP(S) Test Script Recorder. You’ll see here that the port it uses is 8888. We’ll need that for later. Now right-click on the Test Script Recorder, hit Add, and then Listener. We’ll add a View Results Tree to be able to capture the results that we need later. Now we’ll click on the Test Plan - right-click - and then we’re going to Add a Thread Group. Now let’s right-click on the Thread Group and go to Add > Logic Controller > Recording Controller. This is where the requests that we’re going to send are going to be recorded, and we’ll see both the requests and the responses in View Results Tree.

Step 3: Set up your mobile to use the HTTP(S) Test Script Recorder as a proxy

Now, the third step is to set up your mobile to use a proxy. So, back on the mobile, go back to Settings and hit Connections. And then click on Wifi. Now verify that the wifi network that you’re connected to on your mobile is the same one as the wifi network that you’re connected to on your laptop. If it’s the same, click the gear icon. And then hit Advanced all the way at the bottom. Now the Proxy settings are on the second option here so hit the down menu here and then hit Manual. Now, for the Proxy host name we’ll have to go back to the computer to see what our local IP address is.

So, in order to do that, you just have to go to System Preferences. I’m doing it from the Apple menu here and from System Preferences, click on Network. And once you’ve selected the wifi network, hit Advanced, and then TCP/IP. So what you’re looking for will be this IPv4 Address. Now you’re going to type that into the Host name field on your mobile. So let me just type that in.

You’ll note that this is actually a local IP address. It’s not your public IP address. Now for our proxy port, we’ll go and look at the laptop to see the Test Script Recorder Settings and it was 8888, so that’s what we’re going to put in here. All right. And then click on Save.

Step 4: Start recording

Step 4 is recording the traffic. Now, go back to your laptop on JMeter and click on the Test Script Recorder. Now we’re going to click on Start. It’ll come up with this popup box here and you can just click OK since we already trust the source. You’ll see this little popup here to confirm that we are recording traffic. Now let’s see if we’ve actually configured it correctly.

On our mobiles, let’s go back and open the app that you want to use. So I’m going to be using the IowaReporterApp. I just did an article on this on The New Stack (link in the description below) about how I would load test the Iowa Caucus Reporter App. So that’s what I’m going to be using for this tutorial as well. So I’m going to open that (“getting ready to caucus”) and I don’t have a valid precinct ID so I’m just going to put flood.io and I’m going to click Login. So it’s telling me that the connection isn’t private. I don’t actually expect this to work, because the app has been taken down and I don’t have valid login credentials, but I’m still just going to go forward. So, Proceed. You might have to do this as well if your certificates haven’t been set up correctly.

Okay, so now we’ve gotten this error here and it says, “Oops, something went wrong”. So it’s handy to know what it looks like on your mobile so that you can go into JMeter and see if that’s what you see as well.

So on JMeter let’s go to the View Results Tree and it looks like Google Play tried to do a few things beforehand but what we’re really looking for is this IDP caucus thing. So, these are the requests that my phone made and these are the responses. So this is where we got the certificate error because it says “certificate unknown”. Now this is the actual request we sent for login. You can see the request here - the full request - and also the full response. Now, just to check we’ll make sure we got the same response in the body here on JMeter that we saw on our mobile phones.

And here it is; it says “Oops!, something went wrong”. You can of course just search for this. So we’ll stop the recording now because we’ve confirmed that we’ve gotten what we wanted. So in the recording controller, we can see the requests that were recorded from our mobile app and the first one looks like it was one to Google so that’s not one we want. If we were doing this again, we could actually go into the Test Script Recorder, go into this Requests Filtering tab, and we could just filter out or exclude the domains that we don’t want to test, like Google. But this looks like the request that we were actually looking for. You’ll see the full request here with all the parameters and the values.

What’s next?

And that’s how you record traffic from a native mobile app on JMeter. From here, you can then play around with the request. Like we saw, there were a lot of dynamic values that were recorded in the request. We would probably want to correlate those and make sure that we’re doing those correctly. But this is a really great starting point for testing a mobile app.

Till the next Ask a Flooder, happy Flooding!

Ask a Flooder 05: How do I get started with Flood Element?

Tue, 17 Mar 2020 20:58:12 +0100

Originally posted here.

On this Ask a Flooder, I talk about getting started with Flood Element. Element is an open-source tool that is a good Selenium alternative for browser-based load testing. It runs Puppeteer under the hood to drive real browsers at scale and can be paired with Flood for cloud load testing.

‍

TRANSCRIPT:

If you’re sold on browser-based load testing and want to get started on Element, I think the easiest way to do that is by going to element.flood.io and clicking on Install on the upper left corner. You can follow those instructions to get started, and you can also click on the links on the left side if you get stuck.

Once Element is installed, open up the directory and look at the examples folder. There you’ll find a lot of sample scripts that we’ve already made up for you. The easiest way to get started is to just modify some of these slightly so that you’re testing your site. For example, you could just change the URL and step names to make it appropriate for what you’re testing.

Then, you can run it locally using the command element run the name of your test script, like test.ts, and then I also like to include the no-headless flag.

element run test.ts --no-headless

The no-headless flag is going to run a real browser, a Chromium browser, on your machine, so that you can see what the script is actually doing. This is a great way to debug and troubleshoot.

Once you’re ready to run your load test on the cloud, you can simply drag and drop your Element script into the Flood stream editor and then start your load test as usual.

Check out the links in the description below for more information about Element, how to get started, and how to run an Element script on Flood.

More about Element

Getting started with Element

Ask a Flooder 04: Why would your load testing tool report higher response times than you expect?

Wed, 11 Mar 2020 21:02:48 +0100

Originally posted here.

In this Ask a Flooder, I tackle a question we get all the time from users regarding results, particularly for cloud load testing. Sometimes you might notice that response times that are reported in by your load testing tool do not match up with the response time you see from your own browser. What causes this, and how can we fix it?

Transcript:

It’s always confusing when your load testing tool reports higher average response times than what you can see while you’re manually accessing your application on your browser. There are many things that might cause this, but one of the most common causes for it is that you’re overloading your node generators.

As a general rule of thumb, you’ll want to keep your CPU utilization at 80% or less throughout the duration of the test. It is normal for it to spike in the beginning, but it should ideally come to a more stable spot below 80%. If your node’s CPU utilization is going over 80% for most of the test, then what that means is that your node is struggling to get those requests out, which may lead to some inaccurate results.

You can confirm whether or not this is the cause of your high response times by checking the logs and checking the resource utilization on your nodes.If you’re running on Flood, you can do this while the flood is running by going over to Grids, selecting the grid, and then scrolling down on the panel on the right to see the CPU and memory as well as the network throughput.

There are a couple of things you can do to fix this. The first one is to increase any sort of delays or think times in your script — and, if you don’t already have any, you should add some. Think times make the script a lot more realistic and a lot closer to what real users would do. If these values are too low, then your load testing tool is just going to run the script as fast as possible, consuming more resources than necessary.

The second thing you can do is reduce the number of users you’re running per node. This will free up more resources per node and make it a lot easier to scale up your load tests more effectively.

Check out this article for more detailed information on resource over-utilization and other possible reasons for differences between response times in a test and on a browser.