Jmeter on Nicole van der Hoeven

Testing tool cheat sheet for load testers who don't just care about numbers

Mon, 17 Jan 2022 16:30:07 +0100

It’s not enough to decide how much load you want to generate. You also have to know how you’re generating it.

Your approach affects what part of your application is tested, the results returned, and the budget you’ll need to run your test.

Here are three main categories of load testing tools:

Protocol-based load testing tools:

These tools simulate the underlying messages sent back and forth between an application and a client.

Choose a protocol-based tool if: you want to generate massive load, you want to load test before components are integrated, you want to test APIs, or you are budget-conscious.

Best in class tools: Gatling, JMeter, k6, Locust, Neoload

Browser-based load testing tools:

These tools automate real user interactions with an interface.

Choose a browser-based tool if: you want to measure end-to-end performance, you want to reuse your browser automation scripts, you want front-end performance metrics, or you have a higher budget.

Best in class tools: Cypress, Flood Element, k6, Playwright, Puppeteer

Hybrid load testing tools:

This approach is a mixture of the previous two. You can do this in a few ways:

Use a protocol tool to generate the majority of the load, and a browser tool for a handful of virtual users (best functionality, highest effort/cost)
Use a browser-based tool for all load (expensive, simplest)
Find a tool that can do both (ideal solution, few options)

Whatever you choose, compare like with like.

Happy testing!

Realistic load testing

Sat, 07 Aug 2021 22:15:13 +0200

This is a conversation I recorded from Clubhouse in the Performance Engineers group created by NaveenKumar Namachivayam today, August 7th, 2021. I started by introducing myself and then let everyone know that I would be recording the conversation. Enjoy!

Comparing k6 and JMeter for load testing

Fri, 29 Jan 2021 23:01:18 +0100

Originally posted on k6.io.

Introduction

When I joined the k6 team in late November, the foremost question on my mind was “How does this compare to JMeter?” You see, I come from a performance testing consulting background, and in the last few years, JMeter has been far and away my tool of choice.

So what am I doing on the k6 site, talking about how much I love JMeter? Well, firstly, it turns out that the k6 team is pretty open and transparent, especially about potential improvements in k6. Secondly, I want to share my reasons for wanting to explore k6 as a load testing tool– the same reasons that ultimately convinced me to join the team that builds it.

What is JMeter?

JMeter is an open source load testing tool built entirely in Java by the Apache Foundation. It was first released in 1998, and it made waves because of its audacity in taking on more popular but proprietary load testing tools. JMeter took what other companies were charging a lot of money for and published it online for everyone to use– for free. Although scripts can also be extended using code, majority of the scripting in JMeter is done using the UI. At the time of this writing, the latest version of JMeter is 5.4.

What is k6?

k6 is an open source load testing tool released by LoadImpact in 2017. LoadImpact had previously consisted of the SaaS platform (now k6 Cloud) and professional services. k6 is written in Go, but scripts are written in Javascript. Taking the opposite tack to JMeter, k6’s main selling point is its code-based scripting and heavy prioritization of the developer experience. The latest version of k6 is 0.30.

Which is better?

This article isn’t about which tool is better, because as much as we might prefer a clear-cut answer, there’s no “best tool”. Everything depends on the situation your team is in: what, how, why, and when you’re testing.

The question that I want to answer here is: What situations does each tool excel in?

When does JMeter excel?

When you want to get started quickly

JMeter has a GUI

Many testing teams make decisions on load testing tools based on this fact alone. Sometimes, if you’re in a situation where people haven’t done load testing before and need to learn a new tool, a GUI-driven tool like JMeter is simply the most frictionless option. A tool with a GUI will be less daunting for many testers that are used to the mostly no-code UIs of the likes of Postman or SOAPUI.

I’ve always thought of JMeter as user-friendly, but a developer friend recently pointed out that the Test Plan screen that greets you when you first start JMeter doesn’t provide any indication of how to create an HTTP sampler. None of the icons add elements to the test plan. It takes a bit of exploring to realize that right clicking on the test plan, or going to Edit > Add, will show the “Add” menu that you need. He has a point there: UI is subjective to a certain degree. However, I’d argue that it’s definitely easier for non-developers to explore a UI than a bit of code.

k6 doesn’t have a GUI packaged within the tool, but the k6 Test Builder is available for free. It’s a way to create tests with a GUI interface, and despite its inclusion in k6 cloud, it doesn’t require any subscription to use. However, it isn’t as fully-featured as JMeter.

JMeter supports many protocols and features out of the box

Here’s the official list of the protocols that JMeter supports:

Web - HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, …)
SOAP / REST Webservices
FTP
Database via JDBC
LDAP
Message-oriented middleware (MOM) via JMS
Mail - SMTP(S), POP3(S) and IMAP(S)
Native commands or shell scripts
TCP
Java Objects

In contrast, k6 supports the following protocols:

Web - HTTP/1.1, HTTP/2 (Java, NodeJS, PHP, ASP.NET, …)
WebSockets
gRPC
SOAP / REST Webservices

The protocols above are those that are natively supported. However, both tools allow the development of third-party plugins, adding support for more protocols. The number of k6 community extensions, through xk6, is growing, but JMeter plugins still outnumber them. In many cases, the protocols your application uses may make the decision for you of which tool to use.

JMeter also has most of the features you’ll need for a basic load test, and you don’t have to code any of it up yourself. Its parent-child element structure means that you can modify a specific element, like an HTTP request, or modify all HTTP requests, with equal ease. Adding think time to all requests is as easy as adding a Uniform Random Timer to your thread group, and all requests within that group will inherit it. With code-based load testing tools, it can be a bit more difficult to find out what’s possible without a UI. Some of the gap, however, can be bridged by an IDE with good autocompletion.

When you want a mature community with tons of documentation

JMeter’s old, and sometimes that’s a good thing. It’s been around since 1998, so at this point, it’s had 22 years to improve and to build a following. JMeter has a presence on nearly every community networking site, and no matter how specific your use case for JMeter is, the chances are that it’s been done before. Searching “how to load test X with JMeter” is bound to yield thousands of hits and some videos showing you exactly how to do it.

JMeter’s Component Reference is proof of JMeter’s extensive and thorough documentation. Every element, function, and property is discussed in more detail than most people could ever want, and that’s just the “official” documentation that’s on the Apache site. There are thousands of JMeter books, tutorials, and courses by its passionate users.

The JMeter GitHub repository has 17,290 commits as of December 2020 - evidence of a community of developers that have committed some time and experience to JMeter over the last two decades.

In comparison, the k6 community is growing, but still small. There is an official and central community forum that allows users to share their experiences with k6, which is something missing from the JMeter community. The k6 repository does have more stars, but significantly fewer commits at 3,795. Much of the disparity, of course, is because k6 is very new - the open-source tool k6 was only released in 2017.

When you need a cost-effective way to do distributed load testing

One of JMeter’s absolute best features is that it gives you a framework for running distributed load tests with it. This is pretty special for a free and open-source tool.

Distributed testing means ramping up the amount of load you’re generating with your load tests, usually by increasing the number of virtual users and running multiple instances of your script on other load generators. JMeter accomplishes this by designating a controller node and letting you set up worker nodes. Each worker node needs to have a copy of jmeter-server, a utility included with every JMeter installation (in jmeter/bin).

Source: Apache JMeter

These worker nodes then run copies of your test plan and exert load against your application server. You can use on-premise or cloud machines as your worker nodes. A distributed execution like this takes a bit to set up, but it does work surprisingly well.

k6 doesn’t have a native distributed load testing mode. You can containerize it yourself and deploy it via Kubernetes, for instance, but it does require a more involved setup. The team is also working on the k6 Operator, which makes setup significantly easier, but it is experimental and comes with drawbacks such as not supporting threshold synchronization between instances. For now, the easiest way to scale up on k6 is by using the paid service k6 Cloud, which reduces setup time and complexity significantly.

It’s important to note that JMeter can be quite resource-hungry, and thus is more likely to require multiple load generators to generate load than k6. Depending on the resources of the load generator, JMeter can run about a thousand virtual users on average, and scaling up your test beyond that point will require a distributed execution setup. A single instance of k6, on the other hand, can run tens of thousands of virtual users given the same resources. More on that later.

When you want canned reports built into your load testing tool

JMeter’s many listeners allow you to decide exactly what format you want to view results in. Debugging? Use the View Results Tree. Wanting metrics for your load test? Use the Summary or Aggregate Report.

In addition, JMeter can generate HTML reports based on your load tests. They’re currently quite limited, but functional, and they have several default graphs to help you analyze your load test. They are not interactive, but they are pretty easy to generate and a good starting point for putting together a report template for your tests.

k6 does not have built-in reporting. Its modular nature lends itself better to integrating with many other data analytics tools, including Grafana, Datadog, New Relic, Amazon CloudWatch, and k6 Cloud– all of which are better analysis tools and can provide more insights than what JMeter’s reports can provide, but they do require you to set up another tool to analyze your tests. If you’re running low on time and can make do with its reports, JMeter does really well at providing that extra value out of the box.

When does k6 excel?

When you want to get started quickly

Yup, I put this one in both the JMeter and k6 sections. JMeter and k6 can each be easy to get started with, but for very different reasons.

Installation

k6 installs extremely quickly - on my first installation of it on macOS, it took 7 seconds via brew install k6. It doesn’t require nodeJS or any other dependencies, either.

JMeter itself is fairly easy to install, but before you install JMeter, you need to install Java. Especially if you’re using Windows, it’s a bit of a rite of passage for a new JMeter user to run into issues adding the environment variables needed for Java to function.

It can also be confusing to determine which version of Java is the right one. There’s JDK, JRE, and recently Oracle has added to the confusion by announcing that some versions of Java (but not all) are going to require a license.

Source: Oracle Java

While JMeter is still free via OpenJDK, I’ve spoken to a few testers who mistakenly thought they’d have to pay for JMeter. It’s a little disconcerting to see Oracle flexing its muscles here, especially if you’ve built a whole performance testing suite on Java through JMeter.

Plugins

k6 supports plugins, but they’re not necessary. For most use cases, k6’s built-in functionality should be more than sufficient. This is in sharp contrast to JMeter, where everything is a plugin … even the Plugin Manager!

Most JMeter users would argue that there is a standard set of plugins that you should download before you even begin to use it. A new JMeter user might not necessarily know where to find these and may have a less-than-satisfying experience out of the box.

Here are a few things you won’t have if you stick to a vanilla, no-plugin JMeter installation:

Thread groups: No custom load profile, stages, rendezvous
Controllers: No parallel controller to execute requests in parallel
Samplers: No HTTP/2 support
Test Data: No randomization of lines within a CSV file (sequential only)
Timers: No throughput shaping timer for setting throughput in stages

These functions are all available within k6 from the get-go.

Scripting

k6 scripts are written in javascript, making them simple to write and run. All you need is one installer (on Microsoft Windows) or a single command on your terminal and any text editor. For JMeter, you’ll need to install a few things locally before you can get started: Java, JMeter itself, and the standard set of plugins you’ll need. This can be confusing for new users. Who hasn’t struggled with environment variables at some point when learning to use JMeter, especially on Windows?

It also takes time to learn any new UI, whereas javascript is near-ubiquitous and it’s a good language to learn when you’re load testing web sites anyway.

When you want to maximize performance and efficiency

Test your test tools!

Why does all this matter?

When we’re talking about simulating many virtual users, slow performance = high cost. There is a cost to the number of load generators you use, whether it’s the cost to acquire and maintain them (on-prem) or the cost of a cloud service.

It makes sense to consider the performance limitations of your load testing tool before you use it to improve your own application’s performance.

k6 builds on Go’s stellar performance

k6 is written in Go, and Go is built with performance in mind. “Go is a compiled language and does not need to be interpreted, unlike Java or Python. There is no added layer of complexity.” It also has no external dependencies, which is a good thing. Fewer moving parts for a performance tester means fewer potential sources of a bottleneck.

The simplest and most common way for a load testing tool to run a virtual user is to assign one virtual user to one kernel or OS thread. However, the 1 Thread: 1 Virtual User paradigm has a serious flaw: when a virtual user is waiting for a response or executing a programmed sleep(), the thread is also blocked, and cannot process other work.

In k6, each virtual user is run on a goroutine, not a thread. What difference does that make? Well, goroutines can be controlled by something called the Go Scheduler, which acts like a traffic cop. It reuses idle threads and intelligently assigns work, by allowing “work stealing” and work hand-offs between threads. Does this sound familiar? This is the same principle that load balancers are built on: An external monitor that oversees the flow of work improves general performance. Go itself is intrinsically load-balanced in a way that many programming languages aren’t, which makes it the perfect foundation for a load testing tool.

Being able to leverage Go’s innate performance optimizations also translates into significantly less memory utilization. One thread running k6 doesn’t go above 100 kb, whereas a JVM thread like JMeter uses, for instance, uses the default of 1MB. That’s 1000% more than k6! Of course, Java allows a user to tweak an app’s memory utilization, so the difference is usually not quite as stark, but it’s still interesting to note that Go has a much lower starting point.

… creating [a] new thread in Java is not memory efficient. As every thread consumes approx 1MB of the memory heap size and eventually if you start spinning thousands of threads, they will put tremendous pressure on the heap and will cause shut down due to out of memory. Also, if you want to communicate between two or more threads, it’s very difficult. - Keval Patel

Performance benefits in practice

Fewer load generators needed

k6’s comparatively better performance means it needs fewer load generators to execute a given amount of load. Rafaela Azevedo made a comparison of the memory used by k6 and JMeter, and here are her results:

JMeter took up 760 MB of memory.

Source: Rafaela Azevedo

Meanwhile, k6 took up 256 MB of memory.

Source: Rafaela Azevedo

Rafaela’s findings are further affirmed by our own benchmarks.

Source: Ragnar Lönn, in our Load Testing Tool Comparison

k6’s lower memory footprint is the reason it can run more virtual users and generate more load than average. Whether you’re using load generators that are on-premise or in a cloud, you’ll pay less for provisioning costs when using k6. This cost saving makes k6 a great tool for budget-conscious teams.

The common misconception of many load testers is that distributed execution (ability to launch a load test on multiple machines) is required to generate large load. This is not the case with k6. - Rafaela Azevedo

k6 is different from many other load testing tools in the way it handles hardware resources. A single k6 process will efficiently use all CPU cores on a load generator machine. A single instance of k6 is often enough to generate load of 30.000-40.000 simultaneous users (VUs). This amount of VUs can generate upwards of 300,000 requests per second (RPS).

Unless you need more than 100,000-300,000 requests per second (6-12M requests per minute), a single instance of k6 will likely be sufficient for your needs. - k6 documentation

No out of memory errors

k6 is also good for when you’re short on time. Using JMeter means getting familiar with how to performance tune Java and how to fix the most common Java performance problems, because there are several. The one that I’ve personally run into the most while using JMeter is:

ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError:
Java heap space.

You’re not really a JMeter load tester until you’ve seen this error. :)

The cause for this error is usually that there was an insufficient amount of Heap memory allocated to JMeter. Yup– in addition to monitoring your load generator’s memory usage, you’ll also need to monitor the JVM’s memory usage. You can set the amount of heap memory allocated to JMeter by modifying the JMeter binary itself and changing this line:

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
: "${HEAP:="-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m"}"

Even JMeter official documentation suggests that you increase the Java Heap size beyond 1GB before you even run your first load test.

Increase the Java Heap size. By default JMeter runs with a heap of 1 GB, this might not be enough for your test and depends on your test plan and number of threads you want to run - Apache JMeter

No GUI means no resource overhead during load tests

GUIs typically add significantly more overhead to an application, which is one of the reasons why k6 doesn’t have one.

When you’re running a load test, the best practice is to run it from the command line anyway– because otherwise, it interferes with your results. Code-based load testing tools are performant from the beginning.

JMeter is peppered with warnings about the GUI’s effect on performance. Here’s one from the “JMeter documentation:

Don’t run load test using GUI mode ! - Apache JMeter

And here’s a message that occurs upon startup of JMeter:

Not having a GUI means there’s one less gotcha. Any load testing scripts you create in k6 are ready for prime-time execution whenever you’re ready.

When you want to do goal-oriented testing

The most important question a load tester can ask when starting a new load testing project is “Why”? Test plans should be made to directly address a team’s reasons for wanting to do load testing, and that should be reflected in the non-functional requirements. Non-functional requirements, in turn, should then give some guidance as to the bounds of acceptable performance. A common example for these bounds, or thresholds, is an average response time of less than 3 seconds for all transactions during the test. k6 does this natively with global thresholds worked into the script, and you can also create your own metric to use in a threshold:

import http from 'k6/http';
import { Rate } from 'k6/metrics';

const myFailRate = new Rate('failed requests');

export let options = {
 thresholds: {
 'failed requests': ['rate<0.1'], // threshold on a custom metric
 http_req_duration: ['p(95)<500'], // threshold on a standard metric
 },
};

export default function () {
 let res = http.get('https://test-api.k6.io/public/crocodiles/1/');
 myFailRate.add(res.status !== 200);
}

Source: k6 documentation

JMeter doesn’t natively support thresholds at the test-level. There are a few ways that I’ve been able to work around this:

Duration Assertion: You can add a Duration Assertion to any sampler in JMeter, which allows you set an acceptable response time for that request in milliseconds, beyond which the request (and the transaction that encompasses it) would be marked as failed. However, this is still just at the individual request level, and even putting the duration assertion element at the Test Plan level only applies the same threshold to each child request. It doesn’t constitute pass/fail criteria for the test as a whole.
Timers: For criteria related to throughput, you can use the Constant Throughput Timer or the Throughput Shaping Timer (plugin). This approach controls how many requests per second are spent. In my experience, it’s worth testing how these timers behave when using nested transaction controllers - it may take some rearranging to get right.
Performance Plugin: When integrating JMeter with Jenkins, you can use the Performance Plugin to set error thresholds and mark test runs (“builds”) as unstable or failed within Jenkins.
Custom Code: You can use the JSR223 Sampler to write a bit of Groovy or BeanShell code that collects and parses response metrics.

The problem is that even though these options make setting thresholds possible in JMeter, they’re still workarounds that don’t come with JMeter out of the box or don’t adequately address the creation of different types of thresholds (error, response time, throughput, CPU, memory). What I end up doing, more often than not, is exporting the raw data from JMeter and doing the analysis myself using another tool.

When you’re part of a team of scripters

k6 shines in situations where more than one person could be doing the scripting.

It strikes a middle ground that developers and testers can compromise on

Software quality should not be the sole responsibility of testers; quality should be baked into every activity when building software. One of the challenges in implementing this concept, however, is that there has traditionally been a segmentation in tools and languages used by team members with different functions. Developers use one tool; testers use another. It’s difficult to encourage developers to test and testers to code in this environment, and many testing tools only further encourage this divide. A developer can’t use a tool like JMeter for anything but testing, so the incentive to invest the effort in learning it is lower.

k6 bridges the gap by bringing testing concepts and features to an environment and language that developers are already spending their time in. Javascript is useful for more than just writing load testing scripts, and k6 works well with any IDE or text editor. A frictionless developer experience is central to the k6 philosophy.

It’s easier to collaborate on scripts

Once you’ve convinced both devs and testers to contribute to load testing scripts, how do you keep track of changes? How do you manage scripts that multiple people are working on at the same time? With tests as code, the easiest way is to just use versioning tools like Git, the way you probably are already using for your application code. Again, no new framework or processes to learn here. k6’s conciseness and portability really proves to be an advantage in collaboration situations. To illustrate this, let’s consider a simple test script that contains a few things:

A GET request to our test site, https://test.k6.io
A transaction that calls this request 01_Home
A check to fail non-HTTP 200 responses or responses that don’t have the text Collection of simple web-pages suitable for load testing
Variable think time
We’ll just run that as a single user, for a single iteration. Here’s what that script looks like in k6:

import http from 'k6/http';
import { sleep, check } from 'k6';

export default function () {
 let res = http.get('https://test.k6.io', {tags: { name: '01_Home' }});
 check(res, {
 'is status 200': (r) => r.status === 200,
 'text verification': (r) => r.body.includes("Collection of simple web-pages suitable for load testing")
 });
 sleep(Math.random() * 5);
}

You can track JMeter scripts on Git, too. Here’s what the same script looks like in JMeter:

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.3">
 <hashTree>
 <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
 <stringProp name="TestPlan.comments"></stringProp>
 <boolProp name="TestPlan.functional_mode">false</boolProp>
 <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
 <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
 <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
 <collectionProp name="Arguments.arguments"/>
 </elementProp>
 <stringProp name="TestPlan.user_define_classpath"></stringProp>
 </TestPlan>
 <hashTree>
 <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
 <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
 <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
 <boolProp name="LoopController.continue_forever">false</boolProp>
 <stringProp name="LoopController.loops">1</stringProp>
 </elementProp>
 <stringProp name="ThreadGroup.num_threads">1</stringProp>
 <stringProp name="ThreadGroup.ramp_time">1</stringProp>
 <boolProp name="ThreadGroup.scheduler">false</boolProp>
 <stringProp name="ThreadGroup.duration"></stringProp>
 <stringProp name="ThreadGroup.delay"></stringProp>
 <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
 </ThreadGroup>
 <hashTree>
 <TransactionController guiclass="TransactionControllerGui" testclass="TransactionController" testname="01_Home" enabled="true">
 <boolProp name="TransactionController.includeTimers">false</boolProp>
 <boolProp name="TransactionController.parent">true</boolProp>
 </TransactionController>
 <hashTree>
 <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
 <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
 <collectionProp name="Arguments.arguments"/>
 </elementProp>
 <stringProp name="HTTPSampler.domain">test.k6.io</stringProp>
 <stringProp name="HTTPSampler.port"></stringProp>
 <stringProp name="HTTPSampler.protocol">https</stringProp>
 <stringProp name="HTTPSampler.contentEncoding"></stringProp>
 <stringProp name="HTTPSampler.path"></stringProp>
 <stringProp name="HTTPSampler.method">GET</stringProp>
 <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
 <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
 <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
 <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
 <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
 <stringProp name="HTTPSampler.connect_timeout"></stringProp>
 <stringProp name="HTTPSampler.response_timeout"></stringProp>
 </HTTPSamplerProxy>
 <hashTree>
 <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
 <collectionProp name="Asserion.test_strings">
 <stringProp name="-745788246">Collection of simple web-pages suitable for load testing</stringProp>
 </collectionProp>
 <stringProp name="Assertion.custom_message"></stringProp>
 <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
 <boolProp name="Assertion.assume_success">false</boolProp>
 <intProp name="Assertion.test_type">16</intProp>
 </ResponseAssertion>
 <hashTree/>
 </hashTree>
 </hashTree>
 <UniformRandomTimer guiclass="UniformRandomTimerGui" testclass="UniformRandomTimer" testname="Uniform Random Timer" enabled="true">
 <stringProp name="ConstantTimer.delay">1000</stringProp>
 <stringProp name="RandomTimer.range">4000</stringProp>
 </UniformRandomTimer>
 <hashTree/>
 </hashTree>
 </hashTree>
 </hashTree>
</jmeterTestPlan>

Which one is easier to understand? If you had to modify the script to add a request, which format would you choose to do it in?

Scripting in XML makes merge conflicts more likely and breaking changes in automation pipelines way more difficult to pinpoint.

I’ve collaborated with others on JMeter scripts in the past, and we always opened up scripts in the GUI. It’s way easier to spot changes there - but that also created some new issues.

When you want something easy to maintain

Versioning for load testing

When ramping up load tests to more than one load generator, keeping track of the versions of necessary components ensures that one load generator’s traffic is the same as another’s. The value of reducing variables in scientific experiments holds true for load testing.

With k6, you just need to worry about the version of k6 and the script. No other dependencies are required. If you’re using plugins, you’d need to track those too, but as mentioned previously, plugins in k6 aren’t as necessary as they are in some other tools – and k6 is available as a Docker image to reduce those issues.

With something like JMeter, there are more levels to consider. There’s the version of JMeter itself, Java, versions of plugins, and then the script. Every new layer introduces more changes for incompatibilities. In previous projets I’ve worked on, someone would inevitably try out a new plugin and forget to discuss it with the team, and anyone who tried to open that script in their copy of JMeter would see this error:

The error means there’s a plugin that the script uses that JMeter doesn’t recognize, but depending on the plugin, it’s not always this easy to pinpoint which one (it was the Ultimate Thread Group in this case). You don’t want that happening in the middle of your load test because you forgot to add a plugin to some of your load generators!

It’s easier to link test code with application code

If your application code is in Javascript, k6 is a shoo-in as a load testing tool. You could build tests that are tightly integrated with application code by importing objects and having your tests interacting with them directly. This approach would reduce a lot of time spent in refactoring test scripts when objects are changed.

If you’re already using VS Code, k6 even has a VS Code plugin.

Test as code gives you control

Having your scripts be pure code reduces ambiguities in that you’re not limited by poor UI design or bugs in third-party plugins. k6 scripts are Typescript-typed, so most IDEs support autocompletion while you’re scripting - the code equivalent of helpful tooltips, except way more universal. k6 scripts give you the freedom to import and use JS libraries, so you can build on code others have already written. While k6 is not NodeJS, there are even some NodeJS packages that you can bundle for use with your scripts.

No external dependencies

k6 doesn’t require NodeJS or anything else to run, which also reduces the likelihood of bugs that might affect your scripts. For example, since JMeter runs on Java, it is also dependent on Java. When Java receives a new version, that often requires a JMeter update as well. JMeter versions understandably lag a bit behind Java versions as developers refactor code, but this gap can lead to potential security issues in a previous version of Java that may remain unpatched in the meantime.

With k6, development can proceed independently of any third-party component, so major issues can be addressed more easily.

Comparison table: JMeter vs k6

Feature	JMeter	k6
Written in	Java	Go
Scripting language	Limited: some Java (Groovy, Beanshell, etc)	Javascript
Protocols	Supports most protocols via plugins (native support for HTTP/1.1, SOAP, FTP, JDBC, LDAP, MOM via JMS, SMTP, POP3, IMAP, shell scripts, TCP, Java objects)	Supports fewer, modern protocols natively (HTTP/1.1, HTTP/2, WebSockets, gRPC)
External dependencies	Java	none
Resource utilization	Poor; One load generator can simulate a few thousand virtual users	Very good; One load generator can simulate tens of thousands of virtual users
Memory management	JVM heap memory must be set	Uses load generator memory natively
Threading model	1 Thread: 1 Virtual User; Slower performance, higher resource cost	1 Goroutine: 1 Virtual User; Faster performance, lower resource cost
Ease of scripting	GUI-driven, with code blocks	code-driven; VSCode plugin
Test-level thresholds	No, individual request only	Yes
Script format	XML	javascript
Collaboration	Difficult to work on simultaneously; tester-friendly; need the GUI app to edit	Developer-friendly, easy to version; Javascript format promotes collaboration
Maintenance	Verbose scripts; XML format is difficult to read	More concise scripts; JavaScript is easy to read
Community	Since 1998; Many third-party tutorials; Extensive documentation; No central community	Since 2017; Extensive documentation; Fewer third-party tutorials; Official community
Plugin support	Requires plugins for many features, but there are lots of plugins available	Most features are natively supported, but plugin support is new and availability is sparse
Native distributed load generation	Yes	No (Premium only)
Pre-generated reports	Default and custom HTML reports; logging via listeners	No built-in pre-generated reports; integration with analytics tools with third-party dashboards
Website	jmeter.apache.org	k6.io
Source Code	Link	Link

Conclusion

The best advice, when attempting to choose a load testing tool, is to do a proof of concept with the most promising candidates. Some features or bugs may prove more or less important when used in an actual testing cycle. However, don’t put too much weight on comparisons of the load test results from two different tools - each tool records metrics differently, and it’s more meaningful to compare results against previous runs with the same tool. When switching tools, reestablish a baseline in the new tool each time.

Here’s a summary that might help you decide between JMeter and k6.

Both tools do well at:

Generating protocol-level load on an application server by scripting complex user flows
Realistic scripting with the use of dynamic think time, test data generation and reuse, and customizable workload models
Documentation of features and consistency of releases

Neither tool supports:

Generating browser-level load and interacting with DOM elements, especially for SPA
Detailed results analysis (JMeter has pre-generated HTML reports and listeners, but they are far from ideal). Users should expect to integrate results with databases and data visualization software.

JMeter is best for:

traditional testing teams
those looking for a GUI-driven testing tool with tons of third-party tutorials and extensive protocol support
previous users of commercial tools like LoadRunner and NeoLoad

k6 is best for:

collaborative, cross-functional engineering teams where testing spans acrosss multiple roles
those looking for a simple and lightweight, yet fully-featured load testing tool
teams looking to integrate testing into existing development workflows and CI/CD pipelines

Load testing scripting tools are by no means the most important consideration for the success of load testing. Knowing why you’re testing, what the requirements are, and understanding and communicating results are all arguably more important. The right tool will enable you to address those concerns while providing as little friction as possible. There’s no unequivocal “best” tool; there’s only the right tool for your project and context.

Week of Testing Wrap-up: My thoughts on k6 as a load testing tool

Thu, 17 Dec 2020 15:29:57 +0100

I decided to make a final wrap-up video for my “Week of Testing” with k6… ok, so it’s actually three weeks later now, but I wanted to explore k6 a little bit more before I gave my opinion on it.

And here’s the audio version:

Thank you, k6, for being awesome enough to agree to this video series before knowing what I’d say. Gotta love a company that welcomes transparency and feedback.

Cuándo JMeter no alcanza: Pruebas de carga simulando navegadores

Wed, 16 Dec 2020 18:18:23 +0100

Aquí está el video de mi presentación en JMeter Scripting Day, un evento organizado por JMeter en Español el 31 de Oktubre, 2020.

Y la versión audio:

Introducción

Hola a todos! Yo soy Nicole van der Hoeven, y soy performance tester. JMeter es una de mis herramientas favoritas porque es gratis, simple, y por muchos casos hace exactamente lo que quiero y no más. Sin embargo, como performers es parte de nuestro trabajo elegir la herramienta adecuada para cada proyecto, y hay que admitir que hay situaciones en que JMeter simplemente no es suficiente.

Amazon - UI + Dev Tools

Esto es el sitio web de Amazon.com. Tengo abierta la pantalla de Chrome Developer Tools, que me muestra las solicitudes (o requests) que componen la página. Cuándo vamos a la página, el navegador (Chrome en este caso) envia estas solicitudes para nosotros, pero eso occure en el fondo. Un usuario normal no las ve.

JMeter - Creando un script

Lo que JMeter hace es grabar estas solicitudes y replicarlas. Aqui vemos una ejecución de un usuario virtual de un script en JMeter. Cada solicitud tiene la respuesta y el tiempo de respuesta. Pero eso no nos da una idea completa del rendimiento de la aplicación. ¿Por qué?

Por que JMeter, como otras herramientas de nivel protocolo, tiene unas limitaciónes. La primera es que si hay javascript o otros script que deben estar ejecutado por el navegador, JMeter no los ejecuta. JMeter solo los descarga. El tiempo de respuesta mide la descarga, no la duración de la ejecución. Entonces, siempre habrá una diferencia entre los tiempos de respuesta en una prueba de JMeter y los tiempos de respuesta en práctica.

A veces esta diferencia no es tan importante. Pero hay más y más aplicaciones hoy en día que se componen principalmente de estos javascript. Por esas aplicaciones, simplemente no es posible hacer un script en JMeter porque la mayoría de la lógica de la aplicación occure en el navegador, no en el servidor.

Flood Challenge

La segunda razón es que el aumento de la seguridad de aplicaciones conduce a solicitudes más complejas que incluyen parámetros dinámicos. Estos parámetros hacen que sea más difícil para un intermediario lanzar un ataque, pero también hacen que sea más difícil para nosotros scripters, porque tenemos que extraer información desde la última respuesta y ponerla en la próxima solicitud. Es posible, pero es la pesadilla de cada performance tester.

Podemos verlo en la aplicación de muestra Flood Challenge. Ya no trabajo para Flood, pero todavía es un buen ejemplo.

Correlación en JMeter - Flood challenge script

Esto es mi script en JMeter, y vemos que hay muchos parámetros que he tenido que correlacionar. La solicitud para simular hacer clic en el botón “Start” necesita los parámetros authenticity token, step ID, y step number, que son dinámicos. El contenido de los parámetros cambian cada vez que usamos esta aplicación.

La experiencia del usuario

La tercera razón es que JMeter tampoco mide la verdadera experiencia del usuario. Solo mide la duración de la comunicación entre el navegador y el servidor, pero no mide cosas como la representación de elementos en la página web, ni el efecto de un script que dura mucho tiempo para terminar.

Eso es importante porque estas cosas afectan significativamente la experiencia del usuario. De hecho, Steve Souders, un conocido el la industría de performance, dice que hasta 90% del tiempo de respuesta que experimenta el usuario está en la interfaz. Con este numero podemos estar ciertos que una prueba con una herramienta de nivel protocolo no va a poder encapsular el rendimiento final de una aplicación.

Developer Tools - Lighthouse

Entonces, ¿qué podemos hacer?

Lo más fácil es intentar de medir la interfaz durante una prueba de cargas. Mientras ejecutamos una prueba con JMeter, podemos ir manualmente a la aplicación web en Chrome para usar Developer Tools. Podemos usar la pestaña Lighthouse, que nos da más métricos del perspectivo del usuario y sugerencias para mejorar el rendimiento.

WebPageTest

También uso WebPageTest a veces. Eso tiene más opciones para ejecutar la prueba automáticamente varias veces desde un lugar que puedo elegir. Aún podemos elegir un dispositivo para simular la experiencia de un usuario en un iPad, por ejemplo.

Herramientas alternativas para pruebas de carga

Tambien podemos elegir herramientas alternativas para ejecutar nuestras pruebas de carga. Lo que tienen en común es que fueron diseñadas para pruebas de automatización, y utilizan un navegador real.

Selenium es probablemente la más popular de esas. Selenium WebDriver en particular. Es compatible con Ruby, Java, Python, C# y Javascript. El problema con Selenium es que puede ser muy pesada. En una machina probablamente podemos ejecutar cinco usuarios de Selenium, más o menos. Eso es importante si estamos tratando de simular miles de usuarios.

Hay muchas herramientas basadas en Javascript que también podemos usar. Hay varias framework que se utilizan para adaptar Javascript y nodeJS.

Puppeteer fue desarollado por Google. Tricentis Flood también ha diseñado Element que utiliza Puppeteer, pero con un enfoque en el rendimiento.
Playwright es una creación de Microsoft.
Jest fue desarrollado por Facebook
Mocha es otro framework que es la base para Cypress.

Todos esas herramientas son open source y gratis. También hay alternativas propietarias, pero prefiero las soluciones open source porque en la mayoría de casos hacen la misma cosa y se actualizan con más frecuencia.

Element script

Esto es un ejemplo de la misma acción, hacer clic en el botón Start en la aplicación Flood Challenge.

Otra ventaja de simulando navegadores en vez de simulando las solicitudes es que es más simple crear un script. No tenemos que saber cada parámetro. Solo tenemos que describir lo que queremos que cada usuario virtual haga.

Podemos ejecutar scripts usando estas herramientas al mismo tiempo que un script de JMeter o Gatling o lo que sea, y también tenemos la opción de reemplazar los script de protocolo con estas.

Conclusión

Para resumir, JMeter y otras herramientas de nivel protocolo tiene algunas limitaciones.

No ejecutan scripts del lado del navegador
Pueden ser más complicado correlacionar parámetros dinámicos
No miden la experiencia final de un usuario

Para solucionar estos problemas, podemos complementar pruebas de carga tradicionales, de nivel protocolo, como JMeter, con herramientas simulando navegadores. De este modo podremos tener una comprensión más completa del rendimiento total de la aplicación.

Bueno, se acabó el tiempo, entonces los dejo aquí. Gracias por escuchar, y me disculpo por mi español. Gracias a todos!

Week of Testing: Day 4 - Load profiles on k6

Fri, 27 Nov 2020 19:54:06 +0100

k6 has a cool feature where even if you start a load test locally via the CLI, you can still stream the results to k6 Cloud. It’s an interesting compromise between debugging a test locally (and therefore not paying for cloud infrastructure) and sharing results for load tests run on the cloud. It worked seamlessly, too.

I also wanted to see how to recreate a specific load profile that I commonly use - the stepped load profile. In JMeter, I typically do this with a plugin - the jp@gc Ultimate Thread Group is my thread group of choice. So, I created it in JMeter (I love that it shows a little image of it as well, while you’re building it) and then attempted to recreate it in k6.

I definitely had to look at the k6 documentation for this. I used the constant-vus executor for this, but k6 has several other executors to choose from, and it worked as well as I had hoped.

So far, my experience with k6 has been positive. Things are well-documented, and they just work out of the box. No plugins necessary.

Week of Testing: Day 3 - Running a load test locally using k6

Thu, 26 Nov 2020 19:31:33 +0100

I’ve had my fun playing around with k6 Cloud, the SaaS component. For Day 3, I thought it was time to jump into the heart of k6: k6 the open source load testing tool.

First impressions: logging was a little sparser than I expect, coming from having used load testing tools that just do this for me automatically. I think one of the first things I’ll do with k6 is to create my own little “load testing starting script” that I’ll always use, with all of the appropriate logging already set up.

Otherwise, I was really impressed with how quickly it installed. If you scrub over to the timestamp 1:46 in the video, you’ll hear that I was actually expecting to have to speed up the installation process while editing to save time, but the installation actually finished before I could say that I was going to do so. By my count that was 7 seconds between hitting enter in my terminal and the download finishing.

7 seconds.

I can’t help but compare that to JMeter. JMeter is honestly probably my go-to load testing right now– it’s not good for absolutely everything, but it’s the tool that I reach for when I “just want to test this one thing”. I’ve used it for years, and it’s the tool that I know the most. But one of the issues with JMeter is that there’s so much to do before it’s actually usable.

Before you even open JMeter, you need to install Java, and when I first started in load testing, I spent a long time wrestling with getting setting the Java environment variable in Windows before my computer would even recognize that Java was installed. Then you need to install JMeter– but any JMeter tester will tell you that there are some plugins that you just can’t do without, and that really you should always install. That’s why there are so many “best plugins in JMeter” lists (here’s my top 3)… because most of them are necessary, but you wouldn’t know that if you’re new to JMeter.

The fact that k6 starts up in 7 seconds is… shockingly good.

Ask a Flooder 18: How do I load test video streaming with JMeter?

Wed, 22 Jul 2020 22:49:15 +0100

Originally posted here.

In this video, I answer the question: “How do I load test video streaming with JMeter?”

Or listen to the audio version here:

HTTP Live Streaming (HLS) and performance

One of the most popular protocols for video streaming is HTTP Live Streaming, or HLS, and that’s what I’m going to show you today. An HLS video ends with the [file extension .m3u8‍](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8). According to the HLS protocol, when a video is streamed, the first request sent must be a manifest. A manifest is sometimes called a playlist, and it is a list of all the chunks or parts that make up the video. Then, each chunk is downloaded.

To assess the performance of a video stream, we still want the response time of the manifest, the chunks, and the whole video. However, there are many more metrics that are important in video streaming which aren’t captured by just the response time. For example, how do you determine whether a video was played smoothly or if it stopped to buffer? A video that keeps pausing might just drive a user away, and it’s difficult to tell whether that is happening just from the response time.

How to load test video streaming with JMeter

Downloading and installing the Ubik Video Streaming plugin

Luckily, there’s a plugin for that, created by a company called Ubik Ingénierie. The Ubik Load Pack is a set of enterprise-grade plugins for JMeter. One of those plugins is a Video Streaming Plugin, and it happens to be a really easy way to get started with video streaming. This is a paid plugin, but you can sign up for a free 11-day trial.

Once you do that, you’ll get an email from Ubik with a link to download the bundle. Go to lib/ext and then copy this .jar file into the same directory on your JMeter installation. Start up JMeter, and it’ll ask you for the license file. Browse to the license file in the bundle you downloaded. JMeter will restart.

Using the JMeter plugin in your load testing script

Now let’s start creating the Test Plan. Create a Thread group, and then right click on the thread group, go to Add > Sampler and then select the “ULP Video Streaming sampler”.

Click on the sampler and then paste in the URL to the HLS file that you’d like to test. This should end in a .m3u8. This plugin supports several protocols for video streaming, but we can just leave it at auto and it’ll automatically detect which one it is.

In Play Options, you can choose whether to play the entire video or just a part of it, or even whether you want to include the video subtitles in your test.

This Network Options panel is fantastic because it allows you to limit your network speed. Slow networks, like mobile networks, really affect user experience, so it’s really important to take this into account when running load tests. You might even use a parameter here and do something like throttle to 3G speeds for 80% of your traffic, and leave it on Auto 20% of the time. But I’ll just leave this blank for now.

Next, let’s add a View Results Tree listener. Then, let’s run the script once, just to see if it works.

As you can see, we can see the manifest retrieved first, and then all the chunks. We didn’t get any errors, so let’s run this as a load test. We can disable this listener because it just consumes extra resources on our load generator.

Ramping up your load test on Flood

To ramp this up into a load test, we’ll have to go into Thread Group and change these settings. Ideally, we want to be able to change these values from the Flood UI later so we don’t have to go into the script whenever we change them, so I’ll type in these Flood-specific properties. I’ll leave a link in the description about these properties. I’ll also set it to run Forever so that we can change the duration and the video will just keep playing to fill that duration.

To capture some of Ubik’s cool metrics, let’s add a Simple Data Writer listener here and write the results to a CSV file. On Flood, all flood results are stored on /data/flood/results, so we’ll add an output.csv file in that directory that we can download after the test.

So that’s our script! Now we’ll have to include the plugin and related files so that Flood can access them as well.

Let’s go back to the bundle we downloaded and go to the bin folder to look for user.properties. Let’s open that up, and we’ll change the license path to /data/flood/files, which is the directory that Flood saves files we upload to. To preserve this directory structure, we’ll compress just the lib and bin folders.

On Flood, we’ll start a new stream here and upload three things: the jmx script, the license, and the compressed archive of the lib and bin folders from the bundle.

Then we’ll go through the rest of the process here, setting the test parameters. Time to launch the test!

Analyzing video streaming load test results

Flood shows you results of your load test while it’s running as well as after the test. You can see metrics like concurrency, network throughput, transaction rate, and response time for the test. You can also see these metrics for each individual transaction, including the video or sampler as a whole, the manifest, and the chunks.

To see the CSV that our script generated, we can download the Archived Results from a flood.

In JMeter, click on Tools > Generate Report and fill in the fields, selecting the output.csv from the flood’s Archived Results, JMeter’s user.properties file, and then an output directory for the report.

Ubik’s plugin creates a few custom graphs in the HTML report:

Buffer fill in Millis
Lag ratio in percentage
Lag ratio without buffer fill in percentage
Lag time in Millis

Conclusion

Using Ubik’s Video Streaming plugin is an easy and thorough way to load test video streaming with JMeter, and I highly recommend it. I used an HLS video to demonstrate it, but this plugin supports several protocols:

HLS
Mpeg DASH
Smooth
Adobe HDS

For more information, read Ubik’s tutorial on how to use their plugin with Flood here.

Ask a Flooder 13: How do you do mobile load testing?

Tue, 28 Apr 2020 23:21:08 +0100

Originally posted here.

How do you load test a native mobile app? I discuss the general approach to mobile load testing, what a device farm is, and why a device farm isn’t necessary for load testing. I also discuss the best mobile load testing tools to use.

Or listen to the audio version here:

Hi everyone, Nicole here again, back with another Ask a Flooder, and today’s question is:

How do you do mobile load testing?

Now, this question came from a customer who was under the misconception that in order to do mobile load testing, they would need to use what we call a device farm. A device farm is a collection of different kinds of mobile phones. You would ideally have the most popular makes of mobile phones (the hardware models) and the operating systems and system versions. You can imagine that this can very easily get a little out of hand to coordinate because of fragmentation. There are just so many devices out on the market, and then there’s the question of how you actually generate the load from that.

You would somehow need some sort of test automation script that was running and then coordinate os that they all run at the same time, and you would use that as a load test. As you can imagine, whether you use real devices or simulators, this kind of thing would get complicated and expensive very quickly. But the good news is that it’s unnecessary if it’s load testing that you want to do. Now I say that because, for functional testing or compatibility testing, you ARE going to want to test your application on a variety of different browsers. But with load testing, you’re really more concerned about the back end part of the equation. You want to generate load on your application servers. In reality, most mobile apps work by sending API requests to that server, usually by some sort of restful API. Luckily, API load testing has been around for a while, so we do have some industry-tested ways to run this.

What are the best tools for API load testing?

Essentially, you’re going to want to test protocol-level tools. That means that you don’t want to be using browser-level tools in which you’re using scripts to drive a full browser. All you really need is something that sends requests on the protocol-level. That’s great because it’s actually way more efficient.

You have two really good open-source options for this, JMeter and Gatling. I’ve already gone a little bit into the differences between these two in my previous video on how you choose a tool for load testing, which I will leave a link to.

And here are some other resources that will help you get started with mobile load testing. I also have another video about how to record network traffic from a native mobile app on JMeter, which sounds a little complicated, but I promise it’s not. In that video, I go over the steps with you, and I’m actually doing it in real-time with you as well. I show you the screen recording from my mobile and from JMeter on my laptop, so you can always pause and watch it at your own pace.

The second thing is that I already wrote a blog post, which is sort of a guide for mobile load testing, which goes into the steps into way more detail than I can here.

And lastly, I have written blog posts about two real-life examples of mobile load testing. The first one was the Iowa Caucus Reporter app. I ran through it from the workload modeling to the scripting and then the actual execution. More recently, I also wrote about the Robinhood app. Be sure to check out the links in the description to find out more.

Until next time, happy flooding!

Browser-based load testing in 2020

Thu, 23 Apr 2020 16:48:09 +0100

Originally posted here.

We first announced our open-source browser-based load testing tool based on Puppeteer, Flood Element, in February 2018. Despite our decision to make it an entirely separate tool from Flood, we’ve enjoyed using it so much as a team that several of us have written about it:

Tim Koopmans: Load testing for DevOps… with Browser Level Users (BLU)

Kevin Dunne: A new approach to load testing with browser level users

me: Prepare for viral videos with Flood Element

Lachie Cox: Why should you load test with real browsers?

our friend Eduardo Riol, as translated by Antonio Jimenez: Puppeteer vs. Selenium for test automation: is a new leader emerging?

Clearly, we’re huge fans of load testing a web application with real browsers. But does the wider software testing industry share that sentiment?

Browser-based load testing in 2020

Browser-level load testing is still pretty new, and there aren’t too many providers offering it right now. Luckily, we just so happen to have access to a database of load tests run on Flood since 2013. Here’s a bar chart visualizing data we’ve collected about the load testing tools flooders have used (we support Selenium, JMeter, Gatling, and Element).

Load tests executed on Flood over time, grouped by load testing tool

Browser-level load testing is increasing in popularity

In 2020, almost 35% of the load tests run on Flood are browser-based load tests.

While the total number of load tests run across all tools increased across the board, the percentage of tests that utilized Selenium and Element, as compared to JMeter and Gatling, has also increased from 2018. Both Selenium and Element are browser-based load testing tools.

By contrast, JMeter, a protocol-level load testing tool which previously enjoyed a 70% share of the traffic on Flood, now accounts for only a bit over 50% of usage. JMeter’s share of the protocol-level pie is higher than Gatling’s, but the percentage of usage for both tools appears to be contracting when compared to that of browser-based tools.

Element use is increasing faster than Selenium use

Despite the fact that Selenium has been around longer, 25% of all floods today are run using Element. This is a significant increase in just two years, and one that bodes well for the future. While we’re still fans of Selenium and still support it, one reason we see for the dominance of Element in today’s market is resource efficiency.

In our baseline tests, we’ve seen that Element can run from 20-50 users per Flood node (an AWS m5.xlarge instance), whereas Selenium can only run up to 5. This leads to some significant cost savings, as running the same number of users with Element requires fewer load generators than with Selenium.

Another reason might be the fact that Element uses Puppeteer in the background. Puppeteer, developed by Google, understandably shines when driving Google Chrome (and recently, Firefox). Puppeteer load testing is growing in popularity due to the wealth of metrics it has access to.

How browser-based load testing works

Traditional load testing is done on the protocol level by sending messages directly to the web server. These messages are sent without simulating the application interface, and the responses that are received from the server are not visualized, either. “Users” in protocol-level load testing scripts are actually threads, with each one sending a list of messages at predefined intervals, and we refer to these users as Protocol-Level Users (PLU). JMeter and Gatling use PLU to generate load.

The screenshot below shows what one request for a single action might look like in a JMeter test plan.

Example of an HTTP request with parameters on JMeter

In browser-based load testing, real browser instances are opened on load generators, and the load testing tool plays the part of a real user in navigating the web application using the GUI. We refer to the virtual users in browser-based load testing as Browser-Level Users (BLU).

Here’s an example of the same action in the JMeter example above, but this time in an Element test script:

step('Step 1', async browser => {
 //Click Start button
 let startButton = await browser.findElement('.btn-default')
 await startButton.click()
})

This version simply describes the button to be found and clicked, whereas the protocol-level version needs to be more precise in what exactly is passed to the server.

Both types of load testing tools can apply the same load on an application (as long as it supports both methods— see the next section). The main difference is in how the load is generated.

A protocol-level script will contain instructions on which messages to send, which protocols to use, which headers to include, and other information that a typical end-user would not see.

A browser-level script will contain instructions on how to interact with the application: which buttons to click, what to type in text fields, which links to follow, and other on-screen elements that an end-user would be familiar with. This leads us to one of the big advantages of browser-based load testing.

Pros and cons of browser-based load testing

Pros

Simple scripting— or no scripting at all

Protocol-level tools require in-depth knowledge about how an application works, what format it expects requests to be in, and how it responds. On the browser-level, you don’t need to know any of that. When scripting a login process, for instance, you just need to tell the script to type a username and password in the right fields and then click Log In. The script doesn’t need to know that the form submits an encrypted request using the Auth0 API (something you’d need to know for a protocol-level script).

In some cases, you might be able to get away with using something like our qTest Explorer plugin to record a business process, generating a script without any scripting at all. Even when you have to write or modify the script, though, it’s a lot less intimidating than most protocol-level scripts.

They’re also easier to troubleshoot. In addition to opening a real browser instance on your computer when you run a script locally, Element can also display this information on the console (shown here on iTerm):

‍ Screenshots upon execution on Element using iTerm

‍This allows you to understand what’s going on in the script and in the application, letting you detect bottlenecks faster.

Reduced test complexity

The streamlined requirements for scripting lead to a straightforward test. There’s no chance of getting request parameters or values wrong.

Ability to test entire stack from the user perspective

Unlike testing APIs, you don’t need to decide which server you want to hit or make sure you have the appropriate requests to hit all the servers. A browser-level test script interacts with page elements like a real user would, so all underlying application servers are tested. Pair this with cloud-based infrastructure on Flood, and you’ve got a compact package for testing your application.

Capable of testing any user behavior

Protocol-level load test tools struggle with some types of applications, such as single-page apps (SPAs), where a lot of the processing is done on the client side. Browser-level load testing tools shine here, because scripted interactions for these apps are no different for SPAs than other web apps.

Record network and user interaction times for front-end optimization

Browser-level tools show you response times from the user’s perspective: how long after a user clicks the log in button does the user see his or her account balance? This metric is a practical one that may differ wildly from the traditional response times that a protocol-level tool reports. Measuring real user experience is a great value add for your test.

Easier to maintain

Lower script complexity means that a browser-based test script is more resilient to changes in the codebase. A change in the underlying requests sent will not require a change in the test script, making it perfect for regression testing.

It’s also easier to pass on to other members of the team due to high readability. Here’s an example of how to select a value from a dropdown menu in Element:

await browser.selectByValue('#challenger_age','34')

It’s easy to see that the script is looking for an element with the ID “challenger_age” and selecting the value “34”. This step could correspond to dozens of requests that need to be scripted if testing on the protocol level, which makes those scripts harder to understand without a proper handover.

Cons

Browser-based load testing tools have a higher CPU and memory footprint than protocol-based tools. Test scripts in JMeter and Gatling send messages to and record responses from a server without really needing to parse responses. By contrast, browser-based tools parse the server response in order to access the Domain Object Model (DOM) elements on the page.

Using real browsers might be closer to the real user experience, but it also takes its toll on node resources. It is possible to run in headless mode, where browsers are started “invisibly” without actually graphically rendering them. However, even then, a browser-level tool can run fewer concurrent users on a node than a protocol-level tool.

Best tools and applications for browser-based load testing

While other commercial tools exist for this, we’re partial to open source tools at Flood.

Selenium

Selenium load testing is appealing because Selenium is already well-known for test automation and functional testing, and a lot of companies already have full test suites in Selenium. The possibility of reusing those resources and just running them at scale as load tests is, by far, the most compelling reason to use Selenium.

Puppeteer

Google’s just getting started with Puppeteer, and the amount of metrics it exposes is exciting: it seems like anything in Chrome Developer Tools can be accessed by Puppeteer. Imagine running a load test on ONE tool that gives you statistics about back-end performance (response times, the effect of bandwidth restrictions, resources returning errors) AND front-end performance (client-side script console logs, render times, time to first paint, etc). It’s hard to imagine a future where this ISN’T a valuable tool in the performance arsenal.

Under Puppeteer, of course, is Element— our load-flavored version of Puppeteer.

Is protocol-based load testing dead?

No. Testing on the protocol-level may have its drawbacks, but it’s popular for a reason. If you’re testing mobile apps or API endpoints, protocol-level testing tools allow you to focus on hitting what you want to test and only what you want to test. Sometimes you don’t want to test the whole stack. It really depends on your test scenarios.

Plus, protocol-level tools tend to be really resource efficient. We’ve been able to easily run about 1000 users per Flood node with JMeter and Gatling, easily outstripping browser-level tools. It makes sense, since JMeter and Gatling don’t need to start browser instances to generate load.

It’s significantly cheaper to generate very high loads with protocol-level tools. This efficiency makes protocol-level tools a good choice for high concurrency load testing and especially stress testing.

What are we doing to support BLU?

Clearly, our usage metrics show that browser-based load testing is on the rise, and we’re planning to ramp up our offerings in this area while also maintaining support for protocol-level tools. Here’s what you can look forward to from us:

New releases
- We just released version 1.2.3 of Element a few weeks ago, which addresses some bugs and improve stability for certain actions.
Dedicated Element team
- We’ve put together a team of developers who will be focusing on improving Element exclusively. Despite Element being an open source tool, we’re putting our own resources behind it to improve it faster.
Partnering with consulting services
- It can be difficult to get started on a new tool, and Element is no exception. While we provide a load testing platform to run Element scripts, some companies want help to create the scripts in the first place. To this end, we’ve partnered with expert consultants who have already been using Element with great success: Foulk Consulting and Planit Testing.

Conclusion

When choosing a performance testing tool, it’s important to consider the advantages and disadvantages of protocol-level and browser-level testing tools. We think both types of tools deserve to be in a solid performance and load testing strategy. They can also be used in tandem in a hybrid load testing approach, improving overall application performance and scalability.

In 2018, we predicted that testing web-based applications with real browsers would increase in popularity, so we’re thrilled to see our theory validated in 2020. Browser-based load testing is increasing in usage, and our open source tool, Element, is leading the charge. It’s exciting to be at the forefront of this revolution, and we intend to increase support for load testing with real browsers even more this year.

Ask a Flooder 11: What are the best JMeter plugins?

Tue, 14 Apr 2020 19:47:53 +0100

Originally posted here.

In this video, I talk about how to install the JMeter Plugin Manager as well as other plugins, what the best JMeter plugins are, and why you need them. JMeter is an open-source load testing tool that is already robust, but there are some fantastic plugins that have been developed for it that you might want to use in your next load test.

Audio version here:

Hi everyone, Nicole here, back again for another Ask a Flooder, and I hope everybody is staying alert, not anxious, during these crazy times. In today’s Ask a Flooder, our customer asks: “What are the best JMeter plugins?”

What is a JMeter plugin?

So first, I want to talk about what a JMeter plugin is and how you can actually install one. Apache JMeter is a really great, robust load testing that is one of my personal favorites. There are still some things that could be done better or could be extended, and the great thing about being an open-source tool is that anybody, pretty much, can just jump in and develop these plugins for JMeter to extend functionality, so that it supports more protocols or to customize load profiles in a way that is easier. We already have a blog post on this, and I will leave a link to that article as well in the description because it goes into a lot more detail. However, I still want to give you my personal top 3 JMeter plugins.

How do you install a JMeter plugin?

You can install a plugin for JMeter in two ways.

One is to download it from JMeter Plugins, which I will give you a link to, and then you paste that into either the lib — it’s usually in the lib folder, but it could be in the ext (extensions) folder. Each plugin is going to differ as to what you need to copy into what folder, but that’s why I prefer the second method, which is the Plugin Manager.

The Plugin Manager is a plugin itself, so you will initially need to do that copy and pasting of the jar file. Once you have the JMeter Plugin Manager installed, downloading and installing a plugin is as simple as choosing which ones you want and then clicking install. JMeter restarts, and that’s pretty much it. So I do recommend that the Plugin Manager be the first plugin that you install. But that’s still not my top 3 favorite - that’s just an optional step.

What are the top 3 JMeter plugins?

Custom Thread Groups

Now for the actual plugins. My #1 favorite is the Custom Thread Groups. See, with JMeter, you get the standard default thread group, and all that does is it asks you how many users you want to run, how many loops, the duration, and the ramp-up. That’s it. So that’s good for a lot of uses, but it doesn’t really help you in cases where maybe you want to plan a spike test, and you want to make sure that all the users have fully ramped up and gotten to a certain point, and then all hit the server at once. Or maybe you want to do a step load. Previously I’ve discussed stress testing and how a step load can be useful for that. In that case, the Thread Group doesn’t quite cut it because it only ramps up a user, and then it continues at the same level of load for the duration of the test.

So, if you want any sort of variation in the middle (increasing users or decreasing), then you’re going to want something else.The Custom Thread Groups plugin has a lot of different types of thread groups. My favorite is the Ultimate One because you can probably do almost anything on that one. It also has the Stepping, Concurrency, Arrivals, and Free-Form Arrivals, so you have quite a few options.

Throughput Shaping Timer

Plugin #2 is the Throughput Shaping Timer. I love it because not a lot of load testing tools, even the proprietary ones, actually have this. This is really handy when you have a project where you’re given a requirement that isn’t in terms of the number of users; it’s in terms of hits per second, or requests per second, or something to do with the actual throughput of the test.

The problem with that is that the throughput tends to change, so you can’t actually take as a given that it’ll remain constant for the duration of the test. Things like response time or latency are going to affect that throughput. For instance, when you have one user or one thread that’s running, and it sends a request and receives the response back in one second, you can’t necessarily say that that’s going to be the same when there are a thousand threads doing the same thing. The server’s under more load, so maybe it’ll take five seconds instead of one second— but that’s already changed your throughput. Before, one thread was doing one request per second, and now it’s doing one request every five seconds.

So it can get a little complicated, and the Throughput Shaping Timer is one of my favorites because it promises to solve all that. The Throughput Shaping Timer dynamically changes the think time, so at the beginning of a test, for instance, when there are no response time issues and requests are being returned really quickly, it might increase the think time so that it slows down your load test. So you’re not going over that throughput that you wanted to maintain. And then later on in the test, when all the users are ramped up, then it’ll decrease the think time that it uses. So it’s great because you just have this consistent throughput throughout your test. It’s a really intelligent way to test.

I do want to say, though, that you do also want to play around with this a bit and make sure you understand how this works. One page could have several requests in it, and you’ll want to take that into consideration when you’re choosing what to set in that timer.

HTTP/2 Request Sampler

My #3 is the HTTP/2 Sampler. If you don’t know what HTTP/2 is, it’s an upgraded or next-generation protocol to the normal HTTP that we already use. The HTTP/2 protocol is supposed to be faster and more efficient, so more and more web apps are switching to it because of that.

The problem is that JMeter’s HTTP request sampler is only really for HTTP sites, so the HTTP/2 sampler is an upgraded version of the HTTP sampler. It’s really easy to use. The experience is pretty much going to be the same for you, but in the background, it uses the faster HTTP/2 protocol rather than the slower one.

My colleague, Jason Rizio, actually wrote a whole blog post about this, so if you want to learn more, I will put a link in the description.

I hope that answered your question, and until next time, happy flooding!

Ask a Flooder 09: What are the best tools for load testing?

Tue, 31 Mar 2020 20:35:36 +0100

Originally posted here.

One of our most commonly-asked questions is: “What are the best tools for load testing?” The answer will depend on a few things: your reason for load testing, the application you want to test, what kind of load testing you want to carry out, your team’s experience with scripting, and how many virtual users you want to run.

Note: In this video, I say that traditional load testing tools can’t simulate requests that are triggered by interactions on the client-side. This isn’t technically true: if you know what the requests are, you can build those requests manually on a protocol-level tool (like JMeter or Gatling). However, this takes more work compared to browser-level tools that just do this natively. ‍

Transcript:

Hi everyone, Nicole back again with another Ask a Flooder, and this time I’m tackling one of the questions we get asked A LOT, which is:

What are the best tools for load testing?

Why open-source load testing tools are awesome

Now, I want to preface this by saying that we really love open-source tools here at Flood, because 1) they’re really pretty full-fledged these days. You don’t have to make any compromises on features, and, in fact, some of these open-source tools have more features than some newer load testing tools just because they’ve been out longer. And secondly, they have large communities. Because they’re open-source, lots of people are using it, and lots of people are developing for it. So there are lots of plugins normally, and they support any kind of protocol that you can think of. And lastly, cost. It’s hard not to mention cost because it can be a really important factor depending on the budget of your project. And if you’re starting out as a load tester, I think that it might even be the best way to start with load testing, because you can take that knowledge with you and apply it to other projects and other companies regardless of the budget that they have.

Questions to ask yourself in choosing a load testing tool

Why are you doing load testing?

I always advocate starting with WHY. Why are you actually load testing? Is there a particular production defect that you noticed that you’re trying to solve? Is there already a problem, or are you just preparing to launch? These are things that are going to inform the type of tool that you use because you might want to make sure that that tool supports the protocols that you need.

What are you load testing?

Secondly, what type of app are you testing? Because if you have a single-page app, where your end-user is interacting with the page and client-side scripts are running to make requests to servers, then that’s something that most traditional load testing tools aren’t going to be able to do.

There are two general types of load testing tools: there’s browser-based, and there’s protocol-based. I’ll leave a link in the description talking about the differences between the two, but in general, if you want to go for browser-level testing, I would suggest Selenium and Element. Now, they’re both very good options. Selenium is probably the more robust in terms of automation of the two, but Element is better one in terms of efficiency because it was built with performance in mind. So Selenium’s automation-first and Element is really performance-first.

Now, for protocol-level tools, you can’t go wrong with either JMeter or Gatling.

What experience does your team have with scripting?

Thirdly, what’s the level of technical expertise that your team has? I think it’s important to be realistic about what your team can do, given the amount of time that they have. So if you have a team that has never done load testing and has never done any scripting before, then I would suggest using our Test Builder or something similar. On Flood, we have a Test Builder that is really just an easy interface that actually runs JMeter under the hood, but it just kind of simplifies JMeter and boils it down to what are the URLs that you want to test. That is a really easy way to get started, and it’s also a great [way to run a] proof of concept. If you have some automation testers on your team that are trying to do the load testing, well, maybe you should stick to tools that they already use. So it might be easier for them to get up and running with either Selenium or Element rather than learning a new tool. And if you have developers on your team, you might consider Ruby JMeter rather than the vanilla JMeter because Ruby JMeter is a lot easier to deal with— because you don’t have to deal with the XML files of a JMeter file (a normal JMX script). You can just plan out your load tests in Ruby. Or, Gatling is also a really good option because it allows you to write scripts in Scala. Scala is really really powerful, and you can create an entire framework with Gatling, which is also something to keep in mind.

How many virtual users do you want to run in your load tests?

And lastly, how many users do you want to simulate? Now protocol-level tools are going to really shine here because Element and Selenium both require a higher usage of resources on your load generators just because of the complexities involved in starting a browser instance, whereas protocol-level tools are just super efficient at sending these requests. So I would suggest that if you’re talking about thousands of users, I would use the protocol-level tools unless there’s a really good reason why that’s not going to give you the results that you’re looking for. Between JMeter and Gatling– well, they’re both excellent tools, but Gatling has been shown to particularly be good for really high levels of usage. But JMeter is also a good alternative, and you can run thousands of users of JMeter without any problem.

Just get started!

As always, I’d suggest just getting started. Pick whatever tool sounds like it might be the best for you, do a proof of concept on that one, and you can always try the other ones as well. On Flood, we don’t restrict you to just using one tool, so you can use one; you can use all four; you can run them at the same time if you want. It all depends on what you need for your testing.

Till next time, happy flooding!

Ask a Flooder 06: How to record traffic from a mobile app with JMeter

Thu, 19 Mar 2020 20:50:44 +0100

Originally posted here.

How do you record the traffic from your native mobile app using Apache JMeter? In this video, I explain step-by-step how to begin load testing a native mobile app.

‍

You might want to be doing this because you’re load testing a mobile app but either you don’t have access to your developers right now so they can’t help you with how the requests are made or what the requests are, or maybe you just want to verify what you’ve already been told. Either way, I’m going to show you how using my Android device (it’s a Samsung Galaxy Note 10+) and a Macbook Pro 13". However, the same principles should apply to other mobile devices and laptops.

Step 1: Download and install the mobile app

The first thing you’ll want to do is download and install the app. Now, if the app has already been released, then that’s no problem. You just go to the Google Play Store and download it. However, if it hasn’t been released yet and you’ve been given an APK file for the app, then you’ll have to tweak your settings a little bit in order to get your phone to allow you to download it.

And I’m going to do this right with you. So go into Settings: pull down the little notification screen and hit the little gear icon, and you’ll see a bunch of settings there. Scroll down all the way to the bottom until you see About phone. Click that and then click Software Information. You’ll see the Build number halfway through the screen here– and I know this is weird, but click on it seven times. You have to draw your pattern if you have one set up, but you’ll see here that it says Developer Mode has been enabled. That’s all you need to do; from now on you can install any APK, but be careful, because if you don’t trust the source, you know, they could do some damage to your phone.

Step 2: Set up the HTTP(S) Test Script Recorder on JMeter

The second step is setting up an HTTP recorder on JMeter. So let’s head over to our laptop. Make sure you’re connected to a wifi network. Now, this is a completely new JMeter test plan here. So just right-click on Test Plan and then hover over Add, and then you’re going to go down to Non-Test Elements. Select the HTTP(S) Test Script Recorder. You’ll see here that the port it uses is 8888. We’ll need that for later. Now right-click on the Test Script Recorder, hit Add, and then Listener. We’ll add a View Results Tree to be able to capture the results that we need later. Now we’ll click on the Test Plan - right-click - and then we’re going to Add a Thread Group. Now let’s right-click on the Thread Group and go to Add > Logic Controller > Recording Controller. This is where the requests that we’re going to send are going to be recorded, and we’ll see both the requests and the responses in View Results Tree.

Step 3: Set up your mobile to use the HTTP(S) Test Script Recorder as a proxy

Now, the third step is to set up your mobile to use a proxy. So, back on the mobile, go back to Settings and hit Connections. And then click on Wifi. Now verify that the wifi network that you’re connected to on your mobile is the same one as the wifi network that you’re connected to on your laptop. If it’s the same, click the gear icon. And then hit Advanced all the way at the bottom. Now the Proxy settings are on the second option here so hit the down menu here and then hit Manual. Now, for the Proxy host name we’ll have to go back to the computer to see what our local IP address is.

So, in order to do that, you just have to go to System Preferences. I’m doing it from the Apple menu here and from System Preferences, click on Network. And once you’ve selected the wifi network, hit Advanced, and then TCP/IP. So what you’re looking for will be this IPv4 Address. Now you’re going to type that into the Host name field on your mobile. So let me just type that in.

You’ll note that this is actually a local IP address. It’s not your public IP address. Now for our proxy port, we’ll go and look at the laptop to see the Test Script Recorder Settings and it was 8888, so that’s what we’re going to put in here. All right. And then click on Save.

Step 4: Start recording

Step 4 is recording the traffic. Now, go back to your laptop on JMeter and click on the Test Script Recorder. Now we’re going to click on Start. It’ll come up with this popup box here and you can just click OK since we already trust the source. You’ll see this little popup here to confirm that we are recording traffic. Now let’s see if we’ve actually configured it correctly.

On our mobiles, let’s go back and open the app that you want to use. So I’m going to be using the IowaReporterApp. I just did an article on this on The New Stack (link in the description below) about how I would load test the Iowa Caucus Reporter App. So that’s what I’m going to be using for this tutorial as well. So I’m going to open that (“getting ready to caucus”) and I don’t have a valid precinct ID so I’m just going to put flood.io and I’m going to click Login. So it’s telling me that the connection isn’t private. I don’t actually expect this to work, because the app has been taken down and I don’t have valid login credentials, but I’m still just going to go forward. So, Proceed. You might have to do this as well if your certificates haven’t been set up correctly.

Okay, so now we’ve gotten this error here and it says, “Oops, something went wrong”. So it’s handy to know what it looks like on your mobile so that you can go into JMeter and see if that’s what you see as well.

So on JMeter let’s go to the View Results Tree and it looks like Google Play tried to do a few things beforehand but what we’re really looking for is this IDP caucus thing. So, these are the requests that my phone made and these are the responses. So this is where we got the certificate error because it says “certificate unknown”. Now this is the actual request we sent for login. You can see the request here - the full request - and also the full response. Now, just to check we’ll make sure we got the same response in the body here on JMeter that we saw on our mobile phones.

And here it is; it says “Oops!, something went wrong”. You can of course just search for this. So we’ll stop the recording now because we’ve confirmed that we’ve gotten what we wanted. So in the recording controller, we can see the requests that were recorded from our mobile app and the first one looks like it was one to Google so that’s not one we want. If we were doing this again, we could actually go into the Test Script Recorder, go into this Requests Filtering tab, and we could just filter out or exclude the domains that we don’t want to test, like Google. But this looks like the request that we were actually looking for. You’ll see the full request here with all the parameters and the values.

What’s next?

And that’s how you record traffic from a native mobile app on JMeter. From here, you can then play around with the request. Like we saw, there were a lot of dynamic values that were recorded in the request. We would probably want to correlate those and make sure that we’re doing those correctly. But this is a really great starting point for testing a mobile app.

Till the next Ask a Flooder, happy Flooding!

Load Testing Lessons from the Iowa Caucus

Fri, 06 Mar 2020 21:59:02 +0100

Originally posted here and here.

TL;DR? Here are the highlights in this article:

Shadow, Inc. created a mobile app for the Iowa Caucus in February 2020, and the app failed, causing reported results to be delayed. The app reported different vote totals due to a system integration bug where the output was in a different format than what it needed to be. Here are the steps to load testing a mobile app:

Test planning: Does the app require load testing? If yes, what is the underlying technology? How do we create a workload model?
Scripting: Which mobile load testing tool should we use? How do we record traffic from a mobile device?
Execution: What load test scenarios should we have? How can we scale up the load generated? Conclusion: While the Iowa Reported App failure was not due entirely to load testing, load testing could have helped make it more robust.

A company called Shadow, Inc. made headlines earlier this week when they developed an app to count and report votes during the Iowa Democratic Caucus in the US. This app, called IowaReporterApp, ended up failing on several fronts, resulting in a days-long delay in publishing the total votes, during which manual methods had to be employed.

Source: @ShadowIncHQ on Twitter

This delay led to significant criticism from the software testing industry as well as from the general public. Some developers raised questions about the code quality of the app. The app inaccurately reported voting totals that differed from the Democratic Party’s own manual reports, suggesting an issue with functional testing, particularly system integration testing. Shadow’s refusal to accept the Department of Homeland Security’s help with security testing also raised eyebrows. What was perhaps the most damning, though, was the question of whether such an app was necessary in the first place, when cheaper and more effective solutions existed.

What I’m most interested in, however, are the load issues that the app faced. Let’s rewind a little to before the caucus. How would I have load tested an app like the IowaReporterApp?

Planning a mobile load testing

One of the first things to figure out when planning a load test is whether a load test is even necessary.

Do you even need to load test your mobile app?

To answer that, I had to find out more about the app.

In the caucus, residents of Iowa who registered themselves as Democratic voters gathered to vote on whom they thought should be the party’s candidate for the presidential elections later this year.

There are a few things that I think make this app the perfect candidate for load testing:

It was highly publicized. The caucuses would always have been an event of international interest, but the hype surrounding electronic voting increased media attention on the caucus even further.
It was mission-critical. The precinct chairs would use the app as the primary device for capturing votes.
It needed to handle significant simultaneous load. There were over a thousand precinct caucuses organized, each with a precinct chair that would be using the app.

For these reasons, the answer is a resounding “Yes.”

What type of app is it?

The best way to learn about an app is to ask its developers. In this case, I opted to do a little research on my own by downloading the .apk file that Motherboard (Vice) made available and looking through the code myself.

The app is a mobile-only app for Android and iOS devices, written mostly using Facebook’s React Native package. The primary function of the app was to sent the vote totals to Shadow Inc.’s Google Cloud Functions server.

The technology involved is worth noting because it can help determine which load testing tool to use. Since the Iowa Reporter App, like most mobile apps, sends requests to a RESTful API, JMeter or Gatling, both well-tested, robust, and open-source protocol-level tools with large user bases, were perfect for the job. I decided to go with JMeter, my personal tool of choice.

How do we create a workload model?

Now let’s talk about the workload model, which is a plan for how the load we generate needs to look. The objective here is to simulate the load that matches the expected real-life load as closely as possible.

At this point, I would typically look at historical data to try to see the amount of load and the load profile. If I had metrics on the server traffic after previous releases of the app using something like Google Analytics or an APM tool, I would have used that data as a starting point for the workload model of this release. However, in this case, the app hadn’t been publicly released before, so I needed to make some educated guesses.

According to Wikipedia, there were a total of 1,777 Iowa precinct caucuses, of which 99 were global satellite caucuses set up for Iowa residents temporarily not able to attend the local caucuses.

In a Democratic caucus, the actual voting process generally takes about two hours. In the Iowa caucus, the precinct chairs would have used the app twice - once for each round of voting. Around the end of the two hours, each chair’s app would have sent the main payload, containing total votes for each candidate, to the backend server. All this is done simultaneously across all 1,777 precincts.

At a minimum, there are two main events here:

The login - Precinct chairs needed to log in to the app, which would presumably have happened around the same time across all precincts towards the beginning of the voting process, because the app also contained instructions on how to run a caucus.
The submission - This is the transmission of the total vote counts for each candidate, sent to Google Cloud Functions, and would also have happened around the same time.

It’s possible that the app also sent other requests between these two events, but since most of the app is inaccessible without a valid login, I’ll focus on the initial login request.

In a situation like this where I don’t have previous data to show how spread out those requests was, it’s prudent to err on the side of caution. Assuming the precinct chairs triggered those requests within 10 minutes of each other and rounding up the number of requests for some wiggle room, I estimated that there would have been 1,800 login requests within 10 minutes.

I decided to test this as a gradual ramp-up of users within 10 minutes.

Scripting for mobile

Okay, time to write the script. Since I didn’t have the luxury of being able to ask the developers for the format of the login request, I did the next best thing and tried to find out myself by recording the app traffic in action.

Recording mobile traffic on a laptop with JMeter

Here’s how I was able to see the request from the mobile app:

I downloaded the app to a spare Android mobile by using the APK link above.
I set up an HTTP(S) Test Script Recorder on JMeter on my laptop with the port 8888. I also made sure my laptop was connected to my wifi network.
I set up my mobile to use a proxy using my laptop’s local IP address on the network and port 8888.
Then, I went back to JMeter on my laptop and started the HTTP(S) Test Script Recorder.
Finally, I navigated to the app, typed in flood.io in the Precinct ID field (obviously not a real ID), and clicked the “Click here to log in” button.

Here’s the traffic as seen on JMeter’s View Results Tree listener afterward:

And here’s the recorded request within the recording controller:

Analyzing the request

It looks like at least the initial login request is a GET to an Auth0 service, with several parameters being passed in the body. It also has some authorization headers.

The parameter values also look like they might be dynamic, meaning they change from session to session. At this point, I would typically change some of these from being hardcoded values into dynamic ones so that the application receives a request in the right format, and I could look through Auth0’s API documentation to find out how the authentication message should be formulated.

Since I don’t have a valid login for testing purposes, I skipped this step and just used the raw request for demonstration purposes.

Preparing to run

Think time is a delay between requests meant to simulate the time it takes for a user to think before taking another action. Without think time, JMeter would fire off requests one after another, which often isn’t realistic and drains resources. I don’t want that, so I added a JMeter timer. There are plenty of timers available, but I settled on a Uniform Random Timer.

The CEO of Shadow told Motherboard that many precinct chairs were confused by the login process as they had to put in three strings of numbers in the right sequence. So, I decided to work that into the script with the timer. I set it up with a constant delay of 2 minutes and a random delay of 1 minute so that the delay would vary from 2 minutes to 3 minutes to simulate a precinct chair attempting to log in a few times.

Execution: Running your mobile load test

According to Motherboard, Shadow has already taken the app servers offline. However, it’s not a good idea to run a full load test against an endpoint that you don’t own, so I decided not to run a test.

Usually, though, I’d want to start with a shakeout test of 10 users or so, and then gradually increase the user load until the peak of 1,800 users. I also would have played around with changing the load profile, such as by increasing or decreasing the ramp-up or steady-state durations, experimenting with stress tests and soak tests, and adding traffic from another availability zone to simulate the 99 satellite precincts around the world. Running multiple load generators can quickly get out of hand if you want to use physical machines, so I would run load tests on the cloud with something like Flood. All of these little changes would have given valuable data that would help me gauge the app’s performance in production-like situations.

Conclusion

Building quality software is not a trivial task. There are many moving parts to account for, and critical errors can be introduced in every stage of an app’s development. While load issues were not the main culprit in the failure of the IowaReporterApp, a simple load test like the one I describe here, might have improved app stability and reliability. Even in cases where load testing doesn’t seem necessary, it can still uncover gross issues in configuration, usability, or security for an app developed on a deadline.

How AirTasker uses Ruby JMeter and Flood for their load testing

Mon, 24 Feb 2020 18:23:31 +0100

Originally posted here.

In the recent RubyConf 2020 held in Melbourne, Australia, Nancy Cai from AirTasker talked about how they built a performance testing framework using Ruby JMeter and Flood. Luckily, we had some pleasantly surprised Flooders in attendance to take notes during this fantastic talk!

AirTasker is a platform allowing the outsourcing of everyday tasks. When discussing the reasons that AirTasker decided they wanted to carry out performance and load testing, Nancy told a fascinating story about an overly successful marketing campaign—a good problem to have, but a problem nonetheless. The Sydney Cats and Dogs Home, an animal shelter, teamed up with AirTasker to pay one lucky person $500 to spend the day playing with the resident animals. The prospect of this dream job created an enormous surge in AirTasker’s traffic that caused alarm bells to go off on the underlying application servers, resulting in an end-user experience that was less than desired.

Nancy included this hilarious slide about some common feelings regarding application performance.

Steps for running a performance test, the AirTasker way

To avoid repeating that mistake, Nancy and her team set out to run performance testing by using the following simple steps:

Determine performance criteria.
Configure the test environment to be as production-like as possible.
Plan and design based on user behavior, including preparing test data.
Implement test design and set up the framework (which in AirTasker’s case involved Ruby JMeter as the performance testing tool and Flood as a load testing solution).
Run test.
Analyze and fine-tune results.

Setting up a load testing tool and framework

When it came to selecting the right tools for the job, AirTasker wanted tools that met the following criteria:

Ability to simulate production-like traffic easily
Widespread use in the industry
Ease of creation and maintenance of tests for everyone
Historical results aggregation
Visualization of performance metrics
Integration with CI and monitoring tools

Nancy’s team compared several load testing tools and found they enjoyed using the Apache JMeter tool, which is a popular open source load testing software. However, they quickly ran into an “out of memory” error after attempting to run about 400-800 concurrent users on a single machine. She and her team decided to use a distributed load testing platform to run the load they required to scale up more effectively.

Enter Flood.

Why AirTasker chose Flood

We also chose Flood.io […] because it comes in with a really good open-source gem called Ruby JMeter. It is a Ruby-based DSL for building JMeter test plans. - Nancy Cai, AirTasker

Nancy and her team enjoyed the benefits of being able to write load tests natively in Ruby and then using Flood’s Ruby JMeter gem, which is a Domain-Specific Language (DSL). Flood allowed them to turn their Ruby scripts into .jmx files, which is the native XML format that JMeter uses.

When they were ready to scale up, AirTasker ran the same JMeter script in Flood.

The real-time Flood dashboard allowed them to see that there was a problem on one of the API endpoints, called “Get pet reviews.” Further investigations revealed a performance issue due to a long-running query against a database server in the code, which AirTasker’s developers could fix.

Best practices

Nancy gives some great advice for anyone running a performance test, including:

asking the right questions as a load tester
running baseline tests
being aware of the difference between your test and production environments
keeping in mind the caching and autoscaling mechanisms that may drastically change the results
the differences between stress testing and other types of performance testing

We were so thrilled to hear Nancy’s story on how Flood helped AirTasker run their performance tests. Catch the entirety of Nancy’s talk below, and check out the resource links at the end of the post to follow Nancy’s story.‍

Flood’s Ruby JMeter gem

Nancy’s GitHub repo where you can find a copy of her test scripts

Follow Nancy on Twitter

Follow Nancy on GitHub

Performance Testing Interview Questions, Part 2: JMeter

Thu, 20 Dec 2018 14:47:18 +0100

Originally posted here.

In this part, we’ll go over some questions that are commonly asked in technical interviews about the use of the most popular open-source load testing tool, Apache JMeter.

Using JMeter, you record a key business process. Upon playing it back, however, you notice an error on the first page. What could have gone wrong and how would you fix this?

While recorders have gone a long way towards making scripting easier, modern security measures like the implementation of viewstates have also made scripting more complicated to record. Some more complicated applications will pass dynamic values in responses that it expects to see in the succeeding request, otherwise the request is blocked. This is great for security, but it also means that you will need to manually extract these values in your script.

In JMeter, you would do this by adding a response extractor to your sample that will extract the dynamic string according to rules you set, and then save the string to a parameter. This parameter must then be used in the next request that you send.

While running your test plan, you get the error: java.lang.OutOfMemoryError: Java heap space. How do you fix it? The easy fix is to edit jmeter.bat and increase the heap memory (search for HEAP) and increase heap space allocated like this: HEAP="-Xms512m -Xmx2048m"

The part in bold increases the heap space to 2 GB.

However, it’s also worthwhile looking at your test plan and removing any obvious memory hogs. One of the things we see a lot is enabled listeners that aren’t even being used. Listeners are great while debugging, but should be disabled when running a load test. Writing too many lines to the log is another culprit here that should also be commented out when running important tests. And of course, you should always run JMeter in non-GUI mode during load tests!

After running a load test, the results show that 300 transactions were executed with a PASSED status and returning responses of HTTP 200. However, the DBA tells you that there were no records written to the database as a result of the transactions. How would you troubleshoot this?

I see this happening with clients all the time, so I think this makes a great technical interview question. This is also why I said earlier that you shouldn’t count on a test tool to tell you a test has passed. It’s possible for a transaction to have the status PASSED and return an HTTP 200 but still be hiding an error.

This is mainly due to poor scripting that does not include validations that check whether the script has gotten to the right page. For example, you could use a Response Assertion as a child of your home page sample with the words “Welcome, Nicole” to determine whether the user Nicole has logged in successfully.

This is a critical step, because without it, it’s possible for a server to return a nice error page, or an incorrect one, such as the home page, with an HTTP 200.

How do you use a data file to read user account details from a CSV in JMeter?

JMeter has a CSV Data Set Config Element built specifically for this purpose. You can simply add one to your test plan, browse to the CSV file that you’d like to use, and select the options for how to share that data among the threads in your JMeter script.

How would you go about testing a website that uses HTTP/2 on JMeter?

There is a JMeter plugin called HTTP/2 Sampler that accomplishes just this. My colleague Jason just wrote a blog post on this very topic, but basically the plugin adds new HTTP/2 versions to the standard HTTP request and request defaults samplers as well as the Simple Data Writer and View Results Tree listener. These elements will need to be used in order to test an application using HTTP/2.

What can you do to synchronise the execution of multiple JMeter instances on different load generators?

Apache JMeter has released this handy PDF file to describe the steps necessary to run JMeter in distributed mode. This involves installing Java and JMeter on all machines and setting them up in a master and slave configuration so that one machine controls the rest. On the master, you’ll need to add remote hosts to the jmeter.properties file. When everything is set up, you should be able to open up JMeter on the master and do a a remote start of your current test plan on any or all slaves that you choose.

The problem with this setup is the clunkiness of working with different machines and essentially having to set them up the same way. After the test, there’s the daunting task of logging back onto each machine, gathering the data, and transferring them back into a central repository. Then you’re faced with consolidating the data from each injector in order to see a picture of the whole test. You can see that this set up quickly goes from time-consuming to a nightmare, especially when potentially thousands of different machines are involved.

If only there were an easier way…

Should I Upgrade to JMeter 4.0? 5 Reasons to Upgrade

Fri, 23 Feb 2018 23:43:25 +0100

Originally posted here.

Apache has just launched JMeter 4.0, a big step up from their previous release, 3.3, which was only released in September last year. Apache seems to be stepping up the frequency of it’s releases, and we’re not complaining. JMeter is an amazing open source load testing tool and the most popular one used at Tricentis Flood by far.

The big question is: Should you upgrade? Are the new features worth the leap? Will the new black UI make you a better load tester? While this choice should be made on a case-by-case basis and we certainly can’t pretend to give one-size-fits-all advice, here are some good arguments to do it:

#1: Java 9 support

JDK 9 was released back in September 2017, and if you were one of the people who upgraded to it when prompted, you would have noticed to your frustration that JMeter 3.3 did not play well with it, yielding the error:

Error: Java version -- 9.0.1 -- is too low to run JMeter. Needs a Java version greater than or equal to 1.8.0 errorlevel=3

Good news! This is now fixed. JMeter 4.0 runs smoothly on JDK 9. We’ll continue running on JDK 8 at Flood for consistency amongst all our open source load testing tools.

#2: New Boundary Extractor

Previously, my preferred way to extract dynamic values in JMeter was to use a regular expression extractor, but there’s no doubt that it can sometimes be tricky to get just the right text out of a response. Enter the Boundary Extractor.

The boundary extractor allows you to simply set the text immediately before and after the text you want to extract. This saves you wondering whether you should have used a + or a * in your regular expression, and is also easier to validate.

#3: New JSON assertion element

While XPath assertion was great, there was previously no counterpart for JSON. The JSON Assertion element fills that void nicely, using JsonPath to quickly assert the existence (or non-existence, with the Invert assertion option ticked) of a particular element.

#4: New throughput timer

We have a new throughput timer to play with, and it’s called Precise Throughput Timer.

Whereas the Constant Throughput Timer also had the disadvantage of constant pacing of requests, the Precise Throughput Timer is based on the Poisson Random Timer and randomly generates the delay between each request. The implementation seems to be well thought out, with the component reference section listing a lot of scenarios they’ve tested for that the previous Constant Throughput Timer might have struggled with. Throughput timers are one of my favourite JMeter features, and still rather unique in the industry.

#5: Test Plan Autosave

This one’s more of a usability tweak, but it’s a good one. JMeter now automatically saves your script when you run a test, saving you from those all-too-familiar “Wait… did I CTRL+S before I hit CTRL+R?” moments.

Click here to see the full release notes for version.

BONUS reason: Tricentis Flood support

You can now upload JMeter 4.0 scripts to flood.io and be up and running with your first test hosted on one of our grid nodes around the world. Our customer support team is always happy to help you make the switch to 4.0.