Element on Nicole van der Hoeven

Browser-based load testing in 2020

Thu, 23 Apr 2020 16:48:09 +0100

Originally posted here.

We first announced our open-source browser-based load testing tool based on Puppeteer, Flood Element, in February 2018. Despite our decision to make it an entirely separate tool from Flood, we’ve enjoyed using it so much as a team that several of us have written about it:

Tim Koopmans: Load testing for DevOps… with Browser Level Users (BLU)

Kevin Dunne: A new approach to load testing with browser level users

me: Prepare for viral videos with Flood Element

Lachie Cox: Why should you load test with real browsers?

our friend Eduardo Riol, as translated by Antonio Jimenez: Puppeteer vs. Selenium for test automation: is a new leader emerging?

Clearly, we’re huge fans of load testing a web application with real browsers. But does the wider software testing industry share that sentiment?

Browser-based load testing in 2020

Browser-level load testing is still pretty new, and there aren’t too many providers offering it right now. Luckily, we just so happen to have access to a database of load tests run on Flood since 2013. Here’s a bar chart visualizing data we’ve collected about the load testing tools flooders have used (we support Selenium, JMeter, Gatling, and Element).

Load tests executed on Flood over time, grouped by load testing tool

Browser-level load testing is increasing in popularity

In 2020, almost 35% of the load tests run on Flood are browser-based load tests.

While the total number of load tests run across all tools increased across the board, the percentage of tests that utilized Selenium and Element, as compared to JMeter and Gatling, has also increased from 2018. Both Selenium and Element are browser-based load testing tools.

By contrast, JMeter, a protocol-level load testing tool which previously enjoyed a 70% share of the traffic on Flood, now accounts for only a bit over 50% of usage. JMeter’s share of the protocol-level pie is higher than Gatling’s, but the percentage of usage for both tools appears to be contracting when compared to that of browser-based tools.

Element use is increasing faster than Selenium use

Despite the fact that Selenium has been around longer, 25% of all floods today are run using Element. This is a significant increase in just two years, and one that bodes well for the future. While we’re still fans of Selenium and still support it, one reason we see for the dominance of Element in today’s market is resource efficiency.

In our baseline tests, we’ve seen that Element can run from 20-50 users per Flood node (an AWS m5.xlarge instance), whereas Selenium can only run up to 5. This leads to some significant cost savings, as running the same number of users with Element requires fewer load generators than with Selenium.

Another reason might be the fact that Element uses Puppeteer in the background. Puppeteer, developed by Google, understandably shines when driving Google Chrome (and recently, Firefox). Puppeteer load testing is growing in popularity due to the wealth of metrics it has access to.

How browser-based load testing works

Traditional load testing is done on the protocol level by sending messages directly to the web server. These messages are sent without simulating the application interface, and the responses that are received from the server are not visualized, either. “Users” in protocol-level load testing scripts are actually threads, with each one sending a list of messages at predefined intervals, and we refer to these users as Protocol-Level Users (PLU). JMeter and Gatling use PLU to generate load.

The screenshot below shows what one request for a single action might look like in a JMeter test plan.

Example of an HTTP request with parameters on JMeter

In browser-based load testing, real browser instances are opened on load generators, and the load testing tool plays the part of a real user in navigating the web application using the GUI. We refer to the virtual users in browser-based load testing as Browser-Level Users (BLU).

Here’s an example of the same action in the JMeter example above, but this time in an Element test script:

step('Step 1', async browser => {
 //Click Start button
 let startButton = await browser.findElement('.btn-default')
 await startButton.click()
})

This version simply describes the button to be found and clicked, whereas the protocol-level version needs to be more precise in what exactly is passed to the server.

Both types of load testing tools can apply the same load on an application (as long as it supports both methods— see the next section). The main difference is in how the load is generated.

A protocol-level script will contain instructions on which messages to send, which protocols to use, which headers to include, and other information that a typical end-user would not see.

A browser-level script will contain instructions on how to interact with the application: which buttons to click, what to type in text fields, which links to follow, and other on-screen elements that an end-user would be familiar with. This leads us to one of the big advantages of browser-based load testing.

Pros and cons of browser-based load testing

Pros

Simple scripting— or no scripting at all

Protocol-level tools require in-depth knowledge about how an application works, what format it expects requests to be in, and how it responds. On the browser-level, you don’t need to know any of that. When scripting a login process, for instance, you just need to tell the script to type a username and password in the right fields and then click Log In. The script doesn’t need to know that the form submits an encrypted request using the Auth0 API (something you’d need to know for a protocol-level script).

In some cases, you might be able to get away with using something like our qTest Explorer plugin to record a business process, generating a script without any scripting at all. Even when you have to write or modify the script, though, it’s a lot less intimidating than most protocol-level scripts.

They’re also easier to troubleshoot. In addition to opening a real browser instance on your computer when you run a script locally, Element can also display this information on the console (shown here on iTerm):

‍ Screenshots upon execution on Element using iTerm

‍This allows you to understand what’s going on in the script and in the application, letting you detect bottlenecks faster.

Reduced test complexity

The streamlined requirements for scripting lead to a straightforward test. There’s no chance of getting request parameters or values wrong.

Ability to test entire stack from the user perspective

Unlike testing APIs, you don’t need to decide which server you want to hit or make sure you have the appropriate requests to hit all the servers. A browser-level test script interacts with page elements like a real user would, so all underlying application servers are tested. Pair this with cloud-based infrastructure on Flood, and you’ve got a compact package for testing your application.

Capable of testing any user behavior

Protocol-level load test tools struggle with some types of applications, such as single-page apps (SPAs), where a lot of the processing is done on the client side. Browser-level load testing tools shine here, because scripted interactions for these apps are no different for SPAs than other web apps.

Record network and user interaction times for front-end optimization

Browser-level tools show you response times from the user’s perspective: how long after a user clicks the log in button does the user see his or her account balance? This metric is a practical one that may differ wildly from the traditional response times that a protocol-level tool reports. Measuring real user experience is a great value add for your test.

Easier to maintain

Lower script complexity means that a browser-based test script is more resilient to changes in the codebase. A change in the underlying requests sent will not require a change in the test script, making it perfect for regression testing.

It’s also easier to pass on to other members of the team due to high readability. Here’s an example of how to select a value from a dropdown menu in Element:

await browser.selectByValue('#challenger_age','34')

It’s easy to see that the script is looking for an element with the ID “challenger_age” and selecting the value “34”. This step could correspond to dozens of requests that need to be scripted if testing on the protocol level, which makes those scripts harder to understand without a proper handover.

Cons

Browser-based load testing tools have a higher CPU and memory footprint than protocol-based tools. Test scripts in JMeter and Gatling send messages to and record responses from a server without really needing to parse responses. By contrast, browser-based tools parse the server response in order to access the Domain Object Model (DOM) elements on the page.

Using real browsers might be closer to the real user experience, but it also takes its toll on node resources. It is possible to run in headless mode, where browsers are started “invisibly” without actually graphically rendering them. However, even then, a browser-level tool can run fewer concurrent users on a node than a protocol-level tool.

Best tools and applications for browser-based load testing

While other commercial tools exist for this, we’re partial to open source tools at Flood.

Selenium

Selenium load testing is appealing because Selenium is already well-known for test automation and functional testing, and a lot of companies already have full test suites in Selenium. The possibility of reusing those resources and just running them at scale as load tests is, by far, the most compelling reason to use Selenium.

Puppeteer

Google’s just getting started with Puppeteer, and the amount of metrics it exposes is exciting: it seems like anything in Chrome Developer Tools can be accessed by Puppeteer. Imagine running a load test on ONE tool that gives you statistics about back-end performance (response times, the effect of bandwidth restrictions, resources returning errors) AND front-end performance (client-side script console logs, render times, time to first paint, etc). It’s hard to imagine a future where this ISN’T a valuable tool in the performance arsenal.

Under Puppeteer, of course, is Element— our load-flavored version of Puppeteer.

Is protocol-based load testing dead?

No. Testing on the protocol-level may have its drawbacks, but it’s popular for a reason. If you’re testing mobile apps or API endpoints, protocol-level testing tools allow you to focus on hitting what you want to test and only what you want to test. Sometimes you don’t want to test the whole stack. It really depends on your test scenarios.

Plus, protocol-level tools tend to be really resource efficient. We’ve been able to easily run about 1000 users per Flood node with JMeter and Gatling, easily outstripping browser-level tools. It makes sense, since JMeter and Gatling don’t need to start browser instances to generate load.

It’s significantly cheaper to generate very high loads with protocol-level tools. This efficiency makes protocol-level tools a good choice for high concurrency load testing and especially stress testing.

What are we doing to support BLU?

Clearly, our usage metrics show that browser-based load testing is on the rise, and we’re planning to ramp up our offerings in this area while also maintaining support for protocol-level tools. Here’s what you can look forward to from us:

New releases
- We just released version 1.2.3 of Element a few weeks ago, which addresses some bugs and improve stability for certain actions.
Dedicated Element team
- We’ve put together a team of developers who will be focusing on improving Element exclusively. Despite Element being an open source tool, we’re putting our own resources behind it to improve it faster.
Partnering with consulting services
- It can be difficult to get started on a new tool, and Element is no exception. While we provide a load testing platform to run Element scripts, some companies want help to create the scripts in the first place. To this end, we’ve partnered with expert consultants who have already been using Element with great success: Foulk Consulting and Planit Testing.

Conclusion

When choosing a performance testing tool, it’s important to consider the advantages and disadvantages of protocol-level and browser-level testing tools. We think both types of tools deserve to be in a solid performance and load testing strategy. They can also be used in tandem in a hybrid load testing approach, improving overall application performance and scalability.

In 2018, we predicted that testing web-based applications with real browsers would increase in popularity, so we’re thrilled to see our theory validated in 2020. Browser-based load testing is increasing in usage, and our open source tool, Element, is leading the charge. It’s exciting to be at the forefront of this revolution, and we intend to increase support for load testing with real browsers even more this year.

Tips and Tricks for Using Flood Element

Fri, 23 Nov 2018 23:07:24 +0100

Originally posted here.

In case you missed it, there’s a new performance test tool on the block, and it’s called Flood Element. It’s a scalable browser-based tool that allows you to write scripts in javascript that interact with web pages like a real user.

Using Browser Level Users is a newer approach to load testing that overcomes a lot of the common challenges we hear about traditional methods of testing:

Easier scripting that is akin to common functional tools like Selenium and easier to learn
More realistic results that are based on true browser performance rather than API responses
Ability to test against all components of your web app, including things like Javascript that is rendered via the browser

Given the above benefits, it’s a no brainer to check out Flood Element for your web load testing, especially if you have struggled with the existing tools on the market like JMeter or HP LoadRunner.

Pairing Element with Flood turns it into a pretty powerful load test tool. We have a great guide here that you can follow if you’d like to get started with it. As you can imagine, I’ve been using and testing Element for several months now and I thought I’d share some tips I’ve learned along the way.

Initialising your script

You can always start from scratch, but the quickest way to get started is to type

element init myfirstelementtest

from your terminal, filling in the project name you’d prefer.

You’ll then be asked to type in the title of your test as well as the URL you’d like to script against. After a minute, you’ll see that a new directory has been created:

Element will automatically create a file called test.ts. This file will contain the skeleton of a script, along with some sample code to help you find a button and then click on it. But before you open it, let’s move on to…

Choosing the right text editor

Scripting in Element is already pretty simple, but I’ve found two things that really help while scripting are syntax highlighting and code completion. Syntax highlighting will greatly improve your experience when first learning a new test tool like Element, and code completion will make your scripting lightning fast as you become more experienced in the process. My text editor of choice is Visual Studio Code, which has both of those features. It’s slick and clean, and it does the job.

Syntax highlighting is when the text editor intelligently changes the font colour of your code according to its role in the programming language you’re using. Here’s a screenshot of the test.ts file we generated earlier in VS Code to show you what I mean:

This makes it easier to make sense of the code at a glance: comments are in green, values and labels are in orange, etc.

Code completion is when you start to type something, and VS Code helpfully opens a context menu with suggestions for methods you can use.

I love this because it means I don’t have to remember the exact name of the method. It also suggests names of variables you’ve already defined and highlights code that doesn’t make sense. This will help to make your tests more maintainable and readable for others, which is a great benefit as you look to scale your testing out in the future.

Taking screenshots

One of the most powerful features of Element is its ability to take screenshots. I find it immensely useful when debugging because sometimes it’s just easier to see what’s going on visually. With protocol based tools, debugging can often be a much more involved and technical process.

There are two ways to take screenshots in Element:

Add a setting to automatically take a screenshot when an error is encountered. You can do this by setting screenshotOnFailure to true in TestSettings:

export const settings: TestSettings = {
device: Device.iPadLandscape,
userAgent: 'flood-chrome-test',
clearCache: true,
disableCache: true,
screenshotOnFailure: true,
}

Explicitly take a screenshot at a particular point in the script. You can do this by adding

await browser.takeScreenshot()

to your code.

Viewing Screenshots

Once you’ve taken screenshots within your tests, you will likely want to view them and know that they will be stored for future safekeeping. Whether you are running your test locally on have uploaded it to Flood to run with increased concurrency, Flood Element has you covered.

Locally Run Tests

Screenshots will be saved as jpg files in a timestamped folder corresponding to your run. It should look something like: ...myfirstelementtest/tmp/element-results/test/2018-11-20T135700.595Z/flood/screenshots/ and the screenshots will be uniquely named so that new screenshots, even for the same step, don’t overwrite older ones.

However, I rarely have to look up the screenshots in that folder because I prefer to see them in iTerm2 for macOS. iTerm is an alternative to the terminal that works particularly well with Element. When you take a screenshot, iTerm actually shows it in-line:

‍

Tests Run in Flood

Running an Element script on Flood is ideal when you need larger concurrency. Rather than accessing your screenshot locally, Flood will centralize the images into your account so the images remain even after the cloud load injectors are destroyed. You can get to the screenshot files by downloading Archived Results:

Or you can also choose to click on a step on the dashboard to see a film strip of your test:

Using Logs

You may need to check out the logs for more technical debugging, especially when the screenshots don’t tell the whole story. Whether you are running your test locally on have uploaded it to Flood to run with increased concurrency, Flood Element has you covered.

Locally Run Tests

You can print to the console by typing, for example:

console.log('orderValues = ' + orderValues)

This will print the value of the variable orderValues at that point in the script. You would see this in your terminal if you’re running Element locally.

Tests Run in Flood

If you’re running the script on Flood, you can either download the log (in the same Archived Results zipped file as mentioned earlier) or you can click on the Logs tab:

‍

Fun with Flags

Element comes with a few flags that give you more control over how the script is run locally. Here are a few of my favourites!

Headless Flag

When in doubt, run Element in non-headless mode to see the script actually opening the web app on Chrome and interacting with the page. This is only possible locally, but there’s nothing like actually seeing for yourself what’s happening in real time instead of relying on screenshots and logs after the fact. To enable this mode, add the flag when running your test:

element run myfirstelementtest.ts --no-headless

Watch Flag

Element will automatically close the browser window when it encounters an error or finishes the iteration. Adding --watch will leave the browser window open and then will monitor the script. As soon as the script is saved, it will automatically run it in the same window from the beginning. Simply add this flag like the above example:

element run myfirstelement.ts --watch

Dev Tools Flag

This opens a browser instance and runs the script with the Chrome Dev Tools open, allowing you to find locators for the next action you want to script. Simply add this flag like the first example:

--dev-tools

More Flags

For more flags, use:

element run --help

Try Element

You’ve just gotten a crash course on Flood Element. You’re all set! If you haven’t already, download Element here to start writing functional test scripts and reusing them as load test scripts on Flood. If you don’t have a Flood account, you can easily sign up for a free trial here.

We’re extremely proud to have been able to contribute to the open-source community and can’t wait to have you try this new addition to the Flood line!

Prepare for Viral Videos With Flood Element

Thu, 08 Nov 2018 23:24:31 +0100

Originally posted here.

One of the more interesting things to load test is video. You see, traditional load test tools are protocol-based, which means that their approach to load testing video is to capture and simulate the traffic that the browser generates under the covers, sending the same requests to the application. This approach can still be successful– in fact, one of our favourite clients here at Flood recently used Gatling to simulate 6 million concurrent users streaming video from their application. However, for some use cases, testing at the protocol level can’t simulate everything that happens in a real world use case.

The Challenge

If you’re a video creator, the ultimate goal is often a simple goal: creating a viral video. Unfortunately, preparing for and simulating this all-important use case is typically impossible with protocol based tools. The same feature that makes them so efficient – not using a browser to send the requests – is the same thing that prevents them from being used to simulate the needed browser level events. You see, things like view counts on a video are normally tracked by scripts running on the page. Traditional tools download these scripts along with other page resources, but the scripts themselves are not executed. Enter Flood Element.

A Solution: Flood Element

Flood Element represents a new way of looking at load testing. It is a browser-based tool, which means that it actually drives an instance of the browser to test an application. Rather than telling it what requests to send to simulate a user like you would do with a protocol-based tool, you tell Element which buttons to click or which elements on a page to interact with just as a real user would. Essentially Element IS a user, using your application the way it was meant to be used. This means that it also executes any scripts that would run on the browser side. It’s so realistic that you could even use Element to make your YouTube video viral* by scripting up users to view your content… hypothetically speaking.

Simulating Real User Views

Testing video with Flood Element is actually pretty easy, especially if you are familiar with open source testing tools like Selenium. If you haven’t yet begun using Element, go through the installation guide here. You can download the full script used for this tutorial here. In this example, we’re going to be using a YouTube video of the opening credits the classical cartoon Top Cat (the indisputable leader of the gang) from Warner Bros.

Step 1. Navigate to the video page

Let’s start with a basic script:

import { step, TestSettings, By, Until } from '@flood/element'
export const settings: TestSettings = {
 loopCount: -1,
 clearCache: true,
 disableCache: true,
 actionDelay: 8,
 stepDelay: 10,
 screenshotOnFailure: true,
 userAgent: 'flood-element-test',
}
export default () => {
 step('01_Home', async browser => {
 await browser.visit('https://www.youtube.com/watch?v=6fvhLlrBrPQI')

 await browser.takeScreenshot()
 })
}

This is the bones of a basic script in Flood Element. It consists of a single step that involves doing two things: visiting the starting URL, which in this case is the direct link to the YouTube video, and then taking a screenshot once loading the page. Taking screenshots is particularly useful when testing video because screenshots give you a way to check that your script is doing what you want it to do. It’s good to take periodic screenshots as well, to measure progress.

To help you validate that the script runs well locally before scaling to hundreds or thousands of users, you can use our Element CLI. If you save the above code as youtube.ts, open up your terminal in that directory and then type:

element run youtube.ts

Assuming you’ve got Element set up correctly, you should see something like this: (Note that screenshots are only available when using iTerm as your shell. They are also saved as files under .../tmp/element-results/youtube .) ‍

Oops! It looks like our script does navigate to the page, but it doesn’t play the video. We’ll need to click on that play button. Let’s add that to our script.

Step 2. Click the Play button

We’ll need to add some lines to click the play button after visiting the page but before we take the screenshot, so that we can verify later whether or not the play button was clicked.

step('02_ClickPlay', async browser => {
 //Click on Play button
 let playBtn = await browser.findElement(By.xpath('//button[@aria-label="Play"]'))
 await playBtn.click()

await browser.takeScreenshot()
 })

In this example I used Xpath, but you do have the option to select based on partial text, CSS selector, etc. A tool I like to use to get these is Chrome Developer Tools’ Elements tab, which allows you to get the selectors for a particular element.

Step 3. Take periodic screenshots

To make sure that the video is really playing, we’ll need to take a screenshot a few seconds after the initial screenshot in order to visually confirm that the video has progressed. Even better, we can get Element to take screenshots at regular intervals. So let’s add a third step:

step('03_Play', async browser => {
 //Take a screenshot every 5 seconds until video finishes
‍
 for (i = 0; i <= 60; i++) {
 await browser.takeScreenshot()
 await browser.wait(5)
 }
 })

To use i, we’ll also need to declare it earlier in the script:

import { step, TestSettings, By, Until } from '@flood/element'
‍
var i = 0

In this step, i will correspond to every 5 seconds of playtime for the video, so we’ll need to make sure that the limit of i that we set is sufficient time for the entire video to play. Every 5 seconds, Element will take a screenshot, which will help us verify that the video is progressing after the test.

Click here to download the script before moving on to the next step, unless you’ve been following along at home.

Step 4. Run it on Flood!

The final step is to upload it to Flood and run the test. Click here to see a step-by-step guide for how to do just that. You can choose how many users you’d like to run on Flood and across how many nodes. New users will be given 5 node hours which will allow them to run a test with up to 250 users for 1 hour. Each node will have a different IP address, so plan for that accordingly if your application takes that into account.

One of the things I love about running it this way is that if you click on the arrow to the right of the sole transaction, you can see screenshots taken along the way to prove that the video is being played:

Some Considerations for Load Testing Video with Flood Element

Taking screenshots and using logs are necessary for debugging.

Because the page elements while a video is streaming are static, we’re not going to be able to use those as clues as to where we are in our scenario. To compensate, it’s a good idea to take screenshots and use console.log() to determine progress, especially while debugging. Be careful not to go overboard on these when actually running the load test though, as they will consume system resources.

Videos may behave differently in headless mode.

While writing this tutorial, I realised that YouTube auto-plays videos when on non-headless mode (aka, when opening a video link manually on a browser or with the --no-headless flag in Element). When it’s executed in headless mode via Flood Element, it requires a click on the play button to begin. Again, use screenshots to aid you in spotting these differences.

Conclusion

Element’s browser-level scripting capabilities allow it to simulate users that interact with pages just like human users would. Element and Flood are a powerhouse combo, providing the flexibility to test with thousands of users across the world within a few minutes.

Element seeks to reduce the gap between a scripted user and a real user – and it’s so realistic that it’s hard to tell the difference. We encourage you to take advantage of our free trial offer and run some real world load tests today!

*We don’t advise or condone the use of Element to artificially inflate views on YouTube in any way which may violate its Terms of Service. The purpose of this article is to showcase Element and how realistically it recreates real users.