Puppeteer on Nicole van der Hoeven

Testing tool cheat sheet for load testers who don't just care about numbers

Mon, 17 Jan 2022 16:30:07 +0100

It’s not enough to decide how much load you want to generate. You also have to know how you’re generating it.

Your approach affects what part of your application is tested, the results returned, and the budget you’ll need to run your test.

Here are three main categories of load testing tools:

Protocol-based load testing tools:

These tools simulate the underlying messages sent back and forth between an application and a client.

Choose a protocol-based tool if: you want to generate massive load, you want to load test before components are integrated, you want to test APIs, or you are budget-conscious.

Best in class tools: Gatling, JMeter, k6, Locust, Neoload

Browser-based load testing tools:

These tools automate real user interactions with an interface.

Choose a browser-based tool if: you want to measure end-to-end performance, you want to reuse your browser automation scripts, you want front-end performance metrics, or you have a higher budget.

Best in class tools: Cypress, Flood Element, k6, Playwright, Puppeteer

Hybrid load testing tools:

This approach is a mixture of the previous two. You can do this in a few ways:

Use a protocol tool to generate the majority of the load, and a browser tool for a handful of virtual users (best functionality, highest effort/cost)
Use a browser-based tool for all load (expensive, simplest)
Find a tool that can do both (ideal solution, few options)

Whatever you choose, compare like with like.

Happy testing!

Realistic load testing

Sat, 07 Aug 2021 22:15:13 +0200

This is a conversation I recorded from Clubhouse in the Performance Engineers group created by NaveenKumar Namachivayam today, August 7th, 2021. I started by introducing myself and then let everyone know that I would be recording the conversation. Enjoy!

Cuándo JMeter no alcanza: Pruebas de carga simulando navegadores

Wed, 16 Dec 2020 18:18:23 +0100

Aquí está el video de mi presentación en JMeter Scripting Day, un evento organizado por JMeter en Español el 31 de Oktubre, 2020.

Y la versión audio:

Introducción

Hola a todos! Yo soy Nicole van der Hoeven, y soy performance tester. JMeter es una de mis herramientas favoritas porque es gratis, simple, y por muchos casos hace exactamente lo que quiero y no más. Sin embargo, como performers es parte de nuestro trabajo elegir la herramienta adecuada para cada proyecto, y hay que admitir que hay situaciones en que JMeter simplemente no es suficiente.

Amazon - UI + Dev Tools

Esto es el sitio web de Amazon.com. Tengo abierta la pantalla de Chrome Developer Tools, que me muestra las solicitudes (o requests) que componen la página. Cuándo vamos a la página, el navegador (Chrome en este caso) envia estas solicitudes para nosotros, pero eso occure en el fondo. Un usuario normal no las ve.

JMeter - Creando un script

Lo que JMeter hace es grabar estas solicitudes y replicarlas. Aqui vemos una ejecución de un usuario virtual de un script en JMeter. Cada solicitud tiene la respuesta y el tiempo de respuesta. Pero eso no nos da una idea completa del rendimiento de la aplicación. ¿Por qué?

Por que JMeter, como otras herramientas de nivel protocolo, tiene unas limitaciónes. La primera es que si hay javascript o otros script que deben estar ejecutado por el navegador, JMeter no los ejecuta. JMeter solo los descarga. El tiempo de respuesta mide la descarga, no la duración de la ejecución. Entonces, siempre habrá una diferencia entre los tiempos de respuesta en una prueba de JMeter y los tiempos de respuesta en práctica.

A veces esta diferencia no es tan importante. Pero hay más y más aplicaciones hoy en día que se componen principalmente de estos javascript. Por esas aplicaciones, simplemente no es posible hacer un script en JMeter porque la mayoría de la lógica de la aplicación occure en el navegador, no en el servidor.

Flood Challenge

La segunda razón es que el aumento de la seguridad de aplicaciones conduce a solicitudes más complejas que incluyen parámetros dinámicos. Estos parámetros hacen que sea más difícil para un intermediario lanzar un ataque, pero también hacen que sea más difícil para nosotros scripters, porque tenemos que extraer información desde la última respuesta y ponerla en la próxima solicitud. Es posible, pero es la pesadilla de cada performance tester.

Podemos verlo en la aplicación de muestra Flood Challenge. Ya no trabajo para Flood, pero todavía es un buen ejemplo.

Correlación en JMeter - Flood challenge script

Esto es mi script en JMeter, y vemos que hay muchos parámetros que he tenido que correlacionar. La solicitud para simular hacer clic en el botón “Start” necesita los parámetros authenticity token, step ID, y step number, que son dinámicos. El contenido de los parámetros cambian cada vez que usamos esta aplicación.

La experiencia del usuario

La tercera razón es que JMeter tampoco mide la verdadera experiencia del usuario. Solo mide la duración de la comunicación entre el navegador y el servidor, pero no mide cosas como la representación de elementos en la página web, ni el efecto de un script que dura mucho tiempo para terminar.

Eso es importante porque estas cosas afectan significativamente la experiencia del usuario. De hecho, Steve Souders, un conocido el la industría de performance, dice que hasta 90% del tiempo de respuesta que experimenta el usuario está en la interfaz. Con este numero podemos estar ciertos que una prueba con una herramienta de nivel protocolo no va a poder encapsular el rendimiento final de una aplicación.

Developer Tools - Lighthouse

Entonces, ¿qué podemos hacer?

Lo más fácil es intentar de medir la interfaz durante una prueba de cargas. Mientras ejecutamos una prueba con JMeter, podemos ir manualmente a la aplicación web en Chrome para usar Developer Tools. Podemos usar la pestaña Lighthouse, que nos da más métricos del perspectivo del usuario y sugerencias para mejorar el rendimiento.

WebPageTest

También uso WebPageTest a veces. Eso tiene más opciones para ejecutar la prueba automáticamente varias veces desde un lugar que puedo elegir. Aún podemos elegir un dispositivo para simular la experiencia de un usuario en un iPad, por ejemplo.

Herramientas alternativas para pruebas de carga

Tambien podemos elegir herramientas alternativas para ejecutar nuestras pruebas de carga. Lo que tienen en común es que fueron diseñadas para pruebas de automatización, y utilizan un navegador real.

Selenium es probablemente la más popular de esas. Selenium WebDriver en particular. Es compatible con Ruby, Java, Python, C# y Javascript. El problema con Selenium es que puede ser muy pesada. En una machina probablamente podemos ejecutar cinco usuarios de Selenium, más o menos. Eso es importante si estamos tratando de simular miles de usuarios.

Hay muchas herramientas basadas en Javascript que también podemos usar. Hay varias framework que se utilizan para adaptar Javascript y nodeJS.

Puppeteer fue desarollado por Google. Tricentis Flood también ha diseñado Element que utiliza Puppeteer, pero con un enfoque en el rendimiento.
Playwright es una creación de Microsoft.
Jest fue desarrollado por Facebook
Mocha es otro framework que es la base para Cypress.

Todos esas herramientas son open source y gratis. También hay alternativas propietarias, pero prefiero las soluciones open source porque en la mayoría de casos hacen la misma cosa y se actualizan con más frecuencia.

Element script

Esto es un ejemplo de la misma acción, hacer clic en el botón Start en la aplicación Flood Challenge.

Otra ventaja de simulando navegadores en vez de simulando las solicitudes es que es más simple crear un script. No tenemos que saber cada parámetro. Solo tenemos que describir lo que queremos que cada usuario virtual haga.

Podemos ejecutar scripts usando estas herramientas al mismo tiempo que un script de JMeter o Gatling o lo que sea, y también tenemos la opción de reemplazar los script de protocolo con estas.

Conclusión

Para resumir, JMeter y otras herramientas de nivel protocolo tiene algunas limitaciones.

No ejecutan scripts del lado del navegador
Pueden ser más complicado correlacionar parámetros dinámicos
No miden la experiencia final de un usuario

Para solucionar estos problemas, podemos complementar pruebas de carga tradicionales, de nivel protocolo, como JMeter, con herramientas simulando navegadores. De este modo podremos tener una comprensión más completa del rendimiento total de la aplicación.

Bueno, se acabó el tiempo, entonces los dejo aquí. Gracias por escuchar, y me disculpo por mi español. Gracias a todos!

Ask a Flooder 16: How do I use test data with Element?

Tue, 09 Jun 2020 22:18:41 +0100

Originally posted on Tricentis Flood.

In this video, I discuss how to make your load testing scripts more realistic by including test data. The load testing tool I use is Flood Element, but you can apply these principles to any tool. You’ll find a copy of the Element script I used below.

‍

And, if you prefer, here’s the audio version:

Transcript

Hi everyone, Nicole van der Hoeven here back with another Ask a Flooder, and this time the question is: How do I use test data with Flood Element?

Now let’s talk about why you might want to use test data. In load testing, it’s important to make your scripts behave as similarly to real users as possible so that the load that you generate is production-like. The problem is that if you send the same data over and over again to the server, that server is probably going to cache that response. The response being cached means that the server will return the response way faster than it normally would if it had been a unique request. So we can use test data, instead of static data hardcoded in your scripts, to make sure that this doesn’t happen.

Using test data can also help us test different responses. For example, let’s say we’re testing a loan application system, where a user inputs the application ID, the server looks it up, and returns the status of that loan application. Now, loan applications might be in different stages of the process. For instance, some might be saved but not submitted, or maybe they’ve been submitted but failed the identity verification check.

If we used the same application ID every time, then the server would always return the same status and the same details. Instead, let’s create a CSV file that we can use as test data. It might look something like this.‍

applicationId
A100001
B100002
C100003
D100004

We can have applicationId as the single header here, and we can populate this with a list of different application IDs in different stages of the process. That way, when we run the script, the server will have to respond authentically to every request without it being cached.

Now to create our script. This is an empty Element script. To start with, make sure you’re importing TestData here in the first line of your script.

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'

The first thing we’re going to need to do is to create an interface for the CSV file. Let’s call it AppData. Within this interface, we’ll need to define all the fields that we intend to use in our data file and what type of values we can expect to be in those fields. Looking at our CSV here, we’ve just got the application ID, so I’m just going to copy that, and it’s a mixture of characters and numbers, so I’m going to put that in as a string.

interface AppData {
 applicationId : string    
}

Now we need to tell Element how to find this test data. We’ll say that it’s from the CSV, using the interface we’ve just created. Now we’ll tell Element the filename. In this situation, the CSV file that I’m using is actually in the same directory, so I don’t need to specify a path here, but if you’re putting it somewhere else, then you’ll need to specify the folder and the path to it.

TestData.fromCSV<AppData>('applicationids.csv')

Now let’s use the data in a step. In real life, you might want to use an application ID to type it into a text field, but for demonstration purposes, I’m just going to show you how to output it to the console log. Printing to the console is also a good way to debug your tests, especially if you’re dealing with test data.

We need to let Element know that we want to use AppData. So now, let’s output this to the console. Since we’re just debugging, let’s output the application ID. To actually get the value from the data file, we’ll use this syntax:

step('Print to console', async (browser: Browser, data: AppData) => {
 console.log('application ID: ', `${data.applicationId}`) 
})

‍By default, Element reads the data files sequentially. The problem with that is we only have a handful of application IDs. So if we wanted to run this one script 100 times, we’re going to run out of data. What we can do here is add circular. That way, when the script reaches the very last value of the file, the next value it selects will be the first one, and it never runs out of test data.

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

Another thing you might be wondering is how to make it so that Element chooses a random value every time. To do that, we’ll also have to add shuffle:

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

‍Now let’s try to run the test.

element run test-data.ts

As you can see, it’s been finished. The application ID that was selected was indeed a random one because the first one would have started with A. So, it actually chose the last one randomly.

And that’s it for test data with Element. Now, this user actually had a follow-up question, which was about unique test data. I’m going to answer that in the next video. Until next time, happy Flooding!

Sample script

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'
import * as assert from 'assert'

export const settings: TestSettings = {    
 loopCount: -1,    
 description: 'Test Data',    
 screenshotOnFailure: true,    
 disableCache: true,    
 clearCache: true,    
 clearCookies: true,
}

/*** Test Data * Version: 1.0 */

export default () => {    
 interface AppData {        
 applicationId : string    }    
 TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()    
 .shuffle()    
 step('Print to console', async (browser: Browser, data: AppData) => {        
 console.log('application ID: ', `${data.applicationId}`)    
 })
}

Ask a Flooder 15: How do I install Element?

Tue, 12 May 2020 22:45:18 +0100

Originally posted here.

In this Flood Element tutorial, I demonstrate the steps required to install Flood Element, including dependencies it requires. Flood Element is an open-source browser-based load testing tool created by Tricentis Flood.

Learn more about Flood Element

Installation instructions for NodeJS

Installation instructions for Yarn

Installation instructions for Element

Video: Getting Started with Element

Guides: Using Element with Flood‍

Audio version here:

Transcript

Hi everyone, Nicole van der Hoeven here again, back with another Ask a Flooder, and today’s question is: “How do I install Element?”

Now, I already made a previous video on how to get started with Element, but I didn’t actually go into the details of how to install it, so that’s what I’m going to do in this video today. There are actually a few parts to installing Element, and all of these instructions are going to be on element.flood.io. If you go onto the site and click on Install, you’ll see all of the instructions depending on which operating system you’re on.

First, you need to install NodeJS. This is sort of like how, if you’re familiar with JMeter, you have to install Java before installing JMeter. NodeJS is what Element runs on. Secondly, you’ll need a package manager. You can use NPM or Yarn, but in this video, I’m going to take you through how to do it with Yarn. And thirdly, there’s installing Element itself.

First, download NodeJS. I’m going to leave links in the description to all of the sites that I visit so that you can go through all of the instructions later. On the Node site, download whichever installer suits your operating system. Go through the installation steps, and afterwards, check to make sure that NodeJS is properly installed on your machine. You can do that by going into the terminal and typing:

node -v

Just make sure that the version that’s shown here is the same version that you’ve downloaded.

Secondly, install the latest version of Yarn. On the Yarn site, you’re going to have to choose your operating system and the version. We recommend using the latest stable version. And then go through the instructions for your operating system. After you’ve done that, check your Yarn installation by typing:

yarn --version

And the last step is installing Element. On the Element site, you can find the exact code that you need to type into your terminal. It’s:

yarn global add @flood/element-cli

So you can just copy that into your terminal. As of the time of this recording, the latest version of Element is 1.2.3. Looks like that’s installed. To check the version of Element, type:

element --version

So that’s how to install Element. We’ve gotten a lot of questions recently about Element in particular, so I’m going to answering them in future videos. Hope this helped, and until next time, happy flooding!

Browser-based load testing in 2020

Thu, 23 Apr 2020 16:48:09 +0100

Originally posted here.

We first announced our open-source browser-based load testing tool based on Puppeteer, Flood Element, in February 2018. Despite our decision to make it an entirely separate tool from Flood, we’ve enjoyed using it so much as a team that several of us have written about it:

Tim Koopmans: Load testing for DevOps… with Browser Level Users (BLU)

Kevin Dunne: A new approach to load testing with browser level users

me: Prepare for viral videos with Flood Element

Lachie Cox: Why should you load test with real browsers?

our friend Eduardo Riol, as translated by Antonio Jimenez: Puppeteer vs. Selenium for test automation: is a new leader emerging?

Clearly, we’re huge fans of load testing a web application with real browsers. But does the wider software testing industry share that sentiment?

Browser-based load testing in 2020

Browser-level load testing is still pretty new, and there aren’t too many providers offering it right now. Luckily, we just so happen to have access to a database of load tests run on Flood since 2013. Here’s a bar chart visualizing data we’ve collected about the load testing tools flooders have used (we support Selenium, JMeter, Gatling, and Element).

Load tests executed on Flood over time, grouped by load testing tool

Browser-level load testing is increasing in popularity

In 2020, almost 35% of the load tests run on Flood are browser-based load tests.

While the total number of load tests run across all tools increased across the board, the percentage of tests that utilized Selenium and Element, as compared to JMeter and Gatling, has also increased from 2018. Both Selenium and Element are browser-based load testing tools.

By contrast, JMeter, a protocol-level load testing tool which previously enjoyed a 70% share of the traffic on Flood, now accounts for only a bit over 50% of usage. JMeter’s share of the protocol-level pie is higher than Gatling’s, but the percentage of usage for both tools appears to be contracting when compared to that of browser-based tools.

Element use is increasing faster than Selenium use

Despite the fact that Selenium has been around longer, 25% of all floods today are run using Element. This is a significant increase in just two years, and one that bodes well for the future. While we’re still fans of Selenium and still support it, one reason we see for the dominance of Element in today’s market is resource efficiency.

In our baseline tests, we’ve seen that Element can run from 20-50 users per Flood node (an AWS m5.xlarge instance), whereas Selenium can only run up to 5. This leads to some significant cost savings, as running the same number of users with Element requires fewer load generators than with Selenium.

Another reason might be the fact that Element uses Puppeteer in the background. Puppeteer, developed by Google, understandably shines when driving Google Chrome (and recently, Firefox). Puppeteer load testing is growing in popularity due to the wealth of metrics it has access to.

How browser-based load testing works

Traditional load testing is done on the protocol level by sending messages directly to the web server. These messages are sent without simulating the application interface, and the responses that are received from the server are not visualized, either. “Users” in protocol-level load testing scripts are actually threads, with each one sending a list of messages at predefined intervals, and we refer to these users as Protocol-Level Users (PLU). JMeter and Gatling use PLU to generate load.

The screenshot below shows what one request for a single action might look like in a JMeter test plan.

Example of an HTTP request with parameters on JMeter

In browser-based load testing, real browser instances are opened on load generators, and the load testing tool plays the part of a real user in navigating the web application using the GUI. We refer to the virtual users in browser-based load testing as Browser-Level Users (BLU).

Here’s an example of the same action in the JMeter example above, but this time in an Element test script:

step('Step 1', async browser => {
 //Click Start button
 let startButton = await browser.findElement('.btn-default')
 await startButton.click()
})

This version simply describes the button to be found and clicked, whereas the protocol-level version needs to be more precise in what exactly is passed to the server.

Both types of load testing tools can apply the same load on an application (as long as it supports both methods— see the next section). The main difference is in how the load is generated.

A protocol-level script will contain instructions on which messages to send, which protocols to use, which headers to include, and other information that a typical end-user would not see.

A browser-level script will contain instructions on how to interact with the application: which buttons to click, what to type in text fields, which links to follow, and other on-screen elements that an end-user would be familiar with. This leads us to one of the big advantages of browser-based load testing.

Pros and cons of browser-based load testing

Pros

Simple scripting— or no scripting at all

Protocol-level tools require in-depth knowledge about how an application works, what format it expects requests to be in, and how it responds. On the browser-level, you don’t need to know any of that. When scripting a login process, for instance, you just need to tell the script to type a username and password in the right fields and then click Log In. The script doesn’t need to know that the form submits an encrypted request using the Auth0 API (something you’d need to know for a protocol-level script).

In some cases, you might be able to get away with using something like our qTest Explorer plugin to record a business process, generating a script without any scripting at all. Even when you have to write or modify the script, though, it’s a lot less intimidating than most protocol-level scripts.

They’re also easier to troubleshoot. In addition to opening a real browser instance on your computer when you run a script locally, Element can also display this information on the console (shown here on iTerm):

‍ Screenshots upon execution on Element using iTerm

‍This allows you to understand what’s going on in the script and in the application, letting you detect bottlenecks faster.

Reduced test complexity

The streamlined requirements for scripting lead to a straightforward test. There’s no chance of getting request parameters or values wrong.

Ability to test entire stack from the user perspective

Unlike testing APIs, you don’t need to decide which server you want to hit or make sure you have the appropriate requests to hit all the servers. A browser-level test script interacts with page elements like a real user would, so all underlying application servers are tested. Pair this with cloud-based infrastructure on Flood, and you’ve got a compact package for testing your application.

Capable of testing any user behavior

Protocol-level load test tools struggle with some types of applications, such as single-page apps (SPAs), where a lot of the processing is done on the client side. Browser-level load testing tools shine here, because scripted interactions for these apps are no different for SPAs than other web apps.

Record network and user interaction times for front-end optimization

Browser-level tools show you response times from the user’s perspective: how long after a user clicks the log in button does the user see his or her account balance? This metric is a practical one that may differ wildly from the traditional response times that a protocol-level tool reports. Measuring real user experience is a great value add for your test.

Easier to maintain

Lower script complexity means that a browser-based test script is more resilient to changes in the codebase. A change in the underlying requests sent will not require a change in the test script, making it perfect for regression testing.

It’s also easier to pass on to other members of the team due to high readability. Here’s an example of how to select a value from a dropdown menu in Element:

await browser.selectByValue('#challenger_age','34')

It’s easy to see that the script is looking for an element with the ID “challenger_age” and selecting the value “34”. This step could correspond to dozens of requests that need to be scripted if testing on the protocol level, which makes those scripts harder to understand without a proper handover.

Cons

Browser-based load testing tools have a higher CPU and memory footprint than protocol-based tools. Test scripts in JMeter and Gatling send messages to and record responses from a server without really needing to parse responses. By contrast, browser-based tools parse the server response in order to access the Domain Object Model (DOM) elements on the page.

Using real browsers might be closer to the real user experience, but it also takes its toll on node resources. It is possible to run in headless mode, where browsers are started “invisibly” without actually graphically rendering them. However, even then, a browser-level tool can run fewer concurrent users on a node than a protocol-level tool.

Best tools and applications for browser-based load testing

While other commercial tools exist for this, we’re partial to open source tools at Flood.

Selenium

Selenium load testing is appealing because Selenium is already well-known for test automation and functional testing, and a lot of companies already have full test suites in Selenium. The possibility of reusing those resources and just running them at scale as load tests is, by far, the most compelling reason to use Selenium.

Puppeteer

Google’s just getting started with Puppeteer, and the amount of metrics it exposes is exciting: it seems like anything in Chrome Developer Tools can be accessed by Puppeteer. Imagine running a load test on ONE tool that gives you statistics about back-end performance (response times, the effect of bandwidth restrictions, resources returning errors) AND front-end performance (client-side script console logs, render times, time to first paint, etc). It’s hard to imagine a future where this ISN’T a valuable tool in the performance arsenal.

Under Puppeteer, of course, is Element— our load-flavored version of Puppeteer.

Is protocol-based load testing dead?

No. Testing on the protocol-level may have its drawbacks, but it’s popular for a reason. If you’re testing mobile apps or API endpoints, protocol-level testing tools allow you to focus on hitting what you want to test and only what you want to test. Sometimes you don’t want to test the whole stack. It really depends on your test scenarios.

Plus, protocol-level tools tend to be really resource efficient. We’ve been able to easily run about 1000 users per Flood node with JMeter and Gatling, easily outstripping browser-level tools. It makes sense, since JMeter and Gatling don’t need to start browser instances to generate load.

It’s significantly cheaper to generate very high loads with protocol-level tools. This efficiency makes protocol-level tools a good choice for high concurrency load testing and especially stress testing.

What are we doing to support BLU?

Clearly, our usage metrics show that browser-based load testing is on the rise, and we’re planning to ramp up our offerings in this area while also maintaining support for protocol-level tools. Here’s what you can look forward to from us:

New releases
- We just released version 1.2.3 of Element a few weeks ago, which addresses some bugs and improve stability for certain actions.
Dedicated Element team
- We’ve put together a team of developers who will be focusing on improving Element exclusively. Despite Element being an open source tool, we’re putting our own resources behind it to improve it faster.
Partnering with consulting services
- It can be difficult to get started on a new tool, and Element is no exception. While we provide a load testing platform to run Element scripts, some companies want help to create the scripts in the first place. To this end, we’ve partnered with expert consultants who have already been using Element with great success: Foulk Consulting and Planit Testing.

Conclusion

When choosing a performance testing tool, it’s important to consider the advantages and disadvantages of protocol-level and browser-level testing tools. We think both types of tools deserve to be in a solid performance and load testing strategy. They can also be used in tandem in a hybrid load testing approach, improving overall application performance and scalability.

In 2018, we predicted that testing web-based applications with real browsers would increase in popularity, so we’re thrilled to see our theory validated in 2020. Browser-based load testing is increasing in usage, and our open source tool, Element, is leading the charge. It’s exciting to be at the forefront of this revolution, and we intend to increase support for load testing with real browsers even more this year.

Ask a Flooder 12: Puppeteer vs Selenium: which one is better for load testing?

Tue, 21 Apr 2020 17:36:28 +0100

Originally posted here.

When it comes to software testing, the debate of Puppeteer vs Selenium is a hot one. Nicole tackles it here to determine which one is better for load testing in particular. Spoiler: The answer is still “it depends”.

Or catch the audio version here:

Hi everyone, Nicole here again, back for another Ask a Flooder, and today, the question is: “Puppeteer vs. Selenium: which one is better for load testing?”

We’re actually getting this quite a bit now as Puppeteer grows in popularity, but first, let’s discuss the similarities. They’re tools that you can use to drive browsers. The work by identifying and interacting with elements on a website in the same way that a user would. They are also both open-source, and both can be used for load testing. This is a great strategy because instead of writing tests for an automation suite and then tests for load testing, you can just write the one test suite and re-use it, so it saves a lot of resources in the end. To answer this question, I’m going to focus on load testing because that’s what my experience is in.

Difference between Selenium and Puppeteer

Pros and cons of Selenium

Now for the difference. Selenium supports more browsers than does Puppeteer. It supports Chrome, Firefox, IE, Safari, and Opera, so there are a lot more options to choose from depending on your application.

Selenium is also what I would call automation-first. It was built with automation in mind, which means it’s very good at that. It’s very robust, and if you have a complicated test scenario, Selenium is an excellent choice.

It’s also been around for much longer than Puppeteer, which is important because that means it’s had more time to build up a community. There are way more resources and tutorials for Selenium than for Puppeteer.

It’s also, unfortunately, very resource-intensive. That’s not something that you would typically care too much about from a test automation perspective, but when you’re running thousands of users on the cloud, it IS important to keep in mind that Selenium, just because of how it’s built and its complexity, can only run about five users per node. We’ve found that out at Flood, after baselining specifically an AWS m5.xlarge instance. So you can look at the specs of that and compare it with the machines that you’re wanting to run Selenium on, to see how that would equate for you.

Pros and cons of Puppeteer

Puppeteer is a NodeJS library developed by Google, which means it plays very well with Chrome. There are a whole lot of things that you have access to with Puppeteer, and if you’re using it with Chrome, pretty much anything in Chrome, including rendering times, you can see and access with a Puppeteer script. So that’s a huge advantage.

Unfortunately, that also means it supports fewer browsers. Obviously, it supports Chrome very well, and they recently came out with support for Firefox, although they do have plans to incorporate support for more browsers in the future.

One of Puppeteer’s advantages is that we at Flood actually created a tool on top of Puppeteer. It’s also open-source. It’s called Flood Element, and Element is built to be performance-first because obviously, that’s what we had in mind when we were building it. It takes everything that’s good about Puppeteer and makes it really easy to set parameters that you normally would expect to need for load testing, like users and ramp-up.

This also means that it’s way less resource-intensive. I said that with Selenium, you can run up to five users on an AWS m5.xlarge instance, and for the same instance type, you can run, from our experiments, anywhere from 30-50 Element scripts/users. That is a huge saving when you’re running a really large-scale test.

Should you choose Selenium or Puppeteer for load testing?

I would say that if you already have a large Selenium automation suite, if you have a complex test scenario, or if you aren’t running that many users for your load test, then Selenium is a great choice. If you ARE running thousands of users, though, and resource efficiency, and therefore cost efficiency, is really important to you, then choose Element.

My advice is always to try both. If you have the time, do a proof of concept with both of them. That way, your whole team can better understand what the pros and cons are for each one and decide which one is best going forward. Or maybe you don’t even decide on one. At Flood, we support both of them at the same price, so it depends on what you prefer.

Until next time, happy flooding!

Ask a Flooder 05: How do I get started with Flood Element?

Tue, 17 Mar 2020 20:58:12 +0100

Originally posted here.

On this Ask a Flooder, I talk about getting started with Flood Element. Element is an open-source tool that is a good Selenium alternative for browser-based load testing. It runs Puppeteer under the hood to drive real browsers at scale and can be paired with Flood for cloud load testing.

‍

TRANSCRIPT:

If you’re sold on browser-based load testing and want to get started on Element, I think the easiest way to do that is by going to element.flood.io and clicking on Install on the upper left corner. You can follow those instructions to get started, and you can also click on the links on the left side if you get stuck.

Once Element is installed, open up the directory and look at the examples folder. There you’ll find a lot of sample scripts that we’ve already made up for you. The easiest way to get started is to just modify some of these slightly so that you’re testing your site. For example, you could just change the URL and step names to make it appropriate for what you’re testing.

Then, you can run it locally using the command element run the name of your test script, like test.ts, and then I also like to include the no-headless flag.

element run test.ts --no-headless

The no-headless flag is going to run a real browser, a Chromium browser, on your machine, so that you can see what the script is actually doing. This is a great way to debug and troubleshoot.

Once you’re ready to run your load test on the cloud, you can simply drag and drop your Element script into the Flood stream editor and then start your load test as usual.

Check out the links in the description below for more information about Element, how to get started, and how to run an Element script on Flood.

More about Element

Getting started with Element