Ruby on Nicole van der Hoeven

Ask a Flooder 09: What are the best tools for load testing?

Tue, 31 Mar 2020 20:35:36 +0100

Originally posted here.

One of our most commonly-asked questions is: “What are the best tools for load testing?” The answer will depend on a few things: your reason for load testing, the application you want to test, what kind of load testing you want to carry out, your team’s experience with scripting, and how many virtual users you want to run.

Note: In this video, I say that traditional load testing tools can’t simulate requests that are triggered by interactions on the client-side. This isn’t technically true: if you know what the requests are, you can build those requests manually on a protocol-level tool (like JMeter or Gatling). However, this takes more work compared to browser-level tools that just do this natively. ‍

Transcript:

Hi everyone, Nicole back again with another Ask a Flooder, and this time I’m tackling one of the questions we get asked A LOT, which is:

What are the best tools for load testing?

Why open-source load testing tools are awesome

Now, I want to preface this by saying that we really love open-source tools here at Flood, because 1) they’re really pretty full-fledged these days. You don’t have to make any compromises on features, and, in fact, some of these open-source tools have more features than some newer load testing tools just because they’ve been out longer. And secondly, they have large communities. Because they’re open-source, lots of people are using it, and lots of people are developing for it. So there are lots of plugins normally, and they support any kind of protocol that you can think of. And lastly, cost. It’s hard not to mention cost because it can be a really important factor depending on the budget of your project. And if you’re starting out as a load tester, I think that it might even be the best way to start with load testing, because you can take that knowledge with you and apply it to other projects and other companies regardless of the budget that they have.

Questions to ask yourself in choosing a load testing tool

Why are you doing load testing?

I always advocate starting with WHY. Why are you actually load testing? Is there a particular production defect that you noticed that you’re trying to solve? Is there already a problem, or are you just preparing to launch? These are things that are going to inform the type of tool that you use because you might want to make sure that that tool supports the protocols that you need.

What are you load testing?

Secondly, what type of app are you testing? Because if you have a single-page app, where your end-user is interacting with the page and client-side scripts are running to make requests to servers, then that’s something that most traditional load testing tools aren’t going to be able to do.

There are two general types of load testing tools: there’s browser-based, and there’s protocol-based. I’ll leave a link in the description talking about the differences between the two, but in general, if you want to go for browser-level testing, I would suggest Selenium and Element. Now, they’re both very good options. Selenium is probably the more robust in terms of automation of the two, but Element is better one in terms of efficiency because it was built with performance in mind. So Selenium’s automation-first and Element is really performance-first.

Now, for protocol-level tools, you can’t go wrong with either JMeter or Gatling.

What experience does your team have with scripting?

Thirdly, what’s the level of technical expertise that your team has? I think it’s important to be realistic about what your team can do, given the amount of time that they have. So if you have a team that has never done load testing and has never done any scripting before, then I would suggest using our Test Builder or something similar. On Flood, we have a Test Builder that is really just an easy interface that actually runs JMeter under the hood, but it just kind of simplifies JMeter and boils it down to what are the URLs that you want to test. That is a really easy way to get started, and it’s also a great [way to run a] proof of concept. If you have some automation testers on your team that are trying to do the load testing, well, maybe you should stick to tools that they already use. So it might be easier for them to get up and running with either Selenium or Element rather than learning a new tool. And if you have developers on your team, you might consider Ruby JMeter rather than the vanilla JMeter because Ruby JMeter is a lot easier to deal with— because you don’t have to deal with the XML files of a JMeter file (a normal JMX script). You can just plan out your load tests in Ruby. Or, Gatling is also a really good option because it allows you to write scripts in Scala. Scala is really really powerful, and you can create an entire framework with Gatling, which is also something to keep in mind.

How many virtual users do you want to run in your load tests?

And lastly, how many users do you want to simulate? Now protocol-level tools are going to really shine here because Element and Selenium both require a higher usage of resources on your load generators just because of the complexities involved in starting a browser instance, whereas protocol-level tools are just super efficient at sending these requests. So I would suggest that if you’re talking about thousands of users, I would use the protocol-level tools unless there’s a really good reason why that’s not going to give you the results that you’re looking for. Between JMeter and Gatling– well, they’re both excellent tools, but Gatling has been shown to particularly be good for really high levels of usage. But JMeter is also a good alternative, and you can run thousands of users of JMeter without any problem.

Just get started!

As always, I’d suggest just getting started. Pick whatever tool sounds like it might be the best for you, do a proof of concept on that one, and you can always try the other ones as well. On Flood, we don’t restrict you to just using one tool, so you can use one; you can use all four; you can run them at the same time if you want. It all depends on what you need for your testing.

Till next time, happy flooding!

How AirTasker uses Ruby JMeter and Flood for their load testing

Mon, 24 Feb 2020 18:23:31 +0100

Originally posted here.

In the recent RubyConf 2020 held in Melbourne, Australia, Nancy Cai from AirTasker talked about how they built a performance testing framework using Ruby JMeter and Flood. Luckily, we had some pleasantly surprised Flooders in attendance to take notes during this fantastic talk!

AirTasker is a platform allowing the outsourcing of everyday tasks. When discussing the reasons that AirTasker decided they wanted to carry out performance and load testing, Nancy told a fascinating story about an overly successful marketing campaign—a good problem to have, but a problem nonetheless. The Sydney Cats and Dogs Home, an animal shelter, teamed up with AirTasker to pay one lucky person $500 to spend the day playing with the resident animals. The prospect of this dream job created an enormous surge in AirTasker’s traffic that caused alarm bells to go off on the underlying application servers, resulting in an end-user experience that was less than desired.

Nancy included this hilarious slide about some common feelings regarding application performance.

Steps for running a performance test, the AirTasker way

To avoid repeating that mistake, Nancy and her team set out to run performance testing by using the following simple steps:

Determine performance criteria.
Configure the test environment to be as production-like as possible.
Plan and design based on user behavior, including preparing test data.
Implement test design and set up the framework (which in AirTasker’s case involved Ruby JMeter as the performance testing tool and Flood as a load testing solution).
Run test.
Analyze and fine-tune results.

Setting up a load testing tool and framework

When it came to selecting the right tools for the job, AirTasker wanted tools that met the following criteria:

Ability to simulate production-like traffic easily
Widespread use in the industry
Ease of creation and maintenance of tests for everyone
Historical results aggregation
Visualization of performance metrics
Integration with CI and monitoring tools

Nancy’s team compared several load testing tools and found they enjoyed using the Apache JMeter tool, which is a popular open source load testing software. However, they quickly ran into an “out of memory” error after attempting to run about 400-800 concurrent users on a single machine. She and her team decided to use a distributed load testing platform to run the load they required to scale up more effectively.

Enter Flood.

Why AirTasker chose Flood

We also chose Flood.io […] because it comes in with a really good open-source gem called Ruby JMeter. It is a Ruby-based DSL for building JMeter test plans. - Nancy Cai, AirTasker

Nancy and her team enjoyed the benefits of being able to write load tests natively in Ruby and then using Flood’s Ruby JMeter gem, which is a Domain-Specific Language (DSL). Flood allowed them to turn their Ruby scripts into .jmx files, which is the native XML format that JMeter uses.

When they were ready to scale up, AirTasker ran the same JMeter script in Flood.

The real-time Flood dashboard allowed them to see that there was a problem on one of the API endpoints, called “Get pet reviews.” Further investigations revealed a performance issue due to a long-running query against a database server in the code, which AirTasker’s developers could fix.

Best practices

Nancy gives some great advice for anyone running a performance test, including:

asking the right questions as a load tester
running baseline tests
being aware of the difference between your test and production environments
keeping in mind the caching and autoscaling mechanisms that may drastically change the results
the differences between stress testing and other types of performance testing

We were so thrilled to hear Nancy’s story on how Flood helped AirTasker run their performance tests. Catch the entirety of Nancy’s talk below, and check out the resource links at the end of the post to follow Nancy’s story.‍

Flood’s Ruby JMeter gem

Nancy’s GitHub repo where you can find a copy of her test scripts

Follow Nancy on Twitter

Follow Nancy on GitHub