How to Become a Load Tester
Load testing can be a pretty intimidating field to get into, but it’s easy to get started. Here are some concrete steps you can take to start learning how to load test from scratch.
In this video, Flood co-founder Tim Koopmans and I teamed up at Tricentis User Conference 2020 to talk about the essential ingredients for load testing.
Background required for load testing
Load testing can be quite technical. It involves not just the ability to use different programming languages to create load testing scripts but also familiarity with software engineering practices, DevOps, application infrastructure, site reliability, and networking. A good load tester needs to be able to understand the basics of how an application works and how different components interact with each other. It’s common for a load tester to work on applications employing a variety of technologies, so staying up to date with recent technological trends and switching seamlessly from one language to another, or from one platform to another, is of utmost importantance. This technical side of load testing might make you think that a load tester needs to have formal studies in computer science and several years of experience in software testing. The reality is that there are many load testers who stumbled into the field from wildly unexpected backgrounds. So you might think load testing looks like this:
Photo by Christina @ wocintechchat.com on Unsplash
And sometimes it does. But it more often looks something like this:
Photo by You X Ventures on Unsplash
You see, load testing is an interdisciplinary field. So much of the job involves talking to people. Sometimes the requirements for a project are not clear, and someone needs to talk to the both the business and the developers to make sure that nonfunctional requirements are clarified. A load tester is a detective, tracing a request as it goes from one application component to another, coordinating with different application teams. A load tester is a scientist, using the scientific method to design experiments that yield the results. A load tester is a statistician, making sense of a jumble of data and seeking correlations between variables. A load tester is a public speaker. Even when the actual load testing has been completed and the report has been sent, a load tester makes sure all stakeholders understand the report so that the results can be used to make meaningful changes to improve application performance. All these tasks require a mishmash of skills and competencies that can be difficult to find in just the field of computer science.
Let’s take our backgrounds as anecdotal examples.
Tim grew up and currently lives in a small country town in Australia, and he joined the military straight after school. He later developed a love of automating things, which led him first to use tools like perl to scrape websites and try to win bids with early versions of ebay, and then later to starting a performance engineering consulting career and co-founding Flood itself.
I, Nicole, grew up in Metro Manila in the Philippines (which has 14,000 more people per km2 than Mumbai does). I learned English as a foreign language in school, and I have a university degree in Economics. When I discovered load testing, I had a job in a completely different field– I was working for an airline and making sure engine components were properly maintained and repaired. My love for gadgets and technology took me down an 8-year rabbit hole involving doing performance consulting for clients across a variety of industries in Australia and the Netherlands. At Flood, I help customers around the world improve their application performance.
As you can see, Tim and I came from very different backgrounds and different parts of the world. Neither of us knew what load testing even was when we started working. Neither of us got formal instruction in load testing. But somehow, we still both ended up becoming load testers at Flood.
Reasons to become a load tester
So why become a load tester? Here are some practical reasons.
Load testing is cool
A little-known fact about load testing is that it can be really satisfying to be involved in big-name projects that end up going live.
You may have heard of Flood. But even if you haven’t, we’re sure you’re familiar with the results of load testing done on Flood. In the last few years, we have been helping Hotstar, one of the largest streaming services in the world, load test their systems in preparation for the Indian Premier League.
Hotstar uses an open-source tool that we support, Gatling, to create scripts to simulate users tuning in to watch cricket games. Then, they use Flood to create a “Game Day” simulation. They scale up those load tests and run 5 million users on thousands of machines in the cloud. We help them coordinate execution, spot problems quickly with real-time results from all those machines while the test is still running, and identify bottlenecks so that they don’t occur in production, when India’s playing.
As a load tester, you’ll definitely be working with brands you recognize and love.
Load testing can pay well
Source: Payscale
Source: Payscale
I said this would be a practical list, right?
Here are 261,768 practical reasons. The average performance engineer’s salary is 71% more than that of an average software tester in India, according to Payscale.com. We all know that money isn’t everything, but this is a clear signal from the market about the importance of performance-related skills. Adding load testing to your CV as a tester makes you more attractive to employers. It’s a great way to differentiate yourself in an over-saturated market.
Load testing can literally take you places
Source: Then there are the side benefits of being in load testing. The photos above are real ones from the Flood team. That’s me on the upper left, exploring a castle in Belgrade, Serbia, and that’s Tim in the bottom right, taking a photo in San Francisco, USA.
We’re a fully distributed team on Flood, which means we can work from anywhere. Flood is a web app, so you don’t need to install anything– I’ve been known to start load tests from my iPad in a café. Load testing on the cloud frees you from being tied to running tests from the office. You can start load generators when you need them and shut them down when you don’t– all from a web browser. It’s a job that can come with a great work/life balance, if you find the right company.
Load testing can also be a great way to move to another country. It’s what’s allowed me to move to other countries– twice. In Australia, the average performance engineer’s salary is AU$108,053, or over 5.6 million rupee.
Source: Payscale
So it’s clear that there are some pretty good reasons for you to become a load tester. I’m going to walk you through the practical steps to become a load tester. You may not be an expert by the end of this, but I hope I can convince you that you can get started in load testing whether your background looks like Tim’s or mine.
Load testing adds real value
And the trite, but absolutely true reason: Everyone fails.
As a load tester, you are a gatekeeper of quality. What you do directly affects revenue and customer experience.
Performance is an aspect of software that is often overlooked, but it’s only becoming more important in a world where everything is going online. People are being born who have never known what it’s like not to have the internet, and consumers in general expect applications to be responsive and stable. Would-be customers who can’t get an e-commerce site to load within 3 seconds may switch to a competitor’s site, and we all know there are many of them. Performance-related outages can be seriously damaging to a company’s reputation, and social networking amplifies customer dissatisfaction by effectively giving everyone a megaphone to complain into.
Software functionality is still important. However, if an online store goes live with no critical functional defects, but it fails when the 100th person logs into it, or it takes 10 seconds to load the homepage, can you really say that it was ready for production?
Load testing is a crucial part of ensuring software quality, and quality is the responsibility of everyone on a team, even if the word “performance” isn’t in your job title.
The essential ingredients of a load test
Load testing is like cooking a meal. This is one of Tim’s favorite dishes, aloo matar.
Source: Cubes n Juliennes
You can approach cooking as a science. There are recipes with a list of ingredients, a precise measurement of how much you need for each, and a procedure for creating the dish.
But you can also look at cooking as an art. Cooking the same dish over and over again will give you an experience in working with those ingredients that a novice cook won’t have. There’s room in cooking for adding one ingredient, changing the quantity of another, or just trying something new.
A master chef knows how to use both the science and the art to create something delicious. But you don’t HAVE to be at that level in order to cook.
Anyone can cook. If you know how to follow a basic recipe, you may not create a masterpiece, but if you’re like me and you haven’t had lunch yet, you’ll still end up with something you’ll eat.
So here’s the recipe for load testing.
I prefer kadai paneer myself, but aloo matar requires, among other things, peas. And so does load testing.
P is Preparation. To prepare for a load test, you need to get your ingredients ready. You need to decide what it is you want to test, why you’re testing it, and how. You write load testing scripts that will simulate real users as closely as possible.
E is Execution. This is when you do the actual cooking. You run your load tests and scale them up according to your requirements.
A is Analysis. You figure out whether the dish you created is edible. You determine how your application servers responded during the load test, and decide if that’s acceptable.
Preparation – Execution – Analysis.
Preparing for a load test
“As far as the World Cup is concerned, it is a process. We don’t want to jump to the 50th floor straight away. We must start on the ground floor.” - Sachin Tendulkar, via ESPN
Load testing doesn’t start with writing a script. Load testing starts with building a foundation for later tests.
Source: Inside Sport
Part of that foundation is asking WHY. Why do you want to do a load test? Has there been a performance-related issue before? Are you trying to determine whether an application is ready to go live, or is it already live, and you just want to improve its performance? What type of application do you want to test?
WHOM are you cooking for? I think you’d cook differently for yourself than you would for Sachin Tendulkar. What do your end users typically do on your site? Where do they come from?
These are questions that need to be answered before you think about scripting. Here are some resource for planning your load tests:
Start with Why: How to write requirements for API load testing How to write a test plan for load testing Selecting a tool for load testing Generating a load testing script without coding: Getting Started with Test Builder
Execution of a load test
So now we’re ready to start cooking. This is the Execution phase. You’ll need a machine to use as a load generator. Depending on the number of users you want to run with, you may need more than one machine. You might be tempted to run a load test on your laptop, or on a bunch of machines you have lying around, but that introduces new variables that may affect the results.
Source: Ashwini Chaudhary on Unsplash
Like any good cook, you’ll also need to make sure you have all your ingredients handy. In load testing, that means making sure your scripts and test data files are ready to be executed on every load generator.
Both of these problems – provisioning a load generator and transferring files over to them – are solved pretty easily by running on the cloud using something like Flood.
Designing and executing your load test
Where should you run your load tests? A look at demand, hosted, and on-premise infrastructure
Analysis of Results
The last part is Analysis.
Source: Fine Dining Lovers
This is a pretty crucial moment, because this is when you figure out whether what you cooked was delicious or not. Unfortunately, in load testing, it’s not always easy to tell how your test went. It’s like this photo of pakora (onion is my favorite kind). It’s kind of hard to say what expression that is in their faces. Is it disgust? Or is the pakora so yummy that they just can’t wait to eat more?
Load testing typically generates a LOT of data. You’d have to go copy every single file from every load generator you used. But can you imagine the hundreds of load generators that Hotstar used? Of course they didn’t manually copy those files. Instead, they used Flood.
On Flood, we show real-time graphs as you’re running the test so that you can tell immediately whether transactions are failing or whether a server has stopped responding.
After the test, you can also compare this data to previous tests to spot trends, or even create your own dashboards using an analytics engine.
Real-time metrics during your load test
Downloading results and logs after the test
Is that it?
You’ve prepared for a load test, you’ve executed a load test, and you’ve analysed the results from the load test. Congratulations, you’re a load tester! By following this recipe, you’re already doing better than many people, and you’re certainly doing a lot better than those who do not run any load tests at all.
The cool thing about load testing, though, is that this is just the beginning. There’s always room for improvement.
Source: Pop Up City
Dabbawallas are people who deliver hot lunches, and while the concept is not new, the implementation in India is particularly interesting.
For these dabbawallas, having the meals prepared is not the end of the job. Once the food is ready, it needs to be packed into lunchboxes, picked up, separated according to destination, put on trains, and delivered. Later, the empty lunchboxes are picked up, and the system works in reverse.
It turns out that this is actually a pretty sophisticated system. In fact, it’s been estimated that only one tiffin every two months is not delivered. That’s equivalent to a Six-Sigma quality rating of 99.9999999% (that’s nine nines) or more.
Dabbawallas didn’t get that rating just by delivering quickly. They did it by being consistent and reliable. They did it by having a process.
Once you see the benefits of running a load test, you may also start to understand that the real value of load testing is not in a test that you run once whenever you think of it. Imagine having the benefits of a load test tied to your development process, so that it is automated.
As an advanced topic, adding your load testing to your team’s Continuous Integration/Continuous Development pipelines such as sets up a repeatable process that reaps benefits far into the future, not just whenever official load testing activities are carried out.
Continuous Integration with Flood: Integrating Travis CI, Circle CI, and Jenkins
Conclusion
So that’s the recipe for becoming a load tester.
Preparation – Execution – Analysis.
In Preparation, you determine why you’re testing and what tool best fits your needs. We showed you how to use Element, but there are many open source tools out there. Don’t get overwhelmed by scripting– a lot of times it can just mean replacing a URL with the one YOU want to test.
In Execution, you run a baseline test that will serve as a point of comparison for future testing, choose the number of users and the number of regions you want, and then you run your test. Running on the cloud is the easiest (and usually cheapest) way to get started.
In Analysis, you view the results in real-time, compare historical performance to see trends, and share reports to stakeholders so that your team can make changes to your application.
We also talked about the bonus topic of CI/CD and how you can incorporate load testing into your pipeline to get a continuous feedback during your development cycle.
Tim and I started in very different places and had different experiences before we discovered load testing. But eventually, we ended up on the same path, and that’s what we’ve outlined for you here. We hope that this convinces you that load testing doesn’t have to be complicated.
You absolutely don’t need to be an expert to run a load test. Even if you don’t think you’ll get all these steps right, a load test that is less than ideal is still better than no load testing at all.
In the end, anyone can be a load tester with the right attitude and the right tools.
Learning about load testing
High Performance Websites book by Steve Souders
Ministry of Testing/PerfBytes list of resources
PerfBytes podcast (EN and ES)
PerfGuild conferences by Joe Colantonio
Señor Performo’s YouTube Channel: Señor Performo ENG
Scott Moore’s Performance Tour on YouTube