Ask a Flooder 08: Why is concurrent users an ambiguous measure of throughput?

Originally posted here.

What is the meaning of concurrent users in performance testing? You might have seen concurrent users in JMeter or other tools, but what does it mean and how can you use the term properly?

Transcript:

Hi everyone! Nicole here again for another Ask a Flooder, except this time it’s not really a question– it’s just something I want to talk about.

Source: The Flood Twitter account

So I’m going to put the tweet up here, but last month I actually tweeted about something that somebody else posted. Netflix actually reported some of their viewership figures, and one thing that they reported was that 76 million households watched their new show, The Witcher, which is a large amount of people. So I posted something about why concurrent users is actually an ambiguous way to measure throughput, and I want to expound a little bit more on that.

“Concurrent users” in performance testing - limitations

In load testing, the number of concurrent users is a pretty standard measure of throughput, and generally, it means the number of users that are accessing an application at any one point in time. And even on Flood, we use it because it’s a really good rule of thumb. When we want to get a feel for how big a test is, load testers generally ask, “How many users?” However, we should be aware, as load testers, that that doesn’t always give us the full picture.

What type of user is it?

For instance, the type of user that it is can really change the traffic. Sometimes mobile browsers will optimize requests so that they send fewer because they realize that mobile networks are slower.

What is the user doing?

Secondly, what is the user doing? Are they browsing on different tabs all the time? Or are they just refreshing the content on the same tab? Because if they’re just using the same page and fetching it over and over again, then caching could come into play, so that would create a different load on the system.

How long is the user doing it?

And thirdly, how long is the user doing it? Are we talking about a sustained load for an hour, or is it a shorter time period, like a spike test of five minutes where all the users just ram a system? There are things like the number of requests per second that you need to take into account, which is affected by the think time and the delays that you set in your script. Perhaps your script has a user that just sends a request, then waits for a minute, and then sends another request. That’s very different from a user just sending one request after another as fast as possible.

When does the user do it?

Lastly, it’s also important to take into account the seasonality of the request. When does the user do it? If you have a timesheeting application, for instance, then it might only get traffic on Fridays and a little bit on Monday, so you need to specify, when you report these numbers, whether it’s for the peak time period on Friday - you know, around 3 pm or something - or you’re reporting on figures from a Wednesday when there really wouldn’t be that many people accessing it.

In the case of Netflix, what they counted as a view was only a two-minute watch time. So if you’re one of those people who went into The Witcher and didn’t watch past two minutes, that still counted as a view. So the 76 million users is still not completely accurate, because not all of those people would have watched the entire thing.

Conclusion

Concurrent users is a good way in order to get a quick feel of the throughput that a test outputs, but it really needs to be examined in context along with things like requests per second and the network throughput (kbps) in order to get a really good understanding of how much load a test is generating.

Till next time, happy flooding!

See Also