Originally posted here.

In this video, I answer the question: “How do I load test video streaming with JMeter?”

Or listen to the audio version here:

HTTP Live Streaming (HLS) and performance

One of the most popular protocols for video streaming is HTTP Live Streaming, or HLS, and that’s what I’m going to show you today. An HLS video ends with the [file extension .m3u8‍](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8). According to the HLS protocol, when a video is streamed, the first request sent must be a manifest. A manifest is sometimes called a playlist, and it is a list of all the chunks or parts that make up the video. Then, each chunk is downloaded.

To assess the performance of a video stream, we still want the response time of the manifest, the chunks, and the whole video. However, there are many more metrics that are important in video streaming which aren’t captured by just the response time. For example, how do you determine whether a video was played smoothly or if it stopped to buffer? A video that keeps pausing might just drive a user away, and it’s difficult to tell whether that is happening just from the response time.

How to load test video streaming with JMeter

Downloading and installing the Ubik Video Streaming plugin

Luckily, there’s a plugin for that, created by a company called Ubik Ingénierie. The Ubik Load Pack is a set of enterprise-grade plugins for JMeter. One of those plugins is a Video Streaming Plugin, and it happens to be a really easy way to get started with video streaming. This is a paid plugin, but you can sign up for a free 11-day trial.

Once you do that, you’ll get an email from Ubik with a link to download the bundle. Go to lib/ext and then copy this .jar file into the same directory on your JMeter installation. Start up JMeter, and it’ll ask you for the license file. Browse to the license file in the bundle you downloaded. JMeter will restart.

Using the JMeter plugin in your load testing script

Now let’s start creating the Test Plan. Create a Thread group, and then right click on the thread group, go to Add > Sampler and then select the “ULP Video Streaming sampler”.

Click on the sampler and then paste in the URL to the HLS file that you’d like to test. This should end in a .m3u8. This plugin supports several protocols for video streaming, but we can just leave it at auto and it’ll automatically detect which one it is.

In Play Options, you can choose whether to play the entire video or just a part of it, or even whether you want to include the video subtitles in your test.

This Network Options panel is fantastic because it allows you to limit your network speed. Slow networks, like mobile networks, really affect user experience, so it’s really important to take this into account when running load tests. You might even use a parameter here and do something like throttle to 3G speeds for 80% of your traffic, and leave it on Auto 20% of the time. But I’ll just leave this blank for now.

Next, let’s add a View Results Tree listener. Then, let’s run the script once, just to see if it works.

As you can see, we can see the manifest retrieved first, and then all the chunks. We didn’t get any errors, so let’s run this as a load test. We can disable this listener because it just consumes extra resources on our load generator.

Ramping up your load test on Flood

To ramp this up into a load test, we’ll have to go into Thread Group and change these settings. Ideally, we want to be able to change these values from the Flood UI later so we don’t have to go into the script whenever we change them, so I’ll type in these Flood-specific properties. I’ll leave a link in the description about these properties. I’ll also set it to run Forever so that we can change the duration and the video will just keep playing to fill that duration.

To capture some of Ubik’s cool metrics, let’s add a Simple Data Writer listener here and write the results to a CSV file. On Flood, all flood results are stored on /data/flood/results, so we’ll add an output.csv file in that directory that we can download after the test.

So that’s our script! Now we’ll have to include the plugin and related files so that Flood can access them as well.

Let’s go back to the bundle we downloaded and go to the bin folder to look for user.properties. Let’s open that up, and we’ll change the license path to /data/flood/files, which is the directory that Flood saves files we upload to. To preserve this directory structure, we’ll compress just the lib and bin folders.

On Flood, we’ll start a new stream here and upload three things: the jmx script, the license, and the compressed archive of the lib and bin folders from the bundle.

Then we’ll go through the rest of the process here, setting the test parameters. Time to launch the test!

Analyzing video streaming load test results

Flood shows you results of your load test while it’s running as well as after the test. You can see metrics like concurrency, network throughput, transaction rate, and response time for the test. You can also see these metrics for each individual transaction, including the video or sampler as a whole, the manifest, and the chunks.

To see the CSV that our script generated, we can download the Archived Results from a flood.

In JMeter, click on Tools > Generate Report and fill in the fields, selecting the output.csv from the flood’s Archived Results, JMeter’s user.properties file, and then an output directory for the report.

Ubik’s plugin creates a few custom graphs in the HTML report:

• Buffer fill in Millis
• Lag ratio in percentage
• Lag ratio without buffer fill in percentage
• Lag time in Millis

Conclusion

Using Ubik’s Video Streaming plugin is an easy and thorough way to load test video streaming with JMeter, and I highly recommend it. I used an HLS video to demonstrate it, but this plugin supports several protocols:

• HLS
• Mpeg DASH
• Smooth
• Adobe HDS

For more information, read Ubik’s tutorial on how to use their plugin with Flood here.