Browser on Nicole van der Hoeven

Recording a Performance Test With Grafana Cloud k6 (k6 Office Hours #93)

Fri, 30 Jun 2023 23:33:12 +0100

Senior Frontend Engineer Edvinas Daugirdas demonstrates recording a performance test with Grafana Cloud k6 and some other new features that have recently come to Grafana Cloud. He is joined by Developer Advocates Marie Drake and Nicole van der Hoeven to talk about getting started with k6 for free.

How to Be a Gish - API, browser, and chaos testing in one script (ExpoQA 2022)

Wed, 01 Jun 2022 14:06:49 +0200

For the ExpoQA 2022 conference in Madrid, Spain, I presented a talk about how to be a gish. In this talk, I covered the following topics:

what a gish is, and how it applies to the field of testing
how testing is a multidisciplinary field, and why generalists triumph over specialists in the face of uncertainty
how the tool we choose to do our testing informs the type of testing and the type of results we get
a hands-on example of how to do API load testing, browser automation, and chaos engineering in one script using a tool like k6

There was no recording of the conference talk, but here is a video of me going through my presentation beforehand:

You can find the slides here.

And here are some photos from the event!

End-to-end browser testing using xk6-browser, with Thomas Wikman (k6 Office Hours #51)

Fri, 22 Apr 2022 17:00:12 +0200

Have you ever wondered how to test a platform built for testing other products? Our frontend developer, Thomas Wikman, joins us on k6 Office Hours to talk about his experience being a user of one of our tools - xk6-browser - and how he uses it to test the entire k6 Cloud frontend.

Real Browser Testing With xk6-browser

Wed, 16 Mar 2022 16:55:56 +0100

Transcript:

Without a doubt, my favorite k6 extension is xk6-browser. xk6-browser finally brings browser test automation to k6. So you don’t need to learn a new tool or figure out how to integrate it if you already have load tests. You can create JavaScript-based, browser-driven tests with k6. Here’s how.

Since xk6-browser is an extension, we’re gonna need to build a custom version of k6 that includes that extension. And the thing that builds that is called xk6. What I’m doing here is I’m instructing xk6 to create a build using version 0.35 of k6, that includes xk6-browser. And this is a link to the repository where xk6-browser is stored.

Writing a browser testing script

So now that we have a version of k6 that includes xk6-browser, the next step is to create the browser script. Now you can do that in the one script, but just for ease of debugging I’m gonna put it in another script for now. We’re not going to be using the HTTP requests anymore so I’m going to import launcher from k6/x/browser. And I’ll save that as browser.

I’m definitely going to be using sleep, so let me go ahead and import that now as well. We’ll also need a default function for this. We’re setting up an instance of the launcher that’s going to launch Chromium in particular, and we do want to see it so I’m going to say headless: false. If you set this to true, that means that you won’t actually see the web browser being driven by the script, which is handy to see during demos. So we’ll do that, and then we have to set up the context.

So browser, oops, new context. And then we set up a page. The first one is go to the homepage. The second one is to view a product. Third, add the product to the cart. Fourth is go to the cart and then last, I want to check out. So to do that, I’ve already defined the browser context in page. Now I need to do page.goto http://ecommerce.k6.io and I will also want to wait for waitUntil: networkidle.

Great. I wanted to add some sort of check to make sure that I’ve actually gotten to a place so, let’s open up DevTools here and I wanna make sure that some results are displayed. I don’t necessarily need, you know 1-12 of 17 but, I want this one. Okay, so I’ll also say page.waitForSelector and that’ll be this.

And then let’s do a screenshot, because we just for, just in case we wanna debug later we wanna make sure that we will be able to have a look at at this screenshot to see what exactly went wrong. And I’ll put that in screenshots and I’ll say 01_homepage. Then let’s do a sleep. And I always like to do the random one. Random, and let’s say it from zero to four. This time I’m doing the custom k6 version browser.js.

For browser-based tests, I find it much more useful to actually see what’s going on rather than to look at the output. Because as you can see there are all sorts of errors here and I believe that they’re actually application errors so they can be a little bit misleading. It looks like that happened but let’s just look at the screenshots. Screenshot, yeah. That, that loaded fine. So we can continue.

Next step is to view a product. Okay, let’s go back here. This time I want to click this thing. It doesn’t really matter which, which item it is. I’ll just choose the first one. Looks like this is the actual product link. So let element = page. And this is where we actually paste the selector which is what I got from DevTools.

And then we haven’t done anything yet with it. We want to actually click it. So element.click. After that though, I’m going to want to make sure that we get to the right page. So, let’s see what happens when we click it. What can we look for? I guess we can look for this add to cart button, cuz if the item didn’t show up then we wouldn’t be able to add it to the cart. So let’s grab that. Button, button name is add to cart. Copy that.

So then I’ll say page.waitForSelector, the selector with a button name there. Take another screenshot because why not? I’ll say 02_view-product and then another sleep. From here it was pretty much more of the same. Here’s me scripting adding the product to the cart, viewing the cart and then finally going to the checkout page. And here’s what happened when I played it all back. Here it is going to the homepage, and then clicking on the first product, and then adding that product to the cart, and then clicking view cart, and then proceeding to checkout.

The best part about this is that you don’t have to choose between the browser test and the protocol test. They’re both in k6 and written in JavaScript so you could include them in the same script. You can use stages to schedule when one starts and when one ends. You can change the load profile, add checks and thresholds. And you can also use all of the output options that are available for protocol tests for the browser tests as well.

The catch is that xk6-browser is really new. We’ve just announced it. It’s still in beta so you are likely to come across more than a few bugs. However, I’d still recommend that you give it a go. I’ve been playing around with it and I’ve found it more than serviceable for a lot of my use cases. Plus it’s just really cool to be able to test front-end performance along with back-end performance in the same testing script.

Thanks for watching. I will put a link to the xk6-browser repository below and happy testing.

Ask a Flooder 12: Puppeteer vs Selenium: which one is better for load testing?

Tue, 21 Apr 2020 17:36:28 +0100

Originally posted here.

When it comes to software testing, the debate of Puppeteer vs Selenium is a hot one. Nicole tackles it here to determine which one is better for load testing in particular. Spoiler: The answer is still “it depends”.

Or catch the audio version here:

Hi everyone, Nicole here again, back for another Ask a Flooder, and today, the question is: “Puppeteer vs. Selenium: which one is better for load testing?”

We’re actually getting this quite a bit now as Puppeteer grows in popularity, but first, let’s discuss the similarities. They’re tools that you can use to drive browsers. The work by identifying and interacting with elements on a website in the same way that a user would. They are also both open-source, and both can be used for load testing. This is a great strategy because instead of writing tests for an automation suite and then tests for load testing, you can just write the one test suite and re-use it, so it saves a lot of resources in the end. To answer this question, I’m going to focus on load testing because that’s what my experience is in.

Difference between Selenium and Puppeteer

Pros and cons of Selenium

Now for the difference. Selenium supports more browsers than does Puppeteer. It supports Chrome, Firefox, IE, Safari, and Opera, so there are a lot more options to choose from depending on your application.

Selenium is also what I would call automation-first. It was built with automation in mind, which means it’s very good at that. It’s very robust, and if you have a complicated test scenario, Selenium is an excellent choice.

It’s also been around for much longer than Puppeteer, which is important because that means it’s had more time to build up a community. There are way more resources and tutorials for Selenium than for Puppeteer.

It’s also, unfortunately, very resource-intensive. That’s not something that you would typically care too much about from a test automation perspective, but when you’re running thousands of users on the cloud, it IS important to keep in mind that Selenium, just because of how it’s built and its complexity, can only run about five users per node. We’ve found that out at Flood, after baselining specifically an AWS m5.xlarge instance. So you can look at the specs of that and compare it with the machines that you’re wanting to run Selenium on, to see how that would equate for you.

Pros and cons of Puppeteer

Puppeteer is a NodeJS library developed by Google, which means it plays very well with Chrome. There are a whole lot of things that you have access to with Puppeteer, and if you’re using it with Chrome, pretty much anything in Chrome, including rendering times, you can see and access with a Puppeteer script. So that’s a huge advantage.

Unfortunately, that also means it supports fewer browsers. Obviously, it supports Chrome very well, and they recently came out with support for Firefox, although they do have plans to incorporate support for more browsers in the future.

One of Puppeteer’s advantages is that we at Flood actually created a tool on top of Puppeteer. It’s also open-source. It’s called Flood Element, and Element is built to be performance-first because obviously, that’s what we had in mind when we were building it. It takes everything that’s good about Puppeteer and makes it really easy to set parameters that you normally would expect to need for load testing, like users and ramp-up.

This also means that it’s way less resource-intensive. I said that with Selenium, you can run up to five users on an AWS m5.xlarge instance, and for the same instance type, you can run, from our experiments, anywhere from 30-50 Element scripts/users. That is a huge saving when you’re running a really large-scale test.

Should you choose Selenium or Puppeteer for load testing?

I would say that if you already have a large Selenium automation suite, if you have a complex test scenario, or if you aren’t running that many users for your load test, then Selenium is a great choice. If you ARE running thousands of users, though, and resource efficiency, and therefore cost efficiency, is really important to you, then choose Element.

My advice is always to try both. If you have the time, do a proof of concept with both of them. That way, your whole team can better understand what the pros and cons are for each one and decide which one is best going forward. Or maybe you don’t even decide on one. At Flood, we support both of them at the same price, so it depends on what you prefer.

Until next time, happy flooding!