K6 Office Hours on Nicole van der Hoeven

Celebrating 100 episodes of k6 Office Hours (k6 Office Hours #100)

Fri, 08 Sep 2023 20:56:43 +0200

In this 100th episode of k6 Office Hours, the Grafana Developer Advocates celebrate how it all started! Join us at this live fika with our favorite drinks and discuss our favorite moments and episodes.

What's new in k6 browser? (k6 Office Hours #98)

Fri, 11 Aug 2023 20:37:25 +0200

k6 browser adds browser-level APIs to automate browser actions and collect web performance metrics as part of your k6 test. It’s an experimental module, and there is a good reason why!

In this k6 Office Hours, Developer Advocates Marie Cruz and Nicole van der Hoeven are joined by Software Engineers Ankur Agarwal and Daniel Jimenez to discuss the breaking changes that are about to come to k6 browser! You wouldn’t want to miss this.

Recording a Performance Test With Grafana Cloud k6 (k6 Office Hours #93)

Fri, 30 Jun 2023 23:33:12 +0100

Senior Frontend Engineer Edvinas Daugirdas demonstrates recording a performance test with Grafana Cloud k6 and some other new features that have recently come to Grafana Cloud. He is joined by Developer Advocates Marie Drake and Nicole van der Hoeven to talk about getting started with k6 for free.

From Load Testing to Reliability Testing Tool (k6 Office Hours #92)

Fri, 23 Jun 2023 23:43:31 +0100

In this k6 Office Hours, Developer Advocates Nicole van der Hoeven and Marie Cruz talk about the evolution of k6 from a load testing tool to a reliability testing tool. They also demonstrate how to combine various modules and extensions with k6.

Testing Kubernetes Networking With k6 (k6 Office Hours #91)

Fri, 16 Jun 2023 13:00:50 +0200

How would you go about testing Kubernetes networking? For his initial project with k6, our new Senior Software Engineer, Roberto Santalla, talks to us about how he used k6 to improve end-to-end network throughput and some insights he learned about Kubernetes networking along the way. I also invited my colleague, Leandro Melendez, to help me interview Roberto.

How to use k6 for site reliability engineering (k6 Office Hours #90)

Fri, 09 Jun 2023 18:32:32 +0200

In this k6 Office Hours, our k6 Champion, Sarah Zipkin, chats to the Grafana Labs Developer Advocates on how a Site Reliability Engineer uses k6 to test for reliability.

How Supabase does performance benchmarking using k6 (k6 Office Hours #89)

Fri, 26 May 2023 16:19:51 +0200

In this k6 Office Hours, we talk about how Supabase does performance benchmarking using k6 with Egor Romanov, SDET at Supabase; Théo Crevon, Senior Software Engineer at Grafana Labs; and Nicole van der Hoeven, Senior Developer Advocate at Grafana Labs.

Supabase is an open-source alternative to Firebase. Learn more about Supabase here.

How to Get Core Web Vitals in k6 (k6 Office Hours #86)

Fri, 28 Apr 2023 16:11:52 +0200

In this k6 Office Hours, we discuss how to get Core Web Vitals for your web app using k6, and how it can help you test for user experience. We are Nicole van der Hoeven and Marie Cruz, and in this livestream, we’re talking about frontend performance testing with Grafana k6.

k6 Javascript Ecosystem (k6 Office Hours #85)

Fri, 07 Apr 2023 13:00:42 +0100

Did you know that there is a k6 JavaScript ecosystem on top of the Go extension ecosystem? What’s the difference between the two? What does k6 actually support? What types of things can you do with JavaScript? In this k6 Office Hours, Software Engineer and k6 maintainer Mihail Stoykov joins me to talk all about the different options for k6 available through JavaScript libraries.

Grafana Cloud k6 (k6 Office Hours #83)

Fri, 24 Mar 2023 18:00:00 +0100

With the release of Grafana Cloud k6, you can get observability and performance testing in the same platform. Senior Product Manager Mark Meier and Frontend Lead Simon Legander join me to talk about what Grafana Cloud k6 is, how you can use it, and demonstrate what it can do to improve your system’s reliability.

Experimental modules in k6 (k6 Office Hours #82)

Fri, 17 Mar 2023 16:00:35 +0000

Experimental modules in k6 were released in v0.40, and new modules for distributed tracing and browser testing have been added since. In this video, k6 Senior Software Engineer Théo Crevon talks to Developer Advocates Marie Cruz and Nicole van der Hoeven about what experimental modules are and how to use them.

How to Do Exploratory Testing for Performance (k6 Office Hours #79)

Fri, 24 Feb 2023 23:01:26 +0000

Marie Cruz and Nicole van der Hoeven talk about what exploratory testing is, why you should do it, and how to do exploratory testing to improve application performance. They discuss some heuristics to add structure and intention to exploratory testing for both frontend and backend performance.

Integrating k6 and Tracetest (k6 Office Hours #77)

Fri, 03 Feb 2023 08:00:15 +0000

Tracetest enables trace-based testing with assertions at any point within a request transaction. Using the xk6-tracetest extension, your k6 tests can launch deep testing by unlocking the power of OpenTelemetry.

Join us as Developer Advocates Nicole and der Hoeven and Paul Balogh are joined by founder Ken Hamric and software engineer Oscar Reyes from Tracetest.

Check out Tracetest:

site
xk6-tracetest: repo for k6 extension
docs on how to use Tracetest with k6

How to Organize Testing Scripts in k6 (k6 Office Hours #76)

Fri, 27 Jan 2023 17:00:10 +0000

In this video, we talk about how to organize testing scripts in k6 using features like tags, groups, and functions depending on what type of testing you want to do with your test script. Developer Advocates Paul Balogh, Marie Cruz, and Nicole van der Hoeven show us a few examples of how they would organize and modularize scripts in a test suite.

Distributed Load Testing Using Kubernetes With k6

Sun, 18 Dec 2022 15:54:07 +0000

My colleague Paul Balogh and I demonstrate how to run distributed load testing using Kubernetes with k6 and k6 operator. k6 can already run up to 40,000 virtual users from a single load generator, but if you want to scale that up further, you can use the Kubernetes operator to create a k6 custom resource within your cluster.

Distributed Tracing With Grafana k6 (k6 Office Hours #70)

Fri, 04 Nov 2022 14:03:04 +0100

Just announced: distributed tracing with Grafana k6 x Tempo! Daniel González Lopes joins us to talk about what distributed tracing is, and how traces using the Tempo backend could help you troubleshoot your performance issues and improve overall reliability. This new integration was just announced at ObservabilityCon 2022, and Daniel’s here to tell us all about it!

Resources

All About Load Testing Metrics (k6 Office Hours #69)

Fri, 28 Oct 2022 15:00:07 +0200

Leandro Melendez and Nicole van der Hoeven discuss all about load testing metrics. What do they all mean? Which ones do you REALLY need? Understanding metrics and analysing test results is arguably the most important part of a load test– and also the most difficult! We’re answering all the questions you ever had about the many metrics that come with load and performance testing.

Assertions in k6 (k6 Office Hours #68)

Fri, 21 Oct 2022 16:29:35 +0200

How do you do assertions in k6? CTO Pawel Suwala talks about different ways to implement BDD/TDD in k6, including an assertion library called k6chaijs. We also discuss the differences between checks, thresholds, and assertions.

End-to-end browser testing using xk6-browser (k6 Office Hours #51)
Browser-based testing (notes)
Browser testing and API load testing in the same script with k6 (k6 Office Hours #35)
In search of the best Pokémon: Browser automation and load testing in one script with k6 (presentation at Automation Guild 2022)
Real browser testing with xk6-browser

Load Testing Types and k6 Executors (k6 Office Hours #67)

Fri, 14 Oct 2022 18:36:55 +0200

What are the different load testing types, and how do they relate to the k6 executors? We go through every single k6 executor and how, when, and why you should use each one in this week’s episode of k6 Office Hours.

All About Web Performance (k6 Office Hours #66)

Fri, 07 Oct 2022 17:25:24 +0200

My fellow Developer Advocate and tester Marie Cruz and I talk all about web performance and best practices for making websites faster. What’s the difference between performance testing a website vs. an API? What are some things you should keep in mind when testing a web app?

Performance testing for beginners (k6 Office Hours #65)

Fri, 30 Sep 2022 12:45:08 +0200

My colleague Paul Balogh and I interview the k6 technical writer, Matt Dodson, who asks all the questions about load and performance testing that you were always afraid to ask.

The future of k6 (k6 Office Hours #64)

Fri, 23 Sep 2022 12:18:07 +0200

We’re talking about the future of k6 with CEO Robin Gustafsson and CTO Pawel Suwala. What’s next for k6? What new features are we working on? What’s the plan for further integrations with Grafana? What are the plans for the k6 team as we transition out of the honeymoon period after the Grafana acquisition?

Getting started with xk6-browser (k6 Office Hours #63)

Fri, 16 Sep 2022 12:13:33 +0200

Marie Drake, the newest Developer Advocate to join the k6 team, shares her experience in getting started with xk6-browser: how to install it, issues she encountered, and the state of browser testing with k6 as she sees it.

Load Testing Apache Kafka using k6 (k6 Office Hours #62)

Fri, 09 Sep 2022 12:09:36 +0200

Senior Software Engineer Mostafa Moradian joins us to talk about load testing Apache Kafka using k6 and the extension he created, called xk6-kafka. He describes what Kafka is, the basics of Kafka, and why he thinks load testing it is important. Mostafa works on the backend team at k6/Grafana Labs.

Resilience and Chaos Testing with SteadyBit and k6 (k6 Office Hours #61)

Fri, 02 Sep 2022 16:56:14 +0200

What is the difference between resilience testing and chaos testing? SteadyBit CEO and co-founder Benjamin Wilms talks about how to integrate SteadyBit with k6 for both resilience and chaos, and why he prefers “chaos testing” to “chaos engineering”.

What is browser testing, with Marie Drake (k6 Office Hours #60)

Fri, 19 Aug 2022 17:51:38 +0200

What is browser testing, and do we really need it? What does browser or automation testing have to do with load testing? Our new colleague, Marie Drake, joins us today to talk about why she joined k6 and why she is an advocate for browser testing.

Observability Trends and Future Best Practices (k6 Office Hours #59)

Fri, 12 Aug 2022 16:19:13 +0100

What are some observability trends and best practices in 2022 and going forward? Technical Program Manager Daniel González Lopes joins Developer Advocates Nicole van der Hoeven and Paul Balogh to chat about trending concepts and new areas of exploration in the field of observability. We’re going to talk about distributed tracing, continuous profiling, eBPF monitoring, and more!

1 Year Ago We Were Acquired. Here's what happened next. (k6 Office Hours #58)

Fri, 29 Jul 2022 16:12:34 +0100

In June 2021, CEO Raj Dutt announced that k6 had become part of the Grafana Labs family, and the year that followed has been replete with changes in features, organizational structure, and overall culture within the k6 team. In this video, Nicole van der Hoeven and Paul Balogh reflect on everything that’s happened in the last year and how k6 has fared 1 year after the acquisition.

Workload Modeling for Performance Testing (k6 Office Hours #57)

Fri, 22 Jul 2022 17:00:05 +0100

What exactly is workload modeling for performance testing, and why is it so important? In this video, Leandro Melendez and I talk about the shape of your load profile, why increasing the number of users may not be enough, and how you can make your load tests more realistic.

Why Is Everyone Talking About Chaos Engineering? (k6 Office Hours #56)

Fri, 15 Jul 2022 18:00:13 +0100

Why is everyone talking about chaos engineering anyway? How different is it from testing? What does it have to do with performance? In this k6 Office Hours, k6 Technical Program Manager Vince Huang joins me and my colleague, Paul Balogh, to talk about these topics and more.

How to migrate to Kubernetes, with Carlos Ruiz Lantero (k6 Office Hours #55)

Fri, 08 Jul 2022 17:00:57 +0100

k6 Cloud Backend developer Carlos Ruiz Lantero joins Developer Advocates Nicole van der Hoeven and Paul Balogh to discuss how to migrate to Kubernetes. Carlos has been working on migrating k6 Cloud services from ECS to EKS, and in this video, he shares his best practices for migration and things he wishes he’d done differently.

What Is Grafana Mimir? (k6 Office Hours #53)

Fri, 06 May 2022 17:00:28 +0200

Maintainer Marco Pracucci talks about the new Grafana project, Mimir, a next-generation time-series database for Prometheus. He joins Developer Advocates Nicole van der Hoeven and Paul Balogh to talk about what Mimir is, why it was needed, and how he helped scale it using k6.

API load testing using k6, with Tom Miseur (k6 Office Hours #52)

Fri, 29 Apr 2022 17:00:26 +0200

In this second part of demonstrating the dogfooding we do at k6, I talk to Tom Miseur, Technical Specialist, who shows how he wrote API load testing scripts with k6 to test k6 Cloud– and what his results were after his testing.

End-to-end browser testing using xk6-browser, with Thomas Wikman (k6 Office Hours #51)

Fri, 22 Apr 2022 17:00:12 +0200

Have you ever wondered how to test a platform built for testing other products? Our frontend developer, Thomas Wikman, joins us on k6 Office Hours to talk about his experience being a user of one of our tools - xk6-browser - and how he uses it to test the entire k6 Cloud frontend.

Load testing results in the k6 Cloud App for Grafana, with Edgar Fisher (k6 Office Hours #49)

Fri, 08 Apr 2022 18:00:18 +0200

There are many ways to get k6 load testing results into Grafana, but our frontend software engineer, Edgar Fisher, talks to us about the easiest way of them all: the new k6 Cloud App for Grafana.

Building Reliable Systems With k6 (k6 Office Hours #48)

Fri, 01 Apr 2022 10:02:29 +0200

In this week’s k6 Office Hours, my colleague Pandurovic, an engineering manager/backend lead at k6, joined Paul Balogh and me to talk about how k6 works on the backend, and how the backend team is working to make it even more reliable.

How to Analyze Load Testing Results with k6 (k6 Office Hours #47

Fri, 25 Mar 2022 18:00:56 +0200

In this k6 Office Hours, Paul Balogh and I discuss the different ways to turn k6 metrics into graphs for analyzing performance testing results.

How to Make Kubernetes Clusters Production Ready using Crossplane, with Viktor Farcic (k6 Office Hours #46)

Fri, 18 Mar 2022 17:30:45 +0100

In this video, Viktor Farcic, Paul Balogh and I talk about how to make Kubernetes clusters production-ready. Viktor is a Developer Advocate from Upbound, and he demonstrates a solution that ties together Crossplane, Grafana, Prometheus, and Loki to deploy and manage clusters in production.

Load Testing for DevOps with Victor Farcic (k6 Office Hours #45)

Fri, 11 Mar 2022 11:12:52 +0100

How can you use load testing for DevOps? What kind of performance testing can or should be included in CI/CD pipelines? Viktor Farcic from Upbound and DevOps Toolkit join Nicole van der Hoeven and Paul Balogh to discuss how he sees k6 fitting into the daily duties of a DevOps engineer.

Private Load Zones for Load Testing (k6 Office Hours #44)

Fri, 04 Mar 2022 10:01:57 +0100

Load testing on the cloud definitely has its advantages, but what if you need to load test behind a firewall? Here to talk about using private load zones in k6 are k6 CTO Pawel Suwala and k6 Backend Lead Samuel Regandell. Private load zones are a new feature of k6 Cloud that will allow you to run load tests against applications that are not publicly accessible and also use your own AWS account to generate the load.

Postman for load testing using k6, with Tim Haselaars (k6 Office Hours #43)

Fri, 25 Feb 2022 18:00:11 +0100

Postman is a near-ubiquitous tool for anyone doing API testing, and many teams use collections to create and organize complicated test suites. However, there’s one thing that Postman isn’t good at: load testing. That’s where k6 comes in.

In this video, Tim Haselaars joined k6 Office Hours to talk to Paul Balogh and me about how to use Postman collections to run load tests using k6.

Transcript

Hello everyone and welcome to another k6 Office Hours. I am Nicole van der Hoeven and today I have two guests. One of them may already be familiar to you, Paul Balogh.
Hey everyone.
And our special guest is Tim Haselaars. Tim, would you like to introduce yourself?
Okay, hi, nice to meet you Nicole and Paul. I am Tim Haselaars. I’m living in Belgium. I’m a product manager, and I work at the marketing automation company called Selligent. I have a passion for APIs. I love music, I’m a DJ as well in my spare time.
Oh, nice.
And if I still have some spare time left, I like to contribute to a number of open source projects, which the Postman-to-k6 is one of the projects I’ve been working on together with the k6 team in the past. So that’s a bit of why I’m here today. So nice to meet you both.
Yeah, that’s a pretty big if though, if you have time, you’re a DJ.
It’s a hobby thing, but I’m really proud of it actually, because I’ve been playing a lot, and there’s one festival in Europe, Tomorrowland, I’ve been playing it each year for the past 10 years.
Oh, wow.
So it’s a small stage, but I’m really proud of it.
Nice.
Yeah.
Well, that is awesome. You should be proud of that. I think that kind of makes sense, because product management is such a multidisciplinary role as well. Like, it’s not all technical, and it’s not all people skills. It’s a mixture of both, and a lot of other things.
Yeah, that’s also what keeps it interesting, at least for me, what keeps it interesting, actually, so yeah.
So what do you typically do in your current job, and what does it have to do with k6?
Okay, in my current job, I’m a product manager, so we have a digital platform, marketing automation platform and I’m spearheading the API renewal program. So we had a number of APIs in our company which customers use to connect and exchange data with marketing data, and we’re renewing that, and performance is critical, because we have a number of really big customers in Europe, or , the Dutch luxury brand, and we talk about huge volume, so our APIs has to be scalable, and performant, and we use k6 to guarantee that we keep them up to standard, actually. So that’s what we do. And my role there is to make sure the teams, I help with the API design, and I check with the teams, and I document stuff together with the teams, we try to do that. So what I’m going to talk about is also my experience with k6 and the way we use it at the company a bit more.
Awesome.
Cool, so how did you first start, what made you start to use k6? Did you try other things before that?
Yes. In a previous job, a couple of years ago, I was working at a hosting company, and they were doing a, there was an anniversary of the national lottery, it’s like the lottery company, so it had,, they expected a lot of visitors to the website, and we wanted to make sure the hosting was up to par, and it was able to handle the volumes. And I tried to test it, performance test it, load test it, and I used Jmeter, and the experience was, I struggled a lot with it. And I started browsing for another solution, and then I hit, I found k6. text was really nice. You could really quickly start nice documentation, and that’s how I rolled into k6. And then I shifted jobs, and at the current job, I said we have to use k6, because it’s so much easier than using JMeter. You can script it, you can automate it, and that’s it, yeah. I used it once, and never looked back.
So nice, with the automation, did you put hook that in your CI/CD pipeline as well?
Yes we do. I can, I have a diagram. Like, I’m really proud of it. Maybe I’m a bit, how do you say, biased by it, but I’m really proud of what we’ve achieved with the automation aspect actually. And we came from a far away, a far place, but now, we have a nice automated flow, actually.
That’s awesome.
Which I hope I can showcase a bit later on in the talk.
Yeah.
Yeah, I’d love to see it. And do you use k6 Cloud or k6 OSS?
We had used Cloud in the past, I think right now, my team are mostly using the OSS version of it. I don’t know why. I have to check with them, actually, but I think we had a number of dashboards set up, and they were a bit detailed to what we need, or what I needed at least, and that’s why I think we stick with, for now we stick with the OSS version of k6.
Okay, awesome. So I guess people might be wondering, seeing the title of this video, what Postman has to do with either Selligent or k6.
Yeah, actually that’s a story I want to tell, how did we ended up with k6, and the whole story. So what typically happens if you start developing an API is you build your API, and then you want to test it, you want to play around with it. You want to experiment with it. And the first thing that you do as a non-technical user like, I take Postman. Postman is an application where you can model your request in a nice tool, and then you can just fire them. You don’t have to a curl script, or you don’t have to use any CI tools for making API calls. So that was the logical step. So we build an API, I want to experiment, play around with it, so we ended up using Postman for it. But maintaining Postman is not so easy when you go for automation flows. And then when you want to take a step further, if you want to do performance tests, that’s another a thing. And yeah, that’s why I think we went from what we did to Postman just the next, as a first phase. But sooner, the more end points we have, the more we wanted to test and the more we wanted to play around with it, the bigger everything became. So that’s why, yeah, we wanted to make it an automation level and make it a bit more streamlined so that the manual work would go down a lot.
Sounds like you have a fan out in the viewers.
Yeah.
Yeah. I like to make things visual, because it helps me visualize my thoughts and everything. So yeah.
I’ve used Postman before. I think it’s almost ubiquitous now for anyone who’s doing any sort of API testing. I think at some point people would’ve used Postman just because it is very easy to get started, but I do, I have also run across some of the issues that you describe when you’re scaling up, when you have a very complex test suite or you want to have one, it is, I think I personally still like the scripting approach of being able to have the full, you know, being able to have condition loops and that sort of thing. There’s still a lot more flexibility when, when you’re just dealing with pure code. But I know that I think it was, I mean, relatively recently when I started using Postman, it was just firing off requests, and it was like a single user kind of thing, but what are the features available for performance testing for Postman, you alluded to it, them, and you said that maybe they’re not, they weren’t exactly what you were looking for.
No, no. Maybe I should quickly show a bit how I–
Yeah, sure.
Work with it if that’s okay for you. Okay, I’m gonna showcase my, so well, just as a reference, I created fake endpoint and I was set up a Postman collection. So what you do typically in Postman is you have an API that you define. In our case, I created a k6 office hour CRM API, just plain and simple where you can see, yeah, you could see some context you potentially want to interview in the future, and you model them. So you have your endpoint, you just put them in. You say, it’s a get post, put, whatever you need, and then you can just fire it and it’s response. And so this is an actual API request happening behind the scenes. That’s really nice because as a non-technical user, you don’t have to have any technical expertise. You just enter the endpoint percent and you’re up and running. But it also has some nice features if you want to make complex things, like you said, Nicole, if you want to use variables, because you might want to experiment with your API endpoints in different environments, so you can use variables. That’s what they use with the curly brackets here. So it’s a base URL. So I can shift around so if I want to switch, I have potentially two environments. I have a production one and a dev one. I can easily switch between them and then the variables will also switch along. That’s a really nice thing to do with Postman. The next thing, what I really like about Postman is that you can use it to create cruds and you can create flows with it. So it’s really straightforward. So if you say, I want to create one item and then I want to use the item in my detailed call in my, so I have all leads, create a lead and I want to get the leads, so this is the end point, you could just define it as a variable. You could say, I want to, that this lead ID can be used to fetch the right lead ID. So if you create one, that’s the body, I do a post request. And what happens is you get a new ID back, which is unique because I just created a new contact in our CRM application. And you can use scripting in Postman to say, ah, there is a data response, and I’m gonna store that response in my lead ID variable. So it allows some scripting within a GUI which is really nice, and it also allows you to define tests. Like I want to check what’s the response. It should be 200. You can define a Postman test. They use a bit of assertion, kind of syntax for that. So what started out as something really straightforward, I wanted to just test and play around with it, experiment of the endpoints. It became much more. And we started building like a full suite out of Postman, in Postman itself, a test suite. And that’s really cool because as a non-technical user, you could just share this collection with anybody on the team and they could just run the same request and play around with it. You can even pass it on to partners or colleagues or customers, when they say, when they want to use your API, you can just say, ah, you can use a Postman collection to play around with it. And another nice thing about Postman is that you can run scenarios, like I said, and you can run them at scale. So right now I’m doing get leads, creating a lead. Then I’m gonna fetch the lead that I just created. So I created Elon Musk as a user. I can update that one as well. So I can put a body and touch and I can even delete it. So if I would run this in a cycle, I would test actually the whole flow of a typical crud operation. You can fetch one, create one, edit one, delete one, and then it’s back to normal again. And so lemme talk about, yeah, sorry.
[Nicole] Sorry, I just wanted to, sorry to interrupt, but would it be possible to try and zoom here in this one to see if you can make it just a little bit bigger?
[Tim] Okay, something like that?
[Nicole] Yes. Thank you. Sorry.
[Tim] No, no, no worries. So what you can do Postman, if we talk about test automation, you can just say, I want to run all these, these API requests in the sequence, like they’re organized here, and you can just run them. So if you run them, you see you put in some test, this one is a success, success, creation successful. And so actually testing if your API’s responding as you expected, but you cannot test it at scale or at high volume. The only thing you could do in Postman is if I press run, I could define a number of iterations. So if I would say, I want to do 10 iterations, I’m just gonna run them now, so you can see, you can see it just creates blocks. but while I’m talking, it took to run all these 10, five times 10 requests, it took me, let’s say one minute to run. So we cannot really test at scale if, that’s not the use case of Postman. The use case of Postman is to experiment with request, do some testing on some validation of the response and the inputs are okay and that’s about it. So I needed something more if you want to do that scale. So while I discovered k6, k6 has this nice scripting language, you could define–
[Nicole] Tim before you, sorry. Before you move on, we do have a question. Chaitra asks, can you share this collection please, just to get along with Postman.
[Tim] Okay.
[Nicole] Is there a way that you could share, make that public or something?
You can, I haven’t tried it, but I guess if I click share and I could, can I create a JSON link? Maybe could JSON link, update the link, maybe like that? I haven’t tried this actually before either so.
[Nicole] Awesome, we’re learning in public.
[Paul] Exactly. No pressure, no pressure.
[Nicole] Yeah.
[Tim] Oh, should I put it in the comments, or how would I do that?
[Nicole] Yeah, sure. That would be great.
[Paul] Yeah, otherwise I was gonna say we could probably put something together too for after, you know, the presentation if someone wants to access that, if there’s no issue with that.
[Nicole] Okay, I’ll put that in the comments.
[Tim] So I was asked as well from the k6 team to update our blog post about Postman and k6. So what I’m gonna do after this session is going to review the blog post that they have already on the website and then update it with maybe some content that I’m talking about in this show here.
[Nicole] Great.
[Nicole] And we also got a comment from Jim Munro who says Postman is definitely not designed for load testing.
[Tim] No, no.
[Nicole] And that’s what it seemed like. So it seemed like you could only do iterations. Is that right? You can’t set, you can’t have more threads or users.
[Tim] No, that’s it. So the only thing you can do is you can run a sequence and you can run it in a number of iterations, but you can, you can’t define how many users, you can’t define how long iteration, you can define sleep, but that’s difficult. So Postman is great for what it is. It’s great for testing some of your contracts of your API, is great for experimenting with your API, and really kickstart using your API, but if you want to do volume, to test the volume at scale, I wouldn’t recommend Postman at all. But that’s also a bit the problem, because you already spent a lot of time making a nice collection, preparing it, grooming it, updating all the time. If you then want to go to k6, you would have to rewrite everything. So if you, this is how k6 syntax looks so you can define your iterations. You know this better than me actually. And you can just use a get request. And you can script your request. You can also check it. So that’s the same as you would do in the Postman where you test your response. You could check if it’s a 200. But if you have a complex scripts, you can, they become really large to maintain and to keep them up to date. And that’s what I struggled with because it felt like I’m reinventing the wheel. So I made a diagram out of it just to explain it. So if you have an API and you have to update it, you would have to really update your collection and say, every time something changes an endpoint was added or a property was changed, you needed to update, go into the Postman collection, manually correct everything, extend it, and yeah, keep it up to date. But if you would go to k6, you have twice the work, actually. You have your API that’s changing. So you need to update your collection. You need to update your k6 scripts. So if you have five endpoints, that’s doable. But if you have 20 endpoints for one API, that’s a lot of work to keep them in sync, because this is a moving target, but it forces you to keep in sync to other moving targets because you want to be able to experiment and test them, contract, and you also want to test them at scale with the load. And I was looking really for a way, like how can I leverage this nicely prepared Postman collection and use for k6 scripts? And that’s what I, when I ended up with this scenario where you have your API, I update my post collection, I want to generate a k6 script. And I was looking around like, how can I do that? And I found on the k6 website, I found some blog items on, on how that they provided a Postman to k6 converter. And that really triggered me and I started playing around with it. And I really, really liked it. So it gives me all the flexibility of modeling everything in Postman, preparing the whole flow. So if I want to run it in sequence, I could do that. But then it also converted it into a k6 script where I had no maintenance on it, and that was really nice. And when I found the Postman k6 package, like a year ago or something, the purpose of that was to kickstart your k6 experience. And I said, but you can do much more with it. You could even fully automate it with some parameters because the first time I converted my collection, it had like 20 endpoints. I still needed to do add some minor things, I needed to add some k6 specific stuff to it, like how many iterations do I want to do, is there certain parameters I want to add that are not by default converted over. So I said, okay, what can I do to make it even better? And I don’t have to touch the k6 script anymore. So the first time that you converted it, I had to modify it still. So each time that I would do this kind of a behavior, the API changes, I need to update my Postman collection and I needed to update my k6 script. I want to, my ideal world would be, I don’t have to touch the k6 script at all. I just convert it and I was able to run it in one go. And I can show you how it works, because it’s really straightforward in the end.
Yeah.
If it’s okay, you can export your collection–
[Nicole] Yeah, sure.
[Tim] So if you do export, you can just say export and you download a JSON file and you can do the same thing with your own environment. So if you want to have your test environment and you want to also convert them you can because you can also export these from Postman. And what you end up with, I’m gonna close this one, you end up with two JSON files. So as you can see, it’s the k6 office hours. It has some syntax in it, like the test scripts are there, the endpoints are there, the way you connect to them. So all the request information is in this configuration, in the environment file. You can find all your variables that you have defined. So let me just now showcase it. Sorry, I’m gonna, this one. So if you run the Postman-to-k6 converter, it expects actually two things. You define an input, my Postman collection, and you define an output. So if I would run this, just closely watch what happens there, it does some things behind the scenes, and once it’s finished, you would get, actually, if things go well, let’s see if it’s there, you would get this. Yeah, that that’s the goal of CLI. It should be fast. You get the k6 script. I’m gonna go over that later on, and you get this library folder. The library folder, what it is is because the converter, Postman uses number of functions, some methods that are not mapping exactly towards k6. So once this package, it adds some additional functionality that Postman uses. Like it uses low dash and some encryption, stuff like that, and it just adds the libraries there. So if you would do this manually, you would also have to include the libraries, but it does it for you. It includes all the libraries. And it’s actually a wrap around the, let’s go for the SHIM quickly. If it’s not clear what I’m trying to explain, stop me, Nicole.
[Nicole] Yeah, but we can also answer this question from Carlos Ramirez. You can also integrate, well, not a question, a comment. You can also integrate your process using the Postman API key to request information about your collection and environment. And he also adds, oh, no, sorry, this is from Chaitra Nanjesh, this is interesting, because we use k6 in Tesco, says load test, and I think we can try this to see if it works for us. So people are already liking what you’re sharing Tim.
[Tim] Yeah, that’s the goal, That’s also why I liked it actually–
[Nicole] Do you also do what Carlos was suggesting?
[Tim] Yes, and I even go a step further, but that’s, I’m gonna come back to that later. I’m gonna showcase how we do the Full Monty in the end, actually. I’m building up.
[Nicole] Awesome. Sorry, people are excited.
[Tim] Yeah, I’m also excited, so I hope that I’m coming across properly.
[Nicole] You’re doing great, don’t worry.
[Tim] Okay. Just for reference, you have to think of this Postman-to-k6 thing as it’s wrapping Postman functions and k6 functions and combining them in one place. So it’s abstracting away the k6 part and it matches with the Postman stuff. So that’s what it does behind the scenes. So if you see, if you do an execute request, which is an, it’s a function actually, behind the scenes, it just does the HTTP dot request, which is a k6 function. So it wraps a lot of things together, and I think if I would go to execute test, you would also see it uses the k6 check function. Just for reference, this is the wrapping part. This is all not so relevant. You don’t have to care about it. What’s relevant is what’s being generated for you. So it says on top, it’s an autogenerated file here. It does some imports. It imports also k6 related items and it sets default options. By default, it says maximum redirects. And then it does some strange things which are not k6 related, but that’s typically what the package does. That’s the wrap that you see. It defines a collection with a base URL, and then you will see that my group is there, I have my request get all leads. It showcases the URL. It showcases that it does a test here. So everything that you had defined in your Postman collection, sorry… Get all tests and you could see the test here. It’s translated into a k6 kind of syntax and it’s ready to be used. And you see all the requests on here, all ready to be used. It even matches a response codes that you can match to a variable in Postman. You could do anything. The body is there that I inserted that you can create. So if I were now just say k6, sorry, k6 run k6 scripts. Oh, dunno know why this is happening? Give it a second to start up. There you go.
[Nicole] Cool.
[Tim] So I haven’t touched anything, I just took my Postman collection, converted it with the converter that we built a while ago together with k6 and then you can see the tests there. So I created a number of tests. Postman expects to, the response is expected to be 200. I get all the data. My average is my group duration because you also have a group there. So if you have 20 endpoints, you will test 20 endpoints, but I don’t need to write any k6. So I only do my maintenance for, my maintenance only happens in here. I only have to maintain my Postman collection and my k6 gets generated out of them. So that’s really nice thing here. Yeah, that’s the library that we’re using. No, that’s not, that’s not my library actually. I think it’s moved to–
[Paul] Uh-oh.
[Nicole] Hope this is the right one.
[Paul] Yeah, we gotta update our show notes.
[Nicole] Yeah, we do. Oh, with the API deck one. Sorry about that.
[Tim] No worries. I think I made a pool request a while ago, so.
[Nicole] Ah, I see. Okay, I will update the description.
[Tim] But I, just to continue.
[Nicole] Can you also post, Tim, the link?
[Tim] Yeah, I’ll post it. Of course.
[Nicole] Thanks.
[Tim] But it’s really cool. Maybe cool’s not the right word because the package supports a number of things so that you can even tweak the output even further. So by default, I just say, give me the Postman collection and generate a k6 and it would work. But if you want to automate this on a higher level, need to do work for automation, you wanted to tweak some things. Maybe I want to play around with some k6 parameters, or I wanted to do something else, or I want to exclude stuff, or I want to include my environment because, if you see here, there is no, there’s only my base url, but I have a lot of variables I want to use in my Postman collection that I already defined. I have login information. I have some Postman base URL. So you can take it even further because now you just get a script, but it’s just, the script is already addressed. So you can take it step further. So for example, if you would say, I want to increase the number of iterations, you could just say like this. You have this parameter, says iteration 10. I’m not going to convert it. There we go. And if you see my script now has options already set at 10. That’s nice. That’s handy because you want to automate this, of course. And you can even go, take a step further. I want to include my Postman environment variables. So I’m gonna, there’s also a parameter for that. So you can pass some environment variables. So you can even pass some global variables. This is typically Postman, but you can just take the ones that Postman provides, export them and use them. So if I now do this kind of thing, so this is my file with my Postman variables, like I showed. Let’s do this again. And here, you’ll see all my variables that were exposed in my environment are now available for my k6 script to be used in automation workflows. So that’s really nice, but I can even take it a step further. What if I have special k6 related parameters? I’m just gonna copy paste them here for the sake of the demo, but I might want to define my iterations or my maximum redirects because k6 offers a lot of parameter options. Like I want to use cookies. I want to use headers. The number of redirects. I want to set some text. I want to define a timeout. All these k6 related parameters, I want to make sure that my automation flow was able to process them because I want to run the script without touching it. So I want to be able to just say, take these parameters for this pipeline, these parameters are applicable for production. So you could switch environments. For example, what we do is we have different test scenarios defined. So if we go for a small test, we run it like this, we define stages and thresholds. Well, if we want to really do performance testing, we have different ones. So these options, we use them to pass them along to the pipeline. So I could do something again with the generation part.
[Nicole] Question from Sri. Is it one environment at a time only?
[Tim] Yes, because what you generate is what you inject the script. So it, you would then have to generate a second script, but that’s really easy to do because you would just say, I want to have, for example, just showcasing it here, I’m gonna take this script is for the second test stage. You just run it and then you get a second script next to it. So you don’t have to worry about it. You don’t think about one script. You just fork it as much as you need with different configuration options defined that you would use.
[Nicole] Yeah, and Sri, I wanted to also add here that with k6, you can have, you can test more than one environment at a time. What Tim is showing here is that for, because of the way that the that Postman treats these collections separately, every environment is separate. And so the converter also converts them to separate k6 scripts. However, once they’re in k6 script format, once it’s all JavaScript, you could use k6 executors and scenarios to run them simultaneously, if that’s what you’d prefer or anything else.
[Tim] So in this example, what I’m doing, I’m going to pass on my K6 parameters as a JSON file so that I can put them in a Git somewhere. And if I then just say, press go, you will see that it also just updates this option spot on the top. There we go. So you will see, I have defined it to be in this case five virtual users, 10 iterations, maximum redirects, and just adds it on top. So I can switch around in my pipeline without touching anything. I just prepare everything, and then it’s just a matter of defining which command I should to run and it just does it for me. So that’s really handy and it really makes automation a lot easier in that regard. And there’s one thing I really like. It’s a small thing, actually. Now it generates one big file. So everything is in this one big file, all the request, it’s doable if you have like me, for example, four requests or five requests. But if you have like 20 endpoint or 20 requests, it’s a huge file. And to handle that, you can just pass along one parameter. I’m not sure if it’s commonly used a lot, but I’m a big fan of it. It’s called a separate parameter. So what happens now? The options are there. The request configuration is there. But you just get this in your k6 script. I’m not sure which one, I’m gonna remove one because otherwise I might get confused. There we go. You get this, and this is actually, it splits all your requests in a separate folder like you would see them in Postman and each request would be separated from the rest. And that makes it really easy. If you want to test something specifically, if you would have one script, it would run everything. But now I can just comment out, I don’t want to do the create, I don’t want to do the update and delete. I only want to test to get all leads. You just say k6, k6 run k6 script. Give it a second to warm up. There you go. It will only test that request that I defined. So if your 20 endpoints, your k6 script has been generated for you will have 20 lines and all your requests will be nicely separated from each other. So you can even say, I want to tweak it a bit, or you have all flexibility then, while if you don’t use a separate parameter, it just generates one file with everything in it. Do I make sense still?
[Nicole] Yes you do. But there’s a comment here from one of our engineers who wrote that initial blog post that you may be updating, Mostafa said last time you wrote an article about it, it needed a few changes to make things work, so well done.
[Tim] Yeah. Yes, that was one of my first VRs that I did. I wanted to make separate flow over pretty much everything. And there’s one final thing I want to showcase for, with the k6 part, is now you have to really, like all these parameters, they become very long and difficult. You could also just define them as a JSON file and just pass them along. And now you can just define the one file, put the main Git, and then you could just say, I want, this is the output, my environment. You can even define that it uses a handle summary, or you can add request tagging on top of it. And if you do it like that, you get, you just have to do one parameter, one command, actually. So if I do it like this, you see, it just says, this is my input file. I’ve defined an options file, which I referred to. And if I then just do it, it would take all these parameters that I’ve defined and you would get, again, everything nicely generated for you. All the requests are there and it even added the handle summary thing on top of that, which is something–
[Nicole] Oh nice.
[Tim] So everything is there. And so this is the automation flow that’s really powerful because otherwise you would have to add, if you want to have a summary, you have to add a modified file. But now I just define one configuration file, I define my parameters, I define whatever I need. Just run it, and my script, I don’t need to touch it. I can just say, okay, k6 run k6 script. And you would see that it generates, yeah, it generates a test. It does, I think five or 10 iterations. There we go. And it doesn’t open anymore. It generates the JSON file. If I close my, quickly here, you would see a summary report is here. So it just, yeah, there is no manual work anymore to run k6, just keep your equipment collection up to date.
[Paul] And you’re helping folks out with your example here.
[Tim] Yeah, okay, great. Okay, that’s in summary, the things that I want to showcase about the converter. There are some nice things added since the last releases from the k6 team. So I added some, requested and changed some things, and I’m really happy with it. It works flawless at the moment in our organization actually. So I’m really happy with it.
[Nicole] Yeah, it seems really, I think I tried this initially, like when Mostafa was saying, there are a few things that you, you had to get done, but this seems way slicker, and it seems like you’ve really focused on making it automatic so that don’t, there is no manual intervention, which totally makes sense in a CI/CD pipeline.
[Tim] Yes, correct. And I wanted to put my effort on the Postman and the management of the Postman collection, because that was a tool where we were all working on and maintaining it. And I wanted to keep the workload on k6 as low as possible. And I think we succeeded it quite well on that regard. There are some things that are not available out of the box. Like you cannot do the send request. You cannot set next request. But what I did add was, what I’m also really proud of, the dynamic functions from Postman. You could just add any, pretty much any generated stuff. Let’s see if it shows, if it works like that.
[Nicole] There’s also some nice comments for you. There’s Gertjan De Wilde, who says, Go Tim. There’s Pepe Cano as well from k6 chiming in with his support. James is saying nice to see this getting the love it deserves, much better experience. And Charly is saying, Very useful tool, congratulations.
[Tim] Thank you.
[Nicole] Lots of love in the comments for you.
[Paul] Definitely.
[Tim] Okay. But honesty, we took it a even a level further, but maybe I should, maybe there are, if there are other questions we should talk about or not, I’m not sure.
[Nicole] I’m interested in this. So what were you saying, the dynamic part that you added parameters?
[Tim] Let’s see.
[Nicole] I mean, you also don’t have to demonstrate everything. Can we just like acknowledge the fact that Tim who doesn’t even work for k6 or Postman volunteered to do like a live demo about this. So that’s awesome. Don’t feel like you have to show us everything. You can just talk about it.
Yeah, Postman really provides this nice, what they call dynamic variables. And so it’s, this is a screenshot of it. So if you use curly brackets, it’s random, and then you can say, I want the full name, I want the bank account or anything that uses Faker behind the scenes, and you can make dummy data out of it. So that’s really handy if you want to really make, yeah, realistic data without having to use for loops or anything, just put in the variable. And that was not fully supported in Postman to k6. So we brought in that support as well to make sure that you could have the full functionality of the dynamic variables attached to that. So yeah, there are a number of things we did since the last release of it, and I think dynamic variables was in much demand at that moment. There you go. Yeah, and I’m really happy for Mostafa, I’m not sure how to pronounce it correctly, because I think he was one of the first ones who pointed out in his blog post about functionality. So thank you for pointing it out to me.
Yeah, now that I think about it, I should have asked Mostafa to be here too. Oh, I missed that opportunity. Sorry about that Mostafa. That’s so good about the dynamic variables, because that’s one of the things that you have to think about when you shift from just a purely automization mode, like automated testing mode to load testing is that you can’t, you then have to start thinking about the data that you’re sending too, because if you’re trying to get realistic response times it could trigger caching on the service side if you’re just requesting the same thing. So yeah, totally makes sense. k6 can already, you can already import faker with k6, so great, great job for putting that together.
Yeah, no, that’s very cool though, but yeah, it’s like, you can use Postman for your user, your UI, your, you know, your development environment and do all this stuff and then even goes through there and generates some random data. So, yeah, that’s cool. Very cool.
Oh, go on.
But maybe I’m gonna talk first before I want to show the final piece of what we did.
Well, just, just quickly, I just wanted to ask about the limitations of this approach that you see.
Not all functions of Postman are supported. So that’s the downside of it actually. But for the use is that that we are using it, we have everything covered. Like you have the, the dynamic variables, you have the environment variables, you can define k6 parameters. Yeah, there was nothing stopping us. I think if you want to do complex scenarios and you want to do if-else situations, there it’s not suited for. That’s the same problem you would have in Postman. It would be difficult to make these if-else conditions in Postman as well. So that’s, if you do simple tests, functional tests in Postman, you can really easily port them to k6 and then you have performance tests end points.
And in the spirit of the whole free and open software. I mean, do you have issues where like you’re looking for help? Is there any gaps that you’re still looking to fill?
There are some bugs I cannot reproduce at the moment, something with XML conversions, but right now it’s feels quite feature complete. I haven’t added that much, actually. The base was already there by the k6 team members who built the initial library. So I just added some sugar coating on top to make the automation flows much smoother and less manual work, or linked actually. But if people have great ideas, I think they should just make a PR and add it to the package.
Yeah, did you want to show us something else?
Yes, I want to show how we ran a step further even, because yeah, if you update your API and then you have to update your Postman collection, it then generates a nice k6. But we also use openAPI as a documentation tool where we document and design our API. So every change that you do on your API, you need to add them to your API, and then you, so you still have to maintain not one output, the Postman, but you also have to update your documentation. And I want to flip that around. I want to make the API documentation leading, generate the Postman collection out of it and generate out of the Postman collection, a k6 suite. So that as a product manager, I don’t have to maintain all these documents anymore. I just maintain one thing, and that’s my openAPI and my team or our team then only just need to develop the application itself. And it’s called spec driven approach. It’s nothing new, but I’m a really big fan of it and I’m a really, yeah, we introduced it a year ago and it solved so much and it brought so much quality on the company level, in our documentation, in our test suites. So that’s something I want to share as well, because it’s so valuable once you get how it works and if the right tools and properties, you can really do it. And this is actually where we are right now. I can also demo how it works. So we have, I’m managing an openAPI documentation. We use the formatting and we validate it if it’s valid. And then we use a generated one. And from the generated one, we generate documentation. So public documentation, stop lights, just like whatever you call it, we generate a Postman collection that we can share with the customers and we run contract tests, we run soak tests and we run smoke tests all based on this openAPI documentation. So, and again, there’s no no contact or no changes involved. We just, we managed the spec on its own and it’s really, really something powerful and useful. And I can quickly show how it would work actually if you’re interested.
[Nicole] Yes, but before you do, a wild Mostafa appears.
Hello.
How awesome is that? Let’s just say hello to Mostafa, who just joined last minute since he was already in the comments anyway.
Nice to meet you. And I’m grateful that you wrote the blog post so many, so many months or years ago, so thank you for that.
Yeah, yeah, I really felt the need that we needed this article, so that’s why I wrote it in the first place. And then, yeah, I found a lot of issues that you solved, and thank you, it was a great job you did, and well done.
Thank you. And thank you for putting out the 90% of it. I said I only had to put in the sugar on the top of it and the cherry on the top.
No, no, no, no, that’s not true, that’s not true.
Mostafa–
In the first place just to clarify.
Yeah, Mostafa while you’re here can you quickly introduce who you are and why we’ve randomly invited you onto the stream?
Of course. Okay, so I am a backend developer at k6. I’m almost three years with k6, like it was called Load Impact before being renamed completely to k6, and now Grafana. I’ve been a developer advocate when the first like idea came into place that we want to have a developer advocacy program. And yeah, I’ve been a developer advocate for a year and then came back to back it. So I worked with lots of these tools and tried to help others because like, there were no other resources available. Like there was this read-me on the project and that was it. No mention of it anywhere. Only the hidden knowledge of everyone inside the company of how to use this tool, but nobody knew without outside the company. So tried to learn it from others, specifically, Pepe, thanks for that, from here. And I tried to write that blog post and try to share the knowledge. So that’s basically it.
Great, so we also got a comment: Great with the diagram. Can you share this?
Of course I can. Yeah, I’ll do that.
All right, I guess that’s our cue to head back. Please continue, Tim.
Oh, I’m not gonna, I’m just gonna showcase it, and then I hope everybody will really appreciate and see the power that it brings actually. So, okay, so maybe start with the beginning. I’m gonna close my screens a bit to clean it up. So I have this openAPI spec that I, it’s just a copy of what we just described. So this is how it looks, this is documentation using Docly. You can see it’s the k6 office hours CRM application. You can see the list in there. You can see the create, the get. So it’s just the definition of what the API would look like. This is the documentation part of it. So I’m using now two tools. I’m gonna use Portman as another package that I built together with API Deck and make an API Deck where we can convert the openAPI to Postman, and from the Postman, you can generate a k6 one. And that’s really nice because you only need two commands, one source file, and you have two outputs out of the box. So that’s really powerful. I find it powerful at least. So I’m gonna take my commands correctly.
So If it’s okay, can you, I’m not, I’m not up to speed with Portman. So it creates a Postman collection already given openAPI, is something in openAPI specification?
Yes.
That’s awesome.
There was, Postman built a patch like Mostafa, k6 build a patch where you can convert from Postman to k6, and Postman build a Postman where you can convert from openAPI to Postman. And when I found out, I said, that’s amazing. You don’t have to maintain a Postman collection anymore, but it wasn’t, it was missing some functionality, and the major thing that I was missing, it was just creating the request than Postman, but not the tests. It was not handling the tests of it. It was not assigning variables. It was just taking the Postman request and converting them into openAPI requests, into Postman. So you still needed to maintain and modify them. And I said, it has to be, because an openAPI documentation or specification, it describes everything, the request type, the body, the properties, if you do it properly, you have all, everything in one go. And I said, why not use that to create a test suite, update your variables, do all the dynamic stuff you would expect it to come out of the box. And that’s why we created Portman. It started as a PR on that openAPI to Postman package, but then I met Nick from openAPI, from API Deck, sorry. And then we came together and we said, let’s collaborate, and let’s make a startup package out of it that you can plug in and use it to automate these kind of workflows. And it’s really, yeah, I’m biased again, but I really like it.
No, it sounds awesome.
A question here. May I ask a question here? So have you seen the openAPI generator package that I’ve written?
Yes, I saw that and that I had the same comments as I had initially. It generates a nice k6 script, but it it’s usually you still have to manually adapt a lot of things and correct it.
Yeah, but sorry that I interrupted you. There’s this latest PR point? I think Michael, if I’m currently remembering his name, so Michael added a feature to, to be able to add certain extension to the, to your Swagger or openAPI collection so that it generates everything for you. And if you add examples to your API output, to your responses, then it can extract those examples and put it in your script so you don’t need to modify anything. So you can create basically the same thing you did for Postman, you can do it for the openAPI. So the new revision, the new PR and the new version that is released by Ophelia, the maintainer of the openAPI project helps you create a complete test from start to finish with no modification.
Yeah, I think that’s great, but we do two types, three types of tests. We will do contract tests, and then we use Newman for it. And we want to use k6 because Newman, it can do schema validation, it can do a lot, I think a lot more than k6 can right now. I think it’s also the goal for k6 to do all these contract validation. It can, I guess, but I treat them differently. I think the Newman Postman has more contract testing capabilities, while I think k6 is perfect for performance and load testing at scale, where you only focus on what’s the response, is it 200 or is it not a 200, at least that’s how I use it actually.
That’s fair enough.
Yeah, what Portman does it converts openAPI to Postman. So I’m gonna showcase that quickly. So it just took the, this openAPI that’s here now, and then gives a while to spin it up. There we go. It’s converting it. And if things go well… Let me check something happening here. Here it is. So here’s my Postman collection. It’s just generated it. It does everything for you. So it creates, recreate, so there wasn’t a collection there. It just recreated for you. And I hope things, it’s not showcasing it here. That’s bad, too bad. So it just generates and if I would import in Postman, let’s quickly do that… Give me a second to find everything. There we go. So this is the k6 and it’s generated by Portman. There we go. If you see you have the same endpoints–
[Nicole] Oh, nice.
[Tim] So the first thing it’s great because you don’t have to recreate your Postman collection. You just, you just run it from your openAPI specification and you get, as a result, you get your requests pre-tailed already, also your post exercise is ready. So you can see here, it inserts a body, everything, I don’t have to do anything. It just takes your openAPI and converts it to Postman. Plus it adds a lot of tests already. You see, it does contract testing by design. So it includes automatically 200 contact applications. So it just injects your contract test in there. And that’s really nice because I only need to maintain one document, my openAPI specification, as a result, I get all my requests in Postman, and I get all my tests in Postman without writing any code. I just need to maintain it in the openAPI documentation.
Wow this is really awesome, Tim.
So keep in mind that this was the flow before you had to create your Postman collection and add the tests, and then you could generate k6. Now we turn it around. Now we generate the Postman and request, and then we can use that to generate, okay, six scripts. And so we only need to maintain this part in, yeah, or you know, that’s the only thing I need to maintain as a product manager and the team, of course, they built the API and the service and everything, but I get this and I can focus only on this. And this is easy to manage because it’s a YAML, you can play around, you can add endpoints, you can remove endpoints and just run it. A minute later, I have a Postman collection to play with it and I can run it at scale with k6, doing the volume tests. What else can I showcase quickly here? Let me see if I can. I just–
[Nicole] Some more nice comments here. Such an awesome flow, says Gertjan, and James says his mind is blown and, and I join him on that sentiment right now.
[Mostafa] Exactly. That’s a full test suite, like from start to finish.
Yeah, and like example, schema validation. It comes out of the box. You don’t have to add it. It just does it for you. And the richer you make a documentation, the richer the test suite will become. And it also, Portman, be extended, you can also do variations. So now it just validates if it’s a 200, but you can also have it generate 404s, and then it will remove items from it. If it’s a required property, it’ll create a variation out of it. And then you can test I’d the 404s are not present. And so you get two test suites in one go, actually.
Wow. That is, that’s so awesome. Maybe we can like, try and summarize this as we try to collectively recapture our blown minds. So I think that there are two different tools here, Postman and k6, which are really for different things. The Postman is really better for API testing, and as you mentioned, Tim ,contract testing, and k6 is better for load testing. And their features are pretty divergent after a certain point, but they do have a commonality, and it seems like that’s what you’ve really capitalized on. Mostafa and Tim both have worked on this. and you you’re trying to make, you’ve gotten us closer to this ideal of having just one set of test suites, and it doesn’t really matter whether it’s automation test suites or load test suites or, it can just be the one thing and you no longer have to maintain two separate ones. And that’s having like the amount of time that you need to set it up, but also running it. Because having to explain this to somebody else is also a huge deal when you’re not the only tester on your team.
Nope, exactly.
And the nice thing is as a product manager, I’m not technically, so but I can contribute to it. I can, I just need to maintain the openAPI specification and the whole team benefits from anything that I do. If we need to add a new endpoint, I can design it. Not even five minutes later, one minute later, I can just run it in a pipeline, see if it works, and that’s a bit, the cycle of spec driven development. You put your spec at the center, the spec at the center, and then you design it, simulate it, fill it, Then you build it, you test it. But the spec is a test driver here and then you can even run it on production and yeah, do the management aspect as well, because then you can monitor it as well based on your API specification. So that’s, as a product manager, I’m really proud that we pulled this off because that was one of the ambitions when we wanted a newer, or API program, like how can we use a manual workload, what can we automate, and yeah, I’m really happy because it’s all open source based. There is no other software involved. We just use k6. It has excellent dashboards to individualize reports. We use Postman to generate contract testing and manage it, play around with it. And we benefit from such large ecosystem , Swagger as documentation interface, where you can display or openAPI as well. So it’s almost full cycle or I don’t know what’s the term for it, but yeah, I really, I’m really happy with where we are right now. It’s a good place to be in an API landscape actually, especially from, yeah–
You’ve taken it a step further, right? This is no longer just about test suites. It’s also because you are building on the openAPI specifications, it’s all connected even from not just the development part, but the design part. And Pepe actually has a question about this. He says, can you show us more about the process that updates the APIs? What does it do?
How do you mean? This flow, this one?
Sorry, which one? Yes, I believe so.
The powerful thing about this can also be because if you have an openAPI specification, you can even generate clients or service side code from it so that in the end, your API service can also just be generated from an API specification. There is a really nice openAPI, there’s a really nice openAPI open source project, it’s called I think openAPI Generator, and then you can just run a C# library or an API for that. You just take your openAPI documentation and you run, you run it and you get as a result, you get C# code that you can run in production. And that’s also what we use in our company. You have some teams that just manage to build the openAPI specification and they use that to generate C# code and have APIs up and running in no time. I hope that this is the answering this question.
Yeah, Mostafa, Paul, anything you want to say to Tim.
Yeah, no questions but, yeah, that would come later, but I need to definitely pass this along, this video along to some people that I used to work with, because this will blow their minds as well. This is something that they definitely need.
Yeah.
Do everyone.
Great job, great job.
Yeah, great job, exactly. I’m literally speechless because I’ve worked with all these tools and I know all the–
Clearly not literally.
Well, not the ones that Tim wrote, the ones that we had before. But anyhow, this is a great addition to this mix of open source software that was out there and that we used, and this is basically end-to-end testing. So based on a YAML file, which is an openAPI specification, you can generate tests for your API. You can generate load tests. You can generate, I don’t know, like literally everything.
It’s also documentation. You get documentation, nice looking documentation out the box, a playground with Postman and you get the numbers to back it up with even graphs from k6. I really like your dashboarding. So if you have a customer and they they’re asking about SLA, we just pull up some graphs from k6 and everybody’s happy with that. So we, the full ecosystem has really matured a lot over the last couple of years.
Bravo.
Yeah, this is so freaking awesome. Thank you so much, Tim, for coming on. By the way, he’s not even working for k6. He’s just, and I think you said at one point, oh, I’m a product manager. I’m not technical. I beg to differ. This is pretty amazing things that you’ve put together. There’s a bunch of comments here too. James says he’s literally waiting for the livestream to end, so he can link to the video with a timestamp. Well, let’s just try and drag it on for him, shall we? Thanh says plus a thousand votes, and people are also saying thank you, and outstanding, almost the holy grail. So lots of fans in comments. Thank you so much, Tim, for joining us. I know we’re a bit over time. I’m kind of taken aback by how much value you just showed us. This is amazing, I think we need to have you back.
Yeah, I’m gonna write a blog post for k6 and one for API Deck where I’m going to get more into the details because yeah, it’s a show now, it’s a difficult to but I’m gonna try to write it up and then extend a bit on what, sorry.
Mostafa.
Mostafa, apologies, I’m really bad.
It’s fine, it’s fine, no worries.
Yeah, I think can be really powerful to share with the rest of the world that they can, yeah, maybe use some parts of it or not, but I benefit a lot from it. The whole team benefits a lot from it, actually. I can showcase, this is a pipeline we use, for example. So this is we do tests, then we run Postman and we do tests and if it’s 100%, it gets deployed. So it’s not just a gimmick, me showcasing it. This is really happening in our company. So we use it to build it. We use it to have nice dashboard with k6 attached to that. So yeah, I’m grateful that you put it out there for us to benefit from it.
Thank you.
That’s so great, thank you.
Yeah, I–
Go on, Mostafa..
Review the article that you want to write.
Okay, I’ll keep Pepe in the loop and then you’ll reach out.
Of course, of course.
Great, thank you. Thank you as well, Paul and Mostafa, for coming on, Mostafa, even though, sorry, I should have invited you even earlier. Thank you everybody for watching. Thanks everyone in the chat. Have a great weekend, everybody.
Absolutely.
Thank you for having me.
Yeah, thank you.
Yeah, all right, see you all.
Bye.

How to design a workload model in k6, with the k6 Developer Advocates (k6 Office Hours #42)

Fri, 18 Feb 2022 18:20:21 +0100

On this week’s Office Hours, Paul Balogh and Leandro Melendez join me to talk about how to design workload models in k6 load testing scripts using scenarios and executors.

Transcript

Hello everyone, and welcome to another k6 office hours. I am Nicole van der Hoeven, and for the first time, all three of us developer advocates are in one stream. Could you please introduce yourselves?
Oh, who’s going first?
I guess you are.
All right, age before beauty, as they say. Okay, well, hey, I’m Paul Balogh, the newest DevRel here at k6, nice to meet you.
And on my end Leandro Melendez, also known as a Senor Performo, very happy to contribute here and bringing, usually it’s knowledge, but this time I’m gonna be a little bit on the learning side, which is super exciting.
We learn together.
Oh yeah.
Yeah, I mean, I don’t think you can really have one without the other. I think it’s super valuable to also just ask questions.
And Nicole, I’m a little disappointed that you don’t have a fake mustache to support Leandro and I.
Yeah. That is not something that I just have lying around, unfortunately.
We may have to fix that. I was commenting the equivalent here in Mexico is the unibrow.
I could potentially draw something on my face, maybe, I could try that.
Is that the Frida Kahlo?
Oh yeah.
So, Leandro and Paul are relatively new to k6. I think you both have been here for less than a month.
Just a month here.
Oh, just a month now, that’s awesome.
Three.
But this is the first time, I’ve been here with you with each of you separately, but this is the first time that all three of us are here. It is pretty cool, welcome.
Exciting, hopefully it’s not overload for everybody watching.
Yeah. One thing that I wanted to get out of the way is that we have a very awesome community, we have several community champions, but one of them, NaveenKumar, just posted a couple of videos on how to use checks and thresholds in k6. So I’m going to put that in the comments, as well, because I thought that they were pretty awesome videos. NaveenKumar has a YouTube channel where he goes through a bunch of different tools, including k6, but also like JMeter and Gatling, and he always does a good job, so check those out. Oops, I actually didn’t post his videos, I think I posted the documentation for what we were talking about, so that’s a little premature. Sorry about that, let me put the actual links this time.
Well, this is a live podcast.
Yeah, obviously. Sorry, you get what you get. So if that hadn’t already tipped you off, what we actually want to about today are scenarios and executors in k6. You may have been confused by the thumbnail because it was like, I realized afterwards, that it was the three of us, our photos, and then the words executors in k6.
Here come the executors, the three of us.
I guess that’s our new name or something.
You know, if you invert the k6 logo, it almost looks like a guillotine, so maybe we are the executors.
Oh, okay.
I had not noticed that.
That’s a little morbid.
Oh, you know, hey, I do what I can.
You’re going like play Beatles songs backwards and all that type of stuff, man. I’m sure it was not on purpose.
It looks like we’ve lost access to LinkedIn for some reason, but, oh well, this has happened before. Thank you, Mark, for saying we’re killer performance engineers, love the pun. So the reason that I wanted to talk about this was that, I don’t know if you also have got the same, Leandro, Leandro and I were both in the Automation Guild 2022. We both had a talk there and there are a lot of common questions that come up with regards to load testing. And a lot of it has to do with the results, specifically when, I’m not sure if you can actually hear the wind in my background.
Is that you, oh wow.
Are you okay?
I’m okay, but apparently there’s a storm that’s in Northern Europe.
I was gonna say, hopefully you’ll still be in Kansas, right?
I know. It’s like 140 kilometer per hour winds.
Don’t end up in Kansas.
But can I be the straw man, at least, in this Oz thing?
Sure.
I always liked him.
I guess I’m Dorothy. Or the wizard.
I ask Toto, please.
I’ve got Toto around here somewhere, let her in.
So one question that I was getting a lot, or I saw other people talking about variations of it, was the situation where you might have already run your load test and seen all check marks and everything passed, and the response times were within the expected results, and yet when the application goes live, suddenly there’s all these errors, and performance issues, and maybe production incidents that you never saw in any of your tests. And the question is, how can that be when you’ve done your testing and you didn’t find any of that and you only see it in production?
Yes, of course I experienced that several times, but many things come to mind on, in what environment did you test? What happened in this environment? If you test in production, hopefully ‘cause we are modern performance engineers that are not afraid of testing in production. Settings, changes, what was the workload? I mean, I don’t know, the list is pretty long.
Yeah, there’s a lot that, that could, that’s why that question is hard to answer because there’s no one answer. It’s like, oh and that happens, always check this. And that’s also because it could be that there’s something different with the environment. Like maybe it’s you didn’t test the same version of the code even, or maybe you were testing in not a production-like environment in terms of size, or test data, or something like that. But another reason that I think is easier because we testers normally control it is on the side of the script. It could be that you weren’t testing the things that you actually wanted to test. And a big part of that, I think, is workload modeling.
That’s one that I also wanted to comment, ‘cause that has been a pain in different occasions. On one hand, watch model, ‘cause from what you’re saying, there were these projects where in the end we were told, hey, this is a workload, this is a utilization that we are expecting to have in production or that we actually have in production, we want you to test for this. And it was nothing like that. We were told, yeah, it’s a thousand clicks per day, but you wanna test an emergency center when there’s a tornado. How is that true, how is that relevant, or vice versa? For some reason, our scripts are just simulating a thousand clicks per day where it’s 10,000 billions, gazillions, what would’ve been the right thing. Of course your performance test is gonna be awesome if you underdid it. And the other side of the risk, everyone running and, like, the system is dying, what is happening? Oh, we’re pressing it too hard, sorry, our bad.
Yeah, that’s true, could go both ways.
But definitely you always have to worry about configuration drift anyway. Differences in your environments, unless you’re using something like a terraform or something, infrastructure is code where you can make sure that your environments are equal. So if you do your load testing and performance testing in a lower environment or isolated environment that it gets some of that, that it is truly configured the same.
Yeah, even when the environments are exactly identical, though, let’s just say that it’s possible to say that, yeah, there are still situations where if your load tests weren’t simulating realistic enough load, then you can’t really, you may not be able to apply those results to production either.
And one thing, as I said earlier, on one hand, the problem to getting those realistic numbers, at times, the team doesn’t even know, they have no idea what are you asking for. All they know is we want to be able to handle 10 users in the system. I’m kind of simplifying it, usually it’s 10 billion users or something like that, right. Okay, 10 billion users, what does that translate into in terms of utilization? And they go like utiliti-what? And that’s the first struggle, right. Supposing that you get the right numbers that you know exactly how much to stress each thing, ‘cause there there’s another differentiator, right. We get this action a thousand times, okay, but to get to that action at times or on some systems that are the same, you have to do others many more times, less times, and they are just telling you about this one. What is the proportion, how much to stress the other one that the other one does need not that many times? So that’s one discussion. If you get the numbers right then to landing it into the script and your scenarios, also, it’s a very interesting situation, headache-bringing situation at times.
I think part of the problem is that a lot of times, when we think about load testing, there are only a few parameters that we think about immediately. One them is the number of virtual users. Another one is the test duration. And sometimes you might get like a request per second, like you were saying, Leandro, but those three don’t really describe the shape of a load. It’s kind of like trying to encapsulate test results into an average figure, like when you get the geometric mean of something, it’s not a great way to express the distribution. You know, if you say that the average response time is three seconds, that doesn’t mean that you don’t have really high response times. Like you could still have a handful of requests that are at 15 seconds, but maybe the rest of them are a hundred milliseconds. It doesn’t really
There is context.
Accurately describe the whole context, the big picture of the load. And in the same way, things like the number of virtual users, and duration, and requests per second also don’t show the entire picture.
Yeah, wen you’re describing it that way, it makes me think when you are asked to describe someone, oh, has beautiful glasses. I mean, I was talking about Paul, sorry, Nicole.
Oh, sorry.
No, no, I like your glasses, too. But you need a full picture, right? There are so many elements that a scenario or a workload model includes, so many variations, details that, yeah, as you were saying, I need 10 million users in the system doing what, how often, how long, how many of them are doing which ones? And many of those things, you have to get into the weeds to truly understand and design ‘cause this is another one, you will want different workloads for different type of load scenarios. You should not, I was gonna say you cannot, but you can try, but it’s not recommended at all to try to kill too many birds with a single stone. You may be able to get two, but more than that gets complicated, gets crazy. So you have to create a work model for that specific risk situation, that scenario that you want to test for. Just one at a time gets challenging, getting all those components, all those items and things that you need to be careful putting together to model and simulate something that resembles realistically that risk scenario, that risk circumstance that you want to test for.
And here you go again with that killer mentality there, killing multiple birds with one stone. And now Mark’s making comments here that actually for killers.
Okay, okay, let’s say you may be looking for a needle into multiple haystacks, no killing.
There we go.
I was thinking about what you were saying about the hypothetical scenario of a thousand requests over 24 hours. There’s still so many ways to interpret that, right? Because it could be, one interpretation could be, take a thousand requests divided by 24 hours and that’s the amount of load that you want to test with a one-hour load test. It could be like that, but if it’s emergency services, there are certain times of the day when emergencies are more likely. It could be seasonal, as well, like the incidents of emergencies of ambulances that are sent out increases over the holidays for inexplicable reasons. That kind of thing does give more context than just a thousand over 24 hours. I mean, if you just look at a period of 24 hours, those thousand could have been simulated, could have happened in one hour for all you know.
Step in a little far from the ambulance example, don’t wanna go more on the killer topic. Let’s think of a restaurant, if they say, this Golden Arches restaurant, we sell a thousand burgers per day. Okay. Most probably you are selling 80% of that at lunchtime and the rest of the hours we don’t care about. So those different behaviors during that period of time is what we should focus on, or care for, or have better information. Well, in the morning you get more, what’s the name of the thing, hash Browns, no burgers. Even if you are the same restaurant, which one do you want to simulate? What is the utilization at different periods in time during the day? ‘Cause yeah, probably no one, not that many people get a milkshake in the morning, but most of them getting in the night. So some of those variations, you want to make sure that you are doing the right mix, not only how many at a single number per long period of time, what happens each hour of the day, which ones do you care for. As a restaurant, I would say I don’t wanna simulate a load test of, I don’t know, after lunchtime where everyone is gone and we are super quiet. I don’t care load test-wise, right. But lunch time, breakfast time, I don’t know, 7:00 AM in a gym, I want to test those type of things of what people are doing and focus on, get a good work load model for what is happening at those times, right.
So I thought, Mark has a couple comments here. I think also the number of vusers or threads compared to the target arrival rate or expected response time for me is the health of the workload generation. And he follows up with, also synchronizing or rendezvous vuser requests is underused in my opinion. Do you agree, Leandro.
Well, it’s in my personal experience, there have been a few times, at least on the render view or synchronizing part, there are not that many times where I had to leverage those functionalities. I think it kind of depends on the projects that you work on. One that comes to mind, and I really love this example for workload modeling in general, is I worked at a gaming company, Australia’s largest gaming company, and the project that I worked on was for the Melbourne Cup. So they sold tickets for the Melbourne Cup. And if you don’t know, and probably you don’t unless you’re Australian, but the Melbourne Cup is a horse race. It is like a two-minute horse race that is for some reason incredibly popular in Australia, in my state, in Victoria, ‘cause I lived in Melbourne, we actually got a public holiday for it. So, it’s that big. And for some companies, for a lot of these betting companies, the revenue from that one race could be up to 80% of the entire year’s revenue. So it is that big.
Just in that two minutes.
Yeah, and that’s the thing. Betting is open until the race starts. And yet when you look at the workload model for that, it really, the load spikes, the betting spikes just in the few seconds before the race. And that’s for a number of reasons, people might not have had the chance to bet earlier in the week because they were working or something. And if you have a public holiday, then you do have the time to just be there watching. And also there are, depending on the types of bet that you placed, there could be a difference in odds. So they’re trying to get the last possible information that they can before they have to make a decision about which horse to back. And that’s a good case for a rendezvous because there is a clear cutoff and sometimes you see the reverse, like for example, in a university portal or one of these shopping sales, where at midnight all the sales go live. And so you get people maybe refreshing that page all up to that point and then, bam, all of a sudden they start checking out things. And that would be, I think, a good time to use rendezvous.
No, and in a quick example, you got me thinking, as well, video gaming, when you are on these multiplayer games, most probably everyone watching this are familiar, campaigns, online campaigns, certainly there are so many things happening, there are people playing other things, but a campaign is gonna start at this moment, this schedule. I think there a rendezvous would be a very good use of those type of things. Sadly, in my experience, some of those situations were solved with a spike scenario run out here with an average day in production, so overlapping scenarios. But that’s interesting ‘cause, as well, it depends on how do you want to tackle that problem. We are going to have a normal day and suddenly something’s gonna happen. How do we simulate it? How do we model that workload? What tools do we use? That’s a very good point, and again, with performance testing and load testing, there’s no one size fit all or the way to do everything. No, it depends, and you can get two very similar loads, simulations, workloads with different approaches, with different things. And that’s one that I also wanted to mention, like when modeling, it’s something that when you wanna try to get, there are two elements that I like a lot and I struggle a lot of times, when you are modeling your workloads, you have to somewhat randomize the actions. A big thing that we used to leverage in the load runner days was randomized think time in between steps. But eventually you have to say, okay, it’s randomized, it’s happening, like real people is using the system, but in the end I want just this process, just this virtual user to iterate a thousand times in an hour. Oh, but I have random times in between, okay let’s, that modeling, as well, I can tell got so interesting. And even like mathematic, not too complex, but mathematic functions like, okay, over time we have this deviation, if everything goes in the higher random wait time, we may get up here, we may get down there, so on average we’ll be simulating this. It’s all so challenging.
Yeah, those delays are definitely part of the workload model. In k6, we call them sleep. In JMeter, they’re timers. LoadRunner is think time. There are a lot of different terms for it, but basically a delay is some sort of planned sleep in your script that is meant to simulate a real user maybe taking time to think about what they’re going to do next, or read the text on the screen, or type out something in a form. And it actually can have a pretty significant impact on your load tests. First of all, if you don’t have any think time at all, then you need to be careful that you’re not maxing out your CPU and memory utilization on your load generator because that situation where you’re just very quickly firing off requests may not be entirely realistic. And I think that if you have think time that’s always the same time, like this is what you were talking about, having dynamic ones, if you always use the same value for your think time, you might see the staggering in your results, but that’s due to how your test is set up and not at all to do with how your application is responding. And then if you use dynamic think time, like you said, it can make it harder to plan because it’s more difficult to know exactly how many iterations your test is going to complete. So there’s always trade-offs, right. There’s no one way that’s always the best way.
And it depends, as well, on what are you actually trying to simulate. As you mentioned, there are times where we put think times that want to simulate a real person using the system. There are some others where you just don’t want these events to happen at the same time. To give you an example would be like if your script was a soldier marching one, two, one, two, and you’re adding soldiers at the same pace, one, two, eventually they all will be doing the same step at the very same time. If you are simulating a military parade or something like that, that’s awesome. You have the test. On that time, you may want that to happen or, or work it out with some rendezvous continuously happening. But if not, if you just want to simulate a regular march or people passing in the subway, they are not like one, two, one, it would be very interesting to see, but that’s not how real people walk in the every day. There are situations for everything, and it’s challenging at times, as you say, not only wait in between steps, but waiting after you complete a cycle and you want to start another one. There, you may also want some randomization or not to be that static so that your generally load simulations do not have to be that robotic, even if they are done by robots or virtual users. So there are several trade-offs.
I have an extension to your metaphor, I think you’ll like, Leandro, what if instead, this is a way that your script might change depending on the business processes and the context of the industry. So what if instead of trying to simulate a military, what you’re actually trying to do is simulate Fremen running across the open sand, trying to not have patterns in their steps to avoid attracting Shai-Hulud, the great worm. If you don’t know the reference, I’m sorry. But in that case you do not want anything constant. You want something more dynamic, something that isn’t easily noticeable, that won’t skew the results as much. Oh, we have a question from ChipsAho. I don’t know if it’s just the Filipino thing, but there’s actually a good packet of cookies that’s called Chips Ahoy.
Ahoy, love them.
If you are faced with budget constraints, how would you prioritize the workload model to use? That’s really good. And they follow up with, as performance engineers, I know sometimes our curiosity tends to get the best of us, trying to see how the system behaves in different circumstances.
Well, answering to, oh yeah, ChipsAho, is from Ahoy, very good cookies, put ’em in the microwave eight seconds and you’ll love ’em. So it depends a lot, I would say, and your budget constraints, what are you testing would be my question. What risks does your application will face, or what are the most common reasons? Again, there’s no silver bullet or single test that will clear all your performance risks ever again that you may have under loads. What loads were you? What things may harm your system? Those are the ones that you need to focus and prioritize based on that risk. If you already tested one, you know that it’ll survive, next test, maybe go to the next one and down the ladder, the risk ladder. But first it depends on your system. What is it going to subject to? What are the processes? And the risks may be different a spring ago from today. You may be releasing a new process that is super critical, super dangerous, will be touching databases or doing important or even dangerous queries. This time, you may want to focus on that one. There’s no one test to rule them all, one workload scenario, so it depends.
I think there are a few things that I would think about first. What is, like Leandro said, what’s the biggest risk? Meaning if there’s something where if you don’t load test this component there might be security implications, test that. Anything that’s critical, like payment or something that might affect compliance regulations for whatever industry it is, meaning like how you handle individuals’ data, potentially identifiable data, that would be worth load testing. And also anything that’s very public, so maybe something like you’re about us page is nowhere near as public as your homepage. It can deal your company very significant damage, but hard to quantify damage, in terms of reputation, if your entire homepage is down, rather than if your about us page is down. So that could be a situation where something is not really mission critical or important to the business functionality, but it still would deal a lot of damage. So that’s a lot of reputation risk, I guess. A question here not about executors, but we can quickly talk about that. Do we get any performance HTML report in browser k6 load testing? Not yet ‘cause this person specified browser. So I don’t think that we have that yet for browser, but there are a few extensions. There’s the xk6 dashboard one by Ivan Skiba. And there’s also xk6, what is it called, xk6 HTML report or something like that. Go have a look at the session that I just had, the show that I just had with Paul last week, actually, where we talked about this very thing. They’re both extensions and they’re both awesome. They give like a, the Ivan Skiba one, I think, is the one that you did, Paul, right?
Yeah, and actually I was gonna show that a little bit later here.
Okay, all right, well let me know when you want to.
Stay tuned, yeah.
We might as well jump into that, ‘cause I definitely want to show some things about how to actually implement these workload models in k6. So one way that you can do that is by using scenarios. So I have, let’s just start with this one. So I have one script here, it is a protocol level script. And I’m just putting it in a separate JavaScript file just so this is a little cleaner to look at, so that we can focus on the scenario. In k6, we have something called scenarios that are ways to set things that run independently of each other. So you can have it be consecutive, or run one before the other, or you can have them run simultaneously. So this one just has, this example that I’ve got up now is just one scenario and it has an executor. So let’s talk a little bit about executors because this is something that’s specific to k6. So let’s look here. An executor is basically a way to change the way that the load looks. There are a bunch in k6 and I’m actually going to, I already, ‘cause I get this question a lot, I’m going to put this in the chat because I have this cheat sheet for how to go about thinking, deciding which one to use. So I’m going to post it in the chat and we, I think that the rule of thumb is if you want to just try one, you can, or you want a simple one, it’s probably ramping-vus that you want. And that’s because this is the default one and a lot of load testing tools where you set the number of virtual users and duration and then sometimes you might have stages. This is also what k6 defaults to. So let’s have a look. This is what that looks like. As you can see, I have ramping-vus set here, and this is a way to change the load depending on the number of virtual users. I also have these stages here. The first one is kind of like, I was trying to simulate a ramp up. So from when the test starts, this will start with zero virtual users. I’ve got that set here. And then it’ll gradually ramp up to 10 over five minutes. This is shorter because I think I was trying to, this was from a demo, so I was trying to make it something that we’d actually get results from, but this could be, you know, 30 minutes, if you wanted, depending on your test. And then this is kind of what we call steady state, where the number of virtual users doesn’t fluctuate. It’s just holding steady for this period of time, in this case 10 minutes. And this last one where it goes from 10 virtual users to zero is reflective of a ramp down. Paul, did you want to share your screen ‘cause I know you said you had some things in cloud to show us.
I did, yes. And then unfortunately I wasn’t able to share for my script, there’s something going on with my system here. But let me go ahead and share my browser, which is working. So yeah.
Awesome.
So this is a script that I had done and what I did was, as I was trying to go through and then figure out all these different executors, I wanted to see the pattern that each of these executors applies. So now here, let me, maybe it’s better if I start down here. So this is the actual script that I executed. So if you look here, you’ll see that in these scenarios and hopefully, let me see if I can.
[Nicole] Yeah, I was just going to ask.
[Paul] A little bit more legible, there we go. I’m not the only one that may have old eyes and just can’t see. So I loaded up the options so that I would have all these different scenarios. ‘Cause I wanted to see every one of these executors and I wanted to be able to see what kind of pattern that they applied. And originally you’ll see that some of these numbers are a little bit different, but these are all the very specific things that each one requires. So in this case, the shared iteration scenario, and again, I’m not gonna be the expert on what each of these different executors provides, but they’re in the docs. So yeah, we can go through there. I’m gonna be working on these docs, as well, in the next couple weeks and try to clean these up a little bit more, maybe make ’em a little bit more, I don’t know, state it a little bit differently or to help my understanding anyway. But yeah, so I loaded this up, so I have each one of the different executors set up. So here’s the ramping, so similar to what Nicole was talking about with the different durations. So here over a ten-second period, I had it ramp up to, I think, from zero to 10 vus. And then for 10 seconds I would have it keep going at a level 10, so it would be a flat top. And then the last 10 seconds would be ramping down to zero. All these other ones here, too. So, I put every one of these in there so that way then we could go ahead and compare them and, whoa, let me, all right. So each one of these is kind of denoted here and I guess I can’t, I wish I could isolate these and then what’s really, I guess, the most important here is the purple line and hopefully you’re not, no one’s really color blind or missing the color differences here. But yeah, so the purple line is the actual number of the request coming through. And then this blue line, or the aqua, is the actual response time for the service. We’ll just kind of focus on the purple line that’s what these executors are really affecting more so. And then the gray line, this shaded area is actually the virtual users. So you’ll see that here it started at zero, and then it ramped up for a period of time, and then it started kind of flattening, and then it would ramp down. Now this is for the accumulation of all the different executors. Unfortunately, and this might be something I might put in as an issue to possibly add or an enhancement is that each one of these lines here is a scenario. So this actually has the shape from that particular executor. So it’d be nice to show the vus in here, as well, but you can kind of infer it based on the activity, the request. So again, that’s the purple line again. And so here, you see this executor actually ran pretty quickly to get to the number of requests. I was trying to keep them about the same because I wanted to do, like, a progress bar race. So when I was running this locally, I could see the different areas in which was actually completing the same number of requests faster and all that. But yeah, so this was kind of my playing around and, again, this is on the cloud product. So this is an add feature. But now with talking about the HTML reporting, this is what I actually did here. So let me go ahead and I’m gonna go ahead and run this script again. Now unfortunately, you’re not gonna be able to see the actual execution as I run it locally, but let’s see.
[Nicole] Paul, we got a question about the k6 cloud results that you were showing. Could you create a shareable link so that people can also have a look at it?
[Paul] Well, let’s see. How can I do that?
[Nicole] You can go to the three dots in the upper right. Yep, and then share test results.
[Paul] All right.
[Nicole] We can just copy and paste that into the chat.
[Paul] Cool.
[Leandro] And then they are out into the world.
[Paul] Out into the world, yeah. Awesome, cool. See, today I learned again.
[Nicole] Yeah. Awesome. I’m so glad you did this, though, because I wanted to, but I didn’t get around to doing it. So I’m really glad you did.
[Paul] Yeah, and then I wanted to actually have them separate so I could kind of show each one’s pattern again. That’s the thing that I was really trying to get to, and I’ll probably put together a blog post, so keep an eye out for that, where I’ll kind of go through that, and describe, and show patterns that are created by the executors. And then that is as I learn more. Again, so this is, this dashboard that I’m showing here, this is actually an extension. So I did compile this into the k6 binary, custom with this output. And this is available here. This is this xk6 dashboard, and this is also included in our registry on the site, so you’ll be able to find this as well. Pretty easy to do this, the xk6, to create the extension and build it. But anyway, we can always give help. Anyway, I’m gonna go ahead in this other window, and I’m gonna go ahead and start this test running. So now as we see here, and I shouldn’t have to refresh, but let’s see, go ahead and refresh here. All right. So now this test is running in the background. And then we should start seeing the output. There we go. Now, there is a delay. So behind the scenes, this is actually using, it’s creating Prometheus-based metrics and then it’s serving that up in a little web server locally. So it’s really cool. So I haven’t been able to play around with it nearly as much as I need to, but anyway, but this is something, it’s a nice little tool to be able to use if you don’t want just the plain text display in your console. But yeah, so this test is actually already completed. Now, it’s actually still running in a way, though, because we have this externally controlled executor where now from another terminal I could sit there and I could say, okay, yeah, now I wanna ramp up the vus to 10 or that type of a thing, just via rest API. But unfortunately, due to technical difficulties, I can’t show you that, but I will definitely put that in the blog post when I start putting that together. Anyway, this is one of those cool little tools with this extension that you can kind of, I don’t know, really get some cool results. But yeah, that was the big thing. Here’s for the external access, this is the rest API where I could sit there and call that custom executor and say pause testing or even to ramp up the number of vus. Still figuring this out a little bit. It’s a little, I don’t know, it needs some play around with it a little bit to get a good understanding of it. But yeah, I wanted to show that, again, you can do this stuff, have the graphics there. Now, the HTML report that Nicole was talking about, that’s something where, yeah, you could save that off, and again, it doesn’t have the graphics, I don’t think, if I remember correctly, but it does have it where you could then save that off.
It’s not real time either. But it is more like a report.
Yeah, yeah. That’s it, let me go ahead and stop that test. Oh yeah. So again, I will put together something so that way then if anybody wants these details and obviously you can use that link, so it’s now shareable, so I will make sure to not delete it .
Great. I do want to share something, Paul, if you’re okay.
Yeah, absolutely.
Great. So I also wanted to talk about the fact that we can do it, you can put together a scenario using different executors, but another thing that you could do is have each scenario be a completely different test. One could be a load test. Another could be a chaos engineering test, for instance. Like this is one that I had where this get Pokemon is an HTTP request that just hits the API. And this is intended to be more of a load test, but then you could have a chaos scenario and you can say that at a certain point, you fire off this thing that then terminates one of the pods in Kubernetes, according to certain conditions. This is using the xk6-chaos extension. But in this way, you can see that these are not just load-testing scenarios. These are just testing scenarios and they’re different types of testing that you could use. Another type other than chaos is browser testing now that we’ve got xk6 browser. So now you can do things where you have one scenario that’s doing some sort of protocol level test and the other one is doing a browser level test. And because those two are fundamentally different, you may want to be able to control them separately. You probably won’t be wanting to generate majority of the load with a browser level test, for instance, you probably wanna do that with your protocol level test, but you might want to piggyback off of the protocol test and run a few, like a handful, of browser users just to be able to get some front-end metrics. So, I think it’s kind of cool that you can do that. Another thing that you can do with scenarios and executors, well, it’s something with workload modeling anyway for k6, this is a script that I’ve got for percentages. Now, I think the use case for this, I mean, there are a lot of use cases, but what I actually think this could be used for is an e-commerce scenario where a typical scenario is to have like 100% of the virtual users go into your homepage, and then maybe 50% of them go and browse through the product pages, and then of that 50%, maybe only half of those actually added to the cart. So that is also something that isn’t directly related to scenarios and executors, but it does have something to do with the workload model of your test. So in this example, I’m setting a random number between one and 100, and then depending on that, I have this combination, in this case, I’m setting different clients for it. So let’s say that there’s a really big client that we want to use 50% of the time, you could use that, and then pass that on to some request that you’re sending. And in this way, you’re already making something that’s more realistic than just get a client, just get details for a client. Well, which client is that because maybe that’s being cached? Maybe you do want it to be cached, you’d have to see what you want, what you’re trying to achieve with that particular test, but it all has an impact on your testing results.
Yeah, and if I may add for learning some of these functionalities, it makes, and I think I started a comment how the situations, how the ramp-ups work, how when we want to model these targets in the old days, as I said, getting these, okay, my user takes this amount of time of execution, it has this amount of random timers, and it’s a pain at times when your process starts to degrade a little bit of performance, and because of that, they don’t go as fast as you need them to, as you want to generate, because you did the calculations based on optimal response times, right. So some of these controls will be very useful depending on the scenario that you are wanting to focus and not to do all that type of crazy math that we had to do. I remember with the load runner days, it was like, okay, but if the response time starts to deviate a little bit, this we need to, there’s so much that on this situation with some of these options, k6 can mutate may be the wrong word, but adapt to what you’re trying to do, right, and keep doing the loads and doing those processes that you are aimed for. There are some others that you are like, if the system starts to go slow, the users will slow down and that’s right, that’s okay, that’s normal, but on others it’s like, no, I absolutely want that the users keep coming, keeps hearing the system, even if the system is slowing or many other things that can start to happen. These controls truly make our lives easier. in so many of those situations like crazy.
We have a question from SJ who says, is there a pacing config in k6 or maybe a throughput controller equivalent? So let’s talk about pacing, Pacing and load testing terms is related to think time in that it does impose some sort of timing on the requests, but while a think time is more like a direct sleep on a thread, pacing is more like the space between each iteration and there are different ways to achieve this. k6 doesn’t have something that is exactly pacing. But for instance, the way that JMeter would do this is by using the throughput controller, which is what I think SJ is alluding to. And so instead of specifying the time between each iteration, JMeter does it by allowing you to set a target request per second, some sort of throughput. And we do have that. Let me just share my screen again. So this is the executors that we have. And I think this constant arrival rate would be the best for that. Because if we look at the example here, you can set a rate here, a requests per second, that is your target. What k6 does is it sends a request or it does one iteration, sees how many requests were sent, and then adjusts the next one, waits a certain time so that before it does the next iteration to adjust to that. So if it already did, you know, 200 requests per second the first iteration, then maybe it’ll just do the same thing for the second iteration. And it will dynamically change throughout the test. So maybe at the beginning of the test, response times are faster because not all the users have ramped up yet. In that case, the pacing might be slower because there’s no delay on the server side. And so the script is easily hitting the 200 requests per second, but you’ll find that later on throughout the test, you may experience higher response times, and in that case, the pacing will be quicker. So that number will be lower so that it still reaches the 200 requests per second.
Yeah, it’s part of what I was mentioning that easiness of. JMeter, I remember before they didn’t have that plugin for the throughput, what was the name, I just forgot about it. At the very first times that we started to use
Throughput controller.
Throughput controller, thank you. We didn’t have those type of controllers, ways to managing them. And again, bringing the load runner days, I don’t remember many useful devices to go through this. And on this case, when you have a scenario that at times when we were designing this workflow, this scenario, we would know, okay, I know that per user, I want that in an hour these many iterations happen. So if you want that, you can use the per vu iterations executor that will help you to quickly configure and manage that, you do have the iterations per user, and maybe you have 10 users, 20 users, you don’t care about the number of users. Then you select the shared iterations. I just need this throughput to happen this number of times in this period of time, maybe next time I will have more virtual users, maybe next time I will have less. So you have more control on the end game, to call it in a way, that you’re looking for, rather than, and I cannot say this enough, it was a struggle to try to control, and simulate, and reach those goals that you were trying to get in your simulation, and there are so many reasons, as I said, the wait times in between steps were randomized before, as Nicole says, you have the pacing times in between each cycle, and it was so many, like three factors, main factors, I would say, wait time, pacing time, and processing time, each step would add up to that time that there are so many variables that were somewhat random that you could not control. So here you would have some assistance, some help on hitting those goals that you’re trying to simulate and having it realistic or on the goal that you’re trying to reach, right. It’s that difference, I don’t know if anyone has gym experience or has ever visited one of those things, but they will tell you, hey, I want to work leg today. Okay, do five repetitions three times. That’s if you wanna think of that, how long do you take and do each sit up to go, to give an example, that will be your think time, your wait time in between steps. Okay, you need some rest time before you do the next one. That’s the pacing. So for the ones that are not so familiar with those terms, just wanted to throw out that silly example.
It’s one of those things I’d like to, I don’t know, I keep thinking of playing around with, you know, if you look at say HTTP audit logs or something like that, they have a timestamp on there as far as when everything came in. Well, if you take that and use that as the basis to basically replay your activity, I don’t know, that’s one of the things that I keep thinking about, that would be cool to do, or to have, you know, all your activity going into Kafka, and then being able to repeatedly replay the exact same, perfect storm scenario without having the randomness injected in there. I don’t know, I’m probably going off on a tangent, but that immediately made me think of that, though. It’s like this is one of those things where it’s like I always kinda wanna play around with this idea.
I think that there’s a time for that. I think there’s a use case for it, but there are also some disadvantages, I think. that people should be aware of. If it’s information from a production environment, there are potentially security concerns there for using a user’s data to test or a user’s account to test. And what happens if that actually gets replayed on their account, which it could be, it, it might not be. Sometimes it doesn’t work, sometimes sending the same request, even if it’s exactly the same, won’t work. I think that that’s, I think, more experimental, I guess, when you don’t really know what you’re looking for and you’re trying to narrow it down, but I also think that once you’ve narrowed it down, it’s probably easier and cleaner to script it up so that you can consistently reproduce that scenario. And you can also use test data. Also, you’d be able to kind of break it down, you’d control all the variables, so you know with certainty that when there’s only one thing that changed and it was a negative result, then you know that that was the cause of it. But yeah, I think that there’s a valid reason to use that, as well.
Yeah, there’s definitely data sanitization that you gotta be concerned about unless we’re replaying activity on Leandro’s credit card, then it’s okay.
No, please don’t. Oops, there are 10,000 charges per hour in your credit card. What?
Load testing your credit card. Navin Kumar, whose videos I was just plugging at the start of this has a question.
Very good question.
Can we intentionally include sleep time in the response time? Now, Mihail, who’s one of our developers, if I say something wrong, tell me in the comments, but I believe that there isn’t a way, that there’s no k6 specific function to do this or option. What I’m thinking of is you could create a custom metric because you can create really a bunch of different metrics, counters, and timers, and gauges, and whatever in k6. So you could definitely do that and have that reported and you could set the timing manually, as well. Another thing that I’m thinking of is if I had to do this, I’d probably just use the xk6 browser extension, because that’s already going to include everything, the actual render time. ‘Cause now we’re getting, if you’re including the sleep time or the think time, this is more like getting into the front-end performance side of things and I think it might just be easier to do that.
That was gonna be kind of a question that I wanted to do, too, Naveen. Like why do you want to include the sleep time programmatically that you are generating in your script? I mean, as you say, Nicole, the front end is important to know those render times, those wasted moments in the response or back-and-forth communication, but what are you trying to use it for? It would be very interesting if you can share it. I’m curious what’s the use. Oh, okay, you were just being creative.
Yeah, I think that, yeah, probably if you want to just see front-end experience, I do think it would be more useful if in this case, if you’re including it in the response time, then maybe have a constant sleep, actually, because then it’s consistent because you don’t want the sleep to be influencing your results.
I did use something like that a long time ago, but precisely to model this workload to calculate. I generally recommend do a small-ish version of the load test that you want to do, let it run for a while so that you can massage those values of how much random time to sleep here, how much random pacing you have here, and how is the response time affected so that you can balance it out. That could be an interesting use, real-life example, but that’s a good question, as well.
I mean, injecting a network latency and things like that is always a nice thing for backend services. I mean, with microservices, because one single API call on the front end could be going through 10 different microservices ultimately. And to be able to test something where if halfway through those calls if you have a network hiccup where one of the services is not accessible, but it’s, anyway, but it’s all one of those things where it’s like this feels a little bit more on the chaos engineering aspect of things, you know, how is your application going to behave when something goes wrong? If all of a sudden your network grinds to a halt where it’s going really slow, is your application handling that scenario appropriately?
Okay, and on that note, we are actually a little bit over on time already, that went by really quickly. Thank you both for coming and for sharing your thoughts on these topics and also for demoing the k6 cloud stuff that you already had, Paul. Yeah, you are welcome back anytime and I hope we do this again. And for everybody,
I thought that was the last one.
Oh, well, sorry to disappoint you. You’re stuck now, like you’ve signed the contract, that was it. You didn’t read the fine print. Must come to office hours every Friday.
This would be the last time that you would want me on here.
Well, for those of you who are watching, thank you for watching. Thank you for all the awesome questions we got this week and have a good weekend. And if you are in a place where it’s very stormy, good luck, hope we all survive it.
Stay safe.
Bye everyone, have a good weekend.
Take care everyone, adios.

Getting started with using k6 extensions, with Paul Balogh (k6 Office Hours #41)

Fri, 11 Feb 2022 18:00:24 +0100

In this week’s k6 Office Hours, I was joined by my new colleague Paul Balogh, to talk about his experience in joining the k6 team and his first impressions of the xk6 extension system.

Site Reliability Engineering and performance testing with Stephen Townshend (k6 Office Hours #32)

Tue, 19 Oct 2021 00:09:50 +0000

In this week’s Office Hours, I wanted to talk about the intersection of Site Reliability Engineering and performance testing, so I decided to invite two people with different backgrounds from me to discuss it. Stephen Townshend is a performance engineer who recently changed jobs to become an SRE, and Daniel González Lopes is an SRE who had not had experience in performance testing before he joined k6. It turned out to be a great talk!

Resources mentioned

Load testing with k6 for beginners, with İnanç Gümüş and Viktor Mihajlović (k6 Office Hours #31)

Fri, 15 Oct 2021 01:52:48 +0200

Did you know that every new person to k6 does a Week of Testing? It’s a k6 tradition for new hires to spend their first week seeing firsthand what the experience of a new user of k6 is like. How quickly do they find the documentation? What issues do they get stuck on? Then, at the end of the week, the new employees do a presentation for the rest of the team to show their first impressions and findings.

In today’s Office Hours, İnanç Gümüş, one of the newest k6 team members, shares his presentation and his experience as a k6 beginner. Another colleague, Viktor Mihajlović, a Technical Customer Success Engineer, also shares some of the most commonly asked questions he’s heard on the job.

SLOs in performance testing, with Leandro Melendez aka Señor Performo (k6 Office Hours #30)

Fri, 08 Oct 2021 22:05:22 +0200

In this Office Hours, I invited my colleague Tom Miseur and friend Leandro Melendez, aka Señor Performo, to talk all about different criteria in performance testing, including non-functional requirements, SLAs, SLOs, SLIs, thresholds, and checks. Criteria at different levels is how we determine whether testing, including performance testing, has been successful, so it’s important to understand what they are and incorporate them into testing scripts and processes.

How to use k6 Cloud - a k6 load testing tutorial with Bill Rainaud (k6 Office Hours #29)

Fri, 01 Oct 2021 13:58:03 +0200

Bill Rainaud is an inside account executive at k6, which means he talks to users and potential users of k6 every day. In this Office Hours, he explains how to get started with load testing using k6 Cloud, which is a SaaS platform for running and scaling up load tests on the cloud. We also answered a few frequently asked questions about k6’s capabilities and how to do correlation in k6.

How to load test a GraphQL API built with Hasura using k6 (k6 Office Hours #28)

Fri, 17 Sep 2021 13:58:42 +0200

My co-host Simme and I had invited Adron Hall from Hasura to come talk to us about how to load test a GraphQL API, but he was unable to come due to unforeseen circumstances. Luckily, Tom Miseur, who is part of the Professional Services branch of k6, was able to step in on very little notice!

Tom has done a load testing engagement using k6 that involved Hasura, so he was able to tell us what it is and how it can be tested with k6. It turned out really well, thankfully!

How Gitlab uses k6 for load testing, with Grant Young (k6 Office Hours #27)

Fri, 10 Sep 2021 18:13:26 +0200

Grant Young, Software Engineer in Test at GitLab, joined k6 OSS maintainer Mihail Stoykov and me to talk about how GitLab uses k6. Apparently, they began using k6 internally some time ago, and because they like to eat their own dogfood and use GitLab to test GitLab, they also incorporated k6 into GitLab so that their users could reap the benefits, too.

Here’s the GitLab documentation about how to do load testing in GitLab, and here’s a k6 blog post on how to set it up!

How to do continuous performance testing, with Lee Barnes (k6 Office Hours #26)

Fri, 27 Aug 2021 09:34:13 +0200

Lee Barnes and I got connected because we are both speaking at TestCon Europe 2021 this year, and I thought it would be nice to have a chat with him before the conference to get a sneak peek at what he’s presenting about: continuous performance testing. Lee is the CEO of Utopia Solutions, and he’s been in the testing industry for decades, so I really appreciated his insights on where the industry is going.

Catch Lee’s talk here and mine here!

How to debug k6 load testing scripts with Tom Miseur (k6 Office Hours #25)

Fri, 20 Aug 2021 14:38:11 +0200

Tom Miseur joined k6 at about the same time I did late last year, and we immediately hit it off because we both come from a performance testing consulting background AND we both had worked for different competing testing tool vendors before moving to k6. I was excited to have him on k6 Office Hours to talk about a topic we both had a lot to say about: how to debug load testing scripts, and specifically k6 scripts. We got a chance to do a bit of a show-and-tell of tools that we use that we use, and they are all independent of k6.

I also got to demo a cool HTML reporting dashboard that Ben Coleman, a k6 user, created. While it’s not baked into the tool itself, it’s fairly straightforward to add it, requiring only two snippets of code that can be pasted onto any script.

Kubedim: Microservices testing using k6 for reliability, with Kelvin Zhang (k6 Office Hours #24)

Fri, 13 Aug 2021 15:50:46 +0200

Today, my colleague Daniel González Lopes joined me as a co-host on k6 Office Hours, and we interviewed Kelvin Zhang about his thesis at Imperial College London, using k6 to test microservices-based architectures. He created something called Kubedim, an application that intelligently applies non-uniform brownout theory to intentionally dim application requests for the purpose of improving overall availability.

This was a dense Office Hours to digest, but Kelvin really impressed us with his experiments and findings! It’s definitely an awesome use case for k6 and just a great one to watch for any site reliability engineer.

How we work at k6 - building software & working remotely with CTO Pawel Suwala (k6 Office Hours #23)

Fri, 06 Aug 2021 18:09:05 +0200

For this episode of k6 Office Hours, I got the k6 Chief Technical Officer, Pawel Suwala, to give a bit of a behind-the-scenes of how we work at k6 and what we’re working on next. We talked about the methodology we use for building k6, the different teams we have, and exciting new features that are coming soon!

How to run distributed load tests with k6 Operator (k6 Office Hours #21)

Fri, 16 Jul 2021 18:17:13 +0200

In this week’s Office Hours, Simme and I talked about the k6 Operator, which is a open-source project that Simme created for distributed load testing with k6.

k6 Cloud already offers a no-hassle distributed load testing experience, but for those who prefer to use their own infrastructure, k6 Operator allows the setup of a k6 test as a custom resource object that can be deployed along with a Kubernetes cluster.

Load testing and Azure DevOps with José Luis Latorre Millas (k6 Office Hours #20)

Fri, 09 Jul 2021 21:08:36 +0200

José Luis Latorre Millas, Developer Community Lead and Software Architect at Swiss Life AG, joined Simme and me on k6 Office Hours to talk about how he personally uses k6 with Azure DevOps to add load testing to his team’s CI/CD pipeline.

I got to know José from the k6 Community Slack workspace when he mentioned that he would be presenting on k6 in a few meetups. I went to his livestream with CrossDvlUp (Spanish) and enjoyed his approach, so I knew I wanted to have him on. It’s always great to have real users of k6 (both of the OSS and the commercial Cloud option) talk about how they personally use k6 and even what they found lacking in the tool.

José’s multi-scenario template repo: https://github.com/SwissLife-OSS/K6-MultiScenario-template

Register to watch José’s talk at Azure Summit 2021, later this year: https://azuresummit.live/speaker/jose-luis-latorre-millas/

Acquisition talk with CEOs Raj Dutt (Grafana) and Robin Gustafsson (k6) (k6 Office Hours #19)

Fri, 02 Jul 2021 21:04:13 +0200

This week on k6 Office Hours, Simme and I were joined by two CEOs: k6 CEO Robin Gustafsson and Grafana Labs CEO Raj Dutt. It was a little nerve-wracking to have two CEOs on the stream, but it did give us the opportunity to ask both of them pointed questions on Grafana Labs’s acquisition of k6, including any plans they have for how both companies could integrate better in the future.

How to use k6 to benchmark hosting providers with Kevin Ohashi (k6 Office Hours #18)

Fri, 25 Jun 2021 20:57:42 +0200

Kevin Ohashi joined me and Floor Drees on k6 Office Hours to talk about how he used k6 to benchmark the performance of WordPress hosting providers. He talks about the difficulties he ran into while trying to keep tests repeatable and consistent, and I also make some suggestions for how he could improve his load testing script for future comparisons.

Of particular note for me in his results was how clearly the performance of the hosting providers varied depending on the location Kevin had selected for his load generators. One in particular showed a significant delay when a site was fetched from Sydney, indicating the possibility of a lack of a CDN server in that region. It made a great argument for load testing in the cloud, from different geographical regions.

WordPress Hosting Performance Benchmarks: https://wphostingbenchmarks.com/

Benchmarks methodology: https://wphostingbenchmarks.com/methodology/

k6 is part of Grafana Labs! AMA with CEO Robin Gustafsson (k6 Office Hours #17)

Fri, 18 Jun 2021 22:26:49 +0200

On this week’s Office Hours, Simme and I were joined by Robin Gustafsson, the CEO of k6, to make a big announcement: k6 has been acquired by Grafana Labs.

The folks at Grafana Labs have been fans of k6 for a while, and we’ve been fans (and users) of their work since almost the beginning of k6. While a surprise to many, Grafana and k6 together is a combination that makes sense.

Read Grafana’s press release here to hear more about this milestone.

Accessibility and open source support program with Floor Drees (k6 Office Hours #16)

Fri, 11 Jun 2021 22:38:52 +0200

On this week’s Office Hours, Simme and I were joined by Floor Drees, the newest member to join the k6 team. Every new person at k6 goes through a week of testing, and Floor decided to spend her week of testing doing an accessibility review on the k6 Cloud web app and documentation. We talk about how accessibiltiy is an often-overlooked form of operational testing, and Floor shares specific tools that testers can use to make applications more accessible.

We also discuss the k6 open source support program, which makes k6 free for open-source maintainers and regular contributors.

Resources mentioned

k6 Open Source Support Program Boia.org, a good tool for accessibility reviews AlexJS, for identifying inconsiderate language Paul Bruce’s rant on operational requirements instead of “non-functional” requirements Marie Drake on accessibility using Cypress Chrome extension to get dark mode everywhere Contributing Today - open source funding and sponsorship Contributing Today - on open source licensing

Making k6 extensions using xk6, with Iván Szkiba and Mihail Stoykov (k6 Office Hours)

Sat, 05 Jun 2021 22:28:29 +0200

This week on k6 Office Hours, Simme and I were joined by Iván Szkiba and Mihail Stoykov to talk all about extensions. Iván, who is a System Architect at NNG LLC, is the most active k6 extension creator outside of the k6 team. Mihail is a software engineer at k6, and one third of the k6 OSS team.

We talked about the many extensions Iván has created for k6 so that he can incorporate k6 into his team’s daily testing activities, his considerations for creating extensions, and how he got started. Mihail also shares some inside information on why certain decisions were made about the way k6 is structured, and common mistakes he sees people make in their custom extensions.

Application Performance Monitoring using Netdata and k6, with Odysseas Lamtzidis (k6 Office Hours)

Fri, 28 May 2021 23:24:48 +0200

This week, we had Odysseas Lamtzidis on the show to talk about Netdata, which is an application performance monitoring tool that prides itself on having the easiest setup ever. Odysseas admitted that it might not have all the bells and whistles of other monitoring platforms, it Just Works. I see this as being very interesting especially for testers like me, who just want to have something set up but might not want to be bothered by the implementation.

Learning how to use k6 live with Nočnica Fee from New Relic (k6 Office Hours)

Fri, 14 May 2021 21:53:09 +0100

We were happy to welcome Nočnica Fee from New Relic to k6 Office Hours. She very bravely volunteered to learn how to use k6– live! Talk about pressure. But she handled it like a champ!

We also talked about why observability is both particularly important and particularly difficult in serverless architecture. There will definitely be a part two at some point!

Performance testing tool feature requests (k6 Office Hours)

Fri, 07 May 2021 21:53:09 +0100

We managed to get two other folks from k6 to join our Office Hours: our CEO, Robin Gustafsson, and our Head of Customer Success, Mark Meier. We took advantage of the opportunity to discuss wishlist features for k6 and bug Robin about how likely each feature was to see the light of day!

How to get started with chaos engineering (k6 Office Hours)

Fri, 30 Apr 2021 21:53:09 +0100

Recently, I’ve been learning more about chaos engineering. In this week’s k6 Office Hours, I shared my experience as a tester getting started with Kubernetes, chaos engineering with xk6-chaos and k6, and observability with New Relic.

Shift-left testing and creating a culture of quality with Simon Prior (k6 Office Hours)

Fri, 23 Apr 2021 21:53:09 +0100

Simon Prior from the Testing Peers podcast joined us on k6 Office Hours to talk to us about how he’s managed to foster a culture of quality at his job with EasyJet.

The future of load testing with k6, with k6 CEO Robin Gustafsson (k6 Office Hours)

Fri, 16 Apr 2021 21:53:09 +0100

This week, Simme and I convinced k6 CEO, Robin Gustafsson, to come onto k6 Office Hours to talk about the origins of k6 (spoiler: it began with a company called Gatorhole), things we’re working on right now, and what he sees as the future of the company.

Testing Kubernetes and infrastructure as code with Sid Palas (k6 Office Hours)

Fri, 09 Apr 2021 21:53:09 +0100

For this week’s k6 Office Hours, Simme and I had Sid Palas from DevOps Directive come on board to talk about what Kubernetes is, how it relates to Infrastructure as Code, and why either of those should matter to testers.

Using CircleCI and environment variables with k6 (k6 Office Hours)

Fri, 26 Mar 2021 21:53:09 +0100

Simme and I decided to take this week to answer questions on k6 Office Hours, and we ended up talking about CircleCI, environment variables, and how to use them with k6.

Performance testing vs. load testing with Señor Performo and Pepe Cano (k6 Office Hours)

Fri, 19 Mar 2021 21:53:09 +0100

What’s the difference between performance testing vs. load testing? This week, Simme and I were joined by Señor Performo, aka Leandro Melendez, and Pepe Cano, the Head of Marketing at k6, to talk about this common question.

k6 Office Hours 05: Observability with Daniel González Lopes

Fri, 12 Mar 2021 21:53:09 +0100

Our first repeat guest, Daniel González Lopes, also from k6, joined us today to talk about how to use observability in conjunction with load testing. We talked about the differences between metrics, logs, and traces, and how to implemented distributed tracing.

k6 Office Hours 04: Using Cypress with Marie Drake

Fri, 05 Mar 2021 21:46:56 +0100

Marie Drake joined us today on k6 Office Hours to talk about front-end performance testing using a test tool called Cypress and how it can compliment a back-end performance testing strategy.

k6 Office Hours 03: Chaos Engineering with Daniel González Lopes

Fri, 26 Feb 2021 21:40:25 +0100

In this week’s Office Hours, Simme and I invited Daniel González Lopes, a DevOps engineer from k6, to join us to talk about chaos engineering and what it has to do with load testing. Simme also demonstrated how to use k6 to run simple chaos experiments.

k6 Office Hours 02: Observability with New Relic

Fri, 19 Feb 2021 23:24:58 +0100

For this week’s Office Hours, Simme and I invited Gary Spencer from New Relic to talk to us about why he’s recently started using k6, and why an observability platform like New Relic is the perfect companion for performance testing and observability.

The first ever k6 Office Hours

Fri, 05 Feb 2021 21:47:16 +0100

The k6 virtual office is open!

The first time I heard about office hours was from Notion. Recently, Obsidian went in on this trend as well. The concept involves opening your virtual doors regularly to your community, to answer questions and share best practices.

A big part of being a Developer Advocate is talking to people, and even conferences these days are mostly pre-recorded. I kind of miss being able to interact with people. I miss the immediacy of live. So I talked Simme Aronsson into doing a k6 Office Hours with me.

We’re both new to livestreaming, so there’s a lot that we’ve learned and will be improving in future videos, but here’s our first attempt!

K6 Office Hours on Nicole van der Hoeven

Celebrating 100 episodes of k6 Office Hours (k6 Office Hours #100)

What's new in k6 browser? (k6 Office Hours #98)

Recording a Performance Test With Grafana Cloud k6 (k6 Office Hours #93)

From Load Testing to Reliability Testing Tool (k6 Office Hours #92)

Testing Kubernetes Networking With k6 (k6 Office Hours #91)

How to use k6 for site reliability engineering (k6 Office Hours #90)

How Supabase does performance benchmarking using k6 (k6 Office Hours #89)

How to Get Core Web Vitals in k6 (k6 Office Hours #86)

k6 Javascript Ecosystem (k6 Office Hours #85)

Grafana Cloud k6 (k6 Office Hours #83)

Experimental modules in k6 (k6 Office Hours #82)

How to Do Exploratory Testing for Performance (k6 Office Hours #79)

Integrating k6 and Tracetest (k6 Office Hours #77)

How to Organize Testing Scripts in k6 (k6 Office Hours #76)

Distributed Load Testing Using Kubernetes With k6

Distributed Tracing With Grafana k6 (k6 Office Hours #70)

Resources

All About Load Testing Metrics (k6 Office Hours #69)

Assertions in k6 (k6 Office Hours #68)

Related articles

Load Testing Types and k6 Executors (k6 Office Hours #67)

All About Web Performance (k6 Office Hours #66)

Performance testing for beginners (k6 Office Hours #65)

The future of k6 (k6 Office Hours #64)

Getting started with xk6-browser (k6 Office Hours #63)

Load Testing Apache Kafka using k6 (k6 Office Hours #62)

Resilience and Chaos Testing with SteadyBit and k6 (k6 Office Hours #61)

What is browser testing, with Marie Drake (k6 Office Hours #60)

Observability Trends and Future Best Practices (k6 Office Hours #59)

1 Year Ago We Were Acquired. Here's what happened next. (k6 Office Hours #58)

Workload Modeling for Performance Testing (k6 Office Hours #57)

Why Is Everyone Talking About Chaos Engineering? (k6 Office Hours #56)

How to migrate to Kubernetes, with Carlos Ruiz Lantero (k6 Office Hours #55)

What Is Grafana Mimir? (k6 Office Hours #53)

API load testing using k6, with Tom Miseur (k6 Office Hours #52)

End-to-end browser testing using xk6-browser, with Thomas Wikman (k6 Office Hours #51)

Load testing results in the k6 Cloud App for Grafana, with Edgar Fisher (k6 Office Hours #49)

Building Reliable Systems With k6 (k6 Office Hours #48)

How to Analyze Load Testing Results with k6 (k6 Office Hours #47

How to Make Kubernetes Clusters Production Ready using Crossplane, with Viktor Farcic (k6 Office Hours #46)

Load Testing for DevOps with Victor Farcic (k6 Office Hours #45)

Private Load Zones for Load Testing (k6 Office Hours #44)

Postman for load testing using k6, with Tim Haselaars (k6 Office Hours #43)

Transcript

How to design a workload model in k6, with the k6 Developer Advocates (k6 Office Hours #42)

Transcript

Getting started with using k6 extensions, with Paul Balogh (k6 Office Hours #41)

Site Reliability Engineering and performance testing with Stephen Townshend (k6 Office Hours #32)

Resources mentioned

Load testing with k6 for beginners, with İnanç Gümüş and Viktor Mihajlović (k6 Office Hours #31)

SLOs in performance testing, with Leandro Melendez aka Señor Performo (k6 Office Hours #30)

How to use k6 Cloud - a k6 load testing tutorial with Bill Rainaud (k6 Office Hours #29)

How to load test a GraphQL API built with Hasura using k6 (k6 Office Hours #28)

How Gitlab uses k6 for load testing, with Grant Young (k6 Office Hours #27)

How to do continuous performance testing, with Lee Barnes (k6 Office Hours #26)

How to debug k6 load testing scripts with Tom Miseur (k6 Office Hours #25)

Kubedim: Microservices testing using k6 for reliability, with Kelvin Zhang (k6 Office Hours #24)

How we work at k6 - building software & working remotely with CTO Pawel Suwala (k6 Office Hours #23)

How to run distributed load tests with k6 Operator (k6 Office Hours #21)

Load testing and Azure DevOps with José Luis Latorre Millas (k6 Office Hours #20)

Acquisition talk with CEOs Raj Dutt (Grafana) and Robin Gustafsson (k6) (k6 Office Hours #19)

How to use k6 to benchmark hosting providers with Kevin Ohashi (k6 Office Hours #18)

k6 is part of Grafana Labs! AMA with CEO Robin Gustafsson (k6 Office Hours #17)

Accessibility and open source support program with Floor Drees (k6 Office Hours #16)

Resources mentioned

Making k6 extensions using xk6, with Iván Szkiba and Mihail Stoykov (k6 Office Hours)

Application Performance Monitoring using Netdata and k6, with Odysseas Lamtzidis (k6 Office Hours)

Learning how to use k6 live with Nočnica Fee from New Relic (k6 Office Hours)

Performance testing tool feature requests (k6 Office Hours)

How to get started with chaos engineering (k6 Office Hours)

Shift-left testing and creating a culture of quality with Simon Prior (k6 Office Hours)

The future of load testing with k6, with k6 CEO Robin Gustafsson (k6 Office Hours)

Testing Kubernetes and infrastructure as code with Sid Palas (k6 Office Hours)

Using CircleCI and environment variables with k6 (k6 Office Hours)

Performance testing vs. load testing with Señor Performo and Pepe Cano (k6 Office Hours)

k6 Office Hours 05: Observability with Daniel González Lopes

k6 Office Hours 04: Using Cypress with Marie Drake

k6 Office Hours 03: Chaos Engineering with Daniel González Lopes

k6 Office Hours 02: Observability with New Relic