Text on Nicole van der Hoeven

Frontend vs. backend: How to plan your performance testing strategy

Mon, 03 Apr 2023 13:20:53 +0100

Originally posted here.

There are many aspects of application performance, but they broadly fall into two categories: frontend performance and backend performance. As a tester, it’s important to know the differences between the two and how that impacts the way you approach your tests.

In this blog, I’ll provide a high-level overview of frontend performance testing and backend performance testing, including pros and cons of each one. I’ll also explain why you need both if you really want a holistic view of how your application is performing.

Why (and where) to use frontend performance testing

Frontend performance testing verifies user experience at the interface level. Basically, this entails everything a user sees when they access your web app, whether it’s on a laptop or other devices. This might involve automating user interactions with the site through the browser — for example, clicking on a button, filling out a form, or having scripts that load on the browser and run.

In frontend testing, you’re looking at how components of the UI respond.

The metrics we analyze for frontend performance tests are typically quite different from what we use for backend testing. Let’s say we want to know the time it takes for a page to render after a user clicks on a button that directs them there. This isn’t the same thing as measuring server response time; that total round trip takes a lot more into account — things like latency or the time it takes for a client-side script to run and display the requested information.

Advantages of frontend performance testing

The great thing about frontend performance testing is that it provides a more realistic measurement of your end users’ experiences. They don’t care what’s going on under the hood; they care about the overall experience — and that’s exactly what frontend performance testing measures. Unlike backend performance testing, frontend performance testing looks at everything from a user’s point of view: from the time they visit a webpage to the time they can interact with something on that page.

On top of that, most (about 80%) of the performance issues end users report are frontend issues. It doesn’t make much sense to skip this type of testing; it can be a critical step in finding ways to improve the overall user experience.

Disadvantages of frontend performance testing

The most obvious disadvantage of frontend testing is that it doesn’t look under the hood. If you only do frontend testing, you’re almost certain to miss something. Maybe a component isn’t working, which is creating issues for your users. But you wouldn’t know that because you’re only testing what’s visible from the browser. In other words, you might be in the situation where you know something in the application is wrong and responding too slowly, but you don’t know what.

Frontend performance testing can also be quite expensive compared to, say, protocol-based testing. That’s because you need enough resources (memory and CPU) to actually spin up a browser instance and interact with your application the way that a real user would. The same load generator that could run tens of thousands of backend virtual users may only be able to run 50 frontend virtual users before running out of resources.

Finally, only testing frontend performance can sometimes lead to pretty misleading results, especially as you scale. As load increases, the response time for the frontend remains fairly constant, but that’s not the case with the backend.

While the frontend component of response time remains more or less constant, the backend component of response time increases exponentially with the number of concurrent users.

This phenomenon occurs because most of the issues that frontend performance tests might identify, such as a slow-running client-side script, don’t tend to compound. It’ll be just as slow on each user’s computer regardless of how many users are running it. However, problems uncovered by backend performance testing are on the side of the application, not the users. If a database can’t keep up with 100 users sending queries to it, then 1,000 users sending it queries is likely to yield an even worse response time.

Why (and where) to use backend performance testing

As you might have guessed at this point, backend performance testing targets the infrastructure under the hood. This could be a database that contains customer information, a payment gateway, or an authentication service for customers that have already signed up. These are things that an end user doesn’t see and maybe isn’t ever even aware of, but they still have an effect on response times.

Backend testing addresses infrastructure users might not even be aware of, like different databases, gateways, and services that underpin an application.

Scripting against the backend is quite different from scripting against the frontend. Instead of scripting user interactions like clicking, you’re scripting certain requests that are being sent on the protocol level. So, for example, you could have an HTTP GET request for the page and also for its embedded resources.

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
 http.get('https://test.k6.io');
 sleep(1);
}

The sample Grafana k6 script above is an example of a simplified backend performance test. It makes a single HTTP request to retrieve the contents of https://test.k6.io, and then waits one second.

Backend performance includes things beyond just load times and latency. It includes scalability, availability, reliability, and similar attributes, so the metrics that we measure are different as well. We might look at server uptime or response times per component. We could also trace the path of a particular request through the application stack.

An illustration shows that performance testing can address scalability, elasticity, availability, latency, resiliency, and reliability. Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Application performance is more than just speed. Backend testing helps us verify many different aspects of performance.

Advantages of backend performance testing

As we already discussed, frontend testing doesn’t cover everything, but it’s also more complicated than that. The 20% or so of performance bottlenecks that are backend-related can be the hardest to troubleshoot, identify, and fix. Even worse, a lot of backend performance issues can spiral out of control at higher levels of load and can be quite expensive to fix later on in a development cycle. The worst time to realize that you need to rethink your entire infrastructure is after it’s already failed in production. If you take care of backend performance testing earlier (commonly known as shifting left), it makes such pivots less costly.

Another advantage of backend testing is the ability to target different components of an application. You can choose which API endpoint to hit and how hard, and you can target specific user flows, so it doesn’t always have to be end-to-end testing.

And finally, backend performance tests tend to be less resource-intensive; they’re just protocol-level requests that can be easily ramped up.You don’t need to spin up a browser instance, or even parse through the Document Object Module (DOM) of a webpage or look for specific elements. It’s just a matter of sending these requests, and that’s something that you can increase the scale of pretty cheaply.

Disadvantages of backend performance testing

Backend performance testing won’t tell you anything about what’s happening on the frontend, and a lot of sites these days actually use scripts that have to run on the frontend. So with backend performance testing tools, you just download those scripts. But the response time they measure is the download time, which is not really realistic because those scripts are executed. A backend performance test would fetch the text of the script. A frontend performance test would run it — and that script might do something like perform a calculation. You need frontend performance testing to measure that properly.

Backend performance testing is also much more expansive than front end performance testing, especially with microservices-based applications. There are so many components to track that it can be difficult to determine where to look. You can quickly find yourself going down a rabbit hole of constant, never-ending performance tuning.

Frontend + backend testing for the win

So, as we’ve seen, frontend testing and backend testing both seem to have advantages and disadvantages. So which one should you use? Both, if at all possible! They look at different parts of application performance, so doing both is essential if you want a holistic view of user experience.

Software teams can use Grafana k6 to address frontend and backend performance testing.

If you’re operating at a smaller scale, use your own discretion about how to find the right mix for needs. But if you’re testing at a larger scale with thousands of users and you want to test both the frontend and backend aspects of application performance, I suggest you change your test strategy so the majority of the load is generated with backend performance testing scripts. That’s because of the resource intensiveness of frontend performance testing scripts, which we discussed earlier.

So, if you generate most of the load with the backend and then you have a handful of frontend performance testing scripts running in the background, you should still get a good idea of how both are performing. Remember, frontend application performance remains fairly constant throughout different levels of load, so you’ll still get a lot of useful insights on both types of performance without worrying too much about the scaling costs.

Frontend and backend performance both affect what your end users think of your application. If the underlying application servers respond immediately to all requests, but then a huge picture of a crocodile makes your homepage excruciatingly slow to load, then users aren’t going to like that.

On the other hand, if your homepage is really sleek and quick to load, but the application server hasn’t responded with the data that the browser needs to render, then users aren’t going to like that either.

A common problem when testing both frontend and backend performance is script duplication: because of the differences in the two approaches, you often have to use two separate tools — which means two test suites and potentially two scripting languages. While this is a valid approach, it’s more sustainable to look for a tool that can target both types of performance, such as k6.

Whichever tool(s) you end up using, testing both frontend and backend performance is going to give you the clearest view of how your application really performs.

The easiest way to get started with k6 is through Grafana Cloud k6. The free tier includes 500 virtual user hours/month for k6 testing and more. Sign up for free now!

Guide to Load Testing Websites

Thu, 12 Jan 2023 17:12:42 +0000

Original article here, written for k6.io

This doc explains some key concepts about load testing websites, including:

The difference between backend and frontend performance testing
When to choose between protocol-based, browser-based, or hybrid scripts
Recommended practices about how to load test websites

All load tests try to simulate real user traffic to prevent failures, improve reliability, and release new code with confidence. But your approach to load testing must adapt to the type of application you want to test.

In this guide, learn about strategies to test websites, including specific recommendations for complex scenario scripting and test execution.

Load testing approaches

When you approach the load test, first consider the following perspectives:

Backend vs. frontend performance
Protocol-based, browser-based, or hybrid load testing
Component testing vs. end-to-end testing

Backend vs. frontend performance

Performance has a significant influence on the user experience of a website.

For example, when users want to visualize or interact with some information, they expect the website to react quickly.

To measure performance, testers often look at response times, which are affected by two main factors:

Frontend performance
Backend performance

Frontend performance

Frontend performance testing verifies application performance on the interface level, measuring round-trip metrics that consider how and when page elements appear on the screen. It’s concerned with the end-user experience of an application, usually involving a browser.

Frontend performance testing excels at identifying issues on a micro level but does not expose issues in the underlying architecture of a system.

Because it primarily measures a single user’s experience of the system, frontend performance testing tends to be easier to run on a small scale. Frontend performance testing has metrics that are distinct from backend performance testing. Frontend performance tests for things like:

Whether the pages of the application are optimized to render quickly on a user’s screen
How long it takes a user to interact with the UI elements of the application.

Some concerns when doing this type of performance testing are its dependency on fully integrated environments and the cost of scaling. You can test frontend performance only once the application code and infrastructure have been integrated with a user interface. Tools to automate frontend testing are also inherently more resource-intensive, so they can be costly to run at scale and are not suitable for high load tests.

Backend performance

Backend performance testing targets the underlying application servers to see how they behave under production-like conditions. When it comes to websites, while frontend performance involves how assets included in the page are rendered, backend performance focuses on how those assets are processed by the application servers, served to users, and downloaded by browsers.

Backend testing is broader in scope than frontend performance testing. API testing can be used to target specific components or integrated components, meaning that application teams have more flexibility and higher chances of finding performance issues earlier. Backend testing is less resource-intensive than frontend performance testing and is thus more suitable for generating high load.

Which one should you test?

It depends! Ideally, both.

Frontend testing tools are executed on the client side and are limited in scope: they do not provide enough information about backend components for fine-tuning beyond the user interface.

This limitation can lead to false confidence in overall application performance when the amount of traffic against an application increases. While the frontend component of response time remains more or less constant, the backend component of response time increases exponentially with the number of concurrent users:

Testing only frontend performance ignores a large part of the application, one more susceptible to increased failures and performance bottlenecks at higher levels of load.

Testing only backend performance, on the other hand, ignores “the first mile” of user experience and breadth. Backend testing involves messaging at the protocol level rather than interacting with page elements the way a real user would. It verifies the foundation of an application rather than the highest layer of it that a user ultimately sees.

Testing both frontend and backend performance leads to the best overall performance and user experience for your application. Ignoring one or the other exposes you to performance bottlenecks that significantly decrease a user’s satisfaction with your application.

However, if your testing is smaller in scope, you can choose to focus on either frontend or backend with the goal of eventually building a test suite that encompasses both.

Component testing vs. end-to-end testing

When you’re testing a web app, you may wonder whether how you structure your test scripts. Consider the following two methods.

Component testing

One way you could load test a web app is to load test its components. Maybe you know there are issues with a specific functionality due to previous production issues, or maybe you’d like to target components that are business-critical to reduce risk exposure.

In these cases, the type of test you write may:

be protocol-based
call only those API endpoints that are relevant
have smaller or no sleep/think time
be focused on eventually stress testing a component or service or finding its breaking point

Testing in this way is more flexible. With protocol testing, you can specify the endpoints or servers you’d like to hit, and narrow your test’s target that way. You can test specific functionalities while skipping others that come chronologically before it in a standard user flow. You can more finely control the type of traffic that is generated. If you’re recreating a specific mix of requests, you can script those requests and reproduce them in a repeatable manner.

Doing component testing for load may not always require that your script behave like an end user. In fact, it may be necessary to artificially inflate traffic to more quickly reproduce issues, or to deflate traffic to reduce the noise in the logs. By the nature of this type of testing, scripts don’t contain the full flow of a request that you would expect to see in production, so realism is not a priority.

End-to-end testing

You could also do end-to-end testing against a web app. End-to-end testing seeks to replicate real user behaviour and track its effects across the entire stack. When doing end-to-end testing, you might:

do protocol-level, browser-level, or hybrid load testing
replicate the actions in a typical user flow
think about the performance of the entire workflow as well as how long a request took to be processed by each component
write scripts that access the application the same way users would, such as by browsing to the homepage before finding their way to other parts of the website

This type of load testing is broader in scope than component testing, but shallower in terms of depth. With end-to-end testing, you get a better idea of the full user experience of the application as a whole. However, it can also be more complex to troubleshoot, as you have more components to monitor and more places to look for issues that are found.

End-to-end testing is based on real user behavior, so it’s often important that end-to-end test scripts also be realistic.

Protocol-based, browser-based, or hybrid load testing

The decision of whether to test the frontend, backend, or both will also affect the type of load testing you should carry out and the kind of scripts you should write.

Protocol-based load testing

Protocol-based load testing verifies the backend performance of an application by simulating the requests underlying user actions. For websites, this commonly involves HTTP requests that bypass the user interface of your application and are sent directly to a server or application component.

For example, a protocol-based load testing script might request all the resources on a webpage from the application servers, but those resources are merely downloaded. The response times reported by a purely protocol-based script do not include frontend metrics such as the time taken for images to render on a browser. The load is generated by simulating many requests sent to application servers.

While protocol-based load testing may seem to lend itself better to component testing, you can also do end-to-end website testing with protocol-level scripts.

Sample protocol-based test script

The following is an example of a protocol-based load testing script in k6 that fetches the homepage, along with resources embedded into the page.

import http from 'k6/http';
import { sleep, check } from 'k6';

export function Homepage() {
 const params = {
 'sec-ch-ua': '"Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"',
 'accept-encoding': 'gzip, deflate, br',
 'accept-language': 'en-GB,en;q=0.9',
 };

 // 01. Go to the homepage
 let responses = http.batch([
 ['GET', 'https://mywebsite.com/', params],
 ['GET', 'https://mywebsite.com/style.min.css', params],
 ['GET', 'https://website.com/header.png', params],
 ['GET', 'https://website.com/polyfill.min.js', params],
 ]);
 check(responses, {
 'Homepage loaded': (r) => JSON.stringify(r).includes('Welcome to my site'),
 });

 sleep(4);

 // 02. View products
 responses = http.batch([
 ['GET', 'https://mywebsite.com/products', params],
 ['GET', 'https://mywebsite.com/style.css', params],
 ['GET', 'https://website.com/product1.jpg', params],
 ['GET', 'https://website.com/product2.jpg', params],
 ['GET', 'https://website.com/displaylist.js', params],
 ]);
 check(responses, {
 'Products loaded': (r) => JSON.stringify(r).includes('Add to Cart'),
 });

 sleep(1);
}

Recording browser traffic helps you prototype to test websites on the protocol level.

Browser-based load testing

Browser-based load testing verifies the frontend performance of an application by simulating real users using a browser to access your website.

For example, a browser-based load testing script might include instructions to navigate to a page, click on a button, and type out text on a form. Those user actions then trigger underlying requests on the protocol layer, but only user actions are scripted in browser-based testing.

Unlike protocol-based load testing, browser-based load testing scripts generate load by starting multiple instances of browsers and interacting with your application the way real users would. Testing at the browser level can also be the only option for testing Single-Page Applications where a lot of the application logic is executed by client-side scripts.

Scripting on the browser level usually requires the use of different tools from the ones used to test at the protocol level. However, k6 has an extension called xk6-browser that allows the creation of browser-based test scripts alongside protocol-based ones.

Sample browser-based test script

The following is an example of a browser-based load testing script in k6 using xk6-browser on a dummy website. Instead of making an HTTP request, the script views the homepage, then looks for and clicks on a link to the product page.

import { chromium } from 'k6/x/browser';
import { sleep } from 'k6';

export default function () {
 const browser = chromium.launch({ headless: false });
 const page = browser.newPage();

 // 01. Go to the homepage
 page
 .goto('https://mywebsite.com', { waitUntil: 'networkidle' })
 .then(() => {
 page.waitForSelector('p[class="woocommerce-result-count"]"]');
 page.screenshot({ path: 'screenshots/01_homepage.png' });

 sleep(4);

 // 02. View products
 const element = page.$(
 'a[class="woocommerce-LoopProduct-link woocommerce-loop-product__link"]'
 );
 element.click();
 page.waitForSelector('button[name="add-to-cart"]');
 page.screenshot({ path: 'screenshots/02_view-product.png' });

 sleep(1);
 })
 .finally(() => {
 page.close();
 browser.close();
 });
}

Tips for writing browser-level scripts

The following steps can help you get started with a browser-level test script.

Script user actions, not requests. Determine what a user might do for a particular task, and script interactions with elements on the browser level. For example, script what buttons the user clicks on.

Identify unique selectors. Once you have identified which page elements a user interacts with, use the Element Inspector for DevTools in your browser to find a unique, static, and simple way to identify each element. The script needs selectors to find the right element to interact with.

Use elements to verify responses. After every action, use locators to search for elements on the page that you would expect to find. This verification helps ensure that the script has reached the expected page.

Take screenshots for every action while debugging. One of the advantages of browser-based testing is the ability to take screenshots. After every user interaction the script simulates, use page.screenshot to save a visual image of what the script encountered for later troubleshooting.

Hybrid load testing

Hybrid load testing is a combination of protocol-based and browser-based load testing. While you can use two tools or two scripts to execute different types of load tests (one protocol-based and one browser-based), having both types of tests executed by the same script and the same tool is ideal. Aggregating results between different tools can be difficult at best and inconsistent at worst.

A best practice in hybrid load testing is to generate most of the load using the protocol-level test, and then to run a smaller number of the browser-level testing scripts. This approach:

reduces the number of load generators needed, since protocol-level testing requires fewer machines to generate the same load
measures backend and frontend performance in the same test execution
provides a single source of aggregated output at the end
reduces complexity in the creation and maintenance of scripts

Scripting considerations

When you script a test for a website, consider these recommendations.

Consider factors that affect script realism

Record your user journey. Using the browser recorder can facilitate initial test script creation by capturing all the embedded resources on webpages. Check out the Session Recording guide to learn more about how to auto-generate your load test from a user session.

Correlate data. Recordings often don’t take into account dynamic values that are generated anew each time a request is made. Go through the recorded requests and determine whether you need to extract values from previous responses and use parameters in subsequent requests. This practice ensures your VUs behave more like real users would.

Include or exclude static resources. Determine whether you should include or exclude static resources on pages such as images, JavaScript, etc. Consider including them if you want to measure overall user experience. Consider excluding them if you are using a Content Delivery Network (CDN) that is under a separate Service Level Agreement (SLA).

Exclude third-party requests. Don’t load test servers that you don’t own. Many applications make calls to third-party providers for authentication, social sharing, and marketing analytics. Disable these requests unless you have permission to include them in your tests.

Use concurrent requests. To mimic the way modern browsers download some requests in parallel, use batching.

Determine cache and cookie behaviour. k6 automatically resets cookies between iterations, but you can also change this behavior if maintaining cookies would be more realistic.

Use dynamic think time and pacing. Consider adding varying delays so you don’t artificially stagger a script with completely uniform delays.

Use test data. Real users typically don’t search for or submit the same data repeatedly. Consider adding a test data file for the script to iterate through.

Model test parameters and load profile after production. In k6, you can use test options to determine the exact shape and profile of your load test script. Select the appropriate executors for the job.

Create a reusable framework

Use tags and groups. Organizing requests by tagging and grouping them helps you consolidate like metrics and makes your test scripts more understandable by others.

Use scenarios. When combining protocol-based and browser-based tests, use scenarios to independently control their test parameters and executors.

Modularize scripts. Use modules to separate and organize functions for protocol-level testing and browser-level testing, and then use a test runner script to execute them. This approach means different scripts can be versioned and changed without affecting each other.

Integrate your tests into your CI pipeline. Adopting a “tests as code” approach enables you to tie your load tests more closely into your project’s existing CI/CD processes, helping you get the most value out of every test.

Test with thresholds in mind

Create thresholds for both types of testing. Some browser-level and protocol-level metrics cannot be combined because they don’t measure the same thing. Set thresholds for relevant metrics from both the browser-level script and the protocol-level one.

When possible, use hybrid load testing

Use protocol-based scripts to generate majority of the load. When writing the test scenario, use the protocol-based requests to simulate most of the traffic and use fewer VUs for the browser-based requests. Relying on the protocol-level traffic helps keep resource utilization on load generators down.

Execution considerations

When you run tests, consider the test environment and the load generator location.

Run your tests in the appropriate environment

Testing in pre-production environments and in production environments both add value.

Testing in pre-production environments (staging, test, system integration testing, user acceptance testing, and production replica environments) enables you to identify performance defects early, which can save a lot of time and effort (and reputation) later on. Running in test environments also means you can often afford to be more aggressive with your tests. However, it’s also more critical to get your load profiles right, and the test results you get may not necessarily apply to production.

Testing in production yields the most accurate results, but it’s also more risky. Often, testing production is the only feasible alternative. You can reduce the risk of impact to real customers while testing in production by using lower levels of load when running load tests during peak hours, schedule tests for off-peak hours, choosing load test types that are less risky, using techniques like synthetic monitoring that generate less traffic, using real user monitoring tools to get snapshots of user performance at load, and ensuring that your observability stack is working at peak efficiency.

Run tests where your customers are

The location of the load generator(s), where the traffic is coming from, can also have an impact on your test results. The question is: where are your end users located?

On-premises load testing can be ideal when testing early on in the development a website, or when there are machines that can be repurposed as load generators. However, testing entirely from within a corporate network may also yield false positives, in that response times reported are significantly lower than if the same application servers were accessed from across the country.

Load testing on the cloud is an essential part of the testing strategy for many public-facing websites. Using load generators on the cloud gives you access to test in different states and geographical countries, creating a mix of load generators proportional to your users’ locations. Cloud load generators are easier to provision and cheaper to maintain in the long run than on-premise ones. Load testing on the cloud can help you include the effects of network latency in your tests and yield results that are more realistic.

Recommendations

Here are some recommendations to help you plan, script, and execute your load tests for websites.

If you want to test the last-mile user experience of your website:

focus on frontend performance,
write browser-based test scripts,
and consider doing more realistic end-to-end tests of the user flow.

If you want to test the underlying infrastructure of your website:

focus on backend performance,
write protocol-based test scripts,
and consider starting with component testing and then gradually increasing the scope.

If your website is meant for internal or has limited access:

Use on-premise load generators located within the network that most of your users access the website from.

If your website is external and public-facing:

Use cloud load generators in the load zones where your users reside.

Test in pre-production environments when you can, but also consider testing in production in a limited capacity.

Load testing websites can be complex due to the number of viable testing approaches available, the scope for performance testing, and the potential effects of releasing undertested code. By following the recommendations we’ve put forward here, you can tailor your testing more closely to your objectives.

Realistic Calendar Blocking for the Agile Knowledge Worker

Tue, 01 Feb 2022 16:07:31 +0100

Calendar blocking is the only way to identify and avoid distraction.

In a world that tempts us with “a little bit of everything, all of the time,” resisting distraction is the most valuable skill. Yet distraction isn’t always easy to recognize, especially when it comes in the guise of other things we need to do.

Here’s how to use calendar blocking to gain control over your schedule.

Set realistic schedules.

Setting an overly ambitious schedule inevitably leads to disappointment.

The hard part is figuring out what’s infeasible, and that has to come with time. In the meantime, expect to change your schedule.

Use digital tools.

Calendar Tetris isn’t fun– but there are tools that can help.

If you expect to change your schedule, choose tools that make that easier. I use Reclaim.ai to change my calendar in response to changing priorities.

Do a weekly review.

Review how well you stuck to your schedule and adjust it accordingly.

A good schedule, like a good budget, changes over time. Set aside a time every week to assess the previous week and set priorities for the next week.

Relate your schedule to your core principles.

Your calendar should be the reflection of the values you hold.

But what are those values? Keep a list of the principles you live by (or want to live by). Look at your calendar as a way to keep yourself on track.

Reframe restriction as freedom.

When you don’t set your calendar, others set it for you.

Restriction isn’t following a schedule you set. Restriction is being subject to the passing whims of bugs and meetings that demand your attention.

Calendar blocking defends your time.

It frees you to do things you decide are important.

5 Things the Obsidian Templater Plugin Can Do That Templates Can't

Mon, 31 Jan 2022 22:46:52 +0100

The Templater plugin is what the core Templates plugin should have been.

The name is unfortunate, as it implies that the two plugins are similar. Turns out they’re only similar in that one is a limited version of the other.

Here’s what you can do with Templater– and not with Templates.

Add dynamic variables to your notes

You’re probably thinking, but Templates can already add dates.

Templater can do that, and more. You can use it to fetch values in YAML frontmatter, random quotes from the interwebs, and even clipboard contents.

Make your notes context-sensitive

Templater can create notes with different templates depending on context.

For example, you could set up Obsidian to create a new note using the Person template in the People folder but a Weekly Review note on Sunday nights.

Increment your notes automatically

Templater can automatically increment new instances of a series.

You can set it up to create a session log for your 100th D&D session, a review note for your 12th monthly review, or your 10th book review.

Interact with other plugins

Templater leverages all the best parts of Obsidian’s ecosystem.

You can read from and write to YAML frontmatter using Dataview, or run commands that create notes based on text input with Buttons.

Write and run your own scripts

Templater lets you write and call user scripts within an Obsidian note.

This is the ultimate in extensibility. The ability to run custom JavaScript means you’re limited only by your imagination (and skill).

Templater lets you turn Obsidian into exactly what you want.

And isn’t that what a PKM should do?

10 Mandatory Obsidian Plugins for Every Software Engineer

Sat, 29 Jan 2022 23:58:12 +0100

If you haven’t installed any Obsidian plugins, you’re missing out.

Many other mainstream PKM apps seem to have been built with the intention of hand-holding users through every step using a pretty interface. On the other hand, Obsidian is perfectly suited to power users who want to change the tool to meet their needs exactly. But you can only do that with plugins.

Here are 10 Obsidian plugins you should install if you work in tech:

Enable these Core Plugins.

Core plugins are shipped with Obsidian but are disabled by default.

Daily notes: Creates a new Markdown page according to a specified template that serves as a scratchpad and springboard for every new day.
Templates: Lets you define predetermined formats that can be inserted into notes. Some template examples: Daily/Weekly/Monthly/Yearly notes, meeting notes with agendas, person notes to create your own CRM.
Backlinks: When you have a note open, this shows you all the other notes that have referred to it, explicitly (via [[links]]) or implicitly (unlinked).
Quick switcher: Lets you find any note by hitting CMD+O and showing search results as you type. With this enabled, I never bother with folders.
Workspaces: Lets you save the exact arrangement of notes and sidebar options you have selected, and then load it quickly later. I have workspaces for writing, processing content, playing D&D, and making videos.

Install these Community Plugins.

Community plugins are created by users and open-sourced on GitHub.

Calendar: Adds a real-world calendar to your side panel that shows daily and weekly notes. Use Fantasy Calendar for even more customization.
Editor Syntax Highlight: Formats text in code blocks according to the programming language set. Drastically improves readability of any code.
Readwise Official: Regularly imports your highlights from ebooks, web articles, audiobooks, tweets, and PDFs into your Obsidian vault.
Templater: Extends templating functionality by letting you run JAVASCRIPT. In your vault! This blows the functionality of Obsidian wide open.
Dataview: Turns your Obsidian vault into a database using custom parameters in YAML frontmatter. There’s even a way to write queries in js!

Why All Roads Lead to Testing

Tue, 25 Jan 2022 23:59:03 +0100

Why all roads lead to testing

Testing is a skill every software engineer should develop.

It’s easy to dismiss testing as a responsibility only some carry, but it’s best thought of as a hat that anyone can put on. How well you learn to wear this hat affects your career whether or not you’re a tester. Here’s why.

Ideas are cheap.

In tech, having an idea isn’t what sells– the success is in the execution.

iPhones didn’t become popular because it was the first mobile with a touchscreen. The Nintendo Switch isn’t ubiquitous because of its revolutionary hardware. Behind every successful product is a lot of testing.

Testing isn’t just for testers.

Every good software engineer does testing.

Debugging is a form of testing. Questioning requirements is a form of testing. Testers don’t (and shouldn’t) have a monopoly on testing. The reality is that everyone is doing it, often without calling it “testing”.

Testing is about people more than it is about machines.

The fundamental goal of any app is to address a human need.

Testing is finding new ways to represent and understand this need at all stages of development. Testing is a people problem, not a technical problem.

Humans are irrational.

The biggest challenge in building apps is human irrationality.

We don’t always want what we need, we don’t always know what we need, and even when we’re building for others, we don’t always work towards what we need. Testing is an attempt to make sense of this irrationality.

Testing is an attitude.

Getting good at testing extends to aspects beyond work.

Testing is a discipline of questioning. It’s having curiosity as a default state and trying to meet people where they are. These are things every project, and every community, needs in spades.

Testing is rarely the destination.

But some time, some how, all roads lead to testing.

How to Actually Remember What You Learn, for Engineers with Long Reading Lists

Mon, 24 Jan 2022 21:47:59 +0100

Who says you need the terminology of the Zettelkasten process? Here’s how I’ve learned how to learn and how to apply new ideas to my daily work.

If you’re reading this, you probably make your living on what you’ve learned that other people haven’t. Unfortunately, if you don’t have a written system to keep track of new ideas, you’ll probably unlearn everything in a few years.

Most people think reading = learning.

Here’s why reading isn’t enough.

Our brains are best at linking concepts, not at remembering them.
Writing doesn’t help us learn; it IS how we learn.
You can’t learn to swim by reading about it.

If reading isn’t enough, then what should you do instead?

Step 1: Consume

The first step is to consume content, but selectively so.

Content could be a book, a video, a podcast, an article, or even a conversation. Take notes on the author’s main points.

Example: Shape Up is a methodology emphasising iterative cycles, “betting” on work to be done, and building fully integrated slices of work.

Step 2: Redefine

Next, go beyond facts and add your thoughts.

Think about the broader picture of what the author might be saying, and supplement it with your opinions. Example: Shape Up is an Agile methodology concerned with the limits we should apply to different stages of work.

Step 3: Contextualize

Think about what the content means to you, specifically– how does it stack with other things you’ve learned? Where does it fit into your interests? Accept that your context, and your notes, will change.

Example: Shape Up’s idea of fully integrated work supports the notion of context-switching (see: Continuous integration, Multithreading) but could be in contention with productivity concepts like deep work (David Allen), batching, and Productivity Momentum.

It’s your brain!

In the end, nobody can tell you the “right” way to learn.

My Workflow for Making Tech and Travel Videos Online

Sun, 23 Jan 2022 16:16:33 +0100

In the last 5 years, I’ve made ~220 videos across 4 YouTube channels. Here’s what I’ve learned so far about making videos.

Why make videos?

A 10-minute video takes me anywhere from 7 to 15 hours to produce. I wouldn’t be doing it if I didn’t enjoy it!

It can be the most genuine form of expression.
Videos were a big part of how I got into tech, and I want to pass that on.
I’d rather make a video once than explain the same thing to 100 people.

I don’t think my way is the best way, but maybe sharing my workflow will make it easier for you to get started, too!

Pre-production: Ideation

Ideas are cheap. The best idea is worthless if I can’t convey it well.

I start with the idea, then the title and thumbnail. I search YouTube to see what titles have worked for others. Then, I use @Canva for thumbnail creation and thumbsup.tv to test combinations.

Only then do I start to outline the video or research it. I use @obsdmd as my project management tool and content calendar.

Production: Filming and editing

I like to film with talking points rather than a full script. It saves me time, and it comes across as more natural. My main gear: Sony A6600 + Sigma 16mm f/1.4, Røde PodMic, Elgato keylight.

I edit everything myself in Final Cut Pro, using stock footage from @StoryblocksCo and sound effects/music from @epidemicsound.

Post-production: Publishing

I export and upload the video to YouTube, adding the title, thumbnail, description, cards, end screen… I have a checklist so I don’t forget these.

I send most videos to humans at @Rev for captioning. I then use the captions to create YouTube chapters. From there, it’s just about publishing to appropriate platforms.

Optimize for consistency and fun

When in doubt, I chuck technique out the window and do what will keep video creation fun for me!

How to Do What Kind of Testing When: Testing Types by Scope

Sat, 22 Jan 2022 16:42:49 +0100

Correctly identifying when you test could be just as important as what. Below are types of testing according to scope.

Static testing

Static testing occurs when requirements are written. Question requirements and spell out what each one means.

Concerns: Setting up a system to track and link requirements to test cases, negotiating business needs with technical feasibility, clarifying scope.

Unit testing

Unit testing occurs when coding begins. Practice Test-Driven Development by translating requirements into tests before writing the code.

Concerns: Treating test code like application code, getting developer buy-in, communicating technical implementation, performance, pair programming.

System testing

System testing occurs when a build is released. Employ black-box testing: step back from the code and assess the feature as a whole for consistency.

Concerns: Adherence to common design standards, equivalence partitioning, boundary value analysis, fighting scope creep, usability, creating test data.

System integration testing (SIT)

SIT occurs when integrating multiple components. Use white-box or grey-box testing to target the interactions and traffic between components.

Concerns: API testing, stubbing and mocking, identifying request paths throughout the system, performance benchmarking, observability and analysis.

Business acceptance testing (BAT)

BAT occurs when the code is assessed against overall business goals. Seek out SMEs who can help you verify expected outcomes.

Concerns: Adherence to business requirements and regulations, process testing, simplifying request flows, identifying potential customer pain points.

User acceptance testing (UAT)

User acceptance testing occurs before code is released to production. Map test cases to requirements and test real-world scenarios.

Concerns: Testing for superuser workflows, data cleansing, verifying real user behavior from historical logs, exploratory testing, end-to-end performance.

Regression Testing

Regression testing occurs between builds. Supplement a robust automation test suite with nuanced manual test cases that address known problem areas.

Concerns: Setting up a CI/CD pipeline, deciding what to automate, consolidating test and development tools, tracking failures over time.

Question everything

Remember: testing is an attitude.

How to Use Executors in k6: Adapting Load Test Workload Models to Production

Tue, 18 Jan 2022 22:45:52 +0100

Executors in k6 are like traffic cops.

They define the shape and volume of traffic that your k6 load testing script follows during test execution. Executors can also drastically affect your results if you don’t get them right.

Here are all the executor options in k6 and when you should use each one:

k6 executors

shared-iterations: Fix the number of total iterations for the entire test. Use when you want to create precisely X rows of test data during a prep run.
per-vu-iterations: Fix the number of iterations each virtual user (VU) does. Use when you want each VU to use exactly X rows of test data.
constant-vus: Fix the number of VUs and duration, but allow the number of iterations to vary. Use for straightforward tests or shakeout tests.
ramping-vus: Set stages, each with a number of VUs and a duration, and let the number of iterations vary. Use for easiest comparison with other load testing tools and for custom load profiles.
constant-arrival-rate: Fix the number of requests per second (rps) and duration, and allow the number of VUs and iterations to vary. Use when you have a simple load profile with a request throughput requirement (rps).
ramping-arrival rate: Like constant-arrival-rate, but with stages. For each stage, fix the number of rps and duration. Use for spike tests, multiphase load profiles with distinct requirements, and Game Day simulations.
externally-controlled: Fix max VUs and time, and allow everything else to be changed at runtime. Use for disaster recovery, breakpoint, or stress tests where you want ultimate flexibility in adapting your test to real-time results.

Still unsure?

Use ramping-vus. It’s probably what you’re used to from other tools.

Ready to move up? Use ramping-arrival-rate. Rps is a better (but still not perfect) measure of throughput for comparison than just the number of VUs.

When in doubt, look at production.

Making sure your test looks and acts like real traffic yields the best results!

For more details, check out the k6 documentation

Testing tool cheat sheet for load testers who don't just care about numbers

Mon, 17 Jan 2022 16:30:07 +0100

It’s not enough to decide how much load you want to generate. You also have to know how you’re generating it.

Your approach affects what part of your application is tested, the results returned, and the budget you’ll need to run your test.

Here are three main categories of load testing tools:

Protocol-based load testing tools:

These tools simulate the underlying messages sent back and forth between an application and a client.

Choose a protocol-based tool if: you want to generate massive load, you want to load test before components are integrated, you want to test APIs, or you are budget-conscious.

Best in class tools: Gatling, JMeter, k6, Locust, Neoload

Browser-based load testing tools:

These tools automate real user interactions with an interface.

Choose a browser-based tool if: you want to measure end-to-end performance, you want to reuse your browser automation scripts, you want front-end performance metrics, or you have a higher budget.

Best in class tools: Cypress, Flood Element, k6, Playwright, Puppeteer

Hybrid load testing tools:

This approach is a mixture of the previous two. You can do this in a few ways:

Use a protocol tool to generate the majority of the load, and a browser tool for a handful of virtual users (best functionality, highest effort/cost)
Use a browser-based tool for all load (expensive, simplest)
Find a tool that can do both (ideal solution, few options)

Whatever you choose, compare like with like.

Happy testing!

Building a DM's Brain Attic

Sat, 15 Jan 2022 16:45:22 +0100

Anyone other DMs have this problem? My TRPGs folder, containing paid third-party assets for D&D and other RPGs, is made up of 37,884 items (287 GB).

Here’s how I make sure I actually get value out of this embarrassment of riches: 👇

Mike from @SlyFlourish wrote an article applying Sherlock Holmes’s brain attic for lazy dungeon masters, and he called it the “[DM Brain Attic](https://slyflourish.com/brain_attic.html". The idea is to collect interesting techniques or ideas and file them away for later.

Sounds like a good idea, but for most of us, this attic is theoretical. We read books, we watch videos, and we read blog posts on monsters and settings and plots, and we trust that they’ll all be saved somewhere in our heads.

That’s not how it works. Our minds are good at making connections, not remembering CRs from monster statblocks. So how do we make this DM Brain Attic less theoretical and more practical?

Read and take notes. If you haven’t read everything you’ve bought, you’re buying too much. Read what you’ve got and write down the parts that are actually relevant to you and your game. Here’s what mine looks like for the first Arcadia (@helloMCDM) issue.

Combine like with like. This is the hard part, because it involves creating a central place for monsters, items, spells, etc. This is my attempt to do this for monsters. It creates one page per monster that includes its source. Here’s a monster example.

Here’s an example for items that includes SRD items and Vault of Magic Items (by @KoboldPress). Each one links to a page for the item with full details.

Resurface via search: You may have noticed the metadata in the previous page. The metadata gives me criteria to search monsters by. Here’s how I can query my monster database using the Dataview plugin for Obsidian.

And here’s the result of that query, returning monsters that fit my criteria and, importantly, which of my PDFs they come from.

Resurface via random table generators. Another way that I resurface content is by using random table generators using the Dice Roller plugin for Obsidian by Jeremy Valentine. Here’s my NPC page, compiled from multiple sources.

Each die icon can be clicked to generate a new “roll”. Here’s a gif for how that works, using stuff from Sly Flourish’s Uncovered Secrets, accessible to his Patreons. Each click “rolls” on a table I’ve defined in a separate note.

Is building a DM Brain Attic a lot of work? Yes.

But how likely is it that I’d use my 287 GB of third-party stuff while preparing for my game if I didn’t have this system? Not very.

At least this way, I’m using what I’ve actually bought and putting good content into a very real DM’s Brain Attic that I can actually scrounge around for good ideas in, before or during a game.

If you’d like to know more, check out this video I made about how I use Obsidian to run and play D&D online:

Financial Independence Isn't Just for Old People

Thu, 13 Jan 2022 16:43:46 +0100

5 years ago, at age 31, I became financially independent.

Financial independence (FI) means that if I wanted to retire tomorrow, I could. I’d have to make significant changes to my life to do it, but I’d have a plan, and enough resources at hand to get by.

Unfortunately, just talking about FI is uncommon at best and taboo at worst, even in an industry with higher-than-average salaries like tech.

You might be thinking that I received some sort of inheritance or unexpected windfall. The truth is that 10 years ago, my husband and I were in credit card debt and had no major assets. Now he’s retired, and I have a job I love.

The funny thing is, there’s really nothing magical about how we got here.

Here’s how you can do it too.

Step 1: Track your money.

You can’t fix something you aren’t monitoring.

To this day, my husband and I use (YNAB) to track every cent of our money, whether we’re saving it or spending it. When your financial situation is right there in black and white, it’s hard to ignore.

Step 2: Reduce your expenses.

Too many people focus on earning more, when the reality is that it’s far easier to adjust what you’re spending.

You can afford (almost) anything, but you can’t afford everything– so figure out what matters. Reducing your expenses is even better than getting a raise. Do regular audits of fixed expenses. If you have a partner, consider living on less than one income and banking the other entirely.

Step 3: Automate investments.

Set up a recurring monthly payment for a painful amount of money. Invest in low-cost ETFs like Vanguard’s and then forget about them. Resign yourself to living at your current lifestyle level (or lower), and don’t live it up when you get a raise.

And for those who want a goal post, as I did: Actually calculate the minimum amount you’d need to retire. It may be less than you think.

Money isn’t everything, but it can be. To someone who doesn’t have it, money can buy freedom.

What Tabletop Games Teach Us About Application Perfomance

Wed, 12 Jan 2022 16:30:29 +0100

Tabletop games and application performance have more in common than you might think.

See, it’s all about optimization. Both involve juggling several variables: some that you can control, and some that you can’t.

Here’s what I’ve learned about application performance from playing tabletop games.

#1: Get more workers

In worker placement games, you should always spend your first worker to train more workers.

Similarly, one of the most basic ways to increase concurrency is multithreading: using more than one thread at a time. (See: Golang) This applies to testing tools as much as to systems. The 1 Thread: 1 Virtual User paradigm is flawed.

#2: Spend time on getting your engine going

Get your big combos in place before you tap to attack.

Things like setting up automation frameworks or CI/CD pipelines DO set you back in the beginning, but they are worth their mana cost in the long run.

#3: Draft with others in mind

The trick to drafting is knowing what you’re leaving for others as much as what you’re taking for yourself.

The short game is getting your team to do the testing. The long game is getting everyone in the company involved in testing.

#4: Hope for a nat 20, plan for a nat 1

Sometimes proficiency, expertise, and a luckstone just aren’t enough.

Instead of wondering whether an app will fail, take for granted that it will, and start planning for it. Keep that Action Surge in your back pocket.

#5: Sometimes, you just have to trade for sheep

The 2:1 port is valuable, but not always the best option.

Learn what you can, but accept that you can’t learn it all. Reach out to others with compatible skillsets and ask the “stupid” questions that nobody else wants to ask.

Testing is a team sport, and everyone gets a chance to play.

Happy gaming!

The Tester's Toolbox Is Flawed

Tue, 11 Jan 2022 16:30:49 +0100

Every tester has a toolbox.

In the toolbox rests a variety of tools for every conceivable purpose. Part of our job, as testers, is to choose the right one for the job. Sounds reasonable, right?

That kind of thinking could be dangerous.

“The right tool for every job” fails for multiple jobs.

You end up with multiple tools. One for accessibility testing. Another for API testing. Another for browser testing. Another for load testing. This is the modern tester’s dilemma: we spend more time working with our tools than with the application under test.

In the real world, including a new tool in your testing stack means a new framework, usually a new programming language, and sometimes even new colleagues with experience in those tools. And what about continuous testing? All those tools have to talk to each other at some point.

More tools = more time, effort, and money.

When choosing tools, prioritize multitools.

A multitool can be used for different purposes and prioritizes flexibility.

Here are a few ways to spot a multitool:

#1: It is a polymath. A multitool is good at many things and is limited in use case only by the imagination of those that use it.

#2: It uses what you know. A multitool is built on languages and frameworks you and your team already know, reducing ramp-up time.

#3 It plays well with others. A multitool doesn’t force you to use it to the exclusion of everything else. It integrates well with best-in-class tools.

When you optimize for individual jobs, you’re missing the big picture. You’re ultimately left holding the ends of multiple tools and creating a kraken of a test suite with all these tools cobbled together.

Instead, prioritize multitools.

Obsidian for the Uninitiated

Mon, 10 Jan 2022 23:07:06 +0100

Obsidian for the Uninitiated

You’ve installed Obsidian and created a new vault. Now what?

Unfortunately, it can be more than a little daunting to get started with. It doesn’t help that there are terms like Zettelkasten and Maps of Content and fleeting notes and digital garden that are all just trying to structure what we all learned to do in primary school.

Start by logging what you’re doing.

Having a rigid structure does wonders for curtailing creativity and plain fun. Instead, I recommend starting small.

Here are some small steps to take after installing Obsidian:

Step 1: Enable the Daily Notes plugin

The most game-changing feature of Obsidian is plugins.

Go into Settings > Core Plugins, and enable the Daily notes plugin.

Then, exit out of the dialog boxes and hit CMD P (Mac) or CTRL P (Windows) to bring up the command pane. Search for and select Daily notes: Open today’s daily note. That creates a new note in your vault with today’s date.

From now on, do this every day and log things you do during the day: things to do, appointments to keep, videos you watched.

Step 2: The next time you consume content, take notes.

Now, the next time you read or watch something interesting, type its title into Obsidian with brackets:[[Cool video I watched]]

It’s now a link. CMD+click on that link, or click the glasses icon on the top right to switch to Preview mode, and then click on the link. You should now have a new note. Write about what you thought of the video.

Step 3: Start identifying and fleshing out ideas.

Keep going with these steps and every now and then, start to identify cool ideas you come across. Create new notes from them just like you did in Step 2. Don’t worry about getting it all; just do it when and until you feel like it.

These three simple steps, over time, will be the foundations of your own Personal Knowlege Management system with Obsidian.

5 Ways to Use Obsidian as a Software Engineer

Sun, 09 Jan 2022 19:20:49 +0100

Yesterday, I wrote about why I think taking notes is the surprising foundation of a tester’s career.

But it’s not just testers. Every software engineer is a knowledge worker, and that means we get paid for what we know. Using a Personal Knowledge Management (PKM) system such as Obsidian.md helps us keep up with the constant barrage of new ideas in tech.

Here are 5 things I use Obsidian for.

#1: My very own Stack Overflow

If you code, you’ve used Stack Overflow.

What if you had your own Stack Overflow, one that contained cheat sheets of every language you’ve ever tried to hack something up in, including only commands relevant to your job? This is exactly what I’ve done for Python recently, and it’s helping me solidify my knowledge.

#2: Tools database

There’s a tool for everything.

To keep them straight, I have a page for every load testing tool I try, with tutorials for how to get started, an assessment of its features, and observations I had while attempting to learn it. Here’s what I have for the load testing tool k6.

Bonus: I also use the Dataview plugin to query tools that fit my criteria.

#3: An Agile workspace

If Agile is good enough for my professional work, it’s good enough for my personal work.

I use the Kanban plugin to keep track of projects, create weekly and monthly review/retro templates, and work iteratively using the Incremental Writing plugin.

#4: A public learning log

Learning in public is the fastest way to improve.

When I’m trying to figure something out, I document my attempts. With Obsidian Publish, I can quickly push this log out to my domain and share them with colleagues to get a quick review so that they can easily follow my train of thought and tell me where I went wrong.

#5: A digital garden of software ideas

Build libraries, not feeds.

I use Obsidian to save things I read/watch/listen to but also to save space for ideas I’ve learned to germinate. I publish notes before they’re fully fleshed out, with the goal of growing everything over time.

Today, I have 4,567 notes, all about things that matter to me.

Every engineer should have a PKM system.

Take Notes, Not Tools

Sat, 08 Jan 2022 17:00:44 +0100

When I first got into testing, I thought that testing was about learning different tools and frameworks.

I learned how to use Neoload, then LoadRunner, then JMeter, and then SeleniumSilkPerformerVisualStudioGatlingPuppeteerPlaywrightk6 in short succession. I took courses on JavaScript, on C, on Java. I ticked off tools like a grocery list.

It didn’t take me long to realize that I still wasn’t a good tester.

Taking notes, not tools, is what makes a good tester.

When you think about it, a lot of things a tester does revolves around taking good notes.

Requirements are just notes on what the business wants to achieve. Test scenarios are notes on situations that the team wants to simulate. Test cases are notes on how a user would use an application. Defects are notes on issues.

The quality of a tester’s notes dictates the quality of their work.

Listen to people, not tool tutorials

Take notes on what users do. Talk to the business. Pore through Google Analytics logs. Piece together a story of the kind of people who use your application, and how they use it.

Take notes on problem areas. Talk to customer support about the most common questions they get– they know more about pain points than anyone else in the company. Talk to the ops engineers who were on call during the last production incident. These problems are what you’ll need to test later.

Take notes on what you’ve learned. If you don’t like learning, you’re in the wrong field. Document everything, even things you think you’ll never forget, because one day, you WILL forget.

Taking notes, not tools, is the foundation every testing career is built on.

Non-Lazy DMs use Obsidian for D&D

Thu, 30 Sep 2021 16:29:54 +0200

Last month, I wrote about how I use Obsidian for D&D as a player in a 4e campaign (I play a half-elf hybrid wizard/cleric). This is about my second weekly game, in which I am the Dungeon Master for a 5e campaign. I’m a relatively new DM (I’ve got 39 games under my belt at the time this writing, and Obsidian is at the center of my workflow for improving my DMing skills, getting inspiration, preparing for a game, running the game, and making sure my players are having fun.

In many ways, my setup is the Non-Lazy DM’s approach to preparing for D&D games: high-effort, high-touch, and high-return. But I only do it because I enjoy it!

Plugins I use

This is going to be a long article, so if you’re just after the plugins I use, here they are up front!

I go into detail for some, but not all of these, in the article below.

Additionally, feel free to skip around to different sections by clicking on the hamburger menu on the upper right-hand corner and selecting the session that interests you.

Outside the game

Although I use Obsidian while playing D&D, I use it most for D&D between games: fleshing out the world, getting inspiration for new adventures, or preparing for specific sessions.

World-building

Many DMs run published campaigns, but I’m not one of them. I don’t think there’s anything wrong with running published campaigns; I just personally enjoy world-building and being able to tailor the environment to what I know my specific players will like. However, I still do read campaign books, for D&D as well as for other systems, for inspiration. I am also a voracious fantasy reader, and consider everything fair game for “stealing” for my games.

One thing I’ve begun experimenting with is processing my fiction reads the way I do my nonfiction reads. What better place to look for inspiration about creating a fantasy world than other fantasy worlds? How could I use a Bene Gesserit-like faction in my campaign?

I recently reread the first Dune book in preparation for the movie coming out later this year, but while I was reading it, I remembered how much I love the world it’s set in. I read books on Kindle, either on my actual physical Kindle ereader or on the Kindle app on my mobile, and I took highlights while I was reading Dune. You can see my full highlights here, synced via Readwise. I highlight turns of phrases I like, character descriptions that sounded cool, or passages that are particularly revealing of the type of world the story of Dune is set in. Below is an example of a quote I highlighted:

Schools were started to train human talents.” “Bene Gesserit schools?” She nodded. “We have two chief survivors of those ancient schools: the Bene Gesserit and the Spacing Guild. The Guild, so we think, emphasizes almost pure mathematics. Bene Gesserit performs another function.” “Politics,” he said. (Location 248)

It really resonated with me as a good characterization of two important factions in the Dune universe.

Then, just as I do with nonfiction books, I create a separate note about the book itself, attempting to summarize the parts of it that might be interesting or useful to me in the future. Below is a screenshot of my page on the first book of Dune. I identify “Key truths” about the world, because I also like to use that terminology when creating secrets or themes for a fantasy world or city I’ve created.

Here are my full notes on Dune.

Then, I also think about tropes or patterns of character, plot, or setting present in the book. I’ve started collecting these in a a page called Forms in my vault. I think it’s useful to spot things that occur again and again in fantasy fiction, whether they’re overused or not. If they’re tropes that I’d rather not repeat, having a “formula” of the trope helps me either avoid it or consciously subvert it in my games. Sometimes, I just take a form wholesale and incorporate it as is.

Here’s an example of a page I created that was inspired by the Bene Gesserit– a short note about “Nuns who think and fight”.

I haven’t been doing it for very long, but it’s already come in useful when thinking of major events and players in my world.

Adventure creation

Adventures are smaller-scale and higher-resolution than campaigns or worlds, so I often need more specific notes than forms.

I read and highlight adventure blog posts and articles using Readwise’s recently announced Reader app, and then they, along with tweets, books, and PDFs, get sent to Readwise and then Obsidian using the Official Readwise plugin for Obsidian. I’ve written about this process in depth, but here’s what that looks like for Mike Shea’s Return of the Lazy Dungeon Master - affiliate link.

An advantage of using Reader (which is currently in a private beta, so I’m not allowed to share screenshots of it) is that I can even highlight text locked behind a paywall. For example, I have a D&D Beyond subscription and I’ve bought all the books available on their store, but most reading apps don’t know that, so it’s been hard to find one that will let me highlight text from one of the WotC sourcebooks. Not so with Reader. Here’s what my highlights from the first chapter of Volo’s Guide to Monsters looks like in my Obsidian vault after it’s synced from Reader:

Readwise is great because I can highlight something on my Kindle and worry about processing it later in Obsidian. I process incoming Obsidian notes daily from 9-10pm– that doesn’t mean I get through them all, just that I have a scheduled time to go through highlights and notes every day.

For D&D-specific notes, that also involves summarizing what I’ve learned into the notes I already have on how to be a dungeon master:

The referencing feature on Obsidian lets me confidently paraphrase without stealing. All I need to do is put a [^lazydm] where I want the superscript to be within the text, and then at the end of the note, I put something like:

[^lazydm]: Shea, M. (2018). _Return of the lazy dungeon master_. [[Return of the Lazy Dungeon Master|My highlights]].

That gets parsed as a footnote by Obsidian and lets me properly credit authors.

When I find good tools that help me flesh out adventures, I try to update my page on DM tools and inspiration, so they’re all in one place.

While creating adventures, I like to use Obsidian Admonition to call out specific parts of the text for me as I’m running the adventure, like this:

Random tables

I definitely have overarching plans about the wider plot in the world at large, but I also like to be surprised. Throwing random elements into games is one of the ways that I can keep myself on my toes and remind myself that I’m not the only one making the story.

There are many random tables in both official and unofficial D&D sourcebooks, but I only use those as starting points. For me, creating random tables that are tailored to my characters and to the things they care about is where it’s at!

I’ve experimented with creating my own random generators for various things, as well as using existing ones like Donjon, but it’s an Obsidian plugin, Jeremy Valentine’s Dice Roller, that I have settled on.

The name “Dice Roller” makes you think it’s only for rolling dice, but it’s much more than that. Its REAL value, in my opinion, is in how it lets me create tables for ANYTHING, and then randomly pull one row out of it. Here’s how it works.

First, you create a table. I had a session where my players discovered a place I called Shipwreck Beach, and had a lot of fun going through several of the ships that were there. Here’s a table of “encounters” I put together for that session.

### Ship encounters
| Roll | Encounter |
| ---- | --------------------------------------------------------------------- |
| 1 | Carnivorous Ship, ToB II p. 56 |
| 2 | Map Mimic: https://www.dndbeyond.com/monsters/1624929-map-mimic |
| 3 | Random curse: p.249 of GM's Book of Random Encounters - d100 |
| 4 | Clockwork weaving spider (1, ToB p. 66) |
| 5 | 1d6 Cobbleswarm (2, ToB p. 68) |
| 6 | a feral little girl wearing a periapt of proof against poison |
| 7 | A large gold coin bearing the crest of the Kryn Dynasty and the queen |
| 8 | Book about the Arms of the Betrayer Gods |
^ship

Note the ^ship identifier at the bottom there– it can be called whatever you want, but you’ll use that identifier when rolling on that table. I also included the Roll column, in case I decided on the spot to let players roll instead of doing it myself.

Next, in another note (or the same one, if you want), include this line:

Shipwreck encounter: `dice: [[Shipwreck Beach#^ship]]`

When you flip to Preview mode, you’ll see this in the place of the line above:

That’s it!! This has made a HUGE impact on my games, because it’s so easy to create or modify tables. I use random encounter tables sparingly during the game, but often use it before a game, to introduce something fresh that I hadn’t thought of.

Session prep

As you might have guessed, I mostly use Sly Flourish’s Lazy DM steps for encounter creation, saved as an Obsidian template:

---
type: session
world: Exandria
campaign: "The Slayers Five"
date: {{date}}
game_date: 836-03-
location: Rumblecusp
characters: [Boris, Lira, Ora, Jamedi, Flint]
tags: [TVZ]
---
## [[Session Summary]]:
^summary
---
## Housekeeping
## Recap
![[^summary]]
## Strong start
>
## Scenes
- [ ]
- [ ]
- [ ]
- [ ]
## Secrets and Clues
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
## [[Loot]]
- [ ]
---
## Log

Before each session, I’ll go through each of these sections.

I add metadata to every session for use with the Dataview plugin later, in case I want to do a search for all sessions that involved a particular NPC or that took place in a certain location, for example.

I fill out the Session Summary section after the game, but I put it at the top so I can immediately see what happened in the session in case I need to refresh my memory on it in the future. I also identify this section with a ^summary so that I can embed it in the next session, as a recap - more on that below.

In the Housekeeping section, I write down things I want to mention before we dive into the narrative of the session. Usually, these topics are scheduling- or rules-related. For example, if my players have just levelled up and I see that one of them has chosen a feat, I might make a note of it here to make sure that they know what it means. My players are pretty new, so I think it’s better to err on the side of more communication rather than less.

In the Recap section, I call the sumary from the previous session and embed it into the current session using something like ![[38_8360310-8360311 Seduction at the Moonlight Chateau#^summary]]. When in preview mode, Obsidian shows just that section labelled with ^summary from the previous game. I like to start the game by either reading out the summary myself, or asking one of the players to recount what they remember. That way we’re all on the same page when we begin.

In Strong start, I write a paragraph or two of prepared text that I use to kick off the story. This is usually the only text I’ll read out word for word during the game.

Scenes are a list of events that I THINK will happen. My players frequently surprise me, which I welcome, so I also list these with checkboxes so that I can keep track of where the story deviated from the plan.

Sly Flourish does recommend a separate section for Fantastic Locations, but I tend to prepare these as separate pages. For example, here are my notes on a possible location, The Charred Circus. I don’t include them within the session note itself, but I do make sure I have a few that characters could go to. Then, I link to all of them from the note I have for the city they’re in, like this one for Port Damali.

Secrets and Clues is where I put a list of 10 things that players MIGHT find out somehow. This is where I look when a player rolls a natural 20 on a Perception check. The idea is to describe what the secrets are, but not get too caught up in trying to plan when or how the players will find out those secrets. I try to include things that might be interesting for the players, as well as a few potential adventure hooks. Here’s a list that I made for a previous game. I have never used all 10 up in one game, but I like having more than I need.

In Loot, I put in any items I know the party will likely get to. If they’re in a boss fight, I usually save in this section all the things I’ve decided beforehand they’ll get. If I don’t have something specific like that in mind, I sometimes randomize a bunch of non-magical trinkets. The Loot page contains items I’ve picked out for specific characters.

Post-session processing

If your players are anything like mine, you’ll often find yourself with a bunch of unused material and a lot of stuff you still had to improvise on the spot, despite adequate preparation. That’s okay! I actually enjoy when my players do something unexpected. I encourage and reward creative play… but it does mean that I have to do a bit of a debrief after each session.

I try not to prepare things that may really be “wasted”. Things like locations, NPCs, or general plot hooks can usually be recycled. After games, I spend some time tying off loose ends, noting down decisions the party made and the reasoning behind those decisions, and tweaking connections between NPCs or places based on how the party reacted.

Fleshing out bits of the world, with each city or NPC getting a page
Tagging cities and NPCs with Location or NPC, and then within the city, doing a query for NPC and city name to get everyone living in that city.

I use metadata for every page in my campaign so that I can use the excellent Dataview plugin later to find whatever I want. You already saw my session template, but here’s an NPC template that I use:

---
type: NPC
faction: []
location: []
world: Exandria
campaign: "The Slayers Five"
date: [{{date}}]
description: ""
race: []
gender: []
class: []
---

I insert this template for every single NPC in my world, whether my players get to it or not. Because I use it so often, I’ve set up a hotkey for inserting templates in the Obsidian settings:

That way, all I have to do is hit CMD + ; and select the right template from the list, and the text above is inserted at the beginning of the note as frontmatter, with all the fields I need to fill out. The date is also automatically generated.

Using the Dataview plugin, I can then embed queries on location pages so that I can see NPCs in that place at a glance. For example, here’s the snippet I put under the People section in my Port Damali page:

\```dataview
table shop as Shop, faction as Faction, description as Description from "Exandria"
WHERE contains(type, "NPC") and contains(location,"Port Damali")
SORT file.name ASC
\```

(If you try this, remove the backticks \ in the snippet above– those are just to show you the syntax for a dataview query.)

That query, when rendered in Preview mode in Obsidian, looks like this:

Having the metadata for every NPC, monster, item, and place in my world is essential for creating queries like this. I’ve also found it useful to maintain a model of the metadata I use:

- `type`
- `NPC`
- `PC`
- `shop`
- `place`
- `item`
- `session` for D&D play sessions
- `quest`
- `race`
- `faction`
- `family`
- `monster`
- `condition`
- `deity`
- `other` for anything else
- `characters` (for sessions): characters involved in this session
- `deceased`
- `true`
- `false`
- `family`
- `faction`
- `race`
- `gender`
- `class`
- `description`
- `date`
- `game_date`
- `world`
- `Exandria`
- `Forgotten Realms`
- `Exandya`
- `campaign`
- `Order of the Wandering Mind`
- `The Slayers Five`
- `Die vergessene Schwester`
- `Temporary White Circle`
- `shop` (for proprietors)
- `shop_type` (for shops)
- `general` : general goods store
- `alchemy`: potions
- `magical items`: sells magical items
- `services`: magical spells, messaging, brothel
- `tavern`: food and drink
- `inn`: lodging
- `herbalism`
- `smithy`
- `location`: city of residence
- `origin`: city of origin

During the game

I keep a “World” page for each of my D&D campaigns, including a list of sessions:

During the game, I use the session page that I mentioned above, and start filling in the Log section. I tend to take way fewer notes as a DM than as a player, just because I’m busy doing other things, but I do always write down things that the players do that I might need to remember later.

One thing that comes up frequently in my games is the need to improvise encounters, NPCs, or locations. I’ve had great success in creating a Random Rolls page that collates many of these random tables in one note. If you click on that link, you unfortunately won’t be able to see the results rendered, because the Dice Roller plugin isn’t available for Obsidian Publish, but maybe you’ll get an idea of things I’m rolling on. Most of the tables referred to are from this Random Tables note, but every now and then I’ll include tables in location notes instead.

I always have this Random Rolls page up as part of my DM workspace, in case I need to quickly improvise something. Here’s what the NPC section looks like:

I can use any or all of those things that I want, or use them as inspiration for something else I make up on the spot. Either way, it means I’ll never be completely at a loss for the name of an NPC. ;)

Initiative Tracker

I use D&D Beyond for character sheets, but I use Jeremy Valentine’s initiative tracker to handle combat because there are a few critical features that D&D Beyond’s Encounter Tracker doesn’t yet support:

Status conditions
(Easy to tweak) homebrew monsters
Hiding monsters’ names from the players

The Initiative Tracker plugin supports all of these. When activated, it appears as a side panel in Obsidian:

I can easily see everyone’s HP and AC at a glance, and also apply special status conditions when necessary. My players don’t have to know whose turn it is– they just have to know when their turn is. I warn them verbally when they’re “on deck”, and they haven’t had an issue with that.

D&D Beyond does allow you to create homebrew monsters, but the only real benefit to doing so is to be able to use it in their Combat Tracker, so I’ve been using another plugin by Jeremy Valentine, the author of this one, to create monster statblocks

Statblocks

The 5e statblocks plugin lets me easily create homebrew creatures just by using this format:

\```statblock
name: string
size: string
type: string
subtype: string
alignment: string
ac: number
hp: number
hit_dice: string
speed: string
stats: [number, number, number, number, number, number]
fage_stats: [number, number, number, number, number, number, number, number, number]
saves:
- <ability-score>: number
skillsaves:
- <skill-name>: number
damage_vulnerabilities: string
damage_resistances: string
damage_immunities: string
condition_immunities: string
senses: string
languages: string
cr: number
spells:
- <description>
- <spell level>: <spell-list>
traits:
- [<trait-name>, <trait-description>]
- ...
actions:
- [<trait-name>, <trait-description>]
- ...
legendary_actions:
- [<legendary_actions-name>, <legendary_actions-description>]
- ...
reactions:
- [<reaction-name>, <reaction-description>]
- ...
\```

(Again, remove the backticks \ at the starting and ending lines of the snippet above if you copy and paste this into your vault.)

I usually start with a “real” monster from official or unofficial sourcebooks, then tweak it from there. For example, here’s a sample one that I tweaked from Tome of Beasts:

The best part about this plugin is that it integrates with the Initiative Tracker. When you create a statblock and save it, that monster then becomes available for use within the Initiative Tracker, as you can see from the screenshot above.

Publishing

I subscribe to Obsidian Publish, partly to give back to the fine Obsidian developers who’ve made a tool I use daily and partly because I’ve found that it’s reduced friction in publishing. I use it primarily for non-D&D-related things, but D&D is part of my life, so it’s there as well. You can find my notes on the D&D campaign I DM here.

Some of the plugins that I use unfortunately don’t render correctly when Published, but the developers are working on that.

I publish my campaign notes mainly because I love seeing other DM’s thought processes and systems, so I thought I’d return the favour by sharing mine as well. In case you were wondering, my players don’t know I publish these notes– or, at least, I hope they don’t!

Player-facing campaign vault

I used to drop handouts into a WhatsApp group with my players, but it became increasingly unwieldy to look for something. We switched to Slack, but that had the same problem of needing to scroll back through lots of photos before finding the right one. Photos aren’t very searchable in those apps.

A few months ago, I switched to Obsidian for handouts. I created a separate vault for the campaign and shared it with my players via Dropbox. I did have to talk them through the setup, but it’s worked out very well for us as a free alternative.

In the vault, I put things like:

Pictures and other handouts
Session videos and summaries
Links for everything we use as a group
House rules

Session page

I create a page for every session we have, which has a section for the video recording of that session (unlisted and just for us, sorry), a player-friendly summary, and a log. The log is where I dump photos I give them during the session. Here’s what that looks like:

I don’t put any other notes in the log; that’s the players’ jobs (if they want to).

I like to have a “Story Thus Far” page that embeds the video and summary sections of all the sessions, so any of us can read through it to remember what’s happened. It’s fun to see how far we’ve come as a group.

About our game

The About our game page has a list of links to all the tools we use, our schedule, and a few things about the campaign and the world.

Rules

I keep a house rules page that shows all the extra rules I’ve put into the game. I mostly play with all the WoTC-produced sourcebooks, but I’ve added some extra rules about things that my players are interested in, such as strongholds (for which I use Matt Colville’s Strongholds & Followers book) and smith’s tools.

I also have some other pages on rules that come up often, such as Ship statblocks (since the campaign is set on a coast) and suggested downtime activities.

Conclusion

I call my approach to game preparation the Non-Lazy DM’s approach as a tongue-in-cheek nod to Mike Shea’s content, but there’s truth to the moniker, too. My setup is extremely detail-oriented, time-consuming, and fun… at least for me! And that’s the key: I wouldn’t be doing any of this if I didn’t find it truly enjoyable to world-build to this level of detail.

If you’re a non-lazy DM too, thanks for sticking around! I hope this has given you some ideas for how you could use Obsidian to run your games and stimulate your imagination, too.

How to import Shortform highlights to Obsidian via Readwise

Mon, 16 Aug 2021 21:40:16 +0200

I’ve always been dubious of those book summary apps. You know the ones– they tend to have obnoxious ads that promise to help you “read 5 books a day” or learn the world’s secrets in 15 minutes. Having tried a few of them, I’ve been disappointed with the quality of their summaries (or lack thereof). They attempt to distill the message of a 300+ page book into a handful of bullet points, which I’ve found neither accurate nor useful. Recently, though, I found myself happily subscribing to one, and here’s why.

What’s Shortform and why is it awesome?

Shortform really took me aback because it isn’t just a book summary. I should mention here that I’m not affiliated with Shortform and gain nothing by recommending them. I’m just a happy subscriber.

When I read their guide for the book Why We Sleep, I was pleasantly surprised to find that Shortform not only summarized it succinctly, but they also added intelligent counterarguments to the author’s points. In some cases, when the author made a false claim, there were Shortform notes that linked to studies that showed why the claim was false. In other cases, Shortform mentioned related concepts proposed in other books that strengthened the author’s viewpoint. I found myself looking up their guides of books I’d already read, just because I was curious to see if I had missed anything. And I had!

In short, Shortform is the only book summary reader I’ve found that goes beyond summarizing and actually adds value to the reading experience, even if you’ve read the book.

They also have a one-page summary of the book AND a summary per chapter, so information is presented at different levels of detail. In my time with it, my most frequent workflow with it has involved reading the one-page summary of a book, buying it if it was interestering enough, reading the book, and then going back to Shortform to go read the chapter summaries. And both the one-page and detailed chapter summaries, including Shortform notes, can be highlighted!

Naturally I wanted to take it further than that. I already sync my Kindle highlights to Obsidian automatically and incrementally process them in Obsidian, so I wrote a script to have my Shortform highlights going to Readwise as well.

My Shortform highlights for the book Mindset, in my Obsidian vault.

The screenshot above shows the final result: highlights I made in the Shortform web or mobile app that have been synced to a page in my Obsidian vault. Here’s how I did that.

Note: Since this article was published, Shortform and Readwise have created an integration, so the script I mention here is no longer necessary. Click here to see what I’d recommend doing today.

What you need

A Shortform account. (affiliate link)
A Readwise account. (This is an affiliate link. If you sign up with this link, you’ll get a two-month free trial instead of one, and I get a small percentage if you sign up for a plan.)
A tool that Readwise can export to, such as Obsidian, Roam Research, Notion, or Evernote. You can also use their CSV and Markdown exports.
A local installation of Python 3, which you can download here.

Obsidian is my Personal Knowledge Management tool of choice, and I use the Readwise Official community plugin to get Readwise to send highlights to Obsidian. However, using Obsidian is not necessary as my script connects only Shortform to Readwise.

The setup

Clone my Shortform-to-Readwise repository on GitHub or download it to your computer.

Enter your tokens

The file variables.py requires two tokens: one from your Shortform account and another from your Readwise account:

# Your authorization token for Shortform
authToken = ''

# Your Readwise API token
# Retrieve this token from https://readwise.io/access_token
readwiseToken = ''

Getting your Shortform token

This is a little hacky, because Shortform doesn’t expose an API token in their web app. So you’ll need to log in to your Shortform account and open up DevTools in your browser. In Chrome, go to Options > More Tools > Developer Tools (or, on Mac, hit CMD+OPTION+I). Click on the Network panel.

You should now see Shortform on one side, and DevTools on another in the same browser window.

Then, go to your Highlights page. Doing so will populate the Network panel with a bunch of requests that look like this:

DevTools open in Chrome, showing requests from Shortform.

Click on the request with the name ?sort=date, highlighted above. That’ll open another side panel that will show Headers. Scroll down to the section marked “Request Headers”, and look for where it says Authorization: Basic followed by a bunch of characters.

Authorization headers for Shortform, in DevTools.

I’ve removed my personal authorization token from the screenshot above, but it’s a long series of characters that comes after Authorization: Basic . Go ahead and right click on it and click Copy Value. Paste it into variables.py as the value for authToken, and then _remove Basic (including the space after it) so that you’re saving only everything that comes afterwards.

Getting your Readwise token

Thankfully, this one’s a little easier. Go to this page on Readwise after logging in. Click “Get Token”, click “Copy to clipboard”, and then paste it into variables.py.

Save variables.py.

Note: Don’t share either of these tokens with anyone else, as they will then be able to access your Shortform or Readwise accounts.

Run the script

Open up your terminal, and cd into the /shortform-to-readwise repository. Then, run:

python3 getHighlights.py

It should finish in a few seconds (depending on how many Shortform highlights you have), and then you can verify that it worked by going to your Readwise library.

Shortform exports showing up in your Readwise library.

You should see books that have the Shortform- prefix. If you do, then you’re done! If you’ve got your Readwise account set up to push your highlights elsewhere (such as Obsidian), then you’re all set!

In case you’re wondering why I added the Shortform- prefix, it’s because I often read the Shortform guide, and then read the book itself. Rather than end up in a sticky situation where the Shortform guide highlights might override the actual book highlights, I titled it differently so that I get two pages for the two sets of highlights.

What about running the script regularly?

The whole point of this, at least for me, was to set it up so that I’ve got highlights going to Obsidian automatically, without needing my intervention. What I did was to use crontab on macOS to run the script every 30 minutes, but you can set up your own schedule.

Give cron full system access

Cron needs to have the right permissions to run scripts on your behalf, so you may have to go to System Preferences > Security & Privacy > Privacy > Full Disk Access on your laptop and add /usr/bin/cron to the “allowed” list.

Find the path for your python3 installation

In your terminal, run:

which python3

This will give you a path. For me, it was /Library/Frameworks/Python.framework/Versions/3.9/bin/python3. Copy this.

Open crontab

Then, from your terminal, run the command env EDITOR=nano crontab -e to open up a list of cronjobs. If you don’t have any other jobs, this will be blank.

Set a schedule

Add a line to crontab for the schedule you want to create. This line has to be in the format

* * * * * command

where the five asterisks correspond to:

minute
hour
day of the month
month
day of the week

In my case, I wanted it to run every 30 minutes, so here’s what my schedule looks like:

*/30 * * * * /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/nic/python/shortform-to-readwise/gethighlights.py

Make sure you change /Users/nic/python/ to the path where you put the repository and your python3 path instead of /Library/Frameworks/Python.framework/Versions/3.9/bin/python3. Now exit out of it and save it (CTRL+X, and yes). The Python script should automatically run every 30 minutes.

That’s it!

I consume a lot of content daily, but without systems in place to capture insights from that content and process it, it all just gets forgotten eventually. That’s why I love services like Shortform and Readwise– they let me get what I’ve read/watched/heard out of my head and into Obsidian, where I can review them more critically and truly learn the important parts.

Both are paid services (which I paid for out of my own money), but I really think they’re worth it.

Note: This post contains affiliate links. Check out My Ethics Statement to see my standards for promotions.

Meet the Grafana Labs team: Nicole Van Der Hoeven, Developer Advocate at k6

Fri, 13 Aug 2021 15:29:37 +0200

Originally posted here.

As Grafana Labs continues to grow, we’d like you to get to know the team members who are working on the cool stuff you’re using. Check out the latest of our Friday team profiles.

Meet Nicole!

Name: Nicole van der Hoeven

Current location/time zone

I currently live in Maastricht, in the Netherlands. I moved here with my husband four years ago from Melbourne, Australia.

What do you do at Grafana Labs?

I’m a Developer Advocate at k6, but I’m not actually a developer — I’m a performance tester! I guess there’s no such thing as a Performance Tester Advocate? My job is part engineering, part content creation: I write load testing scripts and code, but I also write blog posts, produce videos, and give presentations. I found out I was joining Grafana Labs a few months ago when k6 was acquired, but I was happy to join the team! I’ve met some fun and motivated people at Grafana Labs that I’m eager to collaborate with.

What open source projects do you contribute to?

I’m a budding developer and still learning the ins and outs of social coding, but some things I’ve contributed to recently are a chaos engineering extension for k6 (created by my colleague, the talented Simon Aronsson), an application called Poké API that I’ve added chaos experiments to, and a Python script for pushing my Kindle highlights to my note-taking app of choice, Obsidian.

What are your GitHub and Twitter handles?

GitHub: nicolevanderhoeven

Twitter: n_vanderhoeven

What do you like to do in your free time?

I play Dungeons & Dragons eight hours a week, take a ridiculous amount of notes on everything, and vlog about my (pre-COVID) travels.

A battle map in one of my Dungeons & Dragons games.

What’s your favorite new gadget or tech toy?

My new Rødecaster Pro, an audio mixer that has quickly become an essential part of my audio and video production process. It takes up a huge amount of precious real estate on my desk, but it is absolutely worth it!

My current desk setup, including the Rødecaster Pro peeking out from underneath the laptop in the upper right.

What’s the best part of your job?

I got into Developer Relations and advocacy in general because I realized that, as a performance testing consultant, I was being asked the same questions and solving the same problems. So I started to share my experience online, and I’ve found that significantly more efficient. Why explain something repeatedly to one person at a time, on the spot, when you could post a more considered explanation online and potentially help an unlimited amount of people? A large part of my job is simplifying technical concepts and making it accessible to beginners, and I love being able to help others by creating the content I looked for when I was a beginner, too.

Here’s one of my YouTube videos:

What’s the last thing you binge-watched?

I have binge-watched the entire two campaigns of Critical Role, a livestream of a Dungeons & Dragons game with a bunch of voice actors, which is over 1000 hours of watch time. Totally worth it!

What superpowers do you have?

I speak several languages: Tagalog (my native language), English, Dutch, Spanish, Esperanto, German, and French — in that order and to varying degrees. I am learning and have learned more, but don’t yet feel confident enough to claim to “speak” them! I choose travel destinations based on how interested I am in learning the local language.

What’s your Zoom virtual background and why?

I don’t use virtual backgrounds. The thing I like most about video is that it’s the most honest form of expression: writing, podcasting, and photography can all be edited far more easily. I like that my home office gives clues about who I am and what I like, and I would never want to hide that — mess and all!

The easiest way to sync Kindle highlights to Obsidian: Readwise to Obsidian Redux

Tue, 10 Aug 2021 19:50:40 +0200

Almost exactly six months ago, I wrote a blog post detailing my workflow for processing content. Specifically, I wrote about how I used a python script to take my highlights from Instapaper, Kindle, Twitter, and Airr and automatically put them into Obsidian.

That method worked, and you can read the original article in the link above. However, today, Readwise announced that they now have an official Obsidian community plugin, and I disabled my trusty old python script for good.

You see, although my old script worked, there were quirks– not the least of which was that making changes to it meant programming in Python. I’m not a developer, so while I made do, it was never a painless experience. I also ran into issues with time zones and invalid characters being used in titles. So I’m thrilled to have something significantly better from Readwise itself! Here’s a video from the Readwise team, or read on for instructions on how to get started.

From Readwise on YouTube

Install the Obsidian plugin.

In your Obsidian vault, go to Settings > Community plugins. You’ll need to turn Safe mode off for community plugins if you haven’t already.

Then, click Browse and search for Readwise Official.

Install it, and then enable it.

Connect your Readwise account to the Obsidian plugin

Head back to the Settings menu and scroll down to the “Plugin Options” section, and then click Readwise Official.

Click connect as highlighted in orange above, and a browser window will open, showing the Obsidian Export options. You’ll need to sign into your Readwise account if you haven’t already.

At this point, your Readwise account will have been connected to the Readwise Official plugin, but you won’t have triggered any syncs yet. If you don’t want to change any of the configuration settings, you can then skip ahead to triggering the sync on Obsidian, but let’s run through the options anyway.

Modifying configuration settings

On the Obsidian Exports page, you’ll be presented with a few options.

Group Files in Category Folders

By default, this will be enabled. When enabled, Readwise will put your highlights in different folders depending on its source. For example, Kindle highlights will appear in the folder Books. Disabling this option means all new notes will be created in the directory specified in the Obsidian plugin settings, which we’ll cover later. I left this enabled in case I read a book that has the same name as a blog post about the book, for example.

Use Custom Formatting

Custom formatting is disabled by default, and lets you choose how the final note looks. Here are the things you can change in this section

Page title
Page metadata
Highlights header
Highlight
YAML front matter
Sync notification

I did make some changes in this section just to tweak how I want information to be displayed.

Page metadata

I don’t see the need for Page metadata since this is what I use YAML front matter for, so I removed the matadata section and the document tags (I never use those either). So my Page metadata section looks like this:

{% if image_url -%}
![rw-book-cover]({{image_url}})
{% endif -%}
{% if url -%}
URL: {{url}}
{% endif -%}

Highlight

By default, highlights are entered in bullet-point form. That’s fine when I’m taking notes, but when I’m highlighting someone else’s words, I like to be very careful to use blockquotes. I frequently do block searches in Obsidian, and I do NOT want to accidentally pass off a quote as something I said.

To this end, I added a > before each highlight and two newlines (carriage returns) at the end to delineate each highlight. That way, I can quote just one highlight at a time.

> {{ highlight_text }}{% if highlight_location and highlight_location_url %} ([{{highlight_location}}]({{highlight_location_url}})){% elif highlight_location %} ({{highlight_location}}){% endif %}{% if highlight_note %}
- Note: {{ highlight_note }}{% endif %}

As you can see above, I also removed highlight tags, which I never use.

YAML front matter

Metadata has to be quite specific, so I added parameters I already use for other notes in this section. The TVZ is a tag I use that means “To Verzettelen”, meaning pages that I need to process further.

Title: {{full_title}}
Author: {{author}}
Tags: TVZ, readwise, {{category}}
date: {{date}}

Sync notification

As someone who has worked on a syncing plugin for Readwise to Obsidian, I know that things go wrong sometimes. Enabling this outputs a synchronization log to a note, which has come in handy MANY times in the past. Has that book been synced, or did something go wrong? So I copied what the Readwise team suggested:

- [[{{date|date('Y-m-d')}}]] {{time}} — Synced {{num_highlights}} highlight{{num_highlights|pluralize}} from {{num_books}} document{{num_books|pluralize}}.
{% for book in books %} - {{ book.num_highlights_added}} highlights from {{ book.title }}
{%endfor %}

Select Items to be Exported

In this section, you can choose which books/podcasts/other content in Readwise you want to export. It is disabled by default. If you haven’t synced Readwise to Obsidian before, and you just want to pull in everything, leave it disabled.

In my case, I have already synchronized everything using the plugin I mentioned, and didn’t want to create another few hundred duplicate notes in my vault. So I enabled it and selected a few more recent books.

A weird thing about this Export settings page is that there is no saving– it just automatically saves it. Just a note, because I was scrolling up and down that page looking for an indication that it had been saved before I realized it is done automatically.

Initiate Sync from Obsidian

Head back to Obsidian and to the Readwise Official plugin options page. You should see a “Initiate Sync” button where the “Connect” one was before.

That’s a good sign - that means your Readwise account has been connected, and your configuration settings from the previous step will be fetched.

Before you click the button, note that the Customize base folder option is set to Readwise. This is the directory that all your Readwise notes will be created. So if you enabled Group Files in Category Folders earlier, your Kindle highlights will be saved in Readwise/Books/booktitle.md. You can change the base folder if you prefer, but I just kept the default.

Time to click the button!

You’ll see a few messages popping up on the upper right corner of your screen, and you’ll also see a Synced! message appear next to the button you just clicked.

Verifying results

In your Obsidian vault, search for one of the items you synced. Here’s what my note for Pat Flynn’s Superfans looks like:

Great! It looks exactly how I want it to look.

Set up automatic sync

If you leave it there, the plugin will only sync when you tell it to, so you’d need to go into the plugin options each time. That’s not particularly useful for me, because I want it to just happen.

If you feel similarly, head back into Plugin Options one more time and select your desired frequency in the Configure resync frequency dropdown menu. I chose 1 hour, the minimum. I also enabled the automatic sync when Obsidian opens.

Conclusion

I love this new plugin! It was very much needed, because existing options for accomplishing the same thing were flawed in some way, mainly because they were community-driven– meaning the people who created these workarounds (myself included) were “just” users who had other jobs. I’m grateful that the Readwise team has acknowledged the popularity of Obsidian and taken on the maintenance burden. I’ve experimented with the plugin quite a bit and haven’t found any major issues so far. It is a seamless way to get the best of the content I’ve consumed from various sources into one single source of truth: Obsidian. I highly recommend it to anyone looking to streamline content processing and creation processes.

Note: This post contains an affiliate link for Readwise. Clicking on that link will give you a two-month trial of Readwise, double the usual amount, and they give me a small percentage if you sign up for a plan. Check out My Ethics Statement to see my standards for promotions.

How I use Obsidian for D&D - Player's Edition

Mon, 09 Aug 2021 17:46:51 +0200

I’ve previously written about how I use Obsidian for work, and I’ve also talked about how I use Roam Research for D&D. Shortly after the latter post, however, I moved to Obsidian. That was six months ago, so I thought it was time for an update.

Before the game

World

My World page is the main jumping-off point for all my notes for a specific campaign, and each campaign gets one. It contains:

The main locations in the world
The player characters, and links to their individual pages with information about what my character knows or thinks of them
A link to Open Loops (Quests and Questions and Agenda, as discussed below)
The list of all the sessions I’ve played for the campaign
Campaign-specific house rules that the DM has introduced

Quests and Questions

I like to play high-intelligence characters, so I like to keep track of unanswered questions and other adventure hooks my DM has dropped that the party hasn’t quite gotten around to resolving yet. Some of them are critical quest or story arcs, and others are random tidbits of information that NPCs have mentioned. I organize them by location, and then by quest, so that it’s easy to follow a thread depending on where the party is or how much downtime we have. Here’s an example of that page:

Agenda

While the Quests and Questions page records unfinished business that my character is actively thinking about but may not quite know how to explore yet, my Agenda page is where I write down specific tasks. I loosely follow the Getting Things Done methodology and divide this page into Next Actions and Waiting For. I group tasks by person and/or by location.

I use the Obsidian Completed Area plugin so that hitting a CTRL + ENTER on a task automatically checks it and then moves that task to a separate Completed section at the bottom of the note. This helps me keep the agenda clean and relevant.

During a game

Workspace

One of the best parts about Obsidian is that it allows me to save and load custom workspaces. This is useful because I use Obsidian for work as well as play, so I can set up the two workspaces differently. I also DM one campaign and play in another, and I have a workspace for each of them because I need different information available to me. The workspace features lets me customize my dashboard. Here’s what my workspace looks like for the 4e game that I play in:

At the beginning of a game, I’ll create a new session page in the last pane, enter into it, and then stay in the workspace for the rest of the game. I have my character’s stats sheet handy (which I need to do manually since I haven’t found a D&D Beyond-like service for 4e) and all the powers and abilities I need to refer to.

Session template

I love Obsidian templates! I use templates for things like inserting an HP tracker with all my party’s names, their max HP, healing surges (4e); a defense table to keep track of a creature’s defenses (again, 4e); and campaign-specific session log formats.

At the beginning of the game, I’ll create a new page in the Session List, click into it, and insert a session log template (I have this hotkeyed to CMD+;). Here’s what a fresh one looks like, in Edit mode:

I add metadata to every session so that I can query it later using the excellent Obsidian Dataview plugin, though I do this more often for the campaign I DM rather than the one I play in.

We usually do a recap at the beginning of the game, so I jump into the Recap of the last session section and embed the summary from the previous session using ![](/assets/86_20210717 Day 16 Hunting the Ice Phase Spider#^summary), which pulls in the section marked with ^summary from the last session. That embeds just that portion of the last session in the current session, like this:

This helps me get back up to speed quickly instead of doing the usual “Where were we again?” that takes up time.

HP table

My character is a multiclassed healer, so it’s important for me to know who needs heals or other buffs. I keep track of this separately, and it helps out my DM as well because it’s easy to miss something in the excitement of a battle. I do include the HP table in my workspace, in Preview mode, but here’s what it looks like in Edit mode:

Tables in Markdown can be a pain, but the Advanced Tables plugin goes a long way towards making this easier by adding some nice features like tabbing to the next cell, adding commands for inserting and deleting a column or row, and just automatically formatting a table so that it’s not a jumble of lines. I also heavily use this plugin for my characters stats and skills, both of which I keep in tables.

Keeping track of fights

When a fight breaks out, I insert an intiative template, which is just a list of all the characters in the party:

Initiative!
- Ka'ira
- Maestro
- Xio
- Eraithe
- Keren

This makes it easier to add the initiative value rolled, as well as that of any other baddies we’re fighting. I use the Outliner plugin to be able to easily move people on the list up or down as needed without having to copy or paste. I have the shortcuts set to CMD+SHIFT+UP and CMD+SHIFT+DOWN so I don’t even need to use my mouse.

I also have a template for Defenses. D&D 4th Edition has four different kinds of defenses, and since my character’s powers target a few of them, I keep track of the defenses of each monster. Here’s what that template looks like, in Edit mode:

| Defense | Value |
| --------------------- | ----- |
| [[Reflex Defense]] | |
| [[Fortitude Defense]] | |
| [[Will Defense]] | |
| [[Armor Class]] | |

You’ll notice that I also link to the pages I have for each defense type, in case I need to remind myself which stat a defense is comprised of.

Character powers

Like many, I have migrated from playing games on a literal tabletop to doing it virtually. These days, I play 100% online, using a bunch of different tools. For the most part, though, I still prefer to roll actual dice. There’s just something satisfying about the feel of a die in my hand. Since I switched to Obsidian, though, I have looked into virtual dice. On the off chance I don’t have any dice (ha) or all my dice have betrayed me (much more likely), I use the Dice Roller plugin. It turns

`dice: 1d20+7`

into a random die roll. Here it is when incorporated into my character’s list of powers:

Sometimes, when I’m feeling superstitious about my dice, I’ll switch to using this dice roller “just to be safe”. It has other cool features too, which I’ll cover more in detail in the Dungeon Master’s edition of this article.

Search, linked, unlinked mentions

Frequently, something comes up in a game that I’m sure we’ve heard of before, but I can’t quite remember where or when. Actually, sometimes something comes up that I’m quite sure we’ve never heard of before, but it turns out we have. Luckily, Obsidian is quick to tell me that using the linked/unlinked mentions and the search features.

Here are the mentions in my note for the word “Netheril”:

Unlinked mentions show up too, in case the word came up somewhere and I didn’t think to specifically create a page on it.

On monster pages, mentions remind me when we fought them last, and any tips for fighting them. It can be handy when one of the other players has forgotten something that happened to them, too. If I want to remember how Ka’ira got her festering wound, I can type it into the Obsidian search bar:

and be able to confidently say that Ka’ira got her last festering wound on session 68, in the Blue Crystal Fortress that we now know to be the Twisted Fane, while fighting aberrations.

After a game

I think I might use Obsidian even more between games than during the game, because after the game is when I process everything my character learned.

Session summary

Immediately after the game is when I create a short session summary. I like to do this when the session is still fresh in my mind. Then, I add ^summary at the end of the paragraph to identify it and to be able to pull it into the next session’s recap. Block references and block embeds are some of my most frequently used Obsidian features.

Creating new pages

I then create new pages for any people, places, magical items, or events that we encountered. I like to use a standard frontmatter:

---
type:
location: [Neverwinter]
world: "Forgotten Realms"
campaign: "Order of the Wandering Mind"
date: 2021-06-23
---

Here’s an example of the page for an NPC:

If my DM has given us character art, I include it in the page.

And here’s an example of a page for a location:

For this page, I used a plugin called Obsidian Leaflet, which lets me add maps to my notes! In this case, my DM gave us this map of the structure as a guide, but during combats we didn’t use a map. To keep myself oriented between games, I added red pawprint pins where we had spotted or fought monsters, green thumbtacks to mark where we’d ended sessions, and green markers where we had spotted allies. I also like to add links to monsters or sessions on the pins/markers so that I can jump to those pages quickly if I need to.

History

To keep historical events straight in my head, especially since we tend to discover them out of sequence, I use the Obsidian Timelines plugin, which lets me create pages like this:

by using this code in the timeline page:

```timeline
fr
\```

(without the backtick) and this for every event that I want to include:

---
date: [2021-06-23]
tags: [timeline, fr]
---
<span
 class='ob-timelines'
 data-date='1443-00'
 data-title='Xionerys is born'
 data-class='orange'
 data-img = 'assets/xionerys.jpg'
 data-type='range'
 data-end='1443-00'>
 Xionerys is born in Calimshan.
</span>

Using this method, I’ve been able to call out discrepancies in what two different NPCs told us and figure out that one was lying– or at least misinformed.

Replace all

When I inevitably get a fantasy name wrong, having pages as flat .md files is great because I can do a find and replace on ALL mentions of a certain NPC name/place I’ve misspelled (those fantasy names!). I do this with my usual code/text editor, VSCode, just because I usually already have it up, but you can do it using terminal commands. If it’s just the name of a page, updating the spelling in Obsidian changes it everywhere else it’s linked to as well, which is useful.

Updating open loops

To finish processing the last session, I update Open Loops: I add new points or update existing points in Quests and Questions, and then add tasks to Agenda that I want to do in the next session. I find that it’s best to do it after a game because it’s a week, sometimes more, between games and there’s a high likelihood I’ll have forgotten what I want to do by then!

Publish

I also publish my D&D notes, just for transparency. My DM has joked that he looks in there sometimes when he’s forgotten something. Even if he doesn’t, I enjoy having my notes so easily accessible. I often navigate there myself when I have a spare moment, remembering old sessions and thinking about some loose threads. If you’re interested, here they are! Just be warned that the plugins won’t work in the web version, so sometimes you’ll come across bits of code that didn’t render. I also migrated these notes from Roam Research, and haven’t gone back and updated all the older notes.

Conclusion

Since I’ve switched to Obsidian, it’s been an indispensable part of my workflow for D&D. Without it, I’d be just another murder hobo with no notes. And no DM wants that!

Gear I Use

Mon, 12 Jul 2021 21:18:55 +0200

Gear I Use

One of my favorite obsessions is watching or reading about the gear other people use. I’ve watched almost every “What’s in my tech bag” video on YouTube, it feels like, and I’m always prowling blog posts that detail what someone is using. In case there’s anybody out there that is as interested in these things as I am, here’s an overview of the gear that I use to make videos, write both blog posts and code, and generally mess around on the internet.

Edit: After a reader suggested I provide links to these items, I’ve included them below, and some are affiliate links. You are under no obligation to click these links, but if you do, I get a tiny percentage of the sale. Thanks!

Sections

There’s a lot here! I’ve broken it down into sections, so you can skip to the section you’re interested in.

Daily hardware
Home Office 1
Home Office 2
Everyday Carry
Remote Working Tech Bag
Remote Work Setup for a month

Daily hardware

(back to top)

Here are the main devices I use every day.

Computers

Macbook Air 15" (2023): 24 GB RAM, 2TB SSD, M2 chip - this is my main workhorse
Macbook Pro 13" (2020): 2,3 GHz Quad-Core Intel Core i7, 32 GB RAM - secondary laptop
Mac Mini (later version here): M1 chip, 16GB RAM, 1 TB SSD - desktop, always on

Mobiles

Samsung Galaxy Z Fold 5: 512GB - my main mobile, always with me
Apple iPhone 15: Pro Max - secondary mobile, always with me
Samsung Galaxy Note 10+: extra mobile

Tablets

Apple iPad Pro 11" (2020) tablet - still works great!

Readers

Amazon Kindle Scribe: my couch reader, for nonfiction and PDFs
Amazon Kindle Oasis: my bed reader, for fiction
Kobo Sage: for non-Amazon ebooks

Wearables

Oura: health and sleep tracking ring
Samsung Galaxy Watch 6: smartwatch
American Express Prada wearable (not available for purchase separately): for contactless payments

Audio

Apple AirPods Pro: second generation, for Apple products
Bose QuietComfort Earbuds II: for non-Apple products

Home Office 1 (the Netherlands)

(back to top)

Video production gear

Not on desk

Elgato green screen (long, black metal object on the left side of the photo, on the floor)
Canon 6D Mark II with a Canon 16-35mm f/2.8L (upper left, on tripod with teleprompter)
Manfrotto 290 tripod
TeleprompterPAD iLight Pro 10"
Neewer 5-in-1 reflective reflector set (gold, silver, white, black, and transparent)
Neewer two-way rotating aluminum light stand
Manfrotto universal smartphone clamp (mounted on light stand above desk)
Neewer Camera Monopod with rotating column for panoramic shooting
2 x Nanlite PavoTube 6C II tubes on the shelves in the background

On desk

Aputure AL-MC portable light (attached magnetically to the back of my external monitor)
Blue Yeti microphone - (pictured, but retired for the next item)
Røde PodMic microphone - my new main mic
Elgato Key Light
Canon M50 (between my laptop and monitor screens, used as my main webcam)
Elgato multi-mount (holding up the M50)
Blackmagic Atem Mini video switcher
Røde Wireless Go wireless microphone (secondary microphone when I need to move around)
Rødecaster Pro for crispy audio on calls and videos
Samsung Portable SSD T5, 1TB (always connected to my laptop) the drive I edit off
Macbook Pro 13" (2020), 2,3 GHz Quad-Core Intel Core i7, 32 GB RAM - main laptop

Not pictured but in frequent use

Sony A6600 with Sigma 16mm f1.4 lens (main shooter, used to take this shot)
Røde Wireless Go II two-person microphones (used as on-board camera mics)
SwitchPod tripod
Peak Design Travel tripod: Pricy, but the only full tripod I use these days. Excellent for travel!

Other tech

Synology DS920+ 4 Bay NAS with 4 x WD RED 4TB drives (just outside the photo)
Amazon Echo Dot (on the bookshelf)
Google Wifi wireless extender (just outside photo)
2 x Nintendo Switch (on bookshelf; one Animal Crossing edition)
Nintendo Lite Zacien and Zamazenta edition (on bookshelf)
Amazon Kindle Oasis (bookshelf)
Amazon Kindle Paperwhite (bookshelf)
Kobo Aura One (bookshelf)
Risa Electric Solid Tenor Ukulele, black (bookshelf)

Non-tech gear

Wyrmwood Gaming Cherry magnetic dice tower bundle including dice tower, personal dice tray, and dice vault
Kimi Rainbow Acrylic kalimba
Visconti Homo Sapiens Lava Midi, steel age fountain pen (the only fountain pen of many that I use regularly)

Non-video gear on my desk

I already listed the video-related gear on my desk in the first section above, but here’s everything else.

Samsung Galaxy S10 (secondary phone, primarily for timelapses and overhead videos)
Samsung Galaxy Wireless Buds, rose
Samsung Galaxy Note 10+
Apple Macbook Air 11" (2011, used as my presentation laptop)
Apple Magic Trackpad
Apple iPad Pro 11" (2020) tablet
Niz Plum Atom 68 electro-capacitive keyboard with Pikachu keycap
Logitech MX Master 3 the best mouse I’ve used!
Bose NC700 wireless noise-cancelling headphones
Oura health and sleep tracking ring

Home Office 2 (Portugal)

(back to top)

I live between two countries: the Netherlands and Portugal. Since I spend a few months in each place, that also means I need a place to work in each apartment! My home office in Portugal is larger but definitely less well-equipped. It’s a little embarrassing, but I thought it would be interesting to show a more realistic, in-progress setup.

Why yes, I do indeed have clothes hanging on a rack behind my desk to try and soak up sound.

Desk

Elgato Key Light Air x 2: I bought two of the smaller key lights than what I have in the Netherlands, because this room is larger and requires more light
Hidrate Spark Pro: bluetooth water bottle
Samsung 32" 4K UHD monitor: a bit overkill on the size front, but nice
Apple iPad Pro 11": This is from 2020, but still going strong
Elgato Stream Deck MK.2: 15 programmable keys for shortcuts and macros
Keychron K6: hotswappable, RGB mechanical keyboard
Logitech MX Anywhere 3: my travel mouse
Blackmagic Atem Mini: my video switcher
Røde Rodecaster Pro II: my audio mixer
Røde PSA1: the best mic arm
Røde Podmic: great XLR microphone
Samsung Galaxy Note 10+: my secondary phone
Baseus 17-in-1 USB-C Docking Station: for connecting my laptop to all the peripherals
Nintendo Switch Lite in coral: my travel gaming console
Fully monitor and laptop arm: arms to hold my monitor and laptop
Macbook Pro 13" (2020): same laptop that I bring around with me
Sony A6600 with Sigma 16mm f1.4 lens (main camera that I take with me from the Netherlands)
SwitchPod DSLR Tripod: best travel tabletop tripod

Free-standing

Fully Jarvis standing desk: bamboo-topped motorized desk
Neewer 77 inch tripod: basic tripod
Neewer Photo Studio Backdrop Support System: I hung an acoustic curtain and some clothes on this to help with echo for now

Not pictured

Samsung Galaxy Z Fold 3: used to take this photo

My everyday carry

(back to top)

What about what I carry every day?

Samsung Galaxy Active 2: my smartwatch
d20 spinner ring: a ring that acts like a 20-sided die for roleplaying games on the go
Samsung Galaxy Z Fold 3
Aer Day Sling 3 X-Pac: my sling bag of choice
Nitecore Tini 2 USB C flashlight: tiny USB-C rechargeable flashlight
Nite-Ize locking S-biners: for a little extra security for zippers
Orbitkey holder: to hold my keys
Nano bag 2.0: compactable bag
Kable Card: all essential cables and adapters, a light, a mobile stand, and a memory card reader all in one card
Aurtech bluetooth keyboard: portable bluetooth keyboard
Samsung Galaxy S Pen Fold Edition: stylus for my mobile
Sony WF 1000 XM3: wireless noise-cancelling earbuds
Spigen Arc Powerstation 20W: small GaN charger
Secrid mini-wallet: my minimalist wallet
Dispel Dice Lilac Song: a full-sized d20 I carry around
Sprawlopolis: small cooperative card game
LIE: fun party game for 2-6 players and is the size of a pack of gum
Victorinox Jetsetter: small, travel-friendly multitool
Mighty Hanks: microfiber on one side, cotton on the other
Burts Bees: lip balm
Small red Sharpie: mini marker
Joby Gorillapod GripTight Smartphone Micro Tripod: tripod for my mobile
Samsung Galaxy Note 10+: secondary mobile
Topo designs micro pouch: small pouch
Tile Mate Pro: bluetooth tracker
Trezor Model T: hardware wallet and security key

My Remote Working Tech Bag

(back to top)

If I need to go somewhere to work, my EDC bag won’t cut it! Here’s what I bring with me to work and play remotely:

Aer x Ministry of Supply Lunar Pack
Platypus soft bottle. Similar here.
PacSafe Retractasafe 250
iWalk 10,000 mAH USB C power bank
Critical Role Nott the Brave face mask (2 pack)
Apple iPad Pro 11"
Apple Pencil
The Crew: Mission Deep Sea (card game)
Kindle 2022
Popsockets PopGrip
Nintendo Switch Lite
Aer Day Sling 3 X-Pac
SwitchPod
Peak Design Tech Pouch
Keychron K3 travel pouch
Nuphy Air 60 keyboard
Roost laptop stand
Nedrelow Magic Sleeve
URWizards soft wool felt dice tray
Shure Aonic 215 wired sound isolating earbuds
Logitech MX Anywhere 3: travel mouse
2 x Lexon Mino X bluetooth speakers: awesome USB C-rechargeable bluetooth speakers that can be paired together
Logitech Spotlight Wireless Presentation Remote. Similar here.
2.5mm to 3.5mm audio adapter. Similar here.
Yubico Yubikey 5C security key
Yubico Yubikey 5C nano security key (always plugged in to laptop)
8Bitdo Zero 2 bluetooth Switch mini controller
Anker USB C cables
USB C adapters. Similar here.
Røde Wireless Go wireless microphone system
UGreen GaN 100W USB C charger
Hyper Drive USB hub. Updated version.

My Remote Work Setup for a month

(back to top)

Here’s what I bring with me when I need to be able to work while travelling continuously for a month. This setup lets me create videos, run livestreams, do programming work, and do presentations at conferences with minimal bulk and maximal functionality.

Platypus soft bottle. Similar here.
Samsung Galaxy Active 2: my smartwatch
Le Chivée USB C portable wireless charger for Samsung Galaxy Active 2 watch. Similar here.
Bose QuietComfort Earbuds II.
Apple iPad Pro 11".
Apple Pencil
Nedrelow Magic Sleeve
3 x Dispel Dice d20s.
Nuphy Air 60 keyboard
Logitech Spotlight Wireless Presentation Remote. Similar here.
Apple Magic Trackpad.
2 x Lexon Mino X bluetooth speakers: awesome USB C-rechargeable bluetooth speakers that can be paired together
Røde Wireless Go wireless microphone system
Roost laptop stand
Macbook Pro 13" (2020), 2,3 GHz Quad-Core Intel Core i7, 32 GB RAM - main laptop
Yubico Yubikey 5C nano security key (always plugged in to laptop)
Sony A6600 with Sigma 16mm f1.4 lens (main shooter)
Peak Design Travel tripod: Pricy, but the only full tripod I use these days. Excellent for travel!
Elgato Cam Link 4K.
USB C adapters. Similar here.
Peak Design Tech Pouch
Anker USB C cables
Lume Cube Panel Mini LED light.
Joby Gorillapod GripTight Smartphone Micro Tripod: tripod for Lume Cube
Samsung X5 2TB SSD. Better here - T7.
Samsung Galaxy Z Fold 3
(not pictured, but since added) Logitech POP Mouse: travel mouse, even though I hate that it requires an AA battery.

My WFH desk and office setup

Mon, 12 Jul 2021 21:18:55 +0200

One of my favorite obsessions is watching or reading about the gear other people use. I’ve watched almost every “What’s in my tech bag” video on YouTube, it feels like, and I’m always prowling blog posts that detail what someone is using. In case there’s anybody out there that is as interested in these things as I am, here’s an overview of what my home office looks like and the gear that I use to make videos, write both blog posts and code, and generally mess around on the internet.

I’m only mentioning larger items of note that I use frequently.

Video production gear

Not on desk

Elgato green screen (long, black metal object on the left side of the photo, on the floor)
Canon 6D Mark II with a Canon 16-35mm f/2.8L (upper left, on tripod with teleprompter)
Manfrotto 290 tripod
TeleprompterPAD iLight Pro 10"
Neewer 5-in-1 reflective reflector set (gold, silver, white, black, and transparent)
Neewer two-way rotating aluminum light stand
Manfrotto universal smartphone clamp (mounted on light stand above desk)
Neewer Camera Monopod with rotating column for panoramic shooting
2 x Nanlite PavoTube 6C II tubes on the shelves in the background

On desk

Aputure AL-MC portable light (attached magnetically to the back of my external monitor)
Blue Yeti microphone
Elgato Key Light
Canon M50 (between my laptop and monitor screens, used as my main webcam)
Elgato multi-mount (holding up the M50)
Blackmagic Atem Mini video switcher
Røde Wireless Go wireless microphone (main microphone, even for voice calls)
Samsung Portable SSD T5, 1TB (always connected to my laptop) the drive I edit off
Macbook Pro 13" (2020), 2,3 GHz Quad-Core Intel Core i7, 32 GB RAM - main laptop

Not pictured but in frequent use

Sony A6600 with Sigma 16mm f1.4 lens (main shooter, used to take this shot)
Røde Wireless Go II two-person microphones (used as on-board camera mics)
SwitchPod tripod

Other tech

Synology DS920+ 4 Bay NAS with 4 x WD RED 4TB drives (just outside the photo)
Amazon Echo Dot (on the bookshelf)
Google Wifi wireless extender (just outside photo)
2 x Nintendo Switch (on bookshelf; one Animal Crossing edition)
Nintendo Lite Zacien and Zamazenta edition (on bookshelf)
Amazon Kindle Oasis (bookshelf)
Amazon Kindle Paperwhite (bookshelf)
Kobo Aura One (bookshelf)
Risa Electric Solid Tenor Ukulele, black (bookshelf)

Non-tech gear

Wyrmwood Gaming Cherry magnetic dice tower bundle including dice tower, personal dice tray, and dice vault
Kimi Rainbow Acrylic kalimba
Visconti Homo Sapiens Lava Midi, steel age fountain pen (the only fountain pen of many that I use regularly)

Non-video gear on my desk

I already listed the video-related gear on my desk in the first section above, but here’s everything else.

Samsung Galaxy Note 8 (secondary phone, primarily for timelapses and overhead videos)
Samsung Galaxy Wireless Buds, rose
Samsung Galaxy Note 10+ (primary phone)
Apple Macbook Air 11" (2011, used as my presentation laptop)
Apple Magic Trackpad
Apple iPad Pro 11" (2020) tablet
Niz Plum Atom 68 electro-capacitive keyboard with Pikachu keycap
Logitech MX Master 3 the best mouse I’ve used!
Bose NC700 wireless noise-cancelling headphones
Oura health and sleep tracking ring

Downloading files uploaded to Roam Research for use with Obsidian

Wed, 02 Jun 2021 00:27:59 +0200

If you’ve recently transitioned from Roam Research to Obsidian, you might run into a few issues, even after you’ve run the “Markdown format importer” core plugin. One of the most annoying ones that I ran into was that files uploaded to Roam (images, pdfs, etc.) are automatically uploaded to Google Firebase.

Roam assets are linked to in the Markdown files like this:

![](https://firebasestorage.googleapis.com/v0/b/firescript-471a1.appspot.com/o/imgs%2Fapp%2FMyNotes%2FxRFXx7l-ZK.png?alt=media&token=1e03b680-5472-33dc-9b06-a127301af37f)

Now, since Roam relies on security by obscurity on this front, the links to your assets will still be accessible in your Obsidian vault, and they’ll still render correctly. However, I still think it’s worth downloading the files for three reasons:

If you don’t intend to use Roam anymore and cancel your subscription, they will likely delete those assets at some point, leaving you with a lot of broken links in Obsidian.
Part of Obsidian’s appeal is that you own your assets. Not so if they’re on someone else’s server.
You can’t access Firebase files without an internet connection.

So, instead, you want to download the contents of that link and replace it in your notes with something like this:

![](assets/newfile.png)

That way, you’ll have full control over all your assets, and you won’t be dependent on those assets being stored on Firebase by Roam.

To fix this, I wrote a Python script that will:

Go through every file and every folder within a specified root vault directory
Look for links to Google Firebase
Download all assets from Firebase to a local folder (/assets by default), using the timestamp and index number as a filename
Replace the Firebase links in your vault with links to the newly downloaded files, accounting for PDFs, whose links are formatted differently in Roam as {{ pdf: ... }}

Before you run this script, I highly recommend backing up your vault or trying this out on a dummy vault.

After downloading and saving the script, run python3 downloadfirebase.py on your terminal.

This worked really well for me, and I can breathe a little more easily now that my vault is self-contained.

End-to-End Guide for Load Testing

Mon, 24 May 2021 10:48:55 +0200

Originally published on the Abstracta blog.

If you have a public-facing web application, the odds are you should probably be thinking about load testing. Any tester can (and should) add load testing to their repertoire.

Functional tests are often carried out with a single user’s experience of your application in mind, but what happens when hundreds or thousands of users are on your site at the same time? Load testing helps prepare an application for this eventuality by simulating that situation before the application goes live. That way, you have time to fix any issues you find, and there are no nasty surprises in production.

In this guide, I’m going to walk you through the basics of the load testing process, which fits in closely with traditional functional testing phases as well:

Planning: Why are we testing? What are we testing?
Writing test scripts: How are we testing?
Execution: Running test scripts against the application.
Analysis and reporting: What happened, and what do we need to fix?

Planning for load testing

During the planning phase, the team gets together and determines what load testing will involve, including the objectives, scope, resources needed, and test scenarios. The deliverable for this phase is a test plan, but that can be as formal or informal as the team requires.

Planning for any testing is a team activity, and you’ll want to get input from all the stakeholders – that’s anyone who is involved in or might be affected by the testing. The first thing you need to clarify is the reason for load testing.

Requirements

Why are we doing load testing? seems like a very simple question, but you may be surprised by the varying answers from your stakeholders. Their reasons may require different test scenarios, so it’s a good idea to agree on the collective objectives of this load testing cycle so that expectations are set accordingly. A requirement is a concise statement of how an application should behave in a given situation.

Phrasing objectives as nonfunctional requirements (NFRs) is a good way to formalize them. An NFR defines the desired state of the application and can includes performance thresholds that the team decides on. For example: Logging in as a registered user must take no more than 3 seconds (90th percentile response time) is a requirement based on response time. Good requirements are SMART (Specific, Measurable, Agreed Upon, Realistic, and Timely).

Here are some other things that should be decided on during this phase:

Scope. What are we testing? Deciding on a shared objective for load testing lets you narrow down what areas of the application need to be targeted. Make sure everyone is clear on what won’t be tested.

Photo by Daria Nepriakhina on Unsplash

Entry Criteria. What do you need to begin? This is the time to start thinking about resources that are required for testing, including the number of people who need to be involved, the time it takes to carry out the testing, and environment requirements (including test data). You don’t always need a separate testing environment to do load testing, but if you do, that should be discussed up front so that the DevOps team knows you can’t begin without an environment.

Workload Modeling. A workload model is a set of characteristics that describe the load you’re attempting to simulate. If you want your load testing to simulate registered users accessing your site, what does that entail? This might require you to look through your site analytics to determine which pages are visited most often and which application features are most commonly used, or arrive at an estimate if analytics are not available. Are most users visiting from their laptops at lunch on work days, or do they use your site from mobile devices after work? Are most of them located in the US or in Europe? You’ll want your scripts to match real load as much as possible, so take note of key characteristics that might have an effect on the traffic your application experiences.

Server monitoring. Which application components will be involved? You want to have a plan in place to have monitoring, alerts, and logs set up on every server that will be tested by your scripts. If your load testing tool shows performance degradation, you want to be able to have information to look through on the relevant servers to troubleshoot that issue. Keep in mind that any sort of monitoring adds some resource overhead, so decide as a team what information you absolutely need to track.

Test Scenarios. Which situations do you want to recreate? A test scenario objectively describes the situation to be simulated. Do you want to recreate the load profile of a typical day, or one from a very high-traffic period, such as during a sale? Do you want to recreate a production outage? You can use analytics data to come up with parameters for your each test scenario such as duration, number of concurrent users or requests, ramp up period (how quickly the application went from 0 users to your highest number of users), or how “spiky” the load needs to be (how the number of requests per second fluctuates throughout the test).

Everything discussed in this phase should be written in some sort of a load testing plan, which could be a formal, multi-page document, an epic with many user stories, or a bunch of Trello cards. Use the tools your team already uses. The format isn’t important; formalizing the outcome of the discussions is what counts. The test plan makes sure everyone has the same intent with regard to the testing.

Writing load testing scripts

Now it’s time to start writing scripts! You’ve already front-loaded a lot of the important decisions to the planning phase, so all that’s left to do in this step is to implement the plan in the form of load testing scripts.

Tool selection

There are many load testing tools available on the market, and the reality is that you don’t always get to be the one to choose what you use. Often, companies have already made agreements with tool vendors for licenses before you begin your load testing, and it would be financially infeasible to switch to another tool. However, if you’re lucky enough that you can choose the load testing tool, here are a few considerations for assessing different tools:

Resources and costs

How much does it cost? Does the pricing model allow for scaling usage up and down as necessary?
Should you build or buy an end-to-end load testing solution?
What technical skills or experience with tools does your team have? Who will be doing the testing?

Features

Does the load testing tool support the protocols and technologies used by the application?
How easy would it be for the team to learn this tool and/or scripting language?
Does the tool make it easy for multiple people to collaborate on the same script?
Does it integrate with other tools in the testing stack (for test management, test result analysis, server monitoring, team communication)?
Could you add this tool to your CI/CD pipeline?
Does the tool have an easy way of setting thresholds or SLOs for performance?
Are the test results displayed in usable reports, or does the tool integrate with existing data visualization tools you are using?

Support

If something goes wrong, does customer support cost extra?
Is there ample documentation available online for getting started without help?
Is there a community of users of the tool whose experiences you could learn from?

Scaling up load testing

How many virtual users or requests per second can a single load generator simulate using the tool?
How many load generators would you need if you used this tool?
Can the tool handle the amount of test data required by your tests?
Can you execute tests from different geographical locations, depending on your the location of your users?
Does the tool allow you to generate load from the cloud?

These questions are not exhaustive, but you can use them as a starting point for your comparisons of load testing tools.

For this guide, I’m going to outline a basic script in k6 for a few reasons:

It’s open-source, so anyone reading this can download the tool and the script and try it out for themselves;
It’s written in JavaScript, which makes it easy to learn for front-end developers or testers who have used tools like Cypress, Puppeteer, or Playwright.
It’s capable of generating tens of thousands of users on a single machine, so it’s cost-effective even with load ramped up.

However, the principles I discuss will be applicable to other tools. I highly recommend you conduct your own proof of concepts with several tools to come up with a tool that fits your particular situation, test scenarios, and company.

Installation and setup

If you’re on a Mac, the easiest way to install k6 is to use Brew. Run brew install k6 from your terminal, and you’ll be ready to go in a few seconds.

If you’re on a different operating system, check here for installation instructions that apply to you.

A basic k6 test script

A barebones k6 test script looks like this:

import http from 'k6/http';

export default function () {
 let response = http.get('https://test.k6.io');
}

This script sends an HTTP GET request for the URL https://test.k6.io, which is a simple test site. If you copy that code to your favorite text editor and save it as test.js, you can then run it with the command k6 run test.js.

You’ll get something like this in response:

A sample run of a simple local test in k6

The output above means that you’ve executed 1 iteration of your script with 1 virtual user from your local machine. The test run consisted of 1 HTTP request, which had a response time of 128.2 ms (http_req_duration).

It might be tempting to start scaling up the test at this point, increasing the number of virtual users and the duration of the script. But there’s one problem with the script as it is: it’s not realistic.

Making a test script realistic

The purpose of a load testing script is to simulate a real user who accesses your site. In most cases, that involves more than the simple GET request from the script above. Load testing scripts don’t have to mimic a real user exactly, but they do have to be close enough so as to apply the same amount, shape, and distribution of the load on your application server. If your load testing scripts aren’t realistic enough, you may get false positives during testing and discover too late that there is a performance issue in an application component that your scripts didn’t test.

Here are some ways to make your scripts more realistic:

Add static resources

If a page has static resources like images and scripts that are retrieved by a browser automatically, make sure your script retrieves those too, as they will have an impact on the response times you get. Consider excluding static resources fetched from third-party providers unless you’ve gotten their permission to run a load test on their servers.

Review your load testing tool’s default cache and cookie management settings and change them, if necessary, to match the scenario you want to test. First-time visitors to a site won’t have resources cached locally, but repeat visitors may be retrieving resources from the cache.

Add think time

Real users don’t usually fetch the same URL over and over again without stopping. They navigate to your site, then spend some time reading the contents of the homepage before they interact with the page again. Think time is the amount of time that a user spends “thinking”— that is, the delay between requests. Adding a think time that varies between 1 and 5 seconds, for example, prevents the application server from getting artificially bombarded by requests in a way that wouldn’t occur in production.

Using different user paths

A user path is the flow of how a user interacts with the application. What pages do they visit? What information do they type in? So far, you’re only testing one request, but perhaps your users may be performing other actions. The workload model that you put together in the Planning phase will come into play here. Other actions like logging in or paying for items in a cart exercise different parts of an application than browsing does, so the load testing script should reflect the most common user flows through your system. In k6, this is done by adding scenarios, but other tools may call them thread groups or execution groups.

Adding test data

In the same way that users don’t (typically) search for the same keyword over and over again and don’t all log into the same test account, a script shouldn’t use the same data repeatedly. This behavior could cause responses to be cached, leading to misleadingly quick response times, or increased error rates, in the case of multiple virtual users attempting to log into the same account. To prevent that, modify your script to take data from a CSV file instead, letting you add some natural variation.

Setting test parameters

Test parameters are the main characteristics of the test and include the number of users, ramp up, duration, and ramp down. They control the shape of the user load, which you can see when you plot the number of virtual users over time during the test:

The number of virtual users over time during a load test, as graphed in k6

Ramp up is a period of time during which virtual users are gradually added, usually at the start of the test, and ramp down is when they are gradually stopped. For most applications, ramps are more realistic than all users clicking a link at exactly the same time.

Setting failure criteria

Part of simulating real users is determining what constitutes failures in user experience. You can incorporate failure criteria into scripts by adding response verifications, checks, and thresholds (such as those for response time or error rate). Building these into the script is a good practice to get into, because it defines the limits of acceptable performance from the beginning.

The suggestions in this section are for general use, but there are many situations where taking the opposite approach would be more realistic. Spend some time thinking about what is “realistic” for your application and test scenario.

For your reference, here’s a k6 script that started from the basic script I showed earlier, but has been modified to be more realistic. That link includes a CSV file with login credentials.

Executing load tests

In the course of writing the test script, you probably would have already executed small verification tests, called shakeout tests. It’s a good idea to start with these shakeout tests on a new environment to make sure that both your test artifacts (your script and any other data files) and the environment are ready for full testing.

Running the test

If you’ve been following along with the k6 script so far, all you need to do to run a load test is to execute the following command:

k6 run test.js

This command will run the k6 script according to the test parameters set within it. The test will be executed locally. This is best used for shakeout tests, or if you prefer to run your load test on your own infrastructure.

Realism in test execution

One way can make executing load tests realistic is by matching the location of load generators to the physical location of your users. When you run a load test on machines in your data center, in the same network as your application servers, you will likely see response times that are significantly faster than your users would see. These false positives can look good on the report, but they don’t help you measure real user experience.

Running on the cloud is an easy way to make tests more realistic, especially if most of your end users are external to your organization.

If you’re following along with the k6 script, you can sign up for an account with k6 Cloud (the first 50 tests are free). Since k6 itself is open source, you can also run it on your own cloud infrastructure, but using k6 Cloud is the most convenient way to get started.

Once you have an account, copy your API token from here. Then, run this command in your terminal to give your local k6 installation permission to run on your account:

k6 login cloud --token <YOUR_K6_CLOUD_API_TOKEN>

Once you are successfully authenticated, you can continue to run tests locally using k6 run, but you now have the option of running on the cloud as well:

k6 cloud test.js

k6 will then begin to execute your load test, and you’ll see a screen like this:

Note that the execution mode is now cloud, to remind you where you’re running the test.

By default, your test will run on the k6 AWS account, in the Ashburn region. However, you can override this and specify other regions, including percentages of your virtual user to run on each, using cloud execution options.

Analyzing load testing results and reporting

Although I’m listing analysis as a different step in the process from execution, the two often (and should) overlap. The output from k6 in the screenshot above contains a link to the dashboard on k6 Cloud, so that you can see the results of the test in real time, as the test is running. Monitoring the test in real time lets you immediately spot issues as they arise, and, if necessary, stop the test to fix them. If the test is being executed by multiple load generators, having a consolidated dashboard is helpful in giving you an overview of the entire test.

![An example of the results after a load test in k6 Cloud](/assets/Finished k6 run.png) An example of the results after a load test in k6 Cloud

This is also a good time to monitor the health of the application servers and the load generator servers, which you set up monitoring tools for during the planning phase.

A load test is only useful when it’s providing information about the application. If errors occur in large quantities, or if the response time makes it clear that a certain component has failed to handle the load, don’t be afraid to stop the test. Your objective is to get the information you need to suggest performance improvements and solutions for any bottlenecks you find.

Often, you’ll run into issues you can’t solve or even identify during a test. That’s why it’s useful to explore the data that you have after a test. k6 lets you create shareable links to dashboards so that you can involve relevant engineering teams who may be able to help you understand what happened and why. Here’s an example of what that dashboard looks like.

Some projects require formal documentation of test execution by way of a Test Summary Report, but in more agile teams, it may be enough to share the test results dashboard and some supplementary information about what type of tests were run and what problems were encountered.

The most important part of load testing isn’t the execution itself; it’s what you do with the results. Communicating results clearly to stakeholders will make it more likely that performance issues are fixed.

What’s next

The phases I’ve outlined here are a good guide for anyone starting with load testing, but I always recommend that you take what works for your situation and leave the rest. In reality, the phases overlap with each other considerably, especially in more modern engineering cycles.

Another activity I’ve not covered here, but one that I recommend, is adding load testing to your CI/CD pipeline. This continuous testing approach involves automating load testing so that it occurs in regularly, in parallel with other development and testing activities. Continuous testing is a great way to get more value out of load testing, and it ensures that performance is front-of-mind for everyone on the application team.

Happy testing!

How I Use Obsidian at Work

Tue, 18 May 2021 22:33:18 +0200

Note-taking and personal knowledge management get talked about in academic circles, by both students and teachers. But there’s not much said about its use outside of the academe.

I’ll admit, due to my job description, I may be in a better position than most to take advantage of note-taking in a professional context. My background is in software performance testing. My job for several years has been to make sure that applications are prepared to withstand traffic from many users accessing it, usually by writing code to simulate that traffic before it’s released to real users. Currently, though, I’m a Developer Advocate at k6.io, which means a significant part of my job now is sharing with people what I’ve learned.

I consider myself a knowledge worker: a significant part of my job depends on how quickly I learn new technologies and how clearly I can impart what I’ve learned to others. I’d argue, though, that most people who work in a field as fast-moving as tech are knowledge workers too. It’s naive to think we won’t one day forget some skills we’ve learned, even if we were using them daily at one point. Yet nobody talks about note-taking.

Recently, my note-taking and personal knowledge management tool of choice has been Obsidian.md. I have no affiliation with the Obsidian team, and in fact I’m an investor of a competing product. Here are some ways I use Obsidian at work that might help you understand why Obsidian is my daily driver.

My setup

I have multiple Obsidian vaults, and each of them is a version-controlled Git repository, hosted on GitHub. I keep my main Obsidian vault in a Dropbox for extra version control and as a backup. Just for overkill, I also use Obsidian Sync as another backup and as the easiest way to get access to my Obsidian vault on my iPad Pro and my Android mobile.

I use Obsidian Publish to make some of my notes publicly available on a custom domain directly from my vault, as an experiment in learning in public that’s gone pretty well so far.

This setup isn’t free. All up, I’m giving Obsidian about US $16 a month. However, I want to point out that Obsidian itself is free for personal use, and if mobile access, cloud sync, and publishing your notes online aren’t important to you, you can really go a long way with using it for free.

Processing information

Remaining relevant in tech pretty much requires that you learn a lot, learn quickly, and enjoy the hell out of it. Luckily, Obsidian is pretty great at this.

Consume and collect

I consume a lot of content, and in different formats: articles, books, videos, presentations, podcasts are all great sources of free education. I initially capture my notes on them in different ways, but they all lead back to using Readwise, which I use as a collector and sorter of all these notes. Then, I import data from Readwise to Obsidian using a process I describe in more detail here.

For now, suffice it to say that all these notes and highlights end up in Obsidian for processing. Here’s an example of my notes during a presentation (called In the kitchen - a sprinkle of fire and chaos by Ana Medina, about chaos engineering).

When trying to learn something, many people stop at this phase. They’ve read the thing, they’ve highlighted passages from the thing, and now they think they’ve learned the thing. That hasn’t been my experience, because I forget many things I consume shallowly.

Process iteratively

Then, I process notes iteratively, first restating things into my own words and then creating my own notes about concepts I’ve learned. Each time I do that, I try to identify key concepts and create backlinks for them, which in Obsidian just means typing [[Chaos engineering]], with a topic between double brackets. If I click on that new link, it creates a page for it. Over time, I end up with a page like this one on Chaos Engineering that is a consolidation of what I’ve learned.

Interestingly, if I go to the Chaos Engineering page, I can also see all the instances I’ve linked to it from other notes:

What if I didn’t have the foresight to link to Chaos Engineering from an article I read? Obsidian also tracks unlinked mentions as well. I can see all the times I’ve used the words “chaos engineering” in other notes, even if I haven’t created formal links using [[Chaos engineering]].

Over time, things I’m interested in start to emerge this way, and topics grow deeper and more detailed, with branches of pages spinning off from it.

Situate and solidify

Then, I try to think about where this new piece of knowledge fits into the hierarchy of things I’m interested in. This is less about trying to classify the world and more about putting it somewhere on my personal map of interests so that I’m more likely to use that knowledge later. For me, chaos engineering fit into my page on Nonfunctional testing. Doing so helps me pit it against other things I’ve classified as forms of nonfunctional testing and think critically about whether it belongs there, or whether other things I already know contradict it in some way. Sometimes I’ll make notes about my reasoning for classification, such as this note on why I think chaos engineering is a testing discipline.

None of this is set in stone. I’m always changing notes around as I learn more things that change my mind. For example, I’m considering merging my nonfunctional testing and functional testing pages after a conversation where someone questioned whether it wasn’t a misnomer to call application performance “nonfunctional”. The point is to create a living, breathing second brain that helps me make connections between concepts and remember what I’ve learned.

Obsidian’s built-in graph view illustrates this point nicely:

This is my main Obsidian vault as of the time of this writing - all 5443 notes of it. The graph view plots every note and the connections between notes, so I’ve ended up with a slick visual representation of what I’ve been thinking and writing about. The graph view is actually functional, too– I can assign colors to nodes based on the directory they’re in, their tags, words they can contain, and a lot of other criteria. Because each node is sized according to how many links lead to it, this view makes it easy to see clusters in things I’ve thought or written about.

Publish

One of the advantages of Obsidian over something like Notion or Roam Research is how easy it is to publish notes from it.

Obsidian Publish (paid)

There’s the paid Publish service, which I’ve subscribed to for US $8 a month. After setting it up to publish on my own domain rather than the default at publish.obsidian.md, going from a note to a published post is as easy as clicking the Publish icon, selecting the notes to add, delete, or modify, and then selecting Publish.

Open folder as vault

If you don’t want to pay for Publish, Obsidian still saves all files locally in Markdown (.md) format. This is the same format that many static site generators take for content. This site that you’re reading this on is generated using Hugo, so everything written here is kept as a Markdown file before it’s turned into HTML. Since an “Obsidian vault” is just a folder with Markdown files, I can just open the directory containing the files for my site and write blog posts in Obsidian with no noticeable difference.

At work, we happen to store our main product documentation as Markdown files in a GitHub repo, which make it perfect for opening in Obsidian as well. I prefer to write documentation in Obsidian over VSCode for a few reasons:

I can install plugins, like Advanced Tables to make Markdown tables easier to deal with, create templates to standardize the format of notes you make often, see an outline of what you’re writing based on Markdown headings.

Outline core plugin for Obsidian

When I want to link to another page, it’s easier to just hit CMD + O to search through existing notes instead of finding it on GitHub.
I can choose an Obsidian theme that matches the theme of the published site. This “Gitsidian” theme has an Edit view in dark mode (my preference) but a rendered Preview mode that looks more like GitHub’s Markdown format, so I can see what it will actually look like for users.

When I’m done, I can still jump to my terminal and push my change to GitHub as normal.

Obsidian URIs

Another cool feature is the fact that every note in Obsidian has what’s called an Obsidian URI - it’s a link that opens up a note within a specified vault. The URI looks like this:

obsidian://open?vault=docs&file=src%2Fdata%2Fmarkdown%2Fdocs%2F03%20cloud%2F02%20Analyzing%20Results%2F02%20Performance%20Insights

Adding it and publishing that to internal documentation means I can switch easily from browsing on GitHub to modifying the same page in Obsidian on my local machine, which is a great little bonus.

Embedding published work

After publishing, I use iframes in Obsidian to update my notes with the final result. Here’s a video on YouTube about chaos engineering that my colleague Simme and I planned out collaboratively on Obsidian, embedded into Obsidian:

The video plays in Obsidian with all the same controls. The Obsidian desktop app is an Electron app, so it behaves just like a normal browser for playing back the video. To embed the video, I just used the Share > Embed code on YouTube and pasted it straight into Obsidian:

<iframe width="560" height="315" src="https://www.youtube.com/embed/gVwJZPo30rk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Meetings

I take meeting notes to help me focus on the meeting, and I create templates for recurring meetings. Templates let you define the format of a note. Once created, you can insert templates into notes with a keyboard shortcut (I set mine to CMD + ;). Here’s an example of what that looks like, with the template on the left and the inserted template on the right:

Template (left) and inserted template in a new note (right) in Obsidian

You may notice that the frontmatter includes {{date}} in the template, which gets automatically converted to today’s date when the template is used.

If anyone asks for a copy of my meeting notes, that’s no problem– any note can be exported to Obsidian by clicking on Options > Export to PDF.

Options for exporting a note to a PDF in Obsidian

I don’t want to actually publish any real meeting notes, so here’s what my note on the chaos engineering process looks like as a PDF, straight from Obsidian. And here’s what it looks like as a note.

Presentations

The easier way

Obsidian has a presentation mode! It’s basic, but it’s functional. You can create a presentation really easily by adding --- between paragraphs of text to delineate slides, right within a note. Here’s what that looks like:

Then you can click Options > Start presentation, and you get something like this:

A little plain, but given the minimal amount of effort to “convert” a note into a presentation, it’s pretty awesome.

The harder way

Sometimes the basic slides just don’t cut it, and it can be handy to be able to publish the slides in presentation format (since publishing them straight from Obsidian Publish would only publish the Markdown version). For those cases, I’ve figured out a way to do just that while still keeping my slides as Markdown files in my vault. Here’s a whole post on how to do that. This is definitely more complicated, but at the end of it, you get a nice page like this to send to attendees after a presentation. (Tip: Hit s to see the speaker notes while on those slides.)

Task management

I use a few community plugins to keep track of things I want to.

Kanban

The Kanban plugin turns Markdown task lists into full-on Kanban boards.

On the left is the markdown note, and the on the right is the rendered Kanban board. You can customize the titles of the lists that are there, and you can add due dates to each task.

Calendar

The Calendar plugin is Obsidian’s most popular plugin, and with good reason.

It lets me add a calendar on the sidebar:

I have the weekly notes option enabled, so clicking on the W column in the calendar takes me to a note that is automatically created for that week. I use it to keep track of weekly goals and to do a review at the end of the week.

Clicking on a date in the calendar takes me to the…

Daily Note

The Daily Notes core plugin comes with Obsidian, but you have to enable it. Once enabled, clicking on a day in the calendar creates/takes you to that day’s corresponding note. I use daily notes like a log or workbench for that day, with timestamped headings to help me remember later on what I did at what time. It’s a loose way to do time management as well.

Dataview

The Dataview plugin is my favorite recent discovery, and it’s changing the way I do frontmatter. This plugin brings something akin to SQL (albeit not as powerful) to Obsidian. It lets you query your notes based on some file-related metadata as well as parameters you set in your frontmatter.

For example, this query:

table file.ctime as Date
from "Meeting"
where file.ctime >= date(today) -dur(1 month)

when rendered (i.e., viewed in Preview mode in Obsidian), returns a list of all my notes under the subfolder Meeting in the last month:

This is huge! I’m still updating my notes with the appropriate frontmatter to make more use of this feature, but here are some use cases I can think of that would be handy at work:

Find all customers with tag annual that I haven’t spoken to in the last year.
Find a load testing tool with feature Slack_integration = true.
Find that woman I met at the Automation Test Guild conference by searching for job = tester and tag = AutomationTestGuild.
Return all notes tagged #plugin_ideas.

And that’s just the beginning!

Collaboration

Obsidian itself does not (yet) have collaboration features. This is a bummer, but given how quickly they’ve delivered on features that the community has clamored for (block references and mobile apps come immediately to mind), I’m hopeful that they’ll eventually improve in this area.

In the meantime, I’ve still found ways to work on the same vault with other people.

Remember how any folder of .md files can be opened as an Obsidian vault? Well, I’m technically collaborating with other people on that repo for work documentation, even if most people don’t use Obsidian. All I had to do was add the .obsidian directory to the .gitignore file, and now, nobody really even needs to know I’m using Obsidian to view/edit the documentation. They just see me pushing to the repository as if I had used a text editor like VSCode.

I’ve also had the experience of collaborating with someone who also used Obsidian. Since we were working closely together on the same document, we opted to put our shared vault on Dropbox. Dropbox’s main advantage over the Git solution is that Dropbox let us edit notes and see each other’s edits in real time. If you’re looking for an experience similar to Google Docs, this way got the closest to that. We did run into some quirks: if the Dropbox vault is also a Git repository, be aware that checking out a different branch will also change the contents of the directory for the other person.

Conclusion

Note-taking and personal management aren’t just for students and teachers; I think they’re particularly useful in tech, where knowledge work is highly prized. I’m talking about Obsidian because that’s my current tool of choice, but I think the tool you use is less important than the systems you put in place to help yourself learn and create.

If you’d prefer the video version, here’s a video I made showing this process:

Presentation Slides as Code

Tue, 02 Mar 2021 19:07:31 +0100

I was working on a presentation for later this month when Marie Drake told me about slides.com. It’s basically a Powerpoint replacement (which, let’s be honest, needs replacing) in that it lets you create slides and it lets you present from it. It even goes beyond Powerpoint in that it also saves your slides publicly for people to view later.

Before she’d mentioned slides.com, I had been planning on creating a presentation on Obsidian, my note-taking tool of choice. The advantage of this approach is that presentations in Obsidian are just Markdown files with --- between lines of text to delineate slides.

The problem with using Obsidian on its own is that it’s severely lacking in features, and it’s also not shareable. You can, of course, publish the Markdown file, but people won’t be able to view them in slide format. After playing with slides.com, however, I also discovered it had its own weaknesses. It is completely UI-driven, so there’s no way to create slides just from a Markdown file. It’s also hosted by them, so there are limitations imposed, depending on which pricing tier you pay for.

So I got to Googling. And I think I found a pretty good setup for my needs.

The end result

I write the presentation in Obsidian, in Markdown. Here’s what that looks like in a sample presentation where I’m trying out a bunch of features.
I use Hugo to generate a static website and convert the Markdown files into javascript and html, using reveal.js and the reveal-hugo theme.
I push the static website to a remote repository on GitHub. GitHub Pages serves that site (for free!) on my custom subdomain.

Check out the final product here: https://slides.nicolevanderhoeven.com/sample

The setup

Prerequisites

An Obsidian vault (or you can actually just use any text editor)
Git installed and set up with credentials
A GitHub account
Hugo installation (brew install hugo)

Create a new Hugo site

Open up a terminal window and cd to a folder in your Obsidian vault. Then, create a new Hugo-generated site:

hugo new site slides

Replace slides with the name of your new site.

You should have a new directory, slides, in your Obsidian vault.

Add the reveal-hugo theme

cd into slides and clone the reveal-hugo repo. The documentation suggests using a submodule:

git submodule add git@github.com:dzello/reveal-hugo.git themes/reveal-hugo

Personally, I just cloned it because I know I’m probably going to be messing around with it substantially:

git clone git@github.com:dzello/reveal-hugo.git themes/reveal-hugo

You should have a /themes/reveal-hugo folder that contains the theme.

Configure reveal-hugo theme

Set up your new Hugo site to use the reveal-hugo theme by adding the following to /config.toml:

theme = "reveal-hugo"

[markup.goldmark.renderer]
unsafe = true

[outputFormats.Reveal]
baseName = "index"
mediaType = "text/html"
isHTML = true

The Reveal part is what lets you output text in slide format.

Create your slides

Go to /content and create a file there, called _index.md. The filename is important, as is its location (I learned that the hard way). This file is going to be the first page people see, so it should probably be a list of your presentation slides. For now, put in something like this:

+++
title = "My presentations"
outputs = ["Reveal"]
+++

Here are my presentation slides!

[sample presentation](/sample)

Then, create a new folder for your presentation, such as sample. In /content/sample, create a file called _index.md. Again , the filename is also important here (guess how I learned that).

This index file is going to be where you actually create your slides. Here’s the context of my /content/sample/_index.md:

Each slide is separated by ---, just like in Obsidian, which apparently uses the same converter that reveal-hugo does, reveal.js. Note that if you want to try out having an image as a background to one of the slides, you’ll need to add that to your /sample folder and change the filename in the slide in this part:

slide background-image="lucrezia-carnelos-AsM4T46tqcU-unsplash.jpg"

Generate your site

Now it’s time to generate your site! Wait - didn’t we already do that? Sort of that. That was creating the framework for the site, but now you need to get Hugo to convert those Markdown files you’ve just created into something publishable. So cd to the root directory slides and run this:

hugo

That command should generate a slides/public folder, where all the publishable code will go.

At this point, you can also check your work by starting a local server with hugo server and navigating to http://localhost:1313.

Push your site to a remote GitHub repo

cd into your public folder. This is important, as I initially started the repo in slides and it took me ages to realize that GitHub Pages requires an index.html in the root directory to function.

Initialize a Git repo:

git init

Stage and commit your changes:

git add .

git commit -m 'Initial commit'

Create a repository on your GitHub account called username.github.io where username is your GitHub username. (If you already have a repo called that, you can alternatively push the contents of public to a branch called gh-pages). The repo name matters.

Push your changes to your repo.

git push -u origin master

You should now have a repository on GitHub called username.github.io, containing everything in public.

Set up GitHub Pages

On GitHub, go to the repository settings and scroll down to the GitHub Pages section. Select the right branch (master in most cases) and click Save.

(Optional) Using a custom domain

If you want to use a custom domain, enter it in the repository settings from the previous step.

Then, go to your domain registrar and add a CNAME record pointing www to username.github.io. Here’s more information on that from GitHub.

It can take a few hours for DNS records to propagate, so after you set this up, take a break before you convince yourself you’ve set something up wrong. ;)

Check out your new site!

When you navigate to your new site, you should first see the contents of slides/content/_index.md, which has a link to your sample presentation. Clicking on that should reveal (ha) the slides one by one. You can navigate through them with the arrow keys, and you can also see speaker notes (if you put any in, which I did on the third slide if you copied my sample slides).

Other issues I faced

I ran into a few issues during this setup because:

I was already tracking changes in my Obsidian vault with Git
I already had a repo called nicolevanderhoeven.github.io, and I needed to create a branch called gh-pages to still use GitHub Pages.
My Obsidian repo on GitHub is a private one, but I wanted the slides to be public.

If you have a similar setup, maybe the following instructions will help.

Use worktrees, not subtrees or submodules

After attempts to make this setup work with submodules and subtrees, both of which were way too complicated for what I needed, I settled on using worktrees.

Add public to .gitignore. From slides:

echo "public/" >> .gitignore

Create a worktree for public, to be tracked on the branch gh-pages:

git worktree add public gh-pages

After you generate the static code via hugo:

cd public
git add .
git commit -m 'Commit message'
git push origin gh-pages

Your gh-pages branch should contain only the files in public.

GitHub Pages with a private repo

GitHub Pages is free, but if you want to be able to use it with a private repo, you can’t. Sorry!

You can look into using GitHub Actions to copy public from your private repo to a public one, or you could just spend the $4/month to remove the restriction. I opted to pay, because I love GitHub anyway.

Presentation as code

That’s it! I ran into a few snags while I was attempting to implement this, so I thought I’d document it here in case it helps anyone else. Despite the setbacks, I think it’s worth it– I can now whip up presentations in Markdown within Obsidian and have that published to a site I control for people to look through at their own leisure. This should speed up my presentation creation process considerably.

Theoretically.

If only I could stop procrastinating on creating this presentation by setting up a framework for creating presentations…

Readwise to Obsidian

Sat, 06 Feb 2021 09:55:36 +0100

August 10th, 2021: Readwise just released their own Obsidian plugin, Readwise Official, which is available now! I now recommend this method over the one outlined in this article.

I’m about 9 months into heavily using Roam Research as my primary note-taking, task management, capture-everything system, and for the most part, that’s gone very well. Obsidian is another excellent personal knowledge management (PKM) system, though, and I’ve been curious about it for almost as long as I’ve been using Roam. I’ve had it installed for a while, but I never really seriously attempted to make the switch. Until now. I’ve decided to use Obsidian as my primary PKM and zettelkasten for the next month, to see if it’s something I can switch to.

Why I’m considering switching from Roam to Obsidian is another post, probably one I’ll write at the end of this experiment, when my thoughts and opinions are more solidified. For now, I want to address one of the things I struggled with when switching over: Readwise integration.

Readwise is an app that’s all about importing notes and highlights you’ve made from all over the internet and exporting them all into a single format. Readwise supports far more import sources than I use, but here are the ones I’ve got hooked up to it:

Instapaper (for highlights and notes on online articles)
Amazon Kindle (for highlights on ebooks)
Twitter (for interesting tweets)
(Readwise) Web highlighter (for straight highlights off any webpage)
Airr (for quotes and notes from audio podcasts)

Readwise collates my activity in all of these input sources and automatically sends them to my Roam Research account. Here’s what that looks like when it gets pulled into my Daily Note:

Readwise also pulls metadata into Roam, depending on the source. Here’s what the page for the book “Ready Player Two” looks like in Roam:

Readwise integration is an essential feature for me in any PKM, because I can take notes wherever I’m consuming my content and just have it automagically appear in my notes, tagged and ready for processing. So it’s absolutely something I need to have in my Obsidian workflow.

That’s where things get tricky, because Readwise doesn’t have an Obsidian integration.

Source: Readwise Exports

The closest thing is the Markdown export, but that’s not automatic. You can manually trigger an export, and then copy those exports over into your Obsidian vault, and thne make sure they’re in the right format with the right tags. I knew I was never going to do that, so I found a way to automate it.

readwise2directory

I discovered that Readwise has an API, and briefly thought about building my own integration. Happily, I found this Python project on GitHub by nicrivard, who had already done that! Why rebuild the wheel? His solution already did what I needed:

It connects to Readwise via your Readwise access token
Checks for new highlights or updated ones
Exports the new/updated highlights to a local folder of your choosing (presumably your Obsidian vault)

It even pulls in highlight tags from Readwise, if you’ve created any (which I haven’t).

I forked the project here to add just a few minor touches:

Changed date format of imported notes
Added a section to output the import date to a comment block (an Obsidian Insider build feature) rather than YAML frontmatter, so that the date would actually be linked and backlinked.
Customized tags and filename for my use
Sort imports into folders within my Obsidian vault depending on source (Article/, Tweet/, and so on).

This works out pretty well. Here’s an example of an import from the podcast app Airr into Obsidian, in Edit mode.

It’s placed within my Podcast folder inside my value, is tagged appropriately, and contains links to the source. Switching to Preview mode shows an imported cover photo and even a playable recorder for the part of the podcast that I snipped on Airr:

So that works out well, except for one thing: firing off the script that does all this is still manual.

Scheduling the script

Luckily, it’s pretty easy to schedule a Python script to run at certain intervals. I’m on macOS, so I used a built-in utility called cron.

Give cron full system access

Cron needs to have the right permissions to run scripts on my behalf, so I had to go to System Preferences > Security & Privacy > Privacy > Full Disk Access on my laptop and add /usr/bin/cron to the “allowed” list.

Open crontab

Then, from my terminal, I ran the command env EDITOR=nano crontab -e to open up the list of cronjobs I had currently.

Set a schedule

I added a line to crontab for the schedule I wanted to create. This line has to be in the format

* * * * * command

where the five asterisks correspond to:

minute
hour
day of the month
month
day of the week

In my case, I wanted it to run every hour, so here’s what my schedule looks like:


0 * * * * cd /Users/nic/git/readwise2directory/ && /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/nic/git/readwise2directory/readwise-GET.py >> /Users/nic/git/readwise2directory/readwiseGET.log

Python issues

I had some trouble getting cron to work until I realized it was because I had multiple versions of Python installed. I ran a which python3 to find the path to the version I wanted to use and added that path to the command. I also had to cd to the project directory, for some reason.

While I was troubleshooting, I found it really useful to output to a log so I could see what had gone wrong.

Once I added the schedule as in the example above, though, it worked a treat!

Conclusion

This solution did require some setup at the beginning, but doesn’t need any maintenance once it’s going. I’m pretty happy with it as it means I can keep taking notes on various places from anywhere, and it’ll all be waiting for me in my Obsidian vault whenever I’m ready to sit down and process them into my system. Thanks to nicrivard for the repo that made it possible!

Comparing k6 and JMeter for load testing

Fri, 29 Jan 2021 23:01:18 +0100

Originally posted on k6.io.

Introduction

When I joined the k6 team in late November, the foremost question on my mind was “How does this compare to JMeter?” You see, I come from a performance testing consulting background, and in the last few years, JMeter has been far and away my tool of choice.

So what am I doing on the k6 site, talking about how much I love JMeter? Well, firstly, it turns out that the k6 team is pretty open and transparent, especially about potential improvements in k6. Secondly, I want to share my reasons for wanting to explore k6 as a load testing tool– the same reasons that ultimately convinced me to join the team that builds it.

What is JMeter?

JMeter is an open source load testing tool built entirely in Java by the Apache Foundation. It was first released in 1998, and it made waves because of its audacity in taking on more popular but proprietary load testing tools. JMeter took what other companies were charging a lot of money for and published it online for everyone to use– for free. Although scripts can also be extended using code, majority of the scripting in JMeter is done using the UI. At the time of this writing, the latest version of JMeter is 5.4.

What is k6?

k6 is an open source load testing tool released by LoadImpact in 2017. LoadImpact had previously consisted of the SaaS platform (now k6 Cloud) and professional services. k6 is written in Go, but scripts are written in Javascript. Taking the opposite tack to JMeter, k6’s main selling point is its code-based scripting and heavy prioritization of the developer experience. The latest version of k6 is 0.30.

Which is better?

This article isn’t about which tool is better, because as much as we might prefer a clear-cut answer, there’s no “best tool”. Everything depends on the situation your team is in: what, how, why, and when you’re testing.

The question that I want to answer here is: What situations does each tool excel in?

When does JMeter excel?

When you want to get started quickly

JMeter has a GUI

Many testing teams make decisions on load testing tools based on this fact alone. Sometimes, if you’re in a situation where people haven’t done load testing before and need to learn a new tool, a GUI-driven tool like JMeter is simply the most frictionless option. A tool with a GUI will be less daunting for many testers that are used to the mostly no-code UIs of the likes of Postman or SOAPUI.

I’ve always thought of JMeter as user-friendly, but a developer friend recently pointed out that the Test Plan screen that greets you when you first start JMeter doesn’t provide any indication of how to create an HTTP sampler. None of the icons add elements to the test plan. It takes a bit of exploring to realize that right clicking on the test plan, or going to Edit > Add, will show the “Add” menu that you need. He has a point there: UI is subjective to a certain degree. However, I’d argue that it’s definitely easier for non-developers to explore a UI than a bit of code.

k6 doesn’t have a GUI packaged within the tool, but the k6 Test Builder is available for free. It’s a way to create tests with a GUI interface, and despite its inclusion in k6 cloud, it doesn’t require any subscription to use. However, it isn’t as fully-featured as JMeter.

JMeter supports many protocols and features out of the box

Here’s the official list of the protocols that JMeter supports:

Web - HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, …)
SOAP / REST Webservices
FTP
Database via JDBC
LDAP
Message-oriented middleware (MOM) via JMS
Mail - SMTP(S), POP3(S) and IMAP(S)
Native commands or shell scripts
TCP
Java Objects

In contrast, k6 supports the following protocols:

Web - HTTP/1.1, HTTP/2 (Java, NodeJS, PHP, ASP.NET, …)
WebSockets
gRPC
SOAP / REST Webservices

The protocols above are those that are natively supported. However, both tools allow the development of third-party plugins, adding support for more protocols. The number of k6 community extensions, through xk6, is growing, but JMeter plugins still outnumber them. In many cases, the protocols your application uses may make the decision for you of which tool to use.

JMeter also has most of the features you’ll need for a basic load test, and you don’t have to code any of it up yourself. Its parent-child element structure means that you can modify a specific element, like an HTTP request, or modify all HTTP requests, with equal ease. Adding think time to all requests is as easy as adding a Uniform Random Timer to your thread group, and all requests within that group will inherit it. With code-based load testing tools, it can be a bit more difficult to find out what’s possible without a UI. Some of the gap, however, can be bridged by an IDE with good autocompletion.

When you want a mature community with tons of documentation

JMeter’s old, and sometimes that’s a good thing. It’s been around since 1998, so at this point, it’s had 22 years to improve and to build a following. JMeter has a presence on nearly every community networking site, and no matter how specific your use case for JMeter is, the chances are that it’s been done before. Searching “how to load test X with JMeter” is bound to yield thousands of hits and some videos showing you exactly how to do it.

JMeter’s Component Reference is proof of JMeter’s extensive and thorough documentation. Every element, function, and property is discussed in more detail than most people could ever want, and that’s just the “official” documentation that’s on the Apache site. There are thousands of JMeter books, tutorials, and courses by its passionate users.

The JMeter GitHub repository has 17,290 commits as of December 2020 - evidence of a community of developers that have committed some time and experience to JMeter over the last two decades.

In comparison, the k6 community is growing, but still small. There is an official and central community forum that allows users to share their experiences with k6, which is something missing from the JMeter community. The k6 repository does have more stars, but significantly fewer commits at 3,795. Much of the disparity, of course, is because k6 is very new - the open-source tool k6 was only released in 2017.

When you need a cost-effective way to do distributed load testing

One of JMeter’s absolute best features is that it gives you a framework for running distributed load tests with it. This is pretty special for a free and open-source tool.

Distributed testing means ramping up the amount of load you’re generating with your load tests, usually by increasing the number of virtual users and running multiple instances of your script on other load generators. JMeter accomplishes this by designating a controller node and letting you set up worker nodes. Each worker node needs to have a copy of jmeter-server, a utility included with every JMeter installation (in jmeter/bin).

Source: Apache JMeter

These worker nodes then run copies of your test plan and exert load against your application server. You can use on-premise or cloud machines as your worker nodes. A distributed execution like this takes a bit to set up, but it does work surprisingly well.

k6 doesn’t have a native distributed load testing mode. You can containerize it yourself and deploy it via Kubernetes, for instance, but it does require a more involved setup. The team is also working on the k6 Operator, which makes setup significantly easier, but it is experimental and comes with drawbacks such as not supporting threshold synchronization between instances. For now, the easiest way to scale up on k6 is by using the paid service k6 Cloud, which reduces setup time and complexity significantly.

It’s important to note that JMeter can be quite resource-hungry, and thus is more likely to require multiple load generators to generate load than k6. Depending on the resources of the load generator, JMeter can run about a thousand virtual users on average, and scaling up your test beyond that point will require a distributed execution setup. A single instance of k6, on the other hand, can run tens of thousands of virtual users given the same resources. More on that later.

When you want canned reports built into your load testing tool

JMeter’s many listeners allow you to decide exactly what format you want to view results in. Debugging? Use the View Results Tree. Wanting metrics for your load test? Use the Summary or Aggregate Report.

In addition, JMeter can generate HTML reports based on your load tests. They’re currently quite limited, but functional, and they have several default graphs to help you analyze your load test. They are not interactive, but they are pretty easy to generate and a good starting point for putting together a report template for your tests.

k6 does not have built-in reporting. Its modular nature lends itself better to integrating with many other data analytics tools, including Grafana, Datadog, New Relic, Amazon CloudWatch, and k6 Cloud– all of which are better analysis tools and can provide more insights than what JMeter’s reports can provide, but they do require you to set up another tool to analyze your tests. If you’re running low on time and can make do with its reports, JMeter does really well at providing that extra value out of the box.

When does k6 excel?

When you want to get started quickly

Yup, I put this one in both the JMeter and k6 sections. JMeter and k6 can each be easy to get started with, but for very different reasons.

Installation

k6 installs extremely quickly - on my first installation of it on macOS, it took 7 seconds via brew install k6. It doesn’t require nodeJS or any other dependencies, either.

JMeter itself is fairly easy to install, but before you install JMeter, you need to install Java. Especially if you’re using Windows, it’s a bit of a rite of passage for a new JMeter user to run into issues adding the environment variables needed for Java to function.

It can also be confusing to determine which version of Java is the right one. There’s JDK, JRE, and recently Oracle has added to the confusion by announcing that some versions of Java (but not all) are going to require a license.

Source: Oracle Java

While JMeter is still free via OpenJDK, I’ve spoken to a few testers who mistakenly thought they’d have to pay for JMeter. It’s a little disconcerting to see Oracle flexing its muscles here, especially if you’ve built a whole performance testing suite on Java through JMeter.

Plugins

k6 supports plugins, but they’re not necessary. For most use cases, k6’s built-in functionality should be more than sufficient. This is in sharp contrast to JMeter, where everything is a plugin … even the Plugin Manager!

Most JMeter users would argue that there is a standard set of plugins that you should download before you even begin to use it. A new JMeter user might not necessarily know where to find these and may have a less-than-satisfying experience out of the box.

Here are a few things you won’t have if you stick to a vanilla, no-plugin JMeter installation:

Thread groups: No custom load profile, stages, rendezvous
Controllers: No parallel controller to execute requests in parallel
Samplers: No HTTP/2 support
Test Data: No randomization of lines within a CSV file (sequential only)
Timers: No throughput shaping timer for setting throughput in stages

These functions are all available within k6 from the get-go.

Scripting

k6 scripts are written in javascript, making them simple to write and run. All you need is one installer (on Microsoft Windows) or a single command on your terminal and any text editor. For JMeter, you’ll need to install a few things locally before you can get started: Java, JMeter itself, and the standard set of plugins you’ll need. This can be confusing for new users. Who hasn’t struggled with environment variables at some point when learning to use JMeter, especially on Windows?

It also takes time to learn any new UI, whereas javascript is near-ubiquitous and it’s a good language to learn when you’re load testing web sites anyway.

When you want to maximize performance and efficiency

Test your test tools!

Why does all this matter?

When we’re talking about simulating many virtual users, slow performance = high cost. There is a cost to the number of load generators you use, whether it’s the cost to acquire and maintain them (on-prem) or the cost of a cloud service.

It makes sense to consider the performance limitations of your load testing tool before you use it to improve your own application’s performance.

k6 builds on Go’s stellar performance

k6 is written in Go, and Go is built with performance in mind. “Go is a compiled language and does not need to be interpreted, unlike Java or Python. There is no added layer of complexity.” It also has no external dependencies, which is a good thing. Fewer moving parts for a performance tester means fewer potential sources of a bottleneck.

The simplest and most common way for a load testing tool to run a virtual user is to assign one virtual user to one kernel or OS thread. However, the 1 Thread: 1 Virtual User paradigm has a serious flaw: when a virtual user is waiting for a response or executing a programmed sleep(), the thread is also blocked, and cannot process other work.

In k6, each virtual user is run on a goroutine, not a thread. What difference does that make? Well, goroutines can be controlled by something called the Go Scheduler, which acts like a traffic cop. It reuses idle threads and intelligently assigns work, by allowing “work stealing” and work hand-offs between threads. Does this sound familiar? This is the same principle that load balancers are built on: An external monitor that oversees the flow of work improves general performance. Go itself is intrinsically load-balanced in a way that many programming languages aren’t, which makes it the perfect foundation for a load testing tool.

Being able to leverage Go’s innate performance optimizations also translates into significantly less memory utilization. One thread running k6 doesn’t go above 100 kb, whereas a JVM thread like JMeter uses, for instance, uses the default of 1MB. That’s 1000% more than k6! Of course, Java allows a user to tweak an app’s memory utilization, so the difference is usually not quite as stark, but it’s still interesting to note that Go has a much lower starting point.

… creating [a] new thread in Java is not memory efficient. As every thread consumes approx 1MB of the memory heap size and eventually if you start spinning thousands of threads, they will put tremendous pressure on the heap and will cause shut down due to out of memory. Also, if you want to communicate between two or more threads, it’s very difficult. - Keval Patel

Performance benefits in practice

Fewer load generators needed

k6’s comparatively better performance means it needs fewer load generators to execute a given amount of load. Rafaela Azevedo made a comparison of the memory used by k6 and JMeter, and here are her results:

JMeter took up 760 MB of memory.

Source: Rafaela Azevedo

Meanwhile, k6 took up 256 MB of memory.

Source: Rafaela Azevedo

Rafaela’s findings are further affirmed by our own benchmarks.

Source: Ragnar Lönn, in our Load Testing Tool Comparison

k6’s lower memory footprint is the reason it can run more virtual users and generate more load than average. Whether you’re using load generators that are on-premise or in a cloud, you’ll pay less for provisioning costs when using k6. This cost saving makes k6 a great tool for budget-conscious teams.

The common misconception of many load testers is that distributed execution (ability to launch a load test on multiple machines) is required to generate large load. This is not the case with k6. - Rafaela Azevedo

k6 is different from many other load testing tools in the way it handles hardware resources. A single k6 process will efficiently use all CPU cores on a load generator machine. A single instance of k6 is often enough to generate load of 30.000-40.000 simultaneous users (VUs). This amount of VUs can generate upwards of 300,000 requests per second (RPS).

Unless you need more than 100,000-300,000 requests per second (6-12M requests per minute), a single instance of k6 will likely be sufficient for your needs. - k6 documentation

No out of memory errors

k6 is also good for when you’re short on time. Using JMeter means getting familiar with how to performance tune Java and how to fix the most common Java performance problems, because there are several. The one that I’ve personally run into the most while using JMeter is:

ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError:
Java heap space.

You’re not really a JMeter load tester until you’ve seen this error. :)

The cause for this error is usually that there was an insufficient amount of Heap memory allocated to JMeter. Yup– in addition to monitoring your load generator’s memory usage, you’ll also need to monitor the JVM’s memory usage. You can set the amount of heap memory allocated to JMeter by modifying the JMeter binary itself and changing this line:

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
: "${HEAP:="-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m"}"

Even JMeter official documentation suggests that you increase the Java Heap size beyond 1GB before you even run your first load test.

Increase the Java Heap size. By default JMeter runs with a heap of 1 GB, this might not be enough for your test and depends on your test plan and number of threads you want to run - Apache JMeter

No GUI means no resource overhead during load tests

GUIs typically add significantly more overhead to an application, which is one of the reasons why k6 doesn’t have one.

When you’re running a load test, the best practice is to run it from the command line anyway– because otherwise, it interferes with your results. Code-based load testing tools are performant from the beginning.

JMeter is peppered with warnings about the GUI’s effect on performance. Here’s one from the “JMeter documentation:

Don’t run load test using GUI mode ! - Apache JMeter

And here’s a message that occurs upon startup of JMeter:

Not having a GUI means there’s one less gotcha. Any load testing scripts you create in k6 are ready for prime-time execution whenever you’re ready.

When you want to do goal-oriented testing

The most important question a load tester can ask when starting a new load testing project is “Why”? Test plans should be made to directly address a team’s reasons for wanting to do load testing, and that should be reflected in the non-functional requirements. Non-functional requirements, in turn, should then give some guidance as to the bounds of acceptable performance. A common example for these bounds, or thresholds, is an average response time of less than 3 seconds for all transactions during the test. k6 does this natively with global thresholds worked into the script, and you can also create your own metric to use in a threshold:

import http from 'k6/http';
import { Rate } from 'k6/metrics';

const myFailRate = new Rate('failed requests');

export let options = {
 thresholds: {
 'failed requests': ['rate<0.1'], // threshold on a custom metric
 http_req_duration: ['p(95)<500'], // threshold on a standard metric
 },
};

export default function () {
 let res = http.get('https://test-api.k6.io/public/crocodiles/1/');
 myFailRate.add(res.status !== 200);
}

Source: k6 documentation

JMeter doesn’t natively support thresholds at the test-level. There are a few ways that I’ve been able to work around this:

Duration Assertion: You can add a Duration Assertion to any sampler in JMeter, which allows you set an acceptable response time for that request in milliseconds, beyond which the request (and the transaction that encompasses it) would be marked as failed. However, this is still just at the individual request level, and even putting the duration assertion element at the Test Plan level only applies the same threshold to each child request. It doesn’t constitute pass/fail criteria for the test as a whole.
Timers: For criteria related to throughput, you can use the Constant Throughput Timer or the Throughput Shaping Timer (plugin). This approach controls how many requests per second are spent. In my experience, it’s worth testing how these timers behave when using nested transaction controllers - it may take some rearranging to get right.
Performance Plugin: When integrating JMeter with Jenkins, you can use the Performance Plugin to set error thresholds and mark test runs (“builds”) as unstable or failed within Jenkins.
Custom Code: You can use the JSR223 Sampler to write a bit of Groovy or BeanShell code that collects and parses response metrics.

The problem is that even though these options make setting thresholds possible in JMeter, they’re still workarounds that don’t come with JMeter out of the box or don’t adequately address the creation of different types of thresholds (error, response time, throughput, CPU, memory). What I end up doing, more often than not, is exporting the raw data from JMeter and doing the analysis myself using another tool.

When you’re part of a team of scripters

k6 shines in situations where more than one person could be doing the scripting.

It strikes a middle ground that developers and testers can compromise on

Software quality should not be the sole responsibility of testers; quality should be baked into every activity when building software. One of the challenges in implementing this concept, however, is that there has traditionally been a segmentation in tools and languages used by team members with different functions. Developers use one tool; testers use another. It’s difficult to encourage developers to test and testers to code in this environment, and many testing tools only further encourage this divide. A developer can’t use a tool like JMeter for anything but testing, so the incentive to invest the effort in learning it is lower.

k6 bridges the gap by bringing testing concepts and features to an environment and language that developers are already spending their time in. Javascript is useful for more than just writing load testing scripts, and k6 works well with any IDE or text editor. A frictionless developer experience is central to the k6 philosophy.

It’s easier to collaborate on scripts

Once you’ve convinced both devs and testers to contribute to load testing scripts, how do you keep track of changes? How do you manage scripts that multiple people are working on at the same time? With tests as code, the easiest way is to just use versioning tools like Git, the way you probably are already using for your application code. Again, no new framework or processes to learn here. k6’s conciseness and portability really proves to be an advantage in collaboration situations. To illustrate this, let’s consider a simple test script that contains a few things:

A GET request to our test site, https://test.k6.io
A transaction that calls this request 01_Home
A check to fail non-HTTP 200 responses or responses that don’t have the text Collection of simple web-pages suitable for load testing
Variable think time
We’ll just run that as a single user, for a single iteration. Here’s what that script looks like in k6:

import http from 'k6/http';
import { sleep, check } from 'k6';

export default function () {
 let res = http.get('https://test.k6.io', {tags: { name: '01_Home' }});
 check(res, {
 'is status 200': (r) => r.status === 200,
 'text verification': (r) => r.body.includes("Collection of simple web-pages suitable for load testing")
 });
 sleep(Math.random() * 5);
}

You can track JMeter scripts on Git, too. Here’s what the same script looks like in JMeter:

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.3">
 <hashTree>
 <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
 <stringProp name="TestPlan.comments"></stringProp>
 <boolProp name="TestPlan.functional_mode">false</boolProp>
 <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
 <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
 <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
 <collectionProp name="Arguments.arguments"/>
 </elementProp>
 <stringProp name="TestPlan.user_define_classpath"></stringProp>
 </TestPlan>
 <hashTree>
 <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
 <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
 <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
 <boolProp name="LoopController.continue_forever">false</boolProp>
 <stringProp name="LoopController.loops">1</stringProp>
 </elementProp>
 <stringProp name="ThreadGroup.num_threads">1</stringProp>
 <stringProp name="ThreadGroup.ramp_time">1</stringProp>
 <boolProp name="ThreadGroup.scheduler">false</boolProp>
 <stringProp name="ThreadGroup.duration"></stringProp>
 <stringProp name="ThreadGroup.delay"></stringProp>
 <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
 </ThreadGroup>
 <hashTree>
 <TransactionController guiclass="TransactionControllerGui" testclass="TransactionController" testname="01_Home" enabled="true">
 <boolProp name="TransactionController.includeTimers">false</boolProp>
 <boolProp name="TransactionController.parent">true</boolProp>
 </TransactionController>
 <hashTree>
 <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
 <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
 <collectionProp name="Arguments.arguments"/>
 </elementProp>
 <stringProp name="HTTPSampler.domain">test.k6.io</stringProp>
 <stringProp name="HTTPSampler.port"></stringProp>
 <stringProp name="HTTPSampler.protocol">https</stringProp>
 <stringProp name="HTTPSampler.contentEncoding"></stringProp>
 <stringProp name="HTTPSampler.path"></stringProp>
 <stringProp name="HTTPSampler.method">GET</stringProp>
 <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
 <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
 <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
 <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
 <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
 <stringProp name="HTTPSampler.connect_timeout"></stringProp>
 <stringProp name="HTTPSampler.response_timeout"></stringProp>
 </HTTPSamplerProxy>
 <hashTree>
 <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
 <collectionProp name="Asserion.test_strings">
 <stringProp name="-745788246">Collection of simple web-pages suitable for load testing</stringProp>
 </collectionProp>
 <stringProp name="Assertion.custom_message"></stringProp>
 <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
 <boolProp name="Assertion.assume_success">false</boolProp>
 <intProp name="Assertion.test_type">16</intProp>
 </ResponseAssertion>
 <hashTree/>
 </hashTree>
 </hashTree>
 <UniformRandomTimer guiclass="UniformRandomTimerGui" testclass="UniformRandomTimer" testname="Uniform Random Timer" enabled="true">
 <stringProp name="ConstantTimer.delay">1000</stringProp>
 <stringProp name="RandomTimer.range">4000</stringProp>
 </UniformRandomTimer>
 <hashTree/>
 </hashTree>
 </hashTree>
 </hashTree>
</jmeterTestPlan>

Which one is easier to understand? If you had to modify the script to add a request, which format would you choose to do it in?

Scripting in XML makes merge conflicts more likely and breaking changes in automation pipelines way more difficult to pinpoint.

I’ve collaborated with others on JMeter scripts in the past, and we always opened up scripts in the GUI. It’s way easier to spot changes there - but that also created some new issues.

When you want something easy to maintain

Versioning for load testing

When ramping up load tests to more than one load generator, keeping track of the versions of necessary components ensures that one load generator’s traffic is the same as another’s. The value of reducing variables in scientific experiments holds true for load testing.

With k6, you just need to worry about the version of k6 and the script. No other dependencies are required. If you’re using plugins, you’d need to track those too, but as mentioned previously, plugins in k6 aren’t as necessary as they are in some other tools – and k6 is available as a Docker image to reduce those issues.

With something like JMeter, there are more levels to consider. There’s the version of JMeter itself, Java, versions of plugins, and then the script. Every new layer introduces more changes for incompatibilities. In previous projets I’ve worked on, someone would inevitably try out a new plugin and forget to discuss it with the team, and anyone who tried to open that script in their copy of JMeter would see this error:

The error means there’s a plugin that the script uses that JMeter doesn’t recognize, but depending on the plugin, it’s not always this easy to pinpoint which one (it was the Ultimate Thread Group in this case). You don’t want that happening in the middle of your load test because you forgot to add a plugin to some of your load generators!

It’s easier to link test code with application code

If your application code is in Javascript, k6 is a shoo-in as a load testing tool. You could build tests that are tightly integrated with application code by importing objects and having your tests interacting with them directly. This approach would reduce a lot of time spent in refactoring test scripts when objects are changed.

If you’re already using VS Code, k6 even has a VS Code plugin.

Test as code gives you control

Having your scripts be pure code reduces ambiguities in that you’re not limited by poor UI design or bugs in third-party plugins. k6 scripts are Typescript-typed, so most IDEs support autocompletion while you’re scripting - the code equivalent of helpful tooltips, except way more universal. k6 scripts give you the freedom to import and use JS libraries, so you can build on code others have already written. While k6 is not NodeJS, there are even some NodeJS packages that you can bundle for use with your scripts.

No external dependencies

k6 doesn’t require NodeJS or anything else to run, which also reduces the likelihood of bugs that might affect your scripts. For example, since JMeter runs on Java, it is also dependent on Java. When Java receives a new version, that often requires a JMeter update as well. JMeter versions understandably lag a bit behind Java versions as developers refactor code, but this gap can lead to potential security issues in a previous version of Java that may remain unpatched in the meantime.

With k6, development can proceed independently of any third-party component, so major issues can be addressed more easily.

Comparison table: JMeter vs k6

Feature	JMeter	k6
Written in	Java	Go
Scripting language	Limited: some Java (Groovy, Beanshell, etc)	Javascript
Protocols	Supports most protocols via plugins (native support for HTTP/1.1, SOAP, FTP, JDBC, LDAP, MOM via JMS, SMTP, POP3, IMAP, shell scripts, TCP, Java objects)	Supports fewer, modern protocols natively (HTTP/1.1, HTTP/2, WebSockets, gRPC)
External dependencies	Java	none
Resource utilization	Poor; One load generator can simulate a few thousand virtual users	Very good; One load generator can simulate tens of thousands of virtual users
Memory management	JVM heap memory must be set	Uses load generator memory natively
Threading model	1 Thread: 1 Virtual User; Slower performance, higher resource cost	1 Goroutine: 1 Virtual User; Faster performance, lower resource cost
Ease of scripting	GUI-driven, with code blocks	code-driven; VSCode plugin
Test-level thresholds	No, individual request only	Yes
Script format	XML	javascript
Collaboration	Difficult to work on simultaneously; tester-friendly; need the GUI app to edit	Developer-friendly, easy to version; Javascript format promotes collaboration
Maintenance	Verbose scripts; XML format is difficult to read	More concise scripts; JavaScript is easy to read
Community	Since 1998; Many third-party tutorials; Extensive documentation; No central community	Since 2017; Extensive documentation; Fewer third-party tutorials; Official community
Plugin support	Requires plugins for many features, but there are lots of plugins available	Most features are natively supported, but plugin support is new and availability is sparse
Native distributed load generation	Yes	No (Premium only)
Pre-generated reports	Default and custom HTML reports; logging via listeners	No built-in pre-generated reports; integration with analytics tools with third-party dashboards
Website	jmeter.apache.org	k6.io
Source Code	Link	Link

Conclusion

The best advice, when attempting to choose a load testing tool, is to do a proof of concept with the most promising candidates. Some features or bugs may prove more or less important when used in an actual testing cycle. However, don’t put too much weight on comparisons of the load test results from two different tools - each tool records metrics differently, and it’s more meaningful to compare results against previous runs with the same tool. When switching tools, reestablish a baseline in the new tool each time.

Here’s a summary that might help you decide between JMeter and k6.

Both tools do well at:

Generating protocol-level load on an application server by scripting complex user flows
Realistic scripting with the use of dynamic think time, test data generation and reuse, and customizable workload models
Documentation of features and consistency of releases

Neither tool supports:

Generating browser-level load and interacting with DOM elements, especially for SPA
Detailed results analysis (JMeter has pre-generated HTML reports and listeners, but they are far from ideal). Users should expect to integrate results with databases and data visualization software.

JMeter is best for:

traditional testing teams
those looking for a GUI-driven testing tool with tons of third-party tutorials and extensive protocol support
previous users of commercial tools like LoadRunner and NeoLoad

k6 is best for:

collaborative, cross-functional engineering teams where testing spans acrosss multiple roles
those looking for a simple and lightweight, yet fully-featured load testing tool
teams looking to integrate testing into existing development workflows and CI/CD pipelines

Load testing scripting tools are by no means the most important consideration for the success of load testing. Knowing why you’re testing, what the requirements are, and understanding and communicating results are all arguably more important. The right tool will enable you to address those concerns while providing as little friction as possible. There’s no unequivocal “best” tool; there’s only the right tool for your project and context.

Python for Everybody Specialization

Tue, 26 Jan 2021 23:01:18 +0100

I’m starting off the new year by learning something new. I am a big fan of non-traditional education, and Coursera is a platform I’ve used a few times before. You may not get courses that qualify as university credit, but I’ve found it to be a useful resource for learning. I might be biased (my university degree is in Economics, which I’ve never used), but in some ways, I actually prefer a MOOC (that’s a Massively Open Online Course) to a typical formal university course for a few reasons:

MOOCs are entirely online, which means it’s more accessible - I don’t need to physically be in the US to take a course at Harvard
MOOCs are usually self-paced - perfect for someone working full-time like me
MOOCs are free or cheap - no huge student loans here; MOOCs are democratizing education and making it available to more people
Good online education tends to be more practical. There is none of the intellectual posturing that I’ve seen in physical classrooms; there are no random classes you need to take to get a fancy piece of paper. I love being able to choose a topic and learn ONLY that– and get right down to business in the first lesson.
Formal education is often centered around getting students jobs, not helping workers do their jobs well. I already have a job, and I want to learn real, usable skills, not “marketable” ones.

My main complaint with MOOCs was that they tend to be short (in comparison to a formal course), but Coursera has addressed that somewhat with their specializations. A specialization is a series of courses, but it’s more than just disparate courses that are strung together; it’s a thoughtfully designed curriculum that builds up on knowledge from previous courses. Last year, I began learning Python.

Why Python? Others might have different reasons, but here were mine:

Python is one of the best languages to learn for data analysis (the other being R). I’ve always been interested in data visualization, and as a load tester, I’m frequently exposed to large quantities of data that I somehow need to make sense of.
Python is pretty good at retrieving data from websites, and I wanted to explore how good it is to use in load testing. I’m always eager to learn different ways to test, and I’ve wanted to try Locust for a while.
Python is increasing in popularity, and I wanted to see why.

So I decided to take the Python for Everybody Specialization from the University of Michigan (via Coursera) to get an introduction to this cool language. The specialization consists of five modules, each of which I had to take separately:

Programming for Everybody (Getting Started with Python)
Python Data Structure
Using Python to Access Web Data
Using Databases with Python
Capstone: Retrieving, Processing, and Visualizing Data with Python

The specialization was really fun, and the teacher, Charles Severance, is fantastic. I’d taken courses from him before, so I knew what to expect. He didn’t disappoint. He has a way of explaining daunting technical concepts in a way that’s easy to digest for beginners.

You can check out my certificate here. I’m pretty happy with my progress. I’m by no means a Python expert now, but I’m definitely eager to keep learning how to do more things with this new knowledge.

Cuándo JMeter no alcanza: Pruebas de carga simulando navegadores

Wed, 16 Dec 2020 18:18:23 +0100

Aquí está el video de mi presentación en JMeter Scripting Day, un evento organizado por JMeter en Español el 31 de Oktubre, 2020.

Y la versión audio:

Introducción

Hola a todos! Yo soy Nicole van der Hoeven, y soy performance tester. JMeter es una de mis herramientas favoritas porque es gratis, simple, y por muchos casos hace exactamente lo que quiero y no más. Sin embargo, como performers es parte de nuestro trabajo elegir la herramienta adecuada para cada proyecto, y hay que admitir que hay situaciones en que JMeter simplemente no es suficiente.

Amazon - UI + Dev Tools

Esto es el sitio web de Amazon.com. Tengo abierta la pantalla de Chrome Developer Tools, que me muestra las solicitudes (o requests) que componen la página. Cuándo vamos a la página, el navegador (Chrome en este caso) envia estas solicitudes para nosotros, pero eso occure en el fondo. Un usuario normal no las ve.

JMeter - Creando un script

Lo que JMeter hace es grabar estas solicitudes y replicarlas. Aqui vemos una ejecución de un usuario virtual de un script en JMeter. Cada solicitud tiene la respuesta y el tiempo de respuesta. Pero eso no nos da una idea completa del rendimiento de la aplicación. ¿Por qué?

Por que JMeter, como otras herramientas de nivel protocolo, tiene unas limitaciónes. La primera es que si hay javascript o otros script que deben estar ejecutado por el navegador, JMeter no los ejecuta. JMeter solo los descarga. El tiempo de respuesta mide la descarga, no la duración de la ejecución. Entonces, siempre habrá una diferencia entre los tiempos de respuesta en una prueba de JMeter y los tiempos de respuesta en práctica.

A veces esta diferencia no es tan importante. Pero hay más y más aplicaciones hoy en día que se componen principalmente de estos javascript. Por esas aplicaciones, simplemente no es posible hacer un script en JMeter porque la mayoría de la lógica de la aplicación occure en el navegador, no en el servidor.

Flood Challenge

La segunda razón es que el aumento de la seguridad de aplicaciones conduce a solicitudes más complejas que incluyen parámetros dinámicos. Estos parámetros hacen que sea más difícil para un intermediario lanzar un ataque, pero también hacen que sea más difícil para nosotros scripters, porque tenemos que extraer información desde la última respuesta y ponerla en la próxima solicitud. Es posible, pero es la pesadilla de cada performance tester.

Podemos verlo en la aplicación de muestra Flood Challenge. Ya no trabajo para Flood, pero todavía es un buen ejemplo.

Correlación en JMeter - Flood challenge script

Esto es mi script en JMeter, y vemos que hay muchos parámetros que he tenido que correlacionar. La solicitud para simular hacer clic en el botón “Start” necesita los parámetros authenticity token, step ID, y step number, que son dinámicos. El contenido de los parámetros cambian cada vez que usamos esta aplicación.

La experiencia del usuario

La tercera razón es que JMeter tampoco mide la verdadera experiencia del usuario. Solo mide la duración de la comunicación entre el navegador y el servidor, pero no mide cosas como la representación de elementos en la página web, ni el efecto de un script que dura mucho tiempo para terminar.

Eso es importante porque estas cosas afectan significativamente la experiencia del usuario. De hecho, Steve Souders, un conocido el la industría de performance, dice que hasta 90% del tiempo de respuesta que experimenta el usuario está en la interfaz. Con este numero podemos estar ciertos que una prueba con una herramienta de nivel protocolo no va a poder encapsular el rendimiento final de una aplicación.

Developer Tools - Lighthouse

Entonces, ¿qué podemos hacer?

Lo más fácil es intentar de medir la interfaz durante una prueba de cargas. Mientras ejecutamos una prueba con JMeter, podemos ir manualmente a la aplicación web en Chrome para usar Developer Tools. Podemos usar la pestaña Lighthouse, que nos da más métricos del perspectivo del usuario y sugerencias para mejorar el rendimiento.

WebPageTest

También uso WebPageTest a veces. Eso tiene más opciones para ejecutar la prueba automáticamente varias veces desde un lugar que puedo elegir. Aún podemos elegir un dispositivo para simular la experiencia de un usuario en un iPad, por ejemplo.

Herramientas alternativas para pruebas de carga

Tambien podemos elegir herramientas alternativas para ejecutar nuestras pruebas de carga. Lo que tienen en común es que fueron diseñadas para pruebas de automatización, y utilizan un navegador real.

Selenium es probablemente la más popular de esas. Selenium WebDriver en particular. Es compatible con Ruby, Java, Python, C# y Javascript. El problema con Selenium es que puede ser muy pesada. En una machina probablamente podemos ejecutar cinco usuarios de Selenium, más o menos. Eso es importante si estamos tratando de simular miles de usuarios.

Hay muchas herramientas basadas en Javascript que también podemos usar. Hay varias framework que se utilizan para adaptar Javascript y nodeJS.

Puppeteer fue desarollado por Google. Tricentis Flood también ha diseñado Element que utiliza Puppeteer, pero con un enfoque en el rendimiento.
Playwright es una creación de Microsoft.
Jest fue desarrollado por Facebook
Mocha es otro framework que es la base para Cypress.

Todos esas herramientas son open source y gratis. También hay alternativas propietarias, pero prefiero las soluciones open source porque en la mayoría de casos hacen la misma cosa y se actualizan con más frecuencia.

Element script

Esto es un ejemplo de la misma acción, hacer clic en el botón Start en la aplicación Flood Challenge.

Otra ventaja de simulando navegadores en vez de simulando las solicitudes es que es más simple crear un script. No tenemos que saber cada parámetro. Solo tenemos que describir lo que queremos que cada usuario virtual haga.

Podemos ejecutar scripts usando estas herramientas al mismo tiempo que un script de JMeter o Gatling o lo que sea, y también tenemos la opción de reemplazar los script de protocolo con estas.

Conclusión

Para resumir, JMeter y otras herramientas de nivel protocolo tiene algunas limitaciones.

No ejecutan scripts del lado del navegador
Pueden ser más complicado correlacionar parámetros dinámicos
No miden la experiencia final de un usuario

Para solucionar estos problemas, podemos complementar pruebas de carga tradicionales, de nivel protocolo, como JMeter, con herramientas simulando navegadores. De este modo podremos tener una comprensión más completa del rendimiento total de la aplicación.

Bueno, se acabó el tiempo, entonces los dejo aquí. Gracias por escuchar, y me disculpo por mi español. Gracias a todos!

Taking Notes on a Video: my YiNote to Roam Research Workflow

Tue, 15 Dec 2020 22:00:34 +0100

Listen to the audio version of this blog post here:

I was working on an article yesterday comparing the performance of k6 with JMeter, and I had a chat to Mihail, one of the devs of k6. He mentioned that a large part of why k6 is so performant compared to other tools (particularly Java-based ones) is that it’s written in Go. He talked a bit about Goroutines, and when I told him I was a Go noob, he recommended this very excellent video by Kavya Joshi, which explains a bit more about the Go Scheduler and how it manages to achieve performance for Go apps that exceeds the typical 1 Virtual User: 1 kernel thread paradigm that many load testing tools (like JMeter) use.

Anyway, I got around to watching the video today, and because it was a video I knew I’d learn from, I took notes - and ended up improving part of the workflow (more on that later). Other people have commented on my notes before, so I thought I’d share how I process videos into my personal wiki, currently held in Roam Research.

Install YiNote Chrome extension

I use a free app called YiNote to take video notes in particular. I use their Chrome extension.

Take Notes

I was pretty dubious about using an app just to take notes for video specifically, but there were a few things that sold me:

When you click on the text field to take notes in the extension, YiNote automatically pauses the video so that you can type without missing what’s being said.
YiNote also automatically saves the timestamps associated with notes you create, so that you can click on it afterwards to jump to the specific section that you wrote a note on.
YiNote has an export to Markdown option - great compatibility with Roam or Obsidian
You can also export your notes to a PDF, which is great for sharing. It saves screenshots of the presentation and puts your notes next to the relevant screenshot.

Here’s what it looks like while taking notes:

Export YiNote notes

After watching the video, I click the Settings wheel from the Chrome extension panel.

I click the Export as Markdown button.

That lets me save a .md file to my local drive. I always save YiNote files in the same folder as the script below.

Run reformatter script

This is the part that I tweaked today. YiNote’s exported Markdown file is formatted in a way that doesn’t really play well with Roam. Here’s what it looks like, fresh from YiNote:

<!-- Generated by <a href="https://www.yinote.co/#installation">YiNote</a> -->

<!-- # [GopherCon 2018: Kavya Joshi - The Scheduler Saga](https://www.youtube.com/watch?v=YHRO5WQGh0k) -->

<!-- ## [1:19](https://www.youtube.com/watch?v=YHRO5WQGh0k&t=79) -->

The scheduler is what makes sure the Goroutines run concurrently.

<!-- ## [1:29](https://www.youtube.com/watch?v=YHRO5WQGh0k&t=89) -->

All the information is there, but importing that to Roam, with the comments, isn’t really useful. I want to see those timestamps as clickable links, and I want my notes to be nested underneath the corresponding timestamp so I know what belongs where.

So I dusted off my Python skills tonight and wrote up this little script to do just that. Warning: I am not a developer; I just occasionally code. So I’m sure that this isn’t the most concise way to do it (contact me and let me know how it could have been better if you have any ideas). But it’s functional. And, in the spirit of learning in public, here it is:

The script results in an output file that is better formatted for Roam.

Import resulting file to Roam

On Roam, I hit Options > Import Files, select the output file, hit OK, and that creates a new page in a way better format than the one directly from YiNote. After adding the date, author, and “Recommended By” fields, this is what it looks like in Roam:

That’s it!

I don’t do this for every video I watch, but I find this process useful for more technical videos that I really want to learn from. It’s really important to me to get all my notes from disparate sources into Roam, so that they then get processed like any other note into my Zettelkasten - but that’s a process for another day.

How I use Roam Research for D&D: Session Notes

Mon, 14 Dec 2020 20:13:13 +0100

In case you thought this site was all about performance and load testing, surprise! It’s not.

If you’ve talked to me personally for longer than ten minutes in the last year, there’s a pretty good chance I’ve mentioned Roam Research, or the note-taking and learning methodology called Zettelkasten. Sorrynotsorry. I’ve been obsessed since I discovered it.

To say Roam Research is a note-taking device is like saying Bitcoin is currency: yes, it IS that, but also so much more. To me, it’s my second brain. It’s a record of what I know and what I want to know, and sometimes even things I don’t know yet. The real reason that Roam and the Zettelkasten idea resonated with me so much is that I can use it a lot for my work. But I’ve discovered another use case that Roam is perfect for: Dungeons & Dragons.

This became longer than I thought it would be, so I’ll just talk about how I handle Session Notes in particular.

Starting a new session

Everything starts with sitting down to play D&D. I write a log of events and information that comes out while I’m playing. Here’s what a new session log looks like:

I preface pages in this campaign with D&D/ to make it a separate namespace. A namespace is like a folder, and I use it mainly because I have two D&D campaigns, both of which would have the page Campaign, Characters, Locations, etc, even though the two campaigns are on different worlds. To keep them separate, and to keep from getting mixed up (Was Gloomtoes an NPC in this campaign or that one?) I use different namespaces for each campaign.

I add the real-world date in the page name, and then an in-game date reference. The in-game date reference is useful for keeping track of time since I might go a couple weeks (okay, maybe one week; let’s not get too crazy) without playing this campaign, but no time in the in-game world has elapsed. Then, I end the page name with a descriptive title, which I always end up changing after the session. Sometimes I’ll title a page Exploring Castle Never, but then we end up getting carried away in another part of the city instead. That’s okay - when I change a page name, Roam then updates all references to that page that I’ve made from other pages.

I don’t type all this out manually - I use a TextExpander template so that to start a new session, all I have to do is create a page in Roam, type ;s.dnd and that is automatically replaced with prompts for a few fields.

Date: This is the real-world date of when I played this particular session.
Tags: When I first create session notes, this is prepopulated with #TVZ, which stands for To Verzettelen. It’s just my personal way of marking a note as one that needs to be processed.
Location: Here I note down the in-game places that my character visited this session. Usually I’ll just put city names, but I sometimes also split locations up into districts within a city or rooms within a dungeon.
Characters: I’ve found it helpful to keep track of which characters (players and NPCs) I’ve come across in this session.
Session Summary: This is a section I leave blank when the session is initially created.

Filling in the session log

The rest of the page, after Session Summary, is where I’ll spend most of the time writing. I just write whatever comes to mind and whatever I have time to write, in bullet form. I don’t worry about formatting or grammar or linking to other pages. I just type all my thoughts out in plain text.

HP

I like to play healers, which means I care a lot about how everyone in the party is doing so that I can figure out who best needs healing. {{[[table]]}} in Roam lets me create a custom table, and I have a ;hp snippet in TextExpander that recreates this table, with all the names of the people currently in my party, so that I just worry about filling in the values. Sometimes I still get it wrong, or miss a bit of damage that the DM calls out, but it’s enough to give me an idea of who needs heals.

I also write down initiative orders, the defenses/vulnerabilities/immunities of monsters we face, descriptions of NPCs we meet, suspicions or questions I might have… basically anything and everything that comes to mind. Session time is pretty much a brain dump, with no formatting.

Equipment

If I acquire a new item during a session, I’ll tag that line with #[[D&D/Carrying]]. This lets me stay on the Session Notes page while still maintaining an up-to-date record of what I’m carrying.

A tag in Roam is the same as a page, so when I click on that link, it takes me to the D&D/Carrying page, which conveniently also shows me all the references to that page from other pages.

Roam shows me not just the page, but the exact block where I made the reference to D&D/Carrying, so it’s easy to see at a glance what I have and where I got it.

Next Actions and Waiting For

I use David Allen’s GTD methodology in D&D too. When I want to remember to do something later, I tag it with #[[D&D/Agenda]] and #[[Next Actions]] or #[[Waiting For]]. This presents me with the same opportunities as D&D/Carrying, where I can see references to those pages later as well.

Processing the session

Sometime after the session, I’ll go and fill in the metadata from my template, including the session summary, which is just a paragraph or two describing what happened in the session at a high level. This is mainly to jog my memory in the next session, when I’ve already forgotten what we did in the previous one.

Here’s what it looks like, all filled in:

Adding links and fleshing out pages

I then go back through the notes themselves and add turn any people, places, or interesting objects that I mention into links. In Roam, that’s a matter of highlighting them and hitting [[. Then, I click on the page I’ve just created and try to add information that I’ve learned. This isn’t strictly necessary, since all mentions of that page will show up as a linked reference, but it’s an opportunity to consolidate my information on that subject and see how it fits into what else I’ve written there.

Here’s a page for an NPC:

Having these links means that when an NPC says “I’m from Leilon”, I can type Leilon into Roam:

… and say, “Oh, Leilon. I heard you had a dragon problem there recently. Did your family get out okay?” Boom! Instant advantage on a Diplomacy check.

Sometimes we’re in a dungeon with many rooms, and this is the point where I draw a quick map on my iPad Pro and Apple Pencil using Notability, and then I upload it right from my iPad to Roam.

This map is in my page for Castle Never:

The little blue “8” on the upper left of the map is versions. Information rarely comes all at once in D&D; it usually comes in trickles. As I learned more about the castle, I revised the map and added a new version to it, so that I can still see previous versions and what I thought the map was like when we first explored it.

At this stage, I also add any tags I forgot to add during the session.

Reviewing quests and questions

After I go through the session notes, I go to the page called D&D/Quests and Questions, where I keep a running list of all the open quest hooks and any unanswered questions I might have.

Not everything on this page is meant to be resolved, but I find it comforting to track everything anyway, in case I happen to find myself in a city with nothing in particular to do. I mark these with a checkbox ({{[[TODO]]}}) in Roam) and tick the box when one is resolved. Roam lets me filter out the resolved quests so I only see things that are outstanding.

This is where I get the most value out of taking notes. It’s at this point where I’ll add our progress towards quests, make connections between two seemingly disconnected quests, and situate what I’ve just learned in the session within the broader context of the world and the campaign. I also review and update items on the Next Actions and Waiting For.

This is the part that takes the longest, but it’s also what most prepares me for when I get to the table again next week.

Tying it all together

I use a D&D/Campaign page as my landing page for this campaign. It includes D&D/Quests and Questions as well as a list of all sessions to date, so that I always know what happened and when.

Why do I take notes for a game?

You might be wondering why I go to so much trouble. Why do I take complicated notes about something I do “just” for fun?

Taking notes helps me offload this all safely from my brain.
I take my fun as seriously as I take my work.

Week of Testing: Intro

Mon, 23 Nov 2020 20:45:31 +0100

Today was my first day at k6.io! I’m happy to announce that I’ve joined as their newest Developer Advocate for Performance Testing. k6 is a company I’ve had my eye on for a while, even (or maybe especially) when I was working for a competitor, Tricentis Flood. I always admired k6’s stance on open source, their involvement with the developer community, and the slick web app that brings load testing into the hands of many developers and testers.

As a load tester, I try to be tool-agnostic. I have to admit, though, that I do have a strong bias towards opensource tools. I joined Flood because of their focus on tools I already loved like JMeter, Gatling, and Element (based on Puppeteer). (And Selenium, but it’s a stretch to say I love Selenium.) But these are tools I’ve used for years, most of them from even before I joined Flood. What kind of load tester would I be if I could only really talk about scripting in a handful of tools?

So I thought it was time for a change, and k6 was the first and only company on my list for a couple reasons:

The free and open source tool k6, which recently crossed over to 10,000 stars on Github, is often touted as the developer’s load testing tool of choice.
k6’s cloud offering integrates seamlessly with the OSS tool, and it adds some killer features like performance insights that add real value to teams wanting to scale their load tests.
The k6 team is incredibly responsive to the needs of their community. Unlike most companies, where only marketing, sales, and customer support people talk to customers, k6 developers are active on GitHub, social media, and other blog sites. Everyone is passionate about what they’re building.
Roles in testing are changing, and the lines between tester and developer are blurring. Developers who don’t test are going to become just as obsolete as testers who don’t develop. I see k6’s popularity among the developer crowd as a good opportunity for me to sharpen my coding skills.

So I had to see for myself whether k6 was actually as good as it seemed.

Luckily, it turns out that k6 has a tradition of a “Week of Testing”, where every new member of the k6 team, regardless of role, spends their first week putting the product through its paces. Every part of the whole k6 solution (Cloud and OSS tool) is fair game, and the goal of the Week of Testing is to present feedback to the team at the end of it, while the new member’s eyes are fresh. When they told me about it, they said “We want to see you learn the tool on your own to see what parts are difficult to figure out.” To which I responded, “Challenge accepted!”

I decided to not only give a presentation at the end, but also to video the process for posterity’s sake… and, let’s face it, to impress my new managers and coworkers with my dazzling work ethic.

Step 1: Get the company’s name right.

In the first video, I share the differences between k6 vs. k6 Cloud vs. Load Impact!

How to Become a Load Tester

Wed, 07 Oct 2020 14:15:45 +0100

Originally posted here.

Load testing can be a pretty intimidating field to get into, but it’s easy to get started. Here are some concrete steps you can take to start learning how to load test from scratch.

In this video, Flood co-founder Tim Koopmans and I teamed up at Tricentis User Conference 2020 to talk about the essential ingredients for load testing.

Background required for load testing

Load testing can be quite technical. It involves not just the ability to use different programming languages to create load testing scripts but also familiarity with software engineering practices, DevOps, application infrastructure, site reliability, and networking. A good load tester needs to be able to understand the basics of how an application works and how different components interact with each other. It’s common for a load tester to work on applications employing a variety of technologies, so staying up to date with recent technological trends and switching seamlessly from one language to another, or from one platform to another, is of utmost importantance. This technical side of load testing might make you think that a load tester needs to have formal studies in computer science and several years of experience in software testing. The reality is that there are many load testers who stumbled into the field from wildly unexpected backgrounds. So you might think load testing looks like this:

‍ Photo by Christina @ wocintechchat.com on Unsplash

And sometimes it does. But it more often looks something like this:

‍ Photo by You X Ventures on Unsplash

You see, load testing is an interdisciplinary field. So much of the job involves talking to people. Sometimes the requirements for a project are not clear, and someone needs to talk to the both the business and the developers to make sure that nonfunctional requirements are clarified. A load tester is a detective, tracing a request as it goes from one application component to another, coordinating with different application teams. A load tester is a scientist, using the scientific method to design experiments that yield the results. A load tester is a statistician, making sense of a jumble of data and seeking correlations between variables. A load tester is a public speaker. Even when the actual load testing has been completed and the report has been sent, a load tester makes sure all stakeholders understand the report so that the results can be used to make meaningful changes to improve application performance. All these tasks require a mishmash of skills and competencies that can be difficult to find in just the field of computer science.

Let’s take our backgrounds as anecdotal examples.

Tim grew up and currently lives in a small country town in Australia, and he joined the military straight after school. He later developed a love of automating things, which led him first to use tools like perl to scrape websites and try to win bids with early versions of ebay, and then later to starting a performance engineering consulting career and co-founding Flood itself.

I, Nicole, grew up in Metro Manila in the Philippines (which has 14,000 more people per km2 than Mumbai does). I learned English as a foreign language in school, and I have a university degree in Economics. When I discovered load testing, I had a job in a completely different field– I was working for an airline and making sure engine components were properly maintained and repaired. My love for gadgets and technology took me down an 8-year rabbit hole involving doing performance consulting for clients across a variety of industries in Australia and the Netherlands. At Flood, I help customers around the world improve their application performance.

As you can see, Tim and I came from very different backgrounds and different parts of the world. Neither of us knew what load testing even was when we started working. Neither of us got formal instruction in load testing. But somehow, we still both ended up becoming load testers at Flood.

Reasons to become a load tester

So why become a load tester? Here are some practical reasons.

Load testing is cool

A little-known fact about load testing is that it can be really satisfying to be involved in big-name projects that end up going live.

You may have heard of Flood. But even if you haven’t, we’re sure you’re familiar with the results of load testing done on Flood. In the last few years, we have been helping Hotstar, one of the largest streaming services in the world, load test their systems in preparation for the Indian Premier League.

Hotstar uses an open-source tool that we support, Gatling, to create scripts to simulate users tuning in to watch cricket games. Then, they use Flood to create a “Game Day” simulation. They scale up those load tests and run 5 million users on thousands of machines in the cloud. We help them coordinate execution, spot problems quickly with real-time results from all those machines while the test is still running, and identify bottlenecks so that they don’t occur in production, when India’s playing.

As a load tester, you’ll definitely be working with brands you recognize and love.

Load testing can pay well

‍ Source: Payscale

I said this would be a practical list, right?

Here are 261,768 practical reasons. The average performance engineer’s salary is 71% more than that of an average software tester in India, according to Payscale.com. We all know that money isn’t everything, but this is a clear signal from the market about the importance of performance-related skills. Adding load testing to your CV as a tester makes you more attractive to employers. It’s a great way to differentiate yourself in an over-saturated market.

Load testing can literally take you places

‍ Source: Shamim from Flood

Then there are the side benefits of being in load testing. The photos above are real ones from the Flood team. That’s me on the upper left, exploring a castle in Belgrade, Serbia, and that’s Tim in the bottom right, taking a photo in San Francisco, USA.

We’re a fully distributed team on Flood, which means we can work from anywhere. Flood is a web app, so you don’t need to install anything– I’ve been known to start load tests from my iPad in a café. Load testing on the cloud frees you from being tied to running tests from the office. You can start load generators when you need them and shut them down when you don’t– all from a web browser. It’s a job that can come with a great work/life balance, if you find the right company.

Load testing can also be a great way to move to another country. It’s what’s allowed me to move to other countries– twice. In Australia, the average performance engineer’s salary is AU$108,053, or over 5.6 million rupee.

‍ Source: Payscale

So it’s clear that there are some pretty good reasons for you to become a load tester. I’m going to walk you through the practical steps to become a load tester. You may not be an expert by the end of this, but I hope I can convince you that you can get started in load testing whether your background looks like Tim’s or mine.

Load testing adds real value

And the trite, but absolutely true reason: Everyone fails.

As a load tester, you are a gatekeeper of quality. What you do directly affects revenue and customer experience.

Performance is an aspect of software that is often overlooked, but it’s only becoming more important in a world where everything is going online. People are being born who have never known what it’s like not to have the internet, and consumers in general expect applications to be responsive and stable. Would-be customers who can’t get an e-commerce site to load within 3 seconds may switch to a competitor’s site, and we all know there are many of them. Performance-related outages can be seriously damaging to a company’s reputation, and social networking amplifies customer dissatisfaction by effectively giving everyone a megaphone to complain into.

Software functionality is still important. However, if an online store goes live with no critical functional defects, but it fails when the 100th person logs into it, or it takes 10 seconds to load the homepage, can you really say that it was ready for production?

Load testing is a crucial part of ensuring software quality, and quality is the responsibility of everyone on a team, even if the word “performance” isn’t in your job title.

The essential ingredients of a load test

Load testing is like cooking a meal. This is one of Tim’s favorite dishes, aloo matar.

‍ Source: Cubes n Juliennes

You can approach cooking as a science. There are recipes with a list of ingredients, a precise measurement of how much you need for each, and a procedure for creating the dish.

But you can also look at cooking as an art. Cooking the same dish over and over again will give you an experience in working with those ingredients that a novice cook won’t have. There’s room in cooking for adding one ingredient, changing the quantity of another, or just trying something new.

A master chef knows how to use both the science and the art to create something delicious. But you don’t HAVE to be at that level in order to cook.

Anyone can cook. If you know how to follow a basic recipe, you may not create a masterpiece, but if you’re like me and you haven’t had lunch yet, you’ll still end up with something you’ll eat.

So here’s the recipe for load testing.

I prefer kadai paneer myself, but aloo matar requires, among other things, peas. And so does load testing.

P is Preparation. To prepare for a load test, you need to get your ingredients ready. You need to decide what it is you want to test, why you’re testing it, and how. You write load testing scripts that will simulate real users as closely as possible.

E is Execution. This is when you do the actual cooking. You run your load tests and scale them up according to your requirements.

A is Analysis. You figure out whether the dish you created is edible. You determine how your application servers responded during the load test, and decide if that’s acceptable.

Preparation – Execution – Analysis.

Preparing for a load test

“As far as the World Cup is concerned, it is a process. We don’t want to jump to the 50th floor straight away. We must start on the ground floor.” - Sachin Tendulkar, via ESPN

Load testing doesn’t start with writing a script. Load testing starts with building a foundation for later tests.

‍ Source: Inside Sport

Part of that foundation is asking WHY. Why do you want to do a load test? Has there been a performance-related issue before? Are you trying to determine whether an application is ready to go live, or is it already live, and you just want to improve its performance? What type of application do you want to test?

WHOM are you cooking for? I think you’d cook differently for yourself than you would for Sachin Tendulkar. What do your end users typically do on your site? Where do they come from?

These are questions that need to be answered before you think about scripting. Here are some resource for planning your load tests:

Start with Why: How to write requirements for API load testing How to write a test plan for load testing Selecting a tool for load testing Generating a load testing script without coding: Getting Started with Test Builder

Execution of a load test

So now we’re ready to start cooking. This is the Execution phase. You’ll need a machine to use as a load generator. Depending on the number of users you want to run with, you may need more than one machine. You might be tempted to run a load test on your laptop, or on a bunch of machines you have lying around, but that introduces new variables that may affect the results.

‍ Source: Ashwini Chaudhary on Unsplash

Like any good cook, you’ll also need to make sure you have all your ingredients handy. In load testing, that means making sure your scripts and test data files are ready to be executed on every load generator.

Both of these problems – provisioning a load generator and transferring files over to them – are solved pretty easily by running on the cloud using something like Flood.

Designing and executing your load test

Where should you run your load tests? A look at demand, hosted, and on-premise infrastructure

Analysis of Results

The last part is Analysis.

‍ Source: Fine Dining Lovers

This is a pretty crucial moment, because this is when you figure out whether what you cooked was delicious or not. Unfortunately, in load testing, it’s not always easy to tell how your test went. It’s like this photo of pakora (onion is my favorite kind). It’s kind of hard to say what expression that is in their faces. Is it disgust? Or is the pakora so yummy that they just can’t wait to eat more?

Load testing typically generates a LOT of data. You’d have to go copy every single file from every load generator you used. But can you imagine the hundreds of load generators that Hotstar used? Of course they didn’t manually copy those files. Instead, they used Flood.

On Flood, we show real-time graphs as you’re running the test so that you can tell immediately whether transactions are failing or whether a server has stopped responding.

After the test, you can also compare this data to previous tests to spot trends, or even create your own dashboards using an analytics engine.

Real-time metrics during your load test

Downloading results and logs after the test

Is that it?

You’ve prepared for a load test, you’ve executed a load test, and you’ve analysed the results from the load test. Congratulations, you’re a load tester! By following this recipe, you’re already doing better than many people, and you’re certainly doing a lot better than those who do not run any load tests at all.

The cool thing about load testing, though, is that this is just the beginning. There’s always room for improvement.

‍ Source: Pop Up City

Dabbawallas are people who deliver hot lunches, and while the concept is not new, the implementation in India is particularly interesting.

For these dabbawallas, having the meals prepared is not the end of the job. Once the food is ready, it needs to be packed into lunchboxes, picked up, separated according to destination, put on trains, and delivered. Later, the empty lunchboxes are picked up, and the system works in reverse.

It turns out that this is actually a pretty sophisticated system. In fact, it’s been estimated that only one tiffin every two months is not delivered. That’s equivalent to a Six-Sigma quality rating of 99.9999999% (that’s nine nines) or more.

Dabbawallas didn’t get that rating just by delivering quickly. They did it by being consistent and reliable. They did it by having a process.

Once you see the benefits of running a load test, you may also start to understand that the real value of load testing is not in a test that you run once whenever you think of it. Imagine having the benefits of a load test tied to your development process, so that it is automated.

As an advanced topic, adding your load testing to your team’s Continuous Integration/Continuous Development pipelines such as sets up a repeatable process that reaps benefits far into the future, not just whenever official load testing activities are carried out.

Continuous Integration with Flood: Integrating Travis CI, Circle CI, and Jenkins

Conclusion

So that’s the recipe for becoming a load tester.

Preparation – Execution – Analysis.

In Preparation, you determine why you’re testing and what tool best fits your needs. We showed you how to use Element, but there are many open source tools out there. Don’t get overwhelmed by scripting– a lot of times it can just mean replacing a URL with the one YOU want to test.

In Execution, you run a baseline test that will serve as a point of comparison for future testing, choose the number of users and the number of regions you want, and then you run your test. Running on the cloud is the easiest (and usually cheapest) way to get started.

In Analysis, you view the results in real-time, compare historical performance to see trends, and share reports to stakeholders so that your team can make changes to your application.

We also talked about the bonus topic of CI/CD and how you can incorporate load testing into your pipeline to get a continuous feedback during your development cycle.

Tim and I started in very different places and had different experiences before we discovered load testing. But eventually, we ended up on the same path, and that’s what we’ve outlined for you here. We hope that this convinces you that load testing doesn’t have to be complicated.

You absolutely don’t need to be an expert to run a load test. Even if you don’t think you’ll get all these steps right, a load test that is less than ideal is still better than no load testing at all.

In the end, anyone can be a load tester with the right attitude and the right tools.

Learning about load testing

High Performance Websites book by Steve Souders

Ministry of Testing/PerfBytes list of resources

PerfBytes podcast (EN and ES)

PerfGuild conferences by Joe Colantonio

Señor Performo’s YouTube Channel: Señor Performo ENG

Scott Moore’s Performance Tour on YouTube

How to Write a Load Test Plan

Mon, 21 Sep 2020 16:33:23 +0100

Originally posted here.

People often start load testing by creating a script in their favorite load testing tool, but good load testing starts before that. Planning a load test may seem like a tedious exercise, but it is essential to ensuring the success of your testing.

A test plan should answer why, what, who, when, and how the testing will be carried out.

A good test plan, particularly for load testing, includes the following components:

Requirements
Scope
Entry criteria
Workload modeling
Server monitoring
Test doubltes
Test Scenarios

We’ll dive into each of these, but first, a note on the format of a test plan.

Test Plan Format

Test plans have been traditionally presented as a document, but there’s no hard-and-fast rule. Instead, the length of a test plan, and the level of detail it goes into, should be tailored for the complexity of the project and the backgrounds of the stakeholders.

Teams using Agile methodologies may not have a formal test plan at all, but I would argue that all teams should still have some sort of plan, whether it’s a Word document or a list of bullet points. Written plans clarify and communicate intent, and testing can be an expensive and fruitless exercise without a consensus on its intent.

Technical documentation can be difficult to digest and understand, and that’s dangerous when it comes to software development and testing, where it’s imperative that all stakeholders are on the same page about the state of an application.

Paul McLean, a performance engineer, has a creative solution for this. He creates a companion video for every technical report, giving stakeholders the option to read the written report for details but watch the video for a more high-level and nuanced explanation.

Here’s an example of a companion video he created for a test plan:

Requirements

Everything starts with requirements. Until requirements are identified, load testing can only ever be aimless and explanatory, and any performance bottlenecks spotted will be incidental.

Requirements inform every step of the load testing process. Why are we doing load testing? What exactly do we want to test? How will we know when a test has passed or failed? How will we know if application performance is good enough to go into production? What does “good enough” mean?

This is such an important part of the planning process that I felt it deserved its own blog post. See it here: Start with Why: How to Write Requirements for API Load Testing. Even though I specifically mention API load testing, the principles apply to load testing in general.

Scope

At Flood, we use a product development methodology called Shape Up to keep us on track as we build new features. Projects in Shape Up are called “shaped work”, and one of the basic properties of shaped work is that it is bounded.

…shaped work [also] indicates what not to do. It tells the team where to stop. There’s a specific appetite—the amount of time the team is allowed to spend on the project. Completing the project within that fixed amount of time requires limiting the scope and leaving specific things out.- Ryan Singer, Shape Up: Stop Running in Circles and Ship Work that Matters

Defining scope is setting boundaries. Business priorities need to be weighed against the resource limitations (number of people available to do the work and time available) in order for testing to deliver maximum value.

Some considerations for scope include:

Specific features or key transactions to be tested
Types of tests included (component test vs end-to-end test)
Test scenarios (peak load test vs disaster recovery)
Applications included in testing

As with most things in the planning phase, scope is something that can change during the test when unexpected circumstances arise or when priorities change. But it’s still a good practice to define the scope at the beginning, and update it as it changes.

It’s important, however, to prune with intention. By keeping our objectives in mind, we can keep our work in line with our intentions.

Entry criteria

Occasionally, you may be asked to load test an application that isn’t ready to be load tested. This happens more often than you think. While it’s true that Performance Testing is something that should be baked into all software development from the beginning, this doesn’t hold true for Load Testing. Load testing is only one activity that falls underneath the broader umbrella of performance testing.

Entry criteria are conditions that you need to be fulfilled before the testing actually begins. It’s a good idea to have these conditions communicated beforehand so that everyone is clear on what needs to be set up before you can do your job.

Functional testing: Does the application work?

Load testing cannot realistically be carried out until at least the core functionality has been tested and high-severity defects have been fixed. Depending on the kind of load test you want to execute, you may also want to specify that UAT has been executed, as there’s no point doing an end-to-end load test with 1000 users if it doesn’t work for one user.

Load Testing Environment: Is it production-like?

Nonfunctional testing has stricter requirements for an environment than does functional testing, and you may have to champion this cause. For load testing, it is not enough to have an application staging environment that is a virtual machine that is a quarter of the size of the production environment. It’s important to get as close to a production-like environment as possible in terms of capacity (memory, CPU), codebase (the actual build that will be deployed), and integrations with other environments or servers (if within test scope).

Load testing is not linear: a response time of 5 seconds on a server with half the capacity of the production server does not necessarily equate to a response time of 2.5 seconds in production.

This is also the time to think about your load injectors. Will they be on-premises, or in the cloud? A good entry criterion is the availability of the machines in the right network and with the right tools installed. If you’re using commercial tools, license provisioning should be a criterion. What sort of capacity will your load testing scripts require?

Support: Are there people available with expertise in key components?

Load testing is a team activity. When a load test involves multiple application teams, it’s important to request availability of key persons on those teams during the test. Often as load testers, we are seen as working independently, but the truth couldn’t be farther from that. Load testing is a team sport. We need support from:

business analysts who will be able to tell us how things are expected to work and what the current priorities are

developers whom we can consult when poorly performing code needs to be optimised
functional testers who can show us how the application works
DevOps engineers who can help us provision and monitor servers

and many more!

‍

Test Data: Do you have dummy data for your virtual users to interact with?

Once you have as close a copy of the production environment as possible, keep in mind that it’s still a clean copy, which may not be realistic. If there are databases in production, how much data do they contain? The application server may respond differently when your test database is empty compared to when it must contend with gigabytes of data in the production database.

Using production data for testing purposes can be dangerous, not to mention illegal, in certain circumstances. To avoid this, you can scrub sensitive data or generate your own, either by injecting records into a database or by writing an automation script to create data on the front-end. If you write a script, you may be able to reuse parts of it later for your load test.

Workload Modeling

A workload model is a schema describing the load profile for a given test scenario, and it involves determining what (the key transactions), how much (the load distribution among the transactions) and when (timing of the load) to test.

Workload modeling can be the most difficult part of the testing process because it involves finding out how load test scripting can best mimic what is actually happening in production. It can also be the most critical.

Here’s my account of working on a project in horse racing, where workload modeling was of particular importance: Workload Modeling - Preparing for Large Events like the Melbourne Cup.

‍ When you’re selling bets for the Melbourne Cup, every second matters.

Server monitoring

Executing a load test without monitoring server health is like flying blind. You’ll know when you land safely and you’ll know when you crash, but even if you do crash, you won’t know why—or how you can avoid it next time. Monitoring server health is the black box that will tell you what went wrong.

For load testing, you’ll want to monitor the application servers that you’re testing as well as the load generators that you’re using to run the load tests themselves. That’s right; if you’re not watching carefully, the machines you run the load tests on can be the bottlenecks in themselves, causing unnecessary failures in your load tests.

Joe Colantonio goes over the basics of resource utilization in terms of four main areas of concern:

CPU
Memory
Disk
Network

Here’s another guide I put together on the basic metrics as well as some tools you can use to set up your monitoring.

Regardless of which metrics you identify as key or which tools you use, you’ll want to make sure that they’re all set up to measure server health before you run your load tests.

Test doubles

Part of the strategic planning around load testing is deciding which components need to be tested in conjunction with your stated requirements. Applications can sometimes be complex enough that load testing end-to-end is not feasible due to the number of teams involved or the cost of duplicating application infrastructure.

Keeping it as simple as possible will lead to fewer dependencies, so it’s worth considering creating test doubles to isolate relevant components. Test doubles include stubs, mocks, and full virtual services.

‍

A test double is that part that replaces a complicated component that is not within scope. It’s a “dumber” version that responds to requests enough to allow you to go on with your load testing without actually requiring that component.

Test doubles allow us to abstract out other components we don’t want to test so that we can concentrate on the one that we do. The point of a test double is to remove the variables and noise introduced by other parts of the application so that we can focus on testing how one component responds.

If you are willing to put the time into creating a test double, you can drastically reduce the amount of resources you need to set up an environment and isolate components. Reducing variables in your test allows you to more quickly determine where performance bottlenecks lie.

Test Scenarios

A test scenario is the description of a contained situation or condition under which the application will be tested. A test scenario is usually based on several test cases and includes a plan for how these test cases will be executed.

Choosing your test scenarios means deciding which situation is most likely to yield the data that you require. Employing several different types of scenarios will give you a greater understanding of your application’s capabilities. You should feel free to create your own scenarios that are uniquely tailored to your requirements, but here are some common scenarios to start out with. Take the number of users and durations mentioned as guidelines and not rules.

Here’s what our Flooder amigo, Señor Performo, has to say about test scenarios:

Text version here

Conclusion

Creating a test plan can seem daunting, but the process of writing it serves as a prompt for discussions among your team.

Here’s a pro tip from Scott Moore: Once you come up with a template you’re happy with, save it and reuse it for future projects. Modify it over time based on feedback to see progressive improvement.

Building a set of reusable documents and other deliverables (“templates”) to use in the initial planning phase and final reporting will speed your initial startup time. I recommend working with or learning from others with experience to build out some of these templates.

‍When in doubt, remember that the format of a test plan is not as important as making sure it is meaningful and comprehensible for your team.

‍For more information about the load testing process, check out my book on API load testing.

Pros and Cons of API load testing

Tue, 18 Aug 2020 21:22:49 +0100

Originally posted here.

‍API load testing is one of the most popular ways to run a load test because of the number of open source and commercial load testing tools available on the market that support it. Creating a script to hit your application’s API endpoints is the traditional approach to load testing, but it’s not the only one.

Here are some things to consider when deciding whether or not API load testing is right for your application.

Disadvantages of API load testing

API load testing does not simulate real users interacting with elements of your webpage.

‌In contrast to a customer opening up a browser and filling out forms on your site, an API load testing script will consist only of the underlying requests to the server that are made by clicking on those on-screen elements. Exactly how the customer triggers those requests, and what buttons they’ve pressed, is irrelevant. It’s all about the raw requests.‌

The format of these requests will be dictated by the type of API your application uses, and it may take some technical knowledge or expertise to be able to formulate requests correctly. However, a RESTful API is one of the most common types, and it is considered to be relatively simple to use.

HTTPS requests for amazon.com. This is what an API load testing script would need to replicate.

It doesn’t give you an idea of how user-friendly your application is.

‌API load testing tools don’t give you feedback on how long your pictures took to render on your users’ browsers or whether that “SUBMIT” button is in an obvious spot.

‌It doesn’t measure front-end performance or how quickly pages render in different browsers.

‌While all the resources that the server returns can be downloaded by your tool of choice, there is no browser on which to run them.

‌API load testing doesn’t run client-side scripts.

‌It isn’t what you’re looking for if your application is a single-page web app that relies heavily on JavaScript or AJAX to dynamically populate and update the page. Your load testing tool will download the scripts, but not execute them.‌

‌This includes scripts like those required to trigger Google Analytics, which prompts many an engineer to question the results of an API load test due to the inability to see the traffic come in on Google Analytics. API load testing won’t help you with that. Running browser-level load tests using tools like Flood Element or Selenium may be more useful.

If you’ve decided that API load testing is not the best way to load test your application, you can look at browser-based load testing as an alternative.

Executing a browser-based load testing script using Flood Element. ‌

‌Advantages of API load testing

Now that we’ve talked about what API load testing isn’t good at, let’s talk about what it IS good at.

‌### It allows you to load test specific servers rather than the whole stack.

‌API load testing allows you to tailor your load testing by applying load only on particular servers. This is particularly useful in more complicated applications that involve several components and would require substantial effort to reproduce in a test environment. Using APIs, you can test only the functions you want to test.

It is well-supported. ‌

API load testing has been around for decades. There are lots of robust tools you can choose from, both commercial and open-source, and many of these tools have large communities and extensive documentation around how to script the most common cases. It’s a far cry from the browser-level testing space, which is relatively new and sparsely populated by comparison.

‍ The JMeter site contains a wealth of information, including a User’s Manual ‍

It’s not as resource-intensive as browser-level testing.

You can simulate more users with API load testing than with browser-level load testing. Since most interactions on the browser-level get translated to requests anyway, generating load this way is incredibly efficient, allowing you to hit your server with requests while bypassing the UI layer and the resource overhead that that entails.

For example, at Flood we’ve baselined some of the tools that we support to see how many users we can run on an AWS m5.xlarge instance:

Selenium: 5 users Flood Element: 20-40 users JMeter: 1000 users Gatling: 1000 users

‌You’ll see that the browser-level test tools, Selenium and Flood Element, can run significantly fewer users on the same sized node as can the protocol-level tools, JMeter and Gatling. This isn’t due to an inefficiency in the tools but rather to the fact that browser-level tools inherently need more CPU and memory to start separate instances of browsers per user and render pages graphically. You can run more users per node (or machine) by carrying out API load testing with protocol-level tools.

It’s cheaper.

‌The efficiency in resource utilization translates directly into cost savings because every node that you don’t need to execute your tests on is one fewer node that you’ll have to pay to provision (whether on premises or in the cloud).

‌For this reason, API load testing is one of the most cost-efficient ways you can get started with load testing, allowing you to scale up your load relatively cheaply while getting immediate results.

How to get started with API load testing

API load testing isn’t for every application, but depending on your test scenario, it may be the easiest way to test application performance.

If you’re new to non-functional testing or API load testing in particular and would like some step-by-step tutorials on how to get started, check out my free ebook, API Load Testing: A Beginner’s Guide. It takes you through the basics of API load testing, including planning your first load test, selecting an open source tool, scripting, executing a test on the cloud, analyzing results, and even integrating it into your CI/CD pipelines.

Performance testing the Dutch coronavirus hotline

Thu, 13 Aug 2020 22:18:04 +0100

Read the original Dutch version here. This English version was first posted here.

On June 1st, 2020, the National Institute of Health and Environment (RIVM) announced that everyone with certain symptoms could call the hotline 0800-1202 to arrange a free COVID-19 test. The promise: call today, get tested tomorrow, and receive the results the day after that. The reality, though, did not live up to expectations.

The opening day of the hotline was chaotic, and many Dutch people reported connection problems due to a system overload. Yet this could have been prevented with a better understanding of how human psychology can affect application performance.

The situation

According to Dr. Ann Vossen, chairperson of the task force charged with scaling up mass testing in the Netherlands, the country was ready to process 30,000 tests per day. This number corresponds to the national capacity of all the municipal health clinics (GGDs) in the Netherlands. Despite this claim, though, only 1,146 people were actually tested on the first day of testing (the day after the opening of the hotline). Perhaps more importantly, only 5,748 appointments were set on June 1st. The situation was no doubt exacerbated by the overwhelming 323,000 people who called the hotline on its first day, prompting even the telephone operators to admit that they’d had troubles logging into the appointment system.

Bron: From @ggdghornl, Twitter

[“We see that there are many calls going to 0800-1202. You can call for an appointment until 8pm tonight. Please try your call later to avoid long wait times.”]

On Twitter, users shared horror stories: one claimed to have waited on the line for 6 hours, only to be told that the system was down and no appointments could be made; another complained about getting disconnected after getting in touch with an operator. Yet another posted a video of calling the hotline and a recorded message saying “Al onze medewekers zijn op dit moment in gesprek. We zullen u zo snel mogelijk te woord staan. [All of our operators are currently on other calls. We will answer you as soon as possible].”

The second day was somewhat better: there were 11,000 appointments made, partly because of the lower amount of phone calls and partly because of some fixes of technical problems. In the first week, the number of appointments increased to 50,000 (about 7,142 per day), which was still significantly lower than expected.

There are some lessons that we can learn from this case study.

Source: Coronavirus test procedure from GGD Hollands Midden

English translation:

Do you have these (mild) symptoms?:

cough

blocked or runny nose

fever

loss of smell or taste

Step 1: Make an appointment

Call the national number 0800-1202

Have your BSN with you when you call

Make an appointment to get tested at one of our locations

Step 2: Test

Go to the selected location at the scheduled time

You will be tested at the test location

Step 3: Results and eventual start of source and contact research

You will be called with your results within 48 hours

If you are infected with the coronavirus, the GGD will call you for source and contact research *If you show more serious symptoms, or if you fall within a high-risk group, report to your doctor or to the emergency help line.

The effect of fear on performance

This coronavirus has already shown us a side of humanity that we’d perhaps rather not see: fights over toilet paper, chaos on the stock market due to panic, and sabotage of 5G cell towers based on a belief that they caused COVID-19. As much as we’d like to believe it, we’re not always the most rational beings. We all have the tendency to behave unpredictably, especially when we’re afraid.

It’s no wonder that the coronavirus testing line was so welcome. The fact that there were 323,000 callers on the first day is even more remarkable when we remember that majority of Dutch residents had been in self-isolation for months at that point, which lowered their chances of contracting the virus. In theory, the fear that drove these callers might have been somewhat irrational, but not unpredictable.

In English, we call this FOMO (Fear of Missing Out). We’ve already clearly seen how FOMO can drive users en masse to an application, with effects on its performance. In this case, people were afraid that coronavirus test shortages would mean there weren’t enough tests to go around, and they wanted to be first in line to get tested.

Another problem was that one in four callers just wanted general information about COVID-19, despite government pleas to keep the hotline free for coronavirus test appointments only. It didn’t help that the hotline opened on a Dutch national holiday, White Monday, and more people were home than usual for a Monday. The high number of unrelated calls was unexpected, but nevertheless factored into system overload.

What could we have done to improve this system?

Improvements to the corona hotline

Better estimation

As difficult as it is to predict FOMO’s effects, we can still try. We can make educated guesses based on related statistics.

Just one month before the coronavirus test hotline was opened, Jaap van Dissel, the head of RIVM, estimated that there were up to 700,000 Dutch residents infected by the coronavirus. Why, then, was it such a surprise that more than 300,000 of them had called? Furthermore, the RIVM had encouraged people with a broad list of symptoms (such as colds, coughs, or fever) to call, and it’s reasonable to assume that some people with conditions other than COVID-19 might also have called.

Even if we assume that 20% of the 700,000 people with the novel coronavirus did not have symptoms (according to WHO research), we still arrive at a predicted 140,000 calls, which is still 25 times what was the coronavirus testing system was able to process on the first day.

If we have an educated estimate, we can start testing.

‍

Injecting load

When a resident calls the hotline, the call is routed to one of several operators working from home. This is often done with a telephony server that forwards calls based on some type of round-robin system. We can execute load tests on this server to determine how many phone calls a server can process and route successfully, with minimal delay and without message queueing.

Then, the operator enters personal information from the caller into the appointment system, including their government identification number (BSN) and address. We can surmise that the system checks the identity to verify the validity of the BSN, and then sends some details to a database. The closest municipal clinic to the caller can then use the data to process and schedule an appointment. We can also inject load at this stage by simulating the calls that the appointment system creates with an API load testing tool like JMeter or Gatling.

Using different test scenarios

With such a public announcement of the hotline’s opening, a spike test before release could have yielded some useful information. In a spike test, we simulate a sharp increase in users over a short period of time. For example, we could have simulated an increase in users from 0 to 700,000 on the telephony system over 10 minutes to see how the application would have handled that.

Soak tests could also have been useful in this case. The hotline’s opening hours were from 8 am to 8 pm, so it was open for 12 hours in total. With soak tests, we can generate load on an application for an extended amount of time. Soak tests can reveal memory leaks or other bottlenecks in the processing of data that might occur after hours.

These are ways we could have tested the system as it is, but are there improvements that we could have made in the system to make it more resilient to reduce the FOMO factor?

Using automation to lessen FOMO risk

The system had some manual parts of the process that may have exacerbated load issues. Here’s how this system could have been automated to reduce bottlenecks.

Interactive Voice Response

Telephony servers commonly use Interactive Voice Response (IVR), an automated menu system for voice calls that uses either voice recognition or keypress tones to navigate a user through a series of choices. We’re all familiar with it: “For Dutch, press 1. For English, press 2…” Such a system might have helped reduce the overall load by quickly filtering out callers in the 25% who weren’t interested in creating an appointment. “If you’d like to book an appointment for a coronavirus test, press 1. Otherwise, please call the information hotline at….”

Here’s what an IVR system could look like:

From: Vanguard Networks ‍ Load testing IVR systems can be difficult, and it usually requires the use of highly specialized commercial tools. However, introducing a new tool may also increase the time required for load testing.

Building a web app

Another approach would be to implement a web app to verify identity and capture personal information quickly. In the Netherlands, we already have DigiD, an identity verification system that is heavily used for governmental services. Mijn Overheid, which is a central government portal accessible to every Dutch resident, already interfaces with DigiD as the sole method of logging in. Reusing these existing services could have saved a lot of the work on a web app— not to mention reducing the bottlenecks inherent in a more manual process like a telephone hotline.

‍ Source: Mijn Overheid ‍

Testing the system end-to-end

We’ve identified a few ways to load test various components of the coronavirus hotline, both in its current incarnation as well as in hypothetical improved versions. Isolating each component and running load tests at that level can help us resolve performance issues inherent in the component. However, there are still bottlenecks that are only revealed when the integrations between components is tested as well. That’s where end-to-end testing comes in.

For the coronavirus hotline, end-to-end testing means being able to run load tests on the entire process and seeing how data flows from one to the other: the call forwarding from the telephony server, the identity verification system, the appointment system, the database of personal information, the servers of every municipal clinic, and the email/text notifications with appointment details. As modern applications grow more complex, it can be tempting not to run end-to-end tests, but we do so at great risk of functional or nonfunctional issues.

An area of end-to-end testing that is often overlooked is the human factor, which can also be a bottleneck. Had hotline operators been trained to ask early on in the call whether the caller wanted to make an appointment, to filter out unrelated calls? Can the coronavirus testing and test result processing, heavily involving manual work from medical professionals, match the stated requirements (test within 24 hours, result within 48 hours) even when the digital components are performant? Some people waited for an hour in the cities of Eindhoven and Goes due to traffic congestion from cars near the test centers, leading to delays in the testing schedule.

While it can be difficult to load test these logistical systems with automated tools, they may have significant effects on the overall performance of an application and should be considered.

Conclusion

Often, performance issues can lead to a tangible loss in profits or a more intangible loss in reputation. In this case study, poor performance had a direct impact on people’s health. Potentially sick people were unable to make appointments to get tested, and later treated. Delayed tests could have led to an increased infection rate as people waited for a confirmed diagnosis. Hotline operators also admitted that slowness in the system led to them sometimes giving out test results against instructions to wait for trained medical professionals to do so.

‍

Human psychological factors can make a system more complex to load test. According to Andrea Evers (LUCM), a health psychological professor, “Door de uitbraak van corona zijn mensen in een langdurige stresssituatie beland. Onzekerheid, onvoorspelbaarheid en oncontroleerbaarheid maken dat ze de behoefte hebben om zaken juist wel te controleren of voorspelbaar te maken [The coronavirus outbreak has put people in a situation of chronic stress. The uncertainty, unpredictability, and helplessness of the situation cause a need to make things controllable or predictable].”

As performance testers, we can use statistics and a systematic attitude combined with knowledge of human factors in applications to try to predict the unpredictable.

Ask a Flooder 20: How can fear affect application performance?

Thu, 06 Aug 2020 22:43:28 +0100

Originally posted here.

In this video, I talk about the power of fear, and how it can end up affecting performance test results. ‍

Or listen to the audio version here:

Hi everyone, I’m Nicole van der Hoeven, back with another Ask a Flooder, and today I want to talk about the effect of fear on your performance outcomes.

The Dutch “coronalijn”

I currently live in Maastricht, in the Netherlands. At the beginning of this global pandemic, the only people getting tested actively for COVID-19 were either people who presented with serious symptoms of the disease, or health care professionals who were continually exposed to the disease and therefore at a higher risk of contracting it.

Last month, on June 1st, the Dutch government made testing available to everybody. They set up a telephone hotline that any Dutch resident could call to set an appointment to get tested for a COVID-19 test. A task force was dispatched to make sure that they could test up to 30.000 people per day. Yet on the first day of testing, only 1.146 people were tested. So what happened?

Well, it turned out that the bottleneck of the entire process wasn’t getting enough tests, registering the tests, or even processing the test results. The bottleneck actually occurred way before that, when people called up the hotline to get an appointment.

323.000 people called on the first day that the hotline opened. And not even all of them had called to make an appointment. Some of them were just calling for general information on the coronavirus despite the fact that the government made it clear that that’s not what the line was for, and that there were other lines dedicated to information. There were reports on Twitter of people waiting up to 6 hours on the line just to talk to someone who would make an appointment for them. There were calls that got dropped. People said that sometimes it happened in the middle of a call. Even telephone operators that the appointment system was down or slow and unusable for a lot of it.

As a result, only 5,500 people actually had appointments booked on a day where there were 323,000 that called. That’s 1.70%.

What does that have to do with performance?

As performance engineers, we like to believe that everything we do is reasoned. We design tests based on quantitative data. We look at historical trends and metrics to help us decide how to build a workload model. But the truth is that that’s often not enough.

If anything, this pandemic has shown us just how wrong our guesses can be, whether or not they’re informed by facts, especially when it has to do with human behavior. We’ve seen violent fights erupt over toilet paper. We’ve seen peaks and troughs on the stock market that didn’t have anything to do with the underlying companies. We’ve seen arson on 5G cell towers. Because human beings are irrational, and it’s really hard to capture that irrationality in our model for how users are going to behave.

We may never be able to accurate predict how irrationality is going to affect the load on our application in production. But that doesn’t mean that we shouldn’t try. There are still things that we can do to mitigate the risk. I think that any application that is public-facing should be assessed for exposure to the risk of fear or other human psychological factors.

How do we account for fear in our performance testing?

I actually wrote a whole blog post on this particular situation (the Dutch corona line), and how it might have been possible to predict (maybe not with 100% accuracy but with reasonable accuracy) how many people would have called the hotline.

For example, we can look at statistics like the of the Netherlands or how many people normally get flu-like symptoms or at least report them at this time of the year. These are statistics that are readily available from the Department of Health, and we could have used them to inform our decisions on the workload model for such an application. We may not have gotten it exactly, but we would have at least bin in the same ballpark.

If we have a good enough idea about the workload that our application would have to handle when it’s influenced by fear, we can start running load tests around that. Another good open-ended kind of test to run, especially for applications that are particularly susceptible to fear, is the stress test. In a stress test, you push your application to its limits and you find out what they are. You could, for instance, start with your peak work load, and then add a certain number of users every 30 minutes and see how much your application can actually handle. You can do the same with soak tests, where you’re exposing your application to the same amount of load for an extended period of time. Both of those kind of exploratory tests help you define the limits of your application and how it would respond if fear plays a role in the workload model in production.

If we overlook this critical human element in our test design, then we leave ourselves, and our applications, vulnerable to performance degradation and, perhaps, outages.

Till next time, happy flooding!

Read this blog post for more about the coronavirus hotline.

Ask a Flooder 18: How do I load test video streaming with JMeter?

Wed, 22 Jul 2020 22:49:15 +0100

Originally posted here.

In this video, I answer the question: “How do I load test video streaming with JMeter?”

Or listen to the audio version here:

HTTP Live Streaming (HLS) and performance

One of the most popular protocols for video streaming is HTTP Live Streaming, or HLS, and that’s what I’m going to show you today. An HLS video ends with the [file extension .m3u8‍](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8](https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8). According to the HLS protocol, when a video is streamed, the first request sent must be a manifest. A manifest is sometimes called a playlist, and it is a list of all the chunks or parts that make up the video. Then, each chunk is downloaded.

To assess the performance of a video stream, we still want the response time of the manifest, the chunks, and the whole video. However, there are many more metrics that are important in video streaming which aren’t captured by just the response time. For example, how do you determine whether a video was played smoothly or if it stopped to buffer? A video that keeps pausing might just drive a user away, and it’s difficult to tell whether that is happening just from the response time.

How to load test video streaming with JMeter

Downloading and installing the Ubik Video Streaming plugin

Luckily, there’s a plugin for that, created by a company called Ubik Ingénierie. The Ubik Load Pack is a set of enterprise-grade plugins for JMeter. One of those plugins is a Video Streaming Plugin, and it happens to be a really easy way to get started with video streaming. This is a paid plugin, but you can sign up for a free 11-day trial.

Once you do that, you’ll get an email from Ubik with a link to download the bundle. Go to lib/ext and then copy this .jar file into the same directory on your JMeter installation. Start up JMeter, and it’ll ask you for the license file. Browse to the license file in the bundle you downloaded. JMeter will restart.

Using the JMeter plugin in your load testing script

Now let’s start creating the Test Plan. Create a Thread group, and then right click on the thread group, go to Add > Sampler and then select the “ULP Video Streaming sampler”.

Click on the sampler and then paste in the URL to the HLS file that you’d like to test. This should end in a .m3u8. This plugin supports several protocols for video streaming, but we can just leave it at auto and it’ll automatically detect which one it is.

In Play Options, you can choose whether to play the entire video or just a part of it, or even whether you want to include the video subtitles in your test.

This Network Options panel is fantastic because it allows you to limit your network speed. Slow networks, like mobile networks, really affect user experience, so it’s really important to take this into account when running load tests. You might even use a parameter here and do something like throttle to 3G speeds for 80% of your traffic, and leave it on Auto 20% of the time. But I’ll just leave this blank for now.

Next, let’s add a View Results Tree listener. Then, let’s run the script once, just to see if it works.

As you can see, we can see the manifest retrieved first, and then all the chunks. We didn’t get any errors, so let’s run this as a load test. We can disable this listener because it just consumes extra resources on our load generator.

Ramping up your load test on Flood

To ramp this up into a load test, we’ll have to go into Thread Group and change these settings. Ideally, we want to be able to change these values from the Flood UI later so we don’t have to go into the script whenever we change them, so I’ll type in these Flood-specific properties. I’ll leave a link in the description about these properties. I’ll also set it to run Forever so that we can change the duration and the video will just keep playing to fill that duration.

To capture some of Ubik’s cool metrics, let’s add a Simple Data Writer listener here and write the results to a CSV file. On Flood, all flood results are stored on /data/flood/results, so we’ll add an output.csv file in that directory that we can download after the test.

So that’s our script! Now we’ll have to include the plugin and related files so that Flood can access them as well.

Let’s go back to the bundle we downloaded and go to the bin folder to look for user.properties. Let’s open that up, and we’ll change the license path to /data/flood/files, which is the directory that Flood saves files we upload to. To preserve this directory structure, we’ll compress just the lib and bin folders.

On Flood, we’ll start a new stream here and upload three things: the jmx script, the license, and the compressed archive of the lib and bin folders from the bundle.

Then we’ll go through the rest of the process here, setting the test parameters. Time to launch the test!

Analyzing video streaming load test results

Flood shows you results of your load test while it’s running as well as after the test. You can see metrics like concurrency, network throughput, transaction rate, and response time for the test. You can also see these metrics for each individual transaction, including the video or sampler as a whole, the manifest, and the chunks.

To see the CSV that our script generated, we can download the Archived Results from a flood.

In JMeter, click on Tools > Generate Report and fill in the fields, selecting the output.csv from the flood’s Archived Results, JMeter’s user.properties file, and then an output directory for the report.

Ubik’s plugin creates a few custom graphs in the HTML report:

Buffer fill in Millis
Lag ratio in percentage
Lag ratio without buffer fill in percentage
Lag time in Millis

Conclusion

Using Ubik’s Video Streaming plugin is an easy and thorough way to load test video streaming with JMeter, and I highly recommend it. I used an HLS video to demonstrate it, but this plugin supports several protocols:

HLS
Mpeg DASH
Smooth
Adobe HDS

For more information, read Ubik’s tutorial on how to use their plugin with Flood here.

Performanceverbeteringen voor de coronalijn: de kracht van angst

Sat, 04 Jul 2020 21:35:43 +0100

Originally published on Medium. English version here.

Op 1 juni heeft de landelijke GGD Ghor aangekondigd dat iedereen met klachten een coronatest zou kunnen aanvragen door de afsprakenlijn (0800–1202) te bellen. De belofte: vandaag het telefoongesprek, morgen de coronatest en over twee dagen de uitslag. Echter was het in werkelijkheid niet zo makkelijk. Op de eerste dag was er chaos en veel Nederlanders meldden verbindingsproblemen vanwege drukte op de overbelaste systemen.

Dit had voorkomen kunnen worden door een beter begrip te hebben over de manier waarop de menselijke psychologie de performance van een systeem kan beïnvloeden.

Bron: @ggdghornl, Twitter

De situatie

“Voor die 30.000 tests per dag zijn we nu eindelijk klaar”, zei arts Ann Vossen op 31 mei. Dit aantal verwijst naar de landelijke capaciteit van alle GGD’s. In plaats daarvan werden volgens het AD maar 1146 mensen getest en, misschien belangrijker, een kleine 5478 afspraken gemaakt. De situatie werd erger gemaakt door de 323.000 mensen die de afspraaklijn op de eerste dag hebben gebeld. Sommige medewerkers konden zelfs niet in de systemen inloggen.

Op Twitter vertelden gebruikers enkele horrorverhalen: één beweerde 6 uur lang te hebben gewacht, alleen om een melding te horen dat het systeem geen afspraken kon aanmaken; een andere klaagde over verbroken verbindingen, zelfs toen een medewerker al had geantwoord, en nog een andere een video had gepost van een gesprek dat beëindigd was zonder melding. Ik heb zelf ook de coronalijn gebeld, en nieuwe problemen gevonden.

Gelukkig was de tweede dag beter: er waren 11 duizend afspraken gemaakt, mede vanwege het lagere aantal telefoontjes en mede vanwege verholpen technische problemen. In de eerste week nam het aantal geteste mensen toe naar 50.000 (gemiddeld 7142 per dag), wat toch nog veel lager zat dan verwacht. Er zijn wat lessen die we kunnen trekken uit dit verhaal.

Bron: GGD Hollands Midden

Het gevolgen van angst op performance

Dit virus heeft ons al een kant van de mensheid getoond die we liever niet willen zien: vechten over wc-papier, chaos op de beurs, en sabotage van zendmasten. Hoewel we rationeel willen zijn, is het onmogelijk te zeggen dat dat altijd het geval is. In werkelijkheid hebben we allemaal de neiging gekke dingen te doen. Vooral als we angst hebben.

Geen wonder dan dat de afspraaklijn zo welkom was. 323.000 telefoontjes werden er op de eerste dag verwerkt, wat bijzonder is als we bedenken dat het grootste deel van Nederland in zelfisolatie was, en dus maar een kleine kans had om het coronavirus opgelopen te hebben. In theorie lijkt dit gedrag irrationeel, maar niet onvoorspelbaar.

In het Engels heet het FOMO: Fear Of Missing Out. We hebben al duidelijk gezien hoe FOMO veel meer gebruikers naar een applicatie kan dwingen, met als gevolg bijwerkingen op de performance. Mensen hadden angst dat er niet genoeg coronatesten waren en wilden de eersten in de rij zijn voordat de voorraad helemaal op was.

Ook was er het probleem dat één op de vier telefoontjes niet over een afspraak ging, maar over algemene informatie over COVID-19. Dit werd verergerd door het feit dat de eerste dag van de coronalijn ook Tweede Pinksterdag was. Het teveel aan telefoontjes over andere zaken dan de coronatest was een bijkomende factor in de systeemoverbelasting.

Wat hadden we kunnen doen om dit systeem te verbeteren?

Verbeteringen

Beter inschatten

Hoewel het moeilijk is om de gevolgen van FOMO precies te kennen, kunnen we het wel inschatten. Volgens RIVM-hoofd Jaap van Dissel, waren er begin mei tot 700.000 mensen in Nederland besmet met het coronavirus, een paar weken voor de coronalijn werd opgestart. Waarom was het dan een verrassing dat meer dan 300.000 mensen hadden gebeld? Ook zijn de cijfers van niet-COVID-gerelateerde klachten daar nog niet bij inbegrepen.

Zelfs als we inschatten dat 20% van die 700.000 mensen geen klachten had, komen we toch op een voorspelling van 140.000 telefoontjes uit, wat nog 25 keer meer is dan het werkelijke aantal gemaakte afspraken op de eerste dag.

Zodra wij aantallen hebben, kunnen we testen.

Uit mijn eigen ervaring, toen ik het coronanummer belde, blijkt dat er een telefonieserver was die de oproepen heeft doorgeschakeld naar verschillende medewerkers (die thuis werkten). In deze stap konden loadtesten worden uitgevoerd om zeker te weten hoeveel gesprekken deze server aankon, en dat het snel genoeg was om geen of weinig message queueing te hebben.

Medewerkers hebben dan vermoedelijk gegevens direct in een afsprakensysteem ingevoerd tijdens de oproep, waarop de gegevens waarschijnlijk naar een backend server en database gingen. Ook hier kunnen we testen via een load testing tool op protocol niveau, zoals JMeter.

Met zo’n publieke startdatum zou ik spike testen hebben gevoerd met tot 700.000 gebruikers (misschien nog meer, als ik wat tijd had). In een spike test simuleren we een snelle toename van gebruikers op een kort tijdstip, bijvoorbeeld 1 tot 700.000 gebruikers in 10 minuten. Ook zouden soak testen heel nuttig zijn geweest, vooral omdat de coronalijn van 08.00 tot 20.00 was geopend, dus een stroom van 12 uur. In een soak test voeren we een constante aantal gebruikers voor een langere periode op.

Automatisering gebruiken

Het maken van een afspraak voor de coronatest had veel stappen. Eerst moesten mensen hun BSN en andere details aan de medewerkers geven. Dan moesten medewerkers vragen naar hun klachten. Alleen met de belangrijkste klachten die passen bij het coronavirus konden mensen verdergaan. Uiteindelijk moesten medewerkers adressen controleren en het dichtstbijzijnde testcentrum vinden om de afspraak te maken.

Telefoniesystemen gebruiken soms Interactive Voice Response (IVR) technologie, een systeem van menu’s en keuzen. We hebben dit allemaal gebruikt: “Voor het Nederlands, toets 1. Voor het Engels, toets 2…” Zo’n systeem had het aantal niet afspraak-gerelateerde gesprekken kunnen verlagen met 25% (één op vier klanten belden niet voor een afspraak), en op een geautomatiseerde manier.

Hier is een voorbeeld van een IVR-systeem:

Bron: Vanguard Networks

Op het niveau van het IVR-systeem kan het wel moeilijk zijn om load te simuleren. Empirix Hammer and Cyara zijn twee zeer gespecialiseerde tools die dit kunnen doen.

Nog een optie voor de implementatie van een afsprakensysteem is een web app, met verplichte velden voor BSN, adressen, enz, en met een verbinding met DigiD, een identiteitsverificatiesysteem. Mijn Overheid bestaat al, en ook met DigiD integratie. Er had een soort van beveiligde formulier kunnen worden gebruikt om klantengegevens snel op te nemen.

Bron: Mijn Overheid

End-to-end testing uitvoeren

Vaak spreken we over de performance van één applicatieserver, maar in praktijk zijn er meerdere componenten. Als best practice is het goed om de performance van elk onderdeel van het hele systeem te testen, en dan ook de hele keten.

Voor een afsprakenlijn zou het redelijk zijn geweest om het hele proces te testen: het oproepdoorschakelingssysteem, het afspraaksysteem, de database met gegevens en datums, de servers van elke GGD in het land, en het sturen van email en SMS-berichten met afspraak datums. Het is belangrijk om te weten of alle componenten snel samen kunnen werken.

Ook is er de performance van niet-computer onderdelen. De menselijke component kan ook een bottleneck zijn. Wisten de medewerkers hoe snel ze moesten vragen of een klant een afspraak wilde maken? Kunnen de testen worden uitgevoerd en de uitslagen worden gegeven, zelfs met een snelstromend afspraaksysteem? Het teveel aan afspraken toegewezen aan GGD’s kan ook files veroorzaken (zoals in Eindhoven en Goes).

Conclusie

In dit geval had slechte performance een invloed niet op de winst maar op de gezondheid. Zieke personen die klachten hadden, konden niet allemaal een afspraak maken om getest te worden. Hoeveel andere mensen hadden ze inmiddels besmet? Slechte performance heeft medewerkers ook gedwongen om soms uitslagen te geven “in verdekte termen”, wat tot een privacyschending kan leiden.

De menselijke component kan een systeem lastiger maken om te testen. Aldus Andrea Evers (LUMC), hoogleraar gezondheidspsychologie:

Door de uitbraak van corona zijn mensen in een langdurige stresssituatie beland. Onzekerheid, onvoorspelbaarheid en oncontroleerbaarheid maken dat ze de behoefte hebben om zaken juist wel te controleren of voorspelbaar te maken.

Als performancetesters kunnen we statistieken en een systematische aanpak combineren met kennis van de menselijke factoren om te plannen voor het onvoorspelbare.

Ask a Flooder 17: How do I use unique test data on Element?

Thu, 18 Jun 2020 21:58:49 +0100

Originally posted on Tricentis Flood.

In this video, I talk about how to ensure that Flood Element uses unique test data, using the example of a customer wanting to script unique logins for his application.

Or listen to the audio version instead:

Transcript:

Hi everyone, Nicole van der Hoeven here again back with another Ask a Flooder, and this week’s question is a follow-up question from last week. That user had previously asked about test data in Flood Element. This week he’s asking: “How do I ensure that Flood Element uses unique test data?”

So this user is actually trying to script a login for an application. He has a list of usernames and corresponding passwords that he wants to use. He’s already set both of these up in a CSV file, and he’s also set up the Element script in the same way that I outline in my previous video. However, this user is seeing errors in the test due to the application not allowing a given user account to be used simultaneously in two or more devices. The problem is that the way that he’s set it up is that he has Element running on multiple nodes, and each node has the chance to pick the same user account.

So how do you make sure that it’s a unique 1:1 ratio for an instance of Element and a user account in the CSV file? If you haven’t already watched the previous week’s video, you should pause this, watch that, and then come back here, so it makes a little more sense because the script that I’m using is building on what we already did in the previous week.

Let’s talk about the differences between this script and the previous week’s.

First, I’m declaring a global variable here called globalBrowserID. The idea is to create a unique and globally identifiable string that I can attach to every account in the CSV file.

const globalBrowserID = ENV.FLOOD_GRID_INDEX + '_' + ENV.FLOOD_GRID_NODE_SEQUENCE_ID + '_' + ENV.BROWSER_ID.toString()

The global browser ID consists of three parts:- The first is a grid index, which is a number assigned to every grid, and it’s also globally unique. So if there are three grids being used in the test, then the grid indexes would be 0, 1, and 2.- The second one is a grid node sequence ID, and this is unique within the grid. The node sequence ID is a number that is assigned to every node in a single grid.- The last one is a browser ID. The browser ID is a number assigned to an instance of Element running on a single node.

These are all zero-based, so the very first instance of Element running on the very first node and on the very first grid in the entire test would be 0_0_0.

We’re going to have to modify our CSV file to make use of this browser ID. So whereas the user previously had username and password, I’ve added an ID column. From then, these will have to be sequential. So 0_0_0, 0_0_1... And 0_0_1 would be the second instance of Element running still on the first node, on the first grid. And you can make as many of these as you need.

id,username,password
0_0_0,user1,password1
0_0_1,user2,password2
0_0_2,user3,password3
1_0_0,user4,password4
1_0_1,user5,password5

I’ve also modified the interface UserData to describe the fields in the CSV file. So there’s id, username, and password, and I’ve defined them all as strings.

TestData.fromCSV<UserData>('users_multiplegrids.csv')
 .filter((line, index, browserID) => line.id === globalBrowserID)
 .circular()

This line is telling Element to take this data file with this filename and filter it based on the line, the index, and the browserID such that id is equal to the browser ID. So what happens here is that Element will take the global browser ID, which let’s say is 0_0_0 for the very first one. It’ll then take that 0_0_0 and look at the CSV file for anything in the ID field that has the value 0_0_0. And then it will continue to use the rest of that line.

I’ve also made the CSV file circular just because that’s always a good practice to get into.

And as always, I think it’s a great idea to debug this, especially when you’re dealing with multiple grids and multiple nodes on that grid.

step('Print to console', async (browser: Browser, data: UserData) => {
 console.log('global browser ID: ', globalBrowserID, '| user: ', `${data.username}`, ' | password: ', `${data.password}`)    
})

I’ve also added the username and password here, just in case we need to verify the state that that user is in.

So let’s run the test. Since we’re just running this locally, we’ll see that the first value indeed is 0_0_0, and the user we’re selecting is user1 and password1, which lines up with the username and password values corresponding to 0_0_0 in the CSV file.

I hope that helped, and until next time, happy flooding!

Download the full script and the data file to follow along.

Ask a Flooder 16: How do I use test data with Element?

Tue, 09 Jun 2020 22:18:41 +0100

Originally posted on Tricentis Flood.

In this video, I discuss how to make your load testing scripts more realistic by including test data. The load testing tool I use is Flood Element, but you can apply these principles to any tool. You’ll find a copy of the Element script I used below.

‍

And, if you prefer, here’s the audio version:

Transcript

Hi everyone, Nicole van der Hoeven here back with another Ask a Flooder, and this time the question is: How do I use test data with Flood Element?

Now let’s talk about why you might want to use test data. In load testing, it’s important to make your scripts behave as similarly to real users as possible so that the load that you generate is production-like. The problem is that if you send the same data over and over again to the server, that server is probably going to cache that response. The response being cached means that the server will return the response way faster than it normally would if it had been a unique request. So we can use test data, instead of static data hardcoded in your scripts, to make sure that this doesn’t happen.

Using test data can also help us test different responses. For example, let’s say we’re testing a loan application system, where a user inputs the application ID, the server looks it up, and returns the status of that loan application. Now, loan applications might be in different stages of the process. For instance, some might be saved but not submitted, or maybe they’ve been submitted but failed the identity verification check.

If we used the same application ID every time, then the server would always return the same status and the same details. Instead, let’s create a CSV file that we can use as test data. It might look something like this.‍

applicationId
A100001
B100002
C100003
D100004

We can have applicationId as the single header here, and we can populate this with a list of different application IDs in different stages of the process. That way, when we run the script, the server will have to respond authentically to every request without it being cached.

Now to create our script. This is an empty Element script. To start with, make sure you’re importing TestData here in the first line of your script.

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'

The first thing we’re going to need to do is to create an interface for the CSV file. Let’s call it AppData. Within this interface, we’ll need to define all the fields that we intend to use in our data file and what type of values we can expect to be in those fields. Looking at our CSV here, we’ve just got the application ID, so I’m just going to copy that, and it’s a mixture of characters and numbers, so I’m going to put that in as a string.

interface AppData {
 applicationId : string    
}

Now we need to tell Element how to find this test data. We’ll say that it’s from the CSV, using the interface we’ve just created. Now we’ll tell Element the filename. In this situation, the CSV file that I’m using is actually in the same directory, so I don’t need to specify a path here, but if you’re putting it somewhere else, then you’ll need to specify the folder and the path to it.

TestData.fromCSV<AppData>('applicationids.csv')

Now let’s use the data in a step. In real life, you might want to use an application ID to type it into a text field, but for demonstration purposes, I’m just going to show you how to output it to the console log. Printing to the console is also a good way to debug your tests, especially if you’re dealing with test data.

We need to let Element know that we want to use AppData. So now, let’s output this to the console. Since we’re just debugging, let’s output the application ID. To actually get the value from the data file, we’ll use this syntax:

step('Print to console', async (browser: Browser, data: AppData) => {
 console.log('application ID: ', `${data.applicationId}`) 
})

‍By default, Element reads the data files sequentially. The problem with that is we only have a handful of application IDs. So if we wanted to run this one script 100 times, we’re going to run out of data. What we can do here is add circular. That way, when the script reaches the very last value of the file, the next value it selects will be the first one, and it never runs out of test data.

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

Another thing you might be wondering is how to make it so that Element chooses a random value every time. To do that, we’ll also have to add shuffle:

TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()

‍Now let’s try to run the test.

element run test-data.ts

As you can see, it’s been finished. The application ID that was selected was indeed a random one because the first one would have started with A. So, it actually chose the last one randomly.

And that’s it for test data with Element. Now, this user actually had a follow-up question, which was about unique test data. I’m going to answer that in the next video. Until next time, happy Flooding!

Sample script

import { step, TestSettings, Until, By, Browser, TestData } from '@flood/element'
import * as assert from 'assert'

export const settings: TestSettings = {    
 loopCount: -1,    
 description: 'Test Data',    
 screenshotOnFailure: true,    
 disableCache: true,    
 clearCache: true,    
 clearCookies: true,
}

/*** Test Data * Version: 1.0 */

export default () => {    
 interface AppData {        
 applicationId : string    }    
 TestData.fromCSV<AppData>('applicationids.csv')    
 .circular()    
 .shuffle()    
 step('Print to console', async (browser: Browser, data: AppData) => {        
 console.log('application ID: ', `${data.applicationId}`)    
 })
}

Load testing on the Google Cloud Platform

Thu, 28 May 2020 22:25:06 +0100

Originally posted here.

We’re excited to welcome another cloud provider to the Flood lineup, giving you even more options for running your load tests. You can now use Flood to run load tests on the Google Cloud Platform (GCP).

Many of you have requested an integration with Google Cloud, citing many of its strengths:

Full integrations with open-source tools like Influx and Redis
World-class security
High-performance global network

Seamless access to other apps in the Google ecosystem, such as Search and Maps

We think that adding support for GCP instances opens up a lot of opportunities for flooders wanting to take advantage of these benefits.

Using Flood with Google Cloud

You’ll notice a new Google Cloud section in the Integrations page while you’re logged into Flood:

You’ll need to add your GCP service account key to allow Flood to access your account on your behalf. Once you’ve done this, you’ll be able to start grids and floods right from the Flood interface you’re already using.

For more details on how to set up Flood to use your GCP account, check out our documentation or follow along with the video below:

Anyone with a Flood subscription, including a free one, can use Google Cloud instances at no extra cost from Flood; however, GCP usage may incur separate costs.

GCP is the third cloud provider that we support at Flood, after Amazon Web Services and Microsoft Azure.

Ask a Flooder 15: How do I install Element?

Tue, 12 May 2020 22:45:18 +0100

Originally posted here.

In this Flood Element tutorial, I demonstrate the steps required to install Flood Element, including dependencies it requires. Flood Element is an open-source browser-based load testing tool created by Tricentis Flood.

Learn more about Flood Element

Installation instructions for NodeJS

Installation instructions for Yarn

Installation instructions for Element

Video: Getting Started with Element

Guides: Using Element with Flood‍

Audio version here:

Transcript

Hi everyone, Nicole van der Hoeven here again, back with another Ask a Flooder, and today’s question is: “How do I install Element?”

Now, I already made a previous video on how to get started with Element, but I didn’t actually go into the details of how to install it, so that’s what I’m going to do in this video today. There are actually a few parts to installing Element, and all of these instructions are going to be on element.flood.io. If you go onto the site and click on Install, you’ll see all of the instructions depending on which operating system you’re on.

First, you need to install NodeJS. This is sort of like how, if you’re familiar with JMeter, you have to install Java before installing JMeter. NodeJS is what Element runs on. Secondly, you’ll need a package manager. You can use NPM or Yarn, but in this video, I’m going to take you through how to do it with Yarn. And thirdly, there’s installing Element itself.

First, download NodeJS. I’m going to leave links in the description to all of the sites that I visit so that you can go through all of the instructions later. On the Node site, download whichever installer suits your operating system. Go through the installation steps, and afterwards, check to make sure that NodeJS is properly installed on your machine. You can do that by going into the terminal and typing:

node -v

Just make sure that the version that’s shown here is the same version that you’ve downloaded.

Secondly, install the latest version of Yarn. On the Yarn site, you’re going to have to choose your operating system and the version. We recommend using the latest stable version. And then go through the instructions for your operating system. After you’ve done that, check your Yarn installation by typing:

yarn --version

And the last step is installing Element. On the Element site, you can find the exact code that you need to type into your terminal. It’s:

yarn global add @flood/element-cli

So you can just copy that into your terminal. As of the time of this recording, the latest version of Element is 1.2.3. Looks like that’s installed. To check the version of Element, type:

element --version

So that’s how to install Element. We’ve gotten a lot of questions recently about Element in particular, so I’m going to answering them in future videos. Hope this helped, and until next time, happy flooding!

Innovation in Isolation

Tue, 05 May 2020 22:59:57 +0100

Originally posted here.

We’ve always been a fully distributed team at Flood, but the current world crisis has still changed the way we work. Here are some of the things we’re doing to keep self-isolation from being too isolating— and maybe it’ll help your team too!

Within the Flood team, flooders-in-training (aka children) have made more surprise appearances in team meetings. Work has been punctuated with grocery runs to buy essentials before they run out. Some of us who were out of our home countries when all this happened have had to arrange emergency travel before international borders shut. It would be foolish to deny the impact on team morale as the entire world goes through this collective trauma, eyes glued to numbers that increase by the minute.

Not exactly the most productive work environment.

Yet there’s still work to be done (something we’re grateful for), and there are still many features we’d love to bring to Flood. So how do we keep everyone motivated to release and maintain software?

Encouraging banter

Work shouldn’t be all about work. Understanding everyone’s personal circumstances is vital to a team’s ability to work together professionally. We use Slack as our primary communication tool throughout the day, and we already had quite a few non-work channels on there (#anything, #hobbies, #gaming, #drones, and #travel to name a few). Now we have one for COVID-related chatter as well.

In this channel, we share news from our parts of the world, post photos of things that are sold out in our local supermarkets, talk about new hobbies we’ve taken up to stay sane, and discuss the struggles of co-working with spouses and children who are now also working or studying from home.

Having a dedicated channel gives people license to open up about what’s going on in their personal lives, and it helps the team understand where everyone’s headspace is at. We also use #eng-covid-19 for memes. A lot of memes. It’s easier to work through a tough situation if you can laugh about it together.

Over-communicating

Our co-founder and Product Manager, Ivan Vanderbyl, recently wrote a great article with some tips for keeping communication lines open. This is particularly important during this crisis, where schedules and priorities can change quickly.

Emojis - We use emojis as a non-verbal cue to convey tone, approval, agreement, and disagreement. We add emojis to messages we send and also use them as reactions on Slack to messages from other people. Even when you don’t have an answer or when a response is not required, emojis signal acknowledgment so that nobody feels like they’re shouting into the void. Over time, we’ve developed emojis that are specific to Flood culture, such as the avocado:

Source: Our Twitter account

‍We even have a fruit-based vocabulary of emojis that we can use in our Slack status messages to convey our general mood. Being in Carrot Mode, for instance, means that you’re trying to focus and would appreciate not being direct messaged or @ mentioned.

‍When working remotely, and especially when in self-isolation, everyone has their own cadence for working. It’s on each person to communicate that schedule so that others can respect it.

Text chat - We’re used to working asynchronously due to time zones, so text-based discussions are necessary. We try to over-communicate and “think out loud” before making decisions just in case we have to suddenly stop working due to illness or other pandemic-related reasons. Even something as simple as leaving a link to a document that we’re working on means that someone else will be able to find it if we’re unavailable.
Voice chat - We tend to use voice chat for quick discussions or updates about work-related things. It’s also great for those of us in Europe who are suffering from slower internet connection speeds at certain times of the day due to everyone being home and streaming Netflix.
Video calls - Video calls are for team retros, 1:1 calls with managers, and personal or sensitive conversations where body language is crucial. It’s also great for recreating that “hanging out at a pub” vibe. Speaking of which… welcome to the Flood pub!

The Flood Pub is open on Friday afternoons. It’s where we all hang out together with a beverage of our choice and talk about things that are entirely not work-related. We hop on a Zoom call together, Tim (our General Manager) fires up the “jukebox” (Spotify playlist), and we end the week by getting some facetime with each other—just as we would do (and have done) if we were together in person.

‍Meeting in this low-pressure context lets us talk about things we wouldn’t mention otherwise. It’s also a fun way to check in with each other after another week at work (and in quarantine).

Doing stuff

While communicating openly is productive, getting stuff done as a team boosts spirits. Here’s what we’ve done recently to kick our output into high gear.

BOINC

Like most of you, we’ve been cooped up in our houses, watching events unfold and feeling helpless as things happen that are beyond our control. BOINC gives us a chance to do something to contribute that IS in our control.

BOINC is a platform by the University of California Berkeley that lets you volunteer your spare computer (or mobile device!) processing power to science. From the official site, Boinc “downloads scientific jobs to your computers and runs them invisibly in the background.”

In our case, we wanted to help sequence proteins like the novel coronavirus using Rosetta@Home. BOINC is a perfect fit for Flood because we are in the unique position of starting up and managing thousands of virtual machines 24/7. Our Ops team has been hard at work automating the donation of our computing power to the Rosetta@Home project when these machines aren’t being used.

We’re proud to say our contribution is INcreasing over time as our engineers find new ways to streamline the process. We’re already in the top 5% of BOINC users in the world by contribution.

GIG: Great Ideas Gathering

Every year, the mothership (Tricentis) runs a three-day hackathon called GIG. Two weeks ago was the first-ever GIG where most participants, not just the Flood team, were working remotely.

The theory behind hackathons is that the daily grind can be exhausting after a while, and it can sometimes be beneficial to introduce a short burst of work to revitalize the team. Hackathons provide a framework for small groups to pitch, execute, and report on a single idea without committing too much time to it. Everyone gets a chance to step away from regular duties and work furiously together on something they might not have otherwise gotten to work on.

A hackathon where the team members are separated by distance and timezone is certainly another beast, but here are some lessons we learned about remote hackathons:

Come up with a good idea: This is easier said than done, but a good hackathon idea is a creative solution that delivers a lot of value. Here are the ideas we came up with:
- Floody - Use AI to improve onboarding by walking new users through the steps for running a flood (live on Flood now!)
- Flood Bot - Add smarter checks to detect patterns in load tests and help flooders understand their results (prototype)
- Flood Gate - Use AWS Fargate to reduce costs in provisioning, scaling, and managing virtual servers by about 30% (proof of concept)

When thinking up ideas for GIG, we didn’t commit to releasing them publicly. Even a project idea that doesn’t end up making it into the finished product is an occasional worthwhile detour for team productivity and morale.

Limit scope: GIG is a sprint, not a marathon. While my team (comprised of Jason Rizio and me) was planning out what we wanted to do, we quickly realized that there was a lot more that we wanted to do, but it just wasn’t feasible within the strict timeframe. In the end, we decided to go for an incremental piece of work that we could push to production before the deadline instead of completing just one step towards something abstract and unreleasable.The other Flood teams also needed to whittle down the scope to a smaller, completable chunk: a working prototype.
Clarify assignments: With members on different timezones, it was essential to have an initial meeting to plan out everything that needed to be done and by when. In this situation particularly, we found it useful to split apart the project into independent chunks, to minimize the need to collaborate in real-time.
Touch base regularly: When working independently, team members can quickly diverge off the main path. To avoid that, we met up at least daily by hopping on an audio call on Zoom to spend a few minutes updating each other on the work we’d done. Doing this gives others a chance to help out when one member encounters roadblocks.For anything that didn’t need to be real-time, we worked asynchronously on chat. Here’s a screenshot of the number of messages and files exchanged to our dedicated GIG Slack channel during the event:

‍Yup, 1,535 messages and 60 files in 3 days. That’s collaboration!

We generally prefer public rather than private channels, and we certainly stuck to that throughout GIG. When you’re working with more than one person, it’s counterproductive to have private conversations that can’t be referenced by other people.

We also learned that it’s useful to keep tabs on what the other teams are doing. After the event started, my team realized that another team was changing a page on the Flood app that we intended to use. Luckily, both teams were able to work around it.

It’s also only fair to let other teams chime in on important decisions within the project that might affect future work— like the name for your mascot.

Leave time for the video: It’s not just about doing the work; it’s about communicating what you’ve done in a way that others can appreciate. Here’s a video that my team (composed of Jason Rizio and me) put together for Floody and our improvements to the onboarding process:

Celebrate: After the work is done, don’t forget to pat each other on the back for the hard work! For us, this meant a round of tacos for everyone and a trip to the Flood pub. (Why do we have so many food-based rewards?)

It’s not easy

Keeping a team unified and working efficiently towards a common goal is always tricky, and working remotely, especially during a world crisis, doesn’t change that. Distributed team members need to put in more effort to consciously over-communicate, make room for each other’s unique circumstances, and do little projects that foster creativity when we’re all feeling a little lackluster. Incorporating these tips can create a culture of healthy productivity within a remote team.

‍

On-premise load testing advantages and disadvantages

Thu, 30 Apr 2020 23:06:59 +0100

Originally posted here.

Where you run your load test can affect the results that you get as well as your project’s bottom line. In general, there are two infrastructure options to consider for load testing your apps: on-premise and cloud. Which type of environment is best for load testing both web or mobile apps?

Differences between on-premise and cloud load testing

What is on-premise load testing?

On-premise load testing is running your performance or load testing scripts on load generators you physically control. Typically, these load generators are machines that are either in your office or data center. In on-premise load testing, you are responsible for all aspects of the maintenance of the load generators, including hardware upgrades and software installations such as testing tools, but that also comes with increased control and security.

What is cloud load testing?

In cloud load testing, you run your load tests on a service provider’s machines, effectively outsourcing the provisioning and maintenance of your load generators. Running “on the cloud” means you’re paying someone else to be responsible for these machines. In cloud load testing, you don’t “own” the machines your test is running on; it’s more like renting some time on someone’s data center whenever you need it. Cloud-based infrastructure isn’t necessarily less secure than on-premise infrastructure, but there are certainly more security considerations to keep in mind.

Pros and cons of on-premise load testing

On-premise load testing is no better or worse than cloud load testing; it just depends on what you need. There’s a meme that the cloud is just a fancy term for someone else’s computer— but that’s an oversimplification that glosses over nuances that might make a difference to your bottom line.

Here are some questions to ask yourself when deciding between load testing on the cloud or on-premise.

‍

Test infrastructure can make or break your load tests.

Cloud vs. on-premise load testing: how do you decide?

Why are you running a load test?

If your intention for running load tests is to test your web application’s load balancing, the source of the web traffic, and the location of your environment, may not be as important. If you intend to do stress testing, you may want the scalability of a cloud setup.

Who is the intended audience of your application?‍

Load testing is most accurate when it is as realistic to the production workload and user base as possible. Running a load test on-premises means that the traffic comes from one location, which could be ideal if your application is an internal one.

Do you have the resources in your team/company to maintain load generators?

All machines need to be provisioned, set up, and maintained to function correctly. Depending on how many machines you need, you may even need a team of trained professionals to monitor your environment.

What is your budget?

Both cloud and on-premise load testing have costs. Cloud load testing requires a smaller outlay of cash initially but may cost more to maintain as you continue to pay for the computing time. On-premise load testing requires more money upfront but may cost less to maintain in the long run.

How many virtual users do you want to run?

The number of virtual users you intend to test your application against correlates to the number of load generators you require. In general, the more machines you need, the more expensive it becomes (for both cloud and on-premise load testing), but if you only need a handful of users, perhaps you can find a machine or two on-premise that could do the trick.

Advantages of on-premise load testing

‍

Increased control

Since you own the machines you’re testing on, you can set your environment up with your exact specifications in mind. You’ll be able to make hardware upgrades or update versions of operating systems and load testing tools whenever you want, without needing to liaise with a third-party provider. This option can come in handy if you routinely use non-standard configurations or legacy software. If you’re also doing other types of non-functional testing, like security testing, it may be easier to change the configuration on on-premise machines.

Greater data security

The question of cloud vs. on-premises may be a business decision. Some industries or businesses with stringent compliance and regulatory requirements may have no choice but to run tests on-premise because owning the load generators also means controlling who has access to the data that’s on them. An on-premise environment lets you lock down access if necessary. With cloud load testing, you’d need to have security arrangements with the cloud provider that you choose so that any test data or identifying client information seen or accessed by your test scripts remains confidential.

Lower cost and lower barrier to entry in some situations

If you don’t intend to run too many users of your test script and you already have a spare machine or two at the office, on-premise load testing may be the easiest and most cost-effective way to get started. This way, you avoid having to request the budget for a new tool and needing to train your team on a new interface.

Fewer variations due to latency

Controlling your own load generators means controlling the network they run on. If your corporate network is generally stable, and your test scenarios don’t involve testing from different geographical locations, reported response times (and user experience) more closely reflect server processing time rather than latency and travel time.

Disadvantages of on-premise load testing

‍

Higher cost in some situations

Maintaining your own load generators comes with a higher up-front cost, which may be a showstopper for smaller projects or companies. The more concurrent users you require for your load test, the more machines you need, the more licenses you need for your testing tool (unless it’s open source), and the longer it takes to recover your investment.

Geographical limitations

Since all your machines are in one place, you can’t accurately simulate the user experience for customers who might access your web app from another location. You also can’t fully test the effect of CDNs and other tools that your application might employ to keep response times low. Web apps that can be accessed by anyone on the internet might require performance tests that take into account customers from different regions.

More setup

You need to do all the setup yourself for both hardware and software components. You also need a way to coordinate your test, so that load generators start and end when you want them to. You may want to divert some resources towards environment management.

Less flexibility

With on-premise load generators, you can’t add, remove, or scale up nodes on the fly according to your test— something cloud load testing excels at.

On-premise load testing best practices

The environment you’re testing in is especially important when you’re using your own infrastructure. Here are some best practices for a performance testing strategy that involves on-premise load generation.

Run baseline tests

Treat load testing like a scientific experiment. Before you begin changing variables to improve performance, spend some time figuring out your test configuration so that it’s repeatable. Pay attention to things like workloads, business processes, the number of users you’re running per load generator, test environment configuration values, script pacing and think time set in your testing tool, and test script transactions. Make sure that the resource utilization on your machines remains healthy during a test; otherwise, you’re load testing the load generators themselves rather than your application servers.

Then, run a few tests against your application servers without making any changes. This way, you create a “control group” or control test against which to compare future tests.

This process is called baselining your application performance.

‍ Flood’s Project View lets you track an application’s performance over time and compare recent tests to a baseline ‍

Consider using containers or virtual machine images

Along the lines of reducing variables, it’s especially important to use load generators that are as identical as you can make them. You can manually check that you’re installing the same software and tools on each one, or you can do as cloud providers do and automate this process.

It’s easy to miss software that are necessary for monitoring or the load testing tool you’re using. For example, JMeter and Gatling require certain versions of Java. Test automation or browser-level load testing tools might require web browsers.

You can use containers such as Docker to create a standalone package of everything you need for your test. Alternatively, if you’re using virtual machines, you can also create a full system image that you can use for every virtual machine.

The difference between the two is that a container lets you be more granular about what is included in the package that you then deploy to all your load generators. Virtual machines include hardware components that you can set, whereas containers are software-only, decreasing package sizes.

Using containers or images allows you to keep your load generators as identical as possible (in terms of hardware and software) so that you can assume similar results from each one.

‍‍ ‍

Don’t neglect monitoring

It’s easy to forget that your application servers aren’t the only ones that need monitoring. Your load generators need monitoring as well, so that they don’t end up being the bottleneck in your tests. Here’s a starter list of server metrics that you should consider monitoring.

You might also consider how you can integrate performance management tools you already use into your monitoring strategy.

Spend time on a framework for gathering data

One of the biggest challenges in on-premise load testing is figuring out how to get data from your tests. Each machine runs your test script independently of the others, so at the end of your test, you’re likely to end up with several results files. After you download these off every machine, how do you combine them into a single summary?

Manually collating results is an option, but it’s a tedious one. You can copy all the files into one directory and then concatenate them so that you end up with one giant one. Depending on the size of those results files and how long you ran the test, however, this may not be feasible. For larger files, you might consider aggregating the data and increasing the granularity, so that instead of seeing raw response time for every transaction, you only see the average response time for every 15 seconds. You can use a tool like Logstash for this.

It takes some time and technical expertise to set up a framework for results gathering that works for your purposes, especially if you want to go a step further and have real-time results.

Of course, you can also skip this part by just using the framework we’ve already set up on Flood. Tim Koopmans, our co-founder, created the first prototype of what would become Flood specifically because he got tired of having to set up this framework for every engagement.

How to run an on-premise load test with Flood Agent

‍ The easiest way to get started with an on-premise load test is by using Flood Agent. Flood Agent is an executable that you can install onto a machine that effectively turns it into a full-fledged load generator. Then, you can start a test from the Flood web interface and have your test script start running on your machines.

Agent takes care of a few typical problems we’ve discussed so far:

It uses containers to ensure that load generators are identically provisioned.
It automatically gathers CPU, memory, and network metrics on load generators.
It coordinates the execution of the load test so that you only need to start it from one central place.
It enables real-time monitoring of results while the test is running.
It aggregates and collates all the results for you into a dashboard.

We think Agent addresses pain points that we’ve seen over and over again in our own on-premise performance testing projects as well as in our customers’. Agent is a free download and works with all performance testing tools that Flood supports.

Conclusion

On-premise load testing is still a valid option despite the growth in popularity of cloud-based infrastructure, and it has its advantages and disadvantages. This type of load testing trades a little bit of flexibility for increased data security and control, which is a tradeoff that some companies may be willing to make.

On-premise load testing has traditionally included a lot of tedious setup and maintenance. Still, you can reduce maintenance effort by creating an internal load testing framework upfront that automates coordination, monitoring, and results collection. Using Flood Agent can mitigate time spent on this initial setup.

Ask a Flooder 13: How do you do mobile load testing?

Tue, 28 Apr 2020 23:21:08 +0100

Originally posted here.

How do you load test a native mobile app? I discuss the general approach to mobile load testing, what a device farm is, and why a device farm isn’t necessary for load testing. I also discuss the best mobile load testing tools to use.

Or listen to the audio version here:

Hi everyone, Nicole here again, back with another Ask a Flooder, and today’s question is:

How do you do mobile load testing?

Now, this question came from a customer who was under the misconception that in order to do mobile load testing, they would need to use what we call a device farm. A device farm is a collection of different kinds of mobile phones. You would ideally have the most popular makes of mobile phones (the hardware models) and the operating systems and system versions. You can imagine that this can very easily get a little out of hand to coordinate because of fragmentation. There are just so many devices out on the market, and then there’s the question of how you actually generate the load from that.

You would somehow need some sort of test automation script that was running and then coordinate os that they all run at the same time, and you would use that as a load test. As you can imagine, whether you use real devices or simulators, this kind of thing would get complicated and expensive very quickly. But the good news is that it’s unnecessary if it’s load testing that you want to do. Now I say that because, for functional testing or compatibility testing, you ARE going to want to test your application on a variety of different browsers. But with load testing, you’re really more concerned about the back end part of the equation. You want to generate load on your application servers. In reality, most mobile apps work by sending API requests to that server, usually by some sort of restful API. Luckily, API load testing has been around for a while, so we do have some industry-tested ways to run this.

What are the best tools for API load testing?

Essentially, you’re going to want to test protocol-level tools. That means that you don’t want to be using browser-level tools in which you’re using scripts to drive a full browser. All you really need is something that sends requests on the protocol-level. That’s great because it’s actually way more efficient.

You have two really good open-source options for this, JMeter and Gatling. I’ve already gone a little bit into the differences between these two in my previous video on how you choose a tool for load testing, which I will leave a link to.

And here are some other resources that will help you get started with mobile load testing. I also have another video about how to record network traffic from a native mobile app on JMeter, which sounds a little complicated, but I promise it’s not. In that video, I go over the steps with you, and I’m actually doing it in real-time with you as well. I show you the screen recording from my mobile and from JMeter on my laptop, so you can always pause and watch it at your own pace.

The second thing is that I already wrote a blog post, which is sort of a guide for mobile load testing, which goes into the steps into way more detail than I can here.

And lastly, I have written blog posts about two real-life examples of mobile load testing. The first one was the Iowa Caucus Reporter app. I ran through it from the workload modeling to the scripting and then the actual execution. More recently, I also wrote about the Robinhood app. Be sure to check out the links in the description to find out more.

Until next time, happy flooding!

Browser-based load testing in 2020

Thu, 23 Apr 2020 16:48:09 +0100

Originally posted here.

We first announced our open-source browser-based load testing tool based on Puppeteer, Flood Element, in February 2018. Despite our decision to make it an entirely separate tool from Flood, we’ve enjoyed using it so much as a team that several of us have written about it:

Tim Koopmans: Load testing for DevOps… with Browser Level Users (BLU)

Kevin Dunne: A new approach to load testing with browser level users

me: Prepare for viral videos with Flood Element

Lachie Cox: Why should you load test with real browsers?

our friend Eduardo Riol, as translated by Antonio Jimenez: Puppeteer vs. Selenium for test automation: is a new leader emerging?

Clearly, we’re huge fans of load testing a web application with real browsers. But does the wider software testing industry share that sentiment?

Browser-based load testing in 2020

Browser-level load testing is still pretty new, and there aren’t too many providers offering it right now. Luckily, we just so happen to have access to a database of load tests run on Flood since 2013. Here’s a bar chart visualizing data we’ve collected about the load testing tools flooders have used (we support Selenium, JMeter, Gatling, and Element).

Load tests executed on Flood over time, grouped by load testing tool

Browser-level load testing is increasing in popularity

In 2020, almost 35% of the load tests run on Flood are browser-based load tests.

While the total number of load tests run across all tools increased across the board, the percentage of tests that utilized Selenium and Element, as compared to JMeter and Gatling, has also increased from 2018. Both Selenium and Element are browser-based load testing tools.

By contrast, JMeter, a protocol-level load testing tool which previously enjoyed a 70% share of the traffic on Flood, now accounts for only a bit over 50% of usage. JMeter’s share of the protocol-level pie is higher than Gatling’s, but the percentage of usage for both tools appears to be contracting when compared to that of browser-based tools.

Element use is increasing faster than Selenium use

Despite the fact that Selenium has been around longer, 25% of all floods today are run using Element. This is a significant increase in just two years, and one that bodes well for the future. While we’re still fans of Selenium and still support it, one reason we see for the dominance of Element in today’s market is resource efficiency.

In our baseline tests, we’ve seen that Element can run from 20-50 users per Flood node (an AWS m5.xlarge instance), whereas Selenium can only run up to 5. This leads to some significant cost savings, as running the same number of users with Element requires fewer load generators than with Selenium.

Another reason might be the fact that Element uses Puppeteer in the background. Puppeteer, developed by Google, understandably shines when driving Google Chrome (and recently, Firefox). Puppeteer load testing is growing in popularity due to the wealth of metrics it has access to.

How browser-based load testing works

Traditional load testing is done on the protocol level by sending messages directly to the web server. These messages are sent without simulating the application interface, and the responses that are received from the server are not visualized, either. “Users” in protocol-level load testing scripts are actually threads, with each one sending a list of messages at predefined intervals, and we refer to these users as Protocol-Level Users (PLU). JMeter and Gatling use PLU to generate load.

The screenshot below shows what one request for a single action might look like in a JMeter test plan.

Example of an HTTP request with parameters on JMeter

In browser-based load testing, real browser instances are opened on load generators, and the load testing tool plays the part of a real user in navigating the web application using the GUI. We refer to the virtual users in browser-based load testing as Browser-Level Users (BLU).

Here’s an example of the same action in the JMeter example above, but this time in an Element test script:

step('Step 1', async browser => {
 //Click Start button
 let startButton = await browser.findElement('.btn-default')
 await startButton.click()
})

This version simply describes the button to be found and clicked, whereas the protocol-level version needs to be more precise in what exactly is passed to the server.

Both types of load testing tools can apply the same load on an application (as long as it supports both methods— see the next section). The main difference is in how the load is generated.

A protocol-level script will contain instructions on which messages to send, which protocols to use, which headers to include, and other information that a typical end-user would not see.

A browser-level script will contain instructions on how to interact with the application: which buttons to click, what to type in text fields, which links to follow, and other on-screen elements that an end-user would be familiar with. This leads us to one of the big advantages of browser-based load testing.

Pros and cons of browser-based load testing

Pros

Simple scripting— or no scripting at all

Protocol-level tools require in-depth knowledge about how an application works, what format it expects requests to be in, and how it responds. On the browser-level, you don’t need to know any of that. When scripting a login process, for instance, you just need to tell the script to type a username and password in the right fields and then click Log In. The script doesn’t need to know that the form submits an encrypted request using the Auth0 API (something you’d need to know for a protocol-level script).

In some cases, you might be able to get away with using something like our qTest Explorer plugin to record a business process, generating a script without any scripting at all. Even when you have to write or modify the script, though, it’s a lot less intimidating than most protocol-level scripts.

They’re also easier to troubleshoot. In addition to opening a real browser instance on your computer when you run a script locally, Element can also display this information on the console (shown here on iTerm):

‍ Screenshots upon execution on Element using iTerm

‍This allows you to understand what’s going on in the script and in the application, letting you detect bottlenecks faster.

Reduced test complexity

The streamlined requirements for scripting lead to a straightforward test. There’s no chance of getting request parameters or values wrong.

Ability to test entire stack from the user perspective

Unlike testing APIs, you don’t need to decide which server you want to hit or make sure you have the appropriate requests to hit all the servers. A browser-level test script interacts with page elements like a real user would, so all underlying application servers are tested. Pair this with cloud-based infrastructure on Flood, and you’ve got a compact package for testing your application.

Capable of testing any user behavior

Protocol-level load test tools struggle with some types of applications, such as single-page apps (SPAs), where a lot of the processing is done on the client side. Browser-level load testing tools shine here, because scripted interactions for these apps are no different for SPAs than other web apps.

Record network and user interaction times for front-end optimization

Browser-level tools show you response times from the user’s perspective: how long after a user clicks the log in button does the user see his or her account balance? This metric is a practical one that may differ wildly from the traditional response times that a protocol-level tool reports. Measuring real user experience is a great value add for your test.

Easier to maintain

Lower script complexity means that a browser-based test script is more resilient to changes in the codebase. A change in the underlying requests sent will not require a change in the test script, making it perfect for regression testing.

It’s also easier to pass on to other members of the team due to high readability. Here’s an example of how to select a value from a dropdown menu in Element:

await browser.selectByValue('#challenger_age','34')

It’s easy to see that the script is looking for an element with the ID “challenger_age” and selecting the value “34”. This step could correspond to dozens of requests that need to be scripted if testing on the protocol level, which makes those scripts harder to understand without a proper handover.

Cons

Browser-based load testing tools have a higher CPU and memory footprint than protocol-based tools. Test scripts in JMeter and Gatling send messages to and record responses from a server without really needing to parse responses. By contrast, browser-based tools parse the server response in order to access the Domain Object Model (DOM) elements on the page.

Using real browsers might be closer to the real user experience, but it also takes its toll on node resources. It is possible to run in headless mode, where browsers are started “invisibly” without actually graphically rendering them. However, even then, a browser-level tool can run fewer concurrent users on a node than a protocol-level tool.

Best tools and applications for browser-based load testing

While other commercial tools exist for this, we’re partial to open source tools at Flood.

Selenium

Selenium load testing is appealing because Selenium is already well-known for test automation and functional testing, and a lot of companies already have full test suites in Selenium. The possibility of reusing those resources and just running them at scale as load tests is, by far, the most compelling reason to use Selenium.

Puppeteer

Google’s just getting started with Puppeteer, and the amount of metrics it exposes is exciting: it seems like anything in Chrome Developer Tools can be accessed by Puppeteer. Imagine running a load test on ONE tool that gives you statistics about back-end performance (response times, the effect of bandwidth restrictions, resources returning errors) AND front-end performance (client-side script console logs, render times, time to first paint, etc). It’s hard to imagine a future where this ISN’T a valuable tool in the performance arsenal.

Under Puppeteer, of course, is Element— our load-flavored version of Puppeteer.

Is protocol-based load testing dead?

No. Testing on the protocol-level may have its drawbacks, but it’s popular for a reason. If you’re testing mobile apps or API endpoints, protocol-level testing tools allow you to focus on hitting what you want to test and only what you want to test. Sometimes you don’t want to test the whole stack. It really depends on your test scenarios.

Plus, protocol-level tools tend to be really resource efficient. We’ve been able to easily run about 1000 users per Flood node with JMeter and Gatling, easily outstripping browser-level tools. It makes sense, since JMeter and Gatling don’t need to start browser instances to generate load.

It’s significantly cheaper to generate very high loads with protocol-level tools. This efficiency makes protocol-level tools a good choice for high concurrency load testing and especially stress testing.

What are we doing to support BLU?

Clearly, our usage metrics show that browser-based load testing is on the rise, and we’re planning to ramp up our offerings in this area while also maintaining support for protocol-level tools. Here’s what you can look forward to from us:

New releases
- We just released version 1.2.3 of Element a few weeks ago, which addresses some bugs and improve stability for certain actions.
Dedicated Element team
- We’ve put together a team of developers who will be focusing on improving Element exclusively. Despite Element being an open source tool, we’re putting our own resources behind it to improve it faster.
Partnering with consulting services
- It can be difficult to get started on a new tool, and Element is no exception. While we provide a load testing platform to run Element scripts, some companies want help to create the scripts in the first place. To this end, we’ve partnered with expert consultants who have already been using Element with great success: Foulk Consulting and Planit Testing.

Conclusion

When choosing a performance testing tool, it’s important to consider the advantages and disadvantages of protocol-level and browser-level testing tools. We think both types of tools deserve to be in a solid performance and load testing strategy. They can also be used in tandem in a hybrid load testing approach, improving overall application performance and scalability.

In 2018, we predicted that testing web-based applications with real browsers would increase in popularity, so we’re thrilled to see our theory validated in 2020. Browser-based load testing is increasing in usage, and our open source tool, Element, is leading the charge. It’s exciting to be at the forefront of this revolution, and we intend to increase support for load testing with real browsers even more this year.

Ask a Flooder 12: Puppeteer vs Selenium: which one is better for load testing?

Tue, 21 Apr 2020 17:36:28 +0100

Originally posted here.

When it comes to software testing, the debate of Puppeteer vs Selenium is a hot one. Nicole tackles it here to determine which one is better for load testing in particular. Spoiler: The answer is still “it depends”.

Or catch the audio version here:

Hi everyone, Nicole here again, back for another Ask a Flooder, and today, the question is: “Puppeteer vs. Selenium: which one is better for load testing?”

We’re actually getting this quite a bit now as Puppeteer grows in popularity, but first, let’s discuss the similarities. They’re tools that you can use to drive browsers. The work by identifying and interacting with elements on a website in the same way that a user would. They are also both open-source, and both can be used for load testing. This is a great strategy because instead of writing tests for an automation suite and then tests for load testing, you can just write the one test suite and re-use it, so it saves a lot of resources in the end. To answer this question, I’m going to focus on load testing because that’s what my experience is in.

Difference between Selenium and Puppeteer

Pros and cons of Selenium

Now for the difference. Selenium supports more browsers than does Puppeteer. It supports Chrome, Firefox, IE, Safari, and Opera, so there are a lot more options to choose from depending on your application.

Selenium is also what I would call automation-first. It was built with automation in mind, which means it’s very good at that. It’s very robust, and if you have a complicated test scenario, Selenium is an excellent choice.

It’s also been around for much longer than Puppeteer, which is important because that means it’s had more time to build up a community. There are way more resources and tutorials for Selenium than for Puppeteer.

It’s also, unfortunately, very resource-intensive. That’s not something that you would typically care too much about from a test automation perspective, but when you’re running thousands of users on the cloud, it IS important to keep in mind that Selenium, just because of how it’s built and its complexity, can only run about five users per node. We’ve found that out at Flood, after baselining specifically an AWS m5.xlarge instance. So you can look at the specs of that and compare it with the machines that you’re wanting to run Selenium on, to see how that would equate for you.

Pros and cons of Puppeteer

Puppeteer is a NodeJS library developed by Google, which means it plays very well with Chrome. There are a whole lot of things that you have access to with Puppeteer, and if you’re using it with Chrome, pretty much anything in Chrome, including rendering times, you can see and access with a Puppeteer script. So that’s a huge advantage.

Unfortunately, that also means it supports fewer browsers. Obviously, it supports Chrome very well, and they recently came out with support for Firefox, although they do have plans to incorporate support for more browsers in the future.

One of Puppeteer’s advantages is that we at Flood actually created a tool on top of Puppeteer. It’s also open-source. It’s called Flood Element, and Element is built to be performance-first because obviously, that’s what we had in mind when we were building it. It takes everything that’s good about Puppeteer and makes it really easy to set parameters that you normally would expect to need for load testing, like users and ramp-up.

This also means that it’s way less resource-intensive. I said that with Selenium, you can run up to five users on an AWS m5.xlarge instance, and for the same instance type, you can run, from our experiments, anywhere from 30-50 Element scripts/users. That is a huge saving when you’re running a really large-scale test.

Should you choose Selenium or Puppeteer for load testing?

I would say that if you already have a large Selenium automation suite, if you have a complex test scenario, or if you aren’t running that many users for your load test, then Selenium is a great choice. If you ARE running thousands of users, though, and resource efficiency, and therefore cost efficiency, is really important to you, then choose Element.

My advice is always to try both. If you have the time, do a proof of concept with both of them. That way, your whole team can better understand what the pros and cons are for each one and decide which one is best going forward. Or maybe you don’t even decide on one. At Flood, we support both of them at the same price, so it depends on what you prefer.

Until next time, happy flooding!

Introducing Flood Agent: standalone infrastructure for load testing

Tue, 07 Apr 2020 19:03:42 +0100

Originally posted here.

Working from home during this pandemic and wondering how to start load tests from the office? Today, we’re officially introducing Flood Agent, a way to turn your own physical or virtual machines into load generators that you can control from anywhere with an internet connection.

TL;DR:

Installing Flood Agent on a machine lets you start load tests on it remotely from Flood—even if it’s within a corporate VPN.
Flood Agent is a simple binary that you can install on Linux, macOS, and Windows machines.
Once you’ve installed Flood Agent, your machines show up on the Flood web app to use as load generators or grids.

Boldly going where no one has gone before

Load testing within your company’s VPN and trying to coordinate it from home is like being on a maintenance pod trying to contact your starship, but a Mazarite ship is broadcasting a jamming signal, so comms are down. That Mazarite ship is your company’s firewall, and it can make testing difficult.

‍

The easiest way around this is to provision load generators in the cloud. However, because your application servers are still behind a firewall, you’d need to convince IT security to whitelist the IP addresses of the load generators. Depending on how tight security policies are, this may not be an option even in this new world of remote working.

Flood Agent is a way to start load tests on Flood from the load generators you already have. It’s a cross-platform agent that you can install on your regular load generators, allowing you to control them on Flood.

‍‍

Flood Agent is kind of like the Borg Collective. Each machine that you install Flood Agent on is assimilated into the Borg. A hive mind (you, on the Flood site) controls all Borgs, coordinating test execution with robotic efficiency.

Check out this video to follow along as I walk you through the process of setting up Flood Agent, or you can keep reading for the text version.

‍

Installing and configuring Flood Agent

Flood Agent is a free binary, and it runs on Linux, Mac, or Windows machines. You can download the appropriate version for your operating system and then follow the installation instructions here for more information:

Getting started on Linux

Getting started on macOS

Getting started on Windows

After installing Flood Agent, you can configure it according to how you want to use it. Start Flood Agent in your terminal and use the configure functionality. Here’s what that command looks like on macOS, but you can follow the links above for other operating systems:

./flood-agent configure

Linking your Flood account

The command above yields a prompt for a token like this:

This token allows Flood to run load tests on the load generator that you’ve installed Flood Agent in. To get the token, go to the API Access page on your Flood account (you already have one, don’t you?) and click REVEAL token.

You should see a string of characters like this:

‍

Copy your token (anything after the = sign) and paste it into the Flood Agent terminal window.

Choosing load testing tools

Next, you can choose which tools you’d like to set up and which version of those tools Flood Agent uses to run your tests.

‍

Starting with JMeter, Flood Agent asks you which of the tools you’d like to configure. You can choose to set up your tools at the beginning so that you don’t have to set them up again, or you can go back to this configuration wizard at any point to add another tool. Either way, you need to provide the tools that you select in one of two ways: via a Docker image supplied by Flood, or via a local installation that you already have.

For this example, I chose to configure JMeter and to use a local installation. Local installations give you the most flexibility because you can use the version of the tool that you prefer, along with any special plugins. However, if you already have Docker installed or don’t want to manually install JMeter separately, you can also use our Docker image for JMeter.

Selecting a local installation prompts you to provide the location of that directory:

‍

For JMeter, this looks something like /Users/nvanderhoeven/jmeter/apache-jmeter-5.1.1.

After you enter this file path, Flood Agent checks to make sure the directory is valid. You can also set the configuration for other tools in this dialog.

After you finish the configuration, the wizard writes to a config.yaml file in your Flood Agent directory. You can also edit this configuration file directly in the future if you prefer.

Verifying Flood Agent configuration

To run a final check on whether Flood Agent has been installed correctly, type ./flood-agent check. You should see a summary of the configuration options you selected as well as some connectivity and validity checks, like this:

‍

These green check marks mean you’re good to go! If you’re running Flood Agent from within a corporate network, though, you may see some errors here.

Troubleshooting network connectivity within a corporate network

If your company already has a web proxy, check out our guide on running floods through this proxy.

Please ensure that the load generator that you’ve installed Flood Agent on has access to the following endpoints:

<https://drain.flood.io>
<https://beacon.flood.io>
<https://vault.flood.io>
<https://flood-archives.s3-accelerate.amazonaws.com>
<https://logs.us-east-1.amazonaws.com>
<https://sns>.*.amazonaws.com
<https://sqs>.*.amazonaws.com

You may need to request these from your IT department.

If you get stuck, please contact us at support@flood.io. We’d love to go over your specific situation and get you up and running.

Once Flood Agent returns a successful check, you can start up your own grid!

Turning your machine into a Flood grid

A grid, in Flood terminology, is a group of machines to be used for load generation. To make your machine a grid, run this command:

./flood-agent --grid askaflooder

where askaflooder is the name of your grid.

‍

Once you see those [agent] ready and awaiting next job messages, you’re ready to run your test!

Running a load test on your own infrastructure

Log into your account on Flood, if you aren’t already, and create a new stream like you normally would, uploading your load testing script and data files. When launching your test, though, select the option for Hosted infrastructure and then choose the grid with Flood Agent running on it.

‍

Go ahead and launch your test. Congratulations, you’re now set up to run a load test on your on-premise load generators!

Giving it a go

Flood Agent is a free download, so you’ll only need to pay when you run load tests on Flood. However, we also have a free tier— perfect for a proof-of-concept while you’re working from home.

If you have any questions about Flood Agent, check out our documentation or email us. We’d be happy to get you started!

Download Flood Agent for free here.

‍

Ask a Flooder 09: What are the best tools for load testing?

Tue, 31 Mar 2020 20:35:36 +0100

Originally posted here.

One of our most commonly-asked questions is: “What are the best tools for load testing?” The answer will depend on a few things: your reason for load testing, the application you want to test, what kind of load testing you want to carry out, your team’s experience with scripting, and how many virtual users you want to run.

Note: In this video, I say that traditional load testing tools can’t simulate requests that are triggered by interactions on the client-side. This isn’t technically true: if you know what the requests are, you can build those requests manually on a protocol-level tool (like JMeter or Gatling). However, this takes more work compared to browser-level tools that just do this natively. ‍

Transcript:

Hi everyone, Nicole back again with another Ask a Flooder, and this time I’m tackling one of the questions we get asked A LOT, which is:

What are the best tools for load testing?

Why open-source load testing tools are awesome

Now, I want to preface this by saying that we really love open-source tools here at Flood, because 1) they’re really pretty full-fledged these days. You don’t have to make any compromises on features, and, in fact, some of these open-source tools have more features than some newer load testing tools just because they’ve been out longer. And secondly, they have large communities. Because they’re open-source, lots of people are using it, and lots of people are developing for it. So there are lots of plugins normally, and they support any kind of protocol that you can think of. And lastly, cost. It’s hard not to mention cost because it can be a really important factor depending on the budget of your project. And if you’re starting out as a load tester, I think that it might even be the best way to start with load testing, because you can take that knowledge with you and apply it to other projects and other companies regardless of the budget that they have.

Questions to ask yourself in choosing a load testing tool

Why are you doing load testing?

I always advocate starting with WHY. Why are you actually load testing? Is there a particular production defect that you noticed that you’re trying to solve? Is there already a problem, or are you just preparing to launch? These are things that are going to inform the type of tool that you use because you might want to make sure that that tool supports the protocols that you need.

What are you load testing?

Secondly, what type of app are you testing? Because if you have a single-page app, where your end-user is interacting with the page and client-side scripts are running to make requests to servers, then that’s something that most traditional load testing tools aren’t going to be able to do.

There are two general types of load testing tools: there’s browser-based, and there’s protocol-based. I’ll leave a link in the description talking about the differences between the two, but in general, if you want to go for browser-level testing, I would suggest Selenium and Element. Now, they’re both very good options. Selenium is probably the more robust in terms of automation of the two, but Element is better one in terms of efficiency because it was built with performance in mind. So Selenium’s automation-first and Element is really performance-first.

Now, for protocol-level tools, you can’t go wrong with either JMeter or Gatling.

What experience does your team have with scripting?

Thirdly, what’s the level of technical expertise that your team has? I think it’s important to be realistic about what your team can do, given the amount of time that they have. So if you have a team that has never done load testing and has never done any scripting before, then I would suggest using our Test Builder or something similar. On Flood, we have a Test Builder that is really just an easy interface that actually runs JMeter under the hood, but it just kind of simplifies JMeter and boils it down to what are the URLs that you want to test. That is a really easy way to get started, and it’s also a great [way to run a] proof of concept. If you have some automation testers on your team that are trying to do the load testing, well, maybe you should stick to tools that they already use. So it might be easier for them to get up and running with either Selenium or Element rather than learning a new tool. And if you have developers on your team, you might consider Ruby JMeter rather than the vanilla JMeter because Ruby JMeter is a lot easier to deal with— because you don’t have to deal with the XML files of a JMeter file (a normal JMX script). You can just plan out your load tests in Ruby. Or, Gatling is also a really good option because it allows you to write scripts in Scala. Scala is really really powerful, and you can create an entire framework with Gatling, which is also something to keep in mind.

How many virtual users do you want to run in your load tests?

And lastly, how many users do you want to simulate? Now protocol-level tools are going to really shine here because Element and Selenium both require a higher usage of resources on your load generators just because of the complexities involved in starting a browser instance, whereas protocol-level tools are just super efficient at sending these requests. So I would suggest that if you’re talking about thousands of users, I would use the protocol-level tools unless there’s a really good reason why that’s not going to give you the results that you’re looking for. Between JMeter and Gatling– well, they’re both excellent tools, but Gatling has been shown to particularly be good for really high levels of usage. But JMeter is also a good alternative, and you can run thousands of users of JMeter without any problem.

Just get started!

As always, I’d suggest just getting started. Pick whatever tool sounds like it might be the best for you, do a proof of concept on that one, and you can always try the other ones as well. On Flood, we don’t restrict you to just using one tool, so you can use one; you can use all four; you can run them at the same time if you want. It all depends on what you need for your testing.

Till next time, happy flooding!

Ask a Flooder 08: Why is concurrent users an ambiguous measure of throughput?

Thu, 26 Mar 2020 20:41:22 +0100

Originally posted here.

What is the meaning of concurrent users in performance testing? You might have seen concurrent users in JMeter or other tools, but what does it mean and how can you use the term properly?

‍

Transcript:

Hi everyone! Nicole here again for another Ask a Flooder, except this time it’s not really a question– it’s just something I want to talk about.

Source: The Flood Twitter account

So I’m going to put the tweet up here, but last month I actually tweeted about something that somebody else posted. Netflix actually reported some of their viewership figures, and one thing that they reported was that 76 million households watched their new show, The Witcher, which is a large amount of people. So I posted something about why concurrent users is actually an ambiguous way to measure throughput, and I want to expound a little bit more on that.

“Concurrent users” in performance testing - limitations

In load testing, the number of concurrent users is a pretty standard measure of throughput, and generally, it means the number of users that are accessing an application at any one point in time. And even on Flood, we use it because it’s a really good rule of thumb. When we want to get a feel for how big a test is, load testers generally ask, “How many users?” However, we should be aware, as load testers, that that doesn’t always give us the full picture.

What type of user is it?

For instance, the type of user that it is can really change the traffic. Sometimes mobile browsers will optimize requests so that they send fewer because they realize that mobile networks are slower.

What is the user doing?

Secondly, what is the user doing? Are they browsing on different tabs all the time? Or are they just refreshing the content on the same tab? Because if they’re just using the same page and fetching it over and over again, then caching could come into play, so that would create a different load on the system.

How long is the user doing it?

And thirdly, how long is the user doing it? Are we talking about a sustained load for an hour, or is it a shorter time period, like a spike test of five minutes where all the users just ram a system? There are things like the number of requests per second that you need to take into account, which is affected by the think time and the delays that you set in your script. Perhaps your script has a user that just sends a request, then waits for a minute, and then sends another request. That’s very different from a user just sending one request after another as fast as possible.

When does the user do it?

Lastly, it’s also important to take into account the seasonality of the request. When does the user do it? If you have a timesheeting application, for instance, then it might only get traffic on Fridays and a little bit on Monday, so you need to specify, when you report these numbers, whether it’s for the peak time period on Friday - you know, around 3 pm or something - or you’re reporting on figures from a Wednesday when there really wouldn’t be that many people accessing it.

In the case of Netflix, what they counted as a view was only a two-minute watch time. So if you’re one of those people who went into The Witcher and didn’t watch past two minutes, that still counted as a view. So the 76 million users is still not completely accurate, because not all of those people would have watched the entire thing.

Conclusion

Concurrent users is a good way in order to get a quick feel of the throughput that a test outputs, but it really needs to be examined in context along with things like requests per second and the network throughput (kbps) in order to get a really good understanding of how much load a test is generating.

Till next time, happy flooding!

Ask a Flooder 06: How to record traffic from a mobile app with JMeter

Thu, 19 Mar 2020 20:50:44 +0100

Originally posted here.

How do you record the traffic from your native mobile app using Apache JMeter? In this video, I explain step-by-step how to begin load testing a native mobile app.

‍

You might want to be doing this because you’re load testing a mobile app but either you don’t have access to your developers right now so they can’t help you with how the requests are made or what the requests are, or maybe you just want to verify what you’ve already been told. Either way, I’m going to show you how using my Android device (it’s a Samsung Galaxy Note 10+) and a Macbook Pro 13". However, the same principles should apply to other mobile devices and laptops.

Step 1: Download and install the mobile app

The first thing you’ll want to do is download and install the app. Now, if the app has already been released, then that’s no problem. You just go to the Google Play Store and download it. However, if it hasn’t been released yet and you’ve been given an APK file for the app, then you’ll have to tweak your settings a little bit in order to get your phone to allow you to download it.

And I’m going to do this right with you. So go into Settings: pull down the little notification screen and hit the little gear icon, and you’ll see a bunch of settings there. Scroll down all the way to the bottom until you see About phone. Click that and then click Software Information. You’ll see the Build number halfway through the screen here– and I know this is weird, but click on it seven times. You have to draw your pattern if you have one set up, but you’ll see here that it says Developer Mode has been enabled. That’s all you need to do; from now on you can install any APK, but be careful, because if you don’t trust the source, you know, they could do some damage to your phone.

Step 2: Set up the HTTP(S) Test Script Recorder on JMeter

The second step is setting up an HTTP recorder on JMeter. So let’s head over to our laptop. Make sure you’re connected to a wifi network. Now, this is a completely new JMeter test plan here. So just right-click on Test Plan and then hover over Add, and then you’re going to go down to Non-Test Elements. Select the HTTP(S) Test Script Recorder. You’ll see here that the port it uses is 8888. We’ll need that for later. Now right-click on the Test Script Recorder, hit Add, and then Listener. We’ll add a View Results Tree to be able to capture the results that we need later. Now we’ll click on the Test Plan - right-click - and then we’re going to Add a Thread Group. Now let’s right-click on the Thread Group and go to Add > Logic Controller > Recording Controller. This is where the requests that we’re going to send are going to be recorded, and we’ll see both the requests and the responses in View Results Tree.

Step 3: Set up your mobile to use the HTTP(S) Test Script Recorder as a proxy

Now, the third step is to set up your mobile to use a proxy. So, back on the mobile, go back to Settings and hit Connections. And then click on Wifi. Now verify that the wifi network that you’re connected to on your mobile is the same one as the wifi network that you’re connected to on your laptop. If it’s the same, click the gear icon. And then hit Advanced all the way at the bottom. Now the Proxy settings are on the second option here so hit the down menu here and then hit Manual. Now, for the Proxy host name we’ll have to go back to the computer to see what our local IP address is.

So, in order to do that, you just have to go to System Preferences. I’m doing it from the Apple menu here and from System Preferences, click on Network. And once you’ve selected the wifi network, hit Advanced, and then TCP/IP. So what you’re looking for will be this IPv4 Address. Now you’re going to type that into the Host name field on your mobile. So let me just type that in.

You’ll note that this is actually a local IP address. It’s not your public IP address. Now for our proxy port, we’ll go and look at the laptop to see the Test Script Recorder Settings and it was 8888, so that’s what we’re going to put in here. All right. And then click on Save.

Step 4: Start recording

Step 4 is recording the traffic. Now, go back to your laptop on JMeter and click on the Test Script Recorder. Now we’re going to click on Start. It’ll come up with this popup box here and you can just click OK since we already trust the source. You’ll see this little popup here to confirm that we are recording traffic. Now let’s see if we’ve actually configured it correctly.

On our mobiles, let’s go back and open the app that you want to use. So I’m going to be using the IowaReporterApp. I just did an article on this on The New Stack (link in the description below) about how I would load test the Iowa Caucus Reporter App. So that’s what I’m going to be using for this tutorial as well. So I’m going to open that (“getting ready to caucus”) and I don’t have a valid precinct ID so I’m just going to put flood.io and I’m going to click Login. So it’s telling me that the connection isn’t private. I don’t actually expect this to work, because the app has been taken down and I don’t have valid login credentials, but I’m still just going to go forward. So, Proceed. You might have to do this as well if your certificates haven’t been set up correctly.

Okay, so now we’ve gotten this error here and it says, “Oops, something went wrong”. So it’s handy to know what it looks like on your mobile so that you can go into JMeter and see if that’s what you see as well.

So on JMeter let’s go to the View Results Tree and it looks like Google Play tried to do a few things beforehand but what we’re really looking for is this IDP caucus thing. So, these are the requests that my phone made and these are the responses. So this is where we got the certificate error because it says “certificate unknown”. Now this is the actual request we sent for login. You can see the request here - the full request - and also the full response. Now, just to check we’ll make sure we got the same response in the body here on JMeter that we saw on our mobile phones.

And here it is; it says “Oops!, something went wrong”. You can of course just search for this. So we’ll stop the recording now because we’ve confirmed that we’ve gotten what we wanted. So in the recording controller, we can see the requests that were recorded from our mobile app and the first one looks like it was one to Google so that’s not one we want. If we were doing this again, we could actually go into the Test Script Recorder, go into this Requests Filtering tab, and we could just filter out or exclude the domains that we don’t want to test, like Google. But this looks like the request that we were actually looking for. You’ll see the full request here with all the parameters and the values.

What’s next?

And that’s how you record traffic from a native mobile app on JMeter. From here, you can then play around with the request. Like we saw, there were a lot of dynamic values that were recorded in the request. We would probably want to correlate those and make sure that we’re doing those correctly. But this is a really great starting point for testing a mobile app.

Till the next Ask a Flooder, happy Flooding!

FOMO and performance testing: Why Robinhood went down

Mon, 09 Mar 2020 20:29:40 +0100

Originally posted here.

Angry customers flocked to the internet when the Robinhood investment brokerage app went down last Monday and part of Tuesday, leaving them unable to trade stocks on a day where Wall Street reported historic gains. Could this have been prevented?

The answer is yes. Applications should always be assessed for risk and tested accordingly. Risk-based load testing helps unearth problems that would otherwise only surface in production. Testing applications end-to-end as a system instead of just as a collection of isolated components allows us to build software that is more resilient. The problem is that this type of testing can be difficult to carry out. In this article, I’ll go through the entire process.

What is Robinhood?

Robinhood is an app I’ve personally had my eye on since it opened a waiting list about five years ago for Australians wanting to see the American company come to our shores. It doesn’t look like it’s happening now, but if that waiting list still exists, I’m on it.

Robinhood is a brokerage app that takes inspiration from the legend it was named after to “give to the poor,” bring zero-commission trading to its customers. Instead of earning from commissions, Robinhood makes its money from a variety of other methods, including premium platform fees, interest on customers’ uninvested capital, and payments received in exchange for coursing customers’ orders through third-party market-makers. It’s an enticing premise, and one that the company has used in conjunction with mobile apps to appeal to millennials and technophiles.

Source: From Robinhood’s homepage ‍

What caused the Robinhood app to fail?

Unfortunately, its wild success with the tech-savvy crowd also meant that when it went down last Monday morning in the US, its customers took to the internet in droves to draw attention to the failure. In a blog post from Robinhood, the company’s co-founders, Baiju Bhatt and Vladimir Tenev, admitted that the outage was due to “stress on [their] infrastructure— which struggled with unprecedented load.”

The outages you have experienced over the last two days are not acceptable and we want to share an update on the current situation. Our team has spent the last two days evaluating and addressing this issue. We worked as quickly as possible to restore service, but it took us a while. Too long. - Baiju Bhatt and Bladimir Tenev

Bhatt and Tenev went on to explain that the load on their servers had caused a “thundering herd” effect. This effect describes a situation in which a backend server receives a large number of concurrent requests and, instead of different threads processing these requests simultaneously, all threads attempt to process the same request.

If you’ve ever played four-player Overcooked, it’s like having everyone sprint to a fire extinguisher to try to put out a fire from overcooked pasta, but instead, the constant button mashing means you just pass the extinguisher around and yell at each other while the fire rages on. Before you know it, the entire kitchen is on fire.

Source: From Twitter user @csac0425

So the technical cause of the outage was an infrastructure problem.

But I think the REAL cause was FOMO.

What is FOMO?

FOMO is the Fear Of Missing Out, and it’s a phenomenon that’s been particularly exacerbated by social media. Instead of only hearing news from people you interact with in the physical world, it just takes a few seconds to open up Twitter on your smartphone and see what thousands of people are talking about. Twitter will even helpfully tell you what’s “trending” in your country, and other social networks also employ algorithms to determine what messages to show you.

How does FOMO affect application performance?

We can see FOMO in how people have reacted to the coronavirus COVID-19. Despite pleas to leave masks for medical professionals at real risk, people have bought out the stock for these masks despite living in countries with little to no sign of the coronavirus. In Sydney, Australia, residents are stockpiling toilet paper (of all things) in large quantities, clearing out shelves from supermarkets, and getting into knife fights over toilet paper due to their panic at the thought of doing without.

This photo of an empty toilet paper aisle was taken by our Principal Engineer, Lachie Cox, in a supermarket in Sydney last week. ‍ FOMO of this magnitude ultimately also manifests itself in the stock market. The Dow Jones Industrial Average, which represents the performance of the top 30 largest companies in the US, dropped by 3,500 points last week amidst fears that the coronavirus would stall production in countries most affected by COVID-19, leading to lower returns across the board.

Then, an abrupt about-face. The Dow went from recording its biggest ever loss in a single day on Thursday to posting a record gain of 1,200 points on Monday.

Guess who gets to adjust to all this volatility.

Robinhood’s founders reported that the “highly volatile and historic market conditions; record volume; and record sign-ups” contributed to the infrastructure issues they faced. We can only imagine just how many users they saw on their systems during this period.

FOMO creates mass hysteria and panic, both of which cause customers to behave irrationally and likely very differently than previously recorded. So how do you create a workload model that accounts for this volatility? How do you tailor your load tests for FOMO? I’ll run you through the process, using Robinhood as an example.

Does your application require FOMO load testing?

All testing should be risk-based. Start with why: why should you test an application component? Why does your application require this kind of testing or that one?

In Robinhood’s case, there were a few indicators that pointed to the necessity of FOMO testing:

A customer base made up of a younger, tech-savvy generation. ‍This demographic is very likely to take to social media with complaints, making negative publicity a real risk.
Complexity. ‍Robinhood’s systems must receive customers’ orders, forward and place those orders to market makers, and display real-time market data. Robinhood customers can also purchase options on stocks as well as cryptocurrency, further increasing the complexity.
Financial component. ‍Anything that involves taking people’s money should have a high priority for any testing.

When it comes to your money, we know how important it is for you to have answers. - Baiju Bhatt and Bladimir Tenev, co-founders of Robinhood

Record-breaking growth. ‍Robinhood had 1 million users in 2016, 6 million in 2018, and 10 million last December, according to CNBC. Exponential growth brings some major growing pains.
Industry history of expensive failures. ‍Quartz points out that software or infrastructure failures have cost the financial industry millions in recent years.
Recent legal action. The regulatory body FINRA fined Robinhood US$1.25 million just last December for placing orders for customers without looking for the lowest price. While this isn’t a huge fine for a company that was valued at US$7.6 billion last July, any legal action should make a company tread more carefully.

These are solid reasons to warrant load testing Robinhood’s systems beyond just “expected” load.

Planning for FOMO

Once you’ve determined that your application is susceptible to FOMO, the next thing to do is plan how to structure your load tests to include its effect in your test runs.

What type of app is it?

Robinhood consists of three apps: two for mobile (iOS and Android) and one for the web. It is currently only available to US residents, which means that getting mobile apps downloaded is problematic as it would entail changing countries in the app stores to get access. Since I live in the Netherlands and I don’t want to do that, I’ll focus on the web app specifically when scripting the load test, and I’ll assume that the same underlying application servers service both web and mobile apps.

Workload modeling

Business processes to test

Given that the app went down on a day where markets were up, we can assume that most of the people on Robinhood’s apps were doing one of two things:

Signing up
Buying shares

We should ideally test both, but let’s focus on signups here because it’s easier to test without access to the actual app (which requires a US social security number).

Number of virtual users at peak load

Let’s try to come up with a number of users that we can use as Robinhood’s peak hourly load. This task would be easier if Robinhood shared their analytics, but we can still make some educated guesses with what has been made public.

Source: Image from Apptopia

According to Apptopia, Robinhood’s highest number of mobile installations was in 2018, when it saw 2 million downloads of its mobile app in a quarter. This period coincides with when Robinhood released cryptocurrency trading support, which looks like increased signups—in the third quarter of 2019, this figure was about 1.5 million. 2 million quarterly downloads translates to about 666,666 downloads per month. Let’s call that 700,000 and take this number of mobile downloads as a starting figure.

The stock market is only open on the weekdays, so we can divide 700,000 signups a month by 20 days, and we get 35,000 signups a day. The markets are also only open for about 7 hours a day, but the load probably isn’t evenly spread out across the 7 hours— I would expect that people would have more time during lunch to sign up for Robinhood. So let’s say that the majority of those signups would happen within three hours (from 11 am to 2 pm, for example). By dividing 35,000 signups per day by 3 hours, we get about 11,667 users per hour. Let’s round that up to 12,000.

How do we translate this to the number of virtual users we need to run? Let’s think about how long each user stays on the app. The signup process does require approval, so that’s a hard stop— new users won’t be able to sign up and then start trading immediately afterwards. This approval could take up to a day, or up to seven days if documents are required.

Looking at the application form itself, we can see it is relatively standard and requires only information that most people are likely to have handy (social security number and contact details). I timed how long it took me to go through the part of it I could access, and I estimated that 10 minutes would be sufficient time to go through all the tabs.

Source: Robinhood

We want to spread out users evenly across the hour— that is, we don’t want 12,000 users to sign up and leave, all within 10 minutes. We want equal portions of those users to be signing up throughout the entire hour. Dividing 12,000 by 6 (the number of 10-minute intervals in an hour), we get 2,000. 2,000 is the number of threads we need active at any one point of time. Each of those threads signs up one user for 10 minutes, then signs up 5 more users until the hour is up.

So we have our figure: we need 2,000 virtual users per hour to simulate 12,000 signups in an hour.

Test scenarios

In addition to running the standard battery of load tests, testing for FOMO requires exploring the upper limits of the application. Assuming the peak load testing goes well, we can use the 2,000 virtual users per hour figure from our peak load calculations as a starting point for these more destructive tests. To this end, here are some test scenarios I’d consider running if I were testing the Robinhood app:

Soak testing. ‍Soak testing means running typically a little less than the peak load profile, but over a longer amount of time. In Robinhood’s case, perhaps we could run 1,000 virtual users for 8 hours. We would expect that the response times reported during this test would be the same; otherwise, there is likely a performance bottleneck, such as a memory leak.
Stress testing. ‍Stress testing means increasing the number of users on an application at regular increments until the application crashes. We can start with 2,000 virtual users per hour, and then add 500 users every 30 minutes to see how the application handles it. This test helps determine how much room there is for growth.

Source: Stepped load profile for stress testing, generated in JMeter ‍

Resilience testing. ‍Resilience testing involves running the peak load test and then simulating an outage by turning off key components to see how the application behaves. If two servers share the load, for instance, we could turn one off and check to make sure that the user sessions on the terminated node are moved over onto the remaining node. This test shows whether the application recovers gracefully from an unforeseen event.

All these tests help us prepare for the unexpected, and they might have identified the “thundering herd” problem that contributed to Robinhood’s outage.

Scripting for FOMO

Tool selection

To test mobile as well as web apps, I would typically use a protocol-level load testing tool like JMeter to simulate the load. However, I’ve already done that. I also noticed that Robinhood makes use of some dynamic parameters, such as a csrf token:

‍

Source: Chrome Developer Tools on Robinhood ‍ We can usually find the value of this token in the response of the previous page. It is a security feature to allow Robinhood to verify that the same users make all requests in the same session. Tokens can be scripted around in a tool like JMeter, but it can be time-consuming. So for this article, I decided to show another tool that makes this problem go away. I used Flood Element, which is an open-source tool we created based on Puppeteer. One of its advantages is that it runs on the browser level. Instead of diving into HTTP requests and tokens, I just told Element which buttons to click.

I wanted a script that would do the following things:

Navigate to Robinhood’s home page.
Click on “Sign Up”.
Click on the “Get Your Free Stock” button.
Fill out the signup form.

Note that I stopped short of actually submitting the form because I don’t want to create dummy accounts on Robinhood’s database— I just want to show how this flow can be scripted with Element.

To help me identify the elements on the page that I wanted the script to interact with, I used Chrome’s built-in Developer Tools. Specifically, I used the Inspect Element button (in orange below) to click on a field and find where it was in the code.

‍

For instance, to find out how to identify the email field, I right-clicked on the code on the right, hovered over Copy, and then clicked Copy selector.

‍

This gave me the value #signup_email_field, which I then used in the script like this:

//Type Email address
await browser.type(By.id('signup_email_field'),'nicole@flood.io')

Here’s the script that I ended up with:

import { step, TestSettings, Until, By, MouseButtons, Device, Driver, ENV } from '@flood/element'
import * as assert from 'assert'
export const settings: TestSettings = {
 loopCount: -1,
 clearCache: true,
 disableCache: true,
 actionDelay: 8,
 stepDelay: 10,
 screenshotOnFailure: true,
 userAgent: 'flood-element-test',
}
export default () => {
 step('01_Home', async browser => {
 //Navigate to Robinhood homepage
 await browser.visit('<https://robinhood.com>')

 //Validate text
 let validation = By.visibleText('It’s Time to Do Money')
 await browser.wait(Until.elementIsVisible(validation))
 await browser.takeScreenshot()
 })

 step('02_Signup', async browser => {
 //Click "Sign Up"
 let signupBtn = await browser.findElement(By.xpath('//a[@href="<https://signup.robinhood.com>"]'))
 await signupBtn.click()

 //Validate text
 let validation = By.visibleText('Free Stock Waiting For You')
 await browser.wait(Until.elementIsVisible(validation))
 await browser.takeScreenshot()
 })

 step('03_GetFreeStock', async browser => {
 //Click Get Your Free Stock
 let freestockBtn = await browser.findElement(By.xpath('//a[@href="<https://signup.robinhood.com/?">]'))
 await freestockBtn.click()

 //Validate text
 let validation = By.visibleText('Make Your Money Move')
 await browser.wait(Until.elementIsVisible(validation))
 await browser.takeScreenshot()
 })

 step('04_FillForm', async browser => {
 //Type First name
 await browser.type(By.xpath('//input[@name="first_name"]'),'Nicole')

 //Type Last name
 await browser.type(By.xpath('//input[@name="last_name"]'),'van der Hoeven')

 //Type Email address
 await browser.type(By.id('signup_email_field'),'nicole@flood.io')

 //Type password
 await browser.type(By.xpath('//input[@name="password"]'),'demo123456')


 await browser.takeScreenshot()
 })
}

In every step, I also added a takeScreenshot() , which is one of my very favorite features of Element. It’s so useful to be able to save these screenshots and use them to determine exactly what’s happening in the script. Here’s what it looks like running my Element script locally. The automated browser is on the left, and my terminal (iTerm) is on the right, running Element.

‍

Running a FOMO test

Now we’ve got a script that runs locally. How do we run it at scale? Additionally, Robinhood won’t allow users outside the US to sign up. How do we make sure that our virtual users are allowed to sign up?

The answer is a load testing platform like Flood.

With Flood, running our script is a matter of uploading the script and selecting some options.

The Flood test design interface ‍ Flood also lets us choose which regions to generate load from. Below I’ve got it set up to run for an hour from four different US cities, with each one starting 500 users. This configuration gives us our 2,000 virtual user figure.

‍

If this peak load test goes well, we could then go on to run the other test scenarios that we described earlier. Each one puts pressure on application servers in different ways, identifying performance bottlenecks.

Results and reruns

Since Robinhood is a live application, I didn’t run the test— it’s never a good idea to load test a domain you don’t own when you’re talking about thousands of users. However, here’s a shareable link to another small Element test I ran previously.

Real-time dashboards like that one on Flood allow you to get a quick look into how the load test is running and to make any changes if necessary. When testing for FOMO, I think they’re invaluable. For example, during a stress test where users are added at regular intervals, a real-time dashboard would help you identify increases in response time as a result of the increased number of users. It would also let you monitor the test and stop it when necessary.

From the link above, click on the right arrow on the lone transaction.

That will take you to the Transaction Detail page. Remember those screenshots from Element? If you have any in your script, you’ll see them here, arranged according to time.

This lets you troubleshoot issues during FOMO testing and react quickly.

Conclusion

Even with sound load testing strategies in place, it’s challenging to plan for the massive spike in traffic that FOMO can bring, yet it’s necessary to do so. FOMO, as a phenomenon, is only going to increase in magnitude as social networks make it easier to spread fear, uncertainty, and doubt as well as information.

We may not be able to determine precisely what will break, or when, but what we can do is put systems into place to plan for it. We can assess an application’s susceptibility to FOMO and plan for end-to-end performance engineering of the system as a whole. We can start with the assumption that applications will fail and then determine the most likely candidates for that failure. We can routinely expose systems to traffic and circumstances that are extraordinary, and in doing so, we can improve our preparedness for something like FOMO.

FOMO is irrational and unpredictable— but that doesn’t mean it needs to be unexpected.

Load Testing Lessons from the Iowa Caucus

Fri, 06 Mar 2020 21:59:02 +0100

Originally posted here and here.

TL;DR? Here are the highlights in this article:

Shadow, Inc. created a mobile app for the Iowa Caucus in February 2020, and the app failed, causing reported results to be delayed. The app reported different vote totals due to a system integration bug where the output was in a different format than what it needed to be. Here are the steps to load testing a mobile app:

Test planning: Does the app require load testing? If yes, what is the underlying technology? How do we create a workload model?
Scripting: Which mobile load testing tool should we use? How do we record traffic from a mobile device?
Execution: What load test scenarios should we have? How can we scale up the load generated? Conclusion: While the Iowa Reported App failure was not due entirely to load testing, load testing could have helped make it more robust.

A company called Shadow, Inc. made headlines earlier this week when they developed an app to count and report votes during the Iowa Democratic Caucus in the US. This app, called IowaReporterApp, ended up failing on several fronts, resulting in a days-long delay in publishing the total votes, during which manual methods had to be employed.

Source: @ShadowIncHQ on Twitter

This delay led to significant criticism from the software testing industry as well as from the general public. Some developers raised questions about the code quality of the app. The app inaccurately reported voting totals that differed from the Democratic Party’s own manual reports, suggesting an issue with functional testing, particularly system integration testing. Shadow’s refusal to accept the Department of Homeland Security’s help with security testing also raised eyebrows. What was perhaps the most damning, though, was the question of whether such an app was necessary in the first place, when cheaper and more effective solutions existed.

What I’m most interested in, however, are the load issues that the app faced. Let’s rewind a little to before the caucus. How would I have load tested an app like the IowaReporterApp?

Planning a mobile load testing

One of the first things to figure out when planning a load test is whether a load test is even necessary.

Do you even need to load test your mobile app?

To answer that, I had to find out more about the app.

In the caucus, residents of Iowa who registered themselves as Democratic voters gathered to vote on whom they thought should be the party’s candidate for the presidential elections later this year.

There are a few things that I think make this app the perfect candidate for load testing:

It was highly publicized. The caucuses would always have been an event of international interest, but the hype surrounding electronic voting increased media attention on the caucus even further.
It was mission-critical. The precinct chairs would use the app as the primary device for capturing votes.
It needed to handle significant simultaneous load. There were over a thousand precinct caucuses organized, each with a precinct chair that would be using the app.

For these reasons, the answer is a resounding “Yes.”

What type of app is it?

The best way to learn about an app is to ask its developers. In this case, I opted to do a little research on my own by downloading the .apk file that Motherboard (Vice) made available and looking through the code myself.

The app is a mobile-only app for Android and iOS devices, written mostly using Facebook’s React Native package. The primary function of the app was to sent the vote totals to Shadow Inc.’s Google Cloud Functions server.

The technology involved is worth noting because it can help determine which load testing tool to use. Since the Iowa Reporter App, like most mobile apps, sends requests to a RESTful API, JMeter or Gatling, both well-tested, robust, and open-source protocol-level tools with large user bases, were perfect for the job. I decided to go with JMeter, my personal tool of choice.

How do we create a workload model?

Now let’s talk about the workload model, which is a plan for how the load we generate needs to look. The objective here is to simulate the load that matches the expected real-life load as closely as possible.

At this point, I would typically look at historical data to try to see the amount of load and the load profile. If I had metrics on the server traffic after previous releases of the app using something like Google Analytics or an APM tool, I would have used that data as a starting point for the workload model of this release. However, in this case, the app hadn’t been publicly released before, so I needed to make some educated guesses.

According to Wikipedia, there were a total of 1,777 Iowa precinct caucuses, of which 99 were global satellite caucuses set up for Iowa residents temporarily not able to attend the local caucuses.

In a Democratic caucus, the actual voting process generally takes about two hours. In the Iowa caucus, the precinct chairs would have used the app twice - once for each round of voting. Around the end of the two hours, each chair’s app would have sent the main payload, containing total votes for each candidate, to the backend server. All this is done simultaneously across all 1,777 precincts.

At a minimum, there are two main events here:

The login - Precinct chairs needed to log in to the app, which would presumably have happened around the same time across all precincts towards the beginning of the voting process, because the app also contained instructions on how to run a caucus.
The submission - This is the transmission of the total vote counts for each candidate, sent to Google Cloud Functions, and would also have happened around the same time.

It’s possible that the app also sent other requests between these two events, but since most of the app is inaccessible without a valid login, I’ll focus on the initial login request.

In a situation like this where I don’t have previous data to show how spread out those requests was, it’s prudent to err on the side of caution. Assuming the precinct chairs triggered those requests within 10 minutes of each other and rounding up the number of requests for some wiggle room, I estimated that there would have been 1,800 login requests within 10 minutes.

I decided to test this as a gradual ramp-up of users within 10 minutes.

Scripting for mobile

Okay, time to write the script. Since I didn’t have the luxury of being able to ask the developers for the format of the login request, I did the next best thing and tried to find out myself by recording the app traffic in action.

Recording mobile traffic on a laptop with JMeter

Here’s how I was able to see the request from the mobile app:

I downloaded the app to a spare Android mobile by using the APK link above.
I set up an HTTP(S) Test Script Recorder on JMeter on my laptop with the port 8888. I also made sure my laptop was connected to my wifi network.
I set up my mobile to use a proxy using my laptop’s local IP address on the network and port 8888.
Then, I went back to JMeter on my laptop and started the HTTP(S) Test Script Recorder.
Finally, I navigated to the app, typed in flood.io in the Precinct ID field (obviously not a real ID), and clicked the “Click here to log in” button.

Here’s the traffic as seen on JMeter’s View Results Tree listener afterward:

And here’s the recorded request within the recording controller:

Analyzing the request

It looks like at least the initial login request is a GET to an Auth0 service, with several parameters being passed in the body. It also has some authorization headers.

The parameter values also look like they might be dynamic, meaning they change from session to session. At this point, I would typically change some of these from being hardcoded values into dynamic ones so that the application receives a request in the right format, and I could look through Auth0’s API documentation to find out how the authentication message should be formulated.

Since I don’t have a valid login for testing purposes, I skipped this step and just used the raw request for demonstration purposes.

Preparing to run

Think time is a delay between requests meant to simulate the time it takes for a user to think before taking another action. Without think time, JMeter would fire off requests one after another, which often isn’t realistic and drains resources. I don’t want that, so I added a JMeter timer. There are plenty of timers available, but I settled on a Uniform Random Timer.

The CEO of Shadow told Motherboard that many precinct chairs were confused by the login process as they had to put in three strings of numbers in the right sequence. So, I decided to work that into the script with the timer. I set it up with a constant delay of 2 minutes and a random delay of 1 minute so that the delay would vary from 2 minutes to 3 minutes to simulate a precinct chair attempting to log in a few times.

Execution: Running your mobile load test

According to Motherboard, Shadow has already taken the app servers offline. However, it’s not a good idea to run a full load test against an endpoint that you don’t own, so I decided not to run a test.

Usually, though, I’d want to start with a shakeout test of 10 users or so, and then gradually increase the user load until the peak of 1,800 users. I also would have played around with changing the load profile, such as by increasing or decreasing the ramp-up or steady-state durations, experimenting with stress tests and soak tests, and adding traffic from another availability zone to simulate the 99 satellite precincts around the world. Running multiple load generators can quickly get out of hand if you want to use physical machines, so I would run load tests on the cloud with something like Flood. All of these little changes would have given valuable data that would help me gauge the app’s performance in production-like situations.

Conclusion

Building quality software is not a trivial task. There are many moving parts to account for, and critical errors can be introduced in every stage of an app’s development. While load issues were not the main culprit in the failure of the IowaReporterApp, a simple load test like the one I describe here, might have improved app stability and reliability. Even in cases where load testing doesn’t seem necessary, it can still uncover gross issues in configuration, usability, or security for an app developed on a deadline.

How AirTasker uses Ruby JMeter and Flood for their load testing

Mon, 24 Feb 2020 18:23:31 +0100

Originally posted here.

In the recent RubyConf 2020 held in Melbourne, Australia, Nancy Cai from AirTasker talked about how they built a performance testing framework using Ruby JMeter and Flood. Luckily, we had some pleasantly surprised Flooders in attendance to take notes during this fantastic talk!

AirTasker is a platform allowing the outsourcing of everyday tasks. When discussing the reasons that AirTasker decided they wanted to carry out performance and load testing, Nancy told a fascinating story about an overly successful marketing campaign—a good problem to have, but a problem nonetheless. The Sydney Cats and Dogs Home, an animal shelter, teamed up with AirTasker to pay one lucky person $500 to spend the day playing with the resident animals. The prospect of this dream job created an enormous surge in AirTasker’s traffic that caused alarm bells to go off on the underlying application servers, resulting in an end-user experience that was less than desired.

Nancy included this hilarious slide about some common feelings regarding application performance.

Steps for running a performance test, the AirTasker way

To avoid repeating that mistake, Nancy and her team set out to run performance testing by using the following simple steps:

Determine performance criteria.
Configure the test environment to be as production-like as possible.
Plan and design based on user behavior, including preparing test data.
Implement test design and set up the framework (which in AirTasker’s case involved Ruby JMeter as the performance testing tool and Flood as a load testing solution).
Run test.
Analyze and fine-tune results.

Setting up a load testing tool and framework

When it came to selecting the right tools for the job, AirTasker wanted tools that met the following criteria:

Ability to simulate production-like traffic easily
Widespread use in the industry
Ease of creation and maintenance of tests for everyone
Historical results aggregation
Visualization of performance metrics
Integration with CI and monitoring tools

Nancy’s team compared several load testing tools and found they enjoyed using the Apache JMeter tool, which is a popular open source load testing software. However, they quickly ran into an “out of memory” error after attempting to run about 400-800 concurrent users on a single machine. She and her team decided to use a distributed load testing platform to run the load they required to scale up more effectively.

Enter Flood.

Why AirTasker chose Flood

We also chose Flood.io […] because it comes in with a really good open-source gem called Ruby JMeter. It is a Ruby-based DSL for building JMeter test plans. - Nancy Cai, AirTasker

Nancy and her team enjoyed the benefits of being able to write load tests natively in Ruby and then using Flood’s Ruby JMeter gem, which is a Domain-Specific Language (DSL). Flood allowed them to turn their Ruby scripts into .jmx files, which is the native XML format that JMeter uses.

When they were ready to scale up, AirTasker ran the same JMeter script in Flood.

The real-time Flood dashboard allowed them to see that there was a problem on one of the API endpoints, called “Get pet reviews.” Further investigations revealed a performance issue due to a long-running query against a database server in the code, which AirTasker’s developers could fix.

Best practices

Nancy gives some great advice for anyone running a performance test, including:

asking the right questions as a load tester
running baseline tests
being aware of the difference between your test and production environments
keeping in mind the caching and autoscaling mechanisms that may drastically change the results
the differences between stress testing and other types of performance testing

We were so thrilled to hear Nancy’s story on how Flood helped AirTasker run their performance tests. Catch the entirety of Nancy’s talk below, and check out the resource links at the end of the post to follow Nancy’s story.‍

Flood’s Ruby JMeter gem

Nancy’s GitHub repo where you can find a copy of her test scripts

Follow Nancy on Twitter

Follow Nancy on GitHub

Mobile Load Testing on the Cloud

Fri, 21 Feb 2020 18:44:58 +0100

Originally posted here.

What is mobile load testing?

What is the difference between load testing web apps and mobile apps? In general, your goal is the same: to improve app performance by systematically exposing your app to the load you expect, whether it’s on the web or on a mobile device. However, while the purpose may be the same, there a few extra things to consider when it comes to mobile app performance, and it’s easy to make some key mistakes. In this article, I’ll give you a little bit of the theory behind mobile load testing and then show you practical steps to script and execute a load test.

Challenges in mobile load testing

If you have a web app that is mobile first, and your users typically access your site through a browser on their mobile devices as well as on their laptops, you have more choices for optimizing mobile performance. You can use browser-based load testing tools as well as protocol-level load testing tools to generate load.

If you have a native mobile app that you’d like to load test, though, browser-based load testing tools are a little more complicated. One of the most common approaches to mobile load testing is to use some type of device farm. A device farm is a collection of mobile devices and a way to orchestrate and control them so that they access an application simultaneously. Here’s why they’re not the best choice for load testing.

Device fragmentation

One of the biggest challenges is mobile device fragmentation. Not only are there a lot of mobile devices, but there are also many operating systems and versions. Even narrowing the pool to Android and ios leaves us with a large selection. A common mistake in mobile load testing is trying to account for these hundreds of combinations.

Logistics

Another challenge is the logistics of controlling physical devices with different operating systems, which is not trivial. Unless you have a thousand friends that you can rope into doing this at the same time, you’ll need to pay for some sort of service that can orchestrate this for you.

Cost

There’s also the fact that this process is resource-intensive. Even if your device farm consists of emulators on virtual machines rather than physical mobile devices, that still uses up a lot of CPU and memory, and you’d need to spin up dozens or hundreds of machines.

Luckily, this type of mobile load testing is not the only option available to us.

Better strategies for mobile load testing

A better option for back-end performance testing or load testing your mobile application is by using API load testing, which works on the protocol level. To understand this, let’s talk about what aspects of software we are analyzing when we test for “performance.”

We can generally break up performance testing into two types: front-end and back-end. Both types have the goal of making the app respond faster to end-users, but they do so by optimizing different parts of the application.

Front-end performance

Front-end performance is primarily concerned with the UI of the app. A front-end performance tester optimizes page resources (like CSS, images, JS) to render faster on the page. Since a mobile app runs on a user’s smartphone or another mobile device, its front-end performance would also be affected by things we can’t control, such as the operating system and version of the mobile or its physical configuration (CPU and memory). A front-end issue for a particular mobile device type might not necessarily affect performance on another. Device farms can be good at measuring front-end performance.

Back-end performance

In contrast, back-end performance focuses on the underlying mobile application servers. A back-end performance tester optimizes the things behind the interface, and it is affected by things like the size of the responses that a server sends to a user, server configuration, application infrastructure, and how downstream servers talk to each other. These factors may be invisible to a typical end-user. A performance bottleneck in the back-end is likely to affect all users of an app to some degree.

Load testing for back-end performance

Load testing is about determining how your mobile application behaves in response to traffic, so it targets the back-end components of the application, not the front-end.

Both front-end and back-end performance are necessary to get a holistic view of mobile performance. Still, if you want to load test your mobile app, you need to generate traffic on your application server, and not necessarily to drive the app on real devices. Understanding this opens up options for mobile load testing.

API load testing

Most native mobile apps work by using a RESTful API to send instructions to the application server. Luckily for us, there’s no magic in sending these types of requests from a smartphone— we can simulate this easily by using an API load testing tool to send these requests on a non-mobile device too.

Here are some advantages of using API load testing tools to test your mobile app instead of using a device farm:

It’s easier to set up and maintain: API load testing can be easier because you don’t have to account for changes in the user interface as long as the underlying API calls are the same.
It’s cost-effective: You can run about 1,000 users for an API test on a single machine (we used AWS m5.xlarge machines to arrive at this estimate). This translates into fewer resources that you’ll need to pay for whether you’re running a thousand users or [a million](whether you’re running a thousand users or a million), and, ultimately, a lower cost.
It lets you narrow your load testing scope: If you have a particular endpoint that has historically shown performance issues, you can use API load testing to target just that one endpoint. On a device farm, you would have to create a test automation script that might click a button and send dozens of requests, only one of which you actually wanted to test.

Tools for mobile load testing

You can use many tools for API load testing, but you can’t go wrong with robust open-source tools like JMeter and Gatling. Both tools are well-used tools in the performance industry, and they also have large user bases and ample documentation for helping anyone get started. In this article, I’ll continue with JMeter specifically.

How do you get started with writing your script? Ideally, your developers should be able to point you to the API documentation for your app or be able to describe the format of the request themselves. But what if you wanted to get started on your own?

How to record network traffic from a native app with Apache JMeter

Recently I wrote an article on The New Stack about how I would have load tested the IowaReporterApp, which was responsible for delays in reported vote totals for the 2020 Iowa Democratic Caucuses. To show how to load test that app, I created an actual load testing script for that article, without any input from the app developers. I chose to script using Apache JMeter, a popular open source API load testing tool, but Gatling (also open source) is another good option.

Here are the steps I took on my Android mobile, a Samsung Galaxy Note 10+. The same principles should apply to iOS devices and other Android devices.

1. Download and install the app.

Even if the app isn’t on the Google Play store, you can install an app using the .apk file. You may have to enable Developer Mode on your phone, which is not the same as jailbreaking. All this does is bypass the security feature that would otherwise not allow you to install apps that don’t come from the Play store.

To enable Developer Mode, go to Settings > About phone > Software information. Then, scroll down to the field Build number and tap it seven times. You should see a confirmation message that Developer mode is on.

Now download the .apk, click on it, and follow the prompts to install it.

2. Set up an HTTP(S) Test Script Recorder on JMeter

You’ll need to have JMeter set up already for this step.

Connect your laptop to a wi-fi network.

From a new test plan on JMeter, right-click on Test Plan and then click on Add > Non-Test Elements > HTTP(S) Test Script Recorder. Click on it and take note of the port it uses in Global Settings.

Next, right-click on the test recorder element you just created and click Add > Listener > View Results Tree. This element shows details of requests and responses when we run tests.

Then, right-click on Test Plan again and click Add > Threads > Thread Group.

Finally, right-click on Thread Group and click Add > Logic Controller > Recording Controller.

Your test plan should look something like this:

3. Set up your mobile to use a proxy.

On your mobile, connect to the same wireless network that your laptop is connected to.

Go back to your laptop and find out its local IP address. On a Mac, you can see this by clicking on the Apple logo > System Preferences > Network > Advanced > TCP/IP. The IPv4 Address is what you’re looking for, and it might look something like 192.168.86.125.

On your mobile, go to Settings > Connections > Wi-Fi. Under Current network, click on the settings wheel next to your wireless network. Click on Advanced.

Enter your computer’s local IP address in the Proxy host name field.

In the Proxy port field, enter 8888, or the port that was displayed in the HTTP(S) Test Script Recorder on JMeter.

Click Save.

4. Start recording traffic.

On JMeter, click on the HTTP(s) Test Script Recorder element and click Start.

On your mobile, open the app and log in or perform the action that you’d like to record.

As you use the app, you should be able to see the requests coming in on JMeter in the View Results Tree:

If you click on the Recording Controller, you’ll also see the exact format of the requests that the app made:

Scripting and troubleshooting your mobile load testing script

What you now have is a recording of the mobile traffic you need to simulate. If you stop here and try to run it, you may find that it fails. That’s because you haven’t done any scripting yet.

Here are a few things you’ll need to do to turn a recording into a script.

Add assertions

An assertion is a way to verify that the request is returning the response that you expect. JMeter only marks a request as failed when it returns an HTTP 4xx or HTTP 5xx response code. However, there are times when the HTTP response code returned is an HTTP 200 OK, but the page that was returned was still not the one you expected. Without assertions, JMeter would mark this request as passed, yielding false positives.

For instance, if you’re sending a login request, then a good assertion for the response to that request might be text like “Welcome, Nicole” or “Logged in.” To see the data that should be returned, you can either run through the action manually in your browser and use Developer Tools to inspect the response or click on the response tab in the View Results Tree listener in JMeter from your recording.

Assertions also allow you to pinpoint more accurately where in your script the error is.

Replace hardcoded values with dynamic values

The last screenshot above shows the parameters that the mobile application sends to the server to log in. If you examine the parameter values, you’ll notice that several of them are long strings. The script fails when you just send the same request again because some of these are dynamic. This means that parts of the request need to change before your device is allowed to access the application. You might need to add a timestamp, the right answer to a challenge, a session or client ID, or codes returned in previous requests. These dynamic values help the server determine whether the login request is a valid login attempt.

This is where help from a developer would come in handy because he or she can tell you how each parameter value needs to be formed.

Add think time

Think time is a delay between requests that simulates the pauses that a real user takes to read text on the page or otherwise interact with your application. Most performance testing tools do this. In JMeter, they are called timers, and there are several to choose from.

I like the Uniform Random Timer because it allows you to set a constant value for the delay as well as an additional random delay on top of it. I recommend having a variable delay because setting a completely constant delay makes the test too regular and causes unnaturally high load at the interval you set. Real users don’t coordinate with each other, so stagger your requests by using a variable component to your think time.

Debug your test script

Before you think about scaling up, run your test a few times with one user on your local machine. If you don’t get any errors, try other configurations such as running two sequential users and running two threads at the same time. This way, you can test any correlations you’ve had to make for those dynamic values.

It may be tempting to skip this step, but it’s necessary before scaling up your load test.

Set test parameters

Test parameters are settings like the number of threads, ramp-up, and duration, which allow you to shape the pattern of the load you want to generate according to your test scenarios and test cases.

Number of concurrent users or threads

This is the most obvious way to increase load. The more instances of your script running, the more requests are executed.

You can change this in JMeter by clicking on your Thread Group and changing the “Number of Threads (users)” field:

‍

Ramp-up

In production, load on servers very rarely goes from 0 to 1000 in one second. Even for cases with a very definitive start time (such as an item going on sale at a particular time), the load generally increases gradually. You can simulate this gradual increase by adding ramp-up times in your script. A ramp-up is the amount of time during which new users are added at staggered intervals.

Here’s an example of what that might look like:

Duration

Duration is how long the whole test lasts.

The simplest way to set these values in JMeter is to use the basic Thread Group. You’ll need to change the fields highlighted below:

To use the duration, you’ll have to select “Forever” next to Loop Count and “Scheduler” and type in the number of seconds in the Duration field. Otherwise, you can also run the script according to the number of loops, or iterations, that you want to run. However, since response times vary, using the Loop Count as a proxy for duration yields different results from test to test.

Running a shakeout test against your mobile application

When you’re ready to run on your load generators, resist the urge to jump straight to the peak load you want. This is a great way to incur unnecessary costs. Instead, run a small test with dozens of users. Running locally is for debugging issues in the script. The purpose of running a shakeout test is to spot errors elsewhere, such as in your test environment and your test data.

Baselining your mobile application performance

Finding a baseline is one of the primary goals you’ll have as you start executing load tests. To be able to assess how changes in the environment or code affect application performance, you’ll need a stable point of comparison. To accurately measure the effect of a variable (like a code change), it’s best to keep as much of the other circumstances surrounding the test as fixed as possible. The baseline is your unchanging test scenario that you can use as a point of comparison to assess future performance.

The application you’re scripting against, your tool of choice, and your script all affect your performance baseline, so do some experimenting to find out the setup that works best for your situation. This setup includes characteristics like:

the number of concurrent users
test duration
think time, pacing, and all other waits and delays
the script (including how requests are broken up into transactions)
user density

User Density

User density is the number of threads that you can run on a node or a load generator. Figuring out user density it essential because all load generators, even virtual ones, have finite resources. Each load generator also has a finite amount of load that it can generate, based on its CPU and memory utilization, among other things. Trying to generate too many users on a single node may result in the node itself being the bottleneck for your test.

To accurately assess your application’s performance, your load generators need to remain healthy during the test.

It can be helpful to have a number to start with. At Flood, we’ve found that we can reliably run 1000 users using JMeter or Gatling with one of our AWS m5.xlarge machines. For reference, an m5.xlarge machine has 4 VCPUs and 16 GB RAM.

If your machine is similarly specced, run a test on a single node with 1000 users. While the test runs, watch the CPU and memory utilization. If the test finishes without either of those consistently hitting over 80%, you’ll know that the node can handle that number of users.

Let me reiterate, though, that you should use the 1000 users figure as a starting point only. You can then figure out your number through trial and error. Increase the number of users past 1000 and watch the resource utilization again. If that still looks good, add some more users and rerun. When you get to a test where the resource utilization hovers above 80%, stop and fall back to the previous number of users. You’ll now have your number.

Another thing you might want to play with is the think time and pacing in your script. These waits tend to have a significant impact on resource utilization, so you can expect to be able to run more users per node if you increase your delays.

If you’re using JMeter, always run your tests in non-GUI mode. GUI mode is great for debugging, but is unnecessarily resource-intensive for real load tests. While you’re at it, disable any listeners you may have that you don’t need to capture results.

Figuring out the appropriate user density now prevents you from getting inaccurate test results. Running as many users as you can without overloading the load generators is also cost-effective, as you’ll be making sure that you provision only as many nodes as you need.

Scaling up your load test on the cloud

It’s relatively simple to run a load test on one machine, but if you want to run on two or more machines, things quickly get unwieldy. Here are some options for achieving this.

1. Upload your tool of choice and script to every machine and kick off each test separately.

Instead of running one large load test, think of this method as running several smaller ones at (about) the same time. This is relatively easy to set up, but the main disadvantage is that it’s tedious. You may also need licenses for your tools if you’re not using an open source one. You’ll want to ensure that each load generator is as similar as possible to the others in terms of operating system, tool version, and script version. Since each node kicks off a separate test, you’re not going to be able to see a real-time combined view of all the load tests, and after execution, you’ll need to retrieve results files individually and combine them.

2. Use your tool’s distributed testing mode.

JMeter has a feature that allows you to scale. This involves setting up agents on each load generator and using scripts to coordinate execution and results collection. This method requires a little bit more know-how and time to set up, but it’s a little more cohesive than the first method. Here are links on how to set this up: Remote Testing with JMeter

3. Use a load testing platform.

I’m admittedly biased, but if I were to go back to doing performance testing consulting work, I would choose Flood to run my tests. A platform like Flood that is built for distributed load testing is an excellent choice because it takes away all the setup considerations. This is the easiest option, especially for teams that are new to load testing or are perhaps less technical, because all the setup is done through a UI. Scaling out in this case just means uploading your script, choosing the number of nodes you want to run and in which region, and all the work is done for you in the background.

While you can run on-premises, I would consider running your test cloud-based infrastructure for greater flexibility and ease of scaling unless you already have a data center.

What happens next?

Running your test is just the beginning of mobile performance testing— there’s still metrics collection, results analysis, and resolving performance bottlenecks that you uncover. After all, it’s what you do with the test results that counts. Remember that performance and load testing, whether for web-based applications or mobile ones, are iterative. The real value comes from building the app performance testing principles discussed here into your release cycles, so that load testing becomes a habit rather than just a one-off exercise.

Load Testing Tutorial: Server Monitoring

Thu, 26 Sep 2019 19:30:10 +0100

Originally posted here.

It’s difficult to be a good load tester if you don’t like data, because a lot of the value of the load testing is in the analysis. Even if you successfully generate load against an application, it’s ineffective without enough data to determine how your application coped with the load. At Flood, we’re passionate about data because we know it drives decisions, so we’re always working to improve how we present data generated by our nodes. However, there’s another important source of data that in my experience is too often overlooked by new load testers: your own application servers.

Executing a load test without monitoring server health is like flying blind: you’ll know when you land safely and you’ll know when you crash, but even if you do crash, you won’t know why—or how you can avoid it next time. Monitoring server health is the black box that will tell you what went wrong.

What server metrics should you monitor?

There are a lot of metrics that you can monitor, and we always recommend that you choose what to measure based on your own specific circumstances, such as the purpose for your testing or any previous outages you’ve had in the past. If you’re new to load testing, though, here’s a list of metrics you might consider measuring:

Processor Time - how much the processor is being utilised
Processor Interrupt Time - how much time the processor is spending to handle interrupts
Processor Privileged Time - the time the processor spends handling overhead activities
Processor Queue Length - the number of threads that are waiting to be executed
Memory (Available Bytes) - unused memory available to process new requests
Memory Cache Bytes - the size of the memory cache, which is where copies of frequently accessed data are stored
Disk I/O - number of reads and writes to the disk during the test
Disk Idle Time - time that disks are not doing work
Disk Transfer/sec - average number of seconds that an I/O request takes to complete
Disk Write/sec - average number of seconds that a write request takes to complete
Network I/O - bytes sent and received

How do you determine which one to use?

If you’re still not sure where to start: At a minimum, you’ll need the CPU and memory utilisation (in bold in the list above) of every major component you want to test. These two metrics are vital and if either of these is consistently maxing out at (or close to) 100%, that’s a sign that the component is struggling with the number of requests.

As an aside, CPU and memory over-utilisation is a very common reason for less-than-ideal response times on the load generators, which is why we also show you those metrics on our nodes while you’re running the test.

How can you monitor these metrics?

How you get these metrics depends on your budget and your operating system. I’ll start with the free or lower cost methods and work my way up to enterprise solutions.

If your servers are running Windows, PerfMon is a good alternative. It’s built into Windows and the interface allows you to choose the counters that you want to measure and start recording.

There’s also a JMeter server agent that you can download, confusingly called “PerfMon” but not to be mistaken with Windows’ PerfMon, that you can install on your (Windows or Linux) server and will gather metrics. It’s normally used in conjunction with the PerfMon JMeter plugin.

Nagios Core is another good open-source way to not only collect metrics but also set up alerts when values exceed certain thresholds that you can set.

Moving from free and open-source options to more enterprise solutions, Dynatrace is a powerful tool that allows you to track not just server health but can actually be used to trace individual requests using a custom header.

AppDynamics is another fantastic tool that allows you to really drill down to specific SQL queries that take a long time to execute, for example, feeding you important information to give to your DBAs.

New Relic is another good choice for monitoring production as well as test environments, and we particularly love its thread profiling feature.

Other noteworthy tools are the Microsoft System Center Operations Manager, the Oracle Enterprise Manager, and BlueStripe FactFinder.

Flood offers out of the box integrations with Dynatrace, AppDynamics and New Relic. If you’re already using one of these tools, you can push the load metrics from Flood into the dashboards that you’re already using, allowing you to see both load metrics and server metrics in the same place. Here’s an example of a dashboard our own Antonio Jimenez created in New Relic which combined Flood-provided metrics with server-side metrics:

This view helps with spotting correlations or trends across the different sets of data, such as a response time graph that begins to increase at the same time that server memory utilization does.

Putting It All Together

Having both sides of the puzzle, the metrics from the application servers along with those from our load generators, gives you a more holistic look of your application performance as a whole, and increases the value you get from load testing. We’d encourage you to take advantage of the 5 free node hours we provide in our trial to give this integrated load testing solution a spin. You can head on over to Flood to sign up for a free trial which will include 5 free node hours and you can drop us a note to get help integrating Flood with your APM solution.

Start with Why: How to Write Requirements for API Load Testing

Wed, 19 Jun 2019 21:49:17 +0100

Originally published here.

Every good test begins with requirements.

You may think, “But I’m a tester! I’m not a business analyst!”, and I hear you. But as testers, especially on smaller teams or shorter projects, it’s still part of our job to make sure we know what our clients want. Otherwise, how do we know whether we’ve succeeded or failed?

At Flood, I often speak to project teams who ask for help interpreting reports. Most of those times, the problem is not that they don’t understand the metrics or what they measure. The problem is that they didn’t set pass or fail criteria in the first place, and so they don’t have anything to judge the results against. This is always a big warning sign.

Which performance testing metrics are important? Why are we testing, anyway?

Tim Koopmans, co-founder of Flood, coined the acronym SPEAR to describe the different aspects of a performance that should be considered in load testing. This is a great starting point when thinking about what we want our nonfunctional requirements to cover.

Scalability

Scalability is the application’s ability to cope with increasing demands by increasing the amount of server resources. This could mean scaling up (increasing the resources of the dedicated server) or scaling out (adding more nodes to shoulder the load). What happens when more users than expected sign up in response to a promotion on your site?

Performance

The most common performance metric is page response time, but there are other considerations to be made here, such as throughput (requests per minute) and the number of concurrent sessions that need to be supported. Things like the total size of the resources on the page, whether or not a CDN is being used, and what to cache are also worth discussing.

Elasticity

Elasticity is a relatively newer aspect to performance testing brought about by advances in the cloud that allow application infrastructure to adapt to changes in load. Unlike scalability, elasticity emphasises scaling down as much as it does scaling up. Testing that virtual machines scale up when load increases is important, but testing that virtual machines also scale down when load decreases can also help save on unnecessary costs.

Availability

To test for high availability, ask yourself what would happen when (not if) your application’s server fails. Is there another server that the load balancer will seamlessly send traffic towards? Does the throughput fluctuate wildly? If users are connected to one server that fails, is your application smart enough to make new connections to another server? Or will it simply serve up an error page that users won’t know what to do with? Disaster recovery is best tested when there’s no disaster imminent.

Reliability

Reliability encompasses a lot of scenarios, but they all have to do with whether or not your application returns expected responses. Does your error rate increase when you increase the duration of your load test? Are you adding verification steps to your load testing scripts to check whether or not the HTTP 200 response that the application returned is not actually an error page?

What should my application’s response time be?

Our clients frequently ask us what the industry standard is for response times, wanting to make sure their applications measure up. The answer, however, is more complicated than a single number.

Industry standards for response time are only useful when applications are very similar. Constantly changing technologies used in web development as well as innate differences in business processes, however, make it very difficult to extrapolate a single number that will apply to all, or even most, applications in a certain industry.

The home page of one e-commerce app, for instance, might be several seconds slower than that of their main competitor. However, that doesn’t take into account the fact that their app loads a video showcasing new products. Does that mean that the development team should remove the video in order to fall in line with their competition?

Well, maybe. But not necessarily. It’s a business decision that needs to be made after perhaps using focus groups to determine the impact of the video, forecasting changes in conversion rate due to it, and comparing its projected value to the effects of being slower than the competition. A/B experiments could be used to test these assumptions and gather quantifiable data to support the team decision.

These factors are often not considered in the search for one number to rule them all, which is why a fixation on that number can be detrimental. Instead, I encourage project teams to brainstorm and come up with their own numbers for all metrics that would be more appropriate for their application. Gathering comparative metrics from a competitor may be part of this process.

What is a good API load testing requirement?

A good requirement, just like any good goal, is SMART:

Specific

Vagueness in a requirement leads to vagueness in results interpretation.

Instead of: “The performance of the web application…”

Try: “The average response time of the Login transaction…”

Measurable

Make sure there is a quantifiable way to know whether requirements have been achieved.

Instead of: “Decrease user frustration.”

Try: “The error rate must be below 3% at peak load.”

Agreed Upon

Have the appropriate stakeholders been involved?

Instead of: “The system must be able to generate emails as soon as users register.”

Try: “The system must be able to generate a maximum of 100 emails an hour, after which emails are queued.”

A good example for this is a project I was involved in that aimed to increase the speed with which emails were generated. Unfortunately, the particular email being sent included a big change that many customers were expected to contact the support team about. The problem was that nobody had thought to include Support in the conversation. Once they heard about the expected end result, the support team quickly raised their concern that they would not be able to handle the expected volume of emails unless the emails were staggered. This could have been avoided if they had been brought into discussions from the very beginning, in the requirements gathering phase.

Realistic

Can we meet this requirement given the resources available?

Instead of: “All requests must be returned within 5 ms.”

Try: “90% of the Catalog page requests should be returned within 3 seconds.”

Timely

Especially for nonfunctional testing, consider adding a timeframe to requirements.

Instead of: “The digital code is sent by SMS upon successful client log in at peak load.”

Try: “The digital code will be sent by SMS no later than 5 minutes after successful client log in at peak load.”

Go forth and write great load testing requirements

Requirements represent a great opportunity to think things through and make sure everyone on the project team is on the same page about the goals for your load testing. Too often projects skip this phase, only to realise much later that the tests that were executed didn’t address a key stakeholder’s concerns.

In load testing, as in many things: when in doubt about what to do, start with why.

Performance Testing Interview Questions, Part 3: LoadRunner

Thu, 20 Dec 2018 22:54:24 +0100

Originally posted here.

Like it or not, LoadRunner is one of the most popular commercial load test tools, and with good reason. Its protocol support is extensive, and many a load tester has cut his/her teeth on LoadRunner. In this last part of our interview series, we’ll go over some questions you may have to answer when interviewing for a role using Mercury HP MicroFocus LoadRunner.

What sort of transactions would you put into vuser_init and vuser_end?

Both are used for transactions that would only need to run once, at the beginning and at the end of a script. They’re akin to JMeter’s setUp and tearDown thread groups. Some things you might put in vuser_init are user logins or navigation to get to the start of the business process you really want to test, and logouts and any data cleanup to reset the application for the next test are common things to put in vuser_end. Note that if you want each virtual user to iterate through a list of user accounts during the test, you won’t want to put logins and logouts in vuser_init and vuser_end.

When would you run a script in VuGen and when would you run it in Controller?

VuGen is short for Virtual User Generator, and it’s where the actual scripting is done. Running in VuGen is good for writing or debugging a script, where you can quickly see where a script fails, what the response is, and what you need to change. You can also step through the code line by line (F10) to more accurately see, for example, variable values changing throughout your script. It’s really only for shaking out scripts with a single user. Running in VuGen also only runs from your local machine. Once you’re ready to run a load test, you’d switch to Controller.

The Controller is like the command center in LoadRunner: from the Controller, you can actually ramp up the number of users and choose from different scenario configuration settings in order to tailor your workload model to the scenario you’re trying to test. Controller also runs the scripts headlessly; it won’t open up a separate instance of VuGen for every user, so it’s a lot more resource intensive. Controller will also allow you to configure remote load generators and run scripts on them.

How would you set up LoadRunner to run certain transactions a certain percentage of the time? For example, for every 100 users that browses to a site, you want 25 of them visiting each of the About, Catalog, Contact, and Reviews pages.

You can write the code to visit each page in separate Actions. Then, in Runtime Settings, you can modify the run logic, create an Action Block that has all four actions, and assign a percentage to them (25%).

Of course, you can also hard code this in C within the actions, if you’d prefer, but moving these values to Runtime Settings makes it easier to maintain and reuse by other testers.

How is LoadRunner pacing different from how it’s implemented in other tools?

Most test tools allow you to set a fixed or variable pacing that adds onto however long one iteration takes to complete.

LoadRunner has a pacing feature that allows you to set a specific time and then adds a variable amount of time to your test depending on response times for the included transactions. So if you set pacing to 5 minutes and one iteration takes 3 minutes to finish, LoadRunner will automatically wait 2 minutes before starting the next one. If it takes 6 minutes, it will not add any more wait time and proceed instead to the next iteration. This comes in handy because you can more accurately predict how many transactions the script will do in a certain amount of time.

Why would the traffic from a load test on LoadRunner not be seen in Google Analytics?

Google Analytics runs when the browser of a user downloads and executes a javascript file that then sends information to Google to increment the site statistics for the current visit. However, most traditional load test tools work on the protocol level; that is, they send requests directly to the application server and download resources from a page to simulate users. This means that while they download the Google Analytics script, they don’t execute it, which means the scripted visits aren’t counted.

In order to use Google Analytics to view statistics from a load test, you’ll need to use a browser-level load test tool. LoadRunner actually has this covered with their TruClient protocol, which interacts with elements on the page like a user would. It will also work for testing single-page applications that use AJAX to change the data displayed on a page instead of reloading new pages.

Of course, for open source alternatives to TruClient, we at Flood prefer Element or Selenium load testing, both of that will actually open the site in a browser and executed scripts like a real user.

Performance Testing Interview Questions, Part 2: JMeter

Thu, 20 Dec 2018 14:47:18 +0100

Originally posted here.

In this part, we’ll go over some questions that are commonly asked in technical interviews about the use of the most popular open-source load testing tool, Apache JMeter.

Using JMeter, you record a key business process. Upon playing it back, however, you notice an error on the first page. What could have gone wrong and how would you fix this?

While recorders have gone a long way towards making scripting easier, modern security measures like the implementation of viewstates have also made scripting more complicated to record. Some more complicated applications will pass dynamic values in responses that it expects to see in the succeeding request, otherwise the request is blocked. This is great for security, but it also means that you will need to manually extract these values in your script.

In JMeter, you would do this by adding a response extractor to your sample that will extract the dynamic string according to rules you set, and then save the string to a parameter. This parameter must then be used in the next request that you send.

While running your test plan, you get the error: java.lang.OutOfMemoryError: Java heap space. How do you fix it? The easy fix is to edit jmeter.bat and increase the heap memory (search for HEAP) and increase heap space allocated like this: HEAP="-Xms512m -Xmx2048m"

The part in bold increases the heap space to 2 GB.

However, it’s also worthwhile looking at your test plan and removing any obvious memory hogs. One of the things we see a lot is enabled listeners that aren’t even being used. Listeners are great while debugging, but should be disabled when running a load test. Writing too many lines to the log is another culprit here that should also be commented out when running important tests. And of course, you should always run JMeter in non-GUI mode during load tests!

After running a load test, the results show that 300 transactions were executed with a PASSED status and returning responses of HTTP 200. However, the DBA tells you that there were no records written to the database as a result of the transactions. How would you troubleshoot this?

I see this happening with clients all the time, so I think this makes a great technical interview question. This is also why I said earlier that you shouldn’t count on a test tool to tell you a test has passed. It’s possible for a transaction to have the status PASSED and return an HTTP 200 but still be hiding an error.

This is mainly due to poor scripting that does not include validations that check whether the script has gotten to the right page. For example, you could use a Response Assertion as a child of your home page sample with the words “Welcome, Nicole” to determine whether the user Nicole has logged in successfully.

This is a critical step, because without it, it’s possible for a server to return a nice error page, or an incorrect one, such as the home page, with an HTTP 200.

How do you use a data file to read user account details from a CSV in JMeter?

JMeter has a CSV Data Set Config Element built specifically for this purpose. You can simply add one to your test plan, browse to the CSV file that you’d like to use, and select the options for how to share that data among the threads in your JMeter script.

How would you go about testing a website that uses HTTP/2 on JMeter?

There is a JMeter plugin called HTTP/2 Sampler that accomplishes just this. My colleague Jason just wrote a blog post on this very topic, but basically the plugin adds new HTTP/2 versions to the standard HTTP request and request defaults samplers as well as the Simple Data Writer and View Results Tree listener. These elements will need to be used in order to test an application using HTTP/2.

What can you do to synchronise the execution of multiple JMeter instances on different load generators?

Apache JMeter has released this handy PDF file to describe the steps necessary to run JMeter in distributed mode. This involves installing Java and JMeter on all machines and setting them up in a master and slave configuration so that one machine controls the rest. On the master, you’ll need to add remote hosts to the jmeter.properties file. When everything is set up, you should be able to open up JMeter on the master and do a a remote start of your current test plan on any or all slaves that you choose.

The problem with this setup is the clunkiness of working with different machines and essentially having to set them up the same way. After the test, there’s the daunting task of logging back onto each machine, gathering the data, and transferring them back into a central repository. Then you’re faced with consolidating the data from each injector in order to see a picture of the whole test. You can see that this set up quickly goes from time-consuming to a nightmare, especially when potentially thousands of different machines are involved.

If only there were an easier way…

Performance Testing Interview Questions, Part 1: General

Thu, 20 Dec 2018 12:41:40 +0100

Originally published here.

Preparation is Key for the Perfect Interview

Performance testing interviews can be a nerve wracking experience. We all understand it’s necessary, but we dread it anyway. Each company has its own style of interview, so you may need to prepare differently for each opportunity. This interview series will cover some general sample questions and scenarios as well as my answers to them to help you nail that performance testing interview!

I’ve broken down the potential questions into the most popular categories, so you can prepare easily for your all-important interview:

What is Performance Testing? What is Load testing? What is Stress Testing?

Performance testing is the process of systematically determining, quantifying and improving the speed, reliability, and efficiency of an application with the aim of improving user experience. This includes both front-end and back-end performance testing.

Load testing is a specific type of back-end performance testing where automated scripts are written to send requests in an attempt to simulate the peak user load expected on the application servers.

Stress testing takes that one step further and applies load in excess of what is expected in order to determine the maximum capacity of the system under test.

What is the difference between front-end and back-end performance testing? Which one is more important?

Both front-end and back-end performance testing measure how fast an application responds, but they measure different components of that overall user response time.

Front-end performance is concerned with how quickly text, images, and other page elements are displayed on a user’s browser page. Back-end performance is concerned with how quickly these elements are processed by the site’s servers and sent to the user’s machine upon request. Front-end performance is the part of the iceberg above the water line, and back-end performance is everything underneath that you can’t see.

Both are important because they can both determine whether a user continues to use your application. Front-end performance tends to be easier to test and can provide some quick wins due to the large amount of optimisation tweaks that can be done without writing code. Back-end performance tends to be more difficult to test because it often uncovers problems with the underlying infrastructure and hardware that are of a more technical nature.

Why does performance testing matter?

Performance testing matters because application performance has a significant impact on user experience. A site that is unreachable or slow to load due to an inability to cope with unexpected load will cause users to browse to competitor’s sites and tarnish the brand’s reputation.

How do you know when a load test has passed?

Ideally, you would have discussed your nonfunctional requirements with key stakeholders before load testing begins. This means that you set your own pass criteria before you even run the tests. You would ideally have a list of specific transactions (selected based on criticality or complexity according to the business) whose response time needs to fall under a threshold you’ve predetermined. “Fast” is not specific enough– a number is better. Depending on what kind of tests you’re running (soak, stress, volume, etc) you may have other nonfunctional requirements about duration, resource utilisation on the server side, or specific outcomes to scenarios you’d like to test.

As a general rule, don’t rely on a load test tool to determine whether your load test has passed. Rely on it to report your results, but always compare the results to the requirements to determine successes or failures.

What would you advise to clients who say they can’t afford to performance test because they don’t have the resources to maintain several load generators on site?

This is the main reason that performance testing has for so long been considered a luxury that only big companies can afford. Luckily technology moves on, and in 2018 we’re at a point where everyone can load test. The big innovation here has been the cloud and the ability to spin up thousands of virtual machines with a few mouse clicks. Services like Amazon AWS, Microsoft Azure and Google Cloud make it so that every budding entrepreneur can “borrow” the computing hardware necessary to do cloud load testing with thousands of users and then give them back after the test, without the hassle and cost of maintaining them. I would advise the clients to look for a a cloud load testing solution that utilizes virtual machines on the cloud to run their tests affordably.

You run a load test against a server with 4GB RAM and the results show an average response time of 30 seconds for a particular request. The production server has been allocated 8GB RAM. What would you expect the average response time of the same request to be in production given the same load?

Trick question! While you may be tempted to answer that the response time would be halved to 15 seconds, reality is rarely that convenient. Response times are a factor of so much more than memory. Things like CPU utilisation, network throughput, latency, load balancing configuration, and just application logic are always going to influence load tests. You can’t assume a linear progression in response time just because you’ve upgraded one part of the hardware. This is why it’s important to load test against an environment that is as production-like as possible.

What is a percentile and why would you look at percentile response times when you already have average response times?

A percentile is a statistical measure that describes a value that a certain percentage of the sample either meets or falls under. For example, a 90th percentile response time of 5 seconds means that 90% of the responses took 5 seconds or less to be returned. It can be an important measure because they soften the impact that outliers have on more inclusive measures such as averages. A transaction with an average response time of 2.5 seconds may seem perfectly acceptable to the business, but when the 90th percentile response time is 20 seconds, this is a good reason to investigate further.

What are some trends in performance testing that you think will continue in 2019 and beyond?

Cloud is an easy answer, as the cloud brings some compelling benefits in terms of reduced cost and just ease of use. However, I already touched on that in a previous question, so I’ll talk about another trend: open source.

There’s a reason that open source is still around: it works. The main advantage of open source tools is not that they are free, although that is a big part of the appeal. The real advantage is that open source tools are community-based and community-led, which means features get built for it faster than sometimes commercial tools can keep up with, and they’re built by users of the tool themselves. Open source tools like JMeter, Gatling and Selenium have revolutionized the industry due to their impressive feature sets, built by a growing community that has developed plugins for everything you can think of. More and more, even big companies with big budgets choose to go open source simply because of the wealth of knowledge of these tools that are already available for free.

We’re huge proponents of open source tools at Flood, which is why we recently open-sourced our very own tool, Element. We will continue to support all things open source, because we firmly believe that the future is open source.

Tips and Tricks for Using Flood Element

Fri, 23 Nov 2018 23:07:24 +0100

Originally posted here.

In case you missed it, there’s a new performance test tool on the block, and it’s called Flood Element. It’s a scalable browser-based tool that allows you to write scripts in javascript that interact with web pages like a real user.

Using Browser Level Users is a newer approach to load testing that overcomes a lot of the common challenges we hear about traditional methods of testing:

Easier scripting that is akin to common functional tools like Selenium and easier to learn
More realistic results that are based on true browser performance rather than API responses
Ability to test against all components of your web app, including things like Javascript that is rendered via the browser

Given the above benefits, it’s a no brainer to check out Flood Element for your web load testing, especially if you have struggled with the existing tools on the market like JMeter or HP LoadRunner.

Pairing Element with Flood turns it into a pretty powerful load test tool. We have a great guide here that you can follow if you’d like to get started with it. As you can imagine, I’ve been using and testing Element for several months now and I thought I’d share some tips I’ve learned along the way.

Initialising your script

You can always start from scratch, but the quickest way to get started is to type

element init myfirstelementtest

from your terminal, filling in the project name you’d prefer.

You’ll then be asked to type in the title of your test as well as the URL you’d like to script against. After a minute, you’ll see that a new directory has been created:

Element will automatically create a file called test.ts. This file will contain the skeleton of a script, along with some sample code to help you find a button and then click on it. But before you open it, let’s move on to…

Choosing the right text editor

Scripting in Element is already pretty simple, but I’ve found two things that really help while scripting are syntax highlighting and code completion. Syntax highlighting will greatly improve your experience when first learning a new test tool like Element, and code completion will make your scripting lightning fast as you become more experienced in the process. My text editor of choice is Visual Studio Code, which has both of those features. It’s slick and clean, and it does the job.

Syntax highlighting is when the text editor intelligently changes the font colour of your code according to its role in the programming language you’re using. Here’s a screenshot of the test.ts file we generated earlier in VS Code to show you what I mean:

This makes it easier to make sense of the code at a glance: comments are in green, values and labels are in orange, etc.

Code completion is when you start to type something, and VS Code helpfully opens a context menu with suggestions for methods you can use.

I love this because it means I don’t have to remember the exact name of the method. It also suggests names of variables you’ve already defined and highlights code that doesn’t make sense. This will help to make your tests more maintainable and readable for others, which is a great benefit as you look to scale your testing out in the future.

Taking screenshots

One of the most powerful features of Element is its ability to take screenshots. I find it immensely useful when debugging because sometimes it’s just easier to see what’s going on visually. With protocol based tools, debugging can often be a much more involved and technical process.

There are two ways to take screenshots in Element:

Add a setting to automatically take a screenshot when an error is encountered. You can do this by setting screenshotOnFailure to true in TestSettings:

export const settings: TestSettings = {
device: Device.iPadLandscape,
userAgent: 'flood-chrome-test',
clearCache: true,
disableCache: true,
screenshotOnFailure: true,
}

Explicitly take a screenshot at a particular point in the script. You can do this by adding

await browser.takeScreenshot()

to your code.

Viewing Screenshots

Once you’ve taken screenshots within your tests, you will likely want to view them and know that they will be stored for future safekeeping. Whether you are running your test locally on have uploaded it to Flood to run with increased concurrency, Flood Element has you covered.

Locally Run Tests

Screenshots will be saved as jpg files in a timestamped folder corresponding to your run. It should look something like: ...myfirstelementtest/tmp/element-results/test/2018-11-20T135700.595Z/flood/screenshots/ and the screenshots will be uniquely named so that new screenshots, even for the same step, don’t overwrite older ones.

However, I rarely have to look up the screenshots in that folder because I prefer to see them in iTerm2 for macOS. iTerm is an alternative to the terminal that works particularly well with Element. When you take a screenshot, iTerm actually shows it in-line:

‍

Tests Run in Flood

Running an Element script on Flood is ideal when you need larger concurrency. Rather than accessing your screenshot locally, Flood will centralize the images into your account so the images remain even after the cloud load injectors are destroyed. You can get to the screenshot files by downloading Archived Results:

Or you can also choose to click on a step on the dashboard to see a film strip of your test:

Using Logs

You may need to check out the logs for more technical debugging, especially when the screenshots don’t tell the whole story. Whether you are running your test locally on have uploaded it to Flood to run with increased concurrency, Flood Element has you covered.

Locally Run Tests

You can print to the console by typing, for example:

console.log('orderValues = ' + orderValues)

This will print the value of the variable orderValues at that point in the script. You would see this in your terminal if you’re running Element locally.

Tests Run in Flood

If you’re running the script on Flood, you can either download the log (in the same Archived Results zipped file as mentioned earlier) or you can click on the Logs tab:

‍

Fun with Flags

Element comes with a few flags that give you more control over how the script is run locally. Here are a few of my favourites!

Headless Flag

When in doubt, run Element in non-headless mode to see the script actually opening the web app on Chrome and interacting with the page. This is only possible locally, but there’s nothing like actually seeing for yourself what’s happening in real time instead of relying on screenshots and logs after the fact. To enable this mode, add the flag when running your test:

element run myfirstelementtest.ts --no-headless

Watch Flag

Element will automatically close the browser window when it encounters an error or finishes the iteration. Adding --watch will leave the browser window open and then will monitor the script. As soon as the script is saved, it will automatically run it in the same window from the beginning. Simply add this flag like the above example:

element run myfirstelement.ts --watch

Dev Tools Flag

This opens a browser instance and runs the script with the Chrome Dev Tools open, allowing you to find locators for the next action you want to script. Simply add this flag like the first example:

--dev-tools

More Flags

For more flags, use:

element run --help

Try Element

You’ve just gotten a crash course on Flood Element. You’re all set! If you haven’t already, download Element here to start writing functional test scripts and reusing them as load test scripts on Flood. If you don’t have a Flood account, you can easily sign up for a free trial here.

We’re extremely proud to have been able to contribute to the open-source community and can’t wait to have you try this new addition to the Flood line!

Prepare for Viral Videos With Flood Element

Thu, 08 Nov 2018 23:24:31 +0100

Originally posted here.

One of the more interesting things to load test is video. You see, traditional load test tools are protocol-based, which means that their approach to load testing video is to capture and simulate the traffic that the browser generates under the covers, sending the same requests to the application. This approach can still be successful– in fact, one of our favourite clients here at Flood recently used Gatling to simulate 6 million concurrent users streaming video from their application. However, for some use cases, testing at the protocol level can’t simulate everything that happens in a real world use case.

The Challenge

If you’re a video creator, the ultimate goal is often a simple goal: creating a viral video. Unfortunately, preparing for and simulating this all-important use case is typically impossible with protocol based tools. The same feature that makes them so efficient – not using a browser to send the requests – is the same thing that prevents them from being used to simulate the needed browser level events. You see, things like view counts on a video are normally tracked by scripts running on the page. Traditional tools download these scripts along with other page resources, but the scripts themselves are not executed. Enter Flood Element.

A Solution: Flood Element

Flood Element represents a new way of looking at load testing. It is a browser-based tool, which means that it actually drives an instance of the browser to test an application. Rather than telling it what requests to send to simulate a user like you would do with a protocol-based tool, you tell Element which buttons to click or which elements on a page to interact with just as a real user would. Essentially Element IS a user, using your application the way it was meant to be used. This means that it also executes any scripts that would run on the browser side. It’s so realistic that you could even use Element to make your YouTube video viral* by scripting up users to view your content… hypothetically speaking.

Simulating Real User Views

Testing video with Flood Element is actually pretty easy, especially if you are familiar with open source testing tools like Selenium. If you haven’t yet begun using Element, go through the installation guide here. You can download the full script used for this tutorial here. In this example, we’re going to be using a YouTube video of the opening credits the classical cartoon Top Cat (the indisputable leader of the gang) from Warner Bros.

Step 1. Navigate to the video page

Let’s start with a basic script:

import { step, TestSettings, By, Until } from '@flood/element'
export const settings: TestSettings = {
 loopCount: -1,
 clearCache: true,
 disableCache: true,
 actionDelay: 8,
 stepDelay: 10,
 screenshotOnFailure: true,
 userAgent: 'flood-element-test',
}
export default () => {
 step('01_Home', async browser => {
 await browser.visit('https://www.youtube.com/watch?v=6fvhLlrBrPQI')

 await browser.takeScreenshot()
 })
}

This is the bones of a basic script in Flood Element. It consists of a single step that involves doing two things: visiting the starting URL, which in this case is the direct link to the YouTube video, and then taking a screenshot once loading the page. Taking screenshots is particularly useful when testing video because screenshots give you a way to check that your script is doing what you want it to do. It’s good to take periodic screenshots as well, to measure progress.

To help you validate that the script runs well locally before scaling to hundreds or thousands of users, you can use our Element CLI. If you save the above code as youtube.ts, open up your terminal in that directory and then type:

element run youtube.ts

Assuming you’ve got Element set up correctly, you should see something like this: (Note that screenshots are only available when using iTerm as your shell. They are also saved as files under .../tmp/element-results/youtube .) ‍

Oops! It looks like our script does navigate to the page, but it doesn’t play the video. We’ll need to click on that play button. Let’s add that to our script.

Step 2. Click the Play button

We’ll need to add some lines to click the play button after visiting the page but before we take the screenshot, so that we can verify later whether or not the play button was clicked.

step('02_ClickPlay', async browser => {
 //Click on Play button
 let playBtn = await browser.findElement(By.xpath('//button[@aria-label="Play"]'))
 await playBtn.click()

await browser.takeScreenshot()
 })

In this example I used Xpath, but you do have the option to select based on partial text, CSS selector, etc. A tool I like to use to get these is Chrome Developer Tools’ Elements tab, which allows you to get the selectors for a particular element.

Step 3. Take periodic screenshots

To make sure that the video is really playing, we’ll need to take a screenshot a few seconds after the initial screenshot in order to visually confirm that the video has progressed. Even better, we can get Element to take screenshots at regular intervals. So let’s add a third step:

step('03_Play', async browser => {
 //Take a screenshot every 5 seconds until video finishes
‍
 for (i = 0; i <= 60; i++) {
 await browser.takeScreenshot()
 await browser.wait(5)
 }
 })

To use i, we’ll also need to declare it earlier in the script:

import { step, TestSettings, By, Until } from '@flood/element'
‍
var i = 0

In this step, i will correspond to every 5 seconds of playtime for the video, so we’ll need to make sure that the limit of i that we set is sufficient time for the entire video to play. Every 5 seconds, Element will take a screenshot, which will help us verify that the video is progressing after the test.

Click here to download the script before moving on to the next step, unless you’ve been following along at home.

Step 4. Run it on Flood!

The final step is to upload it to Flood and run the test. Click here to see a step-by-step guide for how to do just that. You can choose how many users you’d like to run on Flood and across how many nodes. New users will be given 5 node hours which will allow them to run a test with up to 250 users for 1 hour. Each node will have a different IP address, so plan for that accordingly if your application takes that into account.

One of the things I love about running it this way is that if you click on the arrow to the right of the sole transaction, you can see screenshots taken along the way to prove that the video is being played:

Some Considerations for Load Testing Video with Flood Element

Taking screenshots and using logs are necessary for debugging.

Because the page elements while a video is streaming are static, we’re not going to be able to use those as clues as to where we are in our scenario. To compensate, it’s a good idea to take screenshots and use console.log() to determine progress, especially while debugging. Be careful not to go overboard on these when actually running the load test though, as they will consume system resources.

Videos may behave differently in headless mode.

While writing this tutorial, I realised that YouTube auto-plays videos when on non-headless mode (aka, when opening a video link manually on a browser or with the --no-headless flag in Element). When it’s executed in headless mode via Flood Element, it requires a click on the play button to begin. Again, use screenshots to aid you in spotting these differences.

Conclusion

Element’s browser-level scripting capabilities allow it to simulate users that interact with pages just like human users would. Element and Flood are a powerhouse combo, providing the flexibility to test with thousands of users across the world within a few minutes.

Element seeks to reduce the gap between a scripted user and a real user – and it’s so realistic that it’s hard to tell the difference. We encourage you to take advantage of our free trial offer and run some real world load tests today!

*We don’t advise or condone the use of Element to artificially inflate views on YouTube in any way which may violate its Terms of Service. The purpose of this article is to showcase Element and how realistically it recreates real users.

Mitad de allá y mitad de aquí: reflexiones de una políglota multinacional

Tue, 28 Aug 2018 23:19:32 +0100

Originally posted on Medium.

Nadie se da cuenta de que soy filipina. Incluso los filipinos desconocidos me miran e inmediatamente hablan en inglés. Siempre están sorprendidos cuando respondo en Tagalog. «¿Pero por qué tienes la piel clara?» me preguntan, como si fuera algo que se puede comprar en una tienda. A través de mi piel no pueden ver mi lugar de nacimiento y juventud en las Filipinas, o cuánto echo de menos la voz del vendedor de taho cuando caminas por las calles: «Tahoooooooo!»

Los chinos me miran siempre como si fuera una de ellos. Pero después de hablar unos segundos en chino conmigo retroceden con una sonrisa. «Disculpa, es que pensábamos que eras china.» No les importa que tengo sangre china. Lo que les importa es que mi chino es horrible.

En EE.UU piensan que soy americana por mi acento en inglés por teléfono, pero cuando me ven, dicen «¡Que bueno tu inglés! ¿Qué eres?» Una mujer? Una ingeniera de software? No — un secreto a voces: No eres una de nosotros. Eso después del hecho de que todos mis estudios han sido en inglés, y que es mi mejor lengua.

En una entrevista de trabajo en Australia, me preguntó el entrevistador de todas las maneras posibles de dónde vengo, y, frustrado con mis respuestas sobre mis trabajos previos en Australia, se alegró mucho al descubrir que mis padres eran filipinos. «¡Todavía no tenemos un filipino aquí!» me dijo. Después, la oferta, y mi vergüenza de conseguir un trabajo por no ser australiana. Yo con una carrera casi completamente hecha en el país.

Un holandés me dijo que hay tanta tolerancia en los países bajos. Todas las personas pueden convertirse en holandeses independientemente de sus orígenes. Pero justo cuando explico cómo es posible tener tres ciudadanías, empieza a sacudir la cabeza. «Pero eso no es justo. ¡Tu tienes que elegir! O no serás una verdadera holandesa.» Eso después de un día hablando en holandés.

Puedes hacerme otras preguntas, las que van a hacerte sentir más seguro de quién soy, o más capaz de etiquetarme. Y ofrezco mis respuestas, una mezcla y combinación de países. Pero lo que quiero hacer es contar mi vida, porque no puedo contestar en una sola palabra: Soy originalmente filipina. Mis padres son filipinos. Estudiaba en inglés en una escuela internacional, y más tarde en EE.UU. Me mudé a Australia y me casé allí con mi marido australiano. Vivimos allí juntos por más de una década. Nos mudamos a los Países Bajos, donde vivimos ahora. Y todo eso con una colección de idiomas, algunos de esos países, y otros, como el español, de otros.

Tengo mucha pasión cuando hablo otros idiomas. Y el aprendizaje de un idioma no es posible sin el conocimiento y la comprensión de la cultura que acompaña cada idioma. ¿En qué idioma pienso? Eso depende — de la situación, de la persona con la que estoy hablando, de mi humor, del tema, de si estoy cansada. Pero pienso en todos. Y, cuando pienso en estos idiomas, también actúo de manera correspondiente a las perspectivas inherentes a ellos. Y me cambian. Soy fundamentalmente diferente por haber aprendido este particular conjunto de idiomas.

Me vas a preguntar a qué país pertenezco en mi corazón?

Elegir un país o una cultura sería como cortarme el brazo. ¿Cómo sería posible para mí vivir sin reconocer la importancia de la familia como una filipina, sin cambiar al inglés para discutir noticias del mundo informático, sin sentir orgullo a la vista de la bandera australiana, sin relacionarme con personas con una mente abierta como una holandesa?

Digo que pertenezco a todos. No voy a elegir. Muchos de los problemas del mundo hoy en día ocurren porque queremos que otros elijan: entre sus países o sus religiones, sus familias o sus sexualidades, sus principios o sus presidentes. La verdad es que no hay una opción sino ser lo que ya son: producto de todos.

¿Y no somos todos así?

Manufactured Chaos: How Netflix Does Performance Testing

Wed, 28 Mar 2018 23:35:52 +0100

Originally posted here.

It’s always ideal to run performance tests against an environment that’s production-like. That way, you can ensure that your tests aren’t being influenced by factors that are not present in production - such as sub-optimal specifications, or the configurations of a test server. So, what does a good performance tester do? They compare their machine specifications and request production-sized boxes to run their tests against. But is that really enough?

Here’s the reality: despite our planning, we still sometimes see things in production that we don’t see, and can’t reproduce, in the test environment. It’s hard to predict every possible failure. Even if you do recognize the danger, it’s still difficult to recreate the perfect storm of circumstances that could potentially lead to its occurrence. The more complex your network architecture is, the more likely it is that these failures will occur. As a result, there’s something of a “don’t breathe or it might break” feeling that comes with deploying to production. So how do you test your application to get the maximum degree of confidence in its performance?

If you’re Netflix, your answer would be to lean into the chaos.

Chaos theory, in a nutshell, tells us that what looks random actually contains patterns that, while unpredictable, are recognizable and thus manageable. Far from representing disorder, chaos is our word for events that may be infinitely complex and interconnected in ways that aren’t immediately apparent. Chaos engineering takes this principle and applies it to software. Developers can code to reduce the occurrence of chaos as much as possible, but they can also code to make sure applications can withstand chaos if it does occur. As testers, one way we can test the latter is to manufacture chaos.

Netflix’s testing strategy puts a lot of emphasis on manufacturing chaos because their business model involves streaming gigabits of data to customers on demand, from anywhere in the world, at a consistently fast enough speed that people aren’t annoyed by buffering in the middle of a rousing speech by the Mother of Dragons. In 2015, Netflix accounted for 36% of traffic in North America. Netflix customers stream 125 million hours of video every day. Netflix used to serve the content from their own private data centers but moved to the cloud in 2008 after a database corruption disrupted their business for three days. Today, Netflix is 100% in the cloud, which provides tangible benefits such as architecture that can scale on demand thanks to their use of Amazon Web Services (AWS) - incidentally, the same CDN that we use here at Flood IO. However, even the cloud isn’t invulnerable to chaos. Enter the Simian Army.

Conjuring up the humorous image of a squadron of monkeys wreaking havoc in a data center, the Simian Army is a series of tools developed by Netflix that code chaos into their tests - on purpose. At Netflix, every build has to pass not just the standard suite of functional and nonfunctional tests, but also the chaotic-but-programmable tests of the Simian Army, all of which Netflix have made available as open-source tools on Github. Each monkey is a real-world manufactured-chaos test that the application under test must pass in order to be deployed into production with a high degree of confidence in its stability and continued performance.

Chaos Monkey selects a node or container within a node at random and terminates it unexpectedly, forcing Netflix engineers to adapt their code to deal with this behavior by quickly rerouting requests to backup nodes and containers.

Janitor Monkey detects unused resources (instances, volumes) in the cloud and terminates them. Ideally, all resources no longer used should be shut down, but in practice this doesn’t always happen. People forget to delete things they’re not using anymore. Applications are changed to use new volumes, and old ones are forgotten. While it’s a good idea to make sure everyone is aware of the need to clean up after themselves, Janitor Monkey assumes that things will fall through the cracks and automatically comes to the rescue. Janitor Monkey also sends a notification to the owner of the resource with enough time to mark the resource as an exception before deletion.

Conformity Monkey scans instances and determines whether they conform to a set of rules specified by the organization. If an instance is not configured the right way, Conformity Monkey sends the owner a notification. It can detect things like an instance being created without the appropriate security groups or tags.

Security Monkey monitors your entire public or private cloud to look for policy changes or incorrectly configured instances that could lead to serious security breaches. It even provides a UI so that you can see a history of changes to important policies and whom they were made by.

The Simian Army is a marked departure from traditional software testing techniques, which are very much procedural and based on predetermined flows. The assumption is that people behave logically, are all experts in their field, forget nothing, and always follow best practices. But is this actually realistic?

Netflix’s approach to testing is innovative in that they don’t assume that everyone involved knows exactly what they’re doing, from the developers to the infrastructure engineers to the testers. Neither do they assume that it is always possible to know the exact cause of failures in production. Sometimes bad stuff just hits the fan and nobody sees it coming. The very nature of our work in technology requires constant change, and it can be difficult to keep up with all the ways in which applications are becoming increasingly more integrated with each other. This will only become more apparent as the Internet of Things becomes more popular.

At Flood, we often see our customers targeting production environments, and there are some strong arguments for doing this. Testing later in the lifecycle means we can start to tackle production-sized problems around availability, scalability and reliability that would have otherwise been difficult to reproduce. Let’s face it. We all sometimes see monkeys in our data centers. Maybe it’s time we started planning for it.

References:

Completing the Netflix cloud migration What is chaos theory? Principles of chaos engineering The Netflix Simian Army Chaos engineering - surviving the failures in distributed systems Chaos engineering series - Part 1: chaos engineering deep dive What is Simian Army?

Why AirBNB Plus Could Make or Break AirBNB

Sat, 24 Feb 2018 23:24:00 +0100

Originally published on Medium.

“We’ve done the anywhere part,” Brian Chesky announced a few days ago, “but AirBNB is still, for some, an alternative way of travelling… until today.”

AirBNB Plus is a new tier of service that AirBNB have introduced for homes and hosts that go beyond the Superhost standard.

Here’s what an AirBNB Plus listing means:

The home has been verified by an AirBNB representative in person to have passed a 100-point checklist of quality. AirBNB has not yet publicly released the contents of the checklist.
The home comes with a list of extra amenities, such as a fully-stocked kitchen, wireless internet, and even paintings on the wall.
The host has received an average rating of at least 4.8 out of 5 stars.
You will be able to check in reliably at any time of the day. This could be due to an electronic keypad, safe lockbox or simply a host that’s willing to come out and let you in when you’re ready.
You automatically get premium support from a designated AirBNB Plus team in case anything goes wrong.
AirBNB has indicated that for now, only those who are already Superhosts will be eligible to request AirBNB Plus status.
According to the Verge, the host needs to pay $149 to apply to be considered for AirBNB Plus status.

AirBNB Plus listings are already available in several cities around the world: Austin, Barcelona, Cape Town, Chicago, Los Angeles, London, Melbourne, Milan, Rome, San Francisco, Shanghai, Sydney and Toronto, so you can go check them out yourself.

On the surface, this sounds like a great idea. I almost always travel with AirBNB, and when I’ve chosen to book a typical hotel, it’s always because I couldn’t find a Superhost listing to book. I’ve had bad experiences with AirBNB in the past, and have since decided to only book with Superhosts. However, even that is not a real guarantee of quality — in contrast to AirBNB Plus, becoming a Superhost has much less stringent rules.

Here’s what a Superhost listing means:

The host has received an average of at least a 4.8-star rating for all stays.
The host reponds to messages within 24 hours at least 90% of the time.
At least 10 guests have stayed at the property in the last year.
The host has never cancelled a reservation.

It’s almost entirely review-based: the requirement to maintain at least a 4.8-star rating is the most difficult hurdle. There are otherwise no quality requirements that the listing must maintain. In effect, Superhosts are community-selected, which is tricky because a string of teenagers used to hostels and leaving glowing reviews on a listing could make it appear more attractive than it actually is. I’ve stayed with Superhosts that had faulty showers, unstable internet and a kitchen that consisted of a sink and a kettle.

So I really like the idea of a higher class of accommodations that have been independently inspected. This is the first time they have ever offered anything like a guarantee for the quality of their things. Previously there was still a bit of a risk with booking an AirBNB, but this should dispel all doubt. AirBNB is doing what every company does after its initial growth spurt: it’s bunkering down, standardising its product… and, in effect, increasing some of its prices.

While price increases are not explicitly mentioned, it seems reasonable to expect that listings in this new category would fetch higher prices. After all, it costs more labour on AirBNB’s part to inspect these listings and it also costs the host at least $149, not to mention extra costs spent to meet the 100-point checklist. AirBNB Plus is no doubt a premium offering, catered towards a wealthier clientele.

Another concern is that the line between AirBNBs and hotels are blurring, and not necessarily in a good way. The original appeal of an AirBNB was that it was someone’s personal home, with all the quirks and defects that come with it. AirBNB’s overwhelming success then brought about a new class of landlords who buy properties with the specific intention to rent them via the platform. Already there are several boutique hotels that have been allowed to list accommodations as well.

The new Plus tier seems to narrow the gap between corporate hotel chains and spare rooms owned by real locals substantially. This feels less like a disruption of the hotel industry and more like a return to the status quo. I remain a huge fan of the brand, and I worry that AirBNB will lose that disruptive “plucky startup” charm — because they’re no longer a startup. They’re a major player, and if this announcement is anything to go by, they’re about to get even bigger.

Should I Upgrade to JMeter 4.0? 5 Reasons to Upgrade

Fri, 23 Feb 2018 23:43:25 +0100

Originally posted here.

Apache has just launched JMeter 4.0, a big step up from their previous release, 3.3, which was only released in September last year. Apache seems to be stepping up the frequency of it’s releases, and we’re not complaining. JMeter is an amazing open source load testing tool and the most popular one used at Tricentis Flood by far.

The big question is: Should you upgrade? Are the new features worth the leap? Will the new black UI make you a better load tester? While this choice should be made on a case-by-case basis and we certainly can’t pretend to give one-size-fits-all advice, here are some good arguments to do it:

#1: Java 9 support

JDK 9 was released back in September 2017, and if you were one of the people who upgraded to it when prompted, you would have noticed to your frustration that JMeter 3.3 did not play well with it, yielding the error:

Error: Java version -- 9.0.1 -- is too low to run JMeter. Needs a Java version greater than or equal to 1.8.0 errorlevel=3

Good news! This is now fixed. JMeter 4.0 runs smoothly on JDK 9. We’ll continue running on JDK 8 at Flood for consistency amongst all our open source load testing tools.

#2: New Boundary Extractor

Previously, my preferred way to extract dynamic values in JMeter was to use a regular expression extractor, but there’s no doubt that it can sometimes be tricky to get just the right text out of a response. Enter the Boundary Extractor.

The boundary extractor allows you to simply set the text immediately before and after the text you want to extract. This saves you wondering whether you should have used a + or a * in your regular expression, and is also easier to validate.

#3: New JSON assertion element

While XPath assertion was great, there was previously no counterpart for JSON. The JSON Assertion element fills that void nicely, using JsonPath to quickly assert the existence (or non-existence, with the Invert assertion option ticked) of a particular element.

#4: New throughput timer

We have a new throughput timer to play with, and it’s called Precise Throughput Timer.

Whereas the Constant Throughput Timer also had the disadvantage of constant pacing of requests, the Precise Throughput Timer is based on the Poisson Random Timer and randomly generates the delay between each request. The implementation seems to be well thought out, with the component reference section listing a lot of scenarios they’ve tested for that the previous Constant Throughput Timer might have struggled with. Throughput timers are one of my favourite JMeter features, and still rather unique in the industry.

#5: Test Plan Autosave

This one’s more of a usability tweak, but it’s a good one. JMeter now automatically saves your script when you run a test, saving you from those all-too-familiar “Wait… did I CTRL+S before I hit CTRL+R?” moments.

Click here to see the full release notes for version.

BONUS reason: Tricentis Flood support

You can now upload JMeter 4.0 scripts to flood.io and be up and running with your first test hosted on one of our grid nodes around the world. Our customer support team is always happy to help you make the switch to 4.0.

How I Became Financially Independent in 5 Years

Thu, 22 Feb 2018 11:40:32 +0000

Here’s my secret: I’m 32 years old and I am financially independent. And my husband and I did it all, from $0 to where we’re at now, in 5 years.

What is financial independence?

But let’s back up. Financial independence means I no longer need to work to support myself and my husband. This does not mean I’m filthy rich; this means only that if I stopped working right now, we would be able to continue living a simple life without needing to worry about paying the bills. If you actually do the calculations, that number is not as high as you might think.

Still, it’s a huge financial milestone that gives me a lot of freedom. Think about it: if you have enough money to cover my basic living expenses, why would you choose a job you don’t like? Why would you spend 8+ hours of my day, 5 days a week, with people whose company you don’t genuinely enjoy? It’s called financial independence because it’s the end of slavery — slavery to the rat race and to the whims of the corporate world. Financial independence gives you the freedom to design and pursue the life you actually want.

Where we started

When my husband Rob and I moved in together, we had a negative net worth. We had his three teenage children living with us and neither of us had a job. We lived in a small town that didn’t have many job opportunities in his industry and I, with my Australian permanent residency still in progress, was not allowed to work in Australia. Finally Rob found a job in Melbourne, and we moved the entire family there. We signed the lease on a ridiculously expensive rental house in order to get the kids in the right zone for the best high school we could afford. Despite Rob’s income, we struggled, borrowing from credit cards to finance expenses we just couldn’t seem to get on top of.

Eventually I got my permanent residency and a job. Even though my income could just barely cover the rent, that money made us feel rich. As the years went by and my salary increased, so did our expenses. We had lunch out nearly every day while we threw out spoiled groceries at home. We went to several concerts and theatres, some of them twice or thrice. We bought a car — on loan, of course. Almost our entire paychecks were immediately spent paying off the credit card bills from the last month; then we’d use the same credit cards to pay off this month’s expenses. In short, we were incredibly irresponsible. Still, thanks to dual higher-than-average IT incomes, we managed to convince a bank to give us a mortgage. Despite our incomes we could only afford a laughable 2% downpayment on the house — while we still owed money on credit cards.

Year 0 actually begins the day Rob’s current project ended. He was a contractor, so no project meant no job. Knowing we’d have to live tight on just my salary to meet the mortgage and our monthly living expenses, I started to search for budgeting tips online. What I found would eventually lead us down the road to financial independence. I took control of our finances and changed them around. Here’s what I did.

What we did

I consolidated all our accounts**.

We had several transaction accounts, savings account and credit cards, and the first step was to have fewer buckets for our money. Having fewer accounts made it easier to keep track of everything, especially in the beginning, and it also meant it was easier to calculate our (then negative) net worth and track changes to it.

I used the envelope system for budgeting to keep a lid on our daily expenses.

Specifically, I used an app called You Need a Budget (YNAB — and that’s an affiliate link), but you don’t need to buy it (although I personally think it’s more than worth it). I tracked every single cent then, and in fact I still do that today. Instead of having one account and one balance to look at, I created categories for every conceivable expense. Before I found YNAB, I just used an Excel spreadsheet. A big shift in habit was when I started to look at the category balance instead of the account balance. If we wanted to buy something that cost more than the budget for its category, we would look to see what other categories we could take the money out of. This helped us understand that every time we said yes to buying something, we were saying no to buying something else. Even if we paid using the credit card, that money still went out of the category balance so that we could at least make sure we weren’t adding to our credit card debt.

I automated transfers to pay back credit card debt and to our savings account.

I discussed with Rob an amount that we would save out of my salary every month for emergencies and I set up an automatic transfer from our normal transaction account to our separate savings account with another bank the day after payday. The automatic part is important — you don’t want to give yourself the chance to self-sabotage. This made sure that even if we went over budget, we would not touch our savings. I did the same for extra debt payments towards credit cards and our car loan until they were competely paid off. I also saved one month’s worth of household expenses as an emergency fund.

The waiting game

By the time Rob found a job again, we were debt free except for our mortgage. We suddenly had a lot of cash to spare, both from his salary and from paying off debts. As soon as that happened, rather than increasing our expenses again, I automated transfers to investment accounts instead. For us in Australia, this meant talking to our employers to salary sacrifice into our superannuation funds, for you, this could be contributing to your 401k. We started salary sacrificing as much as we could per year (at the time, that was $30,000 for me and $35,000 for Rob including employer contributions, but check the current limits with the ATO as they have changed). This was a huge deal because we never even saw the money as it was garnished from our wages. It also saved us heaps of tax (one year I calculated it to be about $8000 in savings for Rob alone). At this point we were saving 70% of our income.

When we got unexpected income increases or tax refunds, we put it towards investments. As our incomes grew, we maxed out our super contributions and started to look for something else to invest in. I invested our money into a low cost index fund, available from any stock broker. In our case, we used Vanguard’s VAS for Australian shares. Again, those payments were automated and transferred regardless of whether the stock market was doing well or not.

Then, we waited.

The result

Within five years we had nearly enough to meet our subsistence-level figure. Some surprise financial windfalls finally pushed us over the edge.

I’m sharing this story not to brag about my successes but to let people know that financial independence is not just possible but SIMPLE.

Let me break down the steps again for you:

Calculate your financial net worth.
Revisit and optimise all your accounts.
Track every single expense for at least a month.
Make a budget.
Automate transfers to improve your net worth: first towards your debt, then savings towards an emergency fund and then to an investment account.
Wait.

Now I’m not saying that everyone can do this in five years. My husband and I were lucky enough to have dual high incomes. But the point is not to do all of this in five years. The point is to start now.