AWS Testing—Optimizing Test Performance and Cutting Costs with Headless Browser Execution

Cutting testing time and improving test performance are the main promises of test execution using headless browsers. Now ... how to achieve that in practice?

This article will walk you through an AWS testing scenario of how to optimally utilize the power of your AWS Lightsail virtual private server (VPS) subscription to run automated UI tests in headless mode and save time and costs.

What’s Behind Headless Test Execution?

Before I jump into the test case I want to illustrate, let’s provide some context. By headless browser testing I mean the execution of functional UI end-to-end tests using a browser that’s stripped of any UI elements. When running a test “heedlessly,” the browser will navigate to a page, log in, perform actions and/or verifications based on the test script but without rendering the UI on the desktop. All Chromium-based browsers offer headless mode. Test Studio supports test execution on Chrome Headless.

The question here is how to benefit from headless test execution when your test automation infrastructure is hosted in the cloud.

AWS Testing: Measuring the KPIs of Headless Test Execution in a Cloud-Based Testing Infrastructure

There are three aspects of AWS testing with headless test execution that are critical to understand when measuring testing KPIs in cloud-based infrastructures, in our case AWS Lightsail.

1. Functional tests executed in headless mode will complete at least 50% faster than non-headless UI tests.
2. Headless test execution can reduce memory and CPU up to 30% (depending on the test scripts).
3. Running headless tests will result in inherently more stable test runs as unexpected operating system events (dialogs, windows errors, client errors, user interaction with the desktop while tests are running) will not affect test execution.

At the same time there are a few characteristics of the Amazon-based virtual machines (VMs) that are directly related to the performance and stability of the test runs.

The most obvious ones are the overall machine specs such as memory, CPU configuration, etc. It is critical to understand though that there are other aspects, which are not so obvious, but will affect test performance. These are CPU burst capacity and CPU utilization performance baseline. Here is where it matters significantly whether tests are run in headless or headed-UI mode.

Credit: AWS Lightsail Pricing

CPU Utilization Performance Baseline and Burst Capacity in AWS Lightsail

Here is a very detailed document provided by Amazon on how to view instance burst capacity in AWS Lightsail. For your convenience, here is a short summary:

CPU utilization performance baseline: This is measured in percentages and time per vCPU. For example, from AWS Lightsail Subscription options above, the recommended tier of $70 per month includes 30% CPU utilization. But what does this mean in a real-life scenario? There is the so-called sustainable zone when the CPUs of the VM are not used more than 30%.

CPU burst capacity: This metric stands for the amount of time that the CPU load was greater than 30%. The burst capacity is basically a bucket that is not endless. It gets refilled though when the CPU is underutilized. There is a set amount of time where the CPU usage may reach 100% or close to 100%, and when this time is over, the CPU will throttle down to the sustainable zone and stay there.

Credit: GitHub

How is this information relevant to test execution and why does it matter if tests are running headlessly or in headed-UI mode?

Test Execution Impact on CPU KPIs

Let’s explain what happens if a VM CPU burst capacity is overutilized. Be aware that it can happen in a few hours of sustained CPU load—for example, if you run tests over a few hours. The machine will slow down; respectively, the CPU will slow down as well. Moreover, it will get extremely slow, because the CPU will not be limited to run with more than 30% capacity (or even 20% if the machine is from a lower tier subscription).

How does this affect me?

Normal operations on Windows-based VMs will be slowed down significantly. Launching a browser will take at least one minute. Loading modern dynamic and complex web apps which calculate on the client side will be extremely slow. Running tests won’t be sustainable, especially if you are executing them in headed-UI mode. Tests won’t be synchronized correctly, and actions and verification steps will time out, causing your test runs to fail. On some occasions, the browsers will turn unresponsive.

Now, since you have this information, I am sure you will strive to meet these two goals at all costs:

1. Never run out of CPU burst speed.
2. If the VM is out of burst speed capacity, then make sure to run the most lightweight tests possible.

The Cornerstones of Headless Test Execution

Still remember the three cornerstones of headless execution?

The first and most important one is execution speed. Speeding up execution will help you by optimizing the machine. A single test, for example, can be executed two or three times in headless mode. For that same amount of time, you will be able to execute that same test only once in headed-UI mode.

Even if the test is executed on a machine where the CPU burst capacity is already down to zero, headless tests will perform better and will have a higher chance to run from start to end successfully, mainly because they are lighter in resource usage.

Real-life Scenarios and Implications

You are probably wondering what all of this means for you. Let’s talk in practical terms.

There is a significant difference between running tests with Test Studio in headless mode on Amazon-based machines compared with running the same tests using headed-UI browsers.

The real-life implications here are that if a user is dead set on running tests* in headed-UI mode for more than a few hours per day on a VM, they will need to invest at least $120 per month per machine (based on the official AWS Lightsail tier pricing).

Running the same tests* with the same intensity but in headless mode can be achieved on a machine for $70 per month (or even $40 per month) with plenty of spare room and capacity for on-demand runs, debugging or other actions that are critical to your testing architecture. In other words, you won’t need to worry you will run out of CPU capacity and will be able to focus on critical work.

* All measurements were done while testing functional end-to-end scenarios with Test Studio. Such a scenario could be Login 3 – select Items from an e-commerce catalog – add these to the shopping cart.

Recap

Since its latest R1 2021 release, Test Studio offers headless browser execution features to help testers and automation engineers modernize their testing cycle and improve test scenario coverage. Establishing testing architectures in the cloud is both modern and powerful, but it certainly requires reading the small print. Headless test execution greatly supports AWS testing by helping utilize your Amazon subscription’s full capacity without running the risk of failed test runs, sabotaged build integrations and compromised release quality.

Try Now