Web and App Performance Testing Checklist

Delivering business value is the name of the game at Experitest and our parent organization Digital.ai. We are both interested in building customer trust via innovative digital experiences. The scope of the entire organization is based around Value Stream Management or VSM, our part of the process as we are sure you already know is helping our customers deliver flawless web and mobile apps. One of the ways we do this is with performance testing which is why I wrote this performance testing checklist, but first.

Our continuous testing platform has two main targets:

1. Creating & executing hundreds of manual or automated tests in parallel on real devices in the cloud.
2. Automating your cross-browser testing by running Selenium tests on 1,000+ browser types, versions, and operating systems.

“But wait”, I can hear you shouting at me in your best Ron Popiel impression, “There’s more”. And you would be right because even if you do all of that testing if your app wastes device memory or battery power it will sink as soon as it gets to the app store. That is why we have prepared this performance testing checklist to help you.

What is performance testing?

The process of testing web and mobile app performance is to test the speed, response time, reliability stability, and scalability of your app using a given workload. The result you are looking for is to get rid of any performance bottlenecks in your web and mobile apps. In some circles, it is known by the slang “Perf Testing” not to be confused with the testing of foam projectile firing toy guns. That would be known as Nerf Testing.

Performance testing focuses on the following:

• Speed – How quickly your app responds.
• Stability – How stable an app is under load stresses.
• Scalability – The maximum number of users that can load the application at once with no issues.

Without performance, testing to eliminate the performance bottlenecks your web and mobile apps are likely to suffer from running slow when too many users are accessing it, poor usability, and inconsistent operation across different OSes.

Here are a few statistics courtesy of Dun & Bradstreet that illustrate the importance of Performance Testing.

• Fortune 500 companies experience almost 2 hours of downtime per week. This adds up to around $46 million per year.
• Google crashed for 5 minutes in 2013 which cost them as much as $550 thousand dollars.
• During a recent AWS outage companies lost sales worth up to $1100 per second.

Armed with this important information let’s take a look at the performance testing checklist starting with the different areas of performance that need to be tested with an example for each one.

Types of performance tests

Load Testing – A process of non-functional web and mobile app testing where performance is measured under the app’s specific expected load. This type of performance testing identifies how a web or mobile app behaves when accessed by many different users at once. The goal of Load Testing is to release app performance bottlenecks, while simultaneously making an app more stable and highly functional before release.

• Example of Load Testing – An Airline experienced many app crashes as millions of people tried to change their flights due to the Covid-19 situation.

Stress Testing – This is the next level of Load Testing. Instead of trying to see how an app performs under expected loads the idea here is to pile on the load until you find the app’s breakpoint. You want to stress test in order to find your app’s breaking point.

• Example of Stress Testing – An airline that is working to upgrade its load capabilities tests their app to the load of 2,500 instead of the 2,000 the app is rated for.

Endurance Testing – Also known as Soak Testing endurance testing checks how long a web or mobile app can function at its peak expected load. It helps to discover issues like memory leakage.

• Example of Endurance Testing – An accounting company has an app that helps people file their taxes. Understanding that this process takes a long time the company would execute endurance tests that run for 24-48 hours to ensure the app is functioning properly.

Spike Testing – Tests how a web or mobile app functions under a sudden user increase. Additionally, it helps understand how apps recover from these sudden bursts of traffic.

• Example of Spike Testing – A social media app experiences huge bursts of traffic when viral news hits. They use spike testing to simulate viral news experiences.

Volume Testing – This is accomplished by adding a large amount of data to your web or mobile app database. The object is to see how the app runs under different database volumes.

• Example of Volume Testing – A retail app needs to add hundreds of new clothing items for the new season. They enter all of that information into the database and then run their performance tests.

Types of Metrics that are measured with Performance Testing

The way that we carry out these different types of performance tests is by monitoring devices and measuring various metrics that will show us how to improve our web and mobile apps. Before we delve in, a small clarification.

• Measurements – All of the data that is collected from an app.
• Metrics – Using Measurements to define result quality (average/total response time etc.).

Here is the performance testing checklist for ways to measure speed, stability, and scalability.

• Response Time – Time it takes to send a request and receive a response.
• Wait Time – How long it takes to receive byte #1 after sending a request. Also called Average Latency.
• Average Load Time – Average time taken to deliver requests. The major indicator of app quality from a UX perspective.
• Peak Response Time – Measurement of the longest time it takes for a request to be filled. If this time is longer than average it might indicate an issue.
• Error Rate – By comparing the percent of requests that result in errors to all requests you will calculate this number. These errors are usually a result of issues with load capacity being exceeded.
• Concurrent Users – Total number of active users at any time. Also called Load Size.
• Requests per Second – Number of handled requests
• Passed/Failed Transactions – The total number of requests whether or not they are successful.
• Throughput – Amount of bandwidth used during the test. Measured in KB per second.
• CPU Utilization – Time it takes for the CPU process a request.
• Memory Utilization – Memory needed to process a request.

Tools to test Web and Mobile App Performance

There are many tools that help you Implement performance testing early and continuously as part of your CI pipeline. These tools help you deliver high-quality software with great customer experiences.

With the increasing number of these tools and the difficulty in differentiating between them, we will simply focus on our tool.

SeeTest’s Performance Testing offering gives you the ability to add mobile and web performance testing to your test automation suite. By utilizing performance testing for mobile and web, you will uncover the cause of performance issues early. This will eliminate delays in delivery and before your users find them.

Our tool also provides In-Depth Measurements of Performance Data, Including:

• Transaction duration
• Speed Index
• CPU consumption
• Memory Consumption
• Battery Consumption
• Network data (upload and download)

Bringing the value with this Performance Testing Checklist

Performance testing helps to reveal how your web and mobile apps behave in different situations. Sure it might run well with 10k users running it concurrently, but what about 100k or more? A desire to provide a high speed, stable, and scalable app experience drives performance testing.

To conclude, we saw above how App Performance testing helps your web and mobile app releases. The truth is that this type of testing should be implemented early in the development process. It will enable you to catch bugs earlier and by extension boost satisfaction while saving time and money.

Jonny Steiner – Content Manager