Top challenges to testing automation in 2021

Agile organizations strive for flexibility and quick response. It follows that they want to conduct software tests as efficiently as possible. Efficient testing leads to faster delivery and increased productivity. In software testing, efficiency doesn’t just speed up development, it also delivers positive results. For example, it can increase the number of defects caught and corrected, while decreasing the number of defects that go undetected and escape into production.

Many organizations see testing automation as one of the holy grails for attaining better testing efficiency. With automation, organizations can cover their bases at scale and minimize latency. This frees up talent and other resources to increase focus on things like innovation, solving problems, and user stories.

Test automation is far from simple to implement. In a 2020 article published in the Journal of Software Evolution and Process, a survey of 72 organizations from eight different countries found that “test management and test automation are considered the most challenging among all testing activities by practitioners.” In order to embrace and expand testing automation with minimal growing pains, your organization should be aware of the technical challenges to automated testing as well as other issues you may face.

High implementation costs

Automation increases testing velocity but requires a significant investment in front-end capital. It is a hard sell to management because a “payback” period can be unpredictable or protracted. In some cases, it may never even happen. This is especially true if best practices aren’t followed, which include capturing data that measures the value created by additional internal team productivity as well as enhanced product performance.

The biggest way to ensure positive ROI after comprehensive testing automation adoption is to implement an automated testing solution that integrates with other products in the ecosystem. This can enable end-to-end features, like robust analytics measured in near real time. One such metric that can be derived is the speed index, which tells users how much time it takes for their application to load, including on-page elements as they populate in real time. Being able to test these factors and aggregate performance across all stages of development can allow for better changes to be released faster.

Migrating away from open-source architecture

Open-source solutions have been very popular due to their affordability, and the fact that they are built around outcomes rather than features. But open-source tools have their limitations. There’s not much of a development budget to speak of for these core projects, which keeps the most active contributing developers from being able to spend much time on them. Lacking financial incentives and dedicated developers, there’s little impetus for open-source tools to adopt advanced features like artificial intelligence (AI) or machine learning (ML).

Organizations looking to advance available open-source solutions must be willing to dedicate their own time and money to the project. This commitment opens up cost/benefit considerations, such as whether a proprietary architecture would better suit the specific needs. There are also considerations involving the license, as some open-source projects explicitly discourage monetization or they encourage new branches of the tool to have their source code made public.

Open-source architectures also present challenges with security, as the fact that the source code is known can allow bad actors to readily explore for exploits. Solutions like end-to-end encryption can make a testing tool secure even when the source code is publicly available, but development teams must consider aspects like this on a case-by-case basis.

In some cases, the best option may still be an open-source solution. But organizations should look at their requirements for testing to determine the best tool for the job, and not just go with the solution that is cheapest in the short run. Instead, they should only embrace open-source tools when they feel that the community surrounding the tool can share their values and their desire for specific functionality based on the most common use cases.

Fragmentation in testing ecosystems

Currently, most organizations lack consistent features and integration capabilities, which can lead to fragmented testing ecosystems. For example, some testing automation platforms may play really nice with your development toolchain, but they lack integrations for implementing operations feedback into the test parameters. Testing environments themselves can also be highly fragmented. Some exclusively apply to mobile, desktop, browser, or other environments.

For this reason, development teams should look for testing solutions capable of providing functionality across the stages of development and with the capability to integrate with other tools across the DevOps environment. They should also choose testing toolsets that observe performance across all of the intended device environments, potentially including legacy devices.

Hard-to-interpret reporting

Easy-to-understand reporting is as important as the tests themselves. Testing models may be built upon highly specific scenarios, and because of this automation architecture, the individual results may not present a view that’s familiar to most parties. In order for the reports to be useful to the parties reading them, they may need to be provided with additional interpretation and contextual data.

When evaluating an automated testing solution, consider its ability to present actionable dashboard-derived insights. At the very least, look at its ability to integrate with a robust analytics platform that can provide the actionable insights desired.

Collaboration and continuity of work between teams

Testers and developers must work together to ensure that the testing process goes smoothly. Testing, and especially automated testing, can lead to hang-ups during the “handoff” phase.

According to Stephen Rosing, director of solutions engineering at ACCELQ, “the most important line of communication to build is between testers and developers. It is critical that the development team understands the basic working of the automation tool to ensure the application lends itself to automation.”

There are several places where communication can break down and cause problems. Testers may misunderstand goals, while developers may fail to incorporate feedback from testers into improving code quality.

“Developers love to use the latest and greatest code libraries, but there is often a delay before automation tools support these,” says Rosing. “By communicating with the development team you can ensure that you aren’t forced back to manual testing because development introduced unsupported technology.”

Reaching complete test coverage

With any testing, there’s always a chance it could miss something. This is especially true in automated testing. There are factors that may not have been considered within the model and may go overlooked.

The key to avoiding these blind spots is to track metrics that can reveal trends of problems associated with low testing coverage, such as a high defect escape rate. Granular analysis of these metrics can reveal what specific types of code or areas of code are getting passed over. This analysis can also prioritize and chase better performance over time.

An integrated, end-to-end analytics solution can automatically monitor performance both inside and outside testing and then bring relevant factors to the forefront. For example, an alert can show up when testing an app feature that has shown security vulnerabilities in the past few weeks. This alert is not necessarily based on internal testing logic or a user story under consideration but rather the actual performance of the app in production. Such functionality can close the gaps in testing while helping teams focus on real-world scenarios, not just the purely hypothetical.

Focusing on automation over results

Many organizations that embrace the benefits of agile processes and the higher velocity of continuous integration and continuous deployment (CI/CD) want to implement test automation where they can. This can become a quest of implementing automation for automation’s sake.

As Amir Ghahrai says, “the problem, especially in agile development, is that QAs take a user story and automate its acceptance criteria. While doing so, their main and only focus is to wrestle with their limited coding skills just to get the test passing.”

Automation, after all, is supposed to make the testing process better. Even though things can be automated doesn’t automatically mean they should. It’s better to be selective on what to automate, and base it on the risk and potential impact to the user or organization. Automate testing that is run on a regular basis, such as regression tests to confirm the system is still functioning.

When implementing automated testing, there need to be other metrics beyond, “is this automated? Yes/No.” Retain a focus of the value and benefits of automation, such as the quicker cycle time, the higher deployment frequency, lower defect escape rate, and less unplanned work. Developers also face a lot of pressure to attain a high speed of testing in order to keep up with sprints and CI/CD goals. Organizations may need to dial back on test automation goals in order to keep their automated testing pipeline fine-tuned.

The goal of testing automation should be fast value, not fast tests

No matter what an organization hopes to achieve with automated testing, the primary objective should be to create value efficiently, not just to complete tests quickly. Tests that run fast but miss defects that will later cause problems aren’t doing their job or adding value to the organization.

A value stream management approach can allow organizations to quantify the value created at all states of DevOps, including testing. Testing automation can also fuel a fast feedback loop to drive optimization of all DevOps processes.

Testing automation can be challenging, as well as expensive. But the results can be better products in the hands of customers faster, with improvements delivered more often. Leveraging value stream mapping and analytics, not just automation, can empower an organizational culture where results are constantly improving, not just moving faster.

See how the benefits of test automation have grown and what to expect from automated testing in the future in our webinar: “Benefits of test automation – past, present, and future”.