Good practice demands not just testing automation, but testing of automation itself

Digital transformation and continuous testing continue to accelerate, and, as a result, organizations will have to implement a smarter approach to test automation. As we previously stated in a recent blog about building a culture of QA, “Testing automation also gives organizations the agility needed to deliver changes fast while minimizing risks. Furthermore, test automation creates an infrastructure within the development so that every stage of work is checked for integrity and performance, as well as priority issues, such as security, governance, and compliance.”

However, Diego Lo Giudice, VP Principal Analyst at Forrester, emphasized the need for organizations to take a smarter approach to test automation that involves testing and auditing the automation itself. He explained that “test automation tools, technology, and practices should and can be used for testing… the automation.” While test automation provides immense benefits for reaching test coverage and overall app quality, test automation practices must be regularly audited for performance, thoroughness, and maximum effectiveness. DevOps teams should ensure that they’re getting the results they need to efficiently deliver value to end customers.

To ensure that automated testing delivers on its promises, organizations can employ strategies such as:

• Expanding their testing practices to include testing of the automation tools themselves.
• Adopting a more comprehensive testing approach for their complex systems.
• Leveraging AI-based testing tools and platforms.

Organizations are moving to a complex automation technology landscape

As the use of complex automation technology grows among businesses, organizations are introducing multiple tools to expand automation into more areas and processes. These tools include:

• AI-infused applications, or automation using machine learning and ML models.
• Robotic Process Automation for automating tasks and procedures.
• Dynamic Case Management, including email and document management.
• Digital Process Automation, which encompasses all of these areas.

Plus, as organizations become more customer-focused, they are automating even more parts of the customer journey by adding more self-service features, mobile or social channels, or CRM. In these instances, automation is being introduced to create pleasing, engaging, and intuitive experiences for the customer.

As more automated and AI-infused processes are introduced, testing has to keep up with this expansion. As Lo Giudice states, “organizations need to ask themselves are they testing all of that automation, or are they limiting their testing to the usual regression tests and the front-end applications?” He asserts that test automation tools, technology, and practices should and can be used for testing the automation tools themselves.

Complex automated systems require large scale testing

As the need to develop tests of the processes and tools becomes more apparent, organizations will have to consider a number of pertinent questions to determine how their testing has to evolve. They need to determine:

• Are the testing tools we’re using relevant to the environment and the multitude of platforms being used?
• Are we testing the automated tools to make sure they’re still performing valid tests?

It’s also imperative to test the entire end-to-end process, which creates even further complexity. Testing the automation requires a large-scale, heterogeneous, end-to-end testing approach that covers private, public, hybrid clouds, multiple browsers, desktops, mainframe, devices, web, mobile, IoT, and POS. Implementing a large-scale testing tool, such as Digital.ai Continuous Testing, can provide a seamless and flexible solution.

Leverage AI to improve automated testing outcomes

There are a number of AI tools that can be introduced to improve automated testing. An article in Forbes outlined some of the ways that AI-based tools can benefit DevOps teams, including “eliminating test coverage overlaps, optimizing existing testing efforts with more predictable testing, and accelerating progress from defect detection to defect prevention.” It also noted, “AI-based software development platforms can identify the dependencies across complex and interconnected product modules, improving overall product quality in the process.”

Meanwhile, here are some specific use cases that illustrate how testing can be improved with AI:

• Tools that can complete self-healing of UI tests by applying AI and ML algorithms to dynamically adapt testing.
• Visually using AI for visual testing to make the process more precise.
• Using AI to generate test cases.
• Insights-driven testing or using AI and ML to optimize what is tested. This can be applied to the overall process and test strategy.

AI can also be used to help determine what should be tested next and what should be automated to improve the test coverage. It’s also important to note that AI doesn’t replace testers, but it does make them smarter. AI tools allow testers to be more effective at their jobs.

Interest and implementation of AI in testing is rising

As digital transformation becomes more widespread across multiple industries, there has been a surge in the number of organizations that are moving to incorporate AI in their testing processes.

Survey results from the World Quality Report 2020-2021 reveal that “Almost 90% of respondents claim that testing with AI and testing of AI are the biggest areas of growth planned in their organizations, and 80% intend to increase the number of AI-based trials and proofs of concept.” Additional results show that “Almost 80% of those organizations surveyed indicated that AI will be used to generate test data and test environments.”

Despite this expansion in AI testing, its implementation remains a complicated process, and can’t be viewed as a cure-all. Like any other automated testing or process, organizations must use metrics to audit and assess how well AI tests are working and whether they are delivering value.

Testing automated testing ensures results line up, maximizing value delivery

When considering the expansion of automation, it’s crucial for organizations to focus on the correct goals when determining the best testing approach. Automating for its own sake is not the goal.

We recently addressed this issue in an article on the challenges of testing automation: “Automation, after all, is supposed to make the testing process better. Even though things can be automated doesn’t automatically mean they should. It’s better to be selective on what to automate and base it on the risk and potential impact to the user or organization. Automate testing that is run on a regular basis, such as regression tests to confirm the system is still functioning.”

The goals of a testing approach should be delivering value, improving quality, and enhancing the testing process. In some cases, an automated testing approach will be best, while in other instances, manual testing will be the most effective.

Good practices include completing regular audits of testing and automation practices, as well as aiming for a mix of wide coverage and high-quality results. Whatever approach maximizes value will be the best solution.

Learn more about how AI/ML fits into testing from our webinar. The future of testing: Smarter and adaptable