Test automation mertics
Blog

Test automation metrics: KPI components

The process should be evaluated while the team activities show certain KPI. Apart from usual methods of KPI measurement I would like to offer you metrics that can be necessary for evaluation of the first three mentioned activities.
29 January 2014
Test automation
The article by Dmitry Tishchenko
Head of global business development at a1qa

In this test automation cycle, I tried to give a detailed overview of the specifics of automation testing services. We went through the process steps, risks, returns. So far, you know that the process bases upon several customary activities:

  • Automated test development
  • Automated test support
  • Test run and result analysis
  • Test automation monitoring.

Though, any process doesn’t finish on receiving the result. The process should be evaluated while the team activities show certain KPI. Apart from the usual methods of KPI measurement, I would like to offer you metrics that can be necessary for the evaluation of the first three mentioned activities. Let’s make it clear where to draw attention.

We start with those necessary for automated test development

  • Automated test coverage (automated tests vs. manual tests). The activity reflects the correlation between automated and non-automated tests. The metric has more an administrative approach than one describing the process specifics. Though used along with the next one it gives quite a transparent activity overview.
  • Cost per automated test. Applying the metric you’ll notice that the price per one automated test decreases. The tendency of cost shrinking is the result of 2 factors: 1) automation solution. The efficient architecture allows engineers to use the code several times. 2) Human factor. People eventually start working faster after careful learning the system of automation solutions.
Manual vs automated test cases
Average test case development cost

Earlier, in one of my previous posts of this cycle, I mentioned the time2market parameter, which is important for the efficiency evaluation. The parameter is closely connected to the activity of the test run and result analysis. But take it as the parameter of the process duration. Apply metrics on this step and analyze the time of automated test execution, use smoke test for instance.

Smoke test duration

The metric concerning the correlation of useful and irrelevant results gives important information. While applying the metric keep in mind the definition of useful and irrelevant results.

Useful vs irrelevant results

  • Useful results: test pass, test failure caused by a defect.
  • Irrelevant results: test failure – problems with the environment, data-related problems, changes in the application (business logic and/or interface).

The irrelevant results are those that decrease the automation efficiency from the economic and duration point.

The percentage of the irrelevant fails is also the data used in metrics. The acceptable percentage level depends upon the project, though the figure of 20% can be used a basic adequate level.

Useful vs irrelevant results

If the rate of the irrelevant results exceeds the acceptable level, pay special attention to the source of the issue. Lower in the diagram you can see how changed the automation process after elimination of the secondary issues. The diagram was made on one of our latest projects.

Irrelevant fails structure

So in a nutshell, being aware of automation process specifics and methods of efficiency evaluation, you have the basic information necessary for tracking the process. Now you are able to find the reasons for your automation project success to keep up with and understand what and why went wrong. Still, there always are some things to know more and I am always ready to provide you with the necessary details.

More Posts

na-st-awards-23
16 November 2023,
by a1qa
3 min read
a1qa shines as the finalist in three categories of the North American Software Testing Awards
a1qa is a triple finalist at the North American Software Testing Awards.
General
Quality assurance
Test automation
3 November 2023,
by a1qa
4 min read
From idea to buying: 7 testing types to make your mobile eCommerce solutions flawless
Read the article to discover 7 QA activities helping boost mobile eCommerce solutions quality and provide end users with exceptional buying experiences.
Functional testing
General
Quality assurance
Test automation
Usability testing
On the way to Web 3.0: key software testing aspects for seamless digital experiences. Part 2
12 October 2023,
by a1qa
4 min read
On the way to Web 3.0: key software testing aspects for seamless digital experiences. Part 2
Let’s analyze essential software testing checks to improve the quality of the business-critical Web 3.0 functionality.
Cybersecurity testing
Functional testing
Performance testing
Quality assurance
Test automation
Usability testing
On the way to Web 3.0: key software testing aspects for seamless digital experiences. Part 1
11 October 2023,
by a1qa
4 min read
On the way to Web 3.0: key software testing aspects for seamless digital experiences. Part 1 
In part 1 of this article, learn about the transformation to a new Internet era, Web 3.0, and its benefits for increasing operational efficiency.
General
QA trends
Quality assurance
Software lifecycle QA
Test automation
goodfirms-lists
26 September 2023,
by a1qa
2 min read
a1qa’s recognition by GoodFirms: A testament to our expertise
Delve into a1qa's recent acknowledgment by GoodFirms and explore the significance of this accolade in the sphere of software testing.
General
Performance testing
Test automation
ai-ml-test-automation-mini
21 September 2023,
by a1qa
5 min read
6 activities to supercharge test automation with AI and ML
How to maximize the benefits of test automation? Learn how to do it via applying AI and ML.
Test automation
QA for financial applications: 4 reasons why it is a must-have
14 September 2023,
by a1qa
3 min read
QA for financial applications: 4 reasons why it is a must-have
Striving to release high-quality financial apps? Read on and discover 4 core reasons why your eBanking solutions need software testing.
QA consulting
Quality assurance
Test automation
gaming-qa
24 August 2023,
by a1qa
4 min read
Ready, steady, test: How QA drives seamless gaming experiences
Why is QA pivotal for delivering unmatched player experiences? How to level up video game quality? Find the answers in the article.
Cybersecurity testing
Functional testing
Localization testing
Performance testing
Quality assurance
Test automation
Usability testing
debated technologies
30 May 2023,
by a1qa
3 min read
a1qa tech voice: Managing director at a1qa, North America, discusses pros and cons of much-debated technologies
Nadya Knysh, Managing director at a1qa, North America, puts a spotlight on 6 current technologies, discussing their positives and negatives.
General
Interviews
Test automation

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.