Test automation mertics
Blog

Test automation metrics: KPI components

The process should be evaluated while the team activities show certain KPI. Apart from usual methods of KPI measurement I would like to offer you metrics that can be necessary for evaluation of the first three mentioned activities.
29 January 2014
Test automation
The article by Dmitry Tishchenko
Head of global business development at a1qa

In this test automation cycle, I tried to give a detailed overview of the specifics of automation testing services. We went through the process steps, risks, returns. So far, you know that the process bases upon several customary activities:

  • Automated test development
  • Automated test support
  • Test run and result analysis
  • Test automation monitoring.

Though, any process doesn’t finish on receiving the result. The process should be evaluated while the team activities show certain KPI. Apart from the usual methods of KPI measurement, I would like to offer you metrics that can be necessary for the evaluation of the first three mentioned activities. Let’s make it clear where to draw attention.

We start with those necessary for automated test development

  • Automated test coverage (automated tests vs. manual tests). The activity reflects the correlation between automated and non-automated tests. The metric has more an administrative approach than one describing the process specifics. Though used along with the next one it gives quite a transparent activity overview.
  • Cost per automated test. Applying the metric you’ll notice that the price per one automated test decreases. The tendency of cost shrinking is the result of 2 factors: 1) automation solution. The efficient architecture allows engineers to use the code several times. 2) Human factor. People eventually start working faster after careful learning the system of automation solutions.
Manual vs automated test cases
Average test case development cost

Earlier, in one of my previous posts of this cycle, I mentioned the time2market parameter, which is important for the efficiency evaluation. The parameter is closely connected to the activity of the test run and result analysis. But take it as the parameter of the process duration. Apply metrics on this step and analyze the time of automated test execution, use smoke test for instance.

Smoke test duration

The metric concerning the correlation of useful and irrelevant results gives important information. While applying the metric keep in mind the definition of useful and irrelevant results.

Useful vs irrelevant results

  • Useful results: test pass, test failure caused by a defect.
  • Irrelevant results: test failure – problems with the environment, data-related problems, changes in the application (business logic and/or interface).

The irrelevant results are those that decrease the automation efficiency from the economic and duration point.

The percentage of the irrelevant fails is also the data used in metrics. The acceptable percentage level depends upon the project, though the figure of 20% can be used a basic adequate level.

Useful vs irrelevant results

If the rate of the irrelevant results exceeds the acceptable level, pay special attention to the source of the issue. Lower in the diagram you can see how changed the automation process after elimination of the secondary issues. The diagram was made on one of our latest projects.

Irrelevant fails structure

So in a nutshell, being aware of automation process specifics and methods of efficiency evaluation, you have the basic information necessary for tracking the process. Now you are able to find the reasons for your automation project success to keep up with and understand what and why went wrong. Still, there always are some things to know more and I am always ready to provide you with the necessary details.

More Posts

why-do-bugs-get-missed
31 August 2022,
by a1qa
4 min read
Why do bugs get missed? Learn the problems and tips to avoid them
Still, finding overlooked bugs after the app goes live? Let’s find out why this happens and how to fix it.
Quality assurance
Test automation
30 June 2022,
by a1qa
4 min read
App software testing for telecom: What are the common issues telco providers face?
Facing problems with the quality of your telecom software products? Read more in the article and find out the ways to address them.
Cybersecurity testing
Performance testing
Test automation
Mobile app testing
31 May 2022,
by a1qa
4 min read
Mobile app testing guide: win the race with five-star software
Which aspects of mobile apps to test first to produce a really high-quality product? Find the answer to this and other questions related to mobile app testing in the article.
Cybersecurity testing
Functional testing
Mobile app testing
Performance testing
Test automation
Usability testing
Test automation in Agile
20 May 2022,
by a1qa
5 min read
Test automation in Agile and DevOps: Maximizing flexibility and speed
Global market tendencies and user behavior are changing rapidly, if not talking about the software itself. Familiar situation? Find out how to get ahead of the curve with test automation in Agile and DevOps.
Agile
Test automation
19 April 2022,
by a1qa
5 min read
What prevents companies from boosting eCommerce customer experience: 4 common mistakes
Dreaming of a flawless online shopping journey for your users? Explore 4 widespread situations that hamper achieving this goal.
Cybersecurity testing
Performance testing
Test automation
Usability testing
2 February 2022,
by a1qa
5 min read
Interview with Mike Urbanovich: How to build a robust test automation strategy?
The Head of testing department at a1qa answers the questions on how to smartly build a winning test automation strategy and talks about the advantages you may obtain with it.
Test automation
31 January 2022,
by a1qa
4 min read
Scriptless test automation: revealing the truth via 3 misconceptions
Planning to go scriptless but in doubt? Explore 3 delusions of codeless test automation and its benefits when introduced smartly.
Test automation
test auto speed
11 November 2021,
by a1qa
4 min read
Test automation: 3 core activities to keep pace with dynamic software changes
Planning to release high-quality IT solutions faster? Find out which test automation activities help keep up with constantly updating products.
Test automation
ai-ml-test-automation-mini
29 October 2021,
by a1qa
4 min read
4 key QA activities to solve test automation challenges via AI and ML
How to address the difficulties caused by implementing test automation? Learn how to do it via applying AI and ML.
Test automation

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.