Release at pace with test automation: What, why, and how to measure success?

An automation-first approach is key to enhancing testing capabilities and increasing overall operational efficiency. However, I would suggest justifying its implementation, so that it can deliver the promised value.
7 December 2022
Quality assurance
Test automation
Dileep Marway
The article by Dileep Marway

Automation testing has been a buzzword to accelerate your software quality needs. I have seen in many C-suites that it is always seen as the only answer to releasing at pace. Though do we really understand the value it can add and also the risks around it? As it can easily eat your company funds if it is not implemented in the correct way.

In this article, I will be covering the basics and will be giving some tips so that you have a steady journey on the automation rollercoaster. Let’s go!

What is automation testing?

It is a technique where specialists use automation testing software tools to execute a test case suite.

Automation testing demands considerable investments of money and resources, and that is why it must be done properly.

Why is test automation beneficial?

From my experience, it is the best way to increase effectiveness, test coverage and increase the velocity by which we release.

Some statements around automation benefits:

  • In some cases, it can be more than 70% faster than manual testing.
  • It generally is reliable if done correctly and maintained smartly.    
  • Automation can be run multiple times and overnight.
  • It reduces the chance of human error and improves accuracy.
  • Automation testing helps increase test coverage.

Automation for me, has been a life saver, where we have not had the capacity or time to test manually. It has helped to catch defects which would have impacted our customers greatly.      

My key advice is to run automation frequently, and the quicker the feedback — the better!

What should we automate?

Now, this is a really important question. We should not automate everything, especially if it leads to flaky tests.

6 cases to automate:

  1. High-risk areas — business-critical test cases.
  2. Test cases that are repeatedly executed.
  3. Positive test cases.
  4. User interface (UI) tests.
  5. Test cases that are very tedious or difficult to test manually.
  6. Test cases that are very time-consuming to run and create.

Example scenarios of what we ‘should’ automate:

  • Compare two images pixel by pixel.
  • Comparing two spreadsheets containing thousands of rows and columns.
  • Testing an application on different browsers and different operating systems (OS) in parallel.

What should we NOT automate?

Now, the counter question, this is key and we should always have this at the back of our mind when automation test design takes place.

We should not automate:

  • Test cases that are newly designed and not executed manually at least once.
  • Test cases for which the requirements are frequently changing.
  • Test cases which are executed on an ad-hoc basis.
  • Negative failover tests.
  • Tests with massive pre-setup.
  • Test cases where the return on investment based on automation effort will take a long time.

The process of test automation

This should be planned out, and some time should be invested to think it through. Many times, I have seen automation fail due to flaky tests and people choosing to automate for the sake of automation.

Here are the steps for automating your software testing:

1. Select the test tool — this is done via a proof of concept and involves many stakeholders so that the right decision can be made.

2. Define the scope of automation — facts to consider:

  • The features — this should be clear
  • Devise the scenarios which need a large amount of data
  • Technology feasibility review
  • Review the complexity of test cases
  • Review whether we want to incorporate cross-browser testing.

3. Planning, design, and development:

  • Automation tool selection
  • Framework design and its features:
    • The most popular open-source web framework is ‘Selenium WebDriver’
  • Define your scripting standards, such as:
    • Uniform scripts, comments, and indentation of code
    • Exception handling
    • User-defined messages are coded
  • In-scope and out-of-scope for automation
  • Automation test bed preparation
  • Schedule and timeline of scripting and execution
  • Deliverables of automation testing.

4. Test execution.

5. Maintenance.

The last phase of ‘maintenance’ is very key — as if the tests are not maintained, the technical debt is not brought down and those ‘flaky’ tests are not removed; you will actually find that automation testing is taking more time and money investment than manual testing.

How to measure the success of your automation suite?

So, we have automation in place. It is important to track success, and if success is not being met, if you have data to track, you can get back on the right path.

Some automation metrics I would recommend to measure:

  • Percentage of defects found by automation.
  • Time required for automation testing for each and every release cycle.
  • Time taken for a release due to automation testing vs time taken if scripts are manually tested.
  • Customer satisfaction index.
  • Productivity improvement.


Automation is awesome, and it can really add to your QA capabilities, though it must be done and thought out properly.

More Posts

debated technologies
30 May 2023,
by a1qa
3 min read
a1qa tech voice: Managing director at a1qa, North America, discusses pros and cons of much-debated technologies
Nadya Knysh, Managing director at a1qa, North America, puts a spotlight on 6 current technologies, discussing their positives and negatives.
Test automation
17 April 2023,
by a1qa
4 min read
Why do bugs get missed? Learn the problems and tips to avoid them
Still, finding overlooked bugs after the app goes live? Let’s find out why this happens and how to fix it.
Quality assurance
Test automation
Mobile app testing
15 February 2023,
by a1qa
4 min read
Mobile app testing guide: win the race with five-star software
Which aspects of mobile apps to test first to produce a really high-quality product? Find the answer to this and other questions related to mobile app testing in the article.
Cybersecurity testing
Functional testing
Mobile app testing
Performance testing
Test automation
Usability testing
31 January 2023,
by a1qa
5 min read
Best of 2022 by executives: 8 most visited a1qa blog posts
Let’s look back and revisit the most visited a1qa articles of 2022!
Quality assurance
Test automation
12 January 2023,
by a1qa
4 min read
The future of software testing: top 4 impactful trends that will dominate in 2023
Consider the major industry trends for the upcoming year to know how to improve current QA strategies and stay ahead of the curve.
QA trends
Quality assurance
Test automation
2 December 2022,
by Dileep Marway
3 min read
What is a сulture of happiness for a QA company?
Great culture drives employee engagement and satisfaction, contributes to an impressive work output, and improves organisational performance. But how do we build a culture of happiness? In this blog, I will share more on the answer to this.
Quality assurance
30 November 2022,
by Dileep Marway
3 min read
Agile QA – what is needed for greater flexibility and speed?
What should your QA team focus on to become truly agile, enable quality at speed, and contribute to lasting performance improvement? In addition to introducing test automation, I suggest considering shared responsibility for software soundness.
Quality assurance
28 November 2022,
by a1qa
9 min read
Interview with Dileep Marway on a series of articles “Agility and speed: Supercharging your business strategies with QA”
We cooperated with the VP of Engineering and Quality at SHL to present you with a series of his blog posts on: culture of happiness, test automation, and Agile-driven QA. Happy reading!
Quality assurance
Software lifecycle QA
Test automation
31 October 2022,
by a1qa
4 min read
Optimizing telecom budgets with QA outsourcing: everything you need to know. Part 2 
Welcome to Part 2 of our blog on QA outsourcing and optimizing telecom budget with it. Let’s delve deeper into the topic!
QA trends
Quality assurance

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.