test-automation
Blog

Release at pace with test automation: What, why, and how to measure success?

An automation-first approach is key to enhancing testing capabilities and increasing overall operational efficiency. However, I would suggest justifying its implementation, so that it can deliver the promised value.
7 December 2022
Quality assurance
Test automation
Dileep Marway
The article by Dileep Marway

Automation testing has been a buzzword to accelerate your software quality needs. I have seen in many C-suites that it is always seen as the only answer to releasing at pace. Though do we really understand the value it can add and also the risks around it? As it can easily eat your company funds if it is not implemented in the correct way.

In this article, I will be covering the basics and will be giving some tips so that you have a steady journey on the automation rollercoaster. Let’s go!

What is automation testing?

It is a technique where specialists use automation testing software tools to execute a test case suite.

Automation testing demands considerable investments of money and resources, and that is why it must be done properly.

Why is test automation beneficial?

From my experience, it is the best way to increase effectiveness, test coverage and increase the velocity by which we release.

Some statements around automation benefits:

  • In some cases, it can be more than 70% faster than manual testing.
  • It generally is reliable if done correctly and maintained smartly.    
  • Automation can be run multiple times and overnight.
  • It reduces the chance of human error and improves accuracy.
  • Automation testing helps increase test coverage.

Automation for me, has been a life saver, where we have not had the capacity or time to test manually. It has helped to catch defects which would have impacted our customers greatly.      

My key advice is to run automation frequently, and the quicker the feedback — the better!

What should we automate?

Now, this is a really important question. We should not automate everything, especially if it leads to flaky tests.

6 cases to automate:

  1. High-risk areas — business-critical test cases.
  2. Test cases that are repeatedly executed.
  3. Positive test cases.
  4. User interface (UI) tests.
  5. Test cases that are very tedious or difficult to test manually.
  6. Test cases that are very time-consuming to run and create.

Example scenarios of what we ‘should’ automate:

  • Compare two images pixel by pixel.
  • Comparing two spreadsheets containing thousands of rows and columns.
  • Testing an application on different browsers and different operating systems (OS) in parallel.

What should we NOT automate?

Now, the counter question, this is key and we should always have this at the back of our mind when automation test design takes place.

We should not automate:

  • Test cases that are newly designed and not executed manually at least once.
  • Test cases for which the requirements are frequently changing.
  • Test cases which are executed on an ad-hoc basis.
  • Negative failover tests.
  • Tests with massive pre-setup.
  • Test cases where the return on investment based on automation effort will take a long time.

The process of test automation

This should be planned out, and some time should be invested to think it through. Many times, I have seen automation fail due to flaky tests and people choosing to automate for the sake of automation.

Here are the steps for automating your software testing:

1. Select the test tool — this is done via a proof of concept and involves many stakeholders so that the right decision can be made.

2. Define the scope of automation — facts to consider:

  • The features — this should be clear
  • Devise the scenarios which need a large amount of data
  • Technology feasibility review
  • Review the complexity of test cases
  • Review whether we want to incorporate cross-browser testing.

3. Planning, design, and development:

  • Automation tool selection
  • Framework design and its features:
    • The most popular open-source web framework is ‘Selenium WebDriver’
  • Define your scripting standards, such as:
    • Uniform scripts, comments, and indentation of code
    • Exception handling
    • User-defined messages are coded
  • In-scope and out-of-scope for automation
  • Automation test bed preparation
  • Schedule and timeline of scripting and execution
  • Deliverables of automation testing.

4. Test execution.

5. Maintenance.

The last phase of ‘maintenance’ is very key — as if the tests are not maintained, the technical debt is not brought down and those ‘flaky’ tests are not removed; you will actually find that automation testing is taking more time and money investment than manual testing.

How to measure the success of your automation suite?

So, we have automation in place. It is important to track success, and if success is not being met, if you have data to track, you can get back on the right path.

Some automation metrics I would recommend to measure:

  • Percentage of defects found by automation.
  • Time required for automation testing for each and every release cycle.
  • Time taken for a release due to automation testing vs time taken if scripts are manually tested.
  • Customer satisfaction index.
  • Productivity improvement.

Conclusion

Automation is awesome, and it can really add to your QA capabilities, though it must be done and thought out properly.

More Posts

a1qa-articles
31 January 2023,
by a1qa
5 min read
Best of 2022 by executives: 8 most visited a1qa blog posts
Let’s look back and revisit the most visited a1qa articles of 2022!
Quality assurance
Test automation
qa-trends
12 January 2023,
by a1qa
4 min read
The future of software testing: top 4 impactful trends that will dominate in 2023
Consider the major industry trends for the upcoming year to know how to improve current QA strategies and stay ahead of the curve.
QA trends
Quality assurance
Test automation
what-is-a-culture-of-happiness
2 December 2022,
by Dileep Marway
3 min read
What is a сulture of happiness for a QA company?
Great culture drives employee engagement and satisfaction, contributes to an impressive work output, and improves organisational performance. But how do we build a culture of happiness? In this blog, I will share more on the answer to this.
Quality assurance
agile-qa
30 November 2022,
by Dileep Marway
3 min read
Agile QA – what is needed for greater flexibility and speed?
What should your QA team focus on to become truly agile, enable quality at speed, and contribute to lasting performance improvement? In addition to introducing test automation, I suggest considering shared responsibility for software soundness.
Agile
Quality assurance
interview-with-dileep
28 November 2022,
by a1qa
9 min read
Interview with Dileep Marway on a series of articles “Agility and speed: Supercharging your business strategies with QA”
We cooperated with the VP of Engineering and Quality at SHL to present you with a series of his blog posts on: culture of happiness, test automation, and Agile-driven QA. Happy reading!
Agile
Quality assurance
Software lifecycle QA
Test automation
optimizing-budget-with-qa
31 October 2022,
by a1qa
4 min read
Optimizing telecom budgets with QA outsourcing: everything you need to know. Part 2 
Welcome to Part 2 of our blog on QA outsourcing and optimizing telecom budget with it. Let’s delve deeper into the topic!
QA trends
Quality assurance
optimizing-budget-with-qa
27 October 2022,
by a1qa
4 min read
Optimizing telecom budgets with QA outsourcing: everything you need to know. Part 1
Learn how to optimize telecom’s quality assurance expenses by relying on a trusted QA partner.
QA trends
Quality assurance
qa-trends-in-telecom
30 September 2022,
by a1qa
5 min read
4 telecom trends for 2023 and how to painlessly implement them with QA
It’s time to explore the telecom trends for the upcoming year. Let’s look at them together and also see the value that QA brings for their smooth deployment.
Cybersecurity testing
Migration testing
QA trends
Quality assurance
Test automation
ai-ml-test-automation-mini
15 September 2022,
by a1qa
4 min read
4 key QA activities to solve test automation challenges via AI and ML
How to address the difficulties caused by implementing test automation? Learn how to do it via applying AI and ML.
Test automation

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.