test-automation
Blog

Release at pace with test automation: What, why, and how to measure success?

An automation-first approach is key to enhancing testing capabilities and increasing overall operational efficiency. However, I would suggest justifying its implementation, so that it can deliver the promised value.
7 December 2022
Quality assurance
Test automation
Dileep Marway
The article by Dileep Marway

Automation testing has been a buzzword to accelerate your software quality needs. I have seen in many C-suites that it is always seen as the only answer to releasing at pace. Though do we really understand the value it can add and also the risks around it? As it can easily eat your company funds if it is not implemented in the correct way.

In this article, I will be covering the basics and will be giving some tips so that you have a steady journey on the automation rollercoaster. Let’s go!

What is automation testing?

It is a technique where specialists use automation testing software tools to execute a test case suite.

Automation testing demands considerable investments of money and resources, and that is why it must be done properly.

Why is test automation beneficial?

From my experience, it is the best way to increase effectiveness, test coverage and increase the velocity by which we release.

Some statements around automation benefits:

  • In some cases, it can be more than 70% faster than manual testing.
  • It generally is reliable if done correctly and maintained smartly.    
  • Automation can be run multiple times and overnight.
  • It reduces the chance of human error and improves accuracy.
  • Automation testing helps increase test coverage.

Automation for me, has been a life saver, where we have not had the capacity or time to test manually. It has helped to catch defects which would have impacted our customers greatly.      

My key advice is to run automation frequently, and the quicker the feedback — the better!

What should we automate?

Now, this is a really important question. We should not automate everything, especially if it leads to flaky tests.

6 cases to automate:

  1. High-risk areas — business-critical test cases.
  2. Test cases that are repeatedly executed.
  3. Positive test cases.
  4. User interface (UI) tests.
  5. Test cases that are very tedious or difficult to test manually.
  6. Test cases that are very time-consuming to run and create.

Example scenarios of what we ‘should’ automate:

  • Compare two images pixel by pixel.
  • Comparing two spreadsheets containing thousands of rows and columns.
  • Testing an application on different browsers and different operating systems (OS) in parallel.

What should we NOT automate?

Now, the counter question, this is key and we should always have this at the back of our mind when automation test design takes place.

We should not automate:

  • Test cases that are newly designed and not executed manually at least once.
  • Test cases for which the requirements are frequently changing.
  • Test cases which are executed on an ad-hoc basis.
  • Negative failover tests.
  • Tests with massive pre-setup.
  • Test cases where the return on investment based on automation effort will take a long time.

The process of test automation

This should be planned out, and some time should be invested to think it through. Many times, I have seen automation fail due to flaky tests and people choosing to automate for the sake of automation.

Here are the steps for automating your software testing:

1. Select the test tool — this is done via a proof of concept and involves many stakeholders so that the right decision can be made.

2. Define the scope of automation — facts to consider:

  • The features — this should be clear
  • Devise the scenarios which need a large amount of data
  • Technology feasibility review
  • Review the complexity of test cases
  • Review whether we want to incorporate cross-browser testing.

3. Planning, design, and development:

  • Automation tool selection
  • Framework design and its features:
    • The most popular open-source web framework is ‘Selenium WebDriver’
  • Define your scripting standards, such as:
    • Uniform scripts, comments, and indentation of code
    • Exception handling
    • User-defined messages are coded
  • In-scope and out-of-scope for automation
  • Automation test bed preparation
  • Schedule and timeline of scripting and execution
  • Deliverables of automation testing.

4. Test execution.

5. Maintenance.

The last phase of ‘maintenance’ is very key — as if the tests are not maintained, the technical debt is not brought down and those ‘flaky’ tests are not removed; you will actually find that automation testing is taking more time and money investment than manual testing.

How to measure the success of your automation suite?

So, we have automation in place. It is important to track success, and if success is not being met, if you have data to track, you can get back on the right path.

Some automation metrics I would recommend to measure:

  • Percentage of defects found by automation.
  • Time required for automation testing for each and every release cycle.
  • Time taken for a release due to automation testing vs time taken if scripts are manually tested.
  • Customer satisfaction index.
  • Productivity improvement.

Conclusion

Automation is awesome, and it can really add to your QA capabilities, though it must be done and thought out properly.

More Posts

Zero trust in retail
27 March 2024,
by a1qa
4 min read
Fortifying retail security posture: embracing zero trust to protect customer data
Why adopt zero trust in the retail sector? How can companies ensure increased resilience to cyber incidents? Find out the answers in this article.
General
Quality assurance
Advancing QA and software testing processes with AI
14 March 2024,
by a1qa
4 min read
Advancing QA and software testing processes with AI
Uncovering the benefits companies gain when revolutionizing QA practices with the help of AI and tips to implement it.
General
Quality assurance
Enhancing Agile and DevOps processes
28 February 2024,
by a1qa
4 min read
4 actionable tips to enhance Agile and DevOps processes 
Streamlining Agile and DevOps workflows? Learn practical recommendations on how to achieve this.
Agile
General
Test automation
Navigating the future: QA trends that will define 2024. Part 2
30 January 2024,
by a1qa
4 min read
Navigating the future: QA trends that will define 2024. Part 2
We continue exploring QA trends, helping businesses remain competitive in 2024.
Cloud-based testing
Cybersecurity testing
QA trends
Quality assurance
Navigating the future: QA trends that will define 2024. Part 1
15 January 2024,
by a1qa
4 min read
Navigating the future: QA trends that will define 2024. Part 1
Discover topical software testing trends that will shape 2024 and empower companies to smoothly implement advanced technologies.
Agile
QA trends
Quality assurance
Test automation
2023-year-end-recap:-a-journey-through-the-a1qa-milestones
20 December 2023,
by a1qa
4 min read
2023 year-end recap: a journey through the a1qa milestones 
As we bid farewell to 2023, join us in recalling noteworthy achievements and unforgettable moments that have defined this year!
General
Quality assurance
The year in valuable conversations: recapping 2023 a1qa’s roundtables for IT executives 
8 December 2023,
by a1qa
3 min read
The year in valuable conversations: recapping 2023 a1qa’s roundtables for IT executives 
From dissecting novel industry trends to navigating effective ways of enhancing software quality — let’s recall all a1qa’s roundtables. Join us!
Big data testing
Cybersecurity testing
Functional testing
General
Interviews
Performance testing
QA trends
Quality assurance
Test automation
Usability testing
Web app testing
a1qa has been included in the Next-Generation Quality Engineering Services PEAK
29 November 2023,
by a1qa
2 min read
a1qa has been included in the Next-Generation Quality Engineering Services PEAK Matrix® Assessment 2023 by Everest Group
Explore how a1qa secured a proud spot in the prestigious PEAK Matrix® by Everest Group.
General
Quality assurance
na-st-awards-23
16 November 2023,
by a1qa
3 min read
a1qa shines as the finalist in three categories of the North American Software Testing Awards
a1qa is a triple finalist at the North American Software Testing Awards.
General
Quality assurance
Test automation

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.