Blog

To automate, or not to automate: that is the question

It goes without saying that you should choose the right tool to work with while testing process. The challenge although is in absence of good advice: When particular tools work better?
20 February 2014
Test automation
The article by a1qa
a1qa

It goes without saying that you should choose the right tool to work with while testing process. The challenge although is in absence of good advice:

  • When particular tools work better?
  • Will they suit the current projects?
  • Is it worth to risk and try something new?
  • Should you try them if deadline is near?
  • How all this tools would work together?

Most of the articles about automated testing are full of ROI, KPI and other stuff which is sometimes complicated. Often it is not clear, how it would work on your project. If there are any examples on real projects, the information is not enough or not relevant. Still, what I want to cover is how to decide which approach suits you exactly.

Below are some points that help to consider which manual tests should or should not be automated. Just because you can automate something doesn’t necessarily mean that you should. Also it is worth mentioning that development projects usually benefit a lot from test automation, but down in the bottom there is a glimpse on automation for projects that already in production.

So you’ve got a project in development and want to get automated tests running. You have some manual tests and struggling to determine which of them should be automated. Here some basic rules in such kind of a decision.

Tests that should be automated:

  1. Tests that need to be run against every build/release of the application. (Usually it is smoke, regression tests, or sanity tests).
  2. Tests that use the same flow but different data for its inputs for each test. (Programmed once they will use different datasets)
  3. Tests that need to use information from different sources during run time, like databases (SQL/NoSQL) and low-level application attributes.
  4. Tests that can be used for performance testing. (It is especially good running such kind of tests in parallel)
  5. Tests that take a long time to perform and may need to be run during breaks or overnight. (Automating these tests maximizes the use of time)
  6. Tests that involve large volumes of data. (Machines are much more faster in handling such a task)
  7. Tests that need to run against multiple configurations. (Different OS, locales and their combination, different browsers)
  8. Tests during which images and video must be captured or some low-level data retrieved to prove that the application behaved as expected.
  9. Tests that should return results ASAP every run. (Usually automated tests much faster than same tests performed by manual tester).

Tests that should not be automated:

  1. Usability tests (So far there is no way to program ISO/TR 16982:2002 and ISO 9241 efficiently).
  2. Tests that you will only run one-time. (This is a general rule. If test steps could be automated quickly without need for support and run in a loop (for large dataset input for example) it will save a manual tester considerable time and effort).
  3. Test that need to run ASAP. (As automating a test is not the fastest way to run it)
  4. Tests without predictable results or on part of application that changes every once in a while. (For successful validation it needs to have predictable results in order to produce pass and fail conditions)
  5. If a test needs to be manually watched to determine whether the result is correct.
  6. Test that cannot be 100% automated should not be automated at all, unless doing so will save a considerable amount of time. (Large and routine test preconditions)

What other options you’ve got to make a good decision? Remember, that automation can increase productivity by reducing the time taken to perform repetitive tasks. Another useful point is that automated tests cannot get bored. Anyone who has to undertake repetitive tasks over and over again will get bored. This will slow down their performance and increase the risks of missed defects.

Besides, with the help of repeated automated smoke scenarios you could monitor your application availability for 24/7/365. Main difference of the automation approach is that the test checks your app during development process as well as after the release.

And last but not the least, the more often automated test is executed, the more efficient it becomes. So, if you have a project in production, now it is possible to decide whether automation is necessary or not.

The article is prepared by Yan Gabis

More Posts

a1qa-articles
31 January 2023,
by a1qa
5 min read
Best of 2022 by executives: 8 most visited a1qa blog posts
Let’s look back and revisit the most visited a1qa articles of 2022!
Quality assurance
Test automation
qa-trends
12 January 2023,
by a1qa
4 min read
The future of software testing: top 4 impactful trends that will dominate in 2023
Consider the major industry trends for the upcoming year to know how to improve current QA strategies and stay ahead of the curve.
QA trends
Quality assurance
Test automation
test-automation
7 December 2022,
by Dileep Marway
3 min read
Release at pace with test automation: What, why, and how to measure success?
An automation-first approach is key to enhancing testing capabilities and increasing overall operational efficiency. However, I would suggest justifying its implementation, so that it can deliver the promised value.
Quality assurance
Test automation
interview-with-dileep
28 November 2022,
by a1qa
9 min read
Interview with Dileep Marway on a series of articles “Agility and speed: Supercharging your business strategies with QA”
We cooperated with the VP of Engineering and Quality at SHL to present you with a series of his blog posts on: culture of happiness, test automation, and Agile-driven QA. Happy reading!
Agile
Quality assurance
Software lifecycle QA
Test automation
qa-trends-in-telecom
30 September 2022,
by a1qa
5 min read
4 telecom trends for 2023 and how to painlessly implement them with QA
It’s time to explore the telecom trends for the upcoming year. Let’s look at them together and also see the value that QA brings for their smooth deployment.
Cybersecurity testing
Migration testing
QA trends
Quality assurance
Test automation
ai-ml-test-automation-mini
15 September 2022,
by a1qa
4 min read
4 key QA activities to solve test automation challenges via AI and ML
How to address the difficulties caused by implementing test automation? Learn how to do it via applying AI and ML.
Test automation
why-do-bugs-get-missed
31 August 2022,
by a1qa
4 min read
Why do bugs get missed? Learn the problems and tips to avoid them
Still, finding overlooked bugs after the app goes live? Let’s find out why this happens and how to fix it.
Quality assurance
Test automation
30 June 2022,
by a1qa
4 min read
App software testing for telecom: What are the common issues telco providers face?
Facing problems with the quality of your telecom software products? Read more in the article and find out the ways to address them.
Cybersecurity testing
Performance testing
Test automation
Mobile app testing
31 May 2022,
by a1qa
4 min read
Mobile app testing guide: win the race with five-star software
Which aspects of mobile apps to test first to produce a really high-quality product? Find the answer to this and other questions related to mobile app testing in the article.
Cybersecurity testing
Functional testing
Mobile app testing
Performance testing
Test automation
Usability testing

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.