Interview with Mike Urbanovich: How to build a robust test automation strategy?

Today, we’ve discussed with Mike Urbanovich, Head of testing department at a1qa, how to set up a solid test automation strategy.

Mike is responsible for high-level team coordination, projects management, accounts management, and coaching. With 9 years of experience in quality assurance, he supports the range of projects for the Fortune 500 list clients representing diverse industries and coordinates a big team of technical and non-technical specialists.

Let’s get started to find out the benefits that companies receive with automated testing, what tests to automate, how to measure the effectiveness of a QA strategy, and the steps to make to implement test automation smartly.

Question #1. Test automation has been a leading QA trend for many years, and remains that way. Mike, what is so special about it?

Just have a look at the benefits (you know, dozens of them) that test automation brings to companies. Probably, this is the reason.

For instance, while manual QA engineers run only one test at a time, automated testing allows executing multiple tests simultaneously. Here, you aren’t limited to only one device, operating system, or platform. That’s why the coverage is higher, helping verify more features while curtailing testing time.

Or let’s take reduced operational costs as an example. By implementing test automation during long-term projects, companies make more profit while decreasing human efforts and redirecting them to core business processes. Of course, automation requires upfront investments at the initial stages. However, a well-designed test automation framework, a well-configured test environment, and properly selected tools help save a lot of money, time, and effort later on during the maintenance and support phase.

Not to mention accelerated velocity, optimized time spent on repetitive tasks, reusability of tests, and the possibility to conduct testing around the clock with the confirmation of the acceptable level of the system quality.

Question #2. If to speak about the automation strategy, what are the main factors to consider when choosing one?

To put it briefly, speed, budget, and reliability. Just three of them.

For me, the central one is speed, assisting in reacting rapidly to business risks and timely covering them. We are living in such an era where boosted release cadence is the priority, and that is indeed understandable. Companies move forward with their processes and wish everything could happen at lightning speed. QA automation can help ensure that. “Quality at speed,” like they say.

Question #3. But with all its benefits, test automation is not always the one to choose. Could you name some cases when automation is really a must?

In fact, test automation is a perfect match for long-term projects lasting more than 6 months. In this case, it speeds up the whole software testing process exponentially.

BTW, 100% automation is a common misconception. So, I suggest concentrating on which tests to automate to do it smartly.

First, smoke tests. By automating them, you timely detect critical errors, such as bugs in logging, loading an application, creating an account, and so on.

The next candidate for automation is regression testing, helping make sure an app functions as expected after code updates in the existing features.

Performance testing, of course, allows you to realize the maximum possible load, whether the system handles everyday load or overload, or is operable in the long run.

And verifying multi-platform operation. Test automation streamlines rapidly testing the application across all necessary environments, operating systems, browsers, and devices along with running them in parallel.

So, to summarize when automation is indispensable: you want to save time, the project is durable, the tests are repetitive and tedious enough to perform them manually, the tests run for every build of the software.

Question #4. What are the main difficulties that companies face when building a test automation strategy?

I’d say there are 3 most common challenges.

Firstly, an attempt to automate everything. Let’s remember that there are situations when manual testing helps receive more accurate results (we know them — usability, ad-hoc, exploratory testing).

Secondly, selecting inappropriate tools. The choice of test automation tools is like the Galaxy: free and open-source, fee-based, those customized by companies. But the point is not to choose the “buzzy” one, but the one that fits your project the best.

Thirdly, lack of skills of an in-house team. Does your QA experts specialize in a particular testing area, have enough knowledge to perform test automation and become a cross-functional engineer? That happens from time to time but it’s a big work to learn tons of things and upgrade to the next level. Good training, true wish to do that, and motivation will work in a bundle, I’m suppose. Otherwise, the dedicated team of QA automation Jedi is the option to save time and get ready-to-go people.

Question #5. About a test automation strategy. What are your suggestions to build it correctly, maybe some steps to follow to achieve the desired results?

Let’s go through the core steps together.

The initial step is to define the scope of testing — determine WHAT to automate. Automated testing is a win-game option for routine and voluminous tests running endlessly throughout the project.

The second step is to select the tools. Ask yourself, “Do I need a tool for a web or mobile IT solution? Do I have enough QA budget? Does the QA team have the necessary skills and possess the programming languages required to use it? Does the tool allow easily maintaining and reusing scripts? What have I forgotten to ask myself?” (smiles)

For the tools to operate smoothly, make sure they are able to interact with the given environment and platforms (iOS, Windows, Linux, and more).

The third step is to set up the environment — the very stage when the QA team prepares software and hardware to run the tests. To make the environment stable, consider a bunch of components: test data, DBs, OSs, tested IT product, network, and many more.

The following steps are writing, running, and supporting tests, as well as documenting the results.

After tests execution, it’s time to report the outcomes to analyze them and track the progress.

Question #6. What metrics to monitor to measure the effectiveness of test automation?

The list is vast. Test duration metric, I’d say, is the most imperative. It facilitates keeping track of the efforts spent on automated testing and monitoring the team’s productivity.

The percentage of passed and failed tests is another one. The name speaks for itself, but it also allows you to control the number of tests to execute further.

As we always stick to our clients’ needs, I find it crucial to monitor the percentage of covered requirements. This is how we show the ones reached, in progress, and those to manage later.

And the test execution metric verifies the progress of tests run and shows whether the obtained results correspond to the expected ones.

Probably, that’s all.

Mike, thank you so much for this interview!

Reach out to a1qa’s experts in case you need a personal consultation on how to build a test automation strategy.