Implementing AI- and ML-based technologies provides fast decision-making and better optimization of problem-solving processes. According to Forbes Advisor, 64% of businesses anticipate AI will increase productivity and 60% think it will drive sales growth.

Within a new era of automation, AI and ML are seen as inevitable parts of software development that strengthen competitive advantage and assist in achieving more accurate results. The same happens with the QA function, including test automation.

In this post, let’s look at the test automation activities where AI and ML may be helpful and learn the advantages they bring to the table.

AI- and ML-empowered test automation. Which side are you on?

According to Statista, the AI market totaled $134.89 billion in 2022 while in 2027 it will reach $511.33 billion.

Source: Statista

It’s not that surprising, considering AI and ML aid to improve the quality of IT products, speed up the testing process, and…

Ensure faster time to market

While traditional test automation helps deliver software faster, introducing AI and ML to the process allows launching IT solutions even more rapidly within ever-evolving software development methods and constantly increasing end-user demands.

Reduce the number of discrepancies

By implementing smart decision-making and automated control of processes, companies decrease human intervention. This frees up QA efforts allowing QA teams to focus on the core objectives.

Improve flexibility

Within continuously emerging changes in software development approaches, QA processes, and customers’ needs, AI and ML-enabled technologies are more adaptable than people, adjusting all the necessary components and workflows automatically in a matter of seconds.

Increase the overall test coverage

Automated test coverage detection allows writing much more scripts in an hour as well as running them and getting results — all with the optimized scope.

Reinforce codeless automation

Manually writing scripts is time- and labor-consuming, while AI and ML simplify traditional automated testing within the ability to perform codeless checks. Smart test automation tools assess QA risks, update test cases, detect issues, and prioritize tasks.

Enable better code maintenance

As software evolves, new features are added, therefore, existing code is modified, so QA experts have to regularly adapt test suites.

To reduce manual efforts on constant updates of the automation code, they leverage AI that assists in understanding the impact of code changes, identifying which tests are affected, and suggesting necessary adjustments to test scripts.

Amplify test data generation and management

AI allows businesses to create realistic and diverse test data sets, covering a wide range of scenarios, which is especially valuable for complex systems. This helps QA engineers perform comprehensive testing and identify data-related issues early in the SDLC.

AI, ML, are you there to help test automation?

According to the World Quality Report 2022-23, last year, companies benefited from implementing automated testing. 52.7% of respondents improved software quality, 52.1% — perceived better cost control, and 51.7% — accelerated software releases. Of course, we all want more. Here’s what you can do for that.

Activity #1. Introduce smart test scripts

While interacting with the software, AI collects data, makes screenshots, and tests the load. These steps undergo repetitions that help ML technologies learn the expected pattern and compare it to the behavior of the program. When it detects some deviations, the ML algorithms mark it as a potential error. After this, QA specialists manually intervene during the testing process to see if the identified bug is a real problem. The QA experts conduct final verifications to decide what to do next.

Activity #2. Optimize test automation with self-healing AI functions

Self-healing AI tools can easily adapt to changes in the app UI. By running tests, these instruments discover all the elements and occurring activities while recording them and assessing potential QA risks.

When AI- and ML-based algorithms detect some modifications, the tests change automatically. This helps remove threats before they occur while distinguishing whether the program’s behavior is normal or abnormal, triggering recovery activities when the software has deviations.

It’s important to mention that AI-based projects provide the whole scope of benefits only in the long-term perspective. The more errors appear for ML, then the more efficient and trusted further performance will be.

Activity #3. Conduct GUI test automation with ML

With ML, testing of the graphic user interface has entered the next level. Now, the old-school process of manually checking UI elements in accordance with mock-ups is automated within ML-powered algorithms.

Image-based testing is becoming the mainstream as ML-powered algorithms recognize different patterns and perform visual verification even on various devices and their configurations.

Given that 81% of consumers are ready to pay more for better UI, there is no room for errors.

Thus, ML-based technologies simplify automated GUI testing and help deliver reliable and user-friendly software.

Activity #4. Leverage AI for API testing

Due to the complexity and volume of API endpoints in modern software, API testing often poses challenges while being a time-consuming task.

With AI at the core, companies can simulate complex user interactions with APIs, identify patterns in API behavior, and predict potential issues based on historical data. They expedite QA processes, streamline the automation of API tests, and boost test case generation.

Activity #5. Optimize regression testing

As applications evolve, it becomes challenging to ensure that new code don’t introduce defects into existing functionality. Though test automation aids to speed up regression processes significantly, AI brings even more value to the table.

AI employs data-driven insights to analyze past test results, code changes, and defect histories to intelligently recommend which tests to run, examine high-impact areas, and uncover critical issues. This approach allows organizations to enhance the efficiency of regression testing, resulting in faster feedback and more stable software releases.

Activity #6. Adopt automated monitoring

Automation has already reduced human intervention and accelerated testing processes compared to those performed manually. AI- and ML-based technologies speed up workflows twice as fast.

Machines are powerless when performing tasks that require human monitoring and decision-making. To ensure software and hardware operate with no downtimes and pauses for controlling, checking, and analyzing, forward-looking companies introduce AI-and ML-enabled algorithms while optimizing routine actions and shortening the delivery time.

Closing remark

AI and ML can be used as part of the QA process and will never replace some manual activities or test automation-related checks. So, for some work, especially mentioned above, AI and ML might be introduced to simplify QA processes.

On the other hand, while introducing them, it’s important to calculate which efforts on QA will be saved, and which of them will become extra for other teams.

With total optimization of operational and business processes, AI- and ML-powered technologies assist in alleviating test automation workflows and minimizing human intervention. By introducing smart test scripts, ML-based monitoring, self-healing AI functions, image-based GUI test automation as well as leveraging AI for regression and API testing, companies are taking leading positions in the market.

Reach out to a1qa’s experts to get support on implementing test automation to enhance your software quality and release error-free and secure software in production.

Today, we’ve discussed with Mike Urbanovich, Head of testing department at a1qa, how to set up a solid test automation strategy.

Mike is responsible for high-level team coordination, projects management, accounts management, and coaching. With 9 years of experience in quality assurance, he supports the range of projects for the Fortune 500 list clients representing diverse industries and coordinates a big team of technical and non-technical specialists.

Let’s get started to find out the benefits that companies receive with automated testing, what tests to automate, how to measure the effectiveness of a QA strategy, and the steps to make to implement test automation smartly.

Question #1. Test automation has been a leading QA trend for many years, and remains that way. Mike, what is so special about it?

Just have a look at the benefits (you know, dozens of them) that test automation brings to companies. Probably, this is the reason.

For instance, while manual QA engineers run only one test at a time, automated testing allows executing multiple tests simultaneously. Here, you aren’t limited to only one device, operating system, or platform. That’s why the coverage is higher, helping verify more features while curtailing testing time.

Or let’s take reduced operational costs as an example. By implementing test automation during long-term projects, companies make more profit while decreasing human efforts and redirecting them to core business processes. Of course, automation requires upfront investments at the initial stages. However, a well-designed test automation framework, a well-configured test environment, and properly selected tools help save a lot of money, time, and effort later on during the maintenance and support phase.

Not to mention accelerated velocity, optimized time spent on repetitive tasks, reusability of tests, and the possibility to conduct testing around the clock with the confirmation of the acceptable level of the system quality.

Question #2. If to speak about the automation strategy, what are the main factors to consider when choosing one?

To put it briefly, speed, budget, and reliability. Just three of them.

For me, the central one is speed, assisting in reacting rapidly to business risks and timely covering them. We are living in such an era where boosted release cadence is the priority, and that is indeed understandable. Companies move forward with their processes and wish everything could happen at lightning speed. QA automation can help ensure that. “Quality at speed,” like they say.

Question #3. But with all its benefits, test automation is not always the one to choose. Could you name some cases when automation is really a must?

In fact, test automation is a perfect match for long-term projects lasting more than 6 months. In this case, it speeds up the whole software testing process exponentially.

BTW, 100% automation is a common misconception. So, I suggest concentrating on which tests to automate to do it smartly.

First, smoke tests. By automating them, you timely detect critical errors, such as bugs in logging, loading an application, creating an account, and so on.

The next candidate for automation is regression testing, helping make sure an app functions as expected after code updates in the existing features.

Performance testing, of course, allows you to realize the maximum possible load, whether the system handles everyday load or overload, or is operable in the long run.

And verifying multi-platform operation. Test automation streamlines rapidly testing the application across all necessary environments, operating systems, browsers, and devices along with running them in parallel.

So, to summarize when automation is indispensable: you want to save time, the project is durable, the tests are repetitive and tedious enough to perform them manually, the tests run for every build of the software.

Question #4. What are the main difficulties that companies face when building a test automation strategy?

I’d say there are 3 most common challenges.

Firstly, an attempt to automate everything. Let’s remember that there are situations when manual testing helps receive more accurate results (we know them — usability, ad-hoc, exploratory testing).

Secondly, selecting inappropriate tools. The choice of test automation tools is like the Galaxy: free and open-source, fee-based, those customized by companies. But the point is not to choose the “buzzy” one, but the one that fits your project the best.

Thirdly, lack of skills of an in-house team. Does your QA experts specialize in a particular testing area, have enough knowledge to perform test automation and become a cross-functional engineer? That happens from time to time but it’s a big work to learn tons of things and upgrade to the next level. Good training, true wish to do that, and motivation will work in a bundle, I’m suppose. Otherwise, the dedicated team of QA automation Jedi is the option to save time and get ready-to-go people.

Question #5. About a test automation strategy. What are your suggestions to build it correctly, maybe some steps to follow to achieve the desired results?

Let’s go through the core steps together.

The initial step is to define the scope of testing — determine WHAT to automate. Automated testing is a win-game option for routine and voluminous tests running endlessly throughout the project.

The second step is to select the tools. Ask yourself, “Do I need a tool for a web or mobile IT solution? Do I have enough QA budget? Does the QA team have the necessary skills and possess the programming languages required to use it? Does the tool allow easily maintaining and reusing scripts? What have I forgotten to ask myself?” (smiles)

For the tools to operate smoothly, make sure they are able to interact with the given environment and platforms (iOS, Windows, Linux, and more).

The third step is to set up the environment — the very stage when the QA team prepares software and hardware to run the tests. To make the environment stable, consider a bunch of components: test data, DBs, OSs, tested IT product, network, and many more.

The following steps are writing, running, and supporting tests, as well as documenting the results.

After tests execution, it’s time to report the outcomes to analyze them and track the progress.

Question #6. What metrics to monitor to measure the effectiveness of test automation?

The list is vast. Test duration metric, I’d say, is the most imperative. It facilitates keeping track of the efforts spent on automated testing and monitoring the team’s productivity.

The percentage of passed and failed tests is another one. The name speaks for itself, but it also allows you to control the number of tests to execute further.

As we always stick to our clients’ needs, I find it crucial to monitor the percentage of covered requirements. This is how we show the ones reached, in progress, and those to manage later.

And the test execution metric verifies the progress of tests run and shows whether the obtained results correspond to the expected ones.

Probably, that’s all.

Mike, thank you so much for this interview!

Reach out to a1qa’s experts in case you need a personal consultation on how to build a test automation strategy.

What have we got? Automated testing speeds up QA processes, accelerates delivery time, and reduces the number of repetitive tasks. What’s the issue? The common challenge is that the ROI is still poor.

According to the World Quality Report (the WQR) 2021-22, only 50% of surveyed companies received ROI from adopting test automation. Oh yes, maintaining automated tests can be pretty much expensive.

To save effort and money on automated testing and keep it at a high level, companies go scriptless.

In the article, let’s disprove 3 misconceptions about codeless test automation to reveal its true nature and talk about the main advantages.

3 benefits of going scriptless

In 2020, the test automation market size exceeded $15 billion, and it’s going to grow over 16% by 2027. That’s clear: companies adopt test automation to reduce QA costs, provide better control and transparency of QA activities, increase test coverage, and so on.

But the recent WQR shows that these days, to obtain more benefits, some businesses introduce advanced test automation techniques, scriptless test automation being among them (48%). However, before completely switching to codeless automated testing, companies may introduce low-code tools as they are easier to deal with, especially in the early SDLC stages. Scriptless test automation is a good solution when the software is stable and doesn’t require a plethora of changes.

Source: World Quality Report 2021-22

Is codeless test automation becoming a new trend nowadays? We’ll see, and now, let’s take a look at the major benefits that it brings to companies.

1. Speeding up test automation activities

According to the WQR 2021-22, the reduction of test cycle time reached 52% in 2021, but with codeless test automation tools, businesses achieve even better results.

Just like code-based test automation, scriptless one allows starting testing at the early SDLC stages, helping identify and fix bugs earlier while accelerating go-to-market time. When implementing no-code test automation tools, it’s possible to avoid writing numerous test scripts by using ready-made prototypes and wireframes.

2. Increasing ROI

Conventional script-based automated testing is cost-effective in the long run. However, it involves some challenges, such as investing in staff training, maintaining tests, purchasing cutting-edge test automation tools, etc. A good budget to have. By swiftly shifting to codeless test automation, expenses are getting really low.

Almost any QA expert, let’s say a manual engineer without deep knowledge of scripting, can use no-code automation to perform software testing. QA automation specialists, in their turn, focus on maintaining and updating tests, conducting other mission-critical types of testing, such as white box, unit, integration testing, etc. They also contribute to designing comprehensive test suites for further scriptless automated testing.

3. Simplifying tests maintenance

Tests maintenance is the very stage of errors occurring most often. But in most cases, tests aren’t reusable.

To address this issue, companies adopt no-code test automation, which traces every repeated script component and provides object-oriented test automation. Therefore, the process of managing and maintaining tests is much easier and budget-friendly, leading to higher ROI.

By implementing scriptless test automation, companies bring true value to their businesses while accelerating the velocity, enhancing ROI, facilitating tests maintenance. Along with the core benefits, no-code automated testing helps eliminate the gap between technical and non-technical experts, such as business analysts, stakeholders, and the client.

Moving to scriptless test automation assists non-technical specialists in diving deep into the project activities while tracking the progress of ongoing QA activities. As a result, it contributes to finding the best solutions, helping the client meet desired business objectives.

Dispelling 3 misconceptions about scriptless test automation

Before replacing script-based test automation with the codeless one, it’s necessary to understand some vital facts about it. Let’s shed light on 3 widespread delusions about it and see how it works in reality.

Misconception #1. No script

A popular belief says that automating test cases with a no-code approach doesn’t need scripting and programming skills. The reality is a bit different. To create a test automation framework, QA specialists need to possess coding and logical skills. And when the framework, the tools, and the tests are completely ready, we talk about their scriptless usage and maintenance.

Moreover, to integrate a new functionality into the existing framework, QA experts have to script it. That’s why scriptless test automation isn’t 100% codeless. In fact, it means less scripting.

Misconception #2. High expenses

Scriptless test automation is a relatively new notion in the IT world. Some organizations believe that it’s extremely expensive. However, the truth is: by implementing scriptless test automation early in the SDLC, the money and effort invested in it pays off and helps increase the ROI.

The companies don’t allocate their budget on hiring third-party specialists, possessing programming skills, being able to write test automation scripts and maintain them. Or businesses save financial resources needed to requalify manual specialists into cross-functional ones. As mentioned in the chapter above, a manual QA engineer can handle no-code tools.

Misconception #3. Unreliability and complicated maintenance

To deliver first-rate no-code test automation tools, developers do their best: analyze various components, including users’ scenarios, deployment environments, types of testing to conduct, etc. to ensure proper interaction of tools with other elements, their smooth functioning, and high reliability.

When designing conventional test automation scripts, the QA team doesn’t consider such factors as reusability and traceability, which leads to a bunch of errors during the maintenance phase. However, these elements are vital for no-code test automation, enhancing its credibility. So, scripts are easy to maintain because test automation experts focus on identifying all repeated components that help create new test cases and reuse them in the future.

In a nutshell

Compared to conventional test automation, scriptless one helps speed up QA activities, increase ROI, eliminate the gap between technical and non-technical experts, simplify test maintenance.

And now, when the misconceptions — no script, high expenses, unreliability — have been disproven, companies can implement codeless test automation with confidence.

Reach out to a1qa’s specialists to discuss how to enhance ROI by designing a smart test automation strategy.

Three words. Quality at speed. That’s what companies look for when there is a need to maintain the pace of constantly updating software and remain competitive.

To achieve that, we have 2 words. Test automation helping to ensure high software quality faster and meet customers’ needs that change as quickly as the apps themselves.

According to the World Quality Report 2021-22, IT executives automate only 15% and 20% of all tests (providing them with better KPIs, of course).

But why not 50% or 70% of all tests?

In the article, learn when companies apply automated testing, how it helps keep pace with ever-evolving IT solutions, and what activities are the core to fine-tune test automation QA processes.

Why do forward-looking companies choose test automation?

Today’s lightning-fast rhythm of life requires rapid decision-making and the same “faster-faster-faster!” pace of delivering IT products to the market. It’s okay.

To keep up with this rush, companies introduce test automation.

• Firstly, when it comes to large-scale applications with a huge number of functions that are tested regularly.
• Secondly, test automation is essential during long-term projects with frequent changes.
• And finally, automated testing is vital when releasing software often.

However, to reach the desired outcomes within the test automation framework, it is necessary to begin by analyzing business objectives, budget, allocated time, and the software itself well in advance. Then the organization has a better chance to build an effective QA strategy with test automation being the focal point.

According to the World Quality Report 2021-22, surveyed companies with test automation at the core of their business strategies received a range of competitive advantages, including enhanced control and transparency of test activities, reduced test cycle time, decreased QA costs, and others.

Source: World Quality Report 2020-21

Another interesting question is, “Why do companies need to change their applications so often?” The problem is that customers’ behavior isn’t permanent. To adjust to all end-user needs, organizations look for the strategies that allow making modifications as well as testing them in the shortest possible time without compromising the quality.

3 core test automation activities to release high-quality software

Finding the right approach to applying automated testing is the first step to achieving desired business outcomes. Introducing key test automation activities helps boost efficiency and set up QA processes.

1. Striking a balance between manual and automated testing

Switching completely to test automation is sometimes not the only way out, and for particular projects, 100% automation is just a widespread myth.

For example, many aspects of UX design require QA engineers to test them manually to understand how easy it is for end users to register on the website. Here, implementing test automation is ineffective, as the main focus isn’t on the speed but rather on the product usability.

However, there are types of tests that are better to automate to accelerate the testing process. Such tests include:

• Repetitive tests running on every build or release (e.g., smoke, regression)
• Tests on different configurations (e.g., multiple operating systems or browsers)
• Tests that use the same script but various or dynamic data for each run (e.g., data-driven tests)
• Tests that can be conducted at night (e.g., time-consuming tests)
• And many more.

According to State of Testing Report 2021, companies automate functional and regression testing (75%), unit testing (43%), load and stress testing (43%).

Thus, to build an effective QA process, a company needs to find the golden mean — wisely implement test automation and complement it by manual testing to obtain quality software.

2. Choosing the right test automation tools

Building a solid test automation process involves multiple factors to bear in mind, and selecting the appropriate test automation tools is one of them.

When the company has a clear plan for the project with defined business goals, the next step is to choose the correct instruments to do the magic!

As the pool of tools is ample, it’s better to start with analyzing critical components:

• Platforms — discover whether the tool supports various platforms (web and mobile apps, desktop browsers, databases) or run only at a particular one.
• Programming languages — make sure that the tool supports languages the QA team is going to use and matches the development stack.
• Budget — find out in advance what costs the instrument entails being ready for additional expenses to support it in the future.
• Project requirements — understanding the list of requirements and the problem that QA engineers need to fix with the software to opt for the most suitable tool.

In case your QA team is strong enough, why not design its test automation framework and create a custom tool to fine-tune and boost the efficiency of QA processes.

3. Polishing DevOps processes with test automation at the core

According to State of DevOps Report 2021, 42% of companies apply test automation within DevOps which provides teams with confidence in mitigated QA risks, reduced downtime, and facilitated pace of changes.

As DevOps favors continuous integration and continuous development, there is a strong need for continuous testing. Being a time-consuming and tedious process, continuous testing can’t dispense with test automation.

Thus, automated testing helps speed up QA processes, increase project flexibility, and gain more benefits within DevOps practices:

• Accelerating velocity without sacrificing IT solution quality
• Enhancing reliability via complete test automation coverage
• Improving user satisfaction through continuous software updates based on their feedback.

A well-designed test automation framework in DevOps contributes to fixing bugs at the initial SDLC stages resulting in delivering a faultless product to the IT market.

Bottom line

The IT world is full of changes, the same is true for software development and customers’ behavior patterns. To respond to the global uncertainty and meet constant modifications of IT products with confidence, companies rely on test automation.

Alongside maintaining business continuity, it speeds up time to market, reduces QA costs, QA risks, etc. However, to benefit even more, introduce core test automation activities to fine-tune QA processes — find the balance between manual and automated testing, apply the right test automation tools, and set up DevOps processes within the project.

In case you want to implement a robust test automation strategy, reach out to a1qa’s experts to get professional support.

Having become a buzzword in the global context, test automation today is an integral part of Agile and DevOps culture. McKinsey states that as of August 2020, 41% of representatives have successfully applied automation across their entire enterprise or fully automated processes within at least one function.

However, these efforts are no longer sufficient due to the boosted development pace and an increased difficulty to withstand the competition. Maintaining business as usual and keeping the risks of automation from blocking timely releases at bay, companies not only shift left but introduce in-sprint automation to improve process reliability and cut release cycles.

In the article, we’ve touched upon its value for a business, outlined the associated challenges of its introduction, and assembled hands-on tips to achieve it with ease.

In-sprint automation: what’s in it for your business?

We all know how time- and effort-consuming regression testing may be, sometimes leaving no room for implementing innovation to obtain a competitive edge.

Completing test automation within the limits of one sprint helps companies significantly fast-track these repetitive activities, maximize test coverage, detect issues earlier in the development life cycle, ease iterations, and obtain a clear overview of the product quality.

Embarking on an in-sprint automation journey allows businesses to test continuously at the speed of DevOps while providing enhanced collaboration opportunities between the core teams ― development, automation, and QA ― enabling them to follow the same pace.

Challenges associated with introducing in-sprint automation

The adoption of in-sprint automation has its specific problems:

• Short sprints provoking the lack of time to automate. In general, the QA team must fix spotted defects within a sprint that may last only two weeks. Which can be insufficient to perform automation, as it requires the collaboration of both automation engineers and developers.
• The choice of suitable tools or frameworks. The toolkit should enable the design and initial creation of tests (logic description, business logic, a stable part of the requirements/application) even without accessing the software UI, which is often subject to changes. Such an approach allows starting automation of API and unit tests without wasting time. Thus, the frameworks should support a myriad of tests, as well as flexibility and allowing any changes introduced by evolving business demands.
• The shortage of the necessary technical expertise. Automation is a tricky activity that calls for deep technical expertise from all the parties involved. Therefore, companies should engage solution architects and establish close interaction between them and engineering teams ensuring smooth design, configuration, and implementation of in-sprint automation.

Top 5 tips to seamlessly attain in-sprint automation

Only 1% of the World Quality Report’s 2020-2021 respondents stated that they confronted no challenges in applying automated testing in Agile. Getting things done in such a small period is only possible when applying a complex strategy.

Below, we’ve assembled the most essential QA tips to help companies achieve faster time to market and exceptional software quality through in-sprint test automation.

Tip 1 – Make the QA team closely collaborate with the business and the Dev team

In-sprint automation presupposes that it’s impossible to automate the entire scope. The choice of the relevant tests should be made after discussing and providing in-depth analysis with project stakeholders who give valuable data on how often would-be clients will use particular features.

Constant and early interaction with the Scrum development team is another vital aspect to streamline testing activities. For instance, by previously agreeing with the software engineers upon the identifiers of UI elements or API methods, the QA experts can start creating automated tests even before writing the code of a particular feature. Developers work carefully on the discussed functionality bearing in mind that they need to create it in a way that allows further automation.

Tip 2 – Write test cases in a simple but meaningful manner

Voluminous test cases with the overcomplicated flow may not be an option when it comes to automating within the same sprint. By checking only one part of functionality at a time using precise steps and clearly defined expected results, the QA engineers decrease the efforts required for their preparation and concentrate more on the automation itself.

Tip 3 – Apply combined test automation approach

Determining a mixture of layers for executing test scenarios considerably eases the testing process, reducing the risk of errors and hastening the speed of running verifications. By leveraging API, the QA engineers access the application UI quicker, which allows avoiding prerequisite steps in the automated scenario and scaling down the time for executing automated tests.

Tip 4 – Include automation in the definition of done (DOD)

By embracing automation as a full-fledged part of DOD, the entire Scrum team focuses on both its great significance and reflects on the emerging bottlenecks or techniques that went well to constantly work on process improvements and attain the desired objective.

Tip 5 – Stick to test-/behavior-driven development approaches (TDD/BDD)

The utilization of such practices enables high overall responsibility for quality within your Agile or DevOps teams. Writing unit (TDD) or acceptance and later unit (BDD) tests before the functional code with their automation following allows for shifting QA activities left, increasing test coverage, detecting issues earlier in the development process, and safe refactoring.

Closing remark

Proficiency takes time. Things that seem overly complicated at first, later become easy to perform. In-sprint automation is among them.

To attain this mission-critical objective, businesses may design a profound automation strategy focusing on close cooperation between the Dev and QA teams, writing simple but descriptive tests, automating wisely, advancing DOD, and adhering to TDD/BDD.

Altogether, these steps contribute to hastening release velocity, spotting issues earlier in the development cycle, and following a quality mindset to help organizations get the extra mile and obtain continuous improvement.

Reach out to a1qa’s experts to get professional help in attaining in-sprint automation.

If you’re finding it challenging to achieve the desired level of test automation, rest assured, you’re not alone. Here at a1qa, we often hear this described as a pain point or challenges to companies of all sizes.

Despite that, it breaks all records of being one of the longest-running QA trends. Over the years, companies that implement test automation have experienced added value when doing so wisely, and are now reaping the benefits of QA.

The popularity of automated testing has accelerated particularly high growth in recent years due to the global response to the spreading of COVID-19. Ensuring productivity and optimizing processes with minimal casualties has quickly become the top priority.

In this blog post, we’ll focus on the test automation side of things, answering the question, “Is test automation worth starting?”

Test automation in Agile

With a value of $12.6 billion in 2019, the global automation testing market is expected to be worth $28.8 billion by 2024. Experts say that this growth is driven by the increasing adoption of Agile and DevOps methodology in the software development lifecycle.

By adopting iterative and incremental ways of working, an agile environment and test automation go together like cookies and milk, reinforcing each other. The latter soon becomes an essential strategy for businesses to maximize the speed and efficiency of their software delivery.

The key question here is, would you prefer to think in the long-term perspective and build a long-term solution or solve the urgent QA-related problem right away?

Under tight deadlines, the obvious solution is to apply a one-off strategy. But what if these strict timelines happen all too often, and operational tasks are also continuing? Will it ever be the perfect time to start creating a long-run QA solution that can pay off in perspective (cheap, fast, and simple) and eliminate the human factor once and for all?

There is no easy answer, although strategic development requires pushing the boundaries and making cost-effective decisions to succeed in the future.

Test automation is one way to speed up, evolve, and keep pace and quality with the modern digital users’ life, helping to deliver the software products at the needed time with the required technical health.

56% of the latest World Quality Report respondents noted a reduction of test costs achieved through test automation, meaning that applying this effective framework helps gain better ROI by performing QA processes faster.

To adopt automated testing at an appropriate level, focus on the age-old indicator of total cost of ownership, and realize when it can save the day without becoming a burden.

Embracing benefits and removing difficulties

Integrating test automation into the overall development lifecycle is a great way to start. To provide the maximum benefit from the software, adherence to the shift-left approach is one of the best ways to decrease risk and eliminate the number of costly bugs from the get-go.

Once the test scenario is automated, no human involvement is required. This eases manual efforts, each test automation engineer can run them at any time, and in case of amendments in the application business logic, make minor updates within the autotests. By performing more cases and eliminating the possibility of human error, one optimizes test coverage and reduces the time required to get feedback on the software quality.

Test automation is not only about replacing manual tests, but also an opportunity to take QA to the upper level and provide better results within DevOps and Agile ecosystems. At a1qa, it has reduced the total testing time by 20 times in one of our projects.

No doubt, test automation requires deep knowledge, appropriate tools, and a new mindset that means quality should start in the early phases of SDLC. How do you get the most from the adoption of test automation?

• At the preparatory stages of test automation implementation, define the core business goals (the fewer, the better) to ensure your priorities are right and correspond to the business playbook.
• Choose the toolset depending on the pros and cons of each instrument. This will help you pick the best of the wide range available, from test automation and CI tools to instruments for coding and creating quality reports.
• When moving forward, leave any fairy-tale expectations in the past. Test automation is made to bring results, but the effects of this won’t always be immediate. It’s worth mentioning that it can be applied if the software product operates smoothly while the functionality is checked on a regular basis. Create a solution that can help you track KPIs when heading towards your main goals and think of test automation as a broad space, not a capability.
• Test automation experts should delve deep into the tested app, the latest trends, tools, and solutions to know their industry inside out and improve the expected results.

Automated testing in DevOps

Businesses are measuring continuous testing (CT) effectiveness via user feedback and business KPIs. Received feedback early and often, allows making code changes before the software release occurs.

Test automation is a stand-alone discipline as well as at the heart of CT, meaning it’s vital to perform checks constantly throughout the product lifecycle and the entire continuous delivery (CD) process. Usually, it starts early in the pipeline and occurs at every stage of the development.

Though, in some cases, it can be conducted in parallel or before the development activities. Autotests are designed when creating the feature, so it’s easier to evaluate its quality after the development process has been completed.

We can achieve that for two reasons:

1. By the end of creating a feature, the test scenario has already been run.
2. The autotest is launched after the developer has implemented the feature.

With CT, automation integrates smoothly in the CI/CD pipelines and helps precisely assess business risks with each potential release version, delivering the product at the right time with no critical bugs.

Bottom line

Faster releases. Higher quality of the product. To achieve these goals and get a forward-looking QA solution, businesses are doing their utmost to adopt test automation as early as possible.

A promising discipline, based on an insight-driven approach to its implementation, test automation can help you to get a high ROI without re-doing QA activities.

Are you re-thinking your business scenario to optimize processes and stay competitive in a post-pandemic world? Ask our experts about how to implement automated testing wisely and get an increase in software quality in 6 months.

Today, software testing is an integral part of the development process. To reduce the time spent on testing, many companies opt for test automation. However, automation capabilities move beyond speed increase and include test coverage enhancement and overall QA costs optimization in the long run.

Moreover, test automation is a well-tailored process with defined stages. In this article, we will talk about the importance of this service and will highlight the steps to apply it correctly.

• Determine the scope

Any process starts with the definition. Therefore, before implementing test automation, you should determine the automation scope. When starting tests development, a QA engineer should first define the order according to the tests’ priority rate.

• Prepare to automate

Having quite profound experience in test automation, I can say the following: automated tests should cover the most stable part of the functionality and the one that is tested for about 3-4 times per week.

As a rule, smoke tests (or other regression tests) are chosen for that very purpose.

• Select tools for automation

As soon as the scope is defined, a QA engineer should select test automation tools. The tested interfaces define the package of applied tools. Different types of interfaces presuppose different tools’ range.

There is no any one-size-fits-all solution. The choice of a test automation tool will depend on the technology the software is built on. For example, QTP does not support Informatica. That means the tool cannot be used for the software. We prefer the most reliable and proven solutions: Selenium WebDriver, Coded UI, Ranorex, TestStack.White, Appium, Xamarin, and many more.

Having decided upon the tools, testers get to framework implementation.

• Develop the framework

The framework is the basis for further automated tests’ development. It provides an opportunity to optimize test development efforts by re-using the code. You can utilize any of the ready-to-use frameworks presented on the market, like the Robot framework for Selenium.

The usefulness of frameworks is hard to underestimate. Thanks to frameworks, it’s possible to maintain consistency of testing and improve re-usability. You can also count on minimizing code usage and improving the structure of tests, which is a perfect scenario for long-term projects.

• Configure the environment

All the tests run in the environment, which is to be well-configured. Upon this step, you should create and support the environment to successfully run automated tests and store the results.

Test automation will require test data, which means you are to prepare the set of files and test accounts beforehand. Otherwise, you tackle risks that may damage the process and provide you with irrelevant test results.

• Start to automate

Eventually, when all the preliminary preparations are done, testers can get down to automated test development. A regular process of providing new automated tests includes the following points:

1. Selection of the manual test case according to the stated priorities
2. Code writing for the automated test
3. Adding the automated test to the debug test execution
4. Adding the automated test to the test execution for newly created builds.

As well as analysis, the tests need support, which presupposes the process of updating automated tests along with the updates of the interface and/or business logic.

When the tests are launched, they should be monitored. You cannot let them go along without tracking the process. While monitoring the automated test, remember to take into consideration the following aspects:

1. Automated test coverage, cost per test
2. Useful vs. irrelevant results after test execution
3. Cost per test
4. The scope of support in comparison with the number of executed tests
5. Economic effect (ROI, return on investment).

Tracking the process, keep in mind, that test automation is much more than computers launching test programs. It is also delivering information about quality.

Ultimately, mastering automation can significantly increase your business value. Automation includes many factors that need to be understood and addressed before the start of the process. Follow the main points to avoid the risks and get your benefits.

Alternatively, you can entrust this process to experts in the field of automation. Contact us to get advice on integrating automation into the testing processes of your software product.

In this blog post, we want to continue the conversation about the main aspects of the test automation service. Let us move on discussing how AI can help make automation more effective and address the common myths alongside with the business benefits one can expect at the end of this journey.

Common test automation misconceptions

Many people consider the automation of tests as some separate area of ​​knowledge not related to testing at all.

Performing automation with wrong assumptions can provide poor software results. This, in turn, can lead to more misconceptions. So let’s address some best-known myths to be able to distinguish the truth and lies.

• Test automation is always more efficient than manual testing.

In fact, there’s no winner in the automation vs. manual testing debate. Each technique has its own pros and cons.

Manual testing plays an important role especially at the initial stage of development when automation cannot yet be achieved due to partial project implementation. On the other hand, when you need to conduct testing very swiftly (hotfix checks), manual testing can also take a leading part because automation cannot be achieved in a too short period.

Alternatively, if the requirements are constantly changing, it is also difficult to implement or apply automation to all requirements – this is where manual testing re-emerges as a major actor of this testing play.

In the end, both test automation and manual testing can co-exist, especially if it’s unsustainable to rely just on manual testing (for instance, when working with a huge amount of data, the human factor should be eliminated).

• Automation completely replaces human interaction.

Manual testing is best suited for tests where you need a person’s opinion, like usability tests and exploratory checks. Automation is better applicable to repeated tests, tests with a large amount of data, or tests that need to be run very often (for example, regression tests). Automated testing allows you to reduce the cost of manual testing, but only to a certain extent, by automating tasks that were mentioned earlier.

In any case, the automation solution demands regular support and a well-considered approach. Manual testing requires such skills as knowledge, experience, logical and analytical skills, creativity and other aspects that can be covered only by a human, not a machine. These skills cannot be replaced by automation.

• With automation, it’s possible to achieve full test coverage.

Yes, by means of thorough testing, QA engineers can improve test coverage. Nevertheless, due to the changing requirements (and it’s a normal thing!), there is no such a practical way to get this goal done.

Instead of aiming at reaching 100% coverage, it’s better to pay more attention to the most critical areas of functionality and conduct automation more wisely. Sometimes one can spend a lot of money on automating what doesn’t make sense to automate at all, so the most intelligent approach is highly appreciated.

• Test automation can help catch all bugs.
  

Automation scripts check what they’re programmed to check. All tests can be successful, still, some defects will remain unnoticed, which may give a false impression on the quality of the product itself. That said, automation can prove the presence of a defect, but cannot prove its absence instead.

Also, as it was mentioned in the previous part of the article, after adding new functionality to the existing one, automated tests can help make sure that the current functionality hasn’t broken (and provide these results very quickly). However, the number of regression defects is a little less, than the number of defects found in newly developed functionality.

As it was stated in the WQR 2018-2019, bringing together AI, ML, and analytics and their active use in conducting smarter automation will change QA and testing in the coming years.

With this in mind, let us discuss how one of the main QA trends to follow in 2019 – artificial intelligence – is shaking the test automation process.

Effect of AI on test automation

In the most general, artificial intelligence is a technology that replicates human-style intelligence processes by means of computer systems. This terminology is known as one of the hottest buzzwords in the software sphere (maybe after Agile and DevOps).

True AI empowers the QA teams to be more productive, as their focus is shifted away from repeatable tasks toward creativity and innovation.

How does AI can drive automated testing?

Without a shred of modesty, it is changing all main aspects of automation by enhancing

• Creation of tests (making this process fully autonomous using AI technologies).

AI automatically locates and identifies hundreds of selectors, analyzes the DOM state, provides page object recognition, and more.

• Test execution (even complex tests can take minutes to execute from everywhere around the Earth, from any device, with any bandwidth, and in all types of environments)
• Maintenance (AI can help by the means of self-correct/-healing tests, computer vision diagnosis, and others).

AI-powered testing also transforms businesses resulting in creating more stable suits, releasing software faster to meet end-user demands, and greater product outcomes.

When we bring AI technology and test automation together, the testing capabilities become infinitely larger. According to the WQR, 45% of respondents use AI for intelligent automation in testing.

The survey points out that one of the keys to making test automation more effective is to implement AI in automation. The engineers need to decide what to automate, what not to automate, what to automate first relying on principles of smart test automation. This is where the smartness or the use of predictive analytics re-emerge.

There is also an increasing focus on intelligent automation, which is expected to accelerate speed-to-market and eliminate human errors.

Possible business benefits

Involved right, test automation can bring back large amounts of effort and time invested. Nevertheless, business benefits are not limited by these advantages.

The WQR showed that the respondents give a higher weighting to test automation comparing with the previous years. They have highlighted the following advantages:

Summing up

Test automation goes a long way toward helping ensure a high standard of quality at all points of development and QA processes.

The trends in this sphere-like predictive analytics, RPA, are changing rapidly, and we believe automated testing to become smarter and even more popular.

Creating a successful test automation strategy and adopting this service properly can help take full advantages of test automation potential and change your business to the best.

Request a free consultation with the a1qa engineers and find out if you need to increase the focus on test automation to solve quality challenges.

Within the last 15 years, the role of test automation and its purposes in IT have significantly extended. From focusing on shortening testing times, now it strives to reach better coverage and more effective usage of test cases.

This can be narrowed to one concept – the so-called “quality at speed.”

Although, the level of adopting test automation is extremely low: in conformity with the World Quality Report, 61% of the interviewed IT specialists emphasized they have faced difficulties automating their QA activities.

The reason is clear – the applications modify too often and too much with every release. Notwithstanding, this proves their endeavor to build a robust and smoothly working software test automation product.

This is basically pure truth. In this article, we will define whether test automation is a “silver bullet” for IT business or not.

Test automation basics

As time goes by, test automation is focused on becoming smarter and more effective, as the confluence of AI, ML, analytics provides the systematic change in the benefits it can deliver.

To achieve valuable results of the projects, companies should consider that automated testing is not about replacing manual testing or getting a supplementary reduction in costs. In the most general sense, it is aimed to deliver quality at speed, take QA to the upper level, and make DevOps and Agile provide greater results.

Broadly speaking, we can divide test automation goals into 3 items:

• Process-specific goals (faster time to market)
• High-level goals (LTV – lifetime value)
• Revenue-based goals (costs).

When considering the targets more thoroughly, we’ll say it helps achieve the following local SDLC values:

The problem to get the utmost of test automation lies not only in implementing the service but also in adopting it successfully. To build the process properly, one has to delineate when to automate and when it won’t work out.

Striking a balance

Manual testing is still meaningful in the QA process. That said, in many cases, it is no longer enough.

To speak generally, conducting manual testing takes physical time and effort, that’s why it can be extremely wasteful for continuous testing.

Meanwhile, test automation increases the speed of testing and saves resources. It reduces the number of tests that have to be run manually (for example, checks of multilingual sites) but does not eliminate it altogether. Though, while increasing overall test coverage, more time will be needed for maintaining test scripts.

A flawless test automation solution should be easily maintained. Even if you have about 5,000 automated tests, there is no need to spend roughly 5,000 hours to make it work.

Certain cases simply don’t mesh themselves well to automation.

• UI testing. When conducting the display checks, manual testing will show how well your UI is in accordance with end-user preference, will provide a close connection with consumers, and help understand the problem on an emotional level.
• UX usability tests. They provide the answer from the consumer perspective if the application is easy to use or not.
• Exploratory testing. It is defined as a type of checking where QA engineers test the application, discover, investigate it on the wing, but test cases are not created beforehand.

It’s perfect if you can build an optimal duet of manual testing and test automation: when they do not diminish the value of each other – instead, perform testing in close cooperation.

Let us focus on some cases of such a successful “teamwork”:

• You have an aim to accelerate the test results and release the new version of the product on the market.
• Manual tests start where test automation ends.
• There’s a large volume of regression testing, and the software product is launching frequently.
• You involve both automation and manual checks to test different aspects of the same functionality.
• Tests are “semi-automatic” and involve some part of manual intervention in order to proceed to the next set of automated tests (e.g., it’s an online store, and the purchase can be made by certain customers, which can be proved only with the help of manual intervention).

According to DZone’s 2019 Guide to automated testing, the shift left model is still growing in strength if to speak about automated testing.

Comparing with the results achieved in 2018, the number of respondents who begin automatically testing software at the development stage has grown by 12%. The number of the interviewed who begin automating at the QA stage has decreased by 9%.

What concerns manual testing, this is a little shift left: contrasting this year with the previous one, the percentage of IT professionals surveyed who begin manually testing software at the development stage has decreased by 2 points. The number of respondents who do it at the testing stage has increased by 4%.

What to automate and where to go the “manual” route?

Let us define three main factors that should be considered in this case.

Clear and visible intent of test automation

If conducted accurately, test automation can be quite lucrative. We will discuss the advantages that can be reached from automated testing a little after, but now we don’t want to miss the other side of the coin when test automation just won’t fit well into your project.

What are these traps one needs to consider?

• The amount of new regression defects is slightly reduced comparing with those detected in the new functionality.
• Implementation of automation is not a one-off effort, as you need to spend time to keep the automated checks relevant and get the most leverage from them.
• Sometimes the results of testing are needed in a matter of hours, especially speaking about the checks of the newly introduced features. In this case, it can be more convenient for the project to move to manual testing and get the results ready ASAP.

However, there is no secret that the benefits of automated testing delivered to the project are clear and substantive. Let’s name some of them.

Test automation is the optimal way to fulfill multiple QA goals with effective resources and time.

It can be probably considered essential for organizations who strive to deliver outstanding software products and stay competitive in the industry.

Connect with us to build a result-oriented automation strategy for your business needs.

According to the latest World Quality Report edition, 99% of IT executives and managers are using DevOps principles while deploying their applications.

The factor that will help derive the full benefits of DevOps is greater automation of development, QA process, and delivery.

Due to automation, the tests are performing cyclically and continuously; the productivity of the specialists increases in several times; long sequences and operations that require heavy calculations are automating.

As a result, the quality of a software product is enhancing, and the development cycle is becoming shorter. At the same time, there are no versatile automation tools: much depends on the characteristics of the IT solution, your expectations, and time and budget capabilities.

In this blog post, we’ll talk about two test automation tools – Docker and Ansible – which greatly simplify the activities of QA engineers and allow more time for testing challenges.

Key terms and definitions

Docker is the end-to-end platform helping automate the deployment and the management of the container-based applications.

Being the lightweight software, it allows packing the product into standardized units (containers); making the application portable to any server (where Docker is installed), and running it in two seconds.

The container is one of the key ways to isolate software from its environment and make sure it performs consistently.

Selenium is an open-source testing tool used to automate web-based applications.

Ansible is an IT automation platform that simplifies application deployment, configuration management, and other software needs.

Ansible uses no agents to install on remote systems and no additional custom security infrastructure, so it’s easy to deploy. Therefore, to quickly reconfigure the system, just add a few lines to the test script.

Docker advantages

Docker is used to deploy and isolate applications in a virtualization environment and microservices implementation.

Benefits of the solution include:

• It is fast. Containerization reduces costs and time and allows running multiple isolated environments on a single host.
• Provides the ability to deploy an isolated test environment without interrupting the other ones.
• It can store environment settings in a version control system that allows you to manage their versioning.
• The container protects the software product from the web server configuration issues.
• It standardizes ongoing operations.

Due to the delivery in small containers, Docker simplifies and shortens the process of deploying applications and supporting projects (it detects the problems, returns to the previous versions to eliminate them), and makes it possible to split the project into small independent and convenient components.

How Docker simplifies the workflow with Selenium Server

While working with Selenium, IT professionals, especially QA automation engineers, face some challenges. Docker helps solve the following ones:

1. To eliminate Selenium tests errors, you do not need to install different browsers or multiple versions of the same browser on the server.
2. When launching the installed web browser, disk space will not be filled with cache contents or temporary files.
3. You can run parallel tests in different programs, and the browser windows will not begin to compete for focus, and many more.
4. Changing the browser settings will not affect the tests that were working previously and will not lead to their collapse.

In this article, we have discussed that a worthy alternative to Selenium Grid is Selenoid. Selenoid can also be used to launch browsers in Docker containers quickly and easily.

The main concept of Selenoid is to launch a new container for each Selenium session and stop it after the session ends. By the way, the number of different browser versions and the launch time are not limited.

The emerging issues can be solved by Selenium containers use, but to get the behavior similar to Selenoid, in addition to Docker Ansible should be implemented.

Ansible advantages

Ansible is a simple tool used to write automated scripts.

Ansible allows you to solve the following issues:

• Install, remove, and configure software
• Manage user keys and passwords
• Create and delete users
• Run scripts, tests
• Create and delete containers, virtual machines.

Key benefits of the solution include:

• It helps methodically update the nodes’ status.
• To describe the configuration files, a readable and simple YAML format is used.
• Additional software shouldn’t be set up on the node machines. Therefore, vulnerable programs are not installed, and the security of using the solution is increased.
• The solution code is quite simple, so it is possible to write additional modules.

How Ansible allows automating workflow with Docker

Ansible offers a convenient solution for automating workflow with Docker. Using Ansible to automate container building and application deployment, you get the following benefits:

Flexibility

Ansible Playbooks – applied to run scripts – can be used across different platforms. If you want to create a container using a Dockerfile, you can open the application only in the Docker container. At the same time, Ansible Playbook allows managing the application in any environment (Docker, Vagrant, in the cloud, etc.).

Controllability

Ansible Playbooks scenarios are easy to repeat and control. Despite container isolation, IT professionals need to understand how the container works, should track potential vulnerabilities and make changes if necessary.

The use of Ansible Playbooks allows you to check the status of the container, fix found vulnerabilities, determine who and when made the code changes.

 Versatility

Ansible helps manage the container itself and the entire environment in which it is set up. Using Docker, you need to start the hosts, configure and coordinate the work of cloud infrastructures or individual machines, and integrate them into the system.

Summing up

To effectively implement these solutions, Ansible should configure everything that is outside of containers, and Docker should be used to virtualize separate applications.

By combining both technologies, QA engineers can greatly simplify their lives while working in any CI environment.

Book a free consultation with the a1qa experts to find out how test automation services can speed up your software development process.

Selenium Grid is a widely spread solution used by test automation engineers. It serves to obtain a distributed environment for parallel execution of multiple tests. However, QA engineers encounter bottlenecks, which can be solved using an open-source project named Selenoid.

This article will figure out how Selenoid can help you save time and streamline automated tests performance, as well as provide a high-quality solution at short notice.

What is peculiar about Selenium Grid?

Selenium Grid is a cluster consisting of several Selenium servers. It’s intended to create a distributed network for a simultaneous launch of tests in multiple browsers. It incorporates a central server (hub) and nodes connected to it.

Although the solution is of high demand, the following peculiarities should be taken into account:

• Time-consuming deployment process

Engineers have to fulfill the following steps in order to deploy Selenium Grid:

• Complex launch commands

The commands to initiate server and node launch look as follows:

• Performance speed reduction

The hub decelerates if multiple browsers are run in different nodes.

• Compatibility issue

Browser and node versions should be compatible to avoid conflicts.

Is there a solution?

Some issues, such as long deployment process, can be optimized by means of Docker. This software platform is developed for a quick application build, debugging and deployment.

Engineers elaborate the image and after that proceed with the implementation of a container. It covers all aspects necessary for the application performance: libraries, system tools, code, and execution environment.

Even official Selenium images with multiple browser versions will hardly optimize browser launch since all Selenium Grid peculiarities remain constant.

Selenoid as an alternative

Quickly and easily customizable Selenoid can become a robust option. This tool helps to run browsers in Docker containers. The process is different, although.

Selenoid launches a new container for each new browser request and stops it after the session is closed.

A certain browser version, the required version of the web driver or Selenium-server, which supports this browser and all necessary dependencies (for instance, graphics libraries) form each container.

All browser processes are fully isolated. This allows launching an unhindered number of multiple browser versions concurrently.

Selenoid is written in programming language Golang and supports all popular browsers.

Do you want to make the life of your team easier? Consider test automation. It will come as a relief to both Dev and QA teams. 

Selenoid vs. Selenium Grid

Both solutions pursue one objective, which is the elaboration of a single environment for a parallel launch of automated tests. However, these solutions differ a lot.

Selenoid advanced features

Selenoid comprises a number of additional functions that contribute to workflow optimization.

• Data storage in RAM

Selenoid stores all temporary files in tmpfs.

Tmpfs is a temporary file repository that allows storing files in RAM, access to which is performed much faster than to the file system of the hard drive.

• Various screen resolution types

QA engineers can configure the appropriate screen resolution for a running container on their own by setting the required parameters in the Browser Capabilities.

• Browser screen display

Selenoid enables engineers to connect to the VNC port during tests execution and access the screen of the desired browser. It’s even possible to interfere in the process of running the automated test using the mouse.

• Video recording of tests

It’s possible to record the video of the tests performed.

For instance, the activation in the Google Chrome browser is implemented by setting the parameter true in the Browser Capabilities:

ChromeOptions options = new ChromeOptions ();

options.setCapability (“enableVideo”, true).

Summarizing

Due to multiple benefits, Selenoid is a robust solution for test automation engineers.

This scalable alternative helps to change configuration settings seamlessly according to the preferences and needs of your team, cut corners on resource consumption, and perform tests in due course.

Any questions left? Feel free to drop a line in the comments.

In the previous blog post, we spoke about all the aspects that should be taken into account to build a proper test automation strategy. Now it’s time to bring in the real-world example.

The client – anti-virus solutions provider – was on a hunt for a test automation vendor.

All of the described above was taken into account by the a1qa test automation team that was hired to deliver the automation projects and achieve the following goals: reduce deployment and smoke testing duration, eliminate X number of manual testing man-hours a week, and ensure the customer’s portal localization to 31 language.

Automation was a part of a huge and costly business project: the development of a new web portal that was meant to provide users with the opportunities to install security apps on their devices, remotely manage licenses and protection features, and purchase new products.

The developed portal should have become the company’s flagship product so it was decided to invest in test automation.

Scope of work

The primary objective was to automate time and efforts consuming regression testing. When this task was fulfilled, it was decided to get down to automating other tests that covered scenarios of high business value.

At that point, the a1qa test automation team was assigned to develop automated tests to check the functionality that was delivering maximum business value.

Automation of localization testing

After the thorough discussion, it was decided to start with the automation of localization testing, which was conducted regularly and covered 31 locales.

The objective for the localization team was to adapt the portal to new cultural regions, which meant working on developing new functionality and making the existing functionality available to new countries in new languages.

The approach implemented by the test automation team made it possible to automatically compare the same pages of different product versions for regression and compare the same pages in multiple localizations.

Due to the application of automated tests, the deployment sped up fivefold.

Automation to monitor availability of third-party services

Another business value functionality was the notification on third-party services failure. The portal functionality greatly depended on the performance of the third-party services. The a1qa engineers developed automated tests to monitor the availability of the services.

In case of any troubles discovered, the system reported the defects and notified the stakeholders (developers, managers, testers) about the issues occurred so that they could be eliminated promptly.

Making right choice brings results

The implemented strategy focused on product strategy, functionality value, and the project goals. It helped achieve the following results:

• Deployment process and smoke testing time in production were reduced fivefold.
• Test automation coverage was equal to 900 hours of manual testing efforts on a daily basis and about 200 additional hours every week.
• Monthly environment downtime was reduced by 26 times.
• Test automation paid off and started to bring a positive financial effect six months after it was implemented.

Summing up

Test automation is a great solution for those who want to deliver business value faster than competitors do.

However, as you see, test automation shouldn’t be a one-side decision, rather than technical and management teams should discuss the benefits, costs and possible risks.

To choose a proper set of test cases to automate, consider the strategy of your product, calculate the cost of opportunity, and prioritize the product functionality by its value.

The well-built strategy will bring you significant benefits and put you one step ahead of your competitors.

Contact us to get more information on test automation service.

Working on a software delivery project? Then one day you’ll definitely face a tough question whether or not test automation should be introduced to help you achieve your goals.

Before starting to look for a team of professionals who’ll select the right tools, set up test environment and develop a suite of automated tests, you should decide for sure why and, more importantly, what you want to automate.

Why to automate?

Well-built test automation makes it possible to respond to a number of common software testing challenges: to get faster test results, reallocate resources efficiently, save QA and testing budget, ensure the possibility of closer infusion of testing into the existing product development processes.

What to automate?

Three factors should be taken into account when deciding on what to automate: your software product strategy, opportunity cost, and business value of the functionality. Let’s specify them all.

Contact us to get an obligation-free consultation on test automation.

Product strategy

This is the first issue that should impact your test automation approach. Do you work on the product that will become the flagship for the company in the next decade? In this case, there is no focus on the speed of release and the development will be conducted in a steady pace.

Or are you developing a prototype of the app and need a fast feedback from the target audience? In this case, time to market will matter most.

Certainly, the approach to test automation will never be the same in both cases. You won’t invest heavily in automating testing of the prototype, while the flagship product may be worth huge automation investments.

Opportunity cost

Cost of automation depends on the readiness of the product and process of the automated testing implementation. Therefore, it’s important to know all of the test automation stages.

Agreeing on how long you can spend in all these stages will determine the scope of the future automation solution.

Business value

Measuring business value is a tough proposition. The value depends on the product developed, its objectives, and the specifics of the business domain. A metrics-based approach may be applied to prioritize the functionality. Identify real-world metrics to estimate the business value of every functionality.

In this blog post, we’ll tell how our test automation team built a highly effective test automation strategy that led to the outstanding results.

In the previous article we spoke about describing test scenarios with Gherkin syntax. Today Sergey Senyuk, Manager of a1qa test automation team, describes how their team develops custom-tailored test automation solutions into the BDD mindset.

Test automation in BDD context

When the end users of the automation solution are QA specialists who have no skills in programming and creating tests themselves, we offer to use the following approach to deliver effective automation solution that will work on the client’s side:

Stage 1: Regardless of the type of the application that is to be tested (mobile, web or desktop) we create framework with the basic logics necessary for application automated testing.

Stage 2: Using the Page Object and other relevant patters we design solution structure, write form classes that should contain the fields that correspond to the app’s UI elements and methods of handling these elements. We also create tests classes with form objects to implement the defined usage methods.

Stage 3: We put test data to the external files, thus removing logic details from behavior tests.

Stage 4: Implementing the BDD approach, we integrate relevant libraries to the project to use Gherkin syntax. Then the steps of test scenarios should be developed using the keywords Given/When/Then.

Graphically the delivered solution will look as follows:

As a result, by adopting this approach we let QA engineers work with Gherkin source files containing feature descriptions in their localized language. The documentation also serves as executable automated tests. By making various steps combinations a QA engineer may easily create new tests on his or her own.

A report of the successfully executed test will look as follows:

Every feature description may contain several user scenarios. Every scenario may consist of one or several steps. Every step may need a number of actions to be taken. All this data will be presented in a tree-type structure.

For every test that has failed there will be the following information available in the logs:

• A step with the mistake occurred marked in red.
• Description of the actual and expected outcome.
• The string of the program code with the failed check.
• List of actions that have been completed before a mistake occurred.
• Application screenshot.

Another question is how to build and maintain effective communication between software testing engineers and automation engineers within the BDD projects.

Communication between automation engineers and QA team

In order to ensure a bug-free test automation solution that will take effect and deliver tangible results to the customer, it’s vital for both teams involved to understand the scope of their duties.

We advise to separate the obligations of QA team from those of test automation engineers.

At the initial stage, of the project, be it a pilot one or Proof of Concept, testing engineers convert the set of test scenarios in Gherkin or create their own scenarios. After that, automation engineers develop automated tests based on the given scenarios within the BDD approach. They also set up a continuous integration test environment to enable tests launch and storage of their run reports.

Upon the project completion, automation engineers submit the following deliverables to the client:

• Test scenarios in Gherkin.
• Automated tests and framework.
• Continuous integration environment for tests launching.
• Test automation solution documentation (architecture description, launching procedure in CI, etc.).

Automation engineers should also train QA team how to handle the delivered solution.

Hereafter, the automation engineers’ duties will be limited to working with the code:

• Maintaining the existing tests and updating them in case of the UI or business logics changes.
• Creating new tests.
• Developing additional steps to improve the variability of the automated tests.

QA engineers, who will act as primary users of the delivered solution, will be expected to do the following:

• Launch tests to check the existing scenarios on numerous data sets.
• Analyze test run results and provide the client with the detailed report containing logs, screenshots, tests execution metrics, etc.
• Create new tests using the keywords and scenarios in Gherkin.

It’s also important to take into account a number of factors that may increase the final costs of the automation project with the BDD mindset:

1. Teaching Gherkin grammar and syntax to QA team.
2. The solution development will take by 10-15% longer due to the necessity of developing test scenarios steps.
3. Converting clients’ test scenarios in Gherkin to enable further creation of automated tests on their basis.

Despite extra costs, the number of BDD advantages will significantly prevail.

The key benefits will be the following:

• Documentation and automated tests are stored in one project and the feature description is actually an executable test. There is also an option to develop the automation solution so that the scenarios in Gherkin will be stored in any test-tracking system and automated tests will request them when necessary.
• The solution can be handled by users who have no programming skills. QA specialists will be able to launch tests, analyze their runs and create new test scenarios.
• The steps in Gherkin syntax are reusable and the keywords may be combined in many different ways to create different scenarios.
• The system description in Gherkin will be common for all teams involved. As a result, their vision of the system will also be the same.

Let’s sum it all up

Combination of BDD approach and Gherkin language takes a number of positive effects on projects where QA team has no skills in writing automated tests. However, in order to ensure maximum effect, it’s vital to take into account and calculate the costs of QA team training and converting test scenarios in Gherkin.

Judging by our experience, the benefits of implementing BDD in test automation prevail if done properly.

Sergey Senyuk, Manager of Test Automation Department at a1qa, speaks about Gherkin, a language designed for non-programmers that can help to describe system behavior and create automated tests without deep knowledge of programming.

What languages do automation engineers speak?

To build an automation framework and create a scope of automated tests, the developers may use a variety of languages: C#, JavaScript, Ruby, and many more. Each of the languages has its own syntax.

The choice of the language may be conditioned by client’s preferences, automation team expertise or by a set of specific libraries for a certain tech stack. And while the programming skills are a matter-of-course for engineers who are engaged in developing automation solution, the end-users of the solution aren’t necessarily experts in coding.

However, they’ll have to launch tests, analyze the reports and support them, i.e. bring up-to-date after the UI or application business logic has been changed.  The automation solution shall be beneficial for the end-client, easy-to-use and convenient.

Users of automation solution – who are they?

After the automation solution is developed and handed over to the client, it’ll be used by the following specialists:

• Automation engineers themselves (obviously, there shall be no problems in understanding the language);
• Clients’ developers (generally the solution will be created using the same tech stack as the in-house developers do, therefore there also shall be no troubles in script understanding);
• But most commonly the solution will be handled by the client’s QA department. And this is the point where troubles I want to talk about occur.

The essence of the problem was perfectly defined by one of our potential clients after he saw our automation framework and learnt our approach.

He imparted his apprehensions for the ability of his QA team, who had no experience in writing code, to support the scripts, say nothing about writing their own scripts.

When testing engineers without programming skills are the end-users of the solution, our team suggests building a solution by implementing the BDD (Behavior-driven development) approach.

Why is BDD approach so remarkable?

Primarily, BDD is the methodology of software development and it suggests that the expected behavior of the system should be described in terms of plain-text scenarios. After that the described functionality is developed and, when delivered, the predefined testing scenarios should be successfully executed.

A scenario is the description of the system functional behavior written in human language by a definite pattern. As the automation engineers and testing engineers should be able to work together with the solution, scenarios are written in a ubiquitous language.

A ubiquitous language is a living language that can be used to facilitate communication between programmers and non-programmers and, at the same time, can be perceived and interpreted by the automation scripts. As a result, there is no need to improve or modify the documentation for the development or QA team usage. All scenarios are described in the same way and can equally be used by all teams involved.

Describing test scenarios with Gherkin

Executable specifications can be created in Gherkin – plain-text language that can be learnt by non-programmers. Gherkin fulfills two goals simultaneously: it serves as project documentation and automates testing process.

The system behavior is described with the keywords Given/When/Then. Every new line in Gherkin starts with one of these words which is followed by a description.

According to BDD approach, a desired behavior should be specified in terms of user story in the following way:

Title. It should be clear and explicit.

Narrative. A brief, introductory section specifying:

• who is the stakeholder of the story or who will derive benefits from it
• what effect the stakeholder wants the story to have
• what business value the stakeholder will derive from this effect

BDD does not have any formal requirements for exactly how user stories must be written down, but it insists that each team using BDD should come up with a simple, standardized format for writing down user stories which includes the elements listed above.

A brief example of this might look like this:

As a [X]
I want [Y]
So that [Z]

where Y – is a feature, Z – is a benefit of using the feature and X – is a user or group of users who will benefit from the feature.

However, behavior specification in Gherkin is completely another story. Every line here is a definite step which corresponds to one of the code methods.

Every scenario shall start with a precondition. It may consist of one point or of many of them. The precondition starts with the keyword [Given].

After that the actions that will take place in the scenario are described starting with the word [When].

Finally goes the outcome description with the starting word [Then].

If there is more than one step in any of the blocks, each of them should be written down in a separate line with a keyword And.

Example of specification in BDD

Feature: Log in to the app

As a user

I want to log in

So that I make a payment

In Gherkin this scenario will be described as follows:

Given User is on Login Page

When User enters following credentials and submit

|Name         |Value         |

|Login          |test_user  |

|Password   |pass           |

Then ‘Welcome to our site’ message should be displayed

Most of the BDD frameworks are multilingual Cucumber, for example, supports 60+ languages with this number constantly growing.

Key reasons to learn Gherkin

1. System behavior is specified in a plain way that is clear for all project participants.
2. Files that contain specifications are executable automated tests at the same time.
3. Test documentation and automation scripts indivisible and are saved in one project.
4. Natural language constructions allow to build variable scenarios and to create new automated tests without reviewing the code.

With that all said, we recommend all our QA colleagues to learn Gherkin as this is the easiest way to start talking with automation engineers the same language.

In the next article we’ll talk on how to build effective communication between testing and automation engineers on BDD projects.

In recent years, automated testing has become a trend in software development. However, it is labor-intensive and expensive to implement and maintain. That is why automated tests should be used only when it is really necessary.

What are the reasons for test automation?

The most common way to evaluate the effectiveness of test automation is to calculate the return on investment (ROI) that is the ratio of profit divided by expenses.

Saving efforts on manual testing can be a profit as well. The benefits of test automation may not be apparent, e.g. finding defects in ad-hoc testing by manual testers, who have much free time thanks to conducted test automation. This profit is quite difficult to realize, so you can make an assumption (e.g. 10%) or omit it.

However, test automation is not always implemented to save money or resources. The speed of testing can be significant for business for many reasons. Precisely in these cases, automation helps save not money but time.

Another reason is to avoid the human errors testing the systems that require extreme accuracy and correctness. The price of such errors can be significantly higher than the cost of automated tests development and maintenance.

When is test automation suitable?

Test automation is appropriate and can be implemented in the following cases:

• The application functions smoothly
• The functionality is tested regularly and frequently.

What are the initial data for effectiveness measuring?

To measure the effectiveness of test automation implementation the following input data is required:

• The cost (in man-hours) of manual testing
• Autotests development costs (in man-hours)
• Information about the frequency of testing.

How is the effectiveness evaluated?

Usually, the time period for analysis is a month. But it can be changed according to the project needs. To begin with, calculate the cost of manual testing during each interval.

Then we estimate the cost of automated tests development. Also, it is crucial to take into account the team size. Analyze the difference between the cost of manual testing (without automation) and automated testing.

What can affect the effectiveness of test automation?

• Frequency of test runs

Supposing that one test is run every day and requires 1 man-hour, the automation of the test takes 4 hours. It is obvious that on the fifth day, autotests will become more effective than manual testing.

• Data-driven testing

Often one and the same test is needed to perform with a different set of input data or in different environments, e.g. cross-browser testing or creating a single entity with different data.

In this case, test automation with the use of the data-driven approach is the best solution.

• Tester’s experience

A factor that greatly affects the cost of development is tester’s experience. The tester needs to have enough skills and knowledge to develop, maintain, and analyze automated tests. Otherwise, the time and cost of automation will increase.

• Tools cost

There is a large number of different tools for test automation. Most of the paid applications are better than their free equivalents and allow developing and debugging tests faster.

• Analysis and maintenance

While evaluating the effectiveness of test automation, most engineers forget about such important activities like the analysis and support of test runs. If there are a lot of autotests on the project, it may take a long time.

• Cost of test suits preparation

Often, test automation engineers receive from manual testers test cases that are not suitable for automation. For this reason, sometimes it is necessary to prepare test suits from scratch.

It is obvious that test automation evaluation allows you to answer the following questions:

• “Should we use automation on the project?”
• “When will automation bring a significant result?”
• “How many hours of manual testing can be replaced?”

However, even after receiving the answers to the above-mentioned questions, you should remember that automation can be introduced only when testing processes are stable.

The article by Alexander Panchenko was published on EE Times, you can read the full version here.

As software and hackers get more sophisticated, QA testing will gain importance. That will require more software QA engineers and more test automation. Software QA budgets are on the rise.

Recently, the sixth edition of the World Quality Report 2014-15 was issued. The findings are based on a global market research study conducted among 1,543 senior IT executives. The report highlights recent changes in testing practices and shows that QA and testing budgets are increasing in many industries because of time-to-market pressures and a need to provide an all-channel experience for the consumer.

According to the report, companies spend more than a half of all QA and testing budgets (52%) on new development projects—an amount divided between cloud initiatives (27%), big data and analytics (20 percent each) and mobility (17%).

Based on the analysis presented in the research and our first-hand experiences at a1qa, we can presume that testing trends consistently follow those of development. Today, literally every software company tries to propose the most practical mobile solutions for its clients because applications must be accessible and user friendly on any device. So called “Responsive design” has become extremely popular, providing a unique use experience for each type of device or browser.

Unfortunately, that flexibility could cause security problems. For example, captcha absence in the mobile version of a website exposes it to brute-force attacks. That’s pushing non-functional test requirements such as performance, security, and ease to increase, especially for mobile testing. As a result, automated tests are growing in popularity. Automated tests result in increased test speed, decreased costs, and shorter time to market.

Looking deeper into software testing specifics, you could face a trend of TDD (Test Driven Development), a software-development technology based on the repetition of short development cycles. For example, this occurs when a software tester writes a test that covers a new feature, and then a developer creates code that will pass created test.

Cloud services are another big trend for development and testing of applications deployment and data storage. Cloud services suit the necessity of having information “right here, right now.” More and more data is moving to the cloud, which includes not only personal data, but also application software, servers, and databases. That inevitably results in an increase in cloud use and virtualizations for test environments.

Because of scandals around information leaks (Sony being the latest high-profile victim), however, many companies aren’t in a hurry to transmit their data to cloud services — even knowing the advantages of cloud computing override possible risks of using it. That’s because configuration security is still an unknown. Giving the data to a provider, a customer is convinced that it will be protected by default, although reality shows this isn’t always the case.

Read the second part here. 

With the KDT approach, it should first be confirmed that the test plan consists of actions suitable to KDT. Typically, such tests appear when the program under test provides the user with a common interface operating objects containing different structure or logic.

For example, MS Word guarantees the possibility to work with texts, charts, pictures, etc. using the same interface and set of instruments. Testing of business logic for such generic interfaces is quite complicated for a data-driven approach; KDT eases the process.

When used in situations suitable to its strengths, the KDT approach can help any automated testing company save money on support activities and achieve better ROI in the long term. The graph below shows the variations in cost of development and further support between KDT and some other approaches. In addition to saving money and time, the KDT approach works better if you got more test cases. Also, the majority of effort with KDT is concentrated within the initial stage of the test development process.

A keyword-driven approach significantly cuts down development effort due to several reasons:

• A significant part of the automation process can be performed by people without programming skills.
• Script writers can develop automated tests at a higher level. This allows them to concentrate on the design of automated tests instead of low-level operations.
• GUI changes can be made centrally. In other words, it is enough to change the map with interface elements if you need to add a new button to an existing form. When the keywords are developed, they are not connected with certain interface elements. Thus, keywords can easily be used for similar standard elements.
• Refactoring of non-centralized functionality is realized by “Find/Replace” operations in the test description tables. For example, consider the implementation process of a new form in a Wizard sequence of pages. In the table of metadata, you can find all the tests including the previous screen, and change the data to add the new form. Using standard architecture, you would need to add functions and go through the code to define all the dependent functions and change them afterwards.

Finally, let’s compare KDT to data-driven testing (DDT), the most common testing method used to date, using a concrete example. KDT is, in fact, considered by many to be DDT’s “logical next step”.

a1qa recently compared the costs for development and support of keyword-driven and data-driven testing. While these two methods are similar in that both are a part of the automation process test data preparation, a key differentiator is that KDT test data has a much more important role and is able to fully replace the test scripts. To estimate benefits of KDT in practice, we chose a large, stable scope in terms of business logic and user interface changes. This guaranteed that there would be enough keywords to make measurements and to check test development abilities using keywords; 200 test cases were selected for automation. We performed daily test executions, which was necessary in order to speed up the comparison process.

One of the required factors was to get a situation where test development efforts on the keyword-driven test framework were less in comparison with the actual automation process for an existing solution. We achieved this level when 150+ cases were developed.

As a result of this testing, we found KDT had no restrictions for the usage, and the proof of concept (POC) stage showed positive results. We began measurements after test development, compared them and found similar characteristics for the critical automation process for both solutions. However, based on the experience above, we would suggest using the KDT approach when covering hundreds of test cases in long-term projects with complex business logic, which assumes a high level of keyword re-usage.

Read the first and the second parts of the article.

The article was published on Computer Technology Review. Read the first part of the article here. 

According to the KDT approach, there are three roles in the automation testing process:

• Test designer – creates the actual keyword-driven test cases, including pre- and post-conditions and keyword parameters. This is the role of a non-technical subject matter expert/business user.
• Automation engineer – maintains the core framework and adapts it for any new customizations or enhancements. This is the role of a technical person who is well versed with coding and scripting language tools.
• Test runner – serves as the continuous integration (CI) administration technician. This role does not require a purely technical person but requires understanding of what input data is needed in order for the automated script to run.

The KDT process includes the following steps:

• Testing plan development (includes definition of automation testing keywords).
• Automation testing scripts development.
• Keywords implementation by automation engineer.
• Launch of automation tests.
• Implementation of automated tests in the regular pool (CI).

The picture below shows the different steps of the testing automation process for each of the aforementioned roles.

Let’s consider a very simple example to get a better idea of how KDT works. Imagine you have to test an application that adds two numbers. For the step of “defining business requirements and testing plan development,” you must determine what numbers can be put in and what interface elements might be used. Let it be positive integers. The app’s GUI has two text fields, a button and an inscription to show the result.

The “definition of automation testing words” step includes creation of a limited word set and the requirements for it. This step can be executed by a test designer.

We need three words possessing the following parameters:

• “Edit”: two parameters (entry field name and entry text)
• “Click result”: No parameters
• “Verify”: one parameter (validation text)

The test designer describes the core of the tests – minimizing the quantity of used words. Using the KDT methodology, he/she analyzes tests on a high level. Therefore, it is easier for the test designer to identify the frequently repeated functions that should be automated for tests. As a result, the automation engineer receives a well-defined and simplified task.

For “testing scripts development,” we implement the words defined above, compose a test and input data:

• Words implementation is a duty of the automation engineer. His/her task is to provide efficient execution of every test operation, even if the interface reacts slowly. He/she should also thoroughly validate the operation result and describe the words using “HOW TOs” for a test designer.
• The tasks of test development and data preparation can be submitted to the test designer. He/she should provide tests for full-scale functionality coverage. To successfully fulfill this task, the test designer should understand the application requirements, choose the testing strategy and develop the cases.

The variant can be implemented by means of two test automating instruments – Robot Framework and the AutoIt plugin for robot framework.

The solution will include three tables and the code:

Whenever changes are made, everything should be verified immediately. You can apply high-level test tables to detect changes. As every keyword in KDT is used in multiple tests, it allows you to launch integration testing for already-checked keyword implementation. Thus, you can easily detect and fix all the issues caused by incorrect keyword usage.

As was mentioned, the above example is simplified, but the automation process for real-life projects is quite similar. However, with real projects involving test automation, testers handle many tasks at once. For example, before the test launch, the test runner should prepare the infrastructure and organize the multithreaded launch of automated tests, and the test designer should add information into the application database, as is shown in the picture below.

So what does all of this really mean?

To be continued… 

Automation testing is a topic that is under constant discussion and improvement. The IT industry often tries to find more and more ways to make automation more effective and applicable, though more efforts should be made to fit the QA process; this is where keyword-driven testing (KDT) excels.

KDT, also known as table-driven testing or action word based testing, is a software testing methodology suitable for both manual and automated testing. This method separates the documentation of test cases – including the data to use – from the prescription of the way the test cases are executed.

As a result, it separates the test creation process into two distinct stages: a design and development stage, and an execution stage. This approach is particularly attractive for automation testers without more advanced programming skills.

Also, the analysis and integration of very similar tests with keywords reduces the amount of code required in automated tests. Thus, a person who prepares the description of a test using already implemented keywords can, in fact, create an automated test at the same time. This methodology considers creating a table in the form of keywords for such automated actions as “Add Item” or “Delete Item.”

For example, consider a keyword for a log file messages check. First, check the message type and its code. Then, write a test method to perform automated testing with a couple of parameters: the prefix and postfix. Store the test data in a text file as a table, but use an additional middle column to transform the table into ordinary English text. Separate the records with new lines and separate three columns by tabs.

The test engine library will read through this test data and invokes the test method. For simplicity, we will not consider the test engine and test method implementation just yet. The scheme of an automated test is shown below:

A properly organized test framework according to the KDT approach can reuse the test method for a whole test scope. Keyword usage could be enough for an automated test design. Automated tests in the form of ordinary English text and stored inside text file are easy to develop and maintain. There are several KDT frameworks that implement test engine functionality. We’ll dive into some more detailed examples later, using Robot Framework as a test engine and Python as a programming language.

Read the second part here.

The article was published on Computer Technology Review.

In the previous post we had a look over keyword testing that has a somewhat dubious reputation with QA consultants and covered some steps of the comparison we made. This I`d like to provide you with the test results we`ve got.

Several charts below illustrate our findings regarding last week of experiment. One should also note tests were executed daily.

The first comparison had respect to test execution time. Keyword driven assumes that some additional logic of parsing keywords works at the background. The impact on test execution duration was mainly under our focus.

The results indicate test execution time is comparable and we may apply Keyword driven in future without additional efforts/hardware for optimizing test duration.

Efforts required for analyzing test results became another comparison point here. In fact it shows the similar picture. Despite keyword driven required a little bit more efforts to support tests, we got the same trend there. The reason may lie in the sphere of business logic. This part of measurements should be developed further.

Another point is test support time. We may see that both solutions are comparable by this indicator as well.

As for the test development efforts – manual QA Engineer was able to develop test case description using existent keyword. Taking this into consideration we may consider spending to be effective comparing to current solution.

Let’s sum up the outputs

The first conclusion we are able to mention is that Keyword driven has no contraindications for use and POC stage showed positive results. On the other hand based on our measurements we see no extraordinary results or substantial breakthrough.

We started the measurements after test development efforts of the new cases became comparable and we’ve got pretty the same characteristics for critical automation process on both solutions.

So, based on this research one can suggest using keyword driven when you are going to cover hundreds of test cases in long term projects with a business logic which assumes high level of keyword re-usage.

Having this as a consequence we decided to apply Keyword driven practice to several upcoming projects.

Automation is a topic that is under constant discussion and improvement today.  Its consistency has been already proved.  But for IT industry in general this practice pretends to be more and more effective and applicable, to fit QA process needs better.  One of its means is cost effective usage of testing. Just imagine that you can double return on investment KPI. On the one hand you save money, on the other can increase quality using the same budget or mitigate QA budget risks on stabilization and delivery stage.

There are lots of interpretations of keyword driven approach in the world of test automation. Different tools allow using keywords.  But not all that cases may be a true keyword driven. The way I used to describe it in simple words is: “manual testers mostly create test cases using keywords and test developers create code which implements particular keywords via available interfaces”.

What is coming out that meaning?

1. The keyword driven approach helps to achieve better ROI in long term saving money on support activities.
2. It optimizes (moves down) technical threshold for stuff who may be involved in automated tests development process i.e. persons who prepare description of tests which is pretty the same like in test automation in terms of keyword driven. We may say that a person who prepares description of test using already implemented keywords creates automated test the same time.
3. The last but not least – it maintains the relations between test developers and testers themselves regarding automated tests which positively impacts QA budgets and overall process.

Of course, there are patterns for using that approach like for data driven or other ones we are aware of.  For instance, you may apply scripting technique which requires less money for initial test development, if your test process assumes coverage of stable critical functionality (meaning smoke test) the frequency of builds is closer to daily basis, and testing is required for more than 4-6 upcoming months.

Thinking of Keyword driven approach you should ensure that you have a lot of business logic. There are similar actions which can be used in different test cases (potential keywords), the business logic is too complex for Data-driven approach, which may be extremely effective in some other particular cases.

So, you never know without trying! We decided – it’s worth doing. I’d like to share some of the most important findings. One of the crucial requirements here is that special description format should be applied from the very beginning of testing activities. Taking into account this fact and absence of experience with tools that are specially designed for Keyword driven approach implementation we decided to split our first Keyword driven practice in two steps.

• Step 1 – Proof of Concept (POC). This is the base for the article you are reading. Mentioned step purposed to try out a particular tool, check basic automation indicators like execution/support efforts, check if manual tester might be able to handle test case description.
• Step 2 – Using Keyword driven approach. The second one is to apply this approach from the very beginning of the particular project and cover the whole QA cycle using it.

Thus, we have found a “guinea pig” for the first step and here is a description (without mentioning concrete customer, of course).

Solutions comparison

Basically, we have selected quite stable scope in terms of business logic and user interface changes. It was decided to select rather large scope which guaranties that there will be enough keywords to make measurements and to check test development abilities using keywords only. As a matter of fact, 200 test cases were selected for automation.

It should be mentioned that selected project was quite dynamic and required daily test executions. It was necessary to speed up solution comparison process.

One of the required factors was to get a situation when test development efforts on the keyword driven test framework are less labor-intensive in comparison with actual automation way for existent solution. We achieved mentioned result just after 150+ cases were developed.

Some good lessons have been also learned about Keyword driven while tests were developing. There are top-3 issues you should consider applying keyword driven practices to the project:

• TBD 1– First tests on Robot Framework KDT requires quite more than average efforts (5-6 hours when scripts with DDT regular approach requires 3-4 hours). Efforts decrease with list of base keywords being completed and available for re-use (Down to 1-2 hours).
• TBD 2 – The tester is required to learn several special formats & keywords for creating the customized keywords. It takes some time to get familiar with framework specifics on tests development (IDE, libraries, etc.)
• TBD 3 – Robot Framework execution engine doesn’t help contain default mechanisms implementing itself to solve parallel test execution problem.  It was quite a big challenge to implement this mechanism. We tried to run scripts simultaneously on a single VM by using several jobs in Jenkins, but it failed because of AUT UI specific interactions: some UI elements were available only in the currently available browser instance.

A soon as a number of tests were ready, we started actual comparison.

In the next post we`ll provide you  with the results we got.

The article by Dmitry Tishchenko was published in UK magazine “European Software Tester”.

Robot Framework is an acceptance-driven test automation framework basically in Python (can run also on both Ironpython and Jython). Robot framework (RF) provides a lot of synergy in test automation. It makes a lot of automation processes flexible and simple. But before going further into RF architecture we have to cover its keyword-driven concept.

The best way to use Robot-framework is to use it on unix-like systems because of its pythonic origin. With the help of tools like easy-install and pip you could extend it with open-source libraries almost infinitely without any problems. This also helps a lot when it’s coming to CI systems and deployoing test libraries on the build node.

However, because of same pythonic nature this tool could be easily used on any platform where you can run python 2.7. If you think that windows based installation of robot-framework will have lack of support – blame on you. There’s a lot of platform specific librariesmodules out there. For example, AutoIt library that is windows-only GUI automation tool.

As for Robot framework drawbacks should be mentioned lack of parallel test execution, not-easy-to-customize html reports (however, there is an option to generate xUnit formatted short reports) and some skill to create own low-level libraries.

As for other pros and cons – they are pure approach ones, based on which one you would use. In Robot-framework you could use Keyword-driven approach aside with data-driven and behavior-driven ones. However, behavior ones could be described via keywords or via Given-When-Then scenarios.

Considering the approach benefits and requirements you know how to apply the method in test automation to gain value.

The article is prepared by Yan Gabis.

Keyword-driven approach in automated testing is one step away from behavior driven approach. It’s not far from behavior driven approach at comprehension standpoint but much more flexible and easy to re-factor. It is one step away from Object-driven approach, too.

You can use keywords as methods and “keyword arguments” as method arguments. So there is a possible usage of encapsulation, which could help re-using code and reduce time to write new tests. Actually, there are some tools like Robot Framework, which could use keyword driven, behavior driven and object-oriented approaches as well as data-driven one.

The main reason to use keyword driven approach in the project is maintenance costs and in some cases development costs. Maintenance costs are lower than in many other approaches because of easy keyword modification. To refactor or modify them your team should not have specific programming knowledge, the basics would be enough. Also because of inherited from object-oriented approaches modular structure of keywords they could be reused often, which cuts test development costs and lowers maintenance costs even more.

The reasons against the keyword driven approach are the same that could affect your test automation on the whole:

1. Tests that you run only one time
2. Tests that need to run ASAP
3. Tests that based only on user experience

Keyword-driven approach could also be used in manual testing! This will give you flexible testing steps representation and test implementation. And one of the most exciting and debatable part of – you can combine manual steps with automated ones!

Pros

1.Re-usage of keywords

2.Divide et empera:

• Test case implementation does not require learning specific tool or programming language, only SUT domain knowledge.
• Keyword implementation does not require strong domain expertise and rely mostly on tool/programming language knowledge

3.Easy-to-implement abstraction layers

4.Less costly maintenance because of easy test modification, readability and keyword re-usage

Cons

1.Higher learning curve than with behavior driven testing

2.Division of responsibilities requires more intensive process management than with other approaches

Considering the approach benefits and requirements you can decide how to apply the method in test automation or manual testing to gain value.

The article is prepared by Yan Gabis.

In software development it’s hard to find anyone who considers software architecture useless, the same applies to automation testing. By test architecture I mean all code that will support and facilitate test cases development and execution (e.g. test framework), the project structure and any management that will take place (e.g. test data management and documentation).

First of all there are 3 key points that reveal test automation framework uses:

• Code support
• Test support
• Analysis

Analysis plays an important role for code support as well as test support – it is the most customer-oriented part, tightly connected with framework. Now let’s take a look at automation project architecture where test framework takes place.

• System under test is processed with test tools through application interfaces (GUI/API)
• Test tools actions and test libraries methods are invoked through its API by test framework.
• Test framework uses test data, usually specially formatted for it.

This is the most common architecture. But is test framework really helpful?
First of all – forget about linear tests. Implementing test libraries makes your code reusable. Why doing same things over and over again when you can simply implement it once and use several times?

Main idea of test framework: a little bit more efforts once – much less efforts each time later. A common solution for this is design patterns. With design patterns we can improve some tests capabilities.

For example Page Object pattern reduces the maintenance impact, helps to implement similar tests much easier. In web testing Singleton can be used to provide a driver instance to tests instead of inheritance. If you need to verify a database, the Entity can be used to reduce the quantity of database connections.

However using design patterns along with test framework is recommended, but not obligatory.

When you need to decide what architecture to choose and answer the question “Should we have test automation framework?” try first to answer on this questions:

• Who will develop the tests and analyze results?
• Does this person have the right technical skills and domain knowledge?
• Do you need to separate test description and/or data from low-level code?
• How many third-party tools you use to access SUT?
• Do you have time and budget to implement it internally or it would be better to get a professional help?

The article is prepared by Yan Gabis

It goes without saying that you should choose the right tool to work with while testing process. The challenge although is in absence of good advice:

• When particular tools work better?
• Will they suit the current projects?
• Is it worth to risk and try something new?
• Should you try them if deadline is near?
• How all this tools would work together?

Most of the articles about automated testing are full of ROI, KPI and other stuff which is sometimes complicated. Often it is not clear, how it would work on your project. If there are any examples on real projects, the information is not enough or not relevant. Still, what I want to cover is how to decide which approach suits you exactly.

Below are some points that help to consider which manual tests should or should not be automated. Just because you can automate something doesn’t necessarily mean that you should. Also it is worth mentioning that development projects usually benefit a lot from test automation, but down in the bottom there is a glimpse on automation for projects that already in production.

So you’ve got a project in development and want to get automated tests running. You have some manual tests and struggling to determine which of them should be automated. Here some basic rules in such kind of a decision.

Tests that should be automated:

1. Tests that need to be run against every build/release of the application. (Usually it is smoke, regression tests, or sanity tests).
2. Tests that use the same flow but different data for its inputs for each test. (Programmed once they will use different datasets)
3. Tests that need to use information from different sources during run time, like databases (SQL/NoSQL) and low-level application attributes.
4. Tests that can be used for performance testing. (It is especially good running such kind of tests in parallel)
5. Tests that take a long time to perform and may need to be run during breaks or overnight. (Automating these tests maximizes the use of time)
6. Tests that involve large volumes of data. (Machines are much more faster in handling such a task)
7. Tests that need to run against multiple configurations. (Different OS, locales and their combination, different browsers)
8. Tests during which images and video must be captured or some low-level data retrieved to prove that the application behaved as expected.
9. Tests that should return results ASAP every run. (Usually automated tests much faster than same tests performed by manual tester).

Tests that should not be automated:

1. Usability tests (So far there is no way to program ISO/TR 16982:2002 and ISO 9241 efficiently).
2. Tests that you will only run one-time. (This is a general rule. If test steps could be automated quickly without need for support and run in a loop (for large dataset input for example) it will save a manual tester considerable time and effort).
3. Test that need to run ASAP. (As automating a test is not the fastest way to run it)
4. Tests without predictable results or on part of application that changes every once in a while. (For successful validation it needs to have predictable results in order to produce pass and fail conditions)
5. If a test needs to be manually watched to determine whether the result is correct.
6. Test that cannot be 100% automated should not be automated at all, unless doing so will save a considerable amount of time. (Large and routine test preconditions)

What other options you’ve got to make a good decision? Remember, that automation can increase productivity by reducing the time taken to perform repetitive tasks. Another useful point is that automated tests cannot get bored. Anyone who has to undertake repetitive tasks over and over again will get bored. This will slow down their performance and increase the risks of missed defects.

Besides, with the help of repeated automated smoke scenarios you could monitor your application availability for 24/7/365. Main difference of the automation approach is that the test checks your app during development process as well as after the release.

And last but not the least, the more often automated test is executed, the more efficient it becomes. So, if you have a project in production, now it is possible to decide whether automation is necessary or not.

The article is prepared by Yan Gabis

In this test automation cycle, I tried to give a detailed overview of the specifics of automation testing services. We went through the process steps, risks, returns. So far, you know that the process bases upon several customary activities:

• Automated test development
• Automated test support
• Test run and result analysis
• Test automation monitoring.

Though, any process doesn’t finish on receiving the result. The process should be evaluated while the team activities show certain KPI. Apart from the usual methods of KPI measurement, I would like to offer you metrics that can be necessary for the evaluation of the first three mentioned activities. Let’s make it clear where to draw attention.

We start with those necessary for automated test development

• Automated test coverage (automated tests vs. manual tests). The activity reflects the correlation between automated and non-automated tests. The metric has more an administrative approach than one describing the process specifics. Though used along with the next one it gives quite a transparent activity overview.
• Cost per automated test. Applying the metric you’ll notice that the price per one automated test decreases. The tendency of cost shrinking is the result of 2 factors: 1) automation solution. The efficient architecture allows engineers to use the code several times. 2) Human factor. People eventually start working faster after careful learning the system of automation solutions.

Earlier, in one of my previous posts of this cycle, I mentioned the time2market parameter, which is important for the efficiency evaluation. The parameter is closely connected to the activity of the test run and result analysis. But take it as the parameter of the process duration. Apply metrics on this step and analyze the time of automated test execution, use smoke test for instance.

The metric concerning the correlation of useful and irrelevant results gives important information. While applying the metric keep in mind the definition of useful and irrelevant results.

Useful vs irrelevant results

• Useful results: test pass, test failure caused by a defect.
• Irrelevant results: test failure – problems with the environment, data-related problems, changes in the application (business logic and/or interface).

The irrelevant results are those that decrease the automation efficiency from the economic and duration point.

The percentage of the irrelevant fails is also the data used in metrics. The acceptable percentage level depends upon the project, though the figure of 20% can be used a basic adequate level.

If the rate of the irrelevant results exceeds the acceptable level, pay special attention to the source of the issue. Lower in the diagram you can see how changed the automation process after elimination of the secondary issues. The diagram was made on one of our latest projects.

So in a nutshell, being aware of automation process specifics and methods of efficiency evaluation, you have the basic information necessary for tracking the process. Now you are able to find the reasons for your automation project success to keep up with and understand what and why went wrong. Still, there always are some things to know more and I am always ready to provide you with the necessary details.

Today test automation is seen as a strategic step and gains a lot of focus in software testing world. Applying to automated tests company saves costs, decreases overall time of testing cycle enhances product quality, and enables the staff to focus on testing details.

Nevertheless, there are risks associated with test automation; paying attention to each one of them you are able to manage the process successfully. When automated tests are under go, changes made in the interface can cause serious problems, like irrelevant test results.

The occurred problems cause additional costs to provide support and analyze outcomes.  Anyway, to prevent the problem you need some kind of mitigation plan. First of all, the development team should share the interface changes with QA team, and of course, monitor the irrelevant test results related to the changes.

Similar problems may occur because of changes in business logic. Apart from the irrelevant test results, support, costs increase, it can yield to changes in test data structure and can impact other cases. The cautions are pretty much the same: share the changes with QA team and monitor the irrelevant results.

Besides the changes huge number of defects in functionality can also pose threats, such as improper execution of some of the automated tests because of the defects in others (if the tests are interdependent). The consequences are quite unfavorable, first of all, the results are to be analyzed (tests should be covered manually), while the costs get increased and can be compared with manual testing efforts. Along with that test development is more expensive, when functionality is unstable.

Though, the challenge can be eluded, for that very purpose work out necessary rules or instructions and follow them. Firstly, automate critical path functionality (small number of test cases), then monitor irrelevant test results related to defects and, of course, monitor extra test development efforts for such cases.

There`s a risk of the issues related to the environment execution. For example, when background processes are scheduled on test server, it increases test execution time as application works slowly. . In the situation like this the plan of mitigation will consist of monitoring the irrelevant test results related to environment, and tracking the test execution time. The impact of the environment issues is quite small for web projects, though it can be serious for desktop and mobile applications.

Along with that all, parallel test execution may impact on test data, which causes again irrelevant test results. To preclude such problems, it is useful to create independent datasets and implement procedures, which reset datasets.

Mitigation plan is not the only prevention from the arising problems, along with that you are to care about the automation solutions (from technical point of view) that are based on principles that reducing the impact of risks, when they occur. Just keep in mind that is always better to prevent the risk than to remove the consequences.

In the previous article I went through the points, which describe the returns of test automation. Such parameters as improvement of quality rate (achieved by bigger scope & more often releases), decrease of test cycle, and, of course, bigger ROI determine automation efficiency in general.

The data on test automation effectiveness is collected through the whole process. So, when you urgently have to compose the process efficiency report you can apply to some basic metrics, which is the easiest and most rapid way to make a report.

When writing the report keep in mind, that it should be based upon the correlation of manual and automated testing. For an accurate summary deal with points like:

• Cost difference. The result cost of automated tests should be lower, than those of manual work. If the difference is obvious, a bit later you`ll see that the test automation pays its way, despite the large investment in test development. Moreover, when you know the release frequency and economy rate, it is possible to forecast, when the efforts get repaid.
• Duration. The duration of testing cycle in automation must be much less than in manual testing, due to that the results are gained sooner.

When you get to the report and calculations remember, that only a measurable software development process gets the accurate quantity indicators. If the report made to evaluate one BUILD, for example, is based upon the parameters of cost difference and duration, then the report describing the whole automation process is composed upon the points of ROI increase & result achievement.

• ROI is calculated with the use of a well-known formula: ROI = benefits/costs. Where COSTS are the expenses for test development, execution and support; and BENEFITS are the expenses for substituted works excluding the costs.

In practice, if the releases come out 4 times per week, the ROI is gained in 3-4 months, if the cover is correct. Test duration is also quite an important parameter, especially, if tests get parallelized the results are 4 times better. Test duration is easily calculated upon the tests that last for more than 4 hours. But it`s no use to apply to this parameter on short test, the information you get is not sufficient.

The diagram below shows the specifics.

• Time2Market. Due to test automation the result is gained much sooner, which gives the opportunity to accelerate Time-to-Market. If such parameters as development speed and application delivery speed to the production environment can ALWAYS be measured, you should definitely use them to prove Time2Market acceleration, otherwise there is no point in calculations.

The stated points are the general ones that used for description of automation results. Of course, you can apply to some specifics parameters like LOSSES caused by delay during the delivery or new version testing. This parameter can describe the efficiency rate with money equivalent making, which gives a better view of benefits you get.

The parameters can be calculated, even if they weren`t permanently tracked, but this approach is suitable if the report is needed ASAP, though for a more profound report (ROI analysis and time2market) you`ll need to track all the process changes constantly.

This is our first post from the “Test Automation Cycle”. The article series contains 6 pieces of writing telling about:

• test automation returns
• test automation risks
• test automation process overview

Usually, the expected benefit of test automation is, first and foremost, the reduction of costs. However, it would be a serious mistake to concentrate only on this aspect. It`s no doubt that automation is a proven way to achieve a high return on investment savings, but there are also other aspects associated with automation ROI.

Automation brings benefits because automated tests run quickly, can be run anywhere and are conditionally free. And when you start estimating automation ROI consider some major points. Let`s go through each of them, starting with economic effect.

Economic effect

In fact, automation really pays its way. The costs get reduced as automated tests are reused several times. The QA team members utilize the tests they had written once replacing useless manual work.

Scope optimization

Automation covers quite a substantial part of work. When automated tests are under the process, they release tester`s time. Hence, a tester can cover a more sophisticated project part with manual tests. Due to this approach, the team can deal with a bigger scope.

Quality improvement

Automated tests being launched many times on different steps of the software development cycle improve product quality. For example:

• On the development step automation tests are run to check integration after development work, it`s done on a regular basis. Test execution can be done after each commit with the same goal as for nightly builds.
• Tests can be executed by developers before delivery to QA department for further testing. This approach ensures concentrating on useful coverage so QA doesn’t need to spend time for accept a particular build for testing but concentrates on useful areas.

Moreover, the automated tests eliminate possible human mistakes, do not ignore or overlook the things that a real person can.

Test duration decreases

As automated tests are much FASTER than manual work, the duration of a test cycle decreases accelerating Time-to-market. Test automation is also a great advantage as it decreases the regression cycle, when hot fixes or patches need to be applied to a production platform.

The arguments for automation can often be entangled and forcible. Moreover, accurate calculation of automation effectiveness requires much effort and a measurable software development process, though sometimes it can be hard on every project.

In my next article I`ll give a detailed overview on metrics on test automation returns.