Blog

How to use computer vision to improve your test automation

Learn how computer vision helps automate Android and iOS mobile application testing.
14 November 2018
Mobile app testing
Test automation
The article by a1qa
a1qa

Computer vision is used in a variety of modern devices and security systems. However, its application in test automation isn’t widespread. Most often verifications are limited to comparing two images or searching for a particular element in an overall picture. Is it possible to go beyond this? Is there a way to make automated tests ‘see’ all objects on the screen of your tablet or smartphone, for instance?

Read our article to learn how to streamline mobile test automation through computer vision, what tools to choose, and why its application sometimes can be restricted.

In most cases, to automate testing of iOS and Android mobile apps Appium is used, which is a solid tool. However, the scope of its possibilities is at times limited.

For instance, automatic search of certain elements within the same image can become a challenge for Appium.

The issue is that it can’t track location of individual elements of an image on the screen unless developers have previously added the corresponding unique addresses for each of them. Appium will overlook them.

What is the way out? Make use of computer vision.

Computer vision potential

Computer vision is a scientific discipline that allows machines to ‘visually’ analyze their environment. Images/videos usually form the core of this process.

Nowadays, the technology allows recognizing people by their faces and postures, helps unmanned vehicles to distinguish road signs and identify pedestrians accurately. It is also used to process images in medicine and support manufacturing.

This technology serves to solve the following tasks:

  • Object identification

Computer vision assists to detect whether the video contains a particular object. Images search by their content, the assessment of the object’s location, and the defining of characters in the text also belong to the list of this function.

  • Estimating speed of moving objects

This point presupposes processing a sequence of images to determine the speed of each point of the image.

  • Image recovery

Its core goal is to remove noise (fuzzy image of an object in motion).

  • Scene restoration

If there are more than two scene models, it’s possible to recreate its three-dimensional model.

Mobile testing automation using computer vision

Furthermore, the technology under discussion serves to automate the mobile software testing process.

In automation, it is used specifically for searching for individual elements in the overall image. It is necessary to pre-cut these elements and store them. This process isn’t appropriate for mobile testing automation on devices with different screen resolutions.

Therefore, the automation of this process will help to eliminate the need to manually cut elements out, thus optimizing the workflow significantly.

How to find the necessary elements in the whole image?

Computer vision tools serve this purpose. Combining them with Appium, test automation engineers solve the existing issue.

Let’s discuss each tool in detail.

Overview of computer vision tools

OpenCV

This С/С++ based tool is an open source library that contains computer vision and images processing algorithms. It can also be applied for Java, Python, Ruby, and some other languages.

OpenCV performs image binarization by converting a full-color display to a monochrome with only two types of pixels (dark and light).

Binarization is performed by either an adaptive or a threshold method. The first one is applied only to heterogeneously lighted areas of the image. The second one detects a certain threshold, which allows dividing the image into black and white.

Built-in algorithms separate the background from the object itself, as well as search for individual elements by contours. Open source code allows combining these methods for a specific solution.

SikuliX is an alternative tool when searching for an individual element within the overall image. It represents an open cross-platform script development environment focused on programming GUIs using screenshots.

For scripts creation one may use such languages as Ruby and its implementation jRubi, Jython, constructions from Python. The program is applied for Windows, Mac OS X, Linux. SikuliX is based on OpenCV algorithms.

Tesseract OCR

This is one of the most popular and high-quality programs for text recognition from images. It is written in C/C++ and is suitable for Mac OS, Windows, and Linux OS.

This tool supports new programming languages and fonts, while neural network forms the basis of an updated product version 4.0.

Leptonica (image processing library), the latest version of Tesseract OCR, and data for learning the desired language (for example, Spanish) are necessary to start working with this solution.

When combining these tools with Appium, the automated testing of a mobile application looks like this.

A screenshot of an image is created automatically, then it is divided into separate elements. Each of them is added to a configuration file. Engineers already have the coordinates of each element.

After that Appium accesses the configuration file, obtains the necessary information, and interacts with these elements using their coordinates.

Why is computer vision not a versatile solution?

Despite its vigorous development, QA and Dev teams can’t implement it on a daily basis.

It happens due to the following factors:

  • Dependency on the image quality

Sometimes the quality is quite low. That’s the reason why computer vision tools fail to provide an accurate result.

  • Inability to automate all test cases

It happens due to the existing limitations in the process of recognizing animated elements.

To sum up

Appium is a proven and robust mobile test automation tool. However, sometimes test automation engineers opt for combining it with computer vision instruments in order to elaborate a sustainable solution.

The one that will provide you with exact and quick results and can be extended to multiple applications with minimal effort.

Contact us to get more information on test automation service.

More Posts

Shift-right testing
9 July 2024,
by a1qa
5 min read
Embracing shift-right testing to enhance software quality through real-world insights
Let’s reveal the power of shift-right testing for ensuring high software quality.
General
Test automation
27 June 2024,
by a1qa
3 min read
Establishing seamless interaction between development and QA teams to boost productivity
Establishing seamless interaction between development and QA teams to boost productivity
Agile
General
Quality assurance
Test automation
17 June 2024,
by a1qa
5 min read
Shifting to test automation to maximize software quality
Explore in the article why businesses should move from manual testing to test automation.
Quality assurance
Test automation
RPA in QA
28 May 2024,
by a1qa
4 min read
Embracing robotic process automation to drive efficiency in QA
Discover how the convergence of robotic process automation helps reshape software testing practices.
General
Quality assurance
Test automation
Shift-left testing for better software performance
25 April 2024,
by a1qa
4 min read
Optimizing software performance with shift-left testing
Still in doubt whether to include performance testing from the initial development stages? Learn the benefits companies obtain with shift-left performance testing.
Performance testing
QA consulting
Quality assurance
Test automation
Telecom trends 2024
15 April 2024,
by a1qa
5 min read
QA’s role in adopting telecom trends for 2024 
Let’s dive into the transformative trends set to redefine the telco industry in 2024 and discover QA strategies to adopt them with precision.
Cloud-based testing
Cybersecurity testing
Functional testing
General
Migration testing
Performance testing
QA trends
Quality assurance
Test automation
Enhancing Agile and DevOps processes
28 February 2024,
by a1qa
4 min read
4 actionable tips to enhance Agile and DevOps processes 
Streamlining Agile and DevOps workflows? Learn practical recommendations on how to achieve this.
Agile
General
Test automation
Navigating the future: QA trends that will define 2024. Part 1
15 January 2024,
by a1qa
4 min read
Navigating the future: QA trends that will define 2024. Part 1
Discover topical software testing trends that will shape 2024 and empower companies to smoothly implement advanced technologies.
Agile
QA trends
Quality assurance
Test automation
The year in valuable conversations: recapping 2023 a1qa’s roundtables for IT executives 
8 December 2023,
by a1qa
3 min read
The year in valuable conversations: recapping 2023 a1qa’s roundtables for IT executives 
From dissecting novel industry trends to navigating effective ways of enhancing software quality — let’s recall all a1qa’s roundtables. Join us!
Big data testing
Cybersecurity testing
Functional testing
General
Interviews
Performance testing
QA trends
Quality assurance
Test automation
Usability testing
Web app testing

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.