Assuring quality of mobile app. stage 1: strategy

a1qa Company specializes in assuring quality of software products. Mobile apps testing is one of the main services with more than 220 company’s employees certified to engage in mobile testing projects.

Today we offer you a series of articles prepared by Pavel Novik, head of the Mobile Testing Center of Excellence. The series will be made up of 5 articles. One article is mapped to one project stage. We’ll start with the proper strategy development. Then we’ll touch upon test documentation preparation and testing itself. Product release and associated testing activities will be the subjects of the fourth piece. And to make it more interesting for you to cope with all this amount of theory, we’ll tell you how we’ve decided to test the popular mobile game of 2016 and what bugs we’ve discovered.

So let’s imagine that we’ve been addressed to assure quality of some mobile application. The customer’s goal is to release the product to the market without defects.

We’ll start with a strategy. In our context strategy is the selection of testing devices and planning of all testing activities. The strategy should be presented and agreed upon with the customer.

The first step is to analyze the product and get answers to the following questions:  

• What are the project goals?
• What is the target audience of the product?
• What are the app’s technological specifications?

The answers to these questions may help estimate the project scope and assign it to the employees with the right set of skills.

What devices to test on?

Once we’ve received questions to the answers listed above, we have to compile a list of the rights devices to test on.

Consider the following factors:

• Customer’s preferences
• Target markets/regions
• Target audience
• Application specs (oldest and latest OS versions supported, app requirements to hardware, etc.)
• Budget limits and timeframes.

Let’s analyze every point separately.

1. If the app is already available live, the customer may have gathered stats about the user’s devices. Some customers also have their own preferences about the testing devices. Check if there are any in your case.
2. Target region also matters. If the app is aimed at the Asian region, give attention to the Chinese smartphones Xiaomi and Meizu. In Europe Samsung devices are more popular, while in the USA Google (Nexus) and HTC devices hit the market. However, making guesses isn’t a good way. Analyze statistics to make the informed decision.

There are numerous free resources that enable to see, compare and analyze the list of the most and least popular devices. Consider several of them and compile the final list. Some of the most popular are Antutu, Akamai, DeviceAtlas, Futuremark, etc.

For example, the most widely used in North America in December 2016 are the following:

3. Knowing target audience will help to limit the number of devices according to their “age”. Statistically, the youth prefer to use the latest devices, while the elder generation is loyal to the smartphones and tablets of 3-4 years on the market.
4. Analyze the hardware and software specifications and remove the outdated or, vice versa, most recent devices that aren’t supported by the app from the list.
5. And the last but not least important point is to comply with the budget and time limits. Sometimes you’ll find it necessary to test on 10-20 devices to provide full testing coverage. However, it may be impossible due to tough deadline or limited budget.

At this first stage you may come to face some odd questions from the customer. You’ll have to deal with them to avoid misunderstandings and make sure that the final results respond to their expectations.

Most commonly the customer will ask you about testing devices. “Why do you need so many devices?” “Why will it take you so long to click through the app?” Don’t grudge the time to answer them. A brief presentation about testing process, number of checks and the consequences of reducing the number of devices may be of help.

Initiate discussion on other options if you don’t have the required device for testing. Can the customer provide you with it? Can it be replaced by something else?

After addressing all the questions and gathering stats, we start drawing up a schematic matrix of configurations we want to test on. A test configuration is a combination of variable values (type of device, operating system, browser, CPU, database, etc.)

Every device type (smartphone, tablet, wearable devices) should be represented by various screen sizes and operating system types. Most commonly specific defects will depend on the operating system type. That’s why we recommend including the latest versions of OS in configurations (9.0 an 10.0 for iOS, 4.0, 5.0 and 6.0 for Android). Also don’t forget about beta updates. Include them into test plan.

Questions to ask the customer before testing

Is the app integrated with the third party services? What is the application monetization strategy? If it’s applicable, schedule integration and payments testing.

Was the application tested before? Previous testing results will be useful to estimate the quality of the app and to analyze the problems that were addressed by the predecessors. Moreover, ask the customer whether he was satisfied with those results.

Does the app support multiple languages? Then it may be worthwhile to perform localization testing. Negotiate it with the customer.

Finally, don’t forget to request access to the back-end and test accounts. It can be especially useful if you’re engaged in testing game app. Very often it’s not enough to pass the game to the end. Accessing the backend, you’ll be able to generate various playing scenarios and in-game events, say nothing about saving time.

When we’ve discussed all questions it’s time to start planning future testing activities, i.e. preparing of test documentation and assigning QA engineers. This is what we’ll clarify next week.