Blog

Assuring quality of mobile app: how we at a1qa tested Pokemon GO

a1qa couldn’t help testing the augmented-reality app Pokemon GO. Read about the test process and defects our team discovered.
6 March 2017
Mobile app testing
Usability testing
The article by Pavel Novik
Head of testing department at a1qa

In the last article on mobile apps testing we’d like to tell you how we tested on our own accord Pokémon GO that became a smash hit in 2016.

It should be immediately noted that we performed testing at the beginning of August, immediately after the game was released to the market. To date, many of the defects have been fixed. But the objective of the article is not to list the relevant defects but rather what to focus on while assuring quality of mobile apps.

By the way, we recommend mobile testers to test the existing apps from time to time. It helps hone the outside-thinking and investigation skills.

So, last year in July we downloaded the app to our smartphones to immerse into those times when at school we collected Pokémon pogs and rushed home to watch Ash and Misty’s adventures.

While testing we discovered that the games has its fair share of bugs and glitches. What have we found out?

First, we gathered all the information we could about the product.

What did we know about Pokémon GO?

  • Multiplayer online role playing game.
  • Free-to-play.
  • It uses real location information to encourage players to search far and wide in the real world to discover Pokémon.

What else? We also knew that the app was mainly targeted at the users from the USA, Australia and New Zealand and was built on the Unity game engine.

It supported iOS 8 and 9 versions and Android 4, 5, 6. Keep it in mind that testing was conducted in August, and it was still a month before iOS 10 was released.

It’s an important moment that testing on tablets wasn’t required as the app wasn’t adapted to tablets.

To detect user’s location the app would use the GPS module and constant internet connection so these functionalities also had to be tested.

Having analyzed the info and the available statistics we made up a list of the following test configurations:

For iOS we included the devices starting from iPhone 5.

Android devices were more numerous. This was due to the vast number of versions of the system and various devices popular in different regions. We also managed to check the app on 5 different screen resolutions.

What defects did we find in Pokémon GO?

For a number of reasons, such as no access to the app backend (the customer is hypothetical) and no opportunity to get the scope of test data (and test geolocation) we could hardly detect all defects. But even those that we were able to detect would have been enough to compile a multipage test report.

All the defects we grouped according to their severity level:

Blocked defects:

  • Authorization Failure. It seemed like every second user of the Pokémon GO faced this issue. And it wasn’t not clear whether the problem’s roots lied on the client’s or server’s side.
  • Another frequently reported bug took place when the user tried to create the in-game character. The mistakes of the indistinct nature prevented users from starting the game.

Critical defect:

  • The so-called three-step glitch. Pokémon GO is the augmented reality game. It tracks the nearest monsters and the user has to catch them. Initially, the steps printed next to the Pokémon would show the user how far the monster was. Thanks to the three-step glitch this feature became useless. This glitch showed Pokémon in the vicinity as always being three-steps (three paw prints) away, no matter how much you should walk.

As well experienced testers we proved all the defects with screenshots.

Major defects:

  • Broken UI. Relaunching the app or clearing the cache would resolve this issue.
  • No in-game characters and Pokémons images. However, by some indirect indicators we guessed that they should be there.
  • AR mode issues. It turned out that many users couldn’t use the mode. We guess that the essence of the problem was connected to the software and hardware specifications.

Some average defects were related to the application several functions. Catalogue challenges, frequent camera issues and GPS module problems spoilt the monster-catching experience.

Moreover, there were minor defects that would rather make user smile than swear. For example, UI defects (look at the screenshot below) and the opportunity to exceed the number of items in the inventory.

Shall the customer be real, we would have generated the detailed bug report and product quality report. We would also perform another iteration of testing.

Pokémon GO became widely available in July last year and generated $200 million in revenue within 30 days of launch. The game surpassed Candy Crush Soda Saga, the top grossing mobile games of all times.

However, it was unable to keep up the momentum and in September already more than 10 million users stopped entering the game. The reasons are many, but the detected defects can be among them. Who knows how many users would have preserved their loyalty to the game but for all these bugs?

That’s it. We’ve taught the theory and tested the app. We hope you’ve enjoyed the read and will take advantage of it.

Any questions left? Welcome to the comments!

More Posts

5G impact
31 May 2021,
by a1qa
4 min read
5G network impact on mobile app testing
Check out what 5G connectivity will bring to the IT world and how it will modify mobile app testing.
Cybersecurity testing
Mobile app testing
Performance testing
15 March 2021,
by a1qa
4 min read
Mobile app performance testing: getting high software efficiency
Explore 3 cornerstones of mobile app performance testing and QA steps on how to execute it successfully.
Mobile app testing
Performance testing
29 January 2021,
by a1qa
4 min read
3 do’s and 3 don’ts in BFSI software testing
Considering BFSI to be a fast-paced industry, how to keep up with such velocity? We’ve prepared 3 do’s and 3 don’ts that help sustain the rush and high software quality.
Functional testing
Mobile app testing
Test automation
13 November 2020,
by a1qa
5 min read
QA for media and entertainment
Read the article to explore why QA is a must for the media and entertainment sector and how to perform software testing effectively.
Functional testing
Mobile app testing
Performance testing
Test automation
Usability testing
24 July 2020,
by a1qa
4 min read
OWASP as a guide to mobile apps security testing
More apps, more sensitive data, higher security levels... Learn how companies address the challenge of providing secure solutions harnessing unbiased safety recommendations.
Cybersecurity testing
Mobile app testing
29 November 2019,
by a1qa
3 min read
4 cases when you may think you need usability testing but you don’t
Is user experience testing always the one your software product needs? Read our blog post to figure it out. 
Localization testing
Performance testing
Software lifecycle QA
Usability testing
12 December 2018,
by a1qa
3 min read
Benefits of native mobile app testing on real devices
At the early stage of every QA project, the team should make a choice: run tests on emulators, simulators, or real devices.
Mobile app testing
14 November 2018,
by a1qa
4 min read
How to use computer vision to improve your test automation
Learn how computer vision helps automate Android and iOS mobile application testing.
Mobile app testing
Test automation
20 July 2018,
by a1qa
4 min read
How QA providers help telecom companies overcome 21st century challenges
Let's examine the most vital telecom challenges and ways to overcome them with the help of Software Testing and QA companies.
Mobile app testing
Quality assurance

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.