Blog

Assuring quality of mobile app: how we at a1qa tested Pokemon GO

a1qa couldn’t help testing the augmented-reality app Pokemon GO. Read about the test process and defects our team discovered.
6 March 2017
Mobile app testing
Usability testing
The article by Pavel Novik
Head of testing department at a1qa

In the last article on mobile apps testing we’d like to tell you how we tested on our own accord Pokémon GO that became a smash hit in 2016.

It should be immediately noted that we performed testing at the beginning of August, immediately after the game was released to the market. To date, many of the defects have been fixed. But the objective of the article is not to list the relevant defects but rather what to focus on while assuring quality of mobile apps.

By the way, we recommend mobile testers to test the existing apps from time to time. It helps hone the outside-thinking and investigation skills.

So, last year in July we downloaded the app to our smartphones to immerse into those times when at school we collected Pokémon pogs and rushed home to watch Ash and Misty’s adventures.

While testing we discovered that the games has its fair share of bugs and glitches. What have we found out?

First, we gathered all the information we could about the product.

What did we know about Pokémon GO?

  • Multiplayer online role playing game.
  • Free-to-play.
  • It uses real location information to encourage players to search far and wide in the real world to discover Pokémon.

What else? We also knew that the app was mainly targeted at the users from the USA, Australia and New Zealand and was built on the Unity game engine.

It supported iOS 8 and 9 versions and Android 4, 5, 6. Keep it in mind that testing was conducted in August, and it was still a month before iOS 10 was released.

It’s an important moment that testing on tablets wasn’t required as the app wasn’t adapted to tablets.

To detect user’s location the app would use the GPS module and constant internet connection so these functionalities also had to be tested.

Having analyzed the info and the available statistics we made up a list of the following test configurations:

For iOS we included the devices starting from iPhone 5.

Android devices were more numerous. This was due to the vast number of versions of the system and various devices popular in different regions. We also managed to check the app on 5 different screen resolutions.

What defects did we find in Pokémon GO?

For a number of reasons, such as no access to the app backend (the customer is hypothetical) and no opportunity to get the scope of test data (and test geolocation) we could hardly detect all defects. But even those that we were able to detect would have been enough to compile a multipage test report.

All the defects we grouped according to their severity level:

Blocked defects:

  • Authorization Failure. It seemed like every second user of the Pokémon GO faced this issue. And it wasn’t not clear whether the problem’s roots lied on the client’s or server’s side.
  • Another frequently reported bug took place when the user tried to create the in-game character. The mistakes of the indistinct nature prevented users from starting the game.

Critical defect:

  • The so-called three-step glitch. Pokémon GO is the augmented reality game. It tracks the nearest monsters and the user has to catch them. Initially, the steps printed next to the Pokémon would show the user how far the monster was. Thanks to the three-step glitch this feature became useless. This glitch showed Pokémon in the vicinity as always being three-steps (three paw prints) away, no matter how much you should walk.

As well experienced testers we proved all the defects with screenshots.

Major defects:

  • Broken UI. Relaunching the app or clearing the cache would resolve this issue.
  • No in-game characters and Pokémons images. However, by some indirect indicators we guessed that they should be there.
  • AR mode issues. It turned out that many users couldn’t use the mode. We guess that the essence of the problem was connected to the software and hardware specifications.

Some average defects were related to the application several functions. Catalogue challenges, frequent camera issues and GPS module problems spoilt the monster-catching experience.

Moreover, there were minor defects that would rather make user smile than swear. For example, UI defects (look at the screenshot below) and the opportunity to exceed the number of items in the inventory.

Shall the customer be real, we would have generated the detailed bug report and product quality report. We would also perform another iteration of testing.

Pokémon GO became widely available in July last year and generated $200 million in revenue within 30 days of launch. The game surpassed Candy Crush Soda Saga, the top grossing mobile games of all times.

However, it was unable to keep up the momentum and in September already more than 10 million users stopped entering the game. The reasons are many, but the detected defects can be among them. Who knows how many users would have preserved their loyalty to the game but for all these bugs?

That’s it. We’ve taught the theory and tested the app. We hope you’ve enjoyed the read and will take advantage of it.

Any questions left? Welcome to the comments!

More Posts

The year in valuable conversations: recapping 2023 a1qa’s roundtables for IT executives 
8 December 2023,
by a1qa
3 min read
The year in valuable conversations: recapping 2023 a1qa’s roundtables for IT executives 
From dissecting novel industry trends to navigating effective ways of enhancing software quality — let’s recall all a1qa’s roundtables. Join us!
Big data testing
Cybersecurity testing
Functional testing
General
Interviews
Performance testing
QA trends
Quality assurance
Test automation
Usability testing
Web app testing
black-friday
22 November 2023,
by a1qa
4 min read
Get ready for Black-Friday-to-Cyber-Monday shopping: 5 testing types to include in your QA strategy
What’s your nightmare during Black Friday and Cyber Monday shopping? If it’s a loss of sales, read about the ways to prevent this in the article.
Cybersecurity testing
Functional testing
Localization testing
Performance testing
Usability testing
3 November 2023,
by a1qa
4 min read
From idea to buying: 7 testing types to make your mobile eCommerce solutions flawless
Read the article to discover 7 QA activities helping boost mobile eCommerce solutions quality and provide end users with exceptional buying experiences.
Functional testing
General
Quality assurance
Test automation
Usability testing
streaming services
30 October 2023,
by a1qa
3 min read
Enable crash-proof streaming platforms for Holidays season
Ho ho ho! It’s time to prepare your streaming products for the influx of viewers. Read about how to put peak-load anxiety behind by applying rigorous testing of your IT solution.
Cybersecurity testing
Functional testing
Performance testing
Usability testing
On the way to Web 3.0: key software testing aspects for seamless digital experiences. Part 2
12 October 2023,
by a1qa
4 min read
On the way to Web 3.0: key software testing aspects for seamless digital experiences. Part 2
Let’s analyze essential software testing checks to improve the quality of the business-critical Web 3.0 functionality.
Cybersecurity testing
Functional testing
Performance testing
Quality assurance
Test automation
Usability testing
gaming-qa
24 August 2023,
by a1qa
4 min read
Ready, steady, test: How QA drives seamless gaming experiences
Why is QA pivotal for delivering unmatched player experiences? How to level up video game quality? Find the answers in the article.
Cybersecurity testing
Functional testing
Localization testing
Performance testing
Quality assurance
Test automation
Usability testing
6-march-2023-1
21 March 2023,
by a1qa
4 min read
The ultimate QA guide for smoothly migrating to Web 3.0
Find out how businesses can seamlessly migrate to Web 3.0 by relying on quality assurance.
Cybersecurity testing
General
Performance testing
Usability testing
Mobile app testing
15 February 2023,
by a1qa
4 min read
Mobile app testing guide: win the race with five-star software
Which aspects of mobile apps to test first to produce a really high-quality product? Find the answer to this and other questions related to mobile app testing in the article.
Cybersecurity testing
Functional testing
Mobile app testing
Performance testing
Test automation
Usability testing
eLearning and mLearning QA_mini
22 July 2022,
by a1qa
4 min read
6 must-have testing types for eLearning and mLearning software
Discover how to ensure top-notch educational solutions through QA, provide end users with boosted studying experience, and make the grade in the IT market.
General
Localization testing
Mobile app testing
Performance testing

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.