Rainar Ütt is Head of Quality Assurance at InnoGames. He considers his major tasks in developing the integrated approach, enabling company to bring high quality software faster to the market. He explains, “The job at InnoGames is an appealing task in a very dynamic market environment. I like the international appeal of the company and I am looking forward to contributing further to the success of the company, as I believe that high quality standards are one of the key factors for online games.”
Rainar also is a frequent speaker. Recently he participated “Iqnite 2014” – International Software Testing Conference in Dusseldorf and presented his report “Quality management in online games development”. This presentation gives an overview on how online games are developed and tested, what are the bottlenecks and how to survive with Quality Management in such environments. A1QA together with Rainar Ütt decided to have a closer look to testing issues in gaming industry.
A1QA: Оnline gaming industry (according to global statistics) is just on the top of its popularity nowadays. The worldwide video game marketplace will reach $111 billion in 2015, up from $79 billion in 2012, according to Gartner, Inc. Do you consider this trend as a longtime tendency?
Rainar Ütt: Online gaming industry has been growing. As more and more people are getting online and computer hardware is getting faster so you don’t need specialized hardware anymore. If it has reached the top – I cannot predict that, in the 80’s there were consoles, PC games in 2000’s browser games have been taking over and the future is obviously mobile. As technology is getting more and more affordable and people are getting more online, this trend will definitely continue.
A1QA: It is well-known that Quality assurance is a critical component in game development, though the video game industry does not have a standard methodology. Why Quality is so important for gaming and how to evaluate the scale of troubles undetected defects may cause comparing to other fields?
Rainar Ütt: Of course you cannot compare online gaming to software development for healthcare, automotive or space industry, where the quality assurance is affecting human lives. In games development worst effects software defects might cause are related to privacy and data protection and security. Could you imagine someone stealing millions of credit card details from the database or taking control over your computer, by using a cross site scripting vulnerability on game provider’s website?
Also when, for example, your online game is not responding fast enough. Studies have shown if a web browser request is taking more than 2 seconds, customers would prefer to go to a competition rather than staying, that is why web app testing has become crucial to ensure ultimate performance.
Or if due to data inconsistency or functional problem your co-player is gaining advantage of you, while you should have it. In online gaming – defects in your system would have a really strong effect on your business. In banking, if your online bank is having defects, you still can retain your customers, because not all of them are using your online bank. So I would rank the importance a little bit higher than in a bank.
A1QA: Since game QA is considered less technical than general software QA, some people assume game testing as some kind of pure fun while we know this is not the case. Since you are just in the middle of such amazing industry as gaming could you please describe the way games are usually being tested in InnoGames? How do you think this process might be enhanced?
Rainar Ütt: Yes indeed, game QA is often considered as only playing games and reporting defects. At InnoGames we are keeping a healthy balance between game testing and technical QA. When a game project is in the very beginning, concepts are developed and changed rapidly. Often it also happens that a concept is discarded completely. Therefore in the beginning of the game project there is less focus on the technical aspects of QA. In this phase it is important to develop acceptance criteria and checklists for functional testing.
Later, when the architecture evolves and not so many conceptual changes are introduced, it is important to start looking at quality of different components and the integration between those. For core user stories, we are also introducing automated GUI tests, those are then run after automated unit and integration tests are finished at least nightly.
Of course there is still manual testing involved as well – when new features are being developed, tester is giving feedback among other testing types by exploratory testing. Testers also write acceptance criteria, or let me put it that way: information for developers how they are planning to test it, so that developers will make sure to do basic testing on the feature already before it gets integrated.
Cross functional or non-functional testing we are usually starting, a few months before the launch. The architecture of online games is usually quite simple on the backend side, so this process is usually quite straightforward.
Our game testers and tech testers, both work as part of the development team, each product development team has based on their needs game testers and technical QA leads working as part of it.
A1QA: What are the most effective testing tools you have been using for game QA? Do you actually think Test automation is applicable for online games development as well? In other words, is it possible to automate some aspects of this process?
Rainar Ütt: From one side online games are games, from other side they are software – to be more specific we are dealing with mobile and web applications and there is a great variety of free or commercial tools available. Best experience we have made so far with selenium. Our customer base is over 120 million, worldwide, so for us cross browser and cross market testing is a really hot topic. We are using Thucydides framework, which enables us data-driven testing and reporting capabilities. We have created a small core framework that takes care about the configuration of selenium, integration with different tools. For example reporting into our test management tool (Testlink), launching the tests on cloud testing service provider. For maintaining the framework our automation architect takes care of that and all of our tech QA-s write selenium tests with Java.
Of course selenium cannot cover everything, especially frontend technologies, such as flash, which still will remain one of the major technologies in browser game development. For flash we are using a combination of 2 tools: Genie – tool from adobe that can interact with objects in Flash and Sikuli – a screenshot recognition tool. For mobile end to end tests, there was unfortunately nothing on the market, what we could use – so we are in the process of building our own tool that can interact with cocos2dx and integrate it into a commonly used testing tool, either Monkeytalk or Seetest.
For API testing we are using SOAPUI, for cross functional testing JMeter. We have also built our own tools for measuring the frame rate on mobile apps or improved JMeter websocket plugin.
For testing online games automation is definitely possible, sometimes getting started with it is a little bit difficult, because of the skills of the people. We’ve been through that process with InnoGames, when I started here we only had a manual functional QA, which was getting involved in the development process in the very end. During one year, we managed rearrange the way QA is working – instead of a separate team, we introduced QA as part of development teams and became more technical. We made several workshops, where we learned about automation, trained our Java skills, data-driven testing and eventually also developed an Android application. I think for those saying automation is not possible in games development; it is rather a question of mindset.
A1QA: So as A1QA has some experience testing online games like Roads of Rome and Islands Tribe we know exactly how popular they become being properly tested. Can you give us your examples to which extent effective Quality Management influences the success videogames are getting on the market?
Rainar Ütt: For a successful game launch, there are several factors that make it successful. You have to have a brilliant game concept, find the right marketing channels and of course quality.
As mentioned above, if you product response time is too slow, or if you are not able to keep the players interested for playing more than 2 rounds, if the player data isn’t safe, if the graphics aren’t awesome, if the customer cannot have it in its local language – the product won’t be successful. Also another aspect is predictability – when your product is going to be launched. A good timing for a product / feature launch is important and if you can keep the deadline – for example, launching an eastern event in midsummer, wouldn’t bring the same amount of customers as during Eastern Time.
Rainar thanks for sharing your experience and ideas. We`ll be glad to see you and talk to you again.