Software testing: how to choose what to test
Nadya Knysh – Managing Director at a1qa – on how to decide on the right amount of QA and choose the right test strategy.
Is it possible to choose the single right amount of quality assurance? The simple answer here is NO. And here is why.
There are multiple factors that impact the scope of testing and your testing strategy.
#1 Solution scope
In my experience, about 80% of software projects are delivered using Scrum as the project methodology.
Scrum (like most Agile frameworks) is all about change. New requirements are coming in every sprint, priorities are changing, development is following business needs…
What does that mean for QA?
Well, your planning horizon is pretty limited. Of course, you’ll define your team capacity after a few sprints.
However, you cannot be 100% sure of what skill set may be required in the next sprint – are we testing APIs or UX? Will we have new devices in the office by the time we need to test against them? Will the development team’s capacity impact the deliverables dates? All these questions impact your ability to plan and predict how much QA you’ll need and when. That doesn’t mean you shouldn’t plan, though.
#2 External factors
Imagine you have a one-year roadmap to deliver a brand-new solution to the market.
It’s December 2018 when you start developing, which means Apple has already released its 2018 updates while Samsung is still working on its presentation for February 2019.
Oh, and by December 2019, when you plan to release, Apple will have delivered another presentation on 2019’s new features, hardware, screen sizes, and who knows what else.
So, when planning QA, you should look at up-to-date statistics on what devices your target audience is using to interact with your software.
By the way, the iPhone 7 is still pretty popular, so your first thought might be to include it on the list of supported devices.
However, when thinking from a strategic perspective, by December 2019, the iPhone 7 will be three generations old. So the question is – do you care about it? And what if you do all your development and testing around the Apple Watch and then, a few months before the release, Apple recalls all the devices due to some critical issues?
So, my recommendation is: plan but be flexible. It is not the strongest of the species that survives, nor the most intelligent. It is the one that is most adaptable to change.
What are the most important quality aspects to be tested?
That’s a very good question! But I have found a very simple answer.
A long time ago, I was talking to a solution architect who was developing the architecture for a SaaS solution: highly secure, customizable, and with (hopefully) high performance.
When he was walking me through the architecture ideas (in other words, a draft version), he was explaining why he needed this feature and that feature here and there through quality attributes – a concept well defined in ISO 25010.
The full list of quality attributes is now available all over the Internet. So now, when defining a QA strategy for any of our projects, I recommend reviewing this list again and asking a question: is that important for my product now? In most cases, your answer will be YES in terms of functionality and UI/usability. That’s why you’ll plan functional, UI, compatibility, and usability tests.
Compatibility is the one that is often overlooked.
Always remember that you cannot make your audience use the same browser or smartphone that you do. You have to accept their choice and support whatever they like.
Side note: check your audience geography; people in China use very different smartphones from those used by people in the US.
Let’s say you develop an internal accounting solution for your CFO and two to three other analysts or accountants.
In this case, if one report that you only generate annually takes five minutes to generate, you may not care that much about the solution performance. But what if your solution is a stock market software? Performance is critical here: the market changes in nanoseconds and five minutes will cost you a fortune.
Security is another big topic. Do you store personal information (like SSN and DOB) or your customers’ credit card details? Oh yeah, security testing is a must.
And believe me, you’d rather be compliant with HIPAA, FDA, PCI-DSS, and other regulations than go to court against your customers.
Summing up
To test or not to test isn’t a question in our digital age. Now the focus has shifted toward choosing the right testing strategy that will meet the requirements of the software developed, end-users, and business stakeholders.
Flexible planning and setting priorities related to your product is what will help you make the smart choice.
