Nadya Knysh – Managing Director at A1QA – on how to decide on the right amount of QA and choose the right testing strategy.
Is it possible to choose the single right amount of quality assurance? The simple answer here is NO. And here is why.
There are multiple factors that impact the scope of testing and your testing strategy.
#1 Solution Scope
In my experience, about 80% of software projects are delivered using Scrum as the project methodology.
Scrum (as with most Agile frameworks) is all about change. New requirements are coming in every sprint, priorities are changing, development is following business needs…
What does that mean for QA?
Well, your planning horizon is pretty limited. Of course, you’ll define your team capacity after a few sprints.
However, you cannot be 100% sure of what skill set may be required in the next sprint – are we testing APIs or UX? Will we have new devices in the office by the time we need to test against them? Will development team capacity impact the deliverables dates? All of that impacts your ability to plan and predict how much QA you’ll need and when. That doesn’t mean you shouldn’t plan, though.
#2 External Factors
Here is the thing that happens on each and every project. Imagine you have a one-year roadmap to deliver a brand-new solution to market.
It’s December 2018 when you start developing, which means Apple has already released their 2018 updates while Samsung is still working on their presentation for February 2019.
Oh, and by December 2019, when you plan to release, Apple will have delivered another presentation on 2019’s new features, hardware, screens sizes, and who knows what else.
So, when planning QA, you’d normally look at the up to date statistics on what your target audience is using to interact with your software.
By the way, the iPhone 7 is still pretty popular, so your first thought is to include it on the list of supported devices.
However, when thinking from the strategic perspective, by December 2019 the iPhone 7 will be three generations old. So the question is – do you care about it? And what if you did all your development and testing around the Apple Watch and then, a few months before the release, Apple recalls all the devices due to some critical issues?
So, my recommendation is: plan but be flexible. It is not the strongest of the species that survives, nor the most intelligent. It is the one that is most adaptable to change.
What are the most important quality aspects to be tested?
That’s a very good question! But I have found a very simple answer.
A long time ago, I was talking to a solution architect who was developing the architecture for a SaaS solution: highly secure, customizable, and with (hopefully) high performance.
When he was walking me through the architecture ideas (in other words, a draft version) he was explaining why he needed this and that here and there through Quality Attributes – a concept well defined in ISO 25010.
The full list of quality attributes is now all over the internet. So now, when defining a QA strategy for any of our projects, I recommend reviewing this list again and asking a question – is that important for my product now? In most cases, your answer will be YES for functionality and UI/usability. That’s why you’ll plan functional, UI, compatibility, and usability tests.
Compatibility is the one that is often overlooked.
Always remember that you cannot make your audience use the same browser or smartphone as you. You have to accept their choice and support whatever they like.
Side note: check your audience geography – people in China use very different smartphones from those used by people in the US.
Let’s say you develop an internal accounting solution for your CFO and two or three other analysts or accountants.
In this case, if one report that is only being generated annually takes five minutes to generate, you may not care that much about the solution performance. But what if it’s stock market software? Performance is critical here – the market changes in nanoseconds, and five minutes will cost you a fortune.
Security is another big topic. Do you store personal information (like SSN and DOB) or your customers’ credit card details? Oh yeah, security testing is a must.
And believe me, you’d rather be HIPAA, FDA, PCI-DSS, and whatever else compliant than go to court against your customers.
To test or not to test isn’t a question in this digital age of ours. Now the focus has shifted towards choosing the right testing strategy that will meet the requirements of the software developed, end users, and business stakeholders.
Flexible planning and setting priorities related to your product is what will help you make the smart choice.