PROJECT OVERVIEW

Product Overview

The developed SaaS ERP solution covered the following processes run by PHAs in regards to the provisioning of affordable housing across the USA:

  • Applications submission and management, verification of the basis for the subsidized housing
  • Moving in/out processes management, apartment inspection procedures
  • Tasks allocation, notifications, working time management.

The client sought to fulfill the following goals:

  1. Automate current business processes
  2. Reduce the time spent on documents processing
  3. Enable customers to fill in a questionnaire on their own for further cooperation with a Public Housing Authority (PHA)
  4. Gather statistical data / clients' management data to enhance current business processes
  5. Gather financial information to report it to investors.

To meet all quality requirements, the A1QA team considered time-to-market criteria, product specifics, and target audience to deliver a custom-tailored package of quality assurance services.

Testing activities differed over time and depended upon the development stage. Thus, the booming growth of functionality was accompanied by thorough functional testing, alongside with performance testing and test automation. Later on, UX audit was added to improve end users' satisfaction.

Right before the product was shipped to market, end-to-end tests were carried out to assess the software market readiness.

Project Scope

To meet all quality requirements, the A1QA team considered time-to-market criteria, product specifics, and target audience to deliver a custom-tailored package of quality assurance services.

Testing activities differed over time and depended upon the development stage. Thus, the booming growth of functionality was accompanied by thorough functional testing, alongside with performance testing and test automation.

Later on, UX audit was added to improve end users' satisfaction.

Right before the product was shipped to market, end-to-end tests were carried out to assess the software market readiness.

Testing Approach

The QA team had to develop numerous test cases to validate that all the requirements were met. In order to ensure that test cases covered all the requirements, the team implemented a cross-review stage: once the engineer created test cases, he or she sent them over to another team member (the Reviewer) for review.

During test cases creation and review, the A1QA team faced the following problem: some test cases overlapped with other test cases, while the goal was to cover only one check.

It was decided to change the approach and start using the Test Case Matrix, which helped a Reviewer understand whether all the requirements defined for the system were covered by test cases.

The Test Case Matrix also helped to:

  • Cover several checks by one test case
  • Significantly reduce the number of test cases.

At times, when the functionality was changed it required test cases to be updated. A QA engineer had to select all test cases that were subject to update. The Component Matrix was applied to simplify and accelerate the process.

The Component Matrix contained all Test Case Matrixes with test cases for the appropriate component (module) of the system. Once a new story was implemented, the Test Case Matrix was added to the appropriate Component Matrix.

Technical Documentation

The A1QA technical writers were addressed to design and update a comprehensive suite of reference documentation. The documents had to take into account numerous peculiarities of the system and cover all the functionality.

The team did great job and was loaded with the following tasks:

  • Generate context-sensitive Help
  • Develop How-To’s manuals and FAQs pages
  • Develop trainings packages

Throughout the project, the A1QA technical writers prepared over 90 documents with the total number of 1000+ pages.

The product was concisely documented. The client highly valued the responsive and personalized approach backed by considerable experience and expertise to deliver targeted documentation for all requirements.

User Acceptance Testing

Before every new release rollout, the QA team performed User Acceptance Testing according to the scenarios prepared by the Business Analyst and Product Owner. Based on the received scenarios, QA engineers prepared a suite of test cases.

UAT Issues:

  1. Long time spent on UAT scenarios preparation
  2. Necessity to change / create test scenarios for every UAT.

Solutions:

After two rounds of UAT, it became obvious that almost all regression issues were related to the general user flow (revealed during quick sanity checks) and to the new functionality. Besides, new functionality required changes in the existing UAT scenarios or creation of new ones.

The QA team developed UAT scenarios that covered all the general user flows and applied existing test cases that were created in scope of testing stories, for new functionality.
As a result, preparation for UAT narrowed down to the choice of the appropriate test cases.

Later on, all the QSC tests and BAS (Business Analysts Scenarios) were automated. The behavior of every feature, business analysts specified in the BAS. QA team complemented them and passed over to the test automation team.

As a result, for every feature testing there was at least one automated test, which helped to save 3 hours on testing every feature from the UAT suite.

From Scrum to SAFe

Initially, Scrum methodology was implemented to develop the project. But after several years of development, it was decided to migrate to the Scaled Agile Framework (SAFe) cut for large enterprises and enabling them to manage software development across multiple teams more efficiently.

The A1QA team had to gain a deep understanding of the underlying values and principles of SAFe development.

The most obvious benefits of agile scaling turned out to be the following: synchronized work of all teams and faster delivery of the final product.

Test Metrics

Having agreed it with the client, QA regularly monitored the dynamics of the chosen metrics.

Example:

Explanatory note:

Rejected, % — the ratio of reopened (rejected) defects to the number of all checked defects.

Bugfixing quality:

0% — 3.9% rejected -> high

4% — 9.9% rejected -> medium

10% and more rejected -> low

The goal is < 10% of fixed defects.

Sprint defects by Severity – the number of all defects related to stories of the current sprint by Severity. The number of still opened defects is shown in parentheses.

QA Team

Initially, there was one QA team on the project (2 engineers and 1 Team Lead Specialist). By 2014, the QA needs were covered by 7 teams composed of a QA Project Coordinator, QA Project Manager, QA leads and QA engineers, Test Automation engineers, QA Performance team, Tech Writers, and UX experts.

Today, the A1QA team on the project is curtailed to three specialists who provide product support validating bugs fixing and testing additional features implemented upon users' request.

A1QA Value to the Project

The A1QA team has been working on the project from the very start. Smart parallel work of the several QA team helped to effectively distribute the tasks and solve questions that have arisen.

Besides testing itself, that helped to detect 12114 major and crucial defects, the A1QA specialists assured quality of migration to a new ERP system and conducted a series of trainings to the client’s team.

The client have turned to the A1QA managers’ consultations on every new QA hire.

Thanks to careful and diligent documentation developed by the A1QA technical writers, significant time and efforts saving were achieved and QA maturity was brought to a high level.

The product was successfully shipped to market with all the goals achieved on time.

The client has teamed up with our QA team for a number of other projects and the cooperation continues to date.

 

SERVICES OFFERED
  • Test Automation
  • Functional Testing
  • Cross-Browser Testing
  • Localization Testing
  • Usability Testing
  • Performance Testing
TECHNOLOGIES & TOOLS
  • .NET
  • KendoUI
  • ASP.NET
  • Azure
  • VSTests
  • NLog
  • Autofac
  • MVC
  • MSSQL (SSRS, SSIS)
  • Thinktecture Identity Server
  • WCF
  • Entity Framework
CHALLENGES AND SOLUTIONS

Almost 30 thousand test cases to support. To cut down the number of test cases, some improvements were introduced:

  • The Test Case Matrix to cover several checks by one test case
  • The Component Matrix utilization. It contained all the The Test Case Matrixes with test cases for the appropriate component of the system
  • 'Program' field. The field indicated the programs (in terms of which housing authorities cooperated with their clients) the test cases were valid for
  • 'Repeat' steps. If test cases differed in one step only, one test case was created and the step they differed in was added. Two almost identical test cases were replaced with a single one.

Large scope of regression testing:

  • All the sprint activities were put on hold and all the QA teams switched to 'KendoUI+IE11' test, while the DEV team addressed tech tasks and debugging. It took the team about 300 hours to fulfill it and due to the combination of the two tests about 100 hours were saved. Testing was completed within 2 weeks and the QA teams returned to the sprint activities.

Testing in two branches:

  • Full-cycle testing was carried out in both branches. But then the results were analyzed and it turned out that most of the issues are reproduced in both branches. Therefore, the scope of testing in the second branch was minimized with the full test performed in the branch with the current release only.

Communication with Dev and BA:

  • The QA team designed the Questions & Answers page. All the questions from QA engineers and developers and answers from BA were posted in this page. Q&A pages became an additional type of requirements.
CLIENT'S TESTIMONIAL

“When the A1QA QA Lead joined the team, it was as if she had never left the project. She understood all the features we had added, produced correct documentation, ran the triage meetings effectively, and researched issues quickly. Bottom line: her transition was seamless! 

She is effective in the written and verbal word. She communicates difficult concepts in an easy to understand way. Her soothing personality reflects in a very calm approach to all issues. She demonstrates active listening and extracts the right details to move an issue forward. This is extremely important when we are triaging issues and not everyone in the meeting understands the issues we are trying to correct.” 

“On just about every item the team provides direction and they take off and comeback if there are questions that need answered so they can progress with completing the task.”

RESULTS
IN NUMBERS
  • 5+
    years of project duration
  • 12114
    major and crucial defects detected
  • 100%
    delivering on time and within budget