How a1qa testers conducted customer acceptance testing
For those of you who’ve missed our previous post, let’s remind that a1qa has been engaged in the project of testing the new payment system for more than two years. There was nothing unusual about the project. Until one day, the customer gave us the challenge to arrange a demonstration of the first product release. Our team knew the product perfectly, so we immediately agreed, although the task required additional preparations.
The product was to be demonstrated to the customer’s team that knew and understood the requirements, but had only a vague idea of testing processes and terminology.
Acceptance testing challenges we faced
Customer representatives and tech geeks seemed not to see the product through the same glasses. And that became the reason of the first challenge.
What was important for the customer? To make sure that all processes function correctly from start to finish. As for us, our team of functional testers split the processes into small blocks and tested each block separately and in great detail, sometimes losing sight of the product as a whole.
Additionally, we lacked a clear mapping of test cases and business requirements. That was the second challenge. Test cases were based on technical documentation developed by system analysts and architects. Therefore, it was difficult to find the approach that would help demonstrate the expected result to the customer representatives.
a1qa approach to demonstrating first release
Criteria for successful acceptance of the first release there were very loyal: the implementation of 30% of the requirements, pass-rate – 80%, and the absence of critical defects in the implemented functionality.
Taking those criteria, a small number of demonstrated requirements (about 180), and the lack of complete business processes implementation (only some modules were released), we decided to use the following approach:
- We selected the realized requirements, which had to be demonstrated.
- For each requirement one or more test cases were prepared. Test cases were aimed at demonstrating that particular requirement.
- Each test case assigned to the requirement was a standard test case with test data, a precondition, a detailed description of the steps and results. However, we focused on the description of the test case purpose and underlined the connection between the test case and the requirement as clearly as possible.
Thus, the implementation of each requirement was demonstrated by the test case, and the correctness of implementation was confirmed by passing the corresponding test case
Test cases were prepared as a separate Excel file, and attached to the Acceptance Testing (AT) document.
How was AT held?
AT was held by the a1qa engineers in the presence of several customer representatives. a1qa was in charge of demonstrating the product and executing test cases.
The commission included 10 people – representatives of top management and employees, who had to manage the payment system and monitor it.
The technical equipment of AT wasn’t sophisticated and consisted of a projector to show the information about the product under test and a monitor to display the Acceptance Testing document.
The AT process was organized as follows: the demonstrating requirement was named, the test case was described, and each action was commented. The result of each check was documented. During the demonstration there were many questions, comments, and proposals for improvement. Comments were recorded and questions were answered. If we had no answer for a question, the question was recorded and passed to responsible specialists.
The demonstration generally proceeded smoothly. All 230 test cases were passed, and therefore all 180 requirements were implemented correctly. The first release was successfully demonstrated.
However, we weren’t overly optimistic. There were two more releases ahead and it was advisable to start preparing for the next AT as early as possible. Moreover, after the first AT, we had a lot of ideas about improvements for the following demonstration.
Second release: new approach for Acceptance Testing
The main point that we had to improve in the approach was the “Test Methodology” section. Why wasn’t the approach used for the first release appropriate? The thing is, all test cases covered different modules, and didn’t demonstrate the system as a whole. Thus, it wasn’t evident that all processes were covered, and there was no sense of the integral system.
At the first release, the number of implemented requirements was not so large. To demonstrate the final product it was necessary to use a new approach.
Finally, we had to work on the terms used in the test cases. They were understandable for functional testers and other IT professionals, but not for the customer representatives. Therefore, it was necessary to use terminology of business requirements and prepare test data close to the real ones.
Solution found
We decided to separate the test documentation used for functional testing from the AT one. So we arrived at the idea of preparing separate scenarios for demonstrating the system through business processes – end to end (E2E) scenarios. One more activity was to prepare test data.
What is the reason for using E2E scenarios and what is their advantage? Naturally, there are some projects on which acceptance can be carried out using usual test-cases; however it was not our case. Within the framework of our project, the E2E scenarios were more applicable than test cases due to the following factors:
- Reducing the number of test artifacts by grouping several requirements into one scenario. This allowed demonstrating all the implemented functionality faster. By the way, the demonstration of the first release took us five days. Keep in mind that only 180 requirements were implemented.
- Demonstrating the product through complete business processes, close to the real scenarios of working with the system. The customer got an opportunity to test the product in conditions close to real ones.
Formulating ideal scenario
The idea of using business scenarios defended itself completely. Working together with the customer, we were able to develop a certain approach to preparing scenarios, and decided on a list of requirements for this type of test documentation.
Based on the experience of previous AT and the wishes of the customer, we provided the following structure of the test scenario:
Title. A short and clear title of the scenario helps to understand the aim of the check.
Test Case Goal. It was necessary to describe the purpose of this scenario.
Description. This part describes how the scenario should be passed, using business terminology and speaking the customer’s language.
Constraints. On large projects, one of the affected modules might be not implemented or implemented partly. But this is not an excuse to skip demonstration of the functions associated with it. For example, notifications of a certain type aren’t generated or some interface elements are displayed incorrectly.
It’s not that important. The main thing is that you, as QA engineers, have found and submitted these defects. This is what should be described in the Constraints section. This will help avoid the customer’s disappointment.
Test Data Description. On our project, this section was appropriate, since various technical test data were used to describe the scenarios, which were difficult to relate to the business components. Therefore, in this section, we described as detailed as possible what all elements of the database using business terminology.
Correlation of Steps and Requirements. Since one of the AT goals is to prove that requirements are implemented, it is necessary to specify which requirements we demonstrate passing the scenario.
Test Scenario Steps. This part consists of three components:
- The actions description with the help of business terms. Technical terms are also applicable, but only in exceptional cases. The main point is to use terms that are understood by the customer.
- Expected result (again using business terms).
- Method of verification. Here we can use a technical terms to describe technical implementation of previously stated business requirements.
Conclusion
Our team has come a long way from AT test cases to complete E2E scenarios that show the system from the business point of view.
To sum up, let’s note the following points:
- Not every test case is suitable for AT.
- Learn how to write E2E scenarios and your work as a tester will be invaluable.
- Do not be afraid to take on new tasks. If we do only what we know, we will never become better.
Good luck on your projects!
We’ll be happy to answer any questions you might have.
