Blog

How a1qa testers conducted customer acceptance testing

We talk about our experience in moving from ordinary test cases to end to end business scenarios. 
14 November 2017
Quality assurance
Article by a1qa
a1qa

For those of you who’ve missed our previous post, let’s remind that a1qa has been engaged in the project of testing the new payment system for more than two years. There was nothing unusual about the project. Until one day, the customer gave us the challenge to arrange a demonstration of the first product release. Our team knew the product perfectly, so we immediately agreed, although the task required additional preparations.

The product was to be demonstrated to the customer’s team that knew and understood the requirements, but had only a vague idea of testing processes and terminology.

Acceptance testing challenges we faced

Customer representatives and tech geeks seemed not to see the product through the same glasses. And that became the reason of the first challenge.

What was important for the customer? To make sure that all processes function correctly from start to finish. As for us, our team of functional testers split the processes into small blocks and tested each block separately and in great detail, sometimes losing sight of the product as a whole.

Additionally, we lacked a clear mapping of test cases and business requirements. That was the second challenge. Test cases were based on technical documentation developed by system analysts and architects. Therefore, it was difficult to find the approach that would help demonstrate the expected result to the customer representatives.

a1qa approach to demonstrating first release

Criteria for successful acceptance of the first release there were very loyal: the implementation of 30% of the requirements, pass-rate – 80%, and the absence of critical defects in the implemented functionality.

Taking those criteria, a small number of demonstrated requirements (about 180), and the lack of complete business processes implementation (only some modules were released), we decided to use the following approach:

  • We selected the realized requirements, which had to be demonstrated.
  • For each requirement one or more test cases were prepared. Test cases were aimed at demonstrating that particular requirement.
  • Each test case assigned to the requirement was a standard test case with test data, a precondition, a detailed description of the steps and results. However, we focused on the description of the test case purpose and underlined the connection between the test case and the requirement as clearly as possible.

Thus, the implementation of each requirement was demonstrated by the test case, and the correctness of implementation was confirmed by passing the corresponding test case

Test cases were prepared as a separate Excel file, and attached to the Acceptance Testing (AT) document.

How was AT held?

AT was held by the a1qa engineers in the presence of several customer representatives. a1qa was in charge of demonstrating the product and executing test cases.
The commission included 10 people – representatives of top management and employees, who had to manage the payment system and monitor it.

The technical equipment of AT wasn’t sophisticated and consisted of a projector to show the information about the product under test and a monitor to display the Acceptance Testing document.

The AT process was organized as follows: the demonstrating requirement was named, the test case was described, and each action was commented. The result of each check was documented. During the demonstration there were many questions, comments, and proposals for improvement. Comments were recorded and questions were answered. If we had no answer for a question, the question was recorded and passed to responsible specialists.

The demonstration generally proceeded smoothly. All 230 test cases were passed, and therefore all 180 requirements were implemented correctly. The first release was successfully demonstrated.

However, we weren’t overly optimistic. There were two more releases ahead and it was advisable to start preparing for the next AT as early as possible. Moreover, after the first AT, we had a lot of ideas about improvements for the following demonstration.

Second release: new approach for Acceptance Testing

The main point that we had to improve in the approach was the “Test Methodology” section. Why wasn’t the approach used for the first release appropriate? The thing is, all test cases covered different modules, and didn’t demonstrate the system as a whole. Thus, it wasn’t evident that all processes were covered, and there was no sense of the integral system.

At the first release, the number of implemented requirements was not so large. To demonstrate the final product it was necessary to use a new approach.

Finally, we had to work on the terms used in the test cases. They were understandable for functional testers and other IT professionals, but not for the customer representatives. Therefore, it was necessary to use terminology of business requirements and prepare test data close to the real ones.

Solution found

We decided to separate the test documentation used for functional testing from the AT one. So we arrived at the idea of preparing separate scenarios for demonstrating the system through business processes – end to end (E2E) scenarios. One more activity was to prepare test data.

What is the reason for using E2E scenarios and what is their advantage? Naturally, there are some projects on which acceptance can be carried out using usual test-cases; however it was not our case. Within the framework of our project, the E2E scenarios were more applicable than test cases due to the following factors:

  • Reducing the number of test artifacts by grouping several requirements into one scenario. This allowed demonstrating all the implemented functionality faster. By the way, the demonstration of the first release took us five days. Keep in mind that only 180 requirements were implemented.
  • Demonstrating the product through complete business processes, close to the real scenarios of working with the system. The customer got an opportunity to test the product in conditions close to real ones.

Formulating ideal scenario

The idea of using business scenarios defended itself completely. Working together with the customer, we were able to develop a certain approach to preparing scenarios, and decided on a list of requirements for this type of test documentation.

Based on the experience of previous AT and the wishes of the customer, we provided the following structure of the test scenario:

Title. A short and clear title of the scenario helps to understand the aim of the check.

Test Case Goal. It was necessary to describe the purpose of this scenario.

Description. This part describes how the scenario should be passed, using business terminology and speaking the customer’s language.

Constraints. On large projects, one of the affected modules might be not implemented or implemented partly. But this is not an excuse to skip demonstration of the functions associated with it. For example, notifications of a certain type aren’t generated or some interface elements are displayed incorrectly.
It’s not that important. The main thing is that you, as QA engineers, have found and submitted these defects. This is what should be described in the Constraints section. This will help avoid the customer’s disappointment.

Test Data Description. On our project, this section was appropriate, since various technical test data were used to describe the scenarios, which were difficult to relate to the business components. Therefore, in this section, we described as detailed as possible what all elements of the database using business terminology.

Correlation of Steps and Requirements. Since one of the AT goals is to prove that requirements are implemented, it is necessary to specify which requirements we demonstrate passing the scenario.

Test Scenario Steps. This part consists of three components:

  1. The actions description with the help of business terms. Technical terms are also applicable, but only in exceptional cases. The main point is to use terms that are understood by the customer.
  2. Expected result (again using business terms).
  3. Method of verification. Here we can use a technical terms to describe technical implementation of previously stated business requirements.

Conclusion

Our team has come a long way from AT test cases to complete E2E scenarios that show the system from the business point of view.

To sum up, let’s note the following points:

  • Not every test case is suitable for AT.
  • Learn how to write E2E scenarios and your work as a tester will be invaluable.
  • Do not be afraid to take on new tasks. If we do only what we know, we will never become better.

Good luck on your projects!

We’ll be happy to answer any questions you might have.

More Posts

Why do bugs get missed
27 September 2024,
by a1qa
7 min read
Why do bugs get missed? Learn the problems and tips to avoid them
Still, finding overlooked bugs after the app goes live? Let’s find out why this happens and how to fix it.
Performance testing
QA consulting
Quality assurance
Test automation
Soft skills 101
13 September 2024,
by a1qa
4 min read
Soft skills 101: how to supercharge business success in 2025
Discover the value of soft skills for both career development and workplace improvement and learn some tips to sharpen them.
General
Quality assurance
QA to ensure smooth migration to the cloud
15 August 2024,
by a1qa
3 min read
QA to ensure smooth migration to the cloud
Learn how effectively migrate to the cloud by implementing QA activities.
Functional testing
General
Migration testing
Performance testing
Quality assurance
Test automation
Load testing
23 July 2024,
by a1qa
3 min read
7 reasons why businesses need load testing 
Want to optimize software performance or ensure its smooth functioning during peak sales season? Discover how load testing may help.
Quality assurance
Test automation
27 June 2024,
by a1qa
3 min read
Establishing seamless interaction between development and QA teams to boost productivity
Establishing seamless interaction between development and QA teams to boost productivity
Agile
General
Quality assurance
Test automation
17 June 2024,
by a1qa
5 min read
Shifting to test automation to maximize software quality
Explore in the article why businesses should move from manual testing to test automation.
Quality assurance
Test automation
RPA in QA
28 May 2024,
by a1qa
4 min read
Embracing robotic process automation to drive efficiency in QA
Discover how the convergence of robotic process automation helps reshape software testing practices.
General
Quality assurance
Test automation
QA for fintech
7 May 2024,
by a1qa
5 min read
Navigating the fintech frontier in 2024: QA’s role in delivering high-quality financial software 
Unveil the future of fintech innovations and learn to refine their quality with the help of software testing.
Blockchain app testing
Cybersecurity testing
Quality assurance
Shift-left testing for better software performance
25 April 2024,
by a1qa
4 min read
Optimizing software performance with shift-left testing
Still in doubt whether to include performance testing from the initial development stages? Learn the benefits companies obtain with shift-left performance testing.
Performance testing
QA consulting
Quality assurance
Test automation

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.