Blog

How a1qa testers conducted customer acceptance testing

We talk about our experience in moving from ordinary test cases to end to end business scenarios. 
14 November 2017
Quality assurance
The article by a1qa
a1qa

For those of you who’ve missed our previous post, let’s remind that a1qa has been engaged in the project of testing the new payment system for more than two years. There was nothing unusual about the project. Until one day, the customer gave us the challenge to arrange a demonstration of the first product release. Our team knew the product perfectly, so we immediately agreed, although the task required additional preparations.

The product was to be demonstrated to the customer’s team that knew and understood the requirements, but had only a vague idea of testing processes and terminology.

Acceptance testing challenges we faced

Customer representatives and tech geeks seemed not to see the product through the same glasses. And that became the reason of the first challenge.

What was important for the customer? To make sure that all processes function correctly from start to finish. As for us, our team of functional testers split the processes into small blocks and tested each block separately and in great detail, sometimes losing sight of the product as a whole.

Additionally, we lacked a clear mapping of test cases and business requirements. That was the second challenge. Test cases were based on technical documentation developed by system analysts and architects. Therefore, it was difficult to find the approach that would help demonstrate the expected result to the customer representatives.

a1qa approach to demonstrating first release

Criteria for successful acceptance of the first release there were very loyal: the implementation of 30% of the requirements, pass-rate – 80%, and the absence of critical defects in the implemented functionality.

Taking those criteria, a small number of demonstrated requirements (about 180), and the lack of complete business processes implementation (only some modules were released), we decided to use the following approach:

  • We selected the realized requirements, which had to be demonstrated.
  • For each requirement one or more test cases were prepared. Test cases were aimed at demonstrating that particular requirement.
  • Each test case assigned to the requirement was a standard test case with test data, a precondition, a detailed description of the steps and results. However, we focused on the description of the test case purpose and underlined the connection between the test case and the requirement as clearly as possible.

Thus, the implementation of each requirement was demonstrated by the test case, and the correctness of implementation was confirmed by passing the corresponding test case

Test cases were prepared as a separate Excel file, and attached to the Acceptance Testing (AT) document.

How was AT held?

AT was held by the a1qa engineers in the presence of several customer representatives. a1qa was in charge of demonstrating the product and executing test cases.
The commission included 10 people – representatives of top management and employees, who had to manage the payment system and monitor it.

The technical equipment of AT wasn’t sophisticated and consisted of a projector to show the information about the product under test and a monitor to display the Acceptance Testing document.

The AT process was organized as follows: the demonstrating requirement was named, the test case was described, and each action was commented. The result of each check was documented. During the demonstration there were many questions, comments, and proposals for improvement. Comments were recorded and questions were answered. If we had no answer for a question, the question was recorded and passed to responsible specialists.

The demonstration generally proceeded smoothly. All 230 test cases were passed, and therefore all 180 requirements were implemented correctly. The first release was successfully demonstrated.

However, we weren’t overly optimistic. There were two more releases ahead and it was advisable to start preparing for the next AT as early as possible. Moreover, after the first AT, we had a lot of ideas about improvements for the following demonstration.

Second release: new approach for Acceptance Testing

The main point that we had to improve in the approach was the “Test Methodology” section. Why wasn’t the approach used for the first release appropriate? The thing is, all test cases covered different modules, and didn’t demonstrate the system as a whole. Thus, it wasn’t evident that all processes were covered, and there was no sense of the integral system.

At the first release, the number of implemented requirements was not so large. To demonstrate the final product it was necessary to use a new approach.

Finally, we had to work on the terms used in the test cases. They were understandable for functional testers and other IT professionals, but not for the customer representatives. Therefore, it was necessary to use terminology of business requirements and prepare test data close to the real ones.

Solution found

We decided to separate the test documentation used for functional testing from the AT one. So we arrived at the idea of preparing separate scenarios for demonstrating the system through business processes – end to end (E2E) scenarios. One more activity was to prepare test data.

What is the reason for using E2E scenarios and what is their advantage? Naturally, there are some projects on which acceptance can be carried out using usual test-cases; however it was not our case. Within the framework of our project, the E2E scenarios were more applicable than test cases due to the following factors:

  • Reducing the number of test artifacts by grouping several requirements into one scenario. This allowed demonstrating all the implemented functionality faster. By the way, the demonstration of the first release took us five days. Keep in mind that only 180 requirements were implemented.
  • Demonstrating the product through complete business processes, close to the real scenarios of working with the system. The customer got an opportunity to test the product in conditions close to real ones.

Formulating ideal scenario

The idea of using business scenarios defended itself completely. Working together with the customer, we were able to develop a certain approach to preparing scenarios, and decided on a list of requirements for this type of test documentation.

Based on the experience of previous AT and the wishes of the customer, we provided the following structure of the test scenario:

Title. A short and clear title of the scenario helps to understand the aim of the check.

Test Case Goal. It was necessary to describe the purpose of this scenario.

Description. This part describes how the scenario should be passed, using business terminology and speaking the customer’s language.

Constraints. On large projects, one of the affected modules might be not implemented or implemented partly. But this is not an excuse to skip demonstration of the functions associated with it. For example, notifications of a certain type aren’t generated or some interface elements are displayed incorrectly.
It’s not that important. The main thing is that you, as QA engineers, have found and submitted these defects. This is what should be described in the Constraints section. This will help avoid the customer’s disappointment.

Test Data Description. On our project, this section was appropriate, since various technical test data were used to describe the scenarios, which were difficult to relate to the business components. Therefore, in this section, we described as detailed as possible what all elements of the database using business terminology.

Correlation of Steps and Requirements. Since one of the AT goals is to prove that requirements are implemented, it is necessary to specify which requirements we demonstrate passing the scenario.

Test Scenario Steps. This part consists of three components:

  1. The actions description with the help of business terms. Technical terms are also applicable, but only in exceptional cases. The main point is to use terms that are understood by the customer.
  2. Expected result (again using business terms).
  3. Method of verification. Here we can use a technical terms to describe technical implementation of previously stated business requirements.

Conclusion

Our team has come a long way from AT test cases to complete E2E scenarios that show the system from the business point of view.

To sum up, let’s note the following points:

  • Not every test case is suitable for AT.
  • Learn how to write E2E scenarios and your work as a tester will be invaluable.
  • Do not be afraid to take on new tasks. If we do only what we know, we will never become better.

Good luck on your projects!

We’ll be happy to answer any questions you might have.

More Posts

Agile and DevOps in eCommerce QA_mini
30 September 2021,
by a1qa
5 min read
Agile and DevOps: Boosting the quality of eCommerce apps
What benefits do Agile and DevOps bring to eCommerce business, and how QA helps with that? Find it out in the article.
Agile
Quality assurance
AR/VR testing infographics mini
30 August 2021,
by a1qa
< 1 min read
AR/VR testing in retail: turning challenges into opportunities
Welcome to read the infographic on AR/VR in retail: new shopping experiences, issues, and how to address them with QA.
Quality assurance
6 October 2020,
by Dmitry Tishchenko
4 min read
A clear view of smart team scalability
Get to know how to scale your team sagely and gratify end-user needs and fast-paced tech-market requirements.
Agile
Quality assurance
6 August 2020,
by Elena Yakimova
5 min read
How to arrange fruitful joint work with an outsourcing QA team
The head of the web apps testing department sheds light on how to establish more transparent and effective work on the project cooperating with a remote QA team.
Quality assurance
4 June 2020,
by Vitaly Prus
4 min read
SAFe vs. Scrum, and PI planning essentials
Let's shed some light on the SAFe differences from Scrum that are to be considered by the development and QA teams who have migrated from Scrum.
Agile
Quality assurance
29 April 2020,
by a1qa
4 min read
5 lessons we learned from COVID-19
Here are five key lessons the businesses need to learn during this pandemic to somehow achieve the planned outcomes. 
Quality assurance
17 April 2020,
by a1qa
5 min read
QA-focused retrospective: identifying and solving project problems
The a1qa experts came up to consider an effective approach to identify project bottlenecks and get rid of problems successfully.
Agile
Quality assurance
31 March 2020,
by Dmitry Tishchenko
4 min read
QA outsourcing – the respond to unprecedented global challenge
How can companies meet their business-critical needs without health risks? QA outsourcing is the answer. Get to know why it is the right decision in this time of need.
QA consulting
Quality assurance
10 March 2020,
by a1qa
6 min read
Dedicated team model in QA: all you should know about it
Check on everything you should know about when to apply, how to run and pay for a dedicated team in QA.
Interviews
QA consulting
Quality assurance

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.