Blog

How a1qa testers conducted customer acceptance testing

We talk about our experience in moving from ordinary test cases to end to end business scenarios. 
14 November 2017
Quality assurance
The article by a1qa
a1qa

For those of you who’ve missed our previous post, let’s remind that a1qa has been engaged in the project of testing the new payment system for more than two years. There was nothing unusual about the project. Until one day, the customer gave us the challenge to arrange a demonstration of the first product release. Our team knew the product perfectly, so we immediately agreed, although the task required additional preparations.

The product was to be demonstrated to the customer’s team that knew and understood the requirements, but had only a vague idea of testing processes and terminology.

Acceptance testing challenges we faced

Customer representatives and tech geeks seemed not to see the product through the same glasses. And that became the reason of the first challenge.

What was important for the customer? To make sure that all processes function correctly from start to finish. As for us, our team of functional testers split the processes into small blocks and tested each block separately and in great detail, sometimes losing sight of the product as a whole.

Additionally, we lacked a clear mapping of test cases and business requirements. That was the second challenge. Test cases were based on technical documentation developed by system analysts and architects. Therefore, it was difficult to find the approach that would help demonstrate the expected result to the customer representatives.

a1qa approach to demonstrating first release

Criteria for successful acceptance of the first release there were very loyal: the implementation of 30% of the requirements, pass-rate – 80%, and the absence of critical defects in the implemented functionality.

Taking those criteria, a small number of demonstrated requirements (about 180), and the lack of complete business processes implementation (only some modules were released), we decided to use the following approach:

  • We selected the realized requirements, which had to be demonstrated.
  • For each requirement one or more test cases were prepared. Test cases were aimed at demonstrating that particular requirement.
  • Each test case assigned to the requirement was a standard test case with test data, a precondition, a detailed description of the steps and results. However, we focused on the description of the test case purpose and underlined the connection between the test case and the requirement as clearly as possible.

Thus, the implementation of each requirement was demonstrated by the test case, and the correctness of implementation was confirmed by passing the corresponding test case

Test cases were prepared as a separate Excel file, and attached to the Acceptance Testing (AT) document.

How was AT held?

AT was held by the a1qa engineers in the presence of several customer representatives. a1qa was in charge of demonstrating the product and executing test cases.
The commission included 10 people – representatives of top management and employees, who had to manage the payment system and monitor it.

The technical equipment of AT wasn’t sophisticated and consisted of a projector to show the information about the product under test and a monitor to display the Acceptance Testing document.

The AT process was organized as follows: the demonstrating requirement was named, the test case was described, and each action was commented. The result of each check was documented. During the demonstration there were many questions, comments, and proposals for improvement. Comments were recorded and questions were answered. If we had no answer for a question, the question was recorded and passed to responsible specialists.

The demonstration generally proceeded smoothly. All 230 test cases were passed, and therefore all 180 requirements were implemented correctly. The first release was successfully demonstrated.

However, we weren’t overly optimistic. There were two more releases ahead and it was advisable to start preparing for the next AT as early as possible. Moreover, after the first AT, we had a lot of ideas about improvements for the following demonstration.

Second release: new approach for Acceptance Testing

The main point that we had to improve in the approach was the “Test Methodology” section. Why wasn’t the approach used for the first release appropriate? The thing is, all test cases covered different modules, and didn’t demonstrate the system as a whole. Thus, it wasn’t evident that all processes were covered, and there was no sense of the integral system.

At the first release, the number of implemented requirements was not so large. To demonstrate the final product it was necessary to use a new approach.

Finally, we had to work on the terms used in the test cases. They were understandable for functional testers and other IT professionals, but not for the customer representatives. Therefore, it was necessary to use terminology of business requirements and prepare test data close to the real ones.

Solution found

We decided to separate the test documentation used for functional testing from the AT one. So we arrived at the idea of preparing separate scenarios for demonstrating the system through business processes – end to end (E2E) scenarios. One more activity was to prepare test data.

What is the reason for using E2E scenarios and what is their advantage? Naturally, there are some projects on which acceptance can be carried out using usual test-cases; however it was not our case. Within the framework of our project, the E2E scenarios were more applicable than test cases due to the following factors:

  • Reducing the number of test artifacts by grouping several requirements into one scenario. This allowed demonstrating all the implemented functionality faster. By the way, the demonstration of the first release took us five days. Keep in mind that only 180 requirements were implemented.
  • Demonstrating the product through complete business processes, close to the real scenarios of working with the system. The customer got an opportunity to test the product in conditions close to real ones.

Formulating ideal scenario

The idea of using business scenarios defended itself completely. Working together with the customer, we were able to develop a certain approach to preparing scenarios, and decided on a list of requirements for this type of test documentation.

Based on the experience of previous AT and the wishes of the customer, we provided the following structure of the test scenario:

Title. A short and clear title of the scenario helps to understand the aim of the check.

Test Case Goal. It was necessary to describe the purpose of this scenario.

Description. This part describes how the scenario should be passed, using business terminology and speaking the customer’s language.

Constraints. On large projects, one of the affected modules might be not implemented or implemented partly. But this is not an excuse to skip demonstration of the functions associated with it. For example, notifications of a certain type aren’t generated or some interface elements are displayed incorrectly.
It’s not that important. The main thing is that you, as QA engineers, have found and submitted these defects. This is what should be described in the Constraints section. This will help avoid the customer’s disappointment.

Test Data Description. On our project, this section was appropriate, since various technical test data were used to describe the scenarios, which were difficult to relate to the business components. Therefore, in this section, we described as detailed as possible what all elements of the database using business terminology.

Correlation of Steps and Requirements. Since one of the AT goals is to prove that requirements are implemented, it is necessary to specify which requirements we demonstrate passing the scenario.

Test Scenario Steps. This part consists of three components:

  1. The actions description with the help of business terms. Technical terms are also applicable, but only in exceptional cases. The main point is to use terms that are understood by the customer.
  2. Expected result (again using business terms).
  3. Method of verification. Here we can use a technical terms to describe technical implementation of previously stated business requirements.

Conclusion

Our team has come a long way from AT test cases to complete E2E scenarios that show the system from the business point of view.

To sum up, let’s note the following points:

  • Not every test case is suitable for AT.
  • Learn how to write E2E scenarios and your work as a tester will be invaluable.
  • Do not be afraid to take on new tasks. If we do only what we know, we will never become better.

Good luck on your projects!

We’ll be happy to answer any questions you might have.

More Posts

a1qa-articles
31 January 2023,
by a1qa
5 min read
Best of 2022 by executives: 8 most visited a1qa blog posts
Let’s look back and revisit the most visited a1qa articles of 2022!
Quality assurance
Test automation
qa-trends
12 January 2023,
by a1qa
4 min read
The future of software testing: top 4 impactful trends that will dominate in 2023
Consider the major industry trends for the upcoming year to know how to improve current QA strategies and stay ahead of the curve.
QA trends
Quality assurance
Test automation
test-automation
7 December 2022,
by Dileep Marway
3 min read
Release at pace with test automation: What, why, and how to measure success?
An automation-first approach is key to enhancing testing capabilities and increasing overall operational efficiency. However, I would suggest justifying its implementation, so that it can deliver the promised value.
Quality assurance
Test automation
what-is-a-culture-of-happiness
2 December 2022,
by Dileep Marway
3 min read
What is a сulture of happiness for a QA company?
Great culture drives employee engagement and satisfaction, contributes to an impressive work output, and improves organisational performance. But how do we build a culture of happiness? In this blog, I will share more on the answer to this.
Quality assurance
agile-qa
30 November 2022,
by Dileep Marway
3 min read
Agile QA – what is needed for greater flexibility and speed?
What should your QA team focus on to become truly agile, enable quality at speed, and contribute to lasting performance improvement? In addition to introducing test automation, I suggest considering shared responsibility for software soundness.
Agile
Quality assurance
interview-with-dileep
28 November 2022,
by a1qa
9 min read
Interview with Dileep Marway on a series of articles “Agility and speed: Supercharging your business strategies with QA”
We cooperated with the VP of Engineering and Quality at SHL to present you with a series of his blog posts on: culture of happiness, test automation, and Agile-driven QA. Happy reading!
Agile
Quality assurance
Software lifecycle QA
Test automation
optimizing-budget-with-qa
31 October 2022,
by a1qa
4 min read
Optimizing telecom budgets with QA outsourcing: everything you need to know. Part 2 
Welcome to Part 2 of our blog on QA outsourcing and optimizing telecom budget with it. Let’s delve deeper into the topic!
QA trends
Quality assurance
optimizing-budget-with-qa
27 October 2022,
by a1qa
4 min read
Optimizing telecom budgets with QA outsourcing: everything you need to know. Part 1
Learn how to optimize telecom’s quality assurance expenses by relying on a trusted QA partner.
QA trends
Quality assurance
qa-trends-in-telecom
30 September 2022,
by a1qa
5 min read
4 telecom trends for 2023 and how to painlessly implement them with QA
It’s time to explore the telecom trends for the upcoming year. Let’s look at them together and also see the value that QA brings for their smooth deployment.
Cybersecurity testing
Migration testing
QA trends
Quality assurance
Test automation

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.