Building test automation solution in BDD environment

In the previous article we spoke about describing test scenarios with Gherkin syntax. Today Sergey Senyuk, Manager of a1qa test automation team, describes how their team develops custom-tailored test automation solutions into the BDD mindset.

Test automation in BDD context

When the end users of the automation solution are QA specialists who have no skills in programming and creating tests themselves, we offer to use the following approach to deliver effective automation solution that will work on the client’s side:

Stage 1: Regardless of the type of the application that is to be tested (mobile, web or desktop) we create framework with the basic logics necessary for application automated testing.

Stage 2: Using the Page Object and other relevant patters we design solution structure, write form classes that should contain the fields that correspond to the app’s UI elements and methods of handling these elements. We also create tests classes with form objects to implement the defined usage methods.

Stage 3: We put test data to the external files, thus removing logic details from behavior tests.

Stage 4: Implementing the BDD approach, we integrate relevant libraries to the project to use Gherkin syntax. Then the steps of test scenarios should be developed using the keywords Given/When/Then.

Graphically the delivered solution will look as follows:

As a result, by adopting this approach we let QA engineers work with Gherkin source files containing feature descriptions in their localized language. The documentation also serves as executable automated tests. By making various steps combinations a QA engineer may easily create new tests on his or her own.

A report of the successfully executed test will look as follows:

Every feature description may contain several user scenarios. Every scenario may consist of one or several steps. Every step may need a number of actions to be taken. All this data will be presented in a tree-type structure.

For every test that has failed there will be the following information available in the logs:

• A step with the mistake occurred marked in red.
• Description of the actual and expected outcome.
• The string of the program code with the failed check.
• List of actions that have been completed before a mistake occurred.
• Application screenshot.

Another question is how to build and maintain effective communication between software testing engineers and automation engineers within the BDD projects.

Communication between automation engineers and QA team

In order to ensure a bug-free test automation solution that will take effect and deliver tangible results to the customer, it’s vital for both teams involved to understand the scope of their duties.

We advise to separate the obligations of QA team from those of test automation engineers.

At the initial stage, of the project, be it a pilot one or Proof of Concept, testing engineers convert the set of test scenarios in Gherkin or create their own scenarios. After that, automation engineers develop automated tests based on the given scenarios within the BDD approach. They also set up a continuous integration test environment to enable tests launch and storage of their run reports.

Upon the project completion, automation engineers submit the following deliverables to the client:

• Test scenarios in Gherkin.
• Automated tests and framework.
• Continuous integration environment for tests launching.
• Test automation solution documentation (architecture description, launching procedure in CI, etc.).

Automation engineers should also train QA team how to handle the delivered solution.

Hereafter, the automation engineers’ duties will be limited to working with the code:

• Maintaining the existing tests and updating them in case of the UI or business logics changes.
• Creating new tests.
• Developing additional steps to improve the variability of the automated tests.

QA engineers, who will act as primary users of the delivered solution, will be expected to do the following:

• Launch tests to check the existing scenarios on numerous data sets.
• Analyze test run results and provide the client with the detailed report containing logs, screenshots, tests execution metrics, etc.
• Create new tests using the keywords and scenarios in Gherkin.

It’s also important to take into account a number of factors that may increase the final costs of the automation project with the BDD mindset:

1. Teaching Gherkin grammar and syntax to QA team.
2. The solution development will take by 10-15% longer due to the necessity of developing test scenarios steps.
3. Converting clients’ test scenarios in Gherkin to enable further creation of automated tests on their basis.

Despite extra costs, the number of BDD advantages will significantly prevail.

The key benefits will be the following:

• Documentation and automated tests are stored in one project and the feature description is actually an executable test. There is also an option to develop the automation solution so that the scenarios in Gherkin will be stored in any test-tracking system and automated tests will request them when necessary.
• The solution can be handled by users who have no programming skills. QA specialists will be able to launch tests, analyze their runs and create new test scenarios.
• The steps in Gherkin syntax are reusable and the keywords may be combined in many different ways to create different scenarios.
• The system description in Gherkin will be common for all teams involved. As a result, their vision of the system will also be the same.

Let’s sum it all up

Combination of BDD approach and Gherkin language takes a number of positive effects on projects where QA team has no skills in writing automated tests. However, in order to ensure maximum effect, it’s vital to take into account and calculate the costs of QA team training and converting test scenarios in Gherkin.

Judging by our experience, the benefits of implementing BDD in test automation prevail if done properly.