Big data testing for a worldwide leader in advanced analytics and data usage

PRODUCT OVERVIEW

The client is an internationally acknowledged front-runner in innovative analytics and data usage with its focus on driving human health forward by providing pharmaceutical companies with a full cycle of professional services.

The client collects, analyzes, and provides big data related to sales operations all over the world to its end users.

This big data is presented in the form of a database. To ascertain data integrity, consistency, and compliance with the set business logic and requirements, the client was seeking a QA partner to team up with and entrust software testing guidance and support.

a1qa took the mission over and contributed to turning the client’s need into a success story.

PROJECT SCOPE

Two dedicated specialists were assigned to supplement the Dev and testing teams to deliver QA excellence.

Prior to performing testing activities, the team delved into specifics of the big data transformation process that encompassed 2 phases: its development and testing.

The database creation phase, in turn, was split into the following steps:

  • The client sent source files containing structured and semi-structured data for downloading into the system
  • Later they were loaded into multiple staging databases (depending on the number of source files)
  • Following was their shift to a consolidated database, where the data was structured and presented according to proper business logic and the client’s requirements
  • During the final stage, the consolidated database was transformed into an output one containing structured data from multiple sources (e.g. top 50 corporations with the most remarkable sales features, additional measures showing the average sales of products based on the source data, other groupings of data in accordance with the client’s needs).

Having received this voluminous database, the QA team initiated its testing process.

Firstly, the engineers were checking:

  • Business rules (business logic applied) on staging/consolidation database level
  • Compliance of data in source files with the information in the output database after data transformation
  • Database structure
  • Specific cases (displaying data in the form of lists, correct calculation of measures added based on the source data, groupings of data, etc.).

Secondly, due to the specific nature of the project, the team was handling a large number of source files containing voluminous data.

Their testing using SQL queries turned out to be time-consuming. Therefore, to automate big data testing process, the engineers applied such tools as Power query, Macros, Pivot tables. They allowed the team to download data from multiple sources (e.g. from .CSV files, Excel, databases), cluster it, and compare. In the future, the team is planning to implement Power BI for data visualization.

During a subsequent stabilization stage, all the bottlenecks detected by the QA experts were transmitted to the Dev team for fixing (data was missing/trimmed, contained unexpected characters, was downloaded partially, the format was written incorrectly, etc.) on the database level.

Only after polishing the revised version, it was sent to production.

In the scope of the project, a1qa contributed to assuring the quality of 9 databases encompassing 3 to 20 voluminous source files each. Currently, cooperation with the client evolves, and the team continues to reveal bottlenecks and strengthen the quality level.

SERVICES OFFERED
  • Big data testing
TECHNOLOGIES & TOOLS
  • Excel Power query
  • Macros and VBA
  • Pivot tables
  • Connections
  • SSIS
RESULTS
  • The involvement of the a1qa team contributed to revealing bottlenecks in the extensive data amounts and, thus, creating a database operating in accordance with the set logic and the client’s requirements.
IN NUMBERS
  • 2
    QA engineers on the project
  • 5+
    years of the project duration
  • 9
    databases tested during cooperation with the client
QA news and tips delivered right to your inbox
We’ll send you one newsletter a month, jam-packed with amazing QA offers, hottest industry news, and all kinds of Software Testing goodness.