Big data testing for a worldwide leader in advanced analytics and data usage

The client was undergoing the process of transforming big data related to sales operations into a database for its end users – global pharmaceutical companies. To ensure data integrity and consistency, a1qa joined the project.
Big data testing
Dedicated QA teams
Healthcare and pharma
IT and software development


The client is an internationally acknowledged front-runner in innovative analytics and data usage with its focus on driving human health forward by providing pharmaceutical companies with a full cycle of professional services.

The client collects, analyzes, and provides big data related to sales operations all over the world to its end users.

To automate the testing process, accelerate t2m, and ascertain data integrity, consistency, and compliance with the set business logic and requirements, the client was seeking a QA partner to team up with and entrust software testing guidance and support.

a1qa took the mission over and contributed to turning the client’s need into a success story.

Services offered

Big data testing

Project scope

Organizing the workflows

To process gathered big data, a special infrastructure was developed on the project. Partnering with a wide range of end clients, it was necessary to exclude the required data from the sources and group it according to business requirements. Then, the client’s account managers sent the adjusted output DBs to a particular customer and updated them within the pre-defined timeline (weekly, monthly, quarterly, etc.).

To facilitate the process, the client divided the project into subprojects serving particular end customers. a1qa assigned two dedicated specialists for one of such subprojects, which supported a multinational pharmaceutical company by providing high-quality information. The QA engineers were to supplement the Dev and testing teams to eliminate big data faults and deliver high-quality information to the end customers.

Data life cycle

Prior to performing testing activities, the team delved into specifics of the big data transformation process that encompassed 2 phases: its development and testing.

The database creation phase, in turn, was split into the following steps:

  • The client sent source files containing structured and semi-structured data for downloading into the system.
  • Later they were loaded into multiple staging databases (depending on the number of source files) using S3 Bucket — a cloud storage.
  • Following was their shift to a consolidated database, where the data was structured and presented according to proper business logic.
  • During the final stage, the consolidated database was transformed into an output one containing structured data from multiple sources and meeting the end clients’ requirements (e.g. top 50 corporations with the most remarkable sales features, additional measures showing the average sales indicators based on the source data, other groupings of data in accordance with the client’s needs).

Having received this voluminous database, the QA team initiated its testing process.

Services delivered 

The a1qa engineers checked that data met the following criteria:

  • Accuracy

To make sure that big data doesn’t contain duplicates, formatting issues, blank rows, and invalid values. Moreover, the QA team verified that the set of business rules (business logic) are applied in accordance with project requirements.

  • Completeness

To check the compliance of data in source files with the information in the output database after data transformation (e.g., no incomplete data, no data loss happening due to the data transformation process).

  • Reliability

To ensure that delivered information in the database is up to date.

  • Relevance

To ensure delivery of only relevant data to end clients according to the proper database structure.

  • Timeliness

To ascertain that data refresh doesn’t trigger any error while enabling timely data delivery upon end clients’ requests.

The a1qa specialists supported the client throughout the entire data life cycle. On staging/consolidation/output database level, the QA team also checked portions of specific cases like displaying data in the listing format, correct calculations of measures added based on the source data, groups of data, etc.

Automating the big data testing process

The team handled a substantial amount of source files containing voluminous data, and their testing using SQL queries was time-consuming.

Therefore, to facilitate and speed up the process, the a1qa engineers applied automated QA files and reshaped them within the project requirements and the end customer’s expectations.

The mechanism for performing automated checks was the following:

  • To download the data from multiple sources (e.g., from .CSV files, Excel, databases), the engineers wrote the VBA-based connection string in the automated QA files while interacting with the servers.
  • To get the necessary clustered information, the QA team put the corresponding SQL queries into these files and verified each query manually in DBs beforehand.
  • To compare the data with pre-defined requirements automatically and detect drawbacks sharply, the experts leveraged formulas while adjusting them to a particular group of information.
  • To pull up testing results and evaluate the quality state, the a1qa team created separate pages in the files that allowed monitoring which checks had passed and which tests had failed.

By applying Macros, the testing workflow (from uploading the data to checking testing results) was automated within each file. In case of data or DB updates, all the downloaded information, SQL queries, formulas, etc. are also changed automatically.

During a subsequent stabilization stage, all the bottlenecks detected by the QA experts were transmitted to the Dev team for fixing (data was missing/trimmed, contained unexpected characters, was downloaded partially, the format was written incorrectly, etc.) on the database level.

Only after polishing the revised version, it was sent to production.

Improving processes

Each subproject had processes established in various ways, so the customer didn’t have a centralized QA workflow that could adversely impact the teams’ efficiency.

To help the client refine QA activities, the a1qa specialists suggested introducing an Agile board. This initiative made the voluminous number of tasks more structured and transparent while every project member could easily grasp their current statuses and assignees.

The a1qa experts put other Agile practices forward. The client embarked on conducting retrospectives to inspect past processes and create a plan for improvements.

Thus, a1qa provided the customer with complete visibility and a well-tuned QA process on the entrusted subproject while contributing to the enhanced performance of other subteams.

When all is said and done, a1qa helped assure the quality of big data encompassing hundreds of source files while delivering information to the end customers timely and within their expectations.

The project was successfully completed by revealing bottlenecks and strengthening the overall quality. Now, cooperation with the client continues within a range of other projects.

Technologies & tools

  • Amazon Web Services (S3 Bucket, Amazon Workspace)
  • SQL
  • SSIS
  • Excel (formulas, Power queries, Pivot tables, Macros, VBA, Power BI)


  • Detected faults in the extensive data amounts
  • Created databases operating under the set logic and the client’s requirements
  • Sped up testing time in line with tailored automated QA files
  • Increased visibility on the project due to implementing Agile practices
  • Astonished end clients due to data delivered within timelines and budget

In numbers

QA engineers on the project
years of the project duration
databases tested during cooperation with the client
sources of any complexity and structure went through the automated testing process daily

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.