A large test count did not help when every failed run started another investigation.
Director of Engineering
The client’s platform had about 1,100 automated tests, but engineers no longer trusted the suite. a1qa was asked to assess a bounded 420-test subset covering critical eCommerce flows, make a documented retain, replace, or retire decision, and prove that the resulting tests could support release decisions.
The challenge
The critical subset produced an 18% flaky test rate. 24% of reported failures did not represent a reproducible product defect. Outdated selectors, duplicated setup logic, inconsistent data, and weak failure classification forced engineers to rerun jobs and inspect noise before they could evaluate product risk.
Maintenance and rerun work consumed about 45 engineering hours each week. Only 52% of CI runs completed within the agreed 90-minute budget and without rerun. A regression decision for the selected scope took about 2 working days, limiting the product area to 2 planned releases per month.
The client needed evidence about what still worked, what should be modernized, and what was cheaper to replace.
Over a 12-week period, a1qa moved from assessment to pilot delivery through a structured rescue approach. The team included 3 automation engineers, 1 senior quality engineering architect, and part-time DevOps support.
The team started the project with sampling failures across repeated runs, reviewing code structure and dependencies, and separating flaky tests, environment failures, data failures, and reproducible product defects. Each of the 420 critical scope tests received a retain, replace, or retire decision supported by a technical and economic reasoning.
After stabilizing 170 tests whose business intent and architecture remained useful, QA engineers improved selectors, assertions, waits, data setup, and reusable components while preserving traceability to the critical customer journeys.
Next, the team replaced 150 tests that could not be repaired economically. AI assisted repetitive code preparation and conversion of approved scenarios, while QA engineers applied framework conventions, reviewed business logic, and rejected output that did not meet the acceptance policy.
Ultimately, one hundred obsolete or duplicate tests were retired from the pilot scope. The 320 accepted tests were integrated into the client-owned CI workflow, repeatedly validated, and checked against execution time, flakiness, false-failure rate, and maintainability guardrails.
Authentication and customer accounts, search, cart and checkout, order status and customer notifications formed the bounded scope for the assessment. Environment and product defects outside it were logged as dependencies.
QA engineers ensured that retained and replacement tests met business intent, framework and traceability rules, passed human review, and remained stable across the validation window. Failures were classified as test, data, environment, or product issues before a test could be accepted. That policy kept the 320-test result tied to maintainability instead of raw suite size.
As project deliverables, the client received the classification record, repaired code, CI configuration, operating guide, and technical backlog. The handoff also identified which environment dependencies required client action and which future automation work belonged in a separate scale phase.
