OTA update testing: how one bad release can disrupt an entire IoT fleet
What if a software defect affected thousands of vehicles? How many technicians, labour hours, appointments, and replacement vehicles would a manual fix require?
In May 2025, a Rivian R1S using Highway Assist misclassified a vehicle ahead. The driver did not brake, resulting in a low-speed collision with no injuries.
Rivian identified 24,214 vehicles running the affected software and issued an over-the-air (OTA) fix. By the time the recall was filed in September, more than 99% had already been updated remotely.
The case shows what reliable OTA updates can save: field-service costs, technician hours, customer disruption, and weeks of recall coordination.
It is an impressive result, although the story could easily have ended differently. The update had to reach the correct vehicles, install among varying software states, survive interruptions, and leave each car working as expected. A failure anywhere in that sequence could have added another fault to the recall.
Now extend that risk across the 21.1 billion IoT devices that IoT Analytics counted in 2025. The figure grew 14% in one year and is expected to exceed 39 billion by 2030. A small failure rate can leave thousands offline.
How many of them can the business afford to recover by hand? OTA testing ought to answer that question before deployment begins.
Every device brings its own update history
An update package may be identical, but the devices receiving it rarely are. One has the latest firmware and a strong Wi-Fi connection. Another has skipped three releases, has little free storage, and connects intermittently through a low-power network.
Cloud services select the target group, the delivery infrastructure transfers the file, and the bootloader installs it. Hardware, firmware, stored data, and connected components must still work after a restart.
A revised processor may handle memory differently. An older bootloader may reject a valid package. A firmware jump may change stored data, while a third-party vendor’s module may depend on an interface that was removed in the release. In multi-vendor fleets, no single team controls every component that has to survive the restart.
A download reaching 100% proves very little
The most revealing OTA tests begin when normal conditions disappear. What does the device do when power fails during installation, or the network vanishes halfway through a package?
If firmware is written directly over the active image, a power cut can leave the device unable to boot. Dual-bank architecture reduces that risk, although rollback still needs testing. A device has not recovered if the old firmware starts minus its credentials, settings, or stored data.
Network failure creates a different problem. The device should recognize an incomplete package and resume or discard it safely. It should never treat a partial download as ready for installation.
Targeting deserves equal scrutiny. A correctly signed package can still be wrong for the model, region, bootloader, or current firmware. Security checks should also reject altered files, expired signatures, replayed releases, and attempts to reinstall a vulnerable version.
Then comes the easily missed stage: what happens after the system reports success? The device may restart while draining its battery, drop its connection, or send inaccurate sensor data. Installation status and device health are separate measurements.
When remote recovery fails, the economics change quickly. If an OTA can’t be completed, the user may be instructed to visit a physical address to have the update applied manually. For sensors on farms, controllers at mine sites, or equipment offshore, the equivalent visit may require far more planning.
Regulation is making the update quality harder to postpone
The EU Cyber Resilience Act requires manufacturers to provide security updates without delay and free of charge during the defined support period. Vulnerability reporting begins on 11 September 2026, and the wider regulation applies from 11 December 2027. Fines can reach €15 million or 2.5% of worldwide annual turnover, whichever is higher.
The UK Product Security and Telecommunications Infrastructure regime is already in force. It allows penalties of up to £10 million or 4% of qualifying worldwide revenue, whichever is higher, plus daily penalties while a breach continues. Fines are not the only exposure: under both regimes, market surveillance authorities can restrict, withdraw, or recall non-compliant products, effectively obstructing access to the EU and UK markets.
Manufacturers need evidence that updates are authenticated, delivered during the promised support window, installed safely, and monitored after release. Package records, signing information, test results, device logs, rollout histories, and incident reports provide it.
Contain the rollout before it contains a defect
The safest fleet-wide update begins with a very small subset of units. Internal devices and canary cohorts give teams a chance to examine crash rates, battery use, connectivity, and rollback events before expanding deployment.
That safety depends on the controls in place during the rollout. Can the team pause distribution immediately? Can it withdraw a package that has already been downloaded? Can it exclude one hardware revision? How quickly can it trigger a rollback?
QA should test these controls while deliberately creating the conditions teams hope never occur. Cut power during download, verification, installation, and restart. Add packet loss, limited bandwidth, full storage, clock errors, corrupted files, slow servers, and repeated reboots. Each interruption should lead to a defined state that support and engineering teams can recognize.
Fleet-wide averages can hide failures concentrated in one region, network, hardware revision, or previous firmware version. An experienced IoT testing services team can reproduce those combinations before customers encounter them.
A strong percentage can still hide a large recovery queue
A 99.9% success rate sounds reassuring. Across one million devices, it leaves 1,000 failures. The percentage only becomes useful when it is connected to a decision.
Teams can track update success, download and installation failures, rollback time, remote recovery, post-update crashes, and the share of devices inside the supported version window. They should also define which result automatically pauses the rollout.
For example, a team might require 99.95% successful updates, a rollback within 30 minutes, a crash rate below 0.1%, and 98% of devices within the supported window. These are example operating targets, not published industry benchmarks. A connected glucose monitor warrants tighter tolerances than a warehouse sensor.
Make update testing part of every firmware change
Release week leaves little time to discover an unsupported upgrade path. CI/CD checks can validate packages, signatures, version rules, security controls, and core functions whenever firmware changes, and automated regression tests can run across every supported upgrade path — including devices jumping several versions at once. Device labs and simulators can then cover additional hardware and network combinations.
For a US real estate client, a1qa built a lab to reproduce key-card and QR-code access across buildings. The building-access project demonstrates how a controlled physical setup can expose faults that software-only checks miss. Early automation also reduced QA time. Across the engagement, 25% of the defects found were of high or critical severity.
Know the recovery path before pressing deploy
Can a device lose power, lose its network, receive the wrong package, and still recover without a technician? If the answer is uncertain, the rollout carries more operational risk than its success-rate forecast suggests. Teams that can answer it confidently ship faster, spend less on field recovery, and turn update reliability into a selling point rather than a support burden.
a1qa’s specialists can assess OTA architecture, test upgrade and rollback paths, simulate interrupted conditions, and verify post-release monitoring across connected devices. Contact us to discuss how to make your next OTA rollout easier to contain and recover.
