Closed-course testing has been a staple of vehicle safety testing for over 100 years. Ever since General Motors opened the Milford Proving Ground in 1924, these facilities have served as crucial safety checkpoints in the automotive development process, allowing manufacturers to test new models of vehicles in controlled environments prior to public deployment.
As the automotive industry has evolved toward automation, closed-course testing has been adopted by autonomous vehicle (AV) developers working on everything from basic driver assistance systems to self-driving technology spanning SAE levels two through five. AV companies commonly use test tracks to validate the capabilities of their systems under controlled conditions by performing very specific tests. For example, they might test whether a vehicle using a new Adaptive Cruise Control feature can properly react to a lead vehicle that suddenly brakes.
Though the practice is valuable for testing basic autonomy capabilities, it has many limitations that restrict the type and number of safety tests that can be performed.
- Resource intensive: Manually creating scenarios on a test track resembles elaborate movie-stunt coordination, requiring meticulous planning, extensive resources, and significant time to execute even a single test case. Consider the complexity of creating a construction zone scenario with construction workers moving around dynamically, orange cones defining new lanes, and traffic controllers directing traffic. These kinds of coordination demands make it particularly challenging to create intricate scenarios involving multiple actors performing sophisticated maneuvers.
- Lack of repeatability: The manual coordination involved also makes it difficult to reproduce tests consistently and precisely—something that is critical for comparing system performance across different software releases or hardware configurations.
- Lack of diversity: Diversity of scenarios is also limited. Real-world traffic involves millions of unique vehicles of every make and model, countless drivers and pedestrians, as well as all sorts of animals, debris, and unexpected obstacles yet there are typically very few assets to utilize in a closed-course testing facility, creating a significant gap in validation.
- Unable to test safety-critical scenarios: Importantly, the use of real physical objects
Specialized rigs, dummy props and sophisticated timing triggers have been designed as an attempt to solve some of these challenges. They can help with repeatability and marginally increase the diversity of scenarios that are possible to recreate, including some of the more safety-critical tests, but they are expensive and fundamentally unrealistic.
Dummies and other props can help AV developers test certain safety-critical scenarios, but they are fundamentally unrealistic and cannot be utilized to meet the volume of complex, dynamic interactions that AVs must navigate in real-world conditions.
For a practice that was specifically developed to test safety, closed-course testing falls short. Despite the billions invested across the industry in these facilities, this testing modality cannot deliver the volume, diversity, and complexity of scenarios in the realistic and repeatable manner that comprehensive AV safety testing demands.
Until now.
Introducing Mixed Reality Testing
Waabi’s Mixed Reality Testing (MRT) offers a revolutionary alternative that completely transforms what’s possible on a closed-course track.
In the same way that augmented reality goggles blend the physical world with a virtual world, MRT enables the Waabi Driver to drive autonomously down a physical test track while simultaneously experiencing numerous intelligent, simulated actors that coexist in this hybrid reality and react to each other and to the physical world in naturalistic ways. All this is possible by leveraging Onboard Waabi World, a version of Waabi’s neural simulator that runs in a few milliseconds on the onboard compute. As Onboard Waabi World generates new scenarios, the real physical sensor readings are modified instantaneously so the Waabi Driver can react to the blend of real and virtual elements while driving in the physical world. This fusion creates a first-of-its-kind reality that unlocks unlimited testing possibilities previously impossible to achieve safely or practically.
Traffic jams can materialize instantly, motorcycles can weave between lanes, crowds of pedestrians can jaywalk unpredictably, and children can dart into the street from behind parked cars. Animals can wander across the road, debris can appear in lanes creating hazardous obstacles, and complex construction zones can emerge in the vehicle’s path. Emergency vehicles with flashing lights, police officers directing traffic, or multi-car accidents can suddenly appear ahead. The possibilities are endless.
A Dynamic, Intelligent Alternate Reality
All of the virtual elements in MRT exist within a sophisticated 4D neural digital replica of the real world, where AI-powered actors behave with human-like unpredictability and intelligence, mirroring the chaotic nature of the real-world. They can replicate anything from aggressive drivers weaving through traffic to distracted pedestrians or impaired motorists. Infrastructure can be created or modified on demand. Traffic lights can change states and construction zones can shift layouts instantly.
The alternate reality is enabled by Waabi World’s sensor simulation capabilities, which modify the multimodal sensor data from the physical sensors mounted on the truck including the LiDAR and cameras in real-time. The modified sensor data flows directly into the full onboard software stack, causing the Waabi Driver to respond to virtual scenarios as if they were physically present on the track. And thanks to the groundbreaking level of realism of Waabi World, the Waabi Driver thinks the mixed world is reality, triggering the same decision-making processes and responses as if they existed in the first place.
Mixed Reality Testing (MRT). As the Waabi Driver drives in the real world, it acquires data through its sensors of the surrounding physical environment. We modify this data on the fly using Waabi World’s behavior and sensor simulation systems to create an alternative world with a wide variety of new actors, new behaviors, and new environmental and lighting conditions, which the Waabi Driver can then physically interact with and react to. This entire process happens in a few milliseconds on the onboard compute.
This breakthrough eliminates all the previous constraints of closed-course testing. What once required hours of preparation and coordination on a traditional test track can now be executed instantly.
The MRT environment also operates continuously and endlessly, enabling non-stop testing without pause or preparation time between scenarios. One test seamlessly flows into the next, maximizing every real mile driven on the track. This allows Waabi to conduct exponentially more tests than was previously possible.
With MRT, Waabi is able to explore countless safety-critical cases that would require billions of real-world miles to encounter even once. Rare edge cases that were previously impossible to recreate physically can now be tested with extraordinary levels of realism and, importantly, without real-world consequences. There’s no risk in having a virtual drunk driver create a head-on collision scenario in MRT, allowing us to test how well the system minimizes damage in unavoidable crash situations. And since we know everything about the virtual world, we can assess with perfect accuracy, automatically, and in real time the performance of the Waabi Driver.
These capabilities represent an unprecedented advancement in AV testing.
MRT can simulate complex, diverse and safety critical scenarios that would normally be impossible to replicate on a traditional closed-course and that would typically take billions of miles to even observe once in the real world.
Achieving Unprecedented Safety Testing at Scale
Waabi has been utilizing MRT as a central testing approach for more than two years. It has been a key innovation that has allowed Waabi to advance faster, safer and with unparalleled capital efficiency, enabling us to achieve feature-complete autonomous driving capabilities at the beginning of this year and paving the way to our imminent driverless launch.
We have always been driven to establish what would be the ultimate “driving test” for an AI system and, with MRT, we are delivering on that vision. For the first time, we can test as aggressively and comprehensively as the real world demands, without exposing anyone to the real world’s risks. This breakthrough allows us to build an AV we can truly trust and accelerates the path to safer roads for everyone.
