Last year, we launched our revolutionary closed-loop simulator, Waabi World, and illuminated a pathway towards the safe commercialization of self-driving trucks. Next week at CVPR 2023 we are presenting a groundbreaking paper unveiling a core part of the tech that powers Waabi World.
UniSim: A Neural Closed-Loop Sensor Simulator authored by Ze Yang, Yun Chen, Jingkang Wang, Siva Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang and myself is one of our most exciting and significant papers to date.
Why is it so important?
Rigorously testing autonomy systems and training them to handle all possible situations is essential for making safe self-driving vehicles a reality. To achieve this, we need to generate safety critical scenarios beyond what can be collected safely in the real world. Unfortunately, existing solutions fall short. The self-driving industry primarily tests their systems on pre-recorded real-world sensor data (i.e., log-replay), or by driving new miles in the real-world. In the former, the autonomous system cannot execute actions and interact with the scene, as new sensor data different from the original recording cannot be generated, while the latter is neither safe, nor scalable or sustainable. Existing simulation systems, which are typically based on game engines, lack fidelity, lack the diversity of the real world, or only test a small part of the autonomy system (e.g., motion planning). Furthermore, the assets employed by these simulators (e.g., 3D representations of the environment and the traffic participants) are typically created by hand, resulting in a non-scalable solution that is very capital intensive and lacks fidelity.
The status quo calls for novel closed-loop sensor simulation systems that are high fidelity and represent the diversity of the real world. UniSim is our answer. It is based on a simple concept: we digitally clone the world and modify that world to create new situations, answering questions such as “what would have happened if the car in front had more aggressively cut into our lane?” and analyzing how the autonomy system would have responded.
UniSim is a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle and automatically converts it into a realistic closed-loop multi-sensor simulation, enabling testing of the entire autonomy system in a reactive and immersive manner. This means that all data we collect in the real world, regardless of the platform employed to collect it, can be used to simulate any other platform. For example, we can take camera and LiDAR data recorded by a car and use it to simulate what camera and LiDAR data would look like on a truck platform by elevating the sensors to new mount locations and rendering new multi-sensor data of the scene. This contrasts the industry, where data collected can only be used for training the same platform with the same sensors on the same locations. This makes data very quickly obsolete in the standard industry approach. This never happens with UniSim, where every data counts!
This incredible technology underpins Waabi World, enabling us to automatically recreate digital twins of the world with the diversity, scale, and realism of the world we live in, as well as modify reality by removing, adding, or changing the behavior and appearance of “actors” (including the Waabi Driver) in any scenario and re-simulating the sensors in near real-time. This lets us create an endless number of diverse worlds for the Waabi Driver to experience—unlocking the ability to realistically test the full software stack in an immersive and reactive manner and help the Waabi Driver learn sophisticated driving skills, all safely within the simulator.
With UniSim, we demonstrate, for the first time, closed-loop evaluation of an autonomy system on photorealistic safety-critical scenarios as if it were in the real world. I am immensely proud of team Waabi’s unwavering passion and tireless dedication in their pursuit of this remarkable breakthrough. This technology is the future of the industry and is a key component of the safest and fastest path towards commercialized self-driving vehicles.
To learn more, please visit https://waabi.ai/unisim/
