UniSim: A Neural Closed-Loop Sensor Simulator

Top: UniSim takes recorded camera and LiDAR data from a data collection platform and creates manipulable digital twins. Bottom: UniSim simulates realistic, temporally consistent sensor data for new scenarios, enabling closed-loop autonomy evaluation. The autonomy system reactively interacts with the scenario, receives new sensor data, and changes lanes (see planned trajectory inset). All images and LiDAR in figures are simulated by UniSim.

UniSim can accurately reconstruct the original sensor data with little to no artifacts. We show the real log, UniSim camera simulation, reconstructed depth and normal map.

By using a decomposed representation, UniSim can perform actor removal and in-paint the static scene, creating a fresh new canvas to build safety critical scenarios from.

Left: rendering of the original scene. Right: rendering of scene with all actors removed.

We show this at scale on a large variety of logs, from city to suburban areas, and in different lighting conditions.

Along with actor removal, we can manipulate the existing actors’ trajectories to create new scenarios. Here, we make the white sedan that originally took a left turn, change its mind and swerve back into the ego-vehicle’s lane. We complete the unseen regions of the actor and create a realistic counterfactual scenario.

Here’s another scenario where the grey vehicle changes lanes instead of staying in the right most lane.

We can also insert the reconstructed actors from other logs into the scene.

Insert a single actor into existing scene.

Replace existing actors with new actors

We can also move the self-driving vehicle’s viewpoint to the left, right, or change the camera sensor’s position, elevating it or lowering it.

We can do this at scale, enabling scalable creation of manipulable virtual worlds.

UniSim takes pre-recorded real world logs and converts them into a modifiable digital twin simulations. With UniSim, we can specify new counterfactual safety-critical scenarios, simulate realistic sensor data, and run autonomy in a closed-loop fashion. This is more scalable since we can drive anywhere to collect data, more realistic since we leverage real sensor data, and more diverse since we observe and create all types of assets.

In the following video, we modify the scene and add an actor, highlighted in orange on the left, to come head-on into our lane. The SDV perceives the actor and executes a lane change to narrowly avoid collision with the oncoming actor. UniSim can act as a tool to help identify how the autonomy would realistically behave in these situations and help enable safe development.

Original scenario.

Closed-loop simulation with autonomy.

Here’s another example where we modify an actor, highlighted in orange on the left, to merge into our lane. We can seamlessly generate the sensor data for the new actor positions and new viewpoints planned by the autonomy model.

Original scenario.

Closed-loop simulation with autonomy.

Here we show qualitative results comparing to other state-of-the-art methods. We first show image results in the interpolation setting where we replay the original trajectory of the SDV. Our method handles better the decomposition of moving actors and moving SDV within the static background, resulting in fewer artifacts on the held out frames.

We also show results in the more challenging lane shift setting, where we shift the ego trajectory to the left by 2 meters. Compared to the baselines, our method better handles the large shift from the training views. We can in-paint parts of the dynamic actors and reduce artifacts with our hybrid rendering approach.

Comparison on camera simulation (lane shift setting)

UniSim also simulates lidar point clouds, enabling temporally and spatially consistent multi-sensor simulation. We show qualitative comparison results between a recent lidar simulation method and UniSim. Our simulated lidar is smoother due to more accurate geometry reconstructions.

Comparison on LiDAR simulation

UniSim works not only in slower speed urban areas, but high speed highway settings over long time sequences. We first show a sample of the real video captured in the left most lane at high speed.

Now we show a long running sequence that stretches over several kilometers where we perform various manipulations and render with UniSim. Data was originally recorded in the left most lane. We can change the sensor position and add and remove the actors in the scenario even when the recorded data is from sparse high speed data collects, demonstrating the scalability and robustness of UniSim.