Skip to main content
Digital TwinsSensor SimulationSimulation

UniSim: A Neural Closed-Loop Sensor Simulator

By June 8, 2023July 25th, 2023No Comments

UniSim: A Neural Closed-Loop Sensor Simulator

Ze Yang*,  Yun Chen*,  Jingkang Wang*,  Siva Manivasagam*,  Wei-Chiu Ma,  Anqi Joyce Yang,  Raquel Urtasun
* denotes equal contribution
Conference: CVPR 2023 (Highlight)


Rigorously testing autonomy systems is essential for making safe self-driving vehicles (SDV) a reality. It requires one to generate safety critical scenarios beyond what can be collected safely in the world, as many scenarios happen rarely on our roads. To accurately evaluate performance, we need to test the SDV on these scenarios in closed-loop, where the SDV and other actors interact with each other at each timestep. Previously recorded driving logs provide a rich resource to build these new scenarios from, but for closed-loop evaluation, we need to modify the sensor data based on the new scene configuration and the SDV’s decisions, as actors might be added or removed and the trajectories of existing actors and the SDV will differ from the original log. In this paper, we present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle and converts it into a realistic closed-loop multi-sensor simulation. UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene, and composites them together to simulate LiDAR and camera data at new viewpoints, with actors added or removed and at new placements. To better handle extrapolated views, we incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions. Our experiments show UniSim can simulate realistic sensor data with small domain gap on downstream tasks. With UniSim, we demonstrate, for the first time, closed-loop evaluation of an autonomy system on safety-critical scenarios as if it were in the real world.


Top: UniSim takes recorded camera and LiDAR data from a data collection platform and creates manipulable digital twins. Bottom: UniSim simulates realistic, temporally consistent sensor data for new scenarios, enabling closed-loop autonomy evaluation. The autonomy system reactively interacts with the scenario, receives new sensor data, and changes lanes (see planned trajectory inset). All images and LiDAR in figures are simulated by UniSim.


Play with sound.

Scene Reconstruction

UniSim can accurately reconstruct the original sensor data with little to no artifacts. We show the real log, UniSim camera simulation, reconstructed depth and normal map.

Actor Removal

By using a decomposed representation, UniSim can perform actor removal and in-paint the static scene, creating a fresh new canvas to build safety critical scenarios from.

Left: rendering of the original scene. Right: rendering of scene with all actors removed.

We show this at scale on a large variety of logs, from city to suburban areas, and in different lighting conditions.

Actor Manipulation

Along with actor removal, we can manipulate the existing actors’ trajectories to create new scenarios. Here, we make the white sedan that originally took a left turn, change its mind and swerve back into the ego-vehicle’s lane. We complete the unseen regions of the actor and create a realistic counterfactual scenario.

Here’s another scenario where the grey vehicle changes lanes instead of staying in the right most lane.

We can also insert the reconstructed actors from other logs into the scene.

Insert a single actor into existing scene.

Replace existing actors with new actors

SDV Manipulation

We can also move the self-driving vehicle’s viewpoint to the left, right, or change the camera sensor’s position, elevating it or lowering it.

We can do this at scale, enabling scalable creation of manipulable virtual worlds.

Close Loop Sensor Simulation

UniSim takes pre-recorded real world logs and converts them into a modifiable digital twin simulations. With UniSim, we can specify new counterfactual safety-critical scenarios, simulate realistic sensor data, and run autonomy in a closed-loop fashion. This is more scalable since we can drive anywhere to collect data, more realistic since we leverage real sensor data, and more diverse since we observe and create all types of assets.

In the following video, we modify the scene and add an actor, highlighted in orange on the left, to come head-on into our lane. The SDV perceives the actor and executes a lane change to narrowly avoid collision with the oncoming actor. UniSim can act as a tool to help identify how the autonomy would realistically behave in these situations and help enable safe development.

Original scenario.

Closed-loop simulation with autonomy.

Here’s another example where we modify an actor, highlighted in orange on the left, to merge into our lane. We can seamlessly generate the sensor data for the new actor positions and new viewpoints planned by the autonomy model.

Original scenario.

Closed-loop simulation with autonomy.

Sensor Simulation Realism Comparison

Here we show qualitative results comparing to other state-of-the-art methods. We first show image results in the interpolation setting where we replay the original trajectory of the SDV. Our method handles better the decomposition of moving actors and moving SDV within the static background, resulting in fewer artifacts on the held out frames.

 Comparison on camera simulation (interpolation setting)

We also show results in the more challenging lane shift setting, where we shift the ego trajectory to the left by 2 meters. Compared to the baselines, our method better handles the large shift from the training views. We can in-paint parts of the dynamic actors and reduce artifacts with our hybrid rendering approach.

Comparison on camera simulation (lane shift setting)

UniSim also simulates lidar point clouds, enabling temporally and spatially consistent multi-sensor simulation. We show qualitative comparison results between a recent lidar simulation method and UniSim. Our simulated lidar is smoother due to more accurate geometry reconstructions.

Comparison on LiDAR simulation

UniSim on High Speed Highways

UniSim works not only in slower speed urban areas, but high speed highway settings over long time sequences. We first show a sample of the real video captured in the left most lane at high speed.

Now we show a long running sequence that stretches over several kilometers where we perform various manipulations and render with UniSim. Data was originally recorded in the left most lane. We can change the sensor position and add and remove the actors in the scenario even when the recorded data is from sparse high speed data collects, demonstrating the scalability and robustness of UniSim.


  title     = {UniSim: A Neural Closed-Loop Sensor Simulator},
  author    = {Yang, Ze and Chen, Yun and Wang, Jingkang and Manivasagam, Sivabalan and Ma, Wei-Chiu and Yang, Anqi Joyce and Urtasun, Raquel},
  booktitle = {CVPR},
  year      = {2023},