
DIO: Decomposable Implicit 4D Occupancy-Flow World Model
Christopher Diehl*, Quinlan Sykora*, Ben Agro, Thomas Gilles, Sergio Casas, Raquel Urtasun
ICCV 2023
Jeffrey Yunfan Liu, Yun Chen*, Ze Yang*, Jingkang Wang, Sivabalan Manivasagam, Raquel Urtasun
Neural Scene Rasterization. Our method renders urban driving scenes (1920×1080) at high quality and >100 FPS by leveraging neural textures and fast rasterization. We reconstruct driving scenes in the San Francisco Bay Area and show the rendering at four streets on the map.
Play with sound.
Realistic and efficient camera simulation enables safe and scalable autonomy development. Realism enables us to develop autonomous systems in simulation with confidence that they will perform similarly in the real world, while efficiency enables fast and scalable development of the autonomy system on millions of scenarios. Achieving both speed and realism for camera simulation has been a long-standing challenge. Existing neural rendering methods have demonstrated impressive results. However, they struggle to achieve real-time efficiency, particularly in large scenes. On the other hand, traditional rasterization rendering is fast for large scenes but lacks the realism required for self-driving simulation. NeuRas is a novel neural rasterization approach that combines rasterization-based graphics and neural rendering for realistic real-time rendering of large-scale scenes. It overcomes the aforementioned limitations by utilizing a scaffold mesh as input and incorporating a neural texture field and shading to model view-dependent effects. Compared to computationally expensive neural volume rendering, this approach enables high-speed rasterization, which scales especially well for large scenes.
We first create the scene representation for rendering. To start with, our method takes an moderate quality mesh scaffold as input, which can be generated from MVS or Neural 3D reconstruction. We then unfold the mesh and obtain the UV mappings for each of its vertices. Based on the generated UV mapping, we initialize a learnable UV feature map. For far-away regions like sky and distant buildings, we model them with multiple neural skyboxes to enable rendering of full scenes. Similarly, we use neural feature maps to represent these boxes’s texture.
Given the scene representation, we first rasterize the foreground mesh and neural skyboxes to target view, producing a set of image feature buffers. The feature buffers are then processed with MLPs to produce a set of rendering layers, which are composited to synthesize the final RGB image. The MLP and neural features are optimized during training. For rendering, the MLP is baked as a shader in existing rasterization engines for real time rendering.
To encourage the sharing of latent features in visually similar regions such as roads and sky, we apply vector quantization (VQ) to regularize the neural texture maps.
Compared to existing novel view synthesis approaches, NeuRas produces achieves the best trade-off between speed and realism. In particular, our method can achieve comparable performance as the best NeRF-based methods while being at least 30 times faster (> 100 FPS).
NeuRas can render large-scale urban driving scenes with a high degree of realism in real time. These urban driving scenes typically span a distance of over 150 meters in camera movement.
Additionally, we demonstrate rendering large-scenes interactively. To the best of our knowledge, our method is the first of its kind that is capable of realistically rendering large scenes at a resolution of 1920×1080 in real-time.
NeuRas produces competitive realism results and achieves real-time rendering (> 400 FPS) on drone scenes, allowing for interactive visualization.
Our method can speed up popular NeRF approaches by extracting the meshes from nerf representations and learning neural texture for them. We achieve 30x speedup without significant drop in visual realism
@inproceedings{liu2023neural,
title = {Neural Scene Rasterization for Large Scene Rendering in Real Time},
author = {Jeffrey Yunfan Liu and Yun Chen and Ze Yang and Jingkang Wang and Sivabalan Manivasagam and Raquel Urtasun},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
year = {2023},
}
Christopher Diehl*, Quinlan Sykora*, Ben Agro, Thomas Gilles, Sergio Casas, Raquel Urtasun
Ben Agro, Sergio Casas, Patrick Wang, Thomas Gilles, Raquel Urtasun
Ze Yang, Jingkang Wang, Haowei Zhang, Sivabalan Manivasagam, Yun Chen, Raquel Urtasun
Yun Chen*, Matthew Haines*十, Jingkang Wang, Krzysztof Baron-Lis, Sivabalan Manivasagam, Ze Yang, Raquel Urtasun
UniCal: Unified Neural Sensor Calibration
Chris Zhang, Sourav Biswas, Kelvin Wong, Kion Fallah, Lunjun Zhang, Dian Chen, Sergio Casas, Raquel Urtasun
Yun Chen*, Jingkang Wang*, Ze Yang, Sivabalan Manivasagam, Raquel Urtasun
Sergio Casas*, Ben Agro*, Jiageng Mao*十, Thomas Gilles, Alexander Cui十, Thomas Li, Raquel Urtasun
Sergio Casas*, Ben Agro*, Jiageng Mao*十, Thomas Gilles, Alexander Cui十, Thomas Li, Raquel Urtasun
Jack Lu†*, Kelvin Wong*, Chris Zhang, Simon Suo, Raquel Urtasun