STRIDE: Towards Self-Supervised, Generalizable, and Decomposable 4D Driving Scene Reconstruction

Motivation

Reconstructing dynamic 3D scenes from sensor data is a fundamental task in physical AI domains such as self-driving and robotics. It enables creating high-fidelity digital replicas of the physical world that support realistic and diverse simulation for training and testing autonomous systems. To enable scalable counterfactual simulation, 4D reconstruction methods must produce high-fidelity scene representations efficiently, while also ensuring those representations are decomposable for scene editing. However, existing methods do not satisfy all of these requirements. Traditional per-scene reconstruction approaches require multiple optimization steps, making them prohibitively slow for large-scale simulation. As a result, generalizable reconstruction methods were developed, processing sensor data and producing a scene representation in a single forward pass. However, existing generalizable approaches either fail to leverage multi-modal sensor inputs or do not employ scalable transformer architectures. Beyond architectural limitations, these methods generate unstructured geometry primitives without instance-level understanding, meaning they cannot support e.g. actor manipulation for counterfactual scenario generation.

Method

Architecture Overview

Multi-modal Feature Extraction

Our method leverages camera and LiDAR data, which are commonly available on modern self-driving vehicles. LiDAR provides explicit but sparse 3D geometry. On the other hand, images provide dense appearance information in 2D. We seek to leverage complementary inputs by using the full dense observations from the image, while lifting them to 3D via depth completion. To do this, we first project LiDAR points onto the image to create a sparse depth map which is combined with the RGB images. Then, we employ a vision transformer (ViT) to process the input imagery into a dense depth map for lifting, a sky mask to separate the foreground, and image features which enable downstream scene understanding. 

Self-supervised 4D Scene Reasoning

In contrast with prior works which reason purely in the image space with ViTs, we believe that reasoning in 3D is crucial for geometry and flow understanding. Therefore, using the non-sky mask and the dense depth outputs, we lift the image features into 3D to obtain a set of point features. These point features are then processed through a Point Transformer, which leverages attention as the core mechanism for representation learning. Different from traditional transformers, this architecture groups points into smaller patches via a space filling curve, and only limits the receptive field of each attention layer to a local path. This leads to better efficiency and introduces a strong locality prior. The transformer tokens can then be decoded into Gaussian splats with velocity profiles through a lightweight MLP.

Dynamic Instance Decomposition

To support dynamic instance segmentation, and more generally an object centric latent representation, we introduce a set of learnable instance tokens. These tokens are initialized based on the DBSCAN clustering algorithm, and then processed jointly with the other dynamic scene tokens in another Point Transformer. Here the goal is to learn the association between the two sets of tokens in the form of a similarity matrix. We can then take an argmax over the similarity matrix to obtain discrete instance assignments to scene tokens and their corresponding Gaussians. Crucially, our method learns the instance decomposition in a self-supervised manner. We exploit the intuition that Gaussians with similar motion should belong to the same instance. Thus, in a second training stage we leverage the same reconstruction objectives, but instead predict instance token velocities, and compute every Gaussian’s flow from its instance token. With this formulation, flow learning implicitly supervises the instance decomposition. 

Quantitative Results

4D Reconstruction

We evaluate our method against two pioneering generalizable reconstruction works - STORM and Flux4D. We also incorporate STORM+ as a multi-modal variant for fair comparison. Moreover, because Flux4D leverages multiple forward passes for additional refinement, we re-implement a Flux4D-ff variant which runs a single forward pass. We present our results on the Waymo Open Dataset (WOD) here, but additional experiments on other benchmarks can be found in our paper. 

On the WOD reconstruction benchmark, we show that our method produces significantly better scene flow as indicated by the EPE3D and flow angle metrics, enabling better reconstruction quality on dynamic regions. This is primarily due to our Point Transformer backbone which reasons in 3D, while STORM reasons in the image space and Flux4D leverages a 3D convolution backbone restricted by small receptive fields. While STORM+ has marginally better full image PSNR and SSIM, it is significantly worse in dynamic regions and flow estimation, indicating a lower quality scene representation. As Flux4D-ff relies solely on LiDAR for Gaussian initialization, it has weaker performance in the sparse view setting.

Dynamic Instance Decomposition

Our learnable instance module consistently outperforms all baselines in all precision and recall metrics. Since the DBSCAN baseline is identical to the instance token initialization, this comparison shows that improvements come purely from training with self-supervised 4D reconstruction signals. With our learned instances, STRIDE enables controllable scene manipulation, such as actor removal, injection, and placement which we show below.

To justify our ViT and PTv3-based backbone design, we ablate with the backbones used in STORM and Flux4D. We replace our entire backbone with STORM’s cross-time ViT backbone that directly reasons over 2D image-view tokens. Alternatively, we replace our PTv3 with the 3D sparse-convolution UNet used in Flux4D, while keeping our multi-modal ViT feature extractor. Our method outperforms both backbones in almost all metrics. Cross-time ViT is slightly better on full image PSNR, but is noticeably worse on dynamic PSNR and flow angles. SparseConvUNet performs worse than PTv3 for reasoning on sparse multi-modal inputs.

Qualitative Results

With our self-supervised 4D reconstruction and dynamic instance segmentation, we are able to recover appearance, geometry, actor motion and dynamic instances across time. The visualization below shows STRIDE’s rendered images, flows and dynamic instance ids across time in the first row, and the ground-truth images, flows and instance ids in the second row.

4D Reconstruction

Our 4D reconstruction offers both superior photorealism and more accurate flow estimation compared to other reconstruction methods. In the visualization below, we show the rendered images and flows (color hues represent flow directions and color intensities represent flow magnitudes). Comparing to existing baselines, our self-supervised flow detection is more accurate, capturing movements for pedestrians and faraway actors that are missed by some of the baselines.

Dynamic Instance Decomposition

Our self-supervised dynamic instance decomposition is the first of its kind. Against a clustering DBSCAN baseline, our method learns to consolidate over-decomposed initializations and produce accurate instance masks.

Scene Editing

With high-fidelity reconstruction and instance decomposition, our method enables scene editing. We augment the original scene on the left in three ways. First, we are able to remove any actor of choice. Second, we can take an existing actor and change its trajectory. Third, we can even inject actors from other scenes.

Conclusion

We present STRIDE, a self-supervised, generalizable model for reconstructing large-scale driving scenes from multi-modal sensory inputs. By processing a unified spatio-temporal representation with 3D Point Transformers, our framework achieves superior reconstruction quality and scene flow estimation, highlighting the importance of our 3D architectural design. Furthermore, we pioneer a self-supervised approach to learn decomposable scene reconstructions in the feed-forward setting, demonstrating that standard reconstruction signals can improve dynamic instance segmentation. Overall, this work provides important architectural insights for 4D reconstruction, and establishes a foundational step towards learnable object-centric scene decomposition.