TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception

Motivation

In order for a self-driving vehicle (SDV) to safely navigate through complex environments, it must first perceive anything, i.e. segment above ground obstacles from drivable areas at the current time, then forecast the evolution of the scene to plan its future motion. While object-free occupancy approaches improve upon traditional object-based trajectory forecasting, current 3D/4D models remain heavily reliant on expensive, human-annotated voxel grid labels. Consequently, existing labeled datasets are severely limited in scale and are restricted to predefined semantic classes, failing to encapsulate rare, safety-critical events. To bypass these labeling bottlenecks, prior self-supervised methods attempt to learn representations directly from raw LiDAR, camera, or radar data.

In contrast, we propose TriO, the first model that uses all three common sensor modalities (LiDAR, camera, and RADAR) as both input and supervision for occupancy in a self-supervised manner. Our work also introduces the task of anything perception as the ability to segment any object that exists above the road from the ground surface and forecasts its future evolution. TriO produces 4D outputs of occupancy, flow and obstacle segmentation and can be effectively fine-tuned for semantic occupancy prediction and LiDAR point cloud forecasting.

Method 

To learn a robust representation of the world, we leverage the complementary physical properties of the standard SDV sensor suite: active sensors (LiDAR and RADAR) operating on Time-of-Flight principles, and passive cameras. While LiDAR provides high-precision 3D geometry capable of capturing free-space and object surfaces, its sparse returns degrade at distance and in adverse weather. In contrast, RADAR uses longer wavelengths that penetrate fog and rain, and provide instantaneous radial velocity via the Doppler effect. Passive RGB cameras offer dense, high-resolution semantic and textural information, but lack native depth. To overcome this limitation, we extract dense depth and surface normal estimates using recent vision foundation models, enabling the reliable segmentation of obstacles.  We encode LiDAR and RADAR data as well as camera data with separate encoders and then perform voxel attention to fuse the resulting features. This is followed by a deformable attention multi-resolution header outputting occupancy, obstacle segmentation and flow.

Self-Supervised Training

During training, TriO‘s outputs are supervised by 4D (present + future) pseudo-labels as described below:

1. Occupancy pseudo-labels are constructed from LiDAR self-supervision as in our prior works UnO [Argo et al. CVPR 2024] and DIO [Diehl et al. CVPR 2025].

2. Obstacle Segmentation pseudo-labels, as shown below, are generated from separate LiDAR and camera segmentation which are fused.  

3. Flow pseudo-labels are generated from RADAR doppler-velocity measurements.

To provide supervision for obstacle segmentation, we propose a novel method for extracting obstacle/ground segmentation labels from image and LiDAR features in a probabilistic manner, combining the strength of each sensor modality.

For camera-based segmentation, we leverage depth and normal estimates from vision foundation models projected onto LiDAR point clouds, whereas for LiDAR-based segmentation, we use ground-patch estimation to segment obstacles in LiDAR space. This approach ensures that the objects which are distinctly above ground are captured by the LiDAR segmentation, which is robust due to the 3D information it receives, while also detecting smaller close-to-the-ground obstacles that may only be observable in image space.

TriO learns unsupervised instantaneous flow by extracting refined pseudo-labels from filtered RADAR Doppler measurements, combining sensor velocity, radial direction, and Doppler speed to overcome point-cloud noise.

Zero-Shot Obstacle Segmentation 

We evaluate TriO on multiple real-world driving datasets. We first show exemplary visualisations of TriO on Argoverse 2 sensor dataset. Notably, it successfully captures and segments key elements of the scenes like an excavator (A), fallen cones (B), the inside of a truck (C), an open car door (D), and a pedestrian crossing the street.

Importantly, TriO cannot only produce segmentation in 3D but also predict those into the future. In this scene, it accurately produces the multi-modal future distribution of a cyclist, which might continue driving straight or turning left together with accurate details of static elements of the scene. 

Moreover, TriO’s occupancy and obstacle segmentations generalizes to a variety of scenarios as shown below.

Next, we evaluate the model zero-shot on Spotting the Unexpected (STU), a benchmark that uniquely provides 3D LiDAR point labels for rarely seen objects on the road, aiming to test models on general perception. We note that STU was never used during training and features distinct sensor characteristics. We observe that TRIO generalizes remarkably well to new environments and captures multiple rare objects. It also generalizes to new road settings, as Argoverse 2 was mostly captured in urban scenarios, whereas STU covers suburban and country roads.

Failure Modes of Open-Vocabulary Methods 

We also find that text-based open-vocabulary segmentation baselines using vision-language-models (VLMs), even when prompted with the exact text description of the objects in the road, struggle to generalize to these rarely seen classes, reinforcing the findings of related work. Both our pseudo-labels themselves and the trained semantic occupancy predictions outperform several VLM baselines. 

In the following scenes, our pseudo-labels segment rare objects like a chair and two pigeons (scene 1), a ball (scene 2), and trash on the floor (scene 3) from the ground.

Scaling Labeled Data 

Below we scale the number of labeled training samples when fine-tuning pre-trained models (TriO, UnO, DIO) and models trained from scratch (ImplicitO-4D). TriO outperforms at all levels of supervision, and with 102 labeled training examples, reaches similar performance to the strongest baselines with 104 examples. TriO is two-orders of magnitude more sample efficient, underscoring the generalizability of our method. Due to its novel obstacle segmentation supervision TriO better understands obstacles and free space. This underlines the strong representation learned during pre-training and TriO’s ability to encode the geometry, semantics, and dynamics of the scene.

BibTeX

@inproceedings{eccv2026_trio,