Traffic Scenario Orchestration from Language via Constraint Satisfaction
ICRA 2026
Frieda Rong, Chris Zhang, Kelvin Wong, Raquel Urtasun
Autonomous vehicles (AVs) require extensive testing in simulation, but test case generation for driving scenarios is laborious. The desired scenarios are often out-of-distribution and have precise requirements on interactions with the AV policy under test. Manually programming scenarios allows for precise controllability but is difficult to scale. On the other hand, statistical models can leverage compute and data, but struggle with precise controllability when out-of-distribution. We cast scenario orchestration as a constraint-solving problem and present a language-in, simulation-out scenario orchestrator for closed-loop testing AVs. Our approach leverages foundation model reasoning to translate general, natural language descriptions into a set of constraints as a scenario representation. This then allows us to leverage off the shelf solvers to solve for actor behaviors which meet precise testing intentions in closed-loop. Under a benchmark of carefully crafted and diverse scenario descriptions, our approach greatly outperforms our baselines in orchestration success rate. We further show that our closed-loop approach is especially important for scenarios which require ego-reactive specifications.
Motivation
Simulation offers a safe, efficient, and precise way to test and verify AV behavior across a variety of scenarios. However, scenario orchestration is a challenging task. Manual specification gives the scenario author control and precision but can be laborious. On the other hand, learning-based methods are more scalable but struggle to orchestrate precise closed-loop interactions, especially for out-of-distribution scenarios. We want an approach to scenario orchestration that is easy-to-control, precise, and kinematically realistic.
Method
We propose a neurosymbolic approach to scenario orchestration that combines a large language model (“LLM”) with a constraints solver to orchestrate closed-loop, kinematically-realistic scenarios from natural language. Our approach consists of three steps:
Convert natural language to constraints: Given a scenario description and a high definition (HD) map, we use an LLM to identify the actors in the scenario and the routes they drive along. The LLM also specifies each actor’s motion profile along its route and a corresponding set of constraints that govern its kinematics and interactions with other actors and the map.
Solve for initial states and motion profiles: Next, a constraints solver solves for the free variables in each actor’s motion profile, determining its initial state and how it moves along its route.
Replan for closed-loop orchestration: In most cases, the purpose of orchestration is to test the behavior of an AV policy that is not controlled by the orchestrator. Thus, the orchestrator must be able to react to new AV behavior over the course of a closed-loop rollout of the scenario. To this end, we iteratively reinvoke the solver with additional constraints on all actors’ current state to replan each actor’s motion profile.
Results
We evaluate our approach on a benchmark of 80 scenario descriptions. The scenarios describe the interaction between the ego controlled by an AV policy and a hero actor at an intersection and vary in terms of the actors’ intended routes, desired interactions, and trigger conditions (e.g., the hero actor should yield or not yield to the ego). Our primary metric is orchestration success rate, which measures how well each method orchestrates a closed-loop scenario with the desired routes, interactions, and trigger conditions.
Benchmarking Natural Language to Scenario
Compared to a state-of-the-art learning-only baseline (ProSim), our approach achieves substantially higher orchestration success rate, especially in challenging scenarios with stop-and-go interactions. ProSim’s most common failure mode was not executing the scenario to the precise timing of the solver. This suggests that while statistical models may be beneficial in generating more in-distribution scenarios, precise or slightly out-of-distribution scenarios remain a challenge.
Precision & Controllability
To assess the precision and controllability of our approach, we compare its performance against the baseline on a fixed scenario across a discretized grid of parameter values. We find that the baseline systematically fails when the target ego distance to the conflict point is low (see the “x” in the figure). In contrast, our approach is successful across all tested parameter values.
Open-loop vs. Closed-loop Execution
Our next experiment analyzes the importance of our closed-loop approach to scenario orchestration. Specifically, we evaluate our approach on four representative scenarios with closed-loop replanning disabled (“open-loop execution”) and enabled (“closed-loop execution”). We find that closed-loop orchestration achieves a 23% higher success rate over open-loop orchestration.
Qualitative Results
Conclusion
In this paper, we introduced a neurosymbolic framework for closed-loop traffic scenario orchestration. Our approach combines the frontier reasoning capabilities of an LLM with the precision of a constraints solver to generate controllable, reactive, and kinematically-realistic scenarios from natural language. Compared to a state-of-the-art learning-only baseline, our approach achieves higher orchestration success rates, especially for interactive scenarios with precise spatiotemporal requirements. Promising directions for future work include scaling our approach to higher complexity scenarios (e.g., multi-actor interactions, complex map topologies) and exploring methods and metrics which consider statistical realism.
@inproceedings{rong2026orchestration,
title={Traffic Scenario Orchestration from Language via Constraint Satisfaction},
author={Frieda Rong and Chris Zhang and Kelvin Wong and Raquel Urtasun},
booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
year={2026},
}