SMD: Multi-view Safety-Critical Driving Video Generation in the real-world domain

anonymous
Code

Table of Contents

Abstract

Safety-critical scenarios are essential for evaluating autonomous driving (AD) systems, yet they are rare in practice. Existing generators produce trajectories, simulations, or single-view videos—but they don’t meet what modern AD systems actually consume: realistic multi-view video. We present SMD, the first framework for generating multi-view safety-critical driving videos in the real-world domain. SMD couples a safety-critical trajectory engine with a diffusion-based multi-view video generator through three design choices. First, we pick the right adversary: a GRPO-fine-tuned vision-language model (VLM) that understands multi-camera context and selects vehicles most likely to induce hazards. Second, we generate the right motion: a two-stage trajectory process that (i) produces collisions, then (ii) transforms them into natural evasion trajectories—preserving risk while staying within what current video generators can faithfully render. Third, we synthesize the right data: a diffusion model that turns these trajectories into multi-view videos suitable for end-to-end planners. Videos generated by SMD substantially increase collision rates when stress testing multiple end-to-end planners, and reduce collision rates when incorporated into training, improving planner robustness and safety.

pipeline

SMD pipeline
Figure: pipeline of the SMD framework.
  1. VLM-based Adversarial Vehicle Selector: Identifies the adversarial vehicle from multi-view images.
  2. Two-Stage Evasion Trajectory Generator: First generates a collision trajectory, then refines it into a realistic evasion trajectory.
  3. Trajectory-to-Video Generator: Synthesizes multi-view videos from the generated trajectories.

Adversarial vehicle suddenly cuts in; ego vehicle slightly steers right to avoid.

Rear adversarial vehicle suddenly accelerates; ego vehicle changes lane left to evade.

Front adversarial vehicle suddenly slows down; ego vehicle changes lane and decelerates to avoid.

At night, rear adversarial car suddenly accelerates; ego vehicle accelerates forward to evade and avoids the vehicle ahead.

Rear adversarial vehicle suddenly accelerates; ego vehicle first speeds up, then changes lane right to evade.

Front adversarial vehicle suddenly slows down; ego vehicle also decelerates to evade.

Rear adversarial vehicle suddenly accelerates; ego vehicle also speeds up to evade.

Rear adversarial vehicle suddenly accelerates; ego vehicle also speeds up to evade.


Video Comparison

Due to file size limitations, videos are compressed and quality might be affected.


Origin’s video lacks safety criticality, Naive’s video shows distortion at the end, while SMD’s video shows the ego vehicle timely braking when the lead vehicle brakes, ensuring both safety criticality and higher video realism.

Origin’s video lacks safety criticality, Naive’s video shows distortion at the end, while SMD’s video shows the ego vehicle accelerating to evade when the following vehicle suddenly speeds up, ensuring both safety criticality and higher video realism.