Demonstrably Safe AI For Autonomous Driving - Waymo

Autonomous driving is the ultimate challenge for AI in the physical world. At Waymo, we’re solving it by prioritizing demonstrably safe AI, where safety is central to how we engineer our models and AI ecosystem from the ground up. As a result, we’ve built an incredibly advanced AI system safely operating in the physical world at scale. With well over 100 million fully autonomous miles driven, we are making streets safer where we operate — achieving a more than ten-fold reduction in crashes with serious injuries compared to human drivers. Now, we invite you inside the engine room. This post offers a detailed look at Waymo’s AI strategy and how it’s fueling our momentum, allowing us to safely bring our service to more riders, faster than ever before. We will unpack our holistic AI approach, centered around the Waymo Foundation Model, which powers a unified demonstrably safe AI ecosystem that, in turn, drives accelerated, continuous learning and improvement. Waymo’s Holistic Approach to AIUnlike other AI applications that may optimize for capability first and layer on safety later, in autonomous driving, safety cannot be an afterthought. At Waymo, it’s the non-negotiable foundation upon which we build our AI ecosystem. Achieving demonstrably safe AI — where safety is proven, not just promised — requires a holistic approach. Beyond a smart and capable Driver, you also need a closed-loop, realistic Simulator to train and rigorously test the Driver in a myriad of challenging situations, and a sharp Critic to evaluate the Driver's performance and identify areas for improvement.The power is in unity. Developed jointly and with safety at their core, our Driver, Simulator, and Critic are all fueled by the same underlying AI — the Waymo Foundation Model — creating a continuous virtuous cycle.Waymo Foundation Model: Cornerstone of Waymo AIThe Waymo Foundation Model is a versatile, state-of-the-art world model powering our AI ecosystem. Its innovative architecture provides significant benefits over the pure end-to-end or modular approaches. In particular, the model leverages the full expressibility of learned embeddings as a rich interface between model components and supports full end-to-end signal backpropagation during training. At the same time, its additional compact, materialized structured representations like objects, semantic attributes, and roadgraph elements allow for: Powerful correctness and safety validation at inference time in the DriverHighly efficient, physically-correct and realistic closed-loop Simulation at extremely large scaleStrong verifiable feedback signals for evaluation by the Critic and reinforcement learning during trainingThe Waymo Foundation Model employs a Think Fast and Think Slow (also known as System 1 and System 2) architecture with two distinct model components:Sensor Fusion Encoder for rapid reactions. This perceptual component of the foundation model fuses camera, lidar, and radar inputs over time, producing objects, semantics, and rich embeddings for downstream tasks. These inputs help our system make fast and safe driving decisions.Driving VLM for complex semantic reasoning. This component of our foundation model uses rich camera data and is fine-tuned on Waymo’s driving data and tasks. Trained using Gemini, it leverages Gemini’s extensive world knowledge to better understand rare, novel, and complex semantic scenarios on the road. For instance, in an extremely rare scenario where there’s a vehicle on fire on the road ahead, while the physical space and drivable lanes might be clear for passage, the VLM can contribute a semantic signal prompting the Waymo Driver to take a different route or turn around. Both encoders feed into Waymo’s World Decoder, which uses these inputs to predict other road users behaviors, produce high-definition maps, generate trajectories for the vehicle, and signals for trajectory validation. Waymo’s AI Ecosystem: Distilling Knowledge from Teacher to Student ModelsInformed by our holistic approach, the Waymo Foundation Model powers the Driver, Simulator, and Critic. We achieve this by first adapting it to each of these three tasks, resulting in large, high-quality Teacher models that excel in their specific roles. However, these Teacher models are too big to run on vehicles for real-time decision making or in the cloud to simulate and evaluate hundreds of millions of miles, so we safely distill them into smaller Student models. Distillation is key, as it allows us to retain the superior performance of large models within their more compact and efficient versions. As a result (and mirroring similar trends in other areas of AI), by first training powerful high-capacity Teacher models and then leveraging efficient distillation techniques, we are able to achieve much better scaling laws for the resulting students.Driver. Our Teacher Driver models are trained to generate safe, comfortable, and compliant action sequences. Through distillation we transfer their rich world understanding and reasoning capabilities to more efficient Student models, optimized for real-time onboard deployment. To maximize the benefits of distillation, our onboard architecture is designed to mirror the Waymo Foundation Model structure. Importantly, the Waymo Driver employs a separate and rigorous onboard validation layer, which then verifies the trajectories produced by the Driver’s generative ML model.Simulation is an essential tool for closed-loop training and testing of our Driver across a range of diverse and challenging scenarios, including potential collisions, inclement weather, intricate intersections, and unusual behaviors on the road. The Simulator Teacher models are capable of creating high fidelity, multi-modal dynamic worlds to evaluate our Driver. The student models are compute-efficient versions of these larger models that are designed to run the massive scale of simulations that are needed for the robust evaluation of the Driver. The Waymo Foundation Model’s architecture allows us to seamlessly combine compact materialized world-state representations and sensor simulation, unlocking large-scale, hyper-realistic and physically correct, yet computationally efficient virtual environments.Critic. Our world-class evaluation system is designed to stress-test the Waymo Driver, proactively identify subtle edge cases, and enable rapid, targeted improvements. The Critic Teacher models can analyze driving behavior and generate high-quality signals, used for training Student models and for automatically building rich evaluation datasets. Then the Critic Student models analyze driving logs, identify interesting or problematic scenarios, and provide nuanced feedback on driving quality.Powered by the Waymo Foundation Model, all of these components comprise a seamless AI ecosystem and