ECE PhD Thesis Defense: Jimuyang Zhang

ECE PhD Thesis Defense: Jimuyang Zhang

Title: Towards a Scalable and Generalized End-to-End Policy for Autonomous Driving

Presenter: Jimuyang Zhang

Advisor: Professor Eshed Ohn-Bar

Chair: Professor Wenchao Li

Committee: Professor Eshed Ohn-Bar, Professor Venkatesh Saligrama, Professor Yannis Paschalidis, Professor Philipp Krähenbühl

Google Scholar Link: https://scholar.google.com/citations?user=9spN7eUAAAAJ&hl=en

Abstract: End-to-end frameworks for autonomous driving, which map raw sensor input directly to vehicle control signals, offer advantages of integrated optimization for both perception and planning but face challenges in scalability, interpretability, and robustness. In this work, we introduce a series of improvements across representation pre-training, policy learning from diverse supervision, and structured sensorimotor learning to enhance generalization and scalability in end-to-end driving policies. To learn scalable representations via pre-training, we propose a neural volumetric world modeling (NeMo) approach that can be pre-trained in a self-supervised manner for image reconstruction and occupancy prediction tasks, benefiting scalable imitation learning. We further enhance policy learning through diverse supervision. Specifically, we propose the Learning by Watching (LbW) framework that enables learning a driving policy from non-ego vehicles, and SelfD, a semi-supervised framework for learning scalable driving by utilizing large amounts of online monocular images. In the end, to scaffold the difficult sensorimotor learning task, we present Coaching a Teachable Student (CaT), a novel distillation scheme that is trained with richer supervision in feature space and optimized via a student-paced coaching mechanism. We also introduce FeD that leverages advances in Large Language Models (LLMs) to provide corrective fine-grained feedback regarding the underlying reason behind driving prediction failures to improve robustness in complex driving scenarios. Extensive evaluations demonstrate that our approach significantly improves generalization and robustness, bridging the gap between simulation and real-world deployment and advancing the scalability of autonomous driving systems.