CISE Seminar: Ali Jadbabaie, Massachusetts Institute of Technology

The Pitfalls of Imitation Learning when Actions are Continuous

This talk will present our recent study of the problem of imitating an expert demonstrator in a discrete-time, continuous state-and-action control system. Over the past few years, Imitation Learning (IL) has become a topic of intense focus in the Reinforcement Learning (RL), Robotics and autonomy literature. In its simplest form, imitation learning is an approach that tries to learn an expert policy by querying samples from an expert (usually a human). We show that, even if the dynamics of the system is benign and is exponentially stable, and the expert policy is smooth and deterministic, any smooth, deterministic imitator policy necessarily suffers error on execution that is exponentially larger, as a function of problem horizon, than the error under the distribution of expert training data. Our negative result applies to both behavior cloning and offline-RL algorithms, unless they produce highly “improper” imitator policies — those which are non-smooth, memoryless, or which exhibit highly state-dependent stochasticity — or unless the expert trajectory distribution is sufficiently “spread” to have wide support. In addition, we provide experimental evidence of the benefits of these more complex policy parameterizations, explicating the benefits of today’s popular policy parameterizations in robot learning (e.g. action-chunking and Diffusion-policies). We also establish a host of complementary negative and positive results for imitation in control systems.

Ali Jadbabaie is the JR East Professor and Head of the Department of Civil and Environmental Engineering at Massachusetts Institute of Technology (MIT), where he is also a core faculty in the Institute for Data, Systems, and Society (IDSS) and a Principal Investigator in the Laboratory for Information and Decision Systems. Previously, he served as the Director of the Sociotechnical Systems Research Center and as the Associate Director of IDSS as co-founder of its flagship PhD program in Social and Engineering Systems. He received a B.S. degree with High Honors in electrical engineering with a focus on control systems from Sharif University of Technology, an M.S. degree in electrical and computer engineering from the University of New Mexico, and a Ph.D. degree in control and dynamical systems from the California Institute of Technology. He was a Postdoctoral Scholar at Yale University before joining the faculty at the University of Pennsylvania, where he was subsequently promoted through the ranks and held the Alfred Fitler Moore Professorship in network science in the Department of Electrical and Systems Engineering. He is a recipient of a National Science Foundation Career Development Award, an US Office of Naval Research Young Investigator Award, the O. Hugo Schuck Best Paper Award from the American Automatic Control Council, and the George S. Axelby Best Paper Award from the IEEE Control Systems Society. He has been a senior author of several student best paper awards, in several conferences including the American Control Conference , IEEE Conference on Decision and Control, and IEEE International Conference on Acoustics, Speech, and Signal Processing . He is an IEEE fellow, and the recipient of a Vannevar Bush Fellowship from the Office of Secretary of Defense. He is a member of the Bush Fellows Research Study Group, as well as the National Academies Intelligence Science and Technology Advisory Group (ISTEG). His research interests are broadly focused on decision making, optimization and control, machine learning, network science and network economics, as well as quantitative and computational social science.

Faculty Host: Jinglong Zhao

Student Host: Chae Woo Lim