CS Distinguished Lecture: "Towards perceptual AI assistants" with Lorenzo Torresani

Abstract: What if you had an AI assistant that seamlessly observes and understands your actions and is available at any time to remind you where you misplaced your objects, to automatically transcribe "who" said "what" in a meeting, or to interactively guide and track your progress in the execution of complex activities, such as cooking a recipe, or fixing a bike? In this talk, I will present our research towards this vision through the development of multimodal video-language systems that can assist humans in their daily lives by perceiving and predicting their behavior from wearable cameras. Our advances in this field include models that detect and describe not only what the user is doing but also how the activity is performed to provide relevant guidance and improvement suggestions in language form. I will also discuss our contributions to the broad research community by creating and open-sourcing multiple foundational datasets to promote the next generation of multimodal perceptual AI.

Bio: Lorenzo Torresani is a Computer Science Professor and a President Joseph E. Aoun Chair at Northeastern University. He is the author of the first methods for 3D reconstruction of non-rigid objects and influential architectures for image and video analysis (e.g., Classemes, C3D, TimeSformer). His current research interests are in video-language models. He received his Ph.D. in Computer Science from Stanford University in 2005. In the past, he held a tenured Professor position at Dartmouth College and was a Research Director at Meta, Fundamental AI Research (FAIR). He also worked at Microsoft Research, Like.com, and Digital Persona. He is the recipient of multiple awards, including a CVPR Best Student Paper prize, a National Science Foundation CAREER Award, a Google Faculty Research Award, three Facebook Faculty Awards, and a Fulbright U.S. Scholar Award.