Ajjen Joshi: "Personalized Face and Gesture Analysis Using Hierarchical Neural Networks" (PhD Defense)

  • Starts: 11:00 am on Wednesday, July 25, 2018
Abstract: The video-based computational analyses of human face and gesture signals encompass a myriad of challenging research problems involving computer vision, machine learning and human computer interaction. In this thesis, we focus on the following challenges: a) the classification of hand and body gestures along with the temporal localization of their occurrence in a continuous stream, b) the recognition of facial expressivity levels in people with Parkinson's Disease using multimodal feature representations, c) the prediction of student learning outcomes in intelligent tutoring systems using affect signals, and d) the personalization of machine learning models, which can adapt to subject and group-specific nuances in facial and gestural behavior. Specifically, we first conduct a quantitative comparison of two approaches to the problem of segmenting and classifying gestures on two benchmark gesture datasets: a method that simultaneously segments and classifies gestures versus a cascaded method that performs the tasks sequentially. Second, we introduce a framework that computationally predicts an accurate score for facial expressivity and validate it on a dataset of interview videos of people with Parkinson's disease. Third, we collect a unique dataset of videos of students interacting with MathSpring, an intelligent tutoring system, and build models to predict learning outcomes from their facial affect signals. Finally, we propose a novel solution to a relatively unexplored area in automatic face and gesture analysis research: personalization of models to individuals and groups. We develop hierarchical Bayesian neural networks to overcome the challenges posed by group or subject-specific gesture variations. We successfully validate our formulation on the problems of personalized subject-specific gesture classification, context-specific facial expressivity recognition and student-specific learning outcome prediction. We demonstrate the flexibility of our hierarchical framework by validating the utility of both fully connected and recurrent neural architectures. Examining Committee: Margrit Betke (Major Professor and First Reader) Stan Sclaroff (Major Professor and Second Reader) Kate Saenko (Third Reader) Abraham Matta (Chair of Committee) Jacob Whitehill (Additional Member)
Hariri Seminar Room, 111 Cummington Mall

Back to Calendar