Training Precise Language Models for Imprecise Humans - Valentina Pyatkin

Abstract: This talk examines methods for enhancing language model capabilities through post-training. While large language models have led to major breakthroughs in natural language processing, significant challenges persist due to the inherent ambiguity and underspecification in language. I will present a spectrum ranging from underspecification (preference modeling) to full specification (precise instruction following with verifiable constraints), and propose modeling approaches to increase language models’ contextual robustness and precision. Valentina will demonstrate how models can become more precise instruction followers through synthetic data, preference tuning, and reinforcement learning from verifiable rewards. And she will address constrained instruction following generalization challenges and present post-training methods for improvement. On the preference data side, she will illustrate patterns of divergence in annotations, showing how disagreements stem from underspecification, and propose alternatives to the Bradley-Terry reward model for capturing pluralistic preferences. The talk concludes by connecting underspecification and reinforcement learning through a novel method: reinforced clarification question generation, which helps models obtain missing contextual information that is consequential for making predictions. Throughout the presentation, Valentina will synthesize these research threads to demonstrate how post-training approaches can improve model steerability and contextual understanding when facing underspecification. Bio: Valentina Pyatkin is a postdoctoral researcher at the Allen Institute for AI and the University of Washington, advised by Prof. Yejin Choi. She is additionally supported by an Eric and Wendy Schmidt Postdoctoral Award. She obtained her PhD in Computer Science from the NLP lab at Bar Ilan University. Her work has been awarded an ACL Outstanding Paper Award and the ACL Best Theme Paper Award. During her doctoral studies, she conducted research internships at Google and the Allen Institute for AI, where she received the AI2 Outstanding Intern of the Year Award. She holds an MSc from the University of Edinburgh and a BA from the University of Zurich. Valentina's research focuses on post-training and the adaptation of language models, specifically for making them better semantic and pragmatic reasoners.