CISE Seminar: Aditya Gangrade, Boston University

Date: March 20, 2026
Time: 3:00pm – 4:00pm
Location: 665 Commonwealth Ave., CDS 1101

Algorithmic Behaviours in In-Context Learning

In-Context Learning (ICL) is a remarkable phenomenon whereby transformer-based LLMs can use data contained within their prompts to adapt their responses, without changing their weights. This suggests that such models encode learning mechanism. The recent literature has used statistical learning problems as a test-bed to investigate ICL, and established that ICL can be realised for a wide range of function classes. However, the mechanisms these models use to learn are poorly characterised.

I will describe work on extracting and analysing learning algorithms embedded in the weights of transformers trained to perform ICL in two settings: linear-activation transformers for linear regression, and softmax-activation transformers for linear classification. Through the former, I will illustrate a high-level ‘simplify and validate’ strategy that allows extraction, and through the latter, I will describe a symmetry-driven strategy for evoking structure in these weights. In these settings, we recover concrete iterative procedures that employ use existing ideas (Newton-Schulz; mean-shift methods) in new ways that are distinct from gradient descent. Further, we show that transformers trained on variations of these problems implement modified versions of the same dynamics. This suggests that such models recover certain `stable’ algorithmic motifs, and adapt them in response to problem structure.

Based on work done jointly with Patrick Lutz, Themistoklis Haris, Arjun Chandra, Hadi Daneshmand, and Venkatesh Saligrama.

Aditya Gangrade is a research scientist at the ECE department at Boston University. He obtained his Ph. D. in Systems Engineering from Boston University, and previously held postdoctoral positions at Carnegie Mellon University and the University of Michigan. His research interests span theoretical and methodological aspects of machine learning, with recent focus on safety in sequential decision making, and on in-context learning phenomena.

Faculty Host: Venkatesh Saligrama
Student Host: Ilker Isik