ECE PhD Prospectus Defense: Param Budhraja

ECE PhD Prospectus Defense: Param Budhraja

TItle: When Deleting Data Helps: Theory and Practice in Adaptive Reinforcement Learning

Presenter: Param Budhraja

Advisor: Professor Alex Olshevsky

Chair: Professor Yannis Paschalidis

Committee: Professor Alex Olshevsky, Professor Venkatesh Saligrama, Professor Yannis Paschalidis, Professor Brian Kulis

Google Scholar Link: https://scholar.google.com/citations?user=qFNsAOoAAAAJ&hl=en

Abstract: Reinforcement learning (RL) agents achieve strong performance when evaluated in the same environment they are trained in, but this says little about whether a learned policy continues to perform well once the environment changes, as real-world dynamics inevitably do, e.g. through wear on a robot's actuators or an unavoidable mismatch between simulation and reality. We study Adaptive RL, the problem of training a policy that adapts to environments it has not previously encountered from within a structured family. This is formalized as a Contextual MDP in which each environment is indexed by a low-dimensional context (e.g. mass, length, friction) that is unavailable to the agent at deployment. We build on the standard approach to this problem, which decouples estimating the context from an observed trajectory from acting optimally given an estimated context. We find that a simple, counterintuitive intervention, exponential windowing, in which the probability that an old trajectory survives decays exponentially with each round of new data collection, substantially improves this system's performance.