ECE PhD Thesis Defense: Qinzi Zhang

ECE PhD Thesis Defense: Qinzi Zhang

Title: Towards New Perspective On Stochastic Optimization: Bridging Theory and Practice

Presenter: Qinzi Zhang

Advisor: Professor Ashok Cutkosky

Chair: TBD

Committee: Professor Ashok Cutkosky, Professor Alex Olshevsky, Professor Bobak Nazer, Professor Xuezhou Zhang

Google Scholar Link: https://scholar.google.com/citations?user=QYP73uQAAAAJ&hl=en

Abstract: Optimization theory is a cornerstone of machine learning. At the core of training deep neural networks lies the optimization of a loss function. However, despite the success of widely-used empirical optimizers and advancements in theoretical frame works, a notable gap between theory and practice remains. This gap often arises from theoretical assumptions that do not reflect real-world conditions.

This thesis addresses these discrepancies through two main studies. The first study focuses on optimization algorithms with differential privacy guarantees, highlighting the challenges and adaptations required to maintain privacy without compromising efficiency. It introduces online-to-batch style reduction frameworks that convert online convex optimization (OCO) algorithms into private optimization algorithms applicable under varying convexity and smoothness conditions.

The second study delves into non-smooth non-convex optimization by introducing a new convergence criterion that relaxes the standard notion of the Goldstein stationary point. It also proposes a reduction framework that converts OCO algorithms into stochastic optimization algorithms suitable for non-smooth non-convex scenarios. This framework notably proves the convergence of stochastic gradient descent with momentum (SGDM) under these conditions.

Together, these studies deepen our understanding of optimization by developing new theories that better reflect practical conditions and address realistic needs, thereby bridging the gap between theory and practice.