Institute Hosts 4/4 DSI Colloquium with ECE

11:00 PM – 12:30 PM on Tuesday, April 4, 2017
Hariri Institute for Computing
111 Cummington Mall, Room 180

Join the Hariri Institute for Computing and the Electrical & Computer Engineering Department for a Data Science Initiative Colloquium.

Principled Tuning for Large-scale ML Systems
Ioannis Mitliagkas, PhD scholar, Statistics and Computer Science
Stanford University

Abstract: Modern machine learning systems rely on complex and distributed pipelines that require extensive hyperparameter tuning to achieve the desired performance. Careful tuning can result in significant speedups and improvements in solution quality. However, the dimensionality of the hyperparameter space often makes the use of brute-force search prohibitive. To make things worse, components can interact in unexpected ways and make joint tuning necessary. These challenges preclude non-experts from fully utilizing the potential of modern machine learning tools and call for a deeper understanding of the effect hyperparameters have on the quality and performance of a system. In this talk, I will discuss examples of tuning large-scale learning and inference systems. I will focus on recent work that reveals a previously unknown interaction between system and algorithm dynamics when running an asynchronous learning system. Asynchronous methods are widely used for their superior throughput, but have limited theoretical justification when applied to non-convex problems. I will show that running stochastic gradient descent (SGD) in an asynchronous manner can be viewed as adding a momentum-like term to the SGD iteration. This result does not assume convexity of the objective function, so is applicable to deep learning systems. Furthermore, using a hybrid parallel architecture we can control the level of asynchrony, a new hyperparameter. Theory then implies that jointly tuning momentum and the level of asynchrony can significantly reduce the number of iterations, necessary for an asynchronous system to achieve the same solution. This line of work provides a number of necessary components for realizing the vision of an automated machine learning pipeline.

About the Speaker:
Ioannis Mitliagkas is a postdoctoral scholar with the departments of Statistics and Computer Science at Stanford university. He obtained his Ph.D. from the department of Electrical and Computer Engineering at The University of Texas at Austin. His research focuses on large-scale statistical learning and inference problems, focusing on efficient large-scale and distributed algorithms, tight theoretical and data-dependent guarantees and tuning complex systems. His recent work includes understanding and optimizing the scan order used in Gibbs sampling for inference, as well as understanding the interaction between optimization and the dynamics of large-scale learning systems. In the past, he has worked on high-dimensional streaming problems and fast algorithms and computation for large graph problems.