Convergent Seminar in Photonics & Computing: Tianyu Wang

Title: Optical-Neural-Network Image Sensors: Neuromorphic Computing and Sensing in the Optical Domain

Abstract: Machine learning has significantly transformed how real-world data, such as natural languages and scenes, is processed and analyzed. However, due to the ever-growing computational demands of modern deep neural networks, their efficient implementation on novel hardware is an area of active research for various applications, such as autonomous vehicles, industrial inspection, and biomedical assays. Optical neural networks (ONNs) hold the potential to transform deep-learning computation by utilizing efficient optical processes as neural-network operations. Recently, we demonstrated an ONN using less than 1 photon for each scalar multiplication, revealing the potential of ONNs to process information with stochastically quantized photons.

Inspired by the ultra-high energy efficiency of optical processing, we developed ONN image sensors– a multilayer, nonlinear ONN that compresses images in the optical domain, prior to any photodetection or digital post-processing. Through nonlinear optical pre-processing, our ONN image sensor can capture essential information of natural images with only a few camera pixels, allowing machine vision with lower latency, higher throughput, or even fewer photons. Additionally, our ONN image sensors were designed to process incoherent, broadband light directly scattered from real 3D objects, making them suitable for passive detection involving natural light illumination or fluorescence. For example, the ONN image sensor could be used to make image-based flow cytometers that sort 1,000,000 cells per second - with a faction of cost and phototoxicity as state-of-the-art conventional systems.

In this talk, I will discuss the origins of the optical processing advantages for deep-learning computation, and how these advantages can be further scaled for larger models, such as transformers for natural language processing. To achieve even larger advantages for machine vision, we demonstrated ONNs image sensors capable of pre-processing optical information in spatial, temporal, and frequency dimensions, all natively in the optical domain, without speed or energy overheads caused by digital electronics.

Bio: Tianyu Wang received his Ph.D. in Applied Physics from Cornell in Aug 2018 for his research on neuronal calcium imaging in the deep brain with three-photon microscopy under the supervision of Prof. Chris Xu. He is now a Schmidt AI in Science Postdoctoral Fellow in the group of Prof. Peter McMahon at Cornell University. His current research focuses on novel optical information processing techniques, such as optical neural networks and their applications to machine learning, optimization, and image sensing for automation and biomedical sciences. He is the recipient of SPIE Photonics West JenLab Young investigator Prize in 2017.