Artificial intelligence is becoming part of everyday life, powering everything from chatbots to image generators. But behind the scenes, running these massive AI models takes an enormous amount of computer power and electricity.
Professor Ajay Joshi (ENG, ECE) is tackling this problem by designing a new type of computer chip that uses not only electronics but also light. His work is supported by a National Science Foundation (NSF) award, Heterogeneously Integrated Electronic Photonic AI Accelerators (HIEPAA). The award supports groundbreaking work on “heterogeneously integrated electronic-photonic AI accelerators.” While the phrase may sound complex, the idea is simple— make AI run faster and more efficiently, while using less energy, by building chips that combine traditional electronic circuits with photonics.
“Typically in the system there is a single processing chip that contains either a pure electronic solution or a mix of electronic and photonic. But when it comes to GenAI systems, we need many such chips, and having them on the board or across multiple boards is inefficient,” Joshi said.
“We put several electronic chips and photonic chips on a single wafer and connect them through a mix of electrical and photonic communication links, so that we can run entire algorithms in a single system and run them efficiently.”
Most AI systems today rely on electronics alone to handle billions of math operations every second. One of the most common and time-consuming of these tasks is matrix multiplication, where large grids of numbers are multiplied together.
Photonics offers a major advantage— light can perform these matrix operations much faster and with lower energy consumption than electronics.
By moving the heavy-lifting to light-powered chips while leaving other tasks to electronics, the system can strike a balance between speed, accuracy, and efficiency.
But using light comes with a challenge: photonic systems are less precise than electronic ones, meaning they struggle with handling very detailed calculations.
“With photonics, there is a limit on the precision, like how many bits per operation that you can use,” Joshi said. “I can convert a big 64-bit number into multiple 8-bit numbers, then process these 8-bit numbers very efficiently in the RNS domain, and then I can go back to 64-bit precision.”
Another innovative part of the project is the use of wafers, large discs that chips are made from. Typically, a system might include many individual chips spread across one or more boards, using power inefficiently.
Instead, Joshi aims to place many small electronic and photonic chips together directly on a single 12-inch wafer. Connected by a mix of electrical and optical links, the wafer acts like one giant “superchip” designed specifically to run AI models.
This approach could allow companies like OpenAI, Google, Microsoft, and Amazon to run their large language models more efficiently inside data centers, ultimately reducing both costs and environmental impact.
“For a non-engineer, ‘how does that affect my life?’ It would be a case where they are able to get AI-based services in various domains such as healthcare, banking, education, etc. in a timely and efficient manner, even cheaper,” Joshi said.
The NSF award also highlights another priority— education.
Joshi and his team will also train engineering students in both electronics and photonics, preparing them for careers in the rapidly growing field of AI.
“Everybody wants to use GenAI, but we have to build those systems that run GenAI-based applications,” he said. “There aren’t enough engineers who can go and build these systems; we have to train that next generation.”
Students working on the project will gain hands-on experience with cutting-edge technology, bridging the gap between academic research and industry demand.
For Joshi, the NSF award is not just about advancing technology but also about making a lasting impact in the AI field.
“This whole idea of making an impact in this field of Gen AI, from a hardware perspective, is very appealing,” he said. “Any small impact that I can make in this ever-growing field of GenAI would be fantastic.”
Ajay Joshi is a Professor in the ECE department at Boston University, where he leads the Integrated Circuits and Systems Group.
Professor Joshi’s research interests include computer architecture, security, VLSI design and silicon photonics. Prior to joining Boston University, he received his Ph.D. degree from the ECE Department at Georgia Tech in 2006 and then worked as a postdoctoral researcher in the EECS Department at MIT. He was a Visiting Research Scientist at Google in 2017-18. He has received numerous awards, including the NSF CAREER Award in 2012, Boston University ECE Department’s Award for Excellence in Teaching in 2014, Boston University Ignition Award in 2016, Best Paper Award at ASIACCS 2018, and Google Faculty Award in 2018 and 2019. He currently serves as the Associate Editor for IEEE Transactions on VLSI Systems.