SparkNotes for Surveillance
Like it or not, we’re being watched. Surveillance video networks are increasingly hovering over workplaces, street corners, businesses, hospitals, and public spaces. According to a 2008 estimate in Popular Mechanics, there are now more than 30 million surveillance cameras in the United States, recording about four billion hours of video every week. Many of these electronic eyes were installed after the 2001 terror attacks and are meant to make us safer. But, as electrical and computer engineers Prakash Ishwar and Janusz Konrad point out, the collected data still needs human monitoring to be useful. Who can watch that much footage, especially when most of it is mind-numbingly uneventful?
“It’s a problem of information overload,” says Ishwar. He and Konrad are working on a way to mine that data using computer algorithms to condense massive videos into much shorter “digests” that can be more easily reviewed. Their technology could revolutionize security networks and other long-term video surveillance applications, such as environmental and animal habitat monitoring.
Prakash Ishwar and Janusz Konrad
Instead of cutting entire frames or simply fast forwarding, Ishwar and Konrad’s “ribbon carving” algorithm identifies and removes fine slices of “background” pixels—basically, whatever is not moving or changing from frame to frame—from individual frames of the video as it moves through time. Ribbon carving significantly shortens the amount of time required to watch a video, while also minimizing alterations in spatial and temporal relationships between objects moving in that video—cars, for instance, or people, or an endangered species. Currently, the technology developed by Ishwar and Konrad can reduce videos in time by a factor of between five and 20.
Footage of the Massachusetts Turnpike on their website, taken by a video camera on the roof of a campus building, illustrates the concept. Around midday, four lanes of very light traffic speed down the highway during a 12-minute interval. Every few seconds, a car, van, truck, or bus cruises by in one lane or another; occasionally a few vehicles move through the frame simultaneously. Crucially, however, each individual lane has many empty periods, and there are also brief sequences when the short stretch of highway in view is vacant. The algorithm carves out those empty slices. In the resulting condensed video, which can be viewed in under two minutes, the silver sedan still drives by first in the far left lane, followed by the black jeep and the red SUV in the middle lane, and then the 18-wheeler on the far right. But now, the significant gaps in time separating their appearances are gone: each vehicle follows closely on the heels of the last. Inactivity on the highway is kept to a minimum.
A condensed video still requires a human to search through it for anomalies or incidents of interest, says Konrad, but it makes that job much more efficient. One person could review footage from dozens of cameras, he says, “and not get dizzy.” Likewise, adds Ishwar, reviewers would need to go back to the original footage to judge the true timing of things, such as the exact interval between the sedan and the 18-wheeler. In such cases, he says, the condensed video could provide time-saving search cues, much like “a table of contents or an index in a book.”
Video condensation can reduce viewing time by a factor of 20. In a test sequence of traffic footage, vehicles seen in three different frames of the original video (A-C) appear together in the condensed video (D).
Video frames courtesy of Prakash Ishwar and Janusz Konrad
Ishwar and Konrad are now refining their technology by making the code more professional, optimizing the algorithm to condense videos by a factor of 100 to one, and creating a model that can accommodate cameras with the ability to pan and zoom.
In the meantime, plans to commercialize the technology are in the works, led by former graduate student Stephen Chao, who earned a master’s in computer engineering at BU, and current doctoral candidate Ajay Bangla. Listing Konrad as chief technology officer and Ishwar as vice president of engineering, the two students developed a business plan that took first prize in a competition sponsored by BU’s Student Association of Graduate Engineers. Chao and Bangla will soon start investigating patents, raising venture capital, creating a marketing plan, and, hopefully, seeing their efforts turn a technology into a business.
“There are so many challenges remaining that I don’t even know where to start,” says Chao. “But luckily, in our case, the first big challenge was taken care of—coming up with a great idea.”