Between news articles, tweets, Facebook memes, online videos, Reddit threads, and all the other media sources that we routinely tap into, it seems impossible these days to be informed without being overloaded, or to stay connected without getting hopelessly tangled up.
But what if there were a machine or mechanism that could take in all that information—words, pictures, posts, and videos, even in dozens of different languages—and somehow make sense of it all? Better yet, what if that machine could measure public sentiment about any given event, figure out how different media outlets are covering it, and unscramble the relationship between the two?
That’s the goal of a boundary-breaking collaboration between Boston University researchers Margrit Betke, College of Arts & Sciences professor of computer science, Prakash Ishwar, College of Engineering professor of electrical and computer engineering and systems engineering, Lei Guo, College of Communication assistant professor of emerging media studies, and Derry Wijaya, College of Arts & Sciences assistant professor of computer science. In September 2018, the BU team received a $1 million, four-year research grant from the National Science Foundation to advance their work.
The research itself is as much a part of this unusual story as the researchers who pulled it together while coming from a diverse collection of BU schools.
A Lecture. A Spark.
The backstory of how these unlikely collaborators found each other goes back to a lecture hall at the Hariri Institute for Computing. That’s where, in November 2015, Guo, then brand-new to the BU faculty, delivered a packed-house talk as a Hariri Junior Faculty Fellow about using “big data” methods to analyze online communication. Guo, who studies how media influences public opinion (and vice versa), was explaining how she had used computers to help analyze 77 million tweets about the 2012 presidential election, in which President Barack Obama defeated Republican Mitt Romney.
Until that point, media research had traditionally been done manually, with students and scholars laboriously poring over and classifying text. Guo wanted to show that computers could help make sense of data troves—like those election tweets—too big to be parsed by hand.
Betke was in the audience, and she was dazzled. “I was sitting there like, ‘Wow, this is so exciting!’” But Betke studies computer vision, not text. She didn’t see how her computer science expertise could apply to the political problems Guo was working on.
That might have been the end of the story, but Betke’s graduate student Mehrnoosh Sameki (GRS’17) was just as exhilarated by Guo’s talk, and even keener to pursue a partnership. Betke recalls Sameki insisting, “Can’t we just think of something that connects the fields?”
That thought was still simmering a few months later, when the researchers ran into each other at BU Data Science Day, another Hariri Institute event, and began sketching out a collaboration.
The team would also include Ishwar, who studies machine learning—that is, creating computer algorithms that can “learn” to make decisions based on a set of examples. “Machine learning helps us to scale up the processing and analysis of big data,” says Ishwar. “It can never be a complete replacement of human expertise—at least not in the near future—but it is a catalyst which aids, accelerates, and amplifies human expertise–based analysis of data.”
They quickly discovered that their fields had more in common than they thought.
“There are machine learning tools that actually apply to both fields: looking for patterns in images, looking for patterns in text. That enabled us to work together,” said Betke.
But the team wasn’t complete.
The Missing Piece
“We needed a fourth person who is really good at automated analysis of text,” recalls Betke. That person was Derry Wijaya, who joined the BU faculty in September 2018. Wijaya is an expert in natural language processing—that is, making computer programs that understand ordinary speech and writing, not computer code—and is especially interested in multilingual systems that can learn as many as 100 languages, even with minimal input. “We had this perfect team of different backgrounds come together and the expertise is just such a good mix,” says Betke.
The story illustrates the precise kind of planned serendipity that the Hariri Institute wants to cultivate, says its director, Azer Bestavros, CAS professor of computer science. “Computer science is not just about engineering computing devices and platforms anymore—it’s really about the innovation that emerges from integrating our ways of thinking and our ways of doing into every discipline,” says Bestavros. “The vision of the Institute is to make that happen by connecting computer scientists with opportunities that span the landscape of academic disciplines,” he says.
In addition to programs that expose researchers to the possibilities of data science, the Hariri Institute also serves as an “incubator” that disperses small seed grants.
And so, in 2016, with another presidential contest looming, Guo, Ishwar, Betke, and their collaborators applied for and won a Hariri Research Award to develop their techniques and apply them to tweets and YouTube videos about the election. The year after that, they parlayed that success into a Google research award. Their goal: to make a better Google news feed, one that could automatically incorporate diverse viewpoints and serve as an antidote to the “echo chamber effect” that exposes readers only to stories that reflect and reaffirm their existing opinions.
The $1 million NSF grant is the biggest yet for the project, and will unfold in three phases: First comes the data gathering—collecting thousands of news stories, lead photos, video clips, comments, tweets, and more; then, tapping into crowdsourcing workforces like Amazon’s Mechanical Turk, as well as analysis efforts from communication researchers, they will begin manually analyzing the relatively small selection of the media in their library. They will start with basic questions—What is this article about? Who is in it? Where did it take place?—and move on to trickier judgments, like whether the item is largely positive or negative; finally, using machine learning, they will begin “training” the computer to make human-like judgments about the media in their library.
Other big-data studies have used artificial intelligence to pick out the who-what-where from news stories. What makes this one unique is it will be the first to try to determine something subtler: the story’s particular point of view, or frame.
“To analyze a mass shooting, for example, there are a lot of perspectives to talk about it,” says Guo. The same event could become a human-interest story, a policy op-ed, or an economic impact report. “Framing analysis in our field is always done using student coders or researchers to manually go through the article,” says Guo. “We want to see the limits of what machine learning can do.”
The team hopes that including images and video in their analysis will help the computer judge framing more accurately. For example, the lead image on a human-interest story is likely to be a face or faces, while a story on new legislation might be topped with a picture of the White House.
The researchers will also look at how stories are told differently around the world. “We’re expecting that the frames that people use outside the US are different from the ones in the US,” says Betke.
Even before it’s analyzed, the dataset, incorporating media in 100 different languages, will be a first-of-its-kind tool for researchers who study language, says Wijaya: “Having a dataset actually is very important to move further in research, to push the boundaries of what’s possible.”
And the Hariri Institute is there to connect researchers who want to push boundaries, says Bestavros. “How do you empower them, or magnify what they do?” he asks. “You connect them to one another and help them incubate their ideas.”