Beauty is in the eye of the beholder

Visual art has long been regarded as a powerful means of expressing emotions, but perhaps never before has this been so literal. Now a new software program written by a team from BU and Bath University in England can change digital pictures in response to the facial expressions of the viewer.

A camera poised above the digital image registers the viewer’s expression, and the strength and shape of the strokes in the ‘canvas,’ as well as the colors, change accordingly, transforming the image into a nonrealistic representation similar to an impressionist oil. The artwork shows vibrant colors when the viewer has a happy face, but turns blue and the lines smooth out as the viewer expresses, if not genuine sadness, at least a convincing charade.

The project is a collaboration of Maria Shugrina (CAS’07), Margrit Betke, a College of Arts and Sciences associate professor of computer science and Shugrina’s mentor at BU, and John Collomosse, a professor from the University of Bath in England, where Shugrina studied last summer.

The computer science major was amazed by the response to a paper on the subject she presented recently in France. She was flooded with e-mails after an article about the program appeared in newspapers and blogs around the globe. Press inquiries and congratulations came from countries as distant as Brazil, Australia, and India, she says. One artist wrote asking whether the software could work backwards: instead of reflecting a person’s moods, could it affect moods and thus be used as a form of art therapy?

“I guess it is cool if a computer can figure out if you are smiling or sad,” Betke says about the response. But “we are not claiming that we could detect if someone is sad,” Shugrina quickly clarifies. “We are claiming that we can detect if someone is trying to make a sad face.”

The system is “trained” to recognize only Shugrina’s facial expressions at present, but “it’s an ongoing project,” Betke says, and can eventually be customized for other users, or standardized.

Using images collected through a webcam, the software analyzes eight key facial features — such as the position of the mouth and the arch of the eyebrows — and projects them into a graphic that describes the emotional state of the viewer with only two words: pleasure and arousal. Developed by BU, this technology has several applications, including a “facial mouse” that allows people with severe disabilities to click or type on a screen-keyboard by just winking an eye. But this project’s idea of interpreting facial features as emotions is new.

Programs that create nonrealistic representations of digital pictures are already available in simpler versions in most image-processing software systems and even on photo-print kiosks at drugstores. But changing the way the artist — or the system user — creates the outcome, called “empathic painting,” is a new form of painting.

“One of the things I am interested in is providing animators with more freedom,” Shugrina says, “so that they can have more impact on the final outcome.”

The project’s feedback from the artistic community is encouraging. Scott Dasse, lead designer at BU’s Office of New Media, has been working in interactive media for more than seven years and envisions gaming and interactive film applications. “Sounds like a new input device — like a facial mouse — that could be interesting in a virtual reality way,” he says. A potential field of application is computer-generated characters such as Gollum from the Lord of the Rings trilogy, Dasse says ¬— “Imagine a CG character that can read and react to your nonverbal communication.”

“I think it’s a fascinating interface,” says George Fifield, director of the Boston Cyberarts Festival and curator of new media at the DeCordova Museum. “The history of interactive art has been using a great deal of instruments,” he says. Knowing how to use a mouse, a keyboard, or a computer has been increasingly necessary, but with this software, he says, “you just have to be you. It’s fascinating.”

There are many potential uses for the software. Artists could use it to create interactive art or as a painting tool. Betke points to another possible application: homeland security. “Understanding facial features could be used to try to find out if someone is lying,” she says. For now, it’s entertaining to imagine the next Van Gogh making faces to a webcam embedded in his brand-new laptop.

To see a video of an "empathic painting" in action, click here.