A Smile That Controls Machines

Carolyn Michener (SAR’16) (above) is working with Sensorimotor Rehabilitation Engineering Lab Director Cara Stepp to help people use facial movement and sound to control human machine interfaces. Photos by Michael D. Spencer

Imagine turning on the lights, adjusting the thermostat, or operating a DVD player simply by smiling. For people who are visually or verbally impaired, or who have limited motor skills, this could be a major advance in communication. Carolyn Michener (SAR’16) is working to make it a reality.

An undergraduate in the speech, language, and hearing sciences program, Michener says her lifelong stutter and interest in engineering sparked a passion to develop technology to help others communicate. Working in the STEPP Lab for Sensorimotor Rehabilitation Engineering at Boston University’s Sargent College of Health & Rehabilitation Sciences, she’s collaborating on a project to help people use facial movement and sound to work with human machine interfaces (HMIs)—controls like keypads and touchscreens through which people operate machines, systems, and devices.

“An HMI needs some kind of feedback to properly tell the user what it’s doing,” says Michener, who joined STEPP Lab Director Cara Stepp and Sargent research engineer Sylvain Favrot on the project in 2012. Often this feedback is visual—for example, a control panel flashing a colored light or displaying a message confirming that an action has been completed. “But this can be difficult for people who are visually impaired or who find the visual stimuli distracting,” says Michener. The STEPP Lab project enables people to communicate with machines through sound—no seeing or touching required. Plenty of machines already do this—such as iPhone’s Siri, which allows users to send messages or search for information—but these systems often require voice commands, which are not applicable to people with impaired speech. With the new STEPP Lab technology, users can communicate with machines by using facial movements to create sound.

To test the technology, Michener trained study participants in what she describes as an auditory matching game, using preexisting STEPP Lab software that Favrot modified for the project. Sitting in a soundproof booth in the lab, Michener demonstrates how the game works.

She opens communication between the player and a computer, connecting them by way of two electrodes placed on either side of the lips. This connection enables the computer to translate the facial muscles’ electrical signals from the skin, a process called surface electromyography. The player undergoes a quick calibration procedure, dons a pair of headphones, receives Michener’s instructions—and is ready to begin.

A tone plays through the headphones for two seconds. This is the sound the player will try to match. Then, a second tone sounds. This is the player’s starting point, a low pitch in both ears that represents the player’s muscles at rest. The player now has 15 seconds to match the first sound’s pitch and location (left ear, right ear, or both) by contracting his or her facial muscles in just the right combination. Contracting left or right—in effect, smirking—creates a medium pitch in the corresponding ear. Contracting both sides—smiling—increases the pitch and activates the sound in both ears. The trial ends when either the player hits the target for one second or 15 seconds have expired. The player then receives a score representing how well he or she matched the target.

“A [human machine interface] needs some kind of feedback to properly tell the user what it’s doing. But this can be difficult for people who are visually impaired.”    — Carolyn Michener

While the search for the target sound is an auditory task for the user, the game’s software visually records both the target location and the user’s performance on a graph Michener can review on the computer. In 2013, she tested the game on 16 adults, each of whom completed three test sessions lasting 45 minutes.

After three days, users working with auditory feedback were able to communicate at an average speed of 40 bits per minute (bpm). While this speed is 50 times slower than typing on a keyboard and 15 times slower than the quickest computer mouse use, Stepp says, participants using auditory feedback were able to communicate with machines as effectively as participants using visual feedback in similar studies. “We can conclude that auditory feedback is a viable way to allow people to communicate with this kind of system,” says Michener.

Michener cowrote a paper about the project with Stepp and Favrot that she presented at the Acoustical Society of America’s biannual conference in May 2014. She continues to run trials of the game, this time to find out if players with a musical background perform better than others. Stepp says the team is also embarking on collaborations with Madonna Rehabilitation Hospital in Nebraska and the Perkins School for the Blind in Massachusetts to see how people who are blind and individuals with spinal cord injuries perform in and respond to the game.

“Ultimately I would like to see this technology in a device that can be used inside a patient’s home,” says Michener. Patients trained to associate certain musical notes with particular tasks, for instance, could match those notes using their facial movements to adjust the thermostat, operate an electric bed, turn on the TV, or communicate needs to a caregiver. Ultimately, the ability to easily interact with various machines and devices could help patients in rehabilitation and people with disabilities communicate more effectively and live more independently.