Bostonia: The Alumni Magazine of Boston University

The mission to give people like Ramsey back their voice began in the early 1990s, when Guenther created a computer-based model of the neural circuitry that fires every time we speak. The model, the first of its kind, ranges across several parts of the brain, including the areas responsible for the higher-level processes involved in formulating syllables and the motor cortex, which controls the tongue, lips, and jaw (known as articulators). The model also accounts for the feedback mechanism by which our brains compare the sounds we produce to what we meant to say and make any necessary adjustments. It uses inputs from a few thousand networked “neurons” to control what Guenther calls “a virtual vocal tract,” establishing the alignment of tongue, lips, and jaw that in turn produces speech through a synthesizer.

At first, all Guenther could use to build and refine his model were extrapolations from findings about nonspeech brain functions and from studies of people with brain lesions that had somehow short-circuited their ability to talk. But by the late 1990s, he started using functional magnetic resonance imaging (fMRI) technology. This allowed him to compare the model’s predictions to actual brain scans of people speaking words or syllables under particular constraints, such as restricted jaw movement or distorted auditory feedback. “We use the results to either verify the model or in the cases where the measured brain activities go against the model’s predictions, we improve the model so that it now accounts for the new data,” Guenther explains.

Based on the model, he and his fellow researchers at BU’s Cognitive and Neural Systems Speech Lab hypothesized that the premotor cortex, which most scientists believed controls only bodily movement, also contains neurons that generate a mental preview of speech sounds, such as “uh,” “ee,” and “ay.” These so-called formant frequencies in turn inform the motor neurons as they orchestrate the positioning of tongue, lips, and jaw. This idea would prove critical for decoding Ramsey’s neural signals, Guenther says, because the implanted electrode was in this particular region of his brain.

Around midnight on November 5, 1999, Erik Ramsey was in the passenger seat of a friend’s Camaro as they returned home from a movie on a dark, two-lane Georgia highway. They didn’t see the minivan making a U-turn until it was too late. The Camaro slammed into the minivan’s right front fender, flipped, and landed on an embankment. Firefighters needed the Jaws of Life to cut Ramsey free of the wreck.

He was screaming and writhing in pain when his father, Eddie, got to the emergency room. It took fifteen hours of surgery to repair a collapsed lung, a lacerated spleen, a ruptured diaphragm, ripped tendons in his hand, and a femur that was broken in two places. Erik woke up in intensive care, but he didn’t speak, ask for pain medication, or respond to the doctors, his father, or his mother, Sandra. Tests later revealed that a blood clot had caused a brain-stem stroke that cut the connection between his mind and his body.

After a few weeks, the Ramseys took their son home, and with the help of a home health aide, began the daily routine of feeding him through a tube, bathing him, moving his limbs through range-of-motion exercises, keeping his eyes moist with drops, and clearing his lungs with a nebulizer. Soon, he learned to use his eyes to select from a letter board that his father designed, spelling out requests for movies (anything with vampires or other bloodthirsty creatures) and music (Ozzy Osbourne is a favorite). He’d occasionally play small pranks by spelling titles that didn’t exist. And when these tricks were discovered, Erik’s father recalls, “he would just die laughing,” an involuntary, spasm-like response that he still has when something amuses or excites him. But then two bouts of pneumonia robbed him of the stamina and reaction time needed to spell out words with the letter board. He was back to the limited and laborious yes or no of his eyes.

Ramsey would be the first person to have an electrode implanted in a brain region known to be involved in speech. In December 2004, surgeons put a hollow glass electrode with three wires, measuring about a millimeter and a half, six millimeters deep into the left side of Ramsey’s brain.

It was a nurse in the local school district who put the Ramseys in touch with Phil Kennedy, a pioneer in brain-computer interface research, who had been implanting electrodes — first in rats, then in monkeys, and eventually in humans — since 1986, and who had National Institutes of Health backing for a start-up company called Neural Signals, based in nearby Duluth, Georgia. Kennedy’s first implants in humans had allowed paralyzed individuals to move a computer cursor with their thoughts and to work with basic text and drawing applications. Ramsey would be the first person to have an electrode implanted in a brain region known to be involved in speech. In December 2004, surgeons put a hollow glass electrode with three wires, measuring about a millimeter and a half, six millimeters deep into the left side of Ramsey’s brain.

In the months that followed, Kennedy ran tests in which he asked Ramsey to imagine trying to say various words or to think about moving his lips, tongue, or jaw. Kennedy could see that the neurons were firing during these exercises, but he couldn’t interpret exactly what speech sound, such as “pa” or “ooh” or “dee,” corresponded to which tangle of neural data. In 2006, he sought out Guenther, who along with Brumberg readily agreed to review the data that Kennedy had made available online.

The challenge, Guenther says, was that there were no distinct neuron spikes when Ramsey was trying to say one thing or another. “It’s not like there are neurons that start firing when he says ‘ah’ but no other sound,” he says. “All the neurons are firing a little bit all the time, but they change their firing rates. It’s the details in the patterns of those changes that are important.” By zeroing in on the part of the brain where the electrode was, says Brumberg, “the model gave us the guide to decode that part of the signal. It said, ‘Here’s what that part of the brain is trying to represent.’”

Over the next year, they used that guide to build the neural decoder software at the heart of the system they would bring down to Georgia to read Ramsey’s mind.

The BU researchers spent two days in Georgia before meeting Ramsey — installing the decoder software and testing the receiver for the wireless data transmission and the synthesizer that would give that data voice. The goal for Ramsey was to create vowel sounds, a first step in speech, requiring just one constant configuration of the mouth.

Represented in an x and y coordinate system, different vowels appear in different locations. “Uh” is seen in the middle of the screen, while “ooh” and “ee” and “ah” are in the corners. Ramsey’s challenge would be to mimic a computerized voice starting with “uh” and, using only his thoughts, move the target dot from the center to the corner vowel sounds.

“Listen,” the computer would say, “uhhhhhhhoooooooooh.” And then, it would ask Ramsey to “speak,” and a synthesized voice (adapted from a recording of Eddie Ramsey to give it a familiar southern drawl) would decode Ramsey’s attempt to think that same sequence.

On his first five attempts, Ramsey failed to hit even one target vowel. The tension and frustration in the room were mounting, and everybody took a five-minute break. Eddie Ramsey went for a short walk, and the researchers played one of Erik’s favorite CDs — Headbangers Ball — to help him relax.

In his next series of tries, Ramsey started hitting targets. And by the third round, he nailed more than half, and started laughing in his excitement.

“We were cheering him on,” says Guenther. “Everything we’d done in our sixteen years of existence as a lab had been theoretical. It was really heartwarming to see something from our research so directly impact somebody’s life.”

On his first five attempts, Ramsey failed to hit even one target vowel. The tension and frustration in the room were mounting.

Since then, the researchers and Erik have worked through dozens of sessions. Three days a week, Eddie Ramsey rises well before dawn for his early morning shift at the post office. Then he heads home, picks up his son, and takes him to Neural Signals, where the experiments run throughout the afternoon. While Brumberg returns to Georgia about once a month, he and Guenther conduct most of the experiments from Boston via a video-enabled Skype connection with Kennedy’s lab.

Every session follows a similar pattern, with Ramsey improving as the afternoon progresses, lately achieving as much as 90 percent accuracy with his vowels. “We have only about forty neurons here, and there are maybe a billion neurons involved in speech. So we have a very tiny window,” says Guenther. “But we can get him in the ballpark, and with practice he’s able to improve his accuracy.”

Guenther and Brumberg are collaborating with researchers at Georgia Tech to refine the decoder. Each improvement makes it easier for Ramsey to learn, but it also means that his brain must continually adjust and master a new system. “When we learn to speak as infants, it takes us months. It’s not an afternoon-long process,” says Guenther. “For Erik, the situation is as if a child woke up every day with a slightly different brain than he had the day before and had to relearn what he’d already learned.”

As a result, one short-term goal is to develop the decoder to a point where Ramsey can use the same one over and over, and thereby increase the pace of his progress. The researchers are simultaneously working on an “articulatory synthesizer” to get ready for Ramsey’s next challenge — consonants — which are far more complicated than vowels.

“There are more dimensions to work with for consonants,” says Brumberg. “You need to know where the tongue touches the back of the teeth when you’re saying a ‘t’ sound, for example.” And it gets even trickier when consonants and vowels are combined. “Saying something like ‘ahh-dah’ means your tongue has to rapidly go up to the closure of the vocal tract and then back down, all in about a tenth of a second,” says Guenther. Despite the complexity, the researchers are confident they’ll have Ramsey producing consonants within the year.

Roger Miller, program director for neural prosthesis development at the NIH’s National Institute on Deafness and Other Communications Disorders, one of the project’s funders, says Guenther and Brumberg’s work is at the cutting edge of brain-computer interface research. But the speech prosthesis is just one of several BCI projects making headlines. In January 2008, scientists at Duke University were able to make a robot walk on a treadmill in Japan by transmitting neural activity from monkeys in North Carolina. A few months later, researchers at the University of Pittsburgh and Carnegie Mellon University trained monkeys to adopt brain-controlled robotic arms as their own, using them to feed themselves grapes and marshmallows.

For now, nearly all of the mainstream BCI research is directed at helping the disabled. Guenther and Brumberg, for their part, hope the speech prosthesis will one day be used by people who have the neurodegenerative disease amyotrophic lateral sclerosis (ALS) or who have lost their voice following throat cancer surgery. But they’ve also been approached by those interested in creating BCI applications to enhance the capabilities of healthy people, mainly to boost memory. “Their goal,” Brumberg says, “would be to upload and download information between a computer and your brain like it was a flash drive.” But, he notes, that would be extremely complicated, because scientists still don’t know exactly how memory is stored and retrieved in the brain.

Guenther and Brumberg hope the speech prosthesis will one day be used by people who have the neurodegenerative disease ALS or who have lost their voice following throat cancer surgery.

“People in our field do think about things like what it would mean to be human with an enhancement implant,” says Brumberg. “But those are the sort of philosophical questions about the future that don’t seem as important as trying to get our device working its best for people like Erik, who need it right now.”

Indeed, another one of their current projects is devising a system whereby Ramsey can turn the synthesizer on and off on his own, which will be critical when he uses it in an actual conversation. And that’s something the researchers hope he’ll be able to do in about five years.

Ramsey’s father shares their optimism. He believes in a future where his son is not only talking again, but drawing as well, all with the power of his mind. “It’s kind of equivalent to watching your baby learn to walk,” he says. “He’s got the first steps out of the way, and as soon as he’s got his footing under him, he’ll be off.” ■

To read more about Erik Ramsey and research that will help locked-in patients, go to  the Neural Interfacing Research Institute.