Bostonia: The Alumni Magazine of Boston University

Frank GuentherFrank Guentherhas been working for more than two decades on a neural model of speech that he’s using to create software he hopes will translate thoughts to words. Photo by Kalman Zabarsky

One blustery day in February 2008, two Boston University neuroscientists arrived in Atlanta to visit a young man named Erik Ramsey. They’d never met Ramsey in person, but they were well acquainted with his brain, having watched its neural firings for more than a year.

About ten years earlier, Ramsey, then a spirited sixteen-year-old who loved playing football, listening to heavy metal, and drawing monsters in class, was in a late-night car wreck that left him completely paralyzed. He could still see, smell, and hear. His body could still register the itch of a rash or the pleasure of a warm breeze. But he couldn’t speak or make any voluntary movements other than with his eyes. There is no treatment for his condition, which is known to neurologists as locked-in syndrome. Ramsey is trapped, his mind cut off from his body and his ability to communicate entirely dependent on those around him asking the right questions and interpreting the sometimes imperceptible shifting of his eyes — up for yes and down for no.

The BU scientists, Frank Guenther, a College of Arts & Sciences professor of cognitive and neural systems, and Jonathan Brumberg, a postdoctoral research associate in Guenther’s lab, are working to help Ramsey and others who have lost the ability to speak because of stroke or disease. Guenther has been developing a neural model of speech for more than two decades, one that he and Brumberg are using as a Rosetta Stone to create decoder software that they hope can translate thoughts into speech.

Before heading to Atlanta, Guenther and Brumberg analyzed data from about forty neurons picked up by a tiny electrode that a fellow neuroscientist had implanted in a speech-related area of Ramsey’s motor cortex. The system was designed to receive Ramsey’s brain signals wirelessly from the implanted electrode as he imagined speaking and to decode that neural activity into real-time speech via a voice synthesizer. They’d come to Georgia to give the system its first trial run.

The speech prosthesis project puts Guenther and Brumberg at the forefront of brain-computer interface (BCI) research, a new field of science that offers hope for those with paralysis, amputated limbs, neurodegenerative diseases, or sensory impairments and that will eventually raise some very big questions about enhancing the healthy versus aiding the sick and about the distinction between human and machine. But on that February afternoon, all that mattered to Guenther and Brumberg was helping Ramsey utter his first bit of speech in nearly a decade.

“Everybody was excited, but they were also tense,” recalls Guenther. “Erik kept trying to orient his eyes toward Jonathan and me. He’d heard about us for a long time. We were the guys up in Boston working on this problem, and he was excited to get a look at us. But we didn’t know if this thing was going to work at all.”

Postdoc researcher Jonathan Brumberg says philosophical questions about melding humans and computers don’t seem as important as giving Erik Ramsey back his voice. Photo by Kalman Zabarsky

The mission to give people like Ramsey back their voice began in the early 1990s, when Guenther created a computer-based model of the neural circuitry that fires every time we speak. The model, the first of its kind, ranges across several parts of the brain, including the areas responsible for the higher-level processes involved in formulating syllables and the motor cortex, which controls the tongue, lips, and jaw (known as articulators). The model also accounts for the feedback mechanism by which our brains compare the sounds we produce to what we meant to say and make any necessary adjustments. It uses inputs from a few thousand networked “neurons” to control what Guenther calls “a virtual vocal tract,” establishing the alignment of tongue, lips, and jaw that in turn produces speech through a synthesizer.

At first, all Guenther could use to build and refine his model were extrapolations from findings about nonspeech brain functions and from studies of people with brain lesions that had somehow short-circuited their ability to talk. But by the late 1990s, he started using functional magnetic resonance imaging (fMRI) technology. This allowed him to compare the model’s predictions to actual brain scans of people speaking words or syllables under particular constraints, such as restricted jaw movement or distorted auditory feedback. “We use the results to either verify the model or in the cases where the measured brain activities go against the model’s predictions, we improve the model so that it now accounts for the new data,” Guenther explains.

Based on the model, he and his fellow researchers at BU’s Cognitive and Neural Systems Speech Lab hypothesized that the premotor cortex, which most scientists believed controls only bodily movement, also contains neurons that generate a mental preview of speech sounds, such as “uh,” “ee,” and “ay.” These so-called formant frequencies in turn inform the motor neurons as they orchestrate the positioning of tongue, lips, and jaw. This idea would prove critical for decoding Ramsey’s neural signals, Guenther says, because the implanted electrode was in this particular region of his brain.

Locked In: A brain stem stroke blocks signals from Ramsey’s brain from reaching the rest of his body. He can think of words, but unable to move his mouth, he cannot speak. Detecting Signals: A tiny electrode inserted into the part of the motor cortex responsible for the movements involved in speech captures the activity of about forty neurons. The data are transmitted wirelessly to a computer to be decoded. Translation: Software developed by BU neuroscientist Frank Guenther looks for patterns in the firings of the neurons when Ramsey imagines speaking. Based on subtle differences in these patterns, it plots the result on a computer screen and generates the corresponding vowel sound.

Speech prostheses may someday be used by people who have ALS or who have lost their voice following throat cancer surgery.

Around midnight on November 5, 1999, Erik Ramsey was in the passenger seat of a friend’s Camaro as they returned home from a movie on a dark, two-lane Georgia highway. They didn’t see the minivan making a U-turn until it was too late. The Camaro slammed into the minivan’s right front fender, flipped, and landed on an embankment. Firefighters needed the Jaws of Life to cut Ramsey free of the wreck.

He was screaming and writhing in pain when his father, Eddie, got to the emergency room. It took fifteen hours of surgery to repair a collapsed lung, a lacerated spleen, a ruptured diaphragm, ripped tendons in his hand, and a femur that was broken in two places. Erik woke up in intensive care, but he didn’t speak, ask for pain medication, or respond to the doctors, his father, or his mother, Sandra. Tests later revealed that a blood clot had caused a brain-stem stroke that cut the connection between his mind and his body.

After a few weeks, the Ramseys took their son home, and with the help of a home health aide, began the daily routine of feeding him through a tube, bathing him, moving his limbs through range-of-motion exercises, keeping his eyes moist with drops, and clearing his lungs with a nebulizer. Soon, he learned to use his eyes to select from a letter board that his father designed, spelling out requests for movies (anything with vampires or other bloodthirsty creatures) and music (Ozzy Osbourne is a favorite). He’d occasionally play small pranks by spelling titles that didn’t exist. And when these tricks were discovered, Erik’s father recalls, “he would just die laughing,” an involuntary, spasm-like response that he still has when something amuses or excites him. But then two bouts of pneumonia robbed him of the stamina and reaction time needed to spell out words with the letter board. He was back to the limited and laborious yes or no of his eyes.

Ramsey would be the first person to have an electrode implanted in a brain region known to be involved in speech. In December 2004, surgeons put a hollow glass electrode with three wires, measuring about a millimeter and a half, six millimeters deep into the left side of Ramsey’s brain.

It was a nurse in the local school district who put the Ramseys in touch with Phil Kennedy, a pioneer in brain-computer interface research, who had been implanting electrodes — first in rats, then in monkeys, and eventually in humans — since 1986, and who had National Institutes of Health backing for a start-up company called Neural Signals, based in nearby Duluth, Georgia. Kennedy’s first implants in humans had allowed paralyzed individuals to move a computer cursor with their thoughts and to work with basic text and drawing applications. Ramsey would be the first person to have an electrode implanted in a brain region known to be involved in speech. In December 2004, surgeons put a hollow glass electrode with three wires, measuring about a millimeter and a half, six millimeters deep into the left side of Ramsey’s brain.

In the months that followed, Kennedy ran tests in which he asked Ramsey to imagine trying to say various words or to think about moving his lips, tongue, or jaw. Kennedy could see that the neurons were firing during these exercises, but he couldn’t interpret exactly what speech sound, such as “pa” or “ooh” or “dee,” corresponded to which tangle of neural data. In 2006, he sought out Guenther, who along with Brumberg readily agreed to review the data that Kennedy had made available online.

The challenge, Guenther says, was that there were no distinct neuron spikes when Ramsey was trying to say one thing or another. “It’s not like there are neurons that start firing when he says ‘ah’ but no other sound,” he says. “All the neurons are firing a little bit all the time, but they change their firing rates. It’s the details in the patterns of those changes that are important.” By zeroing in on the part of the brain where the electrode was, says Brumberg, “the model gave us the guide to decode that part of the signal. It said, ‘Here’s what that part of the brain is trying to represent.’”

Over the next year, they used that guide to build the neural decoder software at the heart of the system they would bring down to Georgia to read Ramsey’s mind.

Eddie Ramsey, Erik Ramsey, and lab technician Jess Bartels are looking forward to Erik’s next challenge: consonants. Photo courtesy of Eddie Ramsey

The BU researchers spent two days in Georgia before meeting Ramsey — installing the decoder software and testing the receiver for the wireless data transmission and the synthesizer that would give that data voice. The goal for Ramsey was to create vowel sounds, a first step in speech, requiring just one constant configuration of the mouth.

Represented in an x and y coordinate system, different vowels appear in different locations. “Uh” is seen in the middle of the screen, while “ooh” and “ee” and “ah” are in the corners. Ramsey’s challenge would be to mimic a computerized voice starting with “uh” and, using only his thoughts, move the target dot from the center to the corner vowel sounds.

“Listen,” the computer would say, “uhhhhhhhoooooooooh.” And then, it would ask Ramsey to “speak,” and a synthesized voice (adapted from a recording of Eddie Ramsey to give it a familiar southern drawl) would decode Ramsey’s attempt to think that same sequence.

On his first five attempts, Ramsey failed to hit even one target vowel. The tension and frustration in the room were mounting, and everybody took a five-minute break. Eddie Ramsey went for a short walk, and the researchers played one of Erik’s favorite CDs — Headbangers Ball — to help him relax.

In his next series of tries, Ramsey started hitting targets. And by the third round, he nailed more than half, and started laughing in his excitement.

“We were cheering him on,” says Guenther. “Everything we’d done in our sixteen years of existence as a lab had been theoretical. It was really heartwarming to see something from our research so directly impact somebody’s life.”

On his first five attempts, Ramsey failed to hit even one target vowel. The tension and frustration in the room were mounting.

Since then, the researchers and Erik have worked through dozens of sessions. Three days a week, Eddie Ramsey rises well before dawn for his early morning shift at the post office. Then he heads home, picks up his son, and takes him to Neural Signals, where the experiments run throughout the afternoon. While Brumberg returns to Georgia about once a month, he and Guenther conduct most of the experiments from Boston via a video-enabled Skype connection with Kennedy’s lab.

Every session follows a similar pattern, with Ramsey improving as the afternoon progresses, lately achieving as much as 90 percent accuracy with his vowels. “We have only about forty neurons here, and there are maybe a billion neurons involved in speech. So we have a very tiny window,” says Guenther. “But we can get him in the ballpark, and with practice he’s able to improve his accuracy.”

Guenther and Brumberg are collaborating with researchers at Georgia Tech to refine the decoder. Each improvement makes it easier for Ramsey to learn, but it also means that his brain must continually adjust and master a new system. “When we learn to speak as infants, it takes us months. It’s not an afternoon-long process,” says Guenther. “For Erik, the situation is as if a child woke up every day with a slightly different brain than he had the day before and had to relearn what he’d already learned.”

As a result, one short-term goal is to develop the decoder to a point where Ramsey can use the same one over and over, and thereby increase the pace of his progress. The researchers are simultaneously working on an “articulatory synthesizer” to get ready for Ramsey’s next challenge — consonants — which are far more complicated than vowels.

“There are more dimensions to work with for consonants,” says Brumberg. “You need to know where the tongue touches the back of the teeth when you’re saying a ‘t’ sound, for example.” And it gets even trickier when consonants and vowels are combined. “Saying something like ‘ahh-dah’ means your tongue has to rapidly go up to the closure of the vocal tract and then back down, all in about a tenth of a second,” says Guenther. Despite the complexity, the researchers are confident they’ll have Ramsey producing consonants within the year.

“It was really heartwarming
to see something from our research so directly impact somebody’s life,” says Frank Guenther. Photo by Kalman Zabarsky

Roger Miller, program director for neural prosthesis development at the NIH’s National Institute on Deafness and Other Communications Disorders, one of the project’s funders, says Guenther and Brumberg’s work is at the cutting edge of brain-computer interface research. But the speech prosthesis is just one of several BCI projects making headlines. In January 2008, scientists at Duke University were able to make a robot walk on a treadmill in Japan by transmitting neural activity from monkeys in North Carolina. A few months later, researchers at the University of Pittsburgh and Carnegie Mellon University trained monkeys to adopt brain-controlled robotic arms as their own, using them to feed themselves grapes and marshmallows.

For now, nearly all of the mainstream BCI research is directed at helping the disabled. Guenther and Brumberg, for their part, hope the speech prosthesis will one day be used by people who have the neurodegenerative disease amyotrophic lateral sclerosis (ALS) or who have lost their voice following throat cancer surgery. But they’ve also been approached by those interested in creating BCI applications to enhance the capabilities of healthy people, mainly to boost memory. “Their goal,” Brumberg says, “would be to upload and download information between a computer and your brain like it was a flash drive.” But, he notes, that would be extremely complicated, because scientists still don’t know exactly how memory is stored and retrieved in the brain.

Guenther and Brumberg hope the speech prosthesis will one day be used by people who have the neurodegenerative disease ALS or who have lost their voice following throat cancer surgery.

“People in our field do think about things like what it would mean to be human with an enhancement implant,” says Brumberg. “But those are the sort of philosophical questions about the future that don’t seem as important as trying to get our device working its best for people like Erik, who need it right now.”

Indeed, another one of their current projects is devising a system whereby Ramsey can turn the synthesizer on and off on his own, which will be critical when he uses it in an actual conversation. And that’s something the researchers hope he’ll be able to do in about five years.

Ramsey’s father shares their optimism. He believes in a future where his son is not only talking again, but drawing as well, all with the power of his mind. “It’s kind of equivalent to watching your baby learn to walk,” he says. “He’s got the first steps out of the way, and as soon as he’s got his footing under him, he’ll be off.” ■

To read more about Erik Ramsey and research that will help locked-in patients, go to  the Neural Interfacing Research Institute.

Margrit Betke
has developed a “camera mouse” whose software uses a computer webcam to lock onto and track a section of the user’s face and then links that person’s head movement to a cursor on the screen. Photo by Kalman Zabarsky

Every day, our world grows more digital and a little bit more of our lives migrates to an electronic format. But Margrit Betke, a College of Arts & Sciences associate professor of computer science, believes the networked world isn’t nearly as inclusive as it ought to be. Consider that the many people with movement and speech impaired by diseases like multiple sclerosis or ALS can’t type Google searches. They can’t play video games, and they can’t click on a friend’s e-mail.

“The community of people with severe disabilities is not really well served by computer science,” Betke says. So, in collaboration with James Gips, a Boston College professor of computer science, and several of her students, Betke has spent the last eight years developing a “camera mouse” that greatly expands the accessibility of the digital world. The camera mouse software uses a computer webcam to lock onto and track a chosen section of the user’s face — a nostril or the tip of an eyebrow, for example — and then links that person’s head movement to a cursor on the screen. Move right and the cursor goes right. Move left and it reverses direction. Pause for several seconds over a link and it clicks.

Betke and her fellow researchers have adapted a camera mouse to work with several popular programs, such as Microsoft Word. They’ve also created custom camera mouse software that allows computer users with disabilities to type e-mails, edit photographs, create music, and fight space aliens, among other activities.

The camera mouse software uses a computer webcam to lock onto and track a chosen section of the user’s face — a nostril or the tip of an eyebrow, for example — and then links that person’s head movement to a cursor on the screen.

In spring 2007, after a failed attempt to build a company around the new technology, Betke and Gips decided to give camera mouse away online. These days, about 2,500 people download it every month. The researchers get frequent e-mails from people as far away as Australia and Uzbekistan, thanking them for camera mouse and asking for technical assistance.

“With software, there’s always an issue of maintenance,” says Betke. Requests to fix a software bug or make camera mouse compatible with the latest operating system always get highest priority. The request for a camera mouse version of Flight Simulator, on the other hand, becomes a candidate for a student’s directed study or class project. Betke’s students also work as volunteers in places like the Boston Home, a nursing care center for adults with neurodegenerative diseases, whose residents have used camera mouse.

What’s next? Betke hopes to make camera mouse more adaptable to the wide range of mobility limitations, arranging navigation buttons to match a person’s most controlled range of motion, for example, or accommodating the diminishment over time of the user’s abilities.

“It’s a challenge that is facing all human-computer interface research,” says Betke. “We can adapt our own system with user profiles and all that, but to actually have the computer figure it out for us and help us along is a very different story.” ■

Download: Download this Article



On 23 November 2012 at 12:02 AM, Aaron (ENG’11) wrote:

I went to high school with Erik, and remember cutting up with him all of the time. Erik is one of the funniest and intelligent people I have ever met. Im so glad to see there has been so much progress and so many great people able to help him.

On 25 August 2011 at 5:09 PM, Joanne (SED) wrote:

I am just wondering why, in the meantime, the predictive-text technology we all use every day with google and/or texting couldn't be employed to help him communicate. For example, couldn't a large monitor be mounted in his line of sight and provide optional phrases (i.e. things he's communicated before - why should he have to spell everything from scratch every time?) as well as projected wording based on lettering he's agreed to...? And why can't he use something like the technology Stephen Hawking uses - or at least a low-ish-tech version of it (as I said above, posting potential items that he could rest his eyes on)...?

On 1 July 2011 at 6:29 AM, Carol Poirier ('11) wrote:

I knew Erik slightly before his accident and I see him often now. I have a daughter with his younger brother Matt Ramsey. I know i haven't been around for the entire study but since being around him the past couple of years i have had an aching urge to want to know everything about his condition and the study to which he has been involved in. I try not to ask a lot of questions and demand answers because of the on going, still tender wounds the family has about Erik and the state in which he is still in. I now have a better understanding of his medical state he is in from reading this article but now i am left wondering what happened? Did he reach a dead end in the study? Did you run out of money to fund this study? knowing him before and after his accident pushes me to want to know more of what i can do for him and to help him get back to were he used to be before his accident. he's a beautiful person inside and out. I know that i have no medical education or knowledge about this condition but I hope that you will answer my questions and help me better understand the situation and options. I do have love for him and his family and drive to do what i can for them so please tell me what i can do to continue further research for Erik and people like him.

On 15 October 2009 at 12:07 PM, Anonymous wrote:

some one implanted an electrode of some sort can you tell me if any one in southern california is helping to desingh your voice sinthisiser or software thankyou any help would be apreciated.

On 23 August 2009 at 1:06 AM, Thomas Wagner wrote:

Couldn't he be taught morse code, so he could communicate right now?

On 12 April 2009 at 8:23 AM, Judy Ryan (SED'78) wrote:

I am the teacher for a student who was paralyzed in a car accident two years ago. This technology would be invaluable in keeping this nineteen-year-old intellectually alive albeit physically confined. Her pursuit for meaning in life as she now is would be greatly enhanced with access to this technology. Please keep us informed on progress made in the development of this technology and provide further information on how to download it.

On 6 April 2009 at 3:33 PM, Dr. Philip Kennedy wrote:

I enjoyed the article very much. It is a pity however, that Neural Signals Inc and the work we have done for over 4 years with Erik was not more prominent. The above quote summed it up, but it is not complete. I did not even see the link to the web site. More important, in the time between the beginning in 2004 and Frank Guenther's arrival two years ago, we did an immense amount of work, recording three days a week to characterize the area. Papers have been published along with several abstracts each year. I found your summary of our work way to short and not inclusive enough. For example, we did provide evidence that Erik could detect 32 of 39 phonemes using neural net analyses, we did have unit firings related to imagined movements of his articulators, did have stable signals related to eye movements, did have control of the cursor during a learning task, and also studied sensory responses including responses to tone inputs. Whew! Sounds like a lot, doesn't it? So if you are publishing another piece, please mention those achievements. I fully understand writing primarily about the Guenther laboratory's great work and that is most appropriate. I support it fully.

Post Your Comment

The content of this field is kept private and will not be shown publicly.

Which is lightest? elephant, cat, moon, tissue

Persons who post comments are solely responsible for the content of their messages. Bostonia reserves the right to delete or edit messages.

More Feature Stories | Or Click Here forThis Issue's Table of Contents

Benny and Josh Make Movies

Extraordinary films about ordinary lives propel the Safdie brothers from Com to Cannes

Sounds of Silence

BU scientists are helping a paralyzed man utter his first words in ten years

One Nation, Under Gods

In Augustine and the Jews, Professor Paula Fredriksen argues that the pluralism of ancient times trumps the tolerance of today

Cyberbullying Goes to College

For those on the sharp end of harassment, campus life can be a virtual hell