Neural Prosthetics for Speech Restoration
In collaboration with Dr. Philip Kennedy and colleagues at Neural Signals, Inc., we are working to decode neural signals recorded from an electrode implanted in the speech motor cortex of a human volunteer with locked-in syndrome (characterized by full consciousness with near-complete paralysis) while he imagines speaking. The decoded signals are used to drive a speech synthesizer. This work has been featured in a July 2008 Boston Globe article, a November 2007 New Scientist article, a December 2007 CNN story, and a more detailed article in The Dana Foundation’s BrainWork in February 2008. More information about the electrode developed by Dr. Kennedy’s group can be found here.
Real-time Neural Decoding and Synthesis of Speech
In February 2008, we succeeded in providing the volunteer with real-time feedback of synthetic speech (created using a formant synthesizer) decoded from the neural firing patterns of cells in the speech motor cortex while he imagined speaking. The following videos show trials in which the volunteer hears the word “listen” followed by a vowel sequence (in this case “uh-oo”) that he is supposed to imitate. He then hears the word “speak” followed by synthetic speech decoded in real time from his ongoing brain activity. (The synthesizer is quite loud, at the volunteer’s request.) The locked-in volunteer can be seen in the center-right of the picture. A small box on the computer screen contains a blue cursor that indicates his position in the vowel space, with the target vowel “oo” being in the lower left corner of the screen. A successful “oo” is registered when the cursor makes it to this corner.
In our initial studies, the volunteer typically failed to produce the vowel consistently in his first five attempts, but by the sixth attempt his ability to control the synthesizer had improved to the point where he was successful on most subsequent attempts. The videos below represent the sixth trial (first video, 3.8Mb) and sixth through tenth trials (second video, 18.8Mb) for the vowel “oo” on the volunteer’s second day of using the real-time synthesizer. All of these attempts were successful, as demonstrated by the cursor moving to the lower left corner of the display on the computer screen.
August 2008 update. The following video demonstrates the participant’s performance with the synthesizer as of August 2008. Each trial begins with the computer telling the volunteer to “listen”, followed by the sound that he is supposed to say. Then the “speak” command is given and he uses the BCI to produce the sound.
In ongoing research we are working on improvements to the neural decoder and synthesizer, as well as allowing the volunteer to practice with the synthesizer to improve his performance.
Offline Decoding of Speech
In December 2007, we succeeded in decoding brain activity during imagined speech in order to drive a speech synthesizer offline. Click on the pictures below to see videos of the articulator movements and speech sounds decoded from neural signals collected from the electrode while the volunteer imagined producing the vowel sounds “ee”, “ah”, and “oo”, respectively. The red dots represent points on the lips (2 dots), tongue (3 connected dots), and lower front tooth (1 dot). [Videos created by Jonathan Brumberg and Sadao Hiroya utilizing technology developed by NTT Communication Science Laboratories, NTT Corporation.]
You can view our 2007 Society for Neuroscience conference poster describing the offline decoding of vowel sounds from neural signals collected from the electrode here.