Scientists are exploring a number of ways people with disabilities communicate with their thoughts. The newest and fastest comes down to a vintage way of expressing yourself: handwriting.
For the first time, researchers have deciphered the brain activity associated with trying to write letters by hand. Working with a paralyzed participant who has sensors implanted in his brain, the team used an algorithm to identify the letters as he attempted to write them down. Then the system displayed the text on a screen – in real time.
The innovation could, with further development, enable people with paralysis to type quickly without using their hands, says Krishna Shenoy, study co-author, Howard Hughes Medical Institute researcher at Stanford University who jointly supervised the work with Jaimie Henderson, Stanford neurosurgeon.
While trying to write by hand, the study participant typed 90 characters per minute – more than double the previous record for typing with such a “brain-computer interface,” Shenoy and colleagues report in the newspaper Nature on May 12, 2021.
This technology and others like it have the potential to help people with all kinds of disabilities, says Jose Carmena, a neural engineer at the University of California, Berkeley, who was not involved in the study. Although the results are preliminary, he says, “this is a big step forward in the field.”
Brain-computer interfaces turn thought into action, says Carmena. “This article is a perfect example: the interface decodes thought from writing and produces action.”
Communication fueled by thought
When injury or illness deprives a person of the ability to move, the neural activity in the brain to walk, have a cup of coffee, or speak a phrase remains. Researchers can take advantage of this activity to help people with paralysis or amputees regain lost abilities.
The need varies depending on the nature of the disability. Some people who have lost the use of their hands can still use a computer with voice recognition and other software. For those who have difficulty speaking, scientists have developed other ways to help people communicate.
In recent years, Shenoy’s team has decoded neural activity associated with speech in the hopes of reproducing it. They also developed a way for participants with implanted sensors to use their thoughts associated with attempted arm movements to move a cursor on a screen. Pointing and clicking letters in this manner allows people to type around 40 characters per minute, the previous speed record for typing with a brain-computer interface (BCI).
No one, however, had examined the handwriting. Frank Willett, a neuroscientist in Shenoy’s group, wondered if it would be possible to harness the evoked brain signals by putting a pen on paper. “We want to find new ways to allow people to communicate faster,” he says. He was also motivated by the opportunity to try something different.
The team worked with a participant enrolled in a clinical trial called BrainGate2, which tests the safety of BCIs that relay information directly from a participant’s brain to a computer. (The trial director is Leigh Hochberg, a neurologist and neuroscientist at Massachusetts General Hospital, Brown University, and Providence VA Medical Center.) Henderson implanted two tiny sensors in the part of the brain that controls the hand and arm. , which allows the person to, for example, move a robotic arm or a cursor on a screen while trying to move their own paralyzed arm.
The participant, who was 65 at the time of the research, had a spinal cord injury that left him paralyzed from the neck down. Using signals that sensors picked up from individual neurons as humans imagined writing, a machine learning algorithm recognized the patterns their brain produced with each letter. With this system, the man could copy sentences and answer questions at a pace similar to that of a person his age typing on a smartphone.
This so-called “Brain-to-Text” BCI is so fast because each letter sparks a very distinctive pattern of activity, which allows the algorithm to distinguish them from each other relatively easily, says Willett.
A new system
The Shenoy team plans to use an attempted handwriting for text entry as part of a more comprehensive system that also includes point-and-click navigation, much like that used on current smartphones, and even an attempt to decode speech. “Having those two or three modes and switching between them is something we do naturally,” he says.
Next, Shenoy says, the team plan to work with a participant who cannot speak, such as someone with amyotrophic lateral sclerosis, a degenerative neurological disorder that causes loss of movement and speech.
The new system could potentially help those suffering from paralysis caused by a number of conditions, Henderson adds. These include a brainstem stroke, which afflicted Jean-Dominique Bauby, the author of the book The Diving Bell and the Butterfly. “He was able to write this moving and beautiful book by carefully selecting the characters, one at a time, using eye movement,” says Henderson. “Imagine what he could have done with Frank’s handwriting interface!”
CAUTION: Investigation device. Limited by Federal law to experimental use.