On May 12, 2021, Howard Hughes Medical Institute (HHMI) researchers have, for the first time, decoded the neural signals associated with writing letters, then displayed typed versions of these letters in real time. They hope their invention could one day help people with paralysis communicate. The study results were published in the journal Nature.

Scientists are exploring a number of ways for people with disabilities to communicate with their thoughts. The newest and fastest turns back to a vintage means for expressing oneself: handwriting..

The innovation could, with further development, let people with paralysis rapidly type without using their hands, says study coauthor Krishna Shenoy, a Howard Hughes Medical Institute Investigator at Stanford University who jointly supervised the work with Jaimie Henderson, a Stanford neurosurgeon. By attempting handwriting, the study participant typed 90 characters per minute – more than double the previous record for typing with such a “brain-computer interface,” Shenoy and his colleagues report on May 12, 2021.

In recent years, Shenoy’s team has decoded the neural activity associated with speech in the hopes of reproducing it. They have also devised a way for participants with implanted sensors to use their thoughts associated with attempted arm movements to move a cursor on a screen. Pointing at and clicking on letters in this way let people type about 40 characters per minute, the previous speed record for typing with a brain computer interface (BCI).

No one, however, had looked at handwriting. Frank Willett, an HHMI research specialist and neuroscientist in Shenoy’s group, wondered if it might be possible to harness the brain signals evoked by putting pen to paper. “We want to find new ways of letting people communicate faster,” he says. He was also motivated by the opportunity to try something different.

The team worked with a participant enrolled in a clinical trial called BrainGate2, which is testing the safety of BCIs that relay information directly from a participant’s brain to a computer. (The trial’s director is Leigh Hochberg, a neurologist and neuroscientist at Massachusetts General Hospital, Brown University, and the Providence VA Medical Center.) Henderson implanted two tiny sensors into the part of the brain that controls the hand and arm, making it possible for the person to, for example, move a robotic arm or a cursor on a screen by attempting to move their own paralyzed arm.

The participant, who was 65 years old at the time of the research, had a spinal cord injury that left him paralyzed from the neck down. Using signals the sensors picked up from individual neurons when the man imagined writing, a machine learning algorithm recognized the patterns his brain produced with each letter. With this system, the man could copy sentences and answer questions at a rate similar to that of someone his age typing on a smartphone. This so-called “Brain-to-Text” BCI is so fast because each letter elicits a highly distinctive activity pattern, making it relatively easy for the algorithm to distinguish one from another, Willett says.