Dear Fellow Scholars, this is Two Minute Papers
with Dr. Károly Zsolnai-Fehér. Today, we are going to read some minds. A few months ago, our first video appeared
on brain machine interfaces. It was about a paper from Neuralink, which
promised a great deal. For instance, they proposed a brain machine
interface that could read this pig’s thoughts as it was running on the treadmill. And what’s more, they not only read is thoughts,
but they could also predict what the pig’s brain is about to do. So this was about reading thoughts related
to movement. And to be able to use these brain machine
interfaces to the fullest, they should be able to enable some sort of communication
for people who have lost their ability to move or speak. So about writing or speaking? As impossible as it sounds, can we somehow
restore that, or is that still science fiction? Many of you told me that you would like to
hear more about this topic, so due to popular request, here it is, a look beyond Neuralink’s
project to see what else is out there. This is a collaboration between Stanford University
and a selection of other institutions, and it allows brain-to-text transcription, where
all the test subject has to do is imagine writing the letters, and they magically appear
on the screen. And now, start holding on to your papers,
and just look at how quickly it goes! 90 characters per minute with over 94% of
accuracy, which can be improved to over 99% of accuracy with an additional autocorrect
technique. That is absolutely amazing. A true miracle. 90 characters per minute means that the test
subject here, who has a paralyzed hand, can almost think about writing these letters continuously,
and most of them are decoded and put on the screen in less than a second. Also, wait a second, 90 characters per minute? That is about 80% as fast as the average typing
speed on a smartphone screen for an able-bodied person of this age group. Whoa. It is quite remarkable that even years after
paralysis, the motor cortex is still strong enough to be read by a brain-computer interface
well enough for such typing speed and accuracy. It truly feels like we are living in a science
fiction world. Of course, not even this technique is perfect,
it has its own limitations. For instance, we can’t edit or delete text,
have no access to capital letters and the method has a calibration step that takes a
long time, although it doesn’t get significantly worse if we shorten this calibration time
a bit. So how does this work? First, the participant starts thinking of
writing one letter at a time. Here you see the recorded neural activity,
this is subject to decoding. You can see the decoded signals here. And we can’t just give this to a computer
to distinguish between them as is, we project these into a 2D latent space where it is easy
to find which letter corresponds to which region. Look, they form relatively tight clusters,
therefore it is now easy to decide which of the squiggles corresponds to which letter. The decoding part is done by using a recurrent
neural network, which is endowed with memory and can deal with sequences of data. So here, in goes the brain activity, and out
comes the decision that says which character these activities correspond to. Of course, our alphabet was not designed to
be decoded with neural networks. So here is an almost science fiction-like
question - how do we reformulate the alphabet to tailor it to maximize the efficiency of
a neural network decoding our thoughts? Or simpler, what would the alphabet look like
if neural networks were in charge? Well, this paper has an answer to that too,
so let’s have a look. The squiggles indeed look like they came from
another planet, so what do we gain from this? Well, look at the distance matrix for the
English alphabet. The diagonal is supposed to be very blue,
but what is not supposed to be blue at all are the regions that surround it. Look, the blue color here means that in the
English alphabet, the letters M and N can be relatively easily confused. Same with the letters O and C, and there are
many more similarities. And now, look, here is the same distance matrix
for the optimized alphabet. No dark blue in sight outside the diagonal! Much easier to process and decode. If neural networks were in charge, this is
what the alphabet would look like. Glorious! Also, the fact that we are talking about squiggles
is not a trivial insight at all, traditional methods typically rely on movement in straight
lines to select letters and buttons. The other key thought in this paper is that
modern neural network-based methods can decode these thoughts of squiggles reliably. That is absolutely amazing. And wait a second…note that there is only
one participant in the user study. Why just one participant, why not call in
a bunch of people to test this? It is because this method uses a microelectrode
array, and this requires surgery to insert, and because of that, these studies are difficult
to perform, and are usually done at times when the participant has brain surgery anyways
for other reasons. Having more people in the study is usually
prohibitively expensive, if at all possible for this kind brain implant. And note that research is a process, and these
papers are stepping stones. And now, we are able to help people write
90 characters every minute with a brain machine interface, and I can only imagine how good
these techniques will become two more papers down the line. And don’t forget, there are research works
on non-invasive devices too! So, what do you think? Let me know in the comments below. What a time to
be alive! Thanks for watching and for your generous
support, and I'll see you next time!
We're getting closer bois
Having to imagine writting is way to slow. The best method would be to "listen" to the words you think of.
Let's go! This is so cool.
In 2030 we will live in a different world then we live today.
But I'm still waiting for full immersion vr where i can do/feel everything i want.
What a time to be alive!