Thank you very much, Dr. Kim, for this nice introduction. Thank you all for coming. It's great to see you in person. I'm very excited to get this series started and tell you all about what it is like to learn to see again with a bionic eye. As many of you might be aware, there are millions of people worldwide who live with incurable blindness. A lot of sighted people might imagine blindness to be something all or nothing. Complete darkness. But that's not necessarily the case. There are many different causes of blindness. And two of the most common ones in the developed world are shown here. One is called retinitis pigmentosa or RPE, and the other one is age related macular degeneration AMD. Both of these are hereditary diseases, meaning there is no cure for them and they affect your vision by turning you blind, either from the outside in leading to tunnel vision in the case of RPE or from the inside out. In the case of AMD. And so, although there are no cures for these diseases, researchers have been trying to come up with ways to restore vision to affected individuals. There are many different approaches that are currently in development. One idea is gene therapy, because both of these diseases are caused by a malfunctioning gene in the retina. The idea is that maybe we can inject a healthy one into the eye and that could slow the disease or even revert it. Another idea is to use optogenetics, which is a way to make neurons in the brain sensitive to light, even though they have never responded to light before. And that would allow you to shine a light or a laser onto these neurons to activate them. And then the third one is the idea of a micro electrode array, a tiny little chip that gets implanted directly into your eye or into your brain and stimulates the neurons with electrical current. And that's exactly what I want to talk to you about today. This idea of a retinal or a visual prosthesis, as it's called, it's no longer science fiction. In fact, it's being researched all over the world. There are two devices who already made it to the commercial stage. Unfortunately, they're no longer being manufactured. COVID had something to do with that. But there are many other devices that are close to becoming a commercial product. They are already in clinical trials where they're being tested on real patients to test their effectiveness. And then on top of that, there's many different devices that are in development and hopefully might hit the market in five, ten or 20 years. And so as you browse these names on the slide, you might notice that a lot of them seem to be companies, biomedical startups. And that's true. In fact, there are only very few academic groups that do basic research. But I'm very happy to add UCSB to this map as of a couple of years ago. And you can see there are other a couple of other universities that are working on the basic research aspects of of these technologies. And what kind of makes us stand out is that we don't have stock in any particular technology. We actually work with many different groups around the world, actual device manufacturers, to test our theoretical models and work with the patients to see how we can improve their vision. Perhaps the most commonly known device is called the Argus two Device. It was developed by a company called Second Sight based here in L.A. Actually, the way it works is the system comes with a pair of glasses that have a small camera embedded in the front. And so then whatever video the camera records that is being translated into a series of electrical signals delivered to this chip that is implanted in the eye, it seems for my point to work over here. We can zoom in on that a little bit. Zoom to the back of the eye where the retina sits. And you can see this schematic here, this electrode array that sits right on top of all the neurons in the eye. Usually neurons communicate with electrical signals. So what this device is doing is basically hijacking the communication channel. It's electrically stimulating the neurons in the eye and to the brain that's indistinguishable from neurons that just communicate naturally. But this is an artificial signal. And so the idea is if you understand the physiology of the eye, the neuroscience behind it, then maybe we can turn on this device in a clever way such that it will result in pattern vision. So maybe we can draw letters here on this grid and the patient will perceive it as an actual letter. We call this idea the scoreboard model, because if you've been to a large sports stadium, you've probably seen these scoreboards, right? And from far away they look like a coherent image. But as you walk up closer to it, you realize that every pixel in there is actually an independent light source. A light bulb you can turn on or off. And so that's kind of the idea that underlies these approaches here. If you had an image, if you wanted to or needed to translate this into electrical stimulation, you would assume that you could just take each pixel in the image and that would correspond to an electrode in the array. So if you just had a ten by ten grid of electrodes, maybe the image you can produce is not very interesting, but you know, up the numbers. Right? And at some point you should be able to create useful vision. This is the idea behind the scoreboard model where you think that stimulating a grid of electrodes leads to the perception of a grid of luminous dots, little flashes of light. That's nice and theory. Fortunately, it doesn't quite work like that. We know that because we tested this idea. We went out and asked real patients to draw on a touch screen what they see when we turn on a single electrode. Now, of course, they're blind. They can't see what they're drawing. But if we just turn on one thing at a time, they're pretty good at outlining the shape of what they're seeing. So this particular patient started drawing an arc, and that seems weird. That's not pixel. It's not a blob. Maybe it's a fluke. Let's try again on trial to the patient through the exact same streak as do it again and again and again. It's always that streak. To understand where that comes from. He had to, of course, repeat the experiments, not just look at one electrode, but many. So if we just take this guy here called D3, turn it on. We have the patient draw for this particular electrode. The patient drew a vertical line and we can align all the images by their center of mass and average him. So we know on average, this electrode produces a vertical streak and you can repeat it for all possible electrodes and map out the different streaks or shapes that the patient sees. Now this is just one patient so we can repeat that for other patients. And what we find is really a rich repertoire of of shapes. We call each of these a phosphine. It's a perception of light that is artificially created. And these phosphine, as you can see, have very different shapes across the array. We have lines, we have wedges, triangles, or some of them are blobs, Sure. But most of them are pretty streaky. What gives? Do you understand where that comes from? I have to give you a very short crash course on the retinal anatomy. You probably all aware that as light enters the eye here on the left, it gets refracted at the cornea and projected onto the back of the eye where the retina sits. The retina contains your photosensitive cells, the photoreceptors, rods and cones. If we zoom in on the back of the eye onto the retina, we already find an intricate network of neurons here organized into layers. You can see on top the photoreceptors, the rods and cone that actually respond to the light. And then these cells send their signal to the next stage, which are called the bipolar cells. And eventually the signal arrives at the ganglion cells, which are the output layer of the retina. They send whatever signal they receive along these axons, these wires through the optic nerve to the back of the head where the visual cortex sits. Now, this is a drawing of a healthy eye. If you have a disease like ARP or AMD, the first thing to lose are the photoreceptors. There are other changes happening to the retinal network that I'm not drawing here. But with the photoreceptors gone, you're missing your input stage. You no longer have cells that respond to light. So naturally, the idea of the implant is to replace this lost functionality with a chip. There are different locations to implant. If you put it right here, it's called Subretinal. You can also place it outside the eye, which is safer. But then you're really far away from all the neurons you're trying to stimulate. Or you can go at the bottom there, which is the device that I showed you before. Argus two is a so-called epi retinal device. And you can see that it sits directly on top of the ganglion cells, but not just the ganglion cell bodies. Also the ganglion cells, axons, the wires going out of the eye. And so that turned out to be really, really important. Here I'm showing now a bird's eye view of the back of the of the retina. A bird's eye view onto the retina. And each of these fibers is an axon that was extracted from one of 55 cadaver human eyes. And the fibers were aligned by the optic nerve and the phobia, which is the center of vision. And what this shows you is that all these fibers follow a very stereotypical pattern. So this is saying if you're a neuron somewhere up there, you would send your axon along one of these trajectories to the optic nerve. We can describe this with math. Don't worry about the equation. This is just to say that it follows a very nicely described stereotypical pattern. And it now allows us to ask what would happen if we placed an electrode, see right there and we stimulated it. So we wouldn't just stimulate the ganglion cells that sit directly below the electrode. We would also hit axons from other neurons that might live far away. But they just happened to send their axon through this point in space to the optic nerve. If you look at this map, where would these neurons live? They would live to the left of the electrode, because if you are along that pathway, you would send your axons through the same points. Right now, usually if you turn on one electrode, sorry, one neuron in the retina that signals to the brain, Hey, I saw something and it was maybe right over there. So now I'm turning on this guy, but I'm also turning on the neighbor. Some tell the brain I saw something. It was right here, but also right there. And right there. And right there and right there. So what we're doing effectively is we're drawing a streak. Of course, it would depend where the electrode sits. Right. So for down here, you shriek should be bent upwards. And in fact, we spent quite a long time proving that this is really what determines the shape of these phosphine drawings. I'm not going to go through the details here, but I just want to show you that we were able to describe the shape of any drawing with just two parameters that tell you how long is the thing and how thick is the thing. We compared these predictions to real patient data and then developed a computational model that tells us how this the shape change as you, for example, increase the amplitude of your current or as you and what happens when you increase the frequency or the pulse duration. All these other very electrical terms that could go into a stimulus have their distinct effect on the shape. Some make it larger, others make it brighter, and this one makes it less streaky, maybe. So this is all constrained by real data and by putting it into a computational model. We now have a simulation of what people with a bionic eye might see. In fact, we have developed open source software and Python. You can try. Maybe not now, but later you can PIP install it real easily and then try any of the currently available retinal implants yourself. Activate different electrodes. So I want to show you an example here, because as I said, now we can actually make predictions about what people with a bionic eye might see, although there's no way to know for sure. Here is our current best guess of what an Argus two patient would see. So a reminder and Argus two device has six by ten electrodes arranged in this grid. And we are wondering what would happen if someone were in the subway and there's a person walking by and you just pointing your camera straight ahead. What would you see? Now, maybe some of you might expect some sort of predator, a Terminator vision. I'm sorry to disappoint, but here is our prediction of what that might look like. Really, what you're seeing is everything is kind of smeared out and it's really hard to make out any particular details. This gets even worse as we turn up this axonal activation, this streaking of the phosphine. It looks like this. You can still kind of tell since a large object from a person walks from left to right and there was maybe a second one. But that's it. There's no way you could tell what's on the sign up there or what. What else is happening. And so that starts to make sense because people with a real bionic eye struggle with even the most basic tasks, because their vision is described as being kind of blurry and it has these flashes of light appear that to them seem very unnatural. And it's a completely new language that these people have to learn. It's just that we didn't have a good insight into what that actually looks like. And so this model can help us. It can help us not just understand where we are currently. Can also tell us what would happen if maybe we just need more electrodes. Right. So six by ten, even if every electrode were a pixel, we would still only have 60 pixels. That's the same thing I showed before. So maybe we just need more pixels. Right. What would that look like? So here at 12 by 20, we get a little bit of better image video. And then even here, now we can start seeing some. Details show it again, but it's still not enough to make out what's written on a sign, for example, or what's far, far away from you. So there's really there's more to be done. But this tells us is that maybe it's less important to increase the number of electrodes and more important to get rid of these tricks to make the electrodes we have more independent of each other. And so what that means is we essentially need the inverse of what we have right now. We have software, a model that tells us for a given stimulus. What is the predicted output, the predicted percept, when in fact we want to know the opposite. We want to know what is the required stimulus to produce a desired percept. So given the output, how on earth do I produce it? That's the question. And so one way to think about this is a regression problem where you're trying to minimize the error between your target output, let's say the letter E and your predicted output, and now you're searching the space of possible stimuli to find the one that gives you the closest match. As an example here for Argus two, the current way of doing things is really if you want to produce the letter E in the mind of the patient, you basically imprint the shape of it on the retina. It's a pixel display, right? So you just paint the E there. But we can use our model to tell you that will look nothing like any. Instead, what you want is you want the E to appear on the right. So if we do that and we search for the the optimal stimulation pattern, we find something that looks much more like an E. But now, just from looking at the electrode activations over here, you can no longer tell what's going on, but that's okay. So I should point out here, the size of the disk is proportional to the current. You have to apply. So this is saying you really only need the bottom electrode here and the top one there. And then as soon as you activate this one, it will streak out to the left. So it will actually help you draw one edge of the letter E and then you just need a little bit more stuff in the middle and your set. Now, you might say this is kind of a simple example, right? Because the shape of you is already aligned with these axon bundles. So let's do another one. Would anyone like to guess what letter this is? Be brave. Loud. And more and more, we have more guesses. Oh, hey, someone's going to get it. It's an N. So this is using the traditional, conventional way of stimulating. And that's what's coming out on the other end. Instead, if we again search for the optimized stimulus, we find something that looks more like an N, but it's not perfect, right? Point out one challenge here is it's really hard to produce a streak that is perpendicular to these action bundles. Why? Because as soon as you stimulate over here, it's going to smear out to the left. So it's virtually impossible to build a vertical edge at that location. Okay, One last one was this. Si, si, si. At some point, you got to get it. Yeah. Yeah. There's that power in numbers, Z. And so, again, we try to optimize for this one, and we're doing it. So we're doing an okay job. Right? But there are some problems. One that I haven't or only briefly mentioned is that the amplitude actually changes both the width and the length of an edge. So not just the brightness, but also the size. That's another thing you have to take into account. But overall, the message here is that there is some form of transformation happening from stimulation to what we see, and we can model that. And by having a model, we can use that to improve the stimulation for the bionic eye. In our most recent work, we actually use deep learning. So here the decoder is basically our former model that tells us how do you go from stimulus to perception. And so what we did is we had a deep net learn to invert that. Whatever streaks you produce, the deep net learns to take care of that, such that when you give it an input, the digit three here, it will pick the best simulation pattern to produce the exact same output. And that works really well here shown on the data set. So on top we have our proposed approach, which we termed the hybrid neural auto encoder, and we can compare this to two different baselines. The naive approach is what I already showed you, and the middle one is a slightly different approach that's popular, which is to learn an approximation of your for what model. But no matter what it is, you can see that the top row looks much cleaner than the rest. Now this is for small rows and lambdas, which are these parameters that define how lengthy and how thick a streak is. So as we increase that, we can see that the bottom two rows, they really struggle to represent the digits, whereas the top row consistently allows you to draw exactly what you want. So this seems to be a very promising approach, and we can't wait to test this with real patience. But there's more to it. So I haven't really told you about how these devices work in practice. Yes, there's a camera embedded on the head. But actually, what the camera sees, the field of view is very limited. It's kind of like watching TV from across the room. And so that makes it really hard to just use the device in everyday life. And people have to strategically move their head called head scanning to sample the scene around them. And that's very different from natural vision. And on top of that, you have these weirdly shaped phosphors that people describe as like fireworks or flashes of light that come on and go off. And so people have to learn how to interpret these things. They need associative learning and deductive reasoning to figure out, well, I looked over here and I got this big vertical flash and I look up and I just see stars. Oh, and I'm outside. So maybe I am looking at a tree, actually. Or the first part is the trunk, and the second one is the leaf. And you see the light reflect off all the different leaves. So that's really how people have to use the device currently. And then in addition to test how well or to document how well the device works, the companies will use these very complicated tasks. Well, they're not very complicated, but they're actually using the device in the real environment, maybe recognizing a letter which just did that. Sometimes you have to walk around and find a door or follow a line or avoid an obstacle. Well, that's a big jump from what I've shown you before. Or maybe just the shape of one phosphine to walking around and finding your way. That's a big jump, and it's really hard to simulate on a screen. And so that's where virtual reality comes in, because virtual environments are really a great way to address many of these challenges. We already have a simulation of what the world looks like. Now, if we put it in a headset and the virtual reality headset, you all can experience bionic vision and we can build the same environments that people use to test the device. And now to it in virtual reality with sighted participants. We call these sighted participants virtual patients, because really every step in the processing pipeline can be matched to an individual patient. We start by figuring out where does the array sit on the retina. We use the exact same stimulation pattern and we build the exact same room in virtual reality, such that we have a 1 to 1 mapping between the real patients and the virtual patients. We can not only embed these more realistic simulations, but we can fit them to an individual patient. So now you can actually model every individual that we have met in real life, and we can use that to predict how this particular patient might benefit. For example. One quick example is that we rebuilt a hallway where there are different obstacles along the way and people had to walk down the hallway and avoid obstacles, of course. So here's one little glimpse at what that look like. You see at the bottom, right? This is the virtual environment. And then that is fact through our bionic vision simulator. And depending on how big or small these rows and lambdas are, you get streaky things like here or you might get really blobby things. And so these parameters might actually help explain why some patients, real patients do really well and others really struggle. Finally, personally, I think this doesn't go far enough because there's so much more potential technology out there that we could use. One idea is not to represent the scene as naturally as possible, but make it as practical as possible. For example, nowadays we have computer vision. We have deep learning that can help us look just at a single RGV image and tell us where are the important objects in the scene where our edges that separate the road from the sidewalk, for example. And perhaps that is sufficient to convey to the user so that they don't crash in a car. In fact, this preprocessing might differ a lot depending on what task people are doing. If you're navigating outdoors, your needs will be very different than if you were conversing with others indoors. And so maybe there's a mode where we have to highlight the faces, the emotions of people, or inform people when someone leaves or enters the room, and so on and so forth. Maybe you just dropped your keys. And so now it's important to recommend certain objects from the background and so on and so forth. And so the big vision in our lab is to build what I like to call a smart bionic eye. So bionic eye that is powered by A.I. smarts to augment a scene such that it can help specific tasks in the real world. This development would be guided by our virtual reality prototype shown here because we can simulate all that and tested with virtual patients to see what works, what doesn't. And then we can work with the real bionic eye user in the loop to improve our designs. And so in the big picture, this is really where my head is, lab is headed. And I'm really excited about that because I think there's so much potential in using computer vision techniques to improve this artificial vision. However, as it says here, to provide meaningful solutions, we really need to understand the neuroscience. Right. It's the business with the axon fibers that really allowed us to predict how good or bad this prosthetic vision is. And so we have to continue to do that to work with the patient, understand the limitations of visual perception, and understand how the implant interacts with the retina or with the visual cortex, for that is very important, I believe. So you see these realistic phosphine models, the ones that are have been validated with real psychophysical behavioral data, because otherwise your predictions may just be misleading. And so it's important to know that fast things change with amplitude and with frequency and so on and so forth, and to build that into your simulations. And finally, of course, there's an engineering challenge because you need to design these computer vision algorithms that are efficient yet robust. You don't want them to miss the approaching car. Right. That could be life or death. So it has to be really robust. People have to trust it. And so there's still a long road ahead to getting to a functioning, smart bionic eye. But I think once we get to a point where these algorithms, these computer vision algorithms work in the real world, there's a lot of potential avenues to explore. And I'd like you to leave you with that. And thank you very much for your attention. Be happy to take your questions.
