The Future of Machine Learning in Clinical Imaging

[Music] my role is to kind of give you the introduction or really the basics about machine learning and so what we're gonna do is we're gonna talk about how machine learning is very very different from a traditional way that you program a computer and traditionally when you program a computer you have to be very explicit in all of the steps that you teach it to do so just a show of hands how many of you have ever written a piece of computer software okay so you're all many of you are intimately familiar with how explicit you must be and even if you have a period or a comma or a semicolon or one letter wrong that that can totally wreak havoc in in what you're trying to do and so what we're really talking about is a set of tools that allow us to be less explicit about the way that we're teaching the computer to perform a certain task and really do things much the way like a child might learn from experience rather than having to set create a set of explicit rules and so in order to understand this we're going to go through an example here and so what we're gonna do is we're gonna talk about how a child might think or learn about discriminating between a hot dog and a hamburger and then we're gonna talk about how this works for machine learning so a child you know has probably if they've never seen a hot dog before it's unlikely that they're going to be able to identify that and say this is a hot dog but if you show a child two or three different hot dogs it's very likely that they're going to be able to identify that that's a piece of food that I'm gonna eat it it may or may not taste good maybe you're a vegetarian and want veggie dogs I'm not entirely sure but the same thing goes then for a burger so if you haven't seen a burger as a child you don't necessarily know what it is you might from context infer that it's a piece of food when you see other people eating it lots of those types of things but no one had to explicitly sit down and tell you you know this is a hamburger you can eat it it's made of beef and a bun and lettuce and ketchup and that is a burger based on all of its constituent parts it's one of these things that seems more intuitive so when we try to do the same thing with a machine learning algorithm we have this black box and inside this black box we have a bunch of different parameters now this is oversimplified there are only three parameters in this model some of the things that you'll hear dr. Taylor talk about later have hundreds of thousands of parameters or maybe the millions of parameters and so when we have this black box and it's as of yet untrained this is the child before it's seen a hotdog or seen a hamburger each one of these you could little parameters you could think of as a dial so I have all these little dials and little orange thing kind of indicates where the dials are set and when the network is untrained all of those dials are set randomly to different places and so in the training process I can take an image of a hot dog and I can run that image through the network and the network will give me a probability that it thinks that this thing that I fed it is a hot dog so in that very first untrained state getting in a number of 45% is just as likely as getting any other number and so what happens is we say okay we know this is a hot dog and so we tell the the algorithm hey tweak these parameters just a little bit and see what happens and so we tweak the parameters a little bit and we run the hot dog through again and it actually got worse so what happens is that there's a set of mathematical feedback loops that will say okay that wasn't a very good optimization go back to the previous one and try again and so when it goes back to that previous one we spin the dials a different direction and run it through and now you get a higher confidence so after you've done this millions of times it gets to the point where you've set all of the dials to a place where they can more reliably identify a hot dog and one of the things that we do in order to increase the ability of the network to discriminate that is that we feed it not just one picture of a hot dog but lots and lots and lots of different pictures of hot dogs and after you feed it lots and lots and lots of different pictures of hot dogs then what happens is you can start to show it hot dogs that it's never seen before of course if we show it one that it has seen before now it's very confident that this is a hot dog if you show it something new that it hasn't seen before then it may be less confident now the other thing is that all we've done right now if we is that we've trained a hot dog detector okay we haven't trained something that can really identify different types of food because all we've ever done is shown it lots of different hot dogs so if we run some new thing through it that it hasn't seen before the only thing that it's going to be to able to identify in this is that it's not a hot dog so this has a very low percentage possibility of becoming a hot dog so what all of this boils down to is the fact that there's really no free lunch when it comes to machine learning and what we really need to have is a huge amount of labeled data that comes from a variety of sources with lots of different types of pathology on it in order to train something very robust and so some of the latest research papers that are coming out in radiology are really kind of tempering some of the early excitement about this set of techniques because that labelled data and that process is actually very expensive and very tedious and time consuming to execute on so many of you how many of you have heard of computer assisted detection or CAD all right cool so CAD was another way to take an image and say is there a feature in it and probably in clinical radiology the most common example that we have of this is using CAD for mammography so in order to screen women for breast cancer we take a radiograph or an x-ray of the breast tissue that special x-ray is called a mammogram and we run those through a computer algorithm and the computer-aided detection says well I think that there's a problem right here and so what happens in a typical computer-aided detection model is that explicit programming that I was talking about earlier where someone actually creates a program that says okay this is a hotdog what are the types of steps that I would need to do from an image processing perspective to look and determine whether or not this is a hotdog so I might write a piece of software that would identify a stripe of mustard on the image I might write another piece of software that looks for a tubular meat like product I might look for another write another piece of software that says okay if there's brown on either side of the tubular meat like product it's likely to be enclosed in a bun and I might have scores associated with each of those and those are based on features that I know and understand about my definition of a hotdog so if you take this into the radiology space or in the medical space you can do the same thing and write a detector for breast cancer so on the right hand side of the screen here you have a mammogram there are a set of fairly ugly-looking calcifications in the middle those calcifications show up as bright white dots so what I might do is I would write a set of computer algorithms to process the pixels and say if the pixel values are very high and they're arranged in these very kind of bizarre patterns that I might give that a higher weight or a higher score saying this is going to be some sort of breast cancer and then I would do things based on the other types of imaging features that we think about like whether or not there's a mass or architectural distortion and so what happens is we're really identifying and taking all of the medical knowledge that we know about how disease looks in a particular image and trying to teach a computer explicitly how to do that so if we go back to our food example in order to teach our cad algorithm a way to distinguish between food types I have to create all of those little pieces of code to identify lots and lots and lots of different types of food so if I wanted to find pizza I would have to write you know some other set of algorithms that would be able to identify the unique features of pizza and an image and I hope that you can understand that this our ability to write this very descriptive code really doesn't scale very well so the the take-home message here is that computer assisted detection or traditional CAD has really been based on human understanding of images and really human defined features but when we go through machine learning algorithms and things that my colleagues will talk about a little bit later these are really machine detected features and that gives us a much larger opportunity to plow through a ton of data very quickly rather than a small amount of handcrafted features that are written by a programmer or a group of programmers again what this means is that we can scale this very quickly much like early server arrays where people had things computers shoved underneath desktops that's not really our model anymore scalability with things like Amazon Google cloud Microsoft Azure have really replaced a lot of on-prem compute so that scalability is also important to us from a machine learning perspective so one other question here is why did this imaging machine learning get so hot right now and really what we have we have two things to thank for that one is that it turns out that all of the computer technology that's been created for the video game industry in creating really really fast graphical processing units turns out that those processors are particularly well suited to this process of machine learning that we're talking about and so all of the money and innovation that got poured into Nvidia and companies like Intel and ATI really now is being transitioned and leveraged for this machine learning revolution the other really important thing is that there's this thing called the image net competition and so what image net did was essentially create this large repository of labeled data that was required to to do these challenges so I'm just gonna introduce this this concept briefly but the image net challenge actually isn't anything new it's been going on for a very long time back in 2010 you can see these traditional computer vision things as the blue dots here and the blue dots have kind of steadily been marching towards an improvement but this orange dot here I think you'll appreciate was a real outlier in 2012 and this was a network a convolutional neural network which is something that Andrew is going to dive into that really blew this world apart because everybody had been focused on writing each one of those little feature detectors in the past and now they're moving on to these these neural networks and then you can see in 2013 and 2014 everybody has pretty much abandoned the traditional feature pieces of this and they're all focusing on neural networks and if I had the later data you'd see that the error rates just continuing to plummet again what image net is is really a set of those labels a set of information that says this is a hot dog or this is a burger or this is a mountain or this is a bicycle and coupled with these public images that allowed people to get the millions of images required to train a network to have discrimination between lots and lots of different types of things and so what we really need in order to try to move the field here is we need to take all of those images of random you know photographs and really replace those with radiology images that are high-quality and labeled and this is something that's very expensive and it's gonna take quite a long time all right so just a couple of challenges a couple of things to think about not really something that we're gonna dive into today but a couple of hot topics this is a very big one you know if you put machine learning algorithm in place as a diagnostic tool who's going to be to blame if they make a mistake what are the medical legal and malpractice risks that are going to be involved there and we're asking ourselves much of the same question when it comes to autonomous vehicles and lots of other things so we're gonna wait I think for our for the industry to catch up with us there there's also a lot of in a lot of ambiguity amongst the United States about who actually owns the medical record and in California the hospital or the physician actually owns the the data in the medical record so that's not the case in all the states so there are a lot of privacy issues and a lot of legal compliance issues that are going on around creating these big data sets and then of course this is somewhat famous now the apparently the National Health Service got in trouble back in May for giving one of the Google start ups access to a bunch of the their patient data and I think that I highlight this not to paint Google in a disparaging light at all but just to highlight how tricky these issues are and I'll just read this little bit to you it says this response by Google shows that deep mine has learned nothing there may be there may well be lawful reasons for third-party IT providers to process data for direct care of 1.6 million patients unfortunately for google's AI division developing an app is not one of them remember that I told you earlier that having this large volume of labeled data is really the only way to develop these algorithms and so this to me is this was published and Engadget I mean this is not a you know random conspiracy theory type of blog so I just think that this kind of underscores that the legal issues and the privacy issues haven't really caught up with where we where we need to be so on summary here we talked about machine learning we talked about human identified or human defined features that have to be explicitly coded for we talked about how machine learning relies on machine detected features and we talked about some scalability why image net was really important in landmark here and we talked a bit about some the legal issues and with that I'd like to hand it over to Andrew Taylor to take us a little bit deeper into this black box good evening everybody my name is Andrew Taylor I'm an interventional radiologist here at UCSF and one of the main areas of my research along with dr. Mangan is focused on utilizing machine learning to identify critical features on imaging studies I'm gonna spend the next 15 or 20 minutes taking you through some of the concepts behind deep learning and computer vision as its applied to pictures in general as well as an example from our current research working with x-rays it's going to be a conceptual talk there's very little math or computer science but hopefully you'll come away with an understanding of how these things work in a general way which will stand you in good stead at your next cocktail party well you're next Marin County cocktail party I'm not sure it'll hold him down on the peninsula but you know okay so I'm gonna start by describing a clinical problem that we are trying to tackle pneumothorax is a condition where air gets into the chest but it's outside the lung so this can cause collapse the lung and that can be a life-threatening emergency on the left here we have a normal chest x-ray and I really want you to just focus on two things first notice that the lungs are dark because they're mostly air and they have a fine wispy branching pattern throughout them which is the normal appearance of blood vessels and Airways second notice that this wispy pattern goes all the way to the edge of the chest where it meets the ribs contrast this with this picture here on the right this is an example of a very large pneumothorax the the right lung looks pretty normal but but the left chest just looks too empty it turns out that that's because the reason you're not seeing this fine branching pattern is that the left lung is completely deflated and collapsed and that little lump there outlined by the arrows is the left lung now pneumothorax isn't always as obvious as this particular example but here's another one this is an example where you can see that the the lung markings don't make it all the way up into the up right chest and they end at this curved line here which turns out to be the edge of the lung and in fact if you look even more there's another one on the other side that looks pretty similar but it's a little harder to see so here's the problem pneumothorax can be a big emergency and it's usually seen on chest x-ray sometimes very easily sometimes not so easily but a typical Hospital may generate hundreds of chest x-rays in a day and unless a study is flagged for immediate review because somebody's freaking out about it they tend to get read in the order that they're obtained so it may be a little bit of time before that study gets reviewed in the case of a free-standing clinic there may be no radiologists on-site and so then the burden is on the primary doctor to review the images and make sure that there isn't some sort of big problem going on and in some countries there are a very few radiologists period and so there can be a long delay before a film gets officially looked at so our goal here is to develop an algorithm that can screen these images and alert the radiologists to turn their attention to the you know the potentially more serious findings first or can contact that point of care doctor the er the urgent care doc and say you know the algorithm is worried about a significant pneumothorax here so please look at this image and contact the radiologist right away an algorithm class that's well suited to doing this kind of work is a convolutional neural network which you may have heard of because it's the type of network that companies like Facebook and Google use to organize your photos based on who or what is in them and it does play some role in the development of things like self-driving cars if you google convolutional neural network you're going to find lots of pictures that look like this which seem pretty opaque and confusing at first the top picture is actually the grandparent of the algorithm of the US Postal Service uses to decipher writing on handwritten mail and the lower picture is an example of it's a more generic example of a network that's being asked to identify pictures of vehicles so cars trucks bicycles etc they share a lot of common features and as I'm going to show you they kind of break down into two parts on the left is the part that's trained to answer the question of what are the key features in this image and on the right the part that says okay given given these features and how they're arranged in this picture what do I think this picture is so these diagrams kind of typically flow from left to right when a network is analyzing an image but to dig into this deeper I'm actually gonna talk about the right-hand side first and then we're gonna look at the left so looking at the right side first this is the what do I think this picture is side and to understand a little bit better how this works I think it's helpful to know why these networks are named as they are you know why are they neural networks so these are not trying to replicate the brain in silica and in fact if you carry the parallels too far they start to break down and it makes hardcore computer scientists and neuroscientists alike both very irritated but there are two basic ideas that I think are helpful here the first involves how neurons are connected and how they communicate so in the center here we have a neuron and it's receiving input from three other neurons the input may be excitatory it may be inhibitory but the input is being collected and kind of summed up by this green neuron in the middle and based on that input it either fires its messages to downstream neurons or muscle cells or it doesn't you know the the threshold for this action can vary from neuron to neuron and even from moment to moment now this is a schematic of a mathematical model that is a correlate of the picture that I just showed you it's called a perceptron and it was developed in the late 1950s it shows a a neuron or a node here which is really just a place where some computation happens and it's receiving input from some basically four other neurons off to the to the left and it's sending its output off to the right so think of it kind of like the three-way switches in your house where you have more than one switch controlling one light bulb in this example I've set it up so the two inputs are on and two of the switches are off and that that's serving is the input but this perception isn't very impressed by that so it stays in the off position but if you add one more on signal then it reaches its threshold and it sends out a signal that that serves as input to similar units further downstream so this is a little more complicated than that because there are additional numbers involved here which are called the weight and that has something to do basically makes makes it such that the numbers don't have to be only 0 or 1 and not all inputs have to be viewed as equal so a slightly more accurate metaphor might be to use a dimmer switch here but in general this is how you get the idea now if we take a lot of these little discrete computational units and we arrange them in multiple layers where each node receives a lot of input from its upstream neighbors then we may be able to do some much more complex things than just the on/off that I showed you so this is really the basis for that output similar to what dr. Coley was talking about with the whole hot dog not hot dog it's that the that's the basis for that kind of output so this is how you answer though what do I think this is part in that Network that I showed you that looks at cars and trucks and so forth these outputs are generating a number that reflects how likely the picture is that it's been shown a car or a bicycle etc and so part of the learning that this network is doing when it's presented with lots and lots of pictures of cars and bicycles is it's making very small adjustments to all those weights that go with the inputs which hopefully improves its likelihood of making the right call this is part of why these networks are called neural networks and at a conceptual level this is how the network is going to classify or make decisions now the second parallel between these networks and the brain starts to show how the left side of these networks function the side that deals with what are the key features in this picture it comes from some seminal work in neurobiology and physiology done by David Hubel and Torsten Wiesel starting in the 1950s and for which they received the Nobel Prize in 1981 they were looking at how the visual system worked in the cat and they were making recordings from individual neurons in the cat's visual cortex so for this example they're presenting a stimulus in the form of a bar of light which is here and they're projecting it onto the screen in front of the cat and they're recording from a particular neuron and when they first present a horizontal bar of light they don't really get any fire the neuron as they rotate that bar towards the vertical they start to get more and more stimulus and you get maximum stimulation when it's vertical and then it starts to roll off again as the bar heads back down to horizontal so they had discovered that some of the lowest level parts of the visual system were basically feature detectors where they might respond to an edge or a bar of light at a certain angle and other detectors might respond to certain things like collar or motion and this has extensions into the human visual system which I've shown with this diagram where a person's looking at a head or a face they have these sort of lower-level parts of the visual cortex that are responding to these basic edge shapes and then they have connections that go higher up in the chain start to assemble some of these basic shapes into compound or more complex shapes and then you go higher still and you start to see these higher-level representations of objects like eyes or ears or face or head shape and then obviously you can play a lot more information on top of that but this is sort of a the basic idea so how do we do this with a computer algorithm so here's a picture of someone you know your visual system possesses the features excuse-me processes the features and you very quickly say this is a picture of Abraham Lincoln but if you'll pardon the anachronism say you pull out your iPhone to get a selfie of you and Abe and that image is recorded by the phone is a grid of pixels blocks where each block is a single color or a single level of gray but this isn't really what the computer sees either each of these pixels has a number associated with it that describes how light or dark it is except the computer doesn't really know what light or dark means kind of either what it sees is this a grid of numbers so how does an algorithm start to pull out features that might be important in identifying what's in photograph and this is where the convolutional filter comes in a convolutional filter is just a small grid of numbers something like 5 by 5 or 7 by 7 and the numbers in that grid are used to process the pixel values at many positions throughout the image really just using multiplication in addition so the filter is kind of about a postage stamp size relative to the size of most images and so it is convolve Dover the image it is moved or slid like a window across all the positions that it fits in an image and the computer does its multiplication and addition and it generates a single number based on the that grid at that position you know at that time because it's only looking at a very small set of the pixels so this is an example of that multiplication addition that I was talking about the the orange is a little filter being slid across the green image with its pixel values and it's doing multiplication and addition and for each position it's generating a single number which is in the the pink there now that gets assembled into its own little grid and that's called a feature map or an activation map and a convolutional neural network network will use hundreds or thousands of these little filters each with its own unique set of numbers and each one generates its own feature map and then they all get passed on to the next level which often is another set of filters that works kind of just like this and so when you start to stack these things together you start to generate some higher-level feature detection like we saw on this slide where the where the human was looking at the face so here are 2 filters two example filters being passed over an image and these filters both happen to be kind of edge detection style filters and they generate similar but slightly different feature Maps areas that are bright on these feature maps are places where the math has produced big numbers or big activations and dark areas are where the activation is low and so the learning here involves tuning the values inside these filters to produce feature maps and activations that are going to go on to provide helpful input to the second half of the network that classification part that helps it make correct decisions about what's in the image so now it's important to note that I'm showing you things like edge detection which is something that we humans can understand in terms of like oh I get it I get what the algorithm is doing there but in many cases perhaps in most cases the filters and the feature maps may not produce something that humans can really understand which kind of alludes to why dr. Coley started off by saying it was a black box but in some cases it can and this happens to be one of those cases and it's a nice example so this comes from a paper that trained networks using images of human faces and the low-level activations look just similar to what I've shown you before they look like they're excited by curves and edges etc these low-level features start to pass their activations on to higher levels and they're starting to be assembled into things we can recognize like eyes or noses or seems to really like eyebrows too and as we go even higher we see very complex activations like complete faces so in this kind of a network in this example at least some of the ideas have built starting simple and kind of building hierarchically you can see some parallels between this and some of the neuroscience slides that I was showing you earlier so let's go back and look at our diagram of the of this convolutional neural network again and as we train we're typically processing images from starting off going from left to right and the first part which contains these convolutional filters which are the little boxes is the is the sort of can I highlight important features part and after a number of these layers it generates output that's passed to the multi node neural network that I showed you first the what do I think this is this picture is part so the big difference between these algorithms and some other image processing algorithms as dr. Coley mentioned is that we are not telling the algorithm look for edges or look for colors etc the computer is figuring out what it should look for on its own and how is it doing this so in most cases the the picture has been labeled with what it is so the algorithm has shown many many pictures and it goes through its computation and it produces an output which is a series of numbers that mathematically expresses its best guesses as to what each of these images are what class it falls into and that mathematical expression gets compared against another mathematical expression that represents the truth ie what the label that was what applied by a human says the picture is and so then you can basically calculate just how far off the mark of the computer is at least after a given round and then through the magic of calculus the computer derives what direction it needs to go to try to make that distance a little bit smaller basically to make it less wrong and once it has that information it actually then propagates back through the network right to left here and it basically makes very small changes to all of those weights the numbers and the filters that I showed you and the numbers in between the nodes in that neural network that was part of that first example and once it's done that it then goes back through the algorithm from left to right and sees if it's decreased how wrong it is so if you do this many many times back and forth forward and back propagation is it's called you're trying to get the predictions to be closer to the truth through repetition so you'll hear it said that these algorithms are data hungry which is true this is a big difference between humans and computers as Mark alluded to first humans are very very good at generalizing their knowledge based on limited experience at least when it comes to vision so I can show a child this picture and I can add a couple of labels and pretty quickly that child gets pretty good at separating cats from dogs even cats and dogs that don't look like this and cats and dogs that's never seen before so maybe we have to show it a few more versions than just one including the cat with the tutu but it doesn't take very many examples before you start to generalize your experience computers are not good at this so for them to do reasonably well on this same task they need to see a lot of images like a whole lot of images potentially hundreds of thousands or millions of images but if you have the data and you have the label truth then these classifiers start to get very very good at doing this kind of thing so let's return to the clinical problem for a couple minutes we've labeled images as part of our research as having a pneumothorax or not but we haven't provided any information to the computer as so where that is or even really what it is so the computer needs to learn and to develop a set of features that will help it to separate these pictures into the correct categories based on the labels that have been given by the human by the radiologist so what might these filters look like now as I mentioned before many of them are not particularly human interpretable and so these are some hyper some hypothetical examples that are but they're valid in that there are many filters and this is potentially one of them so if we looked at the left upper chest in these two images we could imagine that a filter might have one level of activation due to that fine branching pattern of the lung parenchyma the lung tissue that I showed you before and it may have a very different level of activation when it's presented with this very monotonous dark space that lacks those markings so this might be one feature that helps to distinguish between the presence or absence of lung in an area that's supposed to have it similarly another filter might pick up on the presence of this extra line here that in the upper chest it doesn't seem to be part of the ribs it doesn't seem to be part of the normal configuration of the lung and it's seen a lot of normals so this may to also provide some contribution of there's something about this is different and that may tip the algorithm in favor of abnormal or pneumothorax now needless to say many pictures of new authorities look different and many normal chest x-rays look very different from one another even when they don't have a pneumothorax so really the only way to give a computer a lot of experience so that it can make predictions about images that's never seen before is to have lots and lots of pictures so for this project we've assigned labels to thousands of chest x-rays human radiologists have assigned these labels and we've tried to include as much sort of normal variability as we can offer it and then we've done some things to expand that data set even more so that by the time that we're training the algorithm is potentially exposed to a hundred thousand or more images and this seems to be a reasonable sort of minimum requirement to produce an algorithm that yields decent results for this kind of a clinical problem so how are we actually doing pretty well if I say so myself you know our work continues but we've been able to build models using a lot of the techniques that I've talked about tonight that can catch the majority of these new authorities that might go on to become a serious problem for the patient and we're still working on the problem there's still a lot more to be done but we do seem to be pretty well on our way to having a useful tool that can assist the radiologist or the primary doctor in these situations and the tool too in order for it to be created it kind of has to leverage the wealth of data that we have at a center like UCSF and be guided by the clinical knowledge and experience along with lots of hard work and experimentation but this is a very exciting time in medical imaging I think there's a lot happening and a lot more being promised and as with all exciting times it'll be interesting to see how much gets delivered and how much turns out to be hype but with that I'm gonna stop and let my colleague dr. John Manga tell you a little bit more about that based on his experience so thank you very much I hope you're enjoying tonight's session good evening I'm John Magan I'm the vice chair for informatics for the radiology department and I have the pleasure of talking to you tonight about what I think all this means where this is going in terms of the future of radiology for radiologists and for patients so the future of radiology based on what we've seen tonight you know that radiologists look at images you've heard from dr. Coley and dr. Taylor that AI these deep learning convolutional neural network algorithms in a sense also look at images and these AI algorithms are pretty good at looking at these images and they're rapidly getting much better and there are certain advantages to an AI algorithm over a human radiologist they don't get tired they don't take vacations they don't complain about their office space so a natural question is what does this mean for radiologists and a lot of people have thought about this and have spoken about this and somewhat famously or perhaps infamously this gentleman dr. Geoffrey Hinton who's an emeritus professor of the University of Toronto and now works for Google and this this is a guy who is widely regarded as one of the founders of this sort of recent explosion in neural networks and artificial intelligence and one of the world's leading experts on this gave a speech last year a talk a seminar kind of like this one in which he said it's just completely obvious that in five years deep learning is going to do better than radiologists and just in case the implication of that because it takes five years to train a radiologist was lost on anyone he followed that up by saying they should stop training radiologists right now and there are a lot of people in engineering and computer science who have expressed similar sometimes even more extreme thoughts and positions on this and this is something that's been picked up by the popular press and has led to a series of articles of which this that you may have seen in NPR from a couple of months ago is one of just one of the more recent examples that have made the usual analogies between radiologists and buggy whip manufacturers and have created kind of a popular impression that radiology is really at least for radiologists a field in its waning days so why am i standing here talking to you instead of feverishly preparing my resume for a career change well I would represent to you that while dr. Hinton knows an awful lot about artificial intelligence and neural networks and probably more than I will ever know he doesn't actually know that much about radiology and in general I think these people predicting the end of radiology don't fully understand what radiology is and more specifically what a radiologist does and the key is that radiology is not just a perceptual task to use dr. Coley's analogy it's not just the medical equivalent of distinguishing medical hot dogs from medical hamburgers what a radiologist does is a radiologist as a physician who specializes in using technology to make diagnoses and AI is a new technology and it's a new technology that is getting very good some perceptual tasks but those perceptual tasks are not all of Radiology there's small part of it an AI is a long way from being able to perform the full range of cognitive tasks of diagnosis of the full range of human disease on all the different kinds of imaging studies that we can do so really what we have with artificial intelligence is a new technology that's capable of doing some of the important things that radiologists do perhaps doing those things better than radiologists do but not all of the things that radiologists do so in thinking about where the field of radiology is going we need to try to understand how that something that can do some of what radiologists can do maybe faster maybe more efficient maybe better but not all will affect radiology and it's difficult to predict the future so one thing that we can do is we can look to historical analogy so I'm gonna take a step back from talking about the future of radiology for a minute and talk about the history of radiology and radiology is a field that was founded on scientific discovery and technological innovation and the key initial discovery and technological innovation was the discovery of the x-ray and that the x-ray was was radiology for most of the the first half of the last century to the extent that in hospitals like this one that have a long history you can still find signs that point towards the radiology department that just say x-ray on them and in that in that world that x-ray world you have to have radiographs like this one here of the pelvis and one of the you know one of the key skills that a radiologist had was to be able to look at this image which is a flat 2d image and makes this pelvis looks like it's something flat and recognize that this is actually an image of a complex three-dimensional structure and to be able to sort of understand and synthesize in their head that three-dimensional structure looks like so for example I know from my training and an atomic knowledge and many of you may know as well that the pubic symphysis right here projects forward in this pelvis and the ischial tuberosities here and here project backward but that's not immediately apparent just by looking at this flat image and that ability to perceive the three-dimensional shape based on anatomic knowledge and training was was one of the things that was sort of a key differentiator a key skill that radiologists had that many of the other people in the hospital didn't and in the late 1970s there was a technological advance and CT scanners started to become available and CT scanners are essentially three-dimensional x-ray machines and rather than creating a single image where all that three-dimensional complexity is collapsed down into a single image they create a series of cross-sectional images going across the patient so if we look at a CT of this patient we can see that going from forward to back we see that the first thing that comes up I'll let it play through here again the first thing that we see is this pubic symphysis here in the middle so we know that that's towards the fore and towards the the forward as we go further back we see these issue Barossa T's and so that complexity of trying to understand what the 3d relationship was that was once entirely the domain of people who had been trained to recognize it mostly radiologists then became something that a machine did far more accurately and far more quantitatively and so one could imagine that we could be sitting here 40 years ago in a similar seminar talking about the advent of CT scanners and someone might have advanced the idea that a CT scanner does better than radiologists at evaluating 3d structures and could have proposed that that meant that the future of radiology was that really radiologists were going to go away because now this skill that radiologists had had trained for years to acquire and was a unique talent of radiologists who's now accessible to everyone and anyone looking at the CT scan could obviously see what the 3d relationships of things were and so you know any doctor could then just start looking at these images and you really wouldn't need radiologists anymore we look at what actually happened with the advent of CT scanners CT dramatically increased the diagnostic power of radiology and along with that increase in diagnostic power came a whole range of new complexities and how the machines were used all the new things that you could see and what those meant how you interpreted those how you understood artifacts and so rather than leading to you know the end of radiologists or do a decrease in the number of radiologists and actually led to an explosion in the demand for radiologists as radiologists did what radiologists do which is use technologies to make diagnosis and now they had a new more powerful tool that allowed them to make all different kinds of diagnosis that they had never been able to make before and so I think our friend dr. Hinton has spent some of the last year talking to some radiologists because this year he gave another talk in which he said the role of radiologists will evolve from doing perceptual things to doing far more cognitive things now this is a very very different statement from the one that he made last year because last year he was telling us radiologists pack your bags you're done and this is very different this is saying well there's going to be a role a cognitive role for radiologists to continue doing diagnosis which is what radiologists do and they won't maybe be doing quite as much of the perceptual task because that's something that artificial intelligence is really good at so what this means is that if we look at AI in medicine these convolutional neural networks deep learning in medicine and in radiology really my experience is and and what we're now told by one of the leading lights of the field is that this is not ready to replace physicians it's not that it's not ready do anything you know I'm working on this project with dr. Taylor and I wouldn't be spending a lot of time on that if I didn't think it had any value it does have a lot of value it will I think soon assist with many tasks that radiologists do particularly these perceptual tasks and so what I think that this points to is a future of radiology where we have improved diagnosis that's delivered by human radiologists doing what they do best working in conjunction with AI doing what it does best and I think that there are if this is the model that we're going to which I firmly believe that this is this is the potential this is where we can actually realize benefit for patients from AI in medicine that has some very important implications for how we get there for how we develop these algorithms if you're starting from a premise that you're working on creating these algorithms to look at medical images if you're starting from a premise that you're replacing radiologists then how radiologists work how they think what they need help with really isn't all that relevant because you're replacing them they're going away anyway and so if that's the model that you're working from and this has been a common model of many of the tech startups in the Bay Area and around the world then your development in your development model AI specialists computer scientists really have the central role because they're creating these algorithms and radiologists have at most a minor temporary role in labeling and creating these data sets but once you have sort of extracted their knowledge in terms of label and creating the Davids datasets you can kind of crumple them up and throw them away because you don't need them anymore but we've just discussed that replacing radiologists is really not on the table any time in the near future because these algorithms can't do the whole range of cognitive tasks of diagnosis that you need to replicate what a radiologist does so that model really isn't what we need to be doing because that takes us in a direction that's not going to be effective in producing something that's clinically useful or that benefits patients what we need to be doing what dr. Hinton has suggested is that really the role for artificial intelligence in the foreseeable future in radiology is assisting radiologists and so if you're assisting radiologists you really need to take a different approach in that case what you're doing is you're not replacing the radiologist but you're developing tools that are going to help the radiologist and those tools need to be tailored to effectively and efficiently work with the radiologist so you absolutely need to understand the way radiologists work the way radiologists think what they perceive is the things that they know how to do what they struggle with what they think the problems are because you're creating a tool for someone you can't create a tool for someone to help them do their job if you don't understand them and you don't understand their job in under this model AI specialists are still extremely important because you need to be able to create algorithms that are going to be accurate that are going to be powerful that are going to be useful but radiologists are now equally important because you need them to identify what the problems are to direct the work so that the tools that are created are things that are actually useful to radiologists and fit effectively into their workflow and so another implication that I would suggest is that this suggests that the environment for where we can optimally realize the potential for developing this and this these algorithms much of the early work on this has happened in tech firms in computer science groups but I think that academia and in particular academic med centers are actually ideally suited to pushing forward this work that needs to be a collaboration between computer scientists and radiologists and physicians and some of the reasons for that are we in academia are very experienced with cross discipline collaboration I think in many ways much more so than a lot of startups and a lot of people who are in the pure tech fields many of these tech firms even the ones that are explicitly working on being algorithms for radiology have no radiologists in their company they might have you know a couple who they have as consultants who come in a couple hours a week maybe they only label images some of the really larger firms are now starting to hire a couple of physicians we've got hundreds of radiologists here and thousands of other medical specialists so we have very easy access to the medical knowledge that you need to do this my colleagues talked about these algorithms and these approaches being extremely data hungry needing lots of data this is a major problem for most people in industry trying to work on these problems they don't have access to data we have access to decades of radiological images here once you create one of these things you need to test it you need to see if it actually works well if it actually does help radiologists if it actually leads to better diagnosis and better patient outcomes getting access to a clinical environment where you can do that testing is very difficult for most companies that's something that we're embedded in here and not just any medical environment but a medical environment that is used to in tune to innovation and trying new things and research and testing things the one thing that by design is a public university we don't do directly is turning these things into product and commercializing them but we have a lot of experience of collaborating with corporate partners who do that in tech transfer and IP licensing so we're really very good at you know that translational aspect of partnering with other organizations and companies to move these into the commercial realm where they can actually be distributed outside of our university and impact patients there are resource requirements for doing this for realizing this goal you need to support research time for physicians to do this and this is this is difficult work it's not something you can do on the on the nights and weekends you need to have dedicated time to doing this you need data engineers and data management infrastructure one of the first things that you discover when you start to do this work is that you need to have a way you you know it's very data hungry you have these mountains of data need to have a way to organize and to collate and to manage all this data you have in efficient fashion once you've created these data sets and you can manage them you need data scientists who know how to create these networks how to train these train these networks manipulate the learning parameters so that they train optimally and then additionally you need these GPU graphical processor unit based computers that can execute these algorithms and do this training efficiently and we've laid the foundation for doing work in all of these areas all these different requirements and our we have work underway to expand the capacity in all these areas so that we can realize the potential of really pushing this forward into something that's going to be a tool that helps radiologists and patients so to summarize I think the key points that I'd like you to remember from what I had to say here today is despite what you may see in the popular press and I think that the future of radiology looks very bright both for patients because of improved accuracy and ability to diagnose disease and radiologists because there's going to be new tools that radiologists can employ to be able to make diagnoses that they've never been able to make before and make them more accurately and efficiently I expect that cutting-edge radiologists will be routinely using AI based tools within the next few years and I think that academic medical centers are the ideal environment for bringing together all the elements that are needed to really realize the potential of AI in this you know seven next several decades where AI is something that is assisting physicians to provide better medicine to patients so with that I'd like to thank you for your time I'll invite my co-panelist to come back up here and we'd be happy to take any questions that you might have sure so so the question was we have a lot if I understand correctly we have a lot of diagnostic technology currently in use we like to think that we're pretty good at using it here and we have a pretty high standard of diagnosis so you know we're how will they I improve upon where we already are so you know I'll give my thoughts on this and my colleagues may have has something to add as well I think that predicting the future is difficult I think that you know the the one thing that I can say with certainty is that there will be changes in improvements and that many of them will be things that are unexpected but some of the things that I would expect the work that dr. Taylor and I are working on are is working towards making these diagnoses more quickly so you know if you can prioritize if you can recognize which studies are more likely to have disease and read those first then you can get the diagnosis more quickly and sometimes you know for acute diseases acute problems more quickly is you know is earlier treatment and better outcomes for patients I think that you know part of this may be we've talked about computers AI is doing perceptual tasks and there are some perceptual tasks related to screening that it's possible that in the near future AI may do better it you know not missing things it's difficult to go through thousands and thousands and thousands of images and with a hundred percent fidelity identify every single circle that you should be seeing on there and so you know I think that computer aided detection computer aided by that traditional computer aided detection in many ways never quite got to where we hope that it would but I think that this may help to reduce the what is already a very small fraction but we would like to get down to as close to zero as possible things that are imaged but maybe get get do you have other things that you like that yeah just one other thing to add I think that machine learning today would remember I talked about human derived features and machine derived features so I have friends that are working on training machine learning networks to detect things that humans may not even be able to see so for example if you're looking at the genetic pattern of a brain tumor were they're finding that they can train a network to identify the genetic pattern in ways that the humans don't even really understand how the computer is doing that that's very exciting and very scary all at the same time so having the the capability for the machine to be able to define and detect features that we don't understand I think is another place where that diagnosis can go even farther than where we are today but again we'll use that as a feedback mechanism on our own to improve our own understanding of disease dr. Dylan John you see assessment in the United States there's some variation and then in the world there's even greater variation number of radiologists yeah I think that there's an important application for that in areas that are really underserved by by radiologists you know as as we discussed I don't think that in the near future these algorithms are going to be able that you're going to be able to create a digital radiologist that you're going to be able to replicate the full range of cognitive tasks of of diagnosis so you know even for these underrepresented areas for these you know developing areas I don't see this as replacing a radiologist but I think that it may have great application in terms of prioritizing and screenings so you know there may be a situation where you have you know a country where you know in the United States every every radiograph that's taken is read there are many areas in the world where that's not the case and I think that there may be a lot of utility in this in focusing and prioritizing the time of the radiologists that you do have available towards the images that are more most likely to demonstrate disease and have impact on patients so I think the question is can you give a timeframe for when some of these things might clear some regulatory hurdles and become more mainstream at least in in say US medical centers and then also when do we reach this point of the data is there and and it is it's analyzed in ways that may not be human interpretable but essentially pulls out you know features that we do we don't see well I think that's an excellent question I don't think I have a great number or numbers to give you this is something that that people are still talking about in debating in terms of how is the FDA going to even deal with this there their experience is much more in devices to some degree but but drug discovery and so forth and the development of that and and even the processes by which you need to show steps in development are very very different than these kinds of models and I think there's a second question there which is people are developing the tools right now but as John alluded to in his talk there is the validation side of all of this is it is it going to generalize well in a number of settings and also is it effective if we develop some great detector that does something really really well but it's doing it as well as a radiologist and it takes five times as long then we haven't helped anybody so I think it's going to be I think there will there will be some things that that get announced as look what we can do I think the process of approval is probably still years of sort of sorting that out a few years and then there's the whole validation piece so maybe I would agree with everything that the dr. Taylor said and you know I think I don't think that this is going to be a big bang I don't think that this is gonna be something where you know it goes along for several years and then you show up for work one day and all of a sudden like you're in a fully AI enabled reading room this is gonna be something that phases in gradually and I think the initial applications of these are going to be things where it you know it are going to be the the lower bars things where if the AI messes up it's not that big of an issue so things like prioritizing studies you know if currently you're not prioritizing your studies at all if you have an AI algorithm that prioritizes them even if it doesn't prioritize them quite correctly you haven't really lost anything and if it does prioritize them correctly you've gained something if you have an algorithm that automates some tasks like determining a volume a 3d volume of something and you visually inspect what it's done then if it messes up you see it right away and so there's there's no problem and I think that those are the kind of applications that we'll see first and I think we'll probably be seeing those kinds of applications where really the you know the physician the radiologist is immediately checking every step right away probably you know timelines are tough but I guess three to five years and then as time goes on we'll start to see some of the you know the higher bars the more difficult things you know things where it's you know as dr. Coley talked about things that involve a diagnosis that that a human can't really directly see like what subtype of cancer you know what subtype is this tumor where there's not really any way for a human radiologist to directly verify that that's gonna require a lot more regulatory a higher regulatory bar a lot more validation I think those kinds of things are further down the road ten years maybe and a complete digital radiologist where the CT scan or the mr goes in one end and the complete report comes out the other end I don't expect to see in my career I think it's possible that we will get there someday by the time we get there we will have AI that's sufficiently advanced that it won't just be replacing radiologists it'll be replacing attorneys it'll be replacing reporters and so it will be a big change but it will be a big change for on a much larger scale than just radiology I am not worried about job security I would just add one tiny little briefing and then we'll get to the other questions that the FDA is really drawing lines and saying that diagnosis is really something different and we'll have to have more layers of approval and a more stringent process so they're they're trying to figure out where to draw the lines between the different types of tasks that John outlined but I think some of those things where you're prioritizing work are gonna get they're gonna get approved fairly quickly because there's really not much risk to the patient not much change to the process today thank thanks for your questions so the the question was if if does the image quality is that does it have a big bearing on how the algorithm performs and I think that a big part of that is what it was trained on so if the algorithm was always trained on a perfectly executed frontal chest radiograph and never had any variability in that radiograph exposure or otherwise it's very difficult to for the algorithm to understand the radiograph that's maybe not on axis or maybe not obtained with a good exposure so I've been involved in some other research that I've actually worked in Kenya and collect digital chest x-rays and we were trying to have a computer vision process where we were describing features and cutting out the lungs and all the stuff like I talked about in my talk and that is really quite sensitive to the variability between good quality imaging and poor quality imaging but some of these newer convolution neural networks like dr. Taylor talked about are much less sensitive to that image variation if they've seen it during training yeah I would just add to that briefly the project that we're working on we see and hope for actually quite a bit of variation I realized that you're asking about overall level of quality of the imaging and so forth but the technology that's available in many places is is very good I mean even worldwide in terms of the kinds of images that they're producing and so one of the things that we're striving for is when you get a very large dataset is you want the data to be clean in the sense of you don't want it to be we're not trying to make a classifier for chest x-rays but we're accidentally feeding at scull films or something like that but at the same time we actually really do want as many different ages of patient and what they're getting the chest x-ray for I mean I can tell you a chest x-ray that's taken at your local screening clinic where you walk in under your own tooth you know on your own two feet under your own steam and get that chest x-ray looks very different than you if you are unfortunately in an ICU really sick with something so we actually look for a lot of that variability in the hopes that things will be more robust but you do raise an excellent point that there's going to be sort of a minimum level of quality or even just speaking the language of making sure that the image that you're presenting is what the algorithm was trained to look at and then the variability we see as a strength in that provided that you're kind of playing within the rules I don't see any partnership with industry so you know I think that that's that's still a bit of an open question I think that there are a lot of very encouraging aspects to that the the software tools that people are using for this are largely open-source software that's available to anyone so really you know the the major barrier is having the knowledge to use them and that knowledge is available on the Internet to anyone who wants to sit down and work through the math and work through the computer programming the the compute resources that are needed are not insignificant but as an example the you know nvidia is one of the you know a top-of-the-line GPU compute sort of supercomputer if you will goes for about $100,000 or $120,000 now that's a lot of money but when you consider that a top of the line mr scanner goes for three or four million dollars on the scale of radiology and imaging it's actually not that much money and that's for the top of the line and you can do useful work with a lot less than the top of the line so you know in terms of accessibility of these resources to you know in places where there are limited resources i think it actually looks pretty promising and additionally the the resources that you need for creating these algorithms for training them are much higher than the resources you need for executing them so you need a lot more compute power to create the algorithm once the algorithms been created to just run images through it and get them you know get them classified get the detect you don't need nearly as much compute power just just one great concrete example of that so anybody that has an iphone if you go into your photos you can actually go into the search function and you can do something like type mountain and it will look through all of the photos and your photo stream and it'll pull up the ones that have pictures of mountains it's a fantastic exercise to try i did this it actually found not only the pictures of actual mountains but also paintings of mountains that i so it's the the ability to deploy these algorithms once they're trained takes much less computer hard much less computer horsepower than in order to create these algorithms for all this segmentation which you mentioned earlier volumes disucss burning of active research in auto segmentation and or any sort of open source software that you - for best class so yes there there are some research groups in in our department that are working on that particularly in musculoskeletal imaging I'm not personally directly involved with those efforts I wouldn't be able to to point you to a best-in-class for you know what what software you should use for that but yes there are there are people here working on on auto segmentation and okay so the question was are people using transfer learning and so transfer learning is the idea that you have these networks and you can start as dr. Coley mentioned with random with all the parameters randomized and if you start with them randomized then you need to have an enormous amount of input data to get those parameters into a reasonable state the other thing that you can do is you can say as dr. Taylor was showing the the first layers of these networks really just look at features things like bright spots and dark spots and lines and really that's kind of like the basic wiring of a vision system and those features probably don't vary that much whether you're looking at hot dogs or hamburgers or cats or dogs or pneumothorax or no pneumothorax and so the idea of transfer learning is rather than starting with random weights start with a network that was already trained to do something pretty well like the image net network for instance which recognizes pictures of dogs and cats and boats and trees and all kinds of things and use that as your starting point instead of random weights and just replace the top few layers of that and so with that very long introduction to your question the answer is yes most people who are looking at medical imaging most of the research that's presented uses transfer learning because it's very difficult to generate a data set that is large enough to where you can outperform transfer learning by starting from from random weights just as a quick aside on that actually on a different project but looking at x-rays that we had done last year or was it two years ago now last year I think we sort of we were looking at image net images because the the the images are available as this gigantic data set and so x-rays are generally black white and so we looked at this and said well we should try to do our own sort of we do take advantage of transfer learning but we were also curious as to whether or not we could transfer learn by training on a subset of those images and so we looked at the classes and thought well what's kind of like an x-ray mostly sort of monochromatic and we picked architectural features fungi and something else was at trees or dancers or something something I just things that had a lot of these kind of angles and we're not wildly colorful etc and it worked very well so yes is the answer and and we've had some experience doing that specifically for x-ray projects very long yes yeah so so the question is when you train a neural network you train it towards some particular output and in the AI field we call that ground truth that this image does have this disease on it this image doesn't have this disease on it and the question is that that had that ground truth has to be somehow established and there's going to be some imperfection and some bias and how you establish that ground truth and so how do you how do you deal with that and so that that is a key issue in in these AI these machine learning approaches is you're only as good as your ground truth the best you can the best you can asymptotically approach getting to a hundred percent agreement with the ground truth that you've established in your training set but you will never get better than that and so if there are biases if there are inaccuracies in that then that's going to be reflected in the performance of your network so you know some of the approaches that you take to that some of them are familiar from other research you do you know multiple different radiologists reading and you can look at consensus and that can try and get you towards you know performance that's maybe better than any individual radiologists but a performance that is sort of the a composite of multiple expert radiologists one of the things that I think is really interesting about the potential for this technique is you know looking at things other than radiologists interpretation of ground truth we don't have to be limited to that you can look at you know pathology data you can look at patient outcomes you know what you know rather than looking at whether a radiologist thinks that there's disease on this image you can look at well was this patient still alive or was this patient still able to walk a year later from this and I think that that's really the Avenue by which we're going to try best address these bias issues is by looking at more objective patient relevant outcome measures as our ground truth now that becomes more difficult because there's a lot of noise and those signals and you need even more data than you otherwise would but I think that's really where some of the true potential of these techniques lies so just to add to this question of ground truth if we go back to the FDA question previously as John's indicated you can have a training set of data and you can make that algorithm run really well on that training set of data but what are you gonna do when you go to try to commercialize that and you want to sell it and now you're gonna show it other data that it's never seen before how is the FDA going to validate the claims that are going to be made based on that algorithm for that set of training data and where is the ground truth for the FDA's validation data going to come from so there is a whole myriad of questions that really need to be answered that's a fantastic one and again one that we don't have all the answers for yet okay so there were a lot of aspects to that question that I'll try to unpack so I think that the central question was personalized medicine in many ways involves analysis of very large amounts of data that really require computational techniques and you know certainly no person would be able to for instance sit and analyze a genome by by hand and so given that does that suggest the really computers will take over all this this work so you know I I would agree that the the computational Technol is medicine does generally involve a lot of sort of large data kind of techniques and computers are essential in analysis of those data both with traditional analytics and and calculations and machine learning techniques I think that one of the things that we see is you know as you know in informatics in in biology and medical informatics and personalized medicine is that the there is oftentimes a thought that with automation and with the increase of computers that there will be less need for people working in these areas and I think that what we're seeing what we've seen historically and what we will continue to see in at least the next several decades is actually the opposite of that that you need more and more people to understand how to apply these techniques when they work when they don't work and how to interpret the answers that come out of them and put them in an appropriate context and I think that that's likely to be the trajectory and likely to be what is happening in terms of humans and computers working on analyzing these large collections of data for imaging for personalized medicine for the foreseeable future of the next several decades of my career you know at some level I am a neural network a very complicated neural network as you are and it's possible that at some point we get to the point where we can create a neural network that has similar complexity to you or me and similar cognitive ability and when we reach that point it will be a major major change for Humanity but what my feeling is that that is at least many decades off if not a century and so I think that's something that is interesting to think about but is not a present situation for us to deal with I think we have time for one more question in the third world there's radiologists but ah you know they're good I think that that's something that we we kind of struggle with from time to time whether it's in the third world or whether it's here you know the one of the examples that I was given in the past is you know an airplane pilot is graded on whether or not they made the landing and at the end of the flight you know whether or not they made the landing I think one of our challenges is that I might read a CT scan today and we may not know for 5-10 years whether I actually did that correctly or not because it may take time for that patients disease to manifest itself and so that question is one that we are constantly struggling with and and is is part of the basic biology and the set of problems that we deal with the medicine so on that now I think we've come to the end of our presentation I want to thank you all very much for coming out this evening and for your interesting questions [Music] you

Channel: University of California Television (UCTV)

Views: 9,936

Rating: 4.9101124 out of 5

Keywords: personalized medicine, artificial intelligence, machine learning, radiology

Id: DSg9DqMW1-8

Length: 83min 10sec (4990 seconds)

Published: Mon Nov 20 2017