Artificial Intelligence: Hype, Reality and Future Implications for Diagnostic Imaging

so it's really a pleasure for me to introduce today's speaker who is to say he's an internationally recognized experts in digital imaging would be an understatement dr. Siegel is currently at the University of Maryland where he's the professor and the vice chair for Imaging Research technologies as well as being I think the head of the VA department of radiology and nuclear medicine and Elliott was educated at he has something in common with our chairman he was Ella gated educated as an undergraduate at the University of Maryland and at biomedical engineering and in visual perception I just found out which sort of and he did some graduate work in visual perception which probably got him interested in radiology and he did his resident internship and residency at Maryland he's boarded both the nuclear medicine and diagnostic radiology he also has a holds adjunct appointments in the department of biomedical engineering and computer science which fits nicely with what his career accomplishments have been over the years some of his major accomplishments have been that he was he initiated the first film lestrade e ology department at the VA is that correct in the United States in the world that's slightly bigger than Maryland but not much right and he'll consented yes and he's well traveled he's published extensively in the area of PACs and digital imaging and now artificial intelligence he's published three hundred papers a combination of three hundred papers and chapters he's given many many lectures around the world about topics including PACs and the impact of some of the things we've heard today like machine learning artificial intelligence and advances in sort of what the digital age can bring to medicine he's won numerous numerous awards both for teaching and research his institutions it's very active in the both the radiology societies as well as the computer societies by and other sort of technical imaging societies and you know when people mention deep deep machine learning or artificial intelligence and organizing conferences name his name always comes up and today he's chosen a very provocative title and hopefully the answer to one of the things he's going to talk about will be positive at the end and he's going to talk to us today about artificial intelligence height reality and the implications for long-term implications for diagnostic radiology so I'm going to ask you to join me in welcoming Elliott to UT Southwestern and we look forward to hearing from him on this very interesting topic [Applause] [Music] so thanks so much for the really nice introduction and for the opportunity to be here for research I've learned a tremendous amount and it's really fun for me to have the opportunity to hear all the cool research as you guys are doing and hear about the facilities and government's given me which is incredibly warm welcome I really do appreciate it what you're listening to was composed by Google my and so this is actually the answer the question is can a computer the art or right NuVision answers the answers yes as far as disclosures I've seen a number of disclosure pages that were empty and so I just wanted to say if there's anybody in the audience that has any research sending for us and wants to be added to the list of disclosures be super happy to to talk to you afterwards thanks also for the really nice introduction one of my favorite introductions of all time was actually my unwitting introduction by my own then eight-year-old son and he'd is 't 'add hospital we just opened it up it was all digital and we decided that what we were going to do was to have the open up with a completely filmless in digital department the first in the world brought my eight-year-old son there my son ended up saying you know coming by and visiting I showed him a rotating 3d head and a chest radiograph and a couple of other things and wasn't really sure whether he was even paying all that much attention it's kind of an eight year old and a few weeks later I walked up behind him and a couple of his friends who were playing video games Sega Genesis at the time and one of his friends and they didn't know that I was standing behind them one of his friends said my dad's got like a really cool job because he gets to build skyscrapers which is like really amazing and I'm thinking yeah that does sound like a cool job and then Stevie's other friend says well actually my dad's got the best job in the world because he works in a restaurant he gets to eat all the dessert he wants anytime he wants to which actually really did sound real good to me too and much to my surprise my son said well actually I had been to my dad's work and he's got the best job in the whole world so I'm thinking is Steve going to say dad's in charge of the radiology department or dad's a doctor so Steve says well I've been to my dad's um hospital and he's got the best job in the world because he gets to play video games all day and what I want to talk about a little bit is you know where we are sort of with machine learning and AI and where we're going in the future but what I want to start out with is there's a really interesting trend and we're seeing it more and more all the time and I get emails from all over the world and I just wanted to share a couple from whether they're medical students or residents or fellows or physician radiologists and practice and here's a couple of them this first one comes from jowl auro who says he's a first-year resident in radiology in Portugal and he's asking the question you know should he essentially even stay in diagnostic radiology because of all of the things that he's hearing about artificial intelligence replacing radiologists and certainly our medical students are being told increasingly by their mentors at University of Maryland that you know you might want to not go into radiology because it might not be around in three four five ten years and here's another letter from Andrea who's a Italian second year resident in radiology in Italy and she says that she's seen in the last few months a quote unquote huge number of articles or discussions about machine learning and she says people look at her like she's a fool and make fun of her but she doesn't really think she's overreacting should she switched to interventional to ensure a long career andrea asks if you read any of the magazines lately whether it's time or the Economist or anything you're hearing about artificial intelligence and just last summer there was the addition on march of the machines that talked cool things one thing it talked about is what would happen if our official intelligence did all of our jobs how would it cut the economy work how would we get paid how would economies work in an environment where nobody goes to work and that seems a little bit you know futuristic but these are things that people are actually writing about and in every one of these articles when they pick out the one job that they think they want to mention that is the most easily replaceable amazingly and horrifyingly it's the radiologist and so you know why why exactly is and so in the economist and others they're talking about the fourth Industrial Revolution they fund cyber-physical systems if you look back in the early 1800s about the Industrial Revolution in England this is a quote the substitution of machinery for machine labor may render the population redundant this is back in the early 1800s in England the discovery of this mighty power has come before we knew how to employ it rightly and all of a sudden with each one of these revolutions you know there's the concern about what's going to happen with replacing humans Andrew Nick who some of you may know of if you want to learn about machine learning he's got an online course at Stanford that you can get on YouTube and see seated to get on their site and he does a great job with the machine learning but he like others have said a highly trained and specialized radiologist may now be in greater danger of being replaced by a machine than his own executive assistant in Ezekiel Emanuel who many of you I may know of is kind of the architect of Obamacare essentially at the Wharton School at the University of Pennsylvania he's written and he actually was at the American College of Radiology about a year ago gave the keynote talk on why on the biggest threats to radiology and of course the top threat seemed to be artificial intelligence he also wrote Co wrote an article in the JCR and New England Journal essentially implying that in the next three four or five years there would be major developments that may end up replacing radiologists and that radiologist may be replaced by computers in the next year this is one of my favorite if I'm looking for trash talk about radiologist Jeffrey Hampton who's known as the father of machine learning he's a professor at the University of Toronto what he has essentially said and here's the quote he says I think you should that if you work as a radiologist you're like Wiley coyote in the cartoon you're already at the edge of the cliff but you haven't yet looked down and there's no ground underneath your feet and essentially what he says is it's just completely obvious that in five years deep learning is going to do better than radiologists might be ten years he concedes and his actual words in a recent talk where they should stop training radiologists now and so what I want to do is I want to talk about machine learning AI what it really is how incredibly silly these statements are that are coming by these seeming experts why I think they're incredibly silly and why I think the focus on machine learning and interpreting radiology images is absolutely the wrong focus and what's the right focus ought to be and how we can actually take advantage of the the technology oh and then this is one of my other favorites a CEO of a very well-known startup in Silicon Valley essentially said that he wanted to quote-unquote get rid of the wasted protoplasm sitting in front of the workstation that was the radiologist and replace it with a much better and reliable and consistent alternative in the next few months he said that a couple years ago and he's now actually out of a job and you know I'm not sure the company is moving towards speech recognition rather than a image analysis and this is another one of my favorites this was in the early 1900s this was the guy that was supposed to replace doctors in 1904 so this is not the first time we doctors were going to be replaced this was called the vibratory doctor and it says vigorous strength I'm not sure why the vibratory doctor isn't wearing a shirt exactly and I'm not sure exactly what the vibratory doctor does but you know he was definitely going to replace doctors back in 1904 not sure exactly how but again it's not the first time that that we were going to be related and if you look to Hollywood this is a classic really go to artificial intelligence movie metropolis how many of you recognize a how 2001 a Space Odyssey we're sort of how is the onboard computer and kind of goes awry and sort of takes over over control this mission is too important to me to remind you to represent you however talking about now this conversation can serve no purpose and of course side you may have seen ex machina where at the end the artificial intelligence entity that decides that she sort of wants her own independent and of course the iconic got terminator me you know so sometimes computers aren't evil but just really really cool so let's talk a little bit about some of the ADCs and some of the confusion between what is AI and machine learning etc so you can think of deep learning as falling inside of machine learning as falling inside of artificial intelligence and what is artificial intelligence so it's kind of a umbrella term for a variety of different applications and techniques and so essentially it's been defined as a broad set of methods algorithms and technologies that make software smart in a way that seems human-like to an outside observer John McCarthy back in 1956 is credited with the first use of the term artificial intelligence one of the fascinating things that he complained about was that as soon as something works they don't call it artificial intelligence anymore is a calculator buy this for $1 get it for a dollar on Amazon is the number 2017 a prime number or not well you can find that out in less than a second using a dollar calculator essentially is that an application of artificial intelligence or is that just essentially on something that represents something outside of artificial intelligence and how exactly do we define AI and is it very very narrow applications and so in general artificial intelligence today is there are three types of artificial intelligence and the one that we have today and will for this foreseeable future is narrow artificial intelligence it's also known as weak artificial intelligence so it's like there's an AI program that can beat the world's best chess champion but it's the only thing that it does in fact you can go to Walmart and buy you can your smartphone can beat the majority of grandmasters and you can go to Walmart and download a free PC program that will beat the world chess champion by far today and but that program won't be able to play on tic-tac-toe and so these applications are narrow speech recognition is another example translation from one language to another self-driving cars Siri Alexa they're all very narrow applications there's also artificial general and gents and this is sometimes referred to a strong AI or human-level AI I would suggest that to replace a radiologist we're going to have to essentially strong AI or human-level AI that's when the computers are smart as a human across the board a much harder task and it's not even clear that it's possible to do and there's been a lot of speculation recently about whether or not it's even achievable and it really gets at the question of what is general intelligence Linda gafford's in suggested that characteristics of general AI would be the ability to reason plan solve problems think abstract ly comprehend complex ideas learn quickly learn from experience maybe be able to write a creative short story for example will general AI arrive and if so when well there are a number of conferences on enthusiasts and experts on artificial intelligence 42% believe that general AI will arrive by the year 2030 I certainly do not believe that 25% believe by the year 2050 20% by the year 2100 10% after the year 2100 or so certainly nobody believes it's going to be here in the next four or five years machine learning is a very much misunderstood concept it covers it's sort of a blanket term that covers many different technologies a lot of people if you hear a lot of lectures at the AC R or R sna or other places there's the implication that there's something magical or different about machine learning and then machines have some general understanding and learn as time goes on but it really doesn't refer to learning per se it's another technique and you can think of it as a different class of statistical techniques that can characterize discover or classify data most of these machine learning techniques have been around for decades some 20 30 40 50 years there's not a lot of new math in the machine learning that we're applying you'd think that it all just emerged in the last three years but these techniques including convolutional neural networks and deep networks have been around many many years it's just we're finally discovering they're finally being essentially talked about and we have the graphical processing capability because of advances in video card technology to be able to make things faster but not necessarily better or different as part of artificial intelligence it refers to a wide variety of different algorithms and methodologies and essentially in general machine learning is about recognizing trends from data or recognizing categories that data fit so we'll talk about that in a little bit more so when you hear about machine learning there are machine learning regression techniques neural network support vector machine decision trees Bayesian belief networks literally hundreds of different techniques and each of these techniques can be fine-tuned with parameters and so there's a variety of machine learning techniques and so someone says they've applied machine learning it doesn't really tell you much more differently than if they told you that they applied a statistical technique that don't tell you which one machine learning focus is on prediction based on known properties learned from a training data set can you predict things about a another data set if the two are related to each other data mining unlike machine learning focuses more on discovery of unknown properties in the data and so leo breiman makes the distinction between statistics and machine learning and essentially machine learning can be thought of as a data model or algorithmic model and some statisticians have adopted methods that sort of a combined field between machine learning and statistics called statistical learning so when I started out my residency at University of Maryland because I also was a computer science major and had done a fair amount the Chairman then asked me to give something that was really super scary to me as a first-year resident he wanted me to create a computer program to make the schedules for the entire residency rotation and for all the faculty that created all sorts of interesting political challenges for me as the schedule maker but I also had to figure out how do i how do I write a program to create the schedule there's no mathematical formula to be able to do that and to solve the linear algebra associated with a complex schedule especially where there were rules like resident a doesn't like to be on the same rotation as resident C and resident be an attending number one don't really work together and resident F is going to be out for maternity leave how do you create a mathematical way to write all those well what I ended up doing is I ended up on creating random schedules so it would just come up with a completely random schedule and then I wrote a program to evaluate whether the schedule actually met the criteria well it didn't come up right in the first time but after running for days with quadrillions of different schedules it actually came up with some really good schedules and when I submitted those with a tiny bit of fine-tuning the chief residents and other people were really amazed at how well the computer did against any human schedulers was the computer smart no it was incredibly dumb but did it solve the problem now I could have refined that technique and taken some tentative programs or schedules that were closer to what we wanted and then iterate from there just by than doing it randomly and so you know this is an example of an era t'v algorithmic approach to being able to solve a problem rather than solving it through traditional statistical or other types of methods so if I gave a five question survey to 10,000 people can you tell me if the person responding to the survey is male or female and we ask them silly stereotypic questions is your favorite color pink or blue what would you rather watch on TV Dancing with the Stars or NBA basketball which movie would you rather watch tonight Star Wars or the notebook and and a variety of other questions you know none of these questions is going to discriminate 100% males and females and then you have to start wondering like how would we solve this statistically what has the highest correlation maybe you'd say the highest correlation would be favorite color pink or blue who knows now the next thing we could do with a statistical analysis is we could essentially create a correlation what has the highest correlation then we could do a regret and formula and say here's the relative weighting of each one of those factors and it could be essentially a nonlinear multi regression formula but there are all sorts of insights and exceptions maybe taller women under two hundred pounds who prefer blue over pink prefer the NDA but only if they prefer Star Wars I mean there might be all sorts of interesting observations about the data set how do we with traditional statistical techniques take all of that into account the answer is we can but with machine learning what we can do is we can try and create and derive a simple formula based on those empirical observations from the data set for the first 500 and predict the next 500 so essentially it's a mechanism that's highly related to solving linear algebra equations from complex data using an iterative almost experimental technique where we take the answer we check to see whether or not the variables as they're weighted work and variables may have complex relationships with other variables and so you can have deeper and deeper networks and it turns out that machine learning practically invariably will beat out any statistical technique for analyzing data the data we're talking about are really simple data they're just like two dimensional or five dimensional data but we can analyze much more complex data so here we have smooth versus speculated lung nodules and nodule size and whether or not the nodules represent cancer or not there's no line that you can draw in this diagram on the left that will separate out the benign green lesions from the malignant red lesions but what you can do with the neural network is you can essentially solve for that particular data set with a set of equations and you want to make them as simple as possible using a machine learning approach that discovers what that relationship is what you want to do though is have enough observations so that a training set is generalizable to another set so you don't want just accidental on variations which is why with machine learning you need a lot of data to be able to do it here's another example let's assume pathetically that this is the ratio of what makes a pig beautiful the ratio between the eyes and nose and the ratio between the nose height to nose width so here we've got beautiful pigs and less beautiful pig and as it turns out in this particular case this is the distribution the beautiful pigs are all the red dots and the less beautiful ones are the red are the blue dots is there a formula that you can derive is there a line that we can dry derive or a curve or a statistical technique that allows us to tell the difference well those of you who are looking at it recognize this is a heart shape and as it turns out there's a simple formula for a heart shape x squared plus y squared minus 1 that quantity cubed minus X square times y cube it turns out that a machine learning algorithm could take that pattern and easily and quickly derive this fairly simple formula and without essentially using statistical techniques or solving linear algebra and so it turns out machine learning does a really good job at being able to characterize data now you might say well isn't the data so complex that the formula would have to be really hugely complex but the answer is as this paper from MIT from August of 2016 suggests is that biologic and physics data and astronomy data tend to have relatively simple equations and that's my machine learning work so relatively well for predicting biologics phenomena and associations can we apply machine learning to images and the answer is yes but but images are really hard many of you are familiar with the classic test for artificial intelligence which was called the Turing test he proposed in 1950 an experiment that today would be the equivalent of you're getting a text message from something and you don't know whether it's a human or computer if a computer can fool you even though you can text and ask it questions into thinking it's a human or you're not sure then it's passed the Turing test meaning that it is is artificial intelligence well the Turing test has actually been defeated in the last couple years and so computers are able to fool a panel of expert judges into not being sure whether it's a computer or machine in 2011 there was an article in Scientific American that said instead of the Turing test an even harder test for computers is something like this highlights magazine picture so if you show this to a five-year-old a five-year-old will say oh that's really funny there's a TV on the roof that shouldn't be there or the fork and spoon are really big there's no computer program or algorithm that's anywhere close to being able to beat to beat a five-year-old at highlights magazine today no one has even come close you can teach a computer to find a fork on a roof but you can't teach a computer to generalize and realize what's wrong with this particular picture so fortunately for we in diagnostic imaging diagnostic imaging actually despite the fact that we're the first one thrown to the Wolves as far as what gets replaced in an era of machine learning and AI we actually have the very hardest task of all and so in June 2011 these authors essentially proposed something that would have to do with interpretation of images should actually be the ultimate test for AI or for consciousness and so I think imaging may be the ultimate future or frontier for AI software so let's talk a little bit about imaging it turns out that the same machine learning that we're talking about applying to five variables or ten or two they can be applied to an image itself and in doing that we talked a little bit about a specific type of machine learning algorithm called neural networks or artificial neural networks and in many ways they can be thought of and they were originally pattern from neural activity in the brain so you have one neuron that's able to communicate with other neurons that in turn can communicate with other neurons and so you have essentially an input that might be those five categories in that survey we talked about and an output is it male or female and then you can have a of layers this is a simple three layer neural network pretty much as simple as it gets but you can have incredibly deep and complex networks where every variable can essentially talk to every other variable in every other layer which can in turn talk to other variables so it turns out that essentially you can create all sorts of variants at that neural network and one that's particularly interesting that was mentioned in one of the research papers was convolutional neural networks where the weights are learned predominantly from neurons that are closer to you and so what that means is that pixels that are close and adjacent have a greater weight than ones that are farther apart in this particular type of convolutional neural network and so in general what happens is you can take a complex image and you can essentially detect patterns in that image and once you've detected those patterns then you can use other machine learning or statistical techniques like support vector machine to be able to analyze the pattern so we've already seen some examples of this where what it turns out you can do is let's say you have an image of 240 by 240 pictures maybe it's a dog or a cat or it's a car or it's a bicycle when there's 240 by 240 pixels which by medical standards of course is a relatively small matrix that's 57600 variables it turns out that for every pixel and you can have a variable that represents that particular pixel and if you put it into a convolutional neural network it turns out that what starts emerging magically and really amazingly our patterns and so a line represents a pattern a curve represents a pattern and then you start seeing features like a face emerge for example and so as you get to more and more levels of depths of a network the patterns become more and more complex this is just a human sort of derived way to think about it but the computer doesn't actually see eyes or noses the computer just sees increasingly interesting statistical patterns of information that correspond roughly to we see with our eye as shapes and curves and and other things so deep learning consists of multiple hidden layers in an artificial neural network it's been around for a long time but the biggest problem has been as you get more and more layers to process and get slower and slower so despite the theoretical idea of neural networks when it was first came out 20 30 years ago essentially it was too slow to be practical and in the two biggest applications today as we have faster and faster processing or computer vision and speech recognition so if machine learnings been around for more than 50 years why all the excitement currently well this image net challenge that was essentially issued by Stanford started in the year 2010 and at that point the level of accuracy in 2007 eight nine ten and eleven was around twenty thirty percent or so for this task of telling the difference between a chair and a cat and a car and a dog and a person but if you take a look all of a sudden in like the year 2012 it took a major leap forward and now in 2016 2017 accuracy rates are well into the upper 90s and so essentially if you take a look at the error rate it's gone all the way from about 26% in 2011 and the winner in 2015 inception b3 had an error rate that was lower even than humans as far as doing this classification when you do it quickly and so what happened well there was an incredible revolution that happened in thousand twelve and there was an entry from the University of Toronto called super vision and that super vision algorithm consisted of six hundred fifty thousand neurons arranged in five convolutional layers with about sixty million parameters and so what they were able to do is take advantage of the latest version of video cards back in 2012 and created a cluster of those video cards and had them be able to do tasks that would have taken weeks to do in just a relatively small number seconds.that has created an incredible explosion in human vision where the ability to be able to classify objects like this red fox or hen or Goldfinch or retriever are incredibly high and this has led to folks who do human vision like Tintin like and like others to say well if we can do human vision really well medical images must be like really easy to write we'll just take our same tools and techniques and we'll apply them to medical images will put radiologists out of business and so I think a lot of this has come with major advances in human vision which people assume are going to apply to radiology the fundamental flaw of this particular assumption is that these images are nowhere near as large or complex as medical images it's not at all clear that this can be extrapolated to 3d imaging modalities like CT and MRI and there are other major differences in the tasks we're not just looking to say is there a liver in this CT image we're looking to characterize on what is in the liver itself the shape of the liver the morphology whether there's different pathology to past that we have is fundamentally different than the task in image note the other issue that we have is we're still struggling with how do you test human physicians and trainees how do I know that one resident is confident to practice radiology at the end of their training much less a computer if we had a robot that appeared a Robby the radiologist from 300 years in the future and it came today and it said it reads all the radiology studies and it does a great job better than any human today in reading radiology studies how long would it take before we could test it and actually verify that it does what it claims it could do how about the FDA how long would it take to clear Robbie the robot from 300 years from the FDA would you have to test it in 20,000 different diagnoses I mean how would you know how to debug the software and how would you ever know whether it does what it's supposed to and so those are all really major challenges one of the interesting things I mentioned about machine learning algorithms and it's kind of the old joke about standards it's like oh isn't it wonderful about standards nowadays there's so many to choose from and kind of that same irony is also true for machine learning algorithms so you know the differences for diagnostic radiology is that in general they don't have experience with the same greyscale imaging that we do in in medicine they can identify a chair but they can't tell is the chair broken there's something missing from the chair is there something extra there is it a comfortable chair ugly chair dirty or clean that's a problem another major problem is there's a black box nature of convolution of convolutional neural networks so what do I mean by black box nature well if I have one of the papers that was presented last year at a conference that I ran was somebody taking patients who had a life-support line and I'm taking radiographs of those patients and having other radiographs where there was no life support line and so the question was how good is a computer at determining using a convolutional neural network with no additional training other than yes or no is there a life support line how good is it at finding a life support line the answer was about 78% which obviously is no better you know nowhere close to what any of the radiology residents would be able to do or medical students but the other question was even at 78% you don't know whether it was actually seeing the life-support line or it was just showing really crummy looking chest radiographs that looked like they should be from patients in the intensive care unit and so even though you get the answer right and you can test that there's no way to know exactly how it's coming up with its determination and so how does the FDA clear black box type of things and how does one know how it's going to perform in different circumstances the other thing is I have been issuing for the last 10 or 15 years a challenge saying that anybody who does computer science and image recognition who can find the adrenal gland better than my fifth grade I will go anywhere in the world and wash their car and I've never washed any cars so far now that may change but if there's no computer algorithm that can even find the adrenals better than maybe identifying about 68% of the pixels which a fifth grader can blow away at this point I'm not sure how anybody thinks we're going to be replaced by a computer there's also no general-purpose learning system our diagnostic radiology residents learn every time they learn a new modality it makes it easier to learn another modality every time you look at images it makes it easier because our brains generalize from one to the other in general computers are not doing that now at all and so we have very narrow applications and so you know questions like which algorithm to choose from how do we evaluate which is best for a particular problem blackbox question how do you optimize parameters computational time how do you keep it from taking a really long period of time and then other issues like high dimensional data sets where it gets harder and harder for computers so what are some applications of machine learning and medical imaging well there's lots of applications there's 10 to 20,000 or more papers that have been written about machine learning algorithms in medical image applications fracture detection brain hemorrhage mammography ms diagnosis and quantification bone age determination meniscal tear thousands of different things and as we heard today I mean more and more people are applying machine learning and convolutional neural networks so you've got tens of thousands of algorithms that have been developed over the last twenty to thirty years where are they in clinical practice sit in front of your workstation and how many of those tens of thousands of algorithms are you using today well in general the only one that's in use practical use is CAD mammography so we did a survey for mammography CAD and we asked the question of a large number of mammographer x' what percentage of you actually have are using cad mammography and the answer is 90% and why because it pays about $12 per study how many of you actually trust it what percent of people change their diagnosis based on the CAD program only about two percent what percent of radiologists actually trust and use the CAD program and its output on a regular basis a really small percentage so here we are twenty years or more after expert systems with CAD mammography were demonstrated to be as good as expert radiologists why aren't we using them and why haven't they been implemented I think part of it is that they're implemented the wrong way they're implemented as the second leader rather than a tool for us to use and when are these tens of thousands of algorithms well people will write the algorithm they'll present it at a conference but none of it is getting translated into actual clinical use and we'll talk about that so where are we with non medical applications in machine learning well the answer is there's really exciting stuff going on with non medical applications and there's the potential for incredible advancement in medical applications this is one of the coolest breakthroughs I think this is from a company in the United Kingdom called deep mind that was bought by Google a few years ago and they came up with this formula that I'm not going to go over but it's kind of a general learning formula and so what they ended up doing is they ended up playing Atari video games and they picked like 60 or 70 of these old-school Atari video games and you can see the little controller over on the right the stick goes up down left right and diagonally and you can press the little button so what they did is they got the computer to literally not learn how to play video pinball and boxing and breakout and pac-man in this pac-man but what they did is they just got it to be able to watch the score on the screen and then make random movements with the joystick and see what works and didn't work so here's deep mind learning how to play breakout so in the first 10 minutes of training what happens is it looks at the score and it just kind of randomly moves the little the little bar and the bar kind of sometimes it hits the ball sometimes it does it pretty pitiful right and after a while it starts learn hey if like the little paddle actually hits the ball then I'm gonna be able to get more of a score so this is after two hours of the system playing with itself so now all of a sudden it's like really really good have some of you who are like breakout players yourself are going to say yeah but like I got a strategy that's even better than just hitting the paddle with the ball and what I'm going to do with my strategy is I can actually do better so after four hours of the system playing itself then this is kind of where the magic happens what it learns is is that if it can break a hole in the wall really fast then the ball gets behind the back wall and it starts knocking out all the back pieces so what ends up happening is it starts essentially right from the beginning breaking a hole drilling a hole through the wall and then playing and so this is not with any training there's no training in any one of these 70 games yet it plays all 70 and it beats humans at the majority of the games and that's kind of interesting and maybe kind of scary also and so it's a very different tasks than radiology but having a system that's able to learn over time from watching what happens and experimenting is really cool so now we get to the game of go so the computer science pundits that I used to work with in general said yeah chess I don't really have that many combinations and you can kind of brute force it but now we've got this game go that has more combinations of moves and there are atoms in the universe I don't even know what to pronounce this number as but that's the number of combinations for the game go so they ended up having the computer train itself and train with many many games from masters at this on game go and then after they beat the European champion they challenged the champion of Korea and what ended up happening was essentially the Koreans were so confident that they thought were saying this is the easiest million dollars they'd ever make but what ended up happening was that the computer actually took this world's best player and one beat it at Game one where he resigned after 186 moves in Game two it made a move that was so strange by human standards after moved 37 that Li the world's best go player literally walked out of the room and then resigned later on he went on li went on to win yay for humans game 4 with a move that was called the hand of God move but then he lost afterwards using the same strategy and so playing many many games studying masters and using a completely different non brute-force approach the team was able to to win this is my car which is completely it's an autopilot so my car drives itself it's not careening back and forth this is just a camera that I'm panning back and forth so you know but this is essentially the scariest bridge in Maryland or maybe one of the scariest in the country were about 20 stories up on the Chesapeake Bay Bridge would you trust a car to drive you over the bridge what I mean did somebody debug the software to make sure it doesn't make a hard right or a hard left what happens if there's an obstacle in the road in front of you what does it do and how do you test every possibility and so how do you debug software like autopilot software that drives you over a bridge how would you deal oft where that would do medical diagnosis and how complex would it need to be could Google AI or could AI ever be creative well the answer is yes I mean there's artwork now and there's music that's been done the other question that people ask is well the last thing that a computer could really never do is be empathetic and you they did a survey at Mayo Clinic in 2006 and they asked the question what do you want in your physician and the answer to what do you want was not I want somebody at the top of my class or the smartest person in the class or somebody that's got a bunch of degrees what they want is a physician who's confident empathetic humane personal forthright respectful and thorough and so could a computer ever essentially be amp and could a computer ever demonstrate that I'm Ellie thanks for coming in today I was created to talk to people in a safe and secure environment I'm not a therapist but I'm here to learn about people and would love to learn about you it's watch iris vigilance canister aggressions and wondrous leaning forward or backward how long he waits to respond and dozens of others read with this yes so how are you doing today I'm doing well that's good where are you from originally I'm from Los Angeles oh I'm from LA myself when was the last time you felt really happy when was the last time so can an AI program the empathetic I mean that's a lot more empathetic than a lot of radiologists I know and so I think the answer you know I'm afraid is probably yes or at least give the illusion of empathy so you know why radiologists won't be replaced is because there are tens of thousands of algorithms that are available none are here in clinical practice now and it would be amazing to think that things are going to change in a short period of time in order to replace the radiologist you'd have to find the best of these consolidate them into a package that would work independently and I think a lot of the work that we need to do is in trying to to do that and then assuming you had all these available it would be really difficult and it might take 20 or 30 years just to get FDA clearance on those many algorithms so there's an interesting article in the Harvard Business Review that makes the distinction between computers are really good at making predictions but they make the sign distinction between predictions and judgment so one can make predictions from a lung nodule database about what's benign and malignant but the judgment that's needed to be able to apply that something at this point they think it's going to be a long long time before we have judgement quote-unquote into the computers the three biggest practical challenges for machine learning are the regulatory issues related to this blackbox nature of machine learning the FDA doesn't have a good mechanism currently to be able to evaluate kind of black box type of applications nor applications that are written by other people other than the vendor that's submitting because of the clearance issues and the audit trails that are required the other is horsepower you guys are fortunate to have a tremendous amount of internal computational horsepower most places don't and so they have to do machine learning over the cloud there's a company currently that is doing machine learning for cardiac applications but it's over the cloud and not all institutions are ready to be able to take medical image data and send it out over the cloud with regard to privacy pretty HIPAA issues right now the amount of computational resources required to do real-time machine learning is incredibly high and most places can't do it and then the other issue is the challenge to create a unifying platform that allows me to deliver a subset of these tens of thousands of algorithms that have been created on my pax workstation or or any other workstation so overall I think there's incredibly exciting potential for machine learning in medicine but a lot of it I think is not with medical image analysis and replacing radiologists but in a hundred other applications there's a hundred really cool things that we can do right now with machine learning and image interpretation is only one of them looking at machine learning for readmission evaluation risk assessment population analysis patient tracking personalized medicine genomic correlations treatment planning all of those are fantastic applications that we can do right now so we have actually done some machine learning in an interesting way here you can see a nodule in the left lung posterior Lee and a patient mr. Akamai he's your next piece of sixty two-year-old native hawaiians smoker with COPD he's got a seven millimeter speck related nodule in his left lung what's the likelihood that that nodule is malignant well you might say if it was picked up in a screening study then you could look at the ACR lung rads and try and figure out and tell him well the odds that this is going to be malignant are five four or five percent tell him to wait six months or 12 months and get a follow-up depending on on some other variables and don't worry about it but mr. Akamai your next-door neighbor says that hey I've got a family too died of lung cancer and I'm a smoker and I'm essentially a Native Hawaiian what's my chance of developing malignancy and so we actually did a study where we downloaded the entire National lung screening trial database and then what we did is we essentially tailored the diagnosis with a real-time lookup of the information in the National lung screening trial so I could take mr. Akamai and I could personalize our search through the National lung screening trial which we did real time specifically to patients like mr. Akamai so that we could give him a personalized idea of what's the likelihood that his lung nodule is malignant then what we were able to do is see that when we do that the likelihood in mr. a Kamiya this being cancer might be 11% or even higher and what we found is as the size of a lung nodule increases of course the likelihood of malignancy increases but what we also found was that if you take all the other factors into consideration in a high percentage of patients we ended up recategorize the lung rads criteria either upward or downward personalizing it to the patient so in a high percentage of cases if you look at all the characteristics of the patient you actually change your follow-up time based on personalizing it to them we have the computational capability to analyze a raw data set like the National lung screening trial in a fraction of a second so why aren't we doing that why can't you take your you TSW data set locally which might be different from the National lung screening trial and for every patient who comes in look at all the it's real time and make a machine-learning prediction on that specific patients likelihood of disease based on your own analysis and so the potential to be able to apply machine learning real time to personalized medicine is incredibly exciting not only the data associated the metadata but the image data itself I can characterize each one of these lesions along hundreds of different parameters and here's just a subset of them and then I can essentially say what are what happened in the National egg screening trial database where I've downloaded all the images with nodules that look just like this particular image and then applying them and then the other question is one thing that we ignore with CAD programs and decision support programs is what's the pretest probability of a patient having cancer and so we apply Bayes theorem which computers do really well humans do a really crummy job so this is something that I posed to our medical students and residents as a test of whether or not they think in a Bayesian way and so let's assume that there's a new high accuracy urine test that just gets released the sensitivity of the test is 98% specificity is even higher it's 99% specific you're never going to find a test in radiology that does better than those two so seeing at the airport in Liberia a full 10 out of every 50,000 people that fly actually have Ebola at any given time and now we're going to apply our incredibly high sensitivity and specificity test to all the people who are flying your family member gives you a call and says they tested positive for Ebola what's the likelihood that they actually have Ebola is anybody want to take a guess at what that might be five percent so we hear five percent we hear fifty percent sometimes we hear 99 percent cause of sensitivity and specificity so high and so you know which of these is actually the right answer as to what the likelihood is of somebody testing positive and the answer is actually less than two percent and the reason for that is that as you recall if there's 99% specificity that means there's 500 false positives when you test 50,000 people even though it's 99% that 1% is 500 false positive and remember only 10 people end up having the disease and so you've got 500 false positives with 10 people having it and that works out to a 1.9 percent probability and we don't think that way and so when we're testing for lung nodule cancer or any other predictions that we make knowing the a priori probability is incredibly important so we downloaded a second data set called PLCO 155 thousand patients just like the National lung screening trial this test cost the government a quarter of a billion dollars to collect 16 years of following one hundred and fifty-five thousand patients for 40 different types of cancers and so a native Hawaiian patient like mr. Akamai has an approximately 2.8 fold greater chance of developing cancer in comparison to a Caucasian and even greater in comparison to a Hispanic patient and so a Native Hawaiian has a six-fold greater chance of developing cancer in the next five or six years just because of ethnicity well in the PLCO data set we have 800 variables on our patients so wouldn't it be cool to figure out what's the a priori probability that mr. Akamai is going to develop lung cancer even before he gets the CT scan then when we see is nodule let's compare that and factor in the likelihood that he that he is prone to developing lung cancer and then put the two together so we created another real-time learning program where we use the entire database and raw data you tell me anything you know about a patient are they taking ibuprofen aspirin do they exercise how old are they are they smoker I can tell you instantly the likelihood of them dying in the next X number of years and also on what cancers they may develop and so that's the kind of thing I'd like to see with machine learning that I think would revolutionize the way we end up practice medicine and so I think in the next few years machine learning is going to have a major impact on medicine and diagnostic imaging just not as something that ends up interpreting radiology studies so what do we want I mean in the future the most important applications will be image display and information and dictation and tracking systems with improved intelligence automatic follow-up of recommendations that are made using machine learning repetitive tasks such as finding rib fractures on a CT or lung nodules or pulmonary emboli Peter will machine learning we'll be able to do that but AI software is also going to act like a spell checker or grammar checker you know I've seen a transcription error where we said the aorta has a diameter of four point eight centimeters and what it transcribed instead of four point eight was for play well a human transcriptionist is never going to type four play into a report but the computer actually typed for play well I've never used that word in a radiology report I'm not aware that anybody else has and so what if I built in the intelligence using a machine learning algorithm that as it translates it also translates what is its likelihood prediction that it's level of confidence and then highlight that either don't transcribe subscribe for play or at least make it read because you know I've never used that before and so there are hundreds of really cool machine learning applications that will change the way we practice safety quality efficiency none of those essentially necessarily are dependent just on image recognition so what we need to have develop is a mechanism for delivery of multiple algorithms the ability to index and tag and categorize medical images and then to be able to combine so if I have ten different programs that have the capability of being able to predict likelihood of malignancy on a prostate MRI study I want to be able to take an ensemble of those five algorithms and like five humans have them vote together because what we learn is that machine learning algorithms at can work in synergy and be able to make better predictions when they're combined rather than anyone so I'm here to tell all the worthless protoplasm that are we radiologists and doctors and the rest of you human beings we can continue to ingest food reproduce create waste products without fear of being replaced by computers at least in radiology anytime soon so my advice to Joe and Andrea are please finish your residency there's an incredible amount of work to do before we have a general machine learning AI program that can learn radiology conceptually like a radiology resident there's loads of low-hanging fruit for machine learning techniques and we should be pursuing those way more aggressively now rather than wasting our time trying to replace radiologists but thinking about image processing is one of a hundred different things we can use for machine learning humans a bit around for about one minute and 17 seconds on an earth clock of 24 hours computers have been around closer to just one millisecond who knows what's going to happen with computers in the next Microsoft so I thank you all very much for the opportunity to prevent and really appreciate it and be happy to answer any questions and thanks again for making me feel so much at home [Music] [Applause] we're comment yeah yeah the question is we learned that the machine learning can do some false positive identification sure just like with any statistic a can the machine learning do false negative identifications sure yes absolutely and so I think it's just the classification error so the answer it can do false positives and false negatives equally okay so in this case the Machine the machine learning won't take the job for geologists but can make people other people being a realist can you're saying that essentially it can provide tools that would allow non radiologists to be able to practice yeah so I guess the question really is whether or not I mean what is the specialty of radiology going to evolve to what we're going to see in the next few years are lots of applications that become available to do specific things like find my lung nodule or tell me whether or not a chest radiograph is positive or negative or essentially find rib fractures but as far as a general purpose system that replaces radiologists for non radiologists we're not going to see that will we be able to have non radiologists be able to use specific algorithms that will find intracranial hemorrhage on a CT scan earlier yes but those will be really narrow applications and not to replace radiologists so I really think that the threat of having these algorithms used by non radiologists to essentially replace us even with those other humans is a really remote and far-off threat the other thing I didn't mention is that we radiologists don't give us our selves enough credit because we do so many different things as radiologists we educate we teach we make judgments we make recommendations for follow-up we cross correlate different modalities and different information computers are nowhere near able to begin to do those things and so I really see if I'm giving advice to our radiology residents which I do and medical students etc I think radiology is going to become an increasingly interesting and vibrant field as we move forward and I really don't see the threat from non-physicians when we first created the world's first film Asst radiology department we were told and some of the institutions like Mayo Jacksonville essentially wouldn't let any images outside the radiology department for fear that once we went digital and people didn't have to come down to look at the film's radiologists would be meaning rendered meaningless essentially and everybody would look at their own images and just the opposite happened you know volume in radiology to increase so I think the risk of non radiologists using these tools to replace us is a really small and I'm really optimistic about that that's a good question though so we earlier that was a fantastic talk bill there with fewer radiologists I'm not saying they'll be replaced I'll give you an example from Stanford someone asked me to look at encontramos versus chondrosarcoma he gave me tests 20 cases yeah and I didn't know why he was doing that so when I send back and he said you beat the machine I said what do you mean he said he gave to another five expert radiologists throughout the country and everybody was 85 or something and you but then I thought to myself if he builds up something like that which we have something done on the soft tissue sarcoma site if it builds something like that he can give it to anyone colleges and and right now if my practice is only reading tumors the oncologist won't come to me I mean he'll basically you know if it's benign wasn't malignant right now they mostly come you know what do you think should be biopsy or not yeah so that I think is a problem the way we're saying like our patients are so specific though if you're talking about encontramos that's fine that's one application and right now you go to a website and there's a application that just doesn't count dromos then you have to go to another site and that application may just do rib fractures and and so combining all those different things I think so I can't see even for an oncologist or or others I can't see there's nothing on the horizon even close to being able to combine all of the different things and be able to test it none of those the application showed you as a research application asking when the FDA is going to clear that application and what's required for that so it's a great point and your overall question was is it going to require fewer radiologists let's assume for example that radiologists can be twice as efficient what I've seen is when we went from film to digital impacts the first thing that happened is all my radiologists one of the images to look like film so it drove me a little crazy because when we were reading CT or MRI wanted to stack the images or Senai them me to a radiology resident that sounds ridiculous but they wanted literally it to look like film coming coming across and so as time went on what we found is that when we stack the images we could read a lot more sequences per period of time what ended up happening then is we created more sequences an hour at 640 channel scanners so what we've seen in the last few years is the proliferation of complexity of mr and CT as we've gotten more efficient where we think a pax is twice as efficient why aren't there on half as many radiologists as ten years ago we're four times more efficient than we were 20 years ago why aren't there 1/4 as many what happens is the complexity and the things that we can do become greater and greater so I think there's going to be as many or more radiologists we just want a mundane task we won't spend as much time worrying about whether or not there's loss of hype of one of the vertebral bodies on a thoracic CT because we'll have an index that shows us that and it's the same thing I mean are you a better or worse speller with a spelling checker in Microsoft Word some people would argue one way or the other way some people would say well T it teaches you constantly what the right spelling is and so I think we're going to be seeing more and more of that but I don't think we're I think we're going to be way more efficient and I think paradoxically there will be need for more radiologists too so we're running a little late so I'd like to invite our chairman up to give the closing remarks or join me again in thanking [Applause]

Channel: UT Southwestern Radiology

Views: 5,969

Rating: 4.9499998 out of 5

Keywords: Radiology, UT Southwestern, UTSW, Research Day, RAD, Medicine, Lecture, Dallas, Research, Artificial Intelligence, Future Implications, Hype, Reality, Diagnostic, Imaging, University of Maryland, Informatic Systems, IS, VA Maryland, Healthcare System

Id: _-rbRDXbeA8

Length: 67min 51sec (4071 seconds)

Published: Wed Jun 21 2017