Deep Learning on Retinal Fundus Images, and Lessons Learned

all right hi everybody so we are almost an hour late so I'm gonna kind of really do this talk on the fly and try to use like 10 minutes and hopefully tell you stuff that's interesting and and perhaps useful in what you're doing so I kind of prepared two different chunks of the deck one is sort of an introduction to some of the innards of deep learning and I'll go through that really quickly and then I'm going to talk a lot about the work that we're doing in the retina and particular lessons that we've learned in doing this so there's a there's a saying that you know in theory there's no difference between theory and practice but in practice there is so we learned a lot by actually doing this now I'll share some of those harder and lessons so you can make different mistakes than the mistakes we made so so just real quickly on deep learning you know comparing you know sort of traditional deep learn a traditional machine learning has been around for a long time and one of the big challenges the essential way it worked was if you have a bunch of data this is supervised learning you would come up with a bunch of features that describe it and think of those features as like a vector and then you might have a label these are fraudulent transactions these aren't learn a formula on the features to make the prediction and the challenge particularly in imaging was what are the interesting features right if you're doing facial recognition you could look for eyes and nose and mouth and you know it gets very complicated and you know maybe credit card transactions it's easier and this was always one the challenges and year after year that be new PhD theses about new ways of doing features and really you know kind of jumping ahead through I mean trying to do features that would distinguish those and so so you know deep learning came and that you know I'm not going to go into some of the details one of the big powers is that it learns the features itself by the data so kind of the visualization that you're seeing here is a network that was trained I think to recognize faces and like the lower levels of the network learned edges not the way an engineer would draw those edges but it learnt edges just as we believe is happening in the neurons behind the photoreceptors you have a set of neurons that are sets the photoreceptors on there they're finding edges and they go combination and then textures and features and corners all the way up to the real feature so so this ability to detect features is really critical and I'll talk about this in a second so let me switch gears to the retina so so well actually before I do that let me talk about diabetes so this is the number of people with diabetes in the world it's probably an underestimate the current thinking is that about 10% of the people in the world will get diabetes especially as diets are improving and this is a big problem in developing world's about a third a third of diabetics get a condition called diabetic retinopathy and it's essentially it's a breakdown in the blood vessels in the eye you know diabetics have a lot of vascular problems that's why they have you know amputations and things like that and so for the third that get retinopathy for about a third of them its vision threatening so sort of 10% a third a third it's about a 1 percent prevalence disease but it's the fastest growing cause of blindness in the world and the way it is it's sort of diagnosed by sort of observation there's a there's very very detailed rating guides that the ophthalmologist argue over their Zarn spots that sort of dried plasma the leak from the blood vessel there's dark spots as hemorrhages there's a bunch of these these pathologies that can be diagnosed it's a sort of five point scale traditionally and unfortunately it's only symptomatic at the at the last scale of proliferative so if you can catch it early it's it's much easier to prevent by the time you get to be proliferative you can you can stop it but you can't fix the eye so this is if you know diabetics are supposed to get screened every year please send them in to get screening and you know in places like India and Thailand and in China there's just there's there's just not nearly enough not nearly enough doctors and it's kind of sad I give this talk and I inevitably have people come up to me and say my uncle is blind from this you know that it strikes a lot of people so we did the normal thing we've said okay let's apply deep learning we could we kind of knew that deep learning would work on this I mean you could see the features so obviously if we could pick a cat or a dog we can find exudates and hemorrhages and we did it in kind of a googly way we're able to more quite a bit of resources and we acquired over a hundred thousand images we hired over 50 doctors to give us diagnosis on these images and it worked great it worked absolutely fantastic like way better people been using traditional machine learning on the retina for many many years or some wonderful professor Abramowitz at Utah they've done some great work and the results of this were absolutely stunning there were some bookkeeping predictions that we need to do so so it worked really well and we we actually spent a lot of time to get this published in JAMA which is the journal the American Medical Association it's the journal it's really focused on the practice of medicine and so we actually had to do a clinical trial it was we really treated it as a medical thing rather than a technology thing and everybody knew this would be sort of the big deal and the results in 2016 were as good basically as good as a panel of ophthalmologists which is pretty compelling if you can bring a bunch of ophthalmologists in a box to the developing world it's kind of a big deal and you know not just in the developing world I talk to cab drivers all the time about diabetes talk to everybody about diabetes now and you know to get screened you usually have to take a day off from work and go the doctor make an appointment why can't you just get screened at the pharmacy or at the store or somewhere else so the results were spectacular and so now let me go into some of the problems so this slide we actually had t-shirts made up from this slide so let me explain what it is that the columns are the best of the ophthalmologists we hired and the rows are patient images that were selected especially because they were challenging and the color is the diagnosis that we got from that doctor so you could see that the patient's the two at the bottom are clearly sick that the one the top probably you know the two or three at the top are probably healthy but we got this rainbow of diagnoses and the ones in black they got actually every single diagnosis and so we you know we tracked that and these were the best of the doctors so we tracked the correlation the correlation of cross graders and even the correlation within the greater so we gave the same doctor the same image in a different week and it was only you know two thirds that they would give us the same diagnosis so this this is crazy and so it's funny I put this slide up and rooms full of doctors they look at me and say we're humans what do you expect and I look you know rooms full of engineers like we must fix this this is not OK you know the end thing is that retinopathy is a slow-moving disease so I showed this to some pathologists and they were like these are at the at MD Anderson that one of the best cancer centers in the US and they were like we never get 65% for us it's more like 30% and kind of my jaw hit the floor and they're like well it's not really that bad because pathology grading is very technical and there are more grades than there are treatments in terms of treatment concordance it's it's closer to 85% so what that means is one time and six you're going to be treated differently if a single pathologist looks at your data then if a group of pathologists look at your data so if you take one thing away from this talk it is get a second opinion on your pathology I'm not kidding I mean I give this talk and I you know this is not you know a rural hospital problem this is one of the the lead of one of the top research labs in the country came up to me after this talk and said you know my wife's best friend she had a biopsy they did a double mastectomy they don't they always redo the pathology after they removed tissue from your body she never had cancer there's a you know we're making a lot of progress in breast cancer because we can identify different subgroups of patients that have very different treatment so if you're her to positive breast cancer which my mother was and she got her Septon which is the miracle drug total remission they think about 10% of women are being mistreated around Herceptin so they're either getting it and it's not going to help them or even more tragically they have her2 positive breast cancer the pathologists misread the slide it took billions of dollars years of work to there's a miracle drug on the shelf and she's not getting it because the the pathologist missed it at the last minute so please get a second opinion so so what Google had done you know we had gotten lots and lots of images lots and lots of diagnosis and what we found typically wages we got if it was hell if two doctors thought it was healthy we stopped labeling it if anybody thought it was sick we got seven different diagnoses on it and so then afterwards we were able to do a sensitivity analysis did we need all of those diagnoses and you know we just had some consensus way and it turns out that for the training data we massively overkilled it around to two and a half diagnosis was more than enough but for the test data 7 wasn't enough and you know we never trained on the test data the test data just tells you which of the million-mile models you can make is the right one to pick so we actually started curating the test set and we we hired panels of retinal specialists and we we literally have adjudication panels where it used to be we brought them in a room in Mountain View and they argued over images now we have a whole distributed system and in about 18 months we went from we were as good as an ophthalmologist to now we are as good as a team of retinal specialists and the only we did not change the machine learning this is inception version four like the thing that one imagenet years ago this is like out-of-the-box machine learning it was all in the data so another lesson to take away is do not be afraid of dirty training data but try to exquisitely curate your test set and he was describing some of this this the test set is absolutely critical but but don't be afraid of dirty training data use it but you you know though really the kind of level of accuracy that you need is only going to come from curating the test set and you know just as an example you know I talked about some of these you know the AUC curves you know I would love to set up screening in train stations in India because a lot of people have diabetes and they don't even know it yet so if we could catch them the problem is if it's a 1 percent prevalence disease and we're only 99% accurate every other case is going to be a false positive or going to flood the the you know the medical system so you know 90% is 80% is tolerable 90% is great like 99% is it has even got issues so we have to keep making these systems better and better so so you know we kind of knew that we would be able to do diabetic retinopathy it was a well understood disease but then we started saying well you know the the retina is the place in the body where you can very inexpensively visualize both the nerve the vasculature and the neurons and we said what other signals might be there and there's kind of a funny story behind this we had these images we didn't have medical records but we did know male or female and there was a young woman who had recently graduated who wanted to you know do work on this project she had a different job inside Google and we said well we know male female see if you can predict it and it's not going to work but it'll be a good exercise for you and she came back about two or three weeks later and said yeah I can predict whether someone is male or female from their retina and there is nothing in the literature that says male and female retinas are different we're now at about 97% accuracy so and what's interesting about this one is we haven't been able to explain it it does not localize to any one part in the image and we checked all the obvious about size and things like that we could even cut these images into 64 by 64 boxes and scramble them and we don't quite get 97% but we still get a pretty good accuracy it's mostly around around the macula and the optic disk but we still haven't quite figured it out yet but this is one of those things you know it's come up about explain ability and this is an interesting challenge so in the case of diabetic retinopathy we can explain that we can show which parts of the image contribute to the prediction but for something like like this we really still can't explain it and so you know if empirically it works it's it's tempting but if you can't explain it it's it's it's unfortunate so this is the explained ability is a big deal so what about how old you can tell who you are well we can tell within about 3.2 3.2 6 years whether you're a smoker nah not quite as effective and we actually have a nature paper out where we can essentially tell your blood pressure we can tell your h1 sea level we can we can tell all sorts of things and what we're looking for now is can we tell neurodegenerative disease can we tell whether you're anemic or not as well as glaucoma AMD so you know one of my hopes is that you know right now you go the doctor and they take your blood pressure and they weigh you and they take your temperature that they'll just take a retinal picture and you'll get a whole health assessment we can actually predict your risk having a significant cardiovascular event as effectively as the Framingham score which is essentially the the point system that they use to treat heart disease now so it's it's pretty exciting and you know this is published in Nature so all this stuff is available and so I just want to show you just some example images I mean I've been showing you pretty images but these are what you know this one doesn't look too bad and you know this one you can see there's sort of a consistency to them but you know the actual images where is this showing the actual images that we get start getting you know looking weirder and weirder and weirder and this is where but you have to deal with this in the clinical context like we want to be screaming and sometimes this is what the images look like you know and they get weirder and weirder still so you know to get sort of you know very very accurate images you have to work under a lot of these conditions the other thing that is actually a big in sort of open area is how confident are you of your actual prediction because it's not enough to just say okay we you know we think you know with you have diabetes or not but but how confident is that prediction because you might want to act differently and that's the whole other science again about and there's all sorts of interesting ideas about like dividing your data up into 50 different sections building 50 different models and then trying to understand the variants across all the models and the high variance predictions are the ones that you're going to be less confident over whereas the low variance ones you will be because however you build the model you make it so it's a really exciting area I'm glad to sort of answer questions or anything sorry it was so rushed but thank you very much [Applause] so we work very closely with deep mind so there's an interesting story about this so deep mind was mostly focusing on Oct which gives you sort of a 3d picture of the retina and where we actually have a European regulatory approval and we're screening now live in India and Thailand and one of the things that we discovered is one of the Thai clinics also had OCT and there's a pathology that comes along with diabetic retinopathy called diabetic macular oedema it's the swelling of the macula which is sort of the center of your vision and doctors thought that they could diagnose DME from just looking at the fundus image and we finally got pair training data from the Oct the 3d depth that we could tell whether you have macular edema edema means swelling so we got ground truth and it turns out that doctors are terrible at diagnosing DME they massively over diagnose it so we're able to train models that can can do DME extremely accurately and what we were able to do using gans actually was to be able to build networks that could basically add or subtract DME from any one patient's image and we're able to update we have a paper coming out on this update the grading rubric that says well if you look at this set of images give it a two and this set give it a three so the the combination of this is you know the ability to use expensive imaging to then make predictions off of cheaper imaging is very exciting cuz you know lung cancer is going to be a gigantic problem especially in the developing world is you know the rate of people dying of communicable diseases is going way down and then the they're dying of the same things that people die in Western countries of cancer and heart disease and lung cancer we still don't have good treatments for the trick to lung cancer is to catch it early and there's some great studies and great work we have great stuff published on this if you can detect lung cancer early and cut it out you have a good chance flow so can we link CTS and you know can we make predictions from CTS and then link CTS and x-ray so it's a very exciting area but Google we pulled all the health groups together under one umbrella so you'll see a much more consistent messaging there used to be about five of us doing this research and now now we're all working together so yeah you know we we are very open about sharing some of the stuff the data that we got off often has strings attached to it so we would actually love to produce a data set like this and share it and we've we've shared data sets in a lot of other areas and we've benefited from data sets that other people have shared so we're not quite there yet on the retinal images but as we're doing more and more of these screening programs the government's in in different areas are actually quite interested in doing that stuff like the UK biobank which is just if you're interested in medical imaging I really encourage you to look at the UK biobank it's an absolutely tremendous resource and there's another one coming online and in Southeast Asia and there's a one that the NIH has so we would love to do this but the regulatory environment is is quite difficult but if you go to G dot Co slash brain slash healthcare you'll see all of our research papers on this so so the question is how many images did we use and how much computation did it take so we were we were actually very lucky and that we found a source of lots and lots of images so in this case we used 130,000 images which is a unrealistic number for most medical applications and again we were it was early and deep learning and so we were very aggressive about throwing a lot of data we're getting very very good results on on orders of magnitude less data now we did use lots of computation because we want to try everything and it turned out that just using Inception and a little bit of hyper parameter optimization you know the crazy networks weren't really any better so we figured we would just write along with the existing networks but yeah I have to say that computational power is quite useful in doing this because we don't really know how to optimize these networks people can think around say well I think this is better I think this is better and the ability to just try everything is is absolutely fantastic and like you can sort of rationalize it after the fact that this worked but being able to try everything and especially trying different sorts of cross validation what one of the things that we run into all the time is that you lock in on confounding effects so for example we had a bunch of MRIs and we said well just as a as a test let's see if we can predict gender and it was you know 0.9999 and so it's like oh you know men and women's brains are different so someone had that hobbyist thought of saying what if we cut the brain out of the image and just learn on the brain and then just learn on the image with the brain cut out and with the image with the brain cut out was still in point nine nine nine accuracy because it was looking at facial features and things like that and not at the brain and if you just did it on the brain section it was more like point eight five and the point of this is you just have to be really careful about finding confounding factors so we work a lot in microscopy data and we can always tell the batch the data came from we could tell the lab it came from we could if there's two different operators doing the experiments we could usually tell the operator so for practitioners one thing I recommend is try to predict everything that you know and if you could predict something that shouldn't matter like the batch or the phase of the moon or whatever you know maybe you've got signal maybe you're fooling yourself that you're not really seeing what you think you're doing it's very easy to treat these systems as magic there's another great example of this where I'm we had this thing was called deep dream you can google it where you could say okay I think that looks like a dog let me change the image to look more doglike and there was a whole set of articles about like these are what machine learning is dreaming of and one really interesting thing is when it's thought it's saw a barbell and started materializing the barbells the barbells had arms attached to them and it was crazy it was really freaky to look at it because you know we trained it on barbells and barbells often had arms attached so it's very easy to anthropomorphize what these systems are doing and they're like oh they're seeing what I'm seeing it's like no they're not they're seeing pixels and if the pixels that's arms attached they haves arms attached so it isn't like oh this is metal and this is flesh and you have some higher object representation see have to be extremely careful about looking at these strange con founders and that's where the the the computing power comes in to just keep trying things and trying things but you know there's a new generation of hardware coming out both from Nvidia and Google and Amazon will make it available there are CPUs and GPUs are faster yet there's a new generation of computing power there we call them TP use tensor processing units and again they're not magic it turns out that if you're just doing training and walking the gradient you can do much lower precision arithmetic so you can do 16-bit floats so if you're doing lower precision arithmetic because he again you're just walking up or down you're making it bigger or smaller you can do incredible amounts of processing so the hardware that's coming online and every phone is going to have an amputation trip chip the same way so there's this whole hardware revolution going on behind the scenes to back this one of the exciting things about deep learning is that it might take an acre of computers to train a model but you can impute it the model itself is very small you can run it on a small device which is great for phones it's great for medical devices it's great for security too there's a big push now you know my mother is old and I would love to have something in her house that would alert me if she's in trouble but she's not going to put a camera and a microphone streaming to the to the web in her living room but if we could put the models on the device so that device could recognize it and then just you know send out they alert so you're gonna see a big push towards remote learning models because it's just more more secure and more privacy protecting so again we're in the middle of a revolution here yes oh yeah there's a fantastic research team here yeah oh please get in touch with them there's amazing people here they're doing tremendous research there was one other question over there excuse me Oh G Co slash brain slash healthcare and another one to go to is G Co slash research slash GA s gasp and that's that's actually my team and we do a lot of we do a lot of energy and climate and low level science protein evolution computational chemistry you know simulation machine learning and simulation are really interesting couples because we're learning that we can train models to predict the results of very expensive simulations so you run the expensive simulation you train a model and then the model can you know we can calculate the DFT calculations quantum properties of molecules we can run PD es so like two thirds of the world's supercomputing budget it turns out that there's there's repeating patterns in that data that machines can learn so we're in the middle of a revolution so thank you thank you

Channel: TAUVOD

Views: 4,309

Keywords: אוניברסיטת תל אביב, Tel Aviv University, Yuval Neeman Workshop, Science, Technology, Security, Blavatnik Interdisciplinary Cyber Research Center, Cyber Week 2019, cyberweek AI: Present & Future, cyber, cybersecurity

Id: ViSfhPE6q6Q

Length: 24min 53sec (1493 seconds)

Published: Tue Jul 02 2019