PhD: Machine Learning for medical Image Analysis

each year Microsoft Research hosts hundreds of influential speakers from around the world including leading scientists renowned experts in technology book authors and leading academics and makes videos of these lectures freely available yes oh I'm I'm Tony acrimony gee I'm this one but the big names of the people who really you know made this work so it's dark cozy kitchen been glocca the people who are mainly driving these two projects that you know I will be talking about and of course there is a lot of you know collaboration from many other people both in here and in other institutions like at Addenbrooke's Hospital down the road like Scarlett was mentioning and also the University of Washington in Seattle so we'll be talking about the analysis of medical images in particularly in particular here were interested in radiological images many of you you know I've had the misfortune of you know broken they're breaking their arms or something like that like I did and having some extra images of this type taken this is of you know of chest x-rays of course but in many cases you know patients undergo other types of medical scans such as you know CT scans of this type or Ammar magnetic resonance scans as well and there is many other ways of looking inside your body there is you know pet specs there is young ultrasound of course there is many ways of acquiring medical images and the amazing thing is that in the past 10-15 years the harder for acquiring these images has improved enormously the resolution the quality the signal-to-noise ratio the way in which doctors can really you know peek inside you know the patient's body and trying to figure out what's wrong and what to do about it has improved enormously on the software side have to say the things haven't you know been quite as exciting in the sense that you know we have now multi-million pound or dollar worth of you know machinery to look inside the patient and then what the doctors do is the look at the pixels and eyeball what's wrong with the patient so for instance you know in a case of you know a tumor sort of patient you know you look at the patient data say mr resonance and you say yeah that looks like a tumor it looks like you know a glioblastoma tumor it looks like grade 3 basal looks like and you know then you you get a new set of images you know after a month or a week and you try to assess whether the tumor is growing or shrinking whether the radiotherapy or chemotherapy is working or not and again the difference might be subtle but the direction of that difference is really important is it shrinking by how much is it growing by how much oh it looks like you know ten percent improvement but actually a lot of other things have changed you might have you know eaten a lot more than other in other days and so that the shape of this object would have changed a lot so there is a big problem in the medical image analysis domain which is that of you know quantifying the disease not just you know assessing what type of disease it is for which you know doctors trained doctors tend to be very good at but also quantifying exactly how big it is so in the case of solid tumors there are some very well established guidelines international guidelines which all radiologists are forced to adhere to and there are complete because they really don't work so if I have a tumor of this shape for instance you know the resist guidelines they tell me that the way to measure the only way to measure this tumor is to take the longest segment wholly included within that object and report how long it is is it two centimetres 2.5 centimeters whatever it is but clearly this measurement doesn't work when the shape of this tumor is really unusual away from a spherical sort of shape also the reason why this measurement and it's not just me saying that this measurement is you know very wrong but the doctors themselves say that the reason another reason why this measurement is you know really bad is that it only lives on a 2d plane but clear the tumor can grow in 3d in any direction he wants right and depending on the type of tumor he would have you know very weird shapes you can also have disconnected regions you know called metastasis you you have all sorts of weird things happening and so doctors have got real problem is trying to assess and quantify you know the progress of the disease whether a certain drug is having an effect or not so clearly this is important both for the in clinical practice for the treatment of a patient but also in research so for instance you know I'm I don't know GlaxoSmithKline or Novartis or ever other you know big pharmaceutical company you know of and I'm now trying to test a new drug trying to figure out whether it works or not again there is no good way of doing these sort of things because there is no good way of automatically extracting quantifying information out of medical images so we're trying to fix this problem by using machine learning and why do we use machine learning because is the thing we know works best you know when you're dealing with huge amounts of data with you know pixel or voxel type of data is it clear so far and it outs or questions if at any point you want to raise your arm you know please do so so I'm gonna be focusing mostly on two projects this only two of the most recent projects who have been working on but there is a lot more that we do in our small medical team so in the first project I'm gonna be talking about how we can you know delineate actually and in 3d brain tumors of a specific type and once we have this delineation we can solve the problem of you know measuring how big it is for instance in terms of its volume also shape characteristics of this tumor also other sorts of statistics we can do it completely automatically another thing that we can do completely automatically in project two is look at trauma patients patients have been in car accidents or other types of you know accidents in their you know workplace most of these people when they get admitted to hospital they get a CT scan taken CT scan is particularly good just like x-rays is particularly good for visualizing bony structures so we can visualize the spine and automatically detect the position of the spine and also each individual vertebra and also name them I will tell you later on why this is important and you know what kind of results we achieve so let's start with the first project so on the Left we have images from the same patient this is a single patient we are visualizing the brain of this patient these images are taken sort of horizontally imagine you're halfway through the patient's brain and each of these you know images correspond to different acquisition modalities different ways of visualizing the same data if at least the same patient in exactly the same slice position but one of the very cool things about magnetic resonance is that you know the parameters of the acquisition modality can be set in very flexible ways so that you can for instance in some cases visualize more of the fluid in the brain so you can have the blood you know component show up really bright or you can have the bony component you know visualize being visualized you know very bright and so on you know this last tool is what's called DTI diffusion tensor imaging which is a very special type of magnetic resonance which allows you to reconstruct the orientation of say the white fiber tracts in the brain so the brain as you probably know is made of white matter and gray matter the gray matter is all you know very uniform and anisotropic sorry isotropic well instead the white matter is the opposite is very directional so there are certain fibers they connect you know in a long range or long distance certain parts of the brain to certain others and so DTI is particularly good at visualizing the the structure of these white matter tracks now what's interesting is that whatever type of em are channels we have and you know however in number they can be they can be you know six or ten or just one it doesn't matter what we want to do is we want to achieve this so we want to achieve a semantic what we call a semantic segmentation segmentation means delineation the same thing this is visualized in 2d because it's easier by imagine all of this done in 3d in the original coordinate system of the patient and so the color coding the colors themselves don't mean anything this the the fact that they are different is what's important so in blue we have you know the healthy brain in red we have you know the actively proliferating parts of the tumor so that's the most aggressive part so those are cells which have gone mad and they're proliferating very quickly and the tumor is growing ingredient we have the already dead part of the brain where the the the tumor has been in previous weeks and as left you know devastation there is not much you can do about that and in yellow we have the edema edema means inflammation so that's mostly healthy brain but it's swollen because something you know it's been compressed by the growth of the tumor there might be some - Miral infiltration in the yellow part as well so it's very cool if we can get all of this done and all of this done automatically because then it means that you know I can measure straight away without any human intervention how big say the active cells region is and so if from one week to the next I see a big difference then it means that my you know drugs are working or not working the fact that I can differentiate between the red the green and the yellow parts is extremely important as well because there was a pharmaceutical company I actually don't know which one it was which was trying you know a new type of chemotherapy for this type of tumor and the way they were measuring the effectiveness of the drug was by me in the overall volume of the tumor so without making any differentiation between the different tissue types and so after you know a lot of you know millions of dollars you know been thrown at this project and a lot of you know trials a lot of tests a lot of modifications of the drug you know the outcome was you know this drug is completely ineffective the reason for that is that the volume the overall volume of the tumor wasn't changing but actually when someone had the idea of you know looking at the individual tissues within the volume they figured that what the drug was doing was turning red into green was actually killing you know the active actively proliferating tumor cells and turning them into necrotic hole which is precisely what the drug was designed to do so actually the drug was being extremely effective but because they were using the wrong measurement because they didn't have this sort of tool they couldn't quite figure it out so that's why these sort of tools are important so how do we do this again it's very difficult to you know sit down at a computer and start to write a lot of you know rules you can use MATLAB C C++ you know Java whatever your favorite programming languages and you you say look you know these are different patients it's clear that you know the active cells you know are always you know bright white at least in this is called T one gadolinium modality but actually there are many cases like around this blob where they're not so bright white there are many cases were you know the appearance of the tumor and the necrotic core is very different from other cases like in this one compared this one to this one for instance and then you say oh wait there is an exception then I put in another if statement and you know if the value is between 256 and you know 102 days otherwise to this well that's the wrong way of doing things and if you you know start writing this each date means one after the other you very quickly get lost and there are way too many exceptions too way too many rules so a much better way of you know doing this is by allowing you know your computer to learn from experience from data what are good features to use you know by features I mean what thresholds what what channel am I looking at when reading pixel values is it t1 or t2 or do I need to acquire a different scan like a CT scan for instance or not and have the system learn from all this experience from all this data that has already been labeled into by some experts into tumor non-tumor after cells necrotic core and once I have you know learned this model we call it - more tissue classification engine then I can apply this engine to a new previously unseen patient so now I've got a new image coming for a new patient which the system has never seen before so I pushes into the engine and some data is fed out that hopefully will correspond to the correct map for these three different tissue types so that's the idea underlying all machine learning research really you know the key is really to design something which works first of all and is efficient enough to be able to do experiments with also to run in a short enough time in clinical practice so we first of all need to create the label database so that that's the key ingredient for our machine learning based approach we need to have a good number of patients and for all of those we need to have the ground truth labeling we need to have an expert like a doctor who knows what a tumor looks like and what the different teaching components look like and can actually come up with this map so clearly this is a difficult and very time-consuming thing to do because for an image like this you know you can immediately see roughly where the tumor is but if I told you what is the active are to these Democratic Party what is the edema well you know when we try to view a pixel-by-pixel which on his which is very difficult to do so doing this efficiently even interactively even allowing a user the user to interact with the image and to bring season to you know provide hints you know this is very difficult doing it you know efficiently is very important not just for this project but also for what's called radiotherapy planning so when I have a patient and I detect I figure out that there is a tumor then what I need to do is I need to plan for the radiotherapy treatment the radiotherapy is effectively a machine so the patient you know lies down on a bed and then the reason machine with a sort of robotic arm for instance which goes around the the patient and you know it radiates the patient with x-rays for instance and x-rays we know are dangerous and it's a good thing that they're dangerous because you can use them to kill the tumor but of course that dangerous also when they go through the healthy tissue so this radiotherapy planning you know it's meant to maximize the amount of radiation dose that goes into the tumor and minimize the radiation dose that goes into the healthy tissue so the first step in this planning is for a lot of you know NHS you know the National Health Service employees to sit down in front of a console like this and actually manually you know delineate you know the tumor but also delineate the healthy organs as well so it takes between two and eight hours per patients okay so you can imagine how expensive this job is and how tedious this job is as well so if you can do anything to help you know they'll be highly appreciated if we do have something that helps and you know this tools you know chemi as it might look it has already been used at Addenbrooke's and know the absolutely absolutely so because of tarnis and everything else you tend to make your mistakes you know hopefully not too many but that's just the nature of things so this interactive tool interactive the for assisted there is a human in the loop what you do is you know given an image of this type you provide some hints okay so for instance you just you know draw this green brush green indicates a necrotic core and then around here these red brushes brushstrokes they indicate what I believe is you know they actively proliferating parts of the tumor and then you know after this you know hint is given the computer fills in the rest okay so the rest is done completely automatically so the active room is completely delineated and the necrotic core what's more important is that the interaction happens on one or two slices only only in 2d because that's the best way we know how to interact with images but the computation the the propagation of this information happens entirely on 3d so this is again what's called an axial view of the brain horizontal view this is a what's called a sagittal view so from the side you can see the eyes here you know here's the mouth this is the brain and this is a frontal view which is also called coronal so all this you see that you see green and red all over so this this computation is all done in 3d so rather than having to go on every single slice and click click click click click on thousands and thousands of points this can be done by you know very very quickly you know just acting on a couple of slices of three slices and it can be done in any orientation it doesn't need to be only in one orientation so allegedly you know this is this is a simple example because this is all black inside so you can say all you know some of you might be familiar with you know region growing or you know distance transform sort of techniques it's all anything would work on this and you're right this is a particularly simple example but it works you know these technique works also when applied to more tricky sorts of tumors where the necrotic hole for instance is very highly variable in highly textured so that's very good so we said you know there this is you know a nice small self-contained piece of water has come out of the research you have been doing and has got at least two important applications one is you use it in its interactive form for radiotherapy planning but you can also you give it to your doctor colleagues to create ground truth data which then can be used to train the automatic system the completely automated system and so we've done so so all right before I go there so what we're done is you know we have collected we have now about sixty patients from the Cambridgeshire area all suffering from the same type of tumour is called glioblastoma grade 3 or grade 4 were the grade indicates the level of aggressiveness so that's at the top of the level of aggressiveness and though the most aggressive and they are also the most difficult to detect automatically in images so we have those and now what we need to do is we need to train from this training data we need to optimize my our classification tool they will be able to automatically identify tumors in images for that we use a yeah absolutely you know you're absolutely right so doing that is something the another group hearing in Cambridge is looking at and this is part of the constraint programming group that we have here nsrc so they're looking precisely at that problem so I've got a certain region I want to read it a lot certain other I want to you know preserve it you know save it from from the x-rays figure now and there is a well-defined cost function which once you minimize it will give you the exact path for the x-rays okay so and that's all part of the you know path planning sorry the radiotherapy planning I should say that goes on into radio modern radiotherapy machines now that's another area of research that we are investigating right now I'm not the expert there but there are some people who are and we're working with the you know oncologists in other Brooks to really figure this out because one problem there with modern machines is they they are very very sophisticated but the the planning and the optimization takes forever it takes hours and hours and hours so before you see the results you have to wait hours and then you look at the results and you figure oh gosh I forgot to put in another you know region there I want to preserve it because that's in the bone marrow or something and then you put that in you send it back to the Machine it's it's a mess there is a lot of good stuff that we can do for doctors to improve you know the quality of healthcare they they deliver wrigley and also for the doctors themselves to be able to go home at a certain time in the evening if you be honest so in terms of you know once we have the data we need to choose what machine learning algorithm we are going to use and there we need to choose between many right so machine learning is a very advanced field of research which is a you know good at bad at the same time because there are so many algorithms you can choose from it's difficult to figure out you know which one is the best for for my task so there is a special class of techniques that we've been studying which is called decision forests or random forests and the reason why we're interested in those is because they are particularly efficient and because they're so efficient it means that you know we have some hope of making them work with lots and lots like millions and millions of you know pixels or voxels voxel data and also what we recently discovered is the decision for is canticle you know many different machine learning tasks so in machine learning normally you you hear about no classification about you know regression so this you know answers a little bit like you know your question so when i want to you know given some data i want to figure out whether this data you know is less let's think of images so is this the image of a beach scene or is it an image of a street scene or is it a forest sort of you know landscape so that's a very discrete source of assignments so that's typical of your classification sort of problems well instead in regression you have continuous predictions so for instance your given distance from converging village college which is a very good school in the area what is near the house price so if i'm looking to move to a different house i can check against this model to see whether the house of god you know my eyesight upon is overpriced on the price so this or thing but you know the price or temperature is more of a continuous estimate there are many other things that you can do with the same underlying model same underlying efficient algorithms so for instance you know classification forests are being used and are used in your living room when you play kinect games i'm sure all of you have got Kinect at home if you don't you know what to do so you know the cool thing is that the underlying technology that makes Kinect work it's all based on these sorts of techniques so you probably have heard of the Kinect depth sensor is a commoner which manages to you know read out you know the color of each pixel but also the depth of each point how far these point is from the camera itself but that's not the end of the question now there we have all this information you need to figure out whether what the camera is looking at is a player trying to play a game or is a sofa or a television set you see what I mean and so the way that is done is by assigning effectively color coding to each pixel and each color corresponds to different body parts so for instance dark gray whatever this is for the left shoulder you know blue or light blue for the left wrist and left hand and so on this this works all based on classification forests and it's quite fun to see how the underlying technology is the same what we've been working on for years now is one technology one machine learning model which is then applied to either gaming or brain tumor analysis very very diverse applications but that's what makes you know this type of research fun there are the things that you know we've been using forests for for instance I was mentioning earlier if I have a CT scan like a full-body CT scan which clearly you know includes your millions of voxels and even compressed in my end up you know been something like 2 gigabyte file and I know that no this is a full full bodies of the scan but as a doctor I'm only interested in looking at the left kidney this is a side view of the left kidney so if the two gigabytes resides on a remote database I don't need to transfer the entire two gigabytes before I can you know look at the kidney if the computer has already detected say a bounding box for where the kidney is then you know it be almost instantaneous because it's so much less data and you can also do this you know progressively then you know I click on the left Guinea I just get that and then I get you know the heart and other things if I'm interested in other regions as well so it's is you know this sort of technology is good for you know software engineering as well but how does it work many of us have probably studied about decision trees so decision trees are tree structures is just a data structure where you have nodes internal nodes you have edges connecting the nodes there are no loops and that's why it's called a tree and not a graph and then there are some terminal nodes as well which I'm showing here are squares and those are called the leaves so we can use a structure of this type for instance to do a very basic this is a toy example very basic in the image understanding so if I've got a picture of my kids here and trying to figure out whether this is an outdoor scene or an indoor scene then you can think oh yes I know how to do this you know if the top top part the top half of the image is mostly blue then it's likely to be an outdoor scene okay so you can write this you know in terms of a decision tree so at this another half I insert a question which is you know is the top part blue it's a test and so I can you know follow this branch if the answer is true in this branch if the answer is false if I follow this path then I can have another question which is is the bottom half of the picture also blue in which case maybe I'm looking at the picture of a blue wall and then it's an indoor scene and so you know if it is false and I end up you know in dry answer which is you know outdoor window so at the leaf nodes rather than storing you know a point answer i straight indoor straight out door I can also store probabilities I can say you know 90% probability of it being outdoor ninety percent probability of it being indoor for instance which means that you know these simple models can be probabilistic so not only they can give me an answer is a tumor is it no tumor but also have an Associated confidence so if I say you know there is a tumor in this patient but I'm not very confident about my answer then as a doctor as a human being I can you know look at the piece of information say okay further tests are necessary maybe I need to get a different Ammar scan or something else to make sure that I know the answer so that's why you know it's important to deal with you know probabilities so in I'm not going to go I'm not going to present much mathematics but just to give you a little bit you know more intuition so we have some data points you know which are represented by these little dots here which can be represented in a very high dimensional feature space could be D could be even infinite dimension so for each imagine that these points represent pixels for instance and I'm trying to classify these pixels as belonging to the active region or the edema region then I could extract you know colon information information about you know contextual you know spatial context whether it's close to a certain organ or not also all these sorts of information can be all incorporated within one very very long vector like representation and all of this can be used as input to my tree and the different tests represented here by this h function with some parameters theta can decide whether to send this point to the left or to the right and this tree scheme can grow to be very very large for instance in the Kinect case we have trees which are 20 levels deep so those are you know quite massive trees because you have if it was completely full you had you know two to the twenty you know number of tests associated to each to each node and being able to make you know that sort of system run you know in real time actually more than real time is quite an amazing achievement but the other thing that was discovered recently actually decision trees are very old news so we didn't invent them you know they were invented 40 or 50 years ago but recently they've been revived because he was discovered again not by us but by people before us that if rather than having a single decision tree you have a collection of decision trees which are all similar to one another and yet randomly different from one another the accuracy of your final output becomes a lot better in particular you mitigate a big problem in machine learning which is called overfitting overfitting is what happens to Vince's neural networks which is once they are presented with training data they tend to understand and learn that training data extremely well but then they're not capable of generalizing what they have learned to new previously unseen data which we call test data so the new patient which has never been seen before by the system will not be classified will not be handled correctly by the neural network or by a single decision tree and we cannot claim the forest you know solve that problem completely it is still the biggest problem in the whole of you know the machine learning community but we have seen a lot of evidence that you know that problem is mitigated hugely so they are better than anything we know are generalizing that's that's the important thing ok I'm not going to go through this so this is a very toy like illustration imagine that I've got a two dimensional feature space and I've got some data points so each of these points can for instance represent companies for instance you know in business terms we might have x1 is what is the share price and x2 some other measure of you know how well a company is doing and we have this training data and these four sets of clusters of companies so given a point in any of this you know white space we want to classify that point as belonging to one of those four classes so clearly you say oh this is very easy I can just compute the minimum distance to the training points and that's one way of doing it except it's not a very general way of doing it because you know ones have got you know tons and tons of training points then it becomes very cumbersome and time-consuming to do so but also like I said you know we want to be able to track you know the confidence of my prediction and you know what I'm showing at the bottom here is the output of you know forests trained with some variations so in this case you know this does something there is you know pretty obvious you know whatever is on the top left quadrant is assigned to the yellow class whatever is here is assigned to the blue class and and some and you might say you know that's what I want but actually when you when you look at this other set of results you see that not only a point here is assigned a blue color but also the color becomes washed out which is my way of visualizing a higher uncertainty in the prediction a lower confidence in the prediction which makes a lot of sense so what this shows is that you know points we did are falling between the classes or are very far away from any training point there has been seen previously gets assigned a certain class like red or green or yellow if you really force it to but the system confidence becomes a lot lower and that's really what you want instead you know that doesn't happen here so this shows a little bit of the sort of flexibility of this type of you know machine learning problem models more examples but ok so that's the underlying theory of you know what happens behind the scenes both in the Kinect world and in the medical world but going back to the you know brain tumor problem which is a application forget all this these are you know the results that we get so for each column here you have a different patient these are all test patients which means that they have never been seen before by my computer by my algorithm this is the ground truth segmentation so this is the gold standard this is what we would like to achieve what the expert has told us where the different tissue types are for each voxel and these are the results we get you know using our forest base technique so you see that they are different but they're not too different so it's very very encouraging results so despite the fact that the tumour has got you know a very different appearance location shape a lot of variability in the actual you know size and age of the patient the system still manages to do a very very good job completely automatically these are some pretty pictures really some visualizations just to show that it all works in 3d and again we have a good understanding of not only the tumor per se but also the edema another reason why figuring out where they dima is is because that's that gives us a good idea of where the tumor is gonna spread next if we do nothing so it's it's a great thing to have because right now this is not used when doing radiotherapy planning and it should be because you know I can adjust you know the way in which I the region in which I focus my x-rays so as to cover a little bit of perhaps healthy tissue but in the edema region because just in case I've got a little bit of infiltration of terminal cells there I can prevent them from spreading which should be a fantastic thing to do in this type of tumors so we get fantastic numbers whatever that doesn't matter too much but project to because I don't have too much time is about a different type of medical images they look different and they are different acquired by different type of scanners this is a CT CT scan which is a little bit like x-ray so you can see the bones very well so this is a horizontal view across the abdomen of this patient so here we've got the spine actually this is close to the thorax as well because I can see a little bit of the ribs you can see them in 3d here this is the spine this is a kidney this is the liver so it's actually you're seeing the patient from the bottom this thing here is the author big pipe pumping blood all around you know conveying blood all around and what we want to do is we want to be able to say in this view what which vertebra is this one so you might say wait a trained eye will be able to say this straightaway wrong radiologists have got a lot of issues with this they would love to be able to say immediately straight away just by looking at this image what vertebra that is where it is you know T 1 T 2 T 11 t means thoracic or l1 l2 l3 lumber in the lumbar region and so on why is this important because when in all radiological reports you know if if I discover that we say a tumor in this region then I say is beside the l1 vertebra you know the spine the human spine is the perfect coordinate you know system for the human body it's a reference frame which we have it we were born with and doctors used all the time to refer their findings to the position in the body so being able to discover this you know automatically will be great being able to do this automatically is very challenging why because all vertebra look nearly identical to one another so why do we have any hope of being able to do this at all well because you know in the lower thoracic you know early lumbar region for instance you know we know that we're going to be close to kidneys so if you can discover the context we know there we are the bottom of the liver and we can see a little bit of the kidneys then we know roughly which region of the of the spine we're in and so we need to develop techniques which can make use of context you know contextual spatial reasoning very very efficiently and very actually as well you might say wait you know if I look at the same patient from the side then I can have a better understanding of you know which vertebra I'm looking at well even that is tricky because I need to know water ideologies too is the 10 - sorry they try to figure out where the lungs stop and thus the beginning of the lumbar region and then they count one two three four it's very tedious and very time-consuming so being able to solve this problem is very important even just to make the problem even more interesting quite often in order to reduce as much as possible the amount of x-ray radiation which is thrown at a patient these images are acquired in a very cropped way like only just the region that we interested in is image so that we avoid you know x-rays all over the place because it's bad for the patient and so in this sort of your highly cropped views it's even more difficult to figure out exactly which vertebra is this so we have said you know why this is important and this is the sort of variability we have in our database so we have both almost you know full-body scans so we have very highly corrupt scans we have had a neck scans we have thorax scans or abdominal scans we have you know big patients more patients all sorts of ages I want this system to work in all cases once again in the case of a PACs retrieval system as we were describing earlier this can also be extremely interesting if you know this if the computer knows days ahead of time you can do a lot of good in terms of a more adaptive selective retrieval of the images so what we do is we use a different type of decision forests in this case is called a regression forest again answer your question we can deal with continuous variables continuous predictions rather than discrete ones in almost exactly the same way and we have a a cost function which is defined in a continuous space in this case and then we we refine the results used using a more generative model statistical models of approach in the form of a hidden Markov model although this is already all news and we have replaced this but all right yeah we get good numbers these are you know the more interesting to look at no visual results so red is our prediction and yellow is ground truth so you can see that you know we get very very close you know to the ground truth in all cases but also because we have a whole model for the whole spine we can sort of predict you know what happens outside the image as well which is probably completely useless but it's still a lot of fun to look at and these are some videos of you know our results being done in 3d so one thing for instance a is quite interesting to do is in this video on the left you see that you know because now I know you know the evolution of this curve of the spine I can actually create a special type of frontal view which is actually evolving with the curvature of the spine so I can see all the vertebrae in one frontal view which is something you know very very useful to have you know from from a trauma doctor point of view of something which with our technique would be just impossible to do yes very good question we don't know where the liver is so how can we use the context you know to inform the position of absolutely so there are many ways in which you can incorporate context what you have in mind which is absolutely correct is what we refer to as semantic context as in if I know already what region is the liver or the kidney then I can use that clearly semantic information that tagging to inform the rest but if we don't know that information then I can still use what we call is just us calling out that that way appearance context so for instance in the l1 vertebra which is about here if I look 10 centimeters to the right and 10 centimeters up I expect to see in that region a certain type of tissue density which corresponds to the liver like densities right and then if I look you know to the left a certain amount in a certain direction I expect to see kidney like tissue densities right so I can construct my decision tree automatically of course with a lot of these sort of tests where for a given reference point reference voxel I can look around a certain distance away in a certain direction many many different probe pixels and if they all look in a certain way then I cascade through the tree in a certain you know branch in a certain direction until I get to my destination to my leaf and there will be a certain probability that I read out so that's what we do indeed I did mention context and I didn't explain how we implemented very good question so right now we are working on this even more complex patients where there is huge amount of you know deformation this is scoliosis so the this is a frontal view but the spine is deform to know this you know very weirdly and then this is the same patient after the spine has been straightened up but a lot of nails effectively have been inserted to make sure that the spine is straight the algorithm assistance today cannot quite deal with these sorts of you know patients not yet but we're working towards you know making it work in the case as well so summary you know we can use machine learning to learn how to automatically analyze images today here I talked about medical images which have got you know a clear impact on clinical applications and the way doctors do their daily job and the way patients are cured but what's interesting is they're almost exactly the same technology you know has been developed you know for Kinect and so for a completely different type of application and anything to do with image analysis really where the images are of any type you know we can think of using these sorts of techniques there so you know we're quite excited about this line of research I think I've answered all eight really a lot of questions but I'm very happy to answer all other remaining questions you might have thanks very much for thank you that's on tuck so this question is about some curiosity so I think this technique is really amazing and can it already be used in some Hospital yes and then I think what is the accuracy I think it's much more important if the if the patient really have the humor and you must predict out the humor that you're sorry so answer to your question a lot of the technology that I've shown here has been trialed and tested in many hospitals in particular Adam Brooks here which is one of the main research hospitals in Europe but also the University of Washington in Seattle so the tumor part is being tested in here in other Brookes the spine part in in Seattle but also some other work that we didn't quite talk about today is part of a product called amalga which is a radiology type of product which is installed already in many many hospitals in the United States and in fact even in Milton Keynes man behind you is responsible for that installation apparently and answer to the second question yes whenever we have to deal with patients who suffer from you know very serious conditions we are correctly so scared of you know making mistakes however things are much better than we think they are because doctors in the past used to be very very traditionally it's very conservative not wanting to accept or use anything to do the hard to do with technology but the new wave of doctors that just like us on Facebook every day and you know Twitter and all of that they are much more happy to use technology and even if the technology doesn't work perfectly well they sort of understand how to deal with that so that's a clear advantage that you know is you know coming to bear really in these days but you know still we need to be extremely careful with accuracy and all of that so what we show with the tumor work has been evaluated and compared to existing state of the art and we do way way better in terms of prediction than any existing technique as far as we can tell because there is also the problem of being able to conduct a fair comparison with other techniques there is a lot of problems in this field for instance company or you know academic research team such-and-such come up with another technique for doing glow blastoma detection they publish the results they don't publish the data set how can I compare with their you know algorithm so it's a very unscientific process right now which among the many things we are doing we are also trying to change that we are trying to publish our own dataset with the labels so the people can compare against us and and that's fair comparison as possible you can always say the comparison is unfair but at least we try to help making it fair so short answer to your question good accuracy compared to state or they are is it enough for this being deployed for doing real clinical use for global estimate patients we don't know yet we are working on that with the relevant medical people there's a question here this might be recorded so that's why you might need it so thanks for a nice talking thank you I have two questions actually first one also reality to the predictions so I think there is also one step about this and that is prediction in time regarding to the first topic you usual so you probably have also like time sequences of this data it is possible to extend this method also to that so for example if I irradiate the the tumor somewhere in some of some way how it will change yeah very very good question so in our team there is someone who is an expert in constructing models of this type of tumor and our temporal models so he can tell you with great accuracy how the tumor is likely to evolve for instance in the case of you know gliomas in glioblastoma so we know that they prefer to grow in the direction of the white fiber tracts so they don't grow too much in the gray matter but they grow a lot in the white matter and they as they grow they tend to also destroy those tracts with a lot of implications as well so if we know where the demon is we know where the white matter is which is stuffed now we can know automatically and we know we have these mathematical models for the evolution of the tumor then you know we can put all of these things together right and we know we can predict how the tumor is going to evolve and that can in for our technique and indeed right now you know in the second phase of this project we have now temporal sequences for again Adam Brooks patients for which we have you know the preoperative image we'll also have the image 48 hours after whatever treatment it could be the radiotherapy or surgery the the tumor has been removed and then we have follow-up images as well so we have a lot more data and we are trying to look at you know how to extend this you know algorithm algorithm to work on the whole sequence of data as well I see may is a second one so my sort of notion of these machine learning techniques is that you need a some sort of critical amount of data like there is a threshold above which the method just starts to work so are we quite there yet or do we just need more more collection and yeah yeah so answering that question is a lot more difficult because it depends a lot on the data itself how difficult the problem is inherently but also depends a lot on what type of machine learning technique you use again this is the problem of generalization also saying so it's not entirely true that in order for anything to work you need a huge amount of training data is not true depends on the problem loss depends on how well your algorithm can generalize even for fewer examples for fewer for a smaller set of you know training data so the data we have right now is not even a hundred patients so this is completely you know a toy example it's not something that whatever you know feel comfortable deploying in the real clinical setting but you know this type of work is encouraging our medical partners to acquire more of this data and so perhaps in five years time we will have I will be you know happy to have about a thousand of these you know patients if we have that I will feel very confident that we would have some theory works yes thank you very much I'm interested in about error source so if you can't exactly predict to some of the two tumor or not exactly about the vertebra and then can you somehow decide where ad error coming from is it because of the model or because of parameter or simply because of data or something yes that answering those sort of questions takes most of our time okay or is it because of a bug in the implementation that's the other equation which is very very likely to occur as well much more than I would like it to be yeah a difficult question to answer but with these sort of models you can do a lot of reasoning after the fact so what's interesting is that we with a decision for us the I liked them a lot because they are no longer a black box like neural networks used to be in your networks they feel a lot more like a black box you fill in some data something happens and some data comes out and hopefully that works but when it doesn't work then it's very difficult to figure out what doesn't work with decision trees you can actually you know once the the trees have been trained you can go and inspect every one of the different nodes both the the internal nodes and the leaf nodes and try to make sense of what features are used where what has been chosen to be used where how many times in the trees and so on so there have been occurrences where we have done this analysis and then we figured that for instance you know at the very top part of the tree some of the features that the system automatically chose to use correspond exactly to other features that in previous papers other authors had manually chosen to use so that's fantastic to find because it means that they were right and we are right possibly unless you are both wrong but also the advantage of using the automatic technique is that you have many many many more tests to be applied after the few ones that you can choose by hand so you can do a lot of your reasoning after the fact that helps you debug the system and trying to figure out whether there are some bugs or glaring mistakes so whether you have to design the features in a different way or whether there is just too much noise and images to be able to extract any meaningful information so ok thank you very much

Channel: Microsoft Research

Views: 11,544

Rating: 4.8620691 out of 5

Keywords: microsoft research

Id: G8xRwxqe6gk

Length: 59min 24sec (3564 seconds)

Published: Thu Aug 11 2016