In 2016, JAMA published research demonstrating
the efficacy of a deep learning algorithm. We were able to train a deep learning neural
network to recapitulate the majority decision of 7 or 8 US board certified ophthalmologists in
the task of grading for a diabetic retinopathy. The type of deep learning algorithm
used to detect diabetic retinopathy in that study is called a
Convolutional Neural Network, or CNN. CNNs enable computer systems
to analyze and classify data. When applied to images, CNNs can recognize
that an image shows a dog rather than a cat. They can recognize the dog whether it's a
small part or a large part of the picture - size doesn't matter for this technique. It can also classify the dog by breed. CNN systems have also been developed to help clinicians do their work
including selecting cellular elements on pathological slides, correctly identifying
the spatial orientation of chest radiographs, and, as Dr. Peng mentioned, automatically
grading retinal images for diabetic retinopathy. So let's open the deep learning
black box to understand how this works. First, a CNN is not one process. It's actually a complex network of
interconnected processes, organized in layers. With each layer, the CNN can detect
higher-level, more abstract features. When the CNN is identifying these
features, it uses something called a filter. Here's how Larry Carin, one of the authors of
a JAMA Guide to Statistics and Methods article on CNNs, describes a filter: So,
we think about a medical image, a medical image in radiology or ophthalmology or
dermatology is characterized by local structure, could be textures, it could be
edges, it could be curves, corners, etc. And what these filters are doing are
constituting little miniature versions of each of these little building blocks. And the way that the CNN looks
for these building blocks is the C in CNN, and it stands for convolution. It's a mathematical operation
that looks pretty complex. But, actually, it's very simple. It's a very simple concept. It's kind of like you've got this filter, and
you're walking to every part of the image, and you're just asking the question, how
much does this image look like that filter? Think of it like this: you
have a drawing, that's the image, and you have a stencil, that's the filter. You take that stencil and pass that
stencil over that drawing that you have, and as you do that you will see that some
parts of the drawing become more visible than others as you do that, right? And that process of sliding that stencil across this drawing is essentially
the process of convolution. Now that we've explained what a filter is
and introduced the concept of convolution, let's use an analogy of written language to
understand the relationship between the filters and the hierarchical structure
of the layers in a CNN. We will simplify the explanation
by using an analogy. The analogy is a written document. In order to communicate through writing,
we organize it as a series of paragraphs, which are composed of sentences, those sentences
are composed of words, and the words of letters. So reading a document requires assessing
the relationship of letters to one another in increasing layers of complexity,
which is a kind of "deep" hierarchy, like the hierarchy in image analysis. Continuing with our analogy,
let's say we're looking for the phrase Ada Lovelace in a paragraph. Ada Lovelace was a mathematician and
writer who lived in the 19th century. And she holds the honor of having published
the very first algorithm intended to be used by a machine to perform calculations, which
makes her the first ever computer programmer. In the first layer of the network, a CNN
looks for the basic building blocks of an image. The basic building blocks of
written language are letters. So in this analogy, the filters the CNN
uses in the first layer would be letters. Let's zoom in on the word "Ada." Here is what the convolution process
would look like for the letter A. When the "A" filter overlies the letter
"A" in the original image, the convolution output would
generate a strong signal. This signal would then be mapped
onto something called a feature map. The feature map represents how well
elements in the image align with the filter. If something is there, the signal outputs white. If nothing is there, the signal outputs black. CNNs generate a feature map for every filter. So in our analogy, there would be
a feature map for every letter. These feature maps would then
become the input for the second layer. In this layer, the CNN would spatially align and
"stack" all those maps from the previous layer. This would allow the CNN to then look
for short, specific sequences of letters in all the feature maps simultaneously. So the CNN would use a new set of filters to
look for specific letters that are adjacent to one another in particular sequences. In our analogy, the second layer would
look for places where the letters A, D, and A are in sequence together
making the word "ADA". It would also look for places where
letters A, C, E, L, O and V are adjacent to one another using filters for LOVE and LACE. The output of the second layer
would be the feature maps for those three sequences of letters. In other words, in those feature
maps, strong signals would be present where the sequences ADA, LOVE and LACE
are located in the original paragraph. In the third layer, the CNN would
stack and align these three new maps and perform more convolutions-this
time identifying where longer words and groups
of words are located. So the CNN could at this point identify where in
the original paragraph the sequences of letters and words making the phrase
"ADA LOVELACE" are located. In our analogy, we were looking for
a phrase consisting of only two words. Had we been looking for a longer sentence
or even a paragraph, the CNN would deal with the greater complexity
by having more layers. We've omitted quite a few
details about CNNs for simplicity, but this captures the essence of the model. But what does this look like for actual images, like identifying diabetic
retinopathy from an ocular photograph? Images are made out of
pixels rather than letters. In a digital context, a pixel is the smallest, controllable unit of an image
represented on a display. Each pixel is a representation
of a tiny portion of the original image. Think about pixels like creating
a drawing with dots where every dot has a color value and an intensity. The more dots used, the clearer the
image becomes. The filters a CNN uses in that first layer
are small squares of pixels that correspond to things like textures, contrast
between two colors, or edges. These are the image analysis-equivalents
of the letters used in our analogy. And as a CNN goes up in the hierarchy,
it looks for combinations of these filters, getting more and more complex with each layer. As the complexity increases, the CNN gets
closer to identifying what it's looking for. So the specific features analyzed at each
layer help put the whole thing together. So, for example, some of the earlier work
showed that some layers tend to be better at extracting, sort of like,
edge-like information. Meaning that, for example, if you combine
different kinds of horizontal edges, we might get a continuous line that
resembles the retinal blood vessels. And as you combine more of those and start
to encode more higher-level concepts such as, you know, is there a micro-aneurysm here, is
there bleeding over here, is there other lesions in the image? And right at the very end is
where these, after these multiple layers, the network will try to then condense all of
that information down into a final prediction. In this case, severe diabetic retinopathy. Developing a CNN to help identify diabetic
retinopathy was motivated because many patients with diabetes are not getting
screened frequently enough. We have to screen diabetic
patients once a year or we should, and there are some barriers
to getting that done. Some of it is just, you know, not having
enough trained professionals to do that task. It's also not having that expertise
available where the patient is. It's not that, you know, there
aren't retina specialists in a metropolitan city four hours away, it's that there isn't a retina
specialist at your grocery store. And CNNs could facilitate the
integration of diabetic retinopathy and other screening programs into primary care. But before that happens, more research, especially prospective clinical
trials, are needed. The way we do approach these things is
really the way that medicine usually works, which is to say, "let's do validations
of the method again and again and again until we're sure, we're reasonably confident
that it really works on many kinds of images, in many settings for, you know, many
different patient populations." And so from my perspective that's really at
the end of the day what's most important: does it work on real patients
and is it reliable? The excitement generated by early results has
already spurred several research groups to look into the efficacy of CNNs in clinical practice, which could potentially finally get
CNNs from the bench to the bedside. I think we're on the third or
fourth technological revolution where neural networks are
coming to the forefront, and I really hope that this
time we'll get it right. But there were failures in the past where
people used the technology in suboptimal ways and we don' t want it to happen again. One has to make sure that we have appropriate
and sufficient data for development, validation and testing, and that we're
solving actual clinical problems. At the end of the day, one thing to
take away is that even if, as a clinician, it can be hard to understand
exactly how a CNN arrives at its diagnosis, it can still be a useful tool. And this is similar to how many clinicians
use other widely-adopted technologies. Consider antibodies: You know, as a
clinician I may not know exactly where that part of an antibody kind of binds to, but
I'm comfortable after looking at some of this clinical validation of using
Lucentis, for example, for an injection, right. This is kind of like any new
breakthrough technology: needs validation and needs transparency, but I think,
you know, the medical community in general responds very well to new
technologies that have been validated. This video is meant to be an
introduction into the topic of CNNs. To further understand how machine
learning works in a clinical context, be sure to read the JAMA Guide to
Statistics and Methods article by Drs. Carin and Pencina in the
September 18, 2018, issue of JAMA.
