The following content is
provided under a Creative Commons license. Your support will help
MIT OpenCourseWare continue to offer high-quality
educational resources for free. To make a donation or to
view additional materials from hundreds of MIT courses,
visit MIT OpenCourseWare at fsae@mit.edu. PATRICK WINSTON: It was in
2010, yes, that's right. It was in 2010. We were having our
annual discussion about what we would dump fro
6034 in order to make room for some other stuff. And we almost killed
off neural nets. That might seem strange
because our heads are stuffed with neurons. If you open up your skull
and pluck them all out, you don't think anymore. So it would seem
that neural nets would be a fundamental
and unassailable topic. But many of us felt that
the neural models of the day weren't much in the way
of faithful models of what actually goes on
inside our heads. And besides that,
nobody had ever made a neural net that was
worth a darn for doing anything. So we almost killed it off. But then we said,
well, everybody would feel cheated
if they take a course in artificial intelligence,
don't learn anything about neural nets,
and then they'll go off and invent
them themselves. And they'll waste
all sorts of time. So we kept the subject in. Then two years
later, Jeff Hinton from the University of
Toronto stunned the world with some neural network
he had done on recognizing and classifying pictures. And he published
a paper from which I am now going to show
you a couple of examples. Jeff's neural net, by the
way, had 60 million parameters in it. And its purpose was to determine
which of 1,000 categories best characterized a picture. So there it is. There's a sample of things
that the Toronto neural net was able to recognize
or make mistakes on. I'm going to blow
that up a little bit. I think I'm going
to look particularly at the example labeled
container ship. So what you see here is that
the program returned its best estimate of what it was
ranked, first five, according to the likelihood,
probability, or the certainty that it felt that a
particular class was characteristic of the picture. And so you can see this
one is extremely confident that it's a container ship. It also was fairly
moved by the idea that it might be a lifeboat. Now, I'm not sure about you,
but I don't think this looks much like a lifeboat. But it does look like
a container ship. So if I look at only the best
choice, it looks pretty good. Here are the other things
they did pretty well, got the right answer
is the first choice-- is this first choice. So over on the left,
you see that it's decided that the picture
is a picture of a mite. The mite is not anywhere near
the center of the picture, but somehow it managed to find
it-- the container ship again. There is a motor scooter, a
couple of people sitting on it. But it correctly characterized
the picture as a motor scooter. And then on the
right, a Leopard. And everything else
is a cat of some sort. So it seems to be
doing pretty well. In fact, it does do pretty well. But anyone who does
this kind of work has an obligation
to show you some of the stuff that
doesn't work so well on or doesn't get quite right. And so these pictures also
occurred in Hinton's paper. So the first one is
characterized as a grill. But the right answer was
supposed to be convertible. Oh, no, yes, yeah, right
answer was convertible. In the second case,
the characterization is of a mushroom. And the alleged right
answer is agaric. Is that pronounced right? It turns out that's a kind of
mushroom-- so no problem there. In the next case, it
said it was a cherry. But it was supposed
to be a dalmatian. Now, I think a dalmatian is
a perfectly legitimate answer for that particular picture--
so hard to fault it for that. And the last case,
the correct answer was not in any of the top five. I'm not sure if you've
ever seen a Madagascar cap. But that's a picture of one. And it's interesting
to compare that with the first choice of the
program, the squirrel monkey. This is the two side by side. So in a way, it's not
surprising that it thought that the
Madagascar cat was a picture of a squirrel
monkey-- so pretty impressive. It blew away the competition. It did so much better the
second place wasn't even close. And for the first time, it
demonstrated that a neural net could actually do something. And since that time, in the
three years since that time, there's been an enormous
amount of effort put into neural net technology,
which some say is the answer. So what we're going to
do today and tomorrow is have a look at this stuff
and ask ourselves why it works, when it might not work,
what needs to be done, what has been done, and all
those kinds of questions will emerge. So I guess the first thing to
do is think about what it is that we are being inspired by. We're being inspired
by those things that are inside our head-- all
10 to the 11th of them. And so if we take one of those
10 to the 11th and look at it, you know from 700 something
or other approximately what a neuron looks like. And by the way, I'm going
to teach you in this lecture how to answer questions
about neurobiology with an 80% probability that
you will give the same answer as a neurobiologist. So let's go. So here's a neuron. It's got a cell body. And there is a nucleus. And then out here is
a long thingamajigger which divides maybe a
little bit, but not much. And we call that the axon. So then over here, we've got
this much more branching type of structure that looks
maybe a little bit like so. Maybe like that-- and this
stuff branches a whole lot. And that part is called
the dendritic tree. Now, there are a
couple of things we can note about this is that
these guys are connected axon to dendrite. So over here, they'll be
a so-called pre-synaptic thickening. And over here will be some
other neuron's dendrite. And likewise, over here
some other neuron's axon is coming in here and hitting
the dendrite of our the one that occupies most
of our picture. So if there is enough
stimulation from this side in the axonal tree,
or the dendritic tree, then a spike will
go down that axon. It acts like a
transmission line. And then after that
happens, the neuron will go quiet for a while as
it's recovering its strength. That's called the
refractory period. Now, if we look at that
connection in a little more detail, this little piece right
here sort of looks like this. Here's the axon coming in. It's got a whole bunch
of little vesicles in it. And then there's a
dendrite over here. And when the axon is stimulated,
it dumps all these vesicles into this inner synaptic space. For a long time, it wasn't
known whether those things were actually separated. I think it was
Raamon and Cahal who demonstrated that one
neuron is actually not part of the next one. They're actually separated
by these synaptic gaps. So there it is. How can we model,
that sort of thing? Well, here's what's
usually done. Here's what is done in
the neural net literature. First of all, we've got
some kind of binary input, because these things either
fire or they don't fire. So it's an all-or-none
kind of situation. So over here, we have
some kind of input value. We'll call it x1. And is either a 0 or 1. So it comes in here. And then it gets multiplied
times some kind of weight. We'll call it w1. So this part here is modeling
this synaptic connection. It may be more or less strong. And if it's more strong,
this weight goes up. And if it's less strong,
this weight goes down. So that reflects the
influence of the synapse on whether or not the whole
axon decides it's stimulated. Then we got other inputs down
here-- x sub n, also 0 or 1. It's also multiplied
by a weight. We'll call that w sub n. And now, we have to
somehow represent the way in which these inputs
are collected together-- how they have collective force. And we're going to
model that very, very simply just by saying, OK,
we'll run it through a summer like so. But then we have to decide if
the collective influence of all those inputs is sufficient
to make the neuron fire. So we're going to
do that by running this guy through a
threshold box like so. Here is what the box looks like
in terms of the relationship between input and the output. And what you can see
here is that nothing happens until the input
exceeds some threshold t. If that happens, then
the output z is a 1. Otherwise, it's a 0. So binary, binary out-- we
model the synaptic weights by these multipliers. We model the cumulative effect
of all that input to the neuron by a summer. We decide if it's going to be
an all-or-none 1 by running it through this threshold
box and seeing if the sum of the products add
up to more than the threshold. If so, we get a 1. So what, in the end,
are we in fact modeling? Well, with this model,
we have number 1, all or none-- number 2, cumulative
influence-- number 3, oh, I, suppose synaptic weight. But that's not all
that there might be to model in a real neuron. We might want to deal with
the refractory period. In these biological models that
we build neural nets out of, we might want to model
axonal bifurcation. We do get some division
in the axon of the neuron. And it turns out that that
pulse will either go down one branch or the other. And which branch it
goes down depends on electrical activity in
the vicinity of the division. So these things might actually
be a fantastic coincidence detectors. But we're not modeling that. We don't know how it works. So axonal bifurcation
might be modeled. We might also have a
look at time patterns. See, what we don't
know is we don't know if the timing of the
arrival of these pulses in the dendritic
tree has anything to do with what that neuron
is going to recognize-- so a lot of unknowns here. And now, I'm going
to show you how to answer a question
about neurobiology with 80% probability
you'll get it right. Just say, we don't know. And that will be with
80% probability what the neurobiologist would say. So this is a model inspired
by what goes on in our heads. But it's far from clear
if what we're modeling is the essence of why those guys
make possible what we can do. Nevertheless, that's where
we're going to start. That's where we're going to go. So we've got this model
of what a neuron does. So what about what does a
collection of these neurons do? Well, we can think of your skull
as a big box full of neurons. Maybe a better way
to think of this is that your head
is full of neurons. And they in turn are full of
weights and thresholds like so. So into this box come a variety
of inputs x1 through xm. And these find their
way to the inside of this gaggle of neurons. And out here come a bunch
of outputs c1 through zn. And there a whole bunch
of these maybe like so. And there are a lot
of inputs like so. And somehow these inputs
through the influence of the weights of the thresholds
come out as a set of outputs. So we can write
that down a little fancier by just saying
that z is a vector, which is a function of, certainly
the input vector, but also the weight vector and
the threshold vector. So that's all a neural net is. And when we train
a neural net, all we're going to be able to
do is adjust those weights and thresholds so that what
we get out is what we want. So a neural net is a
function approximator. It's good to think about that. It's a function approximator. So maybe we've got some sample
data that gives us an output vector that's desired as
another function of the input, forgetting about what the
weights and the thresholds are. That's what we want to get out. And so how well we're
doing can be figured out by comparing the desired
value with the actual value. So we might think
then that we can get a handle on how well
we're doing by constructing some performance function, which
is determined by the desired vector and the input
vector-- sorry, the desired vector and
the actual output vector for some particular input
or for some set of inputs. And the question is what
should that function be? How should we
measure performance given that we have
what we want out here and what we actually
got out here? Well, one simple
thing to do is just to measure the magnitude
of the difference. That makes sense. But of course, that would give
us a performance function that is a function of the
distance between those vectors would look like this. But this turns out
to be mathematically inconvenient in the end. So how do you think we're going
to turn it up a little bit? AUDIENCE: Normalize it? PATRICK WINSTON: What's that? AUDIENCE: Normalize it? PATRICK WINSTON:
Well, I don't know. How about just we square it? And that way we're going to go
from this little sharp point down there to something
that looks more like that. So it's best when the
difference is 0, of course. And it gets worse as
you move away from 0. But what we're
trying to do here is we're trying to get
to a minimum value. And I hope you'll forgive me. I just don't like
the direction we're going here, because I like to
think in terms of improvement as going uphill
instead of down hill. So I'm going to dress this up
one more step-- put a minus sign out there. And then our performance
function looks like this. It's always negative. And the best value it
can possibly be is zero. So that's what we're going to
use just because I am who I am. And it doesn't matter, right? Still, you're trying to
either minimize or maximize some performance function. OK, so what do we got to do? I guess what we could do is we
could treat this thing-- well, we already know what to do. I'm not even sure why we're
devoting our lecture to this, because it's clear that
what we're trying to do is we're trying to take our
weights and our thresholds and adjust them so as
to maximize performance. So we can make a
little contour map here with a simple neural net
with just two weights in it. And maybe it looks like
this-- contour map. And at any given time
we've got a particular w1 and particular w2. And we're trying to
find a better w1 and w2. So here we are right now. And there's the contour map. And it's a 6034. So what do we do? AUDIENCE: Climb. PATRICK WINSTON: Simple matter
of hill climbing, right? So we'll take a step
in every direction. If we take a step in that
direction, not so hot. That actually goes pretty bad. These two are really ugly. Ah, but that one--
that one takes us up the hill a little bit. So we're done,
except that I just mentioned that
Hinton's neural net had 60 million parameters in it. So we're not going to hill
climb with 60 million parameters because it explodes
exponentially in the number of
weights you've got to deal with-- the number
of steps you can take. So this approach is
computationally intractable. Fortunately, you've all taken
1801 or the equivalent thereof. So you have a better idea. Instead of just taking a
step in every direction, what we're going to do is
we're going to take some partial derivatives. And we're going to
see what they suggest to us in terms of how we're
going to get around in space. So we might have a partial
of that performance function up there with respect to w1. And we might also take
a partial derivative of that guy with respect to w2. And these will tell us
how much improvement we're getting by making a little
movement in those directions, right? How much a change is
given that we're just going right along the axis. So maybe what we ought
to do is if this guy is much bigger than
this guy, it would suggest we mostly want to
move in this direction, or to put it in 1801
terms, what we're going to do is we're going
to follow the gradient. And so the change
in the w vector is going to equal to this
partial derivative times i plus this partial
derivative times j. So what we're going to end up
doing in this particular case by following that formula is
moving off in that direction right up to the steepest
part of the hill. And how much we
move is a question. So let's just have a rate
constant R that decides how big our step is going to be. And now you think we were done. Well, too bad for our side. We're not done. There's a reason
why we can't use-- create ascent, or in the case
that I've drawn our gradient, descent if we take the
performance function the other way. Why can't we use it? AUDIENCE: Local maxima. PATRICK WINSTON: The
remark is local maxima. And that is certainly true. But it's not our first obstacle. Why doesn't gradient
ascent work? AUDIENCE: So you're
using a step function. PATRICK WINSTON: Ah,
there's something wrong with our function. That's right. It's non-linear, but
rather, it's discontinuous. So gradient ascent requires
a continuous space, continuous surface. So too bad our side. It isn't. So what to do? Well, nobody knew what
to do for 25 years. People were screwing around
with training neural nets for 25 years before Paul
Werbos sadly at Harvard in 1974 gave us the answer. And now I want to tell
you what the answer is. The first part of the answer is
those thresholds are annoying. They're just extra
baggage to deal with. What we really like instead of
c being a function of xw and t was we'd like c prime
to be a function f prime of x and the weights. But we've got to account
for the threshold somehow. So here's how you do that. What you do is
you say let us add another input to this neuron. And it's going to
have a weight w0. And it's going to be
connected to an input that's always minus 1. You with me so far? Now what we're
going to do is we're going to say, let w0 equal t. What does that do to the
movement of the threshold? What it does is it
takes that threshold and moves it back to 0. So this little trick here
takes this pink threshold and redoes it so that the new
threshold box looks like this. Think about it. If this is t, and this is
minus 1, then this is minus t. And so this thing ought to
fire if everything's over-- if the sum is over 0. So it makes sense. And it gets rid of the
threshold thing for us. So now we can just
think about weights. But still, we've got
that step function there. And that's not good. So what we're going
to do is we're going to smooth that guy out. So this is trick number two. Instead of a step
function, we're going to have this
thing we lovingly call a sigmoid
function, because it's kind of from an s-type shape. And the function we're going
to use is this one-- one, well, better make it a little
bit different-- 1 over 1 plus e to the minus
whatever the input is. Let's call the input alpha. Does that makes sense? Is alpha is 0, then it's 1
over 1 plus 1 plus one half. If alpha is extremely big,
then even the minus alpha is extremely small. And it becomes one. It goes up to an asymptotic
value of one here. On the other hand, if alpha
is extremely negative, than the minus alpha
is extremely positive. And it goes to 0 asymptotically. So we got the right
look to that function. It's a very convenient function. Did God say that neurons
ought to be-- that threshold ought to work like that? No, God didn't say so. Who said so? The math says so. It has the right shape
and look and the math. And it turns out to
have the right math, as you'll see in a moment. So let's see. Where are we? We decided that
what we'd like to do is take these
partial derivatives. We know that it was awkward
to have those thresholds. So we got rid of them. And we noted that it was
impossible to have the step function. So we got rid of it. Now, we're a situation
where we can actually take those partial derivatives,
and see if it gives us a way of training
the neural net so as to bring the actual output into
alignment with what we desire. So to deal with
that, we're going to have to work with the
world's simplest neural net. Now, if we've got one
neuron, it's not a net. But if we've got two-word
neurons, we've got a net. And it turns out that's the
world's simplest neuron. So we're going to look at it--
not 60 million parameters, but just a few, actually,
just two parameters. So let's draw it out. We've got input x. That goes into a multiplier. And it gets multiplied times w1. And that goes into a
sigmoid box like so. We'll call this p1, by the
way, product number one. Out here comes y. Y gets multiplied
times another weight. We'll call that w2. The neck produces another
product which we'll call p2. And that goes into
a sigmoid box. And then that comes out as z. And z is the number
that we use to determine how well we're doing. And our performance
function p is going to be one
half minus one half, because I like
things are going in a direction, times the
difference between the desired output and the actual
output squared. So now let's decide what
those partial derivatives are going to be. Let me do it over here. So what are we
trying to compute? Partial of the performance
function p with respect to w2. OK. Well, let's see. We're trying to figure
out how much this wiggles when we wiggle that. But you know it goes
through this variable p2. And so maybe what we
could do is figure out how much this wiggles--
how much z wiggles when we wiggle p2
and then how much p2 wiggles when we wiggle w2. I just multiplied
those together. I forget. What's that called? N180-- something or other. AUDIENCE: The chain rule PATRICK WINSTON: The chain rule. So what we're going
to do is we're going to rewrite that partial
derivative using chain rule. And all it's doing is
saying that there's an intermediate variable. And we can compute how much
that end wiggles with respect how much that end
wiggles by multiplying how much the other guys wiggle. Let me write it down. It makes more sense
in mathematics. So that's going to be
able to the partial of p with respect to z times the
partial of z with respect to p2. Keep me on track here. Partial of z with respect to w2. Now, I'm going to do something
for which I will hate myself. I'm going to erase
something on the board. I don't like to do that. But you know what I'm
going to do, don't you? I'm going to say this is
true by the chain rule. But look, I can
take this guy here and screw around with it
with the chain rule too. And in fact, what
I'm going to do is I'm going to replace
that with partial of z with respect to p2 and partial
of p2 with respect to w2. So I didn't erase it after all. But you can see what
I'm going to do next. Now, I'm going to
do same thing with the other partial derivative. But this time, instead of
writing down and writing over, I'm just going to expand it
all out in one go, I think. So partial of p
with respect to w1 is equal to the partial
of p with respect to z, the partial of z with respect
to p2, the partial of p2 with respect to what? Y? Partial of y with respect
to p1-- partial of p1 with respect to w1. So that's going like a zipper
down that string of variables expanding each by
using the chain rule until we got to the end. So there are some
expressions that provide those partial derivatives. But now, if you'll
forgive me, it was convenient to write
them out that way. That matched the
intuition in my head. But I'm just going
to turn them around. It's just a product. I'm just going to
turn them around. So partial p2, partial
w2, times partial of z, partial p2, times the
partial of p with respect to z-- same thing. And now, this one. Keep me on track, because
if there's a mutation here, it will be fatal. Partial of p1-- partial
of w1, partial of y, partial p1, partial of p2,
partial of y, partial of z. There's a partial of p2,
partial of a performance function with respect to z. Now, all we have to do is figure
out what those partials are. And we have solved
this simple neural net. So it's going to be easy. Where is my board space? Let's see, partial of p2
with respect to-- what? That's the product. The partial of z-- the
performance function with respect to z. Oh, now I can see why I
wrote it down this way. Let's see. It's going to be d minus e. We can do that one in our head. What about the partial
of p2 with respect to w2. Well, p2 is equal to y
times w2, so that's easy. That's just y. Now, all we have to do
is figure out the partial of z with respect to p2. Oh, crap, it's going
through this threshold box. So I don't know exactly what
that partial derivative is. So we'll have to
figure that out, right? Because the function relating
them is this guy here. And so we have to figure out
the partial of that with respect to alpha. All right, so we got to do it. There's no way around it. So we have to destroy something. OK, we're going to
destroy our neuron. So the function
we're dealing with is, we'll call it
beta, equal to 1 over 1 plus e to the minus alpha. And what we want
is the derivative with respect to alpha of beta. And that's equal to d by
d alpha of-- you know, I can never remember
those quotient formulas. So I am going to rewrite
it a little different way. I am going to write it as 1
minus e to the minus alpha to the minus 1, because I
can't remember the formula for differentiating a quotient. OK, so let's differentiate it. So that's equal to 1 minus e to
the minus alpha to the minus 2. And we got that minus comes
out of that part of it. Then we got to differentiate
the inside of that expression. And when we differentiate the
inside of that expression, we get e to the minus alpha. AUDIENCE: Dr. Winston-- PATRICK WINSTON: Yeah? AUDIENCE: That should be 1 plus. PATRICK WINSTON: Oh,
sorry, thank you. That was one of those fatal
mistakes you just prevented. So that's 1 plus. That's 1 plus here too. OK, so we've
differentiated that. We've turned that
into a minus 2. We brought the
minus sign outside. Then we're differentiating
the inside. The derivative and the
exponential is an exponential. Then we got to
differentiate that guy. And that just helps us
get rid of the minus sign we introduced. So that's the derivative. I'm not sure how much
that helps except that I'm going to perform a parlor
trick here and rewrite that expression thusly. We want to say
that's going to be e to the minus alpha over
1 plus e to the minus alpha times 1 over 1 plus
e to the minus alpha. That OK? I've got a lot of
nodding heads here. So I think I'm on safe ground. But now, I'm going to
perform another parlor trick. I am going to add 1, which
means I also have to subtract 1. All right? That's legitimate isn't it? So now, I can rewrite
this as 1 plus e to the minus alpha over 1
plus e to the minus alpha minus 1 over 1 plus e to the
minus alpha times 1 over 1 plus e to the minus alpha. Any high school
kid could do that. I think I'm on safe ground. Oh, wait, this is beta. This is beta. AUDIENCE: That's the wrong side. PATRICK WINSTON: Oh,
sorry, wrong side. Better make this
beta and this 1. Any high school kid could do it. OK, so what we've
got then is that this is equal to 1 minus
beta times beta. That's the derivative. And that's weird
because the derivative of the output with
respect to the input is given exclusively
in terms of the output. It's strange. It doesn't really matter. But it's a curiosity. And what we get out of this is
that partial derivative there-- that's equal to well,
the output is p2. No, the output is z. So it's z time 1 minus e. So whenever we see
the derivative of one of these sigmoids with
respect to its input, we can just write the output
times one minus alpha, and we've got it. So that's why it's
mathematically convenient. It's mathematically
convenient because when we do this differentiation, we
get a very simple expression in terms of the output. We get a very simple expression. That's all we really need. So would you like to
see a demonstration? It's a demonstration of
the world's smallest neural net in action. Where is neural nets? Here we go. So there's our neural net. And what we're
going to do is we're going to train it to
do absolutely nothing. What we're going to do is
train it to make the output the same as the input. Not what I'd call a fantastic
leap of intelligence. But let's see what happens. Wow! Nothing's happening. Well, it finally
got to the point where the maximum error,
not the performance, but the maximum error
went below a threshold that I had previously
determined. So if you look at the
input here and compare that with the desired output
on the far right, you see it produces an output,
which compared with the desired output, is pretty close. So we can test the
other way like so. And we can see that
the desired output is pretty close to the actual
output in that case too. And it took 694 iterations
to get that done. Let's try it again. To 823-- of course, this is all
a consequence of just starting off with random weights. By the way, if you started with
all the weights being the same, what would happen? Nothing because it would
always stay the same. So you've got to put
some randomization in in the beginning. So it took a long time. Maybe the problem is our
rate constant is too small. So let's crank up the
rate counts a little bit and see what happens. That was pretty fast. Let's see if it was a
consequence of random chance. Run. No, it's pretty fast there--
57 iterations-- third try-- 67. So it looks like at my initial
rate constant was too small. So if 0.5 was not
as good as 5.0, why don't we crank it up
to 50 and see what happens. Oh, in this case, 124--
let's try it again. Ah, in this case 117-- so
it's actually gotten worse. And not only has
it gotten worse. You'll see there's a little a
bit of instability showing up as it courses along its
way toward a solution. So what it looks like is that
if you've got a rate constant that's too small,
it takes forever. If you've get a rate
constant that's too big, it can of jump too far, as in
my diagram which is somewhere underneath the board, you can
go all the way across the hill and get to the other side. So you have to be careful
about the rate constant. So what you really
want to do is you want your rate constant
to vary with what is happening as you progress
toward an optimal performance. So if your performance is going
down when you make the jump, you know you've got a rate
constant that's too big. If your performance is going
up when you make a jump, maybe you want to
increase-- bump it up a little bit until it
doesn't look so good. So is that all there is to it? Well, not quite, because
this is the world's simplest neural net. And maybe we ought to
look at the world's second simplest neural net. Now, let's call this--
well, let's call this x. What we're going to do is we're
going to have a second input. And I don't know. Maybe this is screwy. I'm just going to
use color coding here to differentiate between
the two inputs and the stuff they go through. Maybe I'll call this z2 and
this z1 and this x1 and x2. Now, if I do that-- if I've
got two inputs and two outputs, then my performance
function is going to have two numbers in it-- the
two desired values and the two actual values. And I'm going to
have two inputs. But it's the same stuff. I just repeat what I did in
white, only I make it orange. Oh, but what happens if--
what happens if I do this? Say put little cross
connections in there. So these two streams
are going to interact. And then there might
be some-- this y can go into another multiplier
here and go into a summer here. And likewise, this
y can go up here and into a multiplier like so. And there are weights all
over the place like so. This guy goes up in here. And now what happens? Now, we've got a
disaster on our hands, because there are all kinds
of paths through this network. And you can imagine that if this
was not just two neurons deep, but three neurons
deep, what I would find is expressions that
look like that. But you could go this way,
and then down through, and out here. Or you could go this way and
then back up through here. So it looks like there is an
exponentially growing number of paths through that network. And so we're back to
an exponential blowup. And it won't work. Yeah, it won't
work except that we need to let the math
sing to us a little bit. And we need to look
at the picture. And the reason I turned
this guy around was actually because from a point of view
of letting the math sing to us, this piece here is the
same as this piece here. So part of what we
needed to do to calculate the partial derivative
with respect to w1 has already been done
when we calculated the partial derivative
with respect to w2. And not only that,
if we calculated the partial wit respect
to these green w's at both levels, what
we would discover is that sort of repetition
occurs over and over again. And now, I'm going to try
to give you an intuitive idea of what's going on here
rather than just write down the math and salute it. And here's a way to think
about it from an intuitive point of view. Whatever happens to this
performance function that's back of these p's
here, the stuff over there can influence p only
by going through, and influence performance
only going through this column of p's. And there's a fixed
number of those. So it depends on the width,
not the depth of the network. So the influence of that
stuff back there on p is going to end up going
through these guys. And it's going to end
up being so that we're going to discover that a lot of
what we need to compute in one column has already been computed
in the column on the right. So it isn't going to
explode exponentially, because the influence-- let
me say it one more time. The influences of changes of
changes in p on the performance is all we care about when
we come back to this part of the network, because
this stuff cannot influence the performance except by going
through this column of p's. So it's not going to
blow up exponentially. We're going to be able to
reuse a lot of the computation. So it's the reuse principle. Have we ever seen the reuse
principle at work before. Not exactly. But you remember
that little business about the extended list? We know that we've
seen-- we know we've seen something before. So we can stop computing. It's like that. We're going to be able
to reuse the computation. We've already done it to
prevent an exponential blowup. By the way, for those of
you who know about fast Fourier transform-- same
kind of idea-- reuse of partial results. So in the end, what can
we say about this stuff? In the end, what we can say
is that it's linear in depth. That is to say if we
increase the number of layers to so-called depth,
then we're going to increase the
amount of computation necessary in a linear way,
because the computation we need in any column
is going to be fixed. What about how it goes
with respect to the width? Well, with respect to
the width, any neuron here can be connected to
any neuron in the next row. So the amount of work
we're going to have to do will be proportional to
the number of connections. So with respect to width,
it's going to be w-squared. But the fact is that in the end,
this stuff is readily computed. And this, phenomenally enough,
was overlooked for 25 years. So what is it in the end? In the end, it's an
extremely simple idea. All great ideas are simple. How come there
aren't more of them? Well, because frequently,
that simplicity involves finding
a couple of tricks and making a couple
of observations. So usually, we humans
are hardly ever go beyond one trick
or one observation. But if you cascade
a few together, sometimes something
miraculous falls out that looks in retrospect
extremely simple. So that's why we got the
reuse principle at work-- and our reuse computation. In this case, the
miracle was a consequence of two tricks plus
an observation. And the overall idea
is all great ideas are simple and easy to
overlook for a quarter century.
