The amount of random access memory - RAM - in
a typical laptop is probably around 8 to 32 GB. That's the part that directly interacts with the
CPU. Aside from that, your hard disk might have, say, another terabyte or so of memory. But
how much memory do you - that's to say: the human brain - have? And can we even measure
it in bytes? But maybe that's not even the first question we should ask. However much memory the
brain has, where is it? Because every piece of memory in a computer has a physical location. To
access a piece of data in RAM for example, you have to know the binary address associated with
that location. In fact, for the CPU this really comes down to turning on just the right wires
to retrieve the bits at the desired location. Now imagine a different kind of memory.
Instead of specifying the *where* of a memory, its binary address, how about we could specify
the *what*, its content. A memory system where, if we provide an incomplete version of the memory,
it just sort of ... autocompletes. Of course with the right software your computer can already
do this. But it's not how computer memory works on its basic level. The point of this video is
to convince you that autocompleting memories, also known as *associative memory*, is a kind
of natural behavior of networks of neurons. With that it'll become clear that it
doesn't really make sense to measure memory capacity in networks of neurons in
the same way we measure computer memory. The biggest difference might be:
computer memories have a place, a fixed location, but as we'll see, the memories
in an associative network rather have - a time. Computer memory is measured in bits, binary
switches of ones and zeros. A string of eight such bits can represent anything from letters
to integers. For our purposes let's visualize them as patterns of this kind, like these 64 bits
representing this 8x8 image of binary pixels. I always find that there's a piece missing from the
story of bits as memory and that's the following: how do I get to a memory once it's saved, say
in RAM? Because on its own it doesn't do much. It's only when we retrieve it that it becomes
useful. So how do we? Well, broadly speaking, and I'm glossing over tons of technical detail here,
every piece of data in RAM is matched to a binary address. And this binary address really eventually
boils down to a set of wires, in this case eight, that are either turned on or off. Each piece of
data is in a different physical location and can only be retrieved by knowing its address. How the
reading and writing of memories is accomplished, is really the meat of programming and is another
story. What I want you to remember is the peculiar fact that memories are matched to addresses and
that's ultimately the only way to retrieve them. Contrast this with what we believe about the
brain. There isn't a central orchestrator like a CPU and there aren't any addresses. Rather,
there is a constant buzz of activity of many independent units called neurons. In this video,
we'll try to make some sense of the buzzing of activity of networks of neurons by introducing a
mathematical model called the *Hopfield network*, named after the author of this 1982 paper.
And as much as this has to do with memory, more generally this video aims to be a lesson
in modeling itself, which I always think of as the art of the essential. This is a neuron.
I'm almost certain you've seen something like this before. But it sometimes pays to remember
why we always come back to this when we want to understand things about the brain. The reason
is, I think, that it has a rather simple behavior: it integrates electrical signals from other
neurons to determine its own activity and then it broadcasts that activity back to the network.
Mathematically the story goes something like this: there's electrical signals coming in from other
neurons, which we will say are just some numbers. Then the synapses act as multipliers on these
signals - another set of numbers - and then the activity of the neuron is based on the sum of the
weighted inputs, and by "based on" I mean that it's fine to apply some function after computing
the sum. And that's it. So it gets interesting once we turn this into a network, connecting the
outputs of neurons to the inputs of other neurons. This is a special type of neural
network. It's a *recurrent network*, meaning that there are back and forth connections
between the neurons. I haven't drawn them but remember that any such edge is actually two edges
so that the two neurons influence each other. Okay there's details here that we need to
get into, but first, what does this have to do with memory? Well it needs to be somewhere
in here doesn't it? Where? Remember the idea of an associative memory, which is the ability
of a system to sort of "pattern-autocomplete". Let's try a definition of memory that's
slightly wider than maybe what we're used to. Let a memory system be a system that,
after having been in a certain state, a configuration, it has the ability to return
to that state later on. Now our computer memory from earlier actually has this property if
we include the CPU into the memory system. Our network seems different though. So let's get
creative. There's other things in our everyday lives that fall under our definition of memory and
one might be - and hear me out on this - a simple plastic bottle. If it's crushed, in other words
its configuration changed, it can sometimes return to its earlier state, which in that sense could be
said to have been memorized. And the metaphor is not arbitrary. I actually do think that networks
of neurons are kind of like that. What i mean is: a neural network is a system with a pattern of
activity that dynamically evolves. If, somehow, we could construct our network such that it would
have some preferred state and would return to that state over time if it was perturbed, then
that could reasonably be qualified as a memory. This is a network of 64 neurons that I
cleverly constructed such that it memorized this pattern of 8x8 binary pixels. So what's going
on? To describe what this model is actually doing, we need to take the following steps. Remember I
said time was important? We need to describe how the activity of the network changes over time. And
then there is the question of *learning*. How do we actually imprint memories into the network? And
this will have to do with the connections between the neurons. Finally, we need to understand if and
when the network converges to its memory states. The crucial ingredient of our network really
is the fact that it is a dynamical system, its activity changes over time. By activity we mean
that each of the now 16 neurons in the network is described by a number and that this number is
a function of time.Aand let's just assume that time moves forward in discrete steps. Furthermore,
since we are interested in binary memory states, we'll assume that activity means that the
neurons can only be in one of two states: inactive, say minus one, and active, say
plus one. Anyway this all leaves us with 16 minus or plus ones at any given time,
which we will call the *state* of the network What actually happens if we
increase time by one step? Imagine folding out the time dimension in space.
Now we select one of the neurons at random and update its state according to the input
of all other neurons in the network. The rest of the neurons stay the
same. And this we simply continue. But hold on you, might ask, how
do we update the state exactly and why update only one neuron at a time? Well
for the second question, we could, in fact, update all neurons at once but the issue here
is plausibility. Because that would require a global updating signal, almost like a clock,
instructing all neurons to update simultaneously. It's slightly more realistic, although not
too much to be honest, to let them update asynchronously. Okay and for the other question
- yes, what is the actual update equation? Well it's remarkably simple. It's a weighted sum
of the states of the other neurons, "weighted" meaning that each state is multiplied by the
strength of the connection between the neurons. And since the connections in that sense "weigh"
the inputs, from now on I'm actually going to call them "weights". But then of course to ensure
that the result is plus one or minus one again, we'll make use of that function I
mentioned earlier, f, and that's it. Those of you familiar with linear algebra
will have recognized this as computing the dot product between the vector of neuron
states and the vector of connection weights. This is a network of 64 neurons and I'm just
going to tell you, without explaining how, that it has memorized this pattern. Starting
it off in different initial states and then running the equations I just described, selecting
one neuron at a time at random and updating its activity - and wait let's just make this a little
simpler - we can see that the network really has this intriguing property that it gravitates
towards the memory pattern in all cases - or, well, it ends up with the anti-memory in some
cases. We'll ignore that. It has to do with a certain symmetry in the network that we will get
to. Moreover, once it's settled into that state, it doesn't change anymore. The memory
pattern is what is called a *stable state* of the network. So can we be sure that
this always happens? Well, no. For example, things start to get complicated when there is more
than one memory stored in the network. We'll get to that. but for the simple case of just a single
memory - yeah, the network will converge to either the memory or its anti-memory or, as an edge case,
to the completely inactive state. So let's recap: networks are just vectors that evolve in time and
we update their elements asynchronously. Given some configuration of weights, there are stable
states in the network which we call memory states and we have seen, although not proved, that the
network is kind of attracted to these states. That leaves the following question: how do we make
the network memorize a certain pattern? In other words, how can we design the stable states of the
network? This amounts to setting the weights of the network, which turns out to be a matrix with
as many rows and columns as there are neurons. For example, the state of this neuron is determined by
the weights in this row of the matrix. At first, this might seem impossible, especially if we wish
to store more than one memory in the network. So I'm just going to tell you the
magic rule and then I'll motivate it. Given a desired memory state with, say, eight
elements, we are looking for an 8-by-8 matrix. We will just say, seemingly out of nowhere, that
the weight between two neurons, i, j in general, is determined by the product of their states in
the memory. For all of you keen on non-linear algebra: this means that the matrix is an
outer product of the memory vector with itself. Except for the diagonal which we set to 0 since we
don't want any self-reinforcement. But why? Think of it this way: our whole approach was in some
sense to build something interesting from many simple parts. So there really should be a way for
the two neurons to determine the weight between them independent of the rest of the network.
This is a principle called *Hebbian learning* and it is exceedingly plausible because remember
that weights are supposed to be synapses? And synapses in actual neurons also would have no way
of knowing what goes on in the broader network. And so why the multiplication? Well here comes
the reason I coded the binary states minus one and plus one. If you map out all four combinations
of states of the neurons, you'll see that the weight will have positive sign whenever the
neurons in their memory state agree and negative if they disagree. And this should make sense
because a positive weight lets a neuron project its state onto other neurons and a negative weight
lets a neuron flip the state of other neurons. It's almost like the neurons behave like charged
particles, maybe ... One last question before we can see it in action, and that's - how do we store
multiple patterns at once in the same network? We do it by computing the outer products for all
desired memory patterns. This gives us a matrix for each and then we average those matrices. This
gives finer and finer gradations of the weights. Now however things start to become
a little bit more complicated. There's no way to guarantee that all
memory patterns are stable states. The memories will start talking to each other,
fusing into new memories, which I find frustrating and super interesting at the same time. Here's
the network with four memories again. It sometimes converges to one of the memory states, yes,
but in other cases it converges to something in between, a merging of memories, which I
find, well, almost a kind of human mistake. And with this we are finally making some
progress on our question from the very beginning of the video: how much memory do you
have? Originally we might have answered this by giving some number of bytes, but now the question
presents itself very differently. It's more like: how many memory patterns can we store in a
recurrent network of a given size such that they are stable states of the network? And the
answer is, at least for this model, not very many! The original paper showed that this model has only
linear memory capacity. That's to say the number of stable states grows as a linear function of
the size of the network. Plus there's a hidden assumption in this graph, too, which is that all
memory states are uncorrelated, which for any set of pictures like this is totally not the case. I
tried my best picking out some not-so-correlated images to really show what this network can do
and the results are ... well see for yourself. Realistically, that means that this model is maybe
too simple after all and that's not surprising given all the violence we did to these networks
with our many simplifications. But the goal of this video wasn't to convince you that this
model can be used in any practical sense anyway, although in a follow-up video, I want to tell
you what steps people took to make this model actually useful for deep neural networks. What
I wanted to achieve with this video is, first, to warn you of false comparisons. Networks
of neurons don't behave like USB sticks and why should they? But secondly, it's to show
you how with modeling approaches, walking a very thin line between complexity and simplicity,
we can sometimes start to conceive of the world differently than otherwise we would have. You
might have set out picturing memory as something static, but I'm hoping now you're willing to
consider that it might be something dynamic, the invisible stable states of
a network buzzing with activity. you
