10.5: Neural Networks: Multilayer Perceptron Part 2 - The Nature of Code

welcome back I'm going to actually write some code in this video not that much so what I'm doing now welcome I made a few introductory videos covered some background about neural networks and why they exist and where I'm trying to go with this and in this video I'm going to actually begin to write the code for a simple JavaScript neural network library now I've actually already done this it exists here at this repository github Schiffman slash neural - network - p5 I'm designing this library to be used with a set of p5.js examples with a library and ajosh with library called p5 although ultimately this library stands alone on itself by itself you don't have to use it with just p5 so before I can write the code let me come over here to the white board and this is where I last left off talking about how the general structure of a neural network library works neural network system works and so what I need to do here when in the code I create a neural network I want to create three things I want to create an input layer I want to create a hidden layer and I want to create an output layer so when I create a new the way I want to design this library is I want to say new neural network and I want to give it can you see this I think you can I want to give it three arguments the number of input neurons let's just use the word neurons the number of hidden neurons and the number of output neurons so I'm doing something which I typically don't do which is usually I'd like to have a specific problem I've tried to solve and like write the code for that problem and in here the problem that I want to solve is I want to make a generic kind of useful a library that can be used in a bunch of different contexts so I don't know what those numbers are going to be I don't know what the data is I'm just kind of working on the skeleton the structure of the library before I start to apply it to things so let's just make up some numbers let's say there's going to be three input neurons four hidden neurons and two output neurons what this means now is in a feed-forward neural network there are three inputs we could imagine again I'm using this kind of classic example of guessing the price of a house this could be number of bedrooms number of bathrooms square footage so those are like three parameters of a house these will connect to one two three four hidden neurons so this is the input layer this is the hidden layer and then I'm kind of running out of space here there will be two outputs and then this is the output layer so this is the configuration the idea of a and so what I'm building here is what's known as this is a multi-layered perceptron these are individual perceptron units essentially that are have multiple layers and it also is another important term that I want to add here is I want to create a fully connected Network and now there are variations to this that we might see in future examples but the idea of a fully connected Network is that every input is connected to every hidden and every hidden is connected to every output all right so I can draw all those connections and it's not so many that I you know if I were to do some kind of post production I would speed this up but I'm going to just draw this web of all these connections so every input is connected to every hidden and every hidden is connected to every output whoo looks ah I must but I'll get it eventually there you go right so you can see that every every node is connected to every node in the next layer so the idea is that those three inputs come in the data feeds forward and those two outputs come out so this is the structure now we have to get into a lot of details here well how do I keep track of all of these connections how do I actually do the loops to like do all the sums of everything how do I read the outputs I'm going to get to all that but this is the overall structure so let's go back to the code and now let's actually try to like write a little bit of this library very very little so where am I going here okay so this is the code there's nothing yet I'm going to create a new file and I'm going to call this n nsj so this is now going to be ma so here's the thing ultimately I want this to be like a proper JavaScript library but ultimately what is the JavaScript library put a file with some JavaScript in it so I'd like later as it gets more sophisticated optimize it and do some sort of like building process or break it up into multiple profiles but right now I just want to kind of get the pieces going so I am going to you I'm also going to use es5 syntax this is the trajectory that I've been 100 tuned in future videos I will start adopting some yet sixth syntax but ultimately maybe this library else will follow up and come back and kind of I'm going to do a lot of things maybe not in the most optimal or efficient way but hopefully in to look easy to understand and follow away so I want to create a constructor function called neural network okay and I should also mention again while we're here that I built this library already and when I built it I based just about everything out of this book called make your own neural network by Tariq Rashid and so while I'm doing this now kind of a bit more on the fly I'm sure everything is from my brain ultimately came from here and probably some other sources too okay so what do I want to do this core thing that I want to do is I want to create the net neural network with the number of input nodes number hidden number of output so I'm going to add those as arguments here I'm going to say number of input number of hidden number of output I'm going to create a neural network with three arguments and then I'm going to say input nodes I think I'm gonna be long-winded about this equals number of input and so I'm going to create three hidden nodes is this argument and output nodes is this argument is that an oh yes it is okay so this is we've actually written to the code the idea being that what I want to do is say things like var brain and brain is a new neural network that has three inputs with three hidden and one output right this is the idea so I need to figure out what shape and Chevys in the word shape very specifically does the data come in that's how many input knows I want what shape is the output that I want am i looking for a single output am i trying to look for a range of outputs that's how many outputs I want then how many hidden neurons do I want well that's kind of an open question well maybe I want as many as I could possibly fit in well the program running reasonably fast but it sort of depends on the complexity of the problem we'll come back to that later and I should also note that I this is a synth oversimplification of how neural network architectures can be this is by definition a 3 layer Network and this library is only going to allow for a 3 layer Network an input signal hitting and an output but as some of you might think about for the future how would you write the code to have multiple hidden layers because a lot of neural network based learning systems need multiple hidden layers to be able to perform optimally but for now I'm going to keep things very simple ok what is the next step rittenberg we did write some code thankfully roads look good now we got to stop again the next step is the feed-forward process the way that the feed-forward process works is that we receive these inputs so much - - so many pieces of this puzzle I'm excited to get through it all though so let's just say for example we're looking at this hidden neurons you remember from the perceptron videos maybe didn't watch those a lot let's talk about it the idea is that we need to do something called a weighted sum so let's pretend this is the house prediction thing and this was the number of bedrooms three this is the number of bathrooms you know this is the number of the square feet so each one of these connections right the data is going to flow in the data comes in here the number three comes in here and then look at this there's four outgoing connections each one of those connections has a weight to it now ultimately the whole point of doing this learning neural network based learning system is we want to tweak those weights we want to train the brain training the neural network to have optimal weights to get good results results that make sense and that training process the other thing that I'm going to get to I don't have any videos down the road for now but not too far away these weights will typically just start one way of thinking about that is they're going to just have random values between negative 1 and 1 and there's a wide variety of techniques and strategies for initializing random weights or not just random - a neural network but for right now a good way for us get started they all have random weights so even though I'm looking at each one of these flowing out slightly better way for me to look at this with you is actually just look at all the connections flowing in so this particular hidden neuron has three connections flowing in a three and the input values of three - in 1,000 one of those has a week so let's pretend this is like point five let's say this is like you see negative point five and this particular weight is one so I may be using very very simple numbers the idea is that each hidden neuron does something called a weighted sum so it takes the input multiplied by the weight and add that to the other input multiplied by the weight and adds it to the other input multiplied by the weight so we could actually do this 3 times 0.5 is 1.5 plus 2 times negative point 5 is negative 1 plus a thousand times 1 is plus a thousand so this value now is a thousand point five now we can see there's a huge flaw here which is that the fact that square footage is kind of a big number and number of bedrooms number of bathrooms are small numbers means this kind of way of something is going to produce some like odd results this the square footage is going to be weighted so heavily just by the fact that it's bigger numbers so a lot of time in working with a machine learning or neural network based system we need to do some type of cleaning or normalizing of the data we might do something where we you know we sample this down so the you know we actually do the number of bedrooms between 0 & 5 as a value between 0 1 and number bathroom is always the value between 0 1 square footage this would actually turn into 0.1 like because the range is between 0 and 10 thousand square feet or something so we would do some kind of normalization of these values but this is again further down the road when we start to apply the library in an actual project once this weighted sum is complete so result of that weighted sum gets sent out through the outgoing connection but it gets passed through an activation function so I'm going to come back to the activation function this is something we did with a perceptron and that's going to be a separate video where we look at different activation functions and they work right now I want to focus on this weighted son so I could keep going here I could create some type of array of I could create an object that's like each one of these nodes or neurons is an object I can iterate over I could have a connection object so there's a bunch of different approaches I could take but the classic and standard approach is actually to look at storing all of these weighted connections in something called a matrix which is really just like a spreadsheet a grid of numbers looking at the inputs as an array and doing some type of math that basically takes take that array of inputs multiply it by that matrix of weights and generate the outputs of this hidden layer so this is so give me a second here I'm going to erase I'm going to I'm going to I'm going to make the case for this with a simpler scenario so let's look at this diagram which just has fewer connections can be easier for us to unpack so we can think of these inputs as x0 and x1 let's not even worry about the output right now these are the inputs this is the hidden layer right hidden layer so let's think about this and actually let me change these numbers to X 1 and X 2 you know sometimes I like to count from 0 sum was like to count from 1 I don't know why but I feel like in this case let's look call it 1 and 2 so this is really like hidden one hidden to so each one of these connections right each one of these weights you could say here this is a weight that goes from 1 to 1 this is a weight that goes from 1 to 2 this is a weight that goes from 2 to 1 and this is a weights right here that goes from 2 to 2 so notice how there are two inputs two hidden neurons for wait in other words the weights I'm going to draw I'm going to kind of you start to use matrix notation a little bit the weights can be expressed like this 1 1 1 2 2 1 2 2 ok so this is a way of expressing the links and a way of expressing the inputs I could write it like this X 1 X 2 okay so I'm making the case that I have two inputs and I have four weights and I could write it out like a matrix of numbers a two by two matrix and this is essentially a 2 by 1 matrix whenever I'm going to get more into matrices in the next video or my in that video already I don't even remember head of where I am but typically when we talk about a matrix a grid of numbers we reference it rows by columns 2 by 2 2 by 1 ok so let me just show you something remember this we need a weighted sum here this weighted sum is X 1 times weight 1 1 plus X 2 times weight 2 1 ok that's the weighted sum for this neuron or node the weighted sum for this neuron or node is X 1 times the weight from 1 to 2 and X 2 times the weight of 2 to 2 plus X 2 times the weight of 2 to 2 it so happens I could take these two results I could call this like h1 and call this h2 and I could say let me actually think I could I could basically say this times this equals h1 h2 so this is the actual math the way that we described it look at both inputs coming in multiplied by their weights and sons look at both inputs coming in multiplied by their weights and some these are those it written out but it so just happens that this exact math writing it like this and producing this outcome is exactly the math that is part of the field of study called linear algebra linear algebra involves manipulating vectors and matrices a vector being a one-dimensional list of values a matrix being a two-dimensional list of values the inputs are always one-dimensional the outputs for always one-dimensional the weight are always can always be expressed as two-dimensional zitz every input connected to every pin you can think of it very much like pixels every row and every column so this is where I need to stop and what I want to do is do a few videos that cover this notation and math with a bit more detail writing a little JavaScript simple JavaScript matrix library and ultimately once we've done that we can come back here and see how we have that library written we can then use it to do the math between the inputs and the hidden and the hidden to the output and ultimately later it's going to go backwards through the network to tweak values and train it and that's we're also going to use the same matrix math so this is why we need or why we don't mean because it can kind of do it without it but while it's useful to work with this idea your algebra and I should note once again that if we were doing this in something like Python using a library like something called numpy we would get all this stuff for free and there are JavaScript matrix libraries and might but I'm going to kind of unpack some of this and write a lot of code from scratch just we have a sense of how it's working so hopefully as you were watching the video you saw a little annotation this is actually incorrect I mean everything about this math is correct this matches this right the weighted sum is x1 times weight 1 from 1 to 1 x2 times weights from 2 to 1 but actually the notation I the way I wrote this matrix as we go as I go into the next video where I actually look at how the matrix math works this really should be written in 1 2 and this should really be written as 2 1 the reason why that is is this should be x1 times w1 1 plus X 2 times w2 1 which is written right here so that matrix massive I'm going to go ignored you tell in the next video we take this row and multiply it by this column and this row and multiply it by this column and you can see that's what these 2 things are ok so thanks for bearing with me I there's a lot of little pieces but I am going to get back into the code so in the next thing I'm not very confident about the order I'm doing all this in but it's just the way that I'm going to choose to build it and so the neck again I'm saying this again the next video I'm going to look at the matrix math again and then write a generic library that does that math and then come back and put it back into the neural network itself ok so see you in the next video thanks [Music]

Channel: The Coding Train

Views: 126,789

Keywords: live, programming, daniel shiffman, creative coding, coding challenge, tutorial, coding, challenges, coding train, the coding train, nature of code, artificial intelligence, itp nyu, neural network, intelligence creative coding, neural network artist, intelligence and learning, machine learning, perceptron, multilayered perceptron, neural network intro, XOR neural network, neural net, XOR perceptron, perceptron javascript, multilayer perceptron, multilayer neural net

Id: IlmNhFxre0w

Length: 21min 29sec (1289 seconds)

Published: Fri Jun 30 2017