Create a Simple Neural Network in Python from Scratch

hi what's up guys in this video we're going to build a basic neural network without any hidden layers to solve a problem I'm also going to explain pretty in-depth our neural network actually learns so let's start with the problem set each row is a training example of three inputs and one output we want our neural network to be able predict what the output should be for the new situation it's pretty easy so you can yeah pause the video and try to guess for yourself so yeah as some of you might have guessed when the first input is 1 the output should be 1 as well but when it's 0 the output should be 0 so yeah if you want our neural network to be able to tell that this should be one as I said before our neural network will have no hidden layers the neural network like that is called a perceptron this is what a what it looks like so we've got our inputs then the Manor on the outputs and the synapse is the synapse is our the connection between the inputs and our neuron and yeah so this is the basic architecture we can now start adding values first the input values in our case this will be either 0 or 1 after that each synapse is given a weight I'll go further into this later but for now you just need to know that weights are very important for getting the right outcome weights with either a large positive or a negative number will have a strong effect on the Norris output before starting the coding we'll set each way to a random number so these then go to the neuron and we get an output to understand what happens in a neuron we need a little math so the neuron basically gets an outputs by doing a weighted sum of the inputs so this is just you take the sum of all input values multiplied by all their corresponding sign-ups weight values at weighted sum then goes through a normalizing function so that we get an outcome that's between 0 and 1 for our normalizing function will use the sigmoid function now it doesn't matter what value you get here the corresponding value will always be between a 0 and a 1 so we now know enough to build a basic form of the perceptron in Python so let's get coding you all right so first we're gonna have to import numpy as MP then we'll yeah we'll define the normalizing function which in our case is the sigmoid function this takes in an x value or just a value and it returns any value between 0 and 1 so in Python the sigmoid function is written like this one place alright next we will define the training example so we have the inputs which is a 3 by 4 matrix and each row is a training example so NP that array of first row was 0 0 1 then we had 1 1 1 1:01 and finally zero one one okay so this are these are training inputs make sure the final training outputs these are mp3 okay so this first was zero now we get a 1 a 1 and another zero yeah this is a robot will transpose it so that it becomes a 4 by 1 matrix next we have to initialize our weights but we're gonna do this by assigning random values to the weight so it's important that you you get the same random values as I do because we're gonna need them later in the video so well seed the random numbers there which way should basically meet make sure that you and I both get the same random numbers so synaptic weights we're gonna create a 3 by 1 matrix because we have three inputs and one output value and these will be random values from minus 1 to 1 with a mean of 0 there's a lot of science that goes into weight initialization but at this point it's not really necessary to know all that so a tree by 1 and a minus 1 all right yeah so throughout the process we're going to be recording a couple things so it's important that we sometimes prints and stuff so random sorry synaptic weights are wait right so now for the main loop for iteration in we're going to define a range and we're gonna start with one because we haven't really talked about how to train a model yet but yeah stay with me it's gonna be okay okay so let's at the input layer oh yeah let's say the input layer to be our training inputs and then we get the outputs which is yeah that random formula we've seen where we take the weighted sum and then pass that through the sigmoid function so sue that sigmoid of NP that that's basically this is a product that does element-wise multiplication of matrices which is what we need so we multiply the input layer by our synaptic weights then we get the outputs for the first iteration so print outputs after training all right okay let's see what that is so I'm gonna run this and these are the values of the synaptic weights pretty random indeed and these are the outputs after training so normally after training we should get something that looks like this but as you can see yeah all right so let's clear this all up say that we take the first training example and we also have our starting synaptic weights which are these guys let's put those both in the formula so the formula was the ever weighted sum okay like that and if we fill this in we get minus 0.99 okay great if you pass this through the sigmoid normalization function we should get our first output now let's do that so minus 0.99 gives exactly what we need okay great it works of course the outputs are wrong give the weights are initialized as random numbers this is why we need a training process so this is what it looks like first we take the inputs from the training example and put them through the special formula we just saw then we calculate error which is the difference between the outputs we get and the actual output now we can adjust the weights accordingly according to the severe news of the error then we repeat the process 20,000 times so yeah here we calculate error okay so the whole process called back propagation and you might wonder well how do you how do we know by which values we have to adjust the weights now it's a good question and the answer is yeah another formula which is called the aggravated derivatives it looks like this we multiply the average which is a difference between output and the actual output with the input which is either 1 or 0 and then we take the gradient of the sigmoid function at our output all right let's clarify that so firstly you want our adjustment to be proportional to the size of the error which is why this is here and secondly we multiply by the inputs which is either a 1 or a 0 if the input is 0 the weights aren't adjusted and finally we multiply by the gradient of the sigmoid curve why let's look at this if the output is a large positive or a negative number the wage was heavy which indicates that the neuron was pretty confident in its output now we don't want to mess with weights as were pretty confident so let's look at the derivatives at large numbers as you can see these get smaller as the numbers get larger and they get larger as the numbers are smaller small numbers indicate of little trust little confidence and so we need to adjust these weights yeah accordingly with a bigger adjustment now large numbers we don't need to adjust these with big numbers because yeah confident neurons are best left alone now let's see this process in action yeah so first our inputs are given and they go to the synapses to the neuron which then calculates the output now the output to the actual output and error is calculated weighting it updated accordingly and the whole process is repeated now we can quote the first the second part okay so first we're top will will calculate average which is the training outputs minus the actual outputs now the actual output minus the Dreaming training outputs I'm sorry then we'll do the adjustments now for calculating the adjustments we remember we need to dig my derivative so let's just first define that sigmoid derivative takes in a value and returns the derivative of that side so return all right so adjustments equals the error times the sigmoid derivative of our outputs then we can adjust synaptic weights accordingly and this is done like this right again that dot products never input layer and we have to transpose this so that it matches T and we multiply that by T just meant okay now let's print the synaptic weights after training okay yeah of course the iterations we can now set them to the normal number because we actually have the training numbers all right let's do that okay and indeed you can see the outputs are pretty close to the actual outputs if you iterate even more so let's say we trade 60,000 times yeah the outputs will become like just crank that up a little 100,000 times its might take a while so we can get very very close to our output but because yeah the properties of a sigmoid function is that you never really actually get to a one unless you yeah unless you iterate an infinite amount of time and it's yeah an infinite amount of times okay now this was for demonstrating how a neural network actually learns in the next video I'll show you how to convert it's this to a usable neural network where you can provide your own inputs after get that been trained now this code is already available on github if you have to wait for next episode so check it out yeah guys thanks for watching and leave a like if you enjoyed it

Channel: Polycode

Views: 649,610

Keywords: python, programming, neural network, artificial intelligence, deep learning, AI, siraj raval, deep neural networks, developer, software development, neurons, calculus, machine learning, learning, machine

Id: kft1AJ9WVDk

Length: 14min 15sec (855 seconds)

Published: Sat Mar 31 2018