6.5: TensorFlow.js: Layers API Part 1 - Intelligence and Learning

hello welcome to another tensorflow das tutorial now I'm very excited about this one I'm generally excited about a lot of things but in this tutorial everything that I've done so far has just used tensors operations to kind of create lists and matrices of numbers and multiply them and add them and optimize loss functions that kind of stuff now and I could keep going and there's a lot that I could do with just that alone but tensorflow digest I talked about this in the first video it has the sort of core API which has the tensors in it the operations and it's what I use to do a linear regression and a polynomial regression demonstration it also has something called the layers API and the layers API might look familiar to you if you've ever used something called Kerris because these are really Kerris layers caris is a lot machine learning library that is a higher level that allows you to create these machine learning models and underneath the hood a lower level code like tensorflow will be running so when Ted's floater chess was created it was created with both those for lower level stuff and the slightly higher level stuff and I also am working on with a lot of collaborators here at NYU ITP an even higher level library that's built on top of the layers API called ml5 I'll be getting to that eventually in this video I just want to talk about what the layers API is and its core features and so the way that I'm going to do that is by looking at a sort of basic diagram of a neural network and how you would put together that neural network with the layers API and the kind of neural network that I'm going to diagram is the same exact one that I did in my very long tutorial series about built writing a neural network all from scratch so the point this is you don't have to write it all from scratch you can just architect it with the layers API but if you want if you want to like get everything you possibly could ever get you could go back and look at some those videos if you want so what is so I'm gonna look at a simple well it's not simple but a basic feed-forward multi-layered perceptron it's gonna have just two layers so it often looks like three layers because there's also the inputs there's the inputs the hidden and the outputs that's three things but technically there's only two layers and you'll see why so let's consider that we have this neural network it's going to have inputs let's say that it has I don't know two inputs then it has a hidden layer how many nodes are in the hidden layer I don't know let's say four then it has an output layer how many outputs are there I don't know maybe we're doing some kind of classification task and there's three possibilities it's either a cat a dog or turtle so there will be three outputs you know I started diagramming the next the thing the scenario that I'm gonna do in the next video maybe as a coding challenge is I'm going to solve again XOR problem I really would like to get to some real-world applicable problems but still here in the weeds of just like I want to see how things work and use kind of trivial and known problems just to see if I can get the solution that I know I'm supposed to get okay so if this is my diagram if you if you've never seen the neural network before again you could go back and look at some other videos but the ideas are some data X I'm going to call this like x0 and x1 and that data each each of those inputs gets sent to each hidden node there are weights here the inputs get multiplied by the weights added together and then pass through an activation function and then get sent out to each of the outputs and so that happens at each one of these to draw this I'll be sure I missed the connection by the way something called dropout so I've done that correct I've dropout correct me if I forgot to draw something but I'll get to that in another video so then then we have the outputs and those are again the weighted sum of all of the outputs of the hidden layer the hidden nodes comes into the output pass through an activation function and we see their result so the idea here is maybe that I have a image with just two pixels in it and two two pixels come in they get multiplied by all these weights activated of the activation function sent to the outputs and then I get some values that tell me the probability of those two pixels were a cat dog or a turtle that would be an image classification task so how do I use how do I use the layers API to create a neural network with this exact architecture so the first thing that I need to do is create something called TF sequential the TF sequential let's go look at that in the API Docs and I'm right here all right TF win is a secretes a sequential model is any model where the outputs of one layer are the inputs to the next layer that's what it says right there guess what and I know I'm sorry that I've kind of gone a little bit too high in my writing but the outputs of one layer are the inputs to the next layer the inputs go into the hidden the outputs of the hidden go down so that's exactly what I want I should say there is also something in the layers API called TF model which I'll click on here and the key difference is that TF model is more generic so there's kind of more possibilities there but if I really just want a simple basic feed-forward neural network or the data flows in one direction between layers TF sequential will work so if I'm writing some code I would say Const model equals T f dot sequential so there we go I'm done well not exactly so that's just creating a sort of empty so all I've done is created this kind of empty architecture so what I need to do now and by the way this is what makes working with something like the layers API really powerful if you watch my tutorials where I did the whole neural network from scratch it was so much easier just to have one layer and I never kind of got to like multiple lakers cuz i have to rewrite the code and think about the layers and how they're connected and have a loop and all this layer object well guess what the layers api has this for you so i can actually just say TF now add layer so I can I can actually I can create a layer so the kind of layer that I want to create is known as dense so what is a dense layer a dense layer is the terminology for a fully connected layer meaning that every every node in that layer is fully connected to every node from the previous layer and that's exactly what I have here these are dense layers all of the connections are there so and you'll see you'll see there's like other kinds of layers there's a convolutional layer that i'll use one at some point when I talk about convolutional neural networks other things too but dense is where we're gonna get started so let's come back and look at now the API Docs for dense and I think I've opened that up here yes TF layer sorry it's TF dot layers dense so I need to say Const and I'm gonna call this hidden equals TF layers dense Const output equals TF layers dense so I'm missing a lot of but this is the idea so like I will have a model which is it's a sequential and then I have a hidden layer and output layer we'll talk about the inputs in a second and then I would just say model dot add layer hidden model dot add layer output and I'm like think I'd like to say outputs I think is it's the output layer whatever nuts to say output so this is the idea this is how in theory simple it is to build your own model using the layers API now what's missing here like I could weirdly enough let's just like run this code and see if we get any errors so it cannot read property name of undefined so okay so so who knows what that error is thing that I'm really missing here is I need to be more specific like I need to say when I make a layer what is the shape of that layer in other words how many nodes are there with the shape of the inputs how is it connected what activation function am i using those types of things so those are if we look at the API Docs the configuration of the layer so so I need to actually configure the model itself and configure each layer so let's go look at that and see if we can figure that out so let's go look again at TF sequential and you know actually I think we're going to be fine right now with there's an optional this question mark means optional there's an optional optional optional parameter config what I am going to skip that right now and yet it could be I could actually create it with a bunch of layers already and a name but I'm gonna skip that what's more important here is the sorry the the dense the config object for the dense layers which is required so that's why I'm getting error this is required so I need to specify some configuration options and this should be listed for me here units activation use bias kernel initializer bla bla bla bla bla bla bla bla and sorry for these markers here so let's start adding something what that means is I need to create an object I'm gonna call it config I could call it and here is where I'm gonna set up the configuration of this hidden layer so what I'm gonna do here is let's look at what some of these are so units so what I want is what's the unit so in this case I want to have the hidden layer has four units and so I'm gonna say units four maybe I want to specify the activation function so what activation function you use is a fascinating topic that you could go down many rabbit holes for different scenarios but and I'm just going to put sigmoid in there as kind of it for historical reasons and that's also the activation function you in my toy neural network JavaScript library but eventually as I start to build out examples I'm gonna be taking out that sigmoid and using other ones so I'm gonna say sigmoid and I'm gonna put config here I'll just say config 1 or config hidden again I could put the object itself directly in here there's lots of ways you could probably write this code in much shorter way I'm trying to write it as long away as possible to be most clear so let's do that and let's do config output and what did I say how many outputs do I have 3 and I'll also use sigmoid so I'm going to say 3 and I'm gonna say I'm gonna say config output ok so things are going pretty well now I'm gonna hit refresh here config is not defined sketch let's just line 7 oh right config hidden ooh Oh interesting amusingly I got this weird error message which makes no sense at all it actually makes sense but it's because I've got the p5 library involved here and p5 actually has a function called model so let me just write now p5 is irrelevant for this discussion so I'm just gonna comment out the p5 library and then hit refresh here now add layer is not a function so I must have imagined that this is how you add a layer to the model let's actually go and look and where would I find that out once again if I go to so here what I want to look at is all right it's a little hard for me to find things Oh No ok I know what I want to look for us I'm gonna just search for it I want to look for the TF sequential class so TF not sequential the function creates an object that is a TF not sequential so if I look at this we now look that the function is just add there's no add layer function it's just add which is a lot nicer actually so this is meant to just be add and this is meant to just be add ok so now this is really bothering me that it's calling this polynomial regression so I'm going to have to chain this two layers API explanation there we go okay uncaught error the first layer in a sequential model must get an input shape or a batch input shape argument all right so what did I miss this is what I was talking about so the inputs are technically not a layer themselves the inputs are in a way part of this hidden layer they are the inputs to that hidden layer the inputs to the output output layer are the outputs of the hidden layer so one of the things one of those properties that I have to specify is the input shape now interestingly enough it says I need an input shape or an input batch shape and what's interesting about this is what is the difference so here I can clearly say that the input shape is two there are two inputs it's just a no actually it's one well no no it's - it's a regular number to it it's confusing this is what it is it's one-dimensional it's a one-dimensional array with two spots in it but someday there might come a time where I have a data set that is just each each each each record of that data set is two numbers but I want to send in a hundred of them at once that's known as a batch so the shape might be something like 2 comma 100 but this is not super relevant for right now this is what I need to specify so if we come back to the code I should here oops going back to sketch touch is what I need to specify here is input shape - that's it again I made this up like I'm just saying my model architecture has - so now I should be able to no errors so I created that model and I can even take a look at it here there it is now you know all this stuff input layers we can see all sorts of stuff here now I there's not much there actually cuz I've forgotten a really crucial step but let me keep going there's some more stuff to discuss so the input shape is - what's the thing how come I don't need to specify an input Shea here you would think that I might after all I had to say that there were two inputs to hit it do I don't have to say that there's four inputs to outputs well the reason why I don't is because tensorflow digest the layers API can infer the input shape of the outputs layer because it has to be the number of units in that hidden layer and by the way it's you know actually naming these things like hidden and output is almost less relevant now it's really kind of like layer 1 layer 2 that sort of thing so so I could put input shape here and I would put for now let's just see if I have any errors everything's fine now what if I put like a tear am I gonna get an error look I didn't get an error that's weird now why couldn't I get an error well maybe it should give me an air I don't think so though I should get an air though I'm missing a crucial step here I actually have just set up the idea of this sequential model I have the model object I have the hidden layer I have the output layer I've added them both in but I actually haven't like plugged all the pieces into each other yet and finished it off that has to have come as I separate it's not building up the model as you're creating it or configuring it it's at the layers API so you configure it and then call a function called compile so I know I'm making lists of and this by the way was in TF dot layers dot dense but another really important function here compile is part of a sequential object ad ad was the other one if I'm keeping track of the things that I'm looking at so I add I looked at now I also need compile so let's go take a look at that so if I go to the documentation here we can see there's ad evaluate there's a bunch of things I'm looking for compile we maybe compile is not part of TF sequential oh it is it is it's just listed as part of TF model because TF sequential is based off of TF model so this is what I'm looking for this is me needing to compile the model now I need to come pilot with an optimizer and a loss function aha so if you watched my linear regression with tension flow Jas videos you might remember that I had to create something called an optimizer and the optimizers job was to minimize a loss function so the same thing the idea you know I just had y equals MX plus B now I have a more complex architecture to learn about a data set so what I want to do with that architecture is the same though I want to feed it a lot of known data and have it optimize all of the weights of all these connections to fit and this can be a very good fit that data so I need to specify those things so how do I do that so first I'm gonna make I'm gonna call this just um like config and I'm going to say optimizer is and so some options here a string or a chi F train optimizer so I think what I want actually want to create my own optimizer and I'm gonna go do that I'm by saying a constant optimizer equals TF train SGD zero point one so if you remember this is a way this is I mean this you might not have seen before if you're watching this video for the first time but this is exactly the same code that I had in my linear regression example I'm creating an optimizer from TF train the optimizer uses to cast a gradient descent and the learning rate is point one so this will be the optimizer so I'm gonna say SGD optimizer like that then what else do I need a loss function so the loss function can be I could probably to find my own loss function or I can use a string so I'm not seeing here the options for the loss function strings but let's see a loss function like basically what I want to add here is like mean squared error or something that's my loss functional I don't think that's right but let's just see now if what I would do is say my a model dot compile config so now right what is what is Ruth let's review make create the TF sequential object configure some layers add them to that model then configure the the Optima basically make an optimizer and a loss function to find those and then compile the model with those so I'm sure there's gonna be lots of errors here and there's things that I've done incorrectly let's so unknown loss mean squared error so let's figure out how do I look that up so let's see here means squared look I found it in an example ah so losses let's look at this I'm just looking for the list of them but I'm gonna I'm gonna come back I'm gonna find that and then come back and show it to you but now that I see it's actually this is just lowercase M so somewhere the documentation I want to find what the options I could put here are let's run let's now hit refresh oh no errors interesting so this is weird I'm surprised it off this is a bug or not but I'm surprised that I didn't get an error for this input shape here for the output maybe once I started feeding it data again there maybe I never would way I would clarify something really important here because I I kind of put this in here as like a demonstration of some goofiness I was trying to like figure out but the input shape so a couple things let me first let me first mention something somebody in the chat just asked me oh isn't it 12 because there's 4 here and there's 3 here and 4 times 3 equals 12 well the number 12 that I know I just sort of wrote that up too high the number 12 is an important one there are 12 connections meaning 12 Waits but this is now we're in the place of using Valerius api that is specified for us we just need to say there are 4 here there are 3 here and we need to say there are 2 coming into here and there for going into there so too is the input shape to here for is the input shape to here now I have this weird eight in my code because I was like kind of messing around like what happens if you put like the quote unquote incorrect input shape and so this really should if I want my network to be to match what I'm drawing over there this should be a four but I don't need it because it can be inferred by here and I think what I'm gonna do actually now to make this a little more readable is I think it's as much as I wanted to try to write this in a long-winded way I think I'm gonna take the configuration and just put it right here inside the creation of this layer and then I'm gonna take this object I mean I don't need to name these objects and do this I think this is actually easier to follow so now you can see I think this is easier to follow right there's and I you know so there's the model there's the hidden layer the output layer I need to specify the input shape always of the first layer and maybe what I want to do is actually say add that hidden layer then create the output it doesn't really matter what order because I'm gonna compile everything so I have the create that let's review did I already do this I'm I need to do it again I need to create the sequential object I need to make whatever layers I want configure them add them in and then I need to compile it and notice compiling it I need both an optimizer which I created one stochastic gradient descent I could use the add a blonde or any of the other ones again what these are and what the formulas are there's a lot we could go down many different paths and rabbit holes for a lot of depth here but I'm just kind of looking at the higher level point of view here and then I need to compile it with a loss function so me and squared error being one of them and I actually the all of the loss functions are listed out here this is where they are so for example so I can name it by the string or I can actually just reference the function name directly like this so this should also not give me any errors and you could see now that if I wanted to use like Oh cosine distance I heard that that's the loss function I should be using I could put that in here and I could also hit refresh again and now I'm using cosine distance so again what did different loss Czar why should use one versus the other hopefully I might get into these things as I start building examples and have to make those decisions but right now I'm just looking at how you put it together okay um so this is gonna be the layers tutorial part one I'm gonna do a second part to this because all I've done right now is I've created the model and I have compiled it the two things that I need to do with this is I need to send data through it I want to put data through it and look at the outputs what are the things I might want to do the two things I might want to do is use predict predict is a function where I give the model inputs and I get out of predict the outputs presumably I would only be doing predict after I've trained the model so I'd want to train the bottle with some training data it's finished then I can make predictions with new unknown data so how do I train the model I use a function called fit fit is a function that I can basically say I want to fit just like we had to in the linear regression example I had a lot of points and I had to find the line that fits those points I need to find all the weights of this machine learning neural network model that fit the data and guess what this is what tensorflow jess is going to do behind the scenes it's gonna do all of the stochastic gradient descent math it's going to use its own loss function everything underneath the hood so all the stuff I did in that build a neural network from scratch now is now done for us by tension flow yes and it has many more sophisticated options so that's what's coming in the next video looking at fit and predict [Music]

Channel: The Coding Train

Views: 37,412

Keywords: tensorflow tutorial, tensorflow tutorial for beginners, tensorflow js, tensorflow basics, tensorflow example, coding train, shiffman, tensorflow, machine learning, machine learning basics, machine learning tensorflow, tensorflow basics tutorial, JavaScript (Programming Language), live, programming, daniel shiffman, tutorial, coding, the coding train, nature of code, tensors, tensorflow layers, tensorflow neural net, tensorflow.js layers, layers api, tfjs layers api

Id: F4WWukTWoXY

Length: 26min 2sec (1562 seconds)

Published: Tue Jun 05 2018