Build a Deep CNN Image Classifier with ANY Images

have you ever wanted to build your very own deep image classifier well in this tutorial we're going to do exactly that let's do it [Music] [Music] what's happening guys my name is nicholas tronat and in this tutorial as i mentioned we're going to be building a custom deep image classifier using your own data now the nice thing about this tutorial is that you can literally pull down any bunch of images from the web and load it into this pipeline and you'll be able to use it to classify images as a zero or one binary classification type problem now in this tutorial we are going to be very much focused on going through the end to end pipeline so first up what we're going to do is focus on getting some data and loading it into our pipeline we're then going to take a look at some pre-processing steps that we need to perform in order to improve how well our model performs then we're going to build a deep image classifier using keras and tensorflow so we'll build a sequential deep neural network then we'll evaluate it test it and last but not least we'll save down this model so that you can take it elsewhere and use it where you need it ready to do it let's get to it hey nick i was hoping to build a deep learning model to classify some pictures is that something that's possible what are you trying to classify uh it's part of my stealth tech startup we're looking to build a cutting edge mobile app to classify youtuber thumbnails as wait for it shocked face or duck face i'm gonna need you to sign an nda now that you know that so you like the idea right well my other idea is to use ai to classify images of people as happy or sad okay that that we can do we can use image classification to do it nice how do you get started well the first thing we probably need is to get some images of happy and sad people then we can use some of the helpers inside of keras and tensorflow to load it up so we doing this or what yeah let's go alrighty guys so we are going to be going through an end to end image classification deep learning pipeline thing wow that got really long really quick no so we are going to be able to build our very own deep image classifier using pretty much whatever data you want in this particular tutorial this is sort of why i wanted to show you this so first things first what we're going to need to go on ahead and do and this is what we discussed with our client is we need to go and set up and load some data and then in order to do that we are going to do a couple of key things first up we're going to install our dependencies and set them up we're also going to we are also going to remove any dodgy images so i found that sometimes when you download images from the web i.e images the versus images that you haven't captured yourself you can tend to get some dodgy images or images that are corrupt so i'm going to show you how to remove those we are then going to load that data so those are the first couple of things that we need to do so first up installing dependencies in setup and as per usual all the code that you see inside of this including the entire jupyter notebook is going to be available inside of github just check the link in the description below so first things first we need to install dependencies and setup so what we can do in order to do that is run this particular line here so the first line is exclamation mark pip install tensorflow tensorflow gpu opencv python and matplotlib so tensorflow and tensorflow gpu are going to be used as part of the deep learning pipeline itself so we are going to use the keras sequential api in a second to be able to go ahead and do that opencv is going to be used when we go and remove our dodgy images and matplotlib is going to be used to visualize our images so if we go and run that that is going to go ahead and install our dependencies so we we scroll on down looks like we just got a warning and this is just telling us that we should probably update our pip version that in this particular case it's fine we don't need to cool then we can go and validate whether or not we've got the right dependencies installed so if we go and run pip list let's go and take a look so do we have tense flow so we've got tensorflow and tensorflow gpu so we've got 2.8.0 and 2.8.0 if you haven't watched the beginner setup tutorial which i just released make sure you go and visit that particularly if you've got a gpu and you need to set up tensorflow with gpu acceleration i go through it step by step so it's going to make your life a ton easier okay so the next thing that we need to go ahead and do is import some dependencies so in order to do that we're going to import two key dependencies the first one is tensorflow so import tensorflow as tf and the second one is os so we've written import os so os is namely used let me minimize that so os is mainly used to navigate through file structures so typically when i'm using os i'm using it for os.path dot join so let's say for example we need to go into our data directory and we needed to go into a folder called happy right so what this is going to do is return back a file structure which is uh what is it homogeneous regardless of what type of operating system you're using so this is if you're using it on a windows machine it's going to be perfectly fine if you're using it on a mac or a linux machine it's going to return the folder structure in a format which is appropriate for that particular type of operating system now the other main type of function that i use inside of os is os.listia so let's say for example i wanted to show all of the um all of the files or all of the images inside of a particular directory i could actually type in os.listia and let's say i've got a folder called data it's going to list everything that is inside of that particular folder now i don't have anything in there but later on you'll actually see that in use just a couple of key tips on what i use os for all right next line we are going to go on ahead and limit the limit tense flow using up all of the vram on our gpu so by default when you load in tensorflow or you load in data it's going to expand and use all of the potential vram that's available on your machine which tends to become a bit of a nightmare particularly if you're getting into really big models so these two particular lines of code actually help prevent that and it helps prevent something called an oom error or an out of memory error so first thing that we're doing is we're actually grabbing all of the gpus available on our machine so the first line is gpus equals tf.config.experimental.list underscore physical underscore devices and to that we've passed through gpu right so if i actually go and write that line so gpustf.config.experimental dot list physical oh gosh my typing is the shocker devices and then if i type in gpu this is going to give us all of the gpus that we've got available right so you can see i've got one there if i had a giant machine that had multiple gpus you'd actually see multiple there so if i type take a look at the length of that particular line you can see we've only got one gpu available there we typed in cpu you can see again i've only got one cpu available right so this particular case returning my cpu so if you wanted to see all the devices you've got available you can filter through on that so that is what that line is doing there then the next line is actually limiting the memory growth so we're looping through every potential gpu that we've got inside of this line here so four gpu in gpu so for each of the gpus inside of this we are going to run tf.config.experiment underscore memory growth so this is telling tensorflow hey don't use up all of our memory keep it to a minimum or keep it to whatever you absolutely need so by limiting this it ensures that we're going to avoid any of those out of memory errors you might still get some if you've got batch sizes that are too large but that is a solvable error cool so we have now gone and installed some dependencies imported a couple and we've also gone and limited our memory growth this next line is just showing you exactly what i showed there so you don't need to run that but if we go and run that that is going to limit our memory growth so you saw that it wasn't run there i'm just going to run it so we are now good so that is step 1.1 now done so we've gone and installed our dependencies we've got an importer whatever we need and we've also gone and limited our memory growth over here now the next bit is removing dodgy images now before we actually do that we actually need to get some images so the nice thing about this tutorial is that i wanted to build it so that you could actually go and build an image classifier for just about anything you wanted now the nice thing about this is that we can actually go and get a bunch of images from the world wide web so we're going to be building an image classifier an image sentiment classifier in that particular sense so what do we need to do we need to get some images of happy people so let's get images of happy people so i'm just going to go to google go to images and type in happy people and you can see we've got a bunch of overly ecstatic people over here they're so happy who knows what about we're gonna go ahead and download these now i've actually got this extension inside of google chrome which makes my life a ton easier when it comes to this i don't actually know let's what's it called uh more tools extensions it's this one over here download all images so if you go and get that particular extension this is going to be a game changer when it comes to building deep learning stuff right because what we can do is from here i can open up this extension so you can see it says download all images there yeah i can click that and that is going to start downloading all of the images from this page so you saw it just literally when it did that so inside of this zip folder we got all those images so if i go to downloads open this up take a look at that images are happy people how cool is that pretty awesome right so now we're now able to build up a deep learning image data set really really quickly so we're going to grab that particular zip file so i'm just going to cut that and i'm going to take it into my image classification repository so i haven't actually shown you this let's do that so this particular notebook is here and inside of this same folder i've got a directory called data a directory called image classification which is just my virtual environment again if you haven't seen the beginner tutorial go and jump back on over to that because i'll show you how to set this up i've got a folder called logs because that's where we're going to log stuff out we've got a folder called models as well now we are going to grab this data that we just downloaded and put it into our data folder and i'm just going to rename it and call it happy just double check that is the right thing yeah that's cool happy people so happy all right so we are going to go what have we gone and done so we've got a set of image data which is for happy people we now need a set of data which is for sad people kind of like when i don't get to order pizza or beer on friday night so we've got a bunch of people that are sad so again we're going to go to our extensions download those images so you can see it's downloading we're good we're great that is now downloaded over there so again we're going to go to our downloads we're going to grab that repository paste it over here and we're going to call this one assad so we've now got inside of our data folder we've got a 2-zip so we've got a happy zip we've got a sad zip so we're going to extract these now so if we go and extract these and we're just gonna create or extract them to a separate folder so we're gonna extract too happy let me zoom in because that's way too small happy we're gonna extract a happy i know sometimes my code is pretty small guys i'm i work to improve it i promise so we're going to extract a happy you can see we've now got a folder called happy with all of our overly happy people right cool we're going to go and do the same facade and this has gone what's that pop up this is saying the following imp file already exists let's just say yes to all okay so we've now got two folders we've got a whole bunch of happy people yeah cool great now inside of our sad folder we've got a whole bunch of sad people this guy's clearly had a big day at work this guy has as well all right so what we're going to go ahead and do is uh what are we doing so we're going to take these zip folders and we're actually going to move them back into this root folder so that way they don't sort of mess around because when we go and load in our data where the keras utility is just going to look at the subfolders that it needs in here cool so i'm going to close that i'm going to close that i'm going to close it well i've opened up a ton of images we don't need that one open either all right cool so what have we gone down so we're going to download a bunch of images out people a bunch of images happy people and we've now gone and put it inside of our data folder if you wanted more classes you literally just go and download those classes paste them in here you do need to change the neural network a little bit if you want a tutorial on that hit me up in the comments below and maybe we'll make one okay so we've gone and got our image data sets we've got one of happy one of sad now we are kind of good-ish to go so the next thing that we're going to do is we're going to go and remove dodgy images now let me explain why we're doing this so sometimes when you go and get images from the world wide web even though they have the appropriate file extensions they might not open up in python because they might be corrupted they might be mislabeled they might be mis-extended or have the incorrect extension applied this particular block is going to help us get rid of that so first thing that we need to do is import two dependencies so we are importing opencv so this is uh what allows us let me actually show you so if we type in opencv it's a huge open c or open computer vision module that allows you to do a ton of computer vision and good stuff so this is super awesome if you've never checked out opencv by all means do the next thing that we are importing is image hdr so this allows us to check the file extensions for our particular images so two lines there so import opencv and import image hdr the next thing that we're going to do is create a variable to hold the path to our data directory so remember inside of here we've got a data directory which is called data and inside of our data directory we've got two folders one called happy and one called sad right so we're going to go and create a variable which points to that so data underscore dr equals data then we are going to go and create a list of extensions let me minimize that we're not talking about that yet too early so image extensions are jpeg jpg bmp and png so this is just a standard list so if we go and navigate through this uh let's actually run that first so image underscore exts right i haven't typed that in image right it's literally just a list so if we actually go and grab the first one that's jpeg the second one that's jpg the third one is going to be png that's actually the fourth one cool so the reason that we're setting up this is we are now going to loop into it so remember how i told you about import os so now what we can actually go ahead and do let's check this out so if i type in os dot list dr and type pass through data deal this should return happy and sad it returns the folders which are inside of this data directory if i go and run that happy and sad pretty cool right now what would happen if i wanted to go and check every single image inside of the happy folder well i could go os.path.join pass through my data directory and then pass through let's say for example a happy folder now i'm going to return every single image inside of that subfolder pretty cool right so this allows us to loop through every single image that we've got inside of that folder which is exactly what our dodgy image script is going to do in a second but i just remember there's one additional thing that we need to go on ahead and do so before we actually go and run that image script normally what i'll do when i'm getting images from the interwebs is i'll actually go into that data set and just anything that is small or weird i'm going to remove that so anything which is under 10 kilobytes which is probably going to be a really small image and all of these weird microsoft edge vectors we're going to get rid of those so from nine kilobytes onwards so you can see let me zoom in so you can see these are nine kilobytes eight four three so pretty small we're gonna get rid of those so from here to here delete those and if you wanted to add in additional images so go and do like go to another page download more go to another page download more you could definitely do that okay so we've gone and removed the small and weird images from happy let's go and do the same from sad let's go to our details tab so again we're going to find 10 kb and then we're going to delete everything below that you can see 9 we're going to go all the way down we're going to remove anything smaller than that got to get rid of that quick assist it's driving me insane all right so down here let's take a look nine kilobytes one kilobyte remove those boom done all right cool so that leaves what effectively should work but we're still gonna run our dodgy image script as well just to make sure that we don't have any weird stuff there so remember how i said i talked about os.listia that's exactly what we're doing so first up we're looping through every folder that we've got inside of our data directory so for image class inside of os.list dr so this is just going to go inside of our data directory and it should effectively just print out happy inside so but let's actually show you this or let me actually show you this so print image class image image underscore class so we're going to print out happy and we're going to print out sad then what we're going to go ahead and do is we're going to print out every single image inside of those subdirectories so if we go and print that so if i go and print image we are printing out or looping through every single image and then we're just going to do a bunch of checks right so if you wanted to do a specific check you could go and do that i'm just double checking that one we can load the image into opencv and two that our image matches one of these paths over here so if we've loaded it up and it's still weird we're actually going to remove it from that particular folder right so what we're doing is for looping through every single one of our data directory folders so happy and sad we're then looping through every single image that we've got inside of those sub directories so in order to do that we're in four image underscore class in os.listia four image.os or for image in os.listia and then we're going into those subfolders so we're joining the path so os.path.join our data directory and then our image class so this would effectively be um happy and then well no this would be data and then it will be happy and then it will be data and then it will be sad and then we're going into that and we're grabbing every single image explicitly so we're going os or path.join the data directory the image classification folder and then the image itself and we're storing that inside of a variable called image path and then we're passing that through to opencv.imread so this actually opens up an image or it allows you to open up an image using opencv so let's actually test that out right so if i type in cv2 dot i'm read and then we need to pass through the full file path so we're going to pass in os.path.join because it just makes it easier we're then going to pass through our data folder and then let's say for example happy and then let's go and get a file from happy so go to uh image classification and then data happy let's grab this one looks pretty big and we're going to pass through the string so there's a jpg so we need the extension as well the image classification data happy jpg cool dot jpg we need to close that so you can see we've just gone and read in our image as a numpy array so if i type in let's just store it inside of variable so i can show you type image it's a numpy array now what does actu this actually mean so basically let's take a look at the shape so we have an image of the shape what is that three so this guy should be rows first so it's going to be 3744 pixels high 5616 pixels wide and it's got three channels that means it's a colored image so if we actually go and open up that image for d well let's actually use our python capability so we can type in plot.i am show and then pass through our image and we have we not imported matplotlib we have not let's import matplotlib we let's just import up here so let's just add in another line so um from we're probably going to visualize it later but i want to show you now so from matplotlib import pie plot as plt let's go and run this so that is our image there so you can see 3 000 what is it 744 pixels high 5616 pixels wide and we it is a colored image and you're probably wondering nick why is it a weird color this is because opencv reads in an image as bgr matplotlib expects it to be rgb so if we wanted to fix this we can type in cv2 dot cvt color pass through the image and then pass through a color conversion code so this is just going to reorder the channels so we can type in cv2.color underscore bgr2rgb boom yep that should work and there you go so you can see the color is now fixed and we can if we wanted to get rid of this weird line here you can just type in plot.show with that so pretty so again so this particular line effectively shows you one how you can read an image using opencv how you can join paths using os.path.join and how we can go and render those okay but this is just sort of explaining so we don't need this we don't need this we don't need this we don't need this and we don't need that what we want to do is actually go and run through our dodgy image script so again if it's not a valid image not a valid image extension we're going to go and get rid of it using os.remove so os.remove allows you to delete a file so let's go and run this and you can see that this is inside of a try accept block so effectively if it throws up any errors then we're going to fail out of that particular block so let's go and run that see we've got some weird images it's removing them and that is done so now let's just double check we still have a bunch of images inside of our data folders so if we go into data happy still got a bunch of images there so 131 to be exact and inside of sad we've got 74. so it looks like it's a bit of an unbalanced data set so we could definitely fix this up or resample if we needed to but for now we'll be okay cool so we've got a bunch of images we've gone and cleaned them up we're looking good so let's take a look at what we've done so far so we've now installed our dependencies successfully we've also removed any dodgy images the next thing that we want to go on ahead and do is load our data now how are we going to do this well tensorflow actually has a data set api and i originally didn't use this a ton when i was just getting started on my deep learning journey but now i probably use it every single time i'm building a deep learning model why well it actually allows you to build data pipeline so rather than loading everything into memory to begin with it actually allows you to build a data pipeline which one allows you to scale out to much larger data sets but it also gives you a repeatable set of steps that you're going to apply to your data to be honest just makes up a lot cleaner um there's a whole bunch of documentation on this now if you wanted to actually access it you can type in tf.data.dataset and it actually allows you to access this tf.dataset api now we're actually going to use it slightly differently so rather than using the data set api directly we're actually going to use a keras utility which actually allows you to do it but for example if you've in some of the two trials that i've actually got coming up namely iris tracking and which one other one did i use it in i think for the toxic comment detection i actually use it so typically you can load in a data set using um tf.data dataset from what is it from tensors from generator from tensor slices but you can also use it and load in data from a directory by typing in list files so this actually allows you to do a wild card search i just wanted to give you a little bit of background on this but um we're not going to be using it directly in this particular tutorial but there is a ton of capability in this uh tf data set api okay that is the data set api talked about now we've also taken a look at its uh documentation so also pro tip if you've never haven't used jupiter too much if you type in question mark question mark gives you the documentation for that particular line of code or that method which you can see there so next thing there are so we're importing matplotlib there so i imported re-imported it up here but i wanted to show you that image visualization so we are now importing numpy so import numpy as np and then from matplotlib import pi plot as plt so if we go and run that now that's two new dependencies imported and then this is where we're going to load our data so remember how i told you that we've got this data set api here well keras actually has a data pipeline direct function or helper built into it as well and this is how we access it so in order to use it you can type in tf.keras.utils dot image underscore data set underscore from underscore directory and this actually builds an image data set for you on the fly so you don't need to build the labels you don't need to actually build the classes it's going to do it for you and it's actually going to do a bunch of pre-processing out of the box so it'll resize your images as well now let's actually take a look at the doco for that nope we don't want to take a look at that yet so if we go to tf.keras.utils utils dot image so it is actually going to batch your images into this batch of 32 it's going to resize your images into 256 by 256 it's going to shuffle them you can also create a validation split so on and so forth but it's pretty cool right so it actually allows you to do a ton of stuff right off the bat so this is a super useful helper if you're building image classification models so we're going to get rid of that so we're actually going to create this data set now so i've just gone and run this line here so we are then going to store our data set into a folder called or into a variable called data so this means that we can now take a look at the data itself now the nice thing about the data part or well actually it's nice but it's also a little bit painful so if i let's say for example i wanted to go and take a look at this data i can't just go and grab the first instance because this isn't a data set which is pre-loaded into memory already it is actually a generator so on the fly we actually need to go and grab the data that we want now the easiest way to do that is to convert it into a numpy iterator first which is what this next line is doing so we are going to go and create a data iterator so data underscore iterator equals data dot as numpy iterator so you probably would have seen me do this before in the face verification tutorial i actually did it there so this is going to convert or allow us to access the generator from our data pipeline so data underscore iterator equals data as numpy iterator then we can actually get consecutive batches using the dot next method so i'll show you this in a sec so if we go and run that now then our next line of code is actually going to run dot next so this is actually going to get us a batch back but first up let's take a look at our data iterator so you can actually see it is an iterator so also like a generator so you can loop through and continuously pull data batches back right and again this is probably not hyper useful right now but when you get to significantly more sophisticated deep learning models where you've got massive amounts of data gonna make your life a ton easier but for now let's actually take a look so we are then going to grab one batch so if you think about it this way so this is building our data pipeline this is actually allowing us to access our data pipeline and then this is actually accessing the data pipeline itself so this is allowing us to loop through it this is grabbing one batch back so if i go and run this line this batch let's actually take a little batch before we visualize it so if i type in batch so this is actually a batch of data so this should have a shape of two or it should have two variables on two parts of the tuple now you're probably thinking nick what the hell is this number two well there's two parts to this data set there is the images and then there's the labels so the first part is actually the image representation so it's our images from our directory loaded into memory as a set of numpy arrays so if i go and grab that first value we've got a whole bunch of arrays here yeah can you see that now if we take a look at the shape we should have all of our images loaded so if i type in dot shape so we've got a batch size of 32 and remember this is based on the image underscore data set underscore from underscore directory helper method we can configure that if we wanted more images per batch if we actually wanted different image shapes but this is the nice thing about that particular utility it automatically reshapes our images to ensure that they are of a consistent size and it also batches them up into a batch size of 32 but again you can configure all of this so let's say for example um i go let's actually go and grab it so tf.keras.utils.image dataset from directory so if i wanted more images per batch i can increase that let's say for example you don't have as much vram on your gpu you could actually drop that batch size and the way that you would do this is let's say that that's your data directory you would type in batch underscore size equals i don't know 16 for example yeah or you'd go batch size equals eight so you can change that if you wanted your image size to be different you could actually pass through that keyword argument so image and the score size equals i don't know let's say 128 by 128 so it would actually allow you to configure what you want your data set to look like for now the defaults are going to work cool all right so we're taking a look at our batch um so we took a look so these are the images right so this is the images images represented as numpy arrays now let's say what about the labels so if we go to batch z one this should be the labels there you go so in this particular case we've got a flag of one we've got a flag of zero zero zero zero one one one now these actually represent the labels so one is going to represent either happy or sad and zero is going to represent either happy or sat i couldn't quite work out whether or not there's a way to dictate which class belongs to which number now i know there is a way i think there is a way that you can actually configure this i know for sure that you can do it using the tf.data.dataset api because i do it a ton um with this default helper i'm not too sure actually how you actually go about configuring that but there's one easy way around that you just double check which class is assigned to which type of image and this is what this plot is going to allow you to do here so over here we can see that the one flag is actually assigned to sad images and the zero flag is assigned to happy images so you can see one there one sad person zero happy people zero happy people zero happy people now let's say for example we wanted to visualize another batch of data well we can actually go and run this line over here again because we are going to get another batch get another batch from the iterator it's director if we go and run this again our batch size is not going to change but our classes so you should see these numbers change over here right so if we go and run this again you can see that we've now got a new batch of data from our data pipeline and just for our own purposes let's write a note so class one equals sad people people and then class zero equals happy people and if we go and run our visualization again you're gonna see that so zero happy people one sad people one sad people one sad people who is that is that timberland kid cardi i don't know um but over here what we're doing is we're actually using matplotlib and matplotlib's subplots function to be able to plot out four images at a particular time cool but that is our data set now brought in so we've actually got that now read in let's uh let's quickly review what we've done so we have successfully installed a bunch of dependencies and set them up we've removed any dodgy images we might have and we've also gone and loaded up our data using the keras image data set from directory helper let's jump back on over and have a chat to our client and see what's next all right so now that we've got some data loaded what's next we've got to get pre-processing what pre-processing for image data we tend to pre-process by scaling the image values to between 0 and 1 instead of zero to 255 this helps our deep learning model generalize faster and produces better results cool anything else we've got to do yep we're also going to split up our data into training testing and validation partitions to ensure that we don't overfit let's get to it and we are back so now what we're on to doing is pre-processing our data so we've got our data loaded but now we've got to do some pre-processing and this is where step two comes in so there's two things that we're going to do here we are going to scale our data and by that what i mean is that when we loaded in our data over here actually let me tell you so if we actually go to our data so if i type in batch and remember our batch is composed of two parts our images and our labels so our images are going to be in key 0 and our labels are going to be in key one if we go to or index one whatever you want to call it uh if we go to the first thing then this is going to be our images so if we type in dot shape all of our images all right so we've got 32 images of shape 256 by 256 by three channels now when you load in an images of representation of a number of channels which are going to be rgb for opencv it's going to be bgr when you use tensorflow i think it's rgb but it's going to be of the values 0 to 255. so if i type in dot min does that work yep so this means our lowest value is 0.0 and dot max should return 255. now when you're building deep learning models ideally you want the values to be as small as possible this is going to help you optimize a ton faster so what we can do is really quickly is all we really need to do is divide these values by 255 and that is going to give us values between zero and one now so if i type in uh certain let's say um scaled so if i type in scaled and then we type in dot min so the last value is still zero the maximum value now one so this means that we've successfully scaled our data but remember that we are using a data pipeline so we can't necessarily go and do this every single time we load up a batch what we want to do in order to do it most efficiently is just do it as we're loading in the data through the data pipeline using the data pipeline capability so let's actually go and take a look at how we might do this inside of the data pipeline we've actually got a function called map and this allows us to apply a particular type of transformation as our data is being pre-processed through the data pipeline now this means that when we go on pre-fetch data it's going to do that transformation as well so this means that it speeds up how quickly we can access our data from our disk so what we've got in written is data equals data dot map this is really really important guys pay attention the data.map actually allows you to perform that transformation in pipeline so data.map and then we're using a lambda function to be able to go and do that transformation so when we go and access our batch remember we're going to get our images and our labels so x is going to represent our images because those are our what is it our independent features so we are going to be passing those in y is going to be our target variable so that is what you can see there or effectively our labels what we're going to do is that as we load in a batch we are going to go and get our data over here so you can see we've got x and we're going to divide it by 255 that is our scaling and we are going to form no transformation on our y so if we go and run that we've now effectively applied that transformation step on our data pipeline now there are a ton of additional um transformations you can do inside of a data pipeline or a tensorflow data set pipeline right and if you just go to tens flow data set api i know tensorflow data api yeah tf.data right there are a ton of functions so if i go to tf.data where is it here over here so tf.data.dataset there are a ton of functions so over here you can see that there are an absolute ton so we are using the map function which you can see there and again this allows you to do a whole bunch of stuff across elements within a data set exactly what it's saying there there's a whole bunch of other ways that you can use it we're using it in this particular manner over here cool cool the other ones that i tend to use a ton are zip so let's say for example i wanted to combine a set of features and labels zip oh sorry just slap the mic zip is a way to combine those um skip you'll probably see that in a second what's another one from generator tensor slices from tensors these are pretty common as well but i'll include a link to this inside of the description below so you can see it as well but that is effectively our data scaled right so now if we go and take a look at a next batch you can see that we are doing that inside of our data pipeline so this is no different to what we did from up here so rather than going and creating separate variables from our iterator we can actually access our data that like that as well so data dot and then dot as numpy iterator so that gives us access to our iterator and then dot next is going to grab our next batch now keep in mind this is going to if we're not applying shuffling it's going to return back the exact same values each and every time in this case we've got shuffling so you can see that our data is changing let's just grab our first set of at images so again we can grab our images by grabbing the first index if we type in dot max you can see them wait hold on that is a very low max dot min zero dot max 0.03 that seems very low let's grab another batch uh so we can let's call this scaled equals scaled iterator data.as numpyre and then if we go scale to iterator dot next these all look awfully low maybe we've just got uh images which aren't all that bright or aren't that colorful let's go dot max we need to grab our first value again grab the next batch seems awfully low let's actually take a look at it using our using our plotting function over here so if we grab that so let's grab a batch so that's our data then we can grab the iterator and then let's just grab create a batch here kind of weird that they're so low batch equal scale data.next all right so data equals that this is going to be our scale iterator grab a next batch what does this look like they're all black something has gone wrong there let's go and reload our data so grab our data set and skip that bit let's go and run through that pipeline wait hold up so because we've gone and divided our images by 255 they're no longer going to be integers but right over here inside of our visualization function we are converting them to integers but in this particular case they'd all be 0 because they're between 0 and 1. so if i divide this this is now on a set of our scale data right so data's now got a lambda function let's just go and reload our data over here this is reestablishing our data pipeline we're then going and running it through our lambda then if we go and take our next batch let's go through our scale data set this is going to be our scale data perfect so if we go and type dot as type into right that's going to turn them all black no go colored let's just double check so batch okay so that looks a little bit better now hold on so if we go batch zero dot max okay one dot min i don't know what's happening there but that looks like it's okay good good thing that we're double checking that this data is appropriately function maybe we went around that data pipeline twice so we're going and dividing by 255 twice that's the only thing that i can think of there but if we go and reload or re-establish our data set by going and running this line over here and then if we go and apply a lambda again go and test this out right so max is one i'm in is zero now if we go and visualize perfect all right cool so that is our data now scaled so we know that our images are now between zero and one the next thing that we want to go on ahead and do is split our data into a training and testing partition this means that when it comes to actually going and validating our data and ensuring that our model hasn't overfit we've actually got specific partitions so in order to do this we are first up going to establish what our training data sizes should be so let's just quickly take a look at the length of our data in its entirety so we have seven batches and remember each batch is going to have 32 images so there's probably one that's going to be truncated if it's not equal batches but that is the number of batches that we've got so our training set is going to be 70 of that data so that should be what uh 50 what is it 49 so it would be like what five batches or seven times point seven right so we'll have roughly five batches there then we're going to have 20 assigned to validation so that'll probably be one and then this will probably be one as well let's just see what this comes out as so how big is our training size four so it's rounded down how big is our validation size one and how big is our test size zero so all right so we should probably add one to that so that means we'll have four plus one plus one that's only going to give us six so let's add an additional one to our validation partition so train is going to now be four batches validation is now going to be two batches and test is now going to be one batch so train plus or train size plus vowel size plus test size equals seven which equals this right so we're good to go now let me quickly explain what we're doing here so our training data is going to be what is used to actually train our deep learning model our validation data is going to be what we use to evaluate our model while we're training so this means that our model hasn't necessarily seen our validation partition but we're using it to fine tune how we actually build this deep learning model and it's good practice to have a training and validation partition as you're building up these models your test partition your model is not going to have seen until we get to the final evaluation state so we hold that out all the way until the end so these two are using are used during training this one is used post training to do the evaluation and right now all we're establishing is how much data we're going to allocate to each one of these partitions so we are going to have 7 multiplied by 32 image or wait hold on we're going to have four multiplied by 32 images allocated to our training partition we're going to have two multiplied by 32 images allocated to our validation partition and we're going to have one batch assigned to our test size now uh we don't need to take a look at that in order to do this we actually can use the take and skip methods available inside of the tensorflow dataset pipeline so if i actually show you this so over here you can see we've got take and skip so take defines how much data we are going to take in that particular partition so if we go and run this line here so our training size so uh len is for batches aval data is two batches and our test data is one batch so in order to do this first up what we're doing is we're saying train equals data dot take and then we're saying how many batches we want to allocate to our training data now keep in mind our data is already shuffled if your data hadn't been shuffled you want to shuffle it before you do this so train is going to equal four batches so train equals data dot take and then to that we're passing through train size val is gonna equal data dot skip so first up what we're gonna do is we're gonna skip the batches that we've already allocated to our training partition and then we're gonna take the last two into our valve data so data dot skip and we're going to skip the first four which we've allocated to train and then we're going to take two based on the vowel size so take two and then we are going to take two batches and that is what our validation partition is going to look like then our test is going to be everything left over right so test equals data dot skip and we're going to skip the training data and the validation data and then we're going to take the rest which should be our test partition and that is how you establish your train test and validation partitions and that is the last step in our pre-processing stage as well so we've now gone and scaled our data and i sort of showed you how to actually go and apply that lambda layer or the map layer and we also saw that we had a bit of an issue there right so we saw that our data wasn't necessarily scaled and this is potentially because we went and applied this twice now in order to fix that all you need to do is go and re-run this line over here and go and re-run this particular line here and again we're overwriting variables so again that can be a little bit sketchy in terms of how we can actually trace that back so you might actually go and choose to call this um scale data so that way you can't override it twice but in this case we are good so our data is scaled our data is now between 0 and 255 we've gone and taken a look and visualized it and we have also gone and created our train validation and testing partitions let's jump back on over to our client all righty onto the good bit now that our data's pre-processed we can begin modeling i've always told my mom i was destined for the catwalk what uh yeah what we're going to do now is build our deep learning model using the keras sequential api nice so this is the ai bit right right come on gisele let's do this all right so we are now on to the good bit actually doing the deep learning so in terms of doing this there are going to be three components so first up we are going to build a deep neural network and in order to do that we're probably going to need to import a bunch of stuff then we're actually going to go on ahead and train the specific deep learning model in order to do this we can actually use the dot fit method and then we're also going to plot our performance using matplotlib as well okay so first things first let's go and import some dependencies using tensorflow so first up what we're going to be doing is we are going to be importing the sequential api so in order to do that within from tensorflow.keras.models import sequential now the reason that i make this distinction is that there are two really specific models or two specific like model building apis available inside of tensorflow and keras the first one is called sequential and that's what we're going to be using this is great if you've got one data input and one data output and the model sort of just flows from top to bottom there is another type of api and that is the functional api and that is really really powerful if you've got like multiple inputs multiple outputs multiple connections you need to do a whole bunch of fancy stuff with deep learning models so um i've definitely got one of those coming up for you a little bit more advanced but we do have that coming up so sequential is great if you need something quick and easy and that's what we're going to be doing in this case for deep learning classification then the next thing that we're bringing in are a whole bunch of layers so the layers that we've brought in are available through from tensorflow.keras.layers import conf 2d so this is a convolutional neural network layer so if we actually type in conf 2d into google and we don't want the pi torch equivalent we want the tensorflow equivalent so this is a 2d convolutional layer eg spatial convolution over images which is exactly what we're doing so we're bringing in a convolutional 2d or conv 2d layer a max pooling layer which effectively acts as a like think of it like a condensing layer so it actually goes through your images and actually condenses it down so it goes what's the max value in this region only return that region so rather than returning all of the data from our convolution we actually condense it down then we're bringing in dents so this is a fully connected layer available through keras we'll also bring in the flattened layer so this is what allows us to go for a convolutional layer which has channels or kernels namely channels and it actually reduces it back into a format which our dense layer will be able to take so that we can only get one output at the end and then we've got dropout which is typically used for regularization but i don't think we actually use that in this but anyway we've got a whole bunch of layers so typically when you're building neural networks you're going to form an architecture and an architecture is a whole bunch of different layers or hidden layers which form a deep neural network so in order to do that first up we are going to create or let's actually go and import that first up we are going to create a model and we are establishing that as so so model equals sequential now i'm going to show you the way that i typically do this but you can always um pass through the layers inside of the sequential class so i could actually do conv 2d here and then add in all the stuff and then just sort of stack these as so but i typically use the add method and actually sort of chain them on when i'm using the sequential api but you'll see people do it like that as well so um that's just the thing to call out if you see people doing using the sequential api like that perfectly fine this is just a matter of choice so we are establishing our sequential class or an instance of our sequential class so written model equals sequential and a set of parentheses now we're going to add in our layers this is where the magic happens so we have let me break this out we're not actually using dropout we could actually get rid of that would still work over here so we've got three convolution blocks we have a flattened layer and we've got two dense layers so let's break this up so this first section over here is performing or adding a convolutional layer and a max pooling layer so model dot add and remember how i said that you can either stack them or pass them direct to the sequential class or you can use model.add to add layers now i'm adding this sequentially so this means that the first layer in my deep neural network is first up going to be an input so we whenever you're passing through a first layer that first layer needs to have an input or either needs to be an input layer now again i'll talk about that more when we do functional models later on but know that our first layer is a convolution here so model.add and we're adding a convolution and this convolution is going to have 16 filters that is what a convolution has so it's got a whole bunch of filters and it basically scans over an image and tries to condense or extract the relevant information inside of that image to make an output classification so we've got 16 filters and our filter is going to be three pixels by three pixels in size and we are going to strike have a stride of one this means that it's going to move one pixel each time so it's gonna go bang bang bang so on and so forth now you can increase that stride you can increase the number of filters you can increase um the size of the filters these are all what we call architectural decisions now you might have actual like state-of-the-art models which actually prescribe how you do this so if you actually go and take a look at how imagenet does classification there is a specific architecture that is used there how we actually go and change these particular hyper parameters or these particular model parameters influences what our model performs like but typically when you're doing a convolution you have the number of filters the size of the filter and the stride another really important function or another aspect of it is the activation that gets applied so here we are applying a relu activation so let me show you this a relu activation really all we're saying is that we're taking the output from this convolutional layer and we're going to pass it through a function that looks like this so that means that any output that was pre previously below zero is now going to be converted to zero and we are going to preserve the positive values so this effectively means that we are converting any negative values to zero and anything that's positive remains unchanged this allows us to take into account a non-linear patterns right so if you've just got or if you've got no activations and really all you're building is a deep neural network which is linear in nature so it's not necessarily that powerful this is where the power comes from applying all these activations there are a ton of them right so another popular one that we'll use as well is a sigmoid activation and that looks like this now these can sound really complex but just think about it as that we're taking all of our data or all of the output from a layer and we're passing it through this function to modify what the output looks like right so we might have a whole bunch of numbers between um i don't know negative infinity and positive infinity or like really really small or really really big values but by going and actually changing it i'm passing it through one of these activations we're reshaping what that output looks like right and we're reshaping what the function looks like as a result of that as well cool so what's the full line i've gone through and talked about this a ton but so we're going to read a model model.addconf2d so we're going to have 16 filters of shape 3x3 and then we are going to have a stride of 1 and a pass through a relu activation we are also going to specify what our input shape looks like now remember that i said that the keras tense what is that image data set from directory function actually reshapes our images well our images are going to be in the shape 256 pixels by 256 pixels by three channels so they're going to be 256 pixels higher by 256 pixels wired by three channels deep and again we pass this through into our initial we just passed it through in our first layer here then we're applying a max pooling layer so this is just gonna take the maximum value after the relu activation and it's going to return back that value so it's going to scan across and go it's effectively going to condense the information it's not just going to take one number it's actually going to do it over a set region so what is this let's actually take a look what's the default max pooling 2d question mark question mark so it's going to go over a two by two region so you can see two by two two by two and it's going to take the max value out of that two by two region so it's effectively going to reduce our image data i think by half in that particular case so rather than having well yeah half based on the rows or half based on the rows half based on the width as well so it's going to condense that information down and then we've effectively got another one of those blocks over here so again the next set of laser we're adding so model.addconf2d so we've now got 32 filters of shape 3x3 of stride 1 and again we've got relu activation now the max pooling rate so then we've got another convolution block then we've got exactly the same over here this time we've just got 16 filters and then what we're doing is we're actually flattening this data down so when we actually apply a convolutional layer the filters are going to be the last channel so we're going to condense the rows and the width and then the number of filters will form the channel value now when we actually go and pass this to our dense layer we don't want that channel value right because we want to condense it down to a single value so by flattening it that's what we're effectively doing so then we're now going to have 256 values as our output and then finally one value as our output here so if we actually go and run this oh well let's actually talk about this in a little bit more detail so we've gone and flattened it then the last two layers that we apply are dense layers so these are fully connected layers so if you've ever seen a neural network you see a lot of little dots and they're all connected and they form go down to a point these are referred to as fully connected layers so inside of keras we call them dense layers so we've got dents we have 256 neurons and again we're applying relu activations here and then our final layer is a single dense layer this means that we're going to get a single output and that output is effectively going to represent 0 or 1 because we have a sigmoid activation so if we take a look at what our sigmoid activation does let's zoom in our sigmoid activation takes any output and converts it into effectively a range between 0 and 1. now as 0 and one map to our happy and sad classes so if it is zero that is i think what i can't even remember what it is now so if it's zero it's going to be a happy person and if it's one it's going to be a sad person that is what our final layer is going to output there so you can see that there cool right and that is our deep neural network so if we go and run that layer that is successfully run the next thing that we need to do is compile that so this is the next most important bit in your neural network so we type in model.compile and remember our model was initially initialized up here and then to that we pass through a few things we pass through the optimizer that we want to use and in this case we're going to be using the atom optimizer there are a ton of optimizers so if we type in tf.keras actually is it tf.optimizers dot there are a ton so we've got ada delta adagrad adam adam max uh ftrl i haven't seen that one too much uh n atom rms propped sgd so again there's a ton of optimizers that are actually available so in this case we're going to be using the atom optimizer then we're specifying what our loss is and our loss in this particular case because we are performing effectively a binary classification problem it's going to be tf.losses.binarycrossentropy and the metric that we want to track is accuracy you can pass through a bunch more in this particular case but our accuracy is going to tell us how well our model is classifying as either 0 or 1. so if we go and compile it this is my favorite bit so by typing in model.summary you can actually see how our model transforms our data so remember our model takes in an input of shape 256 by 256 by 3 right 256 by 256 by 3. our first convolution layer converts it to be 254 by 254 by the number of filters so that 16 is coming from up here then we apply a max pooling layer so it converts it from 254 to 54 by 16 to 127 by 127 by 16. that 127 remember how i said because it is shape 2-2 it's taking 254 divided by 2 which gives us 127. so it effectively halves the output that we're getting out of this convolutional layer there but it is not a trainable layer right so you can see that there's no parameters assigned to it so it shouldn't effectively increase the size of our model too much it just condenses stuff down then we're going and applying another convolution layer so again this one's going to 125 by 125 by 32 now we could actually preserve the size from over here by applying some padding but i'm not going to delve into that for now so we're going 125 by 125 by 2 and again we've got 32 filters that is coming from over here and then we are applying max pooling again so it's going to be halving our data then we've got our last convolution block 60 by 60 by 16 we're playing convolutions 30 by 30 by 16 and then we're getting our flattened layer right so we've got 14 400 values let me enable scrolling so i can show you this so what we're doing is we're taking the outputs from this max pooling layer and we're converting it into a single dimension so if we type in what is that 30 by 30 by 16 that is the number of outputs which are passed to our flattened layer so you can see that we're getting all of these dimensions or all of these elements here that multiply by that multiplied by that gives us the number of inputs that are going into our flattened layer so we're condensing them down into a single dimension so rather than having a multi-dimension or multi-rank tensor we're converting it down to that then we pass it to our dense layer which is 256 uh what is it neurons which is over here and then we finally go down to a single layer over here a single output layer cool now if you're probably wondering why this is 257 parameters keep in mind there's a bias term as well right so it's it's going to be the value of the weights from your neurons plus the bias term that gives us our total number of parameters so three million six hundred ninety six thousand six hundred and twenty five values so it's pretty uh it's not monstrous right you've probably seen like bert or big bert like those are massive models gpt3 huge as is 3.7 million now we could probably condense that down as well if we really wanted to but that particular case that is our deep neural network now modeled so we can get rid of this get rid of that that is our neural network that is step 3.1 and now done so we've now gone and built our deep learning model right and so we've taken a look at our imports we've taken a look at some of the different types of layers that we'll use and we've also seen how we can stack these layers up now the next thing that we want to do is actually go on ahead and train so first thing we're going to create a log directory and we've already got this created so you can see over here i've got a folder called logs so we are just going to create a variable which points to that and then we're going to create a callback so these callbacks are really really useful if you wanted to save your model at a particular checkpoint if you wanted to do some specific logging which is what we're going to do now in order to do this we're creating a callback so tensorboard underscore callback and we're setting that equal to tf.keras.callbacks.tensorboard and we're specifying the log directory equal to this logs folder over here so when you go and do this you're going to have your tensorboard logs logged out i'm going to talk about this more in a future tutorial but we're setting it up now so tensorboard underscore callback and we're passing through this parameter here so this is going to log out your model training as its training so if you wanted to come back and see how your model performed did it vary at a particular point in time do we need to drop our learning rate you're actually going to see that inside of these boards but we can actually just plot them using the history that you collect from your training step which i'll show you in a sec cool so that's our callback established now we're going to go on ahead and fit our model so there's two really really important methods when it comes to building a neural network model.fit and model.predict so fit is the training comport component predict is when we actually go and make predictions so model.fit is going to take in our training data and remember our training data was four batches of 32 images each our epochs is how long we're actually going to go ahead and train for and one epoch is one run over our entire training set of data we're also going to pass through our validation data so this means that after we've gone and performed a or gone and trained on all of our training data or training batches we're then going to run evaluation on our validation data so we can actually see how well our model is performing in real time then we're also going to pass through our callbacks so if we wanted to go and apply additional callbacks or let me know if you want a tutorial on callbacks guys i'm more than happy to do it so here we're actually specifying that we want to log out all of the information from our model to tensorboard so we could open it up inside a tensorboard later on um and by storing it inside of a variable called hist or history we're going to be able to take out all of the training information from our training data and our validation data and plot them out inside of step 3.3 so if we actually go and run this now this is going to kick off our training run so let's run it and you'll actually see our deep learning model start training so it might take a little while to start but eventually you'll start to see it training so it's going really really fast because we've obviously got a gpu but that is now training you can see it's taking what 377 milliseconds per step our loss is going down so ideally you want to see your loss go down and your accuracy go up and so what you've got over here so this is the loss on your training data this is your training let me zoom in because you probably can't see that this is the loss on your training data this is the accuracy for your training data this is your validation loss and this is your validation accuracy so ideally you want to see them or you want to see your losses decrease both pretty consistently and pretty steadily and you want to see your accuracies go up pretty steadily as well over here so you can see it's definitely performing way better so look our losses drop down significantly and our accuracy has got up to a hundred percent so again it's performing really really well by the end and that was it so it went really really quickly that's how deep neural network trained guys that's how quick it goes so um this is obviously with a gpu so it's a lot faster with out a gpu i think we saw roughly a double increase in the training time so again you'll still be able to do this without a gpu but that is our deep neural network now trained the magic is done now cool thing is because we've gone and saved it inside of or saved our training history inside of a variable called history we can actually get a whole bunch of information from this so if i show you hist there is this shows us that callback so if i type in history history there is a whole bunch of information available in here let me scroll so you can see that we've got our loss information we've got our accuracy information so that's on our training data also got our validation loss information and we should also have our validation accuracy information which you can see there pretty cool right but we can actually plot this performance out which is what i've got here so here i'm just using matplotlib so plot.plot and we're grabbing our training loss and we're grabbing our validation loss and we're plotting those so here what you're going to see is that our training loss has the color teal and our validation loss has the has the color orange let's go and plot it out so you can see that our loss has sort of decreased pretty steadily over time we have we've had one bit of a spike there but it's sort of come back if you start to see your loss going down and your validation loss sort of rising up that that is an indication that your model may be overfitting so it might be time to look at applying some regularization might also mean that we need to apply some data or change some data if you don't see these decreasing at all or if you see your green line like not decreasing or going weird this might mean that you need to take a look at your training data again maybe potentially consider a larger neural network or a more sophisticated neural network because that means that it's not able to learn or reduce the loss for that particular data or that training data overall so it might mean we have a bit of a bias problem okay so um if you the validation law starts tearing off that might mean you have a variance problem so again regularization is your friend in that particular case okay so we've got our loss metrics so visualize those uh now we can also take a look at our accuracy metrics so again this sort of little block here is pretty useful because you can actually visualize just about anything that you want and it's really really useful when you're doing deep learning as well because if you added more metrics so let me go scroll up if we added in more metrics when we're compiling our model you'd actually be able to visualize all of them so in the iris tracking video that i'm working on as well i actually show how to do that for a whole bunch of metrics okay so that is our last visualize now we can also visualize our accuracy so again we're using matplotlib to visualize these so if we go and run this one you can see our accuracy steadily increased over time we had a little bit of a pop out there but again we've come back and it has resolved back up to 100 accuracy which is very very good right obviously this model is performing well there's not a ton of data so ideally you'd want to flesh this out with a ton more data but in that particular case that is our model now trained so we've actually gone and completed our modeling step so we've gone and built our deep learning model and we've used this sequential api we've gone and trained our model using the model.fit function and i've also shown you how to pass through your training data your validation data as well as setting up a callback and we've also gone and taken a look at our training performance so we've visualized our loss and we've also visualized our accuracy let's jump back on over to our client and we might be on to our evaluation step so we've finished training our model now it's time to test it out of course wait what does that mean well think of how you might test an athlete you put the athlete through its paces and review certain metrics so you might measure a sprinter by their 100 meter dash time for our classification model we'll do the same ah got it so we're going to check how fast our model is not quite we've got a few different metrics we'll use for classification these include precision recall and accuracy let's jump to it so we're in the end game now evaluating performance so we've gone and trained our model the next thing that we want to do is actually go and evaluate performance on our testing data because remember we held out that partition that our model has never seen before so there's two things we want to do first up evaluate it and then we'll also grab some random images and test that out so in order to evaluate we're going to import a couple of key metrics so from tensorflow.keras.metrics import precision recall and binary accuracy these are different measures that you typically use for classification problems so i'm going to import those and then what we can do or in order to use them we need to establish instances of them and then what we can do is update our state as we actually go and make predictions so we can go and instantiate those i've written pre and these are pretty crappy variable names i think i was running out of energy so pre equals precision and again we're grabbing this class over here creating it re equals recall and then accuracy equals binary accuracy then in order to go and test them out we're going to loop through each batch in our testing data which i think we only had one batch anyway right the len test nope lower case one batch anyway so four batch in test dot as numpy iterator so that's going to bring back our batch and then we can unpack it so x comma y equals batch so this is going to be our set of images this is effectively our y true value then we're passing through our image data to our model so model.predict is how we make predictions and that is going to return back a set of values between zero and one because remember we've gone and passed it through a sigmoid activation so in order to update our metrics we can then type in or use the update state method so precision.updatestate and we pass through our y true value and our predicted value of i've gone and screwed that up so pass through our y true value and our y hat value so you can see those there and we've gone and done that for precision a recall and for accuracy so we'll then be able to see how it's actually performed on our test data so if i go and run that it should run for a little sec and then we can actually print out those results so to actually make this a little bit easier to read so uh f equals uh precision result so this is precision or ecr and i think we can write dot numpy as well and uh what is this this is recall dot numpy and this is accuracy my head blocking that off nope okay got num hi uh what have we done we haven't finished it gotta close that boom okay i should close that over there all right so there we go so we've now gone and printed out our performance on each of those different metrics our precision is one so a higher precision means your model is performing better a recall is one again high value on recall means you're performing better and that accuracy is one so again higher value means you're performing better now the each of these metrics are between zero and one so this is the highest possible value it can take um now again if you want to deep dive on how these metrics are calculated or let's say for example wanted to do a confusion matrix let me know and i can delve into those but just know that this model is performing pretty much as well as it can on a single batch of data and considering how little data we've actually given it to perform so we've got a precision of 100 a recall of 100 and an accuracy of a hundred percent as well now normally i don't like to leave it there i like to actually go and test it out on data that's outside of that batch so let's actually go on ahead and do this now the first import that we're going to import is cv2 but i thought we've already imported opencv we have we should have we imported it when we're doing the ht yeah we've already got that so we can actually skip that we don't need to import opencv but for now let's just import it anyway i'm wasting time what we're going to do is we're going to read in an image that our model has never seen before so what we're going to do is let's go and grab a random image so let's actually go to page 2 or scroll further down let's grab this dude so i'm going to save this image and i'm going to save it into our image classification folder we're going to call this sad test and let's saved it as jfif i don't know why google does that so frustrating sad test dot jpg is it yes okay cool then we're gonna get an image of a happy person happy peoples find a happy dude so this is completely out of sample right our model's never seen this before let's go this one's great all right let's save this image so happy um that's webp that's not gonna work so again it needs to be jpg it's just gonna make your life a whole bunch easier let's grab this one happy test dot jpg okay cool so i've got two test images so if i open up my folders so you can see why they're not there there we go so we've got happytest.jpg so this is that girl that's super happy and we've got a sad test but this dude dark and stormy is red ready to drop a beat all right now what we're going to do is we are going to read in that image using cv2.iamread and so first up let's read in a happy image so we can type in happy test dot jpg so that should read in our image and we should plot it out right now remember this opencv is going to read it in as bgr rather than rgb do we fix that here no we don't let me show you how to do it cv2 cvt color and then we're going to go and pass through the color conversion code cv2.color underscore bgr2rgb and that should visualize it in its correct color so there you go so we've gone and fixed up the color there it's gone and successfully loaded it now remember when we pass through our data to our neural network and need to be in the shape 256 by 256x3 so 256 pixels high 256 pixels wide and it needs to be three channels so what we can actually do is use tf.image.resize to resize our image before we pass it through to our neural network so if we actually go and run this that is what our image looks like now resized now again we could go and apply the cv2cv2.cvt color method over here and grab this paste it here and we are throwing an error that is because it is an integer from that doesn't look like it's playing nicely anyway we've gone and transformed it now so with uh tensorflow so you can see that we've definitely gone and resized it just because we've read it in using opencv we're getting that weird coloring but let's actually go and test this out now so if we actually go and pass this through to our model so model.predict and then we're passing it through to our neural network so now quick word on what how we're actually doing this prediction our neural network expects us to pass through a batch of images not a single image so what we actually need to do is we need to encapsulate it inside of another set of parentheses or arrays or put it inside of a list so in order to do this and you'll see this done quite a fair bit is we can type in np.expanddims pass through our image which is right now called resize because we've gone and resize it using tf.image.resize pass that into there and if i type in what axis i want to apply the extra dimension you can see that right now it's just stored inside of an extra dimension so let me show you that the first value you can see that we've got a whole bunch of extra dimensions there this is the first value uh let's scroll it so you can see that we don't have an extra set of parentheses there by throwing it in there literally all we're doing is we're encapsulating it in another set of arrays or putting it in inside of another list so if i actually show you the shape now dot shape it's 1 by 256 by 256 by 3. if we take a look at the shape for this it's 256 by 256 by 3. so we're literally just wrapping it in another set of arrays now what we're also doing at the same time is we're dividing about 255 to scale it so if we go and run this take a look at y hat so 0.28 so remember what was a sad or happy person happy person was zero which you can see there so happy is zero sad is one so in this particular case our model has successfully predicted that this particular person is happy now the reason that we're making this assumption is if a particular in our binary classification problem what we're saying is that 50 is our cutoff point so if it's below 50 we round down to zero then this particular case we're saying that our person is going to be happy so it's successfully classified that particular person we've got y hat twice there we don't need that now the way that you can extrapolate this or actually explain this in in regular terms is that our predicted class is happy if it is less than 0.5 so if y hat is greater than 0.5 the predicted class is going to be sad otherwise the predicted class is going to be happy which is exactly what we've said over there so because it's 0.28 it is less than 0.5 which means that our model has successfully predicted it as happy pretty nuts right like we didn't have a ton of data we didn't even train for that long and we've successfully gone and built a deep neural network that performs classification but let's actually go and test it out on our sad person as well so in order to do that we just need to pass through a different image so this one's going to be sad test and if we go and run this so our model has gone and given us a probability of 86.8 which means it's going to be above 50 which means we have predicted sad successfully take a look at that predicted class is sad so really quickly we've gone and been able to perform or build a deep neural network that is able to perform sentiment classification using nothing but a couple of images we've collected from the web let's go jump back on over to our client and give him one last final update alright home run baby you know it the last thing we need to do is save our model so we can reload it at a future date so this means another developer or engineer could use the model right right this could also be deployed as an api or to an edge device let's finish this alrighty we're at the final step so saving the model so this is a really important step so you've gone through all to this and this amount of effort the last thing that you want to do is save this model so you can go and use it again if you did want to now this is relatively straightforward we can import or bring in the dependency from tensorflow called load model so from tensorflow.kerastop models import load model so then what we can do is actually use this load model function to be able to load up our model now first of all we actually need to do save the model and to do that we can just use model.save now we're going to save it inside of the folder models and we can name it just about whatever we want so we could call it um happy sad model and because we're saving it as a h5 model what we're actually doing is something called serialization so we're taking a model and we're serializing it onto something that we can store as a disk so this is similar to what you might do when you zip a data set um when you go and wire a data set h5 is a serialization file format so it basically means that you're going to have a file called happysadmodel.h5 and then we can reload that using the load model function up here so the full line is model.save and then to that we're passing os.path.join and we're going to be saving it inside of our models folder which i've already got created over here nothing in there at the moment and we're going to be saving it as happy sad model.h5 it could be whatever you want it to save it as right if i go and run that inside of our models folder you can see we've now got our happy sad model.h5 file over there so we're looking good now the next thing that we want to do is actually go and reload this model so let's actually go and rename it so we're going to call it new underscore model and we can use load underscore model to load that model back up now we need to pass through the full file path to our saved model in order to reload it so we can actually grab this h5 file name here pass it over to here so we are basically saying it's going to be inside of the models folder so models and then happysav model.h5 so we're basically going to give it this file path to be to the load model function to be able to load it back up so if we go and run this now this is our new model it's a new model is our sequential keras model and if we go and pass data to it so again we're going to pass through our resize image so np dot expand dims passing through our resize image which we're scaling and we should get a prediction again you can see we're getting a sad prediction if we go and run this again so let's call this y hat new grab this block over here so what we're going to say if y hat nu greater than 0.5 you can see still predicting sad that in a nutshell is how to build a deep neural network for image classification so we've gone through a ton of stuff in this tutorial let's quickly recap so we first up went up and set up our image or our environment to be able to set up and load our data so we're going to import a bunch of dependencies we went and removed some dodgy images and we went and downloaded them originally from google now remember you can get that image extension which makes your life a whole bunch easier so what was it called um download all images it's just a google chrome extension nothing fancy there it just makes your life ton easier when getting images we then went and loaded our data set using the image data set from directory method we went and pre-processed our data so we scaled it and we split it we then went and built our deep neural network and went into a bit of detail as to how we've actually constructed this architecture we then went and trained it using model.fit took a look at our performance over time we went and then finally evaluated it over some new images that we've got off the web and last but not least we've gone and saved our model down to our disk so we can bring it back when we need to that in a nutshell is this tutorial done thanks again for tuning in guys peace thanks so much for tuning in guys hopefully you enjoyed this video if you did be sure to give it a big thumbs up hit subscribe and tick that bell and let me know what deep neural network type tutorials we should be doing next did you enjoy this one did you build it yourself and what do you go on about and classify thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 515,810

Keywords: python, deep learning, image classification, cnn deep learning, image classification using tensorflow

Id: jztwpsIzEGc

Length: 85min 5sec (5105 seconds)

Published: Mon Apr 25 2022