Build a Python Facial Recognition App with Tensorflow and Kivy

what's happening guys my name is nicole astronaut and welcome to the course in this course we're going to go from a state-of-the-art research paper and go all the way to the end and build a deep learning integrated app which allows you to perform facial recognition using a model that's pretty similar to what you might encounter using face id on your iphone this is going to be broken up into eight different parts where we start off all the way at the start with installing our dependencies completely from scratch to investigating the paper that we're going to be building taking a look at our data building up our deep neural network and then finally taking that deep neural network and integrating it into a qv app so that you can use it in a practical manner ready to do it let's get to it what's happening guys my name is nicholas renate and welcome to the very first episode of papers to code in this series what we're going to be doing is building our very own facial recognition app using deep learning from scratch so this series is all going to be to do with grabbing the data building up the model and implementing it inside of an application ready to do it let's take a deeper look as what we'll be going through so in the very first episode of this bigger series what we're going to be going through is just our general setup so what we'll do is we'll install our dependencies we're going to import all the different tensorflow specifically deep learning components that we're going to need this includes our layers our models and our inputs we're going to also configure our gpu so if you're using colab or if you're using your local machine this is going to ensure that your don't blow out your gpu completely and then what we're going to do is we're going to set up our data folder so there's going to be a bunch of episodes following this but this is really just the setup ready to do it let's get to it alrighty guys so we're going to keep this pretty simple and easy at least for the first episode in the facial recognition or face id type series so what we're going to be doing in this episode is we are going to be installing our dependencies importing our dependencies setting our gpu growth and then creating our folder structures now what i'm hoping is that this sort of episode of this style of video is going to become a little bit more common on the channel so we'll be able to take papers and convert them to code and actually implement them with practical examples so if you like this style of series do let me know in the comments below i'd love to hear your feedback so what we are going to do in this particular case is first up we're going to install our dependencies now the first part of this series is going to be all inside of jupiter so this is all going to be the data science and deep learning side of things where we actually create our facial verification model and we'll talk a little bit more about the paper that we're actually going to be interacting with a little bit later but we'll see that and then once we've gone and done that we'll actually transition and start coding up in vs code when we go and build our facial verification app so first things first let's go on ahead and install our dependencies all righty so those are our dependencies now installed now let's actually take a look at the line of code that we wrote so first up we wrote exclamation mark pip install so this is your standard python install and as per usual we're going to be taking this entire tutorial step by step so again we're going to write the code and explain it so exclamation mark pip install tensorflow equals equals 2.4.1 so this is going to install the tensorflow 2.4.1 version into our environment then we're using or we've imported or installed tensorflow dash gpu equals equals 2.4.1 and if you don't have a gpu on the machine that you're running on there's a few things that you can do about it you can try running this tutorial again without a gpu but it's going to be a little bit slower to train so which is perfectly fine it'll still work if you want to use a gpu there are a whole bunch of cloud platforms out there that leverage jupiter notebook so colab is a free one that you might take a look at but in this case if you don't have a gpu it's perfectly fine then we've installed opencv and so in order to do that written opencv python and this is going to give us some of our image processing capabilities and we're really going to be using this when we actually go to do real-time testing right at the bottom of this tutorial and then we've gone and installed a matplotlib as well cool so all up we've installed four different dependencies so exclamation mark pip install tensorflow tensorflow.gpu 2.4.1 opencv python and matplotlib so those are our four dependencies now installed cool so that is part 1.1 now done now the next thing that we need to do is import our dependencies so we've sort of got two different classes of dependencies that we want to import here so we've got our deep learning stuff so all of our tensorflow components and then we've got some sort of standard dependencies that are more going to help us actually work with our data so let's go ahead and import our standard dependencies first and then what we'll do is we'll import our tensorflow dependencies after that okay those are our standard dependencies now imported so i've written one two three four five lines of code there and one comment so the comment just says we're importing our standard dependencies again it's good practice to comment your code so what i've written is import cv2 so this is going to import opencv into our notebook then we've imported os so this is just an operating system library and we are going to use this when we actually go and create our folder structures so this just abstracts the operating system from our python code so it just makes it a whole heap easier to work with different directories different file paths then we've imported random this one is a little bit optional but i've brought it in just in case so if we are testing generating new data this comes in handy then i've imported numpy so import numpy as np and numpy is a great library when it comes to working with different types of arrays so tensorflow which is the deep learning library we're going to be working with is all to do with different tenses or working with different tenses numpy helps us work with those so what we're then going to do is import matplotlib and numpy is really just a really great array transformation library so again if you need to resize or reshape erase comes in super handy there so we'll probably use it later on then we've imported matplotlib so from map plot lib import pi plot as plt the main reason that i've brought in matplotlib is because there is a method called plot.iamshow so if i'd have employed.i am show this effectively allows us and question my question mark this effectively allows us to visualize an image so you can see that it says display data as an image so it just makes it super easy to visualize it when we're working with our different data components which we're going to need when it comes to facial verification cool so that's done the all right so we've gone and written those five items code so import cv2 import os import random import numpy's mp and then from matplotlib import pi plot as plt cool so those are our standard dependencies are now imported now the next thing that we're going to go ahead and do is import our tensorflow dependencies so i'm just going to change this so we're going to import tensorflow dependencies and really we're going to be using uh well not really we're going to be using 2k oh we need to import two key things here we need to import our model components and the different deep learning layers that we're going to need now what we're actually going to be doing is we're actually going to be using the tensorflow functional api so a lot of tutorials tend to use the sequential api but what i've actually found is the functional api is a lot more flexible when it comes to building really hardcore deep learning models which in this case this one actually is now the model that we're going to be building is called a siamese neural network so this actually allows us to do something called one shot classification we can actually pass through two images and determine whether or not that image is the same so if i actually bring up that paper let's see if i can bring it up and i'll link to this in the description below as well so basically what we're actually going to be doing is we're actually going to be building up this network described in this paper so siamese neural networks will one shot image recognition and if you actually take a look at the architecture here so this is really important so we are going to have two inputs so you can see that there is a break between these hidden layers here so we've got this input and we've got this input what we're effectively doing is we are passing through two images at the top at the same time and then we're going to have this distance layer so this actually measures the similarity between the two images and we're going to train our neural network to determine what that similarity is like so if it is very similar then we're going to output a 1 which basically means it is verified if they are completely different then we're going to output a zero which means we are unverified now in terms of the neural network that we're going to be building we're effectively going to be implementing this or something very very similar to this i think the input shape that we're going to be passing through is a little bit different we'll have 100 by 100 but you will see that towards the end of this series so we're actually going to take this model and actually implement it in real time which is super cool all right we've digressed so this is the model that we're going to be building we're going to be using the tensorflow functional api and i will link to this in the description below but for what we need to do now is we we actually need to import those functional api components so functional api so let's go ahead and do that okay those are our functional api components now imported and we've also imported tensorflow as well so let's take a look and take a step back as to what we've actually gone and imported so what i've written is from tensorflow.krs stop models import model this is probably one of the most important layers that you're going to need so when you're actually defining your model using the functional api what you effectively do is you type your model and you pass through two things so you pass your inputs and then you set that and then you also pass through outputs and you set that so what we're effectively going to be doing is we're going to be building up our neural network and then passing through what we want our inputs to be and our outputs to be so this is a slightly different to the sequential api but it allows you to define a significantly more sophisticated neural network so ideally what we'll actually have is we'll have our input image put this in brackets our input image and then our verification image and what we're effectively going to get out is a layer which effectively outputs one or zero so that is effectively what our model is going to look like by the time we actually get to it so that is what the model class is used for so in order to do that i've written from tensorflow.keras.models import model and this is effectively giving us this and then a big part of our deep learning models are a whole bunch of different layers so in order to do that we've imported a bunch of different layer types so we've imported or we've written from tensorflow.keras.layers import layer so this layer class is a high level class so this actually allows us to define a custom layer now when it actually comes to doing that we can effectively create a whole new class to actually go and generate that layer so it's effectively class um l1 or the class that we're actually going to be implementing is going to be called l1 distance and we can actually pass that layer through so we can perform inheritance for that layer so this effectively allows us to create a custom neural network layer so we'll actually be going through some really cool concepts as part of this bigger series okay so we've imported from tensorflow.layers import layer and then we've actually gone and imported some standard layers so conv 2d this allows us to perform a convolution so if you've heard of convolutional neural networks that is exactly what that allows us to do we've imported dense so this gives us a fully connected layer which is effectively what you typically see in most neural networks that sort of looks like these layers that you're seeing here then imported max pooling so this allows us to pull our layers together and effectively shrinks the information that we've actually got so this is almost like averaging or in this case it's not averaging it's actually taking the max values over a certain region to effectively reduce how much data we're actually passing through to the next layer and then we've got our input so this is again another base class so if you think about the three most important clusters we've got here it's model layer and input so our input allows us to define what we're passing through to our model and our layer effectively or our model actually allows us to compile it all together and our layer gives us our base class and when we're defining our input we're effectively typing input and then shape and you're specifying the shape there and then the last layer that we've imported is flattened so this effectively takes all the information that you've got from a previous layer and flattens it down into a single dimension so that allows us to pass convolutional neural network data to a dense layer so that's what we're going to be using that for and then we've imported tensorflow as tf so import tensorflow as tf so let's quickly recap on what we've imported to from tensorflow.keras.models import model from tensorflow.keras.layers import layer comma conf 2d comma dense comma max pooling 2d comma import comma and then flatten and then we've imported the tensorflow base or the tensorflow class overall or tensorflow overall so import tensorflow as tf so we're going to use this for our image processing and a whole bunch of other helpers but that is section 1.2 now done so what we can actually do is uh delete this because we don't need that and we can delete that so that is section 1.2 now done so what we can now do is go on ahead and set our gpu growth so this you only need to do if you're doing it on a gpu based machine so if you're not doing this on a gpu bet on a machine that has a gpu you can skip this step what we're effectively doing here is we're limiting how much vram tensorflow can use on the gpu because by default tensorflow is going to expand to take it all over and if you don't go and set this then you're effectively going to run into out of memory errors so this helps avoid this so let's write a comment so avoid out of memory errors by setting gpu memory consumption growth right so think of this about like or think of this as like putting some chains around your gpu to let it go to stop tensile from from going crazy or it's more chains around tensorflow than the gpu but effectively we're going to be locking intense flow down so it doesn't go crazy right so let's go ahead and do this oh okay hold on we've uh list fizz okay so that is our memory consumption growth now set so we've gone and written three different lines of code there so first up what we're doing is we're actually accessing all of the different gpus on the actual machine so if i actually type out gpus you can see that we've actually only got one there so if i take a look at the length of that so you can see we've got one gpu so in order to get that we've written gpus equals tf.config.experimental dot list underscore physical underscore devices and then we've passed through a string which says gpu so what this line is doing is it's actually getting all of the different gpus that are available on our machine then we're actually storing that inside of a variable called gpus which is effectively what i'm showing here right so you can see it says physical device name equals forward slash physical underscore device gpu zero so this basically means that we have a gpu here in this case the gpu that i'm using is a rtx 2070 super so i think it's got around about eight gigs of vram so just something to keep in mind sharing some information you know all right then what we're doing is we're actually going through and we're setting our memory growth so we are looping through all of the gpus so again if you have multiple gpus this is going to work as well which is why we do the loop so for gpu in gpus that's effectively just looping through this four gpu in gpus we can effectively print gpu right so we're effectively looping through them but then what we're doing is we're actually setting our memory growth which is this function which you can see right there so i've written tf.config.experimental.set underscore memory underscore growth and then we're passing through the gpu that we're processing at that time so if you remember we're looping through each one of them up here this is actually going to go and set that gpu and set memory growth equal to true got a bit tongue tied there all right cool so you can see that that is now done so we're grabbing all of our gpus we're looping through each one of them if we've got multiple this works and then we're using the set underscore memory underscore growth method to set our gpu growth equal to true and remember we're passing through that gpu and passing through true to actually set that cool so we can delete this cell down here so let's delete that and that is section 1.3 now done now the next thing that we need to do is a little bit of setup so we actually want to set up our folder structure so that we are in a good place when it actually comes to getting our data so if we actually do let me actually show you where i'm working what i'm working with so the folder that we're current i'm currently working in is looks a little bit like this so right now i've just got a virtual environment i've got some jupiter notebook checkpoints and i've got the actual notebook that we're working in at the moment what we need to do is create a couple of folders so we are going to create three specific folders for our data we're going to create one called anchor one called positive and one called negative now let me explain what each one of these is going to do so if we actually take a look at our neural network let's see if there's an example of it okay this is a good example so when we actually go to perform our facial verification we're actually going to pass through two images we're going to pass through an image called an anchor which is think of this as your input image right so this is if we are going and doing it in real time it would be the image that's captured from your iphone or the image that's captured from your webcam or whatever image that is verifying you in real time so think of this as your anchor we then pass through a verification image or a positive or negative image so what we're effectively going to be doing is saying hey does our anchor match the positive image so in this particular case you can see the soccer ball matches the soccer ball so it outputs same when our neural network does it it will output one we then also pass through a our input image again or our anchor image and then we verify it against a negative and we want to make sure that that is different because we effectively want our neural network to be able to distinguish between ourselves and another person so we're going to call the input image and anchor image we are going to call the verification image which is positive the positive image and we're going to call the verification image which is a negative example the negative image now all the negative data we're actually going to get from a repository called labeled faces in the wild which is a sort of like a standard repository but we're going to be doing that in another tutorial i'll show you all in another episode of this series but i'm going to show you how to set all that up so when we actually go to collect our positive and our anchor data we're going to be using our webcam for that so again we're going to be going through that i believe in episode 2 of this cool that is that uh explained so what we now need to do is actually create some folder structures so what we'll first up do is we're gonna set our paths and then we'll actually use the os library to actually go and create those directories so let's set up our paths first up okay so those are our paths set up and we haven't actually created the folders there we're just defining what our paths are going to look like so this is effectively what our directories are going to look like i'm calling them paths but it's just our directories right so i've written pos underscore path and i've just defined all of our paths in negatives because it just makes it a little bit easier to understand what they are so pos underscore path equals os dot path dot join so what os.path.join does is it effectively joins different directories together to be able to form a full file path so what you'll get out of this is data and then it's either going to be a forward slash or a double backslash depending on what type of os you're using so the file path is going to be data and then forward slash or backwards positive so if i actually take a look at that and type out positive path you can see this is just returning the file path that we're about to create and then we've gone and done it for our negative images and our anchor images so remember our positive images are going to be our positive verification images and negative images are going to be our negative verification images and our anchor images are going to do the same now when it actually comes to implementing this on a or on a real life implementation i've got a slightly novel or special implementation that we're actually going to use but again we're going to do that more later on in this series but um we've got something special to make this actually work really really well all right so we've got positive path negative path and anchor path and then what we are going to do now is create those directories so let's go ahead and do that and let's actually just take a look at the folder so we're going to basically create one big folder called data and then inside of there we're going to have three different folders so one called positive one called negative and one called anchor now what we need to do is actually go on ahead and create those so let's go on ahead and do that so i'm just going to delete this cell here and then create a new one let's do it okay so those are our directories now made let's add a comment there make the directories so in order to do that i've written os dot make dears now keep in mind it is m a k e d i r s not mkds uh make dears actually goes and creates all these subsequent folders so if you don't have a top level directory created it's actually going to create the full file path so i've written os dot make dears and then i've passed through our positive path os dot make dears and then pass through our negative path and then os dot make ds and pass through our anchor pass so we're effectively doing this part this path and then this path now if you wanted to you could put this inside of a list and then loop through them i figured it's just as easy to do this but basically we're going to be creating all of our different directories so if we now go into our folder and mine is inside of youtube and face id you can see let me zoom in on that so you can see it a bit better so you can see i've now got this folder called data and inside of that i've got three folders so one called anchor one called negative and one called positive there's nothing in there at the moment but our three folders are created and you can see that we've also gone and created this top level data folder so everything that we we're going to use when it comes to getting our data is going to be in here and then in one of these three repositories cool alrighty and i think on that note that is us done so we've gone and now created our folder structures so we defined our parts and we went and set up our directory so we've gone and done a whole bunch of stuff in this tutorial so we went and installed our dependencies we went and imported them we then went and set our gpu growth and we also went and created our folder structures to get ready to actually start collecting our images and building our face id or facial verification models so on that note that is this tutorial now done see you in the next one thanks so much for tuning in guys hopefully you enjoyed this video if you did be sure to give it a thumbs up hit subscribe and tick that bell and let me know if you've got any questions or comments in the comments below thanks again for tuning in peace what's happening guys welcome to part two in the series on facial recognition using a siamese neural network where we try to implement a code from an existing deep learning research paper and in this particular case we're going to be doing facial recognition or facial verification so ideally we'll be able to use a webcam or a camera to be able to verify ourselves inside of an application let's take a deeper look as to what we'll be going through in this tutorial alrighty so in part two what we're going to be doing is we're going to focus on getting our data now remember from part one what we needed was three different types of data so we needed our negative images we needed our anchor images and we needed our positive images so in order to collect our negative images what we're going to be doing is we're going to be leveraging a standard image repository or facial image repository called labeled faces in the world so we'll be able to download that and unpack it and get it into the structure that we need for our model we're also going to collect our anchor images and our positive images now in this particular case we're going to be using opencv to do that using a webcam but if you've already got some images of yourself say for example you've collected them using your phone you can definitely use those as well ready to do it let's get to it alrighty guys so in this tutorial what we're going to be doing is collecting our data sets from the labeled faces in the wild data set and then we're also going to be collecting our positive and our anchor classes so this basically means that we're going to have all the data that we need to at least go on ahead and train our model now before we get into it i wanted to sort of explain a little bit more as to or how we're going to actually be using these data sets so let's actually take a look at how this is all going to work now as i mentioned before we're going to have a positive data set and we're also going to have a or we're going to have positive examples negative examples and anchor examples so i wanted to sort of visualize how this is actually going to work so let's take a look at our first positive or our positive examples first so say for example we have an input image well that's a little bit thin let's make that a bit bigger so we've got an input image so imagine this is coming from our webcam move this out of the way webcam and then what we're going to have is a positive image so positive when we actually go and build our model what we're going to do is we're going to pass this or pass both of these images to an embedding model or an encoding model it's probably a better term and these models that you can see here so let me change the color so it's a little bit better to see so this model is effectively going to be our encoding model so it's going to convert our webcam or our anchor representation so this webcam data is our anchor so we're going to convert that model encoding to a data representation and then when we actually go and build our model what we're actually going to be doing is trying to determine the difference between our anchor and our positive so this layer that we're going to implement over here is actually going to be a distance layer so think of it as going all right so we're going and converting our input images to a embedding or an encoding and then what we're going to do is we're going to try to see how similar they are and if they are very similar then what our model is going to do is it's going to output a 1 to basically say we are verified which is effectively saying that the person inside of our positive image is the same person inside of our anchor image right so these are all going to be connected now the cool thing about using this particular type of model is that if you wanted to go and implement this on other people then you definitely could all you would need to do is pass through a different positive image and pass through the same anchor image or password different anchor image and it will be able to verify against a whole range of people now in our particular case we're going to be doing it against one person but that's perfectly fine now let's take a look at a negative class right so i'm going to do this one in blue so again actually let's do it in the same color so we're again going to have our anchor image right and this could be from your webcam it could be an image from your phone it's really going to be what we're passing through as our input to effectively perform our verification then we're also going to have a negative example and again we're going to be using this same embedding layer or the same embedding model or encoding model whatever you want to call it and we're going to be passing these images through so these models that you see here or this model is the same across the board i'll actually draw it uh make it a nicer orange that's nice all right so this everything that you see here that is going to be the same model so effectively what that model is going to learn how to do is how to best represent the input images to be able to ensure that when we actually go and perform our similarity analysis that we're actually accurately classifying them as either positive or they match or they don't match as in negative so when we go and pass through our anchor and our negative what's actually going to happen is when we pass this through to our distance layer our distance layer is going to say hey not the same the same and it's going to output a zero so which means that we are unverified so i figured i'd give a little bit of a visual representation of what we're building because sometimes i think it's very easy to get lost as to how these neural network models work but basically this is in a nutshell how this neural network is going to be built up so what we're actually going to be doing in this tutorial is we're going to be focused on collecting our data so this one's going to be so we're going to be collecting our anchors which are there we're going to be collecting our positives which are there and we're also going to be collecting our negatives now in our particular case our anchors are going to come from our webcam so we're going to do that using opencv and our positives are going to come from our webcam as well not wed and our negative data is going to come from the labeled faces in the wild data set so this is an open source data set that actually has a whole bunch of different faces that in a nutshell is what we're going to be doing today we're going to be focused on all of the stuff in i don't know what do you call this color aqua we're going to be doing that so we're going to be collecting that data alrighty so let's actually get back to it and do some coding so first up what we're going to be doing is we're going to be collecting our labeled faces in the wild and then we're going to collect our positive and anchor classes so let's go on ahead and do this now in order to get our labeled faces in the wild data set you can actually go to this link here so let me actually copy this so you can see it so it is http colon forward slash forward slash vis www.cs.umass.edu forward slash lfw forward slash so this is actually going i'll actually link to this in the description below so you can pick that up as well so don't stress if you haven't picked it up now as per usual all the code that we write inside of this tutorial is going to be available via github so if you want to pick that up you definitely can and i'm actually structuring the code so you can see what we've written after tutorial one or part one of this series what we've done after part two so you'll actually be able to see progression okay but for now what we need to do is get our labeled faces in the wild data set so if we go to this link what you'll actually see is that right over here we've got this link called download so we're going to hit download and then there's a whole bunch of information here so we've got uh what do we have so we've got all images as gziptar file all images aligned with deep funneling all images aligned with funneling all images aligned with commercial face alignment software so there's a whole bunch but what we're actually going to be doing is we're going to be using this one here so all images as a gzipped tar file so let's hit that and this should start downloading so it's about 170 273 megabytes so once you've got that we'll be able to untie it and start working with it inside of python so let's give that a second to download alrighty so that is our data set now downloaded so what we're going to do is we're going to open that up inside of its folder let's zoom out a little all right so you can see that that's downloaded so i'm just going to cut that and paste it into the same folder that we're currently working in so i am currently inside of my d drive inside of youtube and inside of our face id folder so i'm just going to paste that there so once you've gone and downloaded it put it or once you've gone and downloaded that data set grab it and put it inside of the same folder as your jupyter notebook is in so if you're doing this in colab just make sure it's in the same folder as your jupyter notebook so you can see that that is our jupyter notebook that we're working on at the moment that is our data set so lfw.tgz cool alrighty so what we now need to do is we now need to uncompress that so it's a tar gz file so you can see it's tgz so we need to uncompress that so we can go on ahead and do that inside of our notebook so i'm just going to add a comment uncompress gz what is it labeled faces in the wild data set all right so we're going to let's do it okay so that is the command to uncompress and extract our data set so i've written exclamation mark tar and then there's a space dash xf and then there's a space and then there's the name of the file so this command here is what's actually going to allow us to extract our data set and put it inside of the same repository or the same place that it currently is this is actually just passing through the actual name of the data set so if say for example in the future the name of the data set from labeled faces in the world changes you're going to want to change this component here so if we run this now all things holding equal this should uncompress and we should be able to see our data set okay so that's finished running so you can see that we no longer have an asterisk over here so if we actually go and open up our data set again i don't know why i closed the folder that's cool you can see that that's now been uncompressed so we've got this folder here called lfw and we have a ton of images so you can see that it's actually labeled by person's name now we don't we're not actually really concerned with the person's name in this particular case because we're going to be using all of these for negatives but if you wanted to do a different form of facial verifications to say for example you wanted to add triplet loss you could definitely do this and add pairs of images for different people in our particular case we're very much focused on the one person that we want to verify so what we need to do is we need to take all of these images so if you actually open up these folders there's multiple pictures of people right heaps of people list keeps going what we want to do is we want to take these images inside of the labeled faces in the wild folder and inside of these subsequent folders and we want to put it inside of the folders that we created in part one of this tutorial so if you go into the data folder that we created earlier we want to move all of those images from the labeled faces in the wild data set and put it inside of this negative folder so what we're going to do is we're going to write the python code to go on ahead and do that so this is effectively going to put all of our data in the same place and follow the same structure so let's go on ahead and do it so we're going to i'm just going to add a comment so move lfw images to the following repository that's going to be data and then forward slash negative all right so let's go ahead and write that code okay so that is our code to move our data from the lfw folders and the subdirectories into our negative folder now the key thing that i just realized is that i haven't actually gone and run the code that we had in our initial tutorial so if i go and run this again so let's actually take a look at what we've written first and then we're going to go and run our imports and stuff so i haven't actually gone and run the initial steps my bad that's fine um so if i actually go and run this now we're going to get a whole bunch of errors if i run this you can see it's saying name os is not defined perfectly fine we'll solve that in a second so first up what we're doing is we're looping through every single directory inside of our labeled folders in the wild repository or directory then what we're going to do is we're going to loop through every single file inside of those subdirectories so we're effectively saying go into ow where are we surface id so go into this folder and then go into this folder and loop through each of these images because in some particular cases there's multiple images of people so you can see in that case we've got multiple images so we need to move each image into its new folder so in order to do that we first up define the existing path so i've written x underscore path in caps and i've set that equal to os the path dot join and then we've passed through lfw which is the main directory or the root directory pass through the directory that we've extracted from here because remember we're looping through them and then pass through the file name that we're getting from here then we're specifying the new path name so i've written new underscore path in all caps and i've set that equal to os.path.join and then we're passing through a negative path which is from our previous tutorial which we defined over here and we're passing through our file name so this is going to effectively join our negative path and our file name and then we're going to use os dot replace and we're going to pass through our existing path and our new path so this is going to grab it from our existing path and move it to our new path but as of right now this isn't going to run because we haven't run our import so i'm going to go right up to 1.2 run this so that's going to import open cv os random numpy and matplotlib then we'll import our tensorflow dependencies not the i think we'll need them now and then we're just gonna run uh the code under 1.3 and we're going to run the code under 1.4 so that's going to define our different paths our positive path our negative path and our anchor path okay now if we go and run this all things holding equal let me actually just show you quickly how this actually works so if i write os dot list here lfw so this actually returns all the subsequent folders inside of the lfw directory now if i loop through those so for directory in os.listia what's going to happen is i'm going to be able to access each directory so if i write for file in os.list dr and this should be os.path.join i've actually written got an error there so os stop let me actually run it without changing it and we'll see what happens so if i run lfw and then directory that's going to throw an error so this should actually be os.path.join because right now i'm not joining those directories together so if i change that here as well os.path dot join and then if i print a file this is going to print out every single image so now if i actually join these together so os.path.join and if we pass through lfw so os.path.join just joins directory names together so it gives us a full path so if i pass through lfw and then directory and then file and then close that this is going to give us the full path to every single image right so that is exactly what we're doing to get our existing path then what we're doing is we're defining the new path so we're going to do os.path.join and we are going to be passing through our negative path and our file right so we're effectively going to be grabbing this image and then moving it into data and then negative and then aaron eckhart and so effectively it's the same name so we're grabbing this and moving it to here we're doing this we're moving it here so we're just going to loop through and do this for every single image so if i delete that we don't need that anymore and i actually go on ahead and run this this is actually going to move all of our images from those existing stacked directories into our negative path and that is done so it ran reasonably quickly so if we go into that folder now and go into d drive youtube face id if we go into let me zoom in on this data and then negative you can see we've got all of our negative images there pretty good right now i don't think we're going to use all these images for training but you sort of get the idea we've got plenty to work with if we need to so what's happening now so we can actually close this so what we just went and did there so if we go into our lfw folder and you can see that each one of these are now empty because we've actually gone and moved them into a new folder so what we can actually do is delete this lfw folder there's nothing in it we can get rid of it now so if we delete that we are all good cool so that is step 2.1 now done so we've gone and uncompressed our label faces in the wild data set and we have also gone and moved all of those images from the lfw directories in to our negative path which we defined up here from the first episode in this series cool so what's next is that we need to actually go and collect our positive and our anchor classes now for this what we're going to be doing is we're going to be using opencv to access our webcam and we're going to collect those images down and save them down now the size of the images that we're actually going to be collecting are going to be 250 pixels by pixels so by default when we use our webcam your image resolution might be a little bit different so what we actually want to do is we want to ensure that we're collecting images of that size because i believe the sizes from the lfw data set are going to be in that same size as well so let me just double check that so if we go into data and then negative and if i go and open one of these so properties and details yep so you can see that 250 by 250 and i can check another and again 250 by 250. so to make our lives a little bit easier we're just going to ensure that we collect the same or images of the exact same dimensions when we collect our anchor and our positives this is going to make your data processing a whole bunch easier when it comes to training the model all right cool so that is good so what we now need to do is do exactly that so we're going to be using a pretty standard opencv loop to be able to go and collect this with a few tweaks to make our lives a little bit easier but first of what we need to do is ensure we can access our webcam and do that successfully so let's go ahead and do that okay so that is the first part of our video capture code or image capture code now done so i've gone and written one two three four five six seven eight different lines of code there so this is if you've watched any of my computer vision tutorials this is going to look super familiar to you so let's actually comment through this so first up what we do is we establish a connection to the webcam and that is exactly what this line is doing here so i've written cap equals cv2 dot video capture and then i've passed through a video capture device now i think it's going to be video capture device 3 but it might be a little bit different because i've gone and installed some new stuff on my pc but we'll see and then i've ridden so i've actually i think it's actually because i'm actually doing the whiteboarding now but and let me know what you guys thought of that i'm testing that out we'll see if it picks up or if you guys enjoy it if you don't let me know and i'll stop doing it so once we've established our connection to the webcam i've then written while cap is open so this is going to loop through every single frame in our webcam and then we're using cap.read to actually read that capture at a point in time and then what we do is we actually unpack the results that we get from that method there so we unpack it and get a return value and the actual frame so this frame is the actual image then what we're doing is we're rendering that image back to the screen so it just makes it a little bit easier to actually see what we're doing so we've written cb2 dot i'm show so let me add some comments here so show image back to screen so cv2 dot i am show and then we're naming what we want our frame to be named so in this case every image collection can name it whatever you want and then we've gone and passed through our frame which is what we got from over here so this is effectively going to be showing the feed from our webcam on our screen inside of python or inside of a cv2 frame and then everything from here on out is to do with breaking gracefully so what we've written is if cv2 dot weight key one so this is going to wait i believe it's one millisecond so cv2.weight key uh so it says delay what is it uh this function yes it's in milliseconds so it's going to wait one millisecond and it's also going to check what key we've actually pressed so this is actually unpacking what is being pressed from our keyboard so then we've written end zero x f f equals ord q or equals equal so let's check it's doing a comparison check this is really important because we're going to use this a little bit more in a second so when we hit q on our keyboard this should close down our frame what we're also going to do in a second is we're actually going to configure some other ones so that when we hit a it's going to collect an anchor and when we hit p it's going to collect a positive image and i think we're going to collect roughly roughly 300 issue images doesn't matter we'll see i think 300 is probably a good good starting point okay so that is doing that check and then if that check is passed so if it waits a millisecond and we hit q on our keyboard then it is going to break out of this loop up here and then it's actually going to release our webcam so let's actually comment this for once release the webcam and then it's going to destroy or close the image show frame so if ever you are using opencv and you're accessing your webcam and stuff is just freezing up it's not working what you can actually do is run cap.release to release your webcam and then cv2.destroyorwindows to actually close everything down and re-kick things off so if this number up here is incorrect and it all locks up and freezes up what we'll do is we'll stop this cell from running and then we'll run these two commands down here to be able to release whatever webcam we're trying to access and then destroy all those windows so that is uh image collect well that is our webcam code now set up we haven't actually done any image collection yet so if we go and test this out let's see if that works so if that does run successfully you'll get a little pop-up right so that has not run successfully so this error is a common error that i always get asked about so error opencv 4.5.3 blah blah blah blah source dot empty in function so this basically means this here super important let me zoom in on that that basically means that it's not able to access the webcam so whatever we're getting back from that webcam device or that image device is empty which means that we don't have the right webcam number up here so what we want to try to do is try a different webcam number so let's try four that might work a little bit better all right so that's worked so you can see that i've got this little pop-up and you can see our feed all right so sometimes you're gonna have to tweak that right so and this is it's great that i'm showing you this because it'll actually show you how to resolve that error so all we did there is we change our video capture device from three to four and you can see that i've now got the right video capture devices i can see myself in the screen and you can see up there that it says image collection so that image collection label is coming from over here so if we wanted to change that you definitely could okay so the key thing now is that this video frame that we're currently looking at is not in the dimensions of 250 by 250 pixels so if i quit out of this and remember if we hit q so this section of code is going to trigger so let me actually show you this so if i hit q on my keyboard you can see it shuts it down what was that going to say now next okay so the frame so frame dimension so if we actually take a look at frame so the nice thing about this loop is that you're going to be able to access the last set of variables that are being captured so if we actually take a look at frame this is our image and if i run this was the advantage of using matplotlib so if i type plot.i'm show you can see that that's our image there ignore the coloring that's because opencv has a slightly different channel order perfectly fine key thing we're interested in is that if i type in frame dot shape this is not 250 by 250 pixels right now it's 480 by 640 pixels by three channels so we actually need this to be 250 by 250 by three so what we can actually do is we can actually do a little bit of indexing or slicing to actually get the right shape so say for example i just did indexing so say i'll grab the first 250 by the first 250 by everything you can see that we are now getting 250 by 250 by 3. and so all i've done there is just some slicing or some array slicing so if i actually go and plot this now plot.i am show you can see i'm getting the top what is that top right corner top left corner so in this case i'm getting the top left hand corner but you can see that that's not actually accurately capturing my face so it's a little bit of a pain there because this is going to suck when it comes to performing facial verification so what we could actually do is we could actually tweak these numbers so rather than starting from zero which is effectively what this code is doing let's say we started a little bit further in so we started from what's a good what did i actually end up using so 120 and then we did 120 plus 250 and then for our x-axis we did 200 and then 200 plus 250. so if we go and do that now that is a little bit better so we're now at least getting somewhere closer to where our face actually is now when we actually go and collect our images we can effectively render that to the screen so we can see where we are inside of that position that's perfectly fine so let me actually explain what i've done there so what i've actually gone and done is some indexing or some slicing so this first value is going to tell us or it's actually going to determine where our pixels start on the y-axis and where they end so we're effectively starting at 120 pixels and we're by passing through our colon we're saying we're going to go from 120 pixels to 120 plus 250 pixels so this is basically specifying the range of values that we want from our image and we're doing exactly the same on the x-axis except here we're starting at 200 pixels and we're going to 200 to 200 plus 250 so it's starting at 200 and going to 450 pixels for our y-axis we're starting at 120 and we are going to what is that 370 pixels and then by passing through another comma and a colon we're effectively saying that we want all three channels if i pass through just uh zero we're only going to get one color channel if i pass through one we're going to get a different color channel and if i pass through two again we're going to get a different color channel there but if i pass through colon we're going to grab all three so it means we're going to retain the fact that we have a color image okay so what we now need to do is we now need to implement this logic inside of our image capture loop so what we had from up here so let's go ahead and do this so we're effectively going to just grab do effectively this and paste this here so we're going to be taking that slice of our frame from our webcam and we're going to reset the variable so we're going to set frame equal to this sliced version so cut down frame to 250 by 250 pixels so now if we go and run our image capture loop we're only going to be grabbing 250 by 250 pixels so if i go and run that you can see that that's effectively doing this so when when we actually go and collect images i'll probably bring my seat a little bit further down and ensure my head is actually inside of that frame but you can see that that is going to be replicating what we've got from our labeled faces in the wild data set a little bit more accurately than if we just went and grabbed 640 by what is that 480x640 cool all right so that's effectively giving us our 250 by 250 pixels what do we need to do now so we actually now need to write out some images or actually save some images so again i'm going to hit q on my keyboard and close that down okay so what do we need to do so we now need to collect our anchors and positive so i'm just going to add two additional comments collect anchors collect positives and so we're going to use this logic that we had down here so our breaking logic to actually go and collect our anchors and our positives so i'm actually going to copy this and paste it here and paste it here and what we want to do is we want to trigger our anchor collection when we hit a on our keyboard so i'm actually going to change the value inside of our ord function from q to a so this basically means that it's going to wait a millisecond and if we actually hit a within that millisecond it's going to collect an anchor image and we're going to do the same for our positives but rather than hitting q we're going to hit p so when we hit a on our keyboard it's going to collect an anchor and when we hit p on our keyboard it's going to collect a positive image now by stacking these together it does mean that there is going to be a little bit of a lag but instead of implementing a ton of logic i just figured this should be fine we can effectively work around this okay so what are we doing now so we're going to collect our anchors and our positives so what we now need to do is we now need to implement some logic to actually grab our frame and save it to our positives and anchor folders now before we actually do that i'm actually going to import a library called uuid so the uuid library is actually going to make it a little bit easier to actually go on ahead and let's actually ensure that we don't have a screen function so i'm just going to write past there and pass there for now until we actually got to implement that logic so we're going to grab uuid and this is going to ensure that we're able to create unique names for each one of our images so in order to import uuid say we're going to import the uuid library to generate unique image names uh and so import uuid cool so that's uid now imported so i've written import uuid and uuid stands for uniform unique identifier or unique uniform something like that so what is it uid question mark question mark nope lowercase all right universally unique identifiers there you go so basically gives you a specific pattern to generate a unique identifier now in order to use it there's a few different methods if i type in uuid dot there are a bunch of different formats that you can have so it can be uuid 1 uid3 uuid 4 or uuid5 so we're just going to use uid1 so if i type in uuid 1 you can see that it is generating this unique identifier there so we are effectively going to be using that to generate our unique image name so we're just going to implement that there so let's go ahead and do that and we'll take a step back and take a look at what we wrote okay so that is oh we haven't actually finished that's the cv2 dot i'm right let's finish that okay that is us done so what we've effectively gone and done is we've added two additional lines of code there so i've written first up what we're actually doing is we're creating the unique name create the unique file path and then we're actually going on ahead and writing out our image right out visit anchor image okay so let's take a look at what we wrote so first up i've created a variable called image name so img name and i've set that equal to os.path.join and then we've gone and passed through our anchor path because we're going to store our anchor images inside of our anchor path and then we're just creating a unique name or a unique name for our file so if i go and copy that you can see that we're just appending a unique identifier to dot jpg so this effectively means that when we go and do this multiple times we're going to be creating unique identifiers each time is that actually yeah it is changing so you can see that there and then by wrapping it in os.path.join we're effectively going to be creating a full power file path so uh step path dot join if i pass through anc path comma and then this unique file name this is effectively going to be storing our images inside of the data folder inside of the anchor folder and then it's going to be naming it that there and in order to actually go and write out our image i've written cv2 dot i'm right and we're passing through our image name which is what we just created up here and we're passing through our cut down 250 by 250 pixel frame cool so that is that now done now what we can also do is just copy this over here and paste it under our positives and all we need to do is change the anchor path to the positive path because we're going to be storing them inside of different folders so that is effectively that there now done okay i think we're good so if we actually go and run this we should be able to hit a on our keyboard to collect anchor images and p on our keyboard to collect positive images so if i go and let's actually do this side by side so we can actually see our folders so if i go into d drive and then youtube and then face id [Music] and then data and then anchor so we're first i'm going to collect our anchor images so if i actually go and run this code here now yep okay so that's let's run it cool all right so we've got our image of ourself and what i'm going to do is i'm going to put down the green screen let's actually test this out so i'm just going to get my head inside the frame move the mic out if you can still hear me fingers crossed all right and so what we're going to do now is we're going to hit a on our keyboard to collect an anchor so if i hit a make sure we're clicked into it all right so you can see that that is collecting our images pretty cool right so if we keep hitting a we should be able to collect a bunch more i'm just going to move my head around i'm just holding down a all right that looks like we've got 400 images there that should be more than enough so you can see 414 down there so we've got plenty just take a look at a different view all right so we've got a ton of images let's get somewhere we're a little closer as well so if i go and a little bit closer cool all right so that's 489 images that we've collected now what i'm going to do is i'm just going to jump into the positive folder and let's collect some positives again hit p on our keyboard this is going to start collecting positive images i'm just going to move my head around as i'm collecting now it is going to take a little bit longer to collect our positive images because we've got that one millisecond break probably speed this up in editing but that's pretty cool so you can see that we are effectively collecting images so we want around about 300 for each that's what we'll use for training alrighty that should be enough so we've got what 332 positives now collected okay so what we've gone and done there is if we go into our data folder into our anchor we've got a bunch of anchor images collected and remember we're going to have two streams when it comes to building our model we're going to have the anchor image that we passed through and we're also going to have the positive image or the negative image down here so we're effectively going to be verifying whether or not our anchor images matches the negative or matches the positive so it should output a one if it is positive so if it matches the positive and it should output a zero if our anchor is being verified against a negative image okay so we've got our anchor images we've got our positive images and again these have been both collected via our webcam and if we go we've got our negative images as well pretty cool right so if we go and hit q now on our keyboard should hopefully close there we go all right so that is our data all collected so we've gone and done a bunch of stuff so that's effectively this entire tutorial now done so or at least part two so what we've gone and done is we first up went and collected our images from the labeled faces in the wild data set and again i'll link to that in the description below we moved those into the negative folder and then we went and collected a bunch of images using our webcam and we collected both our anchor and our positive so this is actually doing our positive cool so that is now done so again as per usual all this code's going to be available inside of the description below by github but on that note that about wraps it up i'll see you in the next one thanks so much for tuning in guys hopefully enjoyed this video if you did be sure to give it a big thumbs up hit subscribe and tick that bell and if you have any questions comments or queries do let me know in the comments below thanks again for tuning in peace what's happening guys welcome to part three in the facial verification or facial recognition series where we go from a complete research paper all the way through to building a front end implementation so we're gonna be building up the model and then implementing it inside of an application alrighty let's take a look as to what we'll be going through in this video so let's make sure we go to present mode so first things first what we're going to be doing is we are going to be pre-processing our images so we've got a bunch of images that we collected in the previous tutorial what we're going to be doing is starting to pre-process those then we are going to create our positive and our negative samples so remember we took a look at the different architectures so we're going to be passing through our anchor image plus a positive image that's going to go through the pipeline we'll then pass through a anchor image and a negative image that's going to go through the pipeline when we train as well we're going to be able to create those samples and actually load them into step 3 which is our tensorflow data loader so we're gonna be able to do all of this and by the end of this tutorial we should effectively have our data ready modeling so we'll actually be able to get to the deep learning part of this tutorial right after this okay ready to do it let's get to it okay so let's get out of slides that's plenty so what we're going to do is we are effectively up to step three in this process so we're going to get our image or get our image directories we're going to pre-process them so scale and resize and then we're going to create a label data set which is step 3.3 let me add some tea guys and then what we're going to do is we are going to build our train and test partition let me make this a little bit bigger okay so first things first we need to go ahead and get our image directory so remember we went and created our image directories right up here so we created a folder called data and then inside of that there was a positive image and then we created another one called negative and another one called anchor so if we actually go and take a look at our data so if we go into youtube and then face id we've got this folder called data if i zoom in on that so you can see it so we've got a folder called anchor which you can see a whole bunch of mugshots of me there and then if we go into our negative folder got a bunch of images from the labeled faces in the wild directory and then if we go into positive we've got a bunch of additional mugshots in myself that's priceless oh god so there's some some great photos of me in here but anyway we've got our data right now what we need to do is we're going to use the tensorflow data loader so if i bring this tensor flow data loader so this tf data pipeline allows you to build these data pipelines and i've found that this makes building deep learning models a whole heap easier so what we're actually going to do is we're actually going to first up load all the folders from their respective directories we're going to create a pre-processing pipeline which is going to be around about here so we're going to create that pre-processing pipeline and then we're going to associate our positive and our negative classes so we'll actually build up all of this okay so first up things first let's go ahead and get our image directories okay so those are our image directories now obtained or specifically the files within our image directory is now obtained so what i've actually written there is three different lines of code and again i'm going to explain this in great detail so we've gone and created three new variables so anchor positive and negative and then what we've actually done is we've used the tf.data.dataset.listsfiles method to actually go and grab all of the different images within a specific directory so what this is actually doing is creating a data gen or creating a generator or using a generator to be able to loop through and grab all of the files within that specific dictionary or specific directory not dictionary so if we went and grabbed anchor now and i'll explain that in a sec in a little bit more detail so if i said um anchor dot as numpy iterator so i'm just going to show you something briefly so dear test and if i type in dear test dot next so what you can actually see there is we've actually got the full path to a specific image within our directory so this tf.data.dataset.list files actually creates a set or a pipeline for all of the different files within a specific directory okay so what we're effectively doing is we're going two ways or passing through a specific path so our anchor path and then we're doing a wild card search so we're adding backward slash and then start.jpg so this is effectively going all right grab everything inside of our anchor path which has a dot jpg extension so the full line is anchor equals tf dot data dot data set and data set is in caps dot list underscore files and then we're passing through the anchor path and then plus plus backwards star dot jpg so if i grab this little bit here you can see that it's effectively doing a wild card search inside of that directory so it's grabbing a data backwards backward slash anchor backwards backwards jpg and if you take a look inside of our data folder so youtube face id data and then income they should all be jpg files so you can see that so say for example you were working in png files you could uh change the jpg or dot png for example and then what we're doing is we're specifying that we only want to take 300 images because remember when we collected our data in the previous tutorial we went and grabbed i think it was roughly 300 anchor images roughly 300 positive images and then negative we've got plenty from the labeled faces in the wild data set but we ideally want to make sure that we've got a matching number of samples for each of these different classes so anchor positive and negative so we're going to take 300. so in order to do that we're just appending dot take and then passing through the number 300. if you wanted to take more images you definitely could if you wanted to take less images you could do that as well but try it out so see what your performance looks like in the end which we'll obviously get to but in this particular case we're going to work with tray 100 images okay so that's our anchor line done so what we've gone and done is we've gone and done a similar thing for our positive and our negative line so positive equals tf dot data dot data set dot list underscore files and then we're passing through the positive path plus and then backward slash dot jpg so exactly the same as what we had in our anchor line but this time we're using a pos or passing through our positive path which was defined right up here so remember positive negative anchor path these are our different paths and we've gone and done the exact same there so we've also gone an appended dot take 300 to grab 300 images we'll grab 300 image paths right because remember this has only got the paths it hasn't actually loaded our images into a data pipeline yet perfectly fine we're going to handle that in a second and then we've gone and written negative equals tf.data.dataset.listfiles and then this time rather than passing through the anchor or the positive path we're passing through our negative path and again we're taking 300 images and what else so down here i was basically showing you how to actually access this iterator so you can actually type in the name of the data set or the data pipeline and type in dot as numpy iterator and then by using the dot next method you can continuously grab the next component inside of that pipeline so if i run dot next again you can see like watch here so you can see that number is changing so right now it's 8294 because that's the image id changed grab the next one grab the next one grab the next one grab the next one grab the next one and the nice thing about this is that when we actually go and batch our data together the way that our model is going to train is we're going to train batch at a time so we will grab one batch out of this data set which we'll eventually create we'll then go through train it so we'll do a forward pass we will calculate the loss we'll then go and what do we typically do then uh we then go and calculate the gradients or we actually go and yeah go and calculate the updated gradients and go and apply them to all of the different weights in our neural network so basically this iterator or this data pipeline allows you to do that with a little bit more flexibility okay that is step 3.1 now done so we've gone and grabbed our anchor our positive and our negative data sets or specifically our image files and you can see that's an example of what we're actually getting out of here so this particular case we've got we can print it out to make it look a little bit nicer so we've got and didn't actually do anything so uh so in this particular case i've got data backwards backwards anchor backwards backwards and then the image within that directory cool alrighty so the next thing that we need to do is go on to step 3.2 and start doing our pre-processing so what we're actually going to do is we're going to write a pre-processing function that loads our image from our directory we're actually going to resize it and we're also going to perform some scaling so we're going to convert all of that image values from 0 to 255 to 0 to 1. so this helps our neural network optimize a lot easier so rather than having a huge range of numbers having it from zero to one effectively makes the gradient descent a little bit easier and helps us achieve a better performing model a whole heap faster so what we're going to do is we are going to write that function now so let's do it okay so the function that we are going to create is going to be called preprocess i know super unique and creative and then to that what we're going to do is we're going to be passing through this specific file path so we'll actually be passing through one of these file paths to this function so we'll be able to use the map method and again don't freak out if i haven't explained this yet we're going to go into it in more detail but what we'll effectively do is once we've got our data set so let me explain it in a little bit more detail so we'll have a object which is a tensorflow.dataset iterator or generator and we'll be able to run dot map and to that we can use the pre-process function and this is effectively going to allow us to apply the pre-process method which we're defining over here over every single example that we've got within our anchor positive and negative classes but again we're going to go into that in more detail i just wanted to explain why the hell we're writing this in the first place okay so what are we doing so we're going to finish this off so let's do it okay that is our pre-processing function now done so we've gone and written an additional one two three four five different lines of code and these effectively load up our image from a file path they also decode the jpeg they then it then it resizes the image it divides it by 255 so it performs its scaling and it returns the image i'm going to explain this in more detail so uh let's actually see how it works so if i go and run pre-process and if i take this file path here you can see what it's doing is it's returning a numpy equivalent of our image so if we wanted to we could um so let's call it image if i ran plot that i am show so you can see it has effectively loaded up our image from this file path and it's gone and done a bunch of other stuff so it's actually gone and resized it and it's gone and divided it by 255 so it's effectively scaled our image to be between one and zero so if we run um what is this so dot numpy is that gonna work yep so dot min so you can see our minimum value is not less well it shouldn't be less than zero anyway either way but it is the 0.1 or 0.1 and then if we take a look at our max so this is the most important thing you can see the maximum value that we've got is one so it's not going past 255 so we're good there okay let's take a look at each of the lines of code that we just wrote so under our pre-process function so in order to define our preprocess function we've written d f d break pre-process and then we've passed through our file path and then colon then we've gone and written one two three four five different lines of code so first up what we are doing is we're reading in our image so read in image from file path and in order to do that written byte underscore image so we've created a new variable and we've set it equal to tf.io.read underscore file and then we're actually passing through the file path that we originally passed through to our function so think of this like just passing through this path that we've got over here to our function pretty straightforward so that is then going to read our file as a bytes-like object and then what we're doing is we're using the tensorflow decode jpeg image to actually load in the image to load in the image and so the line there is image equals tf dot io dot decode underscore jpeg and then to that we're going to be passing through this byte image that we've got here so you can see we're passing that there so the full line is image equals tf dot io dot decode underscore jpeg and then we're passing through byte image then the next line is actually doing a little bit of pre-processing so think of this as our pre-processing steps and the first one is we're actually resizing the image to be 100 pixels by 100 pixels by three channels so in order to do that with an image equals tf.image.resize and then we're passing through our image which we've originally loaded from up here and then we're specifying the dimensions so we want it to be a hundred pixels by a hundred pixels the reason that we're doing this is if you actually take a look at the original paper i wish i siamese uh why didn't i have it up let me open it up now so i can show you it's in here all right so if we take a look at the neural network network um so the input image that we actually have in the paper is uh one image by 105 by 105. we're going to do it 100 by 100 just to keep things a little bit tweaked but if we wanted to we could actually make this 105 pixels by 105 pixels wouldn't cause an issue perfectly fine we're going to make it 100 by 100 this isn't going to implicitly impact our neural network in any negative way at least from what i've seen but if you wanted to make it 105 by 105 perfectly okay alrighty then what we're doing is we are scaling our image so scale image to be between zero and uh so i've read an image equals image divided by 255.0 so this effectively takes every single value or pixel value which is traditionally from 0 to 255 and divides it from or divides it to be between 0 and 1 now so this effectively scales our image and then last but not least we're returning our image so that we're able to do something like this so this is an example of actually going on ahead and running it and you can see it's actually loaded up our image so we can begin to work with it if we need to all right so the next thing that we need to do well we've actually already gone and shown it so what we can then do is actually begin creating our label data set so what we actually need to do here is we are going to create a positive and our negative example so remember when we actually go and validate we are going to pass through an anchor image and a positive image and this ideally should output once ideally verified and we can also pass through an anchor image and a negative image and that should ideally output zero so what we can actually do is we can actually use a function called tf dot um what is that one's like yup so we can actually output tf.one's like and say for example i pass through one this is effectively going to output an array which represents the number or has the value one but if i change it to be um one to a whole bunch of numbers oh god that's not a great example why is that so what we're actually going to get as the output of that is an array which has the same dimensions as whatever i've passed through the tf.ones like function and x rather than having a whole bunch of different values we're actually going to have a number a set of ones so what we're going to do here is we're actually going to pass through our anchor images our positive images and we are going to specify so that's going to be our think of this as our input and the output that we're going to have as a result of this is going to be a array which is made up of ones so think of this as representing our positive class so we're going to create an example like this which effectively represents our positive examples and we're also going to create a similar version which is going to be anchor and negative images except here rather than having ones we're going to pass through zeros because these are unverified right so this is effectively creating a labeled data set so let me just comment that because we don't actually need it so let's go on ahead and do this so we're actually going to create our label data set now okay it looks like we've got a bit of an error there uh what have i done okay this should be ones hold on zeros and we've also forgotten.zip yeah okay cool alrighty so that is our directories now concatenated together so what we've actually got is we've actually got uh one big data set which it has the shape of uh so it'll be our anchor images and then either a positive or negative file paths and then our either it's going to be ones or zeros so let me explain what i've gone and written there because it is quite a fair bit so i've written positives equals tf.data.dataset.zip so this zips stuff together and allows us to iterate through all three at the same time and then i've got inside of here i've got a set of brackets which effectively creates a tuple and then we've passed through our anchor and our positive so this is effectively going to be our different directories from what we had up here or specifically our file paths and then i've passed through tf.data.dataset.fromtensorslices and then to that we've gone and passed through tf.ones for our positive so this is effectively doing this so if i show you that so tf pf dot ones and then len anchor so uh should be one s so that is creating a big set of array which represents ones so this is effectively creating our labeled positives right because we're effectively saying our anchors plus our positives or if we pass through our anchor and our positive at the same time it should effectively represent a one which is a positive verification or a positive recognition so say for example you also wanted to do this on pictures of other people you would also pass through them as anchors and positive examples of them as positives and you'd be passing through the ones as well we're obviously doing it on one person we could definitely extend this out okay so that is our positive data set or beginnings of our data set now sort of staged so you can also see the shape here right so you can see that it's tf.stringtf.string and then tf.float32 so the string is going to be the file path to our specific image the second string is going to be the file path to either a positive or a negative image and the last value is going to represent uh whether or not it's a positive or a negative verification or recognition which in this particular case our positives are going to have ones and our negatives are going to have zeros so the second line that we wrote is negatives equals tf.data.dataset.zip so again it's zipping all of these together so we can iterate at the same time and then we've gone and passed through our anchor and our negative right and so when we have an anchor plus a negative we want to be or we want our model to be outputting zero because it's a negative recognition or verification and so what we've gone and done is we've gone and created a separate data set for our zeros so tf.data.dataset.from underscore tensor underscore slices and then to that we've passed through tf.zeros which have the shape of our anchor so if i go and type in tf.zeros and then lan anchor that is effectively representing or creating an array of zeros which match the exact shape of our negative classes or our negative samples so you can see that it has its shape 300 and we're effectively wrapping this inside of this tensorflow.dataset class so if i go and do that so again this is just creating a or putting that data inside of a data loader so tf.data.dataset.from underscore tensor underscore slices so we can begin to treat it exactly as what we did for add directories up here so let's create an example so this is let's say this is a class reps or class labels so then what we can do is convert it into an iterator class labels so this is purely for example you don't need to write this um and then what we can do is go iterator labels.next and this is going to get the next value right so we're effectively looping through each one of the labels in our class so this converts it into the same format that our anchor and our negatives are in effectively atf.data.dataclass okay we can delete that so we don't need that and then the last line that we wrote is data equals positives dot concatenate negative so this is just joining our positive and then negative samples together so we've effectively got it in one big data set which you can see there so we could do uh convert this to a numpy iterator uh let's call it samples and then samples dot next so you can see that this is one representation of our data or one sample right so in this particular case we have a so remember that label is going to represent whether or not it's a positive or a negative class so in this particular case we've got a 1 which means it is a positive example so this is a the file path to our anchor this is a file path to our positive image and then we've got a one which represents the label so again we could keep iterating through this and right now it's not shuffled so we're gonna have to go through all of the positives before we get to the negatives but once we shuffle it up or once we do a couple of next pre-processing steps you'll actually be able to see them in a better state okay so that is that now done now what we can effectively do is go on to our last step which is pre-processing these images so right now we haven't actually oh what did i write that so build train and test partition so what we're actually going to do is we're now going to use our pre-process function to pre-process these directories so that rather than just having the directory we've actually got the anchor image we've actually got the positive image and then we've got our label so either one or zero so what we need to do is we need to write a second function which actually processes a twin so right now if you think of our negative or anchor and a positive as a twin we need to pre-process these so load in both of the images and then concatenate it back into or return it back to this data set so let's go ahead and do this okay that is our twin pre-processing function now done so again this one's a little bit more straightforward so what we're effectively doing here is we've written d f pre-process underscore twin we've then passed through our input image or we're going to pass through our input image our validation image and our label which is effectively what we've got over here and out of that what we're going to do is we're going to return the loaded and pre-processed image so remember we're going to use this preprocess function which we defined over here so we're going to run pre-process on the input image pre-process on the validation image so this is effectively our anchor and this is could be either a positive or a negative we're also going to return our label so let's actually test that out so if i run pre-process not caps pre-process t break all right so pre-process twin and then if we uh this it could be an example all right so let's take a look at example uh so what we could do is we could pass this to a prepress test twin function and that effectively is what we're getting back so what i've just gone and done there is i went and spelt example wrong but we did more than that so we're grabbing the one example out of our data iterator which we had from up here and we're passing this through to our pre-process twin function down here and this star is effectively unpacking the values that we've got inside of this tuple so rather than writing out or extracting each one of these or unpacking each one of these it effectively unpacks them and passes it through to our function so could we do it like this no it's going to throw errors we pass through a star that effectively unpacks it now you can see that the result added here so if i type in res equals what i'm actually getting is three things so uh if i grab the first thing so this is ow well let's actually take a look at the length so you can see it right so we've got three different values or objects inside of this result the first thing is going to be our pre-processed image so you can see that this is actually running our pre-processing function and loading our image as 100 pixels by 100 pixels by three channels so if i went and plotted that plot dot i am show you can see that's an image of yours truly and if we take a look at our second value inside of that results variable you can see that that is a positive sample so this should effectively have a label of one so if i go and type uh what is it res two you can see we've got a label of one there so all we've done up until here is effectively this is the state that we want to get to but now we need to do it on every single one of our examples inside of this data variable here so let's quickly do a recap because we've done quite a fair bit so first up what we did is we used the tf.data.dataset.listfiles method to load up every single file inside of our anchor a positive in our negative directory then we went got voice just broke there then we went and wrote this pre-process function to be able to load our image from that directory resize it to be 100 pixels by 100 pixels and we also scaled it to be a value between 0 and 1 because that is going to help our gradient descent and effectively help us to optimize our model so we haven't used that yet effectively i've demonstrated how to use it then what we went and did is we went and created our labeled data set so we're effectively creating sets of twins not triplets so where you might have seen this done in siamese neural networks before you typically create a triplet we're creating a twin or a tuple rather than a triplet because we're going to be passing through two values at the same time and so we've created our positive samples and our negative samples so really our positives are just going to be our anchor image plus our positive image and the label that we're going to get out of this is a 1. we then created our negative samples and our negative sample is going to be our anchor image plus our negative image with a label of zero and we went and concatenated those together so we can begin to work with them then we went and wrote our pre-process twin function and the last thing that we actually need to do is actually go and apply this to this data object over here so we're actually going to use the data.map method to be able to go and apply our preprocess twin function so we've gone and done a lot of hard work already all we now need to do is go and build up our data pipeline so i'm going to do this and then i'm going to show you the results okay that is the beginnings of our data loader pipeline and now done so what i've gone and written is data equals data dot map and this is effectively doing app or running a pre-processed twin function from over here so it's going to take our input image path our validation image path and our label and it's going to return the actual image loaded and pre-processed for our anchor the image loaded and pre-processed for either a positive or a negative image and it's going to return our label so now when we go and take a look at this data set or this data object we should effectively have our images i'll show you that in a second the next two steps that we applied to our data loader pipeline is we wrote data equals data dot cash so we're cashing our images and then probably one of the most important pieces is we've gone and shuffled it up so this effectively muddles up all of our images so that we're not just getting positive images and then negative images they're all mixed up so when we go and split our data into a training and testing partition we're going to have ideally we should have a mixed set of samples so in order to do that i've written data equals data dot shuffle and then we've specified a buffer and then we've specified a buffer size of 1024 so now if we go and take a look at our data object you can see that we've got a shuffle data set which has the shape of 100 pixels by 100 pixels by 900 pixels 100 pixels by none so this is our anchor image this is going to be either our negative or a positive image and then this will be a label so if we go and create a numpy iterator again samples and then if i grab samples dot next you can see that we've got our data all ready for us and ready for training so we've got let's take a look at our length again we've got three values and remember these are going to be either our anchor image or it will be an anchor image as our first value it'll be either our positive right negative image for our second value and it's going to be our label so if i run samples.next and then grab the first value that is going to be our anchor image so we can go and plot that out using plot.i am show cool and then if we go and grab our we should actually save this in a variable otherwise it's going to keep changing um so let's go samp samples.next and if i pass through samp zero that's going to be our anchor image pass through value one or index one that's going to be so this really this is really really good so this should ideally mean that this is a negative class so what does that what label should we be expecting here i'll give you some thinking music all right that's enough thinking music that was terrible okay so ideally our label should be label zero if it's not label zero it's gonna be okay cool all right so we've got our we had our what was it our anchor image we've got a negative image and we've got our label which is 0.0 so that effectively means that we have the correct label in this particular case okay we could do this again so if we go and run our samples.next and if i type in plot.i am show samp zero so that is an anchor image of me we take a look at our sample again so that looks like it's a negative sample and again we've got class zero so if we go do it again let's see if we can get a positive one so myself we need to run this again another image of me that looks like it's another negative sample let's wait for a positive one nope come on okay this should be a positive sample so you can see we've got an image of me we've got another image of me as our positive and if we go and take a look at our label you can see it's one so this is effectively how a neural network is going to work we're going to pass through and this is effectively one shot classification so we can pass through a anchor image or an input image and then you can pass through a separate image which you want to validate against and it's going to tell you whether or not that's the same person so say for example you wanted to do verification on multiple images or recognition or multiple people you would effectively just include more positive examples right so you'd include say for example uh you wanted to do it on your best friend you'd include images of yourself as a positive example so you'd have images of yourself self as an anchor yourself as a positive and the label for that would be one you then include also images of your best friend as an anchor and your best friend as a positive and that label would be one when it comes to actually verifying what you would be doing is you'd be passing through your anchor and if you want to verify against yourself you'd pass through an images yourself as the second input to your siamese neural network if you wanted to do it on your best friend it would be an anchor image and then a verification image of your friend to determine whether or not that's the same person okay but enough on that we've actually gone and created our data set now so the last thing that i want to do is just create a training and testing partition so i'm just going to delete all of this so let's create our training partition first okay that is our training partition now done so i went and wrote three additional lines of code there so train underscore data equals data dot take and this effectively is grabbing a certain percentage of our data sets i've written data dot take and then i've passed through round and then specifically so this is effectively just coming up with how many images right so i want to effectively take 70 as my training partition so this does that calc so we're grabbing the length of all of our data multiplying that by 0.7 and then we're rounding the image which just so happens to be 420 and then what we're doing is we're using the data.take method with that value to take the first 420 images so i could just as easily replace this with 420 here and then we've gone and batched our data so we're now going to be passing through our data as batches of 16 images so i've written train underscore data equals train underscore data dot batch and then to that will pass through the value 16 and then we're using the prefetch method and this effectively um starts pre-processing the next set of images so that we don't bottleneck our neural network when we go and train so written train underscore data equals train underscore data.prefetch and then eight so that is our training partition now done so we went and wrote went and wrote three additional lines of code there's data.take data of traindata.batch traindata.prefetch cool so that is now already and again if we go and take a look at our data set we should if i type in train data we will now so the shape is slightly different now so we've now got a so if we took a look at our data values of data that was what our data value looked like our data objects we had 100 so you've got this is the shape here this is the important bit you would have had 100 by 100 by none 100 by 100 by none and this is 100 pixels by 100 pixels by the number of channels in that image or our anchor same thing for our positive and then we had this value here which represents a single unique value in our train data we've got a slightly different shape and this is because we're now batching it so you can see that this none value is actually representing the number of batch or number of images that we've got in our batch so it should effectively be 16 right because we have gone and created batches of 16. so if i go and grab let's do the iterator thing again so train data dot as numpy iterator i'm going to call this train samples and then i can type in train dot sample or sample equals train samples dot next if we take a look at one sample and take a look at the length it's three uh and if we go and grab the first value this should effectively be 16. cool so we've now actually got 16 images inside of each sample that we actually have so rather than having just one image per value that we get back from our generator we've actually got 16 images now cool so that is our training partition now done let's just create a validation partition i don't know if we're actually going to use this but we'll do it anyway so uh testing partition sorry okay so let's go ahead and create our testing data okay that is our testing partition now done so what we went and did there is we skipped the first 420 observations so that we don't pick up our train sample and then we went and took the last 30 and so again i just wrote two lines of code there so test underscore data and i haven't batched this we might batch it later if we need to perfectly fine for now test underscore data equals data dot skip and then we've passed through again so exact same value that we had up here so this should effectively be 420 420 and then we've gone and written testdata equals testdata.take and then we're passing through the last 30 so if i grab that that is gonna grab the last 180 images cool all right so that is pretty much our data now ready so we've gone and created our training partition and our testing partition we should probably batch this as well let's do it while we're here testdata.batch and i don't know we'll do batches of 16. and prefetch eight cool all right that is our that should be everything now done okay so we went and did a ton of stuff there so we went and grabbed all of our image directory so in this video we went and did effectively four different parts so we went and grabbed all of our image directories using tf.data.dataset.list files so remember we went and grabbed all of the specific strings for the images we then went and pre-processed them so we went and loaded them up we then went and resized them to be 100 pixels by 100 pixels and we scaled them as well we then went and created our label data sets we created our positive samples and our negative samples and i sort of explained how you might go about doing this if you wanted to verify multiple people and we concatenated those together we then went and pre-processed our twins so rather than just working with the strings we actually loaded up our images into our data set or set up our data pipeline to do that and then we went and created our data pipeline so we went and fleshed those out so let me just clean this up so you can see it a little bit better so we built up our data pipeline so we mapped using the pre-process twin function which effectively leverages our pre-process method from up here we then when encased our images shuffled them up we then created a training partition and a testing partition so this puts us in good stead to actually get to some deep learning in the next part of this tutorial but on that note that about does wrap this up so thanks again for tuning in guys hopefully you enjoyed this tutorial if you did be sure to give it a big thumbs up hit subscribe and tick that bell and i will see you in the next one what's happening guys welcome to part four in this siamese neural network series where we try to implement a siamese neural network model from a paper all the way through to a final end application that we're eventually going to be building with giving now this tutorial is the one that i've been waiting for because we're starting to get into the deep learning component of this series so let's go on ahead and take a look as to what we're going to be going through in this video so we are going to be doing three key things here so we're going to first up build an embedding layer and this is effectively going to form almost like a feature mapping pipeline for our specific model so we'll pass through an image this is going to go through our embedding layer and effectively convert our raw image to a data representation that's going to represent what we're going to pass through to our siamese neural network so think about it as though we're effectively translating it to something that's going to allow a neural network to determine whether or not the person is verified or not so it's almost like a data translator to a certain extent then what we're going to do is we're going to create an l1 distance layer so i'll show this a little bit more once we actually take a look at the paper but the way it's sort of going to work is we're going to have two streams of images we'll have our anchor and either a positive or a negative and these are streams are sort of going to be like rivers and the way that we compare them is using this l1 distance layer so we actually bring the rivers together and we use that l1 distance layer to compare whether or not the images or the embeddings are similar enough to be verified or not so that's what we're going to do in step two and then last but not least we're going to compile them together to be able to build assignments neural network and then in the next video in this series what we're actually going to do is start training our siamese neural network model okay but without further ado let's actually get to the tutorial and let's do it so ready to do it let's get to it alrighty so what we're going to do in this tutorial is three key things let me bring the mic a little bit closer so first up what we're going to do is we are going to be building our embedding layer then we are going to build our distance layer so this is going to be our l1 distance layer and then last but not least we're going to make our siamese neural network model now specifically what we're going to be doing so remember we are replicating this paper here so siamese neural networks one shot image recognition now we are specifically we've made a few tweaks and the numbers are going to be a little bit off but that's fine we're going to be building this neural network here now i said right at the start think of this as having two streams of information so we are going to be passing through two input images and we are effectively going to be combining them down over here where it says l1 siamese distance so you're going to have two streams so an anchor and a positive or a negative these are going to be passed through to our embedding layer and then they're effectively going to be compared once you get down to here so we're going to compare them using our l1 siamese distance layer okay let's kick this off so rather than talking anymore let's actually start building it so we are first up going to create a function that takes our input image now the paper uses an image which has the shape 105 by 105 but we've gone and why is that weird we've gone and converted it to be 100 pixels by 100 pixels so these numbers at the top here represent what our output shapes are going to be these values down the bottom specify what are the different layers within our neural network now because we've gone and converted our images to be 100 pixels by 100 pixels they're going to be a little bit off but that's perfectly fine it's still going to work so first up what we now need to do is create a function which builds our embedding layer so we're going to create a function and we're effectively going to be passing through all of our different layers so let's start setting up our function and then we're going to add to it incrementally okay so that is the beginnings of our model so i've gone and written two lines of code there and again we're going to build this incrementally so i've written d e f make underscore embedding so this is defining a new function called make embedding and i've closed it off with a colon and then what we're going to be doing is we're going to be returning our final embedding model now you're probably thinking nick where is this model value coming from or model class remember right at the start in one of our earlier tutorials we went and imported a number of different tensorflow dependencies so namely we imported our model class and we also imported a bunch of different layer type components so we imported the base layer class we also imported conf 2d which is going to be used here we imported dense which is going to be used over here we import and i'm going to explain this in more detail don't stress we import a max pooling 2d which is used over here so we perform a convolution and add a relu activation over here and then we go and perform the max pooling so it's going to be convolution max pulling well convolution relu max pulling convolution rally max pooling convolution relu max pooling convolution relu fully connected so this is where we go and perform our siamese distance layer and then we go and perform another fully connected with a sigmoid then produce an output lots of fancy words but really we're just passing it through a data pipeline okay so what we or neural network pipeline really so what we're going to do is we're going to build this up step by step so first up what we want to do is we want to deal with that input so let's go on ahead and create our input and now this is going to be a 100 by 100 pixels so these numbers are going to be a couple of pixels off but it should be pretty much the same so to define our input we can use our input layer so right up here so let's go ahead and create our input layer cool so that is our input layer now created so i've gone and written inp equals input and then i've specified the shape that we want our input to be so in this particular case it's going to be shape equals 100 pixels by 100 pixels by three channels so if i copy this out of here so that is our input now defines if i've write out input you can see that what we've got is we've got a keras tensor with the shape of none because this represents the batch size and then we've got 100 pixels by 100 pixels by three we can actually pass through the name here as well um name is input image copy that rather than having this weird name over here so you can see that our input layer is going to be called input image cool so that is the first part of our neural network done now if you wanted to stick with the exact same shapes as the paper all you need to do is change the input shape to be 105 by 105. so if i wanted to do that type 105 by 105 and you can see that our input tensor is now going to be 105 pixels by 105 pixels cool all right what's next so the next layer that we want to build in is our convolution plus a relu activation and in this case so remember our convolution is composed of two key things well really three key things but we're going to ignore the last so our convolution takes the number of filters that we want to pass through so in this case it's going to be 64 filters and our filter shape is going to be 10 pixels by 10 pixels now normally you'll take a look at a parameter called stride and this is how far our pixels or our filters actually move across the image but in this case we know stride is going to be one so you don't you can sort of ignore that and we know that there is an activation that we need to apply here as well which is relu so let's go ahead and implement our convolutional layer and this is going to be using the conv 2d layer so let's go ahead and do it i'm going to call this layer c1 or we're going to name it created as a variable called c1 so you'll see that in a second okay that is uh this should be capital d that is our con to the d layer now created so i've gone and written c1 equals conf 2d and then we're going to pass through that we want 64 filters as our paper says we've gone and specified that we want the shape to be 10 pixels by 10 pixels 10 by 10 and we've gone and applied an activation which is a rally which again in the paper says it's got a relu activation there and then in order to pass through or start connecting our neural network together we're grabbing our input and we're passing it through to our convolutional layer so this is how the keras functional api wants its input so let's actually take a look at this so if i grab this layer over here and we're going to paste it up there so we've got our input that we defined already we've got our convolutional layer now so if we take a look at c1 you can see the shape is 96 pixels by 96 pixels by 64 channels which is pretty close to this right but it's not perfect so remember this is this represents our input shape these are the actual layers so here we've got 64 channels and we've got 96 pixels by 96 pixels we've actually got 96 by 96 by six wait that one's perfect so that's perfectly fine okay so we're going to ignore that oh wait it's because we had 105 105 up here i thought there was something weird there so if we change this to 100 by 100 there you go so you can see it's going to be a little bit different so rather than having the exact same shape which is 96 by 96 by 64 we've got 91 by 91 by 64. if we wanted to have the exact same shape we could change this to be 105 by 105 and you can see we're getting the exact same shape now but that's fine we are going to stick with 100 by 100 and then we are going to keep going so we if i change this back to 100 and 100 that is effectively what we've got here now now we've got one more layer that we need to implement which is our max pooling layer and then really everything from here on out is pretty much repeating itself at least for our embedding layer so let's go ahead and implement our max pooling layer and then we should have our building blocks sort of ready so let's do it okay that is our max pooling layer implemented so i've gone and read in there so i've created a new variable called m1 to represent our max pooling layer or max pooling layer 1 because we're going to do it multiple times and then i've set that equal to max pooling 2d pass through that we want 64 units of that so you can see that it's 64 units and then we want it to have a shape of 2 by 2 so it's effectively going to take the max value out of a 2x2 area and return the max value so it's effectively condensing down the amount of data that we've got and we're specifying padding as being the same here so this is something that i noted when i was building this up originally you want to have the padding as being the same in order to replicate a similar output shape and then to that we are going to be passing through a convolutional layer so if again we copy this out and if we take a look at layer m1 you can see it has the shape 46 by 46 by 64. here it's expecting 48 by 48 by 64. but if we go and change our input layer back to 105 you'll see that it mimics that exactly so if i go and do that you can see we have in fact 48 by 48 by 64. 48 by 48 by 64. cool so this means that we effectively have our core building blocks ready for this neural network right so we've gone and implemented a convolution plus relu plus our max pooling layer so these two layers sort of form a core block so you'll often refer to a specific neural network block referred to really frequently whenever you're building neural networks so these two are a block which gets replicated a few different times now obviously they've got different shapes so in this case you can see for at least for our first block we've got a convolution with 64 units or 64 filters which has a shape of 10 by 10 and then a max pooling layer with 64 units which has a shape of 2x2 but if we scroll on you can see the convolution shape changes a little bit the max pooling shape changes a little bit so on and so forth so let's go ahead and implement our next block which is this two of these two over here so we can do that reasonably easily all we need to do is copy these should we copy them or write them from scratch let's write them from scratch okay so this is our first box i'm just going to comment this up so first block and then second block let's do it okay that is our second block now implemented so i've gone and written two lines of code there so i've written c2 equals conv 2d and this one we're going to pass 128 filters with a shape of seven by seven so again our second block is going to have a convolution which is 128 filters with a seven by seven shape with a max pooling oh we'll come back to our max pooling layer all right so convolution with 128 units with a seven by seven shape which haven't has a relu activation relu activation and then we're passing through our in the results of our max pooling layer so remember our max pooling layer is called m1 so inside of parentheses we are appending to the end of it and we're passing through our max pool or the output of our max pooling layer of output from our graph which is going to be m1 storing that inside a variable called c2 so this layer over here is implementing this layer over here so if i zoom out a little it's implementing this over here coolio then we've gone and done our next layer as well which is our max pooling layout so you can see mac's pooling over here and we've gone and written m2 equals max pooling 2d and then i've gone and specified that we're going to have 64 units with a shape of 2x2 which has a same padding and then to that we're going to be passing through our c2 convolution or the output from our c2 convolution which you can see there now if we go and copy this and as per usual paste it up over here so if we take a look at the output from our c2 block we're going to get a shape of 42 by 42 by 128 which is pretty much mimicking this are we on one f yeah so we're on 105 right now which is why we're getting exactly the same shape so 42 by 42 by 128 and then if we take a look at our max pooling layer we're getting 21 by 21 by 128 which is 21 21 by 128 so it all is looking good at the moment so remember this is going to have exactly the same shapes because outside of here we've got the shape 105 by 105 in our actual model we're going to be using 100 pixels by 100 pixels but this is sort of like a good sense check really okay so what's next so we have oh what have i done there nope wrong so we've gone and done this two layers we've gone and done these two layers we've got to do this layer and then we've got to do that over there i think yes and a flattened all right let's go ahead and build our third block four okay that is our third block and now done so we've gone and written two lines again and again pretty much exactly the same as what we've written up here but now we're passing through our map the results from our max pooling two layer so this one over here to our next convolution layer and again we've gone and written the exact same thing or pretty much the exact same thing so c3 equals conv 2d specifying that we want 128 filters with a shape of 4x4 128 filters with a shape of 4x4 so that over there you can see that there and we're specifying that we want an activation of relu activation of relu and we are passing through the output of our max pooling the second max pooling layer or the max pooling layer from our second block as the value that's going to be accepted into that convolutional layer so m2 over here then we've gone and specified another max pooling layer which is pretty much identical to what we wrote over here and what we wrote over here again same exact number of units and a same exact shape so m3 equals max pooling 2d 64 comma and then inside of parentheses our shape which is two comma two and then we've gone and i specified our padding as equaling same and passing through the results of our convolution from over here as the value at the end cool all right so what's left so we've really we don't have too much left now so we've gone and done let's take a look now where we're up to so we've gone and done our input we've handled this we've handled that and with that so this is our first block this is our second block this is our third block so our next two lines are going to be a convolution plus a fully connected layer so this is effectively going to be a convolution and we'll probably need a flatten plus a dense layer over here so let's go ahead and implement that okay i think that is our embedding layer now done so i went and wrote an additional three lines of code there so we've gone and created a final convolutional layer which is specified as c4 equals conf 2d and this got units of 256 filters which is over here so 256 256 with a shape of 4x4 so 256 with a shape of 4x4 with an activation of relu activation of rally cool and then 2 that we are passing through at the results of our third max pooling layer so we're passing through m3 and then we are flattening our convolution so you'll see this in a second but effectively we're taking all of the outputs of our convolutional layer which has three dimensions and we're flatting it flattening it into a single dimension so let's actually take a look at this so i'm going to copy this over here paste it up and this should oh we didn't actually paste our m3 layout there so let's copy this cool all right so if we take a look at the output of m3 which was a max pooling layer from our third and block we didn't actually take a look at those shapes let's actually do this properly so c3 has the shape of 18 by 18 by 128 which is going to be over here so 18 by 18 by 128 output from our m3 layer is going to be nine by nine by 128 nine by nine by 128 and then we are taking this m3 output and we're passing it through to our convolution so if i take c4 you can see it's 6 by 6 by 256 and again keep in mind this is based on the input shape of 105 by 105 so if we change this to 100 by 100 the output shape is going to be a little bit different perfectly fine so uh what were we doing so c4 has shaped 6 by 6 by 256 6 by 6 by 256 now just so happens if we multiply 6 by 6 by 256 we get uh 9216. so those units are going to be flattened in our feature vector so if we take a look at the shape of our f1 shape we have 9216 units which is the output which is effectively this multiplied by this multiplied by this which you could see over there then what we're doing and remember f1 is just flattening all of these elements together so rather than having it in the shape of 6 by 6 by 256 you're just going to have a single dimension so then and that is the output of the flattened layer and then if we take a look what we've gone and done is we've passed our flattened layer to our dense layer which should give us 4096 units back so if i type in d1 you can see that we've got 4096 which is our 4096 feature vector which eventually then gets passed to our siamese distance layer but we're going to come back to that in a second and keep in mind the last activation that we had was a sigmoid as specified over here so fully connected plus sigmoid and again this is a little bit uh densely written up but you can sort of see how these all fit together okay i think that's it now what we need to do is we need to pass this through to our model class in order to sort of compile it because at the moment we haven't actually gone and brought this all together as a model so let's go ahead and do that so when i first wrote the function up here so def make underscore embedding then i also specified this last layer so return and then model inputs equals outputs equals and then names equal so now what we need to do is specify these so let's do it okay and that is the final component done so i've gone and ridden so i've basically just gone and filled this out so i've written model equals inputs and then inside of that i've passed through a set of square brackets and pass through our input which is right up here and then i've gone and specified outputs and again what i've gone and passed through is our final layer which is this dense layer over here so what we're going to be outputting is this big feature vector with 4096 outputs so this is what i meant by the two streams or the two rivers so we're effectively going to have two rivers of data flowing through or two rivers for our neural networks and each of those rivers is going to be outputting a feature vector of 4096 units so this is almost like translating uh input images of our faces into a embedding or a feature vector so yeah what have we gotten done there so i've written model equals inputs equals model and then inside of parentheses i've specified inputs equals and then in square brackets pass through input comma outputs equals d1 which is this and then i've specified name equals embedding so if i copy this bring it over here that is effectively compiling our model so if i write a mod equals model that's our final model so if we uh can we type in summary that gives us our final model so this is the model based on 105 by 105 but that effectively gives you an idea of what our model is actually or actually looks like now the cool thing about this is that if i bring this shape over to this side this is effectively one final model which mimics what we had in our paper so this paper is obviously a little bit small but if we take a look that input image over here so 105 105 uh by one so that's our input image so this looks like it might be a single channel we're going to be doing ours on color hence why it's going to be a 3. but 105 105 by 3 that's our first layer if we take a look at our next layer 96 by 96 by 64 96 by 96 by 64. take a look at our next layer 48 by 48 by 64 48 by 48 by 64. and then if you go all the way to the end i'm not going to do every single layer we've got a dense layer which has 4096. we're in business guys so that is our embedding layer now done now what was i going to do so because our input shape is going to be slightly different if we type in what we're actually going to have is 100 by 100 up here so if i go and run all of these layers again you can see that our input changes the output shape a little bit but eventually we're going to get a 4096 output layer because that's how many units our dense layer has anyway so we're good to go there so that is our make embedding function and now done so if we go and run this we're in business so if we wanted to go and create this model so let's go ahead and do it so if i type out model equals make embedding that is our model now generated and if i type in model.summary that is our model so we've got all of our different input layers and again you could name these and make it a little bit cleaner but i haven't done that i've sort of skipped it but this effectively forms our embedding so which is i'm going to bring this back over here which is pretty much from here all the way through to here done apart from this l1 siamese distance label we're going to do that now but that's that done pretty cool right so that is let me zoom out or not zoom out so that is step 4.1 now done so we've gone and built our embedding layer now what we need to do is we need to bring them together so remember i was talking about the two rivers right so the true rivals as part of our neural network graph we're gonna have our anchor and we are going to have either our negative or a positive image which forms the basis of our one shot classification so if we want to join these rivers together we need some way to compare them right so what we're actually going to do is we're not going to be adding them together so effectively our rivers combining we're actually going to be subtracting them so this is our l1 siamese distance layer and it's going to tell us how similar our images actually are which is effectively what allows us to perform our facial recognition or facial verification so let's go on ahead and define this distance layer so it's going to take the embedding or the output of these embeddings so our 4096 feature vectors going to take the output or the those is the input and it should effectively output a value out of this we're then going to pass that to a fully connected layer and then output a final result so let's go ahead and do this okay so the first part of that that's our distance layer done no so that the first part of this is defining a new class so we're going to be creating a new class for our custom layer now this is actually really really cool so as part of producing this tutorial i actually did a ton of learning but this actually shows you how to create a custom neural network layer so if ever you need to go and do some other custom stuff this gives you a good sort of template as to how to go about doing it and i've also defined this so that when we actually go and export our model we'll be able to bring this layer as part of it as well so let's go and finish this out okay that is our one distance layout now produced so i went and wrote four additional lines after i showed you the class layer and these are the init section is pretty sort of struck itself or the call section is a little bit more important but let's actually take a look at it in its entirety so i've written class l one dist and i've written this in caps you don't need to but it's good practice or sort of common practice with python so l one disk so these are all in caps and then to that we're passing through our layer class now this comes from all the way up here it is the abstracted class or the base class for our keras layers and then what we're doing is we're performing a little bit of inheritance inside of our init function so i've written d e f and then underscore underscore init underscore underscore so it's the base init method inside of a python class and then i've gone and passed through self so we can operate on ourselves and then i've gone and passed through asterix asterix kw args so this allows you to work with the this specific layer as part of a bigger model so when it comes to actually exporting and importing this having this actually defined makes your life a lot easier so when we actually go and export it we're actually going to use the abstracted versions so passing this through means that if we wanted to go and pass through specific keyword arguments it's going to handle them innately cool so def underscore underscore init underscore underscore and then inside of parentheses self comma asterisk asterix kw args and then close parentheses and then colon and then we're just going and performing inheritance so super and then parentheses dot underscore underscore this should actually be underscore underscore init afterwards and then close parentheses cool so that is that now covered now this is where the magic happens so let me write a comment magic happens here so the call function is actually or actually tells this layer what to do when some data is passed to it so i've written d f pass through self and then remember our two rivers are going to combine so our anchor image and either our positive or negative is going to be brought together and we're going to compare their similarity so i've written input underscore embedding so this is going to be the first river and then validation underscore embedding output of our second river so this is effectively going to be our anchor embedding this is going to be either our positive or our negative embedding and then we are returning tf.math.abs so this is going to return an absolute value and we are subtracting the validation embedding from our input embedding so written input underscore embedding minus validation underscore embedding so that is our l1 distance layout now done so when we actually go and call this it's effectively pretty much the same as how we might call another layer so we can write l1 equals l1 dist and has no attribute l1 disk in it have we gone and written something wrong there let's check okay that is not working and the super object has no attribute underscore l1ds to underscore underscore init oh i don't know maybe i typed something wrong there oh actually i didn't uh i didn't go and rerun that cell again my bad all right so that's worked out so remember we had it like that so if i go and run that there you go that's the error that we're getting there so if i just go and pass through those underscores it again we're good so that is our one distance layout now produced so if we go and take a look at it there's not going to be nothing inside of it because we're not passing anything through yet but what we're effectively going to do is we're going to pass through effectively our anchor embedding and our validation embedding and we are then going to combine this into a dense layer which is over root yeah so our fully connected layer and then to produce our final output so that's the last bit which we are going to do over here so that is our custom l1 distance layer and this is a defining characteristic in the siamese neural network sometimes what you will see is that they implement a slightly different function here so they'll actually have three rivers or three streams so that you'll have an anchor and a negative and an anchor plus a positive at the same time and you'll actually compare all three of those at the same run or is part of the same run in this case we're just doing it with two streams or two images as part of our graph so that is perfectly fine so let's take a look at what we did there so we first up so we're creating our siamese distance class uh l1 distance class and we've gone and defined our init method which is pretty self-standard so this performs inheritance and then we're doing our actual magic so this is actually performing a similarity calculation which is effectively just this really we're just grabbing one stream we're subtracting it from the other and we're performing an absolute function over the top of it nothing crazy there cool all right that is our one distance layout now defined so that's 4.2 done the last thing that we need to do is combine all of this together so right now we've got sort of like abstract uh components so we've got our embedding for layer or our embedding model we've got our l1 distance label right now that our streams aren't sort of all running simultaneously so we actually need to bring all of this together to produce assimes neural network so that's what we need to do in step.4.3 so what we're going to do is we're going to create another function so def make siamese model and we're going to bring it all together so let's actually do our first bit and then we'll take a step back and take a look at what we've gone and done okay so we are going to be defining another similar model to what we did up here but we're going to bring it all together now so what i've gone and defined is first up i've written def make underscore siamese underscore model opera open parentheses close parentheses and then colon then i've got a written comment so handle our inputs so first up what we're doing is we're defining our two inputs because we're going to have two streams we need two images that are coming through so we're going to pass through our input image which is going to be specified as input underscore image and then i've set that equal to input name equals input underscore image so this is effectively creating one input here this is creating a secondary input so let me actually separate this so this is our going to be our anchor image input in our network this is going to be the validation image in the network right so input image equals input and then i've gone and passed through two keyword arguments of specified name equals input image and shape equals a hundred by a hundred by three because that's going to be the shape of our input images and then we've gone and passed through our validation image so i've written validation underscore image so that's our variable name and i've set that equal to input and then again two keyword arguments this time the name of that input layer is going to be called validation underscore image and i've set that equal to shape 100 by 100 by 3. cool what we now need to do is we need to take these inputs and actually pass them through to our embedding model because where right now our raw inputs are just going to be we're at this stage we need to take these raw input images and pass them through to our embeddings and then what we're going to do is combine them with our siamese distance layer so let's go ahead and do that okay let's pause there i realized that what we should have done is rather than calling our embedding model just model let's actually call this embedding because otherwise it's going to be a bit weird and it's going to get kind of confusing so i'm going to run this again run this and then convert this to embedding cool so our embedding layer or our embedding model is actually going to be called embedding now rather than it just being called model kind of dumb doing that so we are then going to wrap this up let me actually go and finalize it and then i'll explain it okay so that is our siamese distance layer done and our we are actually passing through our two streams so i've gone and ridden siamese underscore layer equals l1 dist which is beginning to use our distance layer and then i've just gone and named it again i'm super pedantic i like naming so siamese underscore layout dot underscore name equals distance so this means that when we actually take a look at our model summary you'll actually be able to see that name there and then i've gone and created a new layer now so i've written distances equals siamese layer and then to that what we're actually doing is we're passing through this input image to our embedding model which we had from up yeah so again i've wrapped this inside of a function but it's kind of like redundant at the moment doesn't matter we'll take a look at that later or there's possible improvements there so if somebody goes and cleans this up do let me know i'd love to see it so we're taking our input image which is effectively this and we're passing it through to our embedding so if i go and run that and type in embedding and input image that is effectively what we're doing so we're taking this input image which has a shape of a hundred by hundred by three and the output that we're going to get out of that is our 4096 units which are out of this then what we actually going to do so what we do is we actually do this on our validation image as well right and if we go and type in so let's do a input embed equals this and val embed equals embedding and then validation image cool so that is effectively our two sets of data now transformed into our feature vectors which is a 4096 unit output so if we take a look input embedding is going to we're passing through an image which is 100 by 100 by 3 and the output that we get is a unit or an output of 4096 values right and again same thing that we're going to get from our validation embedding 4096 units cool so then what we finally do is we go and take our siamese layer which is going to be called l1 dist and because remember this is going to take two inputs so if we take a look at our call function it takes our input embedding and our validation embedding which is effectively what we've got there so i'm just going to delete that because we don't need that and that so what we're doing is we're taking our siamese layer and passing through our input embedding input embedding and validation embedding and that is going to output a 4096 vector or a vector which has 4096 units which we're then going to finally pass through to another dense layer which is going to pass through either a one or a zero so we've still got to do that later but by passing through these two embeddings to our siamese layer we get a result of 4096 so this represents the distances between our input embedding and our validation embedding now all we need to do is pass this through to our final layer to tell us hey do these embeddings or are these embedding similar enough to consider them the same person so that's our final component that we need to do in here so let's go on ahead and do that and that's done guys so that is our siamese model now produced so remember i said that the last thing that we needed to do was combine these distances into a final fully connected layer with a sigmoids with a sigmoid activation that is what this line is doing so i've written classifier equals dense and then we're going in specifying that we want one unit one unit specifying that we want an activation of sigmoid sigmoid and then to that what we're doing is we're passing through these distances which have a shape of 4096 units so we're passing through 4096 units in and we're gonna get one output value out which will either be a one or a zero so if we take a look if i grab this line and bring it over here this should be distances so let's define that distances equals time means layer boom that is our classifier you can see that we are going to have an output of shape one output of shape one by one now the last line that i wrote is effectively combining all of this together so i've written return model remember this is our base level model class and i've specified our inputs as being our input image and our validation image so our input image and our validation image and then our outputs are just going to be our classifier so remember we're going to have both of our streams running simultaneously they're going to combine together on our l1 distance layer and we're going to output our classification layer which is this final little bit over here yeah so what do we write there so return model and then inputs equals and then inside of square brackets we're passing through two inputs so input underscore image comma validation image so these are these two components over here and then comma outputs equals classifier which is outputting this classification layer there and then we'll specify the name as siamese network so if we go and do this let's actually just copy this over here siamese network equals that did i go and run that yeah okay cool so if i run siamese network that's assimi's network fully done now so if i type in dot summary take a look at that so we've gone and passed our input image part of me so input image a validation image which then goes through our embedding layer they then get combined through to our l1 distance layer which you can see there and we then combine that to a dense layer which then outputs our single value that is pretty cool guys so we've now gone and combined all of that together so that is our siamese neural network now done now if we go and use our function if i go and run this so i'm going to call it siamese network or model and run make siamese model that is exactly the same thing let's write down the bottom dot summary that is our siamese neural network ready for one hot or one shot classification not one hot one shot cool so we've now successfully gone and built this now in the next video what we're going to start doing is start training this model but for now let's actually take a look what we did so i'm just going to minimize that and that so we can actually see so in step one what we went and did is we went and built our embedding layer so we went and defined all of the different blocks that we saw over here we then went and defined our distance layer which represents our l1 distance and remember it's really just subtracting the two rivers from each other to determine similarity and then last but not least we combine them all together inside of our make underscore siamese underscore model function which then returns our siamese model which looks like this and on that note that does wrap it up so thanks so much for tuning in guys hopefully you enjoyed this video if you did be sure to give it a big thumbs up hit subscribe and tick that bell see you in the next one thanks again for tuning in peace what's happening guys my name is nicholas tronite and welcome to part five in the siamese neural network series in this series what we're doing is we're going from a sophisticated research paper and building a full-blown siamese neural network that allows us to perform facial recognition on our face in this video we're going to be focused on training our neural network let's take it if you look as what we'll be going through all right so what we're going to be going through in this video is quite a fair bit so we're actually going to be going through five key steps and these steps are all to do with training a neural network now step one is all going to be to do with defining our loss function and this is going to be the loss function for assimi's neural network we're then going to define an optimizer this is going to help us back propagate our updated weights throughout our neural network we'll then build a custom training step so this is a really useful skill when it comes to building sophisticated neural networks because it allows you to handle multiple different types of inputs something which is definitely required when it comes to building really advanced styles of neural networks once we've built our custom training steps step four is all going to be to do with defining our training loop so we're going to blend our training step with our training loop to be able to go to step five which is training a model on that note ready to do it let's get to it alrighty guys so in this video as i was saying there's going to be five specific steps that we need to go through and this entire video is all to do with training our model so we're actually getting into the nitty-gritty now so first up what we need to do is set up our lost and our optimizer now remember the whole purpose of our neural network or what it's trying to do is minimize that loss because that means that it's performing better for this specific function that we're trying to minimize then we are also and we're going to set up our optimizer as well there so that's going to help us perform back propagation through our custom neural network there's a helicopter flying past my desk right now kind of strange but anyway um okay so then in 5.2 what we're going to do is we are going to establish some checkpoints and this basically means that if we need to or if something screws up in our neural network as we're training we've got almost like a placeholder that we can go back to to reload our neural network from we are then going to build our train step function so this effectively defines what happens when we train on a single batch now the basic process is that we use our model we pass through some data we get a prediction we then calculate the loss so whether or not it is determined with a specific image as being verified or not and we compare that to the actual value so we calculate the loss on that using binary cross entropy then what we do is we calculate the gradients for all of our weights across the network we then use our optimizer to effectively back propagate throughout that network to minimize that loss and this is where our learning rate comes in because we try to minimize that loss effectively to get it as small as possible so once we've gone and done that we then define our training loop our training loop is effectively going through every single batch for every single epoch and applying our training step so training step gets applied per batch a training loop does it over our entire data set over the entire set of ebox then what we're going to do is train our model so we'll actually train our model in this video so you're going to be able to kick things off okay without further ado let's go ahead and set up our loss function and optimizer okay that is our loss function now define again it probably sounds more intimidating than it actually is but we have gone and written a single line of code so i've written binary underscore cross underscore loss equals tf dot losses dot binary cross entropy and then open parentheses close parentheses and certain circumstances you might also pass through from logits equals true but i've got to do a little bit more research as to whether or not we need to apply it here but in this particular case i've tested it without that parameter set it looks to perform okay but again need to do a little bit more research and if anyone has any ideas hit me up in the comments below but for now that is our loss function so we've written binary underscore cross underscore loss equals tf.losses.binary cross entropy and then open parentheses close parentheses the next thing that we need to do is define our optimizer and for this particular model we are going to be using the atom optimizer there's a bunch of them out there there's um stochastic gradient descent uh what is another one that's the one i can think of off the top of my head but there's a whole bunch of others you can search tensorflow optimizers or tensorflow.keras.optimizers and you can see a whole bunch of them so let's go ahead and define our atom loss or atom optimizer okay that is our optimizer now defined so i've gone and written opt equals tf.keras.optimizers.adam and then i've gone and passed you a learning rate of one e negative four which effectively represents 0.001 now again as i was saying there's a ton of optimizers so optimizers for keras let's go and take a look at what we've got over here so again you've got a bunch where are they so these are some really popular ones over here so you've got uh stochastic gradient descent rms prop adam ada delta eta grad ada max n adam and ftrl i've never heard or used ftrl but again there's a whole bunch available here if you want a video on optimizers hit me up in the comments below but for this particular case we're going to stick with this bad boy over here which is adam and we've gone and set our learning rate already cool that is our and our optimizer now defined so that is step 5.1 now done next thing that we're going to do is we are going to establish a checkpoint callbacks which we'll use when we go and define our training loop down here so let's go ahead and define these okay that is let's make sure that works okay that is our checkpoint components now set up so what we've gone and done is we've written three lines there so first up we've gone and defined our checkpoint directory we're then going to define our checkpoint prefix which i'll explain in a sec and then we've actually gone and defined our checkpoint class which we're going to use to checkpoint our stuff out so first things first so the first thing is pretty self-explanatory so we've defined a directory where we're going to save all our checkpoints and this is going to be called training underscore checkpoints so if we haven't defined that already let's go and create that folder now so if we go into the same folder that we're currently working in you can see that we don't have a folder called training checkpoint so let's just create one i think it might actually create it for you but just to be on the safe side i'm going to create a folder let me zoom in on this so you can see what we're doing i'm creating a folder let's drag this over here folder called training uh checkpoints is that what we call the training checkpoints training underscore checkpoints yep training checkpoints all right cool so i've just gone and created a folder called training checkpoints inside of the same repository or same folder that i'm actually building our model in cool so that is that now done that's this first line here so written checkpoint underscore dr equals dot forward slash training underscore checkpoint so this basically represents us from the current folder we're going to go into our training checkpoints folder and save our stuff down here then i've gone and created a checkpoint prefix in order to do that written checkpoint underscore prefix equals os.path.join and then we've defined it as being inside of our checkpoint directory and we are going to prefix all of our checkpoints with ckpt so this basically means that we're going to have all of our checkpoints in a consistent format they'll start with ckpt and then normally they'll end with a set of unique numbers or effectively it goes in sequence so 0 0 1 0 0 2 so on and so forth and then we've gone and defined what we actually want to save out so i've written checkpoint equals tf.train.checkpoint and then i've gone and saved our optimizer so opt equals opt and then our siamese model so siamese underscore model equals siamese underscore model and so this effectively means we're going to be saving out these two components as our set of checkpoints now we could probably just save out our model in this case i've saved the optimizer as well so the full thing is checkpoint equals tf.train.checkpoint and checkpoint is in caps and then open parentheses and then we'll pass true keyword parameters so opt equals opt so i'm saving this and then siamese underscore model equals siamese underscore model so we're saving this siamese model that we created in the last video okay that is step 5.2 now done now we're on to step 5.3 so this is the really good bit so this is one of the things which or one of the components of coding in neural networks that i thought was super fascinating so defining the actual training step so this is what is going to be used to train on one batch of data so effectively what happens is one batch of data comes through to our training step we go and make a prediction we calculate our loss we go and calculate our gradients and then we apply back propagation through our neural network to ideally get the best possible model now these steps are going to be pretty much the same whether or not you're building a siamese neural network or a network for classification or a what's another good example so i'm building the super resolution model at the moment so again exact same train step style now again you can wrap this inside of a model and in some of my future tutorials which are coming up top secret i actually do that um because it just makes it a little bit cleaner but in this case i've gone and done it this way so you'll see how to do it using the tf dot compile or tf.function decorator all right so let's go ahead and start defining this okay so those are the first what is that three lines of code that we are defining for our train step function so i've gone and and again this function's not actually going to do anything as of yet so first up what we're doing is we are wrapping the function inside of this at tf function decorator so the reason that we do this i'm just going to show you is it actually goes and compiles this specific training function so you can see if i actually show you that there so wrapping it inside of at tf.function compiles a function into a callable tensorflow graph a key component of how all tensorflow works is using a concept of a graph so it effectively compiles an entire neural network into a graph and allows us to actually go and train it in an efficient manner so exactly what we're doing is by using this at tf dot function decorator we are compiling what is going to happen under this train step function so again some really nifty coding's gone behind this so just know that when you're going and creating your train step function outside of a consolidated or compiled model you need to have this decorator here and again i'm going to show you how to do it the other method in some of the future tutorials that i've got coming up all right so what we've written is at tf.function that's our decorator that i've described here the next line is actually defining our function so i've written d f train underscore step and then to that i've gone and passed through or we're going to pass through one batch of data so this function or this uh what is this this positional argument is actually going to be our batch of data and then i've closed that with a colon and i've just written pass because we don't have what we're going to do yet so again this is sort of the beginnings of the function now let's go on ahead and write out what we're actually going to do okay so there i've actually gone and written another three lines of code and i've started applying some comments so first up what i've written is with tf dot gradient tape as tape so this allows us to actually start capturing our gradients from our neural network model so if i type in a tf.gradient tape let me show you the dock of this so this actually allows us to start performing or calculating or performing our differentiation so this is what is going to allow us to actually get our gradients for our neural network so what is actually happening here is it's recording every single thing that's happening inside of our neural network and it's trying to look at these function that we're actually defining to go and perform differentiation differentiation is uh i guess a component of calculus and it's what allows us to go and perform our back prop across our neural network so by defining with tf dog gradient tape as tape we're starting to capture all of the functions or all of the operations that are happening inside of our neural network so we can eventually go and calculate those gradients so you'll see this in action in a sec and then i've written a comment so we're grabbing our anchor and a positive slash negative image so remember when we defined our batch so what do we call that data again i'm gonna have to scroll up because i cannot remember now we have called it train underscore data so if we go and take a look at our data again train uh dot let's call it test batch equals train.data.take1 so if we take a look at testbatch we need to let's just convert it to a numpy iterator okay so that's one example of our batch now uh let's store that batch one i'm just going to store that so what i've done is again you would have seen me do this in the previous tutorials i've written test underscore batch equals train underscore data dot as numpy iterator and then in order to get one batch we can use the dot next method and i've stored that inside a variable called batch underscore one and if we take a look at our batch it is made up of it should be made up of two components two or three okay three perfectly fine so it's made up of three components i thought the first two might be consolidated but that's perfectly fine the first component is going to be our anchor image so if we take a look at zero this is the representation of our anchor images and it should be the same shape as how many images we've got in our batch which in this case is 16. so we've got 16 anchor images in the first part of our batch we then would have 16 images as part of our negative or positive image remember which we're going to use for validation so if we take a look at the length there 16 images and then our last thing should be our actual labels you can see we've got all of our labels in this case they're all zeros so that is perfectly fine okay so in this particular case we're looking good so we've got some of our data so what we're doing here is we're effectively extrapolating or extracting our batch so we're setting x equal to batch one and then two oh so we're slicing the first two values so remember if we take a look at our uh what is it len we've got two values there and if we take can we run dot shape no because it's not a numpy array an np dot array you can see we have two components so remember our anchor and either our positive or negative images each batch is made up of 16 images and they are in the shape of a hundred by a hundred by three channels so that is all of our data stored in our x think of these as our features then what we're doing is we're grabbing our y value we're setting that equal to batch two and what have i done there this should be batch underscore one and if we take a look at y these are all of our labels in this particular case i thought it would have been a little bit more shuffled kind of strange that we've got all zeros that's fine we'll leave it for now okay so what we've got there is this is effectively what we're doing here so we're grabbing our features and we are grabbing our labels what we can then do now is actually pass this through to our siamese neural network to actually start getting some predictions so let's go ahead and write out the rest of our train set function okay so those are the next two components of our train step so i've gone and completed our forward pass and we've gone and calculated our loss so what i've gone and written there is y hat equals siamese model so this is actually making a prediction right so we're actually passing some data to our siamese model so y hat equals siamese underscore model which is coming from here and really we should be passing this through to our train step function but that's fine whatever we'll skip that for now um so y hat equals siamese underscore model and then to that we're passing through our x which we're getting from here so our features and we're setting training equals true super important that you set training equals to true because certain layers are only going to activate when this is actually set to true so if you're using batch norm or dropout you need to set training equals true training equals to true in order to activate those layers so we've set it equal to true there when you go and make actual predictions in the real world you don't need to set training equal to true then we've calculated our loss so our loss is equal to binary cross loss which is what we defined over here and we are passing so let's actually take a look at the docker for that um so if i type in tf.losses.binary so what we pass through to this is uh so if we take a look here so you can see there that we first let me zoom in on that so to our binary cross entropy loss which you can see there so tf.keras.losses.binarycrossentropy we first pass through our y true value and then our y predicted value in order to calculate our loss which is effectively what we're doing over here so we're passing through our y true value which is our label set from over there and we are then passing through our y hat value and because these are floats these are interpreted as a probability so i think that's why we can get away with not setting from logic equals to true here but again if anybody knows any better hit me up in the comments below i'd love to hear your thoughts okay so what have we gone and done so i've gone and done our forward pass so this makes our prediction we then go and calculate our loss and we've written loss equals binary underscore cross underscore loss and then we pass through y true so this y here is y true right remember again there's our dot go we pass through y true then why pred in our particular case we're passing through y true and then y pred so y hat is another name for y predicted or the predicted outcome in this particular case perfectly fine calling it y hat you can call it y true if you want okay we are almost there for our training step function and this is super cool guys so this is the building or the true building blocks of actually building a neural network you go and calculate or produce a forward pass you calculate your loss all that's left to do now is actually perform or calculate our gradients and then go and perform our back prop so let's go ahead and do it okay that is our train step function now done now in this particular case i don't think we're actually outputting our loss but that's particular perfectly fine if we wanted to go and output some information we definitely could you could also wrap it up inside of a model in best practice actually probably dictates that you probably should do that i've sort of skipped through it but that's perfectly fine so the last two lines of code that we've gone and written is grad equals tape dot gradient and this is actually starting to use a tape that recorded all of our operations through our neural network to go and calculate our gradient so we're in grad equals tape dot gradient and then we're grabbing our loss and we are going and calculating all of the gradients for all of the different trainable variables inside of our neural network and specifically the siamese model which is our neural network so i've written grad equals tape dot gradient first positional argument is our loss which is this over here and then we're basically saying calculate all of our gradients with respect to this loss for all of our trainable variables so the second positional argument is siamese underscore model dot trainable underscore variables then in order to calculate our updated weights i've written calculate or i've actually written a comment so i've called it calculate updated weights and apply to siamese model then in order to do that i've run opt equals or opt dot apply gradients and then what we're doing is we're zipping all of those gradients and we're actually going and applying them through all of the different trainable variables so then we've written zip grad which is this over here and then we've passed through siamese.model.trainable underscore variables so in a nutshell what these two lines are doing are first up calculating all the gradients for our different weights within our specific model with respect to our loss then our optimizer is actually applying our learning rate and slightly reducing the loss by changing our weights to be a little bit closer to the global optima so effectively what we're doing is imagine we've got this big curve if we know that uh right now our gradient is down here if we apply a learning rate to that gradient ideally what should happen is by applying them in a certain direction we're effectively going to reduce our loss across the board as we go and train our neural network so again if you want a deeper understanding of this i do hit me up i'm more than happy to write a little bit more or give you a little bit more information okay but for now what we've actually got is we've actually got our train step function what we might also do is we might also just write um print loss here so we can see if that actually runs okay not can't remember if we need to do something a little bit more sophisticated here to get our loss but that's perfectly fine we might also need to return it down here return loss we'll see if that works okay so that is our train step function done so we went and wrote a ton of stuff in there but in a nutshell what we do is we record all of our operations here we get all of our data we go and perform a forward pass we calculate our loss we then go and calculate all of our gradients and we update our weights across our neural network and we are going to try to return a loss to see if we get that out so return the one disadvantage to actually training it using this method is that you have to go and define all of your progress bars and all of your loss metrics being output when you actually wrap this up inside of a model it does look a little bit cleaner so that's something that i went and learned after i actually built up this tutorial but that's perfectly fine i'm going to show you how to do that later okay so in this particular case though we've got our train step function now defined now what we need to do is actually go ahead and build our training loop so let's go ahead and kick that off okay so our training loop is going to be called train i know super unique but in this particular case i've just ridden ef train and then to that we're going to pass through our data and the number of epochs that we want to train for and what we're then going to do is we're effectively going to loop through the epochs we're going to loop through uh each batch and then we're going to run our train step here so run train step yeah okay let's go and fill this out now so let's do it okay let me just paste that there for now we'll come back to that okay so that's the first loop now done so we've gone and we're effectively looping through each one of our epochs so i've written four epoch in range one because we're going to start at one and then we're going to increment epochs by one otherwise it's going to start from zero so we've written four epochs in range one and then comma epochs in caps plus one colon and then we're effectively just printing out what epoch we're up to so it'll print epoch and then our current epoch and then all of the epoch so it'll be like one out of 50 or one out of a hundred or two out of a hundred and this will sort of give us a bit of a status update then we're defining our progress bar and again this is where i sort of said that if you wrap this up inside of a model class you don't have to do this explicitly the native api just does it for you but again this gives you a little bit more control so i've written prog bar equals tf.utils.progbar and this is in capitals and then we're going and passing through the length of our training data because effectively we're going to increment every time we go through a specific batch so that is our first loop now done then what we need to do is actually loop through each batch so i've written four idx comma batch in enumerate train data so this is going to give us a counter and the actual batch itself let's go ahead and finish this up okay that is our train step now applied so what we've gone and then applied is our train step function which is this over here and we're applying that to a single batch and then we're updating a progress bar so i've written prog bar dot update and then we're passing through a specific index which we had from up here so effectively every time we go through a batch we're going to update our progress bar and that gives you that cool little keras progress bar that sort of goes across the screen the last thing that we need to do and again this is optional is to go and save our checkpoints so let's go ahead and do that okay that is our checkpoint function now applied so i've gone and written if epoc modulo 10 equals zero so effectively we're going to save it every 10 epochs again you could change this if you wanted to then what we're going to go on ahead and do if this is satisfied is we're going to run checkpoint.save and we're going to set our profile prefix equal to our checkpoint for prefix which is what we defined over here so it's going to start with ckpt and i think that is our train function now done so we've gone and looped through each one of our epochs and defined our progress bar we're then looping through each batch and then applying our training step and then we're going to save each checkpoint every 10 epochs assuming we have that many epochs all right let's go ahead and test this out now so we're now that's 5.4 now done so we can actually go to step 5.5 and actually try to train our model so we're going to define our epochs let's set it to uh we should set it to more than 10 so we actually see whether or not we're saving so i'm going to set it equal to let's say that's 50 for example and then what we can go ahead and do is in order to start training we can run train pass through our training data and then pass through our epochs actually i just realized we've got an error there so we've defined data as data over here but we've gone and defined this as training data this should actually be data over there and data over there because what we're going to go ahead and do is pass through train data to this function rather than the full data so if i pass through data over here then that's effectively going to be replicated over here and over here so that's fine so all that changes so let me undo that is we had train underscore data as the data that we're going to be looping through but what we actually need to do is set that equal to data because we'll pass through training data as the parameter okay that's fine that is all good so what we now need to do is actually go and train our model so let's test this out so i just want to apply that these print loss functions and return loss functions over here so we'll see if that works and then go from there all right so i'm going to write train and then we're going to pass through train data and then pass through our epochs and fingers crossed this will work we're not actually returning a loss we need early eager execution in order to print that up okay that's fine you can see that we are actually training in this particular case so the reason that you're getting these values rather than the actual loss value is because you need something called eager execution set in order to get that printed out but that's perfectly fine don't worry about it we'll be able to see our model trained relatively quickly because you can see it's training pretty fast but in a nutshell that is our model now training so again if you wanted a little bit more information as to how to print out the loss hit me up in the comments below i'm more than happy to help you out i had a feeling this wasn't going to work because i remember i did it slightly differently in some of my other models but that's perfectly fine it's all good our model is training how good is that no errors looks like everything's happening pretty successfully so on that note that about does wrap it up so let's let this finish and we should be good to go in the next tutorial so on that note that about does wrap it up so what we've gone and done is a ton of stuff in this tutorial it's still training at the moment but we have gone and defined our loss and our optimizer we've gone and established our training checkpoints we went and built our train step function which is a custom train step function and again i think the lesson learned that i took away from this series is that it's probably more efficient to actually wrap this up in the model class because you get all of the progress by stuff able to print out your loss function a whole heap easier you don't need to work with eager execution that's perfectly fine it's still going to work we then went and built a training loop and we also kicked off the training of our model so ideally in the next video we'll actually get to evaluate it save it and then perform a real-time test but on that note that about does wrap it up thanks so much for tuning in guys hopefully you enjoyed this video if you did be sure to give it a big thumbs up hit subscribe and click that bell if you've got any questions comments or queries by all means do hit me up in the comments section below i'll be happy to help you out thanks again tuning in peace what's happening guys welcome to part six in the series on facial recognition in this series what we're doing is we're going from a state-of-the-art research paper and building up a full-blown implementation with the final end product being an app which integrates that model and allows you to actually use it in real time in this part of the series we're going to be focused on evaluating and testing out our model which we just finished training let's take a deeper look as to what we'll be going through so in this part of the series what we're going to be focused on is testing out the model so in the previous video or part of this series what we did is we finished training our model using a tensorflow implementation now we're going to test it out so we'll actually be able to see how well it actually performs so we'll test it out what we'll then do is we'll evaluate our performance so we're going to be using precision and recall to evaluate how well our model is actually performing and then last but not least we're going to save our model down so we can begin the processes of actually building this up for deployment ready to do it get to it alrighty guys so welcome to part six in the series on facial recognition using siamese neural networks so in this part what we're going to focus on is step six and step seven so i'm just going to add in a couple of cells so we can see what the hell we're working on here all right so step six we're going to focus on evaluating the model and in order to do that we're going to be using two key metrics these are precision and recoil so ideally these aim to measure ideally these aim to measure how accurate our model is and how well it's actually performing classification now if you wanted a deeper explanation of these specific metrics i'll link to some stuff below so you can see exactly that so what we're going to do here is we're actually going to run those metrics and calculate them for our siamese neural network on specific batches and then we're actually going to make some predictions because up until now we've really gone and done a whole bunch of training and a whole bunch of pre-processing to actually get to this step but we haven't actually gone and done our evaluation so that's exactly what we're going to do here and we're also going to save our model so we can actually go and use it when it comes to actually deploying some stuff out so first things first what we need to do is import some metrics in order to perform our evaluation now remember i said we're going to be using precision and recall so let's go ahead and import these okay so i've gone and written one line of code there so i've written from tensorflow.keras.metrics import precision comma recall so these are two metrics that you're able to get from the keras library so if i type in uh keras metrics there's a metric whatever so there's a whole let's actually bring up the tensorflow version and sort of flow but my typing is a shocker today all right so there's a whole bunch of different metrics that you can use here so what we've chosen to use is precision which you can see that there so right over here and recall so in order to use these we are effectively going to do what you can see down here so we're going to set up the metric we're then going to update its state and then we're going to calculate the result so we can do that pretty easily using our model now we can do something similar for recall it's pretty much the same really we're just calculating a different metric but it is a good practice to look at a number of different metrics whenever you're going and building out models because this gives you a better estimation of what it's performing well versus what it's performing not so well at okay so we've got our two different metrics that we're going to calculate and whenever you're using precision and recall a higher number is a better number so ideally that's what you want to look for whenever you're evaluating any metrics or kpis whether or not a higher number is better or whether or not a lower number is better whether or not a negative number is better whether or not a positive number is better okay so we've got our metrics what we now need to do is actually go on ahead and make some predictions now before we do that we want to grab the battery data so let's grab a batch of data and we're going to do it from ow where is it keep scrolling nope maybe it's a little bit further down here we're going to do it from our test data which i believe we created yeah so we created our testing partition so we're going to be using this data to actually test out our model so let's go ahead and grab a part of our partition and then we'll actually be able to compute these metrics okay so that is our test data now collected so what i've gone and done is again written one line of code and what i've written is first up let's focus on this so what i've written there is test underscore data dot as numpy iterator and then i've passed through a set of parentheses dot next so this is going to convert our tensorflow data set into a numpy equivalent so if i run test underscore data dot as numpy iterator what you're actually going to get back is this numpy iterator so basically it's looping through and dynamically producing one batch of data back now if i type in dot next it's going to get the next batch of data so you can see how tensorflow actually uses this and continuously goes to the next batch next batch next batch next batch so on now what we can do is we can use this to grab a batch of data so what we're actually going to get back is three things so if i store this in a variable so i'm just going to call it a test var super descriptive variable name so if i run testvar and take a look at its shape let's put in a numpy array first uh what have we done that np array could not broadcast shape into what have we done oh because it's going to be three different uh let's just have a look at the length so what we've actually got is we've got three different variables inside of this test var variable here now the first variable is going to be our test input so think of this as eventually what's going to be coming directly through our webcam so if i grab the first value this should be 16 different components or 16 different samples so you can see we've got 16 test inputs now if we take a look at the second value that's going to be our validation data so remember we're going to either be passing through a positive or a negative sample to through our siamese neural network in real time so ideally the second one is going to be whether or not it's a negative example or of a positive example so again we've got 16 of those and for our third value and let's take a look at the shape of that or what it actually looks like so again we've got images here and then the last thing that we're going to have is the labels so this is whether or not the values match or not so basically it's the classification label now we can delete this so what we've actually gone and done as part of writing this line is we've actually gone and unpacked it in real time so we're grabbing the test input the test vowel and y true which is effectively our labels so the full lining code is test underscore input comma test underscore val comma y underscore true equals test underscore data dot as part numpy iterator and then parentheses dot next and then parentheses again and that's what we're getting back so we've actually got these values pre-unpacked so if i take a look at test input those are going to be our input images test val is going to be our validation data and why true is going to be our labels boom cool so now what we need to do in order to calculate our precision and recall is we're going to pass through our test input and let's unscrew that we're going to pass through our test input and our validation data we're going to make predictions and ideally what we'll get back from our model is whether or not it is a verified person or a non-verified person so let's go ahead and make some predictions okay and there we go so we've now got our predictions back so we went and wrote what is that two lines of code there and one of them is just rendering the results but what we've effectively gone and done is we've gone and used this siamese model which we went and ran our training loop on already right up here in steps really in step five so we trained it all up and now because we've gone and trained our model up we can go and use the predict function to actually make a prediction so what i've written is predictions this could just as easily actually let's be consistent so pretty should be equal y hat y underscore hat all right so we've written y underscore hat equals siamese underscore model dot predict and then inside of parentheses we've passed through a set of square brackets and remember when we use our sima's neural network we're going to pass through our two streams we're going to pass through our input data and our validation data so either positive or negative samples in this case what we're doing is we're passing through the values that we got back from our numpy iterator so i've written y hat equals siamese underscore model dot predict and then inside of square brackets to that function we're going to be passing through test underscore input and test underscore valve now remember order is important here as well so the input first and then the validation data and then the next line is just rendering that result so i've written y hat and these are our predictions back so you can see that we've got some positive examples so this is going to be a very small number so 1.0779765 to the power of a negative 5 which means it's going to be a value closer to 0 which means it is a negative sample this is 1 so you can see 1.000 to the power of 0 which means it's going to be a positive example or effectively 1. let's see if we've got any other positive so there's a positive and there's a positive all right so what we can actually do now these look like they're pretty large numbers as well so they're 0.99 effectively what we can do is we can convert these into a classification output so either a zero or one so a binary outcome so in order to do that we basically just need to set some conditions so over a certain threshold we're going to say that that is a one or a positive example so let's go ahead and do that so it's a little bit more interpretable um so what are we doing uh post processing results okay let's do it okay so what i'm doing there is a list comprehension so what we're effectively doing is we're going to be looping through each one of the values inside of our y hat example and we're going to be converting it so the threshold that i've set is 0.5 so let me break this down so i've written 4 prediction in y hat if y hat is greater than 0.5 then return 1 else returns 0. so this is just a list comprehension it's a little bit of a it's very pythonic in the way the interpret data but basically it's just doing a loop and we've got an if statement here so the full line is inside of square brackets i've written one if y hat is greater than 0.5 else 0 for prediction in y hat now the other way that you could write this is for prediction in y hat so you'd have a results array so res equals that array for prediction y hat if y underscore hat is a wait no if for pretty if prediction is greater than 0.5 then uh what are we doing so uh we are going to go res dot append one else res dot append zero so we're effectively converting this series of confidence metrics into a binary outcome so zero one now we don't wanna run it like this because it's not super pythonic so we're going to do it this way and i've gone and done something wrong so for all prediction in y hat if y hat is great uh this should be prediction there we go okay so what we've actually got there is we've got the results of our y hat output over here so you can see that rather than having these values which are a range or a continuous value we've actually got a binary outcome now so 0 1 0 0 1. now what we can do is we can actually compare that to our labels that we extracted from up here so if we go and take a look at that so what was it called y true let's take a look so we've got 0 0 1 1 0 0 0 0 this is actually looking very good so one one one one so that's what six ones one one one one one i think we might have a pretty good model there yeah that's looking pretty good you can see that that actually matches that exactly now rather than doing this manually and visually inspecting it we can actually go and use our metrics from up here to actually test this out so let's actually do that okay so that is our metric now calculated so a couple of errors there i was just typing stuff in or passing through the wrong passing through the inputs in the wrong format so i had this inside of square brackets but that's perfectly fine it's resolved now so let's actually break this down so first up what we're doing is we're creating so we've actually written three lines of code there so we're creating a metric object and this is a metric object for recall so i've written m equals recall and then inside a parentheses or a set of parentheses at the end then what we're doing is we're updating the state so this is effectively like calculating the recall value so i've written m dot update state and we're actually passing through two positional arguments to this function so we're passing through y true comma y hat and this is what actually passes that through to our recall object so then what we can do is access the result using m dot result and then we're converting it to a numpy array so or numpy value so return result of recall result cool so three lines of code and you can see there that our recall is 100 so in this particular case we've got a really good output or a really good model that's actually able to identify us versus other people so again pretty pretty good in this case now if we wanted to we could actually calculate precision as well just to keep you know on that so recall a higher value is going to be better and precision a higher value is also going to be better so again this particular case we've got a hundred percent for recall and if we do precision as well again 100 so in this particular case this is pretty good but just keep in mind we've only done this on a single batch so best practice would be to calculate it over the entire test sample rather than just doing it on one okay now if we wanted to we could actually grab another batch so all we need to do is run this line here again so if i run that run that and then again so just to prove you'll actually see that these values actually change right so we're going to get different predictions boom and if we take a look at our y true values again different values if we go and now calculate precision again 100 we can switch it back to recall so again we've got a pretty good performing model given the fact that we only train for what like 50 epochs and we pass through or is it like 300 lines of data when or 300 examples or samples uh what how much data did we actually pass through it was somewhere up here it should have been take take take take take this is our anchor and i'm positive i'm trying to work out how much data we brought yes all right so we took 300 samples right so with 300 so on just 300 images of data we've been accurately able to identify myself versus other people so again pretty good in terms of facial verification now so that is done now what we can also do is we can also visualize these results i'm telling you that they match but you can't actually see anything at the moment so let's actually go on ahead and plot some stuff out let's also break this out into some sections so this first section here we're going to call it 6.1 so import metrics and then we are going to get a batch of data we'll actually call it make predictions and then down here we're calculating metrics and we can also just copy this down so we've got recall separately and we can have precision separately cool and then the last thing that we want to do is actually visualize our results so let's actually create another section for that what are we up to 6.4 these results okay so what we're going to do now is we are actually going to compare the results for different samples in our batch so let's go ahead and do this okay and that is our data now visualized so can we zoom out a little bit this is a bit big on the screen all right so you can see there that what we've actually got is the positive example so this is going to be our input this is going to be our validation example so by passing through our first instance let's actually take a look at our outcome so you can see that the result for that particular prediction is one which means that this is effectively saying that the people in both of these images actually match which is valid right that is me that is also me hence we've got a positive example now i'll explain this code in a second but let's actually update this and see another example so let's see a negative example so the second prediction is going to be a zero so let ideally what you should see is this first photo is still me but the second photo is going to be someone else or a different person so one and there you go so we've got different people now so that's me but that's definitely not me and the output that we've actually got from our model is the fact that these two people do not match so the third sample should be a matching example so let's go to third which is going to be index two so again you can see me here with my eyes closed we go to our fourth example that is going to be a non-matching example so you can see it's got a zero value over here inside of y true so if we run this non-matching so that is me but that is definitely not me so again you can see that it is pretty accurately working now let's take a look at the code that we write there so we wrote six different lines of code so the first line of code is setting the size of our image or the size of our plot set plot size and for this we're using matplotlib so remember we import a map plot lib right at the top here so we wrote from matplotlib import pipeline as plt we scroll on back down uh where are we yeah so read and plot dot figure and then to that i'll pass through one keyword argument which is fig size equals and then a tuple which has the values 18.8 this controls how big the plot is so if i wanted a smaller plot i could change this so say i said it's 10 comma 8 it's going to make our plot a little bit smaller so we can see it a little bit more accurately so if i set it to 18 it's going to be huge actually 10 is probably a better example because you can see that a bit more a little bit clearer okay so that is the first line which is setting our plot size then because we're actually plotting two images side by side what we're actually doing is we're using a subplot so set first subplot so what i've read in there is plot dot subplot equals one comma two comma one so what we're saying is we're going to have i can't actually remember how this works a plot dot subplot uh to do all right so we're specifying the number of rows the number of columns and the index so the number of rows is going to be one row so you can see that we've only got one row there and we've got two columns so one column and then two columns and then inside of those we're specifying the index so we've got one image and then or the first image and then the second image so i think we could actually tweak this a bit and actually just have two columns let's try that so if i set that equal to one did that nope got errors never mind all right so we've actually got uh we've gone and screwed that up so one this should be there we go cool so we're going and setting our first subplot so we're in plot.subplot and then we're passing through the number of rows the number of columns and the index so that's going to be referencing this image here so if i change this value in our next line which actually determines what to actually plot it would change this value so if i go and change this to two you can see that i'm gone it's me just smiling out of uh out of sheer happiness for doing deep learning so we've gone and changed that second index which is changing this image so that is controlling all of this is controlling this image here this second section is controlling our second subplot right now the second line that we've actually gone and written to render this is plot dot i am show and then we're passing through what we actually want to render so i've passed through test underscore input and then we've passed through that we want index number two which is what we originally got from over here so this is the image that we got from our test data batch so if we wanted to change our image we could do that and remember these two indexes should be the same to be to ideally show that you're looking at the same sample because they're going to be in ordered flow right so the second line is plot.i am show test underscore input and then we're grabbing the third or fourth index which is going to be index key three and then we're effectively doing the same down here for our second plot so i've written plot dot subplot in this time i'm specifying i'm passing through the first row in the second column or in within the two columns and then we're specifying what index number two is which is going to be this image over here so one row two columns and then this is going to be the value in the second column cool then the next line is identical to over this over here but rather than passing through test underscore input or passing through test underscore val which is because we're grabbing our validation data which is coming from over here and then i've written plot.show to actually show it nice and clearly so if i delete this so you can see that we get this little line over here which is just a function from matplotlib if i add plot.show it renders cleanly and there you go those are our images rendered so again if we wanted to go and run this on a different batch what we can go ahead and do is run this line over here because this is going to go and grab the next batch remember so if i go and do this again or run through the pipeline again bang bang bang bang bang calculate our precision metrics which again still looks good but now our images are changing so what are we looking at we're looking at the fourth index which is going to be one two three four which is a non-matching sample which you can see there so these two people don't match if we go and grab the first value these people match they're both me pretty good right so that ideally shows that our siamese neural network is working in this particular case now the last thing that we're going to do and this is really step what is it step six evaluate our model now done so we've gone and done a ton of stuff there so we imported our metrics we made our predictions and remember to make our predictions all you need to do is use siamese model dot predict that's going to allow you to predict or make predictions and then we also calculated some metrics and did some visualization now the last thing that we need to do as is always good practice when building deep learning models is not to add just more roses to actually go ahead and save our model so this is what we're going to do now now there's a couple of nuances here because we have gone and created a custom layer which is our l1 distance layer from i believe this was step four step four section four whatever yeah section four so we went and created this uh l1 distance layer which we need to do a little bit of tweaking to our export to make sure that we save so let's go ahead and do this alrighty so let's go ahead and save it okay so the first thing that we're going on ahead and doing is saving our model weight sovereign siamese underscore model dot save and then to that we're specifying what we want our model weights to be called so in this case i've written siamese model dot h5 which if we go to where we're currently working from so youtube face id you can see it's now saved and so the date is the 14th to the 10th 2021 so you can see that that is our model now save there let me zoom in on that so you can see that a little bit better so you can see that's our model there siamese model dot h5 cool that is the first bit now done now what we actually need to do is do that slight customization to be able to reload this so let's go ahead and do that okay that is our model now reloaded so there was a little bit of customization that i had to do in the second part to be able to load up our custom models but let's actually take a look at the full line of code that we've written and again it is just one line of code i've just split it into separate lines so i believe we can just uh do that to go into separate lines nope never mind cool that's our model now loaded i think we can delete that can't we yeah okay so what we've gone and written is model equals tf.keras.models.load underscore model so this line over here actually allows us to load a model up from our h5 weight so if i go and take a look at that so the documentation says let me zoom in on this so you can see it so this function actually allows us to load a saved a model saved via model.save so that's effectively what we're going and loading up now now what we can also do is i believe we can also load up our weights save model.load load weight so you've also got the ability to do that as well but in this case we're going to do it this way because that's the way i've developed the rest of the tutorial so just follow this way guys please okay so i've written it model equals tf.keras.models.loadunscoremodel then the first positional argument that we're passing through is the name of the file that we've gone and saved as which in this case is going to be siamese model dot h5 which is what you can see there so that's pretty sort of straightforward there's no real issues there the second bit which is very important is actually going and loading our custom objects now because we've got these inside of our notebook already we don't need to do anything else aside from that but when we go and actually build our qivi app this is going to be really really important because we're going to need somewhere to reference to our custom layer but i'm going to show you how to handle that later perfectly fine so the second keyword argument is custom underscore objects equals and then we're passing through this dictionary here so it's squiggly brackets and then we're specifying l1 disk which is our l1 distance layer colon l1 disk which is effectively our l1 distance layer which is what we created from again it was inside of step four and we've done a ton of stuff here which is this over here so this is a custom siamese distance layer all we're doing is we're passing that through as a custom object down here and we are also passing through binary cross entropy which is going to be this key over here has a binary cross entropy then colon tf.losses.binarycrossentropy over there and i believe this is because we probably created a separate key for it now if i remove this this is actually going to cause a whole bunch of errors so what you're going to see if you don't pass through the custom object is you're going to get this error here so it's going to say value error unknown layer l1 disk and this is because that is a custom layer so if we go went and added that back in let's just go and add a one distance layer for now okay so it looks like we didn't need the binary cross entropy bit but in this particular case looks like it's worked so again by passing through that custom object we're able to effectively load up our custom objects now i'm going to leave in that second line because i don't know what that's going to do if i'd remove it but in this particular case that's worked successfully it looks like you might be able to drop this off as well but who knows i need to do some more digging into it if anyone in the comments knows let me know all right cool that is our model now reloaded so if we wanted to go and use this model now we could type in model dot predict and then pass through our what was it test input test val you can see we are now getting predictions so that is effectively showing how we can go and evaluate our model make predictions save it down and then reload it so we can now get to actually deploying and building this up into an app let me just double check that we didn't have anything else there nope and again we can go and take a look at our model summary so again we've got the exact same architecture that we had for our model up there so let me just add some comments so make predictions with a reloaded model and then view our model summary cool so that is about it for this tutorial so we have gone through a ton of stuff so we've gone through insider step six we imported our metrics we made predictions remember we can make predictions by grabbing our batch i'm going to show you in the next couple of episodes how you can make predictions with a non-tensorflow data set so you can see how we'll actually productionize this as well we made a bunch of predictions calculated some metrics have visualized our results we then went and saved down our model so we can begin using it elsewhere but on that note that about wraps it up much for tuning in guys hopefully you enjoyed this video if you did be sure to give it a big thumbs up hit subscribe and tick that bell and let me know how you went with this and if there's any changes or any help that you need thanks again for tuning in peace what's happening guys welcome to part seven in the series on building a siamese neural network where we take a state-of-the-art research paper and go all the way to the end and build a fully fleshed out deep learning model which allows us to perform facial recognition and verification in this part of the series we're going to be focused on our real-time tests so we're actually going to be integrating with our webcam to be able to perform our real-time facial verification let's take a deeper look at what we'll be going through so as i was saying in this video what we're going to be focused on is our real-time test now there's going to be three key things that we need to do in order to get this up and running so first up what we're going to need to do is set up some images that we can use for our verification so ideally we're going to have two separate folders one is going to be for our positive verification images the second is going to be for our negative verification images then we're going to build a verify function which basically loops through all their images and compares it to what we're getting in from our input from our webcam so this is going to be akin to what you might have when you're using face id on your iphone and then we're going to loop it all together and we're actually going to use opencv to perform our real-time facial verification ready to do it let's get to it alrighty guys so it's time to get started on our real-time test so what we're going to be doing here is first up what we're going to do is define a verification function so we're going to call this step 8.1 verification function and the second thing that we're going to do is we are actually going to connect or hook into our webcam using opencv so opencv real time verification coolio all right and let's add a couple of extra cells so we know what we're doing here now rather than just talking through it let's actually jump over to whiteboard nick and see how this verification function is actually going to run right so the first thing that we need to do is go ahead and perform or create our verification function so the way that this is going to work is we're actually going to access our webcam using opencv first up and what we're going to retrieve from our webcam is effectively our input image so from here what we're actually going to do is we're going to use this input image and verify against a number of positive samples now these are really just images that we've already collected as part of our training so we're going to have these inside of a folder and let's say for example we get 50 different images we're going to use each one of these samples and loop through them as we go to perform our verification so our verification is actually performing 50 different predictions so one verification cycle or one full cycle works out to be 50 predictions and this increases our chance that we're going to accurately verify that the person that is in the input image is actually our positive class so what's going to happen is this image from over here this should be red this image from over here is going to be passed through to our neural network as are each one of the samples from our positive class so we're going to have two images and what we're going to do is or what we should effectively get out of this is a 1 if the image is actually verified now it's obviously going to be a range from 0 to 1 because this is the way that our neural network works but what we're actually going to do from here is we're actually going to implement two specific formulas we're going to leverage one function called our verification threshold let me double check what the name of it actually is sorry our detection threshold what color we're going to make this so we're going to have verification threshold and what we're going to do here is we're going to set a threshold above which we count our person as being a positive class now typically this would be 50 so would basically say that from each one of our results from over here if the threshold or if the probability that we get back is 50 then we count that or so let's make that 50 then we count that as a positive sample now we're actually going to make this tunable so we could actually set a higher verification threshold or a lower detect or lower verification threshold now what we're also going to do is we're also going to implement our detection threshold and the way that we're going to calculate this is we're going to count the number of classes from over here that surpass our verification threshold so let's say for example that we get 30 images that pass 50 the way that we're going to calculate this detection threshold is we're effectively going to say we have 30 images which surpass our 50 image threshold from over here so we're gonna have 30 divided by 50 and that is going to then become our detection metric and we're going to be able to also have a separate detection threshold as well to basically be able to say what proportion of positive samples are actually clearing this verification threshold so this basically gives us a secondary detection metric which allows us to determine whether or not we actually have a positive sample on our hand so this gives us a whole bunch of additional levels of control alrighty enough whiteboarding let's actually go and write some code alrighty and we're back so what we're going to do now is we're actually going to start setting up our verification now what we need to do is we first up need to set up that folder of positive samples so we're going to go into the same folder that we're currently working and we're going to create a verification set so i'm going to create a new folder in here and i'm going to call it application data and inside of here i'm going to create a new folder called verification images and we're also going to create another folder called input image so what we're actually going to do let me make this bigger so you can see what i'm doing is it's super small right now so i've created two folders so inside of application data which we just created i want to create a folder called input image which is what we're going to do is we're going to take a capture from our webcam save it into here and then we're going to perform our verification against all of the images that we have inside of here now what we need to do is actually get some images inside of this verification images folder so what we can do is just go to our data folder go to our positive samples and we're just going to grab a bunch of images here so what's that that's already 35 let's grab so that many i'm going to copy that and i'm going to go into our application data verification images paste those here and then let's go back and we need more data so if we go back and go to positive grab some of these these cool all right so we grabbed a bunch of images and what we're going to do is we are going to dump them into this verification images folder how many do we've only got 43 let's go grab another seven so if we go back into data and positive so what do we need another seven so i'm going to grab that that that that that's seven let's just make sure we don't have any overlapping ones so if we go back into application data verification images right so we've got one overlapping perfect so we've got 50 images now what we're going to do as i was saying is we're going to loop through every one of these images and use them to be able to perform our prediction so we're not going to be performing predictions over one sample we're actually going to do it over 50 different examples this is going to give us a better likelihood of actually getting an accurate classification and it ideally bulletproofs your model a little bit more okay so we've got all of those images there so remember we've got we just went and created a folder called application data inside of our root folder we then inside of that we then went and created two new folders called input image and one called verification images and then inside of verification images we just pasted 50 different images from our positive samples that we used as part of our training so we're going to dump those into there okay now what we need to do is actually go on ahead and start building up our verification functions so let's go on ahead and start doing this now as per usual i'm going to take this step by step so you can actually see how it works so let's begin okay so that is the beginnings of our function so i've gone and written d f and then i've called our function verify so this is defining a new function called verify and to that we are going to be passing through four different positional arguments so the first argument is going to be the frame and this is effectively our input image so if you cast your mind back to our whiteboard session remember how we had our webcam and we had our straight line going through with our input image this is what our frame represents then we're going to have our model and this is effectively a siamese neural network that we just went and trained and we're also going to have a detection threshold and a verification threshold now remember a detection threshold is effectively going to be the metric above which a prediction is considered positive so remember in most classification models what you're effectively going to set this 2 is 50 so if the probability that you get back is greater than 50 that is a positive class now what we're going to do is we're actually going to allow ourselves to configure this so if we want a more strict level of prediction we can actually bump this up so this is going to be our detection threshold and our verification threshold is going to be the proportion proportion of uh positive predictions over total positive samples so remember we've got 50 positive samples right in our verification or what do we call it i can't remember now we called it verification images remember we've got 50 images here so this number is going to be so say for example we get 30 positive matches we're effectively going to be calculating 30 over 50 which means our verification threshold will be 30 over 50 which means it'll be 60 so we could set this higher or lower depending on how many classes or how many positive predictions we actually want to have as part of our model cool so what we actually need to go do now is actually make some predictions so let's go ahead and start building this up okay before we go any further let's take a look at what we were at there so we wrote four different lines of code and the first part is we've actually gone and created a results array so we're going to store all of our results inside of a results array to make it a little bit easier to go and format calculations later on then what we're doing is we're looping through every single image inside of our verification images folder so let's actually break this down so i'm going to zoom in on this so remember we've got our folders so we've got application data so we created inside of our root folder we created a folder called application underscore data and inside of that we created two folders so input image and verification images now at the moment we don't have anything inside of input image but that's fine i'm going to break that down in a second so what we've gone and done from there is we're going to be looping through every folder inside of every image inside of a verification images folder so to do that i've written four image in os dot list dr and then we're specifically looping through or listing out all of the images inside of a directory and we've actually gone and passed through this file path so os dot path dot join application underscore data comma verification images so if i actually show you this this is effectively just returning the file path to our verification images folder if i write os dot list here now os dot list or wrap this in os dot list here this is going to list out all of the files that we have inside of that folder which are all of our verification images which you can see there cool and this is remember as part of our whiteboarding session this is part of that full cycle loop we're going to be looping through all of those verification images to make sure that we have a higher chance of getting a proper positive example or an accurate positive example then we're getting our input image so i've written input underscore image equals pre-process and remember this pre-process function comes from way back which we wrote i'm trying to find it now it should be around here somewhere yeah so this is going to be reading in our image from our file path so we pass through our file path to that it's going to load the image using tf.io.d code it's going to resize it to be 100 by 100 because this is what our neural network needs and it's going to scale it so it's going to divide it by 255 as well cool so we're going to use preprocess and to that we're going to pass you the file path for our input image so if i show you that now so this is again using os.path.join so we're going to grab an input image from our webcam and we're going to store it inside of our input image folder so inside of application underscore data inside of input image or inside of the input image folder and we're going to call it input underscoreimage.jpg so what we'll effectively do is we'll loop through from our webcam and we're going to set it up so that if we hit v on our keyboard it's going to take a snapshot from our person at that point in time it's going to run the detection loop and then it's going to output whether or not we're verified or not so when we hit the v we're effectively going to be saving down this image first up as part of our opencv loop and then we'll be able to pick it up using our verify function so the full line there is input underscore image equals preprocess and then parentheses os.path.join and then we're passing through this file path over here so apple os.path.join and then inside of parentheses application underscore data comma input underscore image comma input underscore image.jpg and then we're doing something really similar for our validation image so i've just written validation underscore image equals pre-process and then rather than passing through the file path to our input image we're effectively passing through the file path to our verification image so if i separate um this loop so if i write four image in let's just grab this over here let's grab this line and if we go on print it's probably going to go crazy right so we're effectively looping through every single one of our validation images and printing them out in this particular case but let's just say we wanted the file path to each one of these you can see that we're looping through every single one of these validation images but by wrapping it inside of our preprocess function we're actually going to be loading it in using tensorflow so that's exactly what we've done here and right now we've got our images properly set up to be able to go and perform a prediction so remember as part of our neural network we need two images we need that input image and either a positive or a negative sample so in this case we've got a positive sample so all that's really left to do now is go make a prediction and then calculate whether or not our detection threshold and verification threshold have been met so let's go ahead and do this oops sorry deleted that let's go and do it okay so i've gone and written two additional lines there so first up what i've written is result equals model.predict so this is effectively using our model.predict function which we took a look at in step seven as part of our last video so we're doing pretty much the exact same thing here this time however we're wrapping our data inside of an additional set of parentheses because we've only got a single sample that's exactly what we need to do when we've only got one example otherwise if you had multiple examples it's perfectly fine you don't need to do the wrapping so i've written lists and then inside of parentheses np.expanddim so this just wraps a array inside of another set of arrays and then to that we've passed through an array which the first sample is going to be our input image the second example is going to be our validation image so this and then this and then we've passed through comma access equals one so this is part of the np expandium so that is our first set of parentheses then we've got everything wrapped inside of a list which is there and then that's passed through to our model.predict method then we've stored that inside of a variable called result and we take that result value and we append it to our results array that we had from up here so this effectively means that we're going to have all of our results inside of a big array that we can then use to calculate these metrics so what's left to do is actually go on ahead and return those metrics so let's do that okay so before i go any further that is this is effectively a detection value so let me copy this it's actually separated out so that is the number of detections that we've got so we have the results that we had so from over here so we're basically grabbing all of our results and wrapping it inside of a numpy array so mp.array and then we're passing through our results and we're summing up all of the examples that are surpassing through our detection threshold so if i copy this that is that there so this basically determines how many of our positive predictions are actually surpassing this detection threshold then in order to calculate our verification threshold we basically just need to divide it by the number of positive samples so let's wrap this up okay and that is our verification now done so i've gone and written three additional lines of code there so first up i've written verification equals detection so it's this metric over here divided by the number of images that we've got inside of our verification images folder so it's effectively detection divided by 50. now if you wanted to you could just replace this with 50 but by doing it this way if you wanted to add a more verification images later on it's still going to work and it's still going to calculate correctly and then we've basically gone and checked whether or not we're verified so i've written verified equals verification is greater than verification threshold so this is going to effectively return a true or false so if this proportion is greater than our verification threshold that we pass through to our function we're basically going to say hey the person that's in our webcam right now is verified and is the correct person so i'm going to grab this as well just throw this over here cool and that is our verify function now effectively done so i think we're actually pretty good there so what we might try to do now is actually run a test so that looks okay so we just went and ran that cell what we can now go ahead and do is actually try to grab an image from our webcam and actually pass it through this and all fingers crossed this should effectively work they'll probably be bugs as per usual but that's fine we'll work through them uh so what i'm going to do is i'm going to delete this cell so we've gone and done a ton of stuff there so let's actually just quickly take a step back so remember what we're doing is we're looping through all of the images within our verification images folder which we just set up so it's going to be this folder here so we're going to be looping through all of those we've grabbed our images using the preprocess function that we defined way back we then make a bunch of predictions and then we calculate all of our different thresholds so we first calculate the number of detections that pass our detection threshold so normally that's 50 but we can change it and then we calculate our verification threshold so this is the number of positive or positive classes divided by the total number of positive samples that we've got within our verification images folder from down here cool all right let's go and do it so the next thing that we need to do is actually make some real time detection so it's going to be a pretty standard opencv loop if you've seen some of my computer vision stuff before this is going to be pretty familiar to you so let's go ahead and do this okay so that is our pretty stand i'm just gonna drop this down so you can see a little bit better so that is our standard opencv loop now done now we're gonna need to tweak this a bunch but that's perfectly fine so let's go on ahead and run this first up and see if this works and then i'll walk you through the code so ideally if you run this and all things holding equal you should get a little pop-up down the bottom okay so that's not good so that means that we need to tweak this number so i can't remember which number it is so is it zero i don't believe so it doesn't look that way nope let's try video capture device 2 so normally when you're accessing your webcam using opencv you might need to play around with the video capture device number which is this number here in this case i tried zero just then that didn't work i tried three that didn't work that gave me an error so let's try uh one i don't believe it's going to be one either nope that's the wrong one and so that that is actually a very good example so this is one of my other video capture devices that i've actually got installed on my computer so it's for the sony uh camera that i've actually got integrated so you can see that that's definitely not the right one so we can quit out of that um two i believe is going to be the webcam that i'm probably currently using so it's probably going to be inaccessible so you can see that one's throwing an error so it's not two is it three probably not i think we tried three already didn't we nope so it's not three so what about four okay so it looks like it's four so i can see that my secondary webcam is just activated over here so in this particular case sometimes you need to play around with your webcam number to see which one is the most appropriate device number that you actually want to use okay but what we actually need to do now is cut down this frame so remember when we actually captured our images we only captured i can't even remember what it was i think it was 250 pixels by 250 pixels let's actually just double check that so if i go to properties and then details yep so it's 250 by 250 so what we want to do is we want to mimic this so if rather than taking in the full 480 by 640 pixels we want to get 250 by 250. this is pretty easy because we've actually written this code way back when we actually captured our images so let's actually close this down and i'm going to show you where to get this code so it's right over here so this is what we did under step 2.2 collect positive and anchor classes so we effectively wrote this code which was going to allow us to cut our frame down to 250 by 250 pixels so we're going to copy that and we're going to paste it just under here so let's actually take a look at our full code before we go and run that again so first up what we're doing is we're grabbing our video capture device so written cap equals cv2 dot video capture and then we're passing through video capture device four again you might need to play around with this might change depending on which machine you're using and how many webcams you've got installed then i've written while cap dot is open so this effectively double checks that our webcam is currently opened and active then we're reading our frame so i've written ret comma frame equals cap.read so this effectively captures the frame from our webcam at that particular point in time and gives us this frame value that we can actually work with we don't typically use the ret variable back and then let's pause on this for a second we won't take a look at that just yet and then i've written cv2.iamshow and then we're naming what we want our frame to be named so when we actually run this cell you'll see that it says verification in the uh top bar and then the second positional argument that we're passing through is a frame now also as per usual guys all of this code is going to be available on github throughout this series i've actually been committing them per video so you've actually got one per series or episode in the series okay so on that note then everything else from down here is all to do with quitting gracefully so i've written if cv2.weightkey10 and 0xf equals equals old q so this basically checks whether or not you're hitting q on your keyboard as opencv takes a break then it's going to break out of the loop and it's going to release our capture so written cap dot release and then cv2 dot destroy or window so this just basically shuts everything down this line over here actually allows us to capture only 250 pixels from our frame so it's effectively just performing some index slicing so i've written 120 pixels or start from 120 pixels and then go to 120 pixels plus 250 pixels start from 200 pixels and then go from 200 to 250 and then grab all of our channels so this effectively just cuts down the frame that we get down to 250 pixels so if i run this now we should only get a smaller sample or a smaller frame which is only going to capture our head cool so you can see my head in there so that's looking a lot better for when we actually go and perform a verification okay so what's next so we actually need to go and apply a verification now all right so but you can see that's looking good we'll actually be able to perform that verification there okay let's do this so what we now want to do is we now need to implement some sort of verification loop or verification trigger verification trigger so we've written this huge verify function here but we haven't actually used it yet now what we're going to do is when or what we want to do is when our user hits v on their keyboard we want to trigger that verification function from over here so this is akin to like when you double tap um the right hand side button on your iphone it actually goes into or it actually starts performing um face id to actually go and maybe verify your credit card or something this is what we're going to do here so we can actually use this same line over here to actually trigger a verification so rather than using uh the ord q we're going to change this to ord v so basically when we hit v on our keyboard then we're actually going to verify or kick off our verification loop the first part of our verification loop that we need to all verification step that we need here is to save our input image so save input image to input image folder and remember it's inside of application data yeah forward slash input image folder okay so let's go ahead and do this first okay so that's going to go on ahead and write out our image to our input image folder so i've written cv2 dot i'm right so this is just some standard opencv functionality to write out images so cv2 dot i'm right and then we've written os.path.join and then to that we're basically passing through where we want to save that input image so this is the full path so folders and file name so here we've written application underscore data comma input underscore image comma input underscore image dot jpg and then the second argument that we passed through to the cv2 dot i am right function from over here is the frame which is going to be this over here which you can see there cool so now that that's done i think the next thing that we need to do is actually go and perform our verification so we can actually trigger our verification functioning as a verification function actually let's go and test this out first see if we can actually save through our image so if i run this now it should be okay so if i run that let's take a look inside about input image folder all right so that's our face popping up so if we go and hit v on our keyboard we should see an image pop up there okay cool so that's saved so you can see we've got our input image that's my face there try another one that's my face uh let's just change it so we got larger i come so yep cool so you can see that if i keep hitting v that we are effectively changing the image that we're capturing pretty cool right now now that that's actually saved there what we need to do bump the seat up and if we hit q a couple of times that should close that down what we need to do now is actually trigger verification so we're going to use this big bad boy over here so let's do that so we are going to get results and verified back i just realized that we don't actually use our frame here because we go and pick it up from down here i think i might have tweaked this when i was that's fine we can remove that later um but that's so we're going to save our pass through our model and then we're going to set our detection threshold and our verification threshold i'm going to set these both to uh i don't know 50 and let's see how that performs okay and then let's print out our result okay so what i've gone and written there is results comma verified so this is just unpacking our results that we get from here equals verify which is this function and then to that we're passing through a frame which i think we can actually drop to be honest because we don't actually use it in this frame nope that's not gonna work uh let's just take a look yeah we don't actually need frame here my bad we can actually remove that so our full function now is just verify and then we're passing through our model which is going to be our siamese model so if we take look model dot summary that's our siamese network and then we're passing through our detection threshold and our verification threshold so those are our two last components and then we're effectively going to print whether or not we're verified or not so remember we're going to get true or false from down here so we might need to interpret our results a little bit but that's fine let's go and run this and see if it actually works fingers crossed guys it's been a long time coming getting to this point okay so that's ourselves over there now i'm not super confident that that's going to be verified because we've got a microphone in the way and we've got the green screen on the background so let's try this anyway so that's actually verified ourselves so if i move this it looks like it's verified again verified again what about if we get an image of somebody else on our phone um here's a good example lebron all right let's see if that works this would be great this is a great test of our model here come on focus doesn't seem to want to focus all right let's just try that anyway okay so that has not verified so you can see that we've actually got a false there so if we go uh let's try somebody else um eddie murphy i don't know if i look like eddie murphy but it's worth a test i want to get my fingers out of the way that's verified as true okay so that is a bad example try it again true all right so this is where we might need to do a little tweaking so i wanted to get that cool all right so we're going to quit out of that and let's actually take a look at our results so remember we're going to get our results from down here so you can see that we are getting a lot of 1. so if we go and calculate my mic is just dropping um so if we take a look at our results greater than 0.5 is this a numpy dot array okay so uh we've basically got all of our samples there so if i type in mp.sum so it looks like we've got 38 samples that are passing in this particular case so if i type in 38 divided by 50 that would mean that we need to bump up our either a detection threshold or a verification threshold so say we made this a lot stricter so it looks like at 80 we're still getting eddie passing through so if we had 34 here that looks like it's a 60 threshold so what if we did this to 90. still passing okay so if that would be 32 okay so what we could try doing is let's set this to i don't know 90 and then we're going to move this to 70 so this is effectively changing our detection threshold to be 90 and our verification threshold to be 70 so if we go and run this again let's see if we can block eddie this time hopefully not block ourselves as well all right there we go so we've gone and blocked eddie now if we go and try myself you can just cross you know and verify myself okay so you can see that in this particular case we've blocked eddie and we've managed to verify against myself whoo guys we've got it working i'm so happy alright this is awesome again you can do a lot more tuning so if you wanted to boost um accuracy you can play around with your different thresholds you could even um add more verification images so this will give you a greater scope to fine tune how accurate or not accurate your model is let's try taking the green screen down as well see what that does i'm just gonna put myself there that's still verified as true that's verified as false so again so false what is that false negative true okay so we might have had one little bug how good is that guys so that is our facial verification now working oh my god i'm so happy it's taking a while to get to this point but we have effectively done it and we've gone and tested it out on a bunch of other examples as well so again we can smash q to close that down that is our facial verification and the facial recognition now done my mic keeps falling down that's fine alrighty guys so we've gone and done a ton of stuff in this series but that sort of shows you what's possible right so we've gone and trained and fine-tuned a neural network completely from scratch we've built a really sophisticated architecture and we've actually gone and tested it out to get some accurate predictions now we went and did a ton of stuff in this video but specifically what we went and did is we went and built this huge verification function over here which does a full cycle loop through our verification images folder and then we went and fine-tuned and built up a real-time verification look there is one more video in this series and we're going to take all of what we've learned and what we've built and we're going to integrate it into a qivi app where we've effectively got a button where we can hit verify and it's going to perform face id on ourselves inside of a kivy app but on that note that about does wrap it up thanks so much for tuning in guys hopefully you enjoyed this video if you did be sure to give it a big thumbs up hit subscribe and click that bell let me know how you went with this i am trying to accelerate this series a little bit faster so we can ideally get it all out and in the next video what we're going to be focused on is building out kivia i'll see you then until next time now let's do the almighty robin williams test so this was the one that i was uh a little bit worried about so if we go and test him drop the screen down so we don't have so much glare so you can see he is unverified the model is working let's try me again verified verified verified alrighty guys welcome to part eight in the facial recognition and facial verification series where we try to take a state-of-the-art research paper and go all the way to the end and finally integrate it into an app and it just so happens that this is the final episode in the series and we're going to be doing exactly that so we're going to be taking our train model and integrating it into a qv app so that we can actually use it practically in real time let's take a deeper look as to what we'll be going through so as i was saying we are in the final episode in the series and this is all going to be to do with actually building up our qb app so you actually get some practical use out of the hardcore deep learning model that we've actually gone and trained now we're going to be focused on three things as usual following the rule of threes so what we're first up going to do is we're going to set up qv so qivi is an open source framework that allows you to build applications with python what we'll then do is we'll build a lightweight facial verification app so think of like face id for your phone but we're going to be building it using qivi and then we're going to integrate our tensorflow model so we'll actually build up our verification process to be able to leverage this model inside of kiwi ready to do it let's get to it alrighty guys so we finally got to the good bit and this is where all our hard work finally comes together and where we're really going to be focused on integrating our tensorflow deep learning model in to our qv app now if you haven't heard of qv before basically it's an open source framework that allows you to build applications and i believe you can even build some native stuff all using python which works to our advantage because pretty much everything we've done so far has been in python so in keeping with that theme we're going to be using qivi now specifically in order to go through this we've actually got a number of tasks that we need to go through but before we do that let's actually jump over into our whiteboard session and actually take a look at what we'll be building all right so in this part of the series we are going to be focused on building our verification app so in order to do this we're going to be focused on building our app with qivi now we're really going to have three main components that we need to handle so say for example we've got our app the first thing that we need to be able to do is grab our webcam feed so over here we're going to have our webcam feed and in here we're going to have the person that we're trying to verify from that we're going to have a button which basically says verify and then down the bottom what we're also going to have is we're going to have a label which says verified or unverified so let's say this person is not me then right down here we're going to say unverified now in order to do this we are actually going to be building up this app with a library called qivi and the cool thing about qivi is like it's an open source library for python that allows you to build a whole bunch of different types of apps so if you want to build uh like stuff that integrates with your machine learning models this is actually great now we've actually gone and done a ton of additional work specifically so that we can hook in this bit over here to our webcam so we're actually going to have a webcam that integrates into our qb app to actually give us this video feed and that is the core premise of this app so when we actually go and hit that verify button we're actually going to trigger our tensorflow model so this is going to grab the webcam feed and it's going to go and verify against our siamese neural network so imagine i don't know how you draw it let's actually draw a little basic neural network and from that we're going to get either a one or a zero a verified or unverified and that is going to drive this outcome over here and that in a nutshell is what we're going to be building but we're going to be taking this step by step so don't fret we're going to be building it up together all right back to the code and we're back so in order to actually go and do all of that what we're going to need to do is go through all of these 14 different steps now fret not a lot of the stuff that we've already done already is going to be readily transferable to when we actually build this stuff up so even though we've got 14 steps some of them are going to be a lot easier than others now first things first what we need to do is go on ahead and set up our app folder then we're going to install qivi and then bring over some existing assets we're then going to focus on building our layout and our actual app so it's going to be pretty simple but again you could extend this out if you really wanted to and then we're actually going to bring over our verification components and tweak them a little bit so that we can get it to work with our app but first things first let's actually go ahead and set up our app folder so that we can actually get stuff a little bit more structured so i'm going to go into the same root directory that we've been pretty much doing all of our stuff in and from within there i'm actually going to create a new folder and i'm going to call it app and to begin with let's actually just grab our to-do list and let's actually dump it in there so we've got all of our stuff in the same place and we can see it all together and i'm going to open this folder inside of vs code cool so really it's all we've got is our to-do list inside of there at the moment really nothing too crazy over here now what we actually need to do is we actually need to so we can actually mark this bit off so that's done the next thing that we need to do is actually go on ahead and install qivi now this is pretty straightforward it's just a pip install so what we need to do is open up our terminal and before i'm actually going to activate a virtual environment because i'm using one here but if you're not don't fret just install it into your base python environment so let me activate this environment and then we should be good to go all right so you can see my virtual environment is called face id but that's perfectly fine if you don't have a virtual environment don't fret you don't need to go and do that now what we can do is i'm just going to jump back into our app folder let's go on ahead and install qivi okay so before i run that let's actually take a look at what we've written so i've written pip install qivi and then inside of square brackets full so this is going to install all of qivi including its dependencies and then we've also installed or we've passed through the second argument which is kiwi underscore example so this is going to install the qivi examples components not mandatory for this tutorial but i think it's i actually had it as part of testing so i've just kept it in there to make sure that we don't break anything but i believe you could actually drop this and it should still run fine okay so once you've written that go on ahead and run it and this should go on ahead and install qivi and you can see that it has in fact gone and done all of that so it looks like we don't have any errors there i've got this warning saying i should upgrade pip but that's fine we don't need to go on ahead and do that let's go ahead and clear this and so to clear in a terminal it's just cls um i can't remember what it is on mac uh maybe it's clear i can't remember i don't do it too much anymore but what we now want to do is just double check that we've got q there so if i run pip list and if we go and take a look you can see that we have in fact called kibby there so we've got all of our kivy dependencies that means we are kind of good to go and we can actually mark step two off as done all right that is that done now and next step is to create a validation folder now what i mean by this is remember in the previous video what we did is we set up some validation images and we when we actually went to perform our verification we didn't just do one prediction we actually looped through all of the images inside of our verification images folder to actually go and get an accurate prediction right that is exactly what we're going to do here so what we need to do is grab those images and bring it into our app folder and that's exactly what we're going to go ahead and do so if we go back into our root folder so remember face id is my root folder so i'm just going to grab this application data folder and i'm actually going to copy it and i'm going to paste it into our app folder and inside of that we've actually got two things so we've got one folder called input images so this is the image that our webcam captures to verify against our two channels and we've also got a bunch of other images which are our verification images so these are the ones that it goes and compares against so that's pretty much that step done so that one was pretty easy so we just went and grabbed this application folder here let me zoom in so you can see it so we grabbed that and we pasted it into there so application data is now going into our app folder okay so that is step three now done so we've just brought that over and this is really all just a bunch of setup so nothing too crazy here now the next thing that we need to do is create a custom layer module so this is actually relatively straightforward but let's actually take a step by step so i'm just going to close this terminal now and what we're going to do is we're going to create a new file and this is going to be called layers.pi and for this what we're actually going to need to do is we're actually going to need to bring in a custom l1 distance layer so this is actually going to hold custom l1 distance layer module right so what we actually need to do here is we need to bring in our dependencies and then we're actually going to jump back into our jupyter notebook and bring through our custom distance layer and as per usual guys all of the code that i write in this is going to be available in the description below so if you get stuck or you're not sure where to put stuff it's all going to be available there so you can actually pick it up so first up what i'm actually going to do is i'm going to set my python interpreter so if you've got a base interpreter that's perfectly fine again as i'm using a virtual environment i can actually go into the virtual environment so if you haven't seen how to build a virtual environment before i've actually got an example inside of the five hour tensorflow object detection tutorial so i've just created a virtual environment called face id in that tutorial i think i call it tfod when you're inside a vs code you can go and set that kernel inside of scripts and then i'm just going to choose python cool so that basically means that when i'm coding inside of vs code i'm using the same kernel that i'm using to run stuff that i've actually used to develop pretty much the whole app cool so what we now need to do inside of our layers file is we actually need to first up bring in some dependencies and then we're going to go on ahead and copy over a custom l1 distance layer so let's go ahead and do that okay those are our two main dependencies now imported so i've written two lines of code there so first up import tensorflow as tf so this is going to bring in our base tensorflow library so we can use it across the board and then i've also imported the layer class and this is really important when it comes to actually using or building up a custom layer so for that i've written from tensorflow.keras.layers import layer and layer is in caps just pay attention to that now the next thing that we need to do is bring in the custom l1 distance layer from jupiter so let's go and do that so i'm just going to jump in so i've already got the notebook already set up and it looks like i've gone and killed it off and that's perfectly fine as long as we got it open so i just need to copy over this l1 distance class so i'm going to copy that and this was done in i think episode 4 and it looks like under step 4.2 so we're just going to copy this over and the reason let me actually explain the reasoning so the reasoning that we need this or the reason that we need this is because when we actually go and load our custom model from a h5 file you can see that we need to pass through our custom objects and we need to pass through our l1 distance layer which is our custom object so that's why we're going to need it inside of our qv app so we're going to go on ahead under step 4.2 we're going to copy everything to do with our l1 distance layer and we're going to paste it in to our layers.pi file that is that done so let's just explain why so we need this why do we need this it's needed for to load the custom model and this is going to apply wherever you use custom objects with tensorflow so whether it be a custom layer custom activation function custom optimizer custom loss function you're going to need to do this right so just the key thing to know all right so that is our l1 distance layer now brought in so we've gone and effectively brought that cool so we can actually mark step four now as done so so far what we've done is we've set up our app folder installed kivy we've created our validation folder so we brought that in it's called application underscore data and we have also created a custom layer module now again we've got another easy task here we just need to copy over our h5 model from our root folder into our app folder so i'm going to go and grab that remember when we went and finished training our model out of it we got this model called siamesemodel.h5 so all we need to do is grab that and paste it into our app folder that's that now done cool so we can minimize that and that is step five now done so this is going to or this is needed so that we can actually reload our model from disk so basically when we actually go and use our model we're going to be able to reload those weights up now we're on to the good bit so this is where we're probably going to write a little bit more code so we actually first up need to build our template kibi app so we're going to bring in some dependencies we're going to build our layout we're going to write our update function and again i'm going to go through this in detail but our update function is going to be used to refresh from our webcam so think of your qiv app as standalone we need it to be continuously updating so we can see our head moving inside of our app and then we're going to bring over our pre-processing function from jupyter again cool but first up what we need to do is import some dependencies and you're probably thinking nick where the hell am i importing these dependencies well we actually need to create a app file so we're going to call it face id because that's really what this is all being to do with we're performing our verification so i'm going to create a new file and i'm going to call it face id dot pi and then from here what we're going to start to do is import our qv dependency so we're going to import qv dependencies first and we're going to need some other dependencies as well so import other dependencies uh pen [Music] and these are really going to be like tensorflow opencv pretty much the stuff that we've used to actually build up our app but first up what we're going to do is we are going to import our kiwi dependencies so let's go on ahead and do that now there's going to be quite a few but don't forget we're going to take a step back and actually take a look at what we've imported so let's do it okay that is all of our qb dependencies now imported so we've gone and written what is that six lines like eight lines of code there now let's actually take a step back and take a look at all the stuff that we've wrote so these first two up here all to do with our app layout so for that i've written from kiwi.app import app so this is going to be our base app class then i've gone and imported our box layer so this is just the layout that our actual kibi app is going to take so if we actually go into our documentation and type in box layout so basically this is sort of what we're going to be getting we're going to get a vertical or horizontal box in our case we're going to be using a what is it i think a vertical box because we're going to have our camera at the top then a button and then whether or not we're verified or not inside of that application so we're going to be using this type of layout uh where are we again so we're up here okay so that is our app class imported in our box layout so from qv.uix.box layout import box layout and this is in camel case cool then the next couple that we've imported i really just think of these um as asset classes so import qv i'm going to call them asset or actually the ux components right really so i've written from kivy.uix in dot image import image so this is going to be used for our real time webcam feed but you'll see that later because there's some magic trickery that we need to get in order to work and then i've gone and imported the button component so from qv.uix.button import button then i've imported a label so it's going to allow us to work with text so from qv.uix.label import label so really three ux components so image button and label then we've gone and brought in some other stuff import other kiwi stuff and so the first thing that we've brought in there is from kivy.clock import clock so this is going to allow us to make continuous updates to our app and this is what's going to allow us to get a real-time feed from our qb app because natively we're not actually running a loop so we need something to pre-process those updates to keep getting a real-time feed from our webcam and this is what we're going to be using the clock for then i've imported texture so from kiwi.graphics.texture import texture and this is something that i had to play around with a ton when i was actually setting this up so we actually need to convert our image from our opencv webcam to a texture and then we set our image equal to that texture so it's a bunch of messing around in order to get this to work but it does work i've tested it so we've imported that so from qv.graphics.texture import texture and then i've imported the logger so this is going to be towards the end that we actually use this but it's going to make your life a bunch easier so you can actually see some metrics so from qv.logger import logger and this is particularly nice if you don't want to show this information to your users but you still want to see how your app is performing and what's actually happening behind the scenes so that is uh pretty much all of our kiwi stuff now imported so all up within one two three four five six seven eight different lines of code to import our qv dependencies so let's save that so that is our dependencies for qb now imported now i didn't include it as a separate set but we actually need to import some other dependencies as well so while we're here let's just go ahead and import those okay so those are our last couple of dependencies now imported so we've gone and written one two three four five different lines of code so first up what we're doing is we're importing opencv because we know undoubtedly we're going to need this when it comes to accessing our webcam we've then gone and imported tensorflow so import tensorflow as 2f we've then gone and imported our custom distance layer so this is coming from our layers.pi file so we're importing that so to do that i've written from layers import l1 dist which is bringing in this class over here then we're importing os so this is going to make it easier to work with our file path and then last but not least we're importing numpy so import numpy as np so that is it when it comes to importing dependencies and yes we've brought in a ton so there's a ton of dependencies so the next thing that we actually need to do is uh we can actually mark that as done that's done the next thing that we want to do is actually start building our layout so let's go on ahead and do this and then we're actually going to start our app and see if we can get it running so let's start building our layout so i'm just going to say build app and layout let's do it okay so that is the shell of our app started so i didn't i'm gonna take this step by step as per usual rather than going balls to the wall and actually building this all up but so what we first need to do is define our app so in order to do that written class cam app inside of caps and then we're passing through our app so this is going to ideally perform a little bit of inheritance so we can use our app and then the first method that we've actually defined is the build method so def build and then to that we're passing through self build is a inherent function that you actually need to define when working with kiwi so if you actually take a look can we see it here let's go into our basics so you can see in pretty much this one here is a good example so this is an example qivi app so you can see that we've gone and written or they've gone and written class test app and then they're passing through app and then they're using the build function so again we're doing we're pretty much following that structure as well cool but that is the beginnings of our app now done so we're in class cam app in caps and then we're passing through the app class that we imported from right up here with a colon and then we've written def build and then we're passing through this ourselves so the self object colon and then pass at the moment we're not actually doing anything here but we are in a second and then we're just setting up our run so this is just going to make it cleaner and ensure that we run successfully so if underscore underscore name underscore underscore equals equals underscore underscore main underscore underscore quote colon then we're going to run our cam app so we're going to run cam or so capital c am capital app dot run so if we actually go and test this out now the way that you actually run this app is just make sure you're in the same folder so what we need to do is run uh python face id or pi now we we don't actually have our app actually doing anything at the moment but this is effectively what's going to run once we've got that running or successfully so again nothing's going to open up because we don't actually have anything running but it doesn't look like we've got any errors just yet so it looks like we're all good well okay so we do have a bit of an error but that's perfectly fine we're going to get to that in a second so what we now need to do is finish building up our build function so let's go on ahead and do a little bit more there okay so those are the beginnings of our layout components now written so we've gone and written three additional lines of code there plus one comment so really this is going to form the base structure of our app we're going to have an image at the top which is really going to be our real-time webcam feed we're then going to have a button which we click to go and verify and then we're going to have a label at the bottom which basically tells us either the verification hasn't started it's completed and you're unverified or it's completed and you are verified so let's actually take a look at these so we've got three different objects we've written self.image1 so this is going to be our main image and we could actually make this image but i'd just mindful that we might have certain classes that use image and then we've actually gone and used our image classroom up here and we've passed through one keyword argument which is size hint and set that equal to the full width but 0.8 worth of the horizontal height so basically this is sort of saying setting the dynamic layouts of each one of these components our image is going to take the majority or 80 of our vertical height then we've set our button so self dot button equals button and remember this is going to be importing from the button class up here and we're going to pass through two keyword arguments so we've gone and set our text equal to verify and what we're going to need to do eventually is set the onpress function to be able to run a certain function but because we haven't defined that function yet we're not going to do that we'll come back to that so we've written text and then we've set that equal to verify so our button is actually going to say verify and then we set our size hint equal to one so the full width but only ten percent of the height so we've got one and then comma dot or point one and so remember with that image we said our size hint to one comma point eight inside of a set of braces which means it's a tuple we're passing through our size hint for our button is going to be 1 comma 0.1 cool so really just think of this as text and then how big or dynamically how big you want that button to be and then we've set our verification so this is going to be our text which actually tells us whether or not we're verified or not so i've set that equal to a variable called verification so we've said self.verification equal to a label which is from right up here so we said remember our three core ux components are image button label we're using right here image button and label so again we've gone and set two keyword arguments for our label class we've set text equal to verification uninitiated this is going to be what it starts off as right so eventually once we go and trigger our verification you'll actually see that it goes verified or unverified but for now we are going to set it to verification uninitiated and we've set our size hint to 1 then comma 0.1 so there's just a couple of additional lines that we need to write before we can actually kick this off and see some stuff so let's go ahead and write that up but first up just i guess to recap there we've gone and created three new components so an image a button and a label and these are going to show up in our app in concert in sequential order so we'll see our image first which will eventually be our webcam feed then a button then a label all right let's go and finish this up and at least test this layout okay so that is our layout now pretty much done so i think this is the shell of our app now effectively set up so i wrote an additional five different lines of code there so first up i've written layout i've created a new variable called layout and i've set that equal to box layout so remember how i was describing right at the start that we had this box layout class which basically either gave us a horizontal layout or a vertical layout so we've gone and set that to vertical so our objects are going to appear in sequential order from the top down so we've gone and written layout equals box layout and we've set orientation equal to vertical and then we've just gone and added our three widgets so keep in mind right now we've created these uh ux components so image button and label but we haven't actually added them to our layout so this is akin to um creating a module in python but not actually importing it into your notebook so we've created these ux objects we haven't actually added them to our kivy layout and that is exactly what we do here using the add widget method so i've written layout.add widget and then we're passing through self.image1 so it's coming from up there so it's going to be our image and then layout.add widget and then we'll pass in through self.button so we're adding our button to our layout and then layout.addwidget and then we're passing through self.verification which is going to be our verification text so i think based on this we could probably actually try starting this up and see what we actually get so let's try running python now let's clear this so it's a little bit clearer so we're going to run python and then face id dot pi let's see if that opens up successfully and we've got um so run is missing one positional argument itself oh we've actually gone and defined this this should actually be the actual class so i've gone and missed out a set of parentheses there so let's actually try that again so before we had no parentheses then i was wondering why it wasn't actually working but we actually need a set of parentheses down here because otherwise we're just referring to the class we're not referring to our particular instance cool or we're not creating a new instance so let's actually go and try that that's looking positive and there you go so that is our base layout so it looks kind of shitty right now but that's perfectly fine we're gonna add our webcam in a second but you can see that we've got our button let me zoom in on this so you can see it a bit better so this is eventually going to be where our image or our webcam feed is going to be but we've got our verify button and we've got our label down the bottom which says verification uninitiated so we can click our button right now it's not doing anything because we haven't actually hooked anything up to it but that is our base layout now done so we've got our verify button and we've got verification uninitiated so if we actually close this it's going to stop everything down now just to prove to you how this sort of widget layout works so if we actually went and subbed these around so say we put the button below verification let's actually see what that does to our layout right so let's try running that again right so you can see that our text is now above our button right that actually kind of looks better than what i had but that that sort of gives you an idea as to how this actually works so the layout is in a sequential order so the way that you add the widgets are going to determine how our layout actually forms let's actually add it back i wonder if hot reload will actually pick this up no i'm going to pick it up that's perfectly fine so we're going to shut it down and if we rerun it again you can see that our order will be back to normal so it'll be image button and then label say image button and then label cool that is the beginnings of our layout and now successfully done so what is that so we've actually gone and done step seven now done pretty cool right so that is our the beginnings of our layout now the next thing that we actually need to do is actually set up a feed from our webcam so at the moment you saw that we had that sort of blank area at the top for our webcam but it's not actually doing anything as of yet well what we need to do is we need to set up opencv to be able to grab our webcam and then we need to refresh that image so that we're not we're effectively looping through and continuously reading the feed from our webcam and that is effectively what we are now going to go on ahead and do okay so first up what i'm going to do is i'm going to create a captcha inside of our build function so this is going to be from opencv so let's go ahead and do that and then we need to go and create an update function that basically we use to update our feed so let's first up go and set up our captcha and this is really no different to how you'd go and set up a video capture device when you're working with opencv so let's do it okay so that is a video capture device now at least initiated so i've written self.capture and i've set that equal to cv2.video capture and i've set video device for now you might need to play around with this depending on which device the webcam is i say it all the time and i'm currently working on an opencv series to show you guys how to actually determine what that number is but on my machine it's going to be four um if you have no other webcam devices apart from your main webcam on your machine then it's probably going to be the video to capture device zero but you might need to play around with this in order to get your webcam feed up so i'm gonna set it to four but i might also if i didn't have any other devices on my machine i'd probably set it to zero um to make sure that i actually get a feed if you get something returned back which sort of says like cvt is empty or cv2.mt or cb2 issue having issues with i am right it's probably because either your webcam is unavailable or you've got your video capture device number set incorrectly okay so now that that is done we actually need to do something to actually update and read our feed from our webcam so we're actually going to need to use our clock function or our clock class to go and set a schedule to continuously run this function but first up let's actually go and define it okay so i've gone and set up a new function called update so this is going to run continuously run continuously to get webcam feed so there i've written def update and then i'm passing through two positional arguments i've written self and then we're going to unpack all of the other arguments that are passed to it so really it's just def update and then we're passing through self and then asterix args then i'm closing out the function and i've passed you a colon and at the moment i've just set that equal to pass so if i actually went and ran this right now you wouldn't actually see anything different so again to run it we just run python and then faceid.pi from that folder so if we're going to take a look at this it's really going to be no different right so right now that we haven't actually done anything so we're not actually one rendering our image to our layout and two we're not actually running this update function as of yet so we need to go and first up write this update function and what it's going to do and then we actually need to trigger it from up here so let's go on and start writing up this update function okay so the next two lines that i've written are basically going to read our frame from opencv so i've written a rhett comma frame and these these are really just sort of standard opencv lines and if you actually take a look inside of our jupyter notebook we would have written this inside of our real time test so exactly the same as what we've got up here nothing crazy uh we actually don't have the last channel so if i do that pretty much identical then okay so that is basically reading in our frame from opencv so i've written red comma frame equals self dot capture dot read so this is grabbing a capture device from up here and it's reading the frame and this is going to return two things a return value and the actual frame as a numpy array then we're setting and we're cutting down our frame so rather than having the full what is it uh 480 by 640 pixels we're actually going to cut it down to be 250 by 250 which is effectively what this slicing is doing here so i've written 120 colon 120 plus 250 so we're effectively going to start never remember whether or not it's height or width first so i think um 120 is going to be the shorter so that's going to be height so we're effectively going to be grabbing 250 pixels from our height and then we're going to grab 250 pixels from our width so we're cutting down the image so we've only got a little square box so 120 colon 120 plus 250 comma 200 colon 200 plus 250 comma and then colon to grab all of our channels cool that is our frame now captured now again we could run this but we're not going to see anything to the screen so what we need to do is actually go and update this image from over here so we can actually see it so that's effectively what we're going to do now so let's wrap this up and then we're actually going to set up and trigger this update function okay so that is effectively going to go and apply our image texture to our image from up here now i'm really not happy with calling this image image one so i'm actually going to call it self.webcam so we're going to change that up there and what we also need to do if we're doing that is we need to change that down here so let's actually so what i actually went and did is i went and rewrote to self instead of self.image1 i've written self.webcam so let's actually change that web and that makes more sense cool all right so rather than that being image one that's now going to be webcam all right then let's actually take a look at the four lines of code that we wrote down here so i've written buff equals cv2 dot flip and then we're passing through our frame and specifying zero so this is going to flip our image horizontally and we're going to convert it to a string then what we're doing is we're actually commencing the conversion of our image to an image texture so this is basically just a format that i realized i had to go and convert it into so that we can get our frame to show up so basically there's a bunch of arguments but it effectively allows you to convert this image into a texture so let's actually take a look at that so i'm going to open up the doco texture and it is this so uh so the texture is a class that handles opal open gl textures depending on the hardware some opal opengl texture depending on the hardware some opengl capabilities might not be available blah blah blah so basically what we're doing is we're actually converting our image to this texture and it's pretty much what you can see down here so this allows us to actually see um this image on our screen and we do use this blit buffer function so this actually converts our or can we first up convert our image into a buffer and then we effectively render it as a texture the full line of code or next couple of lines of code that we've written is image underscore texture equals texture dot create and then we're setting that up we're passing through a two keyword argument so we specified size and then to that we're passing through our frame height and width so frame dot shape one comma frame dot shape zero and we're setting that inside of a tuple and then we're passing through a keyword argument called color format and we're setting that to bgr because that is the way that opencv actually brings in an image and we've got a error there i don't think that's wrong expected expression i think this is actually should be fine if i remove that what do we get okay yeah sorry no we actually did have a second one because i added it in there okay that's fine and then the so then what we're doing is we're converting this opencv buffer which we had over here so remember we set that equal to buff and we're actually applying our opencv image and converting it into a texture so that is effectively what we're doing we're taking our image we're converting it into a texture so that we can render it inside of our app so then we're using image underscore texture and then we're using the blip buffer method which i just showed and we're passing through this opencv string function or string value that we had from over here through this split buffer method so we're grabbing this we're passing it through to blipbuffer then we're setting color format equal to bgr again and buffer format equal to u byte then what we're doing is we're actually grabbing our webcam object which was all the way up here and we're converting or we're setting the texture for that equal to image texture so again it's a long-winded bass process really but what we're doing is we're getting our image we're flipping it horizontally we're converting it into a texture and then we're setting our webcam image object from up here equal to that texture so basically it's a long-winded way to be able to effectively render our image in real time okay now that that's done what we can go ahead and do is i'm a little bit concerned that we've got an error there expected square bracket y and we'll leave it we'll come back to that if we get any error so now that that's done we still need to actually go and run this update function and this is where our clock class is going to come in so let's go ahead and do this okay i think that is pretty much it done for our real-time feed so i've gone and written one line there so i've written clock dot schedule interval so think of this as um triggering off a loop but basically it's going to tell our app to go and run a certain thing every x number of periods so i've written clock dot schedule underscore interval and then we've set that equal to self.update so we're actually running this function on this interval so clock.schedule underscore interval and then inside of that the first argument is self.update so what is it that we actually want to run so it's going to be our update method and then we're specifying how frequently we actually want to run that so if we actually go and take a look at our schedule interval function so schedule interval you can see that this is going to be so schedule an event to be called every x second so it says timeout seconds but it's basically how frequently we actually want to go ahead and run this so we are going to be running it once or 33 times every second so that's one dot zero and then backward slash 33 divided by zero so basically it's going to keep running and ideally mimic what you'd expect to see with the human eye okay that's a lot of work to be able to get a real-time webcam feed so it is a little bit tricky with kivy but basically we've gone and defined this huge update function and we've gone and defined our video capture devices so all things holding equal still a little bit concerned about this why we're getting an error there pretty sure it should be fine but that's fine let's go and test it out so i'm just going to clear this that's a little bit cleaner so we are going to run python face id dot pi we're still getting that error why oh because we're not actually grabbing an object this should be frame there we go all right all right so if we go and so basically i was slicing nothing which is why it's thrown it's saying hey we expected a square brackets but right now we don't actually have anything in there because it's actually telling us convert that into an array but we don't want to convert that into an array we want to slice our array so if i type in frame that is going to fix that up all right let's go and run this now all right fingers crossed let's see if that works oh my god it's working so you can see that we now have our webcam feed so that's my little head in there and you can see that we actually have our real web time or real webcam feed or real-time webcam feed now appearing inside of our qb app so it had i know it was a lot of work to be able to get that but you can see that the speed is pretty quick we've got our face there and this is akin to what you might have when you're using like face id on your iphone so that is very good so we've now got that successfully running so i know we can bask in the glory we've got this up and running we it's pretty much it like honestly guys that was successful no we're not going to stop there so let's go ahead and close this go back to act do all right so we have now successfully gone and done that so we've gone and built up our update function and we've gone and let's actually say and render webcam so a lot of effort to go on ahead and do that but basically what we did in that section is we went and first up set up our capture device to be able to go and get our webcam we then went and wrote this huge update function which is basically it's not huge uh which is basically reading our webcam feed it's slicing down our image so that we've only got 250 by 250 pixels we're then converting our opencv array which is an image into a buffer and then we're applying and then we're converting it to a texture and then we're actually rendering that to this webcam object from over here and then we're triggering it we're using the schedule interval function to be able to go and trigger our update function so it's in real time okay that's a lot now the next thing that we need to do is what so we now need to bring over our pre-processing function so remember we need to pre-process our image before we pass it to our tensorflow model so that is what we're going to go ahead and do and it just so happens that we can just grab this out of our jupyter notebook so the pre-process function is towards the beginning i think what do we do this this was probably during the data engineering bit um pre-process twin should be close pre-process we need this so we're just going to copy this and we are going to paste it into our app and we're going to tab that in oh that is our preprocessor function and now brought in so uh we could clean this up a little bit but i'm not going to bother but basically we've written we've copied in the function that we defined in step 3.2 so that meant it would have been in episode 3 where we basically go and pre-process our image from a file path now the reason that we're going to use this is when we go and perform our loop we're going to pick up our images from a file path we're going to pre-process it then pass it to our model but that is that step now done so again pretty straightforward so we grabbed our preprocess function and dumped it inside so if i just zoom out a bit because we've got a lot in here so by now inside of our cam app class we should have three different methods so we've got the build method which is our base method we've then got our update method which is running every one over 33 seconds to be able to update our webcam and then we've got the pre-process method which is what we're going to use in a second so we can save that let's go back to our to-do so we've now gone and done step nine zoom back in okay then the next thing that we need to do is bring over our verification function so let's go on ahead and grab that as well so i'm just going to say let's actually add a comment here so this is going to be used so our preprocess function is going to be used to load image from file and convert to 100 by 100 pixels okay now the next thing that we're going to do is bring over our verification function to bring over verification function we're almost there guys we're we're getting close now so let's go ahead and grab that so that is towards the bottom after that it's after step six wow we wrote a lot of code for this didn't we uh okay so it's over here so we need this verify function over here so i'm going to copy that and i'm going to paste that in here let's tab this in okay so that is that's that's a terrible comment it's a verification function to verify listen okay so what we now need to do is let's save that looks like we've got a few errors so these are easily solved we'll come back to that in a second so that is our verification function now done so we can set that to done now let's quickly take a look at that verification function because we've already got a few errors and this is because it is asking to pre-process our or use that preprocess function but we don't actually have that defined because we need to type in self. self dot pre-process and self preprocess and we are still having errors and that is because we need to pass through self over here oh and we also need to pass through self over here that's why okay cool sorry that's my bad so the reason that we're getting errors is we're not actually passing through our current class to our verify function so as soon as we add self there we should be good to go okay so there's not much left to do now so what we want to do is we are going to we effectively need to just ensure that we save down an image from our webcam because remember when we had our loop previously we effectively took a snapshot from our webcam we then saved it into this path over here so application data and then input image and then input image.jpg so we need to ensure that we save our current image into that folder as well so what we're going to do is do exactly that and then we should effectively be able to go ahead and build this we also need to load our current model so we've got this we can actually probably remove these yeah let's just make sure we can handle any args and then what we're going to do is we're actually going to set our detection threshold let's set that to like 0.5 as per usual uh what was the other one verification threshold so these are going to be the metrics that we use to determine how tight we want our detections to be so we're going to set those first and specify specify thresholds and then what we need to do is capture input image from our webcam and we're going to specify a save path i'm going to set that equal to this so effectively what we're going to do is we're going to capture real time feed from our webcam we're going to save it into this folder which is going to be application underscore data input underscore image and then input underscoreimage.jpg so if we actually go in actually we don't even need to do that we can go from here so if we go an application data input image and we're going to effectively replace this image so that's what we copied over from our existing what was it from our existing code inside of our drupal notebook so we're effectively going to be replacing that so let's finish writing out the code to actually go on ahead and save our image from our real-time webcam into that file path okay i think that is looking good so we've just gone and written uh pretty much replicated what we had from up here in our update function to be able to save out our image so i've written red comma frame equals self dot capture dot read and then we're effectively reshaping our image to be in the shape of 250 by 250 and then we're going to be writing that out to our save path now keep in mind that we've still got a couple of things that we need to handle right so right down here so we took out the existing arguments from our verify function and now we're getting this error over here which saying model dot predict is having errors and this is because we haven't actually gone and reloaded our model and now that we've taken it out of our function so what we actually need to do to solve that is we're actually going to load our model inside of our build function so i'm just going to set this to let's actually go to our build function and load that in first so under our layout i'm going to say load keras model the tensorflow keras model and we're effectively just going to load our mod so we're going to create a new variable called self.model and we're going to set that equal to how we actually load up our keras model so let's go ahead and do that okay so that is our model now loaded so i've written one line there so i've written self.model equals tf keras.models dot load underscore model so that's actually going to load up our model from a h5 file so we've written load underscore model and then siamese model.h5 and then when passing through our custom object so custom underscore objects and then inside of squiggly brackets or inside of a dictionary we're specifying l1 disk or the l1 disk key oh don't screw that we're setting that equal to l1 dist or the l1 distance layer that we had from up here so that is our model now loaded now if we go back down to our function over here inside of this line here which says result we can change that to self.model.predict cool i think that is looking good so wait hold on we've got one more thing to do so we haven't actually gone and updated our verification label right so if we take a look remember we have this verification label here we actually want to output whether or not we've successfully verified or not let's actually quickly take a look at our to-do list so um update verification function to handle new paths and save current frames so we've done that we just did that there so that is effectively what we did here so when somebody goes and clicks verify this is going to trigger so we're going to go on ahead and save our image from our current frame so we're going to run self.capture.read get the current frame from our webcam we're then going to cut it down and we're going to save it inside of this save file path we should effectively see our input image change every time we hit verify but what we actually also want to do is we want to update this verification text so it goes from verification uninitiated to something else right and that is effectively what we're going to do down here and then we should be kind of done what else do we have to do oh we're apt we're actually up to that step so update verification function to set verified text and then we need to specify link the verification function to the button yeah we've got to do that as well okay cool so let's do the text bit first so let's go ahead and do it okay that is our verification text now done so we are going and effectively updating our verification object hold on wait we are overriding this object over here so this is no good we got to change that so this over here we're going to change this so if i leave it as it is but uh we're actually updating this might work let's actually just play it on the safe side so i'm just going to convert this object here to the verification text and i'm going to set this actually let's say verification label and verification label over here so i don't know if you saw what my error that was there but basically we've got from our existing verification function we've already got a variable called verification so rather than have overlap even though we're getting this one from our object i'd rather keep it nice and clean so i'm just going to change this to verification label and remember our verification function is going to return back this verification value here and this is whether or not our detection is actually passing a certain threshold so what we actually want to do is we don't want to go and overwrite this verification object we want to set our verification label equal to verified if that verification comes back as true otherwise we want to set that label to unverified if it's false right so i don't want to go and overwrite that label hence what we did is right up back in here inside of our build function we've gone and converted that verification object from to be called verification and we've converted that over to verification underscore label so that should be okay and then down here what we're doing is we're setting self dot verification underscore label dot text equal to verified if verification comes back as true add a space there else we're going to set it to unverified we go and save that now i think there's only one last thing to do so that is done two more things and so the last thing that we actually need to do to test it out is to link this verification function to the button so what we need to do is go right back up to our build function and inside of this self.button class here or variable or object in this case what we're going to do is we're going to set on press equal to what is it so self.verify then yep so we're going to be setting it running that so it's going to be self or are we typing this delete that right up here inside of our build function we're going to set self.button equal to self.verify we need to run that class oh that's fine cool alrighty i think that is good to go all right we wrote a lot of code there without running so it'd be interesting to see if this works so let's run it looks like it's successfully loading our model down here okay so no errors yet let's see if this runs i've clicked our button looks like stuff's happening down the bottom okay and that's said unverified okay that is perfectly fine but that is good so what we're actually getting now is it looks like our verification function is actually running so if i go back into the screen a little bit more what about if i did it from here make sure that our new image is being writing of being written out i go into d drive and then uh where are we so youtube face id app and then application data and if we take a look at our input image okay so that is good so our input image is being written out unfortunately it looks like we're unverified but if we take the green screen down so if i go and run that okay so we've got a new input image we can see that that's saving still coming back unverified new image okay so that one said verified take a look verified again my hand up unverified unverified verified oh man it's working how good is that if we try it again a little bit blurry we'll see if that works unverified verified cover our face unverified verified success guys that is your very own face id app that is now successfully running now i wanted to show you guys a little bit let's actually test it out a little bit more so what happens if we put another person's face up against well if mike was away what happens if we actually put somebody else's face up against this so if i went and got jim carrey my guy let's try this so if i go and put the glaze a little bit much we verify against all right jim's unverified um let's try someone else robin williams try that come on focus oh it's verified robin williams that's terrible okay okay so now it's said robin williams is unverified weird that's me verified against the green screen huh i wonder why it wasn't working before still verified and up verified again interesting i wonder if this is going a little screwy all right so see here now we're sort of getting on slightly less reliable results let's try dropping the green screen hand up unverified and away verified end up unverified and away now unverified come on verify okay that's verified all right so we need to play around with this a little bit more but you can see here that sometimes we're getting good results other times we're maybe not getting such great results and this is ideally where you want to have some logging setup so that you can actually see how well or not well your model is actually performing because you might need to tweak those detection metrics right so what i'm going to do is i'm going to close this down and i'm going to show you how to log out some data so let's actually go in to our verify function and we're actually going to use our logger method which is well this is step 13 now done we've successfully verified guys i know it's taking a long time but we got there last thing to do is set up our logo so we can actually see what proportion of detections are over a certain threshold so let's actually go and do this so inside of our verify function we're going to use our logger to log out to metrics okay so those are two lines that i've written for our logs so i've written logger.info and to that we're going to be passing through our results which we get from over here and this is from our existing verification function and then i've also started to calculate a couple of metrics so basically this line here is determining how many of our results are actually surpassing the 50 threshold so i've written logger underscore actually this is not written right yeah this should be like this like that my bad so this is actually going in calculating what proportion of results are actually surpassing our 50 confidence threshold so i've written logger dot info and then inside of that we've got our results which is inside of a numpy array and we're specifying what proportion of those results are greater than 0.5 which is 50 and then we're summing those all together so we'll have the number of results that surpass that threshold so what we can also do is just throw out a couple extra of these and let's uh let's actually make a couple more thresholds so let's say 0.2 0.4 0.5 and 0.8 now if we go and run this again we can actually get some additional info as to how well our face id model is actually working so when you actually run the logger you're actually going to see the output inside of the console so this is much nicer if you're a developer actually building this up rather than having this actually shown to your user you can actually just have it sort of inside of your log or if you wanted to actually go and log it out it's going to look a whole lot nicer than actually showing it back to user so also with the qv app it's resizable right so if we wanted to make it a little bit tighter we definitely could so that looks a little bit better right there okay so let's go ahead and test it out so if i go and drop it down here drop a little girl right so we can see what proportion of our detections are showing up there so 50 how many images do we have in inside of our verification folder oh so this is saying 50 over 50 50 over 50 pretty much all of them are actually being matched kind of crazy given we've got a green background let's see about robin all right so we're here we're getting 42 38 37 which is what wait hold on this is coming back unverified now input image so we're getting robin what was that detection threshold so we had 50 50 so 37 would surpass 50 that should come back i would have thought that should come back with verified but it's good i mean it's coming back with unverified let's see it again focus all right let's see that okay unverified so this is basically saying that 32 out of the 31 it's getting 32 out of 31 matches so if we go and bump this up higher that seems a little bit weird let's say the 80 the verification threshold rather than being 50 let's set it to um i don't know 80 let's try robin again let's restart that up and this is some of the tuning that you'd ultimately have to do guys as you're building one of these models so keep in mind that once you train your model it doesn't end there you've got to have to read normally you want to have to be retraining this all right so that is centered ish [Music] let's see now okay so now we're getting less so again he's unverified what about ourselves okay so we are way more accurate over here right this is not making sense we're getting 47 so what proportion are surpassing 80 so we are getting unverified back so these are our results so we can actually see these down here we are getting pretty good detection metrics but our text down here is still saying unverified so we are getting some ones that are super low so take a look at that 0.09 the majority is still pretty good so i wonder if we've got some issues with our detection threshold so let's try that again i wonder if it's delayed a little bit until i'm verified so i'm verified but if we drop the green screen well i'm verified this is because we've gone and bumped up what our detection threshold but down here we're saying that we're still getting 44 surpassing the 80 threshold which would indicate so what is this actually doing so it's going verification equals detection divided by how many images we've got inside of here i would have thought that this would have come back as true so if we actually go and drop this back down so this is some of the stuff that you might need to play with i mean ultimately it works it's just uh fine tuning how accurate our model actually is let's verify now but now we're getting under do we changes back we are at what is what's our detection threshold a detection threshold right now is at 50 percent and our verification threshold is also 50 but we are saying down here that our the proportion of results that are above 0.2 and we're summing them but this i've got a feeling we've got something incorrect over here so this is saying np dot sum let's log out add detection so logger dot info and detection that detection is going to return back the number of results that surpass our detection threshold from up here so let's go and check this out i mean it works i'm just uh i want to make sure that we get this right right so if we hit verify ideally we should get the number that is being used to calculate the metric so that is saying 47 right so this is 47 does that say verified unverified verified if verif so verification equals detection divided by the length of the number of images 47 divided by 50. so print out verification let's delete this we don't need that and let's print out verified logo.info print out verified and so this is the advantage of having your logger right you can actually output and tune these models so rather than just leaving it there you can actually go and fine tune and actually see where stuff is going right and maybe where it's not going so right so here what i'm actually doing is i'm actually logging out the detection value the verification value and then whether or not we're getting verified back so let's see so if we go and run this now we should get back those three metrics okay so we had 43 detections which surpassed our verification threshold and we are getting 86 percent and we're getting back the value true and it's saying i'm verified all right so we do have a bug there so verification guys here it is so this should actually be verified so rather than this value here being verification this should actually be verified so if self.verification underscore label.text equals verify or basically we're specifying if verified equals true then return verified else return unverified so this was verification first which would actually return a detection value divided by that so right it's not actually returning kind of weird that it will getting verification then i don't know let's go around this and that is the advantage of using the logger you can actually find out where stuff is going right or wrong if we go and verify with a blurry image all right so now we're verified cool and we're saying that we had 50 percent surpassing the threshold we had a 100 verified so we now are true so if we go and put my hand in front of it let's see right so that's still uh surprising i think we need to bump this up significantly so our detection threshold let's set that to i don't know eighty percent let's try that again okay so we're getting verified what happens put my hand up we're still getting verified there right so we might need a little bit of fine tuning that's perfectly fine don't put our face right so that's going up kind of weird though when you put your hand up it's still saying you're verified all right let's try somebody else robin oh mate robin it's verifying against robin williams no that is not acceptable so we got to bump this up but let's say a threshold could actually be 90. so what are we getting that we're getting back very accurate results down here so if we actually set this to 0.99 let's see how that works kind of weird that it's verifying against him appropriately alright so that's us verified and we are getting back all of our positive results that are surpassing it let's try jim i don't know maybe robin williams is able to bypass that threshold and it's saying that jim carrey is verified we're getting 0.99 there is something not going quite right here so that input image is saying that we are verified wait hold on where are we running this from let's bump up our thresholds again all right so that's me verified what about jim okay so jim is unverified so i've gone and set the thresholds really really high now that detection threshold is 0.99 and our verification threshold is 0.8 so if i go and do myself we are verified if i go and do gym [Music] gym is unverified okay so in this particular case we had to set our detection threshold super high to ensure we didn't let jim in but in a nutshell that is our kiwi face id app now done guys so again you might need to do some tuning with your detection thresholds and verification threshold so i've gone and set it super super high so basically our detection threshold is 0.99 and our verification threshold is 0.8 now keep in mind what you could also do to hedge against any of these types of issues is you could have way more images inside of that verification folder so that you get a better chance of busting out anybody that's trying to break in versus un letting people who aren't necessarily verified through to your app but in a nutshell that is our app now done you could also go and train your model a ton more as well so we can say that our logger is now done so we've gone and done a ton of stuff guys so we went and imported all of our kiwi stuff we built our camera app we went and wrote our update function brought in our preprocess function and update our verified function and i sort of showed you how to tune um this detection threshold as well so remember the logger is super important and it allows you to see what your detection results actually are like and in this case you don't want to let jim carrey or robin williams through so you want to ensure that you set these appropriately so the metrics that i ended up with were a detection threshold of 0.99 and a verification threshold of 0.8 so again we could set this even higher if we wanted to we could even add more images into our verification folder but on that note that about does wrap it up hold it right there so while this model and this application worked it was far from perfect you saw that robin williams was able to bypass our verification so after two and a bit hours of recording this tutorial i went off to korean barbecue and really thought about what we could do to improve this model now i'm going to walk you through exactly what i did to achieve significantly superior performance alrighty guys so you saw in v1 of the model we had a little bit of an error right so this image was being verified as being positive this image was also being verified now as a business user you'd probably take a look at this and go hey the value of this particular model isn't that high because it's letting someone that shouldn't be verified actually through to our model now what i ended up doing is taking a look at how we could actually improve this model so i sort of went through my standard process normally i try to add more data apply data augmentation train for longer and see if that improves it if not then we could also do a little bit extra in terms of changing our model architecture maybe doing some additional pre-processing and going from there but in this case i want to take you through what i did to significantly improve the performance of this model so if we jump back into our code and again this is the standard code that we actually wrote as part of our siamese neural network series now i did a couple of key things but namely they revolved around data augmentation so what i actually went and did is i went and augmented our existing positive and anchor data to be able to produce significantly more data so this meant that rather than training on 300 images we trained on 3 000. i also went and improved our performance and logging metrics so you could actually see what our precision recall and act no just about precision and recall would actually look like so let's actually take a step by step and see what change in this code so i'm going to zoom in a little so you can see that a bit better so if i scroll on down the first change comes right about here in section what is that section two so what i went and did is for every single image inside of the positive and anchor class i went and applied data augmentation so i defined this new function here called data org and for every single image inside of our anchor class i went and looped nine times so this would mean that for every anchor and positive image we would now have 10 times as many images including the the one that we already had there so we had nine additional images for every single image we had in there originally and i went and applied what is that five different random data augmentations so i went and applied random brightness random contrast random flip left and right so this would flip the image left or right the brightness is obviously going to improve or increase the brightness or decrease the brightness the random contrast would increase the contrast or decrease the contrast this would flip it left to right random jpeg quality would improve or degrade the quality of the jpeg image that we actually had and random saturation would actually change the saturation so went and did all of those different data augmentations so five lines of code there's this one was commented out i think i actually got rid of the crop because it was a bit too much so that gave us nine different images and down here this is actually a redundant code this is just part of my testing but you can see this is what i actually would go and do for a real model so i went and tested on a single image to see what the impact of this data augmentation would actually be let's actually test that see if it works we need to import os and we need to define anc path which was right up here i didn't prep to run this for you guys but i sort of wanted to show you we haven't imported anything let's test this and data org is not defined so if we run that right so this would actually go through and we need to import uuid right so this would actually go through and create another nine images so if i actually go into my anchor path now i just created new images by going to youtube and uh where are we face id application data anchor so if i take a look at data that we just created so today is the 24th of the 10th so you can see i just went and created all of these images here so by running through our data through our data augmentation so you can see that that one i don't know what's happened there so you can see they've got varying levels of quality right this like you can see the striations i don't know if you can see it through youtube's video you can see that that's getting significantly worse that one's got a flip applied that one's got a flip applied that one's got a flip applied again and you can see significantly degraded performance in that image and again so that's even worse right so this is reducing the quality of our image so that our model at least has a better chance of generalizing and performing well so that was all of the data that i went and ended up producing and you can actually see right down here let me zoom in there's 4 899 different images so before we started off with 300 now we had 4 800 in our anchor folder and in our negative we didn't do anything positive we now had 3 320 so a ton of different images so if you actually scroll you can see that we had this probably our original quality image or not even that's probably our original quality image and you can see that we've gone and applied flips we've gone and applied additional data augmentation to actually produce more data to actually go and train them so that was the key change right and this significantly improved model performance so if you actually go through what i went and did is loop through every single image inside of our anchor path and then i switch this over to our positive path buzz path and swap that out there that out there and that out there so that basically gave us significantly more data to actually go and train on then i went and brought that data into our data set so before this would have previously been 300 over here so we only had 300 images per data set i boosted that up significantly into 3 000 images so we now had significantly more data to actually go on ahead and train and then i went and trained so if we scroll on down i didn't change any of this oh actually no i changed it so i changed the shuffle buffer size because what i actually noticed is that when we had a significantly larger buffer we needed to up this buffer size so that we appropriately buffered or appropriately shuffled our data set so i bumped that up as well to buffer size 10 000 and then the final change was really the training loop so again no changes there no changes there and right down here i actually updated this training loop so first up i imported our precision and recall a little bit earlier on and i actually set it up so remember when we were training we didn't actually have loss or performance metrics which is kind of a no-no because you don't really know how well your model is performing and whether or not it's gone completely off the deep end so this significantly improves the ability to log out our precision and our recall so first up i created a recall and a precision object i then updated it for every single batch so r dot update states would update recall and p dot update state would update our precision and then i logged out those metrics so first up i logged out our loss so i managed to get that working as well i logged out our recall so i dot result.numpy would give us our recall and p.result.numpy would give us that precision and right down here you can see the actual performance matrix so on epoch 1 we had a loss of 0.85728246 we would have a recall of 0.944 and a precision of 0.995 loss dropped to 0.16 on epoch 2 got our precision recall of 0.979 our precision of 0.99 and then i trained for five epochs and i think i actually stopped under epoch 6 because we're pretty good right um so loss was 5.01 to the power of negative zero five and then we had uh a recall of 0.996 and a precision of 0.996 so again still a very high performing model but we're not getting those ridiculous precision or equal metrics of 100 then i also updated the evaluation to actually calculate it on the entire test data set so this gave us a recall of a 100 which again still a little bit sketchy i'm always suspicious of that and a precision of 0.99 so again way better the last and final thing that i did is i saved this out as siamese model v2 so again this is part of the process guys when you're actually going and building a machine learning model remember you need to iterate it doesn't just stop after you've trained it once you need to go through ensure that this model is performing well so under v2 i ended up saving that so you can see that right inside of our root folder i then had a siamese model.h5 and a siamese model v2.h5 and what i did is i took that model and i dumped it into our app folder so we then had a siamese model v2.h5 inside of there and to actually bring this into our application i changed the model that we loaded inside of this line here so rather than loading tf.carous.models.loadmodel siamese model so previously would have been this i just changed it to v2 that allowed us to bring in a second version of our model now this is model ops in a little bit of a sketchy way but you can see that really quickly you can go and change update your model so if you want to train this on yourself really easy right you can just go through that same workflow retrain your model to be able to go and train a verification model on yourself but now the part that you've all been waiting for to actually see what the model performance is like so inside of our app i actually dropped the detection threshold and the verification threshold but you can choose to leave it high if you want um i found that a detection threshold of 50 was a little bit more appropriate and i left the verification threshold at 80 but let's actually go and test this out so again we can start it up using python faceid.pi copy right so that's me so if i take this out so on the green screen let's try it so in this particular case i'm getting unverified so let's drop the green screen we're now verified so you can see with a green screen it looks like it's saying hey that looks a little bit sketchy but we've still got a reasonable number of verifications we had 29 and 0.58 dropping the green screen we're now absolutely 100 verified so down here you can see that it's saying all 50 images surpassed and matched we had a 100 verification threshold you can see that the model is now a lot more sensitive right so before it was passing if i put my hand up in the screen so let's try that you can see that that is now zero verified images right so by throwing up my hand now saying no that there's no way that that is actually appropriately verified um now let's do the almighty robin williams test so this was the one that i was uh a little bit worried about because i was like robin williams no way he should he be passing through and getting through our threshold so if we go and test him drop the screen down so we don't have so much glare so you can see he is unverified the model is working right let's test that again just to prove that i didn't mess around and let me show you in the image folder as well so if we go into app data input images so it definitely is picking up old mate robin so let's try it again come on focus right again unverified and verified and verified and see significantly better performance after going and applying that additional data and retraining um let's try another example so i was testing this out on chris rock again he's unverified and let's test out jim old jim carrey can he get through our verification process nope so absolutely zero images verified in that particular case let's try me again verified verified verified verified and there you go that is our app now updated and performing significantly better than what we had before so this really at least gave you the chance to see what it's like to build a real-life data science model it's not going to end after you train just once you need to go through and performance tune to ideally ensure that you get the best possible model that you can definitely get so remember what i ended up doing is i added way more images after performing that data augmentation and improved our logging metrics so we could actually see our performance as it were training and what i'll end up doing is if you want let me know i can try to share the actual trained model might not necessarily work for you but this will at least give you a chance to see how big that model is i think it's about 150 megs i'll have to work out how to get it too but that'll actually allows us to go and ahead and perform our facial verification so in a nutshell we are now finally done and that definitely concludes the siamese neural network series where we attempt to produce a state-of-the-art model from paper all the way through to code and we finally got it working reasonably well but on that note thanks again for sticking around that about does wrap it up thanks so much for tuning in guys hopefully you enjoyed this video and thanks for sticking along for the entire series i know it's been a little bit of a long journey but we finally got there and we finally tested it out if you do have any questions comments or queries hit me up in the comments below i'm more than happy to help you out but thanks again for tuning in guys peace

Channel: Nicholas Renotte

Views: 8,507

Keywords: face recognition, python kivy tutorial, facial recognition, face recognition with python, deep learning, face recognition python

Id: LKispFFQ5GU

Length: 349min 54sec (20994 seconds)

Published: Mon Oct 25 2021