Tensorflow Object Detection in 5 Hours with Python | Full Course with 3 Projects

hey nick what's with the uh green box around your head uh what green box uh the one that's currently around your head they're on to me just kidding if you want to be able to detect just about anything using tensorflow object detection this is the course for you what's happening guys my name is nicholas renate and in this video we're going to be going through the full blown object detection course so whatever object detection use case that you wanted to tackle you'll be able to do it after going through this course now we're specifically going to be going through four different use cases so you'll be able to repurpose the code that we've actually got to be able to specifically detect the exact objects that you want to use and it's all done using open source tools mainly python and tensorflow object detection now again this could be used in a whole different range of use cases so say for example you want to detect a person's face or if you wanted to do sign language detection or face mask detection this is the exact same code that you can use now the nice thing about this as well is that we'll go through all of the setup all of the installation will show training will also show exporting which is a really big part of being able to build object detection models so we'll be able to export this to the tensorflow.js format as well as tf lite so if you wanted to run it as part of a web app or if you wanted to even use it on a raspberry pi you could and we might just do that in one of our projects towards the end of the tutorial so without further ado ready to do it let's get to it alrighty what's happening guys welcome to the object detection course now this course is intended to take you from being a complete beginner when it comes to object detection all the way to being fully proficient and acting as a practitioner now again we're going to take this step by step so you'll be able to walk through at your own pace and be able to learn and leverage the technologies so let's take a look as to what our actual game plan is for this now there's going to be a bunch of stuff that we're going to go through so ideally this course is going to be fully comprehensive so you're going to need every or you're going to have everything that you need to be able to get up and running so there's eight key parts that we're actually going to go through and each of these key parts are pretty important when it comes to building object detection models so first up what we're going to do is we're going to install and set up everything that we need to be able to perform object detection on our local machine or pc now in this case we're going to be using the tensorflow object detection api and we're going to be doing that mainly using python now the reason that we're using the tensorflow object detection api is that it has a whole bunch of additional capabilities that make it a whole heap easier to be able to build object detection models so if you've ever looked at a real-time object detection architecture or model before there is a bunch of stuff that you actually need to build in order to actually get one of these up and running there's pre-processing steps there's post-processing steps there's visualization utilities now again if you're not familiar with any of these terms don't fret we're actually going to take it step by step and walk through each of those components just know that building this from scratch takes a lot of effort so using tensorflow object detection speeds this up a whole heap so what we're going to do is we're going to install and set up the tensorflow object detection api then what we're going to do is we're going to collect images and label them so in order to do that we're going to be leveraging our webcam so we'll leverage our webcam to be able to collect a whole bunch of images to be able to label our objects now you're probably thinking nick what the hell is labeling well ideally what you want to do when you're building an object detection model is you want to be able to label or identify the objects that you want to try to detect so say for example we were building a face detection model well i might use my webcam to take photos of myself and then draw a label around my face so that what i'll then do is i'll take that trained image or that image which has been labeled and i'll leverage that in my trained object detection model to be able to detect that in the future that brings us to our next step training the model so again we're going to be using the tensorflow object detection api and we're going to be passing through both our images and our labels or our annotations to be able to train this model now there's a whole bunch of different types of models that you can actually leverage there and there's a bunch available through the tensorflow object detection api so we're going to be using a mobile net architecture but again i'll talk about that a little bit more once we get to it so we're going to be training that model initially and we'll be able to leverage that throughout the process now if you're not sure what training is so when it comes to traditional programming you typically have data plus rules so think of your data as the feed that you're effectively getting and then you pass it through some rules or some logic to be able to get answers or an outcome right so say for example you get so somebody sets a color variable for example so the color is red you might then take that variable and go if the color is red then do x if the color is not red then do y so that if statement think of those as those rules and effectively what you're doing in that action is your intended consequence now when we actually train a deep learning model or object detection model in this case we're not actually specifying those rules explicitly up front this is the beauty of machine learning and deep learning so rather than defining those rules and defining what to look for when detecting a face we're actually going to do it the other way around and this is normal when it comes to machine learning we're going to give it data plus answers so our data is our image and our answers are our labels or our annotations and we're going to train our object detection model to learn the rules to be able to detect faces so machine learning sort of flips traditional programming on its head so rather than starting off with data and rules and getting answers we're starting off with data and answers to get rules so it's a slightly different paradigm but it ideally gives you a really cool output so once we've trained our model what we're then going to do is we're then going to start detecting some objects now we're going to do this in two different ways so we're actually going to detect an object using an image so say for example you had an image and you wanted to be able to detect different objects in that image we're going to be able to do that so this is normally known as static detection so you're going to be grabbing a particular object from over there or over there and being able to grab that image pass it through to our model and detect an object but then where i think the real good bit is is being able to detect objects in real time so we'll be able to take our webcam face it toward it towards a particular frame or a particular thing and we'll actually be able to detect objects using a real-time feed now the beauty of this and particularly the beauty of the model that we picked in step 3 which is a mobile net model is that it's going to be super fast so we'll be able to detect objects in real time when we've got a computer or a gpu powered machine and we'll be able to detect those objects in real time so we can do a lot with that because it is fast now normally there's a trade-off between speed and accuracy but again we'll talk about that in performance tuning all right so in detecting objects we're going to detect stuff with an image and we're also going to text stuff using a real-time video feed then what we're going to do is we're going to learn how to freeze and convert our model so whenever you're building models ideally what you want to be able to do is save down that model and maybe give it to somebody else so if you're part of a software engineering team and you're the data scientist what you might want to do is you might want to export your train detection model pass it through to your software development engineers or your machine learning engineers to then be able to go and deploy that model this is where freezing and conversion comes into play now again with tensorflow object detection you're able to freeze down your model think of phrasing as saving it and then if you wanted to you can convert it into different formats in this course we're going to cover how to convert it into a tensorflow.js format and we're also going to cover how to convert it to tf lite so those tend to be the two most popular formats that people leverage when using the tensorflow object detection api we're going to cover them both now i normally get a lot of questions on this so performance tuning so we're now up to step six so we're going to be covering install collecting images and labeling training detecting freezing and conversion step six is a lot of the time where people come undone so being able to performance tune your model is so so important so say for example you use your webcam you go to detect some stuff and it sucks it's really really crappy and it's not actually detecting the objects that you wanted to detect well what are the steps that you actually go through to be able to improve the performance of that model to then be able to accurately and effectively detect those objects so we're going to cover what to do when things go wrong and how to actually perform and stream your models and improve that performance so that you can get back on track all right so that's step six then what we're going to do is we're going to learn how to train on collab in a cloud instance now the beauty of this and everything that i cover in this course is that all of the code and the full walkthrough as well as a guide is going to be available on github so if you go to the github repositories that i'll show you later you'll actually see that there's a written guide in terms of how to do this as well so i've written a really big readme and a step-by-step walkthrough to actually get this up and running as well as a couple of extra bonuses to make this a little bit easier on you now in a step eight this is where it all comes together so we're going to be doing three different projects now and again these are projects that i've really wanted to do for quite some time and we're going to be doing them in real time so the first project is actually going to be one that i've always wanted to do we're actually going to be using this microscope which is a usb microscope to be able to detect defects in particular or small pieces so in this case i've got a couple of leds here that i pulled out of my raspberry pi kit so in this case it's really really small let's see if we can see that you can see that right there so it's a really small led and you can see that this one there's no issues with it right but then this one i've obviously got a bent pin so if i bring that a little bit closer you'll see that this particular pin is bent right we're actually going to train an object detection model to leverage this microscope to be able to detect those bent pins now this has a whole wide range of use cases so say for example you had a pcb so printed circle board and you wanted to be able to detect defects in that say for example you wanted to be able to look at a petri dish and be able to detect different things within a particular microscope slide to do that as well so that'll be project number one project number two is going to be able to detect directions so we're going to train a model to be able to detect our hand pointing up left right and down and then we're going to take that model convert it to tensorflow.js and actually integrate it into a web app so say for example you wanted to build up on that and build a real-time game and pass through detections using your hand or did using object detection you'd be able to do that and then our third and final project is actually going to be raspberry pi object detection so again this one tends to be really really popular i have seen a bit of a performance drop off when you do pass it to raspberry pi but we'll take a look at that so we're going to be ideally trying to be able to detect sentiment from our face using our raspberry pi so again i'm going to show you how to do all that alrighty so there's eight steps we're going to go through it all and again take this step by step so ideally what i'd recommend is go through each part take a break review what you've actually gone and done make sure it's all working before going on to the next step take lots of notes and again if you've got any questions hit me up in the comments below and join the discord server that's probably my best tip as well so i'm going to include a link inside of the description as well as in the pinned comment to be able to join the discord server so again a lot of the people following the tutorials that i put out on youtube are on that discord server as well as yours truly so you'll be able to jump on in there and get some help if you need it but i'll also show you how to sort of debug and walk through this tutorial so that's our game plan those eight steps now without further ado let's kick things off and let's step into step one install and set up alrighty install and setup so the first thing that i'm going to sort of show you is where to get the code for these tutorials so in this case i've got three sets of code one is more important than the others but i'll show you each of those so i've gone and written three different or i've gone and built three different sets of github repositories that you're going to be able to pick up and leverage to get up and running with object detection a whole heap plus them and if you've done any of my previous object detection tutorials let's say for example the face mask detector or the sign language detector what i've actually done is i've actually improved that code base i've actually made this flexible enough to work on a windows machine a linux machine work on colab a whole bunch of different environments so it actually detects what environment you're trying to use and runs the appropriate code sorts out the file paths so on so there's three different sets of code so there's the tensorflow object detection course code there's the tensorflow object detection app so whenever i say tfod that's just tensorflow object detection then there's the tensorflow object detection raspberry pi code so you're gonna get three different sets of code now in order to get to these you just need to go to https colon forward slash forward slash github.com forward slash knick knock knack forward slash tfod course forward slash chair phone d app and forward slash tfod rpi so if we actually take a look at those so this is the course code and again you can see that we've actually got our baseline image so here you can see that we're detecting live long and thumbs up now again i'm actually we're actually going to build this course oh we're actually going to build this object detection model first up so it's going to be a gesture based model and the nice thing about each of these repositories is i've actually written steps um or written steps rather than made a video tutorial on top of this so that if you prefer just walking through a step-by-step guide you can just walk through each one of these to be able to build up these models as well and i've done this for the tensorflow object detection course code which is the main training code i've also done it for the app code so it shows you how to deploy the app and i've also done it for the raspberry pi code so it shows you what you need to install what you need to clone to be able to get this up and running so again all of this code is going to be made available so if you check the links in the description below you'll be able to pick all this up you'll be able to leverage it to your heart's content and be able to build a whole bunch of different object detection models so those are the three sets of code that we're going to be able to leverage so you'll see once we kick off or once we actually start the first step that we're actually going to go through is clone this code and again we'll just be using git cloned to be able to do that now the next thing that we're going to be doing is we're going to be creating a virtual environment so again i'm going to explain a little bit of the code before we actually go ahead and do it but i don't want to spend too much time on slides so think of a virtual environment as a separate room in your house so it's like creating a separate room in your house now the reason that we're actually using a virtual environment is that it isolates all of the different python and all of the different library dependencies and different things that you're actually going to need for your tensorflow object detection model now this makes sure that it doesn't conflict with all of the different libraries and dependencies that you've already got installed inside of your python or jupyter environment what we're going to do is we're going to create a new virtual environment so think of it as creating a new room in your house and we're going to install all of our object detection stuff inside of there so it doesn't mess around with any of the other things and it ensures that you get all of the main dependencies that you need and you don't get any dependency conflicts now in order to do that we're going to be using a library called venv so this allows us to create a new python environment and again we can name it whatever i want or whatever you want then what we're going to do is we're going to activate it i'll talk about a little bit about that later we'll upgrade our pip dependency so this makes sure that we have the latest resolvers and all of the good stuff that we need and then we're going to be installing ipi kernel so ipi kernel is a really important dependency because it allows you to associate your virtual environment with your jupyter notebook so ideally so we're going to be working inside of jupyter notebooks now we won't cover installing that but again if you want to see how to install that from scratch i'll link to the other installation video where we do the anaconda install and the jupyter notebook install but more often than not if you've done any data science or object detection tutorials before you'll have jupyter notebooks installed so this is going to cover all the stuff that you need to go from jupyter notebooks a little bit further and be able to build your object detection models all right creating a virtual environment so think of it as a separate room inside of your house where you're going to install all of your object detection stuff so that it doesn't screw up everything else mind it's also like locking yourself in a room and getting stuff done right keeping everything in one place now there's a bunch of dependencies that you need to install whenever you're working with tensorflow now the first and foremost one is going to be the visual c plus build tools i'll show you where to install that so it's really a just download and install nothing too special there so tensorflow needs visual c plus plus build tools in order to run that's pretty straightforward now again i'll link to the other insole tutorial where you can pick that up so that one's pretty straightforward where most people tend to get hung up is installing these two though cuda and c-u-d-n-n so cuda and kuden give you gpu based acceleration when training your tensorflow object detection model so the reason that you use a gpu is that it is exponentially faster to train an object detection model using a gpu versus just using your raw cpu and your raw memory now if you don't have a gpu don't worry about it you can still go through this tutorial it'll just take a little bit longer to train your model now if you do have a gpu and you've gone through an object detection tutorial before and it wasn't actually activating or it wasn't actually being utilized i'll show you how to check that then what it might have actually been particularly if you're using tensorflow is that you might not have had the right cuda and kuden installations available but i'll show you how to actually see whether or not or how to actually find an appropriate combination now the reason that i'm talking about this is because there's consistently going to be new versions of tensorflow coming out so you might need to update your cuda version or update your cd and then version in order to be responsive to that actual change so i'll show you where to do both of those or how to detect both of those and let's actually take a look at that so in terms of visual studio so what you just need to do is go to visual studio community download visual studio so it's going to download everything that you need and then ideally what you want to do is just run this install so this is going to install these c plus build tools now again if you want a full walkthrough so really it's just a matter of downloading this and installing that so it doesn't need a whole bunch of stuff so i'm going to cancel that because i've already got it installed so you just need to download that install that and that's going to give you your c plus plus build tools but again if you want to see a full walkthrough on how to install that i'll include a link to that as well now the next thing which is more important is cuda and cu dnn so this only applies if you've got a gpu and you want to use gpu accelerated training now i'm not going to do this first up because i'm going to show you how to check your tensorflow versions and then map to the appropriate cuda and cud and then libraries now the reason that i say this is if you actually go to this link so tensorflow.org forward slash install forward slash source also this one down here but again links will be available in the description if you actually scroll all the way to the bottom it actually gives you the appropriate versions of tensorflow and cuda and cud and then to use in this case this is just cpu so again you can run this full object detection tutorial on your computer even if you don't have a gpu it'll work fine just be a little bit slow but more importantly these are the gpu combinations so here if you're using tensorflow 2.4.0 you need to be using python version 3.6 to 3.8 you need to have the gcc compiler 7.3.1 bazel 3.1.0 and more importantly kudo and cu dnn so here we've got so tensorflow 2.4.0 runs with cuda 8 and kuda or cd and then 8 and cuda 11. so you need to make sure this is so important you need to make sure that the tensorflow version matches these two cuda and cu dn versions up here so if they don't match it'll still run it just won't leverage your gpu and you'll probably be sitting there thinking hey i've got a gpu wise in this training faster more often than not it's because of that and i'll actually show you how to check your tensorflow version and how to get the appropriate cuda and cu dn versions once we've gone through the initial steps of our insole so this is the linux uh linux mac os combo if you go to the windows one over the left hand side here so in this case the install or the link is for sash install forward slash source underscore windows again same style of thing and again it talks about the visual c plus plus build tools over here so again it just tells you go to visual studio hit visual studio downloads select redistributable download and so and the reason that i'm sort of going through this is i'm sort of showing you where the information that i'm teaching you is coming from so this gives you an idea as to where to go to get help but again you can always hit me up in the comments so again you've got to install visual c plus plus build tools through the downloads now if you go down further towards the bottom these are the combos for windows so again if you're using tens flow or if you using tensorflow gpu you need to be using tensorflow gpu 2.4.0 and cuda 8 and 11 if you're using tensorflow gpu 2.3.1 for example then it's going to fall into this category so you need to be using python 3.5 to 3.8 um you need to use the microsoft visuals visual c plus plus build tools to 2019 so this is just talking about that compiler that we had right up here so it sounds complicated but really it's just talking about that that visual or this thing that we were just downloading um so that's fine so you can use that i would move that so microsoft visual studio 2019 and you need bazel but again that's normally pre-installed and you need so in this case if you're using 2.3.0 you need cuda kudun 7.6 and 10.1 so again there's really specific combinations that you need to be using if you want to be training on your gpu cool all right so that talks a little bit about that but we'll actually come back to that gpu install a little bit later so i'll actually show you how to check your tensorflow version make sure that you've got the right version in your virtual environment think about it as the other room inside of your house and then find the matching cuda and cu dnn versions okay now all of that might seem pretty complicated so you're probably thinking all right nick what the hell am i going to do if i get errors because undoubtedly you will get errors as part of this it's always being updated there's always changes there's a couple of things so the first thing that i'd recommend is if you're getting errors by all means do mention it in the comments below but if you're getting errors during the installation steps or while you're actually running the notebook i've actually put together an error guide so if you go through to the tensorflow object detection course github repository which is available at github.com forward slash knickknack knack forward tfod course now again this link is going to be available in the description below i've actually put together this error guide and this is based on all the errors all the questions that i saw people were asking me as part of the other object detection tutorials that i've got so you can see here that say for example you get a error which says no module named whatever that module name is well more often than not the solution to that is to go on ahead and install that module so say for example i've got actually got an example here so if there's no module named typeguard well what you want to do is then go and install typeguide using pip install another one that's really really common is the pi coco tools one so in this case let's say for example you get this area here so let's zoom in on that so you can see it so value error numpy and the array size change may indicate binary incompatibility expected 88 from c header got 80 from pi object sometimes these errors aren't very descriptive so you probably think of well that's a numpy error but actually it's a pi coco tools error now again all of these errors and all of this error guide is going to be available inside of the github repository and my goal is to ideally be consistently updating this so if there's new errors that pop up or if there's new things that pop up that need a solution i'm going to be consistently posting it in here so i'll be cloning it updating the error guide so that you've consistently got another place to turn but again the other place that you can turn if you need support is to jump onto the discord server again i'll include a link in the description below there's a bunch of people that are already on there that are awesome they're providing help to anybody that's actually going through these tutorials okay so that is the next thing that i sort of wanted to show you so that's the error guide and where to get help if you get stuck okay so that's a lot of slides now we want to do is or now what we want to do is actually get started with this thing and start building up our object detection model so the first thing that we're going to do is as i was saying we're going to start up by creating our virtual environment now in this case i've actually got the steps to create the virtual environment inside of the tensorflow object detection course code which you can see down here now the first step that we ideally want to do is we want to clone this tensorflow object detection course repository which contains our checkpoints which we can get rid of that i don't know why i included that it includes our git ignore file our image collection notebook training it let's zoom in a training and detection notebook error guide which i just showed as well as this readme which gives you all of the steps that you need to go through so let's go ahead and clone this repository so i'm just going to copy this link and go to a new command prompt so in this case i've got a new command prompt because i'm on windows if you're on a mac or a linux machine you'd just be opening up a terminal and i'm going to do this side by side so you can see what we're doing so i'm going to create a new folder inside of my d drive and inside of my youtube folder and i'm just going to call it let's call it tensorflow object detection cool and then what we're going to do is go into that folder so i'll go into cd youtube cd tensorflow object detection and then what we're going to do is clone that repository as i was saying so we'll type in git clone and then that link so again all of those links are going to be available inside of the description below let's just make this so it's a little bit easier to see what we're doing all right so we've written git clone and then https colon forward forward slash github.com forward slash nick knock knack forward slash tfod course so this is going to give us all of that stuff that you saw inside of github right so it's going to give us the notebooks and a bunch of other stuff and it's actually going to clone create a new folder inside of this tensorflow object detector repository or inside of our tensorflow object detection folder that we've already got inside of our file explorer so if we go ahead and run that that should clone down you can see it's cloning we're receiving objects so again the speed at which this clones is going to be dependent on your internet mine over here sucks so you can see i'm getting 2.39 megabits per second in this case that's now done all right cool so we've now got our repository clone so again everything that you saw over here is now over here locally so what we now want to do is start creating our environment so in order to do that as i was saying we're going to use that v e and v package which we had over here so we're effectively going to go through steps two and three now so we're going to create our virtual environment and then we're going to activate it so think of activating it as actually we've actually stepped out of that think of activating it as actually stepping into that room so i said creating a virtual environment is like creating a new room in your house well activating it is actually like stepping into that room so you're now inside of that environment where you're going to isolate your libraries so we're first up going to create a new environment and we can do that using this command here so python dash m v e and v tfod now the tf od i'm just going to make this a little bit bigger the tfod at the end of that is actually the name of the environment that we're going to be creating so if you wanted to name it something differently you could if you wanted to name it my special environment you could you could name it whatever you want so in this case i'm just going to clear this using cls if you're on a mac it's going to be clear and we're going to do that so python dash m v e and v tfod so what i've actually written here is we're using python we're passing the m flag and then we're going to be using v e and v which is the dependency that we're using and then we're going to be creating a new environment called tfod so a virtual environment think of it as a new room it's going to be called t-f-o-d so if i hit enter what you should see is that we're now creating a new uh we should have done it inside of there let's delete that all right that's my error again it's going to be error stuff breaks this is sort of what i want to show you the realities of actually doing this so we're going to delete this here we're actually going to do it inside of this tfod course folder so we're going to see it into that and we're actually going to repeat that command here so you saw that when i created it initially it created it outside of there so we're going to go into this folder now and we'll create it in here so let's do it again so type will type in python m v e and v tfod so now it should create inhale and the reason that i do this is i just like keeping everything isolated so the main training code i like having it in one section the app code i like having it in a separate section and then the raspberry pi code i like keeping that separate as well again you can play around with this you can have your environment elsewhere if you wanted to i just keep them isolated so that i can see what is what okay so you can now see that we've now got our tensorflow object detection repository created so if i make this a little bit bigger this folder over here is that tensorflow object detection virtual environment so if we open this up there's a bunch of stuff in here um so in this case it's gonna vary depending on whether or not you're doing this on a mac or whether or not you're doing it on a windows machine if you're doing this on collab or on a cloud instance you don't need to create a virtual environment so those are set up so that they all run on their own individual pods so you don't need to go ahead and do this but if you're doing it locally which i suggest you do because that allows you to do real-time detection more on that later go on ahead and create a virtual environment so the main thing that i tend to look for is inside of the for windows if i go into libs site packages you can see that we've got a bunch of different uh libraries available there so we've got peer pip dash 19.03.dis dash info a bunch of different packages right now a really important thing is right so if i type in pip list which normally shows me the different python packages that i've got available you're going to see a ton here right so i've got a whole heap of different libraries available right now now the reason that you're seeing all of these here but only a couple here is because i actually haven't activated my environment yet now this happens so often you need to make sure that you activate your environment in order to see or in order to see your isolated packages that you've actually got installed so what we're going to do now is actually activate our environment now the command that you use is going to differ depending on whether or not you're running on linux or a mac so if it's on a linux what you're going to do is run source tfod or whatever the name of your virtual environment is forward slash bin forward slash activate to be able to activate your environment now if you're on windows i don't know why they made this different but it is different it's dot backward slash tfod again or the name of your environment backwards scripts backwards activate so in this case i'm on a windows machine so we're going to be using the second command here so we are going to effectively be going so if i go back into our main folder we'll go into tfod we'll go into scripts and we're going to be running activate which you can see over there so there's an activate file and an activate batch file so let's go ahead and run that so we're going to backward slash tfod backward slash scripts backwards activate and what watch what happens once i actually run this right so right now we don't actually have we we don't we're not actually inside of our virtual environment if i run this you can now see that we've got this little thing over here so this tells us what virtual environment we're actually in so we're now inside about tensorflow object detection virtual environment so again we've now stepped into our new room now if i type in pip list you'll see that it's way shorter right so this actually represents the actual sets of libraries that we actually had available inside of there so remember all of these packages are here we don't have all of these over here now if you wanted to deactivate the environment later on all you need to do is type in deactivate this is going to step out of it right so you can see that the tfod is now gone in this case we actually want it activated so let's do that tfod scripts activate so we're now back in so you can see that we're now there all right we're going to clear this clean it up the next thing that we need to do is install a couple of key dependencies so the first one is we want to upgrade pip so this is going to make sure that we update using the latest resolvers and that we've got the latest pip package and in order to do that we just need to run this command here so python dash m pip install dash upgrade pip so let's do that we're going to type in python dash m what was it again pip install dash dash upgrade pip install dash dash upgrade pip so this is going to upgrade our pip installer and you can see it's uninstalled 19.0.3 and it's in tall installed 21.0.1 and the thing that you'll notice is that as you're updating these packages you can see them updated inside of our site packages folder as well now this just makes your life a whole heap easier because you can actually see what's being installed in real time now when it comes to actually installing tensorflow you'll actually see that this is really useful because sometimes you need to run through the install multiple times to make sure they're installed but i've actually also got a verification script as well to make sure that we've got all the right packages to be able to go on ahead okay so we've now gone and upgraded pip so the next thing that we want to do is install ipi kernel so ipi kernel allows us to associate our virtual environment with our jupyter notebook so we're going to be mainly working with jupyter notebooks if you're installing jupyter notebooks for the first time i'd highly recommend go to anaconda python so you can actually install anaconda and this is going to give us i'll give you a bunch of jupiter notebooks jupyter labs to be able to get this up and running if you don't have jupyter notebooks installed already i'd highly recommend just running through this reasonably straightforward to get up and installed i've got a video on installing that separately that i can link to you as well so let me know in the comments below if you want that also let me know in the comments below if you like this sort of format me sort of just walking through and giving you an idea as to how to actually go through this i'm always welcome for feedback if there's anything you do like or you don't like by all means do let me know okay so where are we up to now so we were installing ipi kernel so let's do that so again ipi kernel allows us to associate our virtual environment to our jupyter notebook so first up we need to install it and then the second line here is actually going to allow us to associate a little bit of a typo there that's fine so let's go on ahead and install ipod kernel so pip install ipi kernel let's make this a little bit bigger and again you can see that we're installing libraries over here we can hide our navigation pane so you can see that these are all being sold now right now if we type in pip list again let's jump back here you can see that we've now got additional packages now the next thing that we want to do is actually install a virtual environment into jupyter so this is often where people come unstuck when using virtual environments with jupiter for data science workloads you need to install it in to your jupiter kernel notebook so this is going to give you the ability to see that virtual environment or associated to your notebooks so in order to do that we're going to be running this command so python dash m ipi kernel install dash dash user dash dash name tf equals tfod so let's go ahead and run that so i pi kernel all right so python dash m what was it again i already forgot ipi kernel install dash dash user dash dash name equals tfod that should be right so actually let's clear this so we can actually see it a little bit better uh what was it python dash m install dash dash python m ipi kernel installed my bad dash dash user dash dash name equals tfod okay let's check that python dash m i by kernel install dash dash user dash dash name tfid okay it's all good now i wanted to call this out so this last bit here he's effectively telling the ipi kernel package what the name of your virtual environment is so if you change the name of your virtual environment you need to make sure that you change this name here so say for example i when i created my virtual environment so i wrote python-m-v-e-nv my special environment when i go to install that virtual environment into my jupyter notebooks i need to change that name here so in that case it would be my special virtual environment whatever you called it so in this case if i hit run that's going to now install it so you can see installed kernel spec tfod and blah blah blah so now we've actually got that particular virtual environment available inside of our jupyter notebook now an easy way to validate that is just start up jupyter notebooks and if you create a new dummy notebook or even if you just hit new so you'll see that we actually have tfod available there so that is our virtual environment now made available so now we can actually leverage that virtual environment when we work with our notebooks cool all right so that brings us a two-step numeral five i don't know how to say five in spanish unfortunately but step five all right so we're now up to here where we're actually going to go on ahead and collect our images so that up until now we've sort of done quite a fair bit so we've gone through and set up our virtual environment and that's our baseline setup now installed now the next thing that we're going to be doing is as i was saying collecting our images so let's go back and we're now going to collect and label our images so part two is collecting and labeling our images so remember cast your mind back what i was saying is when we actually go and leverage deep learning models we're going to be passing through data aka images and answers aka our labels to be able to train our object detection model so collecting and labeling good images is really really important now in this case to be able to collect and label our images we're going to be leveraging this jupyter notebook that you've got here which is one dot image collection and we're going to be using a webcam really important thing to note is if you're doing this on your machine you need to be able to have access to your webcam in order to be able to actually collect images using your webcam now when it comes to labeling we're going to be using a library called label image so if i actually go back i can show you what that looks like so it's label image and this allows you to label different images within an image so in this case it's really really simple and it's really sort of fit for purpose nothing crazy that actually is involved in actually doing this so we're going to be using that as well and i'll sort of walk you through setting that up okay now in terms of best practices there are a couple of things to note so when you're labeling keep in mind the case that you're actually passing through your images so you want to make sure that the names that you pass through when you're actually labeling them uh the specific case that you want to render so in this case if i name or name this completely lowercase livelong when i go to actually train my model the label that i pass through needs to be lived long as well so again i'll talk about that a little bit later later keep your labels as tight as possible so don't be going so in this case i was labeling the live long sign don't create a big square around it right so you want to keep these labels as tight as possible in order to make sure that your model is able to detect specifically that object right this helps a lot when boosting performance and then include pictures of objects at different angles and under different lighting conditions so what i tend to do is i'll actually like so if i'm doing gesture recognition for example let's get my hand in the photos i'll actually move my hand around i'll take it at different angles at different lighting positions so that my model learns to generalize better so ideally what you want your model to do is perform well not only on your training data but also in the real world this is what is known as your model being generalized right so it works well in general now in order to get a model that performs that well you want to give it a whole bunch of different types of instances of the object that you're trying to detect so say for example i was trying to detect pictures of my phone right if i took a photo of my phone let's disable that so if i took a photo of my phone just like this in the real world if i then went and flipped my phone if it had never seen images of a phone that looks like this right with the different cameras it'd be really difficult for it to detect that right so ideally you want a whole bunch of different angles so if you're detecting fish for example you'd want to take pictures of the fish front on side on the other side and the back top bottom so your model has a chance to be able to see different angles for that particular object because if it doesn't it's going to be really hard for it to know that that particular object is what it is another best practice is to start off with 10 to 20 pictures of each class now i'm going to start off with way less than that because i'm going to show you what to do if you get sucky performance and how to add additional pictures how to retrain so we're going to start off with less i think we're going to do five initially then we're going to rebuild and then redo this model so ideally we should get better performance later on and we'll also do it with the green screen and without the green screen so again i'm going to give it a chance to better generalize so if we've got a green screen in the background as i do right here which you probably can't see right now because i've got chroma key on but if i had the green screen up and i only give it images where i've got a green screen available what would happen if i took the green screen down maybe it might not actually perform that well because it has never seen images of my hand without a green screen behind it so ideally you want to give your image the or you want to give your model the best chance possible to actually go on ahead and detect objects okay so on that note let's go ahead and start collecting images and labeling images so in order to do this we're going to be using the one of the jupyter notebooks that we cloned down which is this one here say image collection dot i pi and b so this is step five so we're now up to here all right so remember we started off at jupyter notebook so again i'll cancel this so you can see it so we let's clear it so in order to do this i'm just going to start our droop in a notebook so by typing j u p y t e r notebook so this is going to start up our jupyter notebook server that we're able to use so if we hit enter now that is now started and you can see inside of here we've got a burn system so we've got our virtual environments a tfod we've got that image collection notebook so one dot image space collection dot ipm we've got our training and detection notebook which we'll talk about more later error guide and our readme so we are going to start off with our image collection component all right so if we select that that's going to open up our notebook and then we're just going to step through this particular file so what we're going to do is we're going to step through it so i'm going to hit hold shift and enter on my keyboard this is going to allow me to step through each one of these cells so shift enter and this first line here is going to install opencv so this is a new dependency that we've actually got available now a key thing to know is that when you're running this what i want you to check is either over here in the top left hand corner you the name of your kernel that you're using is the same as your virtual environment so if it's not the same that means that you're not currently working inside of your virtual environment so you're not leveraging that pre-isolated environment that you went and created and all of the stuff that you're going to be installing is on your local machine inside of your baseline virtual environment so that will mean that it's not as controlled as possible another way to check that is if you hit kernel change kernel just make sure you've got the right one selected so in this case just to be safe so i can hit tfod again and in this case i'm already inside of that so now when i go in install pip install opencv i'm 100 sure that i'm actually installing inside of my virtual environment and again you can tell this based on the insole command right so here you can see that it's actually installing inside of this tfod lib site packages library so again installing in the right place really really critical so if you're getting errors where you're installing a library and then you've got to use it and you're like it's not picking up sometimes it can be that so say for example you install it in on your command line and you don't have your virtual environment activated well then it's going to be installing not inside your virtual environment so when you do use it inside your jupyter notebook it won't be picking it up key thing to note make sure this little bit over here is your virtual environment so that you are 100 sure that you're actually using it so again we're going to step through so this next cell so this first upsell is let's zoom in on that so exclamation mark pip install opencv python so this is installing opencv into your virtual environment opencv just stands for open computer vision so it's a really really popular computer vision library really really good for working with the images or video in this case we're going to use it to access our webcam and collect a bunch of images then the next cell is actually importing some dependencies so you can see here that we're now importing opencv so import cv2 actually imports opencv slightly different naming convention this is pretty normal we're then importing uuid so this is now giving us our unique identifier so this will allow us to name our images uniquely so we don't have overlaps we're then importing os which gives us our operating system this is really good when you're working with file paths so i've used os a a lot in both of these notebooks particularly to ensure that we have file paths that are compliant with both linux mac and windows so that you can sort of step around those and we're also importing time so we're going to use this when we actually collect our images to take a two second break between each image that we collect so this will give us time to move ad to our different angles so remember really really important to have images at different angles to help our model generalize okay then we're going to define the different images to collect so in this case we're going to collect four different objects so in this case this is going to be a multi-class object detection model so we're going to be able to detect thumbs up thumbs down thank you which is a sign language pose and then live long so we're going to give it a bunch of different types of images or a bunch of different types of objects to detect so in this case what we've actually got here is an array so which is labels equals square brackets and then we've just got a bunch of different strings so if you wanted to have different objects to detect or different types of objects this is where you change those labels so in this case here we've got thumbs up thumbs down thank you and live long now again you could change this to your heart's content just make sure that whatever the label is named here is actually it's not so important here it's important later on when you actually go to label make sure that you pay attention to the case um in our projects we're actually going to change this up so rather than doing thumbs up thumbs down and live long say for example with our little leds we'd be doing bent and not bent for example in this case we're pretty happy so we can run that cell and if we actually take a look at our labels this is just a list so if we actually type in type you can see it's just a standard list there's nothing super fancy going on there it's just a place that we can hold our different labels in this case and we've also got this line here number of images so this just represents how many images we are going to collect per class so we're going to collect five images of thumbs up five images of thumbs down five images of thank you and five images of live long now remember what i said during the best practices ideally you want to have 10 to 20 of these images just when you're getting started right so i'll talk about this in performance tuning as well when you want to scale this up and produce a model which is way way better you want to use way more images than that but to prototype and to get up and running 10 to 20 images ideal i'm doing five on purpose here because we're gonna performance tune later on okay then the next thing that we're going to be doing is setting up our folders so right now we don't actually have anything created inside of our folder directory right so let's just add a navigation pane again so if i go into our tensorflow object detection repository i don't have any folders here but i'm actually going to set up a little folder directory that's actually going to be serviced really really well when it comes to training and doing all about other stuff so this particular line here is just setting up our images path so what we're doing is we're now going to create a new path which is going to be called tensorflow and then inside of that we'll create a folder called workspace folder inside of that called images and a folder inside of that called collected images probably doesn't make sense right now let me show you so if i hit shift enter it's going to set up our images path right so for windows you're going to get a double backslash if you print this out rather you can see that what it's actually doing is just navigating to tensorflow workspace images collected images if you wanted a different file path you could change this but just bear in mind that the rest of the tutorial is going to follow that file path and then what we're going to do is create those file paths so first up what this line is doing is checking whether or not the path exists then it's checking what type of operating system is being used so in this case os dot name actually checks what type of operating system is being used so if i actually type that out so earth step name because i'm on a windows machine it's going to read out nt you'll see later on when i do this on a colab instance because that's a linux machine it's actually going to say posix so this gives you the ability to work out to run on both of those operating systems so in this case nt is going to be windows posix is going to be linux now that's all good what we're going to do is run this cell and this is actually going to create this folder directory or this folder path so if i hit shift enter you can see that that's run successfully now if we go back you can see let's just make this a list so it's easy to see we've now gone and created a new folder called tensorflow so that's now available there and if i step in what you'll actually see is that it is that it's actually creating the whole file path so tensorflow workspace should zoom in on this by default it's going to create tensorflow workspace images collected images so tensorflow workspace images collected images now what it also does at the same time is it creates a new folder per class so it'll create a new one for thumbs up new one for thumbs down new one for thank you and a new one for live long if you change this add different objects add different labels then it's going to create those folders as well when you run this particular cell here so if you don't have these folders created make sure that you've got your labels defined correctly and make sure that you run the cell it'll create those folders for you now the next thing that we're actually going to do is actually capture our images so this cell over here is actually going to do all of this for us so it's going to loop through all of our labels it's then going to connect to our webcam now this line is really really important here so right so this line connects to our webcam or capture device so right now we're just doing it using our webcam right but later on when we go to do object detection using our microscope i'm actually going to need to change this device number to be able to detect that particular microscope now what might happen is this number that you see here so right now i've got 0 in there if you're working on a different machine which undoubtedly you will be this number or this video capture device is going my theory right so on my mac so on my other mac machine the number that i need to pass through into here in order to be able to connect to my webcam is number two now if you get errors which undoubtedly it might happen so if i actually go back into our error guide so if you get an error that looks like this so value error images must have three or four dimensions more often than not that is because your webcam you haven't got that webcam number correct so if you see a grey box pop up and you're not getting any images back again it could be that as well so sometimes you might need to either change that number or restart your notebook to be able to connect to your specific webcam now in this case i believe video capture device one is correct so we should be good to go so what's going to happen here is we're going to first up print out which label that we're collecting images for so it's going to print out collecting images for and then the label that it's looping through because remember it's going to loop through our labels up here it's then going to give us a break for five seconds that enabled us to get into position so it's gonna sleep for five seconds and this is where that sleep library down here is coming in handy and then it's going to loop through the number of images that we want to collect so remember we're going to collect five images in this case if we change this number up here from 5 to 10 or 15 or whatever it's going to change the number of images that we actually collect so it'll then print out collecting image for a specific label it'll then capture the frame using our webcam and then what it's going to do this is really important so it's going to create a new image and it's going to place that inside of this particular folder path so it's going to place it inside of images path which is let's actually print this out so you'll see it ah it's not going to work if i don't have an image so it's going to put it inside i'll change that now it's going to paste it inside of images path and then label so in this case say for example we are up to thumbs up it would put it inside of thumbs up which is over here and then it'll name the actual file label dot and then a random number so it's going to create this unique identifier dot jpg so all of our files will be jpeg they'll be named the specific label plus the unique identifier and they'll all be inside of their respective label folders so we're going to need to move these around a little bit later but that's fine for now then it's going to use the cv2.iamwrite command to be able to write that down into file and it'll actually show it to a screen it'll then take a two second break again and if at any point we need to escape out of it we can hit the q key on our keyboard to be able to get out of that it'll then close everything down if i zoom out of that now let's go on ahead and run this are we zoomed in no we're not zoomed in all right cool so if we run this ideally what you should see is a little pop-up at the bottom of your screen which indicates that it's now connected to opencv and it's actually rendering so you've got remember you've got five seconds to get into position so in this case the part that we're going to go down is thumbs up thumbs down thank you and then live long so again i'm going to make sure so we might do this once see what where our webcam's position and we might need to fix that up so i'm just looking at where mine is now it should be okay let's give it a crack all right so i'm just gonna run that cell again shift enter should see a little pop up down the bottom looks like it's okay for now you can see that's popped up now and again first position is going to be thumbs up it's very dark let's get back into the center not the best images of thumbs up we'll try it anyway and see what it looks like thumbs down and the next one should be thank you i believe let me switch to this position so again you get a little bit of a break between each of these images so you've got time to move around right if you want to increase the break you can just change that number and again i'm trying to get into slightly different positions each time and then the last one it should be live long so all right that looks good remember right now we've only got our green screen images as well so we might choose to drop the green screen and create new detections later on if we get really really poor performance we'll test this out again this meant to be a live example so you can see what some of the challenges that you might face and how to solve those alrighty cool so it looks like we've now got a bunch of images so again if we go into tensorflow workspace images collected images you can see that we've now got a bunch of images so our live long positions looking fine and the way that i'm sort of judging these the only thing i'm a little bit concerned about with these images is that right now they're all in the same format so let's just enable preview um how do we do this again preview pen here so right now they're all sort of in the same position right so this is not good practice when it comes to building a generalized model so what we might choose to do later on is move into different positions and ideally drop the green screen add more images train this up again so in this case that's our baseline right so and again when you're prototyping a key thing in any data science project i preaches all the time prototype as fast as possible because then you're going to work out whether or not what you're trying to do is feasible and whether or not it's possible so say for example you're trying to detect i don't know say for example you were trying to detect bent uh leds and you're just doing it with a webcam well you're gonna need a really really hyper accurate model or one that's able to see really really fine grain detail to be able to do this so if you were to prototype really really quickly with a mobile net model and just a webcam you might see that you get absolutely sucky performance now if you do this fast at least you get to the end output to realize that hey maybe we need to try a different technique maybe we need a different sensor aka microscope by doing this quickly and prototyping quickly you're able to see and establish a baseline from which you can go on then and try to improve performance in this case we're going to give it a crack and see what it looks like so we've got our live long images let's check thank you these ones look a little bit better again obstructed by the microphone a little bit we'll see how that performs it's looking good thumbs down that looks alright again so you can see that right so i'm giving my model a better chance to generalize by having a thumb position there and a thumb position there that's looking better and again a few different ones again reasonably close together now another idea here right is to do it with different hands so in this case i've just done it with this hand but i could have chosen to do it with this hand right gonna give me a better chance to generalize that way and thumbs up that image is crap so we're actually gonna get rid so this is another thing to know right so that sometimes you might need to delete images so in this case this one's really dark and it doesn't even contain an image of a thumbs up so if you don't have the object included within that particular image it's not going to be used for training as part of the tensorflow object detection api so we can actually just delete this image and delete this one because it kind of sucks and i've got a feeling that after we train this model our thumbs up detection is not going to be good but let's give it a crack and see what it looks like we can always performance tune later on so keep in mind our thumb is really really close to the edge this one is half cut off this one is in front of our face so again we don't have a good run of example images there so if we wanted to we could actually just add more images in now right so rather than collecting images all the way again for all of these labels what i'll do is i'll just delete um or add in a new array because i've got a feeling that those thumb images really aren't that good and we'll just call it thumbs what was it up i think it was thumbs up so now if we take a look at our labels so now we should be able to collect more images for thumbs up so let's go on ahead so i've just gone and added in a new variable so if we wanted to go back to our normal labels we could do that as well now again it's pretty common so you might need to add or capture additional images or capture less images in this case i'm just going to capture a couple more images a thumbs up because we've only got three and all three of them kind of suck so and right so like they're only like half detection so this one barely even counts you can only just see your thumb there so ideally we want to get a number of good images so let's do this now so what i'm going to do is i'm going to run this cell again and see how we go so if i run that ideally so we're going to be trying to collect thumbs up you should see a little pop-up again all right we've got our frame it's looking better all right that looks a little bit better now so again we go back into our folder tensorflow workspace images collected images and thumbs up so now we've got way more right so if we take a look at our latest ones again we've got a bunch of different eyes are closed they're great um so again got a bunch of different images we're giving it a better chance to generalize and now i've got images for both hands as well right all right so let's do that and see how we go all right so we've now gone through installing and importing our dependencies for image collection defining the labels that we want to collect as well as how to collect additional images for a particular class and we've also gone and set up our folders and captured our images now again keep in mind when you are doing this bit you might need to change this number depending on your webcam capture device number so it's fine to change it if you have to change it by all means do make sure that it works for you sometimes i try zero one two three um some particular some actually some devices have virtual video capture devices so you need to take those into account as well just again you might need to try five six seven eight nine just make sure that you try a bunch of them ideally you should see that pop-up appear exactly as i showed now we're going to get to our labeling so as i was saying we're going to be using label image for this so this particular package here now if you want to take a look at this package you can go to github.com forward slash i don't know how to pronounce that t-z-u-t-a-l-i-n zoo talon sorry if i butchered that name massive respect here it's an awesome package all right so we're gonna be using label image to be able to go and label our images now when i talk about labeling really we're just drawing boxes around the stuff that we're trying to detect now also another thing to keep in mind is you can label multiple objects in one image as well so another thing to to keep up your hand now down the bottom here there's a whole bunch of installation steps that you can actually go through now what i've actually done is i've actually included this within the jupyter notebook so rather than having to step out install do all of that stuff we can actually just install it using our notebook so first up what we're going to install is pi qt5 and lxml so if we run that now remember we're inside of our virtual environment so if we actually take a look at what we've got in here now hit this now let's actually run this cell first cool so that's now installed and if we run pip list you can see that we now have a whole bunch of other stuff installed and if we look at the two libraries or dependencies that we just wanted to install so pi qt5 and lxml we now have lxml which is there and pi qt5 as well as all of its other dependencies so we're good there right now the next thing that we want to do is create a little home for our label image file or label image package to be saved so what we're actually going to do is create a new folder called tensorflow and then label image so it'll be inside of our tensorflow folder and then a new folder called label image so it's effectively going to be let's keep this open i don't know why i keep closing that um so tfod course tensorflow so we're going to create a new directory here called label image and that's going to be where it gets stored so if i run this cell let's minimize this and then if we now go ahead and create this what we're effectively going to be doing is first up creating a new directory using make dr or that make deer command there so first up we're actually going to check whether or not that file path exists so if it does already exist as in it's done is installed then it's not going to do anything so if you get any errors what you might want to do is actually just delete the label image folder and run the cell again now in this case we haven't run it so we can run that so it'll create a new directory called label image which you can see there and then it's actually going to go in ahead and clone this thing that i was actually showing you so this github repository it's going to clone that so whenever i'm running these commands through a jupyter notebook it tends to be not a little bit slow but you don't actually see the progress so right now it is running so we've got a little star over there so you can see it so it is running you're just not going to see it if you're running on a windows machine but it is clonings so if we open that doesn't look like there's anything there yet let's wait that for that to clone all right so that is our label image package now cloned so if we actually open that up you can see that we've now got that cloned into that folder so we've got tfod tensorflow label image and inside of that we've got effectively all of this stuff cloned into our local repository now the next thing that we need to do is actually go ahead and install label image i believe that might be a typo there so let's just copy that over here okay and then what we're going to be doing so again we're running on a windows machine so we're going to be executing this path down here so we're going to cd into that label image folder so if i actually show you that label image so again we're going to be going into this folder so tensorflow label image then we're going to be running in this command here so pi rcc five dash oh libs forward slash resources dot pi resources dot qrc this is effectively the installation steps that are recommended from the main package so if i actually go down here so for windows we're going to be doing this so pi rcc five dash zero libs forward slash resources dot pi resources dot qrc and then we'll open it up so effectively we're just running through the same installation steps that you would get from the actual repository itself so if i go on ahead and run this now it should ideally run successfully so it doesn't look like we've got any errors there and then if we go on ahead and run this command what should actually happen now is that it'll actually open up the label image package so the actual application will actually open up so if i actually hit shift enter you can see label image is opening up so we've now gone and installed it from scratch and actually opened up the label image package now the fun bit comes actually going on ahead and labeling our images so what i'm going to do is i'm going to over this left hand side here you can see that this button called opendr i'm going to wave my cursor around it so you can see it so i'm going to hit open directory or open dr and then i'm going to navigate to my folder so i'm going to go to youtube tens flow object detection tensorflow object detection course tensorflow workspace images collected images and remember we've got a repository or a folder for each one of our different classes so in this case here what i'm going to do is step into live long first hit select folder and sometimes it won't actually open that folder it'll just ask you to select that folder again what you need to do is just hit cancel b as long as you've got the files down here it means that you're able to label them so i'm going to start out by labeling my first image so if i double click onto that that opens up our image now just a couple of quick tips so what you can do in order to label this image so there's a shortcut right so all you need to do is hit w and this is going to open up a little labeling thing over here so you can see we've got our cursor to label an image what you then want to do is just select the first point that you want to label and then drag down and create the object now what i tend to do is i find tend to find the outermost point so in this case i'm going right to the top of my finger and right to the edge of my hand then what i'll do is i'll drag it so that i only encapsulate the components of my hand that i want so you can see i'm going just to the edge of my what is that pointer finger and to the edge of my wrist so i'm going to be as tight as possible with my detection or my label and then so remember when i kept on harping on about make sure that your label detections are tight so that's what i meant by tight and then make sure that you pay attention to what you actually name these labels because when we go to train you're going to need to make sure that the names of the labels in your annotations which is what we're doing right now match the actual labels that you pass through to your model that you're training so in this case i'm going to label it live long and i'm going to have a capital l and a capital a capital alpha live and a capital l for long now when i go to set up my labels later on in my training step i'm going to make sure i've got the exact same thing if you don't you're going to get an error right and that error is going to look like uh this so in this case here you can see it's when running generate tf record script blah blah blah blah blah and then it's you're going to get this key error here but i'm actually going to show you what that looks like when we go to do it so in this case keep in mind i've labeled it capital live capital long if i go and screw up when i go to train it's fine you can fix it it's not hard but you are going to get an error so in this case pay a lot of attention to the capitalization and the case of your labels so i'm going to label it live long hit ok and then hit ctrl s to save it and save that's my first image labeled so we're going to go through and do the rest so i can select my next image and again we're going to keep them tight so i can do that and say i screw up right and do that i can still go in and drag this back and tighten up my detection so i can do that make it a bit tighter and you can see that there's this little bit of my image over here that i haven't actually captured so if i zoom in you can see over there that i'm not capturing that so we might pull that in now maybe i'm being a little bit pedantic now but that is my ocd kicking in and it is best practice right so you want to make sure that you get this as accurate as possible so we're going to save this that's annotation 2 done now we're going to go to our next image and again do the same thing so we're just going to do this for all of our images now the beauty of what we're doing right now right is that we can do this for whatever type of object that we wanted to save so say for example we wanted to do i don't know um and i saw a really really cool video the other day of someone that used this to classify which way waves were breaking i thought that that was pretty awesome um you could do it for a whole range of different applications so obviously we did sign language that was a great use case super practical and super altruistic and there's like a whole heap of capabilities like there's people out there that need this technology so if you've got the capability to do it and you're watching this course by all means find something interesting and give it a crack so i'm obviously going to give you um some examples when we actually go to do our projects but again if you get come up with a good idea this exact same process that we're going through define your labels collect your images label them and then we'll get through to the training later on but you sort of get the idea that's what we're going through okay so we're now going to label these images now one thing that i want you to check as you're doing this is if you go into the images folder and in this case we've done live long right that you do have these annotations available right so in this case i'm looking at this image here i should have a matching annotation and it's going to have the exact same name as your image file but rather the extension is going to be xml so if i actually take a look let's go details so in this case i've got the image let's do we need date we don't need date so i've got my image over here which is named whatever that is dot jpeg file and then i've got the annotation which matches that image now if i open up this annotation let's take a look what's actually included in here so inside of my annotation i've got obviously the fact that it's an xml file so it starts off with annotation then tells me what folder i'm in what my file name is the path to that file name now this doesn't matter so much when it actually comes to training because we're going to move these around the source and again the database doesn't matter here the size is quite important because this defines how big our images so in this case we've got width height let's make this a bit bigger we can see a bit better we've got width height and depth so in this case it's a color image so it's going to have r j and b whether or not it's segmented so we're not segmenting at the moment and then this is the bit that's really important so you might have multiple objects in an image in which case this would be repeated multiple times so here we've got the name of our object and again we've got it capital live capital long so if you screw up and you wanted to the names to be different you could just come in and change these names as well we've got pose we're not using that truncated and difficult we're not using that and then we've got the bounding box so again we've got the different coordinates so we've got the minimum x value and the minimum y value so effectively we're just giving it two points and we're giving it a point here and a point here and the square that represents that box is going to be our bounding box so that's really what it comes down to when it comes to labeling so we're effectively creating these xml files okay but right now we've only done our one class so if we just wanted to detect uh live long that's perfectly fine you can go on ahead and jump off to training if you only had one object to do you could do that in this case keep in mind we had live long we had thank you so if we go and do that now so i'm going to hit step into that folder and then hit select folder sometimes you've got to hit cancel then open up our image so again we're going to do the same thing now again i'm a little concerned that we've got the mic there but that's fine let's give it a crack so in this case i'm going to label to my knuckles and again to my wrist so this one is going to be called thank you oh almost had a typo they're dangerous so capital t and capital y hit okay then again control s to save then we'll go to our next one so i'm again different angle so this should ideally give us a better chance of generalizing i'm going to save that save go to our next image and again mic there we'll see what that does again i'm just going to keep going through so we're going to do the next one save next one and again looks like i've only got two angles there sort of up and down so one idea there and i've screwed up my detection so i'm just going to hit ok and then i'm going to tighten that up save cool so we can double check just by clicking through all of them in this case i can see all of my detections are saved and they're good if you forget to save what will happen actually i'll show you in the next one so in this case we should be fine so if we go and check our images again thank you so again remember we've got all of our images and we've got their associated annotation files so ideally we should have a round number so it should be even so we've got 10 image of five images and we've got five annotations so we're good there cool alrighty now we're going to open up our next class i want to show you what happens if we forget to save so let's do thumbs down now so double click right so i'm going to label thumbs down hit ok if i then go and double click into my next one when i come back to that image my annotation isn't going to be saved so you've got to keep in mind that you need to save it right so you won't actually have that file created so again i'm going to just do thumbs down change my label name so it's going to be thumbs down save now if i step into my next image and come back i'm going to have that annotation saved so we'll do our next one so sometimes you might notice you forget one if you're doing a ton of these um again you might um you might sometimes it happens you forget to save now a lot of some people have asked about how would we scale this up so say we've got thousands of images there's normally two ways that i recommend here so you can either outsource this mechanical turkey is one way to go or if you go to like upwork or freelancer that's another way the other way is that there are newer tools these days that actually use a technique called auto labeling so they'll actually train a lightweight object detection model to be able to detect the different objects within the images so it's pretty cool right and then what it'll do it'll ask you to label say for example 10 or 20 or 15 images and then you can run that auto labeling and it'll go and auto label all of the other images based on the lightweight model it's already trained now again those are pretty sophisticated but if you'd like to see a video on that by all means do let me know i'll see if i can spin something up in this case we're just going to use label image so again you can see all of our images looking fine there no issues okay let's do our next directory now so we've done thumbs down now we want to do thumbs up and this was the one that we remember we had a lot of images for so we went and collected more so we are now doing thumbs down let's just i can't remember if i labeled that correctly now so the other one we did label thumbs down right so if we open up our annotation yep thumbs down okay so we're good and thumbs down with capitals perfect all right so now we're going to do thumbs up so let's label it and same thing hit w to get that shortcut and open it up this is going to be thumbs up save next image this is a partial so that's fine let's do it anyway again i'm just going to keep going through so again this can get a little tedious when you go to tiny images but just uh sort of walk through normally i'll grab a coffee and i'll just grind through and label these objects and again prototype so don't start out by labeling uh thousands of images because it might not be the right technique so what i'd always recommend is do 1025 if you really want to keep it super fast prototype train and then evaluate right so don't commit to building a huge object detection model before uh look don't commit to building the whole thing with thousands of images because you want to make sure that you're going down the right technique you're getting the right levels of data you're getting you're using the right types of sensors to be able to do this so prototype first going to make you a lot more efficient i'm going to keep doing our thumbs up and again i'm going to adjust our label this time because i screwed up so that looks a bit better save and then we're going to do that thumbs up save you can see so i i did that thumbs up might perform a little bit better now because we've now got images at a bunch of different angles let's tighten that one up and save it again um but keep in mind we've only got a green screen in the background at the moment so it'll be interesting to see what performance looks like and if it sucks we'll make it better okay cool let's double check our annotations as per usual so we're good good good good good good good good all right cool so that looks all good so we've now labeled all of our images and if we check that let's bring that back so thumbs down it now has all of our annotations so if we're going to live long we're good there thank you we are good there thumbs down we're good there thumbs up we're good there as well okay cool so we've now gone and completed all of our labeling now the next thing which you really need to do is we're now going to split this up into a training and testing partition so when we get to our training step we're going to train our images or train our model on a portion of images and then we're going to separate part of it out or we're going to use the other part of our images to actually test how it is so the reason that we split this up is we want to train it on a segment of images that we're going to use for training and then we want to test it on a set of images that it's never seen before look the images might be similar but we want to give it a chance to try it out and see what performance looks like on images that it hasn't explicitly seen before so to do that so you can see we've got step 6 here so we've now done let's quickly recap so we've imported our dependencies we've defined our images to collect we've set up our folders we've captured our images and we've labeled so we've done quite a fair bit now what we need to do is move them into a training and test testing partition which is step six so what i'm going to do is i'm going to go back into that folder so inside of our images folders i'm going to hold there and i'm going to create two new folders now so i'm going to call one called train and i'm going to call one called test now the names of these folders is really important because inside of the training script it's actually going to be looking for these training and testing folders or the test and train folder explicitly so you need to make sure that you create these folders and you need to make sure that you have images and the associated annotation in it so what i'm going to do now is i'm going to take a random portion of our images from each class so we're going to take some from livelong some from thank you some thumbs down something thumbs up and we're going to put it into that training folder so these are the going to be the images that we use to train our object detection model so if i step in to live long i'm going to get everything including the annotation so maybe we'll leave one image for testing so what we're going to have effectively one two three four images and one two three four annotation so i'm going to cut that paste it into the training folder and i'm gonna do this for each class so we're gonna go into thank you so one two three four five six seven eight so again four images four annotations again paste it into the training folder now we're gonna do a thumbs down and again one two three four five six seven eight got to grab that one gonna go back in and the reason why i'm being so explicit in making sure that we grab the annotation and the image is if you skip out and you don't have the annotation or if you skip out and you don't have the image it's going to screw up when you go to do the training step so you need to make sure that you grab both the image and both the annotation really really important and we're going to do the last step so i'm going to grab let's leave two images for testing there so i'm going to grab 1 2 3 4 5 6 7 8 9 10 11 12. all right so we'll leave two images and two annotations for thumbs up because remember we add more images there all right and then we're going to paste those into our training folder so now we should have an even number so we've got 36 a little bit smaller down there but we've got 36 images so we've got the images and the annotation so we're good for our training now we need to do the same thing for our testing so we're just going to grab the rest paste it into our test thank you and again you could automate this you could write a little bit of python script to randomly select your folders if you do get around to it let me know i'd love to see the code um paste it into test but i found that just doing it manually isn't too bad um particularly if you only got a small segment of images i'm going to grab our thumbs down a thumbs up and then paste that into test right so ideally what you should end up with is an even number of images both inside of your test folder and inside of your train folder so if we take a look at our full file path inside of our tfod course folder we should have tensorflow workspace images test and test has images and annotations and train and again train has images and annotations so these images and annotations are going to be used to actually train our model we're going to do this technique or user technique which converts these images and annotations to something called a tensorflow record and again i've got scripts to help you do that so we're going to convert those and then use that those tensorflow records to be able to actually train our model now if we go back into our images folder we can also just delete these folders as well now because we don't need these there's nothing in them so inside of collected images we can actually just delete that now so again all of our images and annotations are inside of test and train so we're good to go now inside of that image collection notebook i've also got the code to compress these images to be able to push them up into collab but again we're going to do that when we get to our collab training so again hold on for that for now you don't need to run these last two cells which is everything under step 7 which says optional cool all right so that's image collection and labeling done so we've gone and collected our images captured our images labeled them using label image and we've now gone and split them up into our training and testing partition so again remember last step was really really important you need to have a training and a testing folder because this is what our training script is going to be looking for so we've also added to some of those best practices so we were case sensitive we made sure we got our cases right we kept our labels tight and right now we've only done five images of each class except for thumbs up so again we might choose to add more later on so we're good there this brings us two step three training our models so in part three we're actually going to start training our models so we're specifically going to be using a model called a mobile net which is a specific deep learning architecture but you could use a bunch of different types of models and we might actually do that in the performance shooting section as well so when it comes to training a model using tensorflow object detection so there's a bunch of steps that you need to go through and a bunch of dependencies that you're going to be needing now namely because we are using tensorflow object detection one of the core dependencies is going to be tensorflow so we're actually going to go through install all of our dependencies set up our folder structures and actually start training this model now as part of this we're also going to be evaluating our model now i might come back to this a little bit later but i wanted to talk about how to evaluate your models now the nice thing about tensorflow object detection is that you're able to calculate these evaluation metrics after you've actually gone and trained your model now specifically when you run your evaluation you're going to get two key things so you'll get precision and normally you'll get a metric called map which stands for mean average precision and this basically tries to calculate what proportion of your detections were correct right so what it's actually doing is it's calculating the true positive or so how it's actually calculated is it's the number of true positive detections divided by the true positives and false positives so effectively it's the number of detections that it got right so in this case here you can see i've got this little matrix so this is a true positive detection so you can see this is the sign for hello right and it's accurately detected the sign hello so it's a true positive now on the other case you might have a false positive so it's gone and detected a label here which says hello but it doesn't that's not actually correct because it doesn't actually have that label there or it doesn't actually have that object there so even though it's gone and made a positive detection it's a false positive because it's not actually correct you've also got a true negative which in this case there's no object so there's no hello in there so say for example this is just a hello model so there's no actual hello object in there in that case it's actually a true negative now in this case we've got a false negative so there is a hello object in there but it hasn't detected that as hello so it's falsely rejected it right so what you're actually going to get when you calculate your evaluation metrics are precision and recall which are calculated off the number of true positive false positive true negative and false negative detections so in this case your precision metric is going to be true positive which are the number of detections that have accurately detected the correct object divided by the number of true positives and the false positives so effectively what that's giving you in layman's terms is what proportion of my detections were actually correct right so this is correct this is bad so ideally you want this precision number to be as high as possible a higher precision number is a better precision number likewise a higher recall number is also a better number so your recall is calculated as the number of true positives divided by the true positives and the false negatives right so this is basically capturing what proportion of my actual objects did i capture so in this case you can see that we might have captured this but we didn't capture this so really it's trying to measure hey for the objects that were in the screen what proportion of those did we actually get so when we calculate our evaluation metrics later on you'll actually be able to see these um you also see loss talked about a little bit so it's a method of evaluating how well the model is performing against the data provided ideally you want loss to be as low as possible recall to be as high as possible and precision to be as high as possible but again we'll talk about this when we get to that step all right so tensorflow model zoo so there's a whole heap of different type of model that you can actually go on ahead and train right now what you normally have to do is balance a trade-off so you'll balance faster detections ideally being able to detect stuff really really quickly versus lower accuracy so the ones that tend to be faster tend to be less accurate not all the time but most of the time the ones that are highly accurate tend to be slower right so if you have more sophisticated models bigger architectures it's going to take slower to actually make that detection so if i actually show you right so if you actually go to google and type in tensorflow model zoo and make sure you type in tf2 which stands for tensorflow 2. there's a heap of different types of models right so these models are just different model names and different model architectures so if i zoom in here you'll see that we've got a model called centernet we've got another model called efficient det we've got an sse mobile net which stands for single shot detector mobile net single shot detector resonant so each of these just represent different neural network architectures which are used for object detection now you don't need to go into great amounts of detail as to how these are actually built up what you do need to know as a practitioner however is that using different models is going to result in different levels of accuracy as well as different levels of speed so say for example i wanted to use so let's actually take a look at the model that we're going to be using so we're actually using this one here so ssd mobile net v2 fpm light 320 by 320. so this is the model that we're actually going to be using and if you actually take a look at that link there so if i copy this link address the let's actually open up the notebook actually we'll come back to that so the link for this is actually going to be the link that we're actually using in our notebook and i'll show you that in a sec um now this model has a let's take a look here so we've got two components we've got speed and we've got our mean average precision remember we talked about precision here in this case we've got the average precision which is represented over here on the coco dataset so this is just a standard data set so just think of these two columns as how fast the model is and how accurate the model is likely to be so in this case we've got a bit of a balanced model right so our ssd mobile net is going to run each frame in 22 milliseconds so ideally you want this number to be the lower the better and it's going to have a mean average precision of 22.2 so again it's reasonably balanced now remember when i said so you're going to have a bit of a trade-off here so in this case and certain certain architectures can be fast and can be highly accurate as well so just keep that in mind as well so you might get a model that's really really quick but also really really accurate so we could use this one here but say for example we wanted to get a model which is more accurate later on well we might choose to go to this one so ssd mobile net v2 fp and light 640 by 640. so it is going to be slower so it's going to take 39 milliseconds per frame but you can see here that our mean average precision is 28.2 so again there's a trade-off so you might choose to use a so you i guess you can sort of make a decision here so if you need a model that's going to be deployed on a phone or on a raspberry pi you ideally want to optimize for speed right because a raspberry pi isn't going without any acceleration isn't going to be that perform or isn't going to be able to perform that fast anyway so you want to choose the model which is already really really lightweight and it's going to be able to run so in this case this is why i tend to use this model a lot unless i have a specific use case where i need a really really accurate model now again a core difference between this model and this model is that this model is actually going to compress our image from 640 by 480 which we had from our webcam to 320 by 320 when we actually pass it through you don't actually need to do anything as part of this it's actually going to use a technique called pre-processing to compress that down to 320 by 320 and then in post-processing it's going to take the detections that it found and convert it back to your original resolution so it'll actually do that for you and this is one of the advantages of the tensorflow update detection model or api you don't have to do that pre-processing and post-processing which can be a pain sometimes another cool thing about it is it does a technique or uses a bunch of techniques called image augmentation so it might darken your image you might shift it it might flip it and it will do the same for the detection so it does all of that for you certain models will have different levels of data augmentation or image augmentation but just keep in mind the reason that you do those is to ideally get a better performing model so ideally a more accurate model now again you can use a bunch of different types of models there's one really really slow one down here which again runs in 301 milliseconds but the reason that this particular model takes a lot longer is because this actually gives you a bunch of additional stuff as well so this actually uses a mask which in this case this this course covers object detection the mask model which you can see down here actually does something called image segmentation as well so rather than just detecting a box around my face it can actually trace around my face and detect the specific shape of my face not just a box now again because it is a lot more sophisticated it's going to take a lot longer to run so in this case 301 milliseconds so again you've got a bit of a trade-off here as to which model you choose to use now we're going to be using this one over here so ssd mobile net v2 fpm live 320 by 320 but again in performance tuning we might choose to use the different model and see what that looks like so that is tensorflow model zoo so whenever somebody asks you about tensorflow model zoo this is normally what they're talking about okay so that is that now sometimes you might not want to train locally on your machine you rather use a cloud instance so say for example you're not at home you can't access your desktop you might want to train something on collab well you can do that but a key thing to note is that when you train on a cloud instance sometimes it can be difficult to connect to different devices so say for example you want to connect your microscope or connect your webcam sometimes that can be a little bit tricky so that's just one thing to keep in mind but again we'll cover that a little bit later on when we get to that cloud section so on that note it's time to kick off our training so again we're going to jump back into our jupiter notebook which we had over here and before we were operating inside of our image collection notebook but this time we are now going to open up our training and detection notebook so there's a bunch of stuff that we're going to be going through this but we're going to go step by step so all up there are 13 steps all together in this notebook but we are going to go for our training section at least we are going to go up to step 7 so evaluate our model so we're going to effectively go through all of these steps so by the end of this we should have a trained object detection model that we can then begin to use in our detection section so again we're just going to step into our first cell and we're going to run it so in this case we're importing operating system as well and then we're actually going to set up our pre-trained model variables so in this case we're defining what we want our custom model folder to be so again when we go to our performance tuning model we might change this we're setting up the name of our pre-trained model so this is going to be the name of the file that we download from our tensorflow model zoo which is over here so when we actually download this where is it let me find it again so this particular file here if so when we actually download it from here what you'll actually see is the file that we get back is called ssd underscore mobile net underscore v2 fpm lite 320 by 320 coco 17 tpu-8 so again this is something that i've just sort of pre-configured so again if you wanted to customize this you could as well and then what i've also got in here is the pre-trained model url so i was saying before that we've got our model url now where i got this from is if i right click on this and copy this link address what you'll actually see so i'm going into a lot more detail in this than i normally do and i sort of wanted to show you how i built these up so this link that we've got here is the same right so you can see that that's the exact same link so what we're effectively doing is we're going to be downloading that model from the tensorflow model zoom and using it in here right so we've got the name of the file so if we wanted to replace this we just need to make sure that we get the name of the new file right so in this case we can see that it's going to be ssd underscore mobile net underscore v2 fp and light 320 by 320 underscore coco 17 underscore tpu-8.tar.gz so again if we got a different one we could change this pre-trained model name or we'd need to change this pre-trained model name and this pre-trained model url but you don't need to stress about this for now you can just leave it it's going to work and then we're setting up a few different variables so we're setting up a generate tf record script which we'll come to later in our label map name which again we'll come to later so i'm actually going to run through this cell this cell this cell and this cell and what you'll actually notice now is that we've actually got a couple of things so this over here is going to create our different paths so if i actually type in paths so this just makes it a whole heap easier to work with all of the stuff that we've got available within our working directory so we're able to refer to each one of these reasonably easily so say for example we wanted to get where our model is going to be we could just type in paths checkpoint path and you can see that we're now going to have that folder there now let me explain each one of these parts our workspace is going to be a high level workspace so let's open this up so our workspace is going to be this and inside of our workspace we're going to have our annotations which is where we're going to create our tf record files more on that later images was already there from when we collected our images we also have our models folder and our my ssd mobnet folder which represents these two lines over let's find it so we've got our models folder which is this and our which is our model path and we've also got our checkpoint path which is this as well so if you actually take a look at checkpoint path or if we type in model path first up that is this folder here so tensorflow workspace models we type in checkpoint path that is this folder here but again you don't need to mess around with this i'm sort of just giving you a bit of an explainer as to what each one of these are so each one of these represents a different file path that we've got available there so if you want to dig into these a little bit more you can we've also got a couple of variables inside of this files dictionary here so if we type in files you can see that we've got a file for our pipeline config which we don't have created yet our tf record script and our label map but again more on those later this script down here is actually creating all of that directory so you can see before that we didn't have all of these directories created right so our script actually went and did that so if we actually now step into our tensorflow folder we should now have five folders so our label image folder our models folder our pro talk folder again more on that later a scripts folder and our workspace folder and if you step into your workspace folder you're gonna have a bunch more available in there as well okay that is our folder structure now set up now the next thing that we're going to go ahead and do is actually download our tensorflow models from our tensorflow model zoo and we're also going to install tensorflow object detection now this tends to be where a lot of people get stuck so we're actually going to take this step by step and i'm going to show you what to look for and how to validate that you've actually got this successfully installed so again we're going to step into our first cell which is this one over here and we're going to hit shift enter and this is going to install a package called wget so this is just going to allow us to actually pull down our model from tensorflow model zoom and then what we're going to do is actually clone down the tensorflow object detection repository so if i copy this link here let me actually show you that so this is the tensorflow model garden now inside of here there is a bunch of stuff that is available for tensorflow but specifically we're interested in object detection and when people refer to the tensorflow object detection api this specific section which is in research and object detection is what they're talking about so everything that's in here is what represents the tensorflow object detection api now again you can see that there's a bunch of stuff that's available through here there's a whole bunch of documentation on this as well there's also a bunch of open source tutorials that are out there but again i've tended to find this to be a little bit tricky to actually get through so again this is why we're building up this course and while we're going to be going through it in great detail so what we're actually going to go on ahead and do in that next line of code that we're about to run is we're actually going to clone everything that you see here and we're going to clone it into this folder so we're going to clone it into tensorflow or if we start off inside of our root directory so tensorflow object detection course we're going to clone this model garden which is everything you see here into tensorflow and then models so then what we're actually going to do is we're going to install it from there so if i run this cell here it's going to do exactly that now this may take a little bit of time to run depending on the speed of your internet so let's go on ahead and run that okay so that is our tensorflow model garden now cloned so you can see in here that we've got a whole bunch of stuff so we've got community official orbit research so on and so forth that is just mimicking this now again keep in mind there's a lot of stuff in here that you're not going to need for this but we're just cloning down the entire repository so specifically we're going to need inside of the research folder inside of object detection this is us right here this is all the stuff that we're going to need to leverage the tensorflow object detection model api or the tensorflow object detection api now keep in mind this is tensorflow 2 so this is the latest version in tensorflow not tensorflow 1. cool all righty so what are we going to do now so we are going to then go on ahead and install it so right now we've just cloned it so if we take a look let's minimize this we've just cloned it into this repository so again remember inside of research inside of object detection this is all of the stuff that we're going to leverage now what we actually need to do is actually run through the install step now again there's a bunch of stuff required to actually go ahead and install this so if you do get stuck or if you do run into issues by all means check the model guide but i'm actually going to step you through how i actually go on ahead and install this stuff so the next cell is actually going to again detect whether or not we're using linux or whether or not we're installing windows and it's going to do three things so it's at first up it's going to install protoc so protoc so a lot of the tensorflow object detection stuff using uses something called protocol buffers so if you actually type in protoc so what we're actually going to do is we're going to set this up for our tensorflow object detection api so again you don't need to do anything but run that cell it's going to do it for you now normally this can be a little bit tedious so again it's just going to run this cell and do it for us locally so if we run this now it's going to install protoc and then it this line over here is actually going to go ahead and install the official tensorflow object detection api then what we'll do is we'll actually verify our installation so this script down here so let's actually clear our output so we can see that so this cell down here is actually going to run our verification script so this will tell us whether or not we've actually got the tensorflow object detection api installed successfully if it is installed successfully you'll see it read as okay so that will mean that we've successfully installed the object detection api and we can kick things off if not i'm going to show you how to work through it and how to get it installed appropriately okay so what we're going to do now is we are going to run this cell over here so this going to install protox set it up and it's going to install the object detection api so let's run this now again this is going to take a little while to run and ideally if you're running inside of a notebook on windows you might not get the status updates but the way to check if it's currently running is if you go into remember we're inside of our virtual environment yes again make sure that that's enabled and you're running it inside of that particular notebook inside of that environment so what we're going to do is actually show so i'll show you that so if we go into our tfod file now in the inside of libs site packages you'll actually see that it's installing packages so right now it's up to 10 30 or 11 30. so you can see it's installing a whole bunch of different libraries so what you should ideally see is that this is continuously installing stuff now it may say that it's successfully completed but ideally what you want to do is until you get an okay from that verification script it hasn't successfully installed right so this is going to be your golden bible so let's run the verification script so you can see here that it's saying that there's no module named tensorflow so what we're going to do is run this next line down here which actually installs tensorflow for us so what i did is i included some of the lines that i used to solve these errors so by running a verification script this is again a standard tensorflow object detection script that allows us to verify our installation so in this case it's telling us hey even though we've gone and run that script it doesn't look like we've got tensorflow installed so this next line down here is you might not need to run it it might say okay and i'll show you what it looks like when it's successful might work all perfectly if it doesn't then in this case we're going to install tensorflow so let's do that and ideally you should start to see tensorflow installing on the right hand side cool so that should be tensorflow installed even though so right now keep in mind so sometimes you can run into errors with this so even though we ran tensorflow installed it doesn't look like it's in sort tense flow so i can't actually see the tensorflow file or library there so tensorboard tensorboard plug-in tf models official tf slim so if we go and run this again it's still gonna tell us no module installed or no module name tensorflow so run it again and again it should ideally install the second time around don't ask me why it just seems like sometimes it throws up doesn't seem to want to run so now it's installing a bunch of additional packages you can see tensorflow there so ideally this is looking a little bit better this time so let's wait and see so again that verification script is your golden bible right so like if you get the okay sign then you should be good to go so in this case looks like it's still installing it's completed over here let's run our verification script again that's looking a lot better this time so it didn't run really fast so ideally it'll take a little bit of time alright so we've now got an error called no module named matplotlib so this one it looks like again i've got the script to run that now again this no module found error is pretty common so this just means that you need to install a package now i've got this again inside of the error guide so if you get no module named whatever you just need to go in ahead and install it sometimes you need to uninstall it and then reinstall it if you need to uninstall it's just pip uninstall and then whatever library you need to uninstall and then dash yes and then you install whichever one you want so in this case here i'm going to run this next cell which does matplotlib and protobuf for us okay that's looking good so you can see again inside of our virtual environment this is the advantage of having the virtual environment you can actually see visually what libraries are installed so that's looks like it's gone and done that let's go and run this verification script again all right so no module named pil so let's go ahead and install this so this one is one that i didn't have pre-configured right so we can do that so pip install pi i believe pil is pillow right so um so again if you get these types errors you can just google them so no module named pill just means you need to install pillows and more often that they're not people are going to tell you to install pillows pip and saw pillow right so let's go ahead and run that cool that looks like it's installed let's run that now again checking our verification script what are we getting now no module named yaml so this looks like it might not have let's check out so let's run pip list now so have we actually installed tensorflow but 2.4.1 that's looking good okay all right so let's install yaml now and module named yaml so let's see how we install that to install yaml we need to run i believe it's pi yaml yep so it looks like so python piping saw pi ammo so let's do that okay so that looks like it's installed you can see payamal there so let's again run our verification script i can hear my gpu spinning up right now so maybe that looks like a good sign so this is looking good even though it's in red all right beautiful so we're good now if you get to the point where you run this successfully and right down the bottom even though it's in red i know it doesn't look that great it should say okay so if it says okay then that means you have successfully installed the tensorflow object detection library right so basically this means that it's going to go and run all of its verification tests to make sure that it's actually working and if it's run successfully you should get okay down the bottom let me zoom in on that because that if you get to that point that's a huge win right now you saw how i was solving those arrows so if you've got no module x just go and install that module if you've got no module y go and install that module sometimes you need to uninstall stuff and reinstall stuff but again if you get stuck hit me up in the comments below join the discord server but ideally that's going to be the same pattern right so if it says no module x you just go and install module x and because you're inside of your virtual environment it's going to be installing in the correct place now remember our virtual environment is up here so we're using tfod need to make sure you're in there so what did we have to go and do to get ours to run so we installed tensorflow again we installed matplotlib and protobuf and we installed pillow so remember we installed pip install pillow and we also installed pymel now again this might be different depending on your particular environment you might need to install more stuff less stuff it might just run perfectly fine but i wanted to show you a real-life installation because this is me installing from scratch you can start to see what it actually takes to install it okay um so it's not all sunshine and rainbows that's i guess the the core thing that i wanted to point out so i'm not just doing it in smoke and mirrors you can see that you might run into stuff but more often than not it's just a pip install that you need to do now if we run import object detection you might still see that you get a error here where it says no module named object detection what you can do to solve this is just restart your kernel so first up let's clear all of our output from this so it's a little bit clearer if i run import object detection again it's still saying no module so what i normally do here is i'll restart my notebook so if i hit let me show you so if we go to up to kernel restart and then i'll say restart you can see it said kernel ready then what i'll do is i'll go up to here zoom out re-import these you don't need to do all of this again so we just need to run wget and then if we go down here you can see that it's now installed successfully sometimes you just need to restart your kernel to be able to import into your notebook and in that case that's our object detection api now installed and available and ready for use so we went and installed a bunch of stuff there so if we run pip list you can see that we have a heap of additional libraries now installed now remember when i sort of said that if you install tensorflow and you want to use your gpu you've got to install some additional stuff well in this case here we've got tensorflow 2.4.1 so this is the equivalent of 2.4 like falls into the 2.4.0 there is tensorflow 2.5.0 available at the moment so you just need to make sure that you find the appropriate cuda and cud and then version for that now to do that what i normally do is i'll go into google and i'll just search for cuda well let's take a look at what version we need first so again we go to tenseflow.org forward slash install forward slash source windows because i'm on a windows machine we could just as easily go back to the linux and mac os page and do that in this case we're on windows so if i scroll down because i'm using 2.4.0 i need to have kuden 8 org and cuda 11 installed now in this case i've already got it installed but i want to show you where to get it from so if we go to cuda 11 if you just go to google and type in cuda 11 you can see that cuda toolkit 11.0 is available so if i select that and then i choose my operating system so in this case i'm on windows i can choose version at what version of windows and once i'm on windows 10 not server and then you can choose which type of installer you want so you can either choose a network install or a local install which downloads everything so if we choose network install and then hit download i've already got this downloaded so we don't need to wait there so once you've got it downloaded you should get a file called this so you'll get a file let me make that a little bit bigger so it's called cuda underscore 11.02 and then in my case it's windows so if you actually double click that it'll walk you through the package installation and in this case it's going to install into my cuda folder here so if i hit ok and then basically what i do is i just click click and go through that cuda installation there's nothing too crazy there right so you just hit next next next no issues so it's going to install cuda into your machine so rather than sitting through this i'm actually going to show you the next step so once you've gone through and installed cuda so again i've got the full insole video where i show up from scratch but basically you're gonna double click cuda install it walk through the installation then what you need to do and this is the bit where it is a little bit more tricky so once you've got cuda you then need to get the matching insulation of c-u-d-n-n which in this case here is going to be cu dnn 8.0 so again to find that and i'm showing you how to find it on google rather than just giving you the direct links because these are going to consistently change so if we type in cud and then and remember we needed what version do we need we need eight so i'm going to type in cud and then eight because again these consistently change and in this case these are going to be available from the nvidia site just as we got from cuda as well so we're going to go to nvidia cu dnn and we're going to look there should be kudern8 let's see what's available there right so down here we've got a link to gtc now we won't see download cdn now in order to do this or not in order to download cdn 8 you're going to need a nvidia developer account but it's free you just need to sign up so i'm going to select download cu dnn and then we need to log in so this is why you need the developer program membership so in this case i can log in and then log in so do the captcha so what are doing trucks and then it should allow you to actually go ahead and download download cu dnn now you do need both but only if you're training on a gpu if you don't have a gpu on your local machine you don't need to go ahead and do this so in order to download it you just need to hit i agree to the terms and then you're looking for the version that you need so in this case there's two versions available here so there's cu dnn v8 for cuda 11 and there's cu dnn v8.1 for cuda 10.2 so remember that we had for our particular version of tensorflow which is 2.4.x we need cudn 8 for cuda 11. so in this case we are going to get cu dnn which i believe that's v8 1.1 so we actually need to go find a different version right so it's cud and then eight dot zero and let's find it so in this case we're going to go to the archive all right so this is looking better so you can see here that we've got a bunch of eight versions so rather than having 8.1 again these are going to consistently come out with new versions so we need this one over here so we've got cuda 11 which is over there and we've got cu dnn v8.05 so we can download this just find the version for your machine so in this case we've got cdnn library for windows x86 so i can download that and that's going to download so in this case again i've already got this installed so we can cancel that so it's going to look a little something like this so it looks like i've got 11.2 here but what do we need c u d and then okay i guess this one must have worked all right so it's going to look a little bit like this so we've got c u d and then and it's a zipped file so what you want to do is extract it and in this case i'm just going to extract it here and then what we're going to do is we're going to copy the cu d and then files into their respective folders inside of our cuda installation location that was probably a bit of a mouthful but basically you're just copying and pasting stuff okay so we've got our stuff extracted so which one is it it's this one here so if i double click into it now you can see inside of cuda i'm going to have a bin folder which has a whole bunch of cud and then stuff an includes folder which again has a bunch of cu dnn stuff and a lib folder which again has a bunch of cu d and n stuff what we need to do is open this or is copy those folders into our cuda folder so if i actually show you what mine looks like so if we go and find nvidia and i believe it's inside of this folder called nvidia gpu computing toolkit so i've got cuda i've got a bunch of versions pre-installed inside of my bin folder i just copy these bin files so i'd copy all of those over now i've already got them done so i'm good to go but you can see that i've already got these available here so i've copied these cdn and files into the bin file inside of my kudo directory again if i go back i'll have to do it for my include file so again include do the same thing so i've got they should be in here so you can see that i've got all the cdn files i've just copied those into here and again likewise you've got to do it for the lib x86 lib x86 and again we're just doing it back in here so you can see i've got c all of these cu dnn files again i've just copied these over into the respective cuda folder so once that's done on your windows machine you just need to do one last thing and make sure that it's added to your path so in this case if i open up my properties go to advanced system settings environment variables and then go to my path you can see i've got my bin folder or the path to my bin folder the path to my lib nvp folder and the path to my cup ti folder so there should be three so this one this one and this one i don't know let's see if we can make that a bit bigger there you go so these three file paths are the ones that you're going to need added it to your path on a windows machine in order to enable this to work so again this is just pointing to this particular folder over here so if i go in let's make this a little bit easier to see so we are pointing to our so nvidia gpu and then see some pointing of v11 and then bin and then you can see i'm pointing to so my bin folder which is this one my vvp folder which is this one and then my cup ti folder so if i go back in so it's under extras cup ti and then lib xx86 or that doesn't actually look like that one's been defined right that's a bit better lib64 cool so that should be all good but again i'll show you how to check whether or not that's working when we actually go to train so you'll be able to actually see whether or not it's picking up the appropriate version of cuda so that's sort of just an extra in-source step that you need to go through if you want to train on your gpu if you don't have a gpu on your machine should be good to go don't need to worry about that but that is our installation done now more often than not that can be a bit of a nightmare process but you can start to see that there is a flow to it so go through run through your install test your verification script and then just keep running through to make sure that you effectively get that okay symbol at the end when you run your verification script so again we can actually run this again and ideally what we should get is that okay symbol or that okay status right at the end even though it's in bright red as long as you get the okay at the bottom you should be good to go so let's run this and keep in mind we restarted our kernel so even though we restarted we should still see that okay pop up and again looks like it's okay so we're inside of our virtual environment we've got our okay status popping up we're good to go all right so we are now up to this cell down here so this cell over here is actually going to download our model from the tensorflow model zoo so if we cast our minds back our tensorflow model zoo was this one over here so remember we've got a whole heap of models that we can already use for our training now the reason that we do this is because one we get a state-of-the-art architecture that we can use and two we're going to be leveraging a technique called transfer learning now out of the box we can actually use these models to do standard object detection so this is detecting just generic objects so a phone a bottle a towel so on and so forth what we're going to do is we're going to leverage what that machine learning model or what that deep learning model already knows to be able to transfer that knowledge over and detect custom objects so in this case we're going to be detecting our different hand poses right so the reason that we're using that architecture is mainly for those two reasons now what we're going to do is we're going to run this cell and again it's checking which operating system we're in it's then using wget to get it from our pre-trained model url which is what we defined up here and i'll show you how to do a more sophisticated or a different model once we get to our performance tuning section so now that that is ideally what should happen is that this particular cell over here is going to clone down that model from tensorflow model zoo and it's going to paste it or put it inside of a specific folder so if you go into tensorflow workspace pre-trained models you can see that it's actually gone and cloned down that model into that specific folder and this is why i defined the name of that folder earlier on because we are actually going to be leveraging all of the stuff that it's in here so inside of that folder let's go back you're going to have the clone of repository so in this case it's just copied down the compressed file it's then untied that file so that is let me just bring that over here it's then untied it which is over here and it's also moved that particular file into this workspace so now you should ideally have all of this so you've got a checkpoint folder which contains all of the different checkpoints for your file think of this as the different saved components to your model or the pre-trained model you've got a saved model.protobar file and a variables file so again this is everything that you get when you actually freeze your model and again i'm going to show you how to do this later on for your own custom model as well but just think of this as your pre-trained model that you're going to be able to leverage for your custom object detection so that's good now now the next thing that we want to do is create something called a label map so a label map effectively represents a map of all your different labels so remember we had thumbs up thumbs down thank you and live long now i've already got these written here right so i've got thumbs up thumbs down thank you and live long so let's go ahead and run this this should run successfully so no errors there so if we go into our workspace so this is going to be written into our files label map file which was defined over here so it's going to be inside of annotation path and then the label map name and our annotation path is annotations and then a label map is named label map and or dot pb text so if we open this up you can see we've got a item for each one of our objects so thumbs up thumbs down thank you live long now remember when i said the case was really really important this is why so when we go the next step that we're going to normally run through is actually creating our tf records so what this actually does is it creates a file format which is called a tf record which is available to be used to actually go ahead and train your object detection model so if we go into these next two lines so you can skip this one over here so this is only used if you're training on collab so if we go and run this tf record script which is there this is going to run a or clone a file from my github which is called generate tf record so let me show you that so it's going to clone this file and this is the file that generates your tf records again you don't need to go into a great amount of detail for what that actually does just know that it's converting your annotations in your images into a file format that it can use so if i run this cell it's going to clone it down and then you should ideally have a file so if you go into workspace and then where is this this is in tensorflow sorry scripts should get that file so generate tf record or generate underscore tier of record so that's now available there now what we're going to do is run this file here so i'm going to show you once running without errors and then i'm going to screw up a label map to show you why the case is so important so if i run this oh it looks like we've got another module error so in this case we've got a module error so what do we do when we go to module error just going ahead and install that so it looks like it's pi tz so let's check how to install that so it looks like we just need to pip install pi tz so let's go ahead and do that so again if you get any of those no module errors you just need to install it see if that runs now okay that's looking like it's running successfully so again to solve that error it was a no module error so to solve that we just install the module and then when we run our tf record script what it's actually going to do is go on ahead and create so if we go into our tensorflow workspace annotations file we've now got a train and a test record so these are what are going to be used to actually train our model now if i screw up our label map right so say for example i take off the capital u from our thumbs up label this is going to screw up that generation so if i go and run our generate script again watch this so you can see that we've got this key error so here are thumbs up and then kiara thumbs up so this is because we have included the incorrect label so it is case sensitive so you need to make sure that the case is correct so if i go and fix that up now and change our label name to thumbs up and rerun that again so this should now run successfully and again no issues cool now the next thing that we're going to do is pretty straightforward so now what we're going to do is just go and copy our model config so our model config is so remember there were a bunch of different types of let's just close this and this and this so there are a bunch of different types of models inside of our tensorflow model zoo now the pipeline config is what controls what your model actually looks like so if i go into let's start from the start so tensorflow workspace pre-trained models ssd mobile net underscore v2 underscore fp and light underscore 320 by 320 underscore coco 17 underscore tpu-8 this pipeline file over here is what defines the architecture of your model and what it actually looks like now what we're going to do is we're going to customize this for our specific model but this sort of gives you a baseline so in here you're going to have the number of classes you're going to have that image resizer which i was talking about so remember it resizes it 320x320 your feature extractor one that i particularly like is post processing no data augmentation so you can see here that it actually does your data augmentation for you this is important because i'll show you in some of the other models later on you'll get a slightly different data augmentation architecture this is just important to note that it gives you some pre-built data augmentation inside of your model if you scroll down you can see that you've got all of these path to be configured components so we actually need to update these with our components for our model so we need to update fine tune checkpoint we need to update our label map path our input path our label map path and our input path our label map path is our label map dot pb text so that's what we created and our input path in this case our train input reader or our train input path is going to be our train dot record file which is let's go back which is this file here and a test input file which is what's being referred to or our eval input file which is what's being referred to down here is going to be our test file but again you don't need to worry about this i just wanted to give you a little bit of additional background our code is going to do that for us so if we go and step through so first up what we're going to do is copy that pipeline file from our pre-trained models folder so we'll copy it from here into our my ssdmobnet folder so if we run that you can see that that's now being copied and then the next sets of code are actually just going to do all of that configuration for it so if we actually just step through all of that right so it's opened up our config file it's gone and set all of our different file paths and it's gone and saved it so if i open up our customize one now what you'll see is that it's going to change the number of classes so this represents the number of different types of objects we're going to use if we scroll on down let's close this you can see that we've actually also gone and set up our different paths so we've gone and set up our fine tune checkpoint our label map path our input path so on and so forth so set that up set that up set that up so this is now all configured and pretty much good to go for us to kick off training so if we close this now what we can actually do is actually start training our model so we're now finally up to the good bit so before we actually go and train our model let's take a look at what we've done because we have done quite a fair bit so we set up all of our paths we downloaded and installed the tensorflow object detection api so remember we had to go through a number of iterations but as long as your verification script says okay we are good to go we then went and downloaded our pre-trained model from the tensorflow model zoo we created our label map and remember that is inside of our workspace annotations and then label map dot pb text so if i open that up it's really just this we then created our tf records which again are our test and our train files we copied our config to our training folder and then updated that so again that's all just code driven you don't need to do anything there now we're up to the good bit training our model so in this case what we're going to do is hit shift enter shift enter and then print out our command now i'm going to run this outside of a notebook so that's going to allow us to see our progress as our model is actually training now if you run it inside of the notebook the only disadvantage is that particularly on a windows machine you're not going to see the status update in real time so i'm going to copy this over here and then i'm going to open up a new command prompt then navigate to where i've got my root tensorflow object detection course folder so let's do that and i'm going to go into tfod course all right cool so we want to be in this top level folder so you should ideally see the tensorflow folder so let's clear that now what we need to do again is remember we need to activate our environment so to do that we can do backward slash tfod backward slash scripts backward slash activate now if we type that so again we're still inside of our tensorflow folder now what we can do is we can actually copy that command so let me just explain that command a little bit so the first thing is actually running our model training script so what it's doing is it's using the model underscore main underscore tf2 dot pyscript now again i'm going to this into a little bit more detail than i usually would because i really want to explain to you how this actually works so what it's actually doing is it's going into tensorflow models research object detection and then it's running this script down here model main underscore tf2 probably a little bit small to see so model main tf2 down there that was probably obscured by me so it's running that over there so model main underscore tf2 so that is this component here so it's running that then we're passing through a number of arguments so first up we're passing through our model directory so this is where our pipeline.config file is remember our pipeline config had all of our custom paths had a number of our labels and had all of our different details of the specific object that we're going to use then we are passing through our actual pipeline config and we're also passing through the number of training steps that we want to go ahead and run through so in this case we should be good to go now so we're going to copy this command and we are going to paste it into that new command prompt so let's clear that and keep in mind we've activated our virtual environment so your virtual environment has to be activated otherwise you're not going to be able to leverage all of that tensorflow object detection stuff that you've gone ahead and insert so if we paste that command in there now oh the last the last parameter is the number of training steps so you can increase this or decrease this ideally i normally set it to 2000 for my first prototyping step then i'll go and clear out all my training and then i'll set it to 10 000 when i come to do my actual training run so let's go ahead and run that and ideally so you can see that it's successfully opened and looks like we've got an error there okay this is good cool so we've got this area here so value error numpy and d array size change may indicate binary incompatibility expected 88 from c header got 80 from pi objects if we copy this and again go into our error guide and search for that you can see that that particular error is solved by reinstalling pi coco tools so let's go ahead and do this so we can pip and install pi coco tool so let's copy that cool so that's it uninstalled and then we can reinstall it okay so in spite of these that looks fine so it says it's successfully installed by coco tools again so let's go ahead and paste in our command again so if we go back to our notebook we can copy this no module named tensorflow add-on so again this is a no module error so we should be able to pip install tensorflow add-ons now let's try it running a command no module named gin all right so this these no module errors are specifically attached to the tensorflow version so let's just type in pip list first up so even though we've got tensorflow 2.4.1 tensorflow estimator so let's run let's install gin so pip install gin try our training command again so let's search for that one looks like i've encountered that one again and it looks like we need to downgrade so let's do that so let's pip install gen config 0.1.1 so let's try that let's try this okay that's looking good so it looks like we sorted through those errors and again no more dual errors are normally just solved by installing the right library if you can't find the right library just take a look google around see what people have done and more often than not these are common errors so in this case we just needed to install or pip install gin config but now you can see that this is actually running or well i know from experience that this is actually running now so what you're going to see is first up if you're running on a gpu that your gpu memory is taken up so you can see that let me bring this over to this side so you can see that i'm now using the full eight gigs on my gpu and ideally in a couple of seconds you should start to see our loss metrics showing up now one of the things i actually just noticed is that we didn't actually have tensorflow gpu installed so it was taking a long time for that training to kick off which was a little bit weird to be honest so normally it should take no more than 15 10 to 15 seconds to actually kick off so what we're going to do is we're going to so i actually just installed tensorflow and then what we're going to do is make sure that we've got the right versions of tensorflow and tensorflow gp installed so we'll install pip install tensorflow and we add 2.4.1 and we're going to install tensorflow gpu as well because you need tensorflow gpu to leverage the gpu so run tensorflow dash gpu equals 2.4.1 so even though our verification script is fine we do still need tensorflow gpu so before it was fine to run on cpu to leverage the gpu we need tensorflow dash gpu so let's go ahead and install that and give that a couple of moments to install and ideally we should see that we've got tensorflow gpu installed so if you kick off your training and you do have a gpu you've set up cuda and cu dnn the next thing to check if it's not kicking off on your gpu or not starting the train relatively fast is whether or not you've got tensorflow gpu installed so then we're going to copy our command and now you should see that our training kicks off a lot faster let's clear this paste that in and let's see how long this takes now so if it takes a long time double check that as well so whether or not you've got tensorflow gpu installed so these are all pretty normal outputs just looking positive so when it sort of gets to this line and again it might be a little bit different depending on what you're training on so use fn underscore output underscore signature instead so normally that means that it sort of got to the next step so ideally what you should see is the loss metrics appear after this so let's wait and see sort of stalls we'll restart it no there you go all right cool so that looks like it's kicked it off so you can see that now we're getting these metrics here so basically after every 100 steps this is going to be printed out so you'll get step 100 per time so you'll get the time per step so in this case it's 0.82 s or seconds we've also got the loss so in this case our loss is 0.693 now if you cast your minds back to our evaluation set what we're saying is that you want your loss to be as low as possible your mean precision and your or your mean average precision as well as your recall to be as high as possible so in this case we're going to let this run so ideally you're going to see this output after every 100 steps so in this case we've got our first 100 steps and if we also take a look you can see that we've got our checkpoints starting to generate here as well so we're looking good for now we're going to let that run and we'll see how we go so ideally remember we we kicked off our training for 2000 steps in our training command which is right up here so we had 2000 training steps ideally after 2000 training steps we should have our fully trained model so let's let that run and then we'll come back and take a look and do some detections so that is our model fully trained so you can see that we got up to step 2000 and our final loss let me bring this over to this side and our final loss was 0.188 now if you also take a look inside of our my ssd mob net you can also see that we've got all of our checkpoints generated so we've now effectively gone through and trained our model now what we can do is actually go and evaluate it so remember how sort of saying you've got your mean precision metric so you've got precision and recall we can actually go and calculate these using the tensorflow object detection library so what we can do is run our evaluation script so if we go and run our evaluation script model so we've gone and done step six and again our model is successfully trained so it didn't take too long what did it take um so 0.153 seconds per step so again it got pretty fast on the gpu now what we can go on ahead and do is evaluate it so if we go to step 7 hit shift enter print out that command now we can actually run this one uh we'll run it using our command prompt so we'll copy that so if we go and paste that command again so we're making sure that we're inside of our virtual environment we can go and paste this hit enter and this is actually going to go on ahead and calculate our performance metrics for the model that we've currently got in that folder so again it's going to go through all of these different lines write out a bunch of stuff and ideally what you should see towards the end of it is that you get this little table that gives you all of your performance metrics and there you go so you can see here that we're getting our average precision our average recall we should get some loss metrics as well so they're down here so you can see that we've got a bunch of loss metrics but i'd pay most attention to these ones up here so you can see here that our average precision at 0.5 to 0.95 iu so pay attention to this is 0.688 so again it's not the highest precision but it's not too bad our recall is about 70 so again not too bad given the fact that we only gave it what like five or six images per class so again not too bad right now what you can also do is you can also open up these same detection metrics as well as the loss metrics using tensorboard so if you haven't worked with tensorboard before basically it's an interactive monitoring tool that comes well that is open source through tensorflow so what we can do in order to use that so we can just step out of this by hitting control c is inside of our my ssd mob net folder you've got two folders which i haven't really talked about yet so you've got this train folder that's generated when you kick off your model training and you've also got this eval folder which is generated once you go and run this evaluation script so effectively uh script the script under 7 evaluate the model so these two folders are going to be generated after training and after evaluation now what you can actually do is you can actually step into these folders and spin up tensorboard so let's go ahead and do this so right now we are in our root folder so we can go into tensorflow and then we can go into workspace and then i believe cd models and then we want to go into our my ssd mob net folder this is probably going to be right down in the corner cd my st mob net and now we're going to have this train folder and this eval folder so if we go into let's do train first because train is going to if you open up tensorboard inside of the train folder you're going to get lost metrics if you open it up inside of the eval folder you're actually going to get your evaluation metrics so let's do a train folder cd train and then the command that you want to write in order to open up tensboard is tensorboard it's right down the bottom let's clear this first tensorboard dash log dear equals dot so this is going to open up tensorboard in the current folder so if we do that and i've typed in tensorboard wrong this should open up tensorboard and tensorboards installed when you install tensorflow then what you can do is you can go to this url here so localhost 6006 and this is going to open up tensorboard so through here you can see all of your loss metrics so you can see that over time the loss is reduced and remember we want loss to be lower because loss a reduced loss indicates it's performing better on our training data so in this case you can see our classification loss and localization loss and our normalized loss so you sort of want these to be in an inverse hockey stick trend so starting up high and then dropping low now this is all well and good but you're probably more interested on the evaluation metric so again you can sort of navigate through this you can also see the images that we've gone and use so our train images and these are our different samples you can adjust your brightness and this is just viewing them at the moment as well right cool but what we actually want to do is take a look at our evaluation metric so if we step out of this and if we rather than being inside of our train folder if we go back and go into our eval folder so remember that is this folder over here we can run tensorboard again and this is going to give us a slightly different view so again we're just going to run tensorboard dash dash log dr equals dot so remember this is important because it represents the current folder and again so it's going to open up tensorboard it'll start up it's running at the same url so http localhost 6006. so we can actually just refresh this and you can see that this is giving us our evaluation metrics now so you can see that our mean average precision is 6.876 down the bottom there so it's a little bit hard to see and now what is this metric over here this is again our precision we can close this we've also got our recall which is 70 again these are the same metrics that you saw when we ran our evaluation script now what we can also do is we can run this after every training run so we'll actually be able to see what our performance looks like over time as well so again you've got the ability to do that you can also view your loss as well here so this gives you your testing loss on these particular sets of data you can also view your images again this sort of gives you an idea of our real-time performance again you can see that this is what it's actually going to look like when it comes to actually go and detect our models and it looks like it's detecting pretty accurately so that is thumbs down that is thank you that is live long this one looks like it's gone and had a little bit of an issue with thank you there but thumbs up looks like it's performing all right and eve side by side alright looks like i got a little crazy there as well so all right so that looks all well and good so that gives you an idea of how to train and evaluate your model as well now on that note we are now up to detecting objects so first up what we're going to do is we're going to detect our object or detect an object with an image so this particularly useful if you're detecting from a cloud provider that doesn't actually have access to a sensor like a webcam or in this case a microscope so we're going to detect using an image but then we're also going to detect in real time so we'll be able to throw up different symbols so live long thumbs up so on and so forth and we'll actually be able to see those detections in real time once that's done the last thing that we're going to do is freeze our graph and then export so let's go ahead and perform our detections and then we'll run through our last step which is going ahead and freezing and exporting our graph and then we've obviously got our projects and performance tuning as well so let's step back so again using our same notebook what we can do is we'll just kill off our tensorboard instance so this doesn't use up our gpu so what we can now do is we can actually load our trained model from our checkpoint so we've now gone through let's quickly recap so we've set up our paths under step zero so we downloaded our pre-trained model and install tensorflow object detection we created our label map and remember this is just a set of labels that represent our different objects that we want to detect created our tf records copied our config and set that up we then went and trained our model using the command prompt and evaluated as well the next thing that we're going to do is load our model so now if you go into your pre-trained model folder or your trained model folder which is the my ssd mob net folder so remember if we go into tfod tensorflow workspace models my ssd mob net this is your custom model right so everything in here represents your trained custom model so if you wanted to give this to someone else you'd be giving them this folder because it contains your pipeline and all of your checkpoints now the latest checkpoint that we can see here is checkpoint three so this is our most trained model if you train for longer you'd ideally get checkpoints with greater numbers as well so what we're going to do is we're going to run that particular cell so the import and then we're going to change this line here which you can see it says checkpoint 5 to checkpoint 3 because that is our latest checkpoint so we can run this and set it up that's going to import our latest checkpoint and then what we can do is actually detect from an image so let's go ahead and import that dependency under step 9 set up our category index and then we'll grab an image so let's go into our workspace now images and we'll grab a test image so let's grab this one over here so i'm just going to grab the name of that image and i'm going to replace it over here right so basically inside of this line here so under step 9 we've got the third cell which is image path equals os.path.join we're basically just resetting the path to that image so if i type out image path now right it's just the path to that particular test image that we had there so if i go and take a look so basically the full file path is tensorflow object detection tfod course tensorflow workspace images test tensorflow workspace images test and then that particular image so we're good to go so now what we can do is run this cell and this is actually going to run object detection on that image and print out our results which we should see here in a second so let's give that a sec and there you go so you can see right down there it's gone and accurately detected that that particular symbol is live long with a 98 confidence metric so again we could do this on a different image if we wanted to so let's grab another one let's grab thank you so if we grab that label we'll grab that image name and replace this run this again we can delete that now so again that's detected thank you successfully let's try another one thumbs up so again all i'm doing is i'm changing the part to that image if you had your images somewhere else you could do that as well since it accurately detected thumbs up 86 percent confidence so not the greatest on that one this thumbs up the thumbs up was the one that we took new images of as well wasn't it so again we could performance tune there and let's do thumbs down okay so not too bad so 100 on thumbs down so that gives you an idea of how to do it with images now basically what we're doing here is we're loading the image into python we're creating it or converting it to a tensor and running our detection so this line over here is actually using our detection function which we imported and then it's going ahead making those detections and then visualizing it so again if you'd like to see me explain this in more detail or maybe even code it up from scratch by all means do leave a mention in the comments below i'd love to hear your feedback okay so those are our detections so that's all great now the next thing that we actually want to do is make some real-time detections from our webcam so let's go on ahead and run this cell under step 10. now i'm going to not run this cell which is pip uninstall opencv dash python-headless dash y we'll come back to that in a second and that's normally a resolution to a pretty common error but let's go ahead and run our real-time capture and see how we go so it looks like it's running okay so it looks like it's got our feed so it's got live long thumbs up thumbs down that's actually really really good considering we only trained for 2000 steps with not a whole heap of images that's actually performing really well so live long thumbs up thumbs down thank you let's see what it looks like with the green screen down because remember we didn't train with this so live long still performing well thumbs up thumbs down thank you what if we did a joint so thumbs up so thumbs up is not performing the best but if we did live long you can see there that because we maybe haven't done too many images trained on so let's do thumbs up so all right so again so you can see that the performance isn't so good when we try to change it or use different hands so okay that maybe that's performing not too bad so thumbs up isn't performing well with this particular hand so again we might add some additional images we might add images of each hand but again that's performing reasonably well right and like this is dual hand right now so it's it's not too bad thank you it's performing okay how about live long so live long on this hand yeah pretty cool but that sort of gives you an idea of what's possible right so like really really quickly we've built an object detection model that's able to detect a bunch of stuff so thumbs up is perfectly fine or is it the thumbs down okay it's not so bad maybe it needs to be a little bit closer to the camera but again these are the things that you could additionally train for so we could add more images of closer to the camera we could add more images of this hand doing live long sort of gives you an idea of how you can performance tune and update your model to be able to perform better now if you want to quit out of this all you need to do is hit q and that's going to close it off but again that sort of gives you an idea of what's possible with real-time object detection alrighty so we've now gone and also made detections using an image as well as using a webcam now you can also do this with an existing video feed or an existing video if you'd like to see me do a video on that by all means mention it in the comments below i'd love to hear your feedback all right so this brings us to freezing and conversion so part five freezing and conversion so now what we're going to do is freeze our graph and again as i was sort of saying freezing your graph is the equivalent of saving a slice of your model or effectively saving it down so that you can use it next time so we're going to freeze our graph we're also going to export it to tensorflow.js so if we wanted to use it later and we're also going to export it to tier flight and we'll eventually use this in our raspberry pi project we'll use our tfjs export for our web app now these are all reasonably straightforward so i've just got step 10 11 and 12. all you need to do is run through these and it's going to freeze your graph so let's go ahead and run this and i'm not going to copy this command outside of the notebook it's probably just easier to run it here and once you've frozen your graph this is going to appear let's just make our mouse work this is going to appear inside of our export folder so ideally what you should get is inside of if we go back to our root so tensorflow workspace models my ssd mob net inside of our export folder we're going to have our frozen graph made available so let's give that a second so again this might take a little bit of time to run and so that looks like it's now complete so let's just check it doesn't look like any errors there and you can see that you've now got your checkpoint your saved model assets variables and your savedmodel.protobuff file now if you remember correctly remember how i said that we were going to create a similar output to what we had for our pre-trained models you can see that we're now doing the exact same thing so we've now created our own slice of our own custom model which looks almost identical to what we had from our pre-trained model but now again this is based on our custom model so again we've got all of that available inside of our export folders we've got our checkpoint we've got our save model and we've got our pipeline config and this will mimic exactly what we had for our custom one so if i scroll down again so this is currently still pointing at our own file folders you can see that you've got all of the stuff that you custom built for your own model right so this is now available to be used elsewhere so you could actually go and convert this which is exactly what we're going to do but again you could pass this off to someone they could pick this up and use it as well now the next conversion that we're going to do is we're going to export this to tfjs so if we go and run all the steps beneath step 11. so first up it's going to install tensorflow.js converter or the tensorflow.js converter and then again it's going to create this command print it out and then it's going to run it so ideally this should create a model.json file and a number of bin folders and i'm going to show you how to use this inside of our projects but for now we're just going to convert it and we'll be able to leverage it later on so give that a couple of minutes and you should effectively have your converted tensorflow.js model and there you go so you can see that we've now got our bin files so we've got our different shards and we've got this model.json file which is huge so if you actually scroll through this this represents our entire deep learning model in json format now when we load this into our react app or into our tensorflow.js app which is in javascript but again i've written all that code for you you just need to paste it in there this effectively represents the full architecture and the weights and everything which is actually pretty cool now the last conversion that we want to do is conversion to tf light so again same sort of deal except this time rather than our exports being export or tfjs export we're going to export into tf lite export so again let's just go on ahead and run these commands so the tflight conversion takes two parts so first up we convert it to what was it again so we export it to a tf like graph and so we do it using this command and then we go and actually run it using the tf lite converter so this is again a slightly separate app but again you don't need to worry too much about this it's actually just going to convert it and output it to exactly the output that you need to be able to go and leverage that so let's let that run and that brings us to the end of this end-to-end model so you can see that we've now gone and converted and it's still running the second part so let's give that a sec perfect and that looks like it's gone and effectively finished so now if you go into this tier flight export folder double click saved model you've now got this folder called detect.tflight and that is your tflight export so that about brings us to the end of our conversion so we've now gone and frozen our graph exported it to tfjs and exported it to tf lite and it also brings us to sort of the end of our full run-through of actually working with tensorflow object detection so all up what we did is we set up all of our stuff and installed all of the tensorflow object detection dependencies we set up our virtual environment which is called tfod we imported our dependencies defined our new images to collect set up our folders captured our images and labeled them we then went and did all of our installs we trained our model and then we evaluated it detected from an image which you can see here we also detected from our webcam which you saw using the real-time webcam feed we then went and froze our graph and then converted it to tfjs and converted it to tf lite so that is a full end to end walkthrough of how to work with the tensorflow object detection api now what we're going to do is get started with our performance tuning so say your model doesn't actually perform that well what do you go about doing well this is really where performance tuning comes in so normally there's three key things that i'll do to improve the performance of an existing model so the first thing is adding more images of a low performing class to your training data set so you saw that when we're doing thumbs up thumbs down live long and thank you we're sometimes getting not the greatest detections now one of the easiest ways we can fix this is to take the green screen down and take images of both of our different positions using different hands so this is going to give a model a better chance to generalize so when people talk about model generalizing really it's being able to perform well in real world situations the second thing that we can do is train our model for longer so you saw that when we generated our training command we had 2 000 steps now what we can do is we can actually clear out our training folder and bump that up a little bit so say for example we train for 3000 steps this gives our model a better chance at reducing its loss generalizing and performing better overall now the last thing that we can do is also change our model architecture so we talked a little bit about the tensorflow model zoo so you saw that we're using the 320 by 320 model so if we actually take a look at that we were using this model over here so ssd mobile net v2 fpm light 320 by 320 which had a speed of 22 milliseconds per frame and an average accuracy of 22.2 but we could also upgrade this and use a slightly more advanced model now again we're going to have that performance trade-off so you might have a slower model but again it might make better detection so say for example we were to use this model here which in this case takes 39 milliseconds per frame but also has a better accuracy metric so we could do that as well so what we're going to do is we're going to add some of these techniques to our existing model so what we'll do is we'll add more images of our different classes so in this case we know that our thumbs up thumbs down detections weren't working so well so we'll add more images of those and then what we'll also do is we'll train our model for longer so we'll bump this up to 3000 and if we get time we'll change our architecture and i'll show you what that looks like so effectively that's really just creating an entire new model but the nice thing about the jupyter notebook that i've sort of set up is that you can plug in the new url and this will allow you to create an entirely new model and build it using a new architecture alrighty so first things first so adding more images of our low performing classes to our training set so we're just going to add more images of thumbs up thumbs down with our green screen down and we will see how our performance looks like so if we go back to our notebook so remember we've got our image collection notebook and our training and detection notebook let me just check we're zoomed in we're good so what we're going to do is rather than collect images of every single class we're just going to collect some more images of our thumbs up and thumbs down so we can get rid of thank you and live long so that way we don't have to label a ton of stuff right so these were the classes that didn't seem to perform that well particularly when we were doing it with different hands right so when we were switching it wasn't actually performing that great so what we'll do is we'll grab some more images of those and label them now we'll add uh let's add six so we've got an even number so we'll do three of each hand in each position okay so again we're going to import to this and again make sure you're inside of your virtual environment so you can see up here that we're inside of tensorflow object detection which is a virtual environment that we created so what we'll do is we'll import opencv or import our dependencies then we'll update our label map so before let's just have a look so before we had a thumbs up thumbs down thank you and live long we are going to just remove the stuff that we don't need so thank you and live long so we can remove those and we'll change the number of images that we want to collect so rather than collecting five we're going to collect six so three of each run that we can then take a look at our labels so now we're just collecting thumbs up and thumbs down and again so when we actually output these particular images they're going to be inside of our images part so if i go into tensorflow workspace images and collected images remember they're going to appear inside of these folders again so live long thank you thumbs up thumbs down now it's not going to add additional images to live long and thank you just thumbs up and thumbs down so let's go on ahead and do that so if we set up our folders we've already got these folders created so again if we step through this not going to break anything and then what we can do is run our image capture so now i changed this while i was doing some testing so we're going to set that back to zero and again so what we're going to be doing is we're going to be capturing images in our different positions so we'll step back and we'll take images um we might take the green screen down it might not we'll just evaluate and see how it goes so what we'll do now is we'll hit shift enter and ideally you should see the little python pop up towards the bottom so remember it sleeps for a little while before it opens up we're just going to have a thumbs up cool right now we'll do it with our other hand oh that's off the screen then we're going to do thumbs down so three on each again the screen again looks like we had some sucky images there so let's just go take a look at what we collected because i've got a feeling some of those were going to be off the screen so if we go to tense flow tensorflow workspace images and then collected images so remember we collected thumbs down and thumbs up so if we go to looks like there's some images there left from my testing so let me just delete those so these look okay so that's thumbs down thumbs down this one doesn't look so good too let me just make that a little bit bigger so you can see that we don't even have an object on the screen to detect so we can delete that one and delete that one this looks fine this looks fine again ideally we could probably get better detections and these are all of the same side right so it doesn't look like we've captured the other ones that was probably a little bit too slow so let's just uh see that one i'm happy with that one i'm happy with that one's okay let's delete that one and we're going to grab some three more thumbs down ideally with the our other hand okay so let's go to thumbs up again more images from testing okay and so again that's poor image there's no class there it's right it's a little bit dark that's right it's a little bit dark it's fine that's fine as well even though i look a little sleepy there and that's okay so one two all right so we're gonna add in a couple more images and then we'll just sort of evaluate what we actually want to keep so let's do a couple more so again to do it again all we just need to do is run this excel under four capture images and this is just going to capture our images again so if we run this again so we'll do some more thumbs up one switch now it's going to be thumbs down keep going off the screen come on nick okay let's have a look at what we've got now so all right so we've got some more thumbs up this is kind of good so you can see that this is a little bit dark so i mean that might be a good chance to give our model to general or might give our model some additional data to generalize so that's fine and that's okay that's fine that's fine that's fine it's looking better okay so i think we're good there so for thumbs up so it looks like we're gonna have what is this 11 new images and if we're going to thumbs down let's take a look at what we've got it's fine again we're going to get rid of this one because that's off the screen this one because that's off the screen okay cool so we've got a bunch of new images now what we're going to do is we're going to go on ahead and label these again so remember when we went and labeled all we need to do is go through our step 5 and image labeling and this is going to open up label image to be able to label images now because we've gone and installed all of this already we don't actually need to run every single cell so these ones up here so pip and saw pip list these couple these are just doing the installation so everything all of these cells which you can see here these are really just doing the insulation all we need to run is this one so this is actually going to open up label image so if you've already run through the installation you're good you don't need to go on ahead and install it so if we just run this cell is oh we need to define our label image path so let's run this cell so you can see here that it's saying the system cannot find the path specified and that's because we're asking it to open up label image path over there but we haven't defined it yet which is this what this cell is doing so if you run the label image path this should allow you to open it up so if we open this now how a label image platform should open so you can see that that's now open now what we're going to do is as per usual we're just going to go ahead and label our images so again w and this is thumbs down and remember it's case sensitive so thumbs down and we need to save go to the next one thumbs down save comes down save you can see that we've now got varying images right so we've got both hands covered so there's a better chance that the model will perform better a lot of the time this is trial and error so you've got to evaluate and see what's going to make your model perform better but normally this is the first step so i'll go on ahead and i'll add more images of the classes which i wasn't too happy with this one is a bit of a shocker image but we'll wait and see okay and we'll do the next one this thumbs down again cool then we'll open our next directory which is thumbs up and we'll go and do these ones in the same manner again we're just changing the label name paying attention as to whether or not it's case or paying attention when it is case sensitive again we can adjust these if we need to and saving we'll go to the next one save go to the next one save the next one and just keep going through until you've got all of your images labeled again this is something that you could start looking at uh auto labeling tools for obviously to improve at your efficiency also pay attention pay attention if you do start using those tools as to what the performance of those auto labeled images are sometimes even though you use an auto labeled auto labeling tool you still do need to go through and do a bit of a manual review but again something that could potentially improve how quickly you're able to label images so i'm just going to finish the rest of these up save that do the next one okay cool so that is now done now remember whenever we are labeling new images we need to copy them into the training and test folder so no real change there so i'm going to copy the majority of these so let's go that looks like we've got a good mix there so let's copy these and we're going to put them into the training folder and then we're going to go back to our collected images go to thumbs up do the same thing and let's just grab these and that looks okay and remember you need to grab the annotation as well as the image so if you don't have both then you're gonna have a bad time so we are going to grab why do we still have a bunch of thumbs down that's because we didn't cut we copied so we go delete those copy those last couple and we're going to paste the rest inside of our testing folder and remember we did all of those so we're going to paste the rest in to our testing folder all right so now we've just added more images of our classes that we weren't too confident about i thought we pasted those in there already ef7 let's just cut those okay we did all right so now if we go into our collected images folder thumbs down we already did those and a thumbs up cool all right so that is now all well and good so we've now gone and added more images to our testing folder and more images to our training folder now if we are going to retrain we're effectively going to be using our training and detection notebook again or we can also stop label image so if you just hit stop that's going to close this off and we can close that as well all right so now that we've got our images collected the next thing that we need to do is go on ahead and train our model now in order to do this we are zoomed in quite a fair bit there let's just go to 125. so in order to do this we're just going to go back to our training and detection notebook and we're just really going to change one thing so namely i'm going to change this custom model name so when we built our first baseline model we actually named it my underscore ssd underscore mob net now we're going to call it slightly different so we're going to just add on underscore tuned and this is going to give us a new folder so again if we step through this it's going to create a new folder which again you can see there so we've now got my underscore ssd underscore mob net underscore tuned so we've now got a separate folder that's going to contain our tuned model now we can keep going through this and effectively this is going to give us the ability to go on ahead and train a tuned model now we can skip this entire step entirely so we don't need to download any new pre-trained models or reinstall tensorflow object detection we've done that we've gone through that pane that's now done and dusted a label map we can run that again but really we're doing this so we've got this label line available and then we're going to create our tf records again so we are going to now create new tf records and again we don't need to install this oh hit install pi tz again we had that installed we are however going to run this generation so remember this is going to create our test record and our train record so if we run this hopefully fingers crossed no issues and this should give us our new tf records to be able to go on ahead and train with so let's let that run and it looks like we don't have any errors there so we've successfully created our tf record file intense flow workspace annotations train record and tensorflow workspace annotations test.record so we're good to go now now the next thing that we need to do is copy our pipeline config so this is going to copy the pre-trained pipeline config into our new tune model so if we run that inside of our my ssd mob net tune folder you've now got this pipeline file we can hide this we don't need a preview pane minimize that so it's a little bit easier to see so we've got this pipeline config file and then really we just need to make our updates as per usual so this is really just filling in what's inside of this pipeline config so if we take a look at this remember right down the bottom we need to configure all of these paths but we've got our code here that's going to do it for us so we're just going to step through it's going to open up our pipeline config and again this is just the standard one and keep stepping through so this section here is probably one of the most important this is actually updating the pipeline config so if we go and run that run that again or run the next cell now when we open up this pipeline you can see that we're now going to have all of our label map paths so on and so forth defined so that's all well and good now what we can do is run our training script so again this is all sort of pretty modular so you can just sort of run through it and it should ideally work now the one thing that we are going to change is the number of training steps because remember i said the things to try are adding more images and changing the length of training time so we can bump that up and we'll increase that to 3000 and we can run that command and again try this out so you might want to play around you might want to train for longer train for a shorter amount of time the world's euro stick here so again a lot of it is experimental so we're going to copy this and we're going to open up a new command prompt and as per usual we're going to go to where our virtual environment is and activate that so let's go and do that okay so that's where our virtual environment is so you can see that we've got tfod there so we can activate that so remember dot backslash tfod scripts activate so that's our virtual environment now activated you can see it there and then what we can do is paste in our training command so let's go ahead and run this and ideally we should have a trained model come right about after okay so after 3 000 steps of training we finally got our final trained model which had a loss of 0.213 now keep in mind we had more images we had images at different angles so ideally what we want to do is run our valuation script to see how our model is actually performing so remember our last model had a i believe it was mean average precision of 0.7 so now we can see what our model or how our model is performing like this time so to evaluate our model remember we just go through and run the code under step 7 so if we hit that and enter that so this is going to give us our new evaluation code so we can copy this over and paste it into a model or into our command prompt so this should ideally run our evaluation pipeline so it will take a little while and ideally we should get some evaluation metrics generated okay and you can see that we've now got our valuation metrics calculated and it looks like we have managed to improve our model so you can see here that our average precision is now 0.834 so that's a bit of a reasonably large jump compared to what we had initially so before this was about 0.7 and i believe average recall was 0.69 or something along those lines so again they were way lower they were in the sevens and sixes not the 84 to 80 what 84 range so so again we have managed to improve our model and generate a better object detection model so simply by adding more images um adding additional labels and different positions we've ideally produced a model which should generalize better now what we can actually do as well is again load this new model now because we have trained for longer we're going to have a new checkpoint so if we create a let me just show you this so again if we go back into our directory youtube tensorflow object detection tfod course tensorflow workspace models and remember so this was our baseline model that we initially trained and then we trained our tune model now if we open this up you can see we're all the way up to checkpoint four so this is our latest model so in order to load this what we need to do is change the checkpoint that we're going to be loading so rather than loading checkpoint three we need to load checkpoint four and then what we can do is we can actually go and run some detections so rather than detecting from an image let's just jump straight to real-time detection which is effectively this cell here and if we run that that should effectively open up our webcam and give us the ability to make some real-time detection so let's see how that actually performs okay so we've got our detection window thumbs up thumbs down thumbs up thumbs down all right so this is detecting live long so let's move over a little bit there you go way better right so that's again performing ridiculously quickly so we did train for longer we trained with more images so really faster right now we're really really quick and this is switching really fast as long as my hands aren't below and so that sort of gives you an idea as to how you can go about performance tuning so again you can see it's very very quick and we're doing pretty well there awesome all right cool now the next thing that you can also do in order to performance tune is change the architecture so if we cast our minds back we sort of talked a little bit about adding more images and training for longer now the next thing that you can also do is change the architecture now in order to do this you can re all you really need to do with the particular code set that i've got is change the link that you actually want to use so in this case we're using let me just zoom in we are using this 320 by 320 model but say for example we wanted to use uh this one over here so again there's some nuances if you want to tune them but really all it's a matter of doing is copying the link address and then going back over to our detection notebook and changing the pre-trained model url so if i actually run this so you can see it's updated the url for that specific model and then what we need to do so once we've got that we just need to grab this name here so you can see here that it's called ssd mobile at v2 fp and light 640 by 640 coco 17 underscore tpu-8 so we can copy that over and paste it here so this is effectively going to give us the ability to go and grab a new pre-trained model so when i run this it's going to download a new pre-trained model and effectively walk us through the exact same steps now i'm not going to go through this in a huge amount of detail but if you'd like to see a more detailed video on using different architectures by all means do let me know but it sort of gives you a feel for how you might go about this so really you grab the new model link and then this cell over here is actually going to go on ahead and download that new pre-trained model so then what will happen is inside of your pre-trained models folder workspace pre-trained models so you'll get as you can see that i've already downloaded a bunch of different ones so it'll just download a new a new pre-trained model that you're able to use so that about wraps up performance tuning so we did a bunch of stuff there we didn't really need to change our model architecture in order to get better performance really it's about the image quality so rubbish in is going to equal rubbish out quality in is going to equal quality out so having different styles of images different images of different qualities and different images at different angles as well as training for longer more often than not is going to be sufficient for producing a better model so this brings us to our next part training on collab so in order to train on collab you can go through pretty much the same steps the core difference is that you need to have collected your images already in order to be able to do this so let's go ahead and jump onto collab and we'll start doing that so again the core thing that i was saying is that in order to train on collab you do need to have your images pre-collected it just makes your life pre-collected and pre-annotated just going to make your life a bunch easier so let's go ahead and do this now the easiest way to get your images on to collab is inside of our image collection notebook right down the bottom i had this code cell which allows you to compress your images and archive them so if we actually run that cell this is going to compress the images that we've got so if we go into our images folder now workspace images you now you're now going to have this file called archive.tar.gz so this is all of our images from our training and testing folder and now compressed and archived so what we can actually do is jump on over to colab so in this case i've just gone to colab.research.google.com and then i can actually bring in a github file so what we'll do is we'll grab in our main github repository which was where are we so tfod course we can paste that link in and then we're really interested in our training and detection notebook so let's grab that and then the nice thing about this notebook that i've set up is that you can sort of just walk through this notebook now if you don't want to have to deal with all of the installation steps and all of that stuff that we had to do to get us up and running codelab sort of gives you a nice alternative to that core thing you need to do is go to your runtime and change your runtime type to gpu in this case it looks like it's already selected so we can hit gpu and hit save and then what we're going to do is step through our notebook so we're going to hit shift enter and run anyway and this is going to start importing and creating our folder structure so again by running the first four cells we should have our folder structure created you can see we do so tensorflow workspace annotations images now the one thing that you will need to do is upload your files so specifically your images so if we select our images folder what we're going to do is upload the images that we just compressed so we're going to go into where our archive is so tensorflow object detection tfod course tensorflow workspace images and then remember we had this archive so let's upload that so we've now got our archive.targeyz file there what we'll do is we'll copy that into our images folder and then when we actually go through the rest of this notebook so let's just hit shift enter so if we keep going through so this is going to clone our tensorflow models repo which should give us our files in here once that's done and then if we go to our next cell so now we're installing tensorflow object detection so this will take a little while and so keep in mind this is installing all of our dependencies to be able to leverage tensorflow object detection so that's looking promising at the moment looks like it's all installed we can run our verification script see if that's all good so this looking promising at the moment doesn't look like we've got any errors and again whenever you're running the verification script gotta check that it says okay so in this case we've got the all clear so that's looking fine test importing object detection no issues there so we're good to go right so what we can now do is bring in our pre-trained model so again pretty standard so this is going to give us our pre-trained model over here we can then create our label map create our tf records so again this is going to create our tf record so if we go into our annotations we've now got our test record and our train record created we can then go into our so what we'll do is we'll copy our pipeline config through so we're just stepping through this notebook so it doesn't look like we've got any errors so far update our config so this should give us our custom config inside of our models my steam obnet folder so you can see it there and then we can train the model now the nice thing about training in colab is that we can just run through and execute that command in the cell so you can see that now i'm running that command and all things holding equal without errors we should get our model training kick off now again the key thing in here is that you won't be able to perform real-time object detection using your webcam you can still do it using the image so we'll test that out we'll see what that looks like so that's looking like it's running there without issues and then again we just need to wait for our loss metrics to show up so that gives us a good indication that it's running successfully no issues okay and that's our first loss metric appearing so you can see there that we've got our step 100 per step and we've got our loss metrics appearing so that's a good indication that all is successful and it's running in collab so we can let that run and we'll come back evaluate that model load it from a checkpoint and then run our detection from an image but again from here you can sort of keep stepping through if you wanted to once you get to the phrasing and conversion step you can grab so you can actually zip up and export everything inside of this workspace folder so i wrote the code down here so you could if you actually wanted to grab everything you could zip up your entire my ssd mob net folder bring it back down to your desktop and run it from there as well but again this looks like it's all running pretty successfully we check out my ssd mob net folder you can see that we've got our checkpoint appearing there so we've got checkpoint one cool let's let that run and we'll be right back okay so that's our model finish training so you can see we got up to step 2000 right down here with a final loss of 0.247 now what we can do is we can evaluate our model as well so again we can run it in cell and ideally what we should see is our mean average precision and our mean or our average recall as well so ideally this will pop up once we've got that calculated and all we're really doing here is running the code and uh step seven so evaluate the model so this will appear as soon as we've got those detection metric or those accuracy and evaluation metrics calculated so again similar to what we did inside of our command prompt but now you're doing it inside of the cell inside of colab so that's looking promising it's found our checkpoint three and in a second looks like we've got our detection metric so again this is our model and this is our non-tuned model right so remember we trained our tuned model for longer we ran that for 3000 steps this was only 2000 steps so you can see our average precision in this case is 0.71 and our average recall is 0.715 now what we can do is we can also detect our model using an image so we can run through our cells to load up our model in this case our latest checkpoint is checkpoint 3. so all we need to do is change our checkpoint to checkpoint three in this line here which is checkpoint.restore so this going to load up checkpoint3 into our notebook so we can actually leverage that so if we run that cell oh we actually need to stop our evaluation first and then we should be able to continue onwards and detect from an image so in this case we're going to be detecting from an image so what what happened is we our evaluation script is going to keep running and keep looking for a new checkpoint so we actually need to force stop that in order to continue on with ourselves so in this case i just hit stop to get that to stop now what we can do is we can load from our checkpoint so what we really just did there is we changed our checkpoint.restore file and we made sure that we're pointing through to the latest checkpoint so in this case it's going to be 3 which is what we've got over there so if we run that that's all looking good and then we can detect from an image so remember our image name needs to be a valid image inside of our images folder so if we open up images and then test you can see that we've got let's make sure that we've actually got this file so we have uh livelong0253422 so let's take a look to see whether or not we've got that well easier yet we can just copy the name of this file so let's grab this one and we don't need a new code cell so let's just paste the name of the file there and then we should need to run these two cells to bring it so this is going to bring in our dependencies it's going to load up our category index again same as what we did in our baseline notebook and then if we run this cell ideally we should be able to see a detection so this is going to run using our object detection model and try to make a detection using that particular image that we've got over there perfect and you can see it has in fact detected live long so you can see that there again you can make this image a little bit bigger if you wanted to um we could try a different image so let's try a thumbs down so if we copy that file name and we pasted that over here again we can make another detection and this should be thumbs down which in fact it has detected by the looks of it so you can see it's a little bit sketchy there or a little bit small but you can see that it has in fact detected our object so that gives you an idea of how to train on collab now again if you wanted to convert or export all of this i've actually got the code to export it down here so what we can do is run this cell and it will zip up and export your files and if you wanted to you could load that into google drive as well and export it bring it back down locally and run it locally if you wanted to but keep in mind if you want to run it locally you do need to install the object detection api to have that available locally so that about wraps up training on colab and really wraps up our part seven of this course the next part that we're going to be going through are our projects now there's specifically three projects that we're going to go through reasonably quickly so this just sort of gives you an idea as to where you can potentially take this object detection technology further or might give you some ideas as to where you can apply it in your real life so the first project that we're going to take a look at is how we can use different types of sensors so specifically a microscope to be able to use or perform object detection so we'll give that a try and i have noticed that it does get very very close to the object so we'll wait and see and take a look at performance there then we'll take a look at building a web app so again we're able to leverage the existing code base we're just able to repurpose it to be able to build up a tensorflow.js app and i'll show you where to get the code for that and then last but not least we'll export our model and be able to perform uh some raspberry pi sentiment so we'll just be trying to detect one class there so whether or not i'm happy or not so in this this case we're going to be going through each of these three projects but starting off with microscope defect detection then web app directions and then last but not least our raspberry pi sentiment okay so we're going to jump back into our notebooks we can close collab for now and close that and again let's just restart our notebook so we're going to shut these down and again all things start so say for example we're starting off with our microscope defect detection now this particular microscope that i'm actually working with is a usb microscope so you can see it there right and i'll link to this in the description below if you want to test this out or take a look at it yourself so think of this as just another camera so right now i'm using a webcam to be able to perform object detection but we could just as easily use a different sensor say for example you had a better or high resolution webcam or a slightly different webcam you could definitely use that as well now we're specifically going to be trying to train a model to detect defects in these little leds here they're really really small so it might be a little bit difficult for you to see but um so you can see that this is a green led and it's got a bent pin this one is still a green led but this one doesn't have a bent pin i've got a red one as well and again you could do this on other stuff so if you had like a printed circuit board you could try that out as well so let's i'm going to plug in this usb microscope and take a look at what this looks like so you need to get it into focus a little bit so we're going to write some code to be able to see the output of this webcam first before we commit to actually collecting images so what we're going to do now is again we're going to close down our notebooks and we're sort of going to start from scratch so we're still going to be working within our environment that we've already created because we've gone through so much effort to set this up we've got our virtual environment we've got all of our dependencies installed well first i'm going to start out by opening up our image collection notebook and we're just going to create a new section just before capture images and we'll call this uh i don't know 4.x so i'm just going to create a new markdown section and i'm going to say this is uh testing microscope position ning cool and then we'll convert this to a markdown cell and then we're going to import opencv as our dependency and sort of just leave it at that for now so what we'll do actually we should probably put this up let's put it up here so we'll make it 1.x so what we're going to try to do is we're going to try to read from our usb microscope to be able to at least get a feed and then we'll position it try to get some zooms appropriately and then see what this looks like i've never tested this before so i wanted to see whether or not it's possible what our results look like and what it looks like to do object detection with different types of sensors okay so the first thing that we're going to do is we've imported our dependencies from an image collection notebook now we're going to try to read in a feed from this particular microscope so in order to do that we first up need to get a feed so we're going to call or create a new variable called cap equals cv2.video capture and we're going to try out sensor 0 first so remember this is no different to this line down here so we're just grabbing a video capture from our capture device then we're going to include a while loop so while cap is opened so this is checking that our capture device feed is still available so when you actually set up this capture right you can type in cap.read and this is going to read the feed from your microscope so what we can then do is extract the two variables that we get from that so we can type in red comma frame which again is no different to what we're doing when we're capturing the images from our webcam which we did for our different gestures so in this case we're going to be trying to detect two different classes so either a bent pin or not a bent pin but we ideally want to position our microscope appropriately first because i played around with it yesterday and it is a little bit temperamental in terms of focusing so let's position it first and get it uh in the right spot so we're going to write ret comma frame cap dot read and then we need to show our feed so to do that we can type in the cv2 dot i am show and then we name the name of our frame so we're going to call this microscope feed and then we are going to display our frame so a frame that we actually get back from here is actually just an image right it's an image from our microscope so that should give us our feed and then we can copy this little bit over here so this is um these four lines of code let me actually explain that in a sec right so these four lines of code so this basically is what gives us our ability to sort of quit out of our feed and close that little python window and then cap.release is just best practice so this allows us to release our capture device and destroy any of our opencv windows ideally you want to run this after you close any cv2 web capture and sometimes you might get your sensor hung up so say for example it won't detect images anymore it might sort of freeze up running those two lines normally resolves a bunch of those problems so let's go ahead and test this out so in this case we let's just double check so we've written so we're getting our capture so cap equals cv2.video capture and we're testing out device one and we're looping through so while the captcha is opened then grab or read the frame show it back to us and then if we try to break out of it then break out of it so let's run this and see what happens okay so what are we getting there so uh so cv2.iamshow so let's try another capture device okay that's looking more positive now all right so this looks good so i had to change the video capture devices capture device zero is probably my current webcam that i'm actually streaming on so we might need to change that now right now this looks kind of weird so we're not actually seeing anything from our microscope feed but that's because i'm just got it currently pointed to a piece of paper so if i move this around right let's move it there so you can currently see like one of the leds so if we bring that into focus a little bit more nope the other way all right cool so we've now got one of the leds actually appearing so i can move this around and you can currently see that we've got that actually appearing so we've got a bent pin there so you can see that it's definitely broken or it's obviously not going to work if we try to plug it in we can try putting in another led so in this case we've got one that actually is working but right now you can sort of see that we're sort of going down the same path right so rather than having a particular or doing gesture based detection this time we're actually just going to be using a slightly different feed but we're still grabbing pictures from a camera so in this case we're sort of good so i've got got the position based on my sheet of paper so i'm doing it on a white background to ideally allow us to see this a little bit better so we actually want to collect some images of our uh the different pins and whether or not they're broken or not so i'm just going to put it back into position and it's rolling away from me there we go let's bring that a little bit closer there you go nope it still wants to roll okay so i'm going to hold it there stay okay so what we need to do now is actually collect a bunch of images of our particular led so if i bring that back into frame i should have kept it in position but so right now this is pins the pins are fine we need to collect a bunch of images of different styles of pins so again this is probably going to be a little bit finicky we need to obviously get these right into position in order to get the photos appropriate but we'll test this out okay so we've got our webcam position appropriate or our microscope positioned appropriately so we can quit out of that using q and what we probably need to do for this particular project is collect images for two particular classes so really a bent pin and a non-bent pin so we're going to create two different image types so bent pin and i don't know what should we call the other one okay pin actually we'll call it defective and non-defective defective pin non d effective pin so we're going to only collect two different classes this time so defective pin and non-defective pin and we'll try with i don't know five images again and we'll give that a crack so this is going to create two new sets of images and again when we go through and run our setup folders so we can delete this that's left over so this is going to create new folders for our new image classes so if we go into our images we've now got a folder for defective pin and non-defective pin we can actually delete these old ones so live long thank you thumbs up thumbs down we don't need those anymore so those are now all cleaned up now what we also probably need to do is clean up what's inside of our test folder because we don't need those anymore we're doing a new project and we are now going to collect our images for our defective pins and our non-defective pins so we've got the folders created we can now go to our captcha and we're going to test out video capture device one because that's the device that we had to grab up here to be able to grab the images of our pins so let's run this and see how we go okay we're getting our frame bring it back okay so i've got it we probably need to run this again so right now i've got a we had a picture of a non-defective pin so let's stop this and go back into our collected images so these are the okay pins so we want to delete those because those are fine they are they are effectively non-defective but we had those images in defective so we're going to put the images of our defective pin there first up let's just double check we've got the feed right okay so that's looking fine let's actually position it this way right so that looks okay i'm just making sure we can get it in the frame appropriately okay that's looking good so let's just try to bring it in a little bit more that looks good as the trick so i move the paper not the actual pin okay mastery at last okay so what we're going to do now is collect images of our defective pin let's just do one at a time so that we can actually pause and i'm just going to do it for a red one we won't bother with the green ones for now let's do a defective pin oh we need to stop this cell as well so if we just jump back into here hit q that's going to close it down okay so what we're going to do is collect images of this defective pin and i'm just going to move it around slightly and we can probably increase the time that we want to sleep between frames so let's give us a little bit more time so three seconds so this line here which you can see that reads time dot sleep this is how long we actually sleep between each image capture so right now it's set to three seconds so if we run this let's wait for our frame to pop up so right now it's positioned so ideally we should be getting appropriate images already cool all right so we can move it slightly okay so that's done so we've got our images for our defective pin let's do our non-defective pin and remember we're not collecting a whole bunch of images here so this will be interesting to see what performance looks like let's just position it again so we'll run this cell up here and we've added a new cell there so that looks like a typo so let's just run that again okay so we want to position it down okay so that's looking okay all right so let's try that okay and then we can quit that by hitting q just holding the sheet of paper as we're collecting and what we need to do now is type in this is going to be a non-defective pin that we're collecting so we can set that label and then we can run this so this is going to collect images of our non-defective pin or non-defective led but again you could do this on a whole bunch of different use cases right so i'm doing it on leds and specific components and we're just going to move it slightly each time okay cool so let's take a look at our images all right so we've got some defective pins here uh let's show our preview pane so we've got defective pins and if we go into our non-defective pins looks like we've got non-defective pins cool so again now what we're going to do is go through our same process so we're going to label our images so we've got all of this installed we are going to run our labeling line which is this one down here so this will open up label image so let's run that and again we need to set up our label image path which is this really we should bring this down here and then again if you've got any feedback for the code or the notebooks or if you want any clarification hit me up in the comments below join the discord server more than happy to have a chat so let's open up i'll set our label image path open up label image and get to labeling so we're going to open our directory we're going to go into where our images are so in this case it's again in our workspace images folder collected images and defective pin so we're going to hit w and we'll just do let's just do this bit this bit yeah let's do that so we're going to say this is a defective pin right and then we're going to save save again i'm hitting w and i'm just labeling each one of these now if it wasn't so finicky i probably would have done a bunch of different positions but the microscope is super sensitive so wait and see what performance looks like on this so i'm just going to label it hit save all right and then we're going to open up our other directory so again same process we're just using a different sensor and this is going to be pretty consistent with most of the projects and then we're going to label these non-defective pin so save and pay attention to your labels as per usual because it is important and i've got a feeling our model will won't be generalized as much as possible because right now we've only got our pins in one direction so we'll see what that looks like but again you can always go through the performance tuning steps to get this to perform better and we sort of went through that so you add in images of different angles at different positions under different lighting positions that should help out okay so those are our images labeled so if we go through so you can see that they're they've all saved and we've got five of each class so we're good to go there now as per usual we're going to copy our images into our training and testing folders so we're going to grab the majority so we'll only leave one for testing it for each class and we're going to paste these into our training folder if we go back into our images collected images non-defective pin we'll leave one and again back into there and then the last one we're going to put that into our testing folder so cool all right so we are all set up now so we've got our training images or testing images and our training images uh so we should be able to now go through again our training pipeline what we will do however is we're going to probably create a a new folder for our custom object detection model in this case for our project so we'll call it a pin detection or something along those lines and we will do that by opening up our training and detection jupyter notebook and so our last model that we had was my ssd mob net tuned in this case we're going to call it microscope and we are going to leave everything pretty much the same we want to revert this back to our other models so this right now is our custom model we don't want to use that so let's grab the link to our existing one which is going to be this one here copy link address and i'm going to paste that there really this is sort of just making sure that we're resetting back to usual so we've already got this model downloaded so we don't need to download it again but we could if we wanted to all right so we're going to import our operating system set this up go through our paths create our file paths and create those actual directories so now if we go in you can see we've got a folder for our microscope model and then we're going to go through the rest so again we can skip step one we don't need to do that because we've already set up tensorflow object detection and we've already cloned our pre-trained model which is what this cell does what we do need to do however is update our label map so if you cast your minds back when we set up our images so if we go into workspace images train so we need to make sure we get the appropriate label so it's going to be defective pin and it will be what was the name of that sorry i've got to open up the actual annotation and non-defective pin so we're going to update these labels here so again this this whole code base is really really general purpose so you could do a bunch of stuff with it so we're just going to rename our labels so defective pin and i'm doing this under step two label map so this first line here again we probably talked about a while ago but this first line is what sets up our labels then this is what creates a label map which is what tensorflow object detection api needs to actually to actually train i'm going to set up our next label non-defective pin so we'll do that and now if we go and open up that actual label map which will be inside of tensorflow workspace annotations let's open this you can see we've now got one class for defective pin and one class for non-defective pin so we're good to go there now what we want to do is again we are going to create our tf record so we don't need to run this cell here this is what unarchives our images inside of colab because we're not running on collab we're good to go don't need to run that this cell actually clones our generate tf records group so we can bring that in it's not really good because it already is there it's not going to clone it again so we've got a conditional statement there that's going to prevent that happening what we do need to run is this cell here so this cell here is what creates our tf records and remember our tf records are what allow us to actually train our model it's what our tensorflow object detection api uses so if we run that fingers cross we effectively create our tf records so it looks like no issues there and then again we're going to copy our model config to our training folder so remember we created a new training folder called microscope so if we run that we should be able to go into our microscope folder which is in models microscope and you can see that we've got our pipeline there we just need to update our config so again so that's our baseline we make our updates if we reload it this is a cool thing right so you can actually reload that config by running this cell here so config underscore util dot get configs from underscore pipeline file and then we're specifically going to load that up another thing that i had a thought about is if you'd like to actually see me write this code from scratch please do let me know i think it'll be a huge tutorial but if you think it would be useful to be able to learn how to actually do this i'd love to hear your thoughts on this as well maybe we'll do a live stream so in this case that's looking good so we've reset our classes and if we scroll down we'll reset our label map and where our test.record file is let's go on ahead and train our model so again we're going to generate our training script under step six which gives us this year and we probably don't need a train for 3000 steps let's do a regular 2000 so then we'll copy this into a new command prompt and again same folder so we can stick inside of this folder here so the youtube tensorflow object detection tfod course folder and we've got our environment activated we should be able to paste this in and kick off our training so we'll kick this off and we'll be right back as soon as that's done alrighty so our microscope object detection model has now finished training so ideally the end goal for this is when we load this up we should be able to detect a bent pin and effectively or a defective pin and a non-defective pin so i'm interested to see how this goes because i've never tried this so this would be really really interesting so let's go on ahead and do this now so we'll just make sure our gpu is fully released because sometimes it can throw up a whole bunch of issues if it's not now inside of our microscope folder you can see we're up to checkpoint 4. so this is a little bit further than usual so what we'll do is we will load in checkpoint four so that's looking good and so let me just quickly give a recap on this notebook so step zero sets up your path step one or ideally point one is what's going to download your pre-trained models from tensorflow models and install tensorflow step two is where we create our label maps our different labels step three creates our tens flow records four copies our pre-trained model config into our custom model config folder or into our custom model folder step five's updates that config and then step six trains which is where we copy our command step seven goes and evaluates that model so this is purely optional step seven step eight is where we actually initially load our checkpoint so if you get more checkpoints than checkpoint four which is what i wanted to call this out you can change this number to detect or use an older model or use a newer model you sort of worlds your eyes to that so and then step nine is detecting from an image step 10 detect from your webcam and then step 10 11 12 are going to be your exporting and freezing step 13 is your zip and export so again i sort of wanted to drive that point home that there's a logical sequence of this so to run through step eight what we're going to do is run this import cell which you can see here and then we'll load our checkpoint so we've set it to checkpoint4 and then we can detect from an image so let's try detecting from an image first before we mess around with actual the actual microscope so we're going to grab the name of an image from our test folder so we'll grab this defective pin and we're just going to grab the name of that image and then paste it in here right so all i did was i grabbed this particular image i grabbed the name of that particular file and then i've overridden what's in this cell here so this is basically pointing to where our image actually is i'm just updating that then what we can do is run that import which we just did then set up our category index and if you take a look at this category index probably haven't talked about it much this is just another representation of a label map so we can close that again bring in our image and then ideally fingers crossed this should be able to detect whether or not it's defective or not defective so now we're actually running the cell to detect oh my god it actually detected it how cool is that so you can see that it's actually detected the fact that it's a defective pin now this is probably a gimme because i mean they're sort of in the same position but that is is really really cool what's possible there um because this opens up a whole wide range of opportunity right like you can now use different types of sensors you can scan over different types of things let's grab another image so um so what was the other one inside a test so non-defective pin so let's see if that works so we're just going to update the name of the image to non-defective pin run it again oh and it's detected non-defective pin with super confidence the bigger test however is if we detect try doing this in real time and see how it actually performs so video capture zero i think we actually need to go to video capture one so let's test that and see if that opens up that's looking like it's working pretty cool so that's detecting a non-defective pin even if i switch it the other way it's still going pretty okay sweet right and this is all coming through that little microscope that i showed you what if we did a defective one let's bring it into frame look at that uh where are we so this is sort of the concern that i had with having the defective pin in a particular position so you can see that it's it's definitely not performing as well with the defective pin but you sort of get the idea right what's possible there obviously gonna pick it up so it doesn't like the defective pin at least from the images okay so it's actually picking wait it's picking that up as non-defective what if we did a different color i wonder if it's the lighting as well this is me just playing around with defective pins and stuff okay just trying to get this into the center there we go okay so it had a little so what we can actually do here this is actually a good point time to actually show you something so if i quit out of this this detection frame here actually has a bunch of parameters that you can choose to tune now the two that i'm going to get you to place pay most attention to are these down here so these two lines max boxes to draw and minimum score threshold define what level of quality detections you're actually going to show so right now i've got it set to reasonably high right so i'm not going to show anything that has a detection accuracy of less than 80 but in this case we're not be we're not able to pick up the defective pin so what we might do is we might relax this constraint a little bit and drop it to 40 for example so if we run that again okay so now you can see that it's we're obviously getting a whole bunch of detections so this is sort of what happens right so there's a bit of a trade-off as to whether or not you're going to be able to or want to show everything versus not show everything so in this case you can see that it's we're getting 60 so what we might do is not drop it down so much to four let's drop it to 60 and try that again okay so we're now not getting anything all right so you can see there that it's detecting defective pin reasonably well it's interesting that it's not actually doing the bend it's still detecting that it's a defective pin but it's not doing the bend interesting so this is the green one let's try the red one and keep in mind we didn't train on a green image on a green pin the led is rolling away and so that's a non-defective pin let's try another green one which is a non-defective and keep in mind when we train we trained at an angle right so this is going to be obviously skewed so when i put it at an angle for whatever reason it thinks that that's a non-defective pin straight away but if we change it the other way you can see that we're getting some lower accuracy detections for whatever it's detecting my thumb as a defective pin they're pretty cool right like so you can start to see what's possible with this so if i bring that into frame again oh it's detecting my fingers is non-defective pretty cool so let's throw in a bunch of them defective and these are all sort of overlapping right so let me move spread them out a little so we're getting non-defective on that red one which is appropriate what about if we went to this green one saying that's defective so that might be an indication that we need to train a little bit more on that all right spin this around okay so you can see at that angle it's detecting that it's not defective so that might be an indication again that we need to give it a couple of different angles let's try it that's saying it's a non-defective when really it should be defective and you can reduce the number of boxes that you're also going to draw as well so in this case i'm drawing a bunch you can change that parameter again hopefully losing napkins let's move it around pretty cool right so that shows you what's possible with a bunch of other different types of sensors so again you could train this you could performance tune you could get it better so that sort of summarizes project number one and being able to use different types of sensors for our object detection so just to quickly summarize what we did we went through our image collection notebook we repositioned our microscopes or we wrote this custom code here to be able to make sure that we get our microscope positioned appropriately we then defined our images that we wanted to collect set up our new folders captured our images and then we labeled using label image again so again pretty standard process so you're always going to capture your images label your objects then train now we're in our training we sort of went pretty standard here we didn't do anything too crazy we just created a new folder called microscope that means that all of our microscope stuff is going to be stored inside of a folder called microscope now again for our new projects we're going to just create a new folder or for our next couple of projects we're just going to create another folder and build up from that that brings us to project number two so in this case our next model is going to be a web app for direction so what we're ideally going to try to do is build a object detection model that allows us to detect up down left and right then what we'll do is we'll use a little bit more of the conversion code to export this out to tensorflow.js and i'm going to show you how to actually create a web app like what you can see on the screen so that's actually running in a browser at the moment so let's go back to our jupiter notebook and again we're going to start as per usual sort of this is sort of what i wanted to drive home so whenever you're regardless of what you're building you're sort of begun you're sort of going to be going through the same process so you're going to collect your images you're going to label then you're going to do whatever exporting you want or you're going to run real-time detection so here we're going to go back into our image collection code and again we've already got our imports done we don't need to retest or actually we should probably retest to see because i've gone and unplugged the microscope now and i've plugged in a webcam so let's see if that's working okay that doesn't look like it's happy let's try video capture device one plugged it in looks like it's okay all right cool so that's our webcam so i just had to change the video capture device so we're back to video capture device one so if i move this way it's fine if i move this way so it looks like i'm in front of the camera so what we're going to do for this particular model is we're going to be detecting let's get back in front of the mic is we're going to be detecting up down left right so ideally we should be able to detect those inside of our web app now the beauty of doing this web app is that you could go and deploy this as your own website later on and have users be able to actually use that so now this is going to take a little bit of javascript skill but it's not a whole heap and we're also going to need to upload some files but again i'm going to walk you through this and all the code for this is actually available inside of the github repository so now what we're going to be doing is we'll be combining the code from our tensorflow object detection course and the code from our tensorflow object detection app github repository to be able to load this up so let's go ahead and do this so i'm going to quit out of this so we've got the right feed and we're just going to change our video capture device back to zero down here to make sure we've got the right camera and then what we need to do is collect images of left right up and down so we're going to set reset our label so left right up and down okay so now we're rather than collecting thumbs up or thumbs down or live long or whatever or even the defective and non-defective pins we're going to be just defining direction so left right up and down and we'll try with different hands and whatnot so what we can now do is run this particular cell and we can test our labels i don't know why i deleted that and you can see that we've got our correct labels are left right up and down again we're going to set up our folders so if we go into our images we again can delete everything that's in there so if we go into tensorflow workspace images and collected images so we've got defective and non-defective pin so we can get rid of those we don't need them anymore and what we're going to do now is create our folders for left right up and down so if we do that and run through the steps under step three this should create our down left right up and down folders and then what we can do is capture our images as per usual so again we probably don't need three seconds break so we can change that so again we're just going to point in our different directions capture our different images and labels so let's go ahead and kick things off so we should be going left right up and down so let's do it again we should get our little pop up so we're going to be going left first i have to use my mouse with my lip right here or left hand okay so left then right again move your hand around i probably should have collected different images when i collected left right now it's covered on my face is behind it so i'm just going to move my hand around and it should be we should be up to up right so up probably do both hands to help generalize and then down again we'll try to use a different hand oh that's a dual hand cool all right so those are our images collected so we can go and check those out so i've gone to youtube tensorflow object detection tfod course tensorflow workspace images collected images so we've now got images inside down left right and up so that's looking good okay now what we need to do is label as per usual so again we need to define our label image path which we had already and we need to open up label image because keep in mind we've already got it installed so we're just going to open it up so i'm going to go into my image path or image paths and we're going to start labeling so we're going to go into collected images and down and then open those up so that's down and again paying attention to the case on our images beautiful do the next one now again if we get lackluster performance we can always come back and performance tune as well so we're going to do down on that one beautiful all right so that's down done let's grab the next one so left this probably needs to be flipped but that's fine behind the mic yeah that'll be an interesting one and we'll keep on going so left so down had okay images left i'm a little concerned because they're all kind of in front of my face so again we might not get the best performance on here but again it's all about testing and training and trying it out so left then we'll go left again cool so that's all of them done we can check our labels all right so we're good there then we'll go into right let's do that i'm going to change our label again save right save the next one save that save that and then last one over here okay let's check our labels beautiful alrighty last one up and let's do it okay and then last one and keep in mind we're only doing five images per object or per class at the moment but you could definitely do more i normally do 5 10 to 20 when i'm just prototyping and obviously if i'm labeling 20 images in this tutorial it's going to take quite a while so 5 is fine but play around add more images add less images and remember you can collect a varying number of images just by changing this line over here so number underscore images so right now i've got it set to 5 we could set it to 10 20 30 40 50 100 if you really wanted to okay so on that note we have now gone and labeled our images so if we go into our images folder again and again it's probably getting repetitive now but it sort of gives you an idea of again the process is going to be the same so we've got our images in down and our annotations for down and left right and then up and so what we'll do is we'll clear out our test folder of our pin stuff and now what we'll do is we'll go to our collected images and again we're going to do the same thing so we're going to cut them out and paste them into a training and testing split so we've done down let's do left we're going to paste so at the moment i'm pasting four of the available images inside of the training folder so what would that be an 80 20 split and play around if you want to include more or less again sort of trial and error a little bit here and data science and specifically this being deep learning and data science it is an iterative process so you can go through an experiment and you don't need to get it perfectly right the first time it's more than okay if you play around and try different things out okay so we've got our images there what we want to do now is go and grab the rest and throw it inside of test so we're going to have one image per class in our test folder i'm just going to keep doing this and again this is just copying and pasting right so you could potentially automate this if you use the os library and just randomly choose which files to place into train and test but in this case i just find it easier to do it manually because then i've got a little bit more control over it okay so where are we so we've now got our images inside of test so we've got five image or four images so left so down what is that left right and up and again inside of train we've got a whole heap more we're going to go through our standard process now so we're going to go through back into our training folder and again we are going to stop label image because you can see it's still running up there and we've got a little star so we can stop that jump into our training and detection folder and what we'll do is we'll just re we can actually just clear all of our stuff here so we can make sure that we are importing so we're going to import operating system we're going to change our folder name so we're going to call this one directions and then so this will effectively remember we had our my ssd mob net folder my my ssd mob net tune folder we also had our microscope folder now we're going to have one called directions so if we run this and run this and this and this this should create our new file paths go into youtube tensorflow object detection tfod tensorflow workspace uh where i have models you can see we've now got a folder called directions and this is going to be the current model that we build and then again we can skip skip step one because we've already set up our environment we are in our virtual environment already so we don't need to do anything magic there what we do need to do as per usual is update our label map so remember we had left right up and down and they were all capitalized so we're going to create those new labels so left right and then we'll create two new items up and down and again just double check you need to make sure these unique these ids are unique as well so this should be three and four another good thing to know so let me just quickly show you so again we want to check our images so make sure that we've got the right case so if we go into train let's take a look at down again that's capitalized let's look at left again that's the actual image again left is capitalized here so you can see that let's do right so i'm just double checking because obviously checking up front is going to save you a lot of time down the track when you're trying to debug errors if they do come up and up is capitalized as well so we're good there we are good so we've got left right up and down we've got unique identifiers for each one of them so one two three and four i think we're good to go so let's go and run this annotation which is fine so that's going to update our label map create our tf record so again we can skip this cell because it's only for colab so we can import our new generate tf record script again we've got it already so it's going to skip it we already had pi tz installed so we can actually delete that cell we don't need that anymore and then we can run our tf record generations again ideally this should successfully create our train record and our test record we're going to copy our model config to our custom folder again which will go into our directions folder and then we're going to go on ahead and update our config so if we re-import it in this cell you can see that in fact we've gone and updated it looks like we've got the number of classes equals to four so again this is updated if we scroll on down again we are picking up our correct folders this is all looking good okay so again now what we're going to do is train our model so we're going to run our command to generate our training script again print it out we're going to copy this we are going to create a new command prompt and again we've already got this one so this was from our or is this from the defective non-defective pin example of project and we're going to run our training code so if we paste that in this should kick things off and it should kick off our training so again we'll be right back as soon as this is done okay so that is our model train so this is our directions model and eventually what we're going to be plugging into our web app so you can see we went all the way up to step 2000 and ended up with a loss of 0.299 now what we'll do is we'll test this out inside of our jupiter notebook as per usual so again let's take a look at our checkpoint so if we go into our workspace models and then directions so you can see we're now up to checkpoint three this is a key thing to take a look at guys so if you're training your model don't always assume you're going to be at checkpoint 4 after 2 000 steps or checkpoint 3 it could vary so you always want to make sure you pull down the one with the highest index so ideally in this case it's going to be checkpoint 3. so there's checkpoint 1 2 and 3. so we want checkpoint 3. in this case that's exactly what we're going to do so remember for our um what was it our pin or defective pin detection what we had the latest model that we had was checkpoint four so we updated it to that there what we're going to do is we're going to do checkpoint dot restore and we're going to specifically pass through checkpoint three in this case then what we'll do is we'll import this as per usual so all of our dependencies for step eight load up checkpoint three and again we're going to skip right through to real-time detections because that's my favorite we're going to make sure that we set our video capture device back to zero because remember video capture device one was for our microscope number zero was for our webcam so let's try this out so in this case i'm just going to hit run and we'll bump up our minimum score threshold back up to let's say 90 um and give that a run so let's run that we need to import cv2 which is part of this cell we're probably going to need our category index and we don't need that image path so let's do that let's see hopefully we should get a little pop-up and all things holding equal we should get some real-time detections so i can see my webcams just activated so sometimes it takes a little bit of time to start close task manager okay so that's open so up looks like it's detecting appropriately down looks like it i only wanted oh no that's fine so down left oh that should be right okay so we've got an issue with left and right oh so right works well okay over there but if i bring it here so this is what i was saying so remember when we were collecting our images i said that having all of those images across my face might cause an issue because right now it's saying that both left and right over my face is left but if i put right over here in the green it's okay so it's saying right as long as it's not on my face is going to be right now left over my face hmm okay all right so we might want to do a little bit more performance tuning on there but you can sort of see how you might go about building your detection model so what i'd actually do in order to solve this is i'd include more images of me purely pointing what's this this is left so in this case that left is fine left is okay if i do it over my face and if i always did right over my shoulder then i'd be okay but in this case if i did it like this you can see it's maybe not performing as well but you sort of get the idea right so you can go and perform and stream this make it better improve that model accuracy so we've got up working fine down working fine we can do it with that other hand okay so other hand we might need to add in a little bit more tuning there so up is okay down with that hand is not okay you can start to see this is how you might evaluate your model and you'd add in additional images of the particular classes that aren't performing so well so that's sort of how you go about that performance tuning but it doesn't stop there for this particular model we are going to take this one step further so i'm going to quit out of this particular detection and we are going to freeze our graph this time so we're going to run through step 10 and freeze our graph and so we'll go ahead and run that and we're also going to run through everything in step 11 and convert to tensorflow.js as well so this should do two things so inside of our export which once our frozen graph is done we'll get an export here and then we'll also get a tensorflow.js export and then what we're going to do is we're actually going to build up this app so to do that we've got to do a couple of key things so namely what we need to do is go to ibm cloud in this case we're using ibm cloud to host our model and we specifically want to go to cloud.ibm.com forward slash catalog what we need is cloud object store and so i'm just going to log in so what you'll need to do is if you're doing this for the first time you'll either need to log in or sign up so sign up if you haven't signed up before and then you can just plug in your details and again this is all free so you can run this for free at the moment based on your model so what i'm going to do is i'm going to log in and if you actually take a look at the steps inside of the github repository so github.com forward slash tfod app this is actually going to walk you through each of these steps so if you maybe get a little bit lost as to where i'm going through and then by all means do check that out but effectively we're going to be following the steps that you can see there okay so let's actually do this one step at a time so let's make sure that our exports work so it looks like our frozen graph has been generated so let's take a look at that so if we go to youtube tensorflow object detection tfod course tensorflow workspace models and we look at directions inside of export we've now got our frozen graph and if we take a look at tfjs export we've got that done as well so we are in good shape now let's go back to our tensorflow object detection app component and see what we need to do so first up what it's saying is we need to clone this repository so let's go ahead and do that so if we copy this link here and again the link to this all of this stuff is going to be available in the description below so by all means do check it out there and if you need anything else by all means hit me up in the comments so we are going to open up our new command prompt and what i'm going to do now is i'm going to cd out of my tfod course which brings me here actually we should just probably create a new command prompt for this so we're going to go into d drive cd youtube go into our tensorflow object detection repository typing is a bit of a shocker all right so we're in so inside of that folder we've got tensorflow object detection course and then if we go into that tensorflow object detection course that's our root repository for where we've been doing everything so far so you can see if i type in deer if you're working on a linux i'm typing india it's going to be ls if you're on a linux machine so that's all of our baseline root dependencies so image collection and our training and detection jupyter notebook is in there so we're going to jump back out of that and what we're going to do is we're going to clone that repository that we're just looking at so this one over here which is the tensorflow object detection react app so this is using a javascript web development library called react or web development architecture called react and it's just a really good pretty sort of stable library it looks really similar to html so if you've ever done any html but you don't actually need to write anything that's the beauty of this so i've pre-written it all you should be able to pick it up and go so i'm going to clone this repository what we'll do is type in git clone clone that and then what i'm going to do is i'm going to go into that folder so cd and it's going to be called tfod app and then let's take a look at that so we've got all of our stuff there then what we'll do is we'll open this up inside of a code editor so i'm going to open it up in vs code so inside of here there is a bunch of stuff that you can take a look at so there's a cause config more on that later package.json you don't really need to worry about too much of this apart from this line here inside of the app.js file but i will come to that and sort of explain what we need to do so let's go back to our walkthrough steps and do that the next thing that we need to do is install node so node.js is a server package or a basically a server runtime that we can use for running our react app so if you haven't got this installed just go to nodejs.org and again the link is available inside of the github repo download it and install it and you should be good to go then the next thing that we want to do is install our app dependency so under step 3 we're going to go in ahead and do that so if we go back what i'm going to do is inside i'm just going to open up a new terminal inside of vs code and what we're going to do here is we are going to type in the following command so npm install so think of npm as the package manager or the equivalent of pip but for javascript in this case there's a bunch of others or resource managers but in this case we're going to be using npm so an npm comes when you install node so keep that in mind you need to have node installed in order for this to work so we're going to type in npm install and i'm just going to bring that up just to make sure my head isn't covering it and hit enter so this is going to install all of the dependencies for our react app that we're going to need and this might take a little while so let's let that run and then we'll be right back okay so those look like all of our dependencies are now installed and you can tell based on the fact that we've now got this node modules file now this is going to be using tensorflow.js in the background so it's all configured you really just need to keep going through the different steps that we've got inside of our github repo so in this case we've done steps one we've cloned the repo we've installed node i've already got it installed but again i'm showing you sort of where to go to do that the next thing that we need to do is actually create a new free object storage repository on ibm cloud so to do that we're going to hit this link we should do this in a separate link so we'll open a new tab and again this is that that place that i was sort of referring you to so in this case we can create a new cloud object storage and just hit light and then hit create in this case i've already got one created so all you need to do is hit light and then hit create down here and then what you can do is you can go and ahead and open it up so if i go and open up my instance i'll show you what it looks like so once you've got your cloud object storage repository made available it's going to look a little bit like this now what you want to do from here is create a bucket so to do that which is effectively what we are doing in step five so assume we've already gone and created that cloud object storage bucket so what we'll do is we'll create a bucket now which is step five so we'll hit create a bucket or create bucket and then what we're going to do is just hit uh i think we're going to hit customize your bucket and we'll name our bucket so what we're going to do is we're going to name this pay attention to this name because you'll need it later on so we're going to call it directions tfod that sounds good again you can name whatever you want you just make sure you keep note of that particular name and then we are going to select our location so i'm going to choose a you seed but you could choose a bunch of different regions to host this in and then that's about it then we can skip all the way down to the bottom and hit create bucket and so this will spin for a little while and we should be good okay so that is step five now or half done so we've created the bucket we need to store the model.json file and the dot bin files in the bucket so these were the files that we created let me minimize this and this and this so these were the files that we that were created when we went and ran our tensorflow.js export so this line over here is what went and created those particular files now if you go into the folder called tfjs export all of the files that you're going to need to upload are in there so group one dash shard one of three all of these so you need to upload all of these into your bucket to do this we just go back to our bucket hit upload and then we can select this little region over here to actually upload our file so we're going to click into that go into our repository so youtube tensorflow object detection tfod course tensorflow workspace and then we're models and then we're gonna go into our directions folder and then we want our tfjs export and we're just gonna select all of that and hit open and that's gonna upload our folders then what we need to do to actually go on ahead and upload i've got a feeling my head is going to be blocking that so i'll move out of the way we just need to hit upload and that will upload our files into there and then there's two things that we need to do in order to get this all good and running okay so that's looking like it is working fine so we can go on ahead and close that then what i want you to do is grab or select this model.json file oh it hasn't been generated yet so we'll come back to that so we're then going to go on to our next step which is enable a public access policy so this is step 6 over here so to do that we can close this and on the left hand side you're going to see this button over here called access policies so we can select that and what we're effectively doing is we're hosting our model here so this is going to allow us to actually host our model and leverage it from pretty much anywhere in the world so we need to do two specific things we need to put our model in a bucket and then we need to make it available to the world wide web so we can reach it from our app so right now we've got it in our bucket but right now we can't actually access it from our app so we're going to enable public access so once you've hit access policies go on over to public access and then choose so leave the leave it as access group public access and then leave the role as content reader and just hit create access policy and then it's going to give you a warning because now it's going to be publicly available hit enable and that is our access group policy created so then what we want to do is actually go back and grab the url to this particular model.json file so now that you've created the public or enabled public access you should be able to grab this object public url over here so i'm just going to copy that and what we'll do is we'll go over into our app that we created which is this over here so what we cloned down and in order to use that particular line that i just got you to create you just need to go inside of so say this is your main app folder what you're going to need to do is go into source app.js and if you go to line 17 or at the moment it's line 17 but if you just search for the line that says const net equals await tf load graph model so this is actually going to load our react or load our tensorflow.js model into our web app so if i minimize this it looks like that at the moment that's an existing one i tested so we're just going to paste our new link over there and hit save so we're good for now so there's going to be a few other things that we need to update to get this to run now namely we need to enable something called cross origin resource sharing so in order to do this we need to run through a couple of key commands and in order to actually do that we need to have the ibm cloud object store cli installed so to do that you can type in cloud object storage cli installation and this installation over here is going to walk you through how to actually get up and running so you're going to need to install the ibm cloud cli first and then the cloud object storage cli so again i've got a full-blown tutorial on how to install this but if you run through the installation steps on here it's going to walk you through how to do this now if we go through let's actually take a look so if we go to ibm cloud cli install so this is the first one so we've got to do the ibm cloud cli install first up so to do this we can use a browser to access the official ibm cloud cli release github repository and select the installer of your os to begin the download so let's select this and you can see there's a bunch of different installers so you can use mac os windows 64-bit windows 32-bit so we'll do windows 64. and you can see it's downloaded let's run this hit yes so it's going to up perform an upgrade so i've already got it installed so it's just going to upgrade so let's let it install and so you need this for the next step so this is actually going to be what allows us to this is going to allow us to enable something called cross-origin resource sharing but i'll explain this a little bit more in a second and then what we want to do is install the ibm cloud object storage cli so i'm going to show you how to do this so you actually do this through the ibm cloud cli and cli just stands for command line interface so it's basically it allows you to run it through a command line right so to do this next step what we need to do is run this command here so i'm going to show you how to do this so first up what you need to do is log into ibm cloud so i'm just going to clear this so to do that you can just type in ibm cloud log in and by default as soon as you hit log in it's going to ask you to pass through your email and your password and these are going to be the same details that you passed through when you went and created your bucket so when you created your bucket uh right up here i was sort of saying that you would have had to either hit log in or sign up when you hit sign up you would have passed through your email password your password so the details that you put here are going to be the same details in this particular case so you pass through your email and your password so i'm going to go ahead and log in okay so i've now gone ahead and logged in so i just passed through my email and my password i did it slightly different because i've got a different account but that's fine you just pass through your email and your password and you'd be able to log in now what we need to do is install the ibm cloud storage client so to do this we just copy this command here and paste it and so this is going to go on ahead and install that cli so we can just hit yes and we should be good to go cool so that's now done so we can clear this to give us a little bit more space and then if we go back to our tensorflow object detection repository or tensorflow object detection app repository that is now step seven done so we installed the ibm cloud cli and we installed the cloud object storage plugin now what we can do is we can just run this command here so what you need to do is change one single thing and that is the name of the bucket so the full command is going to be let me just zoom out a little it's going to be ibm cloud cos so cloud object store that's the cloud object storage service which you saw over here bucket dash cause dash put dash dash bucket so this is the name in this case the my old bucket name was called live long because i was doing a live long detection then dash dash cause dash configuration and then we pass through the name of the configuration file but i've already got that available for you so you should be able to work with that reasonably easily not reasonably easily you you won't need to make any updates to that or write your own configuration so i'm going to copy this and we're going to update this i'll show you how to update it so let's bring jump back into our app so over here just jump into the readme for now and i'm just going to paste this command let's just paste it in here so what we need to do is namely replace this bucket name with the name of our bucket name so if we go back remember we created the name of our bucket in this case it was called directions tfod so we can copy this and then just replace that there and then in order to run this command line so you obviously need to be logged into the ibm cloud cli which is what we just did you need to be in the top level repository or the top level folder of our tfod app so in this case we've actually got this file called cause config this is going to do what we need it or this is effectively what's going to enable cause for us now cause stands for cross-origin resource sharing so it basically means that our web app can access a url from somewhere else so namely you don't have this enabled because you don't want your web app to be accessing a whole bunch of stuff and namely we're locking down the buckets because we don't want them to be accessible from anywhere but in this case we do because we want our app to be able to get to it so we're going to enable it right and then in order to do that we're going to be running this command so this is going to be going to our cloud object storage service it's going to put our cause policy which is what we've got in this course config file and this is part of the clone repository so you don't need to create anything from here all you really need to do is log into ibm cloud and then just change or whatever the name of the bucket that you created was you need to replace that here so in this case it was live long before we're going to paste in the name of our bucket which is directions tfod so paste that there and then what we're going to do is we're going to run this inside of our command line or terminal so in this case i'm running inside of windows so it's brought up a powershell terminal but if you're doing this on a mac you might be doing it inside of a terminal it's just called terminal or bash terminal so we're going to run that command so let's just make sure we've got that cause config file we do so we should be good to go so let's clear this run that command and it looks like no issues there so it said okay successfully set course configuration on bucket directions tfod so we are looking good now now what we need to do is go through the rest of our steps so step nine is update the following line with the link to your model url so we've actually already done this right so remember we went into app.js and we actually already pasted the name to our custommodel.json file and this is what we uploaded into our bucket over here so this model.json file is our tensorflow.js model and it's going to be what allows us to run our web app using our model so that step 9 is now done step 10 is update the label map inside of utilities.js with your labels so what we can do is we can go into utilities.js over here and all we really need to do is update this label map up there so we need to update it for what names or what our labels actually are and the colors that we want them to represent so in this case we're going to go back to our jupyter notebook and remember our labels were right up here so remember we set up these labels so left right up down so we're just going to do the exact same thing here so if we go back into our app so left right up down and save that so you can add in more if you wanted to so you can add in additional labels you can add in additional colors and you can update this based on your specific app in this case we're just updating it for our different directional models and then make sure you hit save so that should be it in terms of all the stuff that we need to do so we've now gone and done step 10 now step 11 is just starting the app so to do that we'll run npm start let's try this out so again make sure you're in the root directory so tfod app and then what we can do is type in npm start and all things holding equal this should work so let's wait and see so this will automatically open up your browser to the link where our model is actually running and it should access your webcam now it'll take a little while to start up and compile but then you should be able to test it out so let's give that a second and then we'll take a look at our results okay so it looks like our react app has started and you can see that we've got our camera available so if we take a look it doesn't look like it is detecting now if you get errors like this and you're not actually getting detections i want to show you how to actually debug this so in this particular app from line 50 onwards you're actually getting your detection so this line here so line 50. so const obj equals a weight net dot execute async expanded and you can see how i actually built this up if you want to see the full tutorial for how i actually wrote this again i've got the code available and i can more than happy to share that with you if you're getting errors and it's not working so this is the way that i sort of normally process through this first up what i'll do is i'll change the confidence metric and in this case we've got our confidence metric available through this line here on line 61. so let me just check yep should be line 61. so we can change this number here so right now it's set to 90 percent let's just jump out of that so it's set to 90 so we can change that and drop it down so let's drop it a 40 and see if we get any detections nope okay it's just me pointing at myself then the next thing to check is sometimes tensorflow.js will change the order of objects from the converted model so in this case what we actually do is we expect the boxes to be the fourth item in the detection result array we expect the classes to be the fifth item and we expect the scores to be the sixth item but it's not always the case so if we actually type in console dot log this is the equivalent of print in python and if we type in a weight and then if we go through each one of these objects in the object array we can actually determine what is what so if we type in object zero this is going to print out the first object so if we take a look at that and let's just bring this side by side and hit inspect so this is going to allow us to see what object is what so you can see here that we've definitely got something in there and it looks like it's an array and each array has five values so that is not our boxes so let's try object one and so this is sort of how to work out where your objects actually are when you go to deploy this out so what you're looking for and this is a slightly different one so it's not that one oh actually it might be so this looks like so this is the non-normalized result so right now you can see that we're getting add boxes but they've got negatives in them you don't want that one so let's try object two and all i'm doing is i'm changing the number over here so you can see i'm changing that so let's take a look at this so this is okay so this is our classes so this is what we do need so you can see here that we've got one two three four so this represents the different classes that are available for our particular app so what i'm going to do is i'm going to change this line here so line 55 which has our classes we're gonna change that to object two hit save so now we've at least got our classes placed appropriately and then we're gonna go through our next value so we wanna check what's in object three or lit position three in our array or our results array and what's this so this looks like some this looks like non-post process box classes so you can see here that these are very low numbers so if we actually try to use these for our boxes well i'll actually give you an example right so this is object three so if we set this to our boxes let's take a look so it's getting detections but let's wait for it to run you can see that it's detecting all the classes in the top corner so it's really really small so if you set that value to your boxes it's not really going to work so let's try object four see what that is and so it looks like got a hundred that's just the count of objects that have been detected let's try object five this is looking promising array 100 okay so this might actually yeah so this one is or this particular set of arrays is our boxes so it should look something like this there should be a number between zero and ideally one so i believe it's x i can actually tell you correctly so these values represent your y value your x value your height and your width so in this case you've got all of your box values so this is what you'd expect your box value to look like so it's a bunch of numbers between zero and one so there shouldn't be any negative values in there if it is it's the i believe it's the non pre or non post process value so you don't actually want those so what we're going to do is we're going to set the value of the boxes array so line 54 equal to five so let's just minimize this right so now we should be drawing our boxes correctly oh and we've remember we've set our confidence metric really really low so let's bump that up because right now it's losing the plot so we're going to set that back to up to 90. so we're at least getting detections now have we saved that oh we haven't changed our scores so we've got to find where our scores are first as well so let's go and change that for now so let's zoom out a little bit there right now it's gone a little crazy so let's see what object six is this looks like it's just all right so it's not that so this i believe is an index value so we don't want that try object seven let's see what these values are okay these are our confidence scores so there should be a value between zero and one so i believe they're already sorted as well so the highest confidence is going to be the top value so what we need to do is just set this number equal to 7. let's do that and we should have resolved it now so again you just got to work through it and all right so if we take a look beautiful so now we're working so you might need to change this around depending on what the results of your object array come out as so again but that sort of shows you how to sort it through so your boxes are going to be a value it should you should have a number of arrays and each one of those arrays are going to have four values between zero and one so those represent your coordinates your classes are just going to be standard integers so it's going to be one two three and four or the number of different objects that you've got in your model and your scores are going to be the sorted array with the highest confidence value at the top now let's go ahead and test this out right so we've actually gone and done quite a fair bit here so we can minimize this expand it out and if we take a look we're accurately detecting up we go down we're getting down it looks like it's okay the coordinates might be a little bit off that's a bit better and then left right and left so left over my face but anywhere else it's it's detecting right so and then we're getting right over here and again reasonable confidence metrics but that sort of shows you what's possible right like you could do so much with this you could build apps you could build directional apps um you could do sentiment analysis via the web again there's a whole bunch of ideas so if you think of anything awesome let me know i'd love to hear your ideas now you can also change the colors of the different labels so in this case you can just change it by changing the color attached to each different class but on that note that is project two wrapped up so we've now gone and taken a raw object detection model we collected our images again using our image collection script we change it from left to right up and down and again the performance on left and right wasn't the greatest but remember we took images right over our face which isn't always that cool we then went and trained our model using our regular training script we then went and did a bunch of stuff as well right so we uploaded our model into ibm cloud to be able to host it we then set our cause value so we're able to access it from our app and then last but not least we actually went and updated all of our app details so remember when you're updating in here a couple of key things to do so you need to go into your app.js file update line 17 in this case it's line 17 but just look for the line that says tf dot load graph model so you update that with your hosted model update your labels inside of the utilities.js file and then you might need to play around with your object detection array so you might need to change these values but in this case we are pretty much done so for project number two so we've now got our real-time app up and running it's able to detect up down left and right pretty cool right and so what we'll do is if you want to stop that app you can just jump in to vs code and just hit control c um or it'll be command c on a windows machine or on a mac machine so that is our app done so you could do that with a whole bunch of different models so it doesn't just need to be your directions model we did it for the sign language video so again if you want to take a look at that video by all means i will link to that in the description below so you can take a look at that but now we are up to project number three so raspberry pi sentiment so we're going to keep this one relatively simple so we might try one or two classes so maybe happy and then like sad or something like that so we'll so obviously running object detection models on a raspberry pi does have certain things that you sort of need to cater for obviously i don't have a gpu on this particular one it's just a raspberry pi there's nothing fancy happening there just a raspberry pi now you can get a gpu extender like for example coral you can also get like jets and nanos or sort of linux style devices like that to be able to run it but we're going to use a standard raspberry pi box and test this out now i've got it sitting right next to my webcam so we're actually going to remote desktop into our raspberry pi to be able to work with it and we should be able to run similar detections to what we've done already in this case we're just going to be doing it from scratch cool all right so what we're going to do is kick this off and again as we've done before we're going to be doing it very much in the same manner so we're going to be collecting our images training our model the key difference between this particular project and the last two is in the first one we just used a straight up we used a microscope to be able to make those detections but the tfjs app we converted to tfjs to be able to run it now for our raspberry pi model we're actually going to be focused on converting this to tf lite and we're going to be using our specific model in a tf lite format so let's go on ahead and do this now what we'll do is we'll first up open up our image collection notebook so again nothing crazy there and this time rather than having so for our directions model specifically our tfjs app we had left right up and down here we're gonna have let's just try happy and sad so we're gonna update those so we're really just gonna have two objects now you could even try one object depending on performance and see what that looks like so again we're going to import our dependencies there's no need let's actually test our webcam to make sure it's pointing at me appropriately let's wait for that to open up cool that's all good so we'll move over here to collect our images and maybe we'll do like happy and then like sad so like sad is like our face covered so we've got really clear and defined objects because again the camera that i've got on the raspberry pi is not as good as the webcam that i'm using on my pc as well another thing to keep in mind so let's go ahead and do this so we've got our labels and we're just going to collect images of happy and sad we should probably clear out our existing images so if we go into tensorflow workspace images and then collected images remember we had the down left up and right different images so we can delete those we don't need them anymore now what we're going to do is collect our images for our sentiment detector so happy or sad so first up we need to create the paths and this is what step three does so that should have created our path so you can see we've got happy and sad then what we're going to do is collect our images so again we're going to collect five images but you could change this if you wanted to so i'm going to get into position and collect them so let's run this and we'll wait for our little pop-up so i'm a little away from the mic oh no it's running it's not happy right and then i think we're up to sad so so sad will be covering our face sorry you probably can't hear me i'm trying to see through my fingers alright cool all right that should be all of our images collected so let's go and take a look at what we've got so insider happy we probably should i love this photo this is great this is me just dumb struck okay so we're gonna delete that one and we're gonna delete that one this is okay this is now we want me smiling in them all right cool this is fine for sad so sad is us covering our face so we're going to do happy and let's collect some more images of happy so remember we can just isolate what we want to do so we want three more images of happy so let's do that so all i've done is i've just removed sad from our labels array or labels list if it's python and then we want three more images so we've got an even number of happy and sad images so let's run this and then run our capture again and i'm going to try to be happy this time i'm just wait okay cool so let's go and take a look at our images again this one yeah all right whatever we'll count that as happy that's fine that's fine that's fine that's fine okay cool so we've got our images now as you might have guessed it we're going to go and label our images so we need to define our label path as per usual spin up label image and let's go ahead and do it so we're going to go back into our images so workspace images and then collected images happy and do our labeling so we're going to say that this is me happy save that save that and again i'm doing happy in caps okay so pay attention to that because capitalization is super important or case is super important for this now again for the most majority of this course i've been doing it with a green background but you could obviously do it without a green background or a green screen i have just been taking the photos with the green screen but that's okay it will still work without it because you saw that we took it down and we tested it without the green screen now what we're going to do is switch into our sad folder and do the same so that's our already done happy let's do sad so let's capture everything to there so sad sad nick all right then we're going to do the next one and we're good and again i'm just keeping going through i'm making sure my detections or my labels are really as close as possible to correct sad and i'm saving each object again so this makes sure that we've actually we're not going to all this effort and then forgetting to save and just there perfect okay cool let's check him so that's fine that's fine that's one that's one okay cool so that's good let's just open up our happy directory because i wasn't sure if we saved all of those fine fine fine this is also another thing to know right so you can go back and open up the directory of images if you wanted to go and update a label for example so say for example we wanted to do microphones we could go and label the microphone and type in microphone i don't want to do it because it's going to screw up but in this case when we only want happy and sad for this but say for example we wanted to edit it like i didn't capture my fortune we can go and make that update and hit save to be able to go and overwrite oh it looks like i've just gone and killed that off let's go and make sure that we've still got that uh yep so we're all still good you sort of get the idea so you can open up a folder you can make those updates alrighty now as per usual we are going to go on ahead and save our move our annotations into our training and test folders i'm going to go back into our images folder i'm going to delete everything from our test folder and our train folder so these were all about directions and we are going to go and grab our collected images so happy so we'll grab four and four nope yep so those so again remember you got to grab the annotation and the image we're going to paste those into train so this will be a what now it'll be a 80 split again so again we've got one image left inside of our test folder and then we'll do the same thing with sad sorry if there's a little whining on the microphone i've got the raspberry pi running already so it is a little loud i'm going to copy this and then we're going to do our test images oh that's not right so these test images should actually be inside of train so let's cut those paste it into train cool so we should have eight images in train and we should have two images and tests so we gotta go get that other one so let's grab this sad one and let's paste that in okay cool we're good so if we go into our test folder we've got two images and if we go into our train folder we've got eight images pretty cool that you can build it like object detection model of just eight images right so again this is super powerful what we're doing here so happy i'm pretty sure we've already moved that so we can actually go and delete our folders inside of collected images we don't need those anymore okay now the next thing that we need to do is go back into our training model so we're going to go on ahead and train a new model so this time we're going to call our model sentiment and again it's not a full-blown sentiment model it's just a high-level object detection model but you could obviously extend this out and we are running and again make sure you're in your virtual environment so we're going to hit go through all of our path steps so we'll import our operating system set up our new custom model name set up our paths files we'll actually create the paths so now if we go in to our workspace models you can see that we've got our sentiment folder already set up now again we don't need to do step number one because we did that right at the start we set up our tensorflow models our model zoo and we installed tensorflow object detection now all we need to do is update this label map so under step two so we're gonna say what was it happy and sad save that and then we're gonna hit shift enter to create our label map so if we go and check that label map if we go into workspace annotations let's open it up inside of vs code that's looking good so we've got happy and sad we're good to go there and then we can create our tf records and again this step over here we can skip that that's the collab step and run this and this cell and this is going to should ideally create our tf records without issue cool so that's gone and created them as per usual we're going to copy our pre-trained model config to our custom model folder so let's do that then we're going to update it using these steps we can go and double check that it's gone updated so if we go into our sentiment folder and open up this pipeline as per usual let's open up one of vs code got the correct number of classes and if we scroll all the way at the bottom most importantly we've gone and reset the fine tune checkpoint the label map path input path so on and so forth so we're good to go cool now the next thing that we need to do is as per usual train our model so we're going to generate our command and then we are going to go on ahead and paste this into a command prompt so again as per usual we need to make sure we have the virtual environment enabled and we need to make sure we're in the top level directory so again from here you should be able to see tensorflow and tfod and then we're gonna go on ahead paste our training command and then we'll let that run and we'll be right back so i'm gonna paste it in and we'll be right back cool so it's kicking off we'll be right back all righty so that is our sentiment modeled finished training now full transparency i did run it into an error when i kicked off this training run now wasn't anything that we wrote in terms of code it was because all of the vram available on my particular pc was taken up now in order to solve that particular error all i had to do was stop my jupyter notebook which was consuming all of the vram so if i take a look i want to actually show you this error because it is pretty common particularly if you're working on a gpu machine if you see this area anywhere so q blast underscore status underscore alloc underscore failed or anything along those lines that normally means or you can see there's a similar one down the bottom so c-u-d-n-n underscore status underscore alloc failed so again that normally means that your gpu is completely consumed so if you go and take a look inside of your task manager and your gpu memory is completely consumed and you try to run the training run you're going to have a bad time so ideally what you want to do is stop anything which might be using the gpu at any point in time and then kick off your training again now in this case once i did that so i went and ran the training command again so you can see it sort of errored out and it stopped there went and ran the training run again and it ran successfully so right down the bottom again there's lots of lines so right down the bottom you can see that we've got our loss metrics appearing so step 2000 and our final loss was 0.212 and again if we take a look inside of our model folder specifically the sentiment folder we've got all of our checkpoints generated now as well okay now the next thing so because i stopped this notebooks i stopped notebook 2. so you can see that that's definitely not running now so what we need to do is start that back up so we'll close this one open it back up and then what we need to do is import operating system import these paths and then if we scroll on down we need this labels array this is important and then if we go all the way down to step 8 you should be able to reload all of your stuff now again double check what your latest checkpoint is in this case it's checkpoint three so we're fine with checkpoint three there so under step eight we can import our model so we're gonna test it out before we go and push it out to our raspberry pi so let's import that model and again i'm going to skip doing the image but we could if you wanted to um we're going to jump straight through to real time to see what our sentiment looks like so let's run this and cv2 is not defined so that's our error there and that's just because we skipped importing cv2 up here so if we rerun this ideally we should get some detection so we can close our task manager don't need that so i've just seen my webcam start up so i can see the little light on it looks like we've got an error here so category index is not defined so that's an easy fix we just need to re-import this so whenever you restart your notebook there's going to be a few things that you've obviously got to import so if we import our category index try this again this might actually error now because i didn't release my captcha but we'll wait and see oh no we're good all right so saying happy sad how's it looking sad happy pretty cool right so like that's a lightweight sentiment model so it's chasing my head around if i put my head down so what's it actually take for it to go to sad so if i bring my hands up so it's starting to get closer to sad if i go all the way so if i cover my face it's sad if i unhide my face it's happy so again you can sort of see that it's with these models you can pretty much build whatever you'd like this is why i kind of love tensorflow object detect or love the tensorflow object detection api it's really reasonably quick to do once you've got it set up you don't need to go and set up the stuff all the time so what's it looking like so it's still saying that's happy we get in position so that's saying sad so again you can play around with this but i love that it's chasing my head around so cool all right so in this case so we're happy now but we want to take it a little bit further and actually push this out to our raspberry pi so we're going to quit out of this by hitting q on our keyboard we're then going to freeze our graph so we're going to go through step 10. we don't need to go through the conversion to tfjs because we're not doing that this time so we need to do everything under step 10 and everything under step 12 in order to get this to work so what we're going to do is we're going to run through step 10 and freeze our graph so if i run all of those steps including the command we'll run this one from the command or from our notebook this should generate our export file which is over here which is over here and then once that's generated we'll be able to run the tf light conversion so again a lot of these scripts are sort of pre-configured you can sort of pick them up and leverage them let's give that a sec now while that's happening let's take a look at the steps that we need to go through in order to build our raspberry pi detection so again the third and final github repository that i've sort of made available and written up is the tensorflow object detection walkthrough with raspberry pi so again all the other steps that you need to go through to get this up and running are in here so first up we're going to start by going through our tensorflow object detection course to create our tf light files then we'll clone this to our raspberry pi so we can either clone it from our raspberry pi or download it to our pc and then copy it over from our remote desktop connection so in terms of this remote desktop connection i'm actually using something called xrdp which allows you to remote desktop or remote into your raspberry pi so it's actually pretty cool so it makes life a whole lot easier i can code on my raspberry pi from my local pc without having to actually go into the raspberry pi but all the code is actually still executing from the raspberry pi and so once we've done that we've just got to install a bunch of dependencies copy our converted model over and then run it so let's go on ahead and do this okay so we've now gone through and frozen our graph so we can see that in here so we've got our checkpoint our saved model and our pipeline file good to go there now what we need to do is convert it to tf lite so this is all going to happen inside of the tf light export folder so let's go ahead and run this and we should see all of our results pop up let's just bring this over to the side here cool so if we run everything under step 12 i don't know why i focus on putting that to the side all right so we're going to run everything under step 12 so tf flight script this this this this this this and this and ideally this should generate our tier flight export exports what you're looking for is a file called detect.tflight that's the core file that you're going to be able to leverage inside of our raspberry pi now i've already got a raspberry pi remote desktop into so i'm already remoted into it or hooked into it if you want to say that and so we'll be able to jump into that and take a look at that as well so let's give that a sec and then we should be able to kick things off okay so that's the first part of it done in a second you'll see so you can see inside a tier flight export we've got this saved model folder and once the second command runs you'll have detect.tflightmade available so you'll be able to pick that up and leverage it cool so you can see we've now got detect tier flights it's a little bit small there can make a bit bigger it's that file over there so if we take a look at our steps what we've now done and this is this these are the steps for the tfod rpi repository so the raspberry pi code in this particular case now what we're going to do is clone this repository so i'm going to jump into my raspberry pi now and right over up here you can see that i'm just remote desktop into it and i'm going to create a new file or new folder actually and we're going to call this sentiment and we're going to jump into that so we're actually going to open this up inside of vs code so i've got vs code installed on this raspberry pi cool nothing in there yet so we're going to open up a terminal and this is a bash terminal and we're going to type in git clone and then paste that link uh did we have what link do we have there oh we already had a git clone so we don't need to type in git clean let's clear this let's make this a bit bigger as well so we can see what we're doing right so we're going to paste that command into here so git clone and then the tensorflow object detection repository so you can see that we've now got that or it's starting to come through so in there we've got a couple of things so we've got the detect pi file and this is the main file that we're actually going to use to run our detection so all of our good stuff is actually in here then we've got an existing detect.tf lite model so we can actually run this off the bat without actually having to do anything else so in this case it's just a simple basic hello model now before we actually do that though there's a couple of key dependencies that we need to install now they're all listed in this readme here so if we actually bring these up you can see that we've got a bunch of files that we actually need to install so specifically these are all required to be able to work with opencv so and then the last one is actually installing this python tflight runtime so this is really really important to have these dependencies available because it's going to mean the difference between your model running and not running so what we'll do is we'll go through each one of these and we'll install them so i've already got this configured but it's all good let's go ahead and do it so we're just going to paste the first command so let's copy this so that's going to install opencv python looks like it's already installed then we'll install the so all of these are dependencies for um opencv so specifically we need to install these as well in order for this to work so i'm just going to copy these and install them so it looks like package all right so we've got to go through all of the rest of these as well so let's go ahead and install these my copy and paste doesn't seem to want to work so we're going to have to do this manually cool so that's the next one done and again i'm just going through each one of these packages that we need to install and i'm going through and installing them so we'll just copy and paste and i sort of wanted to show it to you from scratch because that at least you can sort of see how to actually get this up and running this next one done and it looks like a lot of these are already installed so it's going through them reasonably quickly it's not actually installing it cool that's that okay so those are all of these packages installed so we've now done all of these right but we've got these couple of packages left to do so let's go ahead and do this one cool and then this one awesome then you should normally do a generic update so we'll do the update and we'll install our tier flight runtime so this last one is the most important one this one over here so this is actually what's going to allow you to leverage the tier flight runtime without explicitly having to go and install the full tensorflow object detection api so once we've got this line done we're pretty much home free so let's let this run for a sec and then we should be good to go once our update is done cool that's done now we'll do the last one so this is going to install our tf light runtimes as i was saying this is the main package which allows you to leverage a tf light model and specifically our object detection model within our raspberry pi so we're going to copy this and again these are all available through the readme so don't fret if you haven't picked up exactly what the command is that's fine all going to be available through there cool so that's already installed now now rather than running our custom model first up i want to show you how to leverage the model that's already in there so you can sort of get a feel for what this is going to look like so to run this let's just clear these commands we just need to cd into that particular directory because right now let's just type in ls right now we've got this subfolder right so tfod rpi so we're going to cd into that and then if we type in ls now so we've got our main folders so to run this all we need to do is run python3 detect dot pi so let's go ahead and run that so python 3 detect dot pi and all things holding equal we should okay so we've got our object our detection box opening up and this is just a straight up hello model so you can see it's a little bit slow and again it's running on a raspberry pi so it's not crazy fast but you can sort of see that it's detecting hello it's detecting my face and again it's not the most accurate model but you sort of get a feel for what's possible here right if i bring my hand a little bit closer and start to see that's making detections on our raspberry pi right okay so if we hit q we can close that as per usual now what we want to do is we actually want to do this with our custom model so in order to do that we just need to go back into our where we actually saved our detect.tflight file copy this and paste it into this repository so what i'm going to do is i'm just going to paste it down and again if you're doing this without idping into it you could definitely transfer it like save it up into google drive or a box folder and bring that over in this case having idp available makes it a lot faster so we're going to copy and then replace it so we'll overwrite cool so that's good and then what we need to do the last thing that we need to do specifically is we need to update this labels.txt file so really all we need to do here is update it for our custom labels and remember if you remember correctly we had two custom labels they were just happy and sad i'm just double checking the order that we had them in so remember we had happy as id1 and sad as id2 so let's go ahead and do that so we're just going to make happy the first object and then sad the second object so we'll hit save now all things holding equal if we run detect dot pi again so python 3 detect.pi we should effectively be running our custom model now so let's clear this let's try that so in order to run this we're just going to run python3 detect dot pi and again we'll see if we get any detection so again we haven't trained for that long so at the moment is a little bit slower and again we're going through idp so it doesn't look like we're getting any detections and that's indicated based on what you're seeing down here so it looks like we've got a random one looks like they're real reasonably low quality so what we might want to do is drop our detection threshold so if you're going to detect dot pi you can actually change this up so actually in this line here so on line 68 if you change this so let's try dropping it to i don't know 20 percent for now so by changing that and then what we can do is rerun this so rerun our command so we're going to run python 3 detect apply again so let's see if we get some detections now so it's looking like we're getting detections already okay so we're getting one big sad we're getting happy it's looking like there's a fake detection over there so there we go it's detecting me as happy and chad at the same time but you sort of get the idea right so you could definitely train this for a lot longer you're getting these false detections over here which isn't that great it was a detecting sad so let's have a look so it doesn't look like it's that great at detecting sad you sort of get the idea so you can obviously train this for a lot longer performance tested we only trained it on eight images as well so obviously this camera on the raspberry pi is going to be different to what we trained on so you're going to get different performance when you do crossover so that's another thing to keep in mind but if we go happy you can see that it's definitely picking us up there is a little bit of lag when going through idp so if we go to sad let's see if we can at least get one sad detection move over a little lots of happiness weirdly enough let's try dropping green screen it's got one big sad but all right so it's detecting sad but it's somewhere up here so it's not the best result i would say drawing a big box around me as being sad if we smile again so it's not the best generalized model but it sort of shows you what's possible right so that gives you an idea of how you can go about making your detections on raspberry pi so again looks like it's pretty happy detecting me is happy maybe sad not so much so if we wanted to performance tune this we're going to add in more images of me being sad in that particular case under different lighting conditions i'd probably drop the lighting because the feed from the raspberry pi isn't as bright as the webcam i've got on my computer but you can sort of see that there are ways to tune this and there's ways to go and deploy your model and a whole bunch of projects that you can sort of take away to go and build yourself now performance for the sentiment model was okay but during editing i couldn't help thinking that this could be so much better so what i ended up doing is performing some additional performance tuning to ideally eek out a little bit of additional performance out of this model my ocd was in overdrive so i decided to let it run free and try to get this performing a little bit better this is what happened so during editing i couldn't help thinking that i really wanted to get better performance out of that sentiment model so again as we went through in step six or part six of this course i went through a performance tuning step so i added more images of the performing classes that weren't performing so well in this case added more of both our sad sentiment and our happy sentiment and i ended up training our model for longer i didn't really need to change the architecture but if i hadn't had a performance boost that would have been my next step so let's take a look at what it is that i actually did so first up i added a number of additional images so all up you can see that i added about 20 additional images and if we actually open these up you can see i've got images at a number of different angles so this gives or this boosts our chances of having a model that generalizes well and i also took down the green screen to ideally see if that had any impact on whether or not our performance was going to boost or not boost so if we keep taking a look you can see i've got images at a whole bunch of different angles and i also had some images of me looking happy as well and you can see i'm moving my head around and this gives our model a better chance of actually being able to detect our head with the green screen up but also with the green screen down so you can see a bunch of different angles and again i went about the same labeling step so all i did was i created new images went and labeled them again that was the first thing that i did the second thing that i did is i went and trained a model for longer so if i actually show you i ended up training i kicked off the training for 10 000 steps so you can see 10 000 there but i actually stopped this at about 5600 steps i was kind of getting impatient and wanted to just test it out so i trained for an additional 5600 steps now what that gave me is a brand new model so i didn't retrain the existing model actually went and performance tuned and created a new model so if you actually take a look exactly as what we did during the performance tuning step so if i went into workspace models you can see here that i've got a model called sentiment two and my checkpoint went all the way up to checkpoint six so you can see that there because i trained for longer i had a checkpoint with a greater index again i went through the same steps i went and converted it to a frozen tf like graph i then went and converted it to a tf light model and then i brought this into my raspberry pi so now this particular detect.tflight model that we had here is the model that's been trained for 5600 steps and it's been trained on 20 additional images those were the core changes that i made here now if we try running this model again let's take a look at performance so if i type in python 3 let's bring that up a little bit detect dot play you can see that we're detecting happy we're not getting all of those false detections so for smile we're good now let's test out sad because remember sad wasn't working right so that was the key thing that we wanted to test out so if i put my hands over my face now you can see that we are detecting the fact that i'm sad so again it's a much better performing model if we take the green screen down let's take a look at performance it's there as well so right now it's performing well with the green screen up but what about if we took the green screen down so again still detecting that we're happy and if we try out sad again so if i put my hands over my face you can see that we're getting significantly better performance now and so you can see the little label in the bottom corner but again performing way better and all i really did was i added those additional images and i trained the 5600 steps so you can see the power of performance tuning and again just because you get not the best performance first time around doesn't mean that you can't go and improve performance after that to be able to generate better results so that really shows you the power of performance tuning in this particular case editing nick out now on that note let's do a quick recap so the last and final project that we did was sentiment analysis so we went and collected our images using our standard image collection capability we went and labeled our happy and sad detections we then also went and created our updated label map and trained our model but the core difference between this and the previous two projects is we obviously went and converted it to tf flight so that we had that capability there we then went an rdp'd into our raspberry pi saved the model and ran out detect the pi script to be able to detect if we're happy or not so it looks like again you could performance tune this and you could make this a lot better with a lot more training a lot more images so you've got the possibility to go and make this better and let me know if you do i'd love to see if you build a better sentiment analysis app it'd be great to see i think it's something super useful um and on that note that wraps up our third and final project so we went and did the first project which is our microscope detection our second project which is our directional model and then our third and final project which was our raspberry pi sentiment model and also on that note that about wraps up our object detection course so hopefully you found this useful and if you've got any feedback comments or need any help by all means hit me up in the comments below thanks and that about wraps it up thanks so much for tuning in guys hopefully you enjoyed the full object detection course now please do let me know if you have any problems at all if you need a hand by all means hit me up in the comments below but also join our discord channel so i'm really active on there so if you've got any questions need some help or want to join the community by all means do jump on there thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 291,336

Rating: 4.9763479 out of 5

Keywords: object detection, tensorflow object detection with python, tensorflow object detection, object detection tensorflow, object detection using tensorflow, object detection raspberry pi, object detection tutorial, object detection python, object detection tensorflow 2.0, tensorflow object detection api, tensorflow object detection tutorial, tensorflow object detection raspberry pi, tensorflow object detection tutorial for beginners, tensorflow object detection api google colab

Id: yqkISICHH-U

Length: 325min 41sec (19541 seconds)

Published: Fri Apr 09 2021

Reddit Comments

[u/v3ritas1989]

Ah damn... how am I supposed to get any actual work done for the last 5 hours of this workday

[u/nicknochnack]

Hey guys OP here. Been messing around with TFOD for a while but realised there wasn't really anything that brought full stack object detection together with some practical projects. Put in a bunch of long weekends to build up this course on Object Detection. Would love any feedback!

[u/GSBattleman]

Haven't had a chance to look at it yet. TFOD is quite computing-heavy, what's the backend you use? A Nvidia GPU, Google Colab, something else?

[u/KommandantJackal]

Stop giving me things to add to my youtube watch later

[u/nikowek]

Excuse me propably rude question, but what's your motivation for those awesome courses... made free of charge?

I mean, great job and I love it, but I wonder what's behind. I am looking for strength to do similar as blog posts.

[u/sblfc1]

Jeez 5 hours! Thanks a lot for making this, gonna try make my way though this today, this plays perfectly into a project I'm working on.

[u/gradi3nt]

I need to find objects in microscope images. I haven’t watched any of your video yet. Do you think this method could be adapted from camera images to microscope images?

[u/mattm220]

I’ve only taken a few classes in programming: C++, Python, and MATLAB (if it counts); but I’ll definitely give this a look later! I’ve got an idea for a project this would potentially help a lot in.

[u/InfiniteBlink]

Could you use a raspbery pi to do some of this? I have this weird little project i want to do.

in my backyard there are squireels, birds, dogs, rabbits, coyotes. I'd love to have a count of the number/frequency of visits by these various types of animals.

wondering if i can have some sort of image capture whenever the frame changes by x amount, takes the pic, sends it to my server for the image recognition. Would that work in theory?