🔴 Sign Language Detection Tutorial with Tensorflow Object Detection | LIVE STREAM

what's happening guys welcome to our first ever live stream so we're going to be testing this out and seeing how it goes now again as per usual by all means if you've got any questions or want to chat about anything just hit me up in the comments below we can have a chat if you've got any errors you want to discuss or anything that you want to go through again hit us up in the chat i can see a bunch of people having a chat already oliver let us know what your error is i'm more than happy to take a look so in this live stream what we're going to try to do is try to replicate our sign language detection model so we want to try to do what we did in the i guess obviously the main video but we're going to do it live so and ideally i'm hoping we get some errors so we can take a look at how to solve these as we're actually going through okay without further ado let's kick this thing off and let's take a look at what we're going to be doing today so again we're going to be focused on jupyter notebooks and we're going to be using or relying on the core real-time object detection tutorial that we already used previously so everything that you see in this particular tutorial is very much going to follow what we did in the actual live video so maybe we'll give it a little bit of a circle wait for everyone to flow in i did kick things off pretty quickly what's happening gustavo how you doing raj welcome to the live stream hopefully you're gonna enjoy this give it a couple more minutes and we'll see so 1201 we'll kick off 1202 what do you say what an internship in silicon valley that is sick awesome work man that is nuts hey how you doing joak sorry i've got my chat over here normally i'm presenting at the same place my chat a little bit weird what's happening miles how you doing welcome to the live stream definitely we're gonna test out this live stream see how it goes i'm thinking it'd be useful to have like a way that i can chat to you as well but i don't know we'll wait and see how that that all turns out yeah a lot of people have been asking for a quick chat i think it's going to be useful like even if we're just talking generic data science how to do stuff um i think that would be useful mick jagger what's happening man i remember we were talking a while ago awesome that you got it up that's sick tisharaz i think that's how we pronounce it how you doing what did you end up using it for by the way mick curious all right what's the time 1202 awesome poop that is awesome i think you're using mask rcnn as well so you are using something a little bit more advanced i think i'm going to do an image segmentation tutorial sometime soon again just got back from holidays trying to get back into the swing of things and start building up some stuff okay with just the asl alphabet but with a glove oh okay so you just had the hand in the i guess in the camera that way that's pretty cool we explain how to work more within an industry environment i guess on that question raj were you more to i get roger's question was um do a video where you explain how to work more for in an actual industry workplace do you mean like how i do this from an like a production grade staging or like how i'd actually something along the lines of what questions to ask in team and how to navigate coding oh okay all right so how to sort of i guess set yourself up for success in that way as a new coder or a new data scientist muted i guess stepping into the industry is that sort of where you're going with that i guess that might be an interesting video so how do we actually get started with development how do we actually get started as a data scientist through different colors and shapes with the glove for a high okay interesting i don't know i'm gonna try to do i'm sort of balancing object detection and reinforcement learning work at the moment obviously there's a lot more reinforcement learning videos that i'm putting out at the moment but i've got fret nut i've got a bunch more object detection tutorials coming i think i've currently got 30 lined up alrighty on that note let's kick so of this let's see are we back guys can you see me now hit me up in the chat if you can see my screen at the moment i'm hoping we're back okay that looks like we're back all right yeah no idea what happened there literally the computer just died like uh there you go at least we're testing out our first live stream and seeing how it rolls all right while we're um while we're up let's see how long we can keep the computer up and running um let's kick things off so again so what i was saying we're going to kick this off and try to recreate the sign language detection model so let me switch back to my screen all right cool so what we're going to do is we're going to clone this repo so again i'm just going to open up a command prompt i've got a folder in my d drive i believe it's called live stream so let's take a look at that live stream so nothing in there at the moment actually i wanted to share these files with you as well because if you're having errors what i realized is a lot of the errors that people are getting and i'll probably put this up on github and share it once once this live stream is done but basically a lot of the errors that i'm seeing people encounter are pretty repeatable and i'm eventually going to sort of compile a big error list so if you get errors you can just run through see what's happening and then go through that alrighty so we're going to close that let's go on ahead and clone this repo so i'll just copy this folder so this is the real-time object detection repo that we've got up here not the sign language one so what i actually noticed is that if you're using the pre-built sign language one the label image package in there does have a couple of errors so if you try to recreate it it kind of screws up sometimes so we're just going to go from this real time object detection tutorial and clone that so i'm going to go into live stream and clone it so git clone paste that in there all right so now if we open up that folder so we've got our real-time object detection stuff in there so you can see we've got all our tensorflow folders generate tf record our workspace some images no it doesn't look like we've got images in there we've got we need to download our pre-trained models and that might be there okay so let's kick things off so we're going to open up a new jupiter terminal now and let's do this so we're going to go into our d drive hit jupiter notebook cool that's jupiter open up over here all right and again i'm going to try to do this as live as possible so um i haven't actually gone and prepared anything before this i want to see how this looks like when i'm sort of in your seat and trying to do this for the first time mind you i do have the object detection libraries installed so if you are having troubles with that by all means go through the installation video i'll log it back in and you guys can take or i'll link it back in the description below and you guys can take a look so we're going to open this up open up tutorial and again this is pretty much the same tutorial that we worked with last time so again no real crazy stuff in here or no real crazy stuff that we're going to need to change either now the one thing that we do need however is the image collection code so i'm going to create a new jupyter notebook and we're just going to copy this down from i believe i had it in the sign language repo so let's open that up it might be this one yep so in here so there's a drop in a notebook called image collection that we wrote in a i think it was the face mask detection tutorial so let's copy this code so it looks like no issues there we'll call this image collection so this first section is really just importing our dependencies and then we're setting up our images path and this images path is real you can name it whatever you want but really this is pointing to let me show you so if we go into so tensorflow workspace images and then from here we just need to create a folder called collected images because i think a key thing that people are sort of getting errors with is if you don't have the folder named when you try to actually write the image out to that particular folder it's going to cause errors it's all going to block up you're not going to you'll see your webcam activate but you won't actually have images written out to that file part so you need to have collected images and hun double check a billion times that the name is the same so i'll just paste it in there that looks all good okay all right and then we're going to copy the labels that we want to collect i don't know if we'll do five what do you guys reckon how many labels should we do five less put five in there for now uh and then we're gonna copy this code here which is our core code that we're going to need to actually capture our images but we won't run that just yet and then i don't i think that's all we need from there okay so we've got our let's take a look so let's quickly step through this code again i want to document it a little bit better so this first section here is importing our dependencies so we're importing opencv which is import cv2 five sounds solid all right yeah cool we'll do five uh all right so opencv import uuid so uuid is going to give us our what's that going to give us it's going to give us a unique identifier that we're going to use to actually name our images we're then going to import operating system and we're also going to import time so where we're using opencv is down here we're using it to connect to our webcam so if you're doing this in a system that doesn't have access to a webcam it's probably not going to work because you're not going to be able to access your webcam if you don't have a webcam so you need to have a webcam key point need to have a webcam so uuid is being used down here and this is really just the naming of the images so by having a unique identifier it means that you're going to be able to write these out there's not going to be any issues you're not going to double up and have conflict when you actually try to write those images os just gives us some directory sort of structures and then time is basically we're only using it for one line and this allows us to sleep between each image that we're actually collecting so rather than like powering through and trying to collect all the images at once we're actually taking a two second break between each one and allowing us to move position so ideally when we're going through our different labels we need hello thanks i love you and then i think it's yes and no something along those lines so what is the asl sign for yes so yes is like this and no is the two fingers all right cool so let's just make sure we've got all this good so we've got our images we've got that okay so this is going to throw an error so this is actually really good so as of right now if i don't change any code this is actually going to throw an error and this is because this particular line right now is connecting to our video capture device 0. now video capture device 0 is already taken up by the live stream so if i actually run this i'm actually going to get an error so let's try running that and see what happens hopefully the live stream doesn't crash again but we'll see so it'll take a while and if all things holding equal oh so it is connecting oh no it's connected to the second one so it's renumbered the devices so now it's actually working okay so this is actually connecting to my second device so let's pause this for a sec okay so that looked like it was fun and that was fine because so what actually happens is so maybe it's going to be the other one that fails out so if i try i want this to arrow guys just so you can sort of see it who knows maybe they'll both work okay there we go so that one's failed out so what's actually happened there is because the camera is already being utilized you're seeing that it's going oh image empty unable to write it out now this is because we're trying to connect to a camera which is already being utilized so if you sort of get this what you can do is either make sure one that your webcam is available to use otherwise what you can actually do is connect to a different webcam restart your kernel so what i normally like to do is i'll type in cap.release add in another line release and then cv2 dot destroy all windows sounds brutal so that's gonna get so that's gonna release our webcam and it should allow us to go on ahead and start capturing images again so in this case that's all fine so what we're going to do is we're going to set this back to zero so in this case zero is the appropriate device and we'll start collecting our different images so i'm obviously behind the green screen so we'll see what performance looks like in this particular case but we'll give it a crack nonetheless so i'm going to jump back in and just delete this folder so we don't get that error actually let's take a look at that error so you can see here that it says a subdirectory of file of tensorflow workspace images collected images hello already exists that's just a warning so it's not actually an error so you can actually keep powering through that it's basically saying that the folder already exists we can't recreate it no issues you can run with that so what we're now going to do is actually go in ahead and collect our images so let's run this so in this case we're going to go through hello so hello thanks thanks i love you yes no so let's do it let's wait for a little pop-up as you can see our folders created that hello folder on the right hand side should pop up in a second there we go all right let's get in position so hello that's a little bit laggy yep cool so got a bunch of images for hello keep moving around so ideally you want to get images in a bunch of different positions so this is going to give you a better chance of actually getting um some good quality images and you don't need to hesitate if you've got bad images just delete them out and and take new ones so in this case we're switching to thanks now so we're collecting images for thanks it looks like we're moving it's looking good try to get the mic out of the way keep moving around that's looking good and so if you wanted to you could sort of get rid of that that break between the image capture but i just think it makes life a little bit easier then we're going to label images for i love you how are we doing for time 12 90 okay okay so we've got i love you put up a little higher collecting images for yes now so you can see the yes folder is now popping up here so you can sort of estimate that we're now up to that particular image capture and again whilst we're doing it for sign language this sort of same structure can be used for a bunch of different tutorials i saw like an awesome demo video on linkedin the other day of someone that had used the code to actually detect which way waves are breaking and i thought it was like absolutely awesome like that's such a cool use case um so now we're switching to no so it looks like we're getting a little bit of lag that's still okay we'll just make it obvious that our hands there go a little bit higher as per usual and again we're just collecting images though nothing like crazy here should almost be done okay i think we're done so if we take a look in our no folder we should have what's that ten images okay cool we're good we're done also another pro tip so if you've got this frame here so remember how saying that you can run cap.release and cv2.destroyer windows this will release your camera and it'll actually close out this frame so if i run that cap.release cv2.destroyer windows it nicely closes out the frame so rather than having to like crash the kernel and sort of destroy it that's just an easier way to close it down okay so in that case now if we go into our collected images folder which is remember where it's going to be written out so if you're getting errors where it's not your images aren't being right or aren't being written out please 100 check it 100 check it 100 times that you've actually got this full folder defined so tensorflow which is this workspace images collected images and then from there it's actually going to start generating images i think people might be potentially skipping the collected images folder which is why stuff isn't being written out but as soon as you add that it should all be good okay any questions on that or like again hit me up in the chat is there anything that you guys are sort of struggling with on that oliver i'd love to answer your question but you got to tell me what you need if there's any like the model like we can actually share the model checkpoints if you'd like to take a look at those as well we'll also need to investigate when the computer crashed originally that's hilarious and not hilarious so as always things things tend to break okay so what have we done now so we've now gone and collected our images we now need a label okay so labeling so again people tend to get a couple or get stuck with the labeling and i think this is because the label image package can be a little bit finicky to work with so in this case we're going to do it from scratch to make our lives a little bit easier so if we go to github and i think we're just searching for label image we can grab that which is this so this is the label image package all right we've got questions let me copy my line data pro w7 what's happening welcome no augmentation necessary for this case i don't think we need too much augmentation it tends to work okay and you can actually have augmentation pipelines inside of your config so i think if we open up config pre-trained models so this was puberty's question so let me open that up yeah so you've got data augmentation here right so it's actually built into the the config that's a really good question though is this live chat going to be uploaded i think the live chat gets uploaded i don't know first time around we'll give it a crack all right so that's the wait what are we doing so we're doing label image okay so what we're going to do now is the steps to install this are actually in here right so if you need to install this on a linux machine you need springfield you'll perfectly fine data pro i know you're a gun you'll work this out but again if you've got any questions got by all means guys hit me up in the chat um i'm more than happy to answer this live or if you want to drop me a line in the comments but uh also hit up the discord server because we're chatting on there all the time as well okay so what we're going to do now is we're going to install label image so what we first need to do is clone this so we're going to open up a new command prompt and we're going to go back into our live stream folder and then we're going to go into real time object detection so in there we've got our so we're right now currently in here so we're in tensorflow image collection tutorial so we've got these three files what we want to do is let's go into the tensorflow folder and we'll clone it into there so we're now effectively in here and what we're going to do is type in git clone paste this link boom this might take a little while to download um but it or to clone but you should be good so let's wait while that clones any more questions what are you guys thinking discord i need to plug it here real quick so i can join let me get a link let's find the link so we go to youtube uh we're gonna have to try to find it now discord discord discord nope here we go here's a link i'll shoot it there okay so we're good with that all right so how's label image coming on so it looks like it's still downloading but we should get there also let me know what you guys think of the live stream format first time trying it out um i do a lot of presentations so i think this is probably easier than actually editing a video but i'm still happy to do the regular videos i think i will keep doing that for a while okay so that is label image now cloned so if we go into our folder now you can see that we've got label image so what actually happened and one of the things that i realized is inside of the what is it inside of the full-blown live or live sign language detection repo that i've got up there and i probably need to fix this up go to sign language sign yeah this one so what's actually missing is this resources.qrc file so you actually need to clone it directly so rather than going in ahead and grabbing this repo you're better off just grabbing it directly from the main repository and again i'll link to that here so this is the main label image repository so you probably want to go off with that and that's because this resources.qrc file i don't believe is in my one so what we're going to do now is we're going to set this up and then kick off label image so let's go on ahead and do that okay so we're going to go into our label image folder and then we're just going to be running these commands over here so the ones that we need are the windows ones because i'm on a windows machine so it's this command here that you need it's hidden between a bunch of stuff but we're going to need this pi rcc command so if we copy that and run that looks like no issues there and then if we now go on ahead and kick off python label image.pi we should be able to start this up all right what are our questions is there a reason you chose object detection over yolo v54 oh okay so this is a good question right so tensorflow model zoo so there's a bunch of um different detection or different pre-trained models that you can actually use so i think i've just used ssd but like so these are the models that you're actually able to use for tensorflow object detection just make sure this is yeah tensorflow 2. yeah so there's a bunch of different types that you can use as well just um so i've obviously used i can't even remember which one we use now i think we used ssd one of these ones somewhere around here and i think i chose that because it's the fastest so you i've obviously optimized for speed but there's obviously sometimes there's a bit of a trade-off between your speed and the performance of the image versus the accuracy so right now i'm working on this tutorial right down here so mask rcnn and it takes a ton more time to train so rather than as going really really fast and being short and sharp that one actually takes a ton of time a ton of resources to actually kick off so again you've sort of got these trade-offs i just realized i'm coding behind my head let's put this up here yeah so you've got trade-offs so i've chosen the one that's that's obviously optimized for speed which i believe is this one yeah 320 by 321 yeah okay so we're now inside a label image what we want to do now is actually kick it off and then start labeling our images so let's run python label image and you can see we're open successfully cool all right so that's a good first step what we now want to do is open the directory of our images so in this case we've got a bunch of images so let's go to d drive live stream real-time object detection tensorflow workspace images collected images hello god i went through that way too quick all right so basically our format is we're going into our tensorflow folder so whatever your top level folder is then tensorflow workspace images collected images and then you're going to have your five folders and each one of those folders represents our different classes so we're going to have hello i love you no fan thanks and then yes and i think a bunch of you have also asked about action detention i am working on it so i am also doing some body analysis stuff as well which is actually pretty cool i found it fascinating so again those tutorials are definitely coming in the coming weeks so we're going to select our hello folder and then you can see our images have opened up down here and then what we're going to want to do is we're going to label so we'll just hit w to label we're going to do that and we are going to say this is hello and then move to our next image and what's happening here i believe we need to save this okay so we're going to save it and then hit save move to our next image which doesn't seem to want to work when i tab through so let's just go down here then w again so hello again save next image so we're just going to go through label all of that images that's hello go to our next one w and so like we're obviously doing it for sign language but you could do it for like a bunch of other stuff right so like if you wanted to do it for like i don't know like phone detection or if you wanted to like label random objects like a bottle or something you could definitely do that so what you're seeing here is the sort of the overall process for object detection but you could definitely apply this to a whole bunch of different use cases so we're going to go to w do that hello oh we got to save these as well right and we didn't save that one so that's my bad just make sure these have all been saved so those looking good okay next one i think there's a way to auto save doesn't look like it's working okay that's fine let's do the rest so i'm just hitting uh and so this image is crap right so like we're just going to delete these ones this one's fine so let's do this one so let's go ahead and delete those images while we're here so we're going to go back into our tensorflow folder workspace images collected images hello so there are those ones where so you can see here that these aren't exactly great images so we're just going to delete those and so we should have effectively 16 images so we've got eight images and then we've got a labeled files so we're good there all right now we're just going to go through and do the other classes so we're going to open up our next directory so i love you hit cancel and then open those and again same thing we're going to do i love you so let's check our comments yeah we're definitely going to keep this live stream so we're just going to post it up as well so you guys can check it out is it possible to add notes or so one of the questions we've got here is it possible to add notes almost like descriptions for gestures detected by the system and have them drawn onto the screen too yeah so i think you can extend this out so if you take a look at the detection history tutorial that i did that was sort of the the idea behind that so you can actually extract these gestures and if you wanted to you could add in additional drawings on the screen as well um so yeah you could definitely do that all right so we're now going to label our new class so in this case it's i love you sorry the text is a little bit small there so we're going to label that hit okay i'm going to go to the next so remember we've got to save save that next one the next button doesn't want to work so i love you here again save next one just checking the time there guys alright so i love you save i'm just saying i love you guys pleasure happy having you here alright so we're going to save this next one i love you save next one [Music] boom save god look at my facial expressions that's a bit of a shocker all right save and i'm just hitting w so you can actually just hit w on your keyboard as well i probably should have called that up you can just hit w and it will open up that rectangle annotation box you can also hit a and d to go backwards and forwards but it looks like that doesn't seem to want to work today so that's fine stuff's broken it happens alright so we're going to go to there hit w and save and then our last one okay i love you and then save cool all right let's double check it's fine fine fine fine fine fine okay cool so that is i love you done let's open up our next directory so we're gonna go to no so this is obviously no now save save save okay w and you can see here that my um annotations are a little bit sketchy so you can actually amend these say i wanted to bring it in a little closer and hit save so that's going to give us some better annotations the next one and next all righty that's all of them done so it looks like let's step through they're all looking good you can see that we've got our annotation still appearing okay we're good there so let's create open up our next directory so we've only got two left thanks so we'll call it thank you and we'll just bring this in a little bit and we'll save that cool all right that's thanks so again i'm just naming each one of these classes differently as i'm going through i'm updating the name of the classes depending on which folder i'm working in so let's make that a bit bigger save it so one of the cool use cases that i was actually looking at was using this same sort of flow but for microscope object detection so you could do things like circuit board defects cracks in particular pieces of machinery so that might be a cool one let me know what you guys think of that that might be another interesting one that we can do so let's power through and try to get all these labels all these images labeled next one and as per user rule just make sure all of your annotations are still there so otherwise you're going to get errors so that's good okay cool so all of our images look like they're working fine now let's do our last one so in this case we're going to go to which one haven't we done yet so it looks like yes is the last one all right let's check comments so installing object detection model so that's going to so dilshan that that'll actually or the tutorial is already sort of on the channel so take a look at that and let me know how you go um again if you get errors by all means just hit me up i'm happy to help uh could have been misinterpreting hold on so like images that don't have the thing you're trying to detect like say images of you with no hand in the frame do you want to elaborate a little bit on that for me mick like like do you want to say not a certain class or like what are you actually thinking there not detect anything since nothing's there um oh quarry so on your bit about speed of the prediction so you can actually slow it down um so once i get to the visualization bit i'll show you how to do that uh what are we labeling now this is yes right all right so yes oh negative images for ssd oh so i guess what you could do with that is um you could have like an image classifier and i guess you could say that i guess you wouldn't even really need a model for that so if you don't detect a particular thing in your image you could just say no objects found and so you could just do that with your results right you could do some pre-process post-processing for that labeling takes a while doesn't it alright so let's keep powering through let's do that save next one save next one save almost that guy said that one okay that's our images labeled so again as per usual check that they're all labeled it's looking good looking good looking good looking good all right cool now close label image that is now done 20 there 20 there and 20 there okay now the next thing that we want to do is we actually want to split these up into train and test now you could apply a much more scientific method to this compared to what i'm going to do but in this case i'm just going to grab some of the stuff and grab a bunch of the images and make sure you grab the image as well as its associated annotation because if you don't grab the annotation then you're going to have a bad time so what we're going to do is we're going to grab let's leave let's leave three of the images per class for eval because we're not really going to do eval as of yet so we're going to grab these so controls let's cut them so we know so i'm going to grab some images out of hello paste them in here and delete this git keep we don't need that so we'll go into i love you and we'll grab up to here i think so then one two three yep so up to there paste that into train so we'll go into no and then we'll grab up to there so then we've got six we'll cut that paste that into train the training folder again and then go to thanks and we are going to go up to there so again we're going to paste this into our training folder and so effectively what we're doing here is just moving images right so we're splitting it up into a training and testing partition okay so we've got 66 items which means we've got 33 images in here for our training so the rest we're going to dump into our testing folder so we'll grab those paste them in there delete the get keep back into here cut that gotta know let's delete that paste those and then last folder yes all right so let's cut that so now if we go through our collected images we don't have anything in hello nothing and i love you nothing in no nothing in thanks and nothing in yes so we can effectively delete this collected images folder now let's take a quick second for questions where are we up to uh does the image size matter no we do some pre-processing and post-processing question was about negative images for ssd you don't actually need the negative images to do the training i'm wondering if there's a way let's have a chat afterwards me so if you jump on the discord link just hit scroll up find the discord link i'm happy to have a chat to you about um goals for the negative images a tutorial on shadow detection okay cool that might be interesting i'm thinking the next object detection tutorial that i'm going to do is on segmentation so object detection is obviously pretty cool but i want to do one on segmentation as well because i think that's going to take it one step further alright so if we take a look at our images now so we've got 66 images there might need to do a part two of this stream and then 30 images there so we're good so we've got 15 testing and 33 training so now that that's done i think we are good to actually start going through our flow now so i'm going to jump back into our jupiter notebook so if i go back let's just shut these down so i'm going to jump back in so again we cloned down this repo and it had so we created the image collection a bit we're now up to the actual tutorial so or the tutorial notebook so really this is the you could call it the training notebook we'll rename that okay so there's a fair bit in here so what we probably shouldn't clap that was probably that my bad um so we've got a bunch of paths here and these just reference uh all the different paths that we've got within our folder structure so if we take actually take a look at our folder structure come on snap what we're basically saying is our workspace path is tensorflow workspace scripts path is tensorflow script so what you see there api model path is going to be tensorflow models which we don't have clones so we're going to need to clean that our annotation path is where all of our different image annotations are going to be and specifically our tf records are going to be our image path is where all of our images sit so again they are in this folder so images and when we actually go to extract them out we're extract actually extracting the train and test partitions as well model path is really just this now that i think about it oh no this is actually our specific so it's going to be workspace path so tensorflow workspace models so it's going to be this model's path rather than the actual tensorflow models but again i'll explain that in a second so let's actually start running this and then i'll show you so let's hit that and then we need to change our label map so in this case we're not doing it on mask or no mask we're going to be doing it on our sign language tutorial so let's change these so we've got at five different classes so let's just grab make sure we've got and another key thing here is that if you're doing custom ones why are these not aligning sorry that's my ocd coming out there we go that's better all right so we need to make sure that the names of these different classes match exactly what you've actually gone and labeled during or using label image so if i say mask and i've actually written mask in the label image step then we're going to have issues there so it needs to match at a case level as well so if we go and take a look at our images and let's just open up train and open up our annotation so in this case it's capital hello so our first class is going to be hello and then if we take a look at an i love you note annotation this is what an annotation actually looks like so you can see it's almost like x oh it is xml so it's got the folder the file name the path the source the size uh the object so on and so forth so what we're going to do now is uh grab this class so i love you so this name bit here is what we're looking for so we're going to grab that and then we're going to scroll on down to the no one and get no should be pretty straightforward we'll write no and then we're gonna have yes and then what did we write i think we wrote thank you thanks thank you one of those thank you okay thank you okay and we're going to chain we're also going to change these id so we're just going to make sure they're consecutive okay so that's our label is now done so we've got hello and we're going to need commas between each of these don't need one there so hello i love you no yes thank you and we're going to have our id's one two three four and five so these ids are important because when you get back your detection you're actually going to get back the id so you need to go back and map through to this if you take a look at the detection history tutorial i go through that in a little bit more detail so if we write this out now we should be good there so this is actually going to write out a label map to the annotations folder so you can see that there now so that's what it should look like so hello i love you no yes thank you make that a bit bigger so you can see that so that's what your and a label map should look like cool and we're just going to delete this git keep we don't need that either all right questions training an already trained version of this detector yes i think i've got to do a tutorial on that like how to retrain or train so this is mick i'm i'm answering that question um so this is actually it's definitely on my list to do like a retraining video so how do you add more images rebuild the the tf records file and then keep training segmentation so segmentation so that's a good question so what is segmentation so right now we're going to be detecting squares right so that is object detection we're just detecting where an object is and putting a square around it when we do segmentation we can actually trace the the object that we're detecting so in this case where where i've gone and drawn my labels using label map i'm just going to be drawing a square around my hand right but when we do segmentation we could actually trace around the entire hand and be able to get the really specific region so it's used quite a fair bit particularly in defect detection so if you're looking at like corrosion analysis or crack detection you tend to see segmentation or even like cell analysis when you're looking at like microscope images used a lot more there and so the model that you can use for that is actually mask rc and then more questions can you make and oh so al on your bit about the sentence so if you actually take a look in the detection history tutorial we show how to actually do that i need to send nick a case of beer or whiskey sounds good mick i'll join you uh can we just do the fourth module manually and skip this section encoding i'm not too sure what you mean there could start from last checkpoint and start inside i think you can start from checkpoint from a different checkpoint but i wonder and i haven't tested this out yet which is why i haven't sort of said it i think you actually need to export a frozen graph rather than just exporting checkpoint zero so that is my thoughts on that when are you planning to do a segmentation tutorial i've actually got all the code done i'm just making sure it all works at the moment so that's actually coming i'm just making 100 sure that this all runs all right so on that note so we're now up to creating tf records so we've already got so this is effectively just going through and running this generate tf record script here so if i actually show you let me actually just print out this command right um how do we do this never mind let's just go and run it so this is actually going through what it's actually doing is it's going into tensorflow scripts it's running this generate tf record script here and we're just passing through the different paths really so if you didn't want to run it inside of the notebook or if you're getting errors you can effectively just run so what's it going to look like it's going to be python and then scripts path which is this up here so tensorflow scripts forward slash generate tf record dot pi and then you're going to pass through these other commands so then it's going to be dash x and then image path dot train so it's this a workspace path which is this tensorflow workspace slash images slash train and then you're going to pass through your annotation parts let's make this full screen now and then you're going to pass through your annotation path which is ideally going to be your workspace which is tensorflow workspace annotations and so here you're just actually naming it so right so if you didn't want to use this command or if you're getting errors when you're actually trying to run this this is effectively the command that you can run at the command prompt so tensorflow scripts generate tf record and then you're passing through dash x so this points to where your images actually are so in this case they're in the train folder but you need to replicate it for test and then we're passing through where we want our annotations to go so in this case they're going to go into tensorflow workspace annotations train record so i think so if i print that out probably should look a little bit better so that's the whole command to actually do this i've obviously just parameterized it so it's a little bit easier to run so if we run this now fingers crossed that shall all run successfully so let's wait and see okay so that's the first one done if we want to change sign name after creating data sets i think you have to retrain or if you're going to add in a new sign or want to change the sign name after that so i believe you can add more images you can recreate the tf records file and actually do it uh will be a little bit different if you use coco format i believe yep okay so that's our tf records and now done so if we now go into tensorflow workspace annotations we've now got these two test and train files here so this actually gives us the files that we're going to use as part of our training so the next thing that we're going to do is start creating a model folder so let's do that so with here we're just initiating our custom model name so this is this cell here you could change this to something else if you wanted to as well it doesn't need to stay as my ssd mob net and then what we're doing is we're actually copying over actually we need to check whether or not we've got tensorflow models oh so we're going to need to run this as well so this is actually downloading the entire tensorflow model zoo repository so we actually need this so we're going to run that that's going to clone it down it might take a little while but we'll give it some time all right questions classical bse format i think we're doing okay copying the model folder to config yeah the pipeline config can be done manually you don't need to use the command in here sometimes it throws up errors actually what i'm going to do is i'm going to do this in another command prompt because sometimes i get a couple of little bugs when i do it through here okay that's not going to run so what we're going to do is go into our d drive cd live stream ls cd into real time object detection and then go into tensorflow and then from here we're going to clone that models repo so is that effectively what we're doing yes and we've ridden git clone twice oh it's already cloned down never mind so this is already happening so if we go into this doesn't want to stop sometimes my windows machine doesn't like running these commands inside of this notebook so let's save this and actually stop it looks like we're gonna have to crash nope that stopped all right cool so we're just going to delete this folder so this models folder and i'm just going to do this manually nope doesn't want to delete let's close this and restart our kernel so let's stop this for now and then let's do it manually so if we go into let's open another command prompt okay so what we're going to do is rather than running that command inside of the notebook i'm just going to cd into it so live stream cd real time cd tensorflow tensorflow yeah so we're gonna do it in here so if we paste git clone and we should have got the command wet before we closed it let's all right let's go restart our notebook and grab it okay uh where were we okay so live stream real time object detection training so we're gonna copy this so we're just going to run this outside of the notebooks again sometimes in my particular machine it doesn't want to run from within a notebook so we're just going to paste that and then run that and so this is going to download all the tensorflow models stuff that we're going to actually need alrighty let's take a look at where we are on comments so it's going to take a little while and again if we don't get through this entire tutorial today that's fine we can always run a part two of this live stream all right he said current can we change the name of the tag in xml oh yeah you could definitely change it inside of the xml but you'll also need to change your what will you need you'll need to change your label map as well dilshan so um yeah so you could do that as well manually instead of having to go and read label all over again cp is not recognized so you need to install git bash for that so that's going to solve that but yeah as mick was saying you can actually just copy over your pipeline config you don't need to run that that cp command so it just makes it a little bit easier yeah awesome i hear your thoughts about collab yeah maybe i'll do that in in um a future video alrighty let's so this has taken a while to download we'll see how long this takes actually we might already have it in another folder that we can just copy over i don't want to be here waiting while you guys are trying to watch me do some training and it's copying all right so let's see might actually have it here oh yeah we got it let's just cancel this so this is the exact same so you can copy it down and just leverage that you can definitely clone it from github in this case i'm just going to skip through and paste it it should be a little bit faster there we go so this git clone is effectively just cloning down the tensorflow object detection utilities my internet is a bit of a shocker so and plus we're live streaming so it's taken up a bunch of bandwidth but again all we're really doing here is running that command or that training command that we saw back up there or the cloning command that we saw over here and because we killed this off so we're just going to need to run this again we don't need to rerun our tf record generation we're sort of up to here okay so once our models have cloned over so you should have all of this stuff available the next thing that we're going to need to do is what are we doing here so we're copying our pipeline config great question we're actually up to this so who was asking about this uh mooney so so what we can actually do is rather than running this command you can actually just go in and copy it over manually and a lot of people are getting stuck on this but all we're really doing here is copying this and pasting it into have we created that new folder yet nope we haven't so let's we just need it we still need to run this line so this is going to create a new folder inside of our models folder so my ssdmobnet and we're just pasting it in there we need to copy that again yeah yeah there you go so if this is causing errors you can literally just copy it if i'd known that it was going to cause um we're going to get errors from that and that's probably my fault a little bit because i'm using git bash on a windows machine kind of a little weird but this is effectively all that these two lines are doing so it's creating a folder inside of the models folder called my ssd mobnet and it's copying that pre-trained pipeline config into here so we've now got it in there okay so now we're up to actually going in the head and training so we're just going to run our import config path so if we actually output that config what's that look like it's looking alright so we can check that and so this is opening up our existing config so we need to change a bunch of stuff in here what we need to do is we need to change the number of classes to the number of classes that we've actually labeled so in this case it's going to be the length of this label array so if we actually change that which is it effectively five so if i type in len labels it's just five so i can type in len labels if we want to parameterize it a bit more everything else i think should be good to go so this is just pointing to our checkpoint um so if we run that run that that's our pipeline config and now done so if we open this up now so this is the custom one remember not the pre-built one what you should see is you should see the number of classes has been updated and if you scroll on down close this close this scroll on down what you should expect to see is that the fine tune checkpoint has been updated this fine tune checkpoint type has been updated label map path and label and the input path have been updated so effectively you should see all of those updated in your pipeline config so if we close that now that's all good to go and then we're up to training the model so we can run that copy this command and ideally all things holding equal actually good thing to note here right so sometimes you're going to get this out of this allocation error and that's because your gpu is running out of memory so if you see that this is already at eight or if you get like a status allocation error then you know then you might need to stop your kernel and run the training separately just a thing to note so what we're going to do now is we're going to try to run it so let's open up a new command prompt go into our live stream folder so cd live stream cd into our realtime object detection folder and then we're going to run it from here so if we just paste our command and hit run and hopefully all things holding equal actually i think i might have an incorrect version of tensorflow installed right now but okay so this is good so you can see that we're getting this errors attribute model tensorflow api v1 has no attribute v2 this is actually because i've got tensorflow 1 in sort of the moment so if i type in pip list let's scroll on up you can see that i'm running 1.1.4 1.15.0 and this is what i wrote for what i was using for for reinforcement learning so but in order to run this particular tutorial because we're using tensorflow 2 models we need tensorflow 2 dot something so in this case i'm going to use 2.3.1 because of the versions of cuda that i've got installed so we can type in pip install tensorflow equals 2.3.1 tensorflow dash gpu equals 2.3.1 and then i'm going to pass through user so this should give me the right version of tensorflow and allow it to run successfully so let's install that and then we'll try training again all righty what happens if we change the type detection of detection is changed to classification i believe it uses a different model and you get a classification model over a detection model but don't quote me on that i think it is a little bit different i have seen it used before i want to test my friend's train model any steps i need to do um nope so you can pick up that checkpoint and actually test it out on your own machine so um you should be able to do that will it work as an image classification program i think it will probably do i mean try it out see what happens let me know how you go i'd be interested to see how what the difference is all right so let's let this reinstall but again this sort of shows you how to solve these errors right like if you start getting errors sort through again google's your best friend find out what's what might have gone wrong um and then try to try different things out so again insul installing this isn't always the most straightforward thing you can you might have to try a bunch of different installs try installing something else in this case this particular error is definitely a versioning error so because we're using a tensorflow to model we need to be using a tensorflow 2 version so in this case installing tensorflow 2 is going to allow us to actually go on ahead and leverage that so we'll let this run for a little while any additional questions guys any new videos or anything that you want to see what's been happening are you enjoying the videos at the moment looks like tensorflow's taking a little while to install we'll just let that run oh internet is not being friendly to me today thanks so much man i'm glad you're enjoying it yeah what do you let me know what you guys think of this stream is it am i hitting the right spot maybe a little bit too long come on tensorflow oh what do we got there we've passed through dash dash user that should be fine tensorflow come on so i might need to wrap up this uh live stream kind of soonish but let me know if you guys want me to do a follow-up so if we let's see where we get to i'll probably we'll give it a little bit more time we'll see how long tensorflow takes to install um do you guys want to follow up and and see how to take this further we'll go all the way to the end action recognition definitely yeah i'm pumped to work on that as well thanks so much shane glad you're enjoying it hmm let's see how long turned is it going to go on ahead and install it's taking a little time now so really all that's left to do now is to get this line running so we just need to make sure that tensorflow 2.3.1 installs oh okay yes all right sweet we're done okay so what we can do now is copy this command and then paste it here i'm crossing my fingers let's see if this runs and the computer doesn't crash it's looking promising it's looking good oh i do have valor i suck at gaming so i've tried i like i just get so bored i don't know if you wanna if you wanna have a gaming session with me let me know okay so this is looking good so if you're getting stuff looking like this so far we haven't actually had any errors so even though we're getting this unresolved object checkpoint that's still fine so it looks like the model is still currently training at the moment and ideally you should start to see your gpu if you're using a gpu so you can see that our gpu has now been fully consumed so we're up to eight and we've got our first 100 time step and our loss being output so this is all good so we're running so we're training our model so as soon as we get a first checkpoint so we won't wait for too long so as soon as we get oh we've got our first checkpoint let's wait for checkpoint two once you start getting these checkpoints out of here that basically means that you've now got a trained model that you can now actually begin to use so let's give it a little time let's maybe let it run for i don't know a thousand steps and and then we'll test it out thanks ahmed glad you're enjoying it yeah wow have we been going for an hour oh damn okay just getting two comments now guys now wow yeah so sometimes running the python command inside of their notebook doesn't work for me either normally i just take them out to run um it seems to take a little while so i don't know if that's something with my particular machine but i just run them outside it's just easier to monitor it tends to work as well thanks so much miles glad you're enjoying it ethan so you're getting random incorrect detections how long do you so ideally you want to so there's a couple of things when you go and build one of these models so you want to have varying images so images of different quality you also want to have so i by quality i'm talking like different images of like different lighting different poses um you can perform some image augmentation but you've actually got that built in as well so really it's lighting quality different poses to ideally make sure that and different variations of that particular pose to get better detections if you're getting crappy detections what is my gpu i'm running a 2070 super so i actually bought it for this youtube channel so i can start doing some more of this stuff okay so it looks like our loss is sort of hovering around 0.343 so we're going to stop it now because we've now got our second checkpoint so you can see that we've got checkpoint 2 here looks like it's just dropped a heap so 264. all right we'll stop it there okay so that is our training now stop let's make sure so it looks like our gpu has released our vram so we're good to do that now the bit that you've been waiting for actually testing this model out so there's key thing so someone did ask about using your friend's model so if you're using your friend's model you just want to make sure that you've got their pipeline config and you also want to make sure that you've got their checkpoints because what we want to do is replace this checkpoint in here so we can see that our latest checkpoint is two uh how do i make this bigger so checkpoint two so you can see that there so we're going to replace the six in there with checkpoint two we're going to import that and it looks like we've got an error what's happening no module name tensorflow core keras okay so now we're getting errors we should have had tensorflow core keras okay that's a weird error let's save this oh wait we reinstalled tensorflow so hold on let's shut this down reopen let's try this now scroll on down so we need that uh let's wait and see if this works okay so that looks fine so remember we reinstalled tensorflow so in this case what we needed to do is just make sure that we restarted our kernel to be able to reload it so what i've done is i just went and ran this node over here and then if we scroll on down we should be able to run this cell so it looks like we weren't getting any errors there load up oh actually wait when that's not the bit that we want we want this so we want that so we're up to step seven we've changed our checkpoint to checkpoint two let's see how that goes no issues there then we can import cv2 import our category index and then our video capture i believe we still use zero so we should be okay oh so if you're getting around 1.5 seconds per step then again that might be a gpu thing so again it's going to train way faster if you've got a gpu so now if we run this detection code we should ideally get a little pop-up and get some detections let's wait and see what this looks like and again you can play around with this train for longer train for less um so in this case i've just trained really quickly so i'm not expecting the greatest model but we'll wait and see okay so we're getting a hello i love you thank you and you can see we're getting a bunch of boxes as well right so ideally it's detecting my headers no probably because it's like around so what we can actually do is change this to increase our detection threshold so if i stop this now and then run so remember our little square box is still down there so what we want to do is run cap dot release and cv2 dot destroy all windows so that's going to release our camera and close that and let's increase our detection threshold to let's say 90. so just by updating this and we can also decrease the number of boxes we want to draw let's just say we want to draw i don't know two or three we'll say three save that and then we can run this again and what are we getting now images must have either three or four dimensions okay so this is another error that you might run into occasionally if you're getting this error it just basically means that you haven't successfully been able to release your camera so what we need to do is just restart this notebook and again we're just going to step back through so we need our custom model file then scroll on down so we're going to then detect and we need our label map util so it's up here configure till okay so we need that from over here so we can import that config path then if we run this now ideally that should work okay we're running there so run that we don't need to run cap.release so let's try that again and then if we run this detection code so we don't need to release because we haven't actually connected to our camera yet so let's start this up and we should be able to get some detection so we actually made it to the end where am i from nick i'm actually from sydney australia so it is what's the time now it's 1 17. it's my bad i know the timing wasn't all that great uh looks like we're getting another error okay so what we'll do is we'll if we're getting this so we obviously connected to it once and it was working successfully so if you keep getting a bunch of errors what the easiest way to sort of solve this is restart your kernel so i'm going to restart that kick it off again go to live stream go to real-time object detection go to our training notebook and let's try this again so rinse awesome vikash what's happening okay so let's run that that looks fine so we need our custom so when we're reloading you don't need to run through everything so you need to run through this cell i believe you need to run through the custom model name cell we need this cell to import our config utils we need our config path don't need those ones you need and then we need to run everything from here so we need this cell this cell this cell this cell we don't need cap dot release we can actually delete that we need a captcha we don't need that cell and then we can run this fingers crossed let's say i can see okay it's popping up sweet there you go so we got hello i love you thank you no come on it's not detecting no well so actually so that might mean that we actually need to pass through a bunch more images for yes and no because right now it's getting a little bit confused so ideally we've got the main one so thank you hello i love you yes and no doesn't seem to want to like so let's try and you know what that's probably because the image is very similar so it's actually no it's performing fine but yes it's not so that might mean we need to train a little bit longer so if i was in this case as a data scientist so what i'd actually be doing is i'd be adding more images of the yes class and i'd be labeling them to be able to better detect so you can see that it's not performing it's not getting yes at all and i think we actually yes was the class that we might have actually had less images of so if we actually go back to our images file how many yes images did we actually have yeah so we only trained on seven yes images so again that might be a case where we need to add in some more images but i mean you can see broadly sort of gives you an idea how to test this out so hopefully you've enjoyed this live stream try without the green screen all right that's it no it's detecting i love you so that kind of sucks right so thank you it still performs well hello i love you no so again that might be and i think the lighting's a little bit weird as well so no it gets it's not getting the rest so again you can try it out and see what actually happens but um yeah so that's what about wraps it up so hopefully you've enjoyed this live stream so again i'll probably share the or let me know if you want me to share this code again it's all the same that's sort of out there but test it out see how you go let me know if there's anything else you'd like to see hopefully you've enjoyed this first ever live stream um and again peace out thanks so much for tuning in guys i'll go through your questions if there's anything else i missed thanks so much mick cheers can i use cap dot release in cv to destroy a window if this statement at the end so that that works if you're running it directly from the command line um the queue thing is from the command if you're running it outside of the notebook it doesn't really work from within the notebook try without the green screen video data set yeah we'll do that at work trying to debug some javascript code that's a bane of my existing vikash yeah i should have kept an equal number of um images for each class so that was remember we i collected some crappy images for um for yes and like it's like it obviously failed so like i'd obviously add in more classes of yes now to be able to get this to perform better but i mean you can see it's performing okay like we only trained it for a thousand steps as well so like doing dual hand is it yeah it's working okay um again you can obviously definitely improve this train for a lot longer but this is obviously real time so you saw it didn't take a lot of time or we didn't train for a very long amount of time next live we haven't even wrapped up the first one thanks um but yeah we'll definitely see if how lives performing and if you guys really enjoyed this by all means let me know if you'd like to see another live stream now you made it to the live stream vikash that's awesome thanks so much dilshan thanks so much rayne glad you enjoyed it thanks laki thanks by you thank you so much mahesh yeah ideally you'd want to have an equal number of images if anything it doesn't matter if they're not necessarily equal but you want ideally more than seven i i obviously kept it quite low um sriram cp is not working scroll back through the the live stream you can actually just copy the config i actually showed how to solve that thanks so much miles nah no stress mick all good no no problems alrighty thanks guys we're going to wrap it up thanks so much for tuning in hopefully you've enjoyed this video and as per usual if you did be sure to give it a thumbs up hit subscribe and tick that bell and let me know if there's anything else you'd like to see you've all been a pleasure to have a chat too thanks again for tuning in peace you

Channel: Nicholas Renotte

Views: 65,123

Keywords: object detection, tensorflow object detection api, custom object detection, tensorflow object detection, computer vision, tensorflow tutorial, object detection tensorflow, object detection opencv, computer vision projects, computer vision python, object detection python opencv, object detection using tensorflow, object detection using opencv python, computer vision machine learning

Id: V0Pk_dPU2lY

Length: 81min 15sec (4875 seconds)

Published: Fri Mar 19 2021