Automatic Number Plate Recognition using Tensorflow and EasyOCR Full Course in 2 Hours | Python

what's happening guys welcome to the full and pr so automatic number plate recognition course in this video we're going to be going through everything you need to do to be able to go up and get started with advanced automatic number plate recognition we'll be able to use our webcam as well as some images to be able to detect license plates and eventually apply ocr to be able to extract the text that we've detected in it ready to do it let's get to it what's happening guys welcome to the automatic number plate recognition full course now in this video we're going to go from start all the way to finish to be able to build a full-blown automatic number plate recognition system something which is pretty pretty cool now in this particular video we're going to cover being able to detect license plates and extract the text from an image as well as from a real-time video feed so later on in the video you'll actually see that we'll throw up a plate on my phone i'll point it at my camera and ideally that should detect the plate from that particular video and perform some ocr to be able to grab the text from the number plate pretty cool right now in this video we're going to cover a bunch of stuff so let's take a look at what we'll actually be going through so first up what we're going to be doing is installing and setting up our environment and everything we need to get up and running then what we're actually going to do is get our license plate data so this video is a little bit different to my traditional object detection videos in that we're actually going to be using a pre-annotated data set rather than one that we've gone and collected and annotated ourselves so it's going to be slightly different in that respect so we're actually going to be using some kaggle data for this particular video now once we've got our kaggle data we're actually going to be training an object detection model and you'll see why we're doing this in the next slide when i explain how the system is going to work so we're going to train an object detection model to be able to detect our license plate so you'll actually see that it detects our region of interest and it shows up as a license plate now the reason that we're doing this is that it's way more accurate and way more resilient than using traditional computer vision techniques so if you watched my previous ampr video you might have noticed that if you threw up different types of number plates or if you applied it to number plates in different regions it might not have been as resilient as you might have liked it to be well this model is going to perform infinitely better than that particular approach which is why we're going through this so once we've detected the number plate we're then actually going to detect some number plates or so once we've actually trained the model we're then going to detect some number plates so we'll do it on an image as well as a real-time video feed and then we're actually going to apply our ocr step which is step five over here so for our ocr we're going to be using a library called easy ocr and this is going to be able to extract the text from the detected license plate feed and we can do this in both real time as well as on a standard image now where i wanted to take this is one step further so step six is actually going to allow us to build up a little bit more of a system now one of the common things that you'll notice in production grade pre-built ampr systems is that it will detect the number plate but it will also save out and log some of the results so you'll actually be able to have and review the regions of interest or the plates that have been detected previously you'll also be able to search through those now i wanted to mimic that a little bit in this particular video so what you're going to see is that we'll detect the plate but then the system's actually going to save that region of interest i know i'm getting excited but it's such a cool thing it's going to save that region of interest to our machine so we'll actually be able to go through and view all the plates that have been detected and we'll have a corresponding csv sheet that'll have the name of the image that represents that region of interest as well as the plate that's been detected or any of the text that's been detected in that plate so all together that should give us a full-blown ampr system now on that note let's kick things off and start taking a look at our setup so in step one we're really going to be going through our setup now in terms of what we're going to be doing let's take a look at how it's all going to fit together first up so the system that we're going to be building is really comprised of two key elements now the first element is our object detection model and the core purpose the sole reason for existing or for that particular component existing is to be able to extract at our regions of interest now it's pretty verbose in what it means it's the region of a particular image or video that we're interested in now in this case our roi or our region of interest is going to be the license plate for that particular vehicle or vehicle now so you can actually see this on the right hand side of our image here so we've got i might actually be covering up this bottom region so we're actually going to be detecting the region of interest for a particular number plate so you'll actually see that over here and this is what our object detection model is actually going to do for us then our second element in our ampr system is actually going to be easy ocr and easy ocr is going to be what allows us to extract the text from the images so again two part system first part detects our license plate second part actually extracts the text now in terms of how this all fits together so we're going to be using our webcam in this case you could go and deploy it you could go and leverage a whole bunch of other systems and this is going to be very gpu reliant in this particular case but we'll be using our webcam to grab an image of a plate so you'll see later on that i throw up a plate on my phone and we'll try to do that then tensorflow object detection will initially kick in to detect our region of interest based on the trained model on our kaggle data set so you'll see that it tries to detect our license plate which is this little green box that you can see here then last but not least we're going to use easy ocr to actually grab this region here i'm probably blocking this text over here but you can see uses easy ocr to extract text and leverages a size filtering algorithm to grab the largest detection region so when i was building this i had to try to think of a way to only pick up the bigger blocks of text so you can see here that we've got bg02qt but then we've also got this slightly smaller area over here as well so what i did is i came up with a small algorithm which effectively tries to categorize the size of the detected region so basically we can tune that particular threshold so so originally what i said is that if the plate so in this case you can see that red line here so the box is actually this if the detected text is 60 of the area of our region of interest then that's going to be our text but again you can play around with this we'll dive into that in more detail later on so that about summarizes how this is all going to work now we need to get up and running and start installing all of the stuff that we're going to need for this particular video so there's going to be two components as i was saying the tensorflow object detection bit and easy ocr so what we're going to do is we're going to get up and running with tensorflow object detection train our model and then we'll go and apply additional overlays to actually be able to go and perform a mpr so let's go on ahead and get this started so i'll show you where you can get this code and we'll start building up so in terms of getting up and running with our tensorflow object detection model we're going to be using the code that we wrote for my full-blown tensorflow object detection course now again that entire video is available on youtube so if you actually go to my channel you can see that it's over here it's a five hour long video so pretty long but it goes through all of the basics of getting up and running with tensorflow object detection in pretty great data so what we're going to do is do exactly that and leverage that particular model to be able to detect our license plate so what we're going to do is copy this link here so this link is github.com forward slash knickknock knack forward slash tfod course and we're going to open up a new command prompt and i'm going to go into my d drive but again you can do this into any drive that you want and i'm going to create a directory called ampl so we're going to just type in mkdr ampl so this is going to create a new folder and then i'm going to go into that folder and if i show you what this looks like it's really just a folder so you can see that i've gone and created this folder here ampr let's zoom in on that so you can see folder called ampr nothing too crazy there right it's just a folder now what we're going to do is we're going to clone in all of those assets from the tensorflow object detection course into that folder so let's go ahead and do that so to do that i'm going to type in git clone paste the link and then at the end of it i'm just going to include dot now what this is actually saying is git clone so clone this repository here so https colon forward slash forward forward slash knick knock knack forward slash tfod course and then the dot at the end of it which you can see sorry my mouse plane up the dot that you can see here is basically saying clone all of the stuff from that particular repository into the current folder so it's going to not create a new folder with all of those assets it's just going to put it into that folder you'll see that in a second so let's go ahead and clone that and we'll take a look okay so that's now finished cloning and if we open up our ampr folder again you can see that we've got all of our stuff so you can see we've got a bunch of different files so we've got some checkpoints we've got a git ignore file again you can probably ignore all of these folders the core one that we're going to be leveraging is this notebook here so it's two dot training and detection dot i pi nb now again we're going to be working mainly with jupiter notebooks in this tutorial i know i'm a data scientist i can't help but using jupiter notebooks but again if you wanted to do this inside a pie charm or a separate idea you could do that you just need to remove the magic commands from the notebook in order to be able to leverage that i'll maybe talk about that a little bit later or if you want to see a video on that by all means hit me up in the comments below but for now i sort of wanted to show you what we've gone and cloned down and you can see that we haven't actually created a new folder in here so it's just gone and cloned all of the stuff from that tfod course folder into our ampr folder and that's the advantage of adding that dot to the end to say clone it into the current folder okay so we're good in that respect now now what we need to do is actually go through the different steps from this github repository so i've gone and written up a really detailed walkthrough to actually get through all of this stuff so ideally this should mean that it's a little bit easier to get up and running now what we need to do is create a new virtual environment activate that environment and then we need to install a bunch of dependencies and then walk through our specific notebook so let's go ahead and do that so first up we're going to create a new virtual environment so if i bring up that same command prompt let's just clear this let's create a virtual environment so to do that i'm going to write python dash m v e n v and then the name of the virtual environment so i'm going to call it ampr amp rcs so this basically you can name it whatever you want i'm just calling it amp rcs because we're building an ampr system but again you could play around with this so python dash mvenv amp rcis now if we run that this will create our virtual environment and that's our virtual environment created so again if we go into the folder that we're working from you can see that we've now got this ampr sys folder here so that is the beginnings of our virtual environment created now what we actually want to do is activate that virtual environment so if we go back into our jupyter notebook or go back into the github repository you can see step three is actually activate your virtual environment now depending on whether or not you're running on a linux machine or a windows machine the way to activate that environment is going to be slightly different if you're on a linux machine it's going to be source tfod forward slash bin forward slash activate if you're on a windows machine it's going to be dot backward slash tfod forward slash scripts or backwards scripts backwards activate and the tfod bit over here is going to be the name of your virtual environment so we've gone and changed ours as it's going to be dot amp rcis or amp assist so we just need to make sure we change that command appropriately so let's go ahead and activate a virtual environment cool so that is our virtual environment now activated so all i wrote there was dot backward slash ampr sys backward slash scripts backwards activate so that you can see there that we've now gone and activated that virtual environment so this little set of brackets plus amp assist means that we're now operating within that virtual environment now what we need to do is install a couple of dependencies in order to get this up and running so if we go back to our jubiter notebook we'll go back to our github repository we effectively need to run these commands here so again this first one is going to upgrade our pip version that we're working with this next one is going to install ipi kernel and ipi kernel is going to allow us to leverage our virtual environment inside of our jupyter notebooks again if you want a crash course on working with jupiter or a crash course on installing jupiter again i'll link the full-blown installation video in the description below so you can take a look at that but in this case we're going to go on ahead and install our ipi kernel first then we'll do this line and then i'll show you how to do that one there so let's go ahead and do this so first up we're going to install ipi kernel so to do that we can run python dash m pip install i pi kernel and this is going to install ipi kernel and ipi kernel is actually going to allow us to associate our virtual environment to our jupyter notebook i'm probably repeating myself a bunch but i like to sort of reinforce that concept and i'll show you that once we've actually gone and set that up where you can actually leverage or see all of those different kernels available inside of your jupyter notebook so let's let that run for a sec and then we'll go and upgrade pip and then we'll go and install our ampr sys virtual environment into our jupyter notebook cool so that's now gone and done and you can see down here that it's saying hey you should consider upgrading via the python dash m pip install up dash dash upgrade pip command well let's go ahead and do that so this is basically going to update our pip or our python package manager inside of our particular virtual environment so we can just copy this command and paste that down here so the full command is python dash m pip install dash dash upgrade pip so it's really just going to be upgrading this pip library here so let's go and run that and again this is all pretty standard i cover this inside of the github repo so you'll be able to pick that up and take a look now that one ran pretty quickly so the next thing that we need to do now is actually go on ahead and associate our virtual environment to our jupyter notebook so we now need to go and associate ampr cis to our jupiter notebook now i'll show you what this actually looks like if we don't have it done right so if i start up a jupiter notebook by just typing in jupyter notebook when we start up this notebook you'll actually see that we don't have ampr sys in there so if i hit new over here you can see i've got all of these different virtual environments i've got one called python 3 one called gpt neo from our jpt neo video one called hands one called rl course and one called tfod but right now we don't have ampr sys in there so we actually need to go and install it to be able to have it in there now i'm going to close this down go back to our command prompt and stop jupyter notebooks just by hitting control c and what we want to do now is install ampr sys as a virtual environment into our jupyter notebook so let's go ahead and do that so to do that we can write python dash m ipi kernel install dash dash name and then whatever the name of our virtual environment is so in this case it's going to be amp rcis so let's take a look at that full command so the command is python dash m and then we're going to be using the ipy kernel library that we just installed so then we've written i pi kernel install and then we're passing through the name of the virtual environment that we want to use so dash dash name equals ampr sys so this will effectively allow us to use our virtual environment inside of our jupiter notebook now often i'll hear clients or people that are working with virtual environments and they'll go hey nick i installed everything into my environment but when i go inside my jupyter notebook it's still saying it's not there or it's not actually appearing more often than not this is because the kernel that you're working with inside of jupiter is not the same kernel that you're installing your python dependencies in doing it via this route means that you're going to be working with the same environment regardless of whether you're working from the command line or from your jupiter notebook so let's go ahead and install this cool so you can see that we've now gone and installed it so it said installed kernel spec ampr sys in c backward slash program data backless jupiter backwards kernels backwards amp assist again this path might be different depending on what computer you're working on or what type of environment you're working on but ideally you should see that it's installed successfully now what we can go on ahead and do is kick this off so if i start jupiter notebooks now when we go and take a look we should effectively have ampr sys available there now so if i select new you can see that we've got ampr syst there so that's our virtual environment now installed on our machine and available inside of our jupyter notebook so we're all good to go in that respect now the next thing that we need to do is actually go on ahead and install the tensorflow object detection api this process can be pretty tricky so again it might take a couple of steps but i'll show you how to work through this and again if you have problems by all means hit me up in the comments below join the discord server if you need some help we're always pretty active on there and we've also got an error guide inside of the tfod course documentation now i'm just going to zoom in on this a little bit so from our jupyter notebook environment i'm just going to open up this training and detection notebook that you can see here so it's called two dot training and detection dot i pi nb so let's go on ahead and open this and now what we need to do is we need to make sure that we're currently working with the ampr syskernal so you can see over here right now it's showing up as tfod this is from a previous video so what we need to do is choose kernel change kernel and then choose amp acis ideally you should see that we're now operating inside of the ampr sys kernel or virtual environment so we're now good to go now this particular notebook is going to walk you through everything that you need to go through in order to get up and running with tensorflow object detection that includes setting up all of your folder structures actually getting your model up and running everything that you really need to be able to go and do to get this up and working now the first thing that we need to do is make sure that we get the tense flow object detection api up and running because that is going to be what gives us the ability to perform object detection so what we're basically going to go ahead and do is just hit shift enter to go through each one of these cells now the first couple should work all fine so you can see that we're up to step zero setup paths this initial block is really just going to set up our folder structure so if i go back into the folder that we're operating in you can see that we've now got this tensorflow folder and inside of here we've got a whole bunch of different folders now as you actually go through this it's actually going to go on ahead and set up all the different folders that you need or if you want a deeper description or deeper walkthrough as to what each one of these folders does by all means do check out the full tensorflow object detection course so that is all well and good okay cool now the next thing that we need to do is actually go through step one which is download our tf models pre-trained models from the tensorflow model zoo and install tensorflow object detection it's probably a lot there so what we're actually going to be doing when we use the tensorflow object detection api is we're actually using a pre-trained state-of-the-art model and we're just fine-tuning that particular model on our specific use case this is often referred to as transfer learning so we're going to leverage the knowledge that's already inside of our object detection model or those pre-trained object detection models and we're just going to fine tune it to be able to detect license plates now in order to do this we need to actually clone down the pre-trained model which happens a little bit later on down here and we need to make sure that we've got the tensorflow object detection api installed now i'm just going to clear these cells just to make it a little bit cleaner so what we're going to do is start kicking that off now in order to do this you just need to hit shift enter and go through these cells and ideally it should all just walk through for you so i've gone and set this up so you can just sort of step through it so if i hit shift enter it's going to install a wget and wget is going to be used to go and grab our pre-trained models then this next one is actually going to go on ahead and clone all of our tensorflow model stuff but so if i show you what that looks like it's actually going to go to this particular repository and clone down the tensorflow model garden so inside of here you've got a whole bunch of the research that's actually happening from the tensorflow model team as well as the object detection api components so what we're going to do is shift enter through that and that will clone down that repository now that might take a little bit of time to clone but as soon as it's done you're going to see the tensorflow model garden inside of this folder here so inside of ampr tensorflow models so we'll let that run for a little while and we'll be right back all righty so that took absolutely forever because my internet is quite slow but you can see that we've now got all of our tensorflow model zoo stuff now clone down into that tensorflow models repository so you can see that everything that we had inside of that folder that i showed you inside of github is now here now we're specifically going to be leveraging the stuff inside of the research and object detection folder so this contains a whole bunch of repository or a whole bunch of resources that are actually all around the object detection use case now in this case we're going to be using a bunch of stuff from here but again i've sort of made it a little bit easier to walk through by using that notebook so that's all well and good we've now gone and cloned that down now the next thing that we actually need to do is actually go on ahead and install the tensorflow object detection model now this will take a couple of goes so if you do get stuck sort of don't hesitate and freak out it is going to take a little bit of time to get this installed now what this is actually doing is first up it's checking whether or not you're running on a linux machine or running on a windows machine now depending on what type of machine you're going to be using it's going to choose or run a different set of commands to get this up and running now i've tested this on colab and i've also tested this on a windows machine so let me know how you go getting this set up so what we're going to do is again we're going to run this cell and this will start installing tensorflow object detection now the way that you can know that this is running is if you actually go into the tensorflow folder or if you're actually going to ampr and ampr sys and inside of there go into the site packages folder what you'll see is that it's starting to install a whole bunch of different types of libraries now in a sec what we'll do is we'll actually run a verification script to make sure that this is all well and running but again we'll come back to this in a second so we're going to let this run and then we'll come back to it so first up it's going to go in ahead and set up something called protoc which stands for protocol buffers and this is what the tensorflow object detection model relies on then it's going to go on ahead and actually install the object detection api in these two lines of code here so let's let this run and then we'll be right back and run our verification script and sort out any additional dependencies we need to run through okay so it looks like that's finished running so you can see that we no longer have our asterisks inside of our notebook and it looks like it's gone and installed a bunch of stuff now if we run pip list within our notebook we can see that we've now gone and sort a whole bunch of additional dependencies but as of right now it doesn't actually look like it's gone and installed tense flow and tensorflow gpu so what we want to do is sort of force prompt that into our folder now rather than going and running that we can actually go and run our verification script and see if this is all working first up so you can see that there's no module named tensorflow so what we're going to go ahead and do is run this line down here which will install tensorflow for us now i'm going to add in one additional line so tensorflow dash gpu and this is going to give us the ability to leverage our gpu now key thing to call out is that in order to leverage tensorflow gpu you do need an nvidia gpu so in this case i've already got an nvidia gpu on this particular machine you can try using an amd gpu in this case rather than using the tensorflow dash gpu library you need to use another one called tensorflow rock m or roc m now in this case what we'll do is i'm keep saying in this case a lot here what we're going to do is we're going to install tensorflow 2.4.1 and tensorflow dash gpu 2.4.1 again if you want to see in greater detail as to why we pick these particular libraries or how to map your cuda and cu dn version to your particular tensorflow version again hit me up in the comments below or do check out the broader tutorial but in this case we're going to install tensorflow 2.4.1 and tensorflow gpu 2.4.1 it's going to give us our core dependency so let's run that and again if we check our libs folder you'll see that tensorflow gpu starts installing into there in a second you can see that we've got movement it's all happening so we'll let that run and then we'll come back and we will test out our verification script again so ideally what you should see is when you run that verification script right down the bottom you'll get some a particular line that says uh xx test run i think it's 21 tests run and it should say okay so if you get that okay line then you should be good to go more or less cool so that looks like tense flows now installed so you can see it's going to run through all of that so if i go if we go and run pip list now we should ideally see tensorflow on there and it looks like we've got tense flow 2.4.1 and tensorflow dash gpu 2.4.1 so it looks like we're good there now again let's try running our verification script which is this line here so you can see it says verification script so again this needs to be reading all well and good in order for this to be running so let's run that and see if we get our okay status towards the bottom all right so it looks like we've got another error so if you get any error that's sort of along the lines of module not found error or module does not exist all you need to do is go and install it so in this case here we can see that we've got no module named matplotlib now if i copy that and google what how to solve that particular error more often than not it's just going to say you need to go ahead and install it so down here let's take a look looks like we just need to run pip install matplotlib so again more often than not these particular module not found errors just mean you need to go and install something so down here i've actually got this particular line which is going to install protobuf and matplotlib so let's go ahead and run this and that should uh skipping matplotlib as it's not installed let's take a look to see if that's actually gone and run let's run pip list again okay so it looks like a matplotlib so that was saying that was throwing up an error for that first line so the first line was actually going in uninstalling matplotlib and then reinstalling it in this case because it didn't have it it's fine it actually went through and installed it so we're all good to go now again when we run pip list down here we should get our matplotlib version so you can see we've got matplotlib there right there so 3.2.0 cool again let's try running our verification script again and so the purpose of this verification script is just to make sure that you've got all of the dependencies that you're going to need later on to be able to go and run our particular model so now we've got another error so it's saying module not found no module named pip now i know that this is the pillow library but let's go and google that error again so you can see the solution to this particular line is going to be to run this line here so pip install pillow so let's run this and this should install our pillow library has it it looks like it's successfully installed even though we're getting these warnings here so let's take a look at pip list again so again you can see to solve those module not found errors you just run that verification script whatever modules it doesn't find go and install it run pip list to make sure it's now there so you can see we've got pillow there so that should resolve our module not found pill error so if we go and run our verification script again let's take a look this time uh what are we getting now so no module named yaml again this one is pi you need to install pi yaml but again let's go and google it so you can see here that in order to resolve this particular error you need to go and install pi yaml so let's go ahead and install that so i'm going to add in another line pip install by yaml run that okay successfully installed payamore down here that's good to go right now verification script again and again i know this is a little bit tedious but it once you've got it set up you're good to go you don't need to run it again or don't need to install a bunch of stuff again you can just go on ahead and leverage this virtual environment whenever you need to go and build object detection models that is the nice thing about it so once you've gone and set it up you're good to go and build a bunch of stuff with it this is looking more positive now so let's scroll all the way down okay cool so right down the bottom here so you can see that it's opened a bunch of libraries successfully but most importantly is if you go all the way down to this particular line down here you can see that it ran 21 tests in 17 dot let me make that a bit bigger ran 21 tests in 17.079 seconds and they were all okay it just skipped one so this means that you've successfully installed the tensorflow object detection model so again it took a couple of iterations we needed to install pillow needed to install tensorflow needed to install pi yaml but once you've gone and got those installed you're good to go you don't need to go and do it again i know it is a bit tedious but once you've got it done you should be fine and again if you get stuck by all means just hit me up in the comments below i'm always happy to help now what we need to do is we need to make sure that we're able to install or run this particular line here which says import object detection if i run this more often than not it's going to say object detection is not found to solve that particular error you just need to restart your notebook so if i run this you can see it's saying it's not found now if i hit kernel and then restart now again you don't need to go and reinstall everything just because you restart your kernel it's fine we just need to hit restart and we're going to be able to import that now and what i'll do is i'll clear all of my output so we can see what cells we have and have it run so if i import os run these again and then import w get up we don't need the seven there import w get now if i import object detection you can see that it's run successfully so sometimes you just need to restart your kernel to be able to go and run that particular cell so we're good to go now so we've gone and installed our verification stuff or we've gone and run our verification script all well and good now if we run this cell here this is actually going to go on ahead and download our pre-trained model so remember i was talking a little bit about transfer learning this is going to download the pre-trained model that we're going to be leveraging so that we can go on ahead and fine-tune it on our number plate detector so run that next cell so the one down here so you can see it's going to be using wget to download our pre-trained model it's then going to move it into our pre-trained model folder and it's going to unpack it so it comes in a rgz file so this just uncompresses it so if you get something that looks sort of like this then you should be good to go the other way to check whether or not that's run successfully is if you go into your ampr folder tensorflow workspace pre-trained models you should have this over here so let me zoom in on that so you should have a file which is called ssd underscore mobile net underscore v2 underscore fpm lite underscore 320 by 320 underscore coco 17 underscore tpu-8 so this is your pre-trained model so this is the model that we're going to be leveraging to fine tune to detect our number plates cool alrighty on that note that is step zero so setting up our paths and step one downloading and installing tensorflow object detection now done thankfully there isn't too much installation left for the rest of this tutorial we're actually going to get into the good bit now so if we take a look we've now gone in install tensorflow object detection easy ocr will do a little bit later now we're up to step two working with our data now in this particular case we're going to be leveraging a kaggle data set so if you want to grab this data set or in this case if you want to go through this tutorial you need to grab the data set it's going to be available kaggle.com forward slash andrew mvd forward slash car plate detection so if we go to that link you can see that i've got it there and this actually already has a whole bunch of car annotations plus car images which is in the perfect format for us to leverage our tensorflow object detection model on so what we're going to do is download these images and again you're going to need a kaggle account to be able to download it's going to ask you to log in set up a kaggle account log in download the data and you should be good to go so let's let that download and then i'll show you what to do with it alrighty so those are our images downloaded so you can see down the bottom we've got this archive 1.0 so we just need to go and grab that particular folder so it's going to be in our downloads folder and i'm just going to cut this now everything in here actually contains al so that particular folder contains the annotations as well as the images so i'm going to show you how to work with these so what we're going to do is go back into our ampr folder so i'm going to zoom in on this so you can see it a little bit better so we're going to go into our ampl folder go into the tensorflow folder workspace and then images folder now what we need to do in here is create two new folders so one called train and another called test and we're going to paste our downloaded images over here to begin with now from here i'm going to extract all of these images so you can see it's going to ask me where i want to extract it to so we'll just hit extract and it'll take a little while to extract but that's fine it's going super slow on my computer and then it should speed up now once that's done what we're going to do is we're going to split up these data sets into training and testing now a core thing to note let me actually show you what's actually in here so we've got two folders in there so one called annotations and one called images so if i open up the annotations folder you've got all of these files which are xml files so if we open one of these up inside of vs code so this is what an annotation looks like so it's got our annotation it's got the folder name the file name the size of our image uh whether or not it's segmented the name of the object this is super important now in this case all of the images have one object and they might have multiple objects haven't gone through every single image but the object name is license and it's in lower case now when we actually get to our second step where we actually need to create our tf records i'll show you where you need to update this so in this case we've got our objects and we've got a bunch of annotations and we've also got the matching images so if we go back into our archive we've got all of the images there so you can see that this is image cars zero so if you can see that we're going to have a matching annotation for that particular image so if i go into annotations this is car 0 so this annotation here is the same or this annotation here is the corresponding annotation for this particular image so what we're basically saying is that this bounding box down here so x min y min y max and y or y min x max and y max actually corresponds to this particular number plate which you can see here now what we need to do is we need to split up these annotations and images into a training and a testing partition so the training partition is going to be what our object detection model is trained on the testing partition is going to be what our object detection model is evaluated on so we train it on one particular bit of data and then we test it on a completely separate bit of data to see how well it's actually likely to perform so on that note let's go on ahead and do that now this is really just a pretty straightforward cut and paste so again you should ideally randomize how you actually do this but i'm just going to be a little bit lazy and just grab all the data so i'm going to grab the 4 00 to 410 again you could choose a different set of data but i'm going to choose the files car 0 to cars 410 for our training data set so if i cut all of those and paste it into the training folder and then what we need to do is do the corresponding thing for our actual images so these are the annotations we now need to do the same thing for the actual images so if we go into back into our archive go to our images so we went car zero let's just convert this to a list what was it details that's better car zero to cars 410 so we're going to cut that and paste that into our train hope i keep zooming in and out that's my bad so let's take a look we haven't pasted that in yet okay so that's looking good now so we've now got our images and our annotations so in this particular case we should have what 820 files so well oh it starts at zero so 822. so in this case we've got our images plus our corresponding annotations inside of our train folder now what we need to do is do the same thing with the rest of the images and the rest of the annotations into our test folder so this is a core difference between using pre-annotated images and files versus actually collecting them yourself but again pretty similar to what you would have seen in the 5-hour tutorial or in the big tensorflow object detection tutorial so i'm going to paste the annotations or the rest of the annotations into test and then i'm going to grab the images as well and paste those in there okay cool so if we go into our archive folder again there's nothing left in annotations and nothing left in images now so we can actually delete this archive one folder now and delete our compressed files so now we should be pretty good to go so that's our data sorted so we've gone and done a few things there so specifically we went to kaggle we downloaded our images what we then went and did is we created two new folders let me bring this down we created two new folders inside of our ampr folder and specifically inside of our ampr folder we went to tensorflow workspace images in here we created two new folders so train and test and then we copied the annotations and the corresponding images into those folders so we now should have all of our data actually up and ready to actually go and train this object detection model that brings us to our next step training our object detection model so step three is training our object detection model so we're now going to take the data that we just went and place it into our train and test folder and train an object detection model to be able to detect the license plates now in order to do that there's four key things that we have to do so first up we have to update the labels and in this case we've only got one label it's license the nice thing about the tensorflow object detection api is that if you wanted to train a multi-class object detection model you could do that so in the past we've built models that are able to do sign language so hello uh thank you so on and so forth as well as like really specific things like performing defect detection with a microscope or building a object detection app so you can do multiple things with this in this case we've just got one label which is license as i showed you in that annotation folder so then what we need to do is create our tf record so tf records are the file format the object detection model requires your data to be in now i've got a script that allows you to do all of that so you don't need to do that from scratch you just need to run through it we then need to prepare our configuration again really straightforward it's all in the notebook and then train our model so first up let's go back into our notebook that we're working in and update our labels so i'm going to scroll on back and then go back to our notebook and what we are then going to do here is go to step two so step two says create label map this is exactly what we're going to do now in this case we've only got one label that we need to work with which is called license so i can delete all of this stuff here so thumbs down thank you liv long and all we need to do is replace this name here and we're going to call it license because that is the name of the annotation of all of the license plates that we had in our downloaded folder now you can check this by going into the file and if you open it up what you're going to see is that this object has a xml of tag called name whatever the name that is that you see in there is going to be or need to be the name of your actual label so in this case it's licensed so we're going to call it license and i think i've already made a typo so it's l-i-c-e-n-c-e i've typed in lic and se so it actually should be ce so those annotations need to map back through to this label map so the name needs to be identical otherwise it's going to throw a bunch of errors so let's go ahead and run this now this will create our label map so that's all well and good the way to check what that's actually gone and done is if you go into your tensorflow workspace annotations folder you can see that we've now got this file called label map which is over here and if i open that up you can see it's just got one class which is called license l-i-c-e-n-c-e not se and it's got an id of one so that's all well and good nothing else to do there now the next thing that we need to do is actually create a tf record so this first cell under step 3 is optional this is really only required if you're running this on colab in this case we don't need to go through this we are up to this step here so this is going to download the generate tf record script that i got from the official object detection tutorial so let me show you this so we're actually going to need to tweak this a little bit so this file over here is actually going to go and convert all of our data from its raw format so in this case our images and annotations and convert it to a tf record format now because of a slightly different format for the annotations we're actually going to need to update this script but i'll show you why so if we actually go and clone this repository here so by running this cell so it's the second cell under step 3 this is actually going to go on ahead and clone at that repository or this script into our folder and it should go into the tf record scripts folder so let's let that go on ahead and run and then i'll show you where that is alrighty so that is our generate record script now clone so you can see that it's called generate underscore tf record now this is going to be what again as i was saying this is going to be what allows us to create our tf records now a core thing is though is that what i notice is the data from kaggle is in a slightly different format to what we normally get when we're labeling using label image so if we go on ahead and run this script now so this next cell down the bottom here we're actually going to get errors so it looks like well the first one is no module name pi tz so let's go ahead and install that first up so this one over here is throwing up a slightly different error so let's run that so again because we've got a no module error we just need to go and google that so how do we solve that particular one so let's google that so it looks like the solution is we just need to install pi tz again pretty straightforward pip install pi tz okay so that's solved now now if we run this script it should be a slightly different error let's wait and see okay perfect cool this is exactly the error that i was expecting to encounter so in this case it's saying index error child index is out of range now the reason that this is occurring is because it's actually looking for the bounding box inside of the fourth object from our annotation now what i actually noticed is that the annotation from the kaggle data set is actually in the fifth object so if i go into an example annotation so if i go into images train and then i open up car0.xml what you can actually see now let me sort of explain this we don't need that open anymore what you can actually see is that the bounding box is down here right now what actually happens in that generate tf record script is it's going all right find the object tag and then go to the fourth object and that's going to or object key 4 and that's going to contain your bounding boxes so what it's going to go ahead and go and do is it's going to go 0 1 2 3 4. but you can see the difficult doesn't actually contain all of our bounding boxes so what we actually need to do is change that index to index 5 which is down here and that should solve that particular error so if we go into that generate tf record script i'll show you how to fix that so if we go into back to our root folder tensorflow and then scripts and if we open up this generate tf record script you can see down here this is where it's actually going and getting the bounding box size now all you need to do is change the 4 over here to a 5 and that should allow us to pick up the bounding box successfully so this is going to mean that rather than going to object 4 and try to get the bounding box it's now going to get object 5 to get those bounding box metrics so if we save this now and run that tf record script again we should run successfully and there you go so we've got the line which is saying successfully created tf record file successfully create a tf record file for train and test and the way to verify this is if you go into again we'll start off at our root folder tensorflow workspace and this time going to the annotations folder which you can see there you should now have two additional files so one called test and one called train and as long as the file size is greater than zero then you know that you've gone and successfully created those tf record files and they should be in the format record file as well okay cool that's all well and good now the next thing that we need to do is copy our model configuration to our training folder so this script under step 4 is actually copying over the pre-trained configuration for our pre-trained files remember we talked a little bit about transfer learning right at the start well a model that we actually download from the tensorflow model garden is actually in the format of a config file so let me show you what that actually looks like so again we'll go into our ampr folder so youtube uh where were we sorry not youtube ampr and then tensorflow workspace and if we go to our pre-trained models and then this long folder here you can see that you've got this file called pipeline and if i actually open this up this contains all of the details about the pre-trained model that we're going to be using now you could choose to use a different model you could choose to use a more accurate one a faster one again if you want to see how this or how to choose these by all means do check out the longer tutorial but in order to get this to work we actually need to update a bunch of parameters in here so you can see that you've got all of these lines which say path to be configured we actually need to change this in order to be able to train our model now again this is all done within the jupiter notebook so first up what we effectively end up doing is we copy this config file from here i don't want to do it because i want the script to do it for us it copies it from here in to this model's underscore my ssd mob net folder so what you'll see in a second is if i go and run this script you can see that it's now going and copying that pipeline over so that's all well and good that's that step done these steps over here now actually go and update all of those different parameters that i was showing in there so if i go and run this well let's actually take a look at the config before we make any changes so if i open this up you can see no different right so right down the bottom you've still got all of these paths to be configured steps down here now if we go and run these cells down here and if we go and open up that config again you can see that that script actually goes and updates all of those configuration lines so you don't actually need to go and do anything additional to that it will do it all for you right now that brings us to the fun bit now we're up to training our model so in order to actually train our model we're now up to step six down here so you can do this inside of the notebook but i prefer to do it externally so that i can see the progress sometimes when you're running on a windows machine if you trigger this training script inside of your jupyter notebook you're not actually going to be able to see the output or the progress so i normally do it externally and i'll show you how to do that so what we're going to do is run the first cell under step 6 train the model and this will create our training command now if i run the second cell and the third cell you've actually now got this training command generated for you now we're going to make one tweak here and we're actually going to tune our model to train for longer so at the moment this particular model will train for 2 000 steps now we want it to be a little bit more accurate than that so we're going to actually train it for ten thousand steps so again this is significantly longer but will ideally give us a much more accurate model so if we change that number run those two cells again you can see that this is now our full command to go and train our model with our number of steps set to 10 000. now in order to run this we just need to open up a new command prompt go to our ampr folder so i'm going to go into d drive cd ampr and this is going to contain all of our stuffs as well as our virtual environment now in order to run this command we need to activate our virtual environment and then run it so let's go and activate that so remember it's just backward slash or dot backward slash amp sys backwards scripts backward slash activate so that's the command to actually go and activate it inside of a windows machine inside a mac it's slightly different i believe it's source let's actually bring that up it's going to be here it's going to be source and then in this particular case it would be amp rcs forward slash bin forward slash activate if you're running on a linux or mac this case we're on a windows machine so that's our activate script and you can see that our environment is now activated so we can clear our command prompt and then what we're going to do is copy that command that we just went and generated inside of our notebook so i'm going to copy all of this and then all things holding equal this should kick off our training run successfully so let's go and try this out now now again we might get errors if we do get errors i'll pause it and i'll show you how to solve it so let's run it and see if that runs okay so it looks like we've got an error there so this particular error is value error and numpy.nd array size change may indicate binary incompatibility expected 88 from c header got 80 from pi object now this is actually a pi coco tools error again it's pretty standard sometimes when you run through and actually install this stuff you're going to get errors to solve this i've actually got the answer inside of the error guide from the github repository so let me show you that so again this is from github.com tfod course all of these so like i try to maintain a repository of errors in here i haven't done it for a little while got to get back to it this weekend but this is that error there so value error numpy dot nd array size change may indicate binary incompatibility expected 88 from c header got 80 from pi object if you take a look at the error that we just got it is exactly that right so let me bring that a little bit better so that error there is exactly this so you can see that over there now in order to solve this you just need to reinstall pi coco tools as i've got in this guide so let's go ahead and run these two commands so first up we uninstall it and then we go and reinstall it let's do that again pi coco tools okay so it looks like it's successfully installed we're getting a whole bunch of these warnings again you can just ignore those so it's gone and successfully installed cython and pi coco tools now let's just clear this and try running our training command again so i'm just going to copy that command because we copied and pasted a bunch of stuff paste that in and let's see what happens now now we're getting no module in named cv2 so this is an opencv error which is a little bit weird but we can do that or install that relatively easily so we can just type in pip install open cv dash python that looks fine try running our training command again let's clear this so again sometimes when you go to in run it you're going to get these module areas just go on ahead and walk through them exactly as we did during the initial setup so let's run this third time's a charm i'm crossing my fingers no module named tensorflow add-ons all right so we now need to go and install tensorflow add-ons again as usual we're just going to copy this let's take a look at how we install that more often than not it's just a pip install so you can see pip install tensorflow dash add-ons so let's go and run that i'm just going to run that command all right that looks like it's successfully installed tensorflow add-ons down here all right what is this fourth time's a charm now let's try running our training command this is all common so if you're getting errors like this this is not uh uncommon see this okay this looking promising no module named gin all right we need to install gin now and again so this one to run that pip install gin config so let's copy that okay that's gin config installed try running our training command again okay this is looking promising so again if you get those module not found errors you just need to go ahead and install those libraries now when i went and ran that training command you can see that it's going through all of this stuff is popping up this means that your training looks like it's running successfully so ideally you'll know that your training has now formally kicked off once you get your first loss metric pop up so i'll show you what that looks like in a sec and then we'll step away let it run and then we'll come back alrighty so this is what you should see once your training is kicked off successfully so you can see here that we're getting tense flow we're getting how long it's taking to train per step so step 100 and it's taking 0.386 seconds per step and we're also getting some loss metrics now in this case we want our loss metric to be as low as possible to ideally get a model which is performing well so again we're going to let this train and ideally by the end of the 10 000 steps so we should be able to see and test out our model which has then been fully trained so we'll let that run and then we'll be right back all righty and that is our object detection model now trained so you can see we went all the way up to step 10 000 and that took a or overall it was taking a time of about 11 to or point 11 to point 12 seconds per step the final run that we sort of had through this left us with a loss of 0.406 but you can see all the way through it sort of varied up and down all the way up to 0.6 and down to around 0.2 i think was the lowest i saw but again this is our model now trained now if you actually go in to the so if we go back into our ampr folder into tensorflow workspace models and then my ssdmobnet you can see we've got a checkpoint all the way up to checkpoint 11. so this checkpoint is actually the latest instance of our trained model and it's exactly what we're going to be able to use to go and be able to perform our ampr now on that note however we have now gone and finished the training of our object detection model so we went and updated our labels we created our tf records prepared our configuration and trained it now it's time to go and detect our plates so in here what we're actually going to be able to do is detect our plates from an image so we'll be able to go and grab out our license plate we'll also be able to detect plates in real time so what we can actually do is we can use that same model hold up our phone or hold up a instance of a plate you could even deploy this in real time and go and run it out in the field and you're going to be able to extract the plate or at least detect the plate then eventually what we're going to do is apply the ocr components so in this case let's go ahead and test out our model so we went to all that work we may as well go and test it out so again we're going to jump back into our notebook in this particular case and this is all still running now what we're going to do is we're going to go on ahead and skip on down to step 8 which is load train model from checkpoint so if we go and run this what we now need to do is in this line here we need to change the checkpoint that we're actually going to be loading into our model now remember i was showing that the latest checkpoint was checkpoint 11. so all we need to do is change this line here which says checkpoint or ckpt dot restore and then it's got a big file path we just need to change this little component here which you can see right now it says checkpoint five just need to change that to checkpoint 11. so if we go and load that this is going to load up our model into our jupyter notebook and allow us to perform some object detection so then if we go to step 9 we can detect from an image so if i run that cell and run our category index all we need to do to actually perform some detection is change the file path for our particular image now in this case we're looking inside of the test folder and at this particular file name here but this is not a valid file name we need to get an image from our data set so let's go ahead and grab a file from our images so if we go into again our root folder ampr then go into tensorflow workspace images and then test let's say we wanted to detect uh this particular plate here so this one is a viper by the looks of it so say for example we wanted to go and do that so let's grab that file name so it's going to be cars411 dot png so if we go and replace this file name here dot png and then if we go and run this next cell this is actually going to go on ahead and perform our detection so if we run this let's take a look and there you go so it's now gone and accurately detected the plate from our car so in this case it's gone and done on this particular image and it's detected that that's our number plate there so you can see it's viper now if we went and did it on a different image we could do that as well so let's grab uh this image here so four to five so all we need to do to do it on a different image is just change the file name here let's run it again you can see it's gone and accurately detected that as well so you can see it's actually performing reasonably well when we're actually going to go on ahead and detect our number plates now this is a key thing and a key reason that i've architected this to use object detection because what you'll notice is that the plates actually vary right so the viper plate actually had a different format this r8 plate had a different format so you want to ideally or leverage this object detection model because it's going to perform much better than leveraged leveraging traditional computer vision methods now in this particular case let's go and try it out from our webcam so i'm going to run this next cell which is cell 10 and let's see if we can go and detect a license plate from our webcam so i'm going to get my phone out and find a plate let's see so as soon as you run this cell you should get a little pop up down the bottom and you can see it's detecting my head now let's get an image of a number plate and see if it's detecting that let's rotate it let's see that so you can see it's accurately detecting the plate from my phone pretty cool right and it's doing it with reasonable degree of accuracy in spite of the reflection that's appearing so you can see already we've already done the bulk of the work so we're now able to grab our plate and detect our different plates in real time now the next thing that we need to do is actually start performing a little bit of ocr now specifically what we're going to do for that is actually extract the region that we need and then go on ahead and actually apply our ocr but as it is if you just wanted to go and build a license plate detector you've sort of got that up and running now it's all working it's able to detect different plates what we're going to go ahead and do now though is take this a little bit further and start applying our ocr so that brings us to step 5 applying our ocr now in order to do this we're going to be using our library called easy ocr so this is going to allow us to take the text or detect text from our particular image and work out what's actually being rendered there now again if you wanted to use a different ocr engine you definitely could this one's an open source one hence why i'm using it there are more accurate ocr models out there so if you wanted to take this by all means go ahead and check that out now key thing is that easy ocr does run on pytorch which means that we need to give a little bit of room to our gpu to be able to go in ahead and run ocr as at right now if i actually show you my performance so right now my gpu is almost completely consumed by tensorflow and by default this is what tensorflow is going to do it's going to take up all of the gpu that you've got available and not leave anything else now what we're going to need to do is allocate a little bit of gpu to or leave some gpu remaining so that we can run our pi torch components so i'll show you how to do all of that now so let's go ahead and jump back into our notebook now what i'm actually going to do first up is we're going to restart our kernel and this should ideally release our gpu so let me show you this so right now it's all consumed now if we go and restart our kernel what you'll see is that we'll release it and you can see that we've gone and released all of our gpu memory there now now what we can actually do is write a little bit of code to prevent tensorflow from consuming our entire gpu so let's go ahead and do this now i'm going to do this just above or actually just below this cell here which is a load train model from checkpoint so we'll leave our imports the same what we are going to do though is we're going to add a new cell here to prevent gpu complete consumption so let's go ahead and write this and then i'll take you through each step okay so before i go ahead and run that let's actually take a look at what we've written there so we've written let me just bring this into a single line so it doesn't look as intimidating as what it actually is so we've got one two three four five five six lines of code now the first line of code is actually going on ahead and grabbing our gpu so in order to do that we're in gpus equals tf.config dot list underscore physical underscore devices and then we'll pass through our gpu so if we actually or if i actually show you that line so if we actually extract that out so this is actually going to show us the gpus that we've got available and if we store that in a variable like so or gpus we can actually take a look at what we've got so in this case i've only got a single gpu on our machine so i'm just going to be setting our memory limit on that particular gpu then what we're doing is we're checking if we do have some gpus available so in this case if we said if gpus print yes we have them so in this case we do have gpus so we're going to continue onwards then what we're doing is we're putting that inside of a try accept block so if we do have a gpu and we can try it out then let's go ahead and run this line here so this line is basically saying try colon tf.config.experimental dot set underscored virtual underscore device underscore configuration and then we're passing through our first gpu which is our only gpu in this case but you could loop through and do multiple gpus if you had multiple gpus then we're passing through this line which is effectively what limits our memory or our memory consumption for tensorflow so inside of square brackets we're passing through tf.config.experiment configuration and then we've specified our memory limit and that's actually spelt wrong memory limit that looks better memory underscore limit equals five one two zero so this means that we're going to be allocating five gigs of virtual memory to tensorflow on our gpu and then we're effectively saying if that doesn't work then we're going to accept our runtime error so accept runtime error as e and then print at e so effectively we're grabbing our gpus we're checking if we've got a gpu and if we do then we're going to set our memory limit five gigs on our single gpu and if that doesn't work then we're going to print out the error so if we run this now ideally this should limit our memory consumption so again it doesn't look like we've got any errors there so we should be good to go and it doesn't look like our gpu's gone and fully consumed yet because we haven't actually loaded our data so now what we want to do is actually go on ahead and load our data so if we run this cell here we need to go and load in all of our different file paths so let's go ahead and run this so to be able to load that we need our file path so if i'm going to run cell 1 2 3 4 and then if we scroll on down we also need the name of our labels so if we run this under section 2 label map and if we scroll on a little bit further we're going to need this config as well so now we should be good to go so if we go and load this up that's run successfully and if we take a look at our gpu so it looks like so you can see down there that we've only consumed 5.8 gigs so again it hasn't gone and completely consumed our gpu this time which is ideal because now we want to use easy ocr so that brings us to our easy ocr component now before we go ahead and do that let's actually bring in an example so if we go and run through our image detection again we're going to need an image to actually test this out so i'm going to run the cell or cell step 9 to actually go and detect our license plate and i believe this was the audi from last time so let's see that run and again we've gone and detected our license now we can actually take a look at this particular line here so this is actually representing our image with its detections so if i grab this you can see that from here we're actually able to get our image plus all of our detections now our detections from our tensorflow model are actually stored inside of this variable here called detections so that contains a whole bunch of stuff so if i take a look at that so in here you've actually got a whole bunch of different keys and specifically you've got the detection boxes the detection scores the classes the raw detection boxes raw scores the multi-class scores the anchor indices as well as the number of detections now what we're actually going to do with this is we're actually going to process that and leverage that to be able to go and apply ocr now first up what we need to do is actually import ocr so let's actually create a new segment here and i'm just going to call it so what are we going to do so this is going to be easy ocr we'll just call it apply ocr detection and then the first thing that we need to do is actually go ahead and import easy ocr well we actually need to install first so let's go ahead and install it okay so to install easy ocr all we need to write is exclamation mark pip install easy ocr you can see it's currently running so we'll give that a couple of seconds and then we should be good to go and that looks like it's all run successfully and this will also import or install pytorch now a key thing to note is that this is going to install pytorch without cuda so if we want to leverage our gpu we need to go and install pytorch as well so to do that you can just go to pytorch.org and then down here we can actually go and grab the appropriate version so we just choose our pi torch build and so again this is all of the pytorch website so i'll include a link to this as well as the entire code inside of the description below so we're just going to choose the stable build windows we're going to choose that we want to install it using pip using python and in this case i've got cuda 11 on my machine so i'm just going to select cuda and i'm going to copy this command here then if we go back into our notebook we can put type in exclamation mark paste that in and i'm going to remove this 3 so this is going to ensure that we actually leverage gpu acceleration when it comes to leveraging ocr so let's run this and this should install pi torch with cuda acceleration alrighty so that is easy ocr and pi torch with cuda acceleration now installed now the next thing that we want to do is actually import easy ocr so to do that we can just type in import easy ocr and that's all imported successfully so if you get any errors just try restarting your kernel this should ensure that you're definitely picking up your latest easy ocr version now the next thing that we want to do is actually go ahead and grab our different detection boxes which pass our threshold so by default when we leverage our tensorflow object detection detection code you're actually able to set this minimum score threshold and this basically dictates over what certain threshold you're going to render your results so in this case it's set to 80 percent hence why we can see our license plate showing up as 88 there but if we set that lower then we'd pick up lower quality detections as well now what we want to do is actually apply that same logic to our ocr so let's go ahead and do this so we're actually going to grab our thresholds over a certain level first up we need to set a variable which is going to be our detection threshold so let's do that first and then we'll go on okay so that is our detection threshold variable now set up so in order to do that i've written detection underscore threshold equals 0.7 so this is our threshold over which we're going to render our results so if it's below that then we're not going to bother now the next thing that we need to do is actually grab our image grab our scores our classes and our boxes that actually pass that threshold so let's go ahead and do it alrighty so we've gone and written four lines of code there now what we're effectively doing is first up we're grabbing our image so image equals image underscore mp underscore with underscore detections and this is the same as the image that we're getting from up here then what we're doing is we're extracting three different components so we're actually grabbing the scores which surpass this detection threshold and for that we're effectively looping through each one of the detections inside of our detection scores key within our detection variable so if i take a look at that detections and then type in detection underscore scores so you can see that here this represents all of the different ac or confidence metrics right so in this case we've got one which is 0.884 which is the one that we had up there so what we're effectively doing in this line is we're looping through each one of these values and we're only returning back the ones that surpass that particular threshold so if i actually type in uh whether or not it's greater than detection threshold you can see that right now only the first one actually surpasses that result so what we've effectively got inside of this scores variable is just the ones or just the scores that surpass that threshold which in this case is that 0.884 value so in order to do that what we've done is written scores equals list and then inside of our list function we've written filter now our filter is effectively what's going to allow us to loop through each one of those values inside of this detection scores array here so the full line is scores equals list and then inside of that filter and then we're using lambda x so this is going to allow us to loop through each one of those variables using a lambda function and then we're basically saying if x is greater than our detection underscore threshold then it's going to bring back that result inside of that list and then the second parameter that we've passed through to our filter function is actually the detection scores array which is that there then we've gone and done and applied some filtering to our boxes and our classes as well so if i show you the detection boxes section these represent all of our box coordinates and then if i show you the classes and these are our box coordinates that are not resized for the size of our webcam so we need to apply that as well so if we take a look at our classes these represent all of our classes and in this case we've only got one class so 0 represents license so to do the boxes we've written boxes equals detections and then true that will pass through square brackets detection underscore boxes so this is effectively accessing a detection underscore boxes key and then we're applying some index filtering so in this case we've passed through colon len and then scores so this is effectively saying grab everything that passes or the same length as our scores array so in this case we only had one value inside of our scores so we're only going to bring back our result which is that then we've gone and done it the exact same thing on our classes but instead of applying our filtering on our detection boxes we've gone and applied it on our detection classes so if we take a look at our classes now you can see that we've got our class back as well now the next thing that we need to do is actually go and grab our image width and our height because we need to recalculate our box coordinates with respect to our image size so let's go on ahead and do that and the reason that we're going to do this is because we're going to then apply some filtering on our image to grab our region of interest so let's do it okay so in this case we've now gone and grabbed our width and our height so in order to do that we've written width equals image dot shape and then we've grabbed the first value using some indexing so square brackets 0 to grab our width and then we've gone and grabbed our height which in this case it looks like it might be the other way around let's just go and change that so this should actually be as one zero wrong way around my bad okay so our width is going to be image dot shape and then grabbing the first value so in this case it's width it's going to be 500 which you can see there it looks like it's about 500 and then we've gone and grabbed our height and our height is going to be our first parameter so in this case the full lines of code that we've written is width equals image dot shape and then grabbing the second value which is index one and then our height is going to be height equals image dot shape and then we've gone and grabbed the first value which is index zero so now we've got our width and our height parameters now what we can actually do is go through and loop through our detections well in this case we've only got one but if we had multiple that pass that threshold we'd actually loop through and extract each one of these and apply some ocr so let's go ahead and do that so first up what we're going to do is we're going to loop through all of the boxes that we've got inside of this boxes variable up here so to do that we're in four idx which represents index comma box in enumerate boxes so this is going to allow us to loop through each one of these now what we're going to do is filter out the region that represents our region of interest which in this case is going to be this region here and then we'll go in ahead and apply ocr so let's do it all righty before we go any further i wanted to sort of show what we're actually doing here so in this case first up what we're doing is we're grabbing the region of interest now in this case our region of interest is going to be our box coordinates multiplied by the dimensions of our image so what we've written is roi equals box multiplied by height comma width comma height comma width so if we actually take a look at our box value so let me print that out print box you can see that we've got all of these coordinates here but these coordinates represent our coordinates for our detection without respect to the actual image size so we need to multiply them by the height and the width to get our actual roi coordinates which if i print out here you can see that these are before pre-processing these are after pre-processing so after they've been multiplied by our image coordinates and then what we've gone and done is we've actually gone and extracted our region of interest so you can see here that we've actually got our plate from our particular image which is this over here now we haven't actually gone and applied any updates to our color so if we actually went and plotted this out or change our color so if we did cv2.cvt color color bgr to rgb so remember when you're loading images or when you are loading images from opencv and visualizing them using matplotlib you need to convert them from bgr to rgb so by default opencv is going to work with the color format bgr but in order to render it using matplotlib it needs to be in rjb so if i do that now you can see that we've got the appropriate coloring exactly as we had up here now to extract this region of interest what we've gone and done is some indexing so we've gone and grabbed our image which we had up here and then we've gone and passed through our coordinates which we grabbed from our region of interest so in this case we've gone and filtered by our x and our y variables to do that written int and then to that of past roi or region of interest coordinate zero colon int and then region of interest coordinate two and then we've gone and pass through comma int roi index one colon int roi index three so this line here is effectively going and filtering in through our numpy array to be able to grab only the region which represents our plate and then to visualize it with written plot dot i am show and then to convert its color cv2 dot cvt color we've passed through our region that we've gone and extracted and then we've gone and passed our color conversion code which in this case is cv2 dot color underscore bgr to rgb which gives us this but we haven't actually gone and applied to any ocr now so let's go ahead and apply our ocr and that should effectively give us our different components okay so in order to go ahead and apply our ocr i've gone and applied three additional lines of code so first up we're setting up our easy ocr reader so to do that i've written reader equals easy ocr dot reader and then two that will pass through the language that we want to use which in this case is english and then we're going and grabbing our result so ocr underscore result equals reader dot read text and to that we've passed through our region so we're actually passing through this block of or this image to our ocr parser to be able to go and extract our plate results and then what we're doing is we're actually going on ahead and printing that out so print ocr underscore result so if we run this now we should effectively get our printed plate or the text from our plate so in this case it's getting vaguely alright so in this case it looks like it's gone and extracted our plate and this is the result there so 6526 gh or jhd which in this case it looks like it's probably grabbed the g incorrectly but in you can see it's a little bit blurry there so it might not be as accurate as possible so for a baseline model that's not working too bad all right so it looks like it's getting a reasonable estimation of our plate so it's actually detecting our plate might not be getting the actual text appropriately but again you could try a different ocr model if you wanted to but this doesn't actually show one of the core problems that i actually noticed when building this up which is if you've got multiple lines of text sometimes the easy ocr reader or any ocr reader for that matter can get confused between what is actually the plate versus what is just periphery text so i'm going to grab another image and sort of show you how this actually occurs and how to actually resolve it so in this case this is the image that i want to take a look at so you can see we've got bg02.qt but then we've also got this text down here which is not the actual plate it's just the state that that particular plate is actually referencing so if i actually grab this image and what i'm going to do is i'm going to place it inside of our test folder and then we'll try it out so i'm just going to go into our tensorflow workspace images and then test and i'm going to actually let's not paste it in test because that will throw off our training if we do it again and i'm going to paste it in there so it's called test image 2 and it's not inside of our test folder so we can actually run our ocr on this so simply by going and changing this so we're going to remove that folder there and we're going to grab the name of the plate which is test image 2. so let's do that test image 2 and then if we run our detection again looks like we've got an error there images must have either 3 or 4 dimensions that normally means that we haven't got the file path right oh okay so it's not inside of our test which i thought we did let's check that is it png might not be it's jpeg that's why it's throwing an error let's try that again jpeg let's change update our image path run that detection now okay so it's detected our plate appropriately and you can see it's got a score of 0.91 which means it's going to pass our threshold of what did we have down here 0.7 but you'll notice that there's actually some extra text in here now so we've got new south wales as well so if we go and run our detection pipelines remember down here this is going to run it so if we just delete this scores bit and delete this height so if we run these three blocks which is getting the image grabbing the scores the boxes and the classes that pass the threshold the width and the height and then applying our roi filtering and ocr we should ideally be able to see the text that we get out of this so let's run this now in this case you can see that we've gone and grabbed a whole bunch of different types of text so this is our result down here okay so we've got bg02 we've got news south and wales detected as well so it's accurately grabbed the plate number in this case it said o is q it said q is actually o that's fine again we've got to improve the ocr we're grabbing the plate appropriately now the problem is that again we are there is more text here than we actually want so what we'd actually need to do is filter through this so if i showed you the ocr result you can see that we've actually got multiple results in there so if we grab result 0 that is the plate if we grab result 1 that is new if we grab result 2 that is south and if we grab result 3 that is whales so what we actually need to do is apply some filtering to be able to grab our main detection which in this case is going to be this region here now the best way that i actually thought through to actually do this is to actually filter based on the size of the region so the nice thing about easy ocr is that you actually get the coordinates for the text as well so in this case this represents all of the four different coordinates for our particular image so in this case it's going to be 591 by 242-836-242-836-3155 so our particular region is going to be this region here right now what we can do is calculate the size of the text region as well and we can actually apply some additional filtering based on the size of the text detected so let's go ahead and write a function to actually do this filtering so i'm going to create a new function let's actually create a new section so we're going to call this region filtering actually it's really ocr filtering and we're going to create a new function called filter text and to that we're going to pass through a couple of keyword parameters so let's apply these parameters and then we'll take a step back and take a look okay so first up what we're going to do is create a new function and we're going to call it filter text so to do that written def filter underscore text and then to that we've passed through three arguments so region the ocr result and then the region threshold so the region is going to be this particular image here or whatever our plate region is the ocr result is going to be our text that we've got extracted from easy ocr so an example of this is that and our region threshold is actually going to be a detect similar to our detection threshold but now the threshold that we're going to pass through here is the size threshold so in this case say we wanted our or we wanted to only detect plate regions which or plate text which is 60 percent of the plate well in this case we'll more than easily pick that up because bg02qt is definitely more than 60 of that plate so ideally it should filter out these detections down here so let's go ahead and apply this so let's actually create a new variable for our region threshold and we're going to set that to 60 then let's go and finish this particular function and we'll take a look at how it works okay before we go any further let me sort of explain what's happening here let's just print out the length the n g th and width okay so our filter text region is first up going to calculate the size of our region and effectively what we're doing here is we're grabbing the height and the width and we're multiplying those so standard rectangle area or area function for our rectangle so width times height equals the area which is what you can see here it's a rectangle underscore size equals region dot shape and then we're passing through the first value which in this case is index zero multiplied region dot shape and then we're grabbing our second index which is index one so again that's going to give us the size of our rectangle and then what we're doing is we're creating a variable called plate and this is going to be where we store our results then what we're doing is we're looping through each of our results inside of our ocr results so this is akin to doing this so for result in ocr result print result so that's effectively what we're doing next so we're looping through each one of these results and then we're grabbing the length and the width of that particular result so in this case what we're doing is we're first up grabbing our first key from our result which is going to give us this so if we take a look result 0 that's going to give us all of our coordinates now if we grab coordinate 1 that's going to give us our initial coordinates and then if we grab coordinate 2 oh if we actually grab coordinate and this is going to be coordinate one so what we're doing is we're subtracting this from this to be able to grab our length so if i subtract those we're effectively getting our length now in this case you can see it's inside of an array so what we can do is just type in np dot sum and this is going to convert it to a single value so that's giving us our length then what we need is our width now in order to grab our width we're just going to change the coordinates that we pull so i'm going to grab the third coordinate and the second coordinate and that's going to give us our width then what we can do is effectively we're going to multiply and this is what these two lines down here are actually doing then what we're going to do is multiply those together to be able to get the size of the region for our ocr text so if i multiply length by width that's effectively going to give us the area for just this text here and just this text here just this text here and just this text here then what we can do is do a bit of a filter so if that particular region passes this region threshold then ideally that should be our text so let's finish this up and then we'll take a look okay that is our filtering algorithm now done so the last couple of lines of code that we've gone and added are if length multiplied by height which are the length and the actually should be length multiplied by width actually this should be height and bad length multiplied by height divided by our rectangle size which is what we had up here so effectively what we're doing is if the length multiplied by the height of just this bg.02.qt text so if this region is more than 60 of the entire plate region then we're going to return back that particular text so if the length multiplied by the height divided by the rectangle size is greater than that region underscore threshold then we're going to append the result from our ocr component so if we grab a result one that actually gives us our text back so this bgo2 qt or ot is effectively what it's detected in that plate so if we go and run filter text now let's try it out filter text if we pass through our region ocr result and our region threshold see if this runs and you can see it's gone and only returned the text which has that or the larger block of text now if we drop that threshold to say for example we set it to two still grabbing everything let's say if we change it to 0.05 so you can see that it's now returning more text from that plate so it's grabbing south and wales so again this sort of gives you the ability to filter through the results that you're actually getting back from your ocr engine now i thought that this was really really important to show because what i actually noticed is that if you couldn't effectively extract just the components that you wanted then you're really grabbing back everything from the plate and not just the components that you need now that's done what we can actually do is bring this function or this component over here into one single function so right now we've got sort of two blocks of code so we've got this ocr code and we've got this ocr filter component but we don't have it brought together so let's create one additional function which brings both of our ocr components together now i'm going to set up this function but really all we're going to be doing is copying down what we've already written so we're going to grab this component and bring it in here and in this case we can get rid of this image line because we're actually going to be passing it through raw and then tap this in and then what we're going to do is grab our width and our height extraction paste that into our new function which is down here and i'm going to comment this up just so we can sort of see how this all fits together and then we're going to grab our looping let's paste that in as well and then what we want to do is actually leverage our filter text so again we'll come back to that in a second so if we take a look at our new function so what i've gone and written is def ocr underscore it so we're going to call it ocr it and then we're going to pass through our image our detection so if we take a look at right now the image that we're working with image mp with detection so it's this so if i type in plot.i am show so we're going to pass through our main image which has everything in it we're then going to pass through our detections then going to pass through our detection threshold and then we're also going to pass our region threshold which is what we need for our filter text because we're going to call it from within there right so and right now it's set pretty low okay so that's looking okay now what we need to do is actually bring in our filter text component actually let's comment this up first so first up we're grabbing our scores above threshold scores boxes and classes and we're grabbing our full image dimensions and then we're going through and enumerating through all of our boxes to grab all of our different components so we grab our roi so i'm just going to delete all these prints because we don't need these just yet so we're grabbing our roi which is remember our box coordinates multiplied by our image dimensions we're actually extracting our region which again is just going to give us this component rather than the whole image we're then using our easy ocr reader to go and grab our ocr results now what we need to do however is use our filter it method which is what or filter text method to go and filter through all of the results that we get from our ocr engine so let's go ahead and apply this down here okay i think that is our ocr it function now done so we've gone and added what is that three lines of additional code so i've gone and applied our filter text method which is what we just defined up here and to do that i've written text equals filter underscore text and then we're passing through our region which is this our ocr result which is this and then our region threshold which we had over here and then we've still got our plot.i am show to show our results we're also going to be printing out the text that we get back from our filter text method and we're also going to be returning that text back so return text comma region so let's go ahead and test it out so i'm going to create a new cell and we're going to call ocr it pass through our image with numpy detections and then we're going to pass through our detections our detection threshold and our region threshold let's see that work okay that's looking promising so it looks like we've gone and accurately extracted our plate and we've also gone and extracted our region now we're getting all of this other stuff as well because we still have a couple of stragglers in terms of our print so we can actually remove this print over here so we don't need that anymore from our filter text and we don't want to what are we also printing it looks like we're printing out all of our text oh we're displaying this region so if we actually extracted these so text comma region sorry my bad this should clean it up there you go all right cool so that gives us our ocr detection method now so we're so if i actually extract or don't extract these variables then it's going to display the entire region which is this here which is all these texts which are showing there which we don't actually need but when we do go to export our results we do want that component but in this case we're fine for now now if we actually extract those variables for text and region then we only get the results back that we need so let's take a look and there you go so to do that or test out our model written text comma region equals ocr underscore it and then to that we'll pass through our image underscore mp with underscore detections which is the broader image so this whole image here and then we've gone and passed our detections our detection threshold and our region threshold now we could change our region threshold to make it a bit higher so we don't grab this south and wales component so let's change it to 0.6 if we go and run it again now we should only get the actual plate number which you can see there so that's all working that is so good so we've now gone and actually gone and grabbed the appropriate component from our plate now what we could actually do is go and apply this ocr method to our real-time detection so this is sort of all doing it on a template image but we could actually go and do it in real time so what we need to do is just update this real-time detection method or this real-time detection cell that we've got here to be able to do that so let's go on ahead and make those updates and then we'll test it out okay so i've gone and written an additional four lines of code there to be able to go and apply our ocr in real time so what i've gone and written is inside of a try except i've effectively gone and just pasted or written this ocr method down here now we're doing almost exactly the same so we've written text comma region equals ocr underscore it and then to that with pass through image underscore mp underscore with underscore detections pass through our detections pass through our detection threshold and pass through our region threshold so ideally when we run this block of code now when we actually go and trigger it looks like it's still running i should quit out of that so when we actually go and run this this should actually go and perform ocr on every valid detection that it's actually got so let's go ahead and test this out so if we go and run this cell now ideally you should get a little python pop up towards the bottom of your screen and if it just pops up gray and closes just run it again okay so that's all well and good so you can see let's bring up a plate on my phone if i bring that up to the screen now so you can see down the bottom it's detecting our license and you can see it's actually in real time it's detecting the plate so again there's a little bit of glare let's bring it back in front of the camera okay bring it down and if we go and take a look at our results so you can see it's going to detected our text there so six t r j two l l so again pretty close but again you can sort of tweak this use a slightly different ocr method but again it's detecting our plate and it's at least extracting some text we try out a different plate let's grab another one about this one let's see if it's grabbing that one looks like it's grabbing it there you go so it looks like it's getting abcd555 abcd five five five so you can see it's working how good is that so that is the first part of our ocr system now up and running so it's been a long bit of a trek to be able to get to this point but what we've now gone and done and we can actually quit out of this so it doesn't smash our gpu so if you open up the little pop-up hit q and that'll close it down so what we've now gone and done is a whole bunch of additional code so we went and wrote the ocr code that actually applies it to our detection we did our ocr filtering we then applied or created our ocr it method now the last thing for us to do is actually go on ahead and save our results now what we can do is we can actually because we've gone and done all of this pre-processing all we really need to do is save our region and save our plate so let's go ahead and wrap this up and then we should be done so what i'm going to do is i'm going to just create another function above section 10 and this is going to be our save results section so save results i'm going to convert this to markdown and then all we need to do is bring in two dependencies so we need csv so import csv and we need uuid import uuid so csv is going to allow us to save our results down to csv and we're going to save the image name as well as the plate detected and uuid is going to allow us to uniquely name our images now all that we really need to do is create a function to actually save our results so let's go ahead and do that and we'll wrap it up okay so before i go any further let's take a bit of a break and see what we've got here so i'm creating a function called save underscore results so to do that we'll run def save underscore results and then to that will pass in through a few different things so we're going to pass through the text that we get from our ocr model the region so as in this over here the file name that we want to export our results to and these are our text-based results and then the folder that we want to store our region images in and then what i'm going to go and do is create a unique image name now in this is the reason that we've brought in uuid so uuid allows you to create a unique uniform identifier so if i type in uuid dot uuid 1 this is going to give me this unique identifier here now the nice thing about this is this means that we're not going to have overlap when it comes to writing out our images so what we're doing here is we're creating some string formatting so this is going to create a unique file name which effectively gives us a unique file name dot jpg for our image so to do that i've written image underscore name equals and then quotes squigglybrackets.jpg.format so this allows us to perform some string formatting here and then to that i've passed through uuid.uuid1 to be able to leverage and create this unique identifier then all we need to do is output our csv results and save down our image so let's wrap this up and then we'll test it out okay that is our save results method now done so i've gone and written an additional four lines of code there so the first one is going and saving our image so cv2 dot i am right and then to that i'm passing through the folder path where i want to actually save that image so in this case i've written os.path.join pass through the name of the folder that we're going to save our images in and we'll set that up in a second and then we'll pass through the name of our unique image which is what we just created up here and then to that we're then passing through the image itself which in this case is going to be our region of interest which we had up here then the last thing that we're doing is we're actually writing out our csv so with open csv underscore file name and then we're setting our mode to append so equals a we're setting our new line formatting as f so we're going to be able to work with our file as the variable f then we're creating our csv writer so csv underscore writer equals csv dot writer passing through our file passing through our delimiter passing through our quote character and passing through how we want our quoting to be in this case quote minimal and then we're using the csv underscore writer dot right row so this is actually creating our csv right here this is actually using the right row method to write out our csv text so csv underscore writer dot write row and then we're passing through our image name as well as the text so if we go and test this out now let's just make sure we still got a region we can work with that looks fine text doesn't look like we've got anything there let's just set it temporarily let's actually go and run this again so if we bring in our image from up here okay so that's looking good then if we so that should give us our detections right so let's try this out now do we have text nope we need to run our ocr on it so let's ocr it cool all right so we've got our text now let's test this out so text is fine region is fine all right cool so what we now need to do is set up a csv as well as folder path so if we go into our folder if we go into our root folder let's just create a new folder called detection images underscore images so we're going to write out all of our image results or all of our regions to this folder here so detection underscore images so what we're then going to do is pass through our csv file name and our folder path which is going to just be detection in images let's go and test this out now save results pass through our text password region pass through our csv file name we'll just call this detection underscore results.csv and then pass through our folder path which you remember is going to be detection underscore images and remember this folder so detection underscore images could be anything that you like so if you change the name here all you need to do is pass through a different folder path in this particular case so if we go and run this now that looks like it's run successfully so it looks like we've got a detection result csv and that's got the name of our image as well as our plate so bgo2ot if we go into our detection images we've got our plate saved so that is our save results method now done now what we can go and do is run this line of code in alongside our ocr method and this should save our results as we're detecting in real time so if we go and do that now so save results and then we're going to pass through our text our region our csv file name so we should change this so let's call it real time results.csv and then we're going to keep our folder name the same so we're going to call it detection images so we're going to pass through the string detection images so all we've done here is we've grabbed our save results method which we set up over here and then what we're doing is we're applying this just under our ocret line so this should effectively run save results and then it's going to grab the text it's going to grab the region and then it's going to write out our results to real time results.csv and write out our images to detection underscore images so that particular folder so if we go and run this now this should kick it off in real time all right so we've got our face showing up there let's just bring our folder structure over here bring this over here okay so right now we've only got that image that one image and that was from our initial test now if i bring up the plate again let's test this out so this the abcd555 plate so let's test it again it's going to go a little bit slow once we get our detections so you can see it's now accurately outputting our images so let that focus in a little bit so you can see it's outputting out our images into our folder it's making our detections pretty sweet right now if we throw up a different plate what about this texas one let's try this so again it's detecting our texas plate pretty accurately pretty cool right so that's your real-time ampr system working check it out right how cool is that now again i'm getting a little bit of glare because i've got some weird lighting at the moment but if you had a better uh if you had better lighting conditions this is obviously going to perform way better now let's actually take a look at the csv results so if we open up realtimeresults.csv you can see so it looks like it's picking up some of them but you might need to tweak this so it looks like the lighting conditions are throwing this out a little bit so the actual plate was lhd 8448 so it looks like it's got lhu8440 but again you could play around with this throw in some different plates test it out so on and so forth but that effectively shows you what's possible with real-time ampr so we've gone and done a ton of stuff in this tutorial so we went and trained our object detection model to be able to detect plates we then went and set up easy ocr built our ocr it function built our filtering algorithm and went and actually applied it in real time now again if you have any trouble with this or you need a little bit of a hand by all means hit me up in the comments below all the code that you see here is going to be available on github so if you want to pick this up and run with it by all means do so and let me know how you go but on that note that about wraps it up thanks so much for tuning in guys hopefully you enjoyed this course if you did be sure to give it a thumbs up hit subscribe and tick that bell and let me know what you thought of the ampr video thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 281,650

Keywords: number plate recognition camera, number plate recognition python, number plate recognition using tensorflow, number plate recognition using python, number plate recognition system, number plate recognition using yolo, automatic number plate recognition, license plate recognition, license plate detection, license plate recognition tutorial, license plate detector, automatic number plate detection, number plate recognition

Id: 0-4p_QgrdbE

Length: 115min 38sec (6938 seconds)

Published: Mon May 03 2021