in this video we're going to work with
automatic number plate recognition and this is exactly what you will be able to do with this
tutorial you can see that we are detecting all the license plates in this video and we're also
reading the text from these license plates we're using 100% python we're going to use an object
detector based on yolo V8 we are going to do object tracking and we are going
to read the text
from the license plates using easyocr
so this will be an amazing tutorial my name is Felipe welcome to my
channel and now let's get started so let's get started with this tutorial today we are going to
work with automatic number plate recognition and let me show you a few resources a few repositories
which are going to be super super useful for today's tutorial the first one is Yolo V8 because
we are going to be detecting license plates and then we're going to be reading the text from
the license plates right and in order to detect our license plates we are going to use an object
detector which is based on yolo V8, so yolo V8 is going to be super super important in today's
tutorial and I'm going to show you more details in a few minutes but for now let me show you the
other repository which we are also going to use in this tutorial and it's going to be super super
important and it's sort it's an object tracking algorithm which is called sort because today we're
going to do object detection and we're also going to do object tracking this is going to be an
amazing tutorial and in order to do object tracking we are going to use sort and then once we have detected
the license plates once we have implemented all the object tracking once we have done everything
we need to do we are going to read the content of the license plate using easyocr so this
is a python Library which is going to be super super super important in this tutorial and now let
me show you the data we are going to use in this tutorial let me show you the video we are going to
use in order to test the automatic license plate recognition software we are going to use in this
tutorial you can see that this is a video of a highway and we have many many cars which are going
through this highway and the important thing about this video is that all the cars... we have like a
very very frontal view of absolutely all the cars and most importantly we have a very frontal view
of all the license plates right you can see that for absolutely every license plate we detect in
this video we have a very very very frontal view and this is an ideal point of view to build a
project like this so this is exactly the video we are going to use in this project and now let me
show you something else if I go to Google and I search for license plate and I go to images let me
show you something you can see that we have a lot of diversity when it comes to license plates right
we have many different types of license plates we have some license plates which are comprised only
with numbers like this one then we have other license plates which are only letters like these
two and we have many many different examples we have many different types many different formats
I would say that absolutely every single country, absolutely every single state, absolutely every
single time in history have its own a license plate format right its own license plate style
its own license plate system right there are many many different type of license plates there's a
lot of diversity when it comes to license plates and obviously that it it's very very challenging
to build an automatic license plate recognition software to deal with absolutely every single type
of license plate right, it's... I'm not going to say it's impossible it's not impossible but it's a
very very challenging task so in order to make it more simple in order to simplify our problem we
are going to focus only on one very specific type of license plate which is this one we are going
to be working with the United Kingdom license plate system, with the United Kingdom license plate
format, which is comprised of seven characters the first two characters are letters then we have two
numbers and then we have three more letters right so we have two letters two numbers and three
letters and this is the exact structure of the license plate type we are going to be working
today in this tutorial right, this is the exact same type we are going to be detecting with the
software we are going to build in this tutorial but today I'm going to show you a very generic
process and a very generic pipeline so by making some adjustments into the code we are going to be
making today you will be able to apply the same process to other types of license plates right we
are going to work with this type in this project but you will be able to make some adjustments in
everything we're going to be doing today so you will be able to apply the same process to other
types of license plates right so that's something I'm going to show you better in a few minutes
but for now let's continue now let me show you something else, when we were starting this tutorial
I showed you that we were going to use an object detector based on yolo V8 to detect license plates
now let me show you the data I used in order to train this license plate detector right this is
exactly the data set I used in order to train this detector, and I'm going to give you a link to this
dataset in the description of this video, and if you want to know exactly how I trained this object
detector I invite you to take a look at one of my previous videos where I show you how I train an
object detector using yolo V8, in that video is the step-by-step guide of how to train an object
detector using yolo V8 and that's exactly the same process I followed when I was creating this
license plate detector so this is the data I used and if you want to know exactly how I trained that
object detector then just take a look at the video I'm going to be posting over there right so now let's
continue I already showed you all the resources we were going to use in this tutorial I already
showed you the type of license plate we are going to be detecting today and now it's time to go to
pycharm so we can start implementing all the code of today's tutorial, and now let's go to pycharm
let's go to this pycharm project and let me show you some files I have over here, you can see I have
many many different files and for now let's just focus on these two: main.py and util.py. main.py
is the file in which we are going to be coding the entire pipeline of this tutorial right you
can see that this is a sequence of steps which we are going to follow in order to build this
automatic license plate recognition software you can see that the first step is loading
the models then loading the video then we're going to read frames and so on this is the entire
pipeline the entire process we are going to be building today and then we have this other file
which is util.py, in this utils file we have five functions let me show you these are the
functions we have defined over here and from all of these functions we are going to focus
on these two which are read license plate and get car, these functions... if I open these functions
you can see that they are completely empty right we need to implement these functions in this
video and then the other three functions they are already implemented right everything is ready
and we're just going to use them and the idea is to focus on these two functions over here because
these two functions are way more important from a computer vision point of view right so these are
the functions we are going to focus the most and this is the util.py file now if I go back to main.py
now it's time we start with this process now it's time we start with this Pipeline and in order to
do so we are going to start importing YOLO so I'm going to say from ultralytics import YOLO and then
we are going to load the models that's the first step in this process and the interesting part is
that we are going to have two models because we are going to be detecting license plates but we
are also going to be detecting cars that's going to be a very important part in this process so
I'm going to be loading two models I'm going to call the first one of these two models coco model
because this is a model which was trained on the coco dataset and this is going to be YOLO and
we're only going to use a pre-trained model from YOLO V8 which is Yolo V8 nano.pt right
we are just going to call this pre-trained model and this is the model we're going to use in order to detect
cars it's very important we detect cars I know we are going to detect the license plates and we are
going to read license plates but detecting cars is going to be super super super important and
you're going to see exactly why in a few minutes then we're also going to load the license plate
detector and we're going to call it license plate detector and this is going to be YOLO
and we need to input the path to this license plate detector and the license plate
detector is located in a directory which is called models and is called license plate
detector.pt so I'm just going to... models... okay now it's time to load the video we are
going to use today and in order to do so I'm going to import CV2 and I'm going to call
CV2 video capture and I'm going to input the video location which is something like the
current directory and it's called sample.mp4 okay and this is going to be cap okay now
we are going to read frames from the video so I'm going to define a variable which is
ret I'm going to initialize it as true and then while ret I'm going to read frames from the
video like this ret frame equal to cap dot read if ret then I am going to continue okay and this is
going to be pretty much all for now so we are reading frames from the video and now it's time to
continue detecting all the vehicles right we are going to be detecting all the cars and therefore
we are going to be detecting the vehicles and in order to do so this is where we are going to use
the first model which is the model trained on the coco dataset so we are going to do something like
this I'm going to call coco model and I'm going to input the frame and this is going to be results
right I'm going to call this object 'detections' and in order to move one
step at the time... I need to access the first element... in order to move one step at the time I'm going to
print detections and I'm only going to execute the first 10 frames otherwise it's going to be
very... this is going to take a lot of time so and frame number lesser than 10 and obviously I
need to Define a variable which is frame number I'm going to initialize it in -1 and
then I'm just going to increment it here okay and I don't really need the pass anymore
and let's see what happens if I print detections okay so everything seems to be working just fine
this is all we got and you can see that this is a lot of information these are all of our detections
so everything seems to be working just fine so what I'm going to do now is we are going
to iterate for detection in detections and this is going to be for detections.boxes
dot data dot to list and let's print detection again so we know exactly how this looks like
and we know how to access all the information okay so this is how each one of our detections
looks like right you can see that we have one two three four five six numbers and the
way this works this is going to be something like X1 Y1 X2 Y2 then we will have the score
and then we will have the class ID right this is detection so remember we are using a
model which was trained on the coco dataset so we are detecting many many different
objects right this is the class ID this is exactly the type of object we are detecting at
every single time at every single one of these detections so this is very important and then
we have the confidence value right this is how confident our object detector is of this specific
detection and then this is the bounding box right so we have X1 Y1 X2 Y2 the bounding box then
the confidence score and then the class ID and something that's very very important we are
doing all of this in order to detect Vehicles so as the coco dataset... as this model which was
trained on the coco dataset is detecting many many different objects we are going to say
something like this if int class ID in vehicles then we are going to continue and vehicles is a
variable which we haven't defined and we are going to Define it with the indexes with the class IDs
of all the vehicles in the coco dataset this is a list of all the objects which we can detect
using this model right you can see that these are a lot of objects and some of these objects
are related to vehicles and some other objects are not for example you can see we have person
bicycle car motorbike airplane bus train truck and so on right so from all this very very
long and very comprehensive list we are going to make sure we are detecting a vehicle so we are
going to say if the class ID we are detecting is either a car or a motorbike or a bus or a truck
then we are going to continue and if not we are going to neglect the bounding box, the detection
we just got, and the indexes we are interested in are 0, 1, 2 for car so we are just going to put
two then three for motorbike four five for bus and then six seven for truck right we don't
really have any motorbike in this video I know for sure because I already watched the video but
nevertheless in order to make this more generic I'm just going to add a motorbike as well so
if our class ID is within our vehicles then we are going to continue and I'm going to
create another variable which is detections_ and this is where I'm going to save
all the bonding boxes of all the vehicles we are going to detect in this video so I'm going
to do something like this if we have detected a vehicle then I'm going to append the bounding
box and the confidence score to this new variable and please mind that I'm not saving the class ID
from now on it's not really that important, we are not really going to care about the specific class
ID we have detected from now on the only thing we care about about our detections is that they are
Vehicles right and we don't really care to know exactly what type of vehicles so this is the new
variable in which I'm going to be working from now on in this tutorial in this process and now let's
continue and now it's the time in which we are going to implement the object tracking remember
we were going to work with object tracking in this tutorial and now it's the time where we are
going to implement this tracking functionality into this project and before we do so let me give
you a very very quick explanation Regarding why exactly we are using this tracking why exactly
we are going to implement this object tracking and basically every time we solve a problem every
time we solve not only a computer vision problem but any type of problem you need to use absolutely
all the information you have available regarding that problem and in this case we are going to be
tracking license plates which are moving through a video right we are going to be detecting license
plates on individual frames and these license plates are objects which are moving through a
video so if we are able to track this license plate through all this video we will have more
information and this additional information is going to be super valuable in order to build a
more robust solution so that's pretty much the reason why we are going to implement this object
tracking and we are going to be tracking... we're not going to be tracking the license plates
themselves but we are going to be tracking the cars, the vehicles, and I'm going to show you exactly
why later on so this is what we are going to do we are going to work with this repository remember
I showed you this repository when we were starting with this tutorial and the first thing you should
do is cloning this repository into your local drive into your local directory you need to clone
this repository into the directory into the root directory of your pycharm project so in my case
this is the root directory of my pycharm project this is where I have all my Python scripts and
this is where I have all my files related to this project and you can clone this repository in one
of these two ways let me show you one of the ways is opening a terminal and typing something like
git clone and the repository URL so I'm going to click here I'm going to copy the repository URL
and then I'm going to paste the repository URL here and then the only thing you will need to do
is to press enter right and that's exactly how you can clone this repository into your local computer
but there is another way in which you can do it and actually this is a much more simple way and
maybe you prefer to do it like this which is just downloading the entire repository as a zip file
and once you have downloaded this file this ZIP file the only thing you need to do is to copy and
paste is to take this directory this sort Master directory into your local directory right that's
the only thing you need to do is to drag and drop this directory into your local computer and that's
it and please mind that this directory is called sort-master but you will need to edit the name
you will need to rename this directory into sort right you can see here in my computer
this is my directory this is called sort if I open this directory you can see these are all
the files which are in this repository so basically remember to rename this directory into sort it's
going to be called sort-master but you need to rename this directory into sort that's very very
very important otherwise you will have some issues possibly you will have some issues with the next
steps in this tutorial so let's go back to pycharm this is the repository you need to clone
into your local directory and remember to call the directory containing this repository remember
to call this directory sort now let's take it back to pycharm and what I'm going to do now is just
importing sort... let's call from sort dot sort I'm going to import everything we are going
to import absolutely everything from this library and then I'm going to call an object I'm
going to create a new object which is called mot_tracker and this is going to be equal to sort
right this is the object tracker we are going to use in order to track all the vehicles
in our video and now let's get back here and what I'm going to do now is just calling
mot_tracker.update and I'm going to input a numpy array... of this list we have created containing all the
vehicles in our video right and this is going to be something like track IDs right so track IDs
is going to contain all the bounding boxes of all the vehicles we have detected in this Frame but
with the tracking information right it's going to add an additional column an additional field
which is going to be the car ID the vehicle ID for each one of the cars we are going to detect
and this vehicle ID or this car ID is going to represent that specific car through all the
video right so let's continue so now we are tracking all of our objects all of our cars and
now it's the time to detect the license plates right so far the only thing we have detected is
the cars in the video but now it's the time to detect the license plates in order to do so we are
going to use this detector over here which is license plate detector and we're going to do it
exactly the same way as we have detected the cars right I'm just going to copy and paste
this sentence and I'm going to replace coco_model by license plate detector right and this
way we are going to be detecting all the license plates I'm going to call this object license
plates and then I'm going to iterate in all the license plates we detected within this Frame and
in order to do so I'm going to call for license plate in license plates dot boxes dot data dot
to list and that's pretty much all and then let's unwrap all the information we got from this
license plate exactly as we did before so this is going to be something like X1 Y1 X2 Y2 score
and class ID this is going to be license plate okay then we will need to assign each license
plate to a given car right because we have detected all the cars in every frame and all
the license plates in every frame but so far we have cars and we have license plates and we
don't really know which license plates belong to which car right and we know for sure that every
single license plate will be on one of our cars but we don't really know which one goes with
which one right so now in this step is where we are going to assign a car to absolutely every
single one of our license plates right and in order to do so we are going to use one of the
functions in our util.py file we are going to use this function which is get car this function
receives a license plate and receives this object we have over here receives this object
with all the tracking information for all the cars in that specific frame and it returns a
Tuple containing the vehicle coordinates and its ID right so we are going to call this function
get car and this function is going to return the car this license plate belongs to right this
is what we're going to do I'm going to import from util import get_car
and now I'm going to call get_car I'm going to input the license plate and
I'm going to input this object which is track IDs remember this object contains all
the bounding boxes and also all the tracking related information right that's very important
and the return will be the coordinates of the car this license plate belongs to so it's
going to be something like X car 1 Y car 1 X Car 2 Y Car 2 and then the car ID
for this car right remember every single car in our video will have an ID
it will have a very unique ID which is going to identify the car through all the
frames in the video that's very important and also please mind that this function is
completely empty for now right this function is only returning some very dummy values and
this function is completely and 100% empty and this is exactly what we will need to implement
in the next step in this project right once we are completely ready once we have completed this
pipeline then at the end of this pipeline at the end of this process then we are going back here
to util.py, to this file to the util.py file, and we're going to implement this function right so
now we have assigned the license plate to a very specific car now we know what's the car this license
plate belongs to and now we can continue with the next step which is cropping the license plate
and this is how we're going to do we are going to call frame and then we're going to input
the license plate coordinates which is int Y1 int Y2 and then int X1 and int X2
right so this is the license plate crop and that's pretty much all we need to do in this
step of this process and now let's continue to the next step which is processing this license
plate right now we are going to apply some image processing filters to this crop we have over
here in order to further process this image so we improve this image so it's much simpler
for the OCR technology for easyocr to read the content from the license plate now it's time to
apply some image processing filters to this crop and specifically the filters we are going
to apply are a grayscale conversion and then we are going to apply a threshold so let's see
how we can do that I'm going to call CV2 dot cvt color I'm going to input the license plate
crop and then I'm going to call CV2 color bgr 2 gray and this is going to be license plate
Gray license plate crop Gray right now we have converted the license plate crop into a grayscale
image and now the only thing we need to do is to call CV2 threshold we are going to input this
grayscale image then is the threshold which I'm going to set in 64 and then it's the value
at which we are going to take all the pixels which are lower than the given threshold right
which is 255 and then I say the value at which we are going to take all the pixels which are
lowered than the threshold because we are going to use the inverse threshold we are going
to use the thresh binary... thresh binary inverse type of threshold and this type of threshold is
going to take all the pixels which are lower than 64 and is going to take them to 255 and all the pixels
which are higher than 64 is going to take them to zero right that's exactly how this threshold works
and if you want more details on how this function works I invite you take a look at one of my
previous videos where I show you an entire course of opencv with python and one of the lessons in
this course is exactly about thresholding right it's exactly about this function so I'm going
to be posting a link to this course somewhere in this video so you are welcome to take a look
at this course and this lesson particularly to get more details on how thresholding works now
let's continue this is going to be equal to a variable which we are not going to use in the
tutorial so it doesn't really matter and then I'm going to call the output license plate crop
threshold right so this is going to be the thresholded image and its exactly the image we are
going to input into our OCR technology into our easyocr algorithm, in order to be more more
clear about the difference between these two images I am going to visualize these images super
super quickly so you see exactly how they look like I'm going to call imshow and I'm going
to input this image which is license plate crop I'm going to call this window crop I'm
going to call it original Crop so it's more clear this is the image we are cropping
from the frame and then I'm going to call cv2 imshow again and in this case I'm
going to be plotting the threshold and I'm going to input this other variable and then the only
thing I'm going to do is to call CV2 wait key and let's take a look at these two images super
super quickly so you see exactly how they look like and this is what we got and you can see
that this is the frame this is the crop we are making from the frame so this is the license
plate and this is exactly how we are cropping this license plate from the frame and this is
the thresholded image right you can see that in this image absolutely every single Pixel is
either white or black and this type of image this thresholded image will make it much much
simpler to easyoce to our OCR technology to read the content from this image right this is the
image we are going to use in order to read the license plate number because this is going to
make it much much simpler to easyocr so it's going to be much simpler to our OCR to read the
license plate so that was like a very very quick way to show you how these two images look like
and now let's continue now it's the time to read the license plate number we are almost there we
have almost completed the this process and this is how we're going to do now we're going to call
another function which is defined in util.py and this function is read license plate and you
can see that this function is not implemented either this function is completely empty we are
returning some dummy values and this is another function which we are going to implement later
on we are going to implement after we are happy with this process once we are completely and
absolutely happy with this pipeline then we are going to move to util.py and we are going
to implement this function as well. But for now we are just going to... we're just going to use
this function so I'm going to import it as well uh no this is not the function name... read license
plate... something like this and now let's see how we can use this function I'm going to call
util Dot read license plate and this is going to return two values let's look at the function
documentation to see exactly what are the values which are going to be returned here... we are going
to... it is going to return a tuple containing the formatted license plate text and its confidence
score so this is going to be something like license plate text and then license plate text
confidence score right these are the two values we are going to be getting from here and the
input should be the license plate crop in our case we are going to input the thresholded crop
right this thresholded version of our crop and that's pretty much all right remember we are just
completing the pipeline the most generic process then we are going to get back here in order to
implement this function and this other function right and now let's continue now the only thing we
need to do is to write the results we are almost there we have almost completed this process and
now obviously if we want to take these results and we want to visualize these results or if we want
to analyze these results whatever thing we want to do with these results we obviously need to write
these results to our local computer so this is how we are going to do in order to write these results
we are going to use another function which is also defined in this util.py file and it's called write csv
and this function is implemented this function is 100% and fully implemented you can see that
this is all the code we have for this function and everything is just ready and we can just use
this function as it is remember in this tutorial and in basically all my tutorials we always focus
on the computer vision part of the problems so writing this csv file is not really that important
from a computer vision point of view so that's why we are not really going to implement this function
live in this video but this is already implemented and we're just going to use it so let's see what
this function does and it says write the results to a CSV file and it receives two arguments which
are the results which is a dictionary containing the results and then it also receives a path to
the CSV file we are going to produce and this is going to be the path in which we are going to
write this CSV file right it's the path in which we are going to save the CSV file we are going to
produce so if we are going to input a dictionary then we need to produce a dictionary in order to
input into this function right we need to take all all of our information and we need to put all of
this information into a dictionary right that's very very important so that's what we are going
to do now because for now the only thing we have done is just Computing all the information but
we have not saved this information into any type of dictionary or anything like that so I'm going
to create a new variable which is called results on results is going to be a dictionary and
then this is where I'm going to save all the information and this is how we are going to
do the first key in this dictionary will be the frame number right we are going to save all the
information and we are going to start with the frame number we are going to have a different
key for absolutely every single frame in our video and then for absolutely every single frame
we are going to save all the information which is related to all the cars we are detecting
and most importantly to all the license plates right so then I'm going back to the end of this
pipeline here and I'm going to say something like... I'm going to make a very quick edit first which
is going back to this function and instead of returning two None I'm going to be returning two
zeros right because we are going to reserve this other output we are going to reserve the None, None
output for those times in which we are going to find an error or we are going to have any type
of issues reading the license plate and this is going to be much more clear later on once we are
implementing this function but for now just bear with me that it's much more convenient to return
some dummy values which are different than None so let's get back here and this is where we're
going to say if license plate text is not None we are going to save all the information about
this license plate in this dictionary we have just created so we are going to take this variable
over here which is results for that specific frame number and we're going to create a new entry with
all the information for the license plate we have detected right and this is how we're
going to do I'm just going to write it first and I'm going to explain it once it's done once
I'm completed and this is what I'm going to do I'm going to say the next key is the car ID right this is going to be results frame number car
ID and then for this car I'm going to create a new dictionary which is going to have two keys one
of them is car and the other one is license plate for car we are going to have another
dictionary which is the bounding box and that's it right and for the license plate
we are going to have another dictionary which is something like bbox... the bounding box
then also the text we have detected then the confidence value for the bounding box and
then the confidence value for the text right okay and that's pretty much all so I'm just
going to format this a little nicer and that's pretty much all now let's see what
exactly we need to input in each one of these fields okay so basically for the car bounding
box we are going to input these values over here which are the car bounding box right
these are the coordinates of the bounding box of this specific car and then for the license
plate bounding box we are going to input these values which are the coordinates for the bounding
box of this license plate and then for the text we are going to input this value which is license
plate text for bounding box score we are going to input this value which is the score in for
in which we have detected this license plate then for text score we are going to input this variable
which is license plate text score and by doing so we don't have any errors and everything is
okay so for every single frame for every single frame number we are going to be saving all the
information which is related to each one of our cars and all the information for each car will
be the information of that specific car where the car is located and then all the information
about the license plate which we have detected in that specific car right and for the license plate
we are going to save all the information we have right and we're going to save all this information
only in those cases in which we have detected the license plate and every time we have successfully
read the license plate number from this license plate so this object is not None we are going
to be saving all these information into this dictionary only in that case only when we have
detected the license plate and when we have read its license plate number right and please notice the
structure I have built for this information for this dictionary because remember every time we
detect a license plate it will not be floating around in space completely isolated no that will
never happen every time we detect a license plate it will be on a given car and this car will
be on a given frame right so this is exactly why this structure I have decided for this dictionary
and once we have created all this information the only thing we need to do is to call... I'm going to
import this function as well, I am going to input the name was something like write csv so let's
import write csv as well and something is going on because we are not really using this import
we have over here so if I scroll down I see I'm not really importing the function itself I think
there we should be okay okay so now let's go back here and I'm going to call this function which
is write csv and I need to input the dictionary so I'm going to input results and I'm also going
to input where I want this CSV file to be saved and I'm going to save all this information into
a CSV file called test.csv so what I'm going to do now is I'm going to execute this pipeline I'm
going to execute this process as it is and then we are going to take a look at this file and then we
are going to continue right then we are going to see if the file we are going to create it
makes sense right so I'm just going to press play okay the execution is now completed and now if I
go to my local directory to the directory of this pycharm project this is test csv so this is the
file we have just created and if I open this file you can see that this is all the information we
have saved and we have extracted from this video right remember we are processing only the first
10 frames we are still processing only the first 10 frames so this is the all the information we
have extracted so far and please remember we are just Computing some dummy values from some of...
from some of our functions so this is this is not really all the information this is all the
information we have compute so far but other than all of these zeros over here and these zeros over
here you can see everything looks pretty pretty well right, we are just producing an entire
CSV file with all the information we have computed from this video we are almost there and actually
we are there we are ready we have completed this pipeline we have completed this process we are
almost almost there the only thing we need to do now is going back to ulil.py because we need
to implement these two functions get car and read license plate and once these functions are
implemented then we are going to be producing a real file right we are going to be using a
file with the entire information here and here right we are going to be producing the real
license plate number and the real license plate score and also the car bounding box and the car
ID for absolutely every single license plate in absolutely every single frame in which we have a
detection right so we are almost there I am super excited and now let's continue to the util.py
file so we can Implement these functions and let's start with get car remember from the main.py
pipeline we were using this function which is get_car in order to assign which car each license
plate belongs to right we have many many cars and many many license plates and for each one of these
license plates we want to know what's the car this license plate belongs to so this is exactly where
we were using this function get car and now let's see exactly how we are going to implement the
function and in order to do so I'm going to show you a few pictures this is a random frame from our
video right you can see that this is a frame we have many many cars and this is only a frame
from the video once we have detected all the cars we are going to have a situation like this we are
going to have many many many detections because at every single frame we are going to have many
many many many cars I don't know how many cars we have in this picture but they are many they are
something like I don't know 20 30 50 maybe 60 cars they are many many cars so for every single frame
we are going to have many many detections which are going to be our cars, we are going to have
many bounding boxes for all of our cars and also at every single frame we are going to have all
of our license plates but please focus please mind that we are only going to have maybe one
or two or three license plates for every single frame right so we are going to have many cars but
only a few license plates and the idea is to know which car this license plate belongs to and the
way we are going to know that is by looking at all of these bounding Boxes by looking at all of
these cars and by finding the car which contains the license plate right by finding the bounding
box of the car which contains the bounding box of this license plate right that's the way we
are going to find what's the car which belongs to this license plate so that's exactly the idea
of what we are going to be implementing in this function now let's see exactly how we can do that
the first thing I'm going to do is unwrap all the information in license plate so in order to do
so I'm going to do something like this because this is exactly the same object license plate
so I'm just going to do this okay then I'm going to iterate in all the cars we have over here I'm
going to say for... let's say for j in a len vehicle track IDs we are going to be iterating in all the
cars we have detected and remember this is the entire information this is the bounding box and
this is also the car ID remember so now we are going to unwrap all the information for each one
of these cars and this is going to be something like x car 1 y car 1 x car 2 y car 2 and car
ID this is exactly the information which is in each one of the elements of this object vehicle
truak Ids and this is vehicle track ID j okay so that's pretty much all we are iterating in
absolutely all the bounding boxes of all the cars we found in this Frame we are iterating
in all these bounding boxes for each one of these bounding boxes we are going to verify
if it contains the license plate right that's exactly what we are going to verify and this is
how we're going to do it we are going to see if X1 is greater... remember X1 is the upper
left coordinate of the license plate if X1 is greater than x car 1 and Y1 is greater than
y car 1 right we are verifying that this coordinate over here it's greater than this
other coordinate over here we are trying to verify if we meet this condition right and then the
other condition we need to meet is if this point we have over here these coordinates
we have over here they are lesser than this other point we have over here right we need to
meet these two conditions and this is exactly how we're going to do it if X1 greater than x
car 1 and Y1 greater than y car 1 and X2 lesser than x car 2 and Y2 lesser than y car 2 then we are we are going to... we
have found the bounding box this license plate belongs to we have found the car on which
this license plate is located right that's what it means if we have met all of these
conditions that's what it means so in this situation we are going to... I'm going to
define a new variable which is foundIt and foundIt is going to be false at the beginning
and then it's going to be true in this case and in this case we're also
going to break the loop right and then I'm also going to Define
another variable which is going to be car_index and car index will be j
okay now if foundIt then return this value which is going to be... okay so if we have found the car which contains
this license plate then we are going to return these values which are the bounding box of the
car and also the car ID and in any other case we are just going to return this output in order to
make it more clear that we have not found the car we are going to return something like this so
it's going to be much more clear so that's pretty much all... that's it, we have implemented this
function which is get car and now let's continue so now let's let's see if everything works well
now we should have the uh the right values for all the cars we are detecting and the only thing I'm
going to do is I'm going to execute this script again and let's see what happens okay I got an
error and I think I know what's the problem I think we need to iterate in range len vehicle track IDs
and now everything should be okay let's try again okay now it's completed and now let's see the new
file we have created the new test.csv file and now you can see that we have some values for car ID
and we also have some values for the car bounding box so we are moving one step at the time but we
are making progress right so now let's continue with the util.py file and now let's move to the
next function which is read license plate now it's time to implement this function over here and
something I'm going to do first is I'm going to do an if over here and I'm going to continue with
this pipeline only if car ID is different than -1 right and now let's continue and let's see
how we can implement this function which is read license plate and the only thing we need to do is
to call easyocr and let's see how we can read the license plate and let me show you some variables
I have defined over here these variables are going to be super super important now this variable are
going to be super amazingly important you're going to see exactly why and then also let me show you
this reader we have here I have already defined I have already initialized this OCR reader and
you can see that I'm calling easyocr and then I'm calling this method which is reader so the
only thing we need to do now is calling reader dot read text and I'm going to input the license
plate crop and this is going to be detections then I'm going to iterate for detection
in detections because remember we could be detecting many many many many different objects
many different text objects in this image so for each one of these objects we are going to
unwrap these objects first and this is going to be something like bounding box text and score
this is going to be the detection right each one of these detections is going to be something
like the bounding box of the text we have detected then the text we have detected and then
the confidence value for which we have detected this text and then we are going to convert this text to
uppercase and we are going to remove all the white spaces right this is exactly how we are going to
do and this will be equal to text okay and now it's the time in which we are going to use this
format right remember when we were starting this tutorial I told you we were going to focus on
this very specific type of license plate right we are going to work with this type of license
plates each license plate is going to have seven characters the first two characters are going to
be letters then two numbers and then three letters this is the format of absolutely every single
license plate we are going to be working with in this tutorial so we are going to make sure every
single text we detect complies with this format and in order to do so I have already created a
function which is license complies format this function returns a Boolean value which is pretty
much the verification of if this license plate complies with the format or not we are going
to be verifying if we have seven characters and we're also going to be verifying the first two
characters are letters and then the second... the third and the fourth characters are numbers and
then the last three characters are letters again right this is exactly what we are doing with this
function and this is a very important function we are going to use now so let me show you
exactly how we are going to use this function if license complies format text then and
only then we are going to return the text and the confidence score we are going to return these two
values, these two variables, which are text and score right only if the text complies with the
format we are asking absolutely all the license plates right only in this case we are going to
be returning these values and in any other case we are going to return None right this is very
very very important and this is going to make our solution way more robust and way way better
and something that makes the solution even better is that we are not going to return the text on
itself we are going to call another function which is format license and let me show you exactly
what we are going to be doing with this function I'm going to call format license text and let me
show you the... let me give you the idea, the high level idea behind this function sometimes
when we are using an OCR technology when we are using a library like easyocr sometimes it's very
challenging to tell some characters apart for example it's very challenging to tell a five apart
from an S right so you can see that the letter S and the number five are very similar and it's
very very very challenging for an OCR to tell the difference between these two characters and
we are going to have exactly the same situation for the letter I and the number 1 or for the
letter O and the number 0 for example right those are characters which are very very hard
to differentiate, they are very hard to tell apart so this function I have over here
format license the only thing it does is going through all the characters in the license plate
in the text and for each one of these characters it fixes whatever issue we may have with
this type of confusion right if for example we are reading this character over here and easyocr,
the OCR technology we are using, it says is the letter S we know for sure it's not the letter
S because we are expecting a number here so if we have detected the letter S then we convert this
value to the number 5 and the same happens here if we are reading this value this character and
we are getting the number 5 we know for sure for a fact that that's not the number 5 because
we are expecting a letter here so we are going to convert the number 5 into the letter S that's
exactly the idea the high level idea of what we are going to be doing with this function
we are going to be going through absolutely all the characters in the license plate and for each
one of these characters we are going to be fixing these type of issues in case we find any type of
issues like this and that's pretty much all and I invite you to take a look at these two functions
to format license and to license complex format and to take a much closer look and to properly
understand exactly how they work right that's your homework that's your homework from this
video so you properly understand how they work so now let's continue and now we are returning
format license text and score if our license complies with our format and we are returning
none in any other case and we are done we are completed now we have completed our process now
let's see what happens now I'm going to execute this file again and let's see what happens I'm
going to make a very very small change I'm only going to execute it for 10 frames but I'm going
to do it like this if ret then if frame number um greater than 10 then I'm going to break the
loop this is going to be much better and now let's see what happens I'm going to execute main again
okay it seems I have a typo over here this is obviously not remove but this is replace I got
confused because I was removing the white spaces but this is obviously not the name of the function
we want to use here so now let's see what happens okay now the execution has been completed and
now we have produced a new test.csv file and if I open this file you can see that we still
have all the information related to the car ID and the car bounding box and now we have all the
license plate numbers we have read from the frames from the license plates and also the confidence
score for each one of these license plates so we made it we have completed this process now
we are completed we are done so everything is ready the only thing I'm going to do now
is to execute this script execute this main pipeline for the entire video so I'm just going
to remove this break over here and that's pretty much all and now I'm going to press play again
and then I'm going to show you how to visualize this data so everything looks like the video I
showed you in the intro so let's see what happens and now let's go back to pycharm so I can show
you exactly how you can create a visualization as the one I showed you when we were starting
this video in order to do so this is where we are going to use these two files visualize.py
and add missing data.py and you're going to find these two files in the GitHub repository
of today's tutorial so you can just go ahead and use them in your project and before using these
two files let me show you something first if I go back here to the test.csv file we have created
let me do something I'm going to filter by car ID I'm going to show you all the data all the
information we have extracted for only one of our cars I'm going to select only the car ID
number three right this is only a random car ID in our data you can see that all the frame numbers we
have detected for this car ID are not consecutive so this means that we have detected the number
zero... the frame number zero then the number one then it jumps to the number four then it jumps to
the number nine then 12 13 14 15 16 17 then 27 so we have many many missing frames right for some
reason we don't have the information for this car ID for many frames which are in between these... these
two for example right we don't have the information for the frame number two the frame number three
or the frame number five six seven eight uh 10 11 right there are many many missing
frames for this car ID so that's something that's going on and remember that we are not saving all
the information because we are only saving the information for those license plates for which
we have detected the car the license plate it is on right? the license plate... the car
where the license plate is located and also we're only saving the information the license plates
for which we have read a license plate... a license plate number which complies with our format right
so we are not saving all the information, there's a lot of information which we are not
saving into this CSV file remember how the OCR Technologies usually work I mean they are very
very good they perform very good but in some cases they have errors they have mistakes so if
in some cases they are not reading a number which complies with this format then we are not going
to be saving the information for those frames so that's the reason why we have some missing
frames over here that's the first thing I want you to notice then another thing which is going to
be much more important is take a look what happens with the license plate numbers now we have read
the license plate numbers in all of these frames and we have read a number which complies with
our format so everything it's okay but you can see that we have many numbers right for example
we have many many different values many different numbers if I show you the number we have detected
in the first frame it's different than the one we have detected here in the frame number four right
and then if I continue scrolling down you can see that we have also detected other values for
example here this is different and if I continue scrolling down this is also different here we have
an N we have a P so for every single car ID we are going to have many many different values for
the license plate and this is a huge issue this is this is a very very important thing we need to
solve because obviously every single car has only one valid value for its license plate
so if we have so many values if now we have so many values for the license plate how do we
make a decision how do we know what's the the real one right what's the real value the
most accurate value for the license plate how do we make a decision what's our criteria that's a
huge problem and this is exactly where the object tracking is involved because for every single car
in the video... because we are going to be tracking the car through all the different frames in the
video, for every single car we are going to have the value for the license plate we have detected
in that given frame for that car so if we want to know what's the value for the license plate of a
given car through all the frames in the video the only thing we need to do is to select the license
plate we have read we have detected with the highest confidence score right you can see this
column is the confidence score in which we have detected every single one of these license plates
so the only thing we need to do is to take a look what's the license plate we have detected with
the highest confidence and that's it, that's going to be your criteria to know what's
the license plate number of this car and that's it that's the way we are going to solve our problem
and that's exactly where the object tracking is involved and that's exactly why it's so important
to track... to apply to implement an object tracking algorithm into this problem because this
is how we are going to solve this problem this is going to be our criteria to select the license
plate number for every single car in this video so remember we have these two problems this
is how we are going to solve this problem and then we have we still have this
other problem which is that we have some missing frames for every single car right this
problem actually is not... it's not really a big problem and the only thing is going to affect is
the visualization right because now we are going to take all this information and we are going to
visualize this information so the only thing is going to happen with all these missing frames is
that we are just not going to visualize the license plate and we are not
going to visualize the license plate value for that given frame so let me show you what happens
if we visualize if we create a video from the text file... the CSV file I just showed you
we will have a visualization which looks like this which will be okay I guess but it's like
um but... it's not an ideal visualization right it's like uh it's it's not really pretty it's not
really good looking this doesn't really look good ideally we would like to have a visualization
which is more stable for every single license plate we would like to see the license plate on
a fixed position through all the different frames in which we are detecting the license plate for
that car right that's exactly what we eould expect and this is not really good looking this doesn't
really look good right so in order to fix this problem which again is not a huge problem
and the only thing it does is to affect the visualization we are going to use one of these
two scripts which is called add missing data and the only thing this script does is interpolate
all of those frames in which we have not detected a license plate or in which we are not extracting
the information for the license plate so the only thing we're going to do is interpolate the
values for the bounding boxes for the car and the license plate in all of those frames, we are
going to interpolate the values and that's it for example in the frame number 41 you can see we have
the information for the frame number 40 and we have the information in the frame number 42 but we
don't have the information in the frame number 41 so the only thing the add missing data.py script
does is going to consider the bounding boxes for this Frame and the founding boxes for
this Frame and it's going to take the average of all the different coordinates and by taking the
average it's going to compute what it's the value of the found inbox in the missing frame right and
it's going to compute exactly the same process in absolutely all the other missing frames so that's
the way we are going to solve this problem all the missing frames remember this is only a problem
of visualization this is a matter of visualization is not a huge problem and then once we have fixed
that issue then we can just create the video and that's it so these two files I'm going to give you
these two files in the GitHub repository of this tutorial and now let me show you how this works
so the first thing you need to do is to execute add missing data and you need to change here the
path to the file name you are going to interpolate right in our case its test.csv and then you need to
specify what's the file name of the CSV you are going to create with the interpolated data let me
show you super quickly how this file looks like I'm going to filter by car ID and I'm going to
select the number three again and you can see that in this case we have computed absolutely every
single frame right we are starting the number zero just as before but now we have computed absolutely
every single... the values for the bounding boxes for absolutely every single frame until the number 65
which is the last frame in which we have detected this car right so this exactly the data
we are creating with add missing data.py and once we have created this data this new CSV
file then we go to visualize.py and then we input something like test interpolated.csv and then we
specify what's the file name of the video we are going to create in this case out dot mp4 and the
only thing we need to do is to execute this file and then to execute this file and then after a
few minutes we are going to have a video which looks exactly like this and this is going to be
all for today my name is Felipe I'm a computer vision engineer and these are exactly the type
of videos and the type of tutorials I make in this channel if you enjoyed this video remember
to click the like button and also remember to subscribe to my channel this is going to be all
for today and see you on my next video [Music]
