Hello world. It's Siraj and today. I'm going to show you how to build your own self-driving car in a simulated environment And it's actually much simpler than what we've been doing like so recall we've been talking about generative adversarial networks and variational [Autoencoders] and all sorts of stochastic models this comparatively is actually surprisingly simple Which is crazy if you think about [it] a self-driving car is simple. No, but it actually is it's nine lines of Keras. I mean the model itself is and then the helper code But we're going to talk about how you can build this and I'm going to show you step by step the entire process [ok] so Here's an example of what it looks like When it's when it's fully trained, and it's running in the simulated environment, so I'm super [excited] I hope you are and at the end. I'm going to answer some questions Just straight up from YouTube comments, so I'm excited for this let's get started so the first thing we want to do I'm going to stop this demo this is what's eventually going to look like and now the car is going to crash into the water because I turned off the Autonomous feature uh-oh, let me just turn off the simulator, ok, simulator is gone. ok, so, Let's first go ahead and talk about What this is and what the whole deal is behind all this a little bit of theory, and then we'll get back into the code ok so Where are we where are we there we are [ok] so how to simulate a self-driving car is the name of this tutorial? And so udacity recently open source their simulated so they built a simulator specifically for self-driving cars so cool, right? It's just [for] self-driving cars And it was it was built for their self-driving car an inner degree students and they recently open sourced it So it was built with unity which is if you haven't used unity before I would definitely recommend at least checking it out If I had more time, I would definitely be diving into unity and building all sorts of cool 3d models that I could play [with] and you [know] maybe even generating 3D models with generative models But yeah, it's a great tool It's very much using in virtual reality and more and more in machine learning [or] for modeling and things like that but everyone check it out, but yeah, they built this thing in unity and They added a bunch of pre-built scripts. They added a bunch of scripts and these scripts Control things like [appetit] and momentum and acceleration and all sorts of things that you would think [would] be in a self-driving car simulation And you can modify these things as well as they're just values that [you've] just change in code [but] you can find the actual simulator itself right here on github And I'll also have [a] link [to] it in the description and in the github readme, but it's a binary file You don't have to compile it from source. That is like an exe. You know you just download it It's like a dot app file And you can just download it for Linux Mac and windows and then boom you're good to go, and it's got two modes it's got a training mode in an autonomous mode, and I'll talk about what each of those modes is but The [high] level is you just download it and it just works no dependencies to install or anything, so it's super super simple [ok] so That's what they built, and so let's talk about the three step process and how this works [ok] there's a data generation part Then there's a training part, and then there's a testing part so the first part is [the] data generation part, and so what happens is we are going to first look at what NVIdia did so To generate the data. What NVidia did was they took their bit to the car? And they did the same exact thing that we're going to do we're going to replicate what NVIdia and udacity did it's the same thing They built a nine layer convolutional network and they attach three cameras to the head of a car [ok] so they attached three cameras to the head of the car and they Had a human driver drive the car while the cameras were attached and all the car has to do was collect Data and [buy] Data I mean the steering up Let me turn off [the] sound here the sirian commands that the human driver was inputting as well as the camera feed so the three Sets of camera images coming from from all three cameras okay at the same time and that was the data generation part okay, and so there were there were images from the center left and right cameras with the associated steering angle speed Throttle and brake so there were four of Physics variables as well as the static images that were coming in and it saved it also a CSV Which is what we're going to do and ideally ideally you have a joystick for this thing but I don't have a joystick ain't nobody got time for that and just because it's like it's Easier it's Smoother with the transitions and stuff using a keyboard the wasd keys is kind of erratic But whatever. I mean it works. It works what you saw was based off of a keyboard training, so whatever it works [so] yeah, so that's for the data generation part, and then there's the training part okay, so What the idea is that we have a human driver, okay? So the human driver is just driving around and we're recording all that data and then we're going to get a machine to clone that Behavior so we call this process Behavioral cloning [ok] behavioral cloning and to do this. We're going to build a nine layer convolutional Network nine layers in Chaos that's just literally nine lines of code and it would based off of NVIdia's and to end learning for self-driving car paper and Yeah, so they trained on model for 72 hours on the whole in a whole bunch of different driving conditions sleet rain Hail all all sorts of things wet rain. You know everything so here's the hardware design So the hardware design was it's obviously a steering wheel But the steering wheel is attached to a controller those that was the so many acronyms here because the controller area network bus and so it's feeding in those four variables that I talked about right and You also have your three cameras and the three cameras are all feeding in Continuous streams like just frame by frame of all the images that the car is seeing and it's fed into this drive Px Computer which is essentially like a motherboard with a cluster of Jeep on-board gPUs attached to it And then and then assaults [a] store it would just store the results. That's it okay So that's the that's the high-level idea behind the hardware behind it and then the software behind it oh and also interesting little tidbit I just recorded from the paper instead of Inputting the steering command are directly they inputted one over R and they found that this may turn smoother, and it [maybe] it made the Autonomous angle that the car moved Independent of the Geometry of the car so the car was facing left. It didn't matter or the car was facing right? it's all about what the cameras with the camera see and then using that as the signal to then move the steering angle so it's it's independent of the Directions the car is facing it sounds counterintuitive like wait you want that But you actually wouldn't and we'll learn [more] about why that is as we get into this code, okay? So the software design for this code And it's so you know it's crazy because going from generative adversarial networks to this is like. That's it That's all it takes for self trying to guard like what but that's that is actually how it is check this out So we here's here's what happens? We have we have two inputs right. We have our Steering angle, and then we have our sets of cameras right which is so three sets of [images] those are our inputs and so We can think of the problem as a supervised learning problem, right we have our input data Which is which are the images and then our output labels which are the if we could [just] abstract all of these? Steering angles into one value. You know like one binary value that would be our label like based on what you see How should you move the car? That's it, right? So we have a CSV just listed of all of those things right and at the same time we have What the car is trying to do so during the training process? Assuming we've generated Data 72 plus [a] 72 hours of human generated driving Data. We can then compare the Both of those results together so we can compute the error or loss between the two right? so if we have let's just say we have the camera Angles and the steering wheel from the autonomous car and then the camera Angles in the steering wheel variable from the human generated Data and then we just vectorize both of those values into one value And then we find a difference between those values and that difference is our error And then we use that propagation to them update our weights based on that error value, okay? So it's taking that that weight value or that back that error value and then back propagating it in our weights Okay, so images of images are fed into the Cnn, and then it computes a proposed during [commit], okay? And that is the predicted steering command And we want to find the difference between the predicted steering command and what the actual steering command would be from the training data, okay? And then the proposed command expects the desired command right label the Target versus the actual output and then the weights of the Cnn are adjusted to bring the Cnn output closer to the desired output and The weight adjustment is accomplished Via backpropagation So eventually you want it strained, then all you're going to need is the [center] camera one camera You feed it to Cnn, and it outputs a prediction when the prediction is the angle of the steering with just one output right? so That's it for training and then for testing right so once once we once we train this Autonomous model for testing [we] could think of it in terms of a simulator as a server client Architecture right the server is going to be the simulator itself that is our you know the app that we downloaded that little 3D game And the client is going to be our python program But what we write and what it's going to do is it's going to just be a feedback loop It's going to be a feedback loop so we can consider it as just so this is how we can consider it It's just the client is piping in steering angles and throttles to the server and the server is typing back up images from the car and steering Angles so that It can train it right. It's just it's just it's just a feedback loop So that's that's the basic idea behind you know how this works high-level, NVIdia has a paper on it and then udacity you know was inspired by it and now I'm bringing it to you guys, so it's just the chain of a Chain of whatever okay, so let's build this thing right enough talk Let's build this thing so [the] [first] step is to install our dependencies And you can do this really easily with just one line of code. There's an environment dot of Yml file, and you can use Anaconda to do this okay? So if we look in the environments fall, we've got like 20 dependencies here But you don't have to you know as long as you run this one line of code It's going to install all of these dependencies and make this bigger all of these dependencies super easily okay, so Yeah, you got [openCV] which we all know is a pain to install so luckily for us Anaconda is going to make it super easy To install okay, so that's the first step now. [I] would do this myself, but I've already installed the dependency here but What I can do, right? Just paste that in [and] once you paste that in then you're going to type in I can show you the command you say source activate both the command I'll put it in the output in the github readme, but you then you activate [your] content by ernment And it's going to be called Behavioral cloning Source activate behavioral cloning and then that activates the environment So it's you install your dependencies in this container this anaconda container, and then you activate it so then all those dependencies are then active in that shell in that session in Terminal ok so once we have that then we can generate our data ok where we've got our dependencies Now step two is to generate our data. So this is a five step process. We're going to install dependencies Which we assume I've done step two is to generate the data. So we're going to drive ourselves but we're going to write to Replicate with it with [NVIdia] did with the human drivers step three is to write the training Script step four is to [train] it then have our agents learn or clone the behavior of us the human and step five is to write the testing script and then Let it run okay, so let's go ahead and generate our data so to generate our data We're going to have to actually drive this baby. Which is going to be a lot of fun now? You just double-click on the app. You know I downloaded it You can double click on it, and then you notice it's got a couple of settings I said it's fantastic because why not of course the choice? But you know fastest you know depending on your processor you would want to change this so I'm going to go ahead and run it at 800 by 600 and it's going to [going] to pop up [just] like that and I'm going to go into training mode okay, so for [training] for training mode I'll just click on training mode and the controls are going to be Wasd okay, so just like this see this is me driving right now. [I] am driving the car Okay, and now I'm going to stop the car and now I'm driving it again. Okay, so this is human driving So what we're going to do is we're going to generate data so that we can train our our car on that data Okay, so to do this We just have to type in just type our right to record our and so then it's going to ask well Where do you want [to] save this data to and I'm going to say let's save it to the desktop, [okay]? let's save it to the desktop and Start recording and now what it's doing It is recording three sets of images from three virtual cameras of the car Okay, assume that the cameras are on top but what it's doing is [its] capturing frames from this game from three different angles and it's also recording the my The four variables, I talked about speed Throttle and the other two steering angle There's one more [to] [look] at but basically what you want to do as you train this thing is you want to complete either up? One between one and five laps, okay? Ideally five but and look I'm off track, but let me just stop recording and so then once you're done You know driving between one and five laps then hit R again? [and] it's going to capture that data so it's going to replay what you've just done And it's going to go up to a hundred percent and when it's done It's going to save it all to a CSV file [that] we're going to look at and see what's inside of there So we could observe it so we've got [fifty] [two] percent and in the meantime let me go to the CSV file and Show you guys that what else we're also going to show you all this captures Well [is] actually [fun] to just watch this let me see what else I can talk about the training and testing script Yeah, we're going to be using cars [95%] here we go done. Okay. Let's check it out. What if what did we save here? so if I go to open, and then I go to my desktop There's my driving log. Just just like we thought it would be okay So let's take a look at what this is let's take a look at what this is What we've got here are three images right and it's the [first] three Columns are images they're pointing to images and these are the images that it's pointing to right here image folder Okay, [their] point their absolute directory values it shows us exactly where they are so see these are the images [that] they're pointing to just like that and These are frames that were captured from the game and we've got three sets of images for the C center For the left and these are labeled with the direction and the right so we've got three sets of images and then we've got four values here, and these four values are those values that I talked about up here which are the which ones are they the steering angle to speed the throttle [and] the [brake] all four values and these are what I input it and See how at the start? I wasn't moving so these are these [are] zero values, but then as I start moving they're going to change okay, so That's it. That's it for our data So what you want to do is right train it between one run one full lap and five laps, okay? So somewhere in between there, so yeah That's that's it for our a data generation part where were we in our process We were at step two so assume that I've generated data. Okay. Just like that It would take a while, but I generated data so the next step is for us to write our training scripts So let's write this training script, okay, so where were we? For our training scripts. I'm going to say Okay, so we've got two files here. We've got a training script Which is modeled up [PI] and then we've got a testing script called drive Dot pi let's write our training script Right we've generated our data and [we] saved it to the desktop and now we want to build a model that will see a convolutional Network that will read from this data, and then it will output output steering command So we've got pandas [10] is going to be our data analysis 2 toolkits numpy is going to help us do Matrix Math We're going to import Scikit-learn just so we can split our training and testing data in two Sets in two sets and then we're going to use [kaos] as our machine learning model Sequential because it's going to be a linear stack of layers very simple model and then [add] [them] for gradient descent model checkpoints So we could save our model, and then we've got a layer types that we're going to use then our helper class utils It's going to define our input shape and generate training images and then our [Arc] parts for command line arguments and lastly our os Module for reading files so what I'm going to do is I'm going to write the code [for] building the model itself But all of all the rest of it is pretty written right so the data loading parts are [pre-written] We're going to load that CSV file that. We just created [write] that driving log and make it bigger Make it bigger the driving log We're going to we're going to generate that data or we're going to read from that data using pandas and then we're going to Say okay, so now we have [that] in a data frame variable very easily parsable variable Thank you, pandas then we're going to say for the center left and right Columns get those values and that [is] going to be our input data that's going to be our x right and then our Output Data is going to be our steering command so we can concatenate that into one variable, right? So then we have our input data and our input in our output labels And we want to find the mapping between the two this is a supervised learning Problem and once we find the mapping between the two then given novel input data Which is novel camera images of what a car sees We can then output the predicted label which will be [the] [herring] command [see] very intuitive right okay? So we've got our [input] data in our output labels And then we can split the data into training and testing data so it's 80% training 20% testing and we can define the size of that in our command line argument, and test size okay, which is going to be? Point two. Oh, this is going to be this value, so yeah so we're gonna have a training our training data and our testing data and our validation data, and we returned back okay, so and then we've got the code for building our model and then training the model and then The main function right so let's forget about these command line arguments So let's just go straight to the main function and let's say that for our main function [after] printing you know whatever our parameters are we're going to load our data using that funk that function that we just talked about then we're going to take that data and use it to Train our model, but first we'll build our model, and then once we built our model and loaded our data We'll train our model right using those two parameters as well as any command-line arguments, so this commit this helper function just [is] For converting a string to a boolean value for a command line argument but this training model function is really interesting let's first build [the] model, and then we'll talk [about] the the training model function okay, [so] for this first part We're going to build this model And it's going to be a sequential model right it's going to be a sequential model very simple Model And we can just pull this straight from the paper serve in a paper write the paper told us the parameters with which we could Build our network, so we'll use those parameters We'll use those parameters build our network, so the first step is for us to say okay We want our first layer to be an image normalization layer, okay? We can totally do that image normalization layer using a lambda function and the land of function is going to help us do so really easily and We'll say okay. [it's] going to be From E type of cell and then I'll talk about it input shape okay, so what's the deal here so x? Yeah, so this is going to Avoid Saturation and nicker gradients work better now notice that these are Magic numbers right here but we but the Authors sound that you know after training and testing out different values these are the ones that work that for Normalizing the images right when we input it to avoid Saturation and make the gradient work better What do what do we mean exactly by that something you know? the images can be Shadowy they can you know any on the lighting it can be certain things could be obstructed or The hue [of] it. You know all [of] these color correction values could be off and they could give us results That are bad, so we want to avoid that right? So that's that's when we when we say image normalization what we really mean is we want to format or reshape these image tensor values into values that would give us good predictions in the end right, so That's it for our first model, and then we've got a bunch of convolutional layers that we did essentially copy Copy and paste once we've written out the first one because they're very [similar] and so the first one is going to have a filter Size of 24 okay, I got you and instead. It's going to be a [5x5] convolution, okay? I can do that as well, and then it's going to have an activation function. Which is going to be e lu which means Exponential linear units so okay. We've got earu here. Okay, which is More suited for this task than rectified linear units And we're going to [subsample] with two by two because that's the length of our strides, and then we'll add our closing Parenthesis okay, so that's it And so why do we use [alou]? It's because it takes care of the vanishing gradient problem. That's why so okay So we've got five of these layers. Let me just paste that one two three four. How many of these five I think that's five that's well, okay, so then we want to make [sure] okay So what's the difference here the filter size here? Is [Thirty] [six] this now you can just read from a paper and just write out the model directly right? it's not that hard they have the parameters and with you know [a] simple library like [Kara's] you can you can build a Model pretty easily, so we've got 36. We've got 48 64 64 Okay, and so then we can remove the subsampling for this for these last two layers since the simplest right size is [one] by [one] respectively for each of these for each of them, so they'll just end in the activation and then We're going to add a drop out layer okay, so it's asking for a [50%] drop [out]. We can do that So we'll say okay. Drop out We'll add It based on what the user says it's going to be and we're going to it's going to we're going to input 50% later and then we're going to say okay, so now that it's dropped out. We're going to flatten the Data and so why we talk flatten it because we're going to start feeding in a series of fully connected layers And why why why a series of fully connected layers? Because the convolutional layers are meant to handle feature engineering [so] that means that the image Processing parts or the filters right so we feed in a set of images and what the convolutional layers will do is Each of them respectively is going to create a set [of] filters and these filters are going to be you know Increasingly abstract right so they're going to start off with low-level features, and they're going to get increasingly more abstract So they're going to be able to detect images, but what we want to output is not an image But we want to what it is that we want to output is a value and that value Is a single value and that is the steering command right? It's a direction How do you want to move this this wheel so to do that to get a single value from these high dimensional image tensors? We have to want we have to squash that data and the way to do that is by applying a series of fully connected layers Okay, and so it's going to happen is Each of these fully connected layers is going to progressively Get smaller and smaller in terms of the number of neurons they have and you'll see what I mean here So it says the first one should have a hundred neurons so we'll say okay So this is a dense which means fully connected layer and [it's] activation function like all the [other] layers is going to be blue and So then what we can do is we can just copy and paste this will say one two three So there's going to be four of them in total Each of them is going to be smaller so 100 neurons 50 neurons 10 neurons And then just a single neuron and this one won't have an activation function Because it's the last layer and that's going to output our driving Value our steering down and say okay So it's a summary so all of those some are all those values to get the summary and in return the model Okay, and so then We'll say How much we have here that there there? We go and then return the model, okay? Cool, so right so for our fully connected layers We start off with a big layer And then we want to get progressively smaller and so the way to do that is by using a model summary or but by using a series of fully connected layers that get smaller and smaller and One of what that's going to do is going to squash our data so it's going to basically be like a triangle in terms of the number [of] Values in that in that [and] those matrices that [are] being propagated forward in our network, so it's Gonna be like you know at N Matrices a matrix with ten indices and then like a Matrix C matrix with five indices and make sure it's with two indices eventually one single scalar value And that is our output and that's what this is going to do right and the line in the paper. They Noted this, but I didn't actually document this they noted that. It's not sure where the image part and the end the steering part begin and end because it's all kind of connected right in terms of What the what the like neural network looks like [but] that's part of the black box Magic of Neural networks. We don't know exactly where One feature starts and one feature ends in terms of where it lies in these in this abstraction Hierarchy But we know that all as a as a whole as a sum total there is some there There is some connectedness right so anyway, so that's it for our model, and then once we have this model We're going to train the model and so let me talk [about] this last training function, so for the training function we're going to say okay take the model and then the The training Data and the validation Data to model and Data and train it right so to do this we're going to run the model checkpoint function and what this does is it saves our model at a in a Check [point] that we bear that we That we say that we want and it's [gonna] be modeled h5 and then we're going to say well What [is] the loss function that we want to monitor? which we're going to call the validation loss not yet, but we're going to call it that and We only want to save the best model, okay? and then we're going to say the mode is going to be auto and Auto mode the Direction is automatically inferred from the name of [the] monitored quantity okay? So that's it for our checkpoint, and then we want to compile our model. So what are we going to do here? We're going to say okay. What is our last function going to be well? There's several log functions that we can use and we're going to do a really simple one called mean squared error what does that mean so our model is going to output a predicted a steering angle And then we have an actual steering angle from with a human driver given those camera Angles and then we want to find a difference between those angles and do that for all [of] the Data points that we have okay and then sum up the differences and then Hold on We want to find the difference Square the difference add up all the differences and then divide by [the] number of them And that's going to be the mean squared error the sum of the squared Errors okay, or the mean of the squared errors, right? [and] so then we're going to use the atom optimizer which is gradient descent Yeah, and so that's how we compile our model, and then we're going to generate some data. So what we're doing Here is we're running the fit generator, okay, and so What the fit generator does is it lets you do real-Time data? Augmentation on images on the cPU in parallel to training our model on the gPU What do we mean by that well? We are generating batches of Data okay? while we are generating those batches of data from from our training Data like the the Buckets or the containers from which we train our model in from the from the huge set at the same time [where] after? We're training our model right so what this fit generator function. Does is it does both simultaneously? [okay], [so] it's super [valuable] and then Cool okay, so then we can train this model Right which will run python models up high and it will start raining on our driving. They'll start training on our drive And we want to then Once it's training. What's done training? we're going to write our testing scripts, so training can take a while it depends on whether you're using it on the cPU or the gPU if you're running on the cloud or On your local machine, but training can take a while and We can definitely speed it up by running it on the gPU [I] ran this thing on my macbook and it took about eight hours To train so yeah it all depends on what kind of specification you have right So once we've so assume that we've trained it. It's going to take hours an hour to train So we're not going to train it right now, but assume We've trained it then we can write our testing [script] okay, so let's go to our testing code. Let's check this out [so] [let's] [just] let's check out our testing code. So for our testing code remember is a server Client Model that's what you think we're using a server client model, and that means that the that the simulator is the server and then our clients is then the script that we're writing so we want to think of it like a think of it like a server client models We're going to use flask [to] do this the web app framework, so let me talk [about] these dependencies and then we'll talk [about] the code so the first thing we want to do is Define our command line arguments a 64 is going to help us the code camera iMages Daytime 4 for [timestamps] of What else we've got high-level file operations socket Io is going to help us work with this as a real time server in terms of typing commands through event handlers We're going to use event lit for concurrent networking and then we have more web framework Dependencies and then image manipulation with [pillo] flask is our web framework and then [bio] to deal with input/output And of course Chaos just to load our model and our helpful class okay? So we'll start off by initializing our server ok it's going to be a socket io server and it's going to [use] flask To do this and we'll initialize our model And image rate as empty because we're going to fill those as we Later on and then we're going to set a max and min speed for our autonomous car So it can't go faster than 25 miles an hour, and it can't go less than 10 miles an hour And then it's we're going to define a speed limit as well what you're going to be at Max speed ok so then Let me skip this function because we're going to write it and let me talk [about] that the other functions So let's go into this main function. So these are the command line arguments that define You know what we're the model and where are the images that? We want to wear the images from the run are going to be saved [so] [we'll] load the model using our load function and Then [we] will tell it via command line where that model is and it's going to create an image folder And then it can either record the run or not depending on how we what we say and then it's going to launch this flat Middleware which is going to let our client communicate with the server and then we're going to deploy it as a WsGI server right and that acronym. What was it again? It was? WsGI Let me brush up on my web my web stuff here WsGI. It's not crud Create read update web server Gateway interface right we have 101 acronyms, okay, so What these functions are are these are event handlers and this is why we're using Socket Io in the server because behind under the hood for the [stimulator] It's got The it's got the architecture necessary to send commands that we need we just need to create event handler student accept those commands, right? So we don't have to code the server part We just have to code the client part So this is really the easiest way to get started with building a self-driving car this simulator that udacity release is the easiest way Obviously, there's brent depth [auto] Which is awesome and stuff, but you can only want that [to] [appear] [on] windows, and then also you [know] there It's got a whole bunch of extra features that we don't necessarily need But yeah, we could also do it in Grand Theft auto But right now this would be easiest if you're looking for the easiest way to get started with building self-driving cars This is the way, okay So so we have event handlers for [connect] which is going to just send it The s id the session Id and then we gotta send control which is going to send the actual commands that our model emits? But then we have the telemetry function and so that is going to be the That's going to be the big the meet of our code, and that's what we're going to write out here, okay? So we're going to write out this part right here. So we're going to say okay. So what this does is it's going to make the prediction and then send it it's going to make the prediction of what the steering angle is going to be and then we're Going [to] send it to the server [okay], so let's go ahead and write this out so if data, so we're going to feed this in our data, and then we're going to create an if statement says if data So once got our data as our as our focus. We're going to say let's get the current angle of the car So we'll say the steering angle We want the steering angle of the car, okay? So that's the first variable that we want and it's going we're going to retrieve it from our data Data frame, okay? And then we want the throttle okay? So remember we want these variables want these values so that we can manipulate them and turn them [into] one single Scalar value that will tell our car where to go and then we want the speed and the speed is going to be a float [it's] gonna be a float value and just like the rest. It's going we're going to pull it from this data frame using the B key which is going to speed Human readable key and Then we're going to get the current image from the center of the camera from the center camera of the car using pillow right? So is the image open so and then we'll use this work lights. [Io] comes into play [it's] what we use it because we want to convert it from Basics [before] this image into What we can read directly into our model so say 864 decode data and then image Okay, and so then we'll say okay, so once we have those things. We'll go ahead and and Do some tensor processing on this image, so we need to feed this image into [our] network right so to do that To do that, we're going to say image let's see what else we need [here] image and then so we're going to convert [it] the image to an array and then we're going [to] apply the Pre-processing step to it and then the model is [going] to expect a [4D] array So that's what that's what this image is going to be and so once we have that then we can say then we can say a Predictive steering angle for the car given that image, so [we're] going to say okay, so let's predict the steering angle using our model Let's say model that predicts just one line given the image and the batch size Would share which is whatever it's going to be also say one for the simple example And then once we have that we're going to say okay, so then given the speed limit that global value that we initialized It's greater than if the speed of the car [is] greater than what we set as our speed limit, then change it say okay? Well the new speed limit is going to be the min speed So then that means [slowdown], right so else if it's not then well then we can say the speed limit is going to be the max speed right so we don't want it to go faster than the [speed] limit is what we're saying here and once we've done that Then we can say okay the throttle is going to be 1.0. - the steering angle, so this is a magic number territory This is where guessing and checking comes in but we'll say - speed over speed limits Times two tours squared okay, and then we can spend the control Using that helper function that we defined before using the steering angle and throttle cool. That's it for this code right, so That's it. We properly pre process the image, and then we fed it [into] our model and that is outputs the The steering angle, and then we can use that to then send that control directly to the server via the send control function, okay? And it's going to admit that as a as a as A packet to the server and then the server will read that in so the actual event the event handling logic is is Written by the is under the hood for [the] simulator and we can just send it via our client That's it, okay So then once we have that then we can compile [it] and it's going to run the script just like I Did right at the beginning all right so to end this I'm going to answer three questions randomly from the Youtube comment And then we're out of here okay, so the first question is let's see what kind of questions We got here how to work with multiple data sets this is [a] great question and it's worthy of a video but essentially there are several ways of thinking about this one way is you can just Combine all your data sets into one big data set using pandas, and that's what I would do. That's [probably] the best way That's the cleanest way But another way You can do it is think about it serially so you'll once you train your model on one data set and then train your model In the next data set but I would I would just combine all [your] data sets into one two more questions I'm going to read from the comments What else we got here? How can [I] discover a foreign language like russian you tell me so this is for the language translation video so discovering a language is Quite a challenging task But think of it has associations so as long as you have some kind of labels and the labels could be a different language But maybe you're trying to discover an ancient language or discover the rules of some language You don't know then as long as you have some kind of associations You can treat it as a supervised learning problem, okay? And I've got two language translation videos and links to those in the description of that video and one more question Once you have this neural net trained can be optimized to run locally or does it have to be run as is every time Yeah, you can react amaizing Network You can [use] the pre trained model or you can use your code you can retrain it again it all depends You can retrain on your data or you can use it or you can use a pre trained model of someone else's train That's it for this session, please subscribe for more programming videos then for now. [I've] got to go Clean my kitchen. [oh], thanks for watching
