20 Installing and using Tenssorrt For Nvidia users

hey guys in this video i'll be showing you how we can install and set up nvidia tensor rt so make your way to this document page documentation page by nvidia just search up tensor rt nvidia install and it should be the first link right here after that's done go to getting started now make sure you have one of the following cuda versions installed i have 11.6 update 2 so i should be fine after that go to number three and press on this link should take you to a bunch of developer stuff you need to go through all that stuff and get to this page where you're able to download this page this page right here i'll be downloading tensorrt version eight which is the newest one agree i'm not only the newest version i'm going to be downloading the package for 11.6 because i have 11.6 i already have it downloaded right now so i'm not going to install that after that's done we need to do the prerequisites before installing this so if you go to zip file installation it nn install docu and it should be the first link right here so after we've made it to this link go to number three installing cuda nnn on windows it's just a bit of a scroll down right here so make sure you have all the stuff correct here go down go down go down installing cuda and then installation guide for windows installing and then you have to register for the nvidia developer program go to here so i'm going to open this link i'm going to go to download cuda and then i'm going to agree to the terms of services and i'm going to download an archived release because according to tensorrt we need version 8.4.1 so i'm going to get 8.4.1 for [Music] cuda 11. i believe i'm using 11.6 update 2. so for windows local installer for windows should download the zip folder after that's done i can exit out of this page go back to the install documentation and follow the steps here navigate to your package path unzip it and copy the following files into the equivalent and video files okay so this should be fairly easy so i'm going to open up a new file i'm going to go to this pc os and it should be located in program files scroll down a bit you should have nvidia gpu computing kit toolkit right here so this is from when we installed cuda double click on here double click on here and here is what they want us to have open so i'm going to leave this open now i need to follow these steps so i need a copy to unzipped package so cuda nnn for windows go to nn move this here move this here so it says that i have to copy bin into bin include and to include and lib into lib so cuda and then bin copy all of this i'm gonna put it in the equivalent cuda bin i'm just going to wait it to finish copying place files and destination i already have it installed so i'll just replace the files in destination now once that's done i'm going to do the same for include so i'm going to copy all of this we drag it and drop it in the include of cuda replace these files in destination do this all for all current items i already have it installed and i need to do the same thing for lib copy all of this drag it and drop it in lib replace files and destination do this for all current items once that's done i'm going to go back to this page select the following environment variable where cuda nnn is located to access the value of path environment variable and follow the following steps add the nvidia cuda bin to path variable okay so we have to add bin to path to this it's fairly easy so let's unzip cuda nn unzip it extract all i'm gonna extract it right here i believe all we have to do is add the cuda bin to the path variable okay so here's the extracted cuda i'm just gonna leave it in my desktop for now but you can put this anywhere you want on your pc so i'm gonna go here i'm going to search up environment edit environment system variables click on this click on this we should be able to edit our path variables here so let's add cuda nn so we're going to copy the path to the bin folder right here copy s path i believe system variables i'll just go to path i'm going to add the path to the cuda nn bin here press ok press ok press ok and that should be everything for the cuda nn installation so i can now close out of this and close out of this now to install tensorrt you install the zip folder once that's done go back to the installation documentation and go down to zip file installation so we have cuda 8.4.1 installed now now we have to unzip the file okay that's all fine so now we have to unzip the tensor rt extract so there are two ways that we can do this uh but i prefer path b as it's much easier but basically we're just going to copy all the dll files from lib to our cuda installation directory so we're going to copy all the dll files from lib into bin this is fairly easy i'm just going to open back up the file so is our tensor rt 1.6 open here rt so we have to copy everything from lib i actually took all of the files we're only supposed to take the dll files and we're only supposed to copy it so click on all the dll files i'm gonna control c click over here ctrl v okay now that copied all our dll files and it should already be in path install one of the tensor rt python wheels so now we have to install one of these wheels to do this we have to go to that's rt go over to python now try all of these paths but basically you'll just be running this command over and over again with each path being different so i'm going to open a new terminal here and i'm going to copy this command i'm going to paste this command here now i'm going to copy the path of one of these so you can try each one not not all of them one of them are bound to work so i'm gonna copy this path paste it here press enter okay that one doesn't work i'm going to do the same thing again just copy this go and paste it i'm just going to actually run this so it's not going to work but then i can just do this i'm going to copy the path of the next one copy path paste it run it that one doesn't work copy the next one copy path paste it run it not working copy the next one copy copy as path paste it run it and this seems to be the correct package as it ran through with no errors so once you're done that step you can go back to the documentation go on to the next one to verify your installations are working we don't actually need to do this step if you're using tensorflow pi torch we have to do this step we are using pi torch so we have to do this step so we have to do copy we have to do the same thing so copy this i'm just going to paste it in here oops i didn't copy it go paste it i'm just going to run it but it's not going to do anything but this will allow me to go up here remove the three at the end here run it again oh i'm just running it so i can just use the arrow keys to navigate back to it so with it back here i'm going to go back to python this time we have to install graph surgeon click on graph surgeon copy this path paste it enter it installs it perfectly fine do the same thing for the next one so the next path we're doing is uff so uff copy this path paste it paste it run it okay here you go that one's installed and one last one we are doing on an x on the next graph surgeon so to do this one just go back here go to on the next graph surgeon copy this path paste it and that should be the installations complete okay so that should be the installation is complete i'm going to test that they're complete in a second and i will show you how you can export your model into the engine version which will allow the model to run possibly two times faster or about like a 60 to 90 improvement in inferencing speed i will see you in a second okay so we are going to export our current model which is in pi torch format into a tensor rt format now here are the different exports that you can do you can export it into tensorflow lite but for our purposes since i have an nvidia gpu tensor rt is the best way to go to improve our speed so to begin exportations scroll down here it should be python export weights and just copy this open your vs code go to new terminal and paste it so we need to replace export dot pi with i believe yellow v5 export dot pi path copy relative path i'm going to paste that path here and our weights file is not called ulv5 but it's called best so your model should be called best or whatever it's named down here just replace that with the name here and we're going to include engine now we're going to add a few more settings so we're going to add device equal device zero so it's going to be running on our gpu and if you really want to make it faster i'm pretty sure you can put half precision on your model should say you can also do add dash dash half to put half precision i believe this can improve your model speed but i'm not gonna do that yet i'm i'll run two different exports i'll run run with half precision and one without it and i believe you need to put the model image size uh let me just check the actual export dot pi so it's just dash dash image and then we input a list so image cs or image then we're going to be a list it's gonna be for me it's 736 by 736. you put whatever your image size is running at the moment but that's going to be mine i'm going to run it and wait till the export dating is done invalid value oh maybe i don't have to put a list i'll just put 736 here run it and it's working okay so i'll see you when the exporting is done okay now that our model is done exporting i'm going to export in additional version except i'm going to put this one at half precision to show you the differences change this rename actually just copy paste it half okay go down to my console put a half okay so the half and the regular model have finished uh exporting i will now show you the performance differences between the two so here is my fps with best.pt so the pi torch format of my model i'm getting about 35 34 fps now i'm going to switch over to best dot engine okay so here i am running fast dot engine and we just went from 35 fps all the way to 50 fps and now i'm going to run half precision dot engine so here i am running half precision with recording all of this is done with recording and i just went from 50 fps using non-half precision to a solid 60 fps with recording software on now that is all for this video in the next video i'll be showing cpu users an option to improve their performance a bit

Channel: SmileMe

Views: 11,035

Keywords:

Id: KoCms6vMH6A

Length: 18min 40sec (1120 seconds)

Published: Wed Sep 14 2022