Episode 9 | How To Export and Optimize an Ultralytics YOLOv8 Model for Inference with OpenVINO

hey guys Morgan's new video in this video here we're going to talk about how you can export pre-trained and also custom trained UV model so yeah we're basically just going to take a model we're going to export open window then we're going to see the performance increase that we actually get or the inference speed so we would like to be able to speed up our model significantly if we have an Intel based system so that it could either be like an Intel CPU an Intel GPU and so on but we're going to see that in this video I'm going to run it on an Intel CPU so you can also do it on the dpu but you will see the performance increase that you will get even on the CPU so you can actually get up to 3x CPU speed up basically just by calling a single line of code it will export the model and then we just need to open it up with the ultralytics framework so let's just jump straight into autolytics documentation we're going to take a look at this open Vino so we can also like integrate it with Robo flow and retune but right now we're going to cover openvino so yeah first of all we just have like this chart or like this figure that shows us we can basically just convert whatever model or eat all these different formats so we have tensorflow pie charts Cafe owner Nicks and so on right now we're basically going to take a pie chart model convert it to over and next and then once we have it in on X we're going to convert it and optimize the performance with the open window framework and then we're going to run it on an Intel CPU on windows so yeah we have this export mode here with yellow V8 we're basically just going to call a single function as we're going to see in just a second and then we can get up to a 3X CPU speed up just by X like doing this so if you have an Intel Intel based system you should definitely do this in your own applications and project like you will get a ton of performance just by doing that so here we can see that we actually have a used example this is what we're going to use in just a second so let's just go in and copy paste it we can see we have some different argument values here we have the format so we're going to use openvino we can also specify the image size and also half position so right now we're just going to use like floating Point 16 quantization instead of like having 32 bits and so on they might support act like better quantization or like lower quantization at some point so now just jump straight down to the open window YOLO V8 Benchmark those are some of the cooler things so right now depends on logging it on an Intel Flex GPU so let's just take a look at here we have the yellow one is the open Vino we can just see like this is as significantly upgrade or like a significant speed up compared to running it's just an on an X but also Pi towards and towards script here we're going to run this on an Intel Core CPU so let's go and go down and take a look at that Benchmark and we can see that we actually get pretty good performance especially like with the larger models here so let's not grow up and copy paste this use an example let's go into a code and see how we can use it so first of all here we import from autolytics we import the YOLO clasp we can create an instance of that we can just directly specify it as we've done in previous videos so definitely make sure that you check those videos out um where we go over after detection segmentation post estimation like how we can train your own custom yellow V8 model use it in your own project we even created like a security project around the yellow V8 model where we're sending emails if you want to give an alert if people are actually like present in the frame so that was actually like a pretty cool project make sure to check that out then we can go in and Export the model so the only thing that we need to do is call this export method we need to choose the format we're going to use the open bino format and then we can just create a new instance of our YOLO model with that format and we can have our open window model let's just go in and act like just create the results so right now we're just going to pass an image through our OB model but let's also have some results for the standard pytorch model so here we're just going to have results original and that will be equal to our standard model here and then we can just pass the exact same image through that there we go and here we're going to call results underscore OB for openvino okay so before we can actually go in and run our program we need to open up a new terminal so I'm just going to do that make sure that you're actually using a lower version than python 3.11 I couldn't get it to work on python 3.11 I could on 0.9 and 0.10 but let's just go down and choose 3.8 so we're going to you open a new terminal with that first of all we need to PIV install ultrolytics so PIV install ultrolytics I already have that on my computer and then we also need to PIV install on the next if you don't already have that on your computer so again and then the last dependency whatever we need is act like open Vino there we go and now we should have all the dependencies ready to go we should be able to run the program and they will do everything for us so first of all here we can see that it loads in the model we're going to get a summary of our YOLO V8 Nano model we can see the parameters the number of layers we can see that we're using CPU with torch and we can also see the CPU we're running it on and then we can see we start the export with O and then next so first of all again I mentioned we're converting from pytesforce to on net X and then after that we convert it to openvino so after we have converted on X it is basically just going to like export it to open window we can now see it exported successfully it took around like one second so let's go down and take a look at our inference results for the first one here this is our open window model we can see the speed first of all we're detecting four persons I want bus it takes um 21 milliseconds so this is for our open window model three milliseconds for pre-processing 21 milliseconds for inference two milliseconds for post processing and we can also see the image shape then the second one here is for our original model so this is basically just our part search model that we load in up here so first of all we have to take the four person one boss one stop sign so here we act like get an additional class detected that we didn't get with the original model or like with openvino model so it's now going to take a look at the image so this act like the image that we are passing it through so it acts like it detects the stuff sign up here in the top left corner um with the original model so I'm not really sure you have to like test that out and act like see what confidence score it gets it is act like correct prediction here with uh with the traffic sign so here we see we're detecting a bus four persons and a stop sign so this is actually like correct prediction so hit down here let's take a look at the speed and fair speed two milliseconds pre-prozing a bit faster doesn't really like matter too much um so yeah we have 66 milliseconds inference one milliseconds post processing so again we can see we have 21 millisecond inference for our open Vino model and 66 millisecond inference for original model so this is a 3X speed up by using this open window framework so this is kind of like the results that you can expect by just calling this single function or like this single method on your model you can also export to all other different types of Frameworks like tens RT or we have the open window here now you can also just use on an X directly torch script and so on so that's this video here guys I hope you learned a ton I hope you have opened up the eyes for the possibility of X like exporting your models to different formats and also different Frameworks so you can optimize it so you can speed up your model significantly without doing anything if you're using the jewel V8 model and just calling the export function so thank you have some watching this video here again make sure to check the other videos out in this tutorial and this playlist where we go over like all of it from autolytics and yellow V8 how we can use object detection object segmentation training on your own custom data set post estimation some cool projects also Linux hop and so on so we're covering everything make sure to check those out or I'll see in the next video guys bye for now

Channel: Ultralytics

Views: 5,849

Keywords: Ultralytics, AI, Artificial Intelligence, Object Detection, YOLOv5, YOLOv8

Id: kONm9nE5_Fk

Length: 7min 28sec (448 seconds)

Published: Wed Sep 06 2023