01 Object+Color detection with OpenCV in AR unity (object detection)

hi guys i hope you are doing well uh welcome to my channel in this clip we want to learn how we can download import and use the opencv in our augmented reality so for download and install the opencv we have to come to assetstore.unity.com and in here write opencv for unity as you can see it is here and it already has a lot of examples which can be useful for every uh scenario for example body tracking and in here face tracking and also at the material and face tracking on the predefined photo and all the other types of examples which are very useful for our projects and and this plugin is cross-platform as you can see it can support ios android support and chrome [Music] chrome os it means webgl or html5 and windows mac and linux and other things as and you can download the example of its from android demo or webgl demo in here and make sure you your unity version has to be 2019.4.31 or higher maybe in the coming days maybe we will see that another version of two point and two point four two point nine or maybe higher so you can you have to check your unity version and then you can buy it it is ninety five dollar and and download it and import it to your unity so i already have this version but i have the version 2.4.7 okay doesn't matter and you need to use your unity version it doesn't matter it is personal or pro or other versions and uh for now i changed the platform to android so for example i need to check off the autographic api and remove the vulcan and also the minimum api level has to be android 7 api level 24 or higher so these options are necessary for our uh project okay now let's get start with uh object detection example of opencv so we will go to the opencv for unity examples and in here we have main modules and dnn we have something it's called and tensorflow inception this is the ex and the basic is example of object detection in uh opencv and you can use it in your project it doesn't matter what is your platform it works on open uh android ios windows and even webgl i tested webgl demo and uh but as you can see if i click on play it doesn't work because it says model file is not loaded please read console message what is the message of console we'll stop the game and here it says okay please read streaming assets dnn set up dnnmodule.pdf but before that you need to know that your streaming assets folder from the opencv for unity folder need to transform to main assets folder so drag and drop and put it on assets so we will have streaming assets in here and then let's go to this address streaming assets the dnn and set up dnnmodule.pdf okay we will say dnn and setup the nn module.pdf okay uh you know this these examples uh need to have some uh assets and resources to be able to work so uh in the previous versions of opencv you could to uh have them you could have them in the main package of the opencv but in the newest version they've decided to put their resources and assets in on the cloud and you need to download them separately so it depends on your project and your your example you need to download the resources and put them in the streaming assets folder so what was the name of our scene and tensorflow inception so you need to come to tensorflow this one setup for tensorflow inception example at the first step is download and unzip this file storage google api dot com this one if you click on it directly you will download this setup for and this zip file i already downloaded it it is here inception 5 is here but you can download it again but yeah after download it says uh copy these two files tensorflow inception graph dot pb and tensorflow and imaginate comp grab graph label strings dot text to this folder inside of the dna and delete inception zip and in simsched folder 5. ok so we will download it here and i double click on this one right click okay so uh i will add these files into my dnn folder all of them imagine a comp graph label strings.txt and tensorflow inceptiongraph.pb and license i put them i put all of them into the dnn folder of streaming assets and i close it okay now let's test perfect it works well it can detect my sunglass and look at this it can detect my sunglasses okay now uh as you can see in the uh object tr and detection you could understand uh the object detected in the specific area is in the process of the detection for example if my glass is here it can say okay this is sunglass but if my glass goes here it cannot detect or it cannot detect in this part it can detect my sunglasses it means need to make sure our object which we want to detect must be in this area of our screen so i want to make sure my object is in the right place but before that i want to save this scene as a separate scene in the scenes folder i call it uh object detection d sorry detection okay and now i want to add something uh for better understanding as a sign for example a quad as a child of this guy and i say one one what's one is too big and as you remember the song the place was here so i will doing something like this and i want to create a material for it for better understanding uh object detection area okay and i say color green and make it transparent and change the alpha to for example 85 okay and i will put object detection area and then drag and drop this material to here okay and put it a little bit front of the screen something like this okay and for example something like this okay it's fine now as you can see if my glass is inside of this area so it can be detected by our system if it is outside it doesn't work okay now uh we need to deep down inside of the object detection system as you can see we have uh just a quad object which has three important components tensorflow inception webcam texture and webcam texture to matte and also fps monitor in the fps monitor we will see the fps number in here and also a label in here to write the final result of our object detection and the duty of the webcam texture is turn on our webcam or device camera and put the screen view as material of this quad but the detection process comes from the tensorflow inception webcam so uh for object detection and use it as a trigger of uh any uh action or any function we need to monitor the final result of object detection so we will come to the uh tensorflow inception webcam script and open it and into the visual studio and in here as you can see we have all uh process steps of the object detection but somewhere we can see the final result so in here in the function of void update in here we can see the final result of object detection and we can monitor it look at this in here the result the final result will appear with this uniform this pose best match class sunglasses okay so we will have two parts uh first best match class and second is the final result so if we can detect and monitor this final result we can use it as a trigger of any function we want i want to say public a string um target name and in here public text as a ui and ui element final is that for example okay and in here because i don't want to use the fps monitor because i for optimization i want to disable the fps monitor so i put it in out of the this condition if for example this final result or maybe target name dot contains final result i copy this part and paste here okay we will say final result dot text equal i found it perfect if it is not contain of it let's find it okay uh find it please okay now come back to the unity editor and we will add uh because i don't want to use these buttons i want to disable all of them and also and click on code and disable the fps monitor and in here i will create a new ui element and legacy text and put it here oh my dear okay and change the color to white and i will say final result okay good drag and drop my final result to this guy okay and in here i will say uh the target is sun glasses or other objects for example mobile or phone or i don't know something else so if our final result is inside between one of these words so we can detect okay this is our target object okay oh sorry i need to check on the fps monitor for seeing what is the current result uh maybe i wrote it wrong sunglasses so glasses okay yes i found it look at this sunglasses it is sunglasses it is not find it please sunglasses i found it perfect for mobile phone maybe it cannot detect it because of my uh [Music] lighting yeah it says its projector yeah okay because my phone is black so it's better to use something is bright so it's fine now let's define it an action for when we detect an object so uh because we don't need to use the fps monitor i want to add something cool in here final read text equal this is the name of it i'm sorry okay and then we will say okay we don't need you my friend and come back to the entity editor and check of the fps monitor in here okay let's come back to the unity visual studio and in here we will add a using unity engine oh sorry unity engine dot events and now we will define public unity event uh order found when we found our target object do these orders and also when we not found okay we will call these functions here found order found dot in work when we found it and when we don't found it order not found that's it okay as you can see we have two separate parameters for import any function and any actions we want so these are our order phone found and these are for our they're not phones for example i will create a simple s object and called it test and i will create a simple functions public uh void change oh sorry change transform okay and i would say trends transform oh what's going on here let me save and remove this guy it has to reload again [Music] uh for example local air anglers new vector three for example zero point twenty and i call another function i will say reset transform okay we have to examples functions sorry and for testing these actions i will create a new cube i called it test object and add this test script to here and i will put this uh object front of the camera where is my camera it is here and i need to put my test object in front of the camera for example here 50 or 50. that's it okay and final step is we need to add the test object into the order found when we found our object for example do this function test change transform you can call any functions from other objects or this object as you want so you don't have any restriction for this part and also when we not found our object do another function reset transform for example okay let's see what will be as a oh i need to change the target name to smallest okay look at the angles of the test object and when we don't find it perfect it works well okay but there is a problem with the naming uh i said in here when we don't find uh when we didn't find it uh let's to say it find it please but in here we say okay let's define let's to write the name of uh object so i'll make it as a comma the comment and let's test it again okay it works well perfect even we can say uh [Music] class or other types i found for example something like this so it will be beautiful i found bravo now we can use this object detection as a trigger to call any functions we want okay in the next step we are going to use a color detection and use it as a trigger of any function we want for example we will find uh and our target is finding uh for example a green uh for example a green glass not any gl uh all the glasses just green glass is our uh target object and try to call the any function we want okay let's go to the next step if you uh like these types of videos please subscribe my channel and like and comment uh below of this video or other videos and related to aar we are annex or in unity

Channel: Abolfazl Tanha - Game & XR maker

Views: 6,645

Keywords: Augmented reality, AR, unity, mobile, unity dev, ar dev, game dev, unity 3d, AR unity, ar foundation, Unity Game, AR game, AR outdoor, AR indoor, Android, iOS, AR android, AR iOS, Android unity, iOS unity, Online, Unityoptimization, modeling, unity graphic, sceneoptimzation, model optimzation, opencv, opencv in unity, unityopencv, opencvforunity, object detection, color detection, object recognition, color recognition

Id: qzAjb1iZ6U8

Length: 26min 20sec (1580 seconds)

Published: Sun Jul 03 2022