Advanced Tutorial-OpenCV in Unity

hello everyone welcome to the advanced tutorial of unity i'm leo glad to see you guys today i'm going to talk about a super interesting and technical functions in the unity that is facing detection by opencv have you ever played this game by your phone this was a really popular game on your smartphone and you can see lots of video in tik tok the people are moving their hats to play this game which is really fun in unity we can also utilize the face detection functions by opencv plus so what is opencv opencv is a library of programming functions mainly aimed at a real-time computer version originally developed by intel the library is cross-platform and free for use under the open source bsd license and it's also supports the deep learning frameworks you can easily access the opencv plus in a unity asset store remember it's free in this video i will tell you how to use opencv plus to detect human face in unity i will teach you step by step so take it easy after that i'm going to talk about how to apply the opencv plus to the actual game at the end of this tutorial we will also discuss the topic about the face detections the limitation of the opencv and some reflection of the process of my learning now let us start to create a demo to show how to use opencv in unity to detect human face firstly we open the unity editor and then open the asset store we can also find lots of really good templates and assets some of them are free so you can use it to practice search opencv plus unity and download it when the download is ready you can input opencv plus unity to your project and you will see the console will promote an arrow because of the nsafe code now we open the project sighting and select law and save code under the player now all the preparation work has been done by the way this asset provides lots of samples that you can use to practice opencv unity if you have time you can use it now we have to create a new scene just pick up the name as you like for your project i named the face detector in this scene we will create a new plane and use the webcam image as the material and then i will add a new script fixed detector to this plane now let's have a look at the script open the script add a new namespace at opencv shop firstly we have to declare our empty texture for this plane and then we create a array to store the list of webcap device in the start of the program we need to load all cameras in my case the camera i want to use is the first one in the list which is the device zero and then we need to make the camera start play under the update we need to use the webcam texture as the render texture okay now let's have a look at the running of the script now you can see that the camera is working now and you can see my face on the screen but the next question is how do we pass the frame to opencv and recognize the face in the camera firstly we need to declare a cascade classifier which is a pre-trained classifier officially provided by opencv we need to put the file under asset and load it and then we need to create a new opencv mat to store the current frame after that we need to create a new function find a new face and then we pass the frame under the find a new face we use cascade to detect whether there is a matching image in the frame and if there is store it as phases now we need to pass the frame to this function and then we use the defaulted scale factor and the minimum neighbor we set the detection type as scale image when detecting the face we print the coordinates of the first phase in the console this part has been almost done so let's test it if i move my face out of the screen the data has stopped updating so we get the coordinates of the face but can we directly see how it is recognized firstly let's comment out the previous render and then declare a new rectangle to store the face here i call it my face when open siri detects a face we start out the face in my face and then we will create a new function to display if my face contains the fixed data we add a rectangle to the frame here we turn opencv mat into texture and add it to material now let us check the final work when you can see the blue rectangle on my face which means my face has been successfully detected by the system now i'm going to talk about how the opencv can be applied in the actual game i have made a sample game by using opencv plus unity as you can see the player needs to move the head to avoid the water paths if you call it with the water paps the game is over to make this game i prepared some basic objects in the ones such as background water paths and game age the code is very simple all the scenes will move to the right at a constant speed when the trigger enter is detected at the age of the game a new one will be instantiated on the right side and an object will be destroyed when the trigger existed okay let's start making games now firstly as we mentioned before we need to create a new plan this time i changed the plan to a new transparent material so that it can blend well with the background we added the script of the previous fist tracker on this plane let's try and it looks good next we add the bird to the scene and add a new script we call it bird we want the bird to move up and down based on the y exists of the face so we go back to the face detector we want to add a new public float my face y to pass the phase coordinates let's go back to the script of the bird we can easily get the face y but there is a problem here we cannot directly use the value of face y we can try it to find a reason obviously the reason is that the coordinates of the birds are different from what we actually imagined so let's go back to the code firstly we need to add a float to record value of the previous frame and then we calculated the difference between the current and the previous frame after that we need to normalize the it so that we can get a relative direction of facial movement let's play the game and try again now we can use our face to control the movement of the bird congratulations we did it [Music] there are many advantages of using face detections in the game design the most important one is that face detection functions can enhance the player game experience by establishing an immersive environment to encourage the player to interact with in the game more interactions events would be feasible involving in face detection technology as phase detection has been widely used in other areas especially for the security works and the smartphone it also can be provided more possibility for game designer as recently ar and vr become increasingly popular in the video game market there will be a huge space for game development due to this technology opencv is not the most effective method for image processing however if you want to learn about image processing and the functions of phase detection it is really easy for you to start with opencv due to its low difficulties and the less knowledge barriers as a long-term technology accumulation the database and resources of opencv are abandoned to support the users to perfectly find a solution in their project for personal perspective the reason why we choose this topic is that we made some projects before by using opencv library in processing and pattern however it is still a challenge for us that this is our first time to utilize opencv in unity based on our research we found that opencv has not been commonly used in the unity project the main reason is that many games that require the face detection technology has usually chosen the smartphone as their platform for example apple has a comprehensive arcade and android also has its own application program interface for ar development the opencv plugin and example that can be used in unity are almost based on opencv shop opencv sharp is too old and we can't find enough resources and documentations to support us to use and learn the only thing that we can do is to deeply learn the source code of opencv in order to find out solutions that can be perfectly matched with unity thus we face lots of pressure and challenges to deal with this topic but we still figure it out in the end we can't say that opencv could be deprecated as for many special hardware equipment opencv could still provide high quality and effective support we believe that one ar or vr become an important component for game platform opencv will play a significant role in the future this is the end of the tutorial thank you for watching

Channel: xiayang zhang

Views: 30,844

Keywords:

Id: lXvt66A0i3Q

Length: 15min 7sec (907 seconds)

Published: Sun Aug 16 2020