Importing a Python Tensorflow model in Unity! (using a websocket)

hey guys and welcome back to unicodeb in this video i'm gonna show you how to import a python model generated using tensorflow into unity now there is a standard way to do this which is using the onnx format and then using barracuda which is a unity library to actually import the python model into unity we are not gonna be doing that today today i'm gonna show you an alternative way i found to import the python model and you might say nico why would we ever use this alternative way well because i was doing some test projects in the past and i actually had some trouble converting tensorflow models which were in different formats like h5 or just a tensorflow save the project into unity because in order to do that and in order to use barracuda i had to convert those models into onnx which is a different format and sometimes because of version mismatch or because of some optimizers i actually couldn't import those models and actually had some trouble this new method that i found is actually much simpler and the only downside is that you have to run an actual python server while your unity app is running the way that it works is basically that we have a python server that has tensorflow which is basically running the model and then we communicate through unity via websocket to receive the inputs of our model and we calculate the outputs in this way we can execute the model but we could also manipulate the code in order to train it as well so right here i have a model that is trained on the e-monist data set which is basically like mnist but for letters this means that i can draw a c and it's going to predict correctly that this is a c i can also i don't know draw another letter and it's going to predict it correctly and i can go again and again well it didn't work this time but you get the gist of it and now i'm gonna show you how this actually works but before doing that please leave a like and subscribe on the video so i don't know we can reach more people and you know more people can be helped by these tutorials or i don't know if you want to share this with your friends or other people that might be stuck and maybe are trying for some reason to import a python neural network into unity and they don't know how let me know how you actually found this video and it would be really helpful to create more content as well also if there is anything you want me to do a tutorial of just leave it in the comments and then i'm going to reply if i know how to do that tutorial and if i don't know how to do it i'm not going to reply because you know i don't want to seem like the guy that doesn't know anything what are you talking okay but jokes aside if you have any ideas for better tutorials leave them in the comments i'm gonna take a look at all of them anyway let's get started first of all i'm gonna show you just briefly how the ui works but it's pretty self-explanatory we have these two sliders right here they basically control the radius of the input area and the strength another thing that we have is constant prediction which basically will send our network which is in the python server a prediction every time the table of pixel right here changes which is really experimental because it can stretch the web socket and sometimes it crashes but it's actually pretty good to see as you can see it predicted really quick okay let's pretend this is a queue i don't know maybe the it doesn't do the d okay there he did it uh let's do i don't know a b but it actually drive it correctly this time okay there you go okay this is really amusing but okay let's see actually how this works in the code we have four simple classes i'm gonna go briefly through all of them naturally the code is on in the description the first two classes are only for the ui so basically this is a script that is on every single pixel uh as you can see we have the pixel prefab right here which is just a pixel that can be black or white or any sheet of gray and so we have these functions to change the color and all of that stuff i guess you don't care about this then we have our pixels handler that basically generates all of the pixels in our case the input is a 21x21 table so we're gonna generate this number of pixels and we're gonna instantiate their prefabs and this is the code that actually draws on the table which is also really simple it's basically doing a physics overlap circle all and modifying all of the pixels that are affected by the mouse simply uh by the strength component which is basically the slider i described before and also the radius is given by the radius slider finally if constant prediction is on so that toggle that i showed before is on then predict every time there is a change we also have a reset button which basically resets the entire table and also the predict ui which happens when you click the predict button okay now that all of the ui is out of the way we can actually go into the core of this which is the python server so this function right here is actually the predict button and every time i want to do a new prediction i'm gonna first read the pixel from all of the inputs which is basically done by getting all of the pixels that i have instantiated and getting their color and this is basically the input which is a 21 by 21 tensor and then i pass this input into my client which i'm gonna show in just a moment and i receive an output which is an array of float naturally this is specified by the model that i have right here the model.h5 which basically returns a float of 26 elements which are the letters of the alphabet and naturally the element which with an ir number means that is the element the network is more accurate about and which is the letter that the network has predicted so i'm gonna get that one and i'm gonna put it into a text funny story you might actually say nico why aren't you right here modifying the text from unity you're actually doing why are you not doing prediction text dot text well the reason i'm not doing this and modifying the text view from unity is because this is not the main thread this is a callback so it's actually gonna give you an error if you do that so this is why i'm storing it into a different variable and then in the update function i'm actually assigning the text to this temporary variable this is just an heads up because i lost quite a lot of time just because of this stupid line of code also we have an error handling function which right now is empty because i'm a really good coder but right here in theory you could put some try catches to see if the socket is gone for some reason and you can open another one and all of that stuff but yeah we're gonna put a little to do here and just forget about it you know this can just stay this way okay but jokes aside let's actually see where we're generating this client we're basically generating it right here oh i'm actually getting it from unity from the inspector but basically this prediction client is this one so in order to communicate from unity to python i'm using an external library which is going to be linked in the description it's basically this one and it's honestly really really helpful so go check that out and give it a star maybe on the unity hub i think it's a it's a heart actually it's not a star but give it whatever means of likeness there is on the website but anyway using that library i'm actually creating a new class which is a runnable thread which is a prediction requester this class is gonna have a request socket which is our client and then it's gonna have two callbacks which are the ones that are gonna happen if the output is received correctly so if we get an array of loads or if there was an exception then we're gonna run this thread and this is basically the code that i got from the library that you can check out in the description and all that it's doing is it's connecting to a local port 555 by the way you can change this of course and then it's retrieving the bytes every time it receives some bytes it's going to copy these bytes and transform them into a float array which is basically the output that we want and then it's going to invoke the callback function the callback function is the one we defined earlier when we actually send the input which is all the way right here this one is the output function the send input function instead is the one used to send the input to the server and as you can see we are doing the exact same thing but backwards we are taking the float array transforming it into a byte buffer and then sending it via client dot send frame byte array naturally all of this has to happen after we initialize our client and we're actually initializing it right here we do initialize server that creates a new instance of the prediction requester that basically initializes it and then we do start to start the thread and this is the predict function you saw before as you can see it accepts an input and output which is a callback and then an exception and as you can see the exception will run if the input fails sending this means if the server or the websocket is down for some reason and this is basically all the code on the client side so now naturally we are communicating with this port but what is on the other side of the port well it's some python code which is doing the exact same thing but much simpler because it's python so of course we are loading the model.h5 which we have in the same folder as you can see it's right here in the project we are using keras models load model once we have our model we are establishing a connection using zmq with the server and then we keep on looping forever and we keep on expecting bytes so you may say why are we expecting all of these bytes this is basically 28 by 28 by four because basically we are sending 28 numbers by 28 so we are sending 28 floats but every single float is composed of four bytes so if we do by four we get exactly this number so this is why we are expecting exactly this number of bytes once we actually receive this number of bytes we are gonna convert them into a float array just because it's simpler and then we're gonna go into our model and using the keras package that we imported we are gonna predict uh the response and we're gonna expect an answer so for some reason the model that i use to once them into this shape which is like one seven eight four uh which i see i assume it's more helpful to the model itself so i mean i guess i'm converting it to this form but naturally this and these are gonna be different depending on the model that you are importing and that you are like depending on your inputs and outputs so yeah once we actually have the prediction we convert the prediction into bytes and we send them back to unity and now just to finish the loop unity gets the prediction back from this function right here runs the on output receive function which is the one that we set right here sorry right here and this predict function basically just calculates the right output and prints it on the screen and that's how the magic works and just in this way we can do the execution of a model without actually converting the model into unity but just keeping it on a python server now there are some things that you can also add to this you can have the server not be local but actually be an outside server maybe you could do it with i don't know a different kind of server a different kind of back end another thing that you need to do though if you have a local server is that every time you run a unity you also need to run the server and you also need to rerun it if it crashes for some reason you could do this via a script and maybe i can show it in a different video or as i said you could just have this server run somewhere else not on the client and that's it i hope you really enjoyed this fun little tutorial i hope to do more soon before college actually starts and you know if you enjoyed this video and it was really helpful to you leave a like and be sure to share the channel with your friends it would mean the most to me uh if you know anyone actually got helped thanks to this video because i actually you know just worked on this for fun but i can clearly see how converting a model into unity can become a problem sometimes with all the different specs and i also and you have no idea how many exceptions i had when trying to convert this model into onlx maybe i was doing something wrong on my end uh so you can tell me in the comments what you guys did in order to convert it but i found this solution which is also pretty nice and if you are doing something just to test a model in unity it could be really viable so let me know if this video helped any of you leave any answers or anything you need in the comments and i'm gonna be sure to leave all of the required materials for this video with the code itself in the description that said i hope you really enjoyed this video leave a like if you did and i'm gonna see you guys in the next one see ya

Channel: uNicoDev

Views: 16,373

Keywords: unity, game, gamemaking, tutorial, how to

Id: yIHGUD4eEoY

Length: 12min 21sec (741 seconds)

Published: Thu Sep 02 2021