Deploy ML model | android studio | Kotlin | deep learning | model deploy | tflilte

hi guys how are you uh in this video we are going to learn how you can deploy your deep learning model onto a Android app by using the kotlin language this time we are going to be kotlin and yes I know we have already done so many videos about deployment of the model on Android app but still you guys are just every day I'm I'm getting new comment on a video my app is getting crashed while I press on the predict button and so on like there are so many similar errors you are you guys are getting and in this video I will try to solve all those errors and also I will uh make sure that you do not get any error in future I would recommend you to watch this video completely and when you are totally free just don't put the video on 2X and just start watching and skipping the parts in between just don't do it then it will going to create more errors so uh that's only the recommendation and I am going to cover this video in few steps the first step would be to have 30 flight model so what what actually happens when you dip you when you train your model it is usually in dot H5 format so first we need to convert our model into the dot TF flight model once we have the model we will create a new Android app and we are going to take that tflight model build the UI build the back end then import the model and just do the prediction so for this particular video we are going to use resnet 50 model which will going to take input as the image and we're going to Output the prediction of 1000 different classes so I hope you have basic understanding of python and deep learning models actually should have since you are watching this video for model deployment and I have this vs code opened over here and just not confuse you right now uh just to make you clear I'm going to convert this resnet 50. H5 model into the TF Lite model since the first step is to have the dot tier flight model which we are going to deploy so first of all I have opened this vs code in the same directory and I'm going to write a script convert dot py so first of all we are going to use tensorflow and tensorflow has a library to convert dot H5 format into 20 effect format you'll see how easy is that first of all we are going to import tensorflow as gear and also you need to import actually from Keras dot models import because first of all we are going to import the model by using load model and the name of the model Is resnet Res net50.h5 which is uh this one resident 50. H5 now once we have this model we are going to create a object of converter which is inside TF dot light dot t f light converter and in this there is a DOT from Keras model and in this you have to pass your model which we have just uh loaded okay which we have just loaded over here now once we have this converter we are going to call a function converter dot convert and this is going to return our light model let's just call this light model now once you run this we are going to have light model but we need to save it in locally so for saving we are going to use the basic functionality with open let's just call this light model dot e applied let's open right binary as uh F and we are going to say f dot right this light model that's it now if you run this script as it is it is going to create a new DOT tflight model in the same directory so to run that you have to say python convert Dot py like that is going to take a while about six to seven or about 10 seconds depending on the speed and you should have dot tflight in the same directory so let's just wait for time okay so finally we have light underscore model Dot tflight and but the problem here you see that it is of 102 megabyte and it is a lot this whole 102 should fit in the Android phone's memory so that's that might not be a problem it can run fine but we need to reduce this size as much as possible so for that we have something called converter which you can specify converter Dot optimizations and you can pass it as the array and for optimization you can say TF dot light Dot I guess optimization if I am not wrong optimizations dot default you can pass it like this and if we try to run this and hopefully it should run if we do not get any error okay we have error has actually it's optimized uh not optimizations so you just say TF dot light dot optimize dot default now you should run okay no errors now again it is going to take some time but this time the size should be reduced from at least 100 into P half at least my guess is there um other than that default you can also provide uh optimize for for latency optimized for size optimized for experimental sparsity you can try it on your own but for me this default works fine if you want I will also link this page in down description you can just check this out all right it is completed and now guys see the size it is 26.2 megabyte and accuracy might be I don't actually know but I tested it the for me the accuracy didn't changed the predictions are still the same but the size is reduced a lot now we have the dotted applied format model now let's move to the next step let's create a new Android Studio project so I would recommend you to go on with me just open your Android studio now it's the time um I'll just close this vs code this code will be in down description don't worry about that um I'll also close the terminal I'll close this and let's create a new project and this time for this video I'm going to use kotlin language I have already created so many videos with Java language there are hardly about one video with kotlin language I guess so let's create a empty activity and let's call our let's say resonant app let's call it resonant app and the language I'm going to use kotlin let's hit finish and we are going to wait for some time for letting it finish it's work okay so while it is loading up uh previously I have told you how you can convert your model into dot tflight format model now this model which I am using is dot H5 format and this takes the input image as BGR format and it is not the RGB format and it is very hard uh to convert the color space of the bitmap from RGB to BGR the ways you you can use uh is by using opencv which is already ongoing series on my YouTube channel so we can do that by using opencv but that is going to increase the size of the app so for this video only I just told you how you can convert your model into dot tflight model and while you are training your model make sure you just uh train your model with the data of the images as RGB format since it will be a bit harder for you to convert the bitmap into the BGR format color space at least for now I tried searching on Google it is quite bit tricky code you have to go with that so for this video I am going to use uh let me just take you to there so for this video I'm going to use this image classification which they provide is download starter model uh v11 model uh this model I'm going to use right now our model uh sorry Android app is created so let's create our UI I'm going to keep the UI very simple as simple as possible so we are going to use relative layout and we will going to have a image View and then two buttons and then uh the text view for showing the predictions so let's do the same let's set the image View and the bits will be actually let's just say match parent then let's give the height of 500 DB let's give the background just to see what is going on let's give the ID let's say it is image View I guess that's enough now let's create two buttons and one um text you to show the result so buttons will be in linear layout let's say fifth is match parent height is wrap content just give ID in layout and it will be below of our image view so the width will be wrap content wrap content let's give the text as let's say select and let's give ID select ptn I guess that's enough let's copy this same button and the it will be predict button ID will be predict button let's Center this horizontal two nope let's set width to wrap content done and then finally we are going to have our text view to show the prediction it will be wrap content wrap content let's give ID as result View and let's give text as prediction prediction and it will be below of our linear layout that's all uh we can give font size update text size text size let's give um let's say 20 DB how much is it is fine let's keep margin left just a little bit 10 DB let's sit and let's give the margin right to this button press it 10 DB all right finish this is our UI if you want to change it just go on with it I've just kept it simple as simple as I could all right so here we are in our backend main game will going to begin over here first of all we are going to initialize our buttons so there will be a select yourself of type button then we will have a predict button of type button let's import it option enter and we'll have a view also so it will be a text text and then we will have finally our image View that's it let's initialize them in our on create so it is a select button equal to find view by ID r dot ID dot select button let's do the same four times we have a predict button we have a predict then we have a result View then finally image View image View okay now when the this select button is pressed we want to open the gallery and the user will going to have to select the image so when this image has been selected we are going to show that image inside this image View let's do that thing now so we have a select button dot set on click listener so when this button is pressed we want to create a new intent which we're going to open gallery allowing us to open the select the image so let's create a where intent of intent equal to intent let's import it option enter and intent dot set action it will be action dot action get content so it is intent dot action get content and intent DOT type set type now the type of data it is going to access it is image type so any type of image now we are going to start activity Simple Start activity for result since we need the result from the activity whatever the image is selected we want to access that data so we are going to pass the intent and then the code let's give the 100 code you can give it anything uh finally uh inside this class outside of this function we are going to override a function you are going to see all activity result that's it now we are going to check if this uh request code is 100 which we have passed over here so if request code is equal to 100 we are going to access the URI first of all so it is Data dot um okay so we are going to say URI is equal to data dot data and it can be now so we are going to give it question mark now we have this URI we are going to uh this is uniform resource indicator I guess so from this location we are going to read the image so the image which we are going to read I am going to store it inside this bitmap since we are going to use this later on when the predict button is pressed so let's import the library so we are going to say bitmap is equal to uh media I guess yeah so you can say media store dot images dot media dot yeah it's get bitmap so you can just pass this content resolver so you can say this dot content resolver and pass this URI once we have the bitmap I'm going to say image view dot set image bitmap and let's just set this bitmap like this everything looks fine over here until now let's try to run this app uh and up till over here and let's see if we get any error until now or not all right so emulator is loaded so when I press the select image button Gallery should be open and I should be able to select the image and the image should be on the screen that's it everything is working totally fine as expected all right now the main part which is what should happen when the predict button is pressed when the predict button is pressed we are going to use this bitmap and we are going to uh send it for the prediction to our model and then get the output from the model so first of all we need our model so for importing your model inside your app we are going to click on file new file new and other and tensorflow Lite model now you need to locate your model wherever it is and just hit open and hit finish and it is going to take a while to install the tensorflow library for your device and finally you should have this sample code so we are going to just copy this sample code if you are doing this with Java you should have to click on this Java and here is the Java sample code okay here here now what we are going to say predict predict button dot set on click listener so when predict button is pressed we are going to load this model let's just get rid of this let's import this option enter and for context you need to pass this transfer buffer that's option enter data type option enter and this byte popper we need to resolve it fetus wait a bit all right so here is our model imported now we need to pass our image over here but we can't we we cannot pass our image as it is for uh having our bitmap into the pipe buffer we need to follow these steps so you need to create a tensor image sensor image like this so you can just say answer image and in this tensor image you can pass the data type which is uint8 for our model you can see over here like data type dot Hue into it so you have to just pass it like this so data type dot unit now once you have your tensor image we are going to say tensor image dot load and you need to pass your bitmap over here which bitmap this bitmap which we have selected by pressing the select button right now from this sensor image we can get the byte profit like this cancel image Dot um buffer just get it like this now if you run your app like this your app is going to crash for sure why because the size of the image we need to convert our image to 2 to 4 by 2 to 4 pixels but your image could be of any size so for that we are going to use image processor which is again provided by uh tensorflow Lite so for that we are going to say let's create image processor where image processor image processor image processor um dot Builder dot add now when you hit dot add you have to pass the image operator to call it image operator so they have a resize operator which can resize the bitmap into the particular size you want so you have to give two to four height two to four feet and then the resize method so you can say resize method Dot and then find you can say dot build finally to use this image processor what you have to do image processor dot process and it expect input to be as tensor image so here we have tensor image and it is going to return that answer image like this okay so if you run your app like this it should run no problem nothing will going to occur now fun point I would like to raise at this point over here okay it is totally fine that uh our current model is using the RGB format image maybe your custom model would expect the image to be in grayscale format so how do you convert the RGB image into the grayscale format luckily for us sensorflow Lite have a normalized tensorflow has a normalized op for us here is tensorflow has normalized op and you can pass it like this and you can give 0.0 F and 255.0 F like this so what this is going to do is normalize your pixels from 0 to 1 if you are using normalization and if you want the grayscale op again here we have a grayscale op but for this default version which this uh automatically loaded in our Gradle script let me just show you zero point uh 1.0 so all right so the version you are going to use is 0.3.1 actually and you can hit sync now so after uh downloading this version you are going to have a grayscale op also so dot add transform so now you have transformed to grayscale op also after using this 0.3.1 version um so this is how you can use image processor to normalize your image from zero to one if you are using uh normalization and you can use a transform to grayscale op for converting your RGB image to grayscape but for our case we are not going to use it since our model do not expect our image to be in that format okay so now the last step is this is the output features which is the output from our the model so you can just get float array from it get float array now from this float area we need to get the index with the highest value so we can just write very simple Loop for that uh you can say output features 0.4 each indexed this one now we have the index and the value so we can say output features 0 maximum idx is greater than current value sorry smaller than current value we can say maximum idx equal to index that's it now finally we can say result uh result View dot set text and maximum in the x dot the problem with this is that it is going to give us the index we need to uh convert this index into the particular label whatever liabilities so with that model which you are going to download they are also going to provide you labels so I'm going to just rename it to labus.exe so it is going to provide you labels of all the Thousand categories so whatever index we get as the maximum index we are just going to use the labels so for that we need to import our labels.txt into our Android app and the location where you need to import should be inside the app you need to create a new directory new directory let's call this directory assets if it do not show like in my case it is also not showing you can click on Project and inside your resnet inside your app here we have the assets let's paste our labels.txt over here that's it we have our assets labels Dot txt and inside your main dot uh mainactivity.kt you need to read this levers.txt so for reading again it is so easy in kotlin you can just say labels equal to application dot assets dot open and give the file name which is labels dot txt dot buffered reader and then finally you can see uh read lines and it is going to return you uh let me just show you if you didn't paid attention to it so read lines is going to return you list of strings so every line will be split at slash and every new line will be any with with a new index so here is our labels and finally when we are saying result view dot set text instead of index we are going to say labels with the max index that's it this is it for our whole app let's try to run it and see if it works fine or not uh I guess our app is crashed let's just try it once again yeah so for resolving you need to at least get the error at least what is the error so for getting the array you can click on run over here now you should have the error like what actually caused this error to be happened in our case it is unable to start activity file not found exception which is labors.txt okay uh it says that labels.txt file not found all right guys actually the error occurred because you need to uh create the assets folder inside this directory Source main then assets so our problem was we have created the assets folder inside this app but you need to create the assets folder inside source and then Main and here should be the assets folder so let's try this once again let's try to run our app once again and see if still there is error okay there is no error no crash error is gone you need to put the assets folder inside source main assets just make sure you do that let's try to select the image apples and click on predict this is Granny Smith Granny Smith it's a green apple store Apple which is called Granny Smith okay it is Apple probably let's try to select this dog because his golden retriever fine let's try to select this banana this is banana let's try to select this apple this is pomegranate yeah it looks like kind of let's try to select this cat it's a CD print card okay we are not getting any error but uh still still if you guys still are getting any error just try to watch the video once again and see if the error can be resolved still if you have any error you can put that error in down comment section you can get what the error is by just clicking on this run button what uh actually the problem which caused that that app to crash still if there are any error the most of the chances which I think is the cause of error is this data type mismatch so probably here could be flow 32 and you might be using unit 8 that is going to create a byte buffer error so you can just check that out over here still if there are any error you can ask them in down comment section we have our model loaded and the predictions are being done fine there are no problems uh this is it for this video I will just end this video over here and I'll catch you in the new video till then goodbye

Channel: ProgrammingHut

Views: 24,324

Keywords: android app, tflite, model deploy android, machine learning android app, latest, buffer overflow error, float32, uint8, buffer size, app crash error solution, how to deploy, dl model, ml model, h5 to tflite, converter

Id: gVJC1j2n9tE

Length: 27min 33sec (1653 seconds)

Published: Mon Dec 26 2022