Image Classification App | Deploy TensorFlow model on Android | #2

hey everyone in this tutorial we will be deploying our custom neural network that we coded in python into an android app for image classification you'll learn how to export your neural network that you coded as a tensorflow lite model then how to use the tensorflow lite model for doing image classification on a mobile device before we get started make sure to subscribe and give this video a like we'll be picking up from our previous tutorial where we coded our own convolutional neural network for classifying fruit i provided a full python code in the video description or you're welcome to use a different neural network that you've already coded because the steps that we'll be taking are the same regardless the most important thing is that you want to have an object or variable for your neural network that you've already trained and in my case i have model which has an accuracy of about 94 our first step is to convert our neural network variable into a tensorflow lite model so we'll do this by using this function and pass in our model variable now to get the tensorflow lite model in that specific format we can use the convert function on our converter variable now that we have this variable over here our model in a tensorflow lite format all that we have to do to save it is in python to write it to a file i'll call it model.tflight wb as f and now we can say f dot write tflight model so to write our saved tensorflow lite model into this file called model.tflight we can go ahead and run this cell in google collab jupyter notebook whatever software you're using and you can see that's complete so if we head on over to the file section on the left of collab you can see that now we have a file called model.tflight i'm going to go ahead click on these three dots and we can download our tensorflow lite model so what will happen now is our next step the second step is to take this model and deploy it into android studio so we can run it on an android device inside my downloads folder i have the neural network that we coded in python represented as a tensorflow lite model file so now what we can do to add this into our android studio project is head on over to android studio right click on the app and the top left hover over new and we're going to go all the way down to other over here and underneath other at the bottom there's a option for tensorflow lite model and this is what we want to add so i'll click on this now we have to provide the model location i'm going to click on the file location over here this is the model that i want to select model.tflight so i click ok and i'm going to press finish over here you can see the android studio is actually providing us with some sample code for how to use our model for classification which we will get to in one second i want to briefly go over the layout for our android app it's very basic there's just an image view a text view and two buttons for taking a picture from the camera or selecting a picture from the gallery i've provided a link for this xml file in the video description you can save yourself some time check out the link now if you take a look at the left side of android studio underneath the app folder we have a folder called lml which android studio automatically added for us when we imported our tensorflow lite model and you can see here it is located instead of my main activity.java i have some very basic code to get us started i have two variables for the buttons one for the image view and one for the text view and you can go ahead and copy down this code now what we're going to do is set on click listeners for our two buttons so camera that's set on click listener and basically when the camera button is pressed all we want to do is just launch the camera but first we have to check whether the permission has been granted so i'll say if check self permission manifest permission dot camera equals package manager dot permission granted then we can launch our camera intent so i'll say intent camera intent equals new intent media store action if you want to understand what this does in more detail i have a tutorial covering how to capture images from the camera on an android device and display it on the app so now that we have our intent we can do act start activity for in for result and then pass in our camera intent otherwise if the permission has not been granted then we have to request permissions so request permissions i'm going to pass in a string array of the permissions we want which is only the permission to take to access the camera i'm going to copy the cameras on click listener and use it for my gallery it's on click listener that way i don't have to retype a lot of the things i'm going to delete the if else statement i do not need that and inside of my onclick method for the galleries clicker a click listener i am going to use instead of this intent this type of intent over here i'm going to modify this to be a new intent intent dot action pic media store dot images dot media dot external content uri one more thing that we have to do regarding taking a picture from the camera is if you head on over to android manifesto xml you have to make sure to specify this over here user's permission android permission there are two more steps that we have to do the first step is actually receiving the image when the camera or gallery are launched and the second step is then passing that image along to our model that way our tensorflow lite model that we have over here can be used for classifying the image as a banana apple or orange or whatever you've trained your model to classify so we're going to override the on activity result method which looks like this and what i'm going to do is i'm going to check if the request code i mean if the result code equals result okay then i'm going to after i have to break it down into two cases the first case is where they took a picture from the camera so i'm going to check if request code equals 3 because 3 is the request code that we used for the camera clicker then what i'm going to do is get the image as a bitmap and another thing is a way to consider the fact that our neural network has been trained to take in square images and do classification on square images which is why we're going to have to resize our bitmap and to do that i can use this bit of code over here basically we get the dimension we get the smallest dimension of our image from the width and height and then we use thumbnail utils.extract thumbnail by passing in our image as well as the dimensions and this will basically rescale our image to fit these dimensions over here and make it a square and now we can say imageview which is the imageview where we want to display the picture that the user took that set image bitmap and then pass in our image variable now what we're going to do is in our case if we go to model.tflight you can see that our model takes in images that are 32 by 32 pixels so we're going to have to resize this image that we took from the camera and i'm going to go ahead and create a global variable called image size that way if your neural network trained on images of size 64 by 64 you can just replace this one variable over here instead of having to repeatedly type 64 all throughout your code image equals bitmap dot create scaled bitmap pass in our bitmap and now pass in the new image size so the width is going to be image size and the height is also going to be image size this prepares our bitmap image to be used for classification from our model eventually down the line we're going to create a function called classify image that will take in an image bitmap and do the classification process the second thing that we have to do before we get there is handling the case where the user wanted to take a picture from the gallery so we're done with handling the case where they want to take a picture from the camera let's move on to the gallery so i'll say else now to handle this we're going to have to create a yuri variable and we'll say bitmap image equals media store dot images the media dot get bitmap this dot get content resolver and then pass in our yuri and we had to surround us with try catch now that we have this bitmap image we can go ahead and save image view dot set image bitmap pass in this um this image variable that we were displaying the picture that the user selected from the gallery and just as we did over here with our camera code we're going to have to create a scaled bitmap of image size by image size in our case 32 by 32 and then we can also call the classify image function so this is the code that we have so far once again this will handle taking the picture from the camera and getting that and resizing it this handle is getting the picture from the gallery resizing it so now we have 32 by 32 images in this variable here and we're ready to write our classify image function which will take in a bitmap image feed it to our neural network and then get the results we can display it in the app and it's going to be a void function if we head on over to model.tflight you can see they provided us with some starter code to do the inferencing with our model so let's copy that we will have to make modifications to it i'll paste it inside of my on classify image function and we're going to replace this context over here with get application context and you can see that we need to create inputs to feed into our model so you can see that what the model is going to be processing is this variable called input feature 0 and we need to load this byte buffer into input feature 0. now the byte buffer is what's going to contain the pixel values from our bitmap so that means we're actually going to create this bytebuffer variable before we can start doing the inferencing so that's why i'm going to create a bytebuffer variable called bytebuffer it's going to be equal to bytebuffer allocate direct and now we have to specify how large this byte buffer should be so the size of the byte buffer is going to be 4 because that's the number of bytes that a float takes up times our image size because this is how many pixels we have and then times three because technically each pixel has an r g and b value so this is going to be the size of an image in terms of the number of bytes and we're allocating that size to our byte buffer now we can say bytebuffer.order now we're going to iterate over all the pixels in our bitmap so i'm going to have an array of those pixel values image size times image size and say image.getpixels i'll pass in this into values array so this is what's going to contain the pixel values and now i'll have a for loop to go over each of the pixels i'll have a variable outside of the loop to keep track of the pixel number that we're on now to get the value of the pixels the interval equals into values of pixel and then i'll increment pixel and then it's important to note that this value over here is the r g and b values all rolled into one so what we want to do is extract those rg and b values individually and get those those numbers on a scale from 0 to 255 for each of these and then add them individually to our byte buffer and this code over here that i copied and pasted will handle that essentially it's using bitwise shift operators to handle extracting those values and something that's important to note over here is this division so in our neural network that we coded in the previous tutorial our neural network has this pre-processing layer where it takes in all of these pixels which range from 0 to 255 and it divides them it rescales them to be from 0 to 1. so it handles the pre-processing for us our neural network already handles that so that's why i'm dividing by one over here however if your neural network is different and it does not do that pre-processing and it's supposed to receive inputs from zero to one then you have to rescale you have to divide these pixels by 255 to get it to be within a range from zero to one in our case that's already handled for us because of that uh rescaling layer that we have in our model so we don't need to do that and now all of this code over here will handle taking our bitmap and providing all that data the pixel data into our byte buffer which we're now loading into this input feature zero variable and we're ready to run the inferencing that's what's happening over here so now the next step is to actually get the output from our model and then display that in the app now that her model has processed our input we're going to get the confidences for our model for each of the classes apple orange and banana as an array using this output feature zero variable and the idea is that the position with the highest confidence or the highest value is what our model thought that that image was of so for example if position 0 has the highest number in this array that means our model thought the image was an apple if we thought if the last position has the highest value our model thought that the orange was the correct image so this basic code over here that i copied and pasted will handle finding the highest position for us it's a very basic algorithm just go through the confidences array and find that position max pause with the highest value and now that we have this position with the highest value we can display this in our text view so i'm going to create a string array of classes that way it's organized and over here i'll type in the classes that our model trained to classify which is apple banana and orange and the idea is i can say classes of max pause to get the name of the fruit that our model thought the images of and i can display this in our textview by saying result that's set text and then i'll say classes of max pause to display the class with the highest confidence and this is these are all the steps that we are required to process the input from our bitmap image put that into a byte buffer run the inferencing get the position with the highest confidence and then display it so let's go ahead and run our app and see how it looks and performs now the app has launched and i'm running it on my mobile device i'll click on launch gallery and then select an image of an orange now you can see that the app has classified that image as an orange which is correct i'll go ahead again click on the button and select an apple image of an apple this time and you can see that it also classified that as an apple lastly i'll take a picture of a banana that i have using the take picture button and you can see that the app has classified this image successfully as a banana and this is really cool because we coded this neural network in python on our laptop and we took this custom neural network that we coded and are now running it on a physical mobile device that you can carry in your pocket so that's really cool i hope you learned a lot about how to code your own neural network in python and then use the machine learning model on an android device please make sure to give the video a like subscribe and share the channel with friends all the quote for these tutorials is in the video description leave a comment below for tutorial ideas or any questions that you may have as always happy developing from ig apps and i'll see you in the next tutorial

Channel: IJ Apps

Views: 86,773

Keywords: IJ Apps, android, machine learning, tensorflow, tensorflow lite, tensorflow android, image classification, tensorflow image classification, android image classification, image classification android, android tensorflow lite, tflite, tflite tutorial, custom TensorFlow model, computer vision, tensorflow convolutional neural network, convolutional neural network, neural network, deploy, deploy tensorflow, deploy tensorflow model, deploy tflite, deploy tflite android, tflite android

Id: yV9nrRIC_R0

Length: 16min 0sec (960 seconds)

Published: Mon Feb 28 2022