I built a Math Solver App using Google's Gemini Model with Flutter

in this video we'll see how we can Leverage The Power of Google's gemin model to make an app that solves math equation and also shows a step-by-step explanation so I know I'm quite late for making a video on gemman but I wanted to make a real world application and hence that took a bit of time anyways let's get to the workflow that we are going to follow to make the app that uses Germany through its API so first and foremost we are going to use flutter for this app because making app in it is relatively easy the app will use camera to get an image of equation and then we'll send that image to Jin via API once we get the response from it we'll put the response back in the text widget that's it that's the whole flow we're not going to set up flutter because some great videos already out there it's pretty easy to set up or you can continue if you know any other STK or you can make a website for it it's up to you since we are only using API calls the concepts will be pretty much the same it's just UI that will be different and it should be according to your use case so without getting further Ado let's get started I'm using vs code and hit Control Plus shift plus P to bring up this window type letter here and select a new project then choose empty application name your application and it will start preparing default code and layout for a basic app once it finishes you'll get some code in a file name main. you can use flutter doctor command to see if you don't have any issues in flutter or in its dependency just for sanity once it gets screen we can simply launch the app using flutter run command I've connected my Android device and selected from here hence the app will show in this device once grle finishes its task when app launches you'll see a basic UI that says hello world in a text widget now we are ready to make our own UI start off by adding dependencies like image picker to get an image from camera image Cropper for croing an image and HTTP for send API request to the server as soon as you save the changes letter will automatically download and install dependencies now get back to code and remove this predefined layout we'll have our own custom homepage a custom title and choose your theme according to your color choice now make a class with the name my homepage that inherit properties of stateful widget so the difference between stateless widget and stateful widget is stateless widget is static while the stateful widget is dynamic it means the dynamic changes in variable gets updated into stateful widget but not possible in stateless widget so we'll need to update image placeholder and response from API in text placeholder dynamically so that's why we are using stateful widget now this showing error if you hover over it then hit Quick Fix then you'll see a bunch of options and it's missing default override function so we can click on that to implement that function so basically this creates a mutable state for a widget instead of directly using this function we'll create a private class which inherit properties of it again hover over it and use Quick Fix to get the required build function where we will be defining our UI now use scap fold which provides visual layout and in that we'll Define app part which is this part of our UI and place app title instead of text widget now Define what should be the body of the layout for now we'll be placing a text that says hello flutter and in the bottom of the layout we'll be placing a floating action button which does nothing on pressing it a tool tip which you can see on Long pressing the button and place a camera icon on it now that we have basic layout ready hit shift plus r to reload the app and we got an error it says that theme data can't be used with const keyword here so just remove it I think I mistakenly place it there you can place more const keyboard to remove some warnings but honestly that doesn't matter for what we are doing and now hit shift R to H reload again and see we have an app bar with a title of our app name a text field that's saying hello flutter and an action button with a camera icon and a tool TI let's move the layout to our use case and change the text from Hello PL to no image is selected and hit hard reload again and see it reflected in UI now we want the the button to open camera on clicking it so let's define a function that opens up camera we'll be using image Pier dependency for that and after initialization use picker to pick the image by selecting source as camera if image picked successfully refer it to Global variable to use it from outside of the function so where we will use it in a image placeholder so let's go and place it in an image widget remove this text widget from here and use stack which places widget on top of one another use the condition to show the text when no image is picked from camera and if yes then place the captured image in the image widget and inside this condition put this line inser State function which will update and reflect the image in image VIs it once it's updated Now call a function to open up camera from Floating Action button Define the function and put the condition that if no image is selected then get image from camera now avoid heart reload this time and close and rebuild app using clutter on command it will take a bit of time again and when it's done use camera button to open camera capture an image and see it's reflected on image widget and also the text which says no image is selected is gone now let's see how the captured image will look like if that image is of our use case which is a set of equations so here are some equations and we'll capture an image of it and see we have a problem that there are several equation in a single image and we need to solve only one so we need to filter out our crop image to only one equation now we'll use image Cropper here so the raw image which was going directly to image widget will be go to image Cropper first you'll crop image there and then you'll be sending cropped image to image widget use captured image as Source I don't remember all aspect ratio so referring to the documentation for doing some copy paste and here is an important part that be missed this needs to go inside Android manifest file in order to get image crer working use copy button to copy this content and in your source code go to Android app Source expand Main and open Android manifest.xml file and place just where the default activity ends and if you're using an iPhone congratulations you need not to do any additional configuration like V Android uses to I scroll down a bit and in this example you can see that preset of expect ratio is there now we can do some hard work select all these and hit contr C to copy and paste it inside expert ratio list and for iOS setting copy the respected line and paste it here quick tip you can always use cons keyword if you don't want this annoying warning line now it's done and now we need to set up the crop image for image reference if crop image is not null of course and hit hot reload and again error so it says that cropped image is an icing function that returns future object so we need to put AWA keyword here and that's it and since we changed Android manifest H reload will not work here so close the application and relaunch it again using flutter run command once the application is launched you can go ahead and capture an image and now you can see the image is not going directly to the image widget instead it is going to the Cropper now you have the option to crop the image crop it according to your preference and see cropped image is reflected on image vet second flow is ready which is capturing the image and cropping it now we just need to send this image on J Min over the API but before that let's place the icon over a condition that if image is already taken then the button icon should change to send icon otherwise it will be a camera icon and see it fixed do the same with the tool tip as well as it should also change with the state of layout now let's set up the most important part of this video the J mini API go to ai. gooogle dodev and click on get API key this will open up Google AI Studio talking about the interface you have option to create a new API key or get an existing API key from left pane on main page you have text placeholder to WR prompt and in the right pan you have options to choose model temperature and several other parameters so we are going to solve an equation from an image so we'll need the model from gini Pro to gini Pro Vision which is made for dealing with images and as soon as you select you can see that there is an example image and prompt here hit tab to get the image and text from hint remove one image and write the desired prompt which we are going to use in our image if you hit get code here you'll see API payloads and how to make the API calls in different languages but there are no example for Dart or flutter here so we can get this Json payload from this SK Tab and as it almost the same for every other language and use it in our flutter code so this is our region of Interest now prepare the code for HTTP request make a function named Send image which takes image file and put a condition just for shity that the image file is not null encode it to base 64 encoding so that we can transfer the file in a Json format and Define a variable for API key now go ahead and copy the payload and paste it temporarily in our IDE now we have to make API key go to get API option and I've already created an API key but if you haven't done yet just click on create API key and it will simply create a new one for you once it's generated you can click on it and copy now paste it here once this part is done Define request body to encode the payload in Json format which we have copied earlier and change the data to our base 64 encoded image now use HTTP to send post request but before that you need to import http use HTTP dopost function to send the payload on URL you can get the URL from where you got the payload copy it from here and paste it in url parer replace this placeholder from API variable define header this two you'll get from the same box copy and paste it here and don't forget to alter it in a dictionary so that API knows that the payload type is in Json format in body pass the request body which is our payload and this is enough to send HTP request to API server for sanity print message on Console if request is successful and a message if failed or got an error and that's it for sending request now go to action button and put a condition here that if image is captured then send the image else open camera to get an image now hit control on app to stop app and using flutter run command to build it again once app installed open up equation page and capture an image of it crop it to single equation and hit send button to send this image to Gemini and it failed after a little bit of debugging I find the culprit I copied ch Pro URL instead of jini Pro Vision And since jini Pro doesn't deal with images the request failed go to jini UI again and this time select J provision and click on get code now copy the URL and replace it with our current Pro URL rebuilding the app again but here you can do H reload that will work capture the image again crop and send it will take a few seconds and see we got the response in console this response contains solution of the equation we have sent but it's also in Json and if we have to get the text we need to go through candidates content and part one by one but before that we need to put a text placeholder where we can show the response that we are getting from from jini so inside stack I'll remove this line and put a single child scroll view so if our response is getting too large we can actually scroll up to the bottom and see inside this Define a column to place image and text wiget in vertical sequence and place our earlier condition here that if image is there then it should be an image widget otherwise the text widget that says no image is selected use sized box to give a margin between widget and then put a container which contains our text widget and give it padding from all the sides and then the child of this container will be our text widget which actually contains response which we are going to get from Jin now we need to Define this variable and update it every time we get response so here set response body to content that we are getting from gini and Define the variable above we know that we are getting response in form of Json from Gemini so we need to decode it to access values from Keys the text Valu is inside candidate then inside content then inside Parts with the key name text I mistakenly placed response body inside function remove and put it back outside to access from within the class defined flag name is sending and set it to false which we will use to place a loading wire between making request and getting response set it to true and false inside set State now get back to layout code and before setting up loading bar add style to text to get the desired font size once you're done with that place a condition that if is sending is true then add a circular progress bar in layout and choose color of it accordingly Now set is sending flag to true on calling Send image function and set it to false if you get response that will remove progress bar once response is received now we are ready to launch hit shift plus r to H reload pick up your phone open camera capture image of equations page crop an equation and hit send wait for Progress bar to disappear and see you have an answer it looks good but we have a bit of problem like we have hardcoded prompt so if we wanted to solve questions of differentiation or integration we need to programmatically change prompt so let's get a step further and add a editable text Tex field where you can decide what you wanted to do with the image Define a variable in which we'll store text from edit text field and put a condition here to proceed with default prompt if there are no custom prompt now make editable text field put a little margin using sized box and Define text field with controller Define controller above and change value of custom prompt with whatever entered in text field you can print content of text field in console if you want now rerun the app again oops I forgot to initialize this now hit H reload again once it runs get some other equation like derivative questions open camera capture image of it and and choose an equation I'm cropping this write a prompt like I wanted JY to solve it and return a step by-step explanation once done hit send button and wait for response and see it solved the equation while explaining every step so this is one among infinite number of use case that can be built on top of large language models like Germany this is bit of large work through than usual so if if you have any question please drop them in comments suggest some topic that you wanted to see watch Quick explain playlist if you wanted to understand things on conceptual level with ease and as always thanks for watching

Channel: Developers Hutt

Views: 3,855

Keywords: gemini, google, machine learning, deep learning

Id: 1-ksrKDrdsI

Length: 20min 9sec (1209 seconds)

Published: Thu Jan 18 2024