How to get Text from Audio in Flutterflow - Using AI

hi my name is Leo meleon and welcome to this video I got a request in my previous video's comment section where someone wanted to make a voice note taking app and they wanted me to show you them how to actually use the whisper uh model from open AI to actually kind of transfer audio Yeah just audio recordings into text so in this video I will show you how to do this in flut flow so how to um either in flut flow upload a audio file or record cord if you're building a mobile app and then transferring that into text that you can store in the fire store super base or wherever or use in some way in flat flow so let's get started as you can see on my screen I'm currently in My Demo project so this is kind of a project I use for a YouTube tutorial is not something that I'm releasing so you can see it looks really bad and I'm just using it for kind of concept overuse or for a tutorial like this so just the only thing you need kind of before this is just an open AI account with a kind of an API key where you can create it after this video if you want to and also a flut flow project that set up the Firebase or you don't need Firebase but in this video I will show you how to actually use the text that's generated from a file and stored in some fire store document so the first thing we need to do is just create some area where we can use some buttons because I will create so we have two buttons and you will see later why I need two buttons because since I'm building on the web here and I can't really record because you can't request microphone permissions on the web in flat flow that's just a restriction they have you can Google it if you want to just request permissions and you will see that H microphone permissions are not um requestable on web but I will show you on the web how to upload an audio file and how to pass that to the AI so the first thing we need to do is actually just to set up the API call for kind of sending the audio file and retrieving the text that was kind of caught from the audio file if that makes sense so the first thing is just go into your favorite browser and yes you go into your open AI API reference so it's loging and then you just press here up here to API reference after doing that the only thing you need to do is just scroll down here to end points and of course we're going to use audio since that's what's related to what we are doing then create speech is if you're trying to turn text into audio which could be useful but in this tutorial we want to transfer audio into text which we'll use here create transcription H which uses the model whisper one so you can see here on the example so here from this documentation we will now transfer this into H your flter flow account so by the way I've already created an an API key so you can see that I'm stored in the per best not taking up ever and the Chrome browser and then just yeah you can store your API whatever you want but I just going to store it here for now just for this tutor tutal um but basically now we can just read here and set up the API call in the FL flow so first you just see oh it's a post request and we can copy this URL here so let's just do that so now we can go into flutter flow go to API calls press add H create the API call so it was a post request pting in the URL and the name could be whatever you want so maybe get text from audio or something simple perfect now we add some headers here so then we just go into documentation again see that we have one header of authorization beer and then here the open key so we can just copy this we can also note that we will have a content type multiart so that's important later when we create the body because we won't use a Json anymore and we can see here that instead of usually on other AP calls with open AI we use content type and application Json but you can see that we now use multiart and this is a great thing to use in h in FL flow when you want to pass an uploaded file which is the code and invites H but you will see what I mean later now if we go to flood flow we can just copy in it we can just use here API key and because we don't want to kind of hardcode our API key here and by the way just I got the request also in the previous video that they was a bit uh worried about the security for using kind of a variable than storing the API key here and a variable here and that's not it's relative safe but usually for all API calls that I use with API keys or any secret tokens I just go here to advanc settings then just make PR private API call so the API call is rooted through a cloud function on the M on the server side so you don't have to expose the call on the front end so that was just a kind of Sidetrack to yeah and talk about security but back to the main thing so here we can see authorization be API key and to use that API key variable we need to create an API key variable with the same spelling so here that will be a string and the default value will just be the API key that I stored here so I just copy this uh did I copy it no okay that wasn't too good uh can we get back perfect sorry my bad there perfect H then we just copy in here add API call let's now add other header so the other header was uh content type multiart so let's just copy this guy over here and then we will look into these two a parameters real real soon so just going to copy this in here as well oh uh there okay perfect so now we done kind of the main part now we're going to look on what parameters we need to send to the AI in order to actually get the text from an audio file so if you go in here to the API key or the API document mation you can see that we got two parameters the one is a file and it looks like a path but this is not a um a URL to an audio file that's s the internet this is actually the um audio object for an uploaded file and this is great because this can be represented by an uploaded file in flutter flow so we don't really have to do any kind of custom code with u kind of transferring one flut flow audio object to this a the object the flutter flow of the object is already adapted to fit this so you will see that this is super easy to set up um later in the FL flow ux so the model we can set up that so the model is a parameter you can see here on the body here that we can select Json but since we have here on the header we have multiart form data for the body instead of J and we're going to select multiart then add parameter then the Model H will just be specific value string and I thought it was whisper one or something I think that's right spell it's going to double check H whisper one perfect so the next thing here is file but instead of copying a value we'll just write in file and we want to be able since we don't want to send the same audio file every time we will actually set up so the file will be sent to this API call as a variable from flutter flow so if we go in back here again we can set up a new that's called file and if we go to variables we can set up a new variable it's called a file and for the type here now it's important you can see that we have uploaded file here at the bottom and an uploaded file is just the same type that that um all the object that the API um expecting so just press that uploaded file then press save oh we haven't paed here of course sorry my bad from variable file save close this save and since it can be hard to test this kind of API call since we don't have we can't really write a file here since uploaded file is more of an object that we can H create via flood X I will not test this API call but you will see um that it's super easy to actually get working so after doing this now we can actually start focusing on the front end or how it will look like for the user when uploading a file or recording so in this your column or whatever you're doing this I'm just doing this this ugly container here we can just add first a button so I will first now show you how we can actually upload a file from your from your what's called uh file explorer so a file you already recorded because this is the only way to upload on audio in uh on the web because as I mentioned the request for microphone is not available I can actually show you how that looks like so if we just uh if you go here in here to actions and we say for example request so request where is it permissions and we want the microphone you can see up here that we get a warning that says um platform support platform permissions actions by button is not support on the web and you can also search just the request permissions and go into flutter flow documentation and you will see that they do not support web for this so we can't record a voice directly in the via the web H but what we can do is actually just upload the file directly so we just write um where is it upload H data upload file we will upload a audm 3 and we will do a local upload so we won't upload to Firebase because then we don't get that audio um or file object that we need to send to the API and then we can just name it whatever perfect now we can actually just do this now we can make the API call so API API call group or get TT from audio now we can set the variable so API key we already a default value so we don't need to set that but for the file here now we can actually set the uploaded local file which is the file we just uploaded here in the action above so that's it actually that's all we need and now H we can actually do this because currently we get the API response back we haven't defined yet um what variable or where in the response object the text or the transcription is so we can just go back into the documentation look on on the response oh it's just text so that was really simple structure so now to actually kind of predefine a variable you just go back in here to the documentation press on uh where is it response and test Jon path for the response and then we just dot Tech uh do text and name uh text is good anything yep that's good enough now we can just go in here and then now here we can if the API call succeed which is hopefully does then we can use the T so for example we could do um we could maybe first we could maybe alert the user about the text that was in there just for for Simplicity so you can say API so the API action which API yeah API six here then we just do on the yeah JS somebody predefined path and then predefined path is just a text variable that we just defined that's a predefined path in the API call now just press confirm so actually I will just kind of show you the alert information dialogue and how to kind of show text but if you want to if you want to kind of um show the text on where because this is has so many applications I don't really know kind of exactly what to show you but for example one example is that if you want to kind of display the text from the audio in here then you could maybe create a um page State variable here that's called um transcript or something or a list of that's called um audio notes and that's a list of strings and then inside of a column so you can create another column here and in there you can just Loop through each um like each note in that text and each time you upload a new file you can basically just go in here and say that you will update the State Management so page date and then you could update that text leaks or text whatever and then just use that um text that we used in this dialogue and then you can just manipulate text whatever you want it's very basic but it could take ups on time of this video and I don't want to make it too long because I need to go over the uh the mobile side as well so let's leave that for now and if you want to use the text in Firebase then you can just basically do where is it back end fire store H create document update document I don't want know what you want to do but for example if you want to um create a document for a note that you want to store in there for example for an audio note taking app you could have a h let's say that you have a collection that's called notes and that's a subcollection of the user maybe and in here you could just have maybe H text or something that that's maybe all that you need uh string uh create so in here after uploading the file and after getting the text you can actually just do this you can then just do a backend fire store create document you you want to create a new H note user is just authenticated user and then you can just say that the text for this document could be the text of the transcript we got back from the AI so then we can say API again H select the right one deer Json body uh predefined and text and that's it but you can use this variable this variable that I find here you can use it however you want and upload a several St Firebase or means some page date you can do whatever so that's it so let's now just um test this in test mode and then after go over to mobile okay the test mode just loaded so let's see if this actually works so let's just press upload file and then I will just select This Record Online voice that I recorded yesterday or something that was I just said something like test one to three let's just press open then let's see if we actually this is a test one to three great so we can see that we actually transferred our voice into text and to actually show you that this was real let's record a new audio so let's just say for example voice recorder I usually just do this online voice recorder record a new one hello this is Leo and I'm recording a video right now and that hopefully gets um some engagement and people like it okay perfect and just save this guy let's now test go over here and upload a new file press this new online recording press press this record a new one hello this is Leo I'm recording a video right now that's hopefully gets some engagement and people like it okay pretty good actually quite funny yeah so you can see that it actually worked now I did a live test so it was more um authenticated or reliable so you don't think I'm lying uh okay yeah let's now go over to mobile and see how we can set out thatt up so most of you when creating an app like this you wouldn't like the user to just upload a file would like to record um some audio so let's actually do that so first we need to set up some um a Boolean State variable that will be like recording so recording which will be a Boolean uh we have that Boolean it will be false perfect and in here we can just say that or wait just uh we don't set to recording or actually let's do that so we can at first in this column just add some text that says like uh recording and that will only be shown so recording uh one two three or something like that that will also margin 20 and then under here we can just have a row of two buttons like start recording stop recording or actually let's just have one button that's uh or no no no sorry yeah sorry for being indecisive okay so let's now add the two buttons so one for starting there recording so here we could just have I don't want to apply too much styl you do start and then and then stop you might want to do some more styling than I'm doing but I'm just car hurrying this up so stop and by the way I won't be able to show you how to do this and the same thei call and everything is all the same it's just the recording or getting the audio file that's a bit different since uh this is for mobile when you can actually record it so here we should do maybe that start so we can say here that if so if not recording H then we can just basically do this we can start audio recording that's named yeah this is good enough we can just copy this conditional actions go into stop oh we also we also need to change the Boolean of course so we also need to change this H State page state to be H recording to true set value to true and then for stop here we can actually asset that we set some that if recording because we don't want to stop the recording or try to um stop it if it's not recording um because that might cause some errors so here instead of start now we can actually say stop or start I mean stop yeah this sound pretty similar uh where is it stop a the recording and stop the recorder so here and now here the action output variable is the audio path for the audio um yeah just audio path that you want to use but we won't use that you will see what I mean because the we will get two variables and from this stop a recording that we use and I was close to being confused by the first variable because you can't actually use that with the API call but then if you go into the widget State you will see another variable that you get from this that you can actually use with the API call so let's just call this m audio note perfect and then we can change the State variable to be stop or not recording anymore so then we're going to say recording uh set false perfect here we can actually now make the API call so now we can actually say for example API API call get texture modu variable and that will be the file so now you will see that we we search for audio note you will see that we get an audio path this is not compatible with that uploaded file a object but instead you can scroll down here to widget State and you will see recorded file and this is the recorded file in that file or audio file object that we need that we can s API and and this is the same as the audio note audio file but in the right format so you can just press that one and and that's basically it so in here you can do the same thing as you did before with Firebase page date H whatever just you can just handle the transcript text on however you want and create some note if you want to but I I won't be able to test this since I'm developing or using this app on the web and I haven't set up any mobile app deployment on this and uh yeah all my testing is done on the web so I won't be able to show you this request and but by the way something important that I forgot to mention before doing this start um or by the way and instead of requesting microphone permissions before every recording you can just do up here on the load page you can just say that P on page load you can do uh re quest uh permissions and then request microphone permissions and then you can actually um have Boolean on the page dat that's called um microone oh okay the perfect this is from yesterday when I tested this out so the microphone you can see you can just create some Boolean page that's called microphone and then after requesting the permissions for the microphone you can just set this um page date wait where is it page date microphone to be the value of whatever the user decided to do because if the the user denies the microphone Mission we don't want them to be able to record since that won't be possible so with then we can actually set this to be microphone permissions microphone so this Boolean will be true if they um accepted and they will be false if they denied so we can store that in this page date and then we can use this page how we want for example we could hide this row if the if the micro micro if the microphone page date is false now just yeah perfect so this is kind of what I wanted to go over how you can actually start recording on mobile how you can actually use the API call for whisper one get the transcript and actually how to overall use this and so I hope this video was helpful and you liked it and if you like my videos you can like And subscribe I guess I don't know what ask but yeah have a great day and I hope you learn something bye

Channel: Leo McMillion

Views: 391

Keywords:

Id: gS1kbbpZkD0

Length: 21min 33sec (1293 seconds)

Published: Fri Mar 15 2024