How to Build a Multilingual AI Voice Assistant in FlutterFlow (OpenAI Text-To-Speech App Tutorial)

in this video we're going to be building a multilingual AI bot completely in flutter flow with the help of open ai's API this is the bot as a website or a web deploy and let's go ahead and give it a try so we'll Click the microphone button and we'll record a quick response what is 2 * 2 2 * 2 = 4 right so we can ask the bot anything we want we can also select the language down below here and once we ask the AI assistant a question it's going to return a response and also showcase a waveform as well we can build this in around 30 minutes in a flutter flow project completely from scratch again just using one page in flutter flow and some API calls uh via open AI so let's actually jump into flutter flow and build this from scratch here you can see I've already started a new project so we have the create project dialogue here's the AI bot uh we also have a package name so I'm going to leave this as is we have the theme that we could select again I'm going to leave this as the default and then I'm also going to toggle on enable web since we're going to be building a web deploy here make sure to leave set up Firebase toggled on and then we'll click next step at this point you could either create a new project just by clicking this button signing into your Google account and flutter flow will create the project on your behalf in Firebase again this project is going to be completely free however I've already set up the project so I'm just going to paste the project Firebase ID here okay so I pasted the key here and I'm going to click connect and there we go it successfully connected the project to a Firebase project and I'm going to click autogenerate configuration files everything looks great here so I'll click generate files as soon as these configuration files are finished we are going to continue the setup here all right so it successfully compl completed the configuration now let's just continue on we'll leave enable authentication off as well as the create user collection we're not going to have any off in this project so anyone can use it so let's click Start building now to match the site that we created we want to start by adjusting some of the colors and fonts uh to make it a bit darker right right now we have the light mode theme and by default flutter flow has both a light mode and dark mode toggled on so let's just disable the dark mode and we will build a a dark theme only so we'll go ahead and jump over to the theme settings and go to colors and we will toggle off the dark mode theme now for the primary background let's add a primary background so we'll add a darker gray here so this looks good and then I'm going to copy this and go to the secondary background and paste it in here and choose something just a little bit lighter we might come back and adjust that we'll see uh now we need to make sure the primary text is white so it stands out against that dark background and the secondary text we will choose white and maybe just make it 50% opacity so now we have the basic colors set up for our project we're going to leave the primary secondary tertiary and alternative or alternate colors as well as the accent colors for now and semantic colors and for the sake of this tutorial we'll just stick with the outfit font as well so let's go back to our page selector here and we can see this blank page that we have we're going to delete the default app bar cuz we also don't need that and at the scaffold level we're going to turn off safe area so now we just have a column within the widget tree and that's all now we need to add the center container right that we have within the page and what we'll do is actually adjust this a bit so we can add a bottom bar as well as the the language selector too to match that original project we worked on in the column we're actually going to wrap this in a stack so I pressed command B on Mac and I can click stack and now we have a stack created so we'll set the width and the height as infinite here so it fills the space and to start let's add that container at the bottom of the page just to sit it and sit there at the bottom and complete the page so at the stack level I'm going to click the plus here and add a container using the alignment tool I'm going to click Center it at the bottom and I'm going to make it infinite width with a height of let's do 60 so now we have a bottom kind of section here to our website so there's not just a random centered container it gives the site more structure so now let's add that Center container so we have the column here right it's filling the space but we need to align it center right within the stack so we'll align It Center here and then we'll set the main axis alignment to Center as well and now we can can add a container within this column so we'll click here to add a widget add a container and let's set the width to maybe 400 and the height to 400 as well for now we'll also add a border radius of 24 just to round it out and the color should be set already for us now I'm going to switch to web view right now we're in the the mobile view so I'll click on that uh now you can see we have a web preview of what the project is going to look like so already looking pretty structured and very similar to uh the the site that we saw all right so now we need to add the actual recording button to The Container so we're going to do that uh in a unique way and this is actually pretty common within flutter flow if we need to hide and show various objects with a conditional Builder and a page State we obviously want to note when the user is recording a dialogue and when they're done so to do that we're going to start by adding a page state so we'll click on State Management and click add field and we'll add is recording we'll set this to Boolean and keep it at false and click confirm so now we can go over to our container and we're going to add a column and we're going to Center that and add some padding that way anything we add in here will have some spacing right and then we're going to add a conditional Builder so let me search here I'll click that the conditional Builder allows us to show or hide specific widgets depending on a condition and since we just set up that page State we can head over to the if statement here and select that page state that we created so if is recording is true it's going to show one set of widgets and if it's false it will show another so we'll go to the if statement and let's search for an icon button this icon button since the conditional Builder is is recording a set to true we want this to be the pause button to stop the recording because if the recording is currently happening if is recording is set to true then we want to show the the stop button right because we know that the user is recording and we want them to give them the option to stop so let's go to this icon button let's make it a lot larger let's do 200 let's add a border radius of 100 and let's make the icon size 80 we'll also search for a pause button or a stop button so let's type stop and here are some options we'll just go with stop rounded all right we'll also adjust the fill color here so I'll select the second or the primary background actually so it's a bit darker and we can remove the Border if we want to now we have our stop button there's one other thing we want to toggle on here just from a UI perspective it's the show loading indicator this will be really helpful when we have the the stop button because we're going to take some action after we press stop and we don't want the user to just see a blank screen a static screen so this will show a loading indicator within the button while everything is loading all right so let's click on this uh icon button we're going to press command C to copy it and we're going to click this I icon next to else to show it and you can see we've switched the condition and we can click in here and paste the icon so now we need to create the record button right so we're going to go down here and select a different an icon so I'll search for a microphone and maybe let's go with this outline rounded microphone I think that looks good so now we have our two buttons we have the recording button and the stop recording button we also want to add that waveform right that we showcase while the AI is speaking so to do that we're going to go to the column level and we are going to search for L animation and add that here I'm going to drag Dr that to the top of the column and I'm going to set the width to let's do 400 and the height to 200 and what we'll actually do is click on the column here so the main access size to shrink and then we're going to remove the height on the container and we can also do it on the width as well so it'll be pretty interesting because this container will dynamically shrink to whatever is being shown within it so we can go to implicitly animated toggle this on and set this to ease in out maybe with 300 milliseconds so there's a slight animation whenever the container changes size depending on what's visible now we're not actually going to be creating a live waveform in this video we're just going to use a lot animation in order to get the point across to the user that there is some audio occurring so we'll head over to L animation and actually just find a waveform and upload that so let me do that now okay so I've gone ahead and found a waveform that I like and I've uploaded it here I want to keep on auto animate and loop so it continuously goes and we have everything already set up here so now we have our waveform and we have our recording buttons now we also need to conditionally hide or show the waveform when the AI is speaking so we need to add another page State and we can name this is speaking right so we'll add a Boolean here just like is recording keep it at false and click confirm then we'll click on the L animation go to conditional visibility and we will select is speaking so when the AI is speaking this waveform will show now we need to go to the conditional Builder and have the opposite so these buttons do not appear when they a i is speaking so we'll click conditional and we will go to page State and select the opposite apply opposite statement to is speaking all right so now we have the conditional visibility set up here for the L animation and the conditional Builder we also maybe want to add an animation for the UI so we'll click on the L animation we'll go to animations here and we'll add a fade and toggle on hide before animating so it'll start hidden and in now let's also apply this to the conditional Builder as well just to make the UI look a little better all right so now we have two animations the conditional visibility now we need to actually begin to set up the actions right so let's start with the record button the record button needs to both set is recording to true and we then need to actually record what the user is is saying so let's first add the action to set the page state is recording to true so we'll click on the icon button we'll go to the action flow editor and we will click add action and search for update page State let's select is recording and this will be set to true now let's go ahead and copy this action and head over to the stop button and we'll add the action here by pasting it and set it to false right so we have true and false now we need to actually add the custom action in order to store the words that the user is saying so let's actually head over to the template project to check out what that looks like and we will just be copying over the custom actions from the Clone wall project the link will be in the description okay so here we're in the clonable project you can can see what we're creating looks very similar to this right we've added that bar at the bottom the container we have the centered container right now so we're pretty close if we go to the custom code section we have three custom actions here start text recording stop text recording and then fetch speech and play what we'll start with is the start text recording so this is using one pubd package and it actually listens to the user and whenever we get a result from listening to the user we update a appstate variable which we're going to create named speech to text response with whatever the recognized words are so it will continuously listen until we trigger the stop text recording action which we will trigger from pressing the stop button using the same pubd package so let's go ahead and add these custom actions to our project we'll start with the action name so we'll copy that so let's create a new custom action and paste the action name then let's go to the Top Line 10 and copy all of the custom code from the start text recording that we need we'll head back and we will paste it here then we also need to add that pubs spec dependency so we'll go back go to the pub spec dependency copy that and add it in our project make sure to hit this refresh button so we load the pubs speec uh package into the code editor here all right so we'll save the action and we need to go ahead and create that App State as well the speech to text response which we reference here so I'm actually just going to click on this copy the name of it so we spell it correctly and we'll go to appstate and we will add the App State variable it's going to be a string and it's not a list and we can keep persisted off and we'll click create now we need to add the stop text recording action so same thing we'll click add and create a new action let's go back to the project get the stop text recording name we'll paste it here we will go to the Top Line 10 of the action and copy all the code below and we again we need to add that Pub speec dependency so let's do that right and again hit the refresh button and click save so now we have this start text recording and stop text recording actions we've also added some console logs here to help debug if you're ever interested in in adding more so we have stopping text recording we also have starting text recording here you would be able to find these in the console uh when you're using this in the web deploy or in test mode or in run mode so let's go back to the page then select the widget tree and and let's go first to the start recording button so we'll click on this icon button we'll open the action flow editor and we will add that brand new custom action start text recording now let's go back and go to the stop recording and we'll click on the icon button open the action flow editor and we will add the other custom action all right so now we've started the text recording and successfully stopped the text recording and all of the text information is going to be saved within this speech to text response appstate variable now we actually need to add an API call to call open AI receive or return a response and then convert that response into an audio message so let's add that new API call this is the basic chat completion API call so again we're going to copy this directly from the project you can find all this information Within in open AI documentation this is the basic implementation to just receive any uh chat completion so we'll just copy the API name add a new API API call paste it here copy the API URL and you could also get all of the open AI API calls by uploading the Swagger file we have a great YouTube video about that and it'll instantly import all of open ai's API calls for use so this is a post request now we need to add the the off header so let's copy this paste it in here we need to adjust the body so again I would just copy this body directly from the clonable project however this is a very basic uh Json body again you could take this right from open AI documentation and we'll actually run through the message that we send here so the content right we send the users message The Prompt which we set up but I have this statement here return a response that could be read aloud in a total of 13 to 15 seconds in and then a language code I did this because I wanted the AI assistant to be more conversational I didn't want it to go on a 2 to 3 minute rant or conversation I wanted it to keep its responses relatively short hence keeping this in the content so now we need to create some of the variables right so here we have one variable API key off so we'll go to variable paste that change it to a string then in the body you'll notice we have two variables prompt and language so we'll add the prompt variable set that to a string and we'll add the language variable and set that to a string as well so now if I go back you can see uh it looks like we almost have everything here oh looks like we forgot to set that to a string so if I go here all of these are purple they're properly implemented and we could click add call so now we have the API call now you'll notice we have this variable API key off and that's because we need to provide an open AI API key to do so we're actually going to create a new state variable App State variable so we'll click here and we'll name this API key and we'll keep this as a string and we'll keep this as persisted and click create we'll also toggle on secure persisted Fields so ideally that is encrypted uh within the device so at this point you could go and get your API key and paste into this field I'm going to skip that for now but again make sure to go and get your open API key paste it in here uh so you can actually use the API call we just added all right so let's go back to our page and we need to add more actions to the stop button so we have this button here and we will open it up so after we stop recording we need to pass that information to the API so let's click plus here and add the API call so let's search for API call and we'll select to get response let's name it uh maybe open AI response and we will add the additional variable such as the API key we'll set this to the appstate variable API key The Prompt which is again that app state speech to text response and then the language Now language is actually pretty unique we can click out of here and this is extremely easy to set up all thanks to flutter flow so we'll go to settings and integration and let's go to the languages tab right now we have no languages set in the project so first let's set up a language since I'm an English speaker I will select English and set that as a primary language but let's add a few more maybe Spanish French let's do Arabic and Chinese so now we have a bunch of different languages and we could click translate all although we don't have any text really used within the project but at that point it would translate all the text in the project we have different languages set up and if we go back to our page let's actually go to the stack and add the language selector widget it's going to appear up here let's do Center bottom alignment and maybe add padding of 100 so it sits just right above the bottom here we could also add it within the container here let's add the background color to the secondary color and let's hide the flag and set a text theme Style so now we have the language selector right so we can actually pass in if we open up the API call again we can go to language and pass in whatever the current language code is so the current language code in internationalization so it will pass you know en n if I have English selected or the specific language code for the language I have selected we automatically add the conditional action here to see if the response worked so for debugging let's add a a failed statement so we'll say show snack bar API call failed obviously make sure to set this up properly if you put this into production but for now we're just setting this for debugging purposes oops change this to purple great now let's add the next step right once we get an API response we need to convert that into an audio message and play that that's where this next custom action comes in so let's click out of the action flow editor and jump over to the custom actions again custom code so let's go back to that other project and now we need to add the fetch speech and play custom action what this custom action does is it gets the prompt text return from that API and then it does a post call to open Ai and returns an audio file and then plays that audio file right so so if you wanted to adjust the voice or the speed you could adjust that here again make sure to refer to open AI documentation but the other thing we do is we calculate the duration of the audio file that's returned that's what this section does here so we play the file the audio file but we also return the duration of the audio file in milliseconds and there's a unique reason why we do this it's actually to set up a timer to show and then no one to hide the waveform L file that we've added so that's why we return a value that's an integer that integer is the duration of the audio file in milliseconds so let's copy this over so copy the action let's add a new action here paste the name go to line 10 copy everything below it we will add a return value an integer and we'll add two arguments here the first is prompt text it's a string and the next is API key only API key is knowable although you could toggle this and just adjust the the function if you wanted to I believe I just left that as knowable although it could be adjusted right so we have prompt Tex and we have API key and if I save this action I might get a popup that says my arguments don't match my function that's okay um as long as you copy everything over properly I'll click yes it should work fine so now we have all the custom actions we need the custom apis everything in here so we can successfully set up the rest of our project so we'll click on the button here and now let's go to the true side right so if the API successfully returns a response we now want to go to that custom action fetch speech and play and we need to return the message so in order to fetch that Json response you could either directly type in the Json path to whatever is return from open AI or we could utilize the Fantastic API call UI that we've put together in flutter flow and utilize some of its features with the response and test tab so the response and test tab allows us to test the API call and then name various Json paths as variables so we're going to do that so I'm going to have to paste my open AI key in here and also provide a random prompt so pretend I pasted my my AI key here my API key and I'll add just test as the prompt then I'm going to click test API call now I'm going to cut this and actually paste in my real API key test it and then uh come right back as soon as i' I've tested it okay so I successfully tested the API I've just quickly removed my API key after doing a test here and you can see uh all we need is the content path so let's scroll down let's go to recommended and here's all the options but what we really need is the reply from the AI which is could you please specify the language code since we didn't pass anything here so we'll click selected we'll go over to the selected panel and name this Json path message and click save now if we go back to the action flow editor and makes it a lot easier to actually select the path of the prompt text so here we'll go to action outputs open AI response we'll select the Json body Json path and since we set up a variable Json path we can just select message and then click confirm the API key again needs to be your app state so that will be API key and that's pretty much everything right so let's name this AI voice response okay so we can name that and we need to name this action output because this is the integer or the duration of the audio file in milliseconds so we need to use this right so we need to give this action output variable a name so we'll leave it as this and actually let's let's just name it uh audio file duration that way we we don't mix it up so we have that now we need to set up a few different things we need to first update the page State since as soon as this action is completed it's going to start playing the audio so we need to show the waveform right so we're going to update the page state that we set up is speaking and set that to True okay we also need to store that audio file duration somewhere so let's store that at the App State so we're going to click off here and we're going to go to the appstate and set up timer value and set this to an integer this is where we're going to store the duration of the audio file so we can use it as the timer value so we'll create this and we'll go back and then we will add another action under here add action update App State and we'll update the timer value to be the output from Fetch speech and play which is the audio file duration so if we go here we'll see audio file duration select that and we also are going to go ahead and clear the speech to text response because we've already passed it to the API we've already returned a response so we can clear that for the next time we want to use it so let's clear that value all right so now we also need to set up a timer so that way we can show the audio for the audio wave for form for as long as we need and then hide it right after the duration so to do that I just kind of add a dummy timer within the page so we'll add a timer and let's just add it uh within the stack right and we could center that here and we could put it behind this container we could set the font size to one make it transparent we just want to hide it entirely right we're only using the the timer action specifically so the countdown time we want to set to the timer value appstate variable maybe let's set the format to seconds so it's even smaller it's a small as possible down here we also want to have a conditional logic is speaking to true so we only want the timer to be shown into start when is speaking is set to true now when the timer goes off once the audio duration has completely finished we want to update the page state to say is speaking is false and we also want to reset the timer so we'll reset the timer and reset timer actually needs to be before updating the page State because we need to reset the timer and then when we update the page state is speaking to false it will then hide the timer so we need to reset it before we hide it so we've set up the timer properly now we actually need to start the timer as soon as that audio starts playing so the custom action occurs then we update the page State set is speaking to true the timer appears on the page then let's do a small wait so the timer has time to initialize 100 seconds and then we will do timer start timer and we'll also at this point oh we actually need to put the App State uh maybe let's do that first so we update the App State and set the timer value then we set is speaking to true to show the timer then we give it 100 milliseconds to just load and then we start the timer on the page and that's everything we should need on the pause widget or the stop widget right so let's actually head on over to settings let's click on web publishing let's leave the settings as is and let's click and actually give it a try so I'm going to click publish and wait for it to uh deploy and I'll come right back as soon as it's done publishing all right so the website successfully published let's actually open it up and see if it works so we'll click here open it in a new tab all right so everything looks to work so far let's Click the microphone button and see what happens can you tell me what flutter flow is in two sentences flutter flow is a low code visual development platform that allows developers to build mobile apps quickly using a drag and drop interface it leverages the flutter framework to create high performance natively compiled applications for both IOS and Android okay so it worked now let's try in French let's give it a try what is 2 * 2 equal to the okay so that's a win in my books this is a beautiful web app that we could utilize we could further customize it in flutter flow again this is a multilingual AI Voice Assistant that we built just with a few custom actions and an API call you can also check out the clonable link to clone this project and start immediately uh or feel free to obviously you follow the tutorial so uh adjust specific parts of this tutorial to create a new UI or or whatever ever else again this could be published responsively on iOS Android and web so feel free to share your Creations too with us on on Twitter and across the flutter flow Community forum

Channel: FlutterFlow

Views: 8,194

Keywords: Flutter, FlutterFlow, No Code, No Code Tools, How To use FlutterFlow, Google Flutter, Dart, No Code App, FlutterFlow App Builder, No Code App Builder

Id: 1GmgjkZiDsk

Length: 33min 9sec (1989 seconds)

Published: Wed Nov 08 2023