Open AI Function Calling | Explanation & Demo

in this video I want to explain what function calling is in open ai's API and show you how to use them in flutter flow but before that here's what function calling and flutter flow allows you to do I've got this little app right here where I can put in a YouTube link and it's going to generate a quiz based on that video so let's go grab a video from the flutter YouTube page maybe this one right here so let's paste it in and analyze and create our quiz okay what is the unique identifier for your app in iOS called I think it's called the bundle ID so let's see if it's right and it's right beautiful so function calling is one of the most powerful features of open ai's API but it's a bit confusing because it's poorly named it should be named generate structured output because that's what it does instead of just outputting a string of text in response to your prompt function calling gives you only the information you tell it to and in the structure you tell it to and this solves a very specific problem you see we all normally use chat gbt like this we ask a question and it gives an answer the answer could be short or long it could contain code or just text and this is great but if you want to use chat GPT in an app that is your program is interacting with it not directly a human interacting with it this might not work for instance maybe I give chat G PT a Wikipedia article and I want to create a multiple choice quiz based on it well in order to have a quiz in my app I need the data structured in my backend in a very particular way like I can't just dump in a normal response that I get from chat GPT with function calling you can do this you can tell chat GPT to give you exactly the structure you need for the quiz or maybe I don't need the structured data at the end of the program but maybe somewhere in the middle so let's say I have an app where the user can ask for the weather in natural language and then the app uses that location to call an API to get realtime weather data because of course chat GPT doesn't have access to realtime data in this example I need chat GPT to take the natural language prompt like the prompt what's the temperature right now in Addis and return just the location so I can call the weather API using that location that's the beauty of function calling so let's build it so I'm here in my app and let's talk about the logic that needs to happen so I'm going to dump a link in here then I need to get a transcription and then take that transcription and send it off to chat GPT with function calling so it returns to me the structured data that I then store in my database okay so let's start at the end that database after all this logic what do I actually need well I've got a collection set up right here called questions and it has four Fields very simple an answer which is just a string the question itself which is a string when I created it and finally all the options which is just a list of strings and this is the structure I'm going to need to get out of chat GPT all right so let's move back up to the top of our logic chain the first thing that I'm going to need is I need to get the transcription of the YouTube video and I'm going to do that using build ship build ship is an amazing tool that allows you to build backends quickly easily and Visually and here's my whole back end right here so we start off with a trigger right here that is this is the trigger that starts this flow of logic here and it's simply a post request to a URL I've just appended this quiz path right here and to see the final URL you can click into here and see it right there there next up I've got this node that's using an npm package for extracting captions so if you just click onto one of these plus where you add a node and search for YouTube you can see I just dropped it in right here and this takes two parameters here the URL of the video that you want captions from and the language it defaults to English that's why I don't have any value in here now you can either hardcode values in right here or you can set them dynamically which is what's going on right here because the URL I want to dump in here is the URL that gets passed in when I make this call it's part of the post body and so to see that you can just click in here and when you hover you can see that the full path is the request body. URL so when you make this call the body of your post should have a property called URL and you can access this by coming into variables request body and hit this URL now where is this coming from well that's coming from up here so if I come in here and I go to edit and I go to Output you can see all the values I'm outputting and I added on to the body right here this string of URL because that's what I'm expecting okay so what do we get out of this node right here when it runs well you can see that if you come into this test mode right here so let's dump in a link to our video right here let's test this node and here's what we get we get this object with a subtitles property which is an array of objects with the start duration and the actual text so we need to do a little cleanup here and we'll do that in our return node so we can just cancel out of this and we see we've got this return node now of course in real life you would want to do better debugging and air handling but just for the demo this is fine and when we go into the editor you can see that I'm returning this object with one property called transcriptions I'm referencing that that subtitles output which I accessed from here you can see we have access now to this YouTube caption extractor because we are after it now and I just came in here and you can see that's what we have I'm just mapping over each item and returning the item text property and there's just some cleanup to get rid of some text I don't need and finally joining together as a string okay great and when you're done you can just ship it and it deploys shockingly fast now you can either copy the end Point URL or export the API as an open API spec that you can directly import into flutter flow if you've got a bunch of workflows that you want to import you can import them all at once by going over to settings and apis and selecting whichever API you want to export if you want to export all of them you can do that and then select if you want it as a yaml file or Json file flutter flow accepts both all right let's jump back into flutter flow so let's come over to our API manager if you want to import that yaml or Json file you would do so right here but I've set up that API call here's a URL it's a post and when you look at our body we've got one thing we're expecting is just the URL and I've defined one variable because this is going to be a dynamic value because every time we make this call we're going to be passing in a different YouTube video and so if we come over to response and test and let's paste in a URL and test that call and beautiful we get our transcription finally we're going to want to grab the path to this so we can use it in our app cuz remember after we get this we're going to want to send it over to open AI so if we go into our Json paths here we have one path which is just the transcription we've selected it given it a name transcription and save beautiful that's half of our logic done finally we've got our open Ai call with function calling you can see the documentation here so let's jump over there and take a look in the past few months the syntax of this API has changed so just make sure you're using the most upto-date version for function calling you just use the normal chat completions endpoint and just like a normal call it accepts the messages which contain different users like system and users and their messages we're going to be using both system and user the system is used for any background information or instructions you want to give to chat GPT about how you want it to handle this if you wanted to use a certain style or act as a certain type of person like an assistant and the user message is just the normal message you'd put into the prompt next you specify the model now if you're using function calling make sure you're using a model that supports function calling these are the current ones that support function calling so let's put these things in here first and then come back for our last thing that we need and that's the function calling itself so I'm here in my call it's a post request there's my URL and make sure you have your authorization with your API key and if you don't know where to get that we've made a video where we go over that that I've linked below but the real meat and potatoes here is in our body which is Jason and so you can scroll down and this is what we've covered thus far we've got our model and we're just going to use 3.5 Turbo with a 16k context that's just to make sure we've got enough tokens to cover a 10 or 15 minute video and next we've got a messages which is an array and we're using two objects one is our system instructions and our other is the user for the system instructions I'm just saying Create a quiz with three options based on input transcription from the user output a question as a string options as an array of three strings and an answer that's identical to the string that is an identical string to the correct answer in the options array okay so that's what I'm telling chat GPT behind the scenes and the user it's just going to be the transcription that's what we're dumping in and of course this has to be a variable because it's going to be dynamic every time we run this logic it will be a different transcription so we've defined this over here as a string beautiful okay finally let's do our function calling so there's an example of function calling over here and we're going to scroll down so we can see it and it's changed recently so it used to be under function call and functions but those are deprecated and they're found under Tools so if we twirl this open you can see that this tools property is a list of tools that the model may call currently only functions are supported as a tool so in the future there will be a bunch of things that you can use here but right now the only tool is function calling and this array has two items one it has a type and because only functions are supported right now the only value that's accepted for this is function then you have have the body of the function itself and here's the real meat of it first you've got a description and this is just a description of what the function does so in our case it's Create a quiz with this structure and this description is used by the model to choose when and how to call the function now we're going to force this function to be called every time and I'll show you how to do that in a second but if you only want it to execute when it thinks it needs to that's also an option next give a name to the function and finally parameters this is where you define exactly what you want Chad GPT to return to you so let me show you the structure of ours so if we scroll down here you can see we've got our tools of course it's a type function and then we've got our function itself the name is create quiz the description is the same one I gave to the system up there and finally we've got our parameters that is the structure of the data that we want want to come out we want it to be an object and the object will have the following properties it will have a question property an answer property and an options property this one will be a string string and this one will be an array then we give a description about what this property is and so I'm just saying it's the question based on the transcription and same with the answer and then finally if you have an array you need to have this items property where you give it a type and these are going to be strings okay great we've only got one other property we need to handle and that is our tool choice right here and this controls which function is called by the model if you give it the value of none that means it won't call the function Auto means that it will make a judgment as to whether it should call it or not or you can specify a particular function by giving it this structure right here and that forces it to call the function that's what we want we want this to be deterministic so this should be of type function and then we give it the name of the function that's the name that we gave up here right here okay let's go do that so we've got tool Choice function with a name of create quiz which is this name up here beautiful all right let's test this call out so let's come over to response and text let's pass in a paragraph from our docs about State Management let's see what it does with that okay beautiful let's scroll down and take a look at this and you can see here in tool calls function and arguments down here is where we get that structure now I know this looks really funny right here and I'll talk about that in a second but you can see in here we've got a question which type of State variable can be access and modified from any part of your app the answer is appstate and the options are appstate page State component State beautiful so this is exactly what we want but what's the deal with this structure here why does it look funny well it looks funny because while this may look like an object or a map it's actually Jason it's Jason nested inside this Jason response so while flutter flow will parse this Jason response automatically we have to do another parsing of this right here because this is just adjacent string now why is it like this well it's probably because open AI wants flexibility in this response here and by allowing this to be just a string here and not have that deterministic structure this response is more protected from error okay but what we need to do now is we need to just grab a reference to this arguments property so if we just scroll down here I've already selected this Json path to our arguments and given it the name function response and we're all good okay that that's all the logic let's see how it's wired up in the app so we have two screens the first one right here with simply a text field and then our button with some logic in here the first thing we need to do is we need to get the transcription of the URL that was dumped in here so we're making a backend call to that build ship endpoint so we've got our backend call and we're referencing that get transcription endpoint it accepts a URL and that's coming from from a widget State on the page which is that text field now if that succeeds so if we get a transcription back then we want to send that transcription over to open AI so we can get that structured data from our function call back so we're doing another API call this time to our open a function call and we're dumping in that transcription which is from an action output variable right there that's that first one up there there and we're referencing our Json path to the transcription remember there's only one property on there just a transcription okay beautiful now what we get back from open AI we're dumping into this variable right here open AI response and we need that because that's what we're going to store in our fire store so we have a backend call to our fire store we're going to create a document in our questions collection and we're just setting all these Fields now here is where we have to do that parsing of the Json that nested Json and we're doing that through a custom function so before we jump in here let's jump over to that function so we can see what's going on so I've got one parse Json function right here we have one argument we're passing in here which is a string that's that nested Json string with all of our beautiful structured data and we're just returning some Json so first here we're just doing doing a null check so we're setting this variable equal to whatever is passed in or an empty string to just make sure we get no null errors and finally we're just calling this Json decode to pars our Json which is coming from this Dart convert package beautiful so we're dumping that string in and getting Jason out of it all right so now when we come back here we can understand what's going on so we have our question let's start there and we want to get it out of that nested Jason so we first want to bind it to our custom function and we pass in the response we get from open Ai and the path that we set up function response that's that nested Json then we get Jason out of here and what we want out of there is our question so if you think back to our parameters that we set up in our post body we're just grabbing that question that we originally set up and all the rest of these work exactly the same way so we have the function we're passing in but instead we want the options out of here and for our answer we want the answer same thing and then finally we just have a created at so we're just giving it a Tim stamp of the current time finally we're navigating to the actual quiz page beautiful so let's take a look at that when this page loads I'm doing a backend call right here I'm looking looking at that questions collection I only want a single document here of course you can change it however you want and I'm just ordering it by when it was created decreasing so I get the most recent one in my collection this is just for the demo okay so now I have that document in here now I just need to bind it to the different widgets I have here so I've got a question right here and I've got that bound to this questions document and to the question beautiful and and we've got the options right here I have that bound here to the options list right there and finally we've got the logic on our submit answer so if we open that up we are just checking that whatever is selected in the radio button so that's in our widget State radio button is equal to whatever the answer is in our document so if it's true I'm going to show a snack bar that's green and I I say correct if not I say try again with a warning color and that's it that's how to use function calling from open AI in your flutter Flow app

Channel: FlutterFlow

Views: 11,709

Keywords: Flutter, FlutterFlow, No Code, No Code Tools, How To use FlutterFlow, Google Flutter, Dart, No Code App, FlutterFlow App Builder, No Code App Builder, open ai, function calling, open ai function calling, open ai api, buildship, rowy

Id: TQEYfIqnwcY

Length: 19min 40sec (1180 seconds)

Published: Wed Nov 15 2023