OpenAI Function Calling - FULL Beginner Tutorial

from that API response from raw text to a dictionary in a controllable way that changes everything alright open AI function calling this is a very exciting new feature but it's a bit of a tricky one you have to see some examples play around with it in order to like fully grasp the ID and the possibilities here but if you're serious about working with large language models building applications around them then this video is a must watch and you really have to understand this concept of function calling because it solves a lot of the problems that you're currently running into when you're trying to build applications around large language models it's really a deterministic way of extracting information from these large language models and they will change everything alright so let's get into it now this will be a Hands-On tutorial like most of my videos I'm really gonna walk you through some code examples which I will make available on my GitHub repository as well so this will be under the Lang chain experiments link will be in the description and today we will be working from the open AI functioncalling dot Pi file with some examples that I will be walking you through now I've checked out some other videos of people like showing how to work with uh function calling from openai but most of them just cover like the basic example that you can see here in the blog post basically and this doesn't give the full scope of what's possible so I really want to give you some more in-depth examples really show you how you could potentially incorporate this into an application like for example I'm incorporating this into a chatbot right now so really getting information from the llm and then based on that information deciding where the call should go next really powerful stuff so those are kind of like the the concepts that I want to explain to you now if you want to follow along you need a python installation and an open AI key no access to gpt4 is required I will be walking you through this example within vs code and if you're interested about copying this setup how I work with my data science projects and AI projects and check out the link in the description I've been getting a lot of questions about that but for now let's launch an interactive python session over here do the Imports load the open AI API key and get started so make sure you're in the correct environment make sure you have these dependencies which are in the requirements that dxt and then we can basically get started and now throughout this demonstration I will be running you through an example through a hypothetical case study of an intelligent agent basically for an airport so this could be a customer server agent that can help customers with with questions with inquiries and before I can really show you how these functions work we must first look at how the API normally functions operates basically so if we load the open AI Library basically and we call the chat completion model we can then put in a model over there so this is the new GPD 3.5 turbo one and we can also put in a message so if you're not familiar with this I would suggest like looking up how this basically works this structure but you specify a role and then you basically ask the question and we can also specify a system role to give the AI basically some instruction on how to behave but for now we just do a simple question when's the next flight from Amsterdam to New York so we can just run that and wait for it to complete and then first have a look at the let's see the completion itself so here we can see the original object that we're getting back from the API so there are various like attributes in here that we can access But ultimately we are interested in the content over here which is under choices content and then basically the first one within that list so what we can then do is we call hey we want the output and if we have a look at that we can say hey I'm sorry I cannot directly provide real-time flight information as the data keeps changing okay so here we've extracted the content the original response basically from the large language model but because it does not have access to the internet it cannot help us with this user inquiry of wanting to understand or get information about the customer's flight so this is not a very intelligent agent and it can only provide us with answers based on what the model was trained on no live or internal data that we can use that we have access to right now so how you would counter this like without functions is you could basically get this query this question basically from the user and then instruct an llm basically to extract the keywords for example Amsterdam and New York from that and then output that as like a Json object so that would literally be you prompt engineering hey we have this inquiry information about Amsterdam New York if it's about flight information convert it or output it in such a way where you have a Json object with location 1 location 2 Etc see where I'm going with this this can really easily become quite tricky especially when your application scales for example and there are lots of questions that you have to be able to handle plus these models are non-deterministic meaning you never know exactly what you're going to get out of it even if you be if you're really specific you see the temperature low based on a user's inquiry a question if they phrase it like completely different like it might throw off the llm and not just produce a Json as output but add some additional text as well and then if you want to capture that within a function parse It Whatever get the arguments you might run into errors then you have to use like regular expressions or whatever like I've said it gets messy real quick alright so now I hope you understand the limitations of working with large language models like this and now we are going to look into how can we buy I bypass this I'm sorry I cannot provide real-time flight information okay so now we have set the stage and we can really talk about what functions are and now before I dive into this function description with you this huge block of text I want to first show you where to put this once it's done and that is within the block of code that you can use to create the chat completion so remember how we in the first place created the response like this we can now get another response by putting the function description over here and then we also added at the additional parameter of the function call Auto which we will get into in a bit but that is just to provide you with some context so now look at what is actually this thing going on over here so it's a description of a function that doesn't exist yet and I think that is a really important concept to understand so this Does this does not look at your python codes this does not look at whatever libraries you've loaded this is just plain text and objects basically that we will send to the API to provide it with context now later like I will show you we are going to actually create the function but it's good to understand that it doesn't have to be created when you interact with the API so now let's look at how to construct this function description and as of right now I believe if you follow the example from openai there are three keys in here that you have to assign that is a name so this is the name of your function then you your description so you describe what this function actually does and then we have the parameters which is an object a nested object basically which has a type and then properties and then requires so let's have a quick look at what's going on over here so this function has the description get flight information between two locations so this gives the API some context okay this is what it's about then we look at the parameters so okay this is a function and it requires some input parameters in order to get the flight information and what are those parameters well first of all it's an object and it has properties and that the properties are first of all the location of the origin and the location of the destination and we can see in here they're both a string so the API also understands this is text output and we can also add a description over here and even provide it with an example of what it should look like and in this case it's the abbreviation basically of the city so Hamburg and Dusseldorf okay so now with this description of the function and as you can see it's within a list so you can also add multiple functions which we will get into in a bit but we start with one what we can do is let me store this in memory and then also store the prompts so we have the same question we use the same API the same model the same message but we add the function description as well and we set the function call to Auto let's see what it will actually do for us so let's wait for that completion and then again it supports the note the function does not exist this is just a description that we're providing the API with and now let's see what the outputs looks like and print it now and this is where it gets exciting so remember before we were getting the answer I'm sorry but I cannot directly provide you Etc but right now if we look at the actual message so we can first look at the whole completion and you can here see the message so if we just focus on that print that we can see role assistant and then we have content and then we have a function called parameter in here function call Key and what we can see in here we have a name get flight info and we have the arguments location origin AMS location destination JFK so this is quite interesting see what's going on over here so it interpreted the question from the user and it understands that it has to convert Amsterdam to AMS to and New York to JFK so it converts it to the actual flight tickers based on the descriptions that we've provided over here and now it also provides just that in the form of an argument's key that we can access here and this is an object that we can use because we've defined that we want an object and now this is really where it gets interesting okay so now we have this these arguments basically but what can we do with it okay so let's continue to the actual function that we've created here in Python and now like I've said the API does not require the actual function but in a real world scenario what you would typically do is you would first create your function so this could be a function with an open a API with an external API or an internal API to gather data customer information whatever do some kind of like calculations just any use case that you would use an API for and you turn it into a function and you define the parameters that is then the starting point so let's have a look at this get flight info so we make sure that the name of the function matches and then also the input parameters of location origin and location destination and let's see what it does so it's pretty simple right now we've hard-coded the information so it gives us some adjacent object back basically on based on this this information but you could easily Swap this out to make a real connection with an API okay so now let's see what we can do with the output that we got from the model so again let me print it one more time here's the output and here what we can do is we can extract the parameters so let me actually show you what's what's going on over here so we start with the output and then we call Dot functioncode.r arguments so function call arguments let's have a look at that that is the object that we're looking for and then we put that into a Json load so as you can see right here it's it's a string right now but if we call it Json loads and put that in we have now converted this into an actual python object it's dictionary essentially under the hood which we can use to get attributes from get keys from so we can now call get and then insert location origin to let me do that get AMS pretty cool right okay so let's store that and we do that in a similar fashion for the destination so let's run that line see what that includes and now that is JFK and we can also without using get function just store the arguments and these are now the params and if we so we can print them all out but let me also show you if we run the programs this is an actual dictionary so did and this is something you really have to understand what we basically did right now is from an unstructured user input question we have converted that into an actual dictionary structured in a way that we have control over and that is such a big thing and this is something if you don't fully grasp it yet it's something you will learn down the line when you start to actually build applications around these models but this is really a big thing getting from that API response from raw text to a dictionary in a controllable way that changes everything because what we can now do based again on the output we can first extract the name of the function using the Python's evolve method so let's quickly run that and now chosen function is our get flight function over here so imagine we can do this All programmatically based on rules if statements based on various functions that we provide the model width and we can then extract the functions that it picked and now for the real magic we can get the chosen function and then put in the params and use two asterisks to unpack the variables so if we look at have a look at the params one more time it's an object and if we use two asterisks we can unpack that and we run the function and we can print the flight so here you can see location origin AMS JFK we have the daytime in here we have the airline and we have the flight and now within this object we have all the information that we need in order to provide the customer with a proper response so let's see what we can do right now if we put this back into the model so now we basically create a second completion so there would be an intermediate step after a customer asks a question you run the API two times first you run it to call the function get the the output and then run it through the function and then here's the key you basically add the function output back to the message with the role function so let's see how that works so we first have the the messages and we add the user question this is the user prompt so again this is the question what's the next flight from Amsterdam to New York but now we add the next one row function name and then we provide output function name so again this is just the name and then for Content this is important we add the flight information Here and Now what this will do this will provide the model with all the information it needs in order to draft the response so let's run this get the response the next flight from Amsterdam AMS to New York JFK is on July 6 2023 at 1826 the airline is KLM and the flight number is kl643 beautiful right this is so cool and if you understand this correctly like I've said you can build some really really cool things alright this already should put you ahead of most people's understanding of how these function calling methods work but I want to show you one more thing about using multiple functions and how to go about that so here's a new function descriptions multiple variable where there are three functions first we have the same one get flight info you're already familiar with that one but then we also have book a flight and in here we have location origin destination daytime and the airline so we need information from the get flight info in order to book a flight and then we also have the file a complaint so this is a function that a customer can use to to file a complaint so we have a name email and a text so let's see let's first store that in memory print it so we have a list with three objects in here pre-descriptions now here's a quick function ask and reply that basically completes the chat completion and then inputs the function descriptions multiple so like I said like let's store that and now let's go through these examples starting with scenario one where we check the flight details so this is similar to what we've done before but now remember that the model or the API can choose from the different functions and now this is where the parameter of Auto comes into place so we have function call Auto which basically means go over all the functions that you have access to and pick the best one based on the names the descriptions the parameters Etc you can also change this to a specific name so you could also call for example file complaint and force it to use file complaint So within various steps within your application it might be desirable to leave it on auto or sometimes it might be convenient to enforce a certain function so that's that so let's do that one more time so if the user prompts and then we print the reply okay so what we do we and just is just to clarify we don't do the extra step where we feed it back into the model because I've just shown you how to do that this is just to check okay what are the functions doing and what are the answers basically the replies that we're getting so here we can see we have a similar uh a similar response it uses the get flight info and we have the AMS and the JFK so we've validated that based on three functions with this query it is able to pick the correct one so then let's process all of that again and then let's print it again this is similar to what we did we can validate it amsjfk and we have the parameters okay but now let's load also the daytime and the airline So based on the response so the flight which is an object we can also extract the information so now we have the flight day time and we have the flight Airline and now remember this is only possible because we first got the function parameters then called our internal function to get the data from our database which was hard coded but that's why this is possible that's how we were able to get the flight daytime and the flight Airline flight Airline okay so now we can move into scenario two where we are going to book a new flight so again this is just hard coded but you can all make this programmatically but now let's change the user prompt to I want to book a flight from origin to destination on flight daytime and then flight Airline okay so now it understood correctly that we need the book flight function and let's have a quick look again the book flight it has location origin destination daytime and Airline and because of that we now have all of that information here alright so now let's make the final step over here scenario three which is file a complaint okay and it understood correctly that we have to file a complaint and it extracted the name we got the email correctly and here is the the message basically so the the complaint itself it was unpleasant surprise uh because the flight was missed now let's make this conversational so this was step by step to check validate basically okay we have multiple functions and what if we interact with this using natural language will it be able to pick up on the correct ones and for this we're going to use the chat openai model from link chain provided with the model and the temperature and store that as the llm and then we have the user prompt over here which is basically everything that we just went through but now it's in one prompt so it's explaining hey I want to do several things I want to get some flight information then book it and also file a complaint here's my information so let's store that and now let's construct the first response and now since we're losing using Lang chain it's a little different but we're still like under the hood like gen generating the messages in a structure so the first response is we take the llm and we predict the message which we put in as content is the user prompt so that is this and then the functions is what we have over here and then we have the response and we say hey function call name get flight info okay so this is still as expected based on this prompts it first identifies that I need to get the flight info but does the function call get flight info and now what's interesting here so remember before we providing multiple functions but one prompt now we have multiple functions and one prompt with multiple questions within it so it gets more complex and now to continue with this we just extend the messages that we're providing to the API now since we're using Lang chain this looks a little bit different so we're using the human message and the AI message template and if we go to Lang chain documentation you have system messages human messages and AI messages which are used for representing information coming from the AI system and um I could only get it to work by providing a double so first like the content and then the description of the actual function but this is probably something that I don't quite understand yet like I've said since it's an additional thing that you have to look into with Lang chain but if we run it like this so we build up the conversation and then do the second try so let's see so first response we were getting the get flight info and now for the second response we have the book flight function remember same prompt some additional information now it understands okay I have the flight information so I can book the flight and we can go on with the third one run that and now it understands correctly that it's now time to file the complaint so it again goes through the prompt checks okay what are the actions that I have to do based on the user query check it off then looks through the history when it makes another call and it decides okay now it's time to file the complaint and then to wrap it all up let's run it one more time so now essentially the both the AI has completed all the tasks with the functions it has access to so let's see what we get so now we don't get a function call like you see content function call in the third response now in the fourth response we just get content I've successfully booked the flight from Amsterdam to New York for you the next available flight is on Jan 1st and it's with Lufthansa okay now this is interesting to see where it seems like it's hallucinating on this one uh it did file the complaint on your behalf regarding the Miss flight and email copy was sent to the correct email thank you for bringing us to our attention alright so it seems to do a pretty good job but it's definitely hallucinating on the date and The Airliner because we have not called the get flight info function within this chain of questioning so if we look at the first response so that is get flight info and we get AMS and JFK and then if we do the second response we do get the book flight but it hallucinates on the dates and times over here so that is where that is coming from and that is because we let's see if we we say that if we book a flight it's required so it needs a daytime and an airline and you can see that it just takes the description so for example this is the data took and this is the airliner that it took so that is I think also it could be a nice thing so when there is no information you can put in some default values but you can also definitely Implement some error handling some catching in all of that and that is just because we did not call the info for the next prompt like we were doing in the first scenario so that is just uh something to keep in mind that if you don't have the information and you have a description it will just fill it in but hey other than that I think this was a nice Loop so we completed all three functions and then eventually it showed hey this is the final message that I can present to the user which you could do in the form of an agent the chess chat message an email whatever all right and that's it so I now hope that you have a solid understanding of how to work with the new function calling methods for the open AI apis like I've said really powerful stuff I will definitely be deep diving into this more potentially create some more videos around this because this is really the key if you want to create cool applications which large language models so if you want to stay updated on that if you want to stay tuned make sure to subscribe to the channel also please leave a like if you found this video helpful and if you're serious about learning data science and artificial intelligence then you should consider subscribing to my newsletter first link in the top comment that I will pin below this video of some exciting announcements planned for that so make sure to stay tuned and now if you want to learn more about creating applications with large language models then should really check out my video on langchang which I will link up here that should give you a really good Deep dive understanding into how the language and Library works and what you can do with it foreign foreign [Music]

Channel: Dave Ebbelaar

Views: 35,496

Keywords: data science, python, machine learning, vscode, data analytics, data science tips, data science 2023, artificial intelligence, ai, tutorial, how to, openai functions, function calling, chatgpt functions, llm applications, function calling openai, function calling gpt, function calling chatgpt, function calling via chatgpt api, function calling via chatgpt, function calling another function python, AAA, ai automation, ai automation agency, open ai, natural language processing

Id: aqdWSYWC_LI

Length: 28min 24sec (1704 seconds)

Published: Thu Jul 06 2023