Function Calling in Ollama vs OpenAI

function calling this has to be the worst name for a feature every company I've ever worked for had marketing folks who had no idea what the feature was and called that feature something that sounded cool but has nothing to do with the actual feature this is sort of related but it's not what the feature is so if you know what the feature is pretend you don't and think of what function calling should mean I would think that function calling in a model means that the model calls functions and that would be a great feature for a model to have but the model has no ability to call functions when you use function calling function calling is all about controlling the format of the output from the model when you get the response in a format you expect then you can reliably call a function but you are calling the function the model is not or even if we expand it out to the company that created the feature open AI does not call the function your program that calls the open a API calls the function don't believe me let's go to the open aai docs and go to function calling scroll down to this four-step process step three you parse the string output of the model into Json then you pull out the values then you call the function I am great with that it's an awesome feature it would just be better if it was the name was accurate so okay so with that settled let's look at how open AI API achieves function calling one of the important steps is to inform open aai about your functions what the function actually does isn't relevant this is just how you let the model know about the schema the output should stick to the rest of this example is using the chat API to ask the model a question and then the chat completion message shows us the output string that we format to Json now let's move on and look at how to achieve this in ama the feature you'll find in the docs is referred to as format Json which is a bit more descriptive about what is really going on there are two requirements to get this working and one bonus one that makes it even better you can do it in the API or even in the CLI if you're scripting something in bash let's use Python for this one and I'm not going to use the SDK just to keep things more translatable to other languages we can start with importing requests now we can define a payload that we're going to send to the chat endpoint this includes a model and an array of messages we don't need to include a system prompt since there's already one in the model in ama so roll user and content uh what is the capital of Germany now set response to a post request to http localhost Port 11434 API chat and setting our payload to the Json now print out the Json from the response to get a streaming response from python we need to iterate through the response so for message in response. iter lines then parse the Json and print message. content let's add and set to nothing to put everything on one line unless the response includes a new line Python fc. piy and we get something saying that the capital of Germany is Berlin awesome now let's make this a touch more Dynamic by taking the country as a command line argument I'll import CIS and then set country to cy. arv1 then change the content in the payload to be an FST string and include country in curly braces python fc. France gets us Paris now let's say we want to find the capital of that country and then calculate the distance between Bainbridge Island where I live and that City so for that we can use the havene package which takes the decimal latitude and longitude of two places and calculates the distance between them so let's change our question to what is a decimal latitude and de small longitude of the capital of an in then curly braces country and run python fc. piy Portugal and we get some coordinates but the current format of the response changes as we use it plus having to parse through the string looking for coordinates well that's going to get annoying the first thing we can do is ask the model to respond as Json let's add it as a new system prompt we can just add this to the top of our messages roll is system system and content is you are a helpful AI assistant the user will enter a country name and the assistant will return the decimal latitude and decimal longitude of the capital of the country output in Json and we can change the user content to just country so this is a good start but notice that we have an explanation of the Json format and sometimes the key names are different so add format Json to the payload and things are a lot better but if you try it enough you may still see that key names change so we need to provide a schema to the model this is the purpose of the function Block in the open aai API for the schema I'm giving a key then type and description since I need the city and coordinates I specify one for each and I'm using the words lat and Lawn to show that my schema is being respected if you run the program a few times you'll notice that it's not the most precise way of getting the latitude and longitude but it's a demo now if the model sometimes responded with something that wasn't in the schema I would do a few shot prompt let me show that just so you can see an example there's a user and I also give the assistant response that I specify for a Json response since we need to get content out of it and we don't do anything until it's complete we should disable streaming then we can set City info to the Json string in theage message content and now we can call havene with my latitude and longitude and the coordinates of the capital city and get a distance now if you run this a few times you may notice that the distance isn't always consistent asking a model for latitude and longitude isn't a great use of the system we can set the temperature to zero to try to get it to be more consistent but it may still be different from what you get with a Google Search and that depends on where exactly both systems put the dot that says this is the spot that represents the capital of whichever country hopefully you now see how you can use format Json to get a consistent output from the model because that's all the terribly named feature of function calling does it gives you the consistent output from the model so that you can call a function with the data which is a very cool thing to be able to do function calling using the new open aai compatible API isn't available yet inama at the time of this recording so you need to use the native olama API for this now I don't think anyone is going to want to do it the open AI way because it's just a lot more complicated and doesn't offer any benefit but it has open AI in the name so maybe that's good enough for some to justify the suffering through the pain what do you think is function calling I mean format Json an important feature for you are you using it today let me let me know down in the comments below and if there's another feature you'd like to see let me know about that in the comments when I started this latest focus on the channel about a month ago I had a simple list of about 50 ideas for future videos but your comments over the last few weeks have upped that to well over 150 keep them coming because there are so many good ones that I never thought of thanks so much for watching goodbye yeah

Channel: Matt Williams

Views: 21,973

Keywords:

Id: RXDWkiuXtG0

Length: 8min 49sec (529 seconds)

Published: Tue Feb 13 2024