Web Dev Fundamentals: Exploring the new OpenAI Assistant API (.NET)

[Music] okay so let's get started uh we will do the in English as usual and our topic today is the new open AI API I'm not sure if some of you had in the past already contact with the existing open AI API uh it has been around for quite a while and this API is currently reworked quite a bit let me show you the website so you can introduce you to the existing concept and to the new Concepts um if we Google for open AI rest API we will get to the documentation of open Ai and here you see the API reference on the top left of the screen you see the menu and here you see the old or let's say the existing chat API this is the API that has been around for approximately a year and many of the applications that you see on the internet besides openi itself are using exactly that API if you want to use this API you typically do not directly interact with the rest API as you can see here it is quite a lot of data so you need to set up the rest API requests yourself that's quite tedious so typically you use an API package depending on the programming language that you use so if you use uh net for instance we can say open AI Azure net and that will probably bring us exactly to the library Azure doai doop AI that is the current release of the API for.net that makes it much much easier to interact with open AI please note that this library is currently also in beta version so it's not for General availability in production applications there are apis for for python for Nots for go and many other platforms some of them are maintained by open Ai and or Microsoft and some of them like for instance the go API which I already tried in the past is maintained by Community developers so these are community libraries okay so if we would just use the existing production API of open AI we would exactly use this nougat package but this is as we are here at an education facility this is kind of boring and therefore we will immediately scroll down and take a look at the new API at a new beta API this beta API was announced I think three weeks ago or something like this at the developer conference of open Ai and this is exactly what we are going to use now what is the big difference between the new beta API and the existing API well the first and most most important difference is that it's way more powerful and way more complicated so by now the existing production API I think we would probably be able to cover the concepts of this API in an hour or something like this and then we are done the new API is way more let's say interesting now what's the idea of the current of the new API it's the so-called assistance API the idea is that you can create one or more assistants an assistant if we take a look at what an assistant is um is is is made of is essentially a description a name and instructions the instructions that's the system prompt that we have already talked about that's the text that tells chat GPT how it should behave you can tell it to be a tutor in a school you can tell it to be a a car salesman you can tell it to be a flower shop assistant you can tell it to talk to to use a language for children to use a a scientific language whatever that's the instruction part so you create this assistant and in contrast to the existing API the new API persists these settings so they are stored you do not need to have your own database where you store all these settings and you have to retransmit everything from the very first bite on in every request so now they are storing data you will get back an assistant ID you will see this ID in a second and later on you can always refer to this assistant ID and open AI will immediately know what you mean so they have stored the instructions for instance for you so we don't need a database that's a very good news the second thing that we need to understand is that now jat GPT has a threads API threads like communication threads up until now if you take a look at the production API you always had to send the entire conversation from the very first message until the very last message to J GPT so you had to store this conversation yourself this is no longer necessary you now can create persisted threads these threads are stored by open AI you will again get back a thread ID and later on you can simply add new messages by referring to this thread ID so again they are taking a little bit of work away from us and that's a good thing however the API is um is not the simplest One the good news is that you are very well prepared when working for working with this API because um this API is a really nicely designed restful web API so the design principles follow exactly what you have learned in this course and you will immediately understand how the paths are separated how the payloads work and so on you will see that's really not difficult that's the thread API now when the user enters something through any kind of UI you have to add messages to the thread so if you take a look at the create message then you will see that to create a message you have to refer to a thread so before you can say something to Jet gbt you have to create the thread and then you store the message into the thread got it so you can have many many different threads if you have a website and you have a lot of users on this website you would probably create a thread for each user and when a user types something in you send the message to exactly the thread related to the user got it the good news again is you do not need to store a list of messages that is done automatically by J GPT however now it gets a little bit complicated if you paste a post a message to a thread jet GPT does nothing it remains passive it just Waits this is where the so-called runs come in the runs mean that you tell chat GPT hey chat GPT please read the entire message history of the thread and create a new response it's a kind of batch chop that you are starting here you are telling the large language model of open AI become active read the messages and create a response okay so this is what you do you you create the run here you see you can create a run but this run will not immediately return the result this is just a batch job it runs in the background so what you have to do is you have to continuously retrieve the run and inside of the run you have a status as you can see it here and you have to ask hey what's the status what's the status what's the status until the status receives or the status um arrives at the completed state when the run is in the completed State you can then ask for the messages here of the thread and you will find jck gpt's response inside of the messages could you follow so far things get a little bit more tricky when we add function calling to the equation but that is something that we will discuss a little bit later first we want to cover the Basics we want to create an assistant using the API we want to create threads using the API we want to send a message and we want to trigger a run and receive the message response from open AI so let's cover the basics first and then we deal with the complicated stuff using function calling and things like that could you follow so far okay very good uh if you want to learn more I highly recommend to First Take A close look at the different rest API endpoints and then there is a dedicated guide here if you take a look at the documentation you see a lot of stuff here it's very well documented that's the production API not interested for us today we want to work with this stuff and this is exactly the beta part the beta API is well it's documented but it's not very well documented so there are a lot of there is a lot of room for interpretation but this is why you have me I prepared a sample let's see how it goes okay but be prepared we have to create a lot of manual work and hopefully in the next few weeks months the apis will catch up and then you can use the N packages but as of today you have to use the rest API good nice now it's very important to understand when you want to play with this stuff that there is an assistance playr ground this assistance playground is super super useful let me log in for a second so I can show it to you uh by the way for this lecture I will give you an API key so you can follow along so hopefully it will work as you can see here this playground I will make it a little bit larger so you can see it better this playground allows us to experiment with the API through a user interface first as can see here we can create assistance see that one so let's try that and let's take a look at the API so we really understand what the apis do do and then we go into coding so let's create an assistant give it first and name um let's say That's The Simpsons a uh Simpsons assistant you are uh helpful assistant answering questions about the Simpsons only answer questions about the Simpsons uh if the user asks questions about anything else say uh can't answer that okay let's try that the assistant is related to a model we can choose between different models the most important selection criteria is here whether you want to use GPT 35 or GPT 4 GPT 35 is cheaper and faster GPT 4 is more intelligent but slower and more expensive in real life so if you build an API and you want to put it on your website and you pay per API request then you have to ask yourself do I need intelligence then take GPT 4 and then you have to accept higher costs or can you work with 35 then you get faster answers and cheaper um API costs okay that really matter Just5 and we will try exactly that okay so I will try it exactly here let's start with the 3.5 and then let's see how good it is okay later on in our Advanced example we will add so-called tools first we can add function calls that will be the topic of today hopefully we will be fast enough to do that in today's series of lecture then there is the code interpretor tool are you all familiar with the code interpretor tool anybody not knowing what the code interpretor tool is h no no no so let me show you what the code interpret tool is um if we go to chat open Ai and ask a question which um which forces chat gbt to to do some calculations for instance um what what could be something like this uh we could say um generate generate 100 um persons with random names and random um [Music] Heights create a CSV file from it and allow me to download the resulting file something like this now take a look what's going to happen now do you see this analyzing if I open that up you will see that chat GPT is now writing python code on the fly but it's not just writing this python code it's really also executing this code so here you see it uses the pandas library from python the random Library it does some random stuff and here you see it's going to take the panda data frame and Export it to CSV and then it runs all this stuff and it gives me the CSV file as a download I can open it up here and it opened up on the wrong monitor let me give it to you here yep name and height I did not specify what I want as the name that this is a a name of a person so it generated just random blah and then the height and meters and that looks pretty okay and code interpreter is the fact that um Ash the J GPT is capable of generating code and running it not just generating it also running it that's code interpreter so if we want we can allow the assistant to write code executed in order to answer the questions of the user got it we will not do that today but this is a very important and super convenient functionality um retrieval means that we can upload files with data that the assistant is capable of using for instance we could take exports of our school's vikii upload it here and then ask questions about our school and chat GPT will take a look in these files HTML files PDF files whatever and answer the questions based on the information which is in these files that would be the retrieval feature okay and then we of course have to use some uh files if we want we don't need that one and we will start very simple so we can click on Save and that saves the assistant the assistant has been created now for us developers this playground is super useful because there is a logs part here where we then when we start with the threads where we can take a look what's going on behind the scenes so now we can ask questions with the gp35 turbo does anybody have a suggestion for a question related to the comedy series The Simpson what is the name of the famous shop owner maybe it understands what I mean let's click on run and then we'll take a look at the apis it runs runs runs and hopefully the famous shop owner in The Simpsons is I will not try to read it I just say Apu who owns the Quicky Mart that is correct now let's take a look at the logs and here you see why this assistant uh playground is so useful we can take a look at each and every web API requests we see that the system first created the thread then it adds a message then it runs the thread and then it continually asks the system for what's the status what's the status what's the status now if we take a look at the last status message and take a look at the response we should see that the status here is now completed that's perfect that's exactly what we are looking for so next is the system will ask for the messages and inside of the messages we see that we have two messages and the first one the first message should be the answer see so we get a message history got it and once we got this message here um the system will ask for the run and I think that's it there is nothing more that nothing interesting here anymore so I think you got the point Yeah question was the second um the second mess watch was our message here as the user what is the name of the famous shop owner got it okay very good so this was not a challenge for jat GPT 35 do you have a more elaborate question a question that is harder to answer so you see for as I said for our application uh 35 is perfectly fine do you see the value of this playground as a developer this is super super useful when it comes to playing with the API understanding the API you save a lot of experimental time you do not need to code every single request on your own you can simply generate them and you get a deeper understanding what it does now there are some severe limitations when it comes to the new API for instance um let me go back here to my gbt 4 and um ask a different question maybe we can ask the question um describe what classes are in CP if you take a look at the answer you will see that the answer is generated token by token you see the tokens are flowing in I will now stop the generation because it doesn't make sense to waste any comput time here but you will see that tokens come in they fly in step by step by step by step now this step bystep generation of answers is called streaming the existing production API supports streaming it uses a protocol which is called server sent events so the server can send events and you can uh get these events and you can process them it's not difficult especially if you use the existing nouet package for C it really works like a charm the problem with the new beta API is that they do not support streaming yet openi has communicated that this is a known limitation and hopefully we will get a streaming API back at the beginning of 2024 but as of today we cannot do streaming so especially now that GPT 4 is pretty slow not having streaming is kind of a big problem because you send a message and then you have to wait until gp4 has generated the entire the entire message understand what I mean and then you can display it well you can get intermediate results that's true but you always have to pull the system you have to pull the system hey give me the new response give me the new response streaming is way more convenient but that is currently not there it will come but we don't have to deal with streaming as of today got it good nice so with that I think let's try to put all this stuff into a c program good let's get started um um to keep things simple to really be able to focus on what we want to learn today I would like to implement a console app together with you if you want you can take the time as a homework to take what you have seen there and put it into an API project but I I want to get the noise to content ratio as good as possible so let's remove the noise and let's really focus on what we want to achieve work with open AI still you will learn a lot about web apis because we will focus on the web client the HTTP client and I can tell you some tricks here good I would like to ask you to create a new and empty project so meaning an empty folder and then you say net new console and this is what we are going to do I will do the same here on my computer I hope everybody was able to generate a new console app I did the same I have a console app here on my screen and and we can do one thing before we go into the break we need to deal with the open AI secret you have to understand that the open AI API wants us to present a secret an API key in order to make sure in order to identify the account that is currently working with the API when using open AI in Azure you don't need that you have the so-called managed identity there it's much more secure but currently we need the open AI key now the question is where can we put this key we could create an app settings Json file but you shouldn't do that because when you take an app settings Json file you add the open AI key there it happens very easily that you check in this app settings Json filing GitHub it might be in public repository and immediately you have lost your open AI ke key and somebody can misuse it okay so never do that if you would like to um maintain secrets in uh in a c application you should always do that with net user Secrets I don't know if we have talked about that in the past if we did let's recap if we didn't well that's something new the first thing that you have to do in order to store some settings um called secretly you have to initialize the net user secrets in your project so please say net user secrets in it this will create a good here you see the good in your CS Pro File this Good Will um be used to store the user Secrets outside of the folder of the project so there um net will automatically make maintain um a settings file outside of the folder therefore if you check in the folder the secrets will not accidentally spill into this checkedin folder okay this is what you do here and then you can say net sorry net user Secrets set and we can for instance say open AI API key or something like this and then we have to specify the key for this lesson I will give you the key in a second but first I will show you how it works just say ASDF or whatever and if you press enter you will see that this is now saved to The Secret store but we do not have an app settings Json in this folder see it that's the important thing so if you need to store some credentials of a database or an API key or whatever and you cannot use managed identity or things like that please use user secrets in the production environment these user Secrets can also come from environment variables but generally for development this is the way that you should do it did it work good so I'll now pause the recording and give you the open AI key and I will delete this key after our lecture okay good welcome back from the break let's get started um before we went into the break we defined a user secret and this user secret as I told you is stored in a different folder you can look up the folder in the documentation that's not the point the point is that the secret is not inside of our folder so we can check in this folder without any problems yes you have a question no that's the same you can use a end file and then exclude this file from checking in however it can easily happen that you accidentally take this file and check it in so using the user Secrets is a little bit more secure because there is simply no file that you could check in but you are absolutely right if you like the end file and you prefer that one don't forget it to put it to put it in G ignore and do not accidentally check it in then everything is good M absolutely good now in order to access these user secrets we need a nougat package and you can add this nouet package using the nouet package manager GUI uh please look up the nouet package um you can find it by typing in configuration and user Secrets if you look for this nouet package you should see here the nug package Microsoft extensions configuration user secrets and you can install it this is this step is not necessary when you use asp.net core when you use asp.net core this nou package is available automatically but if you want to have user secrets in a console app you need to manually add this nouet package again in the nougat package manager look for configuration user secrets and that will give you the nougat package Microsoft extensions configuration user secrets with that we can go to program Cs and in program CS we can start right our first lines of code we are going to create a configuration Builder or Builder equals to new configuration Builder do add user configuration Builder I'm using I'm missing a using statement here okay I don't get intelligence now I get intelligence now it's okay add users secrets of program this line will give us access to configuration values which are stored in user secrets we could also add a Json file here we could add environment variable support we can add all the different sources for configuration values if we want this is what the configuration build is all about in an esp.net core application you do not need this line because this is created automatically by asp.net core you know that already but in a console app it's definitely useful to have that the last step is that we need is to to to build the the configuration um information and we do that by calling do build and with that we can easily retrieve the open AI API key by saying config and then use the name of the variable that we set um using the um net user secret set command and that was in my case o AI API key and now we have the API key in a variable and can use it okay I will zoom in so you can take a look and I will give you a second to catch up and enter this code in your environment nice now we need a possibility to send HTTP requests to open Ai and we do that by using the HTTP client class in C we have used it in the past every now and then but today we need it extensively please note again in the near future hopefully the the nouet package for open AI will support the new apis that is currently not the case this is the only reason why we are going to use HTTP client if you do something like that in real world maybe after your final exam after your matura when you enter the business world please check whether the nou good package was updated prefer the existing apis over directly calling the rest API but now we have no other choice we need to use the HTTP client so let's create a new HTTP client new HTTP client and what we can do to make our lives a little bit easier we can automatically set the so-called Base address now what will be the Base address new URI we can we can look up the Base address by taking a look at the rest API reference if we take a look at the assistant apis and scroll down a little bit you will see that the Base address is that one well essentially we can take the whole thing together with the V1 that is our Base address so we can take this one this is the use of the Base address it's very useful to be able to specify that during construction of the HTTP client please also note that you should not create a dedicated HTTP client if you have multiple requests after each other the HTTP client class was designed to be reused you do not need a separate instance for every call it's essentially a bad thing it's an empty pattern to create an HTTP client per rest API call you should reuse this thing otherwise you can run out of resources uh and that is very inconvenient good let's do another thing if we take a look at the documentation you will see that the system requires us to set as you can see the authorization and um and a header which is called open AI beta assistance equals V1 that's essentially the optin to the the to the API to the new beta API so let's set these two headers and we can do that by saying HB client. default request headers authorization and then in this case GitHub co-pilot is automatically suggesting a meaningful thing here authentication had a value Bearer with the open AI API key that we have retrieved from our from our user Secrets this is the header and this is what the bearer token is all about good very good next one let's set the HTTP client default request header and let's add a new header and then I can copy this one out it says open AI beta this one open AI beer and yeah you see this is a typical limitation of GitHub co-pilot it cannot provide a meaningful request here because the cut of date of GitHub co-pilot is somewhere in 2021 this API didn't exist in 2021 so this thing cannot suggest something meaningful so if we use very new features GitHub co-pilot has very limited support maybe you have heard the GitHub co-pilot AI assistant will be upgraded at the end of this month to hopefully jet GPT 4 then the system will be a little bit smarter but still these apis are too new so we don't get very good support with g co-pilot we have to consult the documentation here let's put it somewhere here not the version I'm sorry I mistyped this of course it should say headers and last but not least let's add let's be good citizens and add the um the accept header so that we tell open AI that we can handle Json responses by saying HTTP client default request headers add no accept and probably yeah exactly GitHub co-pilot will fill out the rest we tell open AI hey with every request we accept Jason as a response this is a very useful strategy configuring The Base address and configuring default headers because now it is very simple to send multiple requests to open AI we do not have to specify authentication over and over again this is done automatically what I will do to structure my script a little bit is I will add some regions here setting up HTTP client and here I will say end region in order to be able to fold that together okay you can do that if you want but obviously this is not a must okay so now we can start interacting with the open AI API we need a pretty long list of um dto and I did that work for you so I sat down yesterday in the evening and wrote a lot of dtos they are not complete so it is not the goal of this sample to support each each and every API that is available it's just enough that we can solve today's lecture I just sent you a link via Discord to a GitHub repository where I have checked in the dto it looks like that here you see region dto for open AI please take this entire region it starts here at line 218 and it goes down until 342 please copy all these lines into your application put it at the end of your application it's a pretty long list you will probably have to add some using statements especially Json calization you can fold it together and forget about it if you would prefer to put them in a in a dedicated file in a separate file feel free to do so yeah you can absolutely do that and again I would like to emphasize the goal of this sample is not to support all features of the open AI API don't get the wrong impression it's not my job to do that this will be done by Microsoft and open AI hopefully in the near future we don't have to write this code by hand in Dynamic languages like python or uh JavaScript you do not need these dto but we are working here with C so we need definitions for these respon and therefore we need the dto I told you I have to admit that I wasn't able to complete everything so I know there is a known bark inside there when it comes to function calling and probably at the end of this lesson we will have to do some debugging together because I tried to fix it today in the morning I fixed a lot but this last step I hadn't enough time I apologize for that but that's a good exercise then we can do some debugging together right good could you copy these dto I think it would be boring typing these dto by hand so I thought why not saving you um the trouble here good now we have the dto we have set up the open AI client now we can start man uh managing the assistant let's start by creating the assistant okay and I would like to do the following I would like to set up an assistant and and then check whether the assistant already exists if it already exists we leave it as it is and if it doesn't exist or is out of date we will update it or create a new one we call this principle um item potency have we have I mentioned this word in the past have you ever heard from item potency before you say yes what is item potency can you try to describe it exactly that's the point you can without any problem repeat the job you can run it two times three times four times and it will not uh lead to bad results so in our case by checking whether it already exists and only trying to create it when it doesn't exist we make our program itm potent that's a very important concept when you Design Systems item potency makes your software more robust because if something bad happens you can simply retry without having to fear that something breaks got it that's also a very important concept when it comes to databases for instance if you have to create a user and then create an order for the user you should make such a process item potent because maybe your first run fails when creating the user then you rerun it and no sorry your program the first first run fails after creating the user so you didn't create the order you want to rerun this but then your program doesn't work anymore because it tries to recreate the user which already exists that would be the opposite of item potency good so let's do that let's first region uh get a list of all assistants and you will see there are some interesting aspects of this API that we should definitely take a close look at um it looks like that VAR assistance equals to await HTTP client and then we can use a get from Json async and there we specify the schema of the output and that would be o AI result of assistant and then we go to the assistants this is how it looks like now let me tell you a little bit of about the API design that open AI has chosen for this use case let's take a look at the list of assistants you see that this is the correct URL so we are using the correct URL but now comes something which is very interesting as you can see open AI supports a list of query parameters and the interesting thing is the paging logic here of course you can have more than 20 assistants you can have hundreds of assistants absolutely no problem but open AI doesn't want to return all of your thousands of assistants in every single HTTP call so therefore you can limit the number of request the number of responses between one and 100 the default is 20 so you will get the first 20 assistants you can specify an order if you want and then you can specify spey for paging if you would like to have the second page the third page the fourth page and that is specified by uh specifying an after object so you take get the first 20 results then you remember the last ID and then you can say give me the next IDs which follow after this ID which I already got got it so let's draw that to make it very clear how this works if this is the list of all your assistants the first call will return by default the first 20 you remember the um assistant ID at the end and with the next call you can say give me the next 20 after the ID that I got before that's a very typical approach for Designing rest apis paging is the concept here well in our case we luckily only have a few assistants I think we are less than 20 people here in the room I think we are I think we are yeah I think we are today less than 20 people so we don't have any problems with um with our paging here so let's forget about the paging to keep things simple but you see in reality it is a little bit more complicated to work with these apis luckily you have this net packages that I was talking about before and they will solve this problem for you now we check whether we have an existing assistant existing existing assistant and then let's see what skub co-pilot suggests yes I think that looks good and now here please everybody choose his or her own assistant name I will say this is raina's assistant please use use a name that is unique for you use your first name and last name just your last name your student ID whatever something which is unique don't just call it assistant because many of you will call it assistant uh and don't use root names please otherwise we get locked and we don't want to do that so just friendly names whatever you want to use so let's get rid of this one now what I essentially do is I'm saying give me all the list of assistants and then I'm I'm checking whether there is an assistant which belongs to me which has a certain name as you can see the open AI API does not support querying filtering for the name using the original rest API they've chosen to build a very simple API which just supports paging the apis that we build together are more elaborate we they are more or capable of what open AI currently supports but this is a beta version it's very likely that they will come up with more advanced versions in the upcoming future very good that should give us the existing assistant if there is one now if you run this program for the first time you will obviously not get your existant assistant but still we can try whether we get something back so what I'm going to do is I'm going to set a break point here and press F5 to debug my application I select the C debugger that should start my app and in a few seconds my app should be up and running the break point should be hit come on here we are at the breakpoint and we see the response status code does not indicate success 40 four not found so what did I what did I do wrong that is not a good idea let's check I think the list assistance it's assistance I think I screwed up here um let's get rid of this V1 here and put this V1 here we'll see try it again come on debugger let's get started see what we have here much better sorry I did something wrong I put the V1 here in the Base address but that is not supported so I took the V1 and put it here and now we can take a look at the assistance and if we take a look at data you will see that we currently have two assistants and these are exactly those two assistants that you have seen in the play playground before so this is the goal please verify in all your environments that you can successfully query the list of assistants your assistant will not be here yet we have to set up uh the assistant on our own but you should be able to successfully call the API very good everybody can ACC can access the API we got a list of assistants now we can set up our own assistant I will create another region and I will say Define data for assistant now we have to do a little bit of prompt design VAR assistant equals to new assistant I created that one for you and first we have to specify the assistant name in my case that will be raina's assistant obviously in real world um we will not use a magic string here but we will use some configuration uh settings or may maybe we ask the user for the name but hey this is just playing around with this stuff so it should be okay next will be a description of the assistant please think of what your assistant should do in my case for the first sample that we are going to do we will not start immediately with fundraising example I will say um assistant for suggesting um what to do in Vienna possibility next one is the model okay which model do we want to use now it's kind of funny GitHub co-pilot suggest the D Vinci model why does it suggest an artist here well that's the old version of the model name of open AI when kitab co-pilot was trained in 2021 DaVinci was the state-of-the-art model but nowadays it's very much outdated so we are not going to use the model here let's check what the model is that we can create here or can use here open AI provides a modal overview page you can get it from the documentation here and if we take a look we see all the different models which are currently supported in the deprecated part you will see old models and somewhere in there we should see the Vinci here you see the Vinci it's that's an old one don't use this one it's outdated the interesting ones that we are going to use as of today that's December 2023 we are going to use GPT for turbo and GPT 35 turbo to be more specific if we take a look at the detailed versions We are going to use GPT 4 version 1106 preview and we are going to use gp35 also version 1106 as developers you have to be more specific than as end users as end users many end users just talk about GPT 4 or gp35 but as Developers we know it we know it better we know that there are multiple different versions than that you can use and here you see there is for instance an older version of gp4 which is 0613 and so on and so on you can read the documentation on your own they have very different properties and also very different pricing the context window of gp4 is pretty impressive with 128,000 tokens for a context window um and the response I think is limited to 4,000 tokens something like this uh but still it is pretty um uh pretty impressive what gp4 turbo can do okay so in our case in order to save a little bit of money I'm going to start with the gp35 turbo 1106 model we can switch to GPT 4 later so this is what we are going to specify here so now you know if you program with open AI where to look for the current model versions and for the exact specifications of the model they are documented uh in open ai's API documentation good next we need to specify the prompt and we can use uh the the multi-line constants here this assistant helps you to find out what to do in Vienna um yeah that's that's possible let's make it a little bit more interesting you are a helpful assistant suggesting what to do in vien answers answer always in German um end every sentence with o you can leave that out we'll see whether gp35 understands the Austrian language okay last but not least we can specify the tools that we have we will need the tools here when we specify that we have some internal function calls this is the function calling part but for our very simple sample we do not need tools at the very beginning so that would be a typical setup of an an assistant in open AI give it a name select the version of the model and then do prompt design and don't underestimate how much work it is to come up with a good system prompt it's especially true for GPT 35 in GPT 35 you have to experiment quite a while until you come up with a prompt which works um properly I highly recommend that you do not do the cromp design here in this API stuff go to the uh go to the playground and try it here so I can for you because you probably don't have access to this playground currently only a few of you will have that um I will simply try this prompt and we will see whether this prompt makes sense so I copy these files down here I will save it I take the GPT 35 clear everything thing um I am in VI overnight what should I do today in the evening first we will see whether gp35 follows our guidance that it has to answer in German ah well maybe I have overdone it with the order but I think you get the idea right it understood the prompt and now you understand what this instruction is all about we can tell J GPT how it should behave what it should do and if you take a look at the logs you will see that this is what we have to do first we have to post it we have to um what what is that oh sorry I have to scroll up we have to create the thread we have to add the message we have to run the thread then we have to wait until the Run status is completed then we have to get the messages and then we will get hopefully an answer okay good so again do prompt design using the tools that open Ai and or Azure provide awesome so now we can write the code which is required to update or create the assistant region update or create the assistant end region how can we do that well let's simply say if existing assistant is null okay if the existant assistant is null then we know that the the assistant does not exist yet okay that is good so we need to create the assistant so we need to create it and that is pretty simple well maybe let's print the status message let's say creating assistant that might be useful when we run our program and debug it and let's see what um giup co-pilot suggests it suggests something which makes sense okay VAR result equals to well I'm going to call it response but in general that is okay yep that that is pretty okay I think good so that would create our assistant and we can immediately try that one so let's run the app. net run and let's see what's going to happen very likely we will end up having a lot of assistance please do not run your application 100 times because then we will end up with 100 assistance and that would be kind of boring run it only once and that should give us in the playground a bunch of assistants let's see whether it worked okay I already have raina's assistant let's see whether you run your apps too at least some of you should run it ah I see Bill's assistant and we have many other assistants probably in a few seconds so oops sorry it worked we can interact with the open AI API and we can create assistance you have a question please nice so we have solved all our tiny little problems and we can complete our application uh what should we do if the assistant already exists let's say else well if the assistant already exists we have to check whether something changed so we can say if assistant that's the one that we have defined in our application name not equal to existing assistant. name yeah and then we can hopefully rely on kitup co-pilot to uh generate the rest description not equal description model not equal model instructions not equal instructions and with that I think we should be pretty okay okay if something has changed we need to update the assistant okay so conso right line updating assistant and here we will see something pretty interesting if I would ask you in an exam which HTP verb are you going to use for updating a record post patch put delete or get what would be your answer H put or patch exactly and guess what open AI did they were not in my class they are using post here um my my guess why they are using post it makes sense that they are using post here is that uh patch and put requests are sometimes in some very restricted environments they are simply not allowed that I've seen customer networks where they only allow get and post they don't allow any other um HTTP verb and I guess this might be the reason why they are using post here strictly speaking if you follow the the HTTP standard by the book if you take a look at rest dpis they typically use put in patch for the these things but open AI made a different decision here we can we can think it's good or not good it is like it is we have to follow their API design we have to use a post here nevertheless here in this school we follow the principle of put and Patch so when it comes to your final exam to your matura and you are uh asked to design an API please use put or patch but you have seen in practice sometimes people use different words that you might expect good so now that we know that we can update using post it shouldn't be a big problem we can essentially use exactly what we used before here but this time we have to specify the existent API in the past so let's use a dollar sign here and here we are going to use the existing assistant ID because if you take a look at the documentation you will see that the assistant ID that already exists is part of the URL exactly like we have done it in the past good nice and what happens if everything is the same well then we don't need to do anything so we will simply write assistant is up to date so that we know that we didn't have to do anything now of course we need to analy whether the update or insert operation was successful and we can do that by taking this response variable and move it outside of the if statement so we can say this is an HTTP response message let's call it response let's set it to null at the beginning and let's remove that that one here and let's remove that one here so if we make an insert or we make an update then response will have a value if the assistant is already up to date everything is fine we don't need to do anything good so after this is if statement we can say if response not equals null we can check some things well the first thing that we can do is we can say response question mark do ensure success status code this small little helper function will help us to very easily verify whether the result was a 200 Response Code 2001 created 200 okay something like this if the Response Code was a 400 Response Code something like not found or bad request or a 500 Response Code internal server error this will throw an exception okay so if you just want to find out hey everything should be okay make sure that everything was okay you can simply call Ure success status code good now let's read the assistant existing assistant equals to await response content read from Jason async uh yep that is what we can do the only thing is here now we can use a um oh we have to check here so we don't need these things here yep should be good like that mhm and with that we should be okay maybe at the end we can write the assistant ID on the screen to make sure that we see it if you managed to write this code I will show it to you in a second let me quickly try it you can run your app and in my case it should now say not creating assistant but assistant is already up to date and I get me my assistant ID you can verify whether it works by changing a little bit for instance in my case I will say end last sentence of your response with order because I overdid it a little bit with the O here and if I run that the system should tell me that the assistant has to be updated it does it correctly it exactly like we want to use it I will let the code here on the screen so you can verify that you have the same code in your code base nobody seems to have problems that is very good oh we have two problems let's fix these problems and then do uh our break and after the break we are going to start sending messages uh creating a thread and then sending messages to the thread okay so let's go into the break but before we go into the break let's fix the small little problems that you have so welcome back after the break let's continue just to uh refresh your memories the last thing that we did before the break is we created the assistant or updated the assistant and we talked a little bit about the API design of the open AI API and next we are going to create threads these are the threads um that are persisted so if you have multiple users on your website Who simultaneously chat with your bot then you will have multiple threads going on at the same time you can also save a thread and when a user comes back maybe an hour or a day later logs in you can continue the thread in the past I told you that before in the past you needed to store the messages yourself and now we have the threads API yes question this work with the mesuse yes they use their own API which chat do you mean use okay you you're talking about the regular chat GPT so you are talking [Music] about this thingy here right yeah exactly um they have their own database and they are using their own apis behind the scenes however the new portal that you see here as far as I know is already based on the new API but on a unpublished version of the new API because it supports um it supports streaming but I don't know the technical details I have never uh taken a close look behind the scenes I've never reversed engineer it but this is what I understood from their presentation okay but yes they use their own persistence API but if you want to build your own API in sorry your own UI in front of it then you have to do this storing yourself okay a lot of of companies are to nowadays building their own UI on top of open AI because they want their own user experience or other things mhm good nice nice so let's create a thread region create thread and region this is what we are going to do and let's take a look at the API of the threads here you see creating a thread is very very simple you will essentially send a post request to um open aivi threads here you see the create thread here you see it with an empty body you you can if you want immediately start with a list of messages to begin with so if your API your user interface would like to for instance automatically add some greeting message then you could do that when you create the thread you can immediately support some messages but we will not do that additionally what you can do is you can attach metadata see you can attach metadata to a thread so you can later on identify the thread or maybe read this metadata to understand what is the user you can use the username the first name the last name some metadata which might be useful for this thread so let's create the thread as I told you we can send an empty message to uh create the thread so V new thread response equals to A wait HTTP client post async now post async requires us to specify as you can see it here a request URI and an HTTP content so the URI is pretty simple if we take a look at the URI the URI is simply open AI V1 threads so we can say V1 SL threads and the second thing is we need to specify content well the thing that we can do is we can specify a string content here say this is an empty string content I want to encode um encoding utf8 and we want to send uh application Json now this call here sends an empty body using the post request to the endpoint V1 SL threads okay let's ensure that it was successful by calling the ensure success status code and then we can deserialize the response War you thread by calling the read from Json async and I created the dto for you that's the created thread result the created thread result essentially only contains the ID we are primarily interested in the identifier the primary key of the thread because this is the thing that we are going to use later on therefore I will make my life a little bit easier by defining a thread ID variable and extracting the ID directly here so I have it in a variable okay this does nothing magic currently no artificial intelligence is involved here it's just creating a record in a database behind the scenes nothing more but also nothing less to make sure that we don't leave anything um laying around let's create immediately another region just to clean up delete the thread that is something that that's kind of um let's say garbage colle ction we don't want to overwhelm the system with an endless number of threads just to try it so what I'm going to do is I'm going to say a wait HTTP client. delete async this one and there we are going to use the threads with the thread ID and that's essentially it that's just cleaning up maybe let's write a status message deleting the thread so we create the thread then we will interact with the thread and at the end we Simply Clean Up by deleting the the thread again strictly speaking this wouldn't be absolutely necessary we could also let the thread lay around I have no idea how long open AI will keep the threads uh I didn't look it up in the documentation I don't know if there is an end date to it however I want to be a good citizen so let's delete what we do not need anymore if you got this code give it a try run it it should say assistant is up to date we should see an assistant ID it should say create the thread and delete the thread again oh I forgot the status message here let's say creating thread so that we see what we do my program works I didn't get any um any mistakes and I will also add a status message here just so that we know the thread ID it's kind of interesting to see which IDs we have um I want to make it a little bit more beautiful so I will add a tab character when displaying the IDS something like this let me try it [Music] again hopefully it works Yep this looks good assistant is up to date I get the assistant ID creating thread I get the thread ID deleting the thread it works for me please please try to follow along and let me know if you have any problems no problems that is good so between creating the thread and deleting the thread we are going as Next Step create add a message end region good let's add a message State this message no problem let's take a look at the API and um analyze what um the API how the API Works let's take a look at the create messages as you can see um open AI is really using the restful web API principle in designing the the the path just as we have learned ourselves they are using the threads here as the top level uh criteria then you have to specify the thread ID and then they use messages which you can use to create messages also query messages so the thread ID is an U mandatory parameter we have to specify that and then in the request body we have to tell the system what is the role currently only user is supported so we can only create user responses okay we can also only say the user has entered this and that then we have to enter the content of the message that's just a string and what we also could do if we use code interpreter or retrieval we could upload a file now this functionality is important when you want to rebuild jet gpt's um attachment functionality you see if you use chat GPT you can type in a message here we know that this is what we are going to do but you can also upload a file together with the text that you just entered so if I click on this paperclip here I will get an open dialogue and I can select any file that I would like to upload if you want to rebuild this functionality you have to use the oops sorry you have to use the file IDs here there is a dedicated API for files I think it is the message file object you see there is a dedicated file uh API with which you can upload files right I think it is also here create assistant files we talked about this with the assistants and now we have the file upload with the messages but we don't care for our example for our example we do not need to update files upload files that would um be a little bit too much for the amount of time that we have today so all that we need to specify is the role which is user and the content essentially the role is fixed it's always user so the only thing that we have to specify is the content good let's do excel exactly that so we say our new message uh response equals to A wait HTTP client and so on let's see V1 threads thread ID messages that looks good I will put that in the next line and here we specify the new uh create thread message and here you can now specify any message input that you would like to specify in in my case I am doing an assistant about sightseeing in Vienna uh so I will tell the system hi I will stay uh overnight in Vienna uh what could I do for the next uh where can I find the best parties why not by the uh by the way I love music very much we'll see we'll see what the system will tell us you get the idea right that's the analogy to entering um data in the user interface but this time we will do it using the new beta API once we sent that one we can of course make sure that uh the Su that the request was successful by simply calling Ure success status code again it doesn't make a lot of sense to run this now because it will not do anything again this is not involving any kind of open AI large language model this is just adding a record to an internal database that we do not see in order to activate open AI chat GPT we need to create a run and then wait until this run reaches the completed State okay so it doesn't make sense to run this program now if it is syntactically correct we can continue good so let's add a new region and let's call it I don't know create R let me point out something which is very important here in this example we simply write the whole process in a single file because we are just experimenting in real life you would obviously separate that into multiple classes and files and you would have a lot more error handling and things like that but I think you got the point okay we only have four hours so we want to keep it we want to focus on what's really interesting today so let's specify a status message and try to take a look at the Run statement at the Run API so let's take a look at the runs here are the runs and here we can create the run now the parameters for the Run are pretty simple they are first the thread ID this is the list of messages that we want to the the chat GPT system to process we need to specify which ID uh which which assistant should process the messages this is kind of interesting listen closely because this is an important concept here of the API we created the assistant at the very beginning of the process but we didn't assign the assistant to the thread the assistant ID is specified when starting the run so what we can do is we can switch the assistant depending on what we want what we want as a result so we can have a chat history and we can start a run with model 3.5 and we can use exactly the same thread to use a second run with model 4 point with model four turbo and then for instance compare the results so assigning the assistant is not a property of the thread it's a property of the Run got it threats are not linked to assistants an assistant is independent of the threat yes question if you open like the one chat you have you ask two different questions you can get two different like two different assistance yes so the answer is fitted to the question yes yes of course but both different assistants follow their instruction you can give them different instructions so if you need for your application to distinguish in this situation I want to have the intelligence of GPT 4 and I'm willing to pay more and in this situation I want the intelligence of 3.5 and I want to pay less you can do that okay please keep in mind all these apis are subject to change they are beta so it is possible that open AI decides to change these things but currently it is like it is you can also specify or let's say override the model this is also interesting you can say I want to use an existing assistant but please do not use the model which I specified at the assistant level but I want to override the model so if you want to do experiments where you use the same assistant but one time with this model and one time with this model you can absolutely do that the important thing here is that this is optional if you do not want to override the model simply do not specify a model and then the system will take the model out of the assistant okay got it good the same is true for instructions the same is true for tools you can override the settings of the assistant per run not per message per run good very nice I simplified the uh creation the API very much because in our situation we simply assume that we do not want to override any features of the assistant so I simplifi the dto but by looking at this um at this data model of the API You Now understand why the nougat package for open AI has not been finished yet it will not be a trivial net API there has to be done quite a lot of thinking how to structure this API um the open AI API got way more powerful and complicated in this new version before it was trivial I would even say trivial now it's a little bit more options and you have to understand more okay let's get let's startop talking let's start running okay so bar new run response equals to um await HTP post s Json threads runs that looks good create run assist yeah uh GitHub co-pilot made it perfect yeah this is exactly what I want let's structure it in two in three different lines so you can see the parameters better so kup co-pilot was uh perfectly capable of generating this call for us not because it understands the API but the open AI API aligns to typical patterns that GitHub co-pilot knows from other rest apis so it was pretty simple to generate the code for a co-pilot another thing that was important is that I used um good naming for my dto for instance here I said there is a dto which is called create run by giving it a good name create and run GitHub co-pilot was capable of automatically suggesting that for creating a run I will use the dto create run you see it's very important to use good names good record names good class names good method names if you want to productively use AI assistance in coding good clear nice now we created The Run of course we need to ensure that everything was okay so let's just call Ure success status code to uh know early that everything is fine and last but not least we need to deserialize the Json to get the Run ID but that shouldn't be a problem we simply serialize the run from the Json and that's it now in new run. ID let's store that in a variable run ID is new run. ID we have the ID of the run and I will print this one out so that we have it on the screen this still isn't very interesting to run because now the run is started but we never see a result so now we have to implement a loop where we regularly check whether the run is already has already been completed got it this is exactly where we are lacking the functionality that was that I was talking before streaming if we would have streaming we wouldn't need to PA we wouldn't need to ask again and again are you done are you done are you done are you done are you finally done we would not need that but currently we need to do that well that is not very complicated let's create a new region and let's wait for the result region wait for completed and region right wait for status completed yeah get up co-pilot will generate waiting for status completed automatically that is good then let's do a while loop while new run do status is not completed okay good let's do that let's again write the console right line come on Console right line um I don't know checking status or something like this checking status and then we need to check the status and we do that by retrieving the run so we have to send a get request with the Run ID and from the result we can extract the status and we have to wait until the status is completed in the in the guide here they describe the process in way more detail if we scroll down you will see that here where is it where is it run the assistant blah blah blah here it is check the Run status this is what we do by default a run goes into cued State you can periodically that's the polling retrieve the run to check its status to see if it has moved to complete it and that's exactly what we want got it if we take a look at the playground we will see exactly the same um where are the logs the logs were gone so let's try it again so that you see it here run thread message run the thread status you see do do you see how the get status was called multiple times here we see that I have to uh to to to change the size a little little bit but I think you get the idea these are the whole uh checks here now let's take a look at the Run status the first status here was in progress see that one and if we take a look we do not have any responses here now if we take a look at the last one we have completed and once we have completed I showed you that before we can get the run and retrieve the messages so we are just waiting until these this thing has been completed if we take a look at the Run steps before uh sorry the this is the status status this is the status here um no the Run steps I'm clicking on the status here where is the status before here is the status before we have the in progress I showed you that already so nothing interesting here good uh by the way are you familiar with these numbers here started at does everybody understand what this number is it's a it's a time it's a time stamp do you recognize the format yeah yeah but it it has a name have you ever heard of the Unix Tim stamp uh there is a this is a very very famous um format so for instance if I enter the the start that created at time here and I converted I will get back December the 14th at 9:30 that's exactly when we tried the excise of course that's GMT Z so you have to keep in mind that we are not at UTC yeah so you have to you have to calculate the time difference here so this format is called Unix timestamp okay of course net is capable of turning Unix timestamps into regular dat times use Google or your favorite AI assistants as assistant to learn more good yep I like that so now we check the status um first before we check the status um maybe we should wait for a second so let's say a wait task delay do you agree if you wait one second and after 1 second we recheck I think that would make sense right let's let's try this one okay then we will retrieve the Run status so we say run response get a sync blah blah blah with the Run ID I think this looks good uh we can even make it a little bit easier we can say get as Json get from Json as and specify directly the run here because this is what we are asking for Run Okay this looks good and maybe print console right Line Print the status on the screen so we can see what the current status is and the system will automatically exit the loop once the status has reached completed and now it makes sense to run the program let's try it I will try it on my side and you can try it on your side too it will first update the assistant create the thread creating the run you see checking status in progress completed done but why doesn't it stop because I made a mistake I'm very sorry this should of course be neon equals to Neon but you already saw that we are reaching status completed so let's run it again to see whether it now ends correctly adding message creating run status in progress checking status deleting thread everything is fine please make sure that the program works we don't print the result on the screen yet but um we essentially see that the run really completes okay before we move on let me make a note regarding security please keep in mind that the open AI API does not distinguish between users if you build a website for instance with um with an assistant and the end users will type in in personal information regarding the questions then everybody who has access to the open AI key can retrieve the messages that the users have typed in Imagine don't do that okay because that is p but imagine you are creating a pot where people can ask for assistance if they are sick that means that people will probably enter their health status that is critical personal identifiable information you are essentially storing health related data in a database at open Ai and I can query all the messages of all my users by using a single AI key a single API key you get the idea that's very relevant for data security that's very relevant for the gdpr for the D shoots K so keep that in mind you need to add an extra layer of security to make sure on an application Level that one user cannot somehow access the messages of another user and if you screw up if you accidentally print the message history of user a although user B signed on you have a critical security vulnerability this is why most of the public apis and most of the public websites that use AI always have a disclaimer at the bottom where they say Do not share personal information like your name or social insurance number or something like this to this AI assistant because this data will be stored previously in the old API this was not the case the old API you had to send the entire message history with every API call so open AI did not store this data but now open API acts as a database and as you might know the gdpr has very strict regulations when it comes to the location of storing personal identifiable information pii uh this is an important uh English term you should definitely remember it again pii personal identifiable information person okay so keep that in mind this is important that is why many of my customers are not using open AI they're using Microsoft Azure because in Azure you can deploy the open AI models but you can choose the data center you would like to run it in so you can Define that your open AI model should run in Switzerland for instance and that's a better situation when it comes to GPR got it so keep that in mind that is something that you as a developer has to have to consider nice once we reached the completed State we can Now list the messages and print the messages on the screen let's do that okay once we got the uh the the status to complete it let's create a new region region print the message result and region so print a a status message let's call it listing messages of the thread something like this I think that is okay and then we can uh get the messages let's take a look at the message Api so that you understand what we are doing we are currently here messages list messages this is the API that we are now using retrieve a list of messages for a given thread good so VAR messages equals to yep that looks pretty good what uh GitHub co-pilot suggests here okay let's iterate over the messages for each our m in messages what sorry messages. data um that's maybe an interesting aspect here um take a look the open AI API always puts lists if you get a list of arrays back always puts it into a dedicated uh property which is called Data so the list is not returned as a top level element but it is inside the property data that's a design decision that open AI made so we need to access the data here otherwise we don't get the list of messages inside the messages we can have multiple pieces of content so we need another loop here for each our C in message content uh messages messages um no M do content this is what I want exactly and then we can write it so let's say um two tabs and maybe specify here C no the the role so that we see whether it was a message from the user or a message from the assistant and then print the status text and with that it should work so let's check what the API generates Rady in progress in progress completed awesome the assistant in my case said hey Indiana keep this I like that response I like that response and here you see that the message history included the original message that we typed in and it includes the response of the assistant I will give you a second to try that on your own and of course you can experiment with uh some uh some instructions and some some questions feel free to do so in real life we would now create a loop on top of it so that the user can type something in get a response from the assistant type something in get another response of the assistant but this does not add any value to the general exercise that we do here because you know how a loop works and it is just a a repetition of what we did in the past few minutes just calling the same apis in a loop so I'm not planning to do that together with you if that is interesting for you you can definitely try that at home but unfortunately you have to deploy your own chat GPT environment either you are a cat GPT plus user or you use for instance your Azure credits to deploy cat GPT in your personal Azure space as you know as students you can get $100 per school year aure credit and with this a credits you can of course play around with open AI okay good I would say we we have three minutes left in this hour but I don't want to start with a new topic we have one hour left and that is perfect that was my goal because in this final hour we want to add function calling so after the break we are going to enhance our code and add a function to our assistant and then we will check how we can process a function call okay this is what we are going to try but let's postpone that after the break because we just have 3 minutes left and it doesn't pay off to start now so enjoy the break 3 minutes more break and after the break we are going to continue with function calling welcome back from the break I think we should continue so that we see a little bit about function calling as promised if we want to get started with function calling we first need to specify which functions we allow and here I would like to save you a little bit of typing so what I did in the sample that I prepared for today is I have set up a sample assistance that you see in the link that I gave you in the first lesson on line 40 please take a look at the link that you got in Discord and it's line 40 until 73 of course you can customize the prompt to your needs if you want but I will simply take this one and then we will take a closer look on this prompt um the assistant name was I think rainer's assistant and the model now I was too fast in replacing that yeah it was raina's assistant and it was gp35 turbo and the rest will be replaced and then we move raina's assistant here and we move the gp35 turbo here and with that it should be fine nearly oh here now we are fine now let me show you what I did in this example here I specified the prompt you are a helpful assistant supporting people doing fundraising by visiting people in their Community just to refresh your mind and also for the people who are sick today and maybe watch the video later last week what we did is we created did this sample this fundraiser sample and we checked in this sample here in GitHub in this example here this was this fundraising example the idea was that fundraisers like for instance the F fire where the fire brigade Go Walks from door to door door and asks for donations that was the idea and in our case we said we want to have an AI that can extract the relevant data for storing a visit out of a textual information so when the user says I just visited B in leing family Str and they were at home it should intelligently extract the street name the town name the house number the family name and the information whether the family was at home from this text this is the use case scenario that we want to fulfill here so the prompt supports such fundraisers fundraisers will tell you about which households they visited Town name street name house number family name additionally they will tell you whether they met someone or not fundraising happens in Austria so town and street names are in German additionally I told the system try to identify the necessary data about the household and the lag whether someone was met or not ask the fundraiser questions until you have all the necessary data once you have the data and now comes the important part call the function store visit with the data as parameter then we have to specify the store visit function we have to understand that this function is not a web API it's just a logical description what our host what our application can do you have to provide the logic in order to execute this function now let's take a look at the API design for function calling because that is particularly interesting how they did it let's take a look at the tools section I already told you about the code interpretor tool we will not use that today I already told you about retrieval tool we are not going to use that today this is where we are interested in the function tool I will provide a function and what you see here is you see that we have to specify a type which says function and then we specify the function with a description and name and a set of parameters the parameters must and that is the interesting thing that I want you to remember here the parameters have to be specified in Json schema you are already familiar with Json schema Json schema is the object description that you know from Swagger do you have Swagger files a little bit in your mind at the end of each Swagger file the data structures are described what types do you have what properties do they have which type does the property have and so on that is Jason's schema and as you can see open AI makes heavy use of Jason's schema they have built the entire API on top of Json schema so if you take a look at what we do here we specify the Json schema for the parameters we have to specify what is the parameter name we have to specify what is the parameter type these are JavaScript types Json so you have string and number and Boolean things like that and then you have to enter a description of of the parameter name normally you do not care about comments very much but here it is super important because the description will be the basis for the understanding of the large language model got it okay so Town name street name house number family name successfully visited and then you have to also specify which parameters are required does that make sense to you so we tell chat GPT that there is a functionality store with it that it can call or to be more specific that it can ask us to call on his behalf on its behalf got it questions so far nope very nice so in order to get something meaningful we of course have to change the uh the the message here I will say um I just visited the family Miller m in L [Music] in I don't know 42 they were at home now this would be a meaningful task for the AI it needs to understand whoops it needs to understand that this is the family name it needs to understand that this is the town name it needs to understand that this is the street name it needs to understand that this is the house number and it needs to derive the value true for successful visit from the sentence they were at home got it that's a non-trivial problem let's see how how the AI um how the AI works you know what we are going to do and that is a best practice do not try that in your API immediately go to the playground let's try the playground first to see whether this stuff happens because what we need to do here we need to um do some prompt design however setting up this function in the playground is not a trivial task so I will do the following I will simply here at the end set a break point just watch okay you can try it in a second I will run the code in the debugger so that it will update my assistant okay it runs it runs it runs and yep continue running come on continue running continue running okay ah I have the wrong break point here sorry okay updating assistant was correct now let's go back here into the playground reload the playground and select the assistant rainers assistant where do we have it here we have it and in the assistant you now see that we have a function see this is what we just created using our API and if we take a closer look at the function you will see exactly the function specification as we had it in our in our application and this is Jason schema this is a Jason schema description of the properties and the name of the functionality okay so that sounds pretty nice so let's copy the message here and let's try that one let's clear it hide the logs enter this message and see what what's going to happen system works works works works works and hopefully we exactly exactly that is what we want to have do you see in this case we now do not get an answer from the AI but the AI tells us that it would like to store the visit it correctly identified the town name Lin the street name ding the house number 42 the family name Müller and successfully visited is true if we take a look at the logs we will see that the Run status here the last run no this is the wrong one what this is the last run status the last run status is now not completed but the last run status is requires action so jgpt tells us hey host I cannot complete this run I need action from you I need you to call this method for me and tell me what the result of this method was so in theory we will probably not have enough time to do that um I can specify here um success true something like this that could be the response I have no idea why chat GPT tries to call the method twice whatever submit and hopefully hopefully come on submit it okay it crashed good cancel run try it again so let me try it again maybe I did something wrong good press run ah okay obviously I screwed up maybe I clicked the button twice or whatever now we can say success true submit see continuing the run and now the assistant says the visit to the Miller family in lining ofi 42 was successfully recorded woohoo that's exactly what we wanted this is how function calling works so instead of waiting until the runs say completed we have to check whether we need to do something for the AI then we have to perform the action and then we have to submit the function response in order to get the response of the AI okay however now I want to demonstrate something which is important I will now be a little bit more stupid as a user I would just say I just visited the mullers What would you expect the AP the AI to do well probably it should ask us okay where do they live see it hallucinates 3.5 turbo says hey I know the MERS they live in hopas in Vienna and we visited that that success the the visit was successful that's rubbish that's simply wrong that's a a a clear case of hallucination and this is where 3.5 is not very good so we would need to spend quite a bit of time to fine tune the instructions so that it does what it should do however if we cancel that run and clear everything and switch from GPT 35 turbo to GPT 4 save this this thing and enter again I just visited the mullus Run let's see what gp4 does it takes longer we know that gp4 is small ah see that is what we want to record our visit properly could you please provide me with the town name street name house number of The Miller's household also were you able to meet someone during your visit they live in lint okay we make the ai's life a little bit hard so we only provide the name of the town thank you for that information street name house number and blah blah blah but did you see what gp4 was capable of doing it recognized that lint is a town name because it is pre-trained this is why it's called a GPT a pre-trained model it understands that ly is is a city well maybe if you take the smallest town near lint that nobody has ever heard of that consists of three farmhouses and nothing more maybe it has some hard uh time to recognize the city but if this is a meaningful or a larger City it will probably recognize it correctly um um now we can say um um it was it is in let's say again thing H 42 um they were not at home for instance and if you specify that it hopefully will trigger the function call yep it analyzed the town name the street name the house number um it had problems with the um you see that's an interesting one it couldn't correctly identify the uml for the Miller okay it is like it is uh but it at least understood that Ms is the plural of the family name it would be interesting if we would have specified the instructions in German whether it would have been better in uh extracting the uml here but at the end of the day it is like it is maybe it's also just a display Pro problem we can check that one let's see post the thread run run run that should now here the Run steps data blah blah blah step details tool calls function arguments no it's not just a display glitch it is really inside of the parameters so it seems that the AI function calling has in this situation a problem with o load okay I'm just the messenger don't shoot me good so you understood how it worked right good let's go back to our application and stop the debugging here let's add the functionality let's see how much time we have yes let's add the functionality to here in the while loop I encourage you to now um join me again because the loop needs to be changed it's not just completed we have to say um and not requires action completed means we are done we are done with uh the execution of the Run required action means that we need to do something on behalf of our AI agent when we print the message result here we cannot simply print the message result we have to make sure that we need that that it's a good idea to print the message result because here we are list listening no we are listing the messages but this does not make sense if we are in the required status case so what we need to do here is we need to here use a a switch statement for instance and switch over the new run. status if the status is case completed we can if we want print the messages on the screen so I will use it like that to a break and here the other case Case Case would be requires action and close the curly braces here we need to add additional code do you understand the difference completed means here you have the response of the agent requires action means sorry the agent cannot give you an answer you need to do something on behalf of the agent this was now a little bit of refactoring so I will wait for a few moments so you can catch up I will highlight the things that I did in approximately line 129 you have to add the check for required actions and then you have to add this switch statement and move the existing code into the completed case Additionally you have to add a requires action case and there we are going to try to extract the function call and the parameters that will probably be the last step that we manage to do today but I think the rest should be pretty clear for you once we have the name of the function that we should call and the arguments we can do whatever we want now we can call our internal Entity framework library to to store data in the database or whatever we want I think that is what we try to achieve when we achieve that the rest is your job okay so here requires action in the requires action we need to get the Run steps okay so let's add a status message here let's say uh run requires action retrieving Run details because there we will find the function name that we need to call so let's get the Run um bar run equals two and that should be get from Jason threads thread ID run run ID I think that is good now let's analyze the data model the function name is in run required action submit tool outputs tool calls and Tool calls is a collection this is a very interesting aspect so jat GPT is capable of telling us that we should not just call a single method but it can tell us to call multiple functions at the same time this is a super interesting and new feature here in the API in the old API jgpt was only capable of calling one method after the other but now it can ask us to do certain things in parallel understand what I mean so it will analyze the the conversation and if the conversation results in two function calls it will not do the first one wait do the second one but it will tell us to call both functions at the same time for us for a simple example we only have one function so we simplify that a little bit and we can simply go to the first one and use the function and the name here in order to get rid of the warnings we will add a question mark So to make sure that nothing bad happens same is to for the argument V function arguments blah blah we have here and then we try to print everything on the screen so let's say console right line and the function name is this one oops come on the function name is this one looks good and the function arguments is this one okay so let me check what my configuration currently is okay the function is here and the question that I sent is I visited the family Müller in lints in ding 42 and I am currently on GPT 35 so I will try it now I will try it on my side we'll see the system updates the assistant creates the thread sends the message waiting for completed and requires action retrieving Run details and Run details come on I don't see Run details did I do something wrong retrieving Run details why does print okay I have to debug something this should work I have to set a break point here and run it in the debugger no problem maybe I use something wrong here let's see okay I don't need that break point let's continue running it in progress okay here we are so let's take a look at what we got the Run ID is set with thre ID is set so let's run this one okay let's take a look at the run here inside of the run ah let's get rid of that one inside of the Run we have the required actions submit tools output tool calls here function is the name store visit and the arguments so why didn't it print it let's see F10 F10 function name oh now it works maybe it was a temporary internet glitch but it works like a charm take a look our GPT 35 model correctly extracted the town name the street name the house number and it didn't even screw up the uml I like that very much and now comes the the part where we need to do some code that where we need to do some work on our own so here here you would execute any business logic um you want to execute in this example we just execute in this example we just store the visit using uh EF core in the database but please use your imagination here you can do what whatever you want if you have a robot connected to your to your computer you can move the robot arm depending on commands that you got in through um a spoke to spoken voice for instance you can do whatever you want now you you have the AI assistant analyzing the intent of the user and you can react on it got it please try it I will give you a few moments to catch up and try whether it works as expected I showed you the differences between 35 and four in 35 we would need to spend way more time to analyze whether the prompt leads to the expected result this prompt that I wrote before is not capable of handling more comple more complex scenarios where the user didn't enter a complete Command it immediately calls the function it's too stupid to um ask questions back to the user but I think if you spend a little bit more time you could make it with 35 with with 352 but I consciously created this example in a way that you see that there is a huge difference between 35 and four okay the last step that is missing and we will not find the time today to complete that one um is this last call where where do we have it ah I need to do it here let me show you that success true you can return any kind of Json it doesn't matter yeah it's I have a timeout so I need to clear that one and try it again come on ah okay cancel the Run clear that one so now I can try it again here we are run it okay let's submit the result success true submitted the the last step that we will not make today is this one submit the function response once you executed the function you need to inform the agent what the result of the function was if it is a function that retrieves data then it would be for instance the name of a customer or whatever whatever you want to do the temperature in a room the the AI will then take the result and generate a written response on top of the result you could create a a function which says get weather in city and then you need to return the current temperature the wind the wind direction the Sun and whatever you can respond with whatever you want so you need to use a post statement to post the result of the function to the agent and based on that the agent will respond with some text the end user will never see this function called This is a Comm communication a conversation that happens behind the scenes between your program and open AI got it good nice yeah and and then you can of course continue create a loop on top of it and I think you got the point nice I am super happy that this worked out I hope it was interesting for you to see how the API Works um it was a a pretty nice use case for rest apis that is very relevant for your upcoming final exam and I think with that we can stop the the coding and maybe you can spend the last 10 minutes that we have discussing a little bit for possible use cases or we will see but that's it for the coding

Channel: Rainer Stropek

Views: 2,731

Keywords:

Id: OpBWa5SVSvI

Length: 134min 40sec (8080 seconds)

Published: Fri Dec 15 2023