Log & Monitor Everything in Azure Open AI with API Management Service

hey guess what I have been vlogging and monitoring all your questions from GPD models on Azure open AI wait what you cannot just read my prompts oh but I did and I found your very interesting questions like how do I delete my digital twin oh that one was for a friend of mine asking you know he got some conflict with his digital twin [Music] um how do I crunchy mgmg stop it please how to rock your hairstyle after using hay removal cream instead of shampoo [Music] that was asking for a friend to write uh yeah [Music] take off your cap take off your cap by the way how did you block all these prompts inputs and outputs log analytics doesn't support it well I added API Management on Azure and now I can beautifully lock everything can you show me how okay then let's go hello everyone this is FG and welcome to another video we all know that in Azure with any resource that you enable you can have logging and monitoring enabled as well similarly for Azure open AI you can start as of now locking any metrics from this service you can see the request coming in the latency of the request so on and so forth but if you want to go one step further to log also the questions that your user are asking through Azure open AI or the output generated by opening models within Azure open AI we have to add some other components to the whole picture like Azure API management and this is what we're going to do exactly in this video to see how we can have a comprehensive logging and monitoring solution for our Azure open AI that not only we can track all those regular metrics that we get from any Azure service but also being able to even monitor all the prompts coming in getting out of the models and even talking utilize per each request so that will help you a lot to get some more insights on what typically your users your company employees are asking from Azure opening like gpt4 GP 3.5 models to help better planning on upcoming large language model applications that you might have or better optimization or scaling Your solution then let's check it out before we start make sure you subscribe and hit the Bell icon so you will get notified for the next video thank you alright this is my Azure open AI service I created before and the main proposed value of the topic of this video are going to discuss is that how we can control monitoring all the logs that's getting generated from this service for giving you an example I want to know uh what kind of questions people are asking from the models deployed in Azure open Ai and what are the model responses so I can let's say aggregate them later to figure out what are the most typical questions that my employees my team members are asking for example right or the total token is being used by each call so sort of making sure that I have full visibility on incoming and outgoing traffic all the way to this open AI models up to all details like even the questions that people are asking well you might say that I can see right now at the left side there's something called diagnostic settings and I can of course see the metrics of this series for example I can see the latency of open AI responses you can see I have me not calling this model too much as today I was doing a couple of tests before recording this video but if I go to diagnostic settings that's the place I kind of start um getting the locks so how I can do it you click on ADD diagnostic settings you click on what specific logs you want to have let's say I want to have all of them and then you say that I want to send these logs to log analytics workspace this is azure service that can host all your logs you can go there and clear your logs and check them out or you want to push these logs to storage I can't even top or partner solution I have already done that you give it a name here like test you click on Save and that's it so I did this before and you can see if I go back to my diagnostic settings there's a test I created and it's going to a specific log analytics workspace if I click on it that's another service like even this one I can send it to multiple log Analytics workspaces but the challenge is this native logs getting generated under this monitoring service by Azure openai doesn't necessarily capture everything for giving you an example it doesn't capture the prompts the user questions and it doesn't even capture the open-air model responses it doesn't capture the total tokens being used within that all request with questions and answers from the models so it's not really that generic enough to capture everything then what is the solution this is exactly what we're going to talk about in this video first of all before I continue I will add the link of all resources that I use for generating some codes or enabling that solution in my Azure portal so we add the link of that reference to Discord Channel and you can find the Discord Channel Link in the video description down below click on it on the reference section of Discord Channel you will see the reference added so as I mentioned the default log logging of azure open AI doesn't necessarily capture everything you can see that it doesn't capture model utilization talk utilization input prompts output of the models but with this solution we're going to talk about not only capture everything also it will capture some stuff that the default Azure opening logging is missing then the question is what is the solution we are referring to this is the solution we're referring to instead of me has number one I am the user I'm gonna call this open AI models on Azure open area and ask my question let's say from gpt4 instead of I directly go to this Azure open AI service or call my Azure open service API to ship my question and get the answer back I want to push my question through API Management Service which is an Azure resource and this guy here as a middleman will call my Azure open AI although there are some virtual network settings and stuff on my question will we manage or secured by firewall settings network appearing we are not going to focus on the network and security portion of the solution today but the main thing we're going to discuss here is adding or introducing you a new service called not a new service something new to the whole picture called API Management on azure so API Management Service First of all let's talk about what this service is regardless of adding this to Azure opening I want to understand what this one is and what's the proposed value of adding this to the whole picture here think about this Azure opinion is maybe just one specific service that you have API out of it you might have machine learning models deployed on Azure ML and each of them they have their own endpoints so you have even more API endpoints there so imagine you're compounding your team in your startup you will have 10 5 20 100 or even thousands of API a points and you wish that there was a way maybe a centralized place that I can have all my apis for other services other tasks in same unified place so I can manage all of them into it this is exactly the place that you're going to be API Management Service on Azure can host all your apis and orchestrate execution of your apis based on the request coming into the series and then this service called your API backends like Azure open AI so the benefits can be security because I can apply rate limit here instead of people bombarding my Azure opening API I can have some rate limits let's say people cannot call my open air model more than 10 times per minute as an example so rate limit IP filtering I can filter people out of that IP range do not coming in call my open AI or key management security is actually one thing monitoring is the key feature that the key reason you're adding API management to Azure opinion with adding monitoring that means when my question from user goes to API management API Management Service will lock my questions from GPT models and the answer of the models when get it from Azure open UI so when I go to the logs of this guy here I can see all the logs for open Ai and also the questions to the models the answers and the total tokens that we discussed about so monitoring is something caching sometimes you have some apis not necessarily Azure Pi sometimes you have some apis that the response of apis doesn't doesn't change that frequent you can cache the results of your previous calls to to make it even a cost efficient you can versionize your apis by API Management Service you can add documentations that you can do cost management here it's like a unified endpoint for all your apis there so lots of benefits again that was a whole high level overview of what is series and the main reason that we're using this with Azure opener is mainly for monitoring for this example okay so now let's go to my Azure portal again so this is my open AI service that I created before let's go back to the overview and now what I'm going to do I want to start creating an Azure API management so how you can do so just simply on the top type API management which I recently opened that it's already here and then just click create and create a new one I'm not going to create that because I created again you can just simply be having a name specified there go ahead and create one for yourself and when you click on it this is what you will see I'm in the overview page of my Azure API management so what I'm going to do here I assume that you already have Azure open AI service created on your Azure portal or Azure tenants if you have not you have to click on Azure open AI or search Azure open AI create one service if you have access to that similar to what we did for API management I created one here and you know that for calling Azure opening there's a key you need to know like for like a password so make sure you copy that you go to Azure open AI if you click on this is the one that I use you go keys and endpoints make sure you grab the key and even the end point save it somewhere we're gonna need that now going back to my API management again now I want to add my Azure open AI as one of the apis in my API Management Service so let's see how we can do so so as we talked for adding an API of azure open AI to the service Azure open AI has a key we just copy that right and I'm going to add that key to API management because API management is going to call that Azure Penny but I'm not going to copy and paste the key manually here so a better practice would be saving that key open AI key as a secret in Azure key Vault and Azure key vault is a service that can host my secret and then I say to API management that hey you want to grab the Azure opening I key go to that key Vault and grab it from there so I'm going to show you how it did it it's optional but it would suggest that certainly follow this so I go to we can just type key vault there you go here's list of the keywords that I have created before I think I have something with the name demo yeah mg demo here I click on secrets I click on generate and you give it a name let's say azure open AI okay it's already exists because I created before recording the video so I click on let's add a random number then enter the secret that you copied from Azure Opening Service that I told you earlier and that's it you click on create you're done so I don't do that because I created before and now I have aoai or Azure opening secret created so now when I go back to my API management I can now add my Azure open AI without being worried about copying the key but I haven't still introduced to this service that there is that key so for doing so go under name values you see that I have created but I'm going to show you how you click on ADD give it a name to this one tree you can have also at display me about it test one two three and the type here you should say that's a key vault and it's going to ask me okay which keyword you're talking about I click on select I say that I have a key Vault with the name I think demo there you go and under that I have a secret created a or Ai and I just showed you how that's it at kittycon this one I click on Save I'm done that's it and you see what happened here I'm not going to save it because I already created and there you go so now I know that where API management can grab the key of my Azure paneer for calling it okay one step done okay now we just added the key but how about the API endpoint by itself is still my API management doesn't know what endpoint we're talking about what what is the backend API which is open AI we're gonna call through API management so you need to go down to the back end and by the way before I go there you remember that the name under name value is the one that we created name is aoai we need that so now go to backend click on back in here you click on ADD now here you have to specify a name for example I want to say this is my response Azure open AI endpoint which is the backend that I'm talking about I already created one so it's going to throw me an error but I'm not going to save it so let me add a number here and here for the URL you have to type https like that and then you have to add the Azure open AI service you created the name of the Azure open AI series mine is testmg okay then dot open AI dot azure dot com slash open AI make sure you add these two and that's it now on their headers you know that when you call this endpoint is a key right and that key yeah their name is API key so type API key here and then on the right side you have to say that I'm not going to give you the circuits here I'm not going to copy and paste that we created that through key Vault and we added that as name values here that's why the name you chose here on name on name values if you remember it was aoai there you go it found it for me that's it so you click on OK and then you click on create again I'm not going to create that because I already did and there you go mine is got created with the runtime URL that we added okay so second is Step done we added name values that was adding The Secret of my opening I key through keyboard and then the back end which is my openai endpoint added also to API management what is next now I'm going to add my API so click on API I already created that and it will be showing that there you go it's the one I created but I'm going to show you how it did so under add API you see there's something called open API you click on it and now you click on full so it's like now here's the place that you're going to Define your open AI API specification or like Swagger you're going to tell that how you're going to call the model how the response is going to get generated how you grab the response what are different parameters when you call the models but you might say that mg is too much I'm not going to add all of them one by one I have to dig into the documentation of open your API but I have good news for you you don't need to manually specify all of it I'm going to give you a file which I grab from a reference that I add to the Discord Channel you grab it from there and that's it so let me show you that link this is the link again I add this to the Discord Channel that link contains a Json file that you need to upload by clicking on select the file here that contains the API specifications of azure opinion you can see it contains all information about the the model type that you're going to choose that those temperature parameters anything that you need to specify in your open AI API to call that it's all here for you so let me select that file there you go it automatically detects a couple of stuff here for me so I just need to click on Create and that's it it got created for me and it figured out oh I have some post in my API because of that API specification Json files and it didn't need to manually specify anything here but there are just some specific parameters we need to add values that's why when you click on Create and it's going to be shown for you under inbound processing click on base then go to code editor this part you need to add it also and that Azure open your endpoint because we want to have that value where it's coming from it's from the backend specification that we have that's the name I use for the backend and now I got it from here that's it just add this one to and click on Save you're good to go oh I close that let me go to my API management again now I have added my API to Azure API management but I haven't told that what I'm gonna lock right so let's go to that API that you create under I think settings let me double check yep there you go you see that I have added subscription required that means there is a subscription key required for calling these open AI API through API management now this is different than Azure open UI Key by the way okay so under the settings there you go under Azure monitor make sure you enable this sample rate put it to 100 percent and make sure the number of payload bytes to maximum which is the number I grabbed from you and then click on Save now what's gonna happen when I call my openai models through API I will call them through API management that I'm going to show you how and then all those inputs that are going to track questions and stuff will go on their logs under API management which is here okay so before we do that let's make sure that our API works I added the API properly so I go to the API again I click on let's create a completion let's chat you click on test and there you go the deployment ID that's going to be a model name let's say let me actually copy that from my reference there you go so the model that I use is g53.5 the API version is this one and actually there's a sample here hey my name is mg let's say this is what I'm gonna ask like from chat GPT which is a weird thing to ask but I just want to see how it works so I'll click on send there you go it was my request and it got to know how to call my Azure open AI service because it grabbed it from that specification or Swagger we uploaded and it sent a request and says hello OMG nice to meet you how can I see you today so I interacted with my Azure open AI service through API management but that was a test if I want to code that here is I I went through my Azure machine learning to just have a place to run my python code you can run it everywhere it doesn't matter right so forget about Azure machine learning here just have this python code I'm calling my Azure open AI service with this model you can parameterize them with this version of the API and I say that hey I want to call this model then the user is that asking this question now can I delete my digital to it and I want to send that request I grab the response and the response is back for me this is for the time that I'm not specifying any key to Azure API management you see that I'm not putting any credentials but a better practice would be enabling the subscription key which I showed you when I was creating the backend API right so if you have that enabled which is by default enabled you need to do the same thing but you need to also add the header something called API management secret key subscription key sorry so I highlighted that I removed that but before that I I also tested out with this key and it works and if you might ask how I grab my subscription key let me show you quickly go back to your API management you see there's a subscription right there you go these are your keys you can click on show hide keys and copy them from here and add it to your code or move them to key Vault or environment variable make it secure then you can call it from there this is the end of the video I'm gonna see how these logs are captured if I can see that weird questions that how can I delete my digital twin stuff are logged or not so I go to my API management I'm not dealing with Azure open AI anymore so in API management I go under logs and there is a well you can have so many queries generated here and and this is a place that you can clear your logs right you have to code to query your logs but the reference link that I'm going to add again for you in Discord Channel gave me some nice um queries examples that are going to use them from the reference to show you what is going to happen so it will go and grab from the logs of API management try to see what chat completion I have locked from Azure open Ai and they have been successful because the Response Code was 200 what models then I can add grab the tokens used the total canceled let's actually run it it's gonna make much more sense so there you go it tells me that from these ipe addresses someone has called g53.5 model with uh prompt tokens 16 14 with the completion of which are the responses of 73 15 and the total which is the the sum of these two numbers with some average total tokens so now I got some info about the total tokens and stuff which I didn't have the model used I didn't have them in Azure opinion and the other query example that I have to capture uh prompt or questions which is this one let me run this again you can grab these queries randomizes from that reference okay there you go now I can see some more stuff prompt text this is what I wanted to know hello mg nice to meet you this is actually the answer of my open-air models to my question so if I want to see what was my question uh I think that was the request yeah and the request hey my name is mg how can I delete my digital remember these are stuff that we asked so it captured them all and even the responses as well which is coming from open air models let me open that up which actually include all the yeah hello mg nice to meet you as an i language I cannot provide instruction how to do the digital twin so that's all so you saw that what is the proposed value of adding API management here as a middle layer for calling our apis here Azure opening as an example you can scale the same idea to any API you have not just Azure open Ai and with doing so you can not only leverage capabilities of API management we just mentioned one of them here which is monitoring more logs we generated but check out API management documentation they're much more capabilities we can have so here we enable that for capturing further logs and make it a unified platform for calling even multiple Azure open your instances for maybe Disaster Recovery or better scalability I hope you enjoyed this overview video I try to show you the art of possibility but it might even get further customized based on the request but at least out of this video you gain this other possibility that it is possible to capture more detailed logs with adding more services further integrated so I'm gonna again add the reference link of the steps that I followed it's a GitHub repo the queries that I copied and all the steps that I took in details and hopefully this video has been helpful let me know if you have any questions or comments and as always I read them thank you so much all great ideas started as a crazy dream David ridiculed doubted laughed at until the event if people think that your ideas are crazy dreaming because you are on the right path so keep dreaming keep practicing and keep working hard on yourself because success is not something you pursue success is something you attract by becoming attractive person so dream big my friends believe in yourself and take action till next video take care [Music] [Music] foreign

Channel: MG

Views: 7,341

Keywords: open ai, machine learning, openai chatbot, openai gpt 3, gpt 3 prompt engineering, open ai chat gtp, Azure, Azure Open AI, ChatGPT in Azure, Open AI in Azure, Open AI in Azure Demo, Azure Open AI demo word embeddings, chat gpt, advanced tutorial chatgpt, chatgpt advanced guide, Customized ChatGPT, Azure Open AI Monitoring, Azure Monitoring, azure log analytics workspace, Azure API Management, Azure Open AI with Azure API Management, Complete monitoring in Azure Open AI

Id: j40RGK5SAeY

Length: 27min 17sec (1637 seconds)

Published: Tue Aug 08 2023