Build an Azure OpenAI powered .NET 8 Chat Bot on your data from scratch | .NET Conf 2023

hello and welcome to netcon I'm excited to show you how to build a chatbot on Azure openai using net8 in this session we will first demonstrate how to make a chatbot work with gbd4 and then we will integrate our data specifically a PDF file and enable the chatbot to answer questions based on the contents of the PDF this way you'll no longer have to read through the pages of TDS text just ask your chatbot first things first in order to get started you will need to go to the Azure portal and create an Azure openai resource if you are a first-time customer you might need to get approval from Azure once the resource is created you will need an API the endpoint and the model creating a model is very straightforward however due to time constraints I won't be going through the entire process of creating an Azure openi resource and model in this session this is one of the models that has already been created and the major things that we need to keep in mind are the keys and endpoints and the model deployments so if you click keys and endpoints you can see you have two different Keys you have the endpoint and the location and the region and then we're going to move to the model deployments once you click manage deployments it should take you to manage all the models you have deployed so far you can see I have a bunch of different models including gp4 32,000 which is 32k model and in order to create a new deployment all you need to do is select one of the models and give it a name of your choice once you click create you should be able to have the resource created within 5 minutes however we're going to skip that step for now Azure openai versus open Azure openai combines the advanced AI capabilities of open aai with the robust security and Enterprise features of Microsoft Azure it offers private networking Regional availability and responsible AI content filtering by co-developing apis with open a Azure open ensures compatibility and a smooth transition between the two platforms that's all I have for slides so let's write some code we will start with using the terminal and creating an MVC project with the name netcon now I'm using visual studio code as well as a Mac but the same code should work on Windows and then of course we will change the directory to netcon and then I will open Visual Studio code now I'm using CI dot because I have two different versions of Visual Studio code but if you have only one version of Visual Studio code you could also say code space Dot and that should pull up your Visual Studio code instance in the same exact directory I would press yes to required uh sorry to build the assets and use Visual Studio to do that and here's a cheat sheet of few different files that we will use as we go in the project and let's talk about these files the first file is the Dov file which is something like this now we're not going to use env. text but um this is just an example file to show you what a EnV or an environment file looks like and as you can see we've got few different values here and we're going to use Caston gp4 model and all all you got to do is add these values your keys your endpoints as well as the model and save it in the EnV file now if you are using just open a and not Azure open a in that case you would only need the key and the model and the model name would be the standard model name like gbd4 32k next we'll go to packages. txt and you can see we only have three dependencies and I like to add all the dependencies in one file and then use a script to make sure that I'm getting all the packages so in this case we have three different packages and as you can see one of them is a beta version so one of the things you may want to do is uh check if by the time you following this there might be an actual release and this is my script in script. RTF and once I use that script I'll be able to download all these packages and you always want to go check CSR to make sure that you got all the packages and the versions you needed thing noteworthy is that itex 7 is version 8 and that's the PDF package we are going to use next we'll go and create a chat controller so I'll copy and paste the home controller and change the name and of course the name of the controller as well as the file I'll also go ahead and remove some of the code that may not be needed and then we'll go to program.cs the first thing I always like to do is make sure that my environment is fully loaded so for that I will use the package netv and using system. environment I'll make sure that I'm able to get the value of the key the endpoint as well as the model loaded now if you have a different file name which is notv you may have to pass the file name in the method however if you use EnV you can just say env. load and then get environment variable and of course the key as in the key value pair in this case that happens to be Azure open AI endpoint and you can always make sure that it actually works so once we print the endpoint after running the project we can be rest assured that all three values are coming in and right there we see Caston AI east. open. azure.com so that's great and it worked now we can copy this code and use it in the controller where we will be creating the functionality for our chatbot of course I need all the three values so I can bring them in the key as well as the model and the endpoint we can also remove um the Privacy code because we don't need it now somehow GitHub co-pilot actually knows that I'm going to create a post method or it got lucky we're going to create an async uh method that will get response and that's just a plain response method and it will take a parameter user message now one thing to keep in mind is that we shouldn't really call it get response but I think that's acceptable for now because it's just demo code otherwise for a post which is HTTP post the name shouldn't be get response so now we're going to we're going to call openi client and create an instance off it and the co-pilot is pretty close to the actual code and now I can pass in the endpoint and the Azure key credential which happens to be key and we'll have to import the package which is using Azure now you can see that somehow the co-pilot thinks I will need a completion request which is also one way I could do this however since it's a chatbot what I like to do is create an instance of chat completions options now chat completion options work for gbd 35 turbo as well as gbd4 and we will pass in a few messages the first one would be a system prompt and that will be chat message with a chat roll of system this is a system prompt and we can say a lot of different things but we'll keep it very simple for the demo one of the things we do when we have our different chatbots is give it a very detailed system prompt for example we're using visual studio code on Mac and make sure the programming language is C and that becomes a default prompt the second one is going to be a user message which could be something very simple like does have you know support for GPD 4 and we can also give it an answer as an assistant because this is just a chat that we're creating and we can give it any answer we like now by default since the training of GPD 4 is dated back to 2021 it usually does not know that gbd4 exists so even the co-pilot may actually think it doesn't exist so as you can see the answer it gives you is says no but it does support gbd3 so that clearly tells us that it doesn't really know that answer which is perfectly okay however we'll go ahead and say Well it actually exists at the same time finally we will add user message which is a variable and this is something we expect the browser to send to us after the user types in the message you could have deleted line number 29 and 30 and this will still work but I just wanted to show you a very simple chat completions option with just messages and Max tokens and this is a number I would say a small number like 400 is good enough because GPD 432k while it's an amazing tool it's also the slowest model when you keep it up to 400 tokens it'll work a little faster than usual now one of the things to keep in mind is I always look at the source code and see what's going on behind the scenes here you can see we have chat completions options and it has a bunch of change tracking list types or even dictionaries and what does that really mean if you've done Entity framework core then you probably are familiar with change tracking and behind the scenes you have and I notify property changed which is a class that helps you track additions modifications and deletions as part of your entities and this is important to understand because behind the scenes you would want some kind of TR change tracking to happen as you can see in this code what you have here is a bunch of messages but these messages could be unending another thing to add is that if the max tokens are let's say just 200 a lot of your answers might be truncated so you know that is something over a period of time once you play with it you're going to realize why the token uh limit is important as well as the max tokens are important now for this particular 30 minute coding session we're not going to get into that the response is of type chat completions and I will use the co-pilot actually the co-pilot didn't get it right this time it's going to be get chat completions async that's because we have an async function and it takes a model as well as chat completions options one thing to remember is it's going to take the deployment name in case of azure open Ai and it usually takes the model name for open Ai and that's it now we have the response and we will take the response value and I think co-pilot got it wrong again this time so we'll change the value to choices do first and then message. content and we can return this as Json and we'll call it response equals bot response next we need to create an index.html in chat folder in views folder and we'll start with creating a div which is going to be an empty chat box where all the chat messages will be appended next we will create an input box with an ID let's say user message this is where we will be passing in our prompt and then we'll create a button that will have a function send message finally we will add the JavaScript we need to make the request happen and in this case the send message will take the value from user message input box and looks like the co-pilot is sp off next we'll make an HP post call to the chat controller and the name of the method is get response and we'll pass in user message and that's not bad at all for the co-pilot now a couple of things we'll we'll just keep it user message for consistency and then we'll use data. response because that's the object that gets returned a small correction this needs to be data. response because that's what we're passing from the API and we should call it cast an AI or the AI chatbot another thing we may also want to add the user message so we have a history of what was asked and what was answered and we'll call that user colon just as a prefix to the message that's going to be passed one more thing to keep in mind is you may notice that data. response is r with with a small R the response is with a small R and here we have you know the capital r and that's because net would automatically take care of it it knows that JavaScript is going to be lowercase on the other end and it's not necessarily lowercase it's camel casing and then of course we can run this application and see how it works next we need to go to the chat controller and type in a question like what is the capital of France and you may have noticed that on purpose I didn't type it correctly but that's something gp35 as well as 4 are capable of handling and we got an error let's look at what the error says oh we forgot the parenthesis and the semicolon we're going to ask the same question again and hope to get an answer this time and that was actually fairly quick so next we should ask it something like what is azure open AI now I asked this question because given its training it should not be able to answer this question correctly and and that is correct as in it does not know what that is in fact it's apologizing for the previous response which was set by us even though it was an assistant response it was actually me coding in that response so it thinks that Azure is a completely different Cloud platform and open AI is a completely independent organization and that's because the training is somewhere around November 2021 now this might change by the time you use use the same model if it has a different training however that tells you why we really need to have our own PDF or our own data work with gbd4 because the training set is quite limited so let's ask it something else that I think it should know which is the total words in all the Shakespeare's works and on purpose Shakespeare was incorrectly spelled but we got an answer which is $884,000 words and I have no way to check if it's true but it's pretty amazing that we got that answer very quickly now one of the another benefits of open AI is that a lot of these models do an amazing job explaining something for example we can ask it explain Docker to a 12year old and let's see what it comes up with Wow Docker is like a magical lunchbox that can hold different meals software applications and all their needed ingredients code files and settings just like how you would carry your lunchbox to school with your favorite meal inside doctor Docker helps developers package everything needed for a software application into a container wow that can be taken anywhere amazing now let's try what is net 8 and uh I'm pretty positive it would not have that answer in its knowledge set and that was definitely expected it doesn't think there is a version of net called net 8 and it's actually trained till net 6 that's also pretty impressive because it's November 2021 now this is exactly why we need our own PDF so why don't we go ahead and build the functionality we need for the same and I'm going to copy and paste the same exact method and make a small change here and call it get response from PDF and I would need to update the UI to make this happen so the UI has been updated to get response from kdf so it calls this particular function we're going to use pretty much the same exact code to create an instance of opening a client pass it endpoint and Azure key credential here's a small observation that this time the co-pilot is showing me chat completions and previously that was not the case co-pilot is learning from my code what kind of code it needs to even prompt and that's pretty cool because I didn't have to copy and paste any of that now that we are here here the thing we need to keep in mind is I have a data folder which has Azure PDF file that's one of the files I had previously copied and pasted to my project and just to have a preview of this file it talks about what's Azure openi service it also has the models available in Azure openi service at least to the date of the documentation and it also has fine-tuning models and you can see there are three of those and what would like to do is ask these kind of questions and say show all the fine-tuning models that you have and how do I get access to Azure openi compare the two and also talk about some of these key Concepts and it's just a four page PDF I mean there's a couple of reasons behind that we are trying to use a small PDF because I wanted to demonstrate that even with small data open AI models do not need any training whatsoever and it's going to work with it it does not require big data and number two we also have a token limit and I would like to keep the PDF under that token limit now usually I do not like static functions but since we only have 30 minutes i' like to create a function called get text which takes a parameter which is the file path for Azure PDF and we're going to hardcode it next we will use PDF document class from itext and we'll create an instance of it and it'll take a PDF reader for that particular file path and then we'll create a string Builder to get all the text in one string Builder with a variable text next we will Loop through all the pages in The PDF document and then append the text together and this code might work and this is something that co-pilot is suggesting however I don't have the time to check if this is going to work so I'm going to take the longer route and write the code that I'm more familiar with so first thing I'll do is create an instance of PDF page and this basically helps us get one page at a time now that we have a PDF page we will use a strategy which is called it text extraction strategy and that's just an interface and we'll create an instance of simple text extraction strategy which is the most popular extraction strategy and it keeps all the text and then of course we'll get the text from that page and store it into current text and then append that text to text now that we have all the text from the PDF file we will now return the text after closing the PDF document and of course we're going to have to do a two string because it was a string Builder next we need to use this particular function inside get response from PDF function and make sure that whatever call we make open AI model is using only this PDF text and that's kind of funny because it I don't know why it thinks I'm on Windows when it's pretty clear that I'm on a Mac but that's the co-pilot code and all we need to do is make sure we have the right path p in here we also need to make sure that we specify the PDF text as a variable and that variable is passed inside a chat message so first we can move couple of these lines that we don't really need and here is where we can specify that the following information is going to come from that particular PDF text and then of course pass that entire text here now one thing that we would not get into is token limits I'm using 32,000 tokens so I should be good with a fullpage PDF but pretty soon you'll run into token limits and that's something I would highly recommend you investigate or use aure search or some other tool to help you there now we'll move the max chus to 1,000 and we'll have temperature equals zero and this is something that's very important to understand temperature equals to zero makes it become more factual and temperature equals to 1 or anything close to one makes it more creative which is where people complain about hallucinations is actually being more creative temperature equals to zero does not mean that hallucinations will completely go away but it reduces it to the bare minimum and that's all we can control so if you want U creativity it will make up things on its own and more hallucinations you can go to one and temperature equals to zero means very factual and it's going to try its best to stay there you can also add a system prompt that could be more detailed and put a lot of conditions and restrictions on it where it does not make up things and those are definitely some of the techniques you can use finally it's the same exact response that we had previously and co-pilot pretty much understands the response and I can use the deployment model and then of course return the B response exactly like I did last time and then return the Json response pretty much the same exact way let's run the application again and hope it works we'll go back to the chat controller and refresh and ask it the same exact question that we asked previously except we have a 500 error and the error seems to be Ah that's the path I think it should work now we'll ask the same exact question again and we're hoping that this time it read the PDF and is giving us a different answer so I don't see a 500 so looks like it's working it might take some time oh wow we got a very different answer Azure open a service is a platform that provides access to open ai's powerful language models including gbd4 gbd 35 turbo and embeddings model series not only that it also talks about the key features of azure open now this is amazing so we have an answer that is not only right it's actually very succinct and that's something that gbd4 is known for by the way notice I did not type which correctly but the answer is perfect it's the same exact three models that we saw previously that happened to be fine tune and again it's a very brief answer gets the job done and is actually pretty quick so now why don't we ask it something like comparing open ey with Azure open ey which previously it thought doesn't even exist that seems like a pretty good comparison it's talking about how both of them have access to powerful language models but there are also some key differences and it also highlights them like platform security and compliance and other things and what I like about this is it's got Microsoft entry ID this is something that was just released or the name change happened to entra just a few months ago so we know for sure that it's going outside its knowledge set and it's actually reading the PDF and giving us the answers and that's pretty incredible given we did not train it at all and this is a great start but this approach has limitations to overcome these limitations I recommend employing the rag pattern which is retrieval augmented generation pattern and use Azure search along with the database of your choice is something like Azure Cosmos DP Azure search or Azure AI search has Vector capabilities and also provides a hybrid search where you could use the same capability that Microsoft Bing has by combining these Solutions users can experience enhanced information retrieval accuracy and relevance effectively addressing the challenges posed by token limitations in AI assistance I trust you found my present ation insightful and engaging should you wish to stay in touch please don't hesitate to connect with me on LinkedIn or follow my company Casten for further updates and insights thank you and have a great day

Channel: dotnet

Views: 5,871

Keywords: .NET

Id: fYJuokUnucE

Length: 29min 40sec (1780 seconds)

Published: Thu Nov 16 2023