Making Enterprise GPT Real with Azure Cognitive Search and Azure OpenAI Service

you're not going to want to miss this episode of the AI sure we talk all about making Enterprise GPT Reel with Azure cognitive search and Azure open AI service with my friend Liam Kavanaugh make sure you tune in [Music] hello and welcome to this episode of the AI show where we're talking all about making Enterprise TBT real with Azure cognitive search and Azure open AI service with my friend Liam Liam it's been a while since you've been on the show my friend it has been a while but hopefully I can make it up with a couple of cool things to talk about today well anytime there's GPT and stuff and search you know I'm super excited Liam for those who don't know you why don't you tell us who you are what you do absolutely hi everyone my name is Liam Kavanagh I am the group product manager for cognitive search and I I lead the product group building edit search and chat experiences or organizational data I I love it so I unders because we've we've we've talked about cognitive search and I I even did full demos with skills and it's the greatest thing ever since sliced bread but now you're telling me about GPT stuff and when people hear gbt they're being Chad GPT so can you tell us a little bit how you're kind of putting these two things together yeah absolutely in fact Seth I I have a demo do you mind if I switch over to the demo so we can kind of walk through that let's go to the screen right here here it is cool so in this demo what I'm going to show you is basically what we've heard from customers ever since chachyb came out what we heard is that they love the experience they love the ability to be able to interact with it in a natural way but we heard what people really wanted to be able to do at least in organizations is be able to do that over their data using their own Enterprise data to be able to interact through that and be able to ask natural questions and so when you ask about search that's where we tie this in is that search is such a great place to be able to retrieve useful information that we can then use as part of the chat gbd experience so can I ask a question though because this is and this is important because I want I want to make sure we're understanding correctly are you getting customer data and putting and like redoing the model or is it something different than that no that's that's a great question because you are absolutely not taking Enterprise or organizational data and injecting it into a model that can actually be you know in many cases with the amount of content organizations very costly very time consuming so a pattern that we're we're using is called reg retrieval augmented generation and what that means is that we're retrieving information from the search service all that information is is stored within an organization's search service that they own and it retrieves that and then it augments the answer using the open AI Azure open AI model I see well I'd love to see this in practice cool so why don't I actually walk through an example in this demonstration will actually um share a link to where you can actually download the code to reproduce this yourself but what I want to do is show you an example where imagine if you will you are in an organization and you wanted to be able to take all of your HR documents such as this one um which talks about the employee handbook we have this which is you know one of the many health plans from a fictitious company called contoso and imagine you wanted to be able to use that information but be able to have a chat experience over it so I'm going to go back into this information or into this application and by the way we have used Azure cognitive search to ingest all of that content so that it's searchable so let me choose one of these example questions is such as what's included in my health plan and what's happening right now is the first stage is it's going cognitive search and it's trying to retrieve for that rag pattern I'm trying to retrieve potentially useful information that it then injects into Azure open AI as a prompt and tries to answer my question and as you can see here it I think it did pretty well yeah yeah and that's cool thanks and so if we see here there's a couple of things that I really want to highlight first there is no hallucination there was nothing where the model made up an answer it was told to only answer if you can find it in that content and let me actually even click on one of these citations where you can actually go and View and actually validate that it's correct and that allows us to tie these together Seth that's cool and I think the cool part about this is if I understood this right and I want to make sure that I'm understanding you're you're basically taking the query that the person or the chat you know message and you're searching an Azure cognitive search for Stuff related to that message and then you're putting that text into the prompt that then the large language model in this case chat GPT or gbd35 Turbo uses in the prompt to generate a grounded answer and not just make stuff up you got it in fact we can actually see this we have in this this demo we have a developer mode so you can actually see this so if I click on the supporting content you can see what the search service had brought back as potentially useful Snippets of information which we then injected into a prompt what we call an uh Azure open AI prompt that explains to the model what to do and it also says you know I want you to only use these sources of information so it allows me to to do that and in fact I can just continue on and just say you know what a better Dental right and I can ask that to further discuss this and you'll notice a couple of really cool things here is that not only did it answer my question but I didn't re-ask that about my plan it just remembered from the context that I have a Northwood and Health Plus plan and it can continue that on through that this discussion that's interesting how does it remember what you said before because the way I understood these things is you're just giving it a prompt and it returns text how does it how are you retaining the conversation that that's one of the great Parts about the chat GPT model is that we can actually continue to keep previous information previous discussions into that prompt so that it continues continues forward so that the context from before continues into the the further discussions so you're you're basically this is cool because it gets into this notion of prompt engineering but with rag the rag pattern where you're retrieving information to augment The Prompt you're you're putting the messages in there because the chat gbt model is very good at you know conversations and then it's giving you answers grounded in the data plus what the person is asking exactly that's cool so is this hard to set up though it it's really not and in fact one of the things let me show you one other example and then I'll show you the uh how this is all backed and how you how easy you can deploy all this through some of the code so let me just clear this out and I'm going to ask a different question I'm going to say can you tell me about performance reviews and while it's doing that I'm actually gonna take that question and I'm going to go in and I'm going to try it in a different way I'm going to go into the Bing Translator and I'm just going to paste in that question sorry I'm going to paste it in and copy it in Spanish Let me take that exact same question and put it in so I'm going to ask the same question in here now what you'll see in a second is that this comes back and what it'll do is it will answer my question in Spanish even though none of the content is in Spanish it was able to to handle that and so to back to your question Seth what we've done is we've provided this GitHub repo and I want to show you something within this GitHub repo we have the prompt that backs this and I'm going to show you just one sentence that we added to the prompt we said if the question is not in English I want you to translate it and I want you to then answer it so you all all of a sudden have a multi-language support with a single sentence that you added to your prompt I hopefully you're seeing oh now you're seeing my face it's basically like because we've been people Enterprises have been one of the things that I've seen is that it's like oh chat gbt is answering things incorrectly well of course it was trained on data that was from before now uh number one number two there's some data that it doesn't it never looked at like what's our contoso employee handbook because if for those of you that may not know contoso is a fictitious company uh so it probably didn't crawl over documents about contoso even though it may have they've always been asking about how can I make it answer questions about my stuff and now what you're saying is with rag or retrieval one's generation specifically with cognitive search you're able to pull stuff back and ground the model and even give it instructions in English to be like and by the way you should translate this and it's that's bananas it's really fun to play with Seth and it's really great a fun fact that we had of this is that all of this content you mentioned contoso being a fictitious company this that what the health plan all the documents purely generated using Azure open AI we just asked it to create content it created over 100 Pages for a health plan it was amazing so super super fun to work with oh my goodness it feels like as programmers we're cheating nowadays it's like this has gotta be cheating it's like the fast mode of this uh but this is really cool what's the reaction Min are there any other details that you'd like to share yeah this is this has been a really popular uh tool for us you can hopefully see how many you know people have actually taken a look at it but I want to go back to your other question about deploying it it's literally a matter of downloading this GitHub repo uh it has a lot of information but all you really need to do is just do a couple of simple steps uh to deploy it just easy azd to log in to your subscription you can initialize it and deploy it so in three simple steps you can get this up and running and going so I hope that you and some of the viewers will take a look at this set I I don't know what to say it's like you've thought of everything it's like I'm gonna blow your mind number one I'm gonna blow your mind again by programming in language and then I'm gonna blow your mind by saying just Do azd up and you can have this too uh in your own uh subscription this is really cool any anything you want people to go read or look at no I I'd love it if they you know take a look at this you know give it a try give us feedback tell us tell us how's working you know what are the types of things that you would like to see in this have you seen any really interesting things so I'm looking forward to hearing what people have to say amazing and I was trying to like lob one up because you told me to put the blog post in a link and there's a blog post on that uh for you all to go look at it Liam Liam's awesome love working with him and also for sure minimally you should go look at the GitHub repo browse around the code and see if it's really as easy as Liam said to to set this up in your own uh subscription any parting words uh Liam now I'm I'm just super happy to be back and hopefully we will be back a little bit more after and with some some additional cool things coming coming down the road I'm gonna hold you to that thank you so much and my friends thank you so much for watching we're learning all about how you can make your own Enterprise gbt real with Azure cognitive search and Azure Cog Azure open AI service thank you so much for watching and hopefully we'll see you next time take care [Music]

Channel: Microsoft Developer

Views: 10,416

Keywords: Azure, Microsoft, Tech, Technology, Dev, Development, Cloud Computing, Enterprise GPT, Azure OpenAI Service, Azure OpenAI, OpenAI, ChatGPT, Azure AI, Microsoft developer, AI Show, developer, Seth Juarez

Id: A_gVmzAHEhU

Length: 12min 38sec (758 seconds)

Published: Mon May 01 2023