The Basics of Prompt Engineering with Azure OpenAI Service

you're not going to want to miss this episode of the AI show where we talk all about the basics of prompt engineering with Azure openai make sure you tune in [Music] hello and welcome to this episode of the AI show we're talking all about the basics of prompt engineering with Azure open Ai and my friend Marco Marco how you doing my friend how you doing Seth I'm having a good day oh that's amazing so why don't you tell us who you are and what you do first off all right well I'm Marco castalena and I am the vice president of products for Azure cognitive services and that includes Azure open AI but it also includes our speech and language and vision and decision capabilities all the fun stuff that we've had on our show tons of time before we even get going Chad gbt has taken over and obviously it feels like a lot of kind of can you help us understand what you mean by the basics of prompt engineering why all of this blew up and what's so exciting about all this yes so check this out and you're going to see it in a minute English is the hottest new programming language that's cool and I tell you well I'm paraphrasing Andre carpathy who used to be uh the head of Tesla AI but anyway the way that you set up chat GPT the way that it actually works behind the scenes you will not believe it that's prompt engineering when we call the prompt is how you feed it plain English or any language really and that's how it works that's how you program it so if I'm understanding correctly this is just a model that you give English to and then English comes out you could look at it that way other languages can come out too but that's right you can tell it what to do and it will do that thing now now I feel like it it's got to be more technical than that is there stuff you can show us that gives us a sense for what's actually happening why it so happens that there is something I can show you let's take a look all right so check this out we're going to start with Bing and if you haven't tried the new Bing chat experience yet it is pretty cool and it's also based on GPT in this case this is actually gpt4 but regardless it kind of works the same way so let's see what happens when I say who I am and what I'm doing and what it kind of suggests to me and then we're going to look at how this actually works behind the scenes so here we go it's doing some searches here and this is something that Bing can do that the raw chat GPT doesn't do it can actually search the web and that helps it inform its answers and now here it comes prompt engineering as it says is an increasingly important skill set and that is absolutely true and that's why we're having this discussion and so believe it or not behind the scenes the way that Bing is programmed is exactly what I'm about to do you'll notice until you see it all right now here we go this is the administrative environment behind chat DBT so I'm in Azure open AI and in here we have this new chat GPT playground in the chat TPT playground well I can have my own chat GPT and by the way there's this common misconception that there is one chat TPT but that's not really true there is one chat.openai.com they're consumer chat GPT but you can make your own chat GPT with Azure open AI so there can be many instances individually of chat gbt and this one this one is mine all right so here's what I'm going to do all right so this thing at the top here the system message or what we call the metaprompt this is very important this is where you give it its instructions this is where you tell it what its tone should be this is where you tell it what not to do for example maybe you say you want to say don't make jokes about politicians or something like that right so you give it all these instructions up here in the system message and I've done none of that so right now I got a wide open system message over here and basically that means it'll talk about anything so down here you know let's say I wanted to talk about the Greek island of Santorini all right all right and you know I submit that and there it goes okay it's telling me whatever it remembers about Santorini that's great but let's say that you know I'm a business like let's say I'm Microsoft and I want to make my own instance of chat GPT but I only wanted to talk about Microsoft stuff like I just want to talk about whatever right so I'm going to have to program this thing so that it only talks about Microsoft stuff and so over here I have done just that this here this stuff this is the programming this is what I meant when I said English is the hottest new programming language so here I've written you're an AI assistant called Soft do you help people find information about Microsoft products you will decline to discuss any topic other than Microsoft products and services so this is like a commandment you will not talk about other stuff and you'll end each response with an emoji that's a Sephora is special I know that's a that's a meta prompt you like to add that's right I've seen you do it now you know down here I have it already typed in tell me about Santorini now Santorini is not a Microsoft product it is a Greek island and it says I'm sorry but as an AI assistant I can't provide information about Santorini because it's not a Microsoft product now of course if I ask it about a Microsoft thing as Microsoft offer uh M365 in Germany for example that most certainly will work so as it's answering I want to ask a couple questions that's okay so this here that I'm looking at and look it did the right answer of course this thing I'm looking at this is this is like chat GPT but specific to you can you explain what this what we're looking at this chrome here yeah so this stuff we're looking at here this is my chat GPT right I got my own chat TPT and the data that goes in and out of this chat GPT and that could be these prompts or the responses that's my data and I get to keep that data and that data doesn't go back and go train other gpts or anything like that this is my own instance of chat GPT that I can program in any way that I want and this here is just basically a playground where you can test stuff out but there's is there like an API endpoint as well that you can use that's only for you and your business absolutely absolutely so when I make a deployment over here uh that creates an API endpoint that I can now use to call my chat GPT from my own business my own website my own mobile app whatever and that's what it's meant then by Azure open AI when you're using Azure specifically cognitive services with your own open AI cognitive service you effectively have your own model for your own data for your own stuff that's right that's what it is amazing all right well keep showing us more stuff this is cool all right so there is more there is more Okay so um now there's something that Bing did that you might have noticed here that this raw chat GPT over here my own chat GP didn't do Bing did something very interesting here see when you give a when you give it when you submit a prompt to chat CPT we call each one of these things a prompt uh when you submit a prompt to chat GPT it just spits out whatever it knows for memory so it just kind of remembered that yes Microsoft offers Microsoft 365 in Germany but it didn't really go look that up to verify that that's true in any way Bing on the other hand does do that so you notice when I made this this query here it actually went and looked up prompt engineering for chat gbt and it used that information that it found so it found this information from all these sites over here and it used this information to uh give me this response so it didn't just make the response from memory it's grounded in data so of course now let's say again that I have my own business let's say I'm uh Microsoft and I would like to put up a conversational experience but unlike Bing I don't just want it to hit whatever on the web I I want to specifically say you know there are these sets of data that I would like it to be able to consider in making its response so I could do that and so here I'm going to show you both how to do that and also I'm going to kind of peel back the covers a little bit on how Bing is doing this so you all can get a look at what's really this is cool this is cool I'm excited I'm excited it's going to be cool all right so here's the deal so I have made this uh bot actually my colleague Pablo made this bot and this is chat GPT on top of azure cognitive search and Azure cognitive search is a search engine but unlike Bing it doesn't just automatically index the whole web it's a search engine that you tell it what to look for and you say you know I only want you to consider these URLs these documents and it'll do that uh and so we have this Azure cognitive search and it is searching across our employer Health Plan let's say that I'm making a kind of an HR benefits bot for my internal employees so we have all these documents in here in the search engine about our health plan I'm going to ask does my plan cover new glasses and here we have it it gives me this response it does cover new glasses actually that's great it tells me the different kinds of plans and so on and this is a conversation I can actually keep going with this because it's asking me for more information but like Bing you notice that it did a query and it actually is returning also citations but these citations are documents that I am indexing that might be only inside my business it might not be publicly available on the web what's really interesting here is if we look at and this is maybe something I wouldn't show you know my my employees or whatever but I love this tab the thought process they call it the thought process and so here's what it was thinking so to speak in informing this so I asked this question up here does my plan cover new glasses and it didn't just send that straight to chat GPT and this is chat GPT that's running behind this but it didn't just send us straight to chat TPT instead it actually generated a query from that first employee benefits glasses coverage so I decided to go query that and it found some documents about that employee benefits glasses coverage so before this ever got to chat GPT it did this query and it found these documents and one of the things that cognitive search does is it chunks up the document so it can take little individual pieces of the documents because what is also true about uh chat GPT is that there is a token limit you can only give it so many characters uh in your prompt before it kind of throws up right so you can't just throw all your documents in there that'll never work you have to have a search engine that will find the most relevant little bits and say these things are relevant and then you can kind of throw that in there which we'll get into in just a minute so we did this query and finally everything under the prompt in this thought process is what it's actually feeding to chat GPT so it's feeding to chat GPT all this stuff down here this first part is the metaprop it's kind of like what I just did and this meta prompt says this this one helps company employees with their health care and it also says uh don't generate answers that don't use the sources below so don't just generate whatever right only if it has substantiation and so on so there's a number of instructions in here and then right into the prompt itself we are injecting the data that it looked up these are the pieces of documents that it found and we're giving this directly to chat GPT and finally underneath all of that is my my little question does my plan cover new glasses so chat GPD by itself could not possibly answer this question without this data grounding in fact if I go back into the raw chat TPT and I say does my plan cover new glasses that is going to give me a nonsensical response because it has no idea what I'm talking about here I'm sorry at least it knows it doesn't though uh but here it has all this information to use and it does use that information to form this response right here that it's created and so this is how you're able to inject your own data into chat GPT which incidentally is covered in this blog post by my colleague Pablo uh and we can maybe give you a link to that so you can read it yourself and use this GitHub repo yourself but this is fundamentally how all of these things like Bing actually work so to summarize because I mean I think this should be surprising to people because you kind of like rip the hood off basically what you've done to make it answer specific internal questions is you've captured the question you've done a search in Azure cognitive search in this case injected relevant documentation then pretended the chat is going on at the bottom and then you left off the answer and you said generate the next most likely thing I mean that's effectively what the model is doing but it sure does give me a good answer doesn't it yeah I mean not only does it give you an answer and that's the cool thing sorry my phone AI was trying to my my watch AI thought I was trying to talk to it go figure we're having a little Inception moment so it's able to actually give you references to the specific documentation because there there is this problem where and I don't like the term hallucinate because my sense is it's always making up the next n tokens but there's a there's a sense that sometimes it's telling you stuff that it's just making up and it's not factual I love that there's actual like go to the document we'll show you hey and you know what's really I mean because it has this instruction at the top that says you know only give an answer if you have the information if I ask it something that it doesn't have information on uh what's the what's the best uh surfboard wax that that's not in my health plan uh and if I ask it that I'm sorry I don't know none of the sources I have mentioned surfboard way and it's true you know so you can kind of restrict its domain both through that metaprompt that we saw and also through the the data that you're searching over now but wait before but wait okay there's more there's more but actually what I'm trying to say is there's less because there's something that I totally did not do here I don't know if you I don't know if you caught this uh what I didn't do in any of this I did not train the model I didn't fine tune a model I didn't do anything with the model I am using the stock this is the chat TPT turbo model and this is definitely the best model to use to get started with Azure open AI today for just about anything you can think of you want to start with this one I understand gpt4 is now in public preview and that's cool but gpt35 turbo is great it's fast it's inexpensive and it gets the job done and that's where you want to start I didn't do anything to this model I am using the stock model out of the box fresh from the dealer I'm using it here and here and here hold on so hold on hold on hold on because I want to make sure that I'm understanding this right you're saying that that all that with the meta prompt and the injecting documentation you just send it to the API without doing anything else that is what I'm saying now of course we did have to prepare the search index up front right so we had to set up Azure cognitive search and tell it what documents we wanted it to index and do the connection which is what Pablo details in this blog post but I didn't train a model that I did not do and that's cool because like everyone knows about search things and if if you're trying to just generate the best output text now I'm getting a sense for you basically in the meta prompt give it instructions you stuff the relevant content however you want to do that into the prompt and then magic comes out that's right and all of these things that you've been hearing about Bing M365 chat the Dynamics co-pilot this is how they all work all of them this is fundamentally what they're all doing this is amazing and this is this stuff is all available now or do folks have to wait everything I'm showing you here today is public well this is this is amazing and like I said I if you haven't tried this like there and I'm not using hyperbole here there's few moments in Tech where you're like oh that was an inflection point it feels like this is an inflection point what say you Marco that's what I'm talking about you know it's crazy it's crazy that you can do this uh but you really can and that's that's amazing so Marco thank you so much for spending some time with us my friend thank you for having me awesome marco has been unveiling the mask of what's going on with Chad gbt by showing us the basic basics of prompt engineering with Azure open AI thank you so much for watching and hopefully we'll see you next time take care [Music]

Channel: Microsoft Developer

Views: 25,787

Keywords: Azure, Microsoft, Tech, Technology, Dev, Development, Cloud Computing

Id: QzZSJDxdUg0

Length: 17min 58sec (1078 seconds)

Published: Mon Apr 03 2023