Let's Get Technical - Build your RAG application with Prompt Flow in Azure AI Studio

hi welcome to a Microsoft reactor live stream series let's get technical so you have joined us previously it's great to have you back so if you're new here here's where you can catch up on our previous let's get technical um topics on demand I'll share the link over here so you can just join us so the let's get technical is a series of three short 55 minute virtual events delivering technical demos and Kno hows on the latest Technologies in Asia so we do have episodes in this series conducted in different languages like English Bahasa Indonesian tagal Tha Vietnamese and we'll love to do more of this so tell us in the comments below what kind of topics you would love for us to cover um or to be honest it would be great to hear from you what timing and time zone works best for you when it comes to learning from live streams like this and you you can of course take this time to hit on the like And subscribe button and that would make myself our speaker today and everyone at Microsoft reactor very happy so today we are super honored to have Dave Glo global principal Cloud Advocate at Microsoft and a familiar phe to many um joining us today so Dave is a passionate software developer and is super interested in many Technologies particularly AI low powered and beta platforms Communications and cloud system so let's invite Dave to join us now hi thank you very much hello Dave great to have you join us from Sydney today thank you it's good to be here I'd love to say it was warm but it's actually pretty chilly here in Sydney at the moment unfortunately winter right so Dave usually it's a practice for us here at let's get technical that we share a fun and Casual side of our speakers so apart from the usual bio and all so um it's usually quite a pleasant surprise for our speakers so I'm quite sure it's going to be good fun so we'll do five of this or that questions are you ready okay can sure cats or dogs uh dogs cool ice cream on a winter day or hot chocolate in a heat wave oh ice cream anytime I'm kind surprised though because you said about like how it's winter and Dre very so like I'm like oh okay surpris okay glass half empty or glass half full uh definely half full person same here okay so we we are on that um Saturday or Sunday probably a Sunday yeah that star to chill out okay I refer Saturday because it's nice to know that the next day is still a non-working day like can sleep in yeah yeah that would go for that too yep okay the last one pizzas or Burgers probably Burger yeah Burgers okay same here so all right so um so this is how we learn more about you Dave in a good 10 seconds but on a more serious note Dave why don't you briefly share today like why the topic that you're sharing today actually matters okay so what um what what we're going to talk about today is around building large language model applications and what I'm going to be showing you is kind of low code ways of doing this and if you think about lots of businesses customers everyone's grappling with the idea about how do you integrate large language models into webs so that's kind of what we're going to be talking about sounds very fun and at this point I encourage you um audience to start your GitHub code space so the link is actually here on the screen and I'm I'm actually going to drop it in the chat as well do that so you can just copy and um get it running right so and before we kick off the session I'll like to draw everyone's attention to the reactor code of conduct so it's pretty lengthy here but in short um we want to encourage you to keep this um environment an enjoyable one so make sure that your comments are respectful constructive and welcoming so at any time um if you do have questions you can drop them on the chat and we'll try to address them um as many as possible during Q&A so yeah so over to you Dave have a good one okay thank you very much okay so just on the note of starting up the code space now the goal of this Workshop is for you to play along and do the workshop as we go now I'm not 100% sure how successful this will be but the first thing you need to do is you need to start up your GitHub code space so there was a link in the page and what you need to do is you need to go to click on code and then I've already started up my code space but there will be a box in here which says start a code space so do that it'll take about a minute to get started and um it'll just be something that's there for up and running so I'm going to imp placees slow down and just let you potentially play around with it and I'll have a bit of a sense of what people are doing um through an admin console so um as I said go to get that link um the link is I'll just go back to the slides in case you lost the link the link is um up with the link that's the link there so aka.ms reactor D rag D CS for space so go and start that up and um hopefully by the time we get to this part of the workshop you'll be up and running okay so anyway a quick introduction to myself so uh we had a bit of a this and that um so I'm um based in Sydney Australia and I'm in the cloud advocacy team and my role is basic an Outreach role so Outreach to the developer Community uh I've been in Microsoft for close enough to 3 over 30 years actually so I've been around for quite some time and um have seen lots of different revolutions across the across the it sector really across the last 30 years if you think about there's been uh personal computers uh there's been the internet there's been mobile and of course now we're looking at Ai and each one of those had a challenges and we're going to talk through some of those things um now also feel free to connect with me on LinkedIn so my Alias on LinkedIn is glove boxes I think I'm probably David Glover on um on LinkedIn and do feel free to uh follow along now the the workshop itself is going to be at this link here aka.ms reactor D Rag and that's got a full Workshop in fact I'm going to pop across to that now and just show you what is in there because what I decided when I was building this Workshop is I wanted something that was highly repeatable so that you could um maybe in your workplace or with your friends or family should you choose to do so um would be able to reproduce this Workshop all the way to I know if you're if you're working with um meetups or things like that it has all of the notes about what you need to provision what you need to go and set up the flow of the workshop everything is in that work in this Workshop to make it super easy for you to go back and reproduce this workshop and including the PowerPoint notes sorry including the PowerPoint slides you'll see there's both English version and Japanese because we happened to do this in Japan but you will see the slides complete with all of my speaker notes um so when you look at this and you think oh that's just what Dave said it's probably because when I was rehearsing and pulling and building the session I wrote the speaker Nots down so it be super easy for anybody else to go and reproduce this session so that's my goal is that you will be to um reproduce a session later on and carry and carry it on now the other thing that I've done with this Workshop is all the resources all of the open AI resources I have granted I have we have what's called an open AI or an AI proxy so I've enabled the resources for the for this Workshop to run for about a week so you don't have to worry about oh gosh I don't have any access to open AI um models I can't do this Workshop I've got I've um set up an event and within that I've allocated out some resources so that you can use them for this Workshop so that's kind of what you can do um as you as you carry on so that's so that's the general gist of it so the goal of this Workshop is that you should be better have a better understanding of what is going on with large language models from Microsoft and what are we doing around building around the platform and tools to make it easy for you to go and build um to to to make it easy for you to go and build these these Tyle of applications so I said all the source code all the documentation is available um for this Workshop okay so we're going to be touching on an introduction to our large language model operations I'm going to be talking about Azure AI studio and and the most of this demo of most of this talk is around prompt flow uh and we're going to talk about promp flow so think about prompt flow is like a low code way of building a large language model application or rest Ender point and we'll get to that in a moment um so we're going to build an application so we're going to build what's called a rag or retrieval augmentation and generative application with pomp flow prompt flow and visual studio code and depending on time we'll also look at some of the evaluations uh that that can be um that you can do okay so now I've already touched upon this already but with all um major revolutions particularly in technology they always come with opportunities and challenges and and that would be things like Mainframe Computing personal Computing internet Ai mobile Etc they all come with opportunities and challenges now what we're doing with large language Ops is we're building out a series of platforms tools best practices and Frameworks to make it easier for you to build this and you'll see this term banded about called large language model operations and it has a life cycle and I'm just going to talk through this if you've done any other kind of um uh apps uh operations life cycle it'll look very similar um so you typically think about you you got an idea so and you want to think about okay how can I've got a business idea or a business need how could I use a large language model to maybe make this application real and then you kind of go through this whole itera process like I'm going to try a few prompts did that work did that kind of Meet the hypothesis do I think I can do this achieve this with this uh larger language model maybe this l language model is no good for my needs and I can go and iterate on different large language models and things like that so there's a whole whole idea of ideating and exploring a different ideas and then the next phase of that is you okay okay you've got the basis of an idea so what are we going to do around trying to make this a bit more real and that's kind of this the kind of the building an augmenting phase where you where you're looking at the application your you're trying out maybe some sample data uh you're seeing how well that response to that sample data you might be using to ground your prompt um you'll be looking at things like evaluation um is this is this application performing well am I starting to get hallucinations is the model starting to respond in ways using my data that maybe I didn't intend so maybe I need to look at the prompts uh the grounding the data I'm using for grounding just to kind of help steer the model in the right direction to kind of give you the kind of responses that you want that you're wanting from these applications um the next or the not the next phase is around oper operation productizing or moving to operations and that the whole thing around that is then you start thinking about well what's the cost management for this um are we are we within budget can we monitor this application now it's in production is it working and and Performing and within the guidelines there's also things like model drift so maybe you've got a new model or maybe you've got a new data set or maybe you've done some different prompts and then you want to think about you want to be testing is this model still and is this application still behaving the way that we originally envisioned so that's kind of come down the whole operation and it's also things like um um uh so safety so is this application does it perform well in the way that we want it to perform that's consistent with our organization or brand and things like that is it secure is it Network secure and things like that so those are kinds of things that we do and inside all that what we do is we wrap this up in a management fabric so then you've got CC pipelines all those types of things you can start testing this application automating the test of this application is it performing the way that I expect this application to be performing if things are that okay so as I mentioned before with all revolutions already come around a um a series of platforms tools best practices and documentation things like that and that's very much what we're doing around this whole Azure AI um now Azure AI studio in particular is really the center of gravity For What Microsoft is doing around um Ai and generative Ai and building applications um in that in to that style so Azure AI Studio has access to thousands of models so you would expect obviously things like open AI models but there's also mattera there's hugging face models um there's mistal models which are it provides Tools in there for grounding uh your applications if you're building rag style applications which is effec what this talk is about so you will have data sets inside your business that you want to Pro provide um for your application so for example you might have product information or customer support information that you want to use to ground the large language model models and those applications so they so they respond back with your data um there's Tools in there for uh prompt evaluation um uh safety and responsible AI and continuous monitoring and the whole idea behind this is that ultimately when we're going to build an application with prompt flow we're going to build this with Visual Studio code as well as the promp flow extension and what we're going to do is we're going to deploy that to Azure AI studio and then that application will then be available to be consumed within your application so basically what's happening is you're going to deploy it as a a large language model rest end point that is grounded on your data so that's kind of what we're ultimately going to be doing okay so we talked about AI studio is kind of the the the center of gravity for what we're doing around all things generative AI so the next thing we want will talk about is prompt flow and what prompt flow is about so prompt flow as I mentioned before is really around providing a low code way of building out um applications so there are a number of styles a number of ways in which you can go and build rag based applications so there can be what what I'd call code Centric or custom code Centric so you might have heard things like L chain you might have heard things like semantic kernel or open AI assistant API those are quite code Centric approaches to building rag style applications but what prompt flow is about is about providing a low code or maybe truer to say lowish code so it can be low code a very much drag and drop but you can extend it and you almost certainly will have to extend it but the goal of prompt flow is to provide a way of simplifying the whole process from prototyping experimenting iterating deploying and monitoring the importance I want to say is emphasize around deploying because a lot of people don't realize they think oh this is great for experimenting but this is this whole end to endend from experimenting all the way through to deploying uh to the cloud dagia and monitoring that application and you can deal with things like it will create you the executables it has support in there for debugging it will create the ability the ability for you to say okay I'm going to do ab testing how does this look how is a performance compare of one set of models and one set of prompts versus another set of models and a different set of problems and you can you can test that and all the way through deploying these large language model applications um into aure okay so I want to talk a bit about what rag style applications are you've probably potentially heard about rag style applications but you're not quite sure why or or what's the relevance of them or how do they the work so the first point I want to make is that kind of like the number one success the number one thing that would determine how successful your large language model application will be is context and that context will be something to do will be related to your data so the context will be your product data your customer support data your frequently asked questions whatever it is it's about your data which is going to provide context into are this large language model app now there are effectively two ways in which you can go and create or build an application a large language model for context now you might have heard of something called fine tuning now what fine tuning is about is you'll take a large language model and what we call a foundational model and you will add your layer to the top of that model and you will tune it with your context so your product set your your data set and things like that now the downside of of fine tuning is it's quite complex and typically a high high initial cost but there will be some use cases where it does make sense to fine tune and maybe at a longer term it might make more cost cost effective sense but my general premise is if you've got data which is quite dynamic like customer order history or product sets that are changing that's probably not the way to go because you have to go and reine tune your models with your data that's changed so that's typically where you go and use rag so I'm going to talk through so there's two ways approach there this fine tuning you go and basically add a layer to the foundational model or you can take this rag which is this retrieval augmented generation approach and the way this works is that you can imagine you've got like a chatbot and you've or an internal support so you got a user who asks a question so what you would do is that you would retrieve um data you basically do a look aside and you would retrieve Lo related data to that user question so then you come back with that data and then you'd augment The Prompt so that prompt might and I'll get to that one what it would look at moment and you'd augment that prompt with data from some backend data store it could be a vector store it could be a relational database it could be whatever and then what you do is that you send that that data off to a large language model and you generate a response and that response will come back um to um back to the end user now there's a question here about how would this integrate with Confluence data well Confluence data could just be a data source that you and did a look aside you you you typically need to have some sort of index and you you go and query that index and it could be inside Confluence and then you bring back relevant data from Confluence or it could be Azure AI search index or redis or whatever there could be a whole lot of backend data sources and you grab that data and then you would um augment The Prompt with that data you grabb from Confluence and then you would send that whole data um prompt off to the large language model and get a response back from it so hope fory that that kind of answers your question okay so now this example here it's it's we're kind of building out um it's like a product support or a product information chatbot so we've been to plenty of sites where you where you've um been in study you want to talk to that and you want to get information about that and as we know that sometimes they can be a bit Hit or Miss um and what large language models can significantly improve that experience that you might be giving to an end user now besides that chatbot there are lots of different other scenarios that um that large language models can play a great part um great great part in the um for using large language models I like to split these things out into risk so high risk and low risk so when you think about if you've got a business idea and you think about well this is quite a high risk um then the general rule of thumb is that you want to collaborate with AI so in which case so give you an example like a doctor um if you want to do some patient diagnosis then if it was me or if I was a doctor I would want to collaborate with that doctor but I'm great using AI to help um with diagnosis of maybe an issue that I've got um so that would be a good example of collaborate but then you might be another situ maybe you got large making large economic decisions well you probably don't want AI necessarily just going off and and creating these economic decisions that may be for the country for a large company um so those are typically collaboration now there are plenty of lowrisk ones like the one we're going to talk about we're going to build customer support uh maybe with customer support with Handover when you sense that maybe the customer is getting frustrated with the data they're getting back you want about to hand it over to a real person but things like information retrieval it could be policy documents inside organization content generation uh language translation great examples education um you might have a scenario where you've got um you're inside a university and you've got all your course curriculum and you've got students who are doing tests and then you can figure out oh that's interesting this student is struggling this particular area I can use generative AI to go and generate content that will help the student learn in areas where they're weak so you can do things like that so general idea break things out into low risk and high risk think about collaborate or automate so low risk automate collaborate for high risk okay so I'm hoping by now that some of you or that that you've opened up um your um GitHub codespace now I think we'll probably so what I'm going to do is I'm going to pop across to um my browser and you remember so you would have been linked off to my repo you'll end up on glove boxes contoso chat proxy and if we go to click on the code button in the code space es you will have in here a green box which says start this code space okay and that that's going to go and create and that takes about a minute to start at the code space and what you will have when that starts up in your browser you will have Visual Studio code in your browser and you can come along here and we're going to go and do a few things on here so if you're used to using visual studio on your desktop this is Visual Studio code running aside a browser but it's backed by um basically virtual machine that's sitting up in GitHub uh it's actually probably a Docker container okay so that's what's going on so you're ultimately this takes about a minute to start up and actually you will see a little warning box down the bottom right hand corner just to say an API is out a date just close that because it w impa this okay so hopefully you can do that um so the link is there aka.ms reactor D r- cace okay so the next thing I want to talk about is ideation exploring prompt flow with codespace now promp flow um with Visual Studio code there's an addin for visual studio prom flow it has um two ways in which you can work you can work with a code first user experience or a designer first user experience and you can they're interchangeable you can switch between the two of them now the code first um approach is super useful for when you want to do things like check code into GitHub or into your code um source code control system and you want to be able to find out line by line what has changed so it's a great way of be able to track changes great way to be able to collaborate with other people in your in your group saying hey look we're working on this promp flow um I need to extend it and do this and do this and that and the design first experience is kind of more drag and drop now the other thing I want to point out is that what we're going to do today is highly stylized it's a it's like the start of a 10 almost certainly your prompt flows are going to be way more complicated so for example I'm assuming in this in this prompt flow we're going to build it's about information retrieval but I might have gone to your site and I might be looking for customer support and it's going to be really frustrating to me if I go and say I want information about I need help with a tent and I'm looking for customer support and then your website tries to start selling me the latest and greatest tent that is obviously not going to go down very well so in the prom flow you often need to deal with customer intent and you will find in that GitHub repo if you go and look at um one of the other folders you'll see things about understanding intent but in this example here we're going to keep it really simple and not deal with intent so just keep that in mind okay so the next thing that this app is going to use and almost all rag applications use is some sort of vector database now the vector database we're going to be using is azure AI search and it's a hybrid database and then what I've done is I've created what are called embeddings now an embedding is just a special type of vector which is being generated from a large language model which has semantic meaning so basically you would have each product description you would send it off to an embedding model something like um open AI uh text embedding Ada something like that or there's a new one called text embedding small um and that would generate this vector and that Vector plus the text you would put into your index and then what happens when a user wants to come along and search excuse me um they would put a query in and what you need to do is you need to go and generate an embedding for that query so I need small waterproof shoes I would create an embedding using exactly the same model that I created the embeddings when I created the search index and then I would send this off to to in this case Azure AI search and aure AI search would do what's called a nearest neighbor lookup of the index as well as a keyword search and it would return back the relevant results in this case here the relevant results would be um research results I need War um shoes the relevant results would be trail trail Walker hiking shoes and there' obviously be some related information to that now there are lots of different Vector stores I'm to be showing you Azure AI search but there's reddis there's postgress there's pine cone there's we8 there's a number of these um Vector searches out there Vector stores out there okay so the next thing I need you to do is this is where you can get access to open AO resources so you need to go to this link so this aka.ms reactor Rag and I'm just going to go to that link myself and it will take you to this page here it's called let's get let's get technical uh rag prompt flow and in here um this is the workshop we're going to do and in here is the link to the workshop and right Mass click on this and open this in another link another page because you're going to want that as you go now as an side this when you access this this will give you access to some open AI gives you access to GPT 35 GPT 4 and text embeddings and a read index read only index for aure AI search um if you're new to prompt engineering you might want to go and do this Workshop here we're not going to do this Workshop but I thoroughly recommend uh doing this if you haven't done this and when you've when you've logged in with your GitHub account which I've already done you will see you've got your API key and your proxy endpoint and you're going to need those for the workshop okay so hopefully a few of you have done that okay so doesn't look like too many have done so I'm probably going to assume that you're going to do this after so we'll see how we go right so you need to set up that and and um get connected to that Workshop right so the first part of this Workshop I'm going to demonstrate this and you'll be how to do this after this Workshop as I said the resources will be available after this workshop and might just be easier for you to play along so we're going to create um and we're going to do we're going to build a b style application and I'm going to show you how you do this inside prompt flow so what I'm going to do is I'm going to go to the workshop and the workshop here in fact if I go to the introduction you will find that the slides I've talked about are in this Workshop so super easy for you to follow on thing I didn't quite understand that what was um going on okay okay so what we're going to do so we're going to so you're you're going to do this event using these resources that have been allocated via proxy to make it easier for you to go and get up and running and the resources that are backing this Workshop we've got Azure AI search and Azure open aai and in here I'll just make this a little bit bigger um you would have created a code space and got up and run building that code space okay so I can see about seven people have done this okay so you've you've you've been off to GitHub code spaces and then from your browser you've opened up that Cod space and You' got that so we're going to ground we're going to learn about grounding your data so we talked about that there's two ways of doing that we're we're going to ground um The Prompt the rag application with Azure data from Azure AI search and from data from a product history um uh customer order history right okay so the first thing you need to do is you need to go into um the code space and you need to up open this file in there you'll see it's called dot environment and you need to add in the proxy endpoint and the Azure AI key that you got from when you signed up so remember this is where the API key is and that's where the endpoint is so you'll do that inside of oh my Cod spos decide it want to close um so hopefully that won't take too long to start up okay so while it's starting up um we're going to go to the workshop and the first thing we're going to do when this starts up we're going to look at the tools that the prompt flow extension adds so we come over here and that is still loading up it should hopefully take about 30 seconds and in the on the side you'll see that there is a promp flow tool and in that you can get access to connections you can see what connections are available for your promp flow and um and what other what other resources are available so we'll come across to okay we're up and running so as I said the first thing you need to go and do is you need to go into um this um environment and you need to add in your endpoint and your key and I'm going to come across here and I'm going to add in my Endo paste that in and my key run that in there save it away and then the next thing you going to do is you're going to go across to the in fact I've this is a trick that I found with prompt flow when you've gone into code spaces you will find that your toolbar is on the left hand side by default I find that with using prompt Flow by far the best way is to have it on the right hand side and this will become more apparent as we go through so what you want to do is you want to open up the connections folder and you want to go and open up a Jupiter notebook called proxy dcre Das connections make sure you open up the one that starts with recommended python environment and then run that and it will run it will take a matter of moments and what that is doing is that that is setting up all your connections so if we come down over here into the uh prompt flow we'll see down the bottom is that that will create a connection to Azure open AI you'll see down the bottom right hand corner and you'll see I've got a connection there called Azure open AI connection and another one called cognitive search now cognitive search is your old name for Azure AI search so this tool hasn't quite caught up and you'll find that there's a definitions for those okay so back in the notes so we have created we've created the um we've created the our environment and we've run that proxy um connection to go and create that the next thing I'm going to do is I'm going to create a very simple um um prompt flow so I'm going to go back into my code space and from the file I'm going to you'll see down the there's a folder called Workshop what you're going to do is you're going to write M click on that workshop and you're going to go new flow in this directory and that's going to fire up a prompt and then you're going to select the third one down which is chat flow with a template okay so what that will do is that that will create you a starter work a template and you'll see at the top here there's a little if you hover the mouse over you'll see that there's a visual editor you go and select that and what that will do is it will start up your chat you're going and click on that and you'll see that there's a little red triangle so you select that and that on the left hand side will bring up the chat the the properties for the chat what are called tools so these are called tools click on the drop down and select the Azure open AI connection okay so you've got that set and the next thing you want to go and set is input so in this case you want to go and select um select inputs and over here you you want to go and say um what tent would you recommend recommend for beginners probably needs two m in there okay so we've set up this and what this is going to do we're going to run this now at the top you will see that there is a number of um uh icons or buttons that you can select so there's one there for run all you can go and hit batch run and you can also debug so in this case here we're going to run this and we're going to run it with standard mode now a few there's a few folk who've just got a little bit caught out and I think for the folk that have got I think there somebody here struggling with the API key and endpoint my recommendation is maybe just watch because we we're going to run out of time otherwise and then just go back so watch and I'm going to go through in quite a lot of detail so just watch what I'm doing and then just go back and reproduce it now when I ran that flow what it did is it came back with an answer you can see down the bottom here for beginners I would recommend a simple and easy Dome Tent now what that's done is that that's grabbed the data for What tent would you recommend it's just grabbed it from the data that the model was trained on it's not using the data that I've that is inside my business that I want this prompt flow to use so we go back over to the workshop so we got up to here and we've created a new prompt flow and we put in what tent would you recommend for beginners and what that's done is it's just grabbed a generic response this is what the data was available and the models come back with a an unreasonable response but it's not relevant to my organization and my data so you'll see all the way through here there's D there's documentation about how to go and do all this now what we're going to do is we're going to now start building out this application to do the whole rag retrieve augment and generate so we've talked all about this and what I'm going to do is I'm going to update the the um the yaml which describes the the flow The Prompt flow so I'm going to come across back to my code space I'm going to close the designer and that will just leave open and what I'm going to leave open the original yaml the flow. dag doyo file I'm just going to delete out everything and paste go and paste in that new prompt and now what I'm going to do is I'm going to switch across to the visual editor now what you'll find is that if you remember I said it's better to have the the um Visual Studio code uh primary bar side barar on the right hand side it leaves space over here for the pro for the properties it just makes you'll find that once it stops bouncing around so much you can move things around there's also some tools in here you can do Zoom to fit and you can play around with this now this works really well on a wide screen by the way okay so what I've got in here is I've got a flow so I've got my input and my input's going to be recommended tense I've got an embedding so what I'm going to do is I'm going to generate an embedding for that question so what it's going to do it's going to send that question off to the text embedding the open AI text embedding 3- small model and it's going to return back and embedding for which is a semantic representation of that question and the next thing I'm going to do is I'm in parallel to this I'm going to look up some data in a customer database and you'll see how there's a little link so this is a python um ex python customer tool and you can go and click on the link and this will take you into the python tool and you can go and come in here in this case I'm just going to set a break point we just set one break point on there okay and the next thing what it's going to do is going to retrieve some data um from both the customer history as well as doing some product lookup so what I'm going to do now is I'm going to run this so now we're doing the first part so I'm running in standard mode that will start the prompt and there'll be some parts in prompt flow workout what can be done in parallel and what needs to be serialized and what it's done is it's brought back data so you can see it's brought back data from my customer order history so you can see in here um these were This was um product product order previous things I ordered ah sorry big your pardon yes it's brought it back my customer order history and within that I can come through here and I can look at my output and I can go look at my output I can open in a new tab and this is the data that's come back from my customer order and my product data okay so the next thing I'm going to do in fact the thing I forgot to do is I'm going to go and run this with a breakpoint or run this in debugging mode so that starts up and what we'll see is that this will stop executing at the breakpoint so as you can see with these custom Tools in this case here I'm going to grab some data from a backend data store and I want to use that for part of my query so that's what's going on there okay so the next part go back to my yo document I'm going to go back to the workshop notes and the next part I'm going to do is I'm going to do augmentation so I'm going to grab that data from that I've grabbed from my product set and my order history and augment it with the customer query so going to come in there copy that come across to the yam document paste that in save it away and I'm going to go to the visual editor and you'll see now what we're doing we've got my customer look look up and then what we're going to do is we're going to create a meta prompt and if I go click on that link this will take you to it's what's called a python Ginger template it's a really standard um template builder in the python world and you can see that this is just going to build out a PR which is ultimately going to be sent off to um the large language model so in this case here we're going to tell um the large language model that it is a an AI agent for Koso outdoor products and as an agent you will answer qu questions briefly and succinctly in a personal manner uh using M and you'll see things out there safety there's a safety section you will always reference factual data so you're going to tell tell the large language model always refer reference large language factur data if you don't have that data don't go make it up um you're having things in here like um if there's insufficient if the if the search based this does not contain sufficient information to answer the question go back and ask the question go back and ask for more data I like this one hiden here when in disagreement with a user you must stop replying in in the conversation you do not want this Lang large language model to start now having an argument with your customer um and then what's going to be doing the next thing we're going to do is we're going to be folding in um the the um the product data which is relevant to the customer query so just basically iterating through the data that came back from my product database and then I'm going to iterate through and through the pre previous order history so now I've got context I know the products that are relevant to this customer's question and I've got relevant products and I've got also relevant order history and then we've got stuff like we've got I want to enter this template I want to put their first name last name so I can just personalize back the response so that's what's kind of going on with this Ginger template so what I'm going to do now is I'm going to run all run to the standard mode and that will now execute and in a moment they'll come back we'll go to outputs we'll open in a new tab window that's not showing entirely what I was expecting we'll see it down the bottom in terminal okay so this was what came back from the that that template so the question was recommend tent for Beginnings for beginners and this is all of the context that's come back from my product set and my customer order history including all the way down to the bottom you'll see that the customer name is Jason Brown and Jason is 50 years old okay and the membership status and stuff like that so that's all going in to this prompt okay so that's the a so we've got rag retrieve we've got a for augment the next thing we're going to do back in the workshop is we're going to go from step two to step three so step three is a generation so we're going to copy the template go across the code spaces again close the visual designer open and just paste that in save that away and down the bottom we're just going to click on the visual editor now what this is going to do we just resize to fit and you'll find this this UI be a bit challenging at times so t a trick have the uh prom Visual Studio code tool bar working on the right hand side it'll stop the properties on the on the left hand side bouncing around on the UI it makes it easy to track and you'll find that you want to constantly toggle um the up and down okay so there was a question why are we doing it inside um Visual Studio code versus just directly an Azure AI studio and you absolutely can do it in Azure AI Studio inside Azure AI Studio it's a design firston experience that's not entirely true but the the code but the code experience doesn't allow for things like debugging and all those kinds of things you want when you're building out these prompt flows so you absolutely can do this inside um uh as your AI Studio but doing inside Visual Studio code gives you a richer debugging experience now by the way I'm doing this in code space but normally I'd probably just do this on my own desktop that's probably what I normally do Okay so we've now got we've now got the application so we've grabbed the embedding from the question we've retrieved product data we've retrieved our customer order history we've augmented that whole meta prompt and now we're going to send the whole thing to a large language model in this case it's GPT 35 turbo so I'm going to run this run it in standard mode and this will take a moment and this will execute and you can see the result comes back it's now remember that first time I test of this this is now highly personalized to my product my customer data my my data so hey Jason Brown for beginners I recommend the sky twers tent catalog item 15 it's the trail Master tent so now we're getting responses back and then what you would do is you're getting back adjacent object with the relevant detail and you'd build out a web application and you'd have a series of links and these links would link off to this data that's sitting behind the scenes in your in your in your backend system so that's how you this that's kind of how You' kind of bring it all together okay now the the bit that I'm not going to show you and that you can't do directly inside this Workshop though this Workshop goes all the way through about how to go and set up the resources yourself and side aure so for the purpose of this Workshop you're it's much more about just learning about how prompt flow works but ultimately you going to want to go and deploy that and to do that you'll need to go and follow the notes around deploying to Azure so just keep that in mind I've shown you all the way up and doing local stuff with large language models but we can't do um deployment because you're not setting up an Azure instance so just keep in mind but if you already got an Azure instance or you going to set up your own uh you can set it one for free you can play around with this and actually go and deploy this and you remember ultimately what this is doing is ultimately this will create you a Docker container that will run this prompt flow and it will create a rest end point that you would call from your application so it could be your web application your mobile application desktop application it's going to call this rest endpoint pass through the the customer question and then run it through that prompt flow and then ultimately this data is going to come back as a Json object and then you're going to do whatever you want to do with that Json object for that customer experience so hopefully that answers um the question about how this is going to work okay so as I said all of the notes and the introduction the slides everything you will find in this um in the in this uh uh GitHub repo now the one thing I haven't covered is that I mentioned this briefly at the beginning there is also in that Workshop work you can do around evaluation and evaluation is super important because you want to about automate how well your your prompt flow is performing and you can set up a number of metrics in this case here I've got groundedness relevance and coherence so um does is does did the model respond back with relevant questions and you might be wondering how are we doing this so there are a number of styles in which you can do this this Workshop is going to actually use GPT version 4 which is a more expensive but slower but more capable capable model and it's going to use GPT version 4 to evaluate the response that came back from a GPT 35 model now you will see that that this this is all documented in that workshop and it'll take you through the process of using that um using that prompt flow to then go and do evaluations of that prompt and absolutely worthwhile doing because it's kind of fascinating that we're using AI to evaluate the performance of another AI model so that's kind of what's going on okay so that's that's everything there so remember the place to get access to all other the resources is on here so HTTP so take a photo or screenshot of that so it's aka.ms reactor Rag and you can do this in your own time I've left the resources available will be available from a week from today so basically what's today Wednesday so be available Wednesday next week so you have a week to play around with this workshop and just try out ideas and different prompts and and things like that so okay so are there any questions so I've seen a few questions that come through so please I think I've been answering the questions pretty much as we go so are there any more questions do do ask away okay so I think we're pretty much on time and um I think we're GNA hand over to suzan and she's going to close up so thanks much for taking the time and um um feel free to reach out to me on LinkedIn uh my LinkedIn ID is glove boxes so you can see at the top feel free to reach out if you have questions ask me on LinkedIn we my GitHub thank you very much thank you so much Dave pleasure thank you right so thank you so much Dave and to our moderator um Eliza who has been working behind the scenes as well so if you have missed the demos you can always watch back today's session that will be available on Demand right away so I'm just going to redrop the link again here and yeah I would also love to invite you to join us for our next let's get technical session where we talk about Azure Quantum ecosystem where the hardware software Solutions are all innovate with um via one cloud service so join us and Switching gears a little um it's the startup AI school so if you want to learn more about AI please join us at startup AI School um it's usually on Thursdays and Fridays or Fridays so if you haven't heard we're actually at the mid of season 1 and you can find all the sessions and details on aka.ms AI school um you can learn about the AI Trends and tools how to use co-pilot um and you know our next session is actually on responsible Ai and that's tomorrow so we would love to see you there so I've also included a survey for today's session if you have time to fill that up we will greatly um appreciate that and this survey is actually important for us to know how we are doing and if we should do something like this again um or what kind of topics would be actually um of an interest so um yeah let us know if you like us to be back so and finally if you enjoyed today um do us a favor hit on the like And subscribe and let us know in the comments that you want more of this so thank you so much for joining us once again we see you next week bye

Channel: Microsoft Reactor

Views: 2,494

Keywords:

Id: OpmqkFV-dPw

Length: 60min 19sec (3619 seconds)

Published: Wed May 08 2024