Make Azure AI Real: Build your own copilot with Azure AI studio

[Music] hello everyone and welcome to the last episode the last episode at least for 2023 of the make Azure AI real series uh well we don't just talk AI we do Ai and I say the last episode for 2023 because maybe we'll get renewed for next season I don't know we need to write a letter to the producers and uh maybe can keep this show going but today we got a special treat we have Naila who is going to be covering it the a Azure AI studio and how we can you know build co-pilots build other reg generative AI applications all of that uh but before we join uh before we go we always start with a question uh again uh says we're talking about co-pilot I tried to do a little bit of travel theme since uh you maybe people are traveling as well for uh the holiday season so the the first the question we have is where's your favorite place to travel to uh and as always Nila we we asked our lovely guests first uh where's your favorite place to travel to um we're a pretty adventurous family we travel a lot uh but my favorite place is probably my parents house uh going oh wow okay this is that was a deep answer I didn't expect that one I need a lot of food I get to be very unproductive um it's it's pretty nice okay yeah I thought you you started out with uh adventurous I was like oh it's just going to like some sort of exotic uh location I don't even country I have to like find or okay parents that's a great answer I love it h I think for me my favorite place I've been I went to Sri Lanka for six weeks and that was probably the best trip I've had in my life uh so Top Choice cool so yeah let us know in the chat if you have a nice nice other travel recommendation uh but the other thing we always ask is uh you know what what you know our series is called making azure AI real uh so what does making AI real mean to you uh I think it's uh it's it's a great question and it changes a lot when I joined uh Microsoft and I joined this team I was working on traditional machine learning um and like classification type problems or what re or predictions right like you take a set of you take an algorithm uh you take a set of input training data and then you make a prediction um and that's that's that's what making AI real was at that time and I think in the last like eight n months it's a very different landscape uh in terms of we're going Beyond predictions to generative AI uh and that making that real is a whole different Challenge and that's what we'll cover to great so yeah let's get into it then uh let's see what what uh take us in the adventure of the into the world of azure AI Studio awesome I'm going to start by sharing my screen and we're going to talk a little bit about kind of what is aure AI studio and then majority of the time we'll just uh cover content in terms of just the product uh and and we'll go uh into it and we'll create a co-pilot nice awesome all right so when we look at a GU Studio One a studio was announced at ignite uh it went oh you're sharing the uh your presenter notes oh oh yeah baby did you get the second screen maybe that's what it where it's showing up at I don't know you know what yeah I heard something connect so yeah perfect all right sorry about that okay so one uh before we even get into generative Ai and what it can do for you let's talk a little bit about what it is uh as I was saying earlier traditional machine learning we make predictions uh based on a set of input training data with generative AI it's a little different we're going Beyond predictions we're using uh these llms large language models uh that are pre-trained on uh data that's out there and you're using those to generate entirely new content and that content isn't limited to text it can be limited it can also be images uh as well as we'll see video as well um and and these are some of the examples in terms of just with our journey within uh generative AI how our customers are using it they're using it for Content generation code generation semantic search being able to retrieve the right information uh being able to ground your model with your own information or your own data and and summarization but when we look at uh adopting generative AI it's pretty difficult um and here are some of the challenges that customers go to I think first is even just getting started um you'll you'll you'll see a lot of cool demos uh on social media around the different capabilities of generative AI um but then you yourself getting started finding the right resources which resources to use is pretty difficult uh when you try to map that to your own scenario uh second is then developing uh development there are a lot of Cutting Edge tools out there there are a lot of different Frameworks out there how do you put them all together and that's a that's not a new problem I think that's always been the problem with any sort of technology is like you'll see tools you'll see Frameworks um and then how do you put them all together stitching them together is pretty difficult uh context um so a lot of times when we're using these llms or you're using them in the playground you you'll realize very quickly that um they may not have uh access to the data you wanted to have access to because they were pre-trained um so then how do you ground these models with your own data uh so that they're retrieve information from from the data that you want them to um another challenge is evaluation again with uh having a lot of choices having a lot of tools uh how do you know the impact of those on your application uh so evaluation and looking at the impact especially before we uh put these things out in production we want to know how they behave under different scenarios so having power Val powerful valuation tools is definitely a challenge um and then last is oper oper oper putting them into production um uh and that is where really concerns our privacy security grounding your model uh making sure you have tools that you can monitor these with over time uh so you're understanding how they're behaving in production as well all right so with that uh we know what generative AI is uh we know the challenges folks have um and then I want talk a little bit about Azure I studio and I think the best way to do that I actually want to show you the product um and instead of kind of going through all of the features and listing them what I'm going to do is I'm just going to show them you and we're going to use it together one question I get um you know obviously Azure AI studio is in public preview now um and I think you within Azure we have many studios uh mainly also people get confused on what's the difference between this and the aure open AI studio uh great question uh one of the so one of the this is a challenge that we get from customers a lot uh they're not which sure not just which studio but which service to use um so with Azure Studio what we're doing is we're bringing those all together um so over time you'll see the capabilities from all these different Studios whether it is uh the cognitive Services Studios or on Vision speech or it is um aure machine learning or um as the example here calls out the um what's it called our Azure open AI Studio those capabilities are all coming into Azure AI studio and they will continue to do that over time uh and you'll be able to see more and more of those capabilities being brought together in here nice Yeah we actually got a question from the audience uh and Andre J I think I should be say better at saying that name uh basically straightforward question here when when GA when is this generally uh available we I don't have a timeline yet that I can share for ga uh but that is definitely top of mind for us and and also a lot of the servic is RGA uh so the studio UI itself is probably where the questions coming from but for example when I come into AI studio today uh I'm going to use uh our Azure openi Service uh that is a ga Service uh that you can take a dependency on um but in terms of the UI I think we haven't closed on that yet nice okay so it's also time you can submit feedback we can still make make it better so if you do see anything and you've been using it I'm sure the product team will appreciate feedback so that's great awesome awesome great questions I I love that folks are already like connecting the dots my next feel was going to be hey we're bringing all of these things together and the question was already that that's what we are about here at make Azure AI real is uh start building that's what everyone wants to do so I love these type of questions too awesome and I love the drive to go uh to G I think uh I'm glad folks are thinking that as well but before we do that let's take a look at the product um so one of the things we said uh earlier was hey we're bringing all of these things together um as uh so our customers don't have to stitch these pieces together themselves they don't have to go to five different Studios all these different Services we're kind of taking that on ourselves and here the way we do that is um the first thing you can do in here is you can create a project uh so here I'm in Azure AI Studio what I'm going to do is let's say I want to build my own co-pilot um and I can create a new project here you can see for example what we do as a part of that is we create something called an Azure AI Source uh I'm not going to create it now I've already created this but in here this is really what's uh creating all of those AI assets for you and it enables you to create or connect to existing uh AI assets let's say you have already have an Azure openi account you don't want to recreate that sorry resource you don't want to recreate that you can just connect to that through here as well and I can show you what that looks like so for example I created this project and within the settings here you can see all of the different connections I have to the different resources uh you can also see here the subscription I'm using you can see all the different Pro uh project members that are added to this these are folks that have permissions to um do different actions whether it's their developer whether their owner the owner can add additional numbers to it you can see all the different infrastructure that my Project's dependent on um and then you can also see the different run times that it's used uh for or station engine so here you're for example you're seeing within the project view you're seeing kind of all of these Services coming together and you not just are able to view these you're also able to manage them within the manage experience here you're looking at all of the different uh Azure AI resources that you've created uh and then you can see the projects within those so the hierarchy really is that I have an Azure AI resource that Azure AI resource has a set of projects within this and each project I I've control at two levels uh I can control kind of this container which is the aure AI resource that has a set of different projects I can have control at that level uh and where I can create shared resources for my organization or I can go within a project uh and I can have control in terms of who uh has access to that project which uh uh which other resources it's connected to and I can add members there as well so we have this hierarchy uh within AI Studio of being able to manage your AI assets both and and you can think think of it in a ways like the Azure AI resources at an organization level um so if you want to control different policies you want to control permissions uh and and uh you want to control folks having access to different assets within your organization you can use the Azure AI resource for that and the project then gives me the ability to within my own project collaborate with folks and add them and give them a different set of permissions so I'm going to go back here going to go back to this project that I created earlier so the first thing I'm going to show you is we're going to go to the playground we're going to remove this so within the playground here when I created this project uh actually let's go to deployment when I created this project the first thing I need is I need a model right I need to be able to deploy a model I want to be able to use it here you see that when I created that I connected to an existing Azure open resource and what that did was it pulled all the deployments uh from that resource uh with in my project and they're now ready for me to use uh I can even uh I can edit these in here as well so for example this is gp35 Turbo deployment uh I get to see here which model version in the advance options I can allocate different quota to here so it's not just you're able to see your deployments you're able to manage them in here as well uh we can leave that so now that I know where my deployments are this this is really what I need to get started I'm going to go into the playground here you can see um same model uh I can use my deployments here I have the GPD 35 turbo model and the first thing I can do is I'll ask this a question and this question uh I'm I'm asking it a question about something called Trail Walker hiking shoes um and this is let's say uh specific to my company data and as expected it doesn't know because again this model has been pre-trained uh and it has no idea what my company data uh is about right so it's saying hey I don't have access to this information um and and it doesn't know um and this is common right so you can use these models but then if you wanted to have access to specific data what you need to do is you need to ground this model uh within your own company data so it's able to retrieve that uh the way we do that is we come in here and we click on add your data uh before you add the data um lonus has actually asked a question maybe this is very actually specific to to his setup but uh is there a specific subscription or Resource Group uh we should be using um I guess to get this working uh no you you use your own subscription and then uh you add resources to that so when you come in here you just use your own subscription you and what the Azure AI resource does within the project initiation flow will create the Azure Azure AI resource for you um what you can also do within there is you'll uh if you already have an Azure open AI resource you can go ahead and connect to that so the project creation step you don't have to do anything just come in and the project creation step uh will allow you to connect to all of the right resources or if you don't have them it'll create them on your behalf for you got it yeah I think that's a good point and maybe another question that I've seen people ask about this but um let's say you don't have access to uh the Azure opening AI uh service um how useful is this can you still use this in in some type of way the AI studio uh yes so we're I'm in this scenario I'm focusing on Azure open AI uh if you'll see later when we go into the model catalog we have uh models from many different providers we have hugg and face models we have a curated set of fion machine learning models we have llama models so you have access to a lot of different models uh we and I'll show a little bit of that later but our key kind of ENT scario that today I was going to cover is around Azure uh open AI um so that's the one I started with first perfect L let us know if we've answered your question or not hope so awesome all right so here what I'm doing is I'm using something called Azure AI search uh and Azure AI search enables me to create a vector index that I've added my product data to so what I did was I took a bunch of different markdown files they're all different product documentation one of those includes the trail Walker hiking shoes and I added them uh to this Vector index um and the way we search this Vector index is through this uh embedding models um so what happens is when I put in a question it uh it it when I when I put in this question it searches this Index this index is able to chunk the data and retrieve the right information for me for that it's using an llm we have many search types uh in here the best is hybrid press semantic um what this does is it's able to understand the user's intent so for example if I ask this a question um uh we're talking about uh in this scenario it's like an outdoor equipment retailer um that that's building a co-pilot so if I asked a question that's related to let's say uh it's raining in Seattle what would you recommend it's going to recommend a waterproof jacket for me for example even though in the question I'm not saying that uh waterproofing is is related to rain but it's able to understand the intent of the question um and it also ranks it so for example within my product documentation if I have documentation that's around um Seattle restaurants or the best places to visit but within my question it's asking about a specific product or saying hey like I need a product recommendation it's going to know that and it's going to rank the product recommendation content at the top of my search um so this is the one I want to pick okay so once I do that that was pretty quick you saw yeah just one update uh so lus has said uh he's like following in real real time but uh he selected the create co-pilot and not the open AI model uh so that was a problem on selecting the resource um cool and also like if if you can can you just zoom in maybe one or two clicks on the browser so might be a little bit there you go that's better uh yeah I don't know yeah perfect that's perfect thanks yeah I think that's good cool all right so now here you see here's my Vector index uh and all the indexes that you add you'll be able to their component within your project so here I'm seeing the playground and you're seeing now me kind of slowly go towards um a scenario where I'm creating and building components for my application in here and those components will be available within your project so here's the index that I just created um and what I'm going to do is I'm going to ask the same question and this time I should know the answer okay so here we can see this time is telling me the trail Walker hiking shoes cost $110 and not only that let's minimize this low this is one of the product documents that was in there and you can see it's pulling it from this document and it's $110 which is accurate so here what I did was I very simple example I took a I took uh my product information I added it to an a vector index and then when I'm chatting within my uh within my playground I'm able to retrieve the right information um this is a very very simple scenario of grounding your model uh but let's say um I now want to understand uh a few other things uh we didn't talk about system messages your system messages are important this tells my application how to behave um so here's where you can edit your system message we also have the ability to edit your parameters in here so now you're starting to see here I'm I'm building variability of different options within my tool um and then I want to understand how does this behave with all of those different variables um one way you can do that is you can do a manual EV valuation so if I click here it'll take my system message same one I can edit this as well it'll take all of my parameters and the data that I've grounded my model with is here as well so now now what I can do is I can input a series of different questions or what could be quicker is I can import an entire data set and here you can upload a data set at U I think I don't know if I have any I do have some in here um let's try it here you can see I have a data set where I have uh questions and ground to truth so basically what I was I asked it a series of questions and I'm saying this is what I'm expecting the answer to be uh I map those and then what I can do is now I can run several different calls all at the same time uh I can compare those results I can flag the ones that I like and I can save this as a new data set um and this just is before I get into a very comprehensive evaluation this just helps me iterate on uh manually on just being able to see what are some of the things that I need to look out for for what are the things that I need to add to my data set perhaps before I even uh go into uh a more comprehensive evaluation so we got a a a flurry I I would say a flurry of questions coming in so are you ready we're gonna go we'll try to get all of these down um so uh SE freed has asked uh if they have a bunch of PDF files containing health insurance policies and plans uh can I load these into the studio yes so so right now I used markon files uh you can use PDF files as well it's you're what you're loading them into is your index so this index these indexes that you see in here uh you can have many different data sources for those you can take PDFs you upload them to your index uh through blob storage and then that your index is able to retrieve that information um so yes you can do that nice great uh Mikel has asked as well uh do you have any links recommended to training to get started so I asked asked followup in the in the chat actually if he was looking for specific to Azure AI Studio or just generative AI so I will take the generative AI part because this is my chance to plug our recently released uh GitHub for uh generative AI for beginners GitHub repo which yeah see Molly's always on uh right when I say it so that's uh a good training for General generative AI knowledge um it's all about building applications we have 12 lessons that we actually use Azure open AI service to build them and but I I'll leave it to you D is there anything that you've uh training Wise from the Azure AI studio now or documentation pages that you can share um we can put that in a collection yes uh I have an entire uh walk through a very a much slow walk through of what I'm doing right now on YouTube um I can share that as well um if you go into actually if you sign out of this and you go into into here I think I believe it's already linked in like the the homepage or the Explorer there's a big video in there of just going through the exact demo um um and then we also have we have a few assets actually we also have a code first experience for the same experience that I'm sharing that has an entire tutorial around it and then we have a tutorial of the studio UI end to end as well so there's several documents and and I can share those links great and we'll add them all to that link that's on your screen so uh perfect then you'll have all the resources for generative Ai and uh the as your AI studio um we had another question channel so everyone's kind of following along which is super exciting to see but also hard because of people have issues and then we can't troubleshoot them because we can't see what they're doing exactly uh but Channel had mentioned uh they encountered an issue uh when deploying a model uh they said that they got the message no quota is available okay yeah oh you already know already know is okay this is good all right so okay we're going to troubleshoot a little bit all right so when you go in here remember I was showing you these are your deployments uh here you can see when I edit that deployment in the advance options you can see I can add more quota here so this is where if you are getting that bug take a look at uh to see where the slider is um and you can always increase the quota um you have to right now you have to go to uh the Azure open ey Studio to request more quota that's something that we'll will add the ability to So based on the capacity and the quota that is allocated for this deployment you should be able to change that here got it and the question we got from Jason was is uh talking about the uh the data itself uh the the data that you loaded uh is your data model A relational database and if so does the model infer the linkages between the table uh to produce the answers right now what I did was I just it's unstructured data I just took a bunch of files and then I threw them in there um and then it's able to through that uh embedding model it's able to chunk that data uh vectorized it and then it's able to uh search that data and retrieve the right information for me and that's what's kind of cool about Azure AI search I didn't have to do a lot to my data I just kind of just threw a bunch of files in there nice just throw throw it at the model i' lovely uh and then yeah CH channels still try to do what you just recommended H they said the bar is grade out for them the uh uh that the bar that you showed okay uh that is probably would be a permissions thing they may so uh I would love to follow up on that um maybe that's something that we can follow with offline um but that likely is because they're uh they don't have the right permissions to be able to change that okay that makes sense got it okay so continue on we'll I I'll I'll take more the questions are coming but we can keep going with the demo there so much you have to follow okay so we up until now it's actually we've seen a very simple scenario right I came in I grounded my model with my data uh I'm able to retrieve information that's kind of cool right and I can do some manual evaluation where I can ask a bunch of different questions and I can see how it is um producing what the output is I can compare them I can save that as a data set so to it's kind of a good ramp up um but the reality is a lot more complex uh customers want a way to be able to add many system messages or prompts that we we call them um and they want to be able to see the impact of those on my app uh they want to be able to try many different models we've just been using two models so far the GPT 35 turbo and the a adaa embedding model I want to try a bunch of different models whether it's OSS models llama models and I want to see the impact of those on the same exact questions um so for all of these what you could do is uh you click open and PR flow um and this helps you customize uh this kind of UI that you see and when you do this uh I've already done this and I can go into this it'll create a what's called a prom flow for you prom flow was one of the first applications for llm Ops um that was in the market what it does is let's take a look at one of these it'll create something called a flow and this flow is that same exact UI but now you're seeing that broken up into nodes and you're seeing literally it's like looking under the hood of how that UI was built and you're seeing code behind each of these nodes that you can change edit and you can run this as well and if I ask the same exact question here how much is the trail Walker I can choose cost I get the same exact answer because it's the same uh it's the same uh UI you're just looking under the hood now here also what you could see you can see all of the files right so up until now um AI studio is an experience for developers and we haven't talked about code yet uh so you can see all of the different files that these uh nodes uh contain and then you can go ahead you can download them locally or what you could do is you can open this in vs code web and you can go ahead and actually I can show that to you once you open this in vs code web what'll happen is we'll create something called um a a compute instance um what this will do is it will create a workstation in the cloud so it'll create a development environment for that has all the packages that you need to be able to uh edit this application in code in vs code web um I've already done that for example it'll go in here and here now you see both my code that contains um uh my files as well as my data files are in here as well and you can also see here within the promp flow extension uh you can see here flows that I created within the UI you're able to see them in here as well you can iterate on those files or you can iterate uh within here as well and the changes that you make here uh will be reflected back in the UI so let's go back to here one of the things that we wanted to do was um we mentioned earlier that we want to be able to have we have different variables now right some of those variables are models uh some of them are the prompts um and and what we want to be able to do or we may want to add another data source right now for example my Cod within my application I only grounded it with product information let's say I want to be able to also ground this in customer information so it can query uh my customer database as well or maybe I want to ground it this with some other data source as well um here you're able to within your application you're able you're able to add those noes um uh and when you do or you're able to make those changes when when you do that then you want to know hey like what is the impact of those on my application this was one of the evaluation gaps that we saw earlier uh so for example let's take an example where what we could do is I have in here I have two different meta prompts uh in here so earlier the one we saw was a very simple one that said you're an AI assistant um I created one more in here that has a lot more information that's telling uh this application uh or this flow uh to be uh more aware of uh safety So it's talking about uh response grounding tone and sympathy response quality and what I want to do with this is I want to test let's say the impact of this like safe meta prompt with the basic one I had uh on a data set um so earlier that you saw we're going to use the same exact data set that we had earlier uh uh remember in the manual evaluation we had a set of questions and answers and we created that we can create that data set we can save it and then I can use the same exact data set throughout the product and I can make changes to this and I can evaluate the impact of those changes um to do that we need to do evaluations um so here you can click and you can do like a built-in evaluation um so I want to compare both of those variants of those prompts and here we can see like an wide array of different metrics that we have um groundedness ensures that the answers that the models generated aligns with the information that's in the input source uh coherence ensures that it's producing uh the outputs flow smoothly that it feeds naturally um we also have similarity what similarity does is it measures the difference between um remember if you recall my data set earlier that we were adding in manual evaluation I had something called Ground truth and that that's just what I'm expecting the the the llm to produce so it Compares that ground truth to the output of this and uh it me it gives it a f star rating from one to five to do that we also have traditional machine learning metrics uh so for example accuracy in here is in here as well so you can select these um the top a set of these are again they're GPT assisted so here you have to select the the model that you're using to calculate these uh metrics and then you can go ahead and submit an evaluation I've gone ahead and done that what I I saw that you selected all of them but is there a reason why you wouldn't select all of them or what's kind of the driving for evaluation like that I kind of like to select all of them actually because I all select all kind of person yeah for sure so I just go for it I'm like might as well select them all um okay you so let me show you when we're done with that what that looks like who is this one so for example when you set those you're going to see that evaluation running here and now you're seeing metrics at the top you see uh metrics for a summary uh AC average scores across all of the 10 questions I had around 10 questions in that data set uh you're seeing all of those at the top and I see for example my relevance is great um I see and this is for actually I want to show one more thing this is for one variant of f so what you could do is you can actually before here you can compare both variations so I had a variance zero and one and then here you see I'm comparing both of those and generally my safer variant is a little higher so I get to compare those for the metrics that I submitted you see those here and then I'm seeing also data index each row within my data set so every row every question that I had I get to compare the output of those for both of those variations and this could be like I right now I did like two different meta promps you can try two different models uh you can try other scenarios as well you can try one where you have two different data sources you can the possibilities are up to you but here for example you're getting to compare both of them um you get to see the input question what the output was what the ground truth was that's the same in both of those and you get to compare all of these in here um and you get to flag those you can also drill into each uh instance of those so I want to see this one and here this is just like one variation of those and for that variation again I can see the scores for every single metric that I had I can see the input question the answer and what's even kind of amazing is I can trace every API call uh that was sent so here I can literally see each API call and I can see the input and output of those and this is great for troubleshooting for example like if if this is uh minor changes in that meta prompt can have very different results on the output of the llm and you need the ability to be able to not just see that the output doesn't look right but you need to actually troubleshoot each and every step within that and you to be able to see the inputs and outputs of those um so you're seeing all of those in here uh both the inputs and outputs of every single call that it makes so you can very easily drill into this and this again is the UI uh the code experience does the same exact thing uh in here you'll be able to see your evaluation runs I think I had it here somewhere uh you can see your evaluation outputs right here as well so again this is like the code equivalent of that I'm seeing all of those for the same 10 questions I'm seeing it all broken down in here all right is there a way to kind of group those uh API calls it or like filter out on anything I think uh right now in the UI you can you you're fil you're viewing it by each uh instance right now um but that is something we could look into if it's needed for example you're saying here I want to be able to group or filter by these calls yeah yeah yeah not right now in the UI we don't have that um I haven't had an ask for it as well but if folks are uh in need of it I'd love to learn why and how what they use it for probably just me just me asking so don't go build it yet uh cool we have some other questions um so Jason who actually already asked about the relational database and and then the answer was that you just kind of threw the data at at it it was unstructured uh he would like to know um if he if I can I keep the mo my model as is uh or do I need to deploy data to an unstructured format like how does he have to make it unstructured to for this to work uh well I'm I'm not following the question uh he's saying can I keep my model as is yeah I think it's his data that what he means or his database uh not the model I think I'll have to like understand a little bit more about what his data is like and what scenario is doing before I can answer that it's not clear to me yeah it the classic it depends I think or a case by case um and then they asked questions about uh providing a link to to use a model stored in a database how's that database created and what are the tables so I think there's a lot of questions around the data part okay um we can we can drill yeah we can drill into a little bit of U what are the different data sources perhaps that's what folks are yeah yeah maybe that's probably the best I think yeah so if we go back into the playground here when I ground my model let's remove this you can have many different data sources and we'll continue to add data sources in here so for example one the earlier the question was hey can I just upload files when you upload files you you need a blob storage but you can pretty much upload files here directly will create a storage account for you and and will'll be able to create a vector index based on that so this is just a bunch of files that you can upload that's what I had done earlier you can also connect this right now you don't see it um but you can also connect this to uh one Lake as well um so you'll be able to see one Lake as an input in here as well um I believe we also have Cosmos DB coming in here as an input as well so we'll we'll continue to add different data sources but there are two different things one is like what are the data sources that you use and second then the vector index through Azure AI search that's an index that inputs that data source um so we have a set of different data sources that we are we are supporting supporting and we'll have a road map for those I don't have all of those at the top of my head right now um that's we that's something we can share offline uh and then what happens is the vector index inputs those uh and it's able to search those uh uh those different data sources and retrieve the right information so there were two things that were in here the data source that has the data and the vector index that's able to search that data okay so we talked a little bit about evaluation we talked about the tools where you can make different changes um you can iterate on this and this is kind of cool right because you can go in the and it's so intuitive now because you can go in and now you're probably wondering hey if I change one of those models to a different model I was using GPT 35 let's say I want to use a different model can I go ahead and iterate on that I would again then do an evaluation run and you can kind of continue to iterate in here and and find the best variation or the best uh output the best uh flow that you want that gives you those results and you're able to come in here and compare all of those um once we have those you can go ahead and deploy these to an end point um so in here for example this is not the one I was looking at but doesn't matter um we were looking I think at this one yeah so in here you can go ahead and you can deploy this to an endpoint and then what'll happen is when you deploy that what happens is it shows up in your deployments and I have several deployments in here um here you're able to see for example all of my deployed flows uh that have been deployed to endpoint and what you can also do is then you can enable let's see if I've enabled monitoring for any of these I haven't yet but what you could do is you can also enable monit tring for these what that does is the same exact metrics that you saw earlier uh right now for example this is now deployed to an endpoint I can test that I can consume that in my application but then I can also monitor it and view its logs and within the monitoring once I enable that what it does is the same metrics that we saw in the evaluation uh whether it's groundedness uh we saw similarity for example those metrics uh we calculate them and we show you uh we monitor you can monitor those same exact metrics in your deployed application to your customer and again you'll be able to do the same exact drill down where if you see certain metrics going down you're going to be able to drill into and and Trace those metrics and you'll be able to trace those calls and see for example when when uh my metrics were lower exactly what was the uh what document was retrieved what did that information look like and you're able to go troubleshoot those as well I know we were uh there's one question I know we were been focused on the the open AI asure the open AI models there uh but Carlos that asked um besides the open a models is there any other model that you recommend going through what you're showing here uh because you know obviously I think you Haven we haven't shown it yet but the the library or the catalog right there's different models that are for the you know different task or the same task so maybe he's also looking for a bit of how to kind of navigate that that whole world I I love like I love that everyone's like two steps ahead of me I'm almost there this is a fast forward show this is I said this is making things real so I'm almost there I'm going to cover one more thing that's super super important and then we're going to go back and we're go to catalog and I'll show you what you're looking for there so here you can see um I we deployed I just showed you how you can deploy uh your uh your flow uh to an endpoint uh you're deploying your application you can test it you can consume it um you can monitor it but you can also add content filters and what this does is it enables you to filter out um uh harmful content from both the input that the user gives into the application as well as the output of the flow um and depending on what your needs are you can change these different thresholds all right so here's content safety filters now let's go back to your question so we've kind of seen an ense scenario we came into a playground uh we looked at all the different variables I need to consider when I'm developing my applications I can add different data sources meta prompts I can try different models we deployed we evaluated the impact of those on our application we deployed it we added content safety filter and now the question is hey like I want to try different models what if I want to do more um let's go back and try that so you go into explore and here we have two um things for you first uh let me that's fine I think we just go look at it here first we have um our model catalog and here you see we have a wide we have thousands of models within here from different collections uh along both Azure open AI uh curated by Azure uh Azure AI these are models that that um we have ensured worked end in the product um we have also have Azure models we have Nvidia models we have hugging face meta models as well um so again now you can see that um a model was just one of the variables within my application you have the ability to use these deploy these and use them in your application so for example let's say you want to use like meta 70 billion Lama 2 uh parameter L 2 model you can go ahead and click on that you can deploy that um and you can use that with you can see in your deployments and you you can use that within your application within the playground and then within prom flow as as we saw uh as well um so this is where you can get access to those um one other thing that we're introducing with AI studio uh I don't know if folks here are using um Azure machine learning today or they're trying to deploy some of these llama models a lot of times they'll run into the same issue where they don't have enough quota um so that's one of the the issues that we are solving with the I Studio where um we're introducing some called Model as a service what that does is uh we've already hosted a set of these models and you don't have to worry about the Cota or managing the infrastructure you're pretty much just calling an API call and you're able to use that model um so that's one thing that we are we've introduced Ani Studio you can go ahead and use these models in a way where you don't have to worry about quota or infrastructure the other thing uh that folks get asked is like okay like I have so many now have thousands of models how do I know which one to pick um within AI Studio you can also Benchmark models here for example the scenario I was doing earlier within that chat was I was doing a Q&A scenario uh within here let's say I want to compare all of these llama and the GPD 35 turbo model that I was using as well um So within the benchmarking experience what you can do is for here we're just looking at accuracy right now but I can compare the accuracy of all of these models that I selected against uh a common public data set um for task and that task was question answering so for example here you can see the GPT 355 turbo model has uh higher accuracy than the Llama 7 uh 27b overall on average but you can also then filter this on different data sets so this can help you uh Benchmark and choose the right model for you uh over time we'll add additional metrics and we'll also add additional models here as well in the data set themselves um is there any way for people to uh not necessarily see the entire data set but know what that data set is about or you know if they want to compare let's say one like you did there right one data set that might be useful to their use case how do they know uh what's in that data set or at least the theme of it uh these are public data sets you can look them up um as well uh and if you select these it'll filter to that data set the other capability that we'll be adding in the future is for you to uh use your own data set to evaluate these and you can do that right today even like if you go into let's say you have your you saw me when I had my own data set I did an entire evaluation with that data set um and uh the the gp35 turbo models what I was using I can do the same exact thing with a different model but on my data set as well and I can compare those within the evaluation experience and pick the best one so you have the ability to do that within the product as well here we just wanted to make it easier uh when you're choosing a model to see it against like a standard like public set of public data sets is there any plans to add more models on that or I see you have okay great we'll be adding both uh more models as well as more metrics in here as well and tasks so we'll be building on top of this this is just a point in time in where we are today nice great so we got 10 minutes left um so if there's any I guess more highlights on what we can do with the uh the studio that'd be great I I would love to cover two more just kind of talk about we don't have time to cover those but they're pretty cool so we talked about the model catalog we talked about how you can choose many different models we've only talked about text so far um so with AI Studio you also have the ability to um there's so much here we can do two hours on this with AI Studio you have the ability to try different modes uh so for example in here I was using chat um we can also use image mode with there uh Dolly 3 is available within the image mode again then what you're doing is um uh you are you can generate images within there uh within completions mode you're generating uh descriptions and content uh and and and so on for example I want to create descriptions for my hiking shoes you can use that um and then we also have multimodal uh that we're enabling in a few weeks what that is is it includes our enhanced uh gp4 V turbo model and that actually inputs videos um and that's pretty amazing as well uh so I think the other thing to cover is just that there's a lot there are a lot of possibilities that are coming in uh through uh multimodality uh that are also within AI Studio we just don't have time to cover all of those today but they're pretty cool so you should check those out the other thing that I get asked a lot is like okay we have these models but let's say I want to introduce bias into these um so let's say I have a model or I have uh an open source model and I want to be able to uh introduce bias by find doing that for my scenario so right now like for example when I'm telling this model how to behave it's really through the system message but at some point like the system message can only contain enough information let's say I want to be able to tell this like uh that I wanted to behave I want to be able to tune this model for let's say like specific medical terminology or uh specific legal terms uh for that we also have the ability to fine-tune uh models as well with an AI studio uh and for those it's similar where uh we we're enabling uh through uh a service for fine-tuning where uh you bring in your data set you call an API and you're able to fine-tune these models and then you see those models again within the same exact experience they're just another variable within uh you building your application uh so they'll be supported both within prom flow as well as within the model playground as well so there's a lot to cover but I want to make sure I mention those because I get asked about those often very nice yes uh maybe you could maybe could quickly I guess if we got we still have around 10 minutes so maybe the fine tuning part because I I do also get lots of questions around that maybe just like what that what are the options what does that sort of look like uh let me show you uh I think let's see hold on let's see so I think it was this project yeah let's see so here for example um I went in um and here you can see um you can pick a model from this list we are supporting these three models right now uh you can fine-tune them um it looks like I have a permission issue no worries um I can follow it happens to all of us everyone it happens no worries um but I can show you what it looks like so basically you go through a flow where you add in uh if that worked uh if I had the right permissions to do that um I would have gone through a flow where I added my data set uh and then I would would have selected fine-tune and you can change your parameters for fine tuning as well when you fine-tune them uh this is what you'll see your models within your fine-tuning tab within your project uh so we can look at a few of these let's look at SWAT these so here for example she's already fine-tuned this model and you can see the metrics here so here you're able to look at different metrics to see if this model is something that I even want to use in my application um and then you can change parameters you can change your data set iterate on this until you get the rate metric and when you do then you can go ahead and deploy this again same thing when you deploy it it'll be within your deployments and then again that is an equation that you can use um that's a model that you can use in the playground you can use it in a prom flow you can use it within the within your application that you're building um so that's something that uh is available in the product um you just have to have the right permissions to be able to do it which is a good thing great I uh we got another question around the um L if there any learn module we actually have the link but I can't post the link on myself so uh to seek free that link that I was going to send you to you is going to be included in the collection um maybe Matt if you still in the background and have that link collection maybe we'll show that as a the closer yeah perfect uh so all the links there and there is a mic there aure learn module for uh the AI studio so even though I would love for you to replay this video and pause and watch it there is another uh self-directed way to do it as well great uh awesome so I guess that about RS it up do you have anything else to cover we don't have any other questions um but I think this was a really great high level like you said you probably talk about this for two hours because there so many uh different features there uh but anything else that you like to to point out before we close it out um no I think we're just really excited to uh help our users go along this gen AI Journey with us uh so for example um the team that I work on within the Azure AI platform team uh and when you look at a lot of times customers look at like our fancy Co pilots like whether it's Bing chat and so on um or M365 these are all built on Azure they're built on the same exact platform what AI studio is doing is it's giving uh you the ability to use the same exact platform the same tooling to build your own co-pilots and that's a pretty exciting Journey so you basically are your P You're the pilot to make your own co-pilots we are the we are we are the the the co-pilots to make your own co-pilots maybe that's a better way to say yeah maybe maybe I don't know but the destiny the Des is in your hands everyone uh really appreciate this this demo I think it was really uh really well done in depth on giving a a little bit Parts but like I always say you know we're trying to make azra AI real so definitely get your hands on this and uh start building we've got all the learn resources there and I appreciate all the questions it's super nice to see people already follow following along and um looking to build something and like I always end with you could have been anywhere on the internet right now but you spent an entire hour with Thea and I so that I'm truly appreciative and that's going to close out this series for at least this year uh maybe we see you in the next year with more uh make as your AI real goodness bye everyone thank you bye

Channel: Microsoft Reactor

Views: 5,233

Keywords:

Id: M1lSDT-5hQE

Length: 54min 22sec (3262 seconds)

Published: Sat Dec 16 2023