Using Keyless Auth with Azure AI Services

hello everyone thanks for joining us for the very first episode of our securing AI apps on Azure series using keyless off with Azure AI Services my name's Anna I'm an event planner for reactor joining you from Redmond Washington before we start I do have some quick housekeeping please take a moment to review our code of conduct we seek to provide a respectful environment for both our audience and presenters and while we absolutely encourage engagement in the chat we ask that you please be mindful of your commentary remain professional and on topic keep an eye on that chat we'll be dropping helpful links and checking for questions for our presenters to answer live our session is being recorded it'll be available to view on demand within 24 to 48 hours on the Microsoft reactor YouTube channel with that I'll turn it over to our presenters thank you both so much for joining us today hello hello hello everyone Welcome to the first session of securing AI apps on aure you can see all of the sessions that are coming up in July in this list here and you can register at this AA aka.ms s-135 I don't know if that's Le speak for anything oh like less s less maybe May uh so yeah register for that and you'll get you know you'll get notified about all the ones coming up and we're covering a lot of great topics here so today I'm joined by marleene do you want to introduce yourself yeah hi everyone I am Marlene and I am on pamis team I am a developer advocate here and at Microsoft and I focus on Python and AI um and yeah I've really been enjoying it so far big fan of of keyless authentication which we are talking about today so um yeah I'm very active in the python community in in general yeah yeah Marlene gives like multiple Keynotes at PE pons every year by very active that's very true I yeah so I'm Pamela Fox I am also in Python Cloud aacy where our job is to help python developers be more successful with Microsoft products whether that's Azure or vs code or GitHub copilot or whatever and it's it is really fun to work with all of them and of course for the last year we've been working a lot with AI so that is what we're going to be talking about today so first we're going to talk about key based authentication and why we don't want to use it this is what lots of developers use today and I may be what you're using and here I'm going to show an example of open AI right so uh the open AI SDK the python package defaults to keybase authentication so that's what it usually assumes people are using and uh in in a way that makes sense because if you are an open.com user like that you're using the API provided specifically by open.com then that only supports an API key for authentication right so there you have to pass in a key if you're using open.com now for as your open aai you can pass in a key that is one option with it so if you're trying to use as your open AI then you know you pass in your as your endpoint uh your API version and then you can pass in a key so if you're using key based authentication today in Python with open AI your code probably looks like one of these uh you might also be using a wrapper on top of the open AI s DEC like Lan chain or light llm or something like that and similarly they they often have uh this way that you pass in a key and Marlene here is going to be talking about Lang chain later so if you are a l chain user stay tuned for Marlene SE uh second half now uh you could also be using keybase authentication with other aure services right so if you're using like as your AI search or as your key well key ball be funny yeah you can use a keybase with KEYT um a your document intelligence AZ your blob storage those are all via the Azure SDK uh here the as your python SDK and it has this a your key credential class so if you're doing key based off with those then you have code that looks something like this right you pass in a key via that key credential class so that's what your code might be looking like if you're using keybase off today however there's a lot of risks of API Keys uh the big obvious one is that API keys can be easily leaked I've seen so many people put keys in PRS and then you know I help them try to get those PRS you know removed from history and and you know taken off the web but it's really really hard right so once something gets leaked you have to first know that it's been leaked and then you know rotate your key wherever you're use it all that stuff so they can be leaked publicly which is a huge risk right uh the other issue is just like internal leakage if you're at a company and uh you know you have multiple developers across different parts of the company using keys you might actually M have multiple uh you know multiple teams using the same resource and not realizing that and then they're like using up each other's quota and maybe causing each other like rate limit throttling issues and not knowing it because they don't realize that they're all using the same key and I've actually seen that happen in some pretty big companies uh and so generally like you can run into these issues with keys and then if you do run into them it can be painful to rotate those keys and and bring in you know new values for the keys and make those changes all you know at the same time so there's a lot of risks of API keys so we're trying to encourage developers to move away from API keys and instead use keyless authentic indication and this is something that is possible with almost all Azure services at this point and it's definitely possible with Azure open Ai and other AI services so that's what we're going to show today is how we can go keyless so first I'm going to talk about how to do keyless authentication for local development and when we talk about keyless authentication it is based off of o2 you don't have to be an expert in o to you know to use it um but uh you know it's might be helpful to know that it's based on O2 uh but basically the idea is that we are instead of using a key we're going to get an aatu token and that token proves we can use a resource so in some way we are using a key but it's like this temporary key that expires really quickly and that we have an existing mechanism for getting a you know getting a new one up right so we'll get a token we'll use that to get a resource and it'll expire in sumon over hours we can get another token you know once that one is expired so we are still passing in a string that proves we can use a resource but it's a token instead of a key which means that it auto expires and that it can be refreshed easily uh and so that's that's a real big advantage of it so the general steps for using kyos authentication while we're doing local development is that first we have to create the Azure service and then we need to give our own user permissions to use that service so we we give oursel a role that says hey I have permission to use that Azure Service uh we log in to Azure locally as our as ourselves our user and then we use the Azure identity Sate to generate either a token or a token provider and we pass that along to the SDK for the service to prove that we can use it so we we going to look at doing this for um you know for as your open AI so here's the project I'm going to be uh using here so you if you want to follow along you can go ahead and Fork this project It Is aka.ms aaii keyless and it has everything set up so that we can do keyless authentication with Azure open AI uh so first thing we need to do is create an AZ your opening eye service if you use this project it'll actually make it for you um but I have a few that are already made in the portal so I'm going to go ahead and go to the portal and find and as your opening ey resource I can go ahead and let me find the one that this one made uh so open AI keyless is the name of it all right so I'm just going to search for this in the portal so I'm going to try and show you multiple ways of setting up key list so that you can use whatever works for you because some people like portal some people like CLI all right so this is my aure open AI resource here uh now um so what so what I can do is go to access control and go to uh add roll assignment so you see this add button here I'm going to add a roll assignment and what you can see here is this list of possible roles you can do and this is a bunch of preset roles that correspond to like kind of usual ways of of using a of using a product so there's this one here which is cognitive Services open AI user and this is the one I most commonly use now we can even we can look at this role and see everything that's available to do right it can um you know list operations it can get policies for responsible AI it can look at deployments right so if you want to you can make completely custom roles that have only exactly the permissions that something needs and that's a good way of like really really locking something down um but a lot of times we can use these existing roles because they correspond to like a very common usage right so I typically use the cognitive Services open AI user built-in rooll and this is these are the permissions it has this is what it looks like in Json so you can see this is what all the data actions corresponds to that that it can do and uh yeah so I can go ahead and select that right let's refresh that page there I did something I did a I did a no no all right so I'm G to refresh this okay and then I'll just do a little filtering here so I can find it so cognitive Services open AI user so I select that then I go to next and here is where I assign it to someone so in this case I want to assign it to just my you know myself the one that's actually logged in here so I'm going to say assign to user and select and I find myself in here okay that's myself great um and that looks good click next I can review like okay I'm going to assign take this role and assign it to this user so then I say review and assign so that is how you can use the azure portal to give access to an Azure open AI uh resource now we could also use the a your CLI so if you like using the AZ your CLI uh we can we can do that let me make it so I can copy paste here all right so the first thing we need is our uh we need a few things we need our PR our our users principal idid that's what we call it it's like the ID for our user so we can say principal ID let me make this way bigger woo okay principal ID equals and then I have to remember how to use bash think we go like this here we go or zush or whatever it is okay so that's our principal ID and then what else do we need we need our subscription ID so we get the subscription ID and the way I get that one is I go to the portal I click on subscriptions and I can see my subscription ID here and I'll just do like this and then we also need a resource Group so I here I'm going to assign it at the resource Group level so there's different Scopes you can assign these permissions at in this case I'm going to do it at the resource Group level you could actually do it uh for just the resource itself depending on what works for you so I'll go ahead and find the resource Group that this is in okay so now I've got my principal ID my subscription ID my Resource Group now I should be able to run this command here and this is the a roll assignment create command so we say this is the roll so this is the funky string that corresponds to cognitive Services open AI user so once you start dealing with keyless off you're going to see these funky strings everywhere and they correspond to these built-in roles and it's a nice idea to like add a comment whenever you have one of these strings so that people know uh you know what what it actually corresponds to I have yet to memorize these all right and then we can roll run this here and it will show us that the roll is created so that's for the Azure CLI so that's one way you could do it one final way you can do it is with bicep and uh this is my favorite way and as most people know I love bicep so bicep it is infrastructure as code it's similar to terraform if you're familiar with terraform the idea of bicep is that we can declare all the resources we want to make and we can deploy all of them together with a single step uh so I really like that because a lot of my architectures are complex and made up of multiple Azure resources and with bicep I can declare them all uh you know in this one place and deploy them all connected together correctly ly at the same time so this is the bicep that's inside the the demo repo so it creates a resource Group it creates an aure open AI account with a single deployment in it and here is where it makes the user so this is my role for myself so I set the principal ID equal to this parameter principal ID I set the RO definition ID to that magic string which means cognitive services is open AI user and I set the principal type to user now this principal ID is coming from this main. parameters. Json and this comes from my azd environment so I always use azd a is a CLI that can uh you know deploy Things based off of bicep so when I do ASD up it checks to see what my log his principal ID is sets the principal ID parameter to it and then sends it into the bicep and then that becomes the the role here so that is the you know the third way that you can create these roles so once we do that we need to actually use the you know use a a uh this Ro base access in our python code so in our python code what we need to do is bring in the aure identity SDK this example is in Python but I think it should hopefully be similar in other languages so we bring in this as your identity SDK we also bring in the openi package because we're going to be using that and what we do is we use this default aure credential from the Azure AI identity STK and what that does is that it actually tries multiple ways of figuring out how you're logged in so it'll check your environment variables it'll check to see if you're logged in with the aure CLI it'll check to see if you're logged in with the Azure developer CLI if if you're logged into Azure Powershell there's so many ways that you can be logged into AER and default Azure credential will go through the chain of all those ways you can also if you want to you can be really specific like if you know that you're always going to be logged in with the a your CLI you could do that you could also do you know uh I think it's I don't remember how to I think it's this maybe uh so you can be really really specific um but it's kind of convenient to not be specific and uh give people give developers more flexibility for how to log in so that's our so we get our credential and then based off that credential we get a barer token provider so we say hey we need we need something that's capable of giving us tokens for this oo scope for this credential so this is not a token itself this is a function that we can call to get new tokens and the reason we want a token provider versus a token is because that this is more compatible with like token refresh and token expiry if we only got a token that token would work the first time we got it but then like an hour from now it would just stop working right so what we do is we get a token token provider and then the open SDK can just call that token provider every time it needs a token and make sure that tokens are you know up to dat and re and they're not expired so that's what we do is we get a bare token provider and um and it's basically a function that can be called by the open AISD and then when we construct our open AI class here we you know specify our endpoint we pass in the token provider and we do not pass in a key so look ma no keys all right so then let's try it out and uh you know I can just run this code here it looks like it's asking for IQ yeah it is and and there we go that works so this is what it looks like to use keyless authentication locally um I see a question from John AIS that it doesn't work with ACD login uh it it does work with ACD login that's the only way that I'm logged in right now I actually don't usually log in with other ways so it should certainly work with addd login um at least the default AZ your credential should work if you look at the intellisense for this it does the very final thing it checks is AZ your developer CLI if you are having issues it might be because it's getting confused by something earlier in the chain so that is the drawback of default Azure credential is that it can get confused by things that are earlier in the chain so I saying you have two options there one is that you can specifically use only the type of credential that you're that you know people are going to be using right if that's like as your developer CLI credential that's class or you can also do this exclude so a lot of times I do exclude shared token cach because I know I'm never going to use that right uh there's a bunch of excludes like this so you could do like exclude uh maybe environment because I have seen people get get uh messed up by that uh so that's another option is that so you can either be really spec if something if it's not working to have it go through the whole chain you can either explicitly exclude things that you know that you're never going to use and that are maybe problematic and tripping people up like the two here or you can be really specific and say the credentials always coming from you know uh a particular you know a particular place like the Azure developer CLI but the thing about default Azure credential is that it's it's going to work a little bit better once we go into production um so yeah you have options is the thing to know great uh great question there okay so so that's working now right so we did bicep we did the SDK now one thing I wanted to show just um as a kind of a bonus thing you can do if you're getting really into keyless off if you're getting super excited you're like yes everything's gonna be keyless no more keys anywhere what you might want to do is actually enforce that keys can never be used with your open Ai and the way to do that is the very confusingly named parameter disable local off so when you hear this you think oh no if I disable it I'm not going to be able to use my open AI locally that is not what it means you have to think of this as like disable key based off that would be a much much better parameter name ignore that uh it would be a much better parameter name than disable local off because as you can see I have disabled local off true and I just ran this you know locally uh and uh so it's a weird name I think it's because traditionally uh people would tend to use Keys when they're developing locally and so that's what it got called and it got called this across multiple Azure services so you're going to see disable local off true in a lot of a lot of my bicep for lots of different Azure Services um but it basically means disable key based off so if you do this then you no one will be able to use keys with this open AI resource so that might be a good thing if you want to make sure to really lock things down uh it could also be a bad thing if you do have some developers on your team that have to use keys for some reason uh so you definitely want to check with your team uh because I have put set this to tr in some of our projects and then heard from developers like oh unfortunately actually some of my people on my team need to have keybase access so that's you know that's a team decision to make and you can also if your team decides like yeah we're on board we don't need keys for anything you can set a policy at the um at the resource Group level or the subscription level that says oh no one can have keys on their cogntive services account in this whole you know Resource Group or sub so that's something you can set H as a policy and there's an article about how to do that here okay so now that we've seen how to do keyless off locally the next thing is how do we do keyless off once our app is deployed on Azure right if it's deployed on app service or container apps or kubernetes or functions like whatever your favorite as your platform is for me it's usually container apps or app Service uh how can we then continue to use keyless off in that environment so that is going to be a little bit trickier um because we need the app to have a way to identify itself and for that app to have permissions to use the open aai resource so what we do is you know create our Azure open AI service create our you know app in this case an aure container app uh then we need to create an identity for that app to use and once we've created that identity we have to give that app identity permissions to use the openai service so similar to how we gave our our own user permissions we'll give the app identity permissions to use the open AI Service uh and then we'll use the a your identity SDK um and make sure that it knows which app identity we're using so for this example here is the AA it's aka.ms key- as your- container apps so if you want a minimal example of how to do keyless off with a your openi and as your container apps this is the sample for you so the first thing is that we need to have an identity for our container app there's actually two options for kinds of identities one is called the system identity and the other is called the user assigned identity now a system identity is one that is is has a onetoone association with a container app so it you know as long as a container app is alive the Sy if you have system identity turned on then that system identity will be alive but if the container app you know gets taken down that system identity will not exist anymore right so it is tied very much tied to the app and since it's a one toone relationship the container app always will have uh this this same system identity the second option is what's called a user assigned identity and there you create that independently of the app and then you tell the app hey you can use this user I send identity so it has a lifetime that is independent of the app and that can be convenient if you want to be able to use the same identity for other things as well and you don't want it necessarily so tied exactly to The Container app and that's actually what I'm going to use in this example here so how how do you assign identity to container WRA so we're going to once again look at different ways of doing that uh so let me go to find my container app here uh so here I've already created it um and if we go into the portal and we look at identity and what we see here is two tabs there's the system assigned Tab and then the user assigned tab right so system assigned we could just turn that on if we want to use a system ident identity you can see I have it turned off because I'm not going to use it in this case but you can just turn that on to get uh to get the system assigned identity I'm using user assign and you can see that I already have it uh I already have it assigned here and you can actually have multiple user assigned entities so that's the interesting thing about user assigned identities is that in theory you can actually associate your container app with multiple user assigned identities so it's not a onetoone relationship it's like a potentially like a many to many relationship actually so you can see here there's an ad and you know I could go ahead and assign another one here if I wanted to and say okay yeah also this one you know is also and identity and then you know and then I could add that to the app uh so what I'll do is just remove this one so then I can show so I can add it back and show that we can add it back uh let's see all right so it's taking few seconds to remove that you can check up here to see the activity log for when it's doing things I don't really remember how long oh there we go okay so it's refresh refreshing refresh refeshing it's still showing there so we'll just go ahead and just re reassign it just from here it'll probably just let us reassign it anyway chat Pro no okay it's not gonna find it because it's still showing there it just really wants to hold on to that so uh this one I did assign using biceps so let's just go ahead to the bicep because really that's how I did it anyway uh so let's look at this repo here so this is the one um that I'm demonstrating right now so what you can see in this repo is that here's the same bicep that I showed in the local Dev section where I you know created the role for my own principal ID and it was a principal type user now when we're doing something in uh you know on AZ your container apps we're going to create a role for its identity so I say okay take the ID of that that I've assigned to that app assign it the same user you can see here that it's the same string so this is cognitive Services open AI user and in this case the principal type is going to be service principal so they do have two different principle types right for me it was a user for local Dev and for the deployed apps identity is going to be a service principle and for this principal ID we can take a look at how I got that in the aa. bicep you can see that this is how you can create a user assigned identity in bicep it's actually really short amount of bicep you just say I'm making a user assigned identity this is the name for it and here's the location and then uh we just pull out the principal ID from that user assigned identity and we also assign it uh to The Container app uh around here is where I assign it in this bicep so we need to create the identity assign it to The Container app get the principal ID from it and then use that principal ID to assign it a role and that sets it all up now that's a lot of steps but once you do have it working then everything is connected uh there is actually uh another one more step which is that we set an environment variable of as your client ID so you see here how I set this environment variable as your client ID and I set that equal to the client ID of the property which is different from the principal ID so we have multiple IDs and that can be a little confusing but we said the as your client ID to the client ID of the property and the reason we do this is because it's going to make it easier to use default as your credential because default as your credential actually looks for as your client ID when trying to figure out how to authenticate so now we can go ahead and look at the dashal code so here um is the code where I say okay I'm going to create a default a your credential you can see I excluded something I knew I wasn't going to use and and then I get the token provider so I didn't have to I didn't have to tell it what the ID was to use because I set as your client ID so if you set as your client ID to the user assigned identity ID then default as your credential will just work if you don't set it you do need to explicitly pass it in so you would have to do something like this like managed identity client ID equals you know if you had it set to a different variable You' do something like this so that's the tricky thing about using user assigned identities is that you know there's a many to many relationship between apps and user iend identities so default aser credential needs to have some way of knowing which user iend identity you're using so you can either set it as an environment variable or pass that ID's explicitly in to default a your credential uh I think it's easier just to set it as an environment variable because then you can have the same python code that works locally and works in production right so I can go ahead and like run this uh locally here right so go uh get my local local local okay here's my local running here all right and let's see let's see how much this is what did Maximus do Maximus is the horse behind me right now I don't know how much opening I knows about Tangled let's see if it make something up here we go captain of the Guard yes so maximum likes security and that's why he's sitting with us today does he does Maximus like security okay and this is working locally so this is using the default a a credential locally so it's that using that user role that we set up locally and now we can also try the same one in uh in production so here I've run ASD up so ASD so this whole all these repos um you know work with ACD so I do ACD up and that creates everything in the bicep H so it creates a resource Group an open AI container app a registry container app environment all that stuff and then deploys it here so we can go and check it out in production all right and in this one I want to talk about caterpillars why are swallow tail caterpillars so cute I have a bunch of them outside on my dealer right now that are really exciting um but what about their horns their horns are so cute there we go the osmeterium they're so adorable okay so this is working in production and we can go ahead and look at the environment variables so if you look at the environment variables you can see that we are specifing as your client ID and this corresponds to that user signed identity and and that's it so we don't have to pass in any key we just have to use the identities and the roles and as long as everything has the right permissions then you know then things should work and you know you can go and you can insect your identity and you can look at its access control and see you know uh that it has the right Access Control if you're trying to debug this sort of thing you can check all of its you know permissions here and there we go so think let's see so we talked about the portal we talked about bicep and I think now we move on to marleene to show linkchain and doing keyless off with linkchain and other cool things so let me go and show start off with this slide and then we'll go over to your code okay thanks Pamela uh so a lot of the things that Pamela has just presented I will be actually putting into practice um with Lang chain so Lang chain is a different um it's sort of a thirdparty python package I will tell you a little bit more about it so the code that you can see right now on the screen is basically most of the code that you can use right now I will show some code as well that is uh for using L chain with Azure with embeddings and and a retriever but um currently we're still waiting for PRS to be merged so I will explain as we go along so um yeah let's go ahead and switch to my terminal um and I will walk you through so for anyone that doesn't know what Lang chain is I thought I would just give you a quick introduction to it so Lang chain is a python and JavaScript framework um for develop for develop developing applications powered by large language models and it aims to simplify every stage of the llm application life cycle so um in the development stage it's also open source which is great and in the development stage so for anyone that is actually working uh let me go ahead and also increase my screen so that you can see that nicely but um for anyone actually working on building out an application one of the nice things about Lang chain is that it acts as a bit of like an abstra Direction so what you're able to do is you're able to use these building blocks so say for example you wanted to build a a chat app that utilizes rag but you also wanted to be able to be really flexible with the the um with the language model that you're going to use so maybe at some points you want to use Azure open AI but other points you want to switch that out with like um maybe fi or whichever whichever other language model that you want to use with Lang chain the base code will remain the same but then as you progress and when you want to switch things out you can just change like one line of code and just replace um the L the language model that you're going to use so I think that's really helpful because yeah it kind of acts as an abstraction in that way um it also allows you to sort of productionize your app in a short space of time so whether that's inspecting your code monitoring and evaluating your code as well and also the uh response from the llms you can do that with Lang chain and then also they have a deployment thing as well where you can turn the chains that you build in Lang chain which are just the components that you build and put together um you can turn those into apis as well um I think it's really particularly for me I think the most interesting part about Lang chain is the development part so really being able to quickly uh take uh you know take building blocks and build out sort of more complex applications in a short space of time I think that makes it really useful so we're going to look at two examples today um in the code and to do this we are going to first need a couple of dep dependencies um for this to work we'll need the Lang chain package itself all of this is in Python um but again you can do this in JavaScript too um we'll also use the Lan chain open AI package so you'll have to install both of those and we'll need Azure identity just like um Pamela mentioned before and the azer search documents um package as well so those are all the dependencies that you'll need uh also if you would like to just try and run this example um I do think we have an AA for that I J like I think you can find the AA in the links in the comments and if you just want to run that you should also have ACD installed um and at some point I'm just going to update the code soon but you should be able to run AG up and have all the resources and keist because of BF have keyless authentication just work and um that's one of the nice things is that if you use addd and bicep together um yeah it takes away a lot of the manual steps of doing things which I think is really great um again a note on key list authentication if you get stuck anywhere I think Pamela has some great uh writing on this and also just this past video um this stream is great as well if you get stuck and then you can also check out and use the Azure open AI key list template too so I've already installed these in this example these are all of the dependencies that I mentioned at the top so the first thing we want to do is we want to actually take a look at using Lang chain in a similar way like we would um with just normal Azure chat open Ai and um in this way we first want to import it from Lang chain so like I mentioned before you don't have to specifically use Azure openai you can use any other language model but here we're importing it from the Lang chain open AI package we're also importing like we saw the default Azure credential and the get Bearer token provider so the same things that pamelo was using in her previous demos we're using those same things here and then we're going to pull from default Azure default credential as well that will be what we'll be using to sort of authenticate our um our application so this is the code to run it in uh Lang chain here the first first thing we're doing is we're setting the version we are also getting the Azure inpoint and that you can just set in a sort of as an environment variable and then we're passing um our token provider which here we had set our Azure credential to be the default credential and we're also uh setting passing through that we want the cognitive Services um is what we want to access and then what also going to be passing through the Azure open AI uh deployment that we uh would also have been provisioned when we run ACD app so all of these things should also should be provisioned in an environment A.V file once you run a uh a app and so we can run this and just like we would with Azure so we can pass it some sort of a a prompt and instead of for in Azure chat open AI when we're just using that we use it's slightly different here we use llm invoke so we're calling this llm here and this is how um sort of chatting using the chat API in Lang chain works so we can just print this which I'm just asking it to write me a Hau about tennis at Wimbledon just because that is what is on right now and I'm really interested in tennis it says grass quot so pristine wimble en's Grand stage awaits Champions rise and so that worked um and it's the response is what we can expect even if we were just using Azure open AI with um without L chain as well so uh that's what a simple I would say what initial beginner um use with Lang chain and Azure open AI looks like uh without keys and then the second example I want to show is a more complex example with embeddings in Azure AI search uh with this example I do want to give a caveat here that like I mentioned uh Lang chain is open source and so we are updating it right now to make sure that it supports keyless with indication I actually have a PR in that both Pamela and I have uh worked on a little bit um and hoping that that gets moved soon and when it does you should be able to to run all of this code at the moment um we're still waiting for the pr to get merged but when yeah when it does this should work so what we want to do in this example is that we want to um use the uh the the open AI Ada text eddings model and we want that to be able to take in a document uh and with Azure search uh create a Victor store where we can give a document the document is is changed into embeddings and then we can do some semantic similarity search on that document and so here how we do that is we want to import the Azure search uh uh Azure search module from the vector stores in Lan chain community and then we're also going to import Azure open AI embeddings as well so that's all we need to get this started again like we saw above we're going to set uh a token provider we're going to pass through default Adder credential and cognitive Services here and that's what we're going to set for the embeddings to work um we'll also make sure that we're specifying which deployment the open AI embedding deployment uh and this is specifically like the deployment name that you named uh that will be named for your Azure deployment so not um even though we're using the open AI Ada text um model that's not what we're passing here which is a little counterintuitive but this instead we're we're um passing through the name of the deployment that we gave in Azure and then the second thing we're passing through again is the endpoint and uh and then the token provider here and then we also can use um oh I need to take the default the other thing away but but here we're going to use Azure search and um we are instead going to pass through the Azure search inpoint here we're going to also specify that we're going to be using the embeddings model that we created here to be able to embed our documents once we pass those in so that's what we're doing here and then um we are also going to be using our Azure default credential here um I think I was just testing things out with this exclude managed identity credential which we should not um but it should still work so let's go ahead and run this code and that works I should let me just delete this here and um we will also it'll be creating for us um Lang chain will create for us uh a vector store an index called Lang chain Vector demo in our VOR store that we just created so here we have the Victor store ready to go to upload documents to it and so in this cell here we are going to uh import a text loader and we're also going to import a character text splitter and these are what we're going to use to be able to load a document up to a vecta store and also split it into chunks so that um so that it's able to to were able to search through and our semantic similarity search is going to return the results that we can expect we're going to use this txt file which I just um imported which is just I think uh a document somewhere from the internet with the president just talking about Co at some point a couple years ago and I'm not going to rerun this cell just because it will take some time for it to actually embed the document but this is what it should look like if it was successful so we can see that it was successful here because we can see some things output onto our screen in our terminal um and then finally here we are going to use the Azure uh search retriever which again we can use with with default Azure credential and um the retriever all it does is it does that it's another sort of component of Lang chain that we can use that's part of search that we can use to retrieve documents that are related to a query so it will go ahead and it will bring back some content you can tell it how many documents um it should return um once you are doing a search so here I will say I want the top one document the I want the first document that is the most relevant to my search and we're also passing through again this is a search this is part of the search um API and so we're going to pass through the search service name or the endpoint here and again we're going to specify that we want to use this index which is the L chain Vector demo index that we created um earlier on as well that we specified here and so when we run this our retriever will be created and we can then use this retriever to look through and and and run some queries queries and it should give us back a document um with information that's relevant so if I press play here and we can see we have some information some content that is returned to us about covid that the president has uh spoken about so this is and this in in this whole notebook here we didn't use Keys once uh which is really good and um we were also using different um different uh available uh sort of apis that Azure provides so we used the Azure chat openai um API here we used the openai the Azure open AI embeddings model here we used Azure search and we also used Azure AI search Retriever and in all of those cases we were still able to use um our default Azure credential we were able to not use keys and this is sort of the ideal workflow I think that we're trying to move to is is really just allowing us even in third-party apps to be able to be keyless and and not use um any yeah to have better security in our code so I will say that if anyone would like to um use this notebook I will have it I think the link is is somewhere in the comments and you should be able to do that too so thanks all right thank you Marlene it's I always like to see like an endtoend rag there just casually casually do that very nice uh so and I also love that Marlene showed that uh that we can use keyless off with other services and that's something that you know we should be doing across the board when uh you know when possible is is using keyless everywhere and a lot of the same things basically like everything we talked about applies to other Azure Services as well in some ways it's actually open AI is harder because usually with open a we're using like a a community SDK with it um versus using like our official aure SDK so the aure SDK actually makes it really easy um if we can switch back over to my screen uh right uh so if you're using just the as your python STK and you have a credential right so we use this default as your credential and then uh with the Azure SDK we can just set up a search client and just pass the credential directly in so if you are using the official as your sdks then it is super duper easy because you just pass in that credential and it knows everything about how to get tokens right it is it is very built into that right so you can see we do that for search we do it for container we for search index client um you know this is a data Lake which is different kind of blob storage you know we just pass in that credential so um yeah the AZ your SDK makes it really easy and then you know the same thing we just have to make sure that we have the roles properly set up for whatever we need so for example for search um you know I actually have here I actually have a few different search roles that I set up uh so I have like a search reader role uh which is used in some situations here this is like uh I need it for you know some optional feature here I do need a reader rooll uh for search contri I only need it if you're using the user upload feature right so we can like optionally set up roles only when we need them um and uh you know we just assign them to the principal ID of the back end and you know and then it will work um you know I've got it for computer vision for document intelligence so these principles apply to pretty much everything you're using just look for what is the role that I need to set and uh you know who am I giving that role to what role do I need locally what role do I need in production just making sure that those are all set up and you can always go into your portal and check the access control and see you know does my user have the role I expect does my app have the role I expect uh it's I will say it's definitely harder to debug this sort of keyless off right like when it's not working my think Mary knows like yeah it's not working you'll get this like dreaded 401 I think right and then it's like H what's gone wrong because if it's not working like well you could have a few problems it could be that you're not actually logged into the account you think you're logged into like maybe you're logged into like your personal account or something like that right so first check to see who you're logged in as um and check try and like check to see what kind of authentication it's using you can usually like kind of see in let me see if I can see it in the logs here if it shows um yeah so here you can see actually in this case it I apparently I am logged into Azure C so you can see that it says that default Azure credential acquired a token from AZ your CLI credential so that would be the first thing I'd look to see is like where is my login status coming from and is you know is that where I want it to be coming from and am I logged in correctly to that one uh and then the next step is seeing does my does you know does this user does this app identity have the access control and that's where you can go into the portal or use aure C to check the access control and make sure that it is set up correctly and make sure it's the right role we've certainly had a few situation where like we weren't using the right role for example uh if you did there's another role that's uh that's as your cognitive Services user which is not to be confused with cognitive Services open AI user if you did let's see if I can get it to come up um uh I can't get it to come up but if you did cognum Services user instead of cognitive Services open AI User it's not going to work so uh that's the sort of thing to to check when you're going on this debugging process here all right and uh there were a couple questions that came up as well about like the cost of this so you should be able to create managed identities for free so it you should not incur any um any extra costs you see here managed identities can be used at no extra cost there are some times some limitations where some like a free tier of a service may not fully support managed identity and I an example is as your AI search supports inbound managed identity meaning that you can authenticate it authenticate keyless but if you need outbound meaning that letting the search index use managed uh identity with another service like with an indexer setup like if you're trying to do like a blob indexer with your AI search that doesn't work on the free tier so there are a few situations like that where there's not full support for keyless off you know in certain tiers of services uh but generally you should be able to use keyless off and not have it be a cost concern so that's so that should not be a blocker for adopting keyless o um so uh you know I think the blocker is more just around complexity and and just getting everything set up that we talked about and yeah for next steps uh if you want to start going keyless with a your open AI there is a great tutorial that shows how to do it in your favorite language uh so we showed lots of things in Python since we're from the python team but we know people are developing in all sorts of languages so we have uh Neto uh let's let me show the actual code because that'll be more fun uh and we've got yeah we've got so many options here as your CLI as your Powershell bicep as your portal and then down here for the default AER credential we've got net go Java JavaScript python so we're really trying to make it easier for folks to use keyless off and this article is about as your open AI but we're adding the same kind of Articles to AZ your AI search and other services as well so if you are having a difficult time using keyless off in some particular service in language do let us know like tweet or two or whatever and uh you know and we can we can talk to the content team about that and uh and then they have samples that you can get started with right so the ones that we showed in in this presentation but we've got other samples as well uh that show how to do it you know on other platforms like app service and also other languages like JavaScript typescript C and we're working on Java uh there's a PR to add manage identity to like Lang chain for J so that should help with the Java front uh there's a lot of PRS in Flight right now as we try to you know get this get this working across all the most popular you know packages for working with as your services so thank you Marlene for working on the PRS it's it can be hard to change H you know libraries that aren't yours right that's true hopefully soon though so yeah yeah uh so yeah so if you want to learn more about securing AI apps on Azure these are our upcoming talks after this one so the next three are going to be all about user off so if you want to have a user login and user log out on uh you know next July 8th we're going to be talking about that and showing external ID if you're doing like a a customer facing app that should be particularly fun but also for Enterprise apps as well uh we'll have another one about uh being able to do that locally we'll have one about doing it for spa apps if you're doing like a react front end you know with whatever sort of back end we'll talk about that on July 10th then we'll talk about data access control for rag applications rag is what Marlene just showed if you need certain users to be able to ask questions about certain documents then you need data access control and that's what we'll talk about there and then finally we'll talk about deploy deploying an AI app to a private Network on Azure and that will incur extra costs in terms of a cost discussion so that's definitely something you need to think about costs uh but it can certainly increase your security so there you go lots of stuff coming up and yeah just uh tweet or toet at us if you have any feedback definitely thank you everybody for joining us thank you so much to our speakers for another awesome session first of a great series you can learn more about this series at the URL at the bottom of the screen register for upcoming episodes we're always looking to improve our sessions and your experience if you have any feedback for us we would love to hear what you have to say you can find the link on the screen or in the chat today's vent code is 2301 0 we'll see you at the next one

Channel: Microsoft Reactor

Views: 417

Keywords:

Id: IkDcQvKoQ8k

Length: 59min 42sec (3582 seconds)

Published: Wed Jul 03 2024