How to use and secure Azure OpenAi using Private Endpoints | Full Demo

hey everybody thank you so much for checking out my channel and for watching this video in this video I'm going to be showing you how you can connect to Azure open AI service first I'm going to show you how you can connect using public endpoint then I'm going to show you how you can secure that using a private endpoint and how you can connect that from an from a v-net and also I'm going to show you how you can connect even if you have a different tenant as long as you have a v-net pairing in between so I hope you enjoy the video let's get to it [Applause] so first things first let's go ahead and talk a little bit about open Azure open Ai and and what it is and how it works well Azure open AI is the implementation of open AI but inside of azure which means that you have the same tools you have the same capabilities however you have them in an environment where you can apply security governance and you can pretty much have it in your environment so you can have users it's just the whole um Azure environment that is around the open AI implementation so what that means is that when you have openai and Azure you can create users you can select security you can pretty much put um private link so that you can access it from inside of your environment so it's just a way that you can bring all those tools in an environment that is controlled by you through governance security and things like that the tools or the models that are available are the same ones well relatively the same ones that are available in openai which is chat GPT that the three the four as well as codecs embeddings are available and Dali 2 is also available however you it's you have to apply for it open AI is also generally available right now however you still have to go through a process that you can that you have to ask for that service to be available in your environment once the service is available in your environment then you can do what we are going to do now which is we're going to go ahead and access it I'm going to show you how you can access it and how you can test the models very quickly so let me go ahead and get to the portal and let's go ahead and get get started okay so before we get into the portal what I want to do is I want to talk about the architecture and what we have and how we're going to do this um I don't have a fancy whiteboard but I do have this surface that I'm going to use so I'm going to say that this is the Azure open AI so this right here is the Azure open AI is not the the the open AI that is out on the internet so when we look at this what we're looking at is we are looking at the portal and you'll see Azure open AI here when you click on it you see your instance here and you can click on it and at that point you can see your end point this endpoint is the public endpoint so when I when I see the public endpoint and I ping it ping the endpoint itself it comes back with a public IP address so this is how you connect into your openai so again when we go back in here you have a public endpoint this is how you connect into into that open AI instance so if you have a v-net and we're going to go ahead and put a v-net here and let's say the v-net IP address this is an internal IP address and it let's say you have a 10.10.10.0.24 if you have a VM here that needs to access this open AI even though they are both inside of Asher you still have to I'm going to go ahead and put the internet here just so that to illustrate even though it's not inside of azure but you'll get the idea the VM will have to Traverse the internet to get to this public endpoint which was a 20 dot X IP address for me for my endpoint so this is how you access the openai instance in Azure now you might think well this yeah this is good but what if I want to secure this in a way that is not available out on the internet well in that case what you can do and I'm going to show you later in this video once we get to the portal the demo you can create what is called a private link or private endpoint private link or private endpoint private endpoint private link and what it does is it creates an internal connection from the open AI instance or open AI service and what it does is it gives you a an IP address inside of your v-net so in this case it would be a 10.10 that seven for example it it's just an IP address that it gives you when you create this private link what it does is it updates your internal DNS because every v-net has a DNS server and the VMS point to that DNS server so what that DNS does is it takes this name that you have and in my case my my endpoint has a name like I showed you when I pinged it when you create this private endpoint it updates DNS and it updates it with this name this open AI name but it points to the internal IP address so any VM you don't need to make any changes you still continue to use the same name however the translation or the you know when when the IP address gets when the DNS gets queried it Returns the private internal IP address not the public endpoint so this is what we're gonna do so let's go ahead and get started what we have here is we have one VM that I'm going to show you and that VM is going to access openai the first thing it's going to do is I'm going to show you it's going to access the public endpoint and then we're going to set up the private link or the private endpoint and then I'm going to show you how it access it from the inside all right let's go to the portal let's start the virtual machine that we have here and I'm going to start this b1s machine so as you can see it stopped so let me go ahead and start it okay the computer is running so let's go ahead and take a look at the IP address copy that put it into putty assuming the security groups are open for SSH [Applause] let's connect into this box okay once you are logged in let's go ahead and look at the files that we have here we have a file that I that I'm going to to use we are going to use this file called chat GPT v2.py which is a python script that we are going to use what is this python script doing it's going to import openai import logging import sys and import OS set up the credentials and this right here is your public endpoint or your endpoint and then you have your openai key the open AI key is the one that you get from openai so let's go back to the portal let's go to open AI click here and here under keys and endpoint is where you find it here is the key you take you copy this key and then you bring it into here and you and you type your key here so then at that point we just have to specify the deployment name a deployment name as The DaVinci customer experience text DaVinci 03 you can find that under model and deployments you have to um it was moved to the open AI Studio but this is where you can have your deployments you must have a deployment otherwise it's not gonna it's not going to work so in my case I have a gpt35 turbo and this is my deployment and this is the one that I'm that I'm going to use in my test here so again you must have a deployment you give it a name and then here after that what we're doing is we are just um just generating a text what I'm what I'm actually doing here is so what I'm doing is I'm taking a file so this is deaf main I'm taking a file and I'm and the file name is transcription.txt and I'm setting up the context and I'm saying the context is read from the file so I'm saying the only the information that you're going to give me is it has to be in this file and then while true this is just what you know it just continues to ask questions question input enter your question and if the question is in lowercase then then then exit so generate a text so this is just what it's doing so the main thing here that we want to test is make sure that it's connecting to the endpoint so let's go ahead and exit out of this go here and let's go ahead and execute this file so chat chat uh chili python I'm going to use Python to execute it python3 chat GPT v2.py so now when we execute it it says enter your question okay which is what we saw in the in the code what is wrong question mark and now what it does is it takes the transcription of the file and it's just a transcription of another video that I used when I explain how Chad gbt saved my Dad's life so what it did is I said what is wrong and it just gave me this an 80 year old male that had a prostate biopsy yesterday and it gave me information okay so the next what I do want to do though is I want to Ping my endpoint so you can see [Applause] okay so that's what it's translating to is translating to a 20.232 IP address that is a public IP address all right next thing I'm going to do is I'm going to come here under private endpoint connections I already have a private endpoint created the way that you create one is you say private endpoint um it's going to ask you for a resource Group you could give it a resource Group you give it a name and then network interface is gonna you can be a little bit more specific open AI connection for example and then you have a Nick remember the connection that is going to be created is going to become a network interface like how we saw here it becomes a network interface in this v-net this is the reason why it's asking for a nickname and then the region again the region is where your v-net is for me I want to create the the connection in west us2 or west us3 I'm going to see West us3 open AI is in the east region so by creating one in the west I'm creating that connection back to here so open AI is in Us East or East US however you want to call it and this is going to be West us3 for example so that private link is going to be able I'm going to be able to connect using this under resource account virtual Network this is where you select the virtual Network that you're going to use and the subnet I'm going to select the test plvnet and the default which is the only one I have and dynamically allocate IP address the DNS automatically is going to create a private DNS zone for this so that way it can translate or you know just give you the right information that is going to resolve to the right name or to the right IP address and then you can do a review and create so what happens when you do a review and create is that it creates this private endpoint I already have one so I'm going to go ahead and and exit out of this because takes time to be to create it so I have this private endpoint here if I click on this private endpoint information what it does is it tells me where it is it says that it is in the test ndi fmd v-net and the default subnet and the network interface is called open aipe Nick and it's a private private endpoint if I click on this Nick it should have a 10 right here 10.1.0.10 IP address okay so everything looks good I do have my private endpoints set up the only thing I need and I'm going to go back to this and you can see it still it's that's the IP address that it was resolving before the only thing I need to do is go to open AI again click on the on this link go to networking go to private endpoint and make sure that is there and then go back to firewalls and virtual networks and set it to disabled I can click on selected networks and private endpoints if I want to select other v-nets if I only want the private endpoint that's all I want is I can just set it to disable once I set it to disabled it says no networks can access this resource private endpoint connections will be exclusive way to access this resource so in other words only private endpoints can access this resource when you set it to disabled so I'm going to set it to disable and I'm going to save it okay now this is saved so now what I want to do is I want to look at my um virtual machines this virtual machine here I want to I want to see where it is so let's click on it and it is in the virtual Network called b1s machine v-net in the default subnet this is not in the v-net where the private endpoint is so in other words what I have is I have another v-net here and this VM that I'm using is here and this this v-net is called b1s machine v-net so the theory if the theory is correct the private endpoint is not in this v-net right now okay so in theory I should not be able to Ping this issue I should not be able to connect to open AI because my machine is in a v-net that there that does not have a private endpoint so let's go back to here and let's go ahead and try to execute this command again python chat GPT what is what is wrong question mark how I get permission error Public Access is disabled so right now what I'm what this is saying is that I'm not able to get to this resource because only devices in the right v-net can access this resources okay I think this test and the IV it's this one is in the right v-net and it is a Linux so let's go ahead and start this one okay so the test ndi fmd v-net VM has started so what we're going to do is we're going to copy this IP address and we are going to set up a session okay we have two virtual machines and two different v-nets this is the this is in the v-net where we have our private endpoint let's see what we have do we have yes we do have a file called open AI test so let's let's do a ping okay so this one when you ping it you can see that it translates to an internal IP address of 10.1.0.10 which happens to be the internal IP address of what we saw earlier so moving forward that is the IP address the configuration of the file is exactly the same so I'm going to do Python 3 and the file name is openai test Dot py and it says enter your question I'm going to say what is wrong question mark and goes out there and it answers the question says there is nothing wrong and the reason is because the file that I put in here is shorter than the than the original file but it what I'm trying to show you is that it responds with with the answer so that means that at this point we are able to get to open AI using the um the connection now if I if I want to make this one work I mean if I want to if I run it what is wrong this one still should not be working okay so now what I can do is I can create a v-net pairing and this two uh there's two v-nets I gotta go to the virtual Network so I need to create a v-net pairing between the b1s machine and the test and the ifmd so let's do that again different regions this is on West us3 this is in central us so let's go to this one let's go to peerings this one has a peering two test plvnet already but I'm going to create one two two B1 as machine v-net appearing link name to I don't know test ndi v-net so allow allow we're gonna use that and we're gonna say subscription is this subscription and the virtual network is this v1s and allow traffic and add okay let's give it a minute this is this connecting and this one's just updating and this one says connected all right let's go to the b1s v-net let's look at the peerings and there it is it's already connected let's go back to this right here let's run this file again what is wrong and public disabled Okay the reason this is still not working is because of the DNS if I ping um so what I the reason is because it's still trying to go through the name and the name still resolving to the public IP address so what I need to do is I need to update the DNS so the easiest way is update the the host file on this host and I'm going to 10.0 10.1.0.10 and I'm going to foreign and I'm going to write in quit so let's run this again the word is wrong question mark and now it should work all right we're supposed to type quit okay exit and now it's um it's back out okay so now going back to what what we did when we created so we started by looking at the v-net West us3 the VM was connecting using the public endpoint then we closed that we establish a private private link or private connection to the West us3 this VM was able to connect to it without any problems because the DNS was updated automatically then what we needed to do is we needed to create a connection from this VM here from the b1s machine v-net that one didn't work because it was trying to go out through the public endpoint which is closed so what we ended up doing is we ended up creating a v-net appearing between these two v-nets and after doing that the network connection was okay however DNS was not set up because remember every v-net has a DNS server so this VM was pointing to that DNS and that DNS server didn't know anything about the endpoint so I ended up updating the in this machine I ended up updating the Etc host file once I did that then the connection started working so as you can see setting up all the little details is not very difficult as long as you have a picture of what you want to do and if you enjoyed this video please give it a thumbs up subscribe to the channel for more content like this and if you have any ideas and any content that I can create something that you would like to see please leave it in the comments down below I always like to see those comments thank you so much for for checking out the channel and for watching this video and until next time take care

Channel: FreddyDubon

Views: 5,947

Keywords: azure ai, openai, azure openai, azure openai privatelink, azure openai private endpoint, azure private endpoint, microsoft openai, chatgpt, chat gpt 4, azure, openai tutorial, openai playground

Id: vpVh9h9i4Eg

Length: 24min 44sec (1484 seconds)

Published: Mon Jul 03 2023