02. Azure using Python SDK: Access and upload files in Azure Blob Storage

in this video we will try to access Azure blob storage using python SDK and this is how our workflow is going to look like we are going to develop our code locally in Python I am going to create a blob storage on Azure inside this blob storage I will create a container and inside that container I am going to upload our files and because this blob storage and everything is going to be on Azure platform and we are going to do our development on our local machine so to access this blob storage we need to create a service principle this service principle we are going to create it on Azure active directory and we need like certain packages so that like we can access the files over here on the blob storage so for that first package that we need to install is going to be Azure identity and second package that we need to install is your storage blob as your identity is going to help us in retrieving the credentials so that we can connect to Azure platform and to work with this storage containers and files we are going to use Azure blob storage inside Azure blob storage we are going to utilize a service which is called like blob service client this client would help us to connect to the containers and the files and the blob storage so keep in keeping this in mind we are going to procure some resources so we need to create a blob storage we need to have a container we need to upload some files also service principle I have already covered in my previous tutorial if you don't know how to create a service principle you can refer that one I have already created in my last video so I am going to just refer that service principle now let's jump to Azure portal first of all we are going to focus on this particular diagram where we have to create a blog storage in container so let me go to Azure portal now and you need to click on storage account to create a blob storage you can see like I already have few of the storage account but let me create a new one so Resource Group is this I am going to select it what I have used in my previous tutorial uh common demo then storage account name I can give it and yeah I can Avail this name so this is okay I will keep it standard here also we don't need to do much I can just click on review because I I want to go with the default value so I can directly click on review and then I will just click create now it is submitting that request and soon we should be able to have this storage account created so deployment is in progress now yeah now it is showing like your deployment is complete so I can directly go to my storage account now I will just click on storage comment demo and inside this one I will click on container and container I can create a container so I will just create storage common container and inside this container we can upload our files so now you can see this upload option so I can click on this upload and I can upload any of the files yeah so I can browse for files I can take like one of the files from here and I can upload it so now you can see like sample 3.txt file if you look at this diagram so we have created a blob storage we have have created a blob container and we have also uploaded a file inside it right now the thing is we want to access this file and read its data from our code so how are we going to do it for writing any code first of all we need to install these packages so you can go to the terminal in your visual studio and then you can type pip install as your identity and it will install the first package for you second package what we need to install is azure storage block install Azure storage and then blog I've already installed it so you can see the requirement is already satisfied but yeah if it is not installed then you have to install it first before actually we can jump on the code now let's go into the code so this is the same thing what we have done in our previous tutorial we are using this Azure identity because we need to get this client secret credentials and then from this Azure storage block we are going to use this blob service client this will enable us to interact with our blog storage block container and the files and then I am just loading the information from my environment file so here I have pasted it the client ID tenant ID and the client secret what we have done before new thing what you could see over here this is your storage URL so in my last video in our keyboard video we were referring the keyword URL but in this particular tutorial we need to pass a list Azure storage URL I can go to the environment file and you can see here we have to pass over Azure storage URL now the question is from where we can get this is your storage URL so for that you need to go to your storage so I will go to the storage and I will click on this Json view here you will see this blob right so I will just copy paste from here and I can just paste it directly here so now we have the Azure storage URL I can again go back to my code here we are just retrieving the client ID Secret in tenant ID from our environment file file which would help us to connect to our Azure platform yeah so one more thing is remaining if you notice in this diagram we need to give access to service principle to this storage account so that like these particular storage can recognize the service functional similar thing what we have done with key Vault so how we can give it so I'm in storage common demo and here I will go to IM then I will go to add rule assignment I will look for a storage something related to storage and because we have to read the data we also have to upload the data so if I go with the storage blob data contributor then this will allow me to read write and delete access to Azure storage blob container and data here if I go for the contributor role yeah so if you only want to read the data from your blob Up Storage then you can definitely go with this storage blob data reader I will go with the contributor one I will select this I will select members and inside members I will search for my service principle and I will give this service principle access to this blob storage so we can jump to our code the first method which I have written it is going to list out the file record so let me go to the container and site container I have uploaded this sample.3.txt file and I can just download it and if you see like we do have only one line written this is sample three file I just want to read the data of this file from our code let's see how we can do that so first of all actually we need to pass a container name what is the container name in our case so we have given container name this storage common container I will just replace it over here and then we need to give the blob name now what is a blob name it is sample 3.txt so I will give this name over here I will save this and after that we need to utilize this blob service client because this particular service is going to help us to interact with blob storage so here you can see the first parameter it is asking the account you are this account URL is going to be what we are passing it over here this is going to be your Azure storage URL and then the second parameter is the credentials so credential is something which we are passing it over here so we are taking this and passing it in the blob service client now using this particular client if everything is correct then this is going to give us access to our blob storage once we get that access to block storage then the second level hierarchy is going to be container once we have created this this blob service client then we can use it to get a container information just to get that we are going to use this service get container client and inside this we are going to pass our container so that's what we have written over here the container name and once you get a container then we can go to our file and this is what actually we have listed out in our diagram also first using this blob service client you are getting overall blob access and after that once you have that client then we can go at the container level and once you get at the container level then at last we can go to the file or pull the level so this is what we are doing in this case also so once we get a blob service client then we have used this get container client to go to specific container once we go to container after that once we have this container client created then we can get the blob information using this get blocked client first we had the get continuous client now we have the get Block client and inside this we can just pass our blob name which is sample 3.txt in this case once you have the blob client then you can read all information from that file and just to read that we first of all we have to download the blog and then we can utilize this read all method and at the last I am just going to print it so let's try to call this method first so let's try to call this method now you can see this is sample 3 file you are also getting like this kind of quote it is coming in binary 64 format so I can decode it so if I want to convert in into simple text then I can just decode this information and let me try to print it again and now you see I'm getting a simple text screen uh text I'm getting a simple string text which is this is a sample three file the second method what I wanted to show you is to list out all blob so suppose like there is a scenario where you have a container in the Azure and you have several files listed over here you want to access all those files how we can actually get all those files information so here everything is going to be same so we are going to have a blob service client account URL credentials and everything and after that once we have this then we have to go at the container level so once we go to container level after that so previously what you are doing because we want to specifically access this file so we were just passing the blob name also and we were creating this block client but here we are going at the container level and after that what we are doing we are utilizing another method of this container client and listing all the blobs available for that we are just looping over the list of the blocks which is going to be retrieved from this container client and we are just printing The Blob name using this we can access all the files what we have uploaded on our blog just for this demo so I am going to upload few more files on the blog so let me upload few more files so we do have this sample and Sample two so I will upload that as well now you can see we have three files listed over here so what I'm hoping once I run this code once I run this code it should be printing all these three files so let me go to this list blob method and I will uncomment the second one one thing what I missed I also need to update the container name so let me update the container name so that we can list out all the files which are over here let me run this code now now you can see it has listed out all these three files information because this is what we wanted to print so we wanted to get the list of all the blobs and then we just wanted to print the name of this file in the next method what we are going to do in this one we have listed out the file name information but what if if you want to access the data of all these files right how we can access it so similarly here actually I am going to update the container name now first of all again we need to do same thing we need to create a service client using the URL and credential and after that the second level we need to go to the container level for that we are going to use this blob service client and then we are going to utilize this get container client so that we can get access to container we can go at the container level and once we go at the container level then we know like just to list out everything from the container then we have to utilize this list block this particular method which we have used earlier now the additional thing would be once you have the information of the blog then also we need to get the data of all these blobs right so what we were following in step one if you go to our first method we were creating this blob client and inside that we are passing the blob name right so similar thing we are doing over here also we are creating a container client then we are creating a Blog client and inside that we are passing this blob information blob name so once you have this blob information then it's going to create this blob client what we have seen in first method and then definitely after that we can just you know download blob and read all this is similar to what we have been doing in our method first but in in that case we were specifically accessing one file here actually we are accessing multiple files and just reading out all the data but if you see at the Baseline everything is same it's just that we are dealing with multiple files over here so now just let me run this code so last time it just printed a file name this time I'm hoping like it's going to print the data and you can see this is sample two sample 3 and in the first file we add more data so it has just printed out everything over here please do note I have used this udf8 to decode the information so that I can get the data in in the text rather than getting it in the bytes and the last method what we are covering over here so we are going to have some file at our local machine and we are going to upload all those files to our blob so let's see like how to do that so first of all we have to pass our local directory where actually we are going to place all these files you can see here in the input folder I have placed all these files and it's going to be same file so let's let me just go over there and delete all this file first now you can see I don't have any files in my blob in this storage common container so two parameters first is local directory where my files are located second parameter is going to be container name here I am going to pass the container name storage common container to get access to this common container first of all I have to create a blob service client and after that I have to use this get container client and inside this one I need to pass my container information after that I am just reading all files from my directory and for this I am using native python Library OS and after that I am just listing out all the file information from this directory and for that I am using this list directory this will give me a list of file and I'm just looping over all those files so here you can see for file name in file names which I have got from here I'm just trying to get a full path when I'm saying full path it means it's going to append the local directory path which is going to be input in this case and then slash sample.tx and which is going to be file name I'm getting getting a file name and from here I am getting a local directory so at the end of the day it's going to give me full path input slash sample.txt once I have the sample path so it's a native way of treating a file in a python I'm just using that to read the file I'm I'm using I'm just utilizing read method and once I have the data then whatever container client which I have created earlier over here now I am using upload blob method and when I'm using upload blob then I'm just passing a file name and the data so let me try to run this and see whether our files are going to get uploaded or not the file should get uploaded and they should be having the data also so let me just comment out this and uncomment upload blog okay so the program has completed let me just check on the portal Let me refresh it and you can see I have all these three files got created here this file is the bigger one and all these files are smaller so this is how we can connect to Azure blob and can perform multiple operation yeah so that's it for this video thank you for watching

Channel: TechyTacos

Views: 7,008

Keywords:

Id: bX-SSTCe2CY

Length: 16min 11sec (971 seconds)

Published: Sun Sep 17 2023