#25. How to pull data from RestApi and write it into Blob Storage|JSON| AzureDataFactory Tutorial |

hey hello guys welcome back to my channel my name is and in this video we will be talking about how to pull data from rest API using ADF okay so in like uh so I have already taught you in the previous videos that what is yesterday API what are the different methods in API and what is a pigeonation concept if in in API using python script and with the help of database platform so if you haven't watched those video you can go to my channel and you can search for this databricks and Pi spark playlist and here you can see this 39 and 14 number video is present over here you know which which tells like what is rest API and what is pagination in stpi so if you can see I have explained also what is get post put delete method of apis and then I have told you like what is pagination concept and like the same way the same procedure will be applied in ADF also but you know with the help of UI thing we will doing we we will be doing uh you know pulling the data but in these in these two videos I have shown you how can we pull data from the API from the same API with the help of python Escape or a small Pi spark code okay so okay so let's let's get started for today's video so for that I have opened this window Pokemon API okay you can see like we have uh this endpoint Pokemon Pokemon so if I click on the submit button then you can see we have this this Json and here here also this is the this is the presentation concept and here you can see we need this like we got this result where we are having name URL name here like that okay okay fine uh cool so let's get started uh so what we're doing for today's video is that we will create a rest API link service then we will create rest API data set then we will create copy activity then blob link service and blob data set okay simple so if I go back to my ADF I will go to this manage option here we can see this link service option click on this new button and search for the rest API rest fine so here we need to pass the base URL so the base URL would be till Pokemon okay so paste it over here and these are the different methods authentication type we have basic Anonymous service principle and so on so today's video we will be talking I'm like since it's a basic video so I will be using Anonymous method so you know in basic when you have a username and password then you will choose basic option and if you have a client ID and secret things then you will choose a service principle option so for now I am choosing anonymous because I do not have any credentials right since we do not have any headers also so I will not be choosing this method and yeah that's it so click on this test connection succeeded click on this create button cool so uh you know we have created a link service so what I will do I will go to this data set option click on this new data set and search again rest continue it to the link service and click on OK fine now click on a pipeline click on new pipeline and here I will choose copy activity why I am choosing copy activity because we need to fetch data from some layer some sources and put it into different sources right so that's why we need our copy data activity so if you can see in general option we have this general information so you can write down anything like based on your need and then you can see this we have source and sync so in Source it is asking a source data set since we have created our data set so I will choose this rest resources and here we have multiple options request method get put I get post here we have uh right and then the request timeout by default it is this request interval in milliseconds this is 10. and then we have an additional header option also with what is like if you need to add some additional headers we can add it but in this case we are not adding it and then this pagination rules okay supposedly like in API we have multiple pages so how can we fetch data from the multiple pages instead of only one pages right so that is the that is the way I mean we can use this this option to fetch data from multiple pages right now we are deleting it because we do not need it and fine so everything is clear over here if you click on this preview data so we will be receiving this response the same response we will be receiving see fine till 19 till 20 okay fine till 20. cancel it click on the sync button so we have a link service already present for the blob storage so I will click on plus Button as your blob storage and click on this Json because the response in the source side is Json object say Json format right so I will choose this Azure blob storage tool okay fine open it oh sorry cancel it and here I have to pass my container name so now container name is sync2 pass it over here and I will pass some name let's say Pokemon Dot Json right correct compression time everything would be same it's not required for now copy Behavior none metadata options nothing file pattern okay fine now if I click on this debug button and if I go to here you can see here right now we do not have any file dot with DOT Json name right so if I refresh it oh let's keep on running it got succeeded and if I flash it again here then you will see that we have this pokemon.json file so if I click on this edit button over here so you will see we have some data so the same data we have it right so if you like see this count one one five four and here also with the count is one one five four and the rest of this will be same um we had gone here right same same same so see these all count next previous let's say there's a scenario that there's only one page okay and you you are you let's say there is not a one page like we have a different multiple pages but you do not need this account next previous this these all other metadata columns right metadata uh attribute so we do not need it in our the in our end layer let's say you know so we do not want these layers uh these columns or attributes so what would you do is that see go to this mapping option okay import schema by default it will you know import now you can see discount next previous site we do not need this so what we will do we will remove it remove it and remove it here okay mapping sync is empty let's say remove it do not need it so we need only result side so I will keep this and here also let's keep the same thing okay and click on this debug button now if I fetch it now if I refresh it again you can see we have this result oh no this count and previous options aligning the attributes have been removed and now we have only result name you are like name URL name URL and name your other things okay because like we do not need that that options or attribute because it is not required in our the case so you can remove it if you want by this method suppose you have multiple things then you can choose multiple names over here multiple you know attribute or scheme up over here and you can you know map it map it so this is the basic video of how to pull data from API using ADF and I will be teaching I will be creating video on how to you know how to do appreciation options in here and how to pass a header and all those stuff in the next videos so I hope you have understood understood this concept and if you liked it please subscribe to my channel and share with your friends and yeah thank you

Channel: GeekCoders

Views: 7,234

Keywords: Geekcoders, Sagarprajapati, Freecontent, Azure, Dataengineer, Python, Sql, Data, Databass, Database, Engineering, Databricks, Azuredatafactory, Spark, Hadoop, service principal

Id: wBLV3mBjNYU

Length: 9min 44sec (584 seconds)

Published: Sat Oct 15 2022