AWS - Kinesis Data Stream - Hands On!

given my name is Manish and in today's video I'm going to talk about Kinesis data stream service Kinesis data stream is a real-time streaming service in AWS platform Kinesis family has some other components like kinases firehouse and kinase estate analytics but in today's video I'm going to focus only on the Kinesis data stream there is a use case which we are going to cover in this video and this is a hands-on video so I will show you step by step like how you can configure the data streaming service to read the streaming data and load into the s3 bucket so we will create an ec2 instance and in the ec2 instance we will have the logs in the real time scenario suppose some application will run on the ec2 instance and it will generate the continuous locks in the ec2 instance and that log will get pushed into the Kinesis streaming service and then from the Kinesis streaming service it will get consumed through Kinesis firehorse and from the Canisius firehouse you can save the data into s3 bucket let's move into the code the first step is you have to create a role you have to just go into the I click on roles create a role you can click on ec2 instance then next here you have to define a policy to do this POC I need to have access on a still pocket click on s3 so I selected the full axis then also I should have access in the kindnesses streaming server again I think that is enough no go to next kinase is Timo so I have created our roll call can assist mo the next step is create a user and that user should have a programmatic access so I will select this option then I'm creating a group here and I'm saying this is also an assist mo and again I can search the policies and then create the group create user you can see like when you will select the programmatic access you will have this access key ID and 6:8 access access key I have downloaded this one I will just go here and I mean copy this one to my main folder that I have kept all other fine so I copied here Oh first thing is done the whole escape that all is created now second is I will create a ec2 instance launched instance click on feed here select our Amazon Linux one I am selecting the free tier I will just use the default configuration so I want just one instance I will use a default V PC network but here only I need to have a role if you click here you can see the role will be reflected here that all which we have created launched so here you have to give up on give a key bear create a new clip here we see two instead instances created ok so meanwhile what I will do I will just go quickly and I will copy this one into the main folder ok I will get a streaming service now and see there are three services when is our streaming service firehouse and Kinesis data analytic I will select our streaming service can assist mo it is asking like how many number of sod you want to give so for this demo I will just used one just to write capacities 1 MB per second or 1000 data per second and that heat capacity is 2 MB per second okay so now the streaming service is getting created so now it is successfully created so with the streaming service you can see there are different options like monitoring options which I will show when I will push the data into the can is a streaming service kind is a state of our house so this is we are creating because once the streaming service will get data from from the logs then we need to push the data into somewhere and kind from currency state a stream we cannot post the data it is just a pipeline some consumer needs to connect with this pipeline so that's why we are using kinase a state offer on date of our house and you can see here that you can have direct put or other sources or you can have you can plug in your currency status streaming services so I selected the currencies data streaming services so it is asking which streaming service all you want to select so I selected the one which we have graded okay and I will say and this demo kind of this host so that will be that way it will be not confusing okay a tan transformation you can enable the data transformation and you can create your own lambda function over here and you can do lots of transformation logic here but for now I will not do it but in the later at one section I will show you how to write a code using lambda and how you can do a basic trans transformation in the firehouse ok and also you can convert the record the streaming record the way you want so you can have a option like you can have a pathway or worse appointment okay but for now I am NOT going to do it but yeah in in one section I will definitely show you how to use the format conversion click on next it is asking like what destination you want to push the data the option is Amazon s3 it saved the last Excel services and is plaintiff for the simplicity I am just going to push the data into s3 bucket but it is asking like do you have a quality pocket created or you want to create a new pocket I did not get any pocket and I still I will just create one bar kind of system click on next it is asking for the buffer size interval okay so I will just change it to minimum click on next here it is asking like what is the I am zone you want to create a new role or you can use the existing Road so I'm going to check if I have existing role ability now okay I can use our new role and it says five hostile every stream is created login into the ec2 instance click on ec2 instance so I will show you how you can get down connection code you can use this connection code to connect to ec2 instance so before you will connect to the ec2 instance the BAM files which we have copied right we have to change the mod of this BAM file so how we can do it okay so they did so now what we do go here then paste it so you are connected to the ec2 instance but now you can just do sudo or connected to the issued instance the next step is install the Kinesis agent in the ec2 instance so assume this is the issue to instance where your application is running and creating a log you have to install the agent in down ec2 instance this is the it of this document let me see how I can copy this one and go here yes if you see here I can after I have installed the kindnesses agent I can see there are several files which God created here log4j agent Jason and a Cindy if you see so this agent is a gentle Jason file has a configuration information that immediately so now next step is on I will create a folder where I will push the locks so I am going to create that folder in there yeah make directory rhythm so I am going to push away that information via the data which I have downloaded from Kegel calm and I have kept here in in the github repository also create down territory okay so one one problem that you create a directory in Dao ec2 instance and when you will try to post the data from your local it will have the access issue I will show you how you will see the access issue and how you can resolve it so this is time very important step check the folder can't create it or not so folder is created right but there is no file here okay so in this file that is a landing zone for locks so when I will post a file here in the bridal folder that configuration the JSON configuration will pick the file and push into the streaming service I have a template here and I will just provide the informations like in this template this is the default with you which comes with our installation I have added access key ID and secret as access key you can just go here and you will get this information over here right so I will copy this one and I will add into my template so this template also I have kept in down github repository okay then you have to copy just a quick key ID and you have to replace that also here okay let me verify it again so now this is done in the flows I have created a folder under the variable log and veto and I'm going to post a dot log file that also I will download from the github repository and the streaming name I have given a Kinesis demo which I have created here in dow AWS can assist mo write partisan keys at random and here i have a data extra code which is a data processing options so what I am doing is I am changing the CSV interlock is in the CSV format so option a my CSV to JSON format I'm changing it and this is the field which are there in my log file like recent country state City Monday year and average temperature how to visualize the data just go here in my github repository and you can just click on the CSV file you can see this is the data which we are going to load into dice the bucket and I have given the same header over here so be careful you should not miss any comma or annotation correct you have to close the bracket correctly I have to download the data already data is also available here if we see instance let me check where I am so now what happened oh do I have a lock let me see how what is the count here so 500 data is there I will later put a cord with 20 data and I will not first don't lock okay and I'll be great this one see the twenty record is created in the first dot log file so I will use that to upload into the ec2 instance so now I have created a I haven't sold an eighth-note agent I have created the file also I have updated the JSON format and next thing is I need to see if ec2 instances working fine or not you have to use a city - I file formation okay this is actually the folder name right so this is the source the local folder name okay so now I use a CP minus I and then I give the PEM file which will be used to connect to the ec2 instance and then I have given our folder information over here and now I will just give the the file which we which I want to upload right now I will use this to user enter it okay so I need to given the ec2 instance name then you have to give : then you have to keep the destination so destination is here is where blog and the folder which we have created right this is to check whether we can copy the data or not let me see I missed one e here you see so I'm just going quickly [Music] so you can say like it is saying permission denied we need to do so much of things that's very important I just wanted to show you folder in now ec2 instance how you can provide the permission so you can Doug you can copy the data from your local to the ec2 instance folder you have to go here in the issues instance right so when you will see here the filter it has a root access okay so first step is you have to create a group so now crew let me clear first crew and so I created a Vader group or okay so now what I will do I will use I will add the ec2 user in that group the next step is so how I can do it so don't shoot them all - a - chain and then give the user next episode you have to exit otherwise not reflect and then again you have to login back to though you shouldn't start and just go to the folder and ok so now you just click group sorry groups so you can see these groups where that is reflecting here right now the next step is you have to change the owner of that folder so you can see still the root is our owner so you just use sudo change honor to beta and so already YouTube correct now next step is Stewart Oh chmod ten to seven seven five so this is clean the directive directive permission and it sub directives to add group write permission and set the group ID on subdirectories credit in the future okay so now we will try and see if we can post up data from local to this folder so user should be us in our okay so you can see the data is getting loaded into the ec2 instance if you go to the ec2 instance yeah that's it's there well you can see that count also good so either don't move for now the cause I did not start up node agent I just installed the node agent but I did not configure and install so next step what I will do because I have kept this template with with some information in it in the local I will copy that template in the ec2 instance and then I will start the kind of seen agent service and then I will push the data okay template okay okay so I pushed invader folder but I recorder and what I can do I have a quickly call to the ETS folder it up yes it is in two previous canisters and I will delete the one which is there because I don't need it so you see I have deleted now what I will do I will go I will copy move command from fire block letter and then agent template correct et Cie database kinases and I will get the name agent dot Jason okay I need to use it so if you see here yes and when you will see the data so yes so whatever update we had done all the updates are getting reflected here right and this is the folder where we will keep the logs and the file form it will be dot log okay everything is looking good and this is the header in the JSON file Clea will start the service so yes that is the important thing we need to start the service also so how we can start the service sync those services it appears kindnesses agent and start good so service got started so how to validate whether services started all correctly you have to just go to the [Music] variable block and then get up this so in this folder one lakh will get generated when you will start the service just to 12 minus one yes so you can see the service is started so now this agent is listening to the weather folder so once I will put the file into the weather folder it will start reading it so let me push the file so I I have kept our 22 contained so I will choose first lock I will push here and I will quickly go here and check whether lock has this informational not good so you can see it is started reading the locks to our King and it is showing like in progress 28 across past and 0 record sent successfully to destination to let again door check for that its processed or not ok it is still walking 20 records and successfully to the destination right so how you can check you can just go quickly and open your streaming service click on money drink yes it came if you see this information this graph is showing up here yes you can see and also can check the delivery stream yes you can see that 20 records came to the delivery stream also from Connie system it moved to the firehouse and then finally if you check in the s3 bucket you will see the folders got created we had created our Kinesis demo so it will create based on here ten month then today's date and then when I have created so if you open the state out good so you can see actually all these 20 records are created and these 20 records are in the JSON format thank you for watching this video

Channel: Manish Singh

Views: 24,073

Keywords: aws kinesis, kinesis, stream, data stream, kinesis data stream, aws data stream, aws kinesis data stream, firehose, data stream lab, kinesis hands-on

Id: ZErwfH-OLk4

Length: 30min 26sec (1826 seconds)

Published: Mon Jun 29 2020