AZ-204 - Connect and Consume Services - Azure Event Hubs

[Music] hi and welcome back now in this chapter i want to go ahead and give an introduction on to azure event hubs so azir event hubs is a big data streaming platform service using the service you can actually go ahead and receive and process millions of events per second you can go ahead and stream data such as your log data your telemetry data or any sort of events onto azure event hubs now over here i've gone ahead and pasted a snapshot from the microsoft documentation so here is the reference url on the architecture of azure event hubs so on the left hand side you have your producers so these are different clients that can go ahead and send events onto azure event hubs the data in azure event hubs is then split across multiple partitions this helps to have better throughput on the data that is consumed in azure event hubs and then you have your event receivers these receivers can go ahead and read the data in as your event hubs so over here giving the different components that are basically part of the architecture so first you have the event producer so this is an entity that goes ahead and sends data onto the event hub the events can be published using either of these protocols then you have the partitions in the azure event hub so here your data is split across multiple partitions this helps for better throughput of your data that means more data can actually be sent onto azure event hubs then you have consumer groups so this is basically a view of the entire event hub you can have throughput so this is the throughput capacity for event hubs and then we have event receivers so and we will be going through subsequent chapters on understanding the different components of azure event hubs the entire purpose of this chapter was just to go ahead and give you an introduction onto azure event hubs now here we are in asia now let's go ahead and start working with azure event hubs for that we need to go ahead and create something known as an azure event hub namespace that namespace can then go ahead and hold multiple event hubs so let me go on to all resources over here let me go on to all resources let me go ahead and click on add and over here let me go ahead and search for event hub so i'll go ahead and choose event hubs over here i'll go ahead and hit on create over here i'll go ahead and choose a resource group now over here i'm going ahead and giving a name for the namespace that will go ahead and hold our event hubs then i'll go ahead and choose my region now over here in the pricing tier if i go ahead and view the full pricing details so we have the basic and we have the standard so let me go ahead and choose the standard when it comes to azure event hubs now over here you can go ahead and decide on the number of throughput units for the moment i'll just leave it as it is i won't add any additional features over here i'll go on tags i'll go on to review and create and let's go ahead and hit on create now this might take around two to three minutes let's come back once we have the namespace in place now once you have the namespace in place i'll go ahead on to the resource next we can go ahead and now add an event hub onto our namespace so over here let me go ahead and click on add event hub over here i'll go ahead and give a name for the hub now over here you can go ahead and decide on the partition count so it can so it can be from 1 to 32 partitions and then you can also go ahead and decide on the message retention so when an event or a message is actually sent onto the event hub it will be retained over here by default for one day over here the maxim retention period is seven days by default you are not supposed to go ahead and persist messages in the event hub so it is used for ingesting data and then you'll have consumers that will actually go ahead and basically read the messages in the event hub and process them accordingly over here you also have the capture option as well so this allows you to go ahead and persist those events onto let's say a storage account so if you want to go out and have the events in a data store for a longer period of time you can go ahead and decide to enable this capture feature for now i'll mark this as off let me go ahead and leave the retention as it is and let me go ahead and create the event hub now once we have the event hub in place so if you go ahead and scroll down over here you can go on to your event hubs you can go on to the event hub itself and now you have your hub in place now currently at this point in time so there is nothing like a way to go ahead and send events or messages onto the event hub from the azir portal so over here we don't have any sort of explorer to go ahead and look at the messages in the event hub or even send messages onto the event hub so in the subsequent chapters we are going to go ahead and look at dot net programs that will be used for sending messages and receiving messages from the event hub so in the previous chapter we had gone ahead and seen how to create an event hub namespace and how to create an event hub now let's go on to visual studio and let's go ahead and create a simple console based application that will actually go ahead and send messages onto the event hub so i'll go ahead and create a console based application i'll go on to next over here let me go ahead and give a name let me go on to next i'll go ahead and create the application now once we have the application in place so the first thing that we need to do is to ensure that we have the right packages in place so i'll go on tools you get package manager manage nuget packages for the solution i'll go on to browse over here let me go ahead and search for messaging dot event hub so i'm going to go ahead and choose azure dot messaging dot event hubs let me go ahead and hit on install i'll go ahead and hit on ok and hit on accept now with event hubs i'm first going to go ahead and define a property that's a connection string that actually allows our program to actually go ahead and interact with our azure event hub so i'll go ahead and define a connection string now where do i get this connection string from now when you go on to your event hub so there are two things one is your event hub itself and the other is the namespace so if you go back on to your namespace your event hub namespace over here you have shared access policies so this allows you to go ahead and define the connection strings or the policies that allow you to interact with event hubs now when you go ahead and define a policy at the name space level it will go ahead and apply that same policy on to all event hubs in that name space but for security reasons if you only want to go ahead and create a policy at the event hub level that is something that you can also do so i'll go on to event hubs i'll go on to my event hub over here i'll go on to shared access policy over here let me go ahead and click on add now the default permissions you have are manage send and listen now over here since i only want to go ahead and send messages onto the event hub i'll go ahead and choose the send permission over here i'll go ahead and give a simple policy name and i'll hit on create i'll go ahead and click on the policy i'll wait for the connection string to pop up over here so i can go ahead and take either the primary or the secondary connection string i'll go ahead and take the primary connection string i'll go on to my program and i'll replace it over here so now we have our connection string in place now i'm going to go ahead and send some events or some messages on to the event hub so this is going to be based on information about different orders so for that i'm firstly going to go ahead and create an order class so over here i'll right click on my program i'll go ahead and add a new class so i'll term this as the order class then i'm going ahead and adding some properties so the order id the quantity the unit price and the discount category i'm also going ahead and overriding the two string method which comes by default for the classes and over here i want to go ahead and basically return a json string of all of my properties over here let me go ahead and ensure that i include system.txt.json now i'll go back on to my main program so over here i'm going to go ahead and first create a list of orders so i want to go ahead and send this as messages on to my event hub let me go ahead and ensure that i use system.collections.generic now it's time to go ahead and send these messages onto our event hub so the first class i need to use is the eventhub producer client this is used for actually sending messages onto the event hub so i need to go ahead and use the namespace of azure.messaging.eventshops.produce shops dot over here let me go ahead and give a name for the object and i'll say new event hop producer and over here i need to go ahead and give my connection string right so i have that in place now next there is a class known as event data batch that allows us to go ahead and actually send a batch of events onto the event hub so over here i need to go ahead and actually use my producer client dot create batch async and over here dot get awata dot get result next now for each of my orders in my orders list i need to go ahead and first construct a message so this will be the message that is sent on to the event hub so there is a class known as event data so over here let me go ahead and ensure that i include it so using azure messaging.event hubs so that class event data can be used to construct that object which basically holds our message now over here we need to go ahead and give the event body which will be in binary format so i need to go ahead and use now the encoding class dot utf-8 let me ensure that i include the namespace dot get bytes and let me go ahead and take each order and two strings so remember i had gone ahead and over in the string method so now all of the information for each order will be returned as a json string and the json string will now be converted into bytes and that is now the body or the payload of my data that is being sent on to the event hub now over here itself let me go ahead and basically use my batch object which i have and there is now a method on a triad to go ahead and add now each object of that event data right so i have that in place and then finally using my producer client so there is a send a sink and i can go ahead and send my batch so get waiter dot get result let me go ahead and just write a statement right and with that we have a program to go ahead and send events on to the event hub right so with that we have our program which is now being used to go ahead and send events onto our event hub so let me go ahead and send our events right so we've got our batch of events that are sent onto the event hub so let's mark an in on to this chapter and we'll move on to the next chapter to see now how do we receive events from the event hub now in the last chapter we had gone ahead and developed a dotnet program so that is now behaving has a producer that dotnet program had now gone ahead and sent a batch of events or messages basically on to our event hub which is part of our event hub name space now remember with our event hub when we created the event hub we mentioned the partition has just one so we just had one partition in place that means all of our events are now sent on to that one partition so keeping this in mind right so if i go back on to my event hub if you go on to the overview for app hub over here you can see the partition count has one now let's go ahead and develop a program that will go ahead and be used to receive messages from our event hub so again for that you need to have a consumer in place you need to go ahead and develop a program that can go ahead and consume the messages please note that there are services in asia such as the stream antics job that can also go ahead and consume events without the need of actually any sort of programming so the stream addicts job can also go ahead and take events from the event hub but since we need to go ahead and know how to develop programs that can interact with azure event hubs let's go ahead so in visual studio let me go ahead and create a new project so again i'll choose console application i'll go on to next i'll go ahead and give a name go out next and let me go ahead and hit on create now as usual the first thing we need to do is to ensure that we have the right packages in place so i'll go on to tools you get package manager match nuget packages for the solution i'll go on to browse and again let me go ahead on to messaging dot event hub now over here when it comes to receiving messages there is a package in place that is known as azure.messaging.eventhubs.processor and this package actually makes receiving messages a little bit more easier but for this example the initial example i'm going to go ahead and show you how to use the normal packages to also go ahead and receive messages so you can still go ahead and use azure.messaging.events to go ahead and receive messages as well it kind of gives you a better understanding on how do you actually receive messages from as your event hops so let me go ahead and choose that package and let me go ahead and hit on install i'll go ahead and hit on ok and hit on accept so i'll go on to my program so as normal the first thing we need to do is to go ahead and define our connection string that allows our receiver to go ahead and connect onto the event hub so i'll go ahead and define my connection string so i'll go on to my event hub i'll go on to shared access policies i'll go ahead and now create a new policy this will have the permission of listen so that we can go ahead and listen or take events or read events from the event hub let me go ahead and hit on create so i'll go ahead on to the policy i'll take the connection string i'll go back onto my program and i'll replace it over here now when it comes to receiving events there is something that we need to specify has a receiver and that is something known as the consumer group so over here going back on to our diagram our receiver needs to receive the events via a consumer group now by default there is actually a default consumer group that is created as part of your event hub so the main benefit of using different consumer groups is you might have different types of applications that might want to go ahead and read events from the azure event hub so let's say your security department wants to go ahead and read messages from the event hub from a security perspective let's say another department wants to go ahead and read the messages from the event up for let's say another perspective so each of them wants to have a different view of the event hub so you can actually go ahead and create multiple consumer groups for these different departments so when you go ahead and basically use a receiver you have to go and mention the consumer group so if you go on to your event hub if you go on to consumer groups you can go ahead and create a new consumer group over here you can see you have the default consumer group in place so i'll go on to my program and let me go ahead and paste in code a property basically for our consumer group now i'm going to go ahead and use a class known as event hub consumer client so this will allow us to go ahead and consume messages from the event hub so i'll go ahead and use the namespace is equal to new event hub consumer client and over here we have to go ahead and give our consumer group and our connection string so let me go ahead and give my consumer group and give my connection string right i have that in place now we can actually go ahead and use the consumer client to go ahead and read events from our event hub so i'm going to go ahead and use the method of read events async now i want to go ahead and read all of the events from my one partition so what i'll do is that i'm going to go ahead and put it in a for each loop so i'll say for each partition event so there is another class for each event each partition event so this is part of the event hub namespace which is just imported our new get package so for each partition event in our consumer client dot read events async let me also ensure to put the await clause over here we can now go ahead and start reading our data now for my main program i need to go ahead and mark it as a sync and task as well i have to include the namespace and now for each partition event if you want to go ahead and look at different properties so if you want to get basically the data you can actually go ahead and get the event body you can get the time that the message was actually inserted into the event hub you can get something known has the offset the partition key the sequence number etc so let me go ahead and paste in code to get some of the properties of our event so over here i am getting basically the partition id the data offset the sequence number and the partition key over here let me go ahead and ensure to use system.txt remember our event body is in bytes so when we send our event onto the event hub it is sent has bytes and over here we can get the string equivalent back of our event body so let's go ahead and run our program right so over here we're getting all of our data in place so note that all of our events are basically part of partition id 0. so if you go ahead and look at the partition id all of them are zero the messages have a sequence number if you look at the offset it's a data offset so basically it's the offset in terms of bytes so the first message that we had had an data offset of zero so starting from zero and since this message took basically 120 bytes or the entire event to 120 bytes the next data offset for the next message is 120 and so on and so forth so when you look at your partition when you look at each message right so this first message basically starts at data offset zero this is in terms of bytes then the next message is 120 the next message is 240 etc and each message also has a sequence number and over here you have the data in place please note these are all important concepts so there's a difference between the different event messaging systems that are there in seo so you have the azure service pass queue the azure service bus topic and over here the azure event hub so the azure event hub is tracking these messages via this data offset and wire these partitions and over here your data is in the binary format now in our last chapter we had gone ahead and seen how to receive messages from the event hub so we had gone ahead and run our program and we read the messages from the event hub now if i go and just run this program again so i can see i'm again getting the list of messages now the reason i'm running the program again is i want y'all to understand the distinction between azir event hubs and as your service bus if you look at the concept of the azios service bus cues over there you normally have producers that will go ahead and send messages on to the queue and then normally you will have consumers that will actually go ahead and receive a message from the queue and then delete the message once the message has been processed but in azure event hubs the philosophy is different over here the message will not be deleted the entire purpose of azure event hubs is to allow multiple consumers to go ahead and read the data at any point in time and then go ahead and process the data accordingly but the data will still be there in azure event hubs remember for us your event hub there's something known as the message retention when we go ahead and create the event hub so if i go on to the overview for my event hub so over here the message retention is currently one day so it's only after one day that this messages will be deleted there is no way to go ahead and delete messages from the azure event hub you have to rely upon this message retention process and i said this is to allow many consumers to go ahead and receive messages from the event hub so this is one important concept i want to bring to your notice the next important concept is over here now my program is actually going ahead and reading the messages from the beginning of the event hub but there are also ways to go ahead and read messages from where you left off so if you want to go ahead and have the program read messages after the last message over here you have to go ahead and ensure to read messages after this data offset so over here there's something that needs to be tracked by a program but there is another package that is available that helps to simplify this tracking process the next point i want to bring to notice is the throughput so the throughput is something that is defined at the name space level so over here based on the amount of throughput that you have at the name space you get the amount of throughput that is available for all of your event hubs in that event hub name space so when you look at one throughput measure when it comes to the amount of incoming traffic that can be accepted by the event hubs over here the limit is 1 mb per second or 1000 events per second remember this is used to process millions of events per second so if you want more throughput you have to go ahead and provision more throughput units and you will be built based on the throughput units and when it comes to the egress traffic when it comes to receiving messages it's up to 2 mb per second for one throughput unit or 4096 events per second if you actually go on to your namespace so if i go on to the namespace itself so if i go on to the overview so over here we have the number of throughput units over here you can go ahead and scale the amount of throughput units accordingly so you can go ahead and change the throughput units for namespace you can go ahead and change the message retention for your event hub the thing that you can't change is basically the number of partitions so if you go on to your event hubs if i go on to my event hub itself so over here the partition count is one and the something that you can't change right just want to bring some important points do you notice before we proceed ahead now in this chapter let's see how to use azir functions to go ahead and receive messages from azure event hub so in azure i have an event hub instance in place so currently there are messages in this event hub so in visual studio let me go ahead and create a new project i'll go ahead and choose azure functions and go on to next over here let me go ahead and give the name of azir functions as your event hub i'll go ahead and hit on create and over here let me go ahead and choose now the event hub trigger so i'll go ahead and choose event hub now over here we have to go ahead and give the name of our event hub so for me the event hub name is app hub and over here we have to go ahead and give a name for the event hub connection string i'll go ahead and hit on create right so over here we have a lot of code in place when it comes to receiving messages from the event hub so over here let me go ahead and first rename my class now let me go ahead and just give a function name now i need to go out and ensure that i put the connection string so let me go ahead and take the connection string from here i'll go on to my local settings.json i'll place it over here now i'll go on to my hub i'll go on to shared acts policy let me go ahead and add a new policy i'll go ahead and give it a name and hit on create i'll go on to the function let me go ahead and take the connection string i'll copied it on to visual studio right so that is in place now in this code in the program code so there's already a lot in place when you can actually go ahead and look at the messages from event hub so over here if you look at the input parameter so the events are sent has an array from the event hub onto a zero function over here it's actually going ahead and getting what is the body of the event data so let me go ahead and just run this as it is because we already have a lot of code in place when it comes to the template and over here you can see it has gone ahead and processed all of the messages in my event hub so over here i just want to go ahead and show you how you can get started when it comes to using azure functions for your azure event hub now in this chapter i want to go ahead and run our dotnet programs against an event hub that is based on multiple partitions so early on for our app hub we only had one partition in place so let me go on to our event hub namespace and over here let me go ahead and now add an event hub so over here i'll just go ahead and give a name for the event hub and over here let me go ahead and just mention it has two partitions and then let me go ahead and hit on create now once i have the event hub in place i'll go on to it so i'll go on to our earlier program which was used to go and send the order information the messages on to our event hub over here i need to go ahead and now replace the connection string so for this new event hub i'll go on to share access policy i'll go ahead and click on add use the permission of send the policy name hit on create i'll go on to my policy let me go ahead and take the connection string i'll go on to my program i'll place it over here and let me go ahead and basically add more orders so what i'll do is that let me go ahead and just add a for loop just want to add multiple orders that's it so over here i'll go ahead and encapsulate everything in the follow over here let me go ahead and remove this from the follow-up over here since i'm just going out and initializing the producer client in the beginning and basically 20 times i want to go ahead and add the list of orders so let me go ahead and run this program right to attend all of the events now i'll go on to our receiver program so let me go ahead and create now a new access policy i'll give it a name i'll mention the permission i'll hit on create i'll go on to the policy i'll take the connection string i'll go on to our receiver program i'll replace it over here and let me go ahead and run the program right so it's going it and reading all of the events that have been sent on to our event hub now over here you can see that messages are part of partition id 1. you can see the data offset as well you can see the sequence number let me scroll on top so if i go ahead and scroll up on top i can see that events are also part of partition id 0. so now over here the azure event hub has gone ahead and basically distributed our events across different partitions so over here it's not trying to go ahead and basically have the events all part of one partition but it's going in and displaying the events across multiple partitions here in our receiver we are going ahead and receiving all of the events in every partition so that's what this particular loop will actually do so no matter which partition the event is it will go ahead and read that particular event so another important note when it comes to the distribution of data so when you have multiple partitions your data is dispute across the different partitions now in the earlier chapter i had shown you an example wherein we had an azure event hub which had multiple partitions and then our azure event hub would go ahead and distribute the messages across the two partitions now you can have your receivers that can actually go ahead and read all the messages from the partition but if you want better throughput if you want to go ahead and process a larger number of messages then even the recommendation for microsoft is to go ahead and have one receiver per partition so let's say that the i.t security department wants to go ahead and process these messages so let's say that you have applications that are sending their log data so in our example i said i'm sending some simple order information based on the classes and the objects that i have but let's say that applications are sending in their log data has messages on to the partitions in azure event hub then the it secure department wants to go ahead and process these messages to look for any sort of anomalies any sort of problems any sort of issues then instead of just having one receiver going ahead reading the messages and processing them they can go ahead and have multiple receivers so the same application but the application is going ahead and reading the messages from each partition individually now in our program we are reading every message in every partition you can also go ahead and ensure that the program only reads from a certain partition and i'll show that but in our case currently our program is reading all the messages from all of the partitions so before i could actually go ahead and show you the concept where you can go ahead and read from a particular partition i want to kind of explain to you that the messages are now distributed across multiple partitions and yes our program can go ahead and read all the messages from all of the partitions so the recommendation is to go ahead and have one receiver per partition and remember your consumer group so i said if this is the i.t secure department they can go ahead and read the messages from these partitions as part of their consumer group now another aspect currently is you can have up to five concurrent readers per partition per consumer group so you could also go ahead and have two applications reading from the same partition because over here you are not bothered about you know the message sequencing has such you know the order the messages you are more bothered about processing the event itself processing the message itself that's the entire basis of using the azure event hub if you have the requirement of sequencing of the messages then you should look towards using the azure service pass but when you have multiple concurrent readers make sure that they do not read the same messages so they should not go ahead and duplicate the process of reading the same messages so having these concepts in place is very important so so we're going step by step and trying to understand the concept of these partitions the messages you know having receivers per partition right so these are important concepts to understand you have your partitions you have your messages you have your readers and you have your consumer groups now in this chapter i want to show you how you can use now a dot net program to go ahead and just read messages from a particular partition so to make this very simple we're going to go ahead and read the messages from just the first partition that's partition number zero so over here i'm going to go ahead and first add a statement where i'm going it and getting the partition ide so for my consumer client i want to go ahead and basically get the first partition so over here i can go ahead and get the partition ids let me go ahead and ensure that i include the namespace of system.link next when i need to go ahead and now read the events i'll go ahead and choose now another method which is read events from partitions and over here i need to go ahead and give the partition id and what is the starting position so over here let me go ahead and give the partition id which i've got over here so my partition id and over here what is the starting position so over here we can go ahead and use event position dot so we could start from the earliest we could start from a particular offset we could go ahead and start from a particular sequence number so for now i'll go ahead and start from the earliest so that means in our program we should be reading all of the messages in our first partition right so we have everything in place let me go ahead and run this program so it's going in and reading the messages but over here you will see that it's only going at and reading the messages from partition number zero over here you can see the sequence number so we're not getting all of the sequence numbers we're only getting the messages from the first partition now in this chapter i want to go ahead and show you how you can use the event hub processor class or the packages in order to go ahead and receive data from your event hubs so over here in my event hub namespace i do have a set of event hops in place over here i can go ahead and choose either hub so i'll go ahead and choose app hub over here i have a set of messages in place now i'll go on to visual studio i'll go ahead and create a new project over here let me go ahead and choose a console based application i'll go on to next and over here let me go ahead and give a name i'll go on to next and let me go ahead and hit on create now once this is in place let me go on tools new get package manager manage nuget packages for the solution i'll go on to browse again let me go ahead and search for messaging.event hub and over here let me go ahead and now choose azure.messaging.eventhubs.processor let me go ahead and hit uninstall over here you can see that it also go ahead and install the required packages for azure.messaging.event hubs as well so the entire idea of the event hub processing is to go ahead and keep track of the messages that have already been read so if your program stops in between and then restarts it will go ahead and start reading messages from the time it left off so that's an advantage where you don't need to go ahead and read all of the messages again now there is already a default implementation in place when it comes to using the event processor you know class now in order to go ahead and implement this event processing solution we need to have a storage account in place so the program needs to go ahead and save that information about you know the tracking information from where it left off in terms of processing of messages has checkpoints in a container in a storage account so in azure let me go ahead and open up storage accounts in a new tab i want to see if i do have any storage accounts in place if not we can go ahead and create a new one so i don't have any in place so let me go ahead and create quickly a new storage account so i'll go ahead and choose my subscription i'll choose a resource group i'll choose my location i'll go ahead and just give a name for the storage account over here let me go ahead and choose locally redundant storage and basically let me go ahead and leave all the settings has there and let me go ahead and hit on create now while this is being done let me go ahead and add some code on to our program so over here i'm first of all adding some properties so first i have the connection string onto our event hub what is the consumer group what is the storage account connection so this is for the azure storage account and what will be the container name that will go ahead and save that checkpoint information so firstly for my app hub let me go ahead on to the shared access policy i'll go on to listen i'll take the connection string i'll go ahead and place it over here my consumer group is fine now next in my main class let me go and paste in some code so over here we need also a blob container client so for that i need to go ahead and ensure i have the right packages in place so i'll go on tools new get package manager manage nuget packages over here let me go ahead and search for blob so i'll go ahead and choose azure.storage.blobs go ahead and install the required package let me go back onto the program ensure that i include the namespace now over here we are going in and using now that event processor client class so over here now the client class will go ahead and put the tracking information in the container client the container client basically points on to our container in the storage account and then when events are detected it will go ahead and send the events onto an event handler so we have to go ahead and define that event handler we can also go ahead and define an error handler as well so if an error is detected then go ahead and send it on to the error handler over here let me go ahead and show that i have it has an async method and then i need to go ahead and add my methods over here so i'll go ahead and paste in some code this will be after my main method so over here one is basically for processing my events and the other is for processing my errors over here i'm just going in and writing this sequence number and basically what is the event body let me go ahead and ensure that i use system.txt over here i need to go ahead and use the azure.messaging.eventhubs.processor and over here you can see it is updating that checkpoint information in that storage container now let me go on to these storage accounts assuming it's done right i'll go ahead on to the resource i'll go on to access keys i'll show the keys let me go ahead and take the connection string go on to my program and let me replace it over here i have to go ahead and define a container so in my storage account i'll go on to containers i'll go ahead and create now an event hub container i'll go ahead and hit on create now once this is in place let me go on to my program and let me go ahead and run my program right so i'm getting all of the information currently i just have five messages in the event hub now let me go ahead and stop the program and run the program again now nothing is showing up in the screen because remember now the message has remembered it has already gone ahead and read the messages and there are no new messages currently now in the event hub so it has gone ahead and remember that tracking information now if i go ahead and open up our previous program which was used for sending messages on to that event hub let me go ahead and run this program so it goes out and send the messages right so it's sent and now if i go back on to my processing client you can see it has gone ahead and read the messages so my client now remembers the tracking of the messages over here so you can see how the event processor you know packages which is available helps for the easy tracking of the messages when they are being read from the event hub now in this chapter i want to show you how you can capture events from your event hub onto a stored account so this is a feature that is available in event hubs you have to make sure that it is using the standard pricing tier when it comes to azure event hubs which we are anyway doing so i'll go on to one of my event hubs over here if i go ahead on to the capture feature over here i can go and turn this on so let me go ahead and hide this so if you want to go ahead and persist now your events on to a data store like azure storage accounts you can go ahead and make use of this feature so now over here in a duration of every five minutes or when the size window is met of 300 mb it will go ahead and start sending messages onto a storage account over here you can go ahead and choose the storage account now i already have a storage account in place so let me go ahead and hit on select container i'll go ahead and choose my storage account now let me go ahead and create a new container over here so i'll go ahead and hit on create i'll go ahead and choose that container hit on select over here you can see what will be the format of the name of the capture file right so let me go ahead and save the changes now once the changes are saved so i'll wait for around 10 minutes now if you have dotnet programs in place which you have seen earlier on to go ahead and send messages onto the event up please do that go ahead and start sending data on to the event hub let's come back after span of around 5 to 10 minutes now i've come back after duration of around 10 minutes now if i go on to my storage account if i go on to my containers if i go on to the capture container so over here i can see the folder for my namespace my event hub and over here you can see the file format that was specified so if i go and drill down now all of this information is actually stored in a separate file format known as the avro file format so this file format is much more efficient when it comes to data storage and retrieval if i go on to the edit section so since the format is different it will not be recognizable over here but you do have tools that have the ability to go ahead and read files in this format over here i just want to go ahead and show you that capture feature which is available for azure event hubs now in this chapter i just want to go ahead and give you an example on how you can stream log data from azure resources onto azure event hubs so most of the services in asia have this feature of sending their diagnostic data or logs onto azure event hubs so over here has an example i have an azure sql database in place so normal as your sql database we've already gone ahead and seen how to create an azure sql database if i go ahead and scroll down if i go on to diagnostic settings so i said this diagnostic settings is normally available in a lot of azure services over here i can go ahead and add a diagnostic setting so over here you can go ahead and stream different log data and you can also go ahead and stream metrics about the underlying database all of this can be streamed onto either a log and x workspace a storage account or onto azure event hub so if you want now applications to go ahead and process this diagnostic data from the azure sql database from azure event hubs you can go ahead and do that if you want to go ahead and process the data a little bit later on you just want to go ahead and store that diagnostic information you can probably go ahead and store it either onto a storage account or onto a log antics workspace but in our case we are going to go ahead and stream it onto an event hub now over here i already have an event hub in place so i have the hub name space and let me go ahead and choose any event hub over here i'll go ahead and choose the hub policy name that is linked onto that event hub namespace so this is required over here let me go ahead and send the basic metric data let me go ahead and scroll on top and let me go ahead and just give a simple name and let me go ahead and hit on save right so once this is done now your information about your database the metrics information will now be streamed on to your azure event hub now i'm going to go ahead and wait for around 10-15 minutes because it does take time for these metrics to be relayed also if you are following along if you already have the azure sql database in place please make sure to go ahead and log into the database perform some sort of activities so that the metric data gets recorded onto azure event hubs now i waited for around 20 minutes now i'll go ahead and basically run a program that i have in place so this is a program that we've already seen to go ahead and receive events from our event hub please note i'm not using the processor host over here i'm using a normal program to go ahead and read all of the events from all of the partitions so let me go ahead and run this program now over here you can see i'm getting a lot of information so now this information is basically the metric log data that is being sent about the azure sql database over here you can see the database name so for example it's sending metrics about the cpu percentage right and it's adding other metrics as well so now all of these metrics are being streamed onto azure event basically in json format and it's up to your basically a program to go ahead and process the data accordingly in this chapter i just want to go ahead and explain to the concept that you know this uh diagnostic setting is available for a lot of as your resources to go and stream their data onto an event hub and that's the entire purpose of the event hub it is a service that is used to go ahead and ingest a lot of data hi and welcome back now in this section we have seen a lot of services messaging services that are available in azure so we've looked at azure event hubs we've looked at azure event grid and we've looked at the azure service bus in the service parts we have the queues and we have the topics so i thought let's go ahead and have you know some points when it comes to comparison of these services now if you are looking at an event-based service so let's say an event is triggered a lightweight notification when a condition or state change happens then you should go ahead and look towards using the azure event grid service remember you can actually go ahead and hook up an endpoint onto the azure event grid such as azir functions and that function can go out and perform some sort of processing on that event so if you want to go ahead and work with events go ahead and use the azure event grid service if you have devices that are sending telemetry data so let's say you have a device that is sending let's say temperatures at different points in time so this telemetry data then you can actually go ahead and look towards using azure event hubs so this is used for telemetry or distributed data streaming if you want a messaging system that needs to be shared across different components of an application then you should go ahead and consider using azure service bus a quick comparison when it comes to the message size so over here when it comes to the azure event grid it's 1 mb which is the size of an event if you're looking at azure event hubs when it comes to the basic tier it's 256 kilobytes and 1 mb for the standard tier when it comes to the azure service bus it's 1 mb if you go ahead and choose premium messaging so again remember that if you want to go ahead and send telemetry data go ahead and choose azure event hubs if you want to go ahead and look at changes in events then go ahead and use azure event grid and if you want to go ahead and have decoupling of your application components then go ahead and use the azure service bus so i just thought that since there are so many services in azure when it comes to messaging i want other clear or tell students you know what is the clear demarcation on when to use which service

Channel: Alan Rodrigues

Views: 1,023

Keywords: AZ-204, Azure Event Hubs

Id: HwZldR8KlKM

Length: 62min 20sec (3740 seconds)

Published: Wed Oct 13 2021