AZURE QUEUE STORAGE From ASP NET Core | Getting Started With ASP.NET Core Series

azure queue storage provides a simple queuing solution for handling large volumes of messages storage queues allows asynchronous communications between different applications this allows to decouple components build resilience and also scale for bursts of traffic in this video let's dive into azure storage queues and understand the different concepts around it we will learn this by doing it from an asp.net core application hello everyone my name is rahul and welcome back to this youtube channel if you're new here i make videos on dotnet cloud and devops so if this is of interest please don't forget to hit the subscribe button without much delay let's dive straight into the code and learn about azure storage queues let's create a brand new asp.net web api application using the.net cli and specifying new and web api which will use the web api template to create a dotnet application i have the application successfully created so let's open this in a ide i will use rider for this example so let's specify writer and specify dot to open the current project i have the project opened up in rider so here we have the program.cs the startup.cs and also a default controller class let's go into the weather forecast controller which has the default get method let's add a new post method that can be used to post weather data to our api now let's assume posting a weather data involves a lot of background work to be done before it can be added to our database for this example i would just be adding it to a queue and not actually save to a database since the intent of this video is to learn about azure queue storage so let's go ahead and add the new post method so let's first specify the http post attribute to create a post method so let's specify public task and specify post because this is going to post data to this api we can take in the weather forecast dto and specify data since this is going to come from the body let's specify the from body attribute now if you're not sure about how this data is getting bound in the api check out my model binding video which will be linked here or in the description below now once we receive the data at this end point let's assume we need to trigger off a series of tasks which are long-running processes in these cases it's ideal to push these messages into a queue and have them processed in the background so that the calling application or the ui does not have to wait long once this is pushed into a queue we can have a background process an azure function or a web job pick up this and then work on that particular message to push this to a queue we need to first create a storage queue let's first understand a few concepts around azure queue storage before we head off into the azure portal to create one the azure storage queue is a service for storing large number of messages you can also access the messages from anywhere using the https or http calls now there are certain restrictions on the message sizes like it should be 64 kilobyte in size an azure storage queue lives under a storage account within a storage account you can have multiple queues in here so the images to download and the images to resize are multiple queues inside these queues we can have multiple messages so each of these messages represent different workloads so this could be our weather data or different data that we need to process based on our application needs if you look at the url format the queues have the storage name with the queue.co.net followed by the queue name itself to identify which queue it is so in this example since we have the storage account as my account the url is myaccount.queue.co.windows.net followed by images to download to actually represent this particular queue here to represent the images to resize the last part would just be images to resize you can get these urls from the azure portal once we create a queue now an azure storage account is what we'll have to create first because that is what contains the queue the azure storage account also supports other types like blobs file shares tables etc so let's see this in action so let's adopt the azure portal to start creating an azure storage account and then an azure queue inside that once we have it we will use this from our application that we just created so let's go to the resource groups i have a resource group delete which i'll be deleting after this video so let's click create to create a storage account so let's search for storage account and select that click create to create a new storage account inside the portal here we can choose the subscription and the resource group that it has to be associated with so let's choose delete in this case and give this storage account a name now there are some restrictions on the characters you can use on the storage account name so you can see it must be 3 to 24 characters and can contain only lowercase letters and numbers so let's name this as youtube video demo let's choose a region so you can choose a region that's closer to you in this case i'll choose australia east choose a performance in this case i'll leave it as standard if you look at the documentation here you can learn about the standard and premium performance tiers let's also choose a redundancy level now there are different redundancy levels if you want to learn about this you can actually go to the learn more tag which will give you an idea on the different data redundancy concepts that is applicable with queues what you are selecting with redundancy is how the copies of the data is being stored in case one of the storage accounts goes down now redundancy ensures that your storage account meets its availability and durability targets in the phase of failures so basically you can choose between locally redundant storage which is lrs or zone redundant storage you can learn more about by reading this in this article here now if we go back we can see we have multiple choices for this redundancy you can also see it has a short description which each of these scenarios in this case i'll use locally redundant storage because this is just for demo purposes and it is also the lowest cost option whenever choosing the redundancy option choose between the cost and the trade-off that you are making so let's leave it at locally redundant and click review and create once this is looking all good let's scroll down and click create to create our storage account the resource is successfully created so let's go to this resource we have the storage account details and you can see the overview details like we just created so it's in australia east it has a subscription id that's associated and the different other properties that we had selected on the left side if we scroll down you can see a data storage option which gives you different options for this video what we are interested is in the queues section so let's navigate to the queues and start creating a new queue inside here to create a new queue let's click this plus button let's name this as weather data because this is going to be weather data coming into this queue let's also add in a ad so that it represents adding a weather data and click ok we have the queue successfully created so if we click we can navigate into the queue and look at the messages that's inside this particular queue since we have all of this set up let's navigate back to our source code and start using this queue from our asp.net application coming back to rider let's add a new get package to start using the azure queue storage here let's search for azure queues to get the new get package that we require we will be using the azure storage.qs nuget package to interact with the queue that we just created so let's click this and make sure to select add to this particular project the nuget package is successfully installed so let's create a queue client to talk to this queue that we have just created let's specify where q client is equal to new queue client now this is coming from the nuget package that we just added which is azure.storage.queues so let's select that and use the overloaded constructor to pass in the connection string and the queue name that this needs to connect to so let's specify connection string as a variable and also the queue name we will just define this in a moment for testing purposes let's define the connection string right here which i'll be moving later in this video so let's specify that and also the queue name so let's specify cue name and pass in the name here for the connection string let's navigate back to the azure portal let's navigate back to the storage account and if you scroll down you can see an access keys section where you can get the connection string so let's copy this particular connection string so let's say show keys and copy this value let's navigate back to rider and paste this connection string here now we will be moving this out from here later in this video for the queue name let's navigate back again and under the queues we can copy the queue name that we created which is add weather data so let's copy that and navigate back and paste this for the queue name now we have a connection to the queue client so let's add in a message to this client so let's use the queue client dot send message method we have a send message and also an asynchronous version of that let's use the asynchronous version and then specify the message text this needs to be a string value so in this case we'll have to convert the weather data object into a string representation so we can use json serializer to do that so let's specify where message is equal to and use the json serializer to convert this object into a serialized string so let's just say json serializer dot serialize and pass in the data now we have the message in the string format which is the json format and then just pass in this message to this method since this is an asynchronous method let's call a weight and make sure to add the sync to this post method as well now here we have created an instance of the queue client called serialize on the data and just used the send message async to send it to the queue let's see this in action let's put a breakpoint here and press f5 to run this application the application is running successfully so we have the swagger endpoint which shows the two endpoints on the api so let's choose the post and try it out and click execute to send this message to our endpoint so let's say execute and this hits the breakpoint as expected if we look at the data we can see this is populated with the data that we have just sent from the endpoint so let's click f5 this will continue the execution if you navigate back into the azure portal we should be seeing a message inside our queue so let's navigate to that and here we have a new message that's dropped inside this queue so if you look at the message text we can see this has the date the temperature and the summary that we had passed from the swagger ui you can also see this has got assigned an id and then the insertion time and the expiration time let's understand a bit more about these different attributes the insertion time is the time when this message was inserted into the queue the expiration time by default is 7 days so you can see in this case i have inserted on 30th of september and this message expires on the 7th of october now this is by default setting to 7 days and after which this message will disappear from this queue if nobody has consumed or worked on this particular message this also has a dq count which we will come to later now when we send a message we can change this default and make this a different value so let's go back to the code and see how we can do that the send message method has different overloads if we look at the next overload that we have this one takes a visibility timeout and also a time to live the time to live is the expiration time that we just saw let's test this so let's pass in a null for the visibility timeout which we will come back in a moment later and specify time span from seconds to specify the expiration time so let's pass in 10 which means this will expire in 10 seconds so let's run this app again and see how this is working now let's change the summary so let's say expires in 10 and click execute now this is successfully posted so let's go back to our queue and refresh this here you can now see that this was posted on 9 30 and this is expiring in 4 35 so once that time is passed this message will expire from this queue so if we refresh again that message is now gone it only stays in that queue for 10 seconds so depending on how long your message needs to be living in the queue you can set this value while you're sending a message the other time that we saw was the visibility timeout earlier when we send a message it immediately appeared into the queue but what if you want the message to appear after a certain delay let's say you don't want the message to come immediately but let's say after 10 seconds or maybe an hour this might be different use cases depending on your application so let's say in an order processing scenario if an order is processed you want to wait for an hour before it goes to the shipping department so you could put a message with a visibility timeout of 1 hour and it will show up in the queue only after 1 r there are different use cases for this time to live property so let's see how that works so navigating back to rider instead of passing null in here let's specify a value so let's click this and specify time span dot from seconds and specify 10 again now since we have given the visibility timeout to be 10 seconds let's make sure the time to live is more than that otherwise the message will never show up into the queue and this is also kind of an error condition so in this case let's specify this to be 20 seconds so the message will appear in the queue after 10 seconds and it will disappear after 20 seconds that's the total time that we are specifying here so let's run this to make sure this works as expected the application is running so let's update this text to be say visible intent and expires in 10 and click execute the message is successfully posted so let's go back to the queue and refresh and this message has not appeared yet this will appear after 10 seconds because that's the timeout that we have specified on it now you can see since the 10 seconds has passed we have the message inside here in this case this was created at 442.04 and it will expire at 440 to 24 because we had given 20 seconds as the expiry time once we refresh again that message would have disappeared so based on your application needs you can use the visibility timeout and also the time to live on how long a message needs to be in a queue and when it should appear if you set the expiration time to be a negative value then it will never expire so if we go back to the source code let's stop this let's pass a null so that the message is immediately visible and specify a negative value inside here so let's say -1 let's modify this message and makes this as never expires and click execute the data is successfully posted so let's navigate back to the queue and refresh this here we can see that this message was inserted at 444.16 and this is expiration time is set to the default value this means this message will never expire one way of exploring the messages in the queue is to use the azure portal you can also use the azure storage explorer which is a standalone tool that you can use on your computer once you have downloaded this you can open this from your computer so i have it already installed so let me open that and show you how you can see the same queue inside the storage explorer now there are different ways you can connect to the storage explorer so if you click the connect button on the left sidebar it gives you the different options in this case since i have a connection string i will use that to connect you can also add your azure account so that all the storage queues and accounts is visible on the sidebar as well so let's choose connection string for this example and select next in this case let's say the youtube demo as the display name and paste in the connection string so if i use the windows and v key i can use the clipboard manager and then select the connection string that i copied a while back so let's paste that in here and click next this gives you a summary so we can specify connect we have successfully connected to the youtube demo storage account so if you expand that and under the queues we can see the add weather data queue if you have more queues that'll all list in here so let's select the add weather data and you can see all the messages inside that here as well so just like using the azure portal you can start interacting with messages from the azure storage explorer you can double click the message to see the details of this message as well now that we have seen how to successfully add a message and also see the messages in the queue let's see how we can start consuming these messages from our application now there are different ways you can run these messages from your application in this video to keep things simple i will use a background task if you're new to background tasks in asp.net core i highly recommend checking out this video linked here or in the description below the other ways you can process these messages is using an azure function or maybe even a web job i will walk through that in different videos later so let's come back to our ide and add a new background job so let's open the solution explorer and create a new class for our background job let's add a new class and call this weather data service so this is the background job that we will be creating now to create a background service we need to inherit it from the background service class so let's do that this has a few missing members so let's make sure to implement them as well this adds an execute sync which is the method that gets called when this background service runs now in this particular case we need to keep pulling on this particular queue as long as our application is running so let's specify while and specify the stopping token dot is cancellation requested so we need to run this as long as this is false so let's make sure to add a negation at the start so as long as this cancellation is not requested which happens when the application shuts down this while loop will keep running since we need to connect to the queue we need the queue client here as well so for now let's simply go back there and copy this three lines so that we create a queue client once we have seen the background job i'll show you how to refactor this out as well so let's go back to the servers and make sure to paste this in here let's make sure to include the missing references and start using the queue client for this example let's simply log the messages from the queue and we'll not do any processing but based on your application you might have different processing needs on the data from a queue message to log let's first add a constructor and add in a i logger interface so let's specify i logger and pass in the weather data service as the category let's specify logger and make sure there is a backing property for this particular field here let's first write logger.log information reading from the queue to start reading the message let's specify our queue message which is going to be the message that's coming from the queue so let's specify the queue client dot read message sync now this has two different methods which is the read message and also the read messages as you expect the message returns one value and messages you can return multiple messages so you can pass in a number of messages that you want to retrieve in a single read for this example let's simply use the read message async method now this is going to read the first message inside this particular queue since this is an asynchronous method let's make sure to add the await call and since i've added the await we also need to make sure the execute async has an sync before it once we have the queue message we need to make sure it has a valid value so let's specify queue message dot value is not equal to null now once the messages has run out this value is going to be null in the next read so let's specify that and start processing this message for this example let's simply write this into the log statement but if you want to convert this into a real data you can also use that so let's say where weather data is equal to we can use the same json serializer dot d specify the weather data object so in this specific case weather forecast and pass in the string value so let's say q message dot value dot message text now this is going to be the value that we have written initially to the queue we know that this is in the json format hence we can deserialize it using the json serializer now if you're using a different format to write the message you can use the similar way to read it back as well let's simply log this message so let's say log log information and specify new message red and pass in the data that we have read so let's say weather data and pass weather data as well ideally inside here you would be doing your application process to process this message for this example let's skip that part now once a message is successfully read we want to wait for a few seconds before we make the next port so in this case we can say await task.delay and specify the time period to wait so let's keep this to 10 seconds in this particular example so let's specify time span dot from seconds and say 10 as the value so every time this background service is going to be run it's going to look at the queue client and then get the top message from the queue and log this out after that it waits for 10 seconds and then pulls this particular queue again to add this weather data service to our application let's go into the startup.cs under the configure services method let's call services dot add hosted servers and pass in the service name in this case this is weather data service so this is going to run this background service on application startup let's run this and see if this is working as expected let's put a breakpoint here and once the application has started up this breakpoint is successfully hit because our background service is loaded and then it calls on to the queue now let's step through this function and see if our queue message has any value in this particular case we get back one of the messages which has the message text as summary string this is the first message that we had dropped into the queue because of this case this is the first one to come out this follows the first in and first out principle so we can deserialize this message into weather data so if you look at weather data we have all the valid data for that particular object and we are successfully logging this now after 10 seconds this is going to read again now if we go back to the queue and refresh it we can see that the messages are not visible here anymore this is because we have read this but if we refresh this after a certain amount of time this will reappear again this is because in our application we had only read the message but we did not acknowledge that we have processed the message because of which after the read it waits for some time and then the message reappears in the queue so if we refresh again we can see the other message will also come back you can see the dq count here has now increased from 0 to b2 this is because this message has been read two times now how do we completely remove this message once we have processed this message so if we come back to our application once you have successfully done your application process we can mark the message as deleted so let's specify a weight let's call in the queue client and pass delete message async method this takes in two properties one is the message id and also a pop receipt that comes with the message so let's call the q message dot value dot message id this is the message id and we can also pass in the q message dot value dot pop received now this is also a property on the queue message so once this is deleted this message will be gone forever so let's run this to make sure this is working as expected let's also put a break point here the application is running so here we have the breakpoint so this is going to delete this particular message so if you look at the weather data we can see this is the message with the string summary so let's click f10 and this will be successfully deleted from the queue now if we go back to the azure storage queue and refresh this this message has gone from here this will not appear back again if we continue this loop let's press f5 this is going to read the next message in the queue after a delay of 10 seconds and delete that as well so this has come again with the next message so if we look at the weather data we can see this is the message with never expires now let's click execute and this is going to delete that message as well now if we come back to the queue and refresh this both the messages are gone now if you keep refreshing this this is not going to appear again anymore so anytime you read a message it's going to make the message disappear from the queue temporarily this is because we don't want any other processes to pick up the same message in these cases we expect the messages to be processed only once so in this particular case if you look at the receive message sync this has an overload which has a timeout as well now by default this is set to 30 seconds this means that when you read a message this will be hidden from the queue for 30 seconds after 30 seconds it will appear again you can specify this to be different values so if you specify 10 seconds you will have to complete your application process within 10 seconds and if you don't respond back the message will appear back in the queue for some other application to process to make sure the message has not appeared back let's go back and refresh and the messages are still gone if your application process has thrown an exception and has not called the delete message sync this message will be available for the next process to pick up again after the default visibility timeout on your receive message sync so in cases where your process can take more time you should use a different timeout when you are receiving messages a sync you also need to make sure in your handler of processing that you are not reprocessing the same message again this will avoid any kind of conflicts that you might have in case the message is getting delivered twice this will allow us to handle for any exceptional scenarios where you might have already processed but not reached back to the queue within the timeout period now that we understand how to write read and delete messages there is also one more method in the message which is peak so if you look at queue client there is a message called peak message and peak messages very similar to read in these cases these messages are not removed from the queue like the receive message we are just looking at the queue to see what the messages are typical use case for this is the azure storage explorer so when we are refreshing this all this is doing is a peek into the storage queue so that the message is still available for somebody else to process it just shows the current state of the message this is the same functionality that is used inside the azure portal when they are listing the messages in the ui here if they use a read then that message won't be available for our application to pick up which is why they use peak so if you are building such similar uis or dashboards for messages you should be using the peak message let's now see how we can remove the connection string and the queue name dependencies in this particular application now rather than hard coding this q client we can dependency inject this q client into our application so let's specify this using the constructor let's say q client and specify q client let's add in a backing property for this so let's say underscore q client now we can successfully remove this from here and add this inside our startup.cs so let's remove that and start using the underscore queue client within our application so let's say underscore queue client which is going to be injected into our constructor so let's save this go to a startup.cs and register the queue client inside here to add azure client libraries inside the dependency injection we can use the microsoft.extensions.azure nuget package this allows us to register different azure clients inside our application so let's see how we can do that so let's copy this nuget package navigate back to our application let's go to the solution explorer and say manage nuget packages let's search for this package and install that into our application so let's click the plus and install this inside this project the package is successfully installed so let's start using this so if you go back to the documentation you can see this uses the services.add azure clients and uses a builder to add the client so let's use something similar so let's say services dot add azure clients take in the builder and register the queue client so let's specify builder dot add client in this case we are going to specify the queue client so let's specify q client it also has the options of q client options which is the options that comes along with the queue client and let's register this client in here this provides multiple overloads that you can use since currently i'm not going to use any of those let's just ignore those parameters so let's specify underscore to discard those parameters inside this code and let's start a new delegate function inside here we need to make sure we return a new queue client so let's paste in the code that we just removed using the connection string inside the startup.cs now this is going to create a queue client so let's return this from here so this acts like a factory which is creating a new queue client using the connection string and the queue name and everything is defined inside the startup.cs class since we removed this from the weather data service let's also make sure we update the weather forecast controller so in the constructor let's add in the queue client and call this queue client let's adding a backing property so let's say underscore queue client and remove this explicit creation of the queue client from here and we can use the underscore queue client to send the message let's run this to see if this is working as expected the application is running so let's drop a new message so let's say never expires new and click execute now this has successfully posted the message so let's go to our queue let's refresh this the breakpoint is hit in the code so i think this is also read by the queue so let's look at this message and we have this new message that we have just created inside the queue so this summary has never expires new so as soon as we create it our background service picked up this message because it was right in time after this delay so let's run this which is going to delete that message from the queue again so as soon as we dropped we saw our background process picking that up and deleting it again after the process has happened now if you have multiple messages some of these messages are going to show up so let's remove the breakpoint and post a few messages from here so let's click execute couple of times and if we go to the queue storage and refresh this we can see those messages inside here now as and when our background servers is going to pick this up these messages are going to disappear from this one by one even when we move the queue client to the startup.cs we still have the connection string being used here now in an earlier video i showed five different ways you can manage connection strings in your application if you're using a connection string i would move this into the app settings and also save this as part of the secrets manager since this is a sensitive value but let's see how we can use managed identity so we can remove the need for this connection string all together if you're new to managed identity i highly recommend checking out the video linked here or in the description below managed identity is a feature from azure where you don't have to explicitly manage these credentials but azure will manage the credentials for you in the backend so all you need to do is specify that applications or users have the access to particular resources let's see how we can do that in this particular example so let's specify where credential and specify new default azure credential which comes in from azure.identity namespace so let's specify that based on where the application is running the default azure credential will use different ways to get the token now since i'm running on my local machine i have in rider in the tools and azure connection setup because of which it will use my id to get the token now if you're running this inside an azure web app it is going to use an azure web app managed identity to get this i show this in the other video on how to set it up and how to get it working so highly recommend checking that out now for this video since i have all of this set up on my local i will just show you how this is going to work so once we specify default azure credential we can remove the connection string and the queue name so let's remove both these lines and use a different overload from this queue client we'll choose the one which has the queue uri and a token credential so let's specify queue uri and the credential that we just created which is the default azure credential so let's specify credential and let's define the queue uri up here in this case this is going to be a new uri and we need the full uri to rq in this particular example so if you go to the azure portal and go back to the azure storage account and copy the url from this queues list so here we have the full url so let's copy this and use this from our application you could move this into an application configuration and this is not any sensitive information i highly recommend checking the video where i show about the five different ways you can manage these configuration values if you are running in an azure environment so once we have this set up this is going to use default azure credential to get a token to talk to this queue client now before we run this application we need to make sure we give access to this queue for my username since i am running this ide under my account i'll have to explicitly do that so let's navigate back to the azure portal let's go into the iam access control on the left side for the storage account and let's click add a role assignment here we can select different roles in this particular case if you scroll down you can see different roles for storage queues so here we have the storage queue data contributor or based on your application you can simply give a reader or a writer role so in this particular case let's choose storage queue data contributor which means i have read write and delete access to the queue so let's select that and search for my username so this is against my email so let's use my email id to find me up and click save now this is going to add a new role assignment which grants me the access to read delete and write messages into the queue the role assignment is successful so let's go to the role assignments click refresh and you can see the assignment down in this particular list if you come back to the application and run this this should all work as expected so let's run this application again the application is running so let's click execute and this has responded with a 200 success code let's click a couple of times this so that we have the message inside the queue let's navigate to the queue let's go to the queues section and navigate to our add weather data now if you refresh you can see these messages appearing again here so as and when the application is picking this up this will disappear from this queue so let's navigate back to the console and here we can see our background service is picking up this message every 10 seconds if we put a breakpoint here you can see this has now successfully picked up the next message so if you look at the console logs you can see the logging messages for each of these messages that's getting picked up so let's resume the execution and we can see the logs coming up here now since we have added the library the azure.core logs is also coming inside our logs if you want to mute these logs we can simply copy the azure.core which is the category go into the app settings if you go into the appsettings.json we can set the logging levels for that library so let's specify azure.code and say only log warning messages so this is going to stop this particular logs coming from here again so if you refresh if you clean this logs you can see from the next time this is going to run it's not going to show the azure dot core if i fix the space correctly so let's save this again which will auto reload this config and then let's clear this again now you can no longer see the azure.core logs coming here so if we go back to the ui and refresh this since all the messages have been picked up let's click execute a couple more times and come back to our console and we can see the logs are much cleaner now again now with managed identity in the startup.cs class we don't have any connection strings or sensitive information managed by this application we have completely removed the need of a connection string or handling them explicitly from our site i hope this helps you to understand more about azure queues and how to use them from an asp.net application when you have long-running processes or you want to decouple applications using a storage queue like this is highly recommended some of the typical examples where you could use storage queues are sending an email or doing long-running processes like these for your application or even for communicating between different applications these are heavily used also with micro services based architecture i will be covering more ways of consuming messages from a queue in following videos so if you like this video please make sure to hit the like button if you have any questions or feedback feel free to drop them below if you want to be notified of future such videos don't forget to hit the subscribe button it also helps me to grow this channel thank you and see you soon

Channel: Rahul Nath

Views: 628

Rating: 5 out of 5

Keywords: azure storage account, azure storage explorer, azure queue storage, azure queue c#, azure queue, queue storage in azure, asp.net core queue, how to use azure queue from asp.net, Azure.Storage.Queues, Microsoft.Extensions.Azure

Id: 5oTX6srQdOE

Length: 41min 16sec (2476 seconds)

Published: Fri Oct 01 2021