App 2025 Episode 4: Simplifying your architecture with Amazon SNS and Amazon SQS

all right hey everybody welcome to this week's episode of app 2025 thank you for joining if it's your first time here I hope that you'll find a lot to take away we are building the app of the future today and today's episode we're gonna talk about Amazon SNS and Amazon SQS and how you can simplify your architecture and at the same time make it a lot more resilient using these two services if you've seen the previous episodes you know I get excited about the services that we talk about because I just I genuinely like to build with them they give you so much power and flexibility to let you think about features and these two in particular are two of my favorites these are old-school service services back five and ten years before that was even a term they're incredibly robust they're sort of foundational in the theory of distributed systems so if you've been building any sort of distributed application the concept should be familiar and there's their you know hugely relevant today they integrate very well with Amazon of Enbridge and we're gonna get into some of that today we've got a nice big packed agenda today there's not gonna be as much building because the pattern itself is pretty straightforward and reusable which i think is a really good thing but we will build I will give you the code for this pattern and we'll look at how all these pieces interact together so I'm going to talk a little bit about Amazon SNS or simple notification service and Amazon sqs simple queue service about their history and what they do I'm gonna talk about how to use them together with Amazon event bridge which if you'll remember is the backbone of our app 20-25 app will go into some specifics on SNS topics and subscribers or subscriber endpoints the different types that you have and what you can do with them we'll talk about two different types of cues the sqs standard and FIFO cues as well as in this pattern whether you should introduce your own cue or not to queue or not to cue I say there that is the question and you'll understand more when we get and then finally we'll build this buffered fan-out pattern that I want to talk to you all about today so that's the meat of it I'm gonna leave us on this screen for a little while instead of going into build because I think if we go into build we'll get a little confused on where we are again as always check the links come in that I just dropped in the chat is gonna give you links to Doc's for both of these services and from the docs you can find pricing quotas all that good stuff you'll also note that first link is a webinar on events queues topics and streams that was just done by my colleague Julian wood outstanding webinar up on YouTube if you want to get a little deeper into this from a to include streams as well and sort of understand outside of the concept of app 2025 so I'd really encourage you to check that out and then I dropped a little link there on idempotency for you we're gonna spend a little bit of time talking about that today should be a familiar concept if you've been building distributed systems but it's still something important to consider so first I want to talk a little bit about SNS let's I lied we're gonna switch it up we're gonna go down here and let's take a look at SNS link blow this up for you a little here it's a fully managed publish/subscribe messaging service so what that means is much like event bridge that we've talked about already you just publish messages on to these topics SNS has this come a concept of topics where you you just put your messages there and again you don't have to worry about who the subscriber is or know anything about it you can you can lash them up that way and we'll see that with our build but the most important thing is you just put those messages on there and then they fan out massively SNS is like I said it's an old-school service it was announced over 10 years ago April 7th 2010 if I'm checking my notes here and it supports a variety of destinations some that you might not think about when you're building distributed systems but that's an bring that bring some incredible capabilities so you can send these topics to HTTP or HTTPS endpoints just arbitrary that you define you can send email messages so this is good for like alerting on infrastructure events and SS qsq can be the target and obviously that's how we're gonna build today AWS lambda functions can be the target direct target you can send SMS messages to users as users and subscribers and this is a direct publish message rather than a fan-out and you can send push notifications to mobile devices and one of the keys on this if you see on the screen here talks about high throughput you can send out millions of messages almost simultaneously right so you're looking a very low latency distribution of these messages if you if you look at some of the quotas here by default this is just a massive service right offers 10 million subscriptions per topic by default and a hundred thousand topics per account and you can request a higher quota so this is for a much broader distribution of messages than we can get so if you need to notify individuals inside of your system or inside of your processes this is a good way to do it do you need to notify sensors or like sensor data that kind of thing multiple endpoints this is a good way to do it as well Amazon sqs or simple queue service is actually the first service that we announced at AWS we announced it on November 3rd 2004 which means it's over 15 years old it actually wound up being the second service released behind Amazon simple storage service or s3 but not by much it was released to the public in July 2006 but these services are also updated with new features SNS received message filtering so you can filter and take different actions based on the content of messages that come into the topics similarly SQS introduced first in first out or FIFO queues in 2016 November 2016 and then just last year right after reinvent announced one-minute metrics so that you get greater granularity and visibility into what's happening in your queues how many messages are in there how far backed up they are average time to clear all these other metrics that you see and again sqs is just a workhorse of a service built to carry so much load when you look at quotas and restrictions here if my links work which they don't that's okay I know how to scroll y'all Oh scroll right past it I mean look at this you can create any number of message queues a single queue can contain an unlimited number of messages you only have 120 thousand in flight in any given time for a standard queue so lots and lots of throughput lots and lots of volume here the the stat that I hope is safe to release that we talked about Amazon uses sqs queues internally and so as a single service we've seen upwards of 25 million messages per second coming on here that's during that's peak load so again when you think about just the sheer capacity there are 25 million messages per second coming into a queue means you're really well buffered from whatever is happening downstream in your services and again just like SNS SQS is an old-school service service back before that was a term you don't have to say I expect this many messages per second or I need you know this many instances at this size you just define a queue and start throwing messages in it same thing with the topics defined topic start spraying messages at it this these things you don't have to think about that for you so they're sort of foundational serverless services for us here at AWS again when when we talk about these you will hear some similarities right with Amazon event bridge but these are all distinct services designed to work together very well so in our app 2025 architecture up until now what we had was a custom event bus running left to right putting business events on that bus and picking them up with workflows those were direct integrations with AWS step functions workflows both Express workflows for quick running processes and standard workflows for using callbacks and the weight per task token type jobs what SNS and SQS enable you to do here is a couple things one remember that the rules that we defined on the event bus are limited to five targets but in a production system you may have certain rules or certain types of messages that you know are gonna be of interest to a large number of consumers and rather than repeating those rules and chewing up the number of rules who are allowed on the bus or rather than trying to you know arbitrate who gets access to those rules you can just have the target for that rule be an SNS topic and then the SNS topic can in turn have these millions of subscribers like we talked about 10 million subscribers by default for a single topic so if you have an event that happens all the time that more than five consumers will want to know about you can use SNS to inform all of those consumers the other so that's that's how you would connect SNS into event bridge once the message is in SNS by introducing an sqs queue you bring a lot more benefits in on buffering on visibility and also on decoupling and abstraction right so if you publish directly from SNS to a lambda function which is a valid use case that we see a lot there's a little more coupling there or you may be connecting to a legacy system that isn't ready to be migrated the fact is that system can still pull that s qsq and pull out the messages that it needs as it's available so again it's that best practice in a distributed system of decoupling and buffering so if your downstream component whether that's a third party API or a legacy system or some sort of batch job that just backs up and then runs these jobs when it hits a certain amount sqs can buffer all those messages for you there because again remember just like we see here on the screen unlimited messages in the queue with 120,000 in flight so you can just pour messages in there as you need to until they're ready and so this is what we call this buffered fan-out pattern right some set of events is happening and going on to your bus quite regularly and the rule picks it up and sends it to its target which is an SNS topic and the SNS topic optionally filters it or just puts it into a queue and then from our number of cubes or you know 10 million queues it's up to you but the fact is between each interaction there are no direct subscriptions there are no direct couplings so you've always got this buffer in place and that's again that's why I like to call that that buffered fan-out pattern and that's what we're gonna be building today so we've talked about a little bit about topics and what subscriber endpoints can be but if you go into the SNS console in the AWS management console we can see what these topics and subscriptions look like so here for our example that we'll be using today we have this legacy operation topic and you see that there's a number of configurable options here we're not gonna get into all of these today but you can control directly who or what can put messages onto the topic or into the topic you can declare retry policies this is for if you're sending messages out to third-party services like web hooks slack notifications things like that come through here inevitably failures are gonna occur so here's where you can define that retry policy and make sure that it gets delivered logging we haven't configured login for our example today but you can log both success and failure deliveries it's especially useful in development or testing so that you can see what's happening with your subscriptions encryption we haven't enabled it in ours but you can do both default encryption or customer provided customer master key encryption for messages at rest inside SQS and then of course tag base and then you create a list of subscriptions as well so in this case we have our endpoint here which is a sqs queue but if we do this manually we can select our topic and these are those endpoint types that we talked about before right so you've got HTTP or HTTPS email or JSON and SQS q lambda functions I haven't set this account up for others like SMS and push notification which is why they don't appear again the filter policy where you can specify quite broadly and specifically meaning you have a lot of options so that you can filter and narrow down quite specifically which messages you're gonna send to the subscription and then read rive policies for dead letter queues so you can tell where to send undeliverable messages so you can see that the topic itself doesn't have to handle filtering right the topic can be here's every message related to customer accounts and the subscription itself can be okay I want all customer accounts that are delinquent more than 90 days or you know last activity was more than 30 days 60 days etc so that the subscriptions themselves give you another level of pairing down to only the messages that you need inside your account then if I go over here let's talk about sqs queues for a little bit because I mentioned there were two types these are just classic queues and distributed computing right put messages in take messages out in general the first message is into the queue are gonna be the first messages out of the queue but that's not guaranteed unless you look at FIFO queues and the difference with FIFO queues and standard queues is that FIFO queues have a little lower throughput but they have strict ordering and deduplication of messages so you you're looking at one time only processing of messages that come in there now don't get sloppy here what this means is that the queue is gonna handle the duplication of incoming messages so that a message is only received and acted on once but if it's sending it off to something else to retry that interaction could always fail happen again not complete in time and be retried but then be completed twice and this is where this concept of idempotency comes up all right even though you're only gonna see that message come in once downstream yeah ii DJ geek says he sought via - he is but he will try yeah you know we all we all make do with what we got right sorry so as these until they figure out I was saying oh man see you make jokes I need to go back and look through my closed captioning and see as these messages come in and are worked on just because they're deduplicated inbound doesn't mean that downstream things can't happen so always be aware of that there's a link in here again I'll drop the links in the chat but I if I can get over to the chat on my computer because who knows how computers work anymore there's a link in there if you see called lambda item potency and that will take you to a post on how to ensure that you're only processing messages once typically you do something like take the message ID and write it out to dynamodb check for the existence of that ID before you begin processing so that you don't duplicate efforts alright and then we had this question about to queue or not to queue idempotence UK yeah thanks Edie J geek for dropping that definition in here it's the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application so think about its it's a good place to put an example in here think about if you got an ATM transaction right and you deposit a hundred dollars and there's a network error after it sends that deposit but before it's recorded and it made it but you haven't gotten the confirmation back and so it retries now you've got to in-flight requests and if you don't think about idempotency you're basically just gonna deposit $100 or units of local currency for wherever you are once and then catch the second message and process it again with idempotency checks you say ok this is transaction ID unique ID ABCD deposit $100 and then you get it again and it says I've already processed this transaction so no matter how many times it receives that similar or that that same message it's not going to modify the outcome you can apply that message over and over but you only ever add $100 one time and that's what idempotency is so thanks for dropping that definition in there big guy appreciate you so we were going to talk about 2q or not to Q because in this buffered fan-out pattern that we're gonna build we have an event bridge rule to an SNS topic to an sqs q2 presumably an AWS lambda function out the other side right but I also mentioned that we can just set up to have an SNS topic public to published directly to an AWS lambda function and in fact that's the same architecture because SNS is gonna handle the retries and the buffering and the polling and the messaging and all of this and it does that by using sqs under the hood the difference in these two is there's two key differences one if you that may be a good pattern for you you have to decide one difference is you won't have visibility into that queue because it's owned by AWS so you won't know how many messages are in there you won't see it in your account you won't be able to flush it purge it you know change the read rive on it after definition the other is it's that tighter coupling with lambda when in actuality you could create a queue and have lambda functions pull messages off and your legacy compute pull messages off at the same time and compare the results to be sure that you are seeing what you expect to see or you could move back and forth so again it goes back to that question of decoupling but again these benefits of retries andrey drives i'm sorry oh i don't know what's going on with my throat today the benefits of the retries andrey drives are already there even if you go directly from your SNS topic to your lambda function all right one other thing that i want to talk about here is service integrations we had done a an episode on service integrations from the AWS step function series a couple weeks ago and these are good examples where all of these can sort of pass messages back and forth to one another as service integrations without executing lambda functions so you know your your SNS topic can publish to your SQS queue your event bridge rule can target an SQ SQ or an SNS topic etc it's just a nice declarative way to define that connection and then you remove the problematic aspects of undifferentiated imperative code right you don't have you know libraries and checks like this you don't have the same deployment issues because you've just given that declarative thing so all of the typical benefits from service integrations stand so let's uh let's jump into Visual Studio code here we'll do a quick refresh on what we have we have this infrastructure here which is a SAM template that just defines our custom event bus and a function that we used in other apps but for this case really all we care about is the event bus our topic today is going to be this operation received or operation performed and when we talk about our any company SAS application it's got core functionality that core functionality we can assume here is performed by a legacy system and we're migrating that into the cloud by allowing that legacy system to put events onto our custom event bus so from there once it's done its job we handle those with work closed so if I go back to the console here the AWS management console really all we're concerned about is that we built this custom event bus and it's named any company and so today we'll be working in operations and this is where I said we were concerned with that named event bus because we have to know where to listen or where our rule attaches it's gonna attach to the any company event bus and again I apologize on my throat let me take a break here and look at the waiver checks here edj geek a great way when using dynamo DB is to use conditional checks and dynamodb that's right that's right I'm pretty sure that he has done that himself all right keep driving on here so this defaults to the value that we've already given it that's great we have our legacy operation rule and again we talked about this buffered fan-out pattern is a rule to a topic to a queue right and so you see in where does that hit line 40 we've already defined all of this now we had to have a parameter here but in 40 lines we've been able to define that there's one more thing we need to put here which is the policy to allow us to publish the messages but that's it this is a straightforward powerful but simple pattern that you can pick this up and use it as a nested stack to create this pattern for any number of resources in your application so again this rule is just copied over from what we've built previously in the customers application it's going to listen on our custom event bus for operation performed events come from our source same as the other we've defined combat any company and then the detail type is the operation performed now a difference to note here when you create a target that has an SNS topic you don't need to create a role for event bridge to assume to invoke that topic that's all handled behind the scenes for you this is in contrast to when we talk about that customer service where are we our customer service we curated a rule which was invoking step functions workflow for that step functions workflow we had to define a role that enabled event bridge to invoke that target this is all wired up properly for us by event bridge for SMS topics so that's great all we have to do here is provide the topic they are in and then this ID is human readable name for it so this block of code takes our ingested event puts it into the SNS topic this block of code is all it takes to define the SNS topic we just need to give it a name which we've seen in the console here there's my SNS kind of my topics this legacy operation topic and then we define a subscription well what's going to subscribe to it we have to give it a protocol and an endpoint in this case our protocol is sqs and the endpoint is just a dynamic reference to this operations queue if that wasn't easy enough the queue gets even easier right this is the simple default now there's a lot more that we can add on here and in fact if we go to the sqs Doc's real quick sqs Doc's I mean let me do this differently AWS sq SQ we have these other properties but they're not required so we can do content-based deduplication for 5o queues we define a boolean if we want to 5 oq we can have a delay in seconds if we want messages to sit in the queue for some reason again KMS keys maximum message sizes retention periods time to wait to receive messages full read rive policies on what to do with these with errors so there's a lot of stuff that we can add into our queue and I'd encourage you to look through those Doc's and get a better understanding of what your use case requires but to just get out the door all we have to do is give it a name and once we've given it a name off we go then the final step here is we need a queue policy so for security reasons queues have HTTP endpoints that are used to put messages in when you create a queue nothing is authorized to put a message in it nothing and that's great that's what we want right but that means that we need to come back out then and at some point it's not useful until something can put a message into it so we create this sqs queue policy this is pretty straightforward it's got two properties it has a list of queues to apply this policy to so you can reuse it and then it has a policy document and it's a pretty standard I am statement where we allow the send message action and this like this hurts my heart but don't worry about this because our condition takes care of us here we allow our send message action from any AWS account so long as this condition is met and again you need to specify the queue name here under queues and you need to specify the queue a are in here under resource so pay attention to this whenever you're defining your policies if you're not able to get messages into your queue this may be why a good way to figure out what's happening there is to enable logging for your SNS topic and send failure logs to a cloud watch logs group so that you can see what happened and then finally this condition is what makes everything okay because the AR n that's the source a our end of the message must equal our SNS topic so what this means is that topic name in another account cannot publish into this queue this account with another topic name cannot publish into this queue anything else EC to EC s whatever cannot publish into this queue so we've got this properly restricted so that the only thing that can do anything to this queue is our SNS topic the find in this Sam template and the only thing you can do is send a message to that queue not like send him a message but you know like I actually put a message in that queue so this is the core of what we want to restrict access to our queue properly and then down here we've just given ourselves a couple outputs so that when we run this in Sam we get what we want now I've actually deployed this already and we've seen all of it in the infrastructure so I just want to take a quick look at the queue we have the permissions that we define down here where everybody can send messages as long as the source Aaron happens to be exactly equal to this account in this region from this topic right we haven't defined readwrite policies or encryption but if we did this is where you'd see them come up and then we can take a look inside this queue this again is the advantage of inserting a queue in between an SNS topic and whatever compute you use is you can watch that volume you get metrics on the average age or the you know number of messages in there lots of useful information especially in depth but also in production right it's always good to be decoupled so we see we don't have any messages in this queue yet what we can do is put some messages let me go back here I've created a file as always in our infrastructure folder with some sample events that match and in this case I've given it five operation performed events at different times so these all match our source and they all match our detail type which is how we defined our event bridge rule so if we go back here we can run events put events episode four and we'll get zero failures five event IDs great hooray that's what we wanted to see and now in a bit of magic we just refresh here and we have five messages available now why mainly because we haven't attached any consumers to this so just as we had built that simulator before we could build a simulator here to pick up the messages from the queue mark them as processed and removed them but I just wanted to make the point that excuse me I just wanted to make the point that these messages are gonna sit here until some sort of compute comes along to get them or until they age out according to the policies that we've set so what happened was we put those five messages on to this event bus where this rule matched all five pick them up and publish them to the SMS topic I've also got them going to cloud watch logs so that we can help ourselves out oops we can also see metrics for this rule here publish them to this topic which has a subscription of that SNS queue and then it put them in the queue so in this way events that come on to the event bus at a high rate can be sent out to a high number of consumers where they're buffered until they can be held hence the buffered fan-out pattern so just a couple things to show you here we've got this data point here where we had this triggered rule and the same the implications thankfully oh that's old failed invocation I'm in the wrong I'm in the wrong block of time here I'll have to wait for those to come up on this graph but so again it's gone through we put these events on to the bus it's gone into the topic it's gone from the topic into the queue and now whatever compute wherever can pick those messages up process them and remove them from the queue when it's done so again if I go back and and review here we've talked about Amazon SNS in its history as a over ten year old service now we've talked about Amazon sqs and its history as the first service we announced 15 years ago we talked about the different types of SNS endpoints that we have HTTP HTTPS email push notifications SMS sqs lambda functions we talked about sqs standard queues where topics just come in aren't deduplicated and are sent out we talked about sqs FIFO queues that Oh y'all I gotta sneeze hold on I didn't get the mute button I apologize I tried I pushed it didn't get it we talked about sqs FIFO queues where messages are duplicated and ordered we also talked about thank you et je we also talked about the need to manage our idempotency even when we're using FIFO queues and then we showed how these can integrate with one another and with Amazon event bridge to create this buffered fan-out pattern so again a little lighter on the code for this episode than normal but I wanted to talk more about the history of these services because of how long they've been around because then I had to look that one up it's good sold height in Dutch if we have any Dutch viewers that can back me up on this again I lost it anyway we we talked a little more about the history of these services because of how many features there are but also because of how stable these are right like we've got with the policy 70 lines of code in this file for a pattern that's applicable for anywhere from one consumer to ten million consumers from on-premise workloads to lambda like it's just pretty easy when you have this code to hook all that stuff up so what I would hope is that you would take this code and take your workloads and go away and think where is my system too tightly coupled or where do I think that I can't move to the cloud and then start to challenge your assumptions there start to break that apart with this pattern because it's a really good way to begin to make your applications more resilient and to make them more distributed so with that I'm gonna wrap it a little early this week go and take care of my throat see what's happening here I appreciate your patience with me on today's stream I hope you learned something I hope you've got something you can take away I hope my voice is better for next week but thank you all for joining us I'll be on the AWS channel next Tuesday same time for AWS step functions and then I'll be back here next week where we're going to talk about service integrations for storing data so again we're gonna move your data between services and get it into persistent storage without you having to use a laminate function to do it so thank you all for joining I hope you have a good rest of your week and a good weekend and I'll see you next week

Channel: Serverless Land

Views: 386

Keywords:

Id: krBKiABQJAk

Length: 39min 0sec (2340 seconds)

Published: Thu May 07 2020