Amazon SNS/SQS with Go | Paul Borlin

[Music] well good afternoon this is a talk on Amazon SNS and SQS I'm gonna have a volunteer at the beginning she's volunteered to pass out some candy so anybody want some candies raise your hand she'll give it to you mmm candy so your applications look like this sometimes 404s you got to get some delay in there ain't nobody got time for that and they're waiting for it and nobody wants an orangutan in pants waiting for application nor do you want a baboon in on the key be what keyboard to be waiting and all of us think sometimes that our users are baboons at the keyboard but sometimes it's our fault so what are you gonna do I'm Paul Borland I'm the senior engineer at Compassion International and I'll also be an architect to the senior engineer at various places like Philips and others but I want to talk about compassion real quick because I have a passion for it we really still drew from poverty and Jesus's name and these kids like this that they use the garbage dump for the playground for their food for their clothing for their toys he's got a toy in hand he just made by himself and we give a better life by connecting sponsors like you guys to these chili 101 and then we have we have over to a million children in the world that we've released from poverty and we support it's amazing the scenes change lives there from what they've done we educate them we give them food we also give them clothing and they hit sustain their life so you can support we're all over Europe and your countries probably listed up here so go support a child and we can change the world together so what are we talking about today so we're talking about a senescent ask us before we do that we need to talk about what our measure Brokers we need to understand that before we can get into technical details what are the some of the bandages you're using a message broker what does an Amazon SNS and what does the Amazon SQS two separate things in it Amazon and then we're going to learn how how we can communicate with that using go now that's our ultimate goal so what is the message broker a message broker this is a stockbroker here's the Wikipedia definition I'm going to read for it you can go online and look at that but essentially it's a service that coordinates messages between other services and they're good at it it's analogous to the postal service so if you have a postal service that routes your mail all over the place you as a publisher of the message you write an address on there and you put it in a one designated spot it gets routed to the other side to another receptacle which is kind of queues up the mail on the other side all I care as a publisher it gave that it gets to that mailbox I don't care if my my cousin or my uncle or anybody else gets it I just wanted to get to that box and get it taken out so just like you have a way place to publish your message it gets routed in his place that gets queued up on the other side a message broker does the same thing with services there's a publisher that publishes the mess of some ways and it gets routed to a destination now I don't actually have to know the directions to my uncle's house most service does that for me same thing with a broker so why use a broker seems like some middleware well why not just do a rest call directly to it well asynchronous communication is probably the best thing for it so what would you rather do standing out line up above or would you like to to put a letter in the mail to deal with your business because you don't want to stand in that line you can't do anything else while you're staying in line maybe you can play on your phone but it's your limit of what you can do so the publisher doesn't have to be be blocked when the message is published you'll have to wait for some other service or some other entity to do something with a message also back pressure can be regulated so certain times a day that line might get longer shorter and you don't have to have different more tellers to deal with that so let's talk about back pressures so this is an example of back pressure in a restaurant you have dishwashers that wash the dishes at a lunch rush or a dinner rush they can be different stacks you bigger and smaller and so if the waiter or waitress had to stop wait there for the dishwasher to do the dishes in the order that they got there the customers be very happy the waiter be unhappy and the dishwasher probably unhappy so a here over a real world example with back pressure when I worked at Philips Respironics these CPAP devices has prescribed people who have sleep apnea or they snore and it gets pressure through through masks to help them breathe at night so when a doctor prescribes this they set the pressures they have to flow and then you know it's working for a patient so this device records that session while it's going at the at the certain times of the day that data is uploaded to our services it gets parsed it gets put into reporting and then the doctors can see the reports the patient's doing well well we tried to ram it randomize the the upload of the data but the problem with that if when you randomize it to help our services patients would unplug the devices they would do all kinds of wonky things with their devices so the data was never getting to us so a marketing person said hey when the blower hit that blower off button upload the data right then it sounded great we released it what do you think the problem is with that everybody wakes up at the same time in the day so our services got crushed there's a problem with our services where we were trying to parse the data at the same time as we're pulling it off so we re architected as we pulled off the data put it on a message broker so that all our reporting and parsing can be done off offline and then we got we're starting any data so at the times where it's low in the trough then the data can be processed and our services we'd have to spit up more nodes our immediate reaction was to spit up more nodes that didn't work so most of message broker was able to be that middleman for us so now I'm going to give some highlights of message brokers really really really fast I have another talk that I talked of these details but for this I want to be able to get to the code so high availability multiple brokers and have our broker has multiple nodes servers can be synchronized for those nodes there's a guarantee of delivery there's load balancing that occurs and it can be alive when your back-end is not not there failover when a node fails others can take over there's no data loss distribution my favorite part of a message broker just like we talked about I could just put the mail and one refs receptacle the distribution you can just publish to a an exchange that gets distributed to the queues this is the queues right there so it's a very analysis be handing out candy so I gave it to one person she distributed the candy I didn't have to stop my talk to figure out who needed candy who got candy a few people so I didn't have to know who you were I have to stop my talk same thing with a service the service doesn't have a stop what they're doing so this is great for an event-driven or reactive design architecture one important thing about a message broker it does no work on the data and it shouldn't just like you don't want your postal carrier open up your mail and dealing with your issues with your lawyer you want it just to be delivered now Amazon does break that with lambda functions inside a queue I would say use that as a last resort applications contact brokers brokers never contact applications it's important distinction brokers communicate in various ways we'll be talking about HTTP because that's what Amazon uses additional features we have retry logic a dead-letter queues time-to-live auto-delete persistence and transient messages cues you have limits on your messages and queues and add different types of distribution various brokers out there but we're talking about sqs and as SNS so when not to use a message broker well I'd say if you have big messages that's probably not a good thing now how big is big you'll have to check your throughput and figure it out but I would say if your messages are getting up into 100 100k think about what you're doing so similar to rest services if you need synchronous communication don't use a message broker I've seen people try it it fails miserably oh and it's just like any other tool if it's if you don't need it don't put it in just to play with it great uses for it you offload work to be done later like an example with my philips respironics now you do a quick response to the user which kind of ties in the first point if you need to handle back pressure at various times a day why spin up nodes and spin down nodes all day long maybe that'll cost you money just have a message broker queue that stuff up it's not if it's not important for that time and more than one receivers the best part of it I think you know how the service doesn't have to do any discovery which other services need that message all right we slow down a little bit we'll start talking about Amazon SNS this is a simple notification in service you have a publisher that publishes a message to S&S and it says which topic he wants to publish it to so a topic is just a named space that in SNS you say I want it put it there on the flip side different things can subscribe to that topic so whoever subscribe to that topic at the time gets that message as soon as you publish it on there now it's very important that if there's no subscribers on an SS topic your message will go into space gone forever so that's an important key to that you can also do message filtering as it comes through ask us a simple queuing service just just that it's a bunch of cues that you put messages here then you have a producer and you have a consumer to each of these cues so here's a topic and cues tied together this is a scanner message broker set up Amazon splits the two out whereas most brokers kind of love them together so there's SNS part with a topic and the sqs with the queues now you have a service that can publish to the topics but the service can also publish to queues but you have to know about a should've it will shoot queue it's nice to have just published to the topic and then you don't have to you don't care which queues or what service are spinning up and spinning down and then you have one of the many services subscribed to a queue HQ so advantage of Amazon SNS and SQS over other brokers HTTP is simple that's the protocol it uses we all know how to use HTTP it's easy the SDK that Amazon built for all their stuff is very very well built and it's easy to use there's no need for a shutdown some other brokers you have to make the connection and shut it down later it's hosted for you how nice is that potential disadvantages that animals on SNS and SQS there's pulling because it's HTTP you say you have a message no you have a message no you have a message yes and it gives you a message there are some ways to ease that we'll talk about that later cost maybe you have a lot of throughput maybe that's gonna wreck your pocketbook so maybe some other alternatives might be there it may not be as fast the brokers there's various reasons for that if you have questions about that afterwards I can talk about that messages have a max life of 14 days that seems like a long time but if something goes on in production you may not want that to start destroying your messages needed advanced features another broker so the other brokers has some more advanced it's this is simple both simple as notification and simple queuing for those two alright enough of my talking of PowerPoint we'll see something cool my let's first go to Amazon and I'll show you how you create a queue where did my thing go I'll skip that I'll come back to that if I got time so I have an application that I built let's do it what that's going alright we'll come back to it so I have an application that I built that interacts with SQ s and s and s so i have a vet various pieces to it i have a way to list topics and i have a way to queue topics so listing topics do list topic and will show you the code unless all the topics i have set up and accuse so these windows here I have I have one publisher and I have three consumers so I'm going to start my publisher and I've told it which I'm going to use the topic and we're which AR end to publish tube and I'm have a subscriber having three cues on this topic go days one go days two and go days three so now as I publish a message it goes out to Amazon SNS and gets distributed to this one that's listen to a queue I have two other consumers I can listen to the other two queues there's number two that message was sitting there waiting for for this one to wake up somebody else ascribed number three that's sitting there as well now if I type more messages they're all going to get a copy so my my publisher never had I never had to redeploy it we can reconfigure he or figure out anything else it just works because I'm publishing to a topic now what do you think would happen if I connect all the same queue what one of them gets them another one it's sort of random and this is due to the polling we do with the HTTP each of these it's just about timing now if I type fast maybe CID season so my command line application I explain that every time I hit enter it sends a message off now if I go subscribe back to the other two queues all those message that I just type we're still sitting there queued up now you'll notice that you might have noticed that some of these are not in order so Amazon standard queue is not first in first out okay just there's various explanations for that amazon has a great definition of why things are a FIFO March FIFO but it this simplest answer is because of the clustering that it does behind the scene all right I'm gonna try one one more time with Chrome I don't know why where chrome went in my it's right there where is that that's a weird icon alright so if you go to Amazon you can you can select up here go SSNs simple notification service simply get there and this is your dashboard you get there you can go to topics and you can create a topic so I want to create a topic I'm gonna call it demo as simple as that now I've have a topic that other things can subscribe to so let's go ask us let's create a queue same thang create queue when I call it demo one now here's your options you have a standard QR first in first out queue and it does give it a simple description here your differences one important thing is you cannot subscribe a FIFO queue to S&S so you have to use a standard queue there's lots of other options of skinny but skipping over but it's a simple demonstration so now I can take this queue and I'm going to subscribe to that SNS so I can say subscribe q2 SNS topic pick my topic demo and I subscribe I can go back over to the SNS and if I hit subscriptions my topic or my subscription will be there somebody go my topic and I'm going to go into my demo topic what publish a message say hello so that's published to the SNS because the demo was subscribed I'm going to look at the queue and here's my message and I open up more detail and here's my message here hello now we've got a bunch of extraneous metadata it was thrown into the message you can change that by going to the subscription and we can edit it and we can tell it I just want the oh this is the topic my bad I can edit this and I can tell it doing the wrong spot why is that subscriptions here we go enable draw message delivery so that tells it I just want the message directly from the service whatever they published so now if I were to publish a message to that that will just gets the message straight all right back to Mike you see what's going on and here it is the second try just not none of that metadata so that some of that metadata you could still get off the headers or the after routes of the of the message and go so you might not want to have to wade through all that stuff maybe you just want your your data alright let's jump to the goal and machine so let's talk about how to set this up and go land or in go so first thing you need to do with anything with Amazon's SDK you need to get a session so I'm gonna get a session gopher skull common now I created myself a common function so that I can use it all over this application and so the build a session we essentially get credentials and my credentials are the region the access key and a secret key now there's various ways you can do credentials in Amazon I chose the access key and secure key for this demonstration so you put that in your configuration for Amazon and you create a new session easy as that configuration new session return recession so back to our demo so the next thing when you want to work with sqs or S&S you have to get the instant stuff of that so make it SS of SNS and my ultimate goal here is going to be to list all the topics in that so we do SNS yeah and new yeah I'll give it a session so now I have a new instance of SNS I call it SVC because that's what Amazon uses in their documentation so now we're list of topics so list output I'm gonna ignore the air like a good developer and SVC dot list topics now I can pass a filter into it but I want all of them so I'm gonna pass nil and then with my list list output now the topics inside of there and I can do a loop on that and printed out if I want to like that so this is a boilerplate and actually was a good developer well I actually created this before so this here this is a full version with the error handling so you get a session to go over this again you get a new instance of S&S with that session then you have your operation let you do you get an output and you deal with your output now you'll start seeing a pattern with an amazon's sdk you have you'll have a an input to the operator so you have an operation you have an operation you have an input for that operation so a list topics input and you always have a operation output list topics output so we same thing with queues i could do that you get a session do a new sqs instance I can list my queues and deal with the output and go through all the queues publishing let's look at publishing so my application you saw it brought running I has a couple met camp command line parameters one was the weather on publish to a topic or a queue and that's that keyword here so the topic I'm passing the function of send SMS and then it's going to loop through the console looking for input so the reader when it reads a new line it's going to then try to send that as a message all right so again it create a new session and we're passing it to a sender since our sender was s and s or this is function here so we get a new instance of it now operation is published we have a publish input and there'll be a publish output so there's a pattern there go with Amazon so the published input essentially takes the message and the AR n where it wants to publish that message so the topic AR n so I published that and then we can get what the output I don't have the output here but one interesting thing about the output is the message ID so when you put a message on NASA NASA ask us it'll give you a unique ID for that message for reference later similar if I had chosen to publish to SQS same thing I have a session I get a new instance of sqs I have my sent message operation sent message input and there'd be a sent message output and say similar to the s and s have a body of the message and we have the URL to send you alright consumer a little bit little bit trickier but it's easy so am I in my command line I just grabbed the AR n that I want to subscribe the cue that I wanted to subscribe to so i'm ascribe to it I get a session and a new instance sqs and then I receive the messages so I'll salute we'll look at that looks like so again we have operation that we want to do receive message recent messages input the input has you have at attributes and you might ask what's there between attribute names and message attribute names this is simply the message this is the attributes that Amazon puts on it and this is the attributes that you put on the message when you publish it you have the cue URL where you're going to publish it to maximum messages so that with the polling hope with that polling when you say hey do you have message no you have a message no if it has up to the some max number of messages it will give that to you so instead of pulling for every message it'll give you bulk amount of messages wait times seconds is handy as well so instead of it immediately returning to you you can say hey wait three seconds before you respond to me so in that three seconds if Amazon has a message for you it'll give it to you so it's saves on the polling and that can be up to a maximum of 20 seconds visibility time out so whenever you get a message Amazon puts us puts that aside for you and nobody else is still in the queue but nobody else can grab it this billet visibility timeout says that hey hold this message outside 20 seconds before you put it back in queue Amazon won't wait for you forever you have to acknowledge it alright so receive a message to get the output there's messages on the output and I'm going to get those messages I'm gonna iterate through those as an iterating through this is how you respond to Amazon that you want you're done with that message unless you delete the message it will put it back in queue for you so again we have a dually up message operation we have a delete message input and there would be a delete message output so the input is the queue URL and there's a receipt handles so on each message this is different to the message ID they will give you a receipt handle what it gives you the message so it gives you that handle that you can use secretly and say okay delete this so something else isn't going to delete your message for you so this is essentially the cycle of consuming and that's the end of my talk thank you very much for for listening and working through my chrome issues I'll put the code up on Twitter later and any questions I used in the past like five years to go or something but heavily and back then there was actually the problem that there was always a certain Latin see it was always delayed when I pushed a message into the queue it could take up to 50 milliseconds with an instance running in the same data center and when I contacted the AWS support I've actually taught me like that's basically by design we guarantee that even if you sent a million messages per second it won't be more but even if you sent just a small amount of messages you always have this delay this get any better not that I know I'm aware of it has to do the short answer again the technical document the the documentation has a technical ask for that but it's the clustering it figure out figures out where in its big cluster of servers where it can put it so that is why I mentioned it might be a little bit slower than something else to bother to use the other questions all right well thank you very much [Applause] [Music]

Channel: GoDays

Views: 1,440

Rating: 5 out of 5

Keywords: #GoDays20, talk, goconference, golang, Berlin

Id: iKC9ctbStU0

Length: 27min 14sec (1634 seconds)

Published: Fri Feb 07 2020