Google Cloud Pub Sub Spring Boot

ladies and gentlemen subscribers and patrons good evening let us play with Google cloud and spring boots and let us do some pops and sub handling this means that if you are if you have tried rabbit in queue if you're trying to CAF cam then the pop and Shop the pub and stuff from Google cloud is will be very natural to you for yeah to use um it it's actually yeah it's quite simple you create some topics just like you do it with Kafka and then you create some subscriptions that can actually read from those topics um so it's it's more simple than I would say it's more simple than rabbit and Q but you have some of it you have kind of some of the the options like you have you can send you can send failed messages to all the failed handling of the messages to a date letter q that a little topic as it's called here um and then there's also retention there's also you also has to think about do you want the messages at least once or at most ones and most of the cases you want you want your messages at least once and then you want your your application straight to the handle that they can get the same message more than once so those are usually the the the the situations I've already created um I've already created an account with Google cloud of course and I've created a project named Mike's demo I have created one topic right here and it's quite easy to let me just create one more so here we say create and here we say demo L2 and here we can actually say this this is my topic so this is where we will actually Place messages too so this is the kind of the yeah this is why we send messages to this is the topic this is where yeah this is where we send message to then wait then we can choose um do we want to add default subscriptions many times if it's um if you have one subscriber to one topic then this is exactly what you want then you just create it then you will get a subscriber with the same name as the topic and then minus sub in the end and that is um that is usually yeah nine out of ten times that's what you want you want one and one topic and then you want one subscriber you'll have one target application or one topic Target uh third party as a third-party partner or something like that that needs to to get those messages right so so that is what you usually do then you can choose if you want to use a schema this is because if you don't if you don't choose to use a schema then you can get anything in into the topic then the message body that illustrations to it you can just set the string you can send a Json a piece of Json you can say you can send whatever you want so you can also send XML but if you use a schema then you are forced to actually um to send that information that you actually expect this will actually um this in many cases this is actually what you want because especially if you're working with third-party vendors then those vendors they they need to be if they get the if they get the error fast you know Fair fast then then you don't have to to communicate with them later on and say that you actually are supposed to send this to send this field it could be from from person or whatever to a person or whatever the two system from system and then they will actually get that error up front instead instead when they try to send a message to the the a topic so this is it's always a good idea so you get some validated data some structured data into your uh into your topic so so here's that um that that's a good tip yes it takes it takes you 10 15 minutes to to Define that schema and yes sometimes you will then actually get some validation errors that uh that that you're not supposed to get because the the validation the field that you actually expect was not supposed to be there anyway but then you've got that conversation with whoever actually sends uh sends messages to the topic and then you've got that sorted out and then you're all aligned on the same page so it's also good as a corporation and a tool I'm not going to I'm not going to edit for this demonstration right here I might make another video where where I use the schema also then we have the message with sentient duration that is not free and then if you if you enable that that means that then you have a cost which would in a Google Cloud this means that the that you can actually leave all of the messages there for some time um and that I'll say that is actually a good that's a good idea if you have a um when you have a real system that the where your subscribers might not be there right now right so you need to have some kind of a retention duration on it um and that's that's what you can set right there there's another place on a subscriber and a subscription then you can actually set the uh for how long the messages should be there after they have been acknowledged but that is another setting so let's just go right here our press cancel right now because I already have my topic which is right here you can also as you can see right here you can export it to bigquery or to to Big to cloud storage if you want to so you can convert it to yeah into one of the other storage features from from the Google in the good cloud of course but that's not that's not usually the use case for this usually this is when you want to uh to integrate systems to um yeah when you have a third party window you have a some division but you you cooperate with and you want to send them information and you also want to receive information then Pub and Shop is an awesome way to do that it's it's very stable and it's a very it's very stable and it's also very fast and efficient way to exchange data um yeah so let us so this is this is the subscription look when we have a topic you can actually have a you'll have a list of subscriptions you're going to have multiple subscriptions right here and the the default one that I have right here is test topic and then minus up right here you can also go click on the on the messages then you need to choose a subscription a subscription but you pull the messages from and then you can go and pull the messages right here I think I've acknowledged all of the messages right now so there will be no messages right now and so let us go back we have this test topic and sub right here and as you can see right here I've already been playing a lot with the with this topic and sub that's why we have these graphs for what's what actually happened uh like 10 minutes ago but now we have been talking enough right so now we know we have a we have a topic we have a subscription I've already created a service user I've downloaded the Json principles for that for that user and I have placed it in this folder named I'll just show you right here I'm not sure the content of course I will probably delete that use that the service use afterwards but uh in your in your springboard application the promises then you can set a credentials location right here and this is right now file and then it's it's placed right here of course you can place it where we want to on your machine or you can also use an environment variable that's uh that points to this file right here then we have another thing that is the project ID that is Mike's demo right now Mike demo uh let me just see it's actually misspells mics demo like this Mike's demo so this is the project that we have up in the top and you can see you have an ID right there that that assigned you want to put there um and then you have so this is these are my custom properties that I've created um but before we get so far let us create the spring boot project so I pressed file new project right here then that shows the next I just shows all of the default Java 17 and Gradle that's fine one thing that is not fine right here you cannot use spring boot 3 right now as it is with the Google Cloud messaging um Library if you're trying you can you will actually get an error so if you say Google Cloud right here we have the we have the messaging gcp messaging right here and you can see right here it says that you need to be below 3.0.0 in one so it means that you actually need to take 277. so text 277 I actually tried to see if I could make it work it is a hassle there's a lot of things you need to wire up and as soon as you get a new version of the Google Cloud messaging then it will fall apart again so don't use springboot 3 and TCP messaging wait until they have um wait until they have updated this module so it is compliant with version three but you take this one off take off gcp messaging like this then you of course you will choose your usual thing like you you might want some lump back of course number the bug is right there then you also want some web of course bring web maybe you also want some some relational database to keep track of which messages you actually have handled or maybe a mongodb so keep keep track of as you give some audience Trail um you get you get a lot with Google clouds you get a lot of insight there but there are some things um you don't give everything you also need to I would say you've uh you you would have to also create to create an auto trader maybe keep track of which messages have been handled and haven't been handled so you when you get the best the same incident twice because that can happen or or 10 times that depends on how you are how the pub and Shop has been set up and your deadline queuing has been set up maybe you have an error in your application so you don't acknowledge your messages then you will continue getting the same messages of course so your application needs to be stable and robust and handled it when you press ok then you end then you get an application like this and then the first thing you're going to do is you're going to create these two configuration parameters because then it is easy to change right so I've I've named this mic pops up topic mic pops up subscription because then I can easily get these two values right here test topic shop and test topic right there and what you need to do of course is you need to create a sub configuration you can name this whatever you want so I I could name this Mike's pop sub configuration and then I get this weird red I get that once and once in a while there's a bit weird is just The annotation processor that has not been configured that's okay but what you need to do is you need to annotate this class you create a class it went outside with configuration promises and then you say that this class should handle and you also get should get the values from mic that pops up and then you create an integrate the fields that matches the last part of your uh of your application properties right here that means you will create one name subscription one name topic and then when you wire that in anywhere in Springfield then you would actually have access to those two properties it's quite cool and the reason why I like to do that is because then I can easily add a new property here if I certainly feel the the urge to do that so that that now we have the configuration but we need to set up of course we need to set up some inbound stuff because whereas we cannot receive anything um yeah just one more thing at your application class at your application class you need you have to go and set enable configuration purposes and then the name of that configuration class that you're just used or else it would not be uh it will not be populated with the values from your application promises file okay so now let us go to look and look at the inbound configuration this is here some mathematics starts you can annotate this for configuration this means that springboot will look inside this class right here and it will look for beans and it will actually enter it will also wire in this Pub sub configuration that we just created and here we are creating two beans one is a message Channel adapter message Channel adapter um yeah and this is just an adapter that will actually handle the introduction setup I'll shoot um how should the message handling actually uh be handled and should the acknowledge mode be manual or also that's very quite important if you set it to Auto that means that if you are when you when you handle the messages later on this is I will this is the screen right here but let us not focus on that right now but when you handle your message later on if you did not throw an exception then then you would actually acknowledge the message this is what you can set right here if you set it to manual that means that your code wise has to actually acknowledge the original message or else it will not be acknowledged acknowledged or what doesn't acknowledge mean acknowledge means that your message is done being handled and there was no errors and that it will be deleted from the topic from the Google Cloud topic that's what it means so you will the message is ready for it to be deleted that that's what acknowledgment means um so this is an adapter we need to create so we need to set up this these things one of the things we set up in our adapter is this inbound Channel adapter here we set a we need to set a subscription name so here we say the subscription name and this is why I use my property that we I just created in my application the process filed get subscription this get subscription right here will actually fetch this value that I have right here so it would actually put this value in that I marked right down awesome awesome so now we have created this message Channel pops up inbound Channel adapter for inbound communication and then we need something like this we need to set up a message Channel and this needs to be a new direct Channel and that's just how it is it does yeah create a channel with default run Robin strategy so that is just that is part of the spring visiting messaging framework that we have to create this one right here basis Channel and that is because we use the message Channel up here we will get this injected up here so that means that's why we have to create this one right here um I have to uh I must say I actually don't know if that's if there are any any other options than the direct Channel I have not I need to dig into that of course but this is this is the default setup and if you read the default documentation from Google cloud and spring boot you can actually go to this in in Spring at the spring documentation this and a chapter regarding Pub and sub Google Cloud then you will get exactly this code right here you will also get exactly this code right here except for the except for the configuration part that I added right there okay so now we have the import channel here we have this is the handing of the messages so this is actually the interesting part right here we get a message in so we create a message receiver and this message receiver needs to be annotated with service activator so this means that when a message arrives and pops up input Channel this means on the subscription that you actually chose that you configured then this method would be called and this method will be called with then with this message right here so we have a message the message has a payload body there's a pillow button with some bytes in it then that of course can be converted into a string and sometimes it will be most of the times it will actually be a piece of Json right so you'll have a structured data format that got in right here and most of the times because you set up a schema on your um on your on your on your Pub on your topic and then you would then you would actually know that this is for instance a spaceship or a price for a flower so something or whatever whatever it is that you expect then we have some headers right here I'm going to leave this out commented because there are a lot of headers on a message the one of the things that we get on a basis is actually a message ID this message ID can actually be used if you want to make sure that you don't handle the same message multiple time if you want to check for that then you can get the message ID and then you can then of course you have Source store somewhere that you already have handled this um this ID and the new just continue to the next message of course you will also get a lot of dates yeah you get a lot of weird stuff there let us let us not focus on that in this video so we have the original message right here so this is because we want to convert it into a basic acknowledgeable Pub and submissions and uh and actually this is the keyword acknowledgeable we want to acknowledge the message that's like that's actually what we want to do so we will get some uh yeah we will we'll just say get the hit us from the um original message and um and then it acknowledges so this is just a conversion so we can actually acknowledge it right here and if we don't have this line then we'll continue handling the same messages again and again I'll show this in just a minute okay let us skip a bit fast forward right here because we also have an outbound configuration this is where we can send messages so now every time the messages will appear on the topic we will receive them and we will we will spray this out in lock right here but we also want to be able to publish messages to to a topic and right now it's it's the same topic as we also subscribe to this means you'll be sending messages to yourself yes it's weird but it's of course it's configurable so that we can just create a new topic and then um listen to that topic later on if we want to um this is just for a demonstration purpose again we have the configuration right here then we have something called service actuator this is an output channel so now we have a pops up output Channel and the output channel is defined um right here so we have the default request channel right here that this is a messaging Gateway a messaging Gateway you can that's like a postman right so now the post office actually this is the post office so here we can here we can actually send this means that every time that we send a message like this then to pops up then we will actually send a message to this channel so the to the spring message channel right here and then we have a service activator right here that would actually um they'll pick up on that and then it will um it will create a new pub sub Handler and then they would use a default template so this is the default it pops up template and then this is the then this is the topic this is the the topic that we have configured and again what it actually means is that it would just send the or just send a message to to the topic so that was a bit it's a bit less cool for the outbound than the inbound but uh yeah and then we have a control arm so this means that we can actually send a rest message now to um if I set that to send slash forward slash and then I I have some kind of message inside the body it could be Jason it could also just be a string then we would actually say them called the messaging Gateway and then we will say send pop to shop listen to pops up sorry yeah and we have the Gateway right here this was the pops up outbound Gateway that we have right here and we have the configuration of it of course right there so this is actually an interface that we we create right here so we have the interface right here and yeah so that is uh a lot of times for spring you just create interface and then spring will create the implementation itself so it's actually not that much we have one two classes actually that you have to to you to create and then the other two classes actually they are those are the controller and the configuration those are of course uh not mandatory this is just something I did to make life easier for myself now let us try to we'll go to Services we'll see if we can find the application run configuration types let us say spring boot right then and then I can press stop because it was actually already running I want to I want to start it up so you can actually see the startup block and what's actually happening so let us start up the application and the first thing that you'll probably experience is that you'll you'll get an error and and you can error that you cannot find that credential file or there's something wrong with your credentials because you need to have the right roles of course set up um on the in the Google clouds and you have to set up a service account of course and you'll have to point to this Json right here um but I don't again I don't have any errors so I'm quite I'm I'm quite happy another you can get when you start up to begin with that is yeah we have the we have some um that is the credentials another area you can actually get that is if you have not created the topic or the subscription if you if you forgot to create those and if you're not creating those programmatically you can't do that but I'll save that for another video but if you forgot to create your topic or your subscription then you it will be quite clear um what you have the problem will be correctly especially with the subscription the I think the topic error you'll get that when you try to send a message the first time okay so that's quite cool now let us get some messages Mike so we can actually handle these messages okay okay I've already created a curl right here so this means I can actually curl in a Hello message I can call in a Hello message which is right here hello this is a test hello this is a test um for Mike let's change a bit so you can see that it is actually live and it works and then I and then I curl this to 80 cents because um that was the endpoint that we created so let's just press play let us see whatever happens so now we're actually sending a message to um we're sending a message to the refers to us bring boot application and then and then it will send it to the topic let's go down and see well what happened here yes send this message to outbound Channel hello this is a test for Mike great message arrived payload hello this is a test for Mike I am so proud of myself already so this means that it actually it actually works so we send it and we send a message from the yeah to the the controller then we got it right here let us play with some more messages because right here we can actually also push we can actually send some messages to the to the topic so if I press topic right here I go to my topic then I go to messages then I can actually say publish messages and then I can say spam they may spam the message system with maybe 20 messages and yeah one second uh apart that's fine and here I can actually create some Json I can create some Json right here this could be and this could be a flower flower and let's put the rose here we have a rose and here we could have a quality something equality that could get a eight hour set baby right so this yeah let us just publish that so what happens now let us see so we can actually see painting messages and success and error messages so this is a way where we can test our topic we can test our topic we can also tip we actually also testing this the the the subscription so we're also testing this the subscription because if we would get any Eros then we will see that right here see look here we get all the payloads so this is our spring boot application and we get all of these payloads right here flower rose flower rose flower rose yes yes yes cool one per seconds do we get more oh we have all of them ended up with success so that is quite awesome right that is quite awesome and this also means if we look at the to see something with the metrics yeah here are the metrics and messages there are no messages right now let us try to skip the acknowledge part actually I'll just I'll comment the acknowledge part right here in the inbounds configuration right here I'll not I'll not acknowledgment the message this means that the message will just line us though and then I'll restart the application I could actually also just stop the application I just want to show you that you can actually see some of the messages in the Google Cloud um in the Google Cloud console and then you can then you will get the fields at least it will not show you the whole message if it's too long but uh yeah let us see what happens now again yeah let me just I'll just um I'll use my package Json right here and here I can actually set the spaceship yeah here I can send a space if I've prepared this spaceship right here model fuel caption let us send that to the the spring application that they would that will then forward it to uh the topic so I'll send this right here curls and chip yes let us see what happened yes we got it we got the spaceship right here model round fuel but we did not acknowledge it this time oh see look now we get it again this is because it's twice to the subscription is actually set up not to um not to have this back off um retention and you should always set up your you should always set up your subscription with in exponential uh back off retention so it will not continue spamming your system with uh with with messages um that can lead to errors in your application this should um yeah you should use the back off for attention and then you should also make sure that they end up on the dead letter topic on it on the little shopping we set up that up there's a bit more complex I want to focus a bit uh on springboard in this video right here but of course you can set it up also um yeah so we continue getting the message right now this also means that if we pull the message if I pull the message is right here and if I'm then I should be able to get the message yeah that is and look what actually happened right there we actually got this is the message but if I go to the right side right here then all of the fields were actually picked out and there and put as a as a column and then you can actually sort on the fuel on the captain um so this makes it a bit easier to find the right um your right messages you can also filter your messages right here you can create some filters so you can see what is happening to your messages right now the deadline exceeded um so yeah so the deadline the deadline just succeeded and that's why you should have a death letter topic so you can get these messages back again and then you also use it for debugging because then you're going to look at these messages and then you can see ah it's because they don't have any destination the spaceship doesn't have a destination field or property right it doesn't have a it doesn't have a destination property so so that's not good right um that it will we will continue trying to we'll continue to to try to get that message right here because we have not set up the the back off retention and that's this deadline that exceeds right here it just shows that something is wrong um which it should have been handled by now um yeah but and then now it goes back to acknowledge so this is because yeah it has some it has some different stages right now it's waiting to be acknowledged and then at some point yeah a deadline exceed it should have been it should have been acknowledged now by the application but it was not so then it knows okay something is wrong and then then it will try again on the subscription so that's how it works that's not that much logic to it but that's why you need the letter Q of course that is actually I I really like pop and Shop it's it's really really easy to to make work you don't have to think about your rapid in queue application and backup and scaling scalability so you don't have to think about all of that that is actually all of that is included in this Google Cloud as usual that is of course the usual benefit of using the cloud you don't have to think about scaling you just scale and pay extra yeah you just pay some more then get the company a credit card and a new then you're good to go right um It's So yeah so there's a lot of things you don't have to think about the same with the Apache Kafka is quite complex actually to set up in uh both rabbit and Q and Apache Kafka it can be quite complex to set up in a cluster of course when you know as usual when you know how to do it it's not complex but it's it's just something that why why would you also have to think about that right so that that's why we love the cloud that's why we choose the cloud right sometimes you would not choose the cloud because when you choose the cloud then of course you are also a bit getting married to whatever uh Cloud you're using the most because um yeah for instance this Pub and Shop it's it's not the same if you go to another cloud and it has another name another brand where they of course but it is awesome asynchronous technology Brandon Cube Apache Kafka Pub and Shop it is cool it's awesome if you haven't tried it out then please do so and I will um I'll share this project as usual on my GitHub account just search for it it's name the pops up right there if I'm not too lazy then I will also place the URL in the description below thank you very much for watching have a great evening or day as you can see it's actually a day right now it's uh the sun is shining that is why suddenly I got a bit transparent in my forehead can you see if there's anything in there yes thank you very much I hope to see you again soon bye-bye

Channel: Mike Møller Nielsen

Views: 7,163

Keywords: demo, tutorial, programming

Id: zZDs6f9EUj8

Length: 29min 48sec (1788 seconds)

Published: Sun Jan 15 2023