Laravel: 1,000,000 webhooks/minute?! Ingest webhooks like a PRO

good morning everybody so today we are going to talk about something that is very cool um we are going to talk about web hook handling so let's get started all right so what is the problem that we're trying to solve well in every single application you have some sort of web hooks for example you have stripe that is calling you whenever there is an update you have um instead of read notifying view of the status of the email that you have just sent there is there's plenty of examples like that right maybe maybe there is an other that you're getting from some some first party all right so this would be like a normal like a very basic example of how web hooks usually work there's a separate party that's hitting your app and there is some other first party as healing your app now what would happen let's say if your app went down so now if you're up out of the question what will the third party do if the third party is handling web hooks properly there will be some time in which they will retry post the data again until the server acknowledges that but we don't know first of all whether or not they Implement proper solution we don't know what's the time in which they will try and most importantly we don't know how often they will retry and this is especially important let's say for for stripe so if your app goes down for five minutes and there is a customer who just you know paid for your subscription they don't have access to anything well that's not really good and we need a better way to handle that what if between our third parties and our app there was some always working relay or something like that and that relay would always accept the payload and keep that locally and your app would just communicate with the relay Whenever there is a chance whenever both of these are online but well basically whenever your app is in line in this way even if your app goes down you're still Gathering all of the information about the web hooks that seems fun doesn't it well let me show you how to implement that well basically we would have a Lambda function that would accept our payload from our third party then it would send that to sqs and put that in a queue and then our larval application will be able to read that job and execute it okay so let's build that in order to do that and to have some help I will use this great package by Aaron Francis called Sidecar and it basically allows us to easily deploy our Lambda functions and execute them in Liberal applications so let's get to the installed okay so now that it has been installed we'll need to configure that but before that let me quickly show you what it does um so we can copy that and inside of our app the auto recommends us to create a new directory called sidecar I'll probably just call this lamp does and inside of here we'll create a new uh stripe stripe Handler let me just paste that let's import this class declare return types and now inside of our resources we can create a new directory called Lambda maybe let's rename that as well it will create stripe.js let's add an export here and then adjust this to write that Handler all right so after we configure everything we'll be able to use this function to deploy it to AWS but as of now we have not yet configured our connection details so we can go to the documentation page and here we can go to installation and after we execute this command it will go it will publish a conflict file we can use the site car configured go through everything we need I will skip this step for you know security reasons but after all of that is completed we can go to our sidecar.php and inside of here we need to register our functions so that would be our stripe Handler so let us deploy that get another error Lambda Center file nor directory um yeah I'll just move that to top level and now it should be good right so now if we go to our Lambda configuration on our AWS account we can see that our alarm that has been created so this is great now there is one thing that this package won't do for us and that's understandable because like I'm just using a side effect of this package um just for like an easy deployment but it won't create as far as I'm aware a function URL for us so I will just create that manually and just set it out to nothing and then save it and now I'll copy that URL because it'll be important for us later once we need to send that to our third party provider like stripe right we'll need to provide this URL for them so that's why we need that that URL um so normally you would you know go to like your services.php and inside of here you would do something like uh Lambda stripe URL and here you do um ripe Handler URL or something like that then you can just reference that but yeah this is this is great um and it seems to be working just fine so now what we need to do is basically create our Handler and we have already established that we'll need some external packages because we'll need to interact with sqs and uh to push the job so we can go to our Lambda and inside of here we can do npm install awssdk lines us and I'll just enable note support we have a nicer highlighting and now we should be able to import sqs from sorry why error right yes okay uh clients and we can establish the connection that is Qs equals new sqs and we just need to specify the region as far as I know because AWS will like automatically know that you're using that and uh because that's it will inject your credentials so instead of hard coding the region here I'll just do process.tnv that is region and uh instead of here what we can do is we can create a public function variables and these will export whatever we type in here to our Lambda configuration our SKS region will be for example our SKS region from our config file which I will change in my EnV so let's go back to our Handler so the way we want it to work is basically you know we'll have some body and if events is not new and event is not undefined then we can do something like body equals Json dot parse basin.stringify event um it's because they even extend as as a text that's like not even prepared for Json which is pretty weird but I'm not in I'm the expert I'm sure there is a way to to you know tackle that to be better so there is nobody we can simply return a status code of 244 for 22. and now we'll need to return you know status code of 200 when everything is alright and inside of here we'll need to do some try catch now let's do error and or we can so error error and we'll just return you know status code to 500 and instead of here we'll do a weight sqs and message and here we need to provide some parameters sqs or Rams and our sqs params will be aren't equal that would be process.nv.qurl it will specify that in a second and then a message body um we'll catch that in a second well we need like some sort of data and here would pass our body I'll get to this in just a second and we need to stringify that as well and now it seems to be correct so in my container I'll just redeploy that Lambda this package under the hood will see that there is a right dysfunction that exists and because of that it will not create duplicates it will just update the existing version hmm however that will fail um simply because we have not I mean well two things right first of all we said that we'll Define this queue URL but we never did let's do Q URL and it'll be our services slam Dash stripe um new name let's go there um let's go to services and we can do qname and this will be like um sqs prefix and Lambda stripe Handler you well let's copy these two and I will Define them in me so sqs prefix will be dependent on your accounts uh but display sqs that region that was and your account name and our queue will be created first this should be and because Normal does not work properly with Vito as far as I know we'll have to use standard and let's just name it stripe and this is important because this is basically you know the message will thank you for four days so if your app goes down or four days you're still able to receive those pills after that I know that there are some packages that try to handle fifo but um this is kind of outside of the scale of this lesson uh so we'll just name it stripe and okay I'm pranks okay let's create that queue and here here you'll have your prefix that's what I mentioned before and then you'll have your Q name which is right so let's go to our Limited and let's just run a test see I mean we need to redeploy that first now let's run test and we'll get the 200. and I'll just open this in our tab I go to sqs you can see that there is one match available that's great that's exactly what we want now the the part of communication between Lambda and rsqs is done so now we'll need to get that sqs job from from that q and executed in arrival the problem is that by default it won't really work because Marvel expects a certain format of those messages so how do we overcome that well there's this great package that was not working for me um so I created my own version here probably supposed to use this one uh honestly it's not very actively maintained but um I have created my own version which you know I suck at open source so like yeah like look here I have no cool thing doing it uh but yeah this this kind of works um so yeah we can we can use it for now that depends are also untrue but uh you know I'm trying to do my best um at some point I will spend more time on that so I'm busy right now either way we can install the package and what it will do for us is it will create jobs based on like any payloads that you have in sqs so this is really useful for something like that it's my version of the package by the way only works with larval 10 plus um so you should probably use um the other one if you have something else it works the same um we'll need to do a vendor publish and it could be number nine for me let's go to SKS plane and uh yeah we can can read that um so let's just do HP artisan make job base handler job let's also do stripe Handler okay these are not even jobs so yeah like if you follow documentation it basically says to just create this and copy that so it shouldn't really queue didn't really do any of that and I'll just clean up this I'll just copy this error base Handler and you know normally we probably don't want anything to hit the base Handler um but yeah okay the way it works is basically you have an association here between the Q name and a Handler so I guess we could do something like new name maps to type enter job now we can remove it so this is a key value map of an sqsq name and a Handler but before this will even work I will need to specify our sqs plane driver in our q.php so let's go to q.php and here I will just paste this um and update those values so basically you'll need sqs prefix here this will be that default you know queue this yeah I would just say that to defaults and always specify the queue um here we can do region nice car access key and uh Secrets no at this point and I'll remove after this so don't even try uh standing here um correct now if we do pH purities in Q work sqs plane and specify a queue so it's stripe we can see that worked but it did not match because I did a noobs here so this should be like a few URL as the full thing and then Q name we just had the full name and we need to update our our reference here to fq URL instead so now if we run that again we'll see no jobs but if around our beautiful Lambda again will be able to see that it's actually executing stripe handler job uh instead of the base Handler here we basically have the data as well oh I'm updating the base Handler that makes sense in our stripe handler job if we VAR damp our data send that again you can see that it contains the actual payload that we have passed from our test so if we adjust it so it's uh hello YouTube and send it we are able to see that here as well so now if our app goes down so you know in a situation like this and we get you know a couple of those callbacks you know nothing will happen on the app end but as soon as it goes live we'll just process all these jobs in no time so that's all I wanted to show you um you can also by the way push events to sqs from that since it will have the same you know formats which is not larpally uh so that can also be nice if you need to communicate between different Services you could use that for like a microservices for example to communicate between different Services if somebody feels like it and has more experience my packages feel free to clean up this mess I will uh try to do something in the near future to make it more pain because demonics really sucks but yeah um that's that's pretty much all I wanted to show you so this allows you to have pretty much an independent scale with your web hooks um I hit the creative solution for this because by the age of there is n't my main solution actually wasn't that nice but there is uh hundreds of thousands of cool bikes that we're getting in very short amount of time and like by creating this you can very much easily you know remove the problem of of handling that on the request time uh so that's all I want to show you I hope you guys did enjoy it and if you did please leave a like below and let me know what do you want to see in the next video thank you so much for watching see ya

Channel: Przemysław Przyłucki

Views: 3,878

Keywords: laravel, vue, tailwind, saas, laravel saas

Id: Yo6UUwGQyeA

Length: 20min 51sec (1251 seconds)

Published: Mon Jul 17 2023