Argo Events - Event-Based Dependency Manager for Kubernetes

[Music] event source decoupled desynchronized sensors can we do something similar event-based asynchronous type of system our lives almost everything we do is asynchronous especially when it requires cooperation with other people yet we tend to design our systems to be synchronous why is that why why do we do that if you take me as an example when i publish a video on youtube i do not know who will watch it i do not know when people will watch it i do not know how many people will see it and that is not important i mean it is important but we get to that later as a publisher my job is to publish a video and i do not know who will do what and how with it that's not part of what i'm doing i'm a publisher you dear person watching this you're a consumer and our tasks are asynchronous they are event based i'm a publisher you're a consumer and then the roles might change you might write a comment and then you become a publisher and i might read that comment and then i'm a consumer or i might not there is no guarantee there is no synchronous process between those actions we perform some action we create some event and then we continue with our lives that's how we operate in a real world outside of software engineering type of setting our systems should do the same some processes should be publishing events and then some other processes should be listening to those events and doing something but those two actions publishing and consuming events should be decoupled desynchronized are not directly related with each other a process publishing an event doesn't need to know who will consume it and the consumer doesn't need to know who published the event those things are decoupled in real life and we should try to do the same for our systems that is how kubernetes works when we execute cube cattle apply we are telling the system that hey that there is this event that somebody something wants to create this object or to update this object or to do something with some resource and then there's completely separate processes inside of the cluster that consume that information and do something with it but when we execute cubecuttleapply we do not wait until that something is done that something is created updated what's or not no no we get only the acknowledgement hey i received your request something will happen sometime but for you the only thing you should know is that you got acknowledgement that i understood what you want go away do something else for example if we create a service mesh interface or smi type of resource we do not necessarily know whether it will be linker d or some other controller that will do something with it we might know because we designed the system but that's not really the job of the publisher we're just telling kubernetes hey i want you to have this and that and then let somebody else something else take care of making that happen and in a very asynchronous way we get acknowledgement and that's about it if we want to know the state of that something if we want to know what happened we need to query at some later moment the system and say hey did you did this happen or not happen but publishing an event is only giving you the acknowledgement and nothing else because it is even based asynchronous and so on and so forth and now comes the question here is the important question can we do something similar to our components our applications our system can we create events without worrying who will listen to those events and can we create consumers without worrying who will publish events can we make those operations completely separate and decoupled and the short answer is absolutely yes and there are many ways we can do that but today we're going to explore how we can create even based architecture or processes using argo events we are about to explore argo events in 20 minutes or less the only thing i prepared in advance is the kubernetes cluster i will be using mini cube but everything i'm showing you applies to any kubernetes basically so it doesn't really matter as long as you have a kubernetes cluster you should be able to follow along and here is the important part if you do want to follow along all the commands i will be executing are in a gist and just the link to the gist is in the description or the comments section so go there get the gist copy paste the commands and all that jazz so the first thing i need to do is to install argo events it can be done in a quite a few ways preferable way is to use argo cd to basically manage everything in your kubernetes cluster including argo events but today i will not use argo cd for a simple reason because i want to go with the simplest possible solution that will be cube cattle apply one or two cube cattle applies and that's it going simple you shouldn't do that in production use argo cd and if you don't know what argo cd is the link is above there above my head uh somewhere up uh you will find the link so just go and follow that if you're not familiar with dargo cd but not today today we are going with the simplest possible installation that is a couple of steps first cube cattle create namespace argo events we need the namespace where we're going to install everything or deploy stuff and come on there we go and then we need to apply a couple of uh definitions uh specifically first we need to create custom resource definitions for arguments so that's cube cattle apply in the then file name and this long address this is the official project maintained the definition manifests of for the all the crds actually i was wrong let's not see this it's crds plus rbac plus deployment for the controller basically it's almost everything everything think of it as everything but everything except one thing we need to install also even the bus i will explain later i might even draw on a screen what all the components are but i will do that at the end for now just trust me apart from the base installation we need even bus and i will explain later what even bus is so cube cattle uh apply and then the namespace where we want to apply these argo events and we want to apply something available in a file which is on this address as i said before it's in the gist so just go there and run it come on come on come on and there is only one step missing uh i think it's only one step missing we're going to clone a repository git clone that i prepared that has sample demo manifests or sample application whatsoever the things we will need is in a repo that i prepared for you and that repo is in https and then github.com and it's me it's my repo and i think it's called argo events demo dot git there we are that's the one oh not found that was silly of me i'm missing the t here argo events there we are now it should work there we are type for everybody makes typos every once in a while so i'm going to go to the directory where i clone the local copy of the local clone or local copy of the repo and that's about it now we can dive into creating events and publishers and consumers and all that stuff all that jazz comes right now when we talk about argo events we normally talk about three different components or pieces one is the even bus that is the one that we installed a minute ago it's the one that manages all the events it is basically a bridge between uh even sources and triggers and now i already mentioned the other two components and that's even sources and triggers hey even sources are sources of events entities that publish some events and those even sources can be many different things i'm going to show you later a list of what can be even sourced but for now think of it as a publisher of some event and then we have triggers and those are entities that will trigger certain actions depending on events created through event sources right so publisher and consumer in a way of events now let's create our first even source there can be many different types of even sources we're going to go with simple one just to demonstrate how how it works so we're going to use not use create a web hook as the source through which we can generate events so let me show you a definition that i created earlier even source and this is the simplest possible one just to demonstrate how it works so we have here an even source called a name we're going to call it webhook webcock means that we're going to expose an entry point in our cluster through which we can send http requests that will uh receive events right that will become the source of our events there can be others my favorite for example would be git or github or gitlab webcook is even source there can be many others but we are going simple just http webhook or endpoint and what do we need for that in specification we need a service uh in our cluster that will uh watch for expose a port and the port it will expose is 12 000 and as a port 12 000 as a target port and then we have the type of event and in this case the type is web hook there can be others i will show you the address in a few moments uh the list of all the currently supported uh even sources but we're using webhook as even source and then we can have one or more uh specifications inside and each of them starts with the name in this case it's called devops toolkit it can be any name you want and then the port through which that webhook should be exposed the entry point which in this case is slashdev's toolkit and finally the method which is set to post and if this works fine we should have the source through which we can generate events and that source will be web cooking simply it will accept http requests so let's create that one and that's cube cattle dash name space argo events we want to what do we want to do apply something defined in a in a file um even source yammer and there we go it is created a couple of things happened first of all if you do cube cattle dash dash namespace argo events we can see that an even source was created get even source right that's the one we just created it's called webhook and that even source created a few things for us and one of those things is that it created a service here we are you see this one web cook even source as you see that's a service that was created for us and uh it exposes port twelve hundred twelve thousand sorry internally right we should normally create now ingress controller that would uh manage incoming requests from outside of the cluster on the cluster ip 12 in the port 12 000 but we're not going to do that i don't want to now dive into ingresses and what's not so we're going to do a simpler way and that is to port forward to the to the pod so we're going to let's take a look at the pods we have here here's the one that was created for this event source so i'm going to copy the name of this spot the name that was selected and store it in environment variable uh even source uh pod or not for pod name equals this right so to put forward i'm going to execute cube cattle this is boring namespace argo events and then i want to port forward don't do this in production this is just a replacement opening the port on localhost instead of creating ingress anyways put forward to the pod event source pod name and the port is 12 000 and it will be exposed it fell as 12 000 on my localhost and they want to run it in background that's the ampersand at the end right okay cool now i'm having access to that service and i can see what can i do with my with the web hook that was just created for me i can for example send uc url curl to send a post request remember we specify that we want to post and then as a header i'm going to use content type to be what is it application jason right and what else do i need i need to send a payload uh some message uh of the event basically i'm now creating my first sending my first event as a web cook and that message will be what will be message i'm using json now and that's the key and the value will be my first first webcook that's it let's see whether this works did i forget anything oh yes i forgot the most important thing and the address is http localhost and then 12 000. and we said that the entry point that's what we specified in event source is devops toolkit right now there we go we got success see here it says success so the system successfully accepted an event uh that contains the message my first webcook and the event is the webco type of event on the port 12000 and the entry point devops to dash toolkit right now what is curious about this is that we got the success and yet we did not define any trigger we did not define what will happen when somebody creates this event we didn't do that and that's normal because those things are decoupled when the system responded with success that means that that is let's say acknowledgement hey i got your event and that's successful i understood what i received the message successfully i do not know whether there is anything listening on those events maybe there is nothing maybe there is something that's not my concern the success means i successfully received the event not that hey i did this i did that we don't know that and right now there is no trigger there is no sensor that will do anything with this event so success means hey i got the message nobody is listening on those events so nothing really happens inside of the system now before we move on let me show you this page in the documentation here we are this page over here let me make it bigger right now those are the type of even sources we can create right it can be menio it can be calendar which is chrome the cron-like type of things uh a file can generation of a file or some action on a file can be an event uh resource a creation or any action on a kubernetes resource can be an event webhook that's the one we used so far amqp kafka mqtt nuts let me make it bigger nuts sns sqs pub sub github github can also be an event source it can be sending us notifications that somebody created a pull request or merge something or whatsoever it can be gitlab it can be hdfs slack storage grid azure event sub stripe emitter redis nsq pulsar and finally it can be generic generic means generic that's the one you're supposed to extend if you want to create your own even sources that are not covered as one of the even sources i mentioned already the link to this page with the list of all the event sources is in the description so anyways we created an event source that is basically accepting events as webhooks in this case and we are not doing anything with those events we are just receiving them so far let's change that let's change that by let's say creating a sensor you don't know what sensor is that's normal i'm going to explain to you what sensor is in a second and the best way to explain what sensor is is to show you one sensor here we are it's a big one isn't it right so uh here we are at let's go at the top there we are here we are creating a sensor it's going to be called web hook mostly because i don't have enough imagination to think of a clever name for it and then we have a service account name which is argo events sa that's the service account that was created when i installed argo events which has decently wide permissions to allow me to do what i'm going to do you don't know yet what i'm going to do but i'm going to show you in a second and then it has some dependencies right and this is basically what connects this sensor this trigger with the event source that we defined before so basically dependency of this uh sensor let's say a trigger in a way is the web hook event source that we created before so whenever somebody sends a web hook event this sensor will be activated and the name of that event uh because if you remember in the web cook event source we can have one or more events we defined only one but there could be more anyway the one that we are talking about right now is called devil's toolkit so that's a sensor that will connect a trigger with even source and the trigger is defined here in the template so let's say let's imagine for the sake of um argument that whenever somebody sends a request on a web hook we want to create a pod right we want to perform some action in our kubernetes cluster and that actually could be to create a pot and this is how we would do that right we would create a trigger in this sensor and the type of the trigger is k8s there could be just like there are many different types of even sources there are different types of triggers and in this case we want to create a pod and the definition of the pod that we want to create is defined in the source section or source resource so we are creating a pod here using using api v1 we are going to generate name payload dash what that means that it could be a name it could be fixed name but then there would be some clashes over there by using generate name instead of just a name we will let argo events how to generate name that has this prefix so it will be payload some random hush and the rest is a typical pod specification right we are not this is exactly the same specification except for generate name as we would use for defining pod normally right it's going to have a container that has a name hello based on alpine image it will echo the message this is the message you sent me and then nothing now the second argument is nothing and this is critical this is important remember it i'm getting to it in a second finally we have some parameters for this for this trigger and those parameters are the payload dependency name if you go back up here you can see that we have this dependency payload which depends on the webhook that we created earlier so we can use that as the parameter so i'm going to take the body and the message of that body from the from the payload and we're going to pass it to something and that something is to spec and pay attention here here is spec and then containers here is the containers and then zero means the first container in the list of containers and then args means the arguments and we want to to inject this uh the message the the message from the payload from the dependency from the event source as the first argument remember it is zero indexed meaning this is argument zero this is argument one so in other words whatever we send as a message to the web hook that we created before will be injected instead of the empty argument over here does it make sense are you following me so far i hope you are i'm not sure whether i'm confusing you or not anyways uh let's create that cube cattle namespace and then we want argo events and we want to apply something in stored in a file sensor.yaml go go there we are what happens if we send see url request like before the one that i created before this one before we were getting uh acknowledgement that the event webcook in this case was received let's see what happens if we and nothing else happened right before let's see what happens now if we send that and yes it is uh successful again but unlike previous attempt now we have also the sensor we have a consumer of those events uh and we can check that easily by retrieving all the pods so where are we here cube now let's clear the screen cube cattle uh dash dash namespace argo events get pods and you see this one payload that's the pod that was created as a result of us sending events so we sent an event that event was processed by uh the sensor and sensor has a trigger to create a pod it could be anything else and if you look at the logs for example logs of the selector what they use is um as labels i think it was app equals payload i think uh here we are this is the the output of the pod that was created as as through the trigger which says this is the message you sent me and the output is uh the message of the and the message of the payload we sent through the through the c url request there are two important things very important left to show you first of all here these are the sensor triggers that we can use actually triggers are here i think or here where is it here yes here are the triggers it could be as a result of having a sensor that is dependent on some event source we can create some triggers and that triggers can be ka tes which is to manipulate communicate with kubernetes api which is the one we use it could create the argo workflow basically if we use dargo workflow here combined with argo events we would effectively have a fully operational kubernetes native cicd system and by the way a couple of you requested asked me to make a video about argo workflows and uh please let me know just still the case i'm very eager to do it i just want to know how interested you are in our google workflows anyways it could be http trigger we could send an http request as a trigger it could be aws lambda we could do something with lavender it could be custom meaning that whatever you want kafka nuts us we could send a slack message we could do things with open risk which we could generate some log entries uh and switch i don't know what switches uh it's probably not anymore used there is deprecated anyway and this list is growing growing as we speak now the most important thing let me draw a picture of what is happening so that you understand what's going on the key piece of the puzzle the the the most important component here is the is and still it was still is even bus right that is the most important thing that uh we installed uh in our cluster and it is important you will see why here now uh let's let's jump to the next component and you will see how even bus matters here we created some even source event source right in our case that even source uh was a web cook right it can be many other things so we created a web cook even source uh we could have other image sources that would do some other listen to some other events what matters for all those even sources is that they are listening to some events happening somewhere right those events can be web cooks uh it could be what else can it be it can be uh some pub pub sub uh events as well it could be something from s3 packets sns doesn't matter it can be many many different things what matters is that even source that we created is listening to some events and whenever it detects an event it writes it to even bus so it goes to even bus and then we created uh some some sensors right sensors sensors are deciding uh to listen to something right so uh this is the error in the arrow in the opposite direction so sensors are listening to the events that were written in even bus those are decoupled right there is no direct relation between an even source and a sensor and just as there can be many even sources there can be many sensors and sensors are basically listening to the events and triggering some actions so we have some actions going on here and those actions can be like in our case to to do something with kubernetes resources uh maybe to generate some uh argo argo workflows uh what else can be can be many different things it can be something with i know lambda for example lambda you can see that i don't know how to write well uh what matters is the sensors are listening to the events uh that are stored written in even bus and then triggering some executing some triggers that will do some things and uh that's all there is to it we have even bus and then even sources are listening for some events and right writing those events to the even bus and we have sensors that are listening to the events written in even bus and executing some action triggering triggering some um some operations and the the best example my favorite example of all that is for example using github as the even source that will listen for web hooks uh whenever we push something to get repository or create a pull request or whatsoever and then sensors that are listening to those events and trigger and triggering argo workflows and if you combine those two for example argo events with argo workflows we have a fully operational ci cd that is cloud native that is designed around events that is cloud native kubernetes first designed for kubernetes and all those things basically potentially and i'm throwing it out there argo events combined with argo workflows is probably the most advanced and potentially the best cicd pipeline type of solution we could have in kubernetes and that is just one of the example of what we can accomplish with argo events it allows us to build systems based on events and orchestrate what will happen when will happen but in a asynchronous decoupled way now i already mentioned a couple of times argo workflows and the reason i have a hidden reason why i'm talking about target events because i want to talk about cargo workflows next and argo events are almost a prerequisite to fargo workflows so if you would like to know more about argo workflows some of you already sent me comments keep sending those comments and i will create a video about argo workflows for now you have argo events and you can use them to design event-based asynchronous type of system so you saw a very quick overview of what argo events are and i will be honest here it takes some time to wire our brains it took at least me some time to wire my brain around events and to start thinking in terms of how i can organize everything to be fairly decoupled asynchronous and yet everything connected and everything working together through even sources and sensors and triggers it is not trivial it takes some time but the results are amazing if you do invest in events it will make your system more resilient easier to design easier even to reason with afterwards so there is an initial investment there is initial change that needs to happen but once it does creating systems especially complex system based on events is very very rewarding on many different levels that's it uh please subscribe to the channel you know what to do hit the bell icon click the like and all that stuff it means a lot to you because you get notifications it means to me because youtube thinks that you liked it because you did right you liked the video anyways so do all the jazz and see you next time see you in the next video cheers

Channel: DevOps Toolkit

Views: 20,750

Keywords: dependencies, event-based dependency manager, argo events tutorial, Kubernetes, event-based, k8s, Argo Project, Argo, Viktor Farcic, Argo Events, events, event based architecture, kubernetes events, event-based system, kubernetes architecture

Id: sUPkGChvD54

Length: 34min 27sec (2067 seconds)

Published: Thu Jan 28 2021