Full Tutorial: AlertManager Set up and PrometheusRules

hello and welcome back to my channel if you're new here welcome my name is Anise and I make videos related to kubernetes and the cloud native EOS system now this video is going to be about alert manager if you're completely new to Prometheus monitoring in your kubernetes class to observability and related topics I highly suggest you to check out one of my previous tutorials one of them is linked up there and the others are linked Below in the description if you're curious now if if you want to learn more specifically about alert manager you are at the right content the Right video tutorial as always if you prefer the written version of this tutorial it's also linked Down Below in the description link below in the description is the link to my blog post what is alert manager a lot of times we set up monitoring in our C cluster now with open source tools that's likely going to be with Prometheus you can visualize the metrix then through grafana dashboard reps or similar and that's the open source stack that you can set up for specifically for metrics from your application now grafana provides an option for you to set up alerts through grafana so let's say uh your application reaches certain thresholds then you're being notified about them you can also set up alerts directly through a tool that comes with Prometheus with the Prometheus operator called alert manager what is alert manager I've sketched this little graphic so this is your cubanus cluster let's assume this is cues cluster and in your cubanus cluster you have for example a monitoring namespace so this is for example the monitoring Nam space and in that monitoring namespace you want to install Prometheus and you can do that easily through a Helm chart that installs the promus operator now I'm a huge fan of the cube from you you stack operator Helm chart and that installs lots of different Tools in your classroom so it will install alert manager in the alert manager configuration it will uh install pot monitors probes uh the Prometheus agent promethus itself Prometheus rules uh scrape configs lots and lots of other useful resources it can also install for you grafana uh inside of your class stud the open source version so we are using the helm chart to install the promethus operator and that runs then promethus in our cluster for example in the monitoring namespace and we will get to the Practical part in a second in this tutorial but basically as part of Prometheus as when you install it you set up the Prometheus configuration now as part of the Prometheus Cube stack operator this one you can also install alongside alert manager and you would configure aot manager the same way when you install it you can also then add configuration after the installation it's up to you but basically everything that's here highlighted in blue is part of uh the hel chart installation now afterwards you can set up Prometheus rules that configure the alerts that you want to be notified about so Prometheus will basically know what to alert on and then tell the alert men maner to fire the alerts that's the relationship between those so Prometheus rules set the alerts everything that you want to be notified about and then it will forward if one of those threshholds one of those alerts are triggered it will forward that to alert manager and alert manager will have different uh configurations and now where to forward the alerts in this tutorial we are going to forward alerts to our public select Channel okay so I'm connected two uh two note cuberes clust it eventss on jke um JK JK JK on Google Cloud platform Google kubernetes engine um as you can see both of my notes are ready now currently I don't have anything installed if you know my tutorials you know that I love the tool called K9s because it allows you to do a double dot I don't know how to call it English and then for example look at your name spaces or I can um go into the default name space and also look for specific um already Parts I can look for deployment able to autocorrect and so on and so forth so in here we're going to create um monitoring namespace so also connected to the same cluster I'm going to say Cube C create namespace monitoring and everything that we install related to Prometheus will be in the monitoring namespace and then we can go back and check our namespaces here's our monitoring namespace now created awesome so we have the monitoring namespace set up now we want to install the C promethus stack with specific configuration right so I've already set up the configuration that I want to install that I want to use this is my monitor security example repository I've used it in previous videos before and I'm going to use it in this video and here we have uh the different rules for the helm charts so we have here our prom values if we just want to install Prometheus with kofana and Loki then I've used these values before now in our case we want to use all of these values and it might look AIT little bit complex right when you look at this at these at this volue CML file that we're going to provide to the promethus helm chart but it's not that complicated I promise you so first of all how do I know what to configure here right if you have never worked with Helm this is going to be overwhelming and I highly suggest you to read the hum book because it has been written by the creators the core maintainers the creators no the the core maintainers of Helm and it's a really good book and I've based lots of my Helm Focus tutorials here on YouTube on that book so you can also if you want to have the gist of it you can also check my Helm related tutorials they also link below in the description of this video but how do I know what I've configured here right that's the next thing so we're going to go back to the helm chart that we want to install this is the helm chart Cube prome stack Helm chart and basically every Helm chart has a value SEMO file it's very smart in that and here are all the different values that you can configure and usually it tells you how you can configure them and why they're configured by default in a certain way so here all the different values if you want to modify a value uh for instance if you want to set the default rules create to false you would need to see what is the highest level here in the yaml right what is the highest level in this case it's the default rules and then you need to set default rules create two faults and so you basically would copy this part and then set it to false if you want to for example set alert manager to false here in this line let me zoom in a bit in this line then you don't need to copy the create if the create is going to be true so basically if you want to set alert manager to false you need to copy default rules you need to copy rules because alert manager is nested in this one and this one and then you can set it to false within that that's how it works basically so that's how I know where I take those values from partially going back to our configuration what do these values mean so this is for promethus the configuration and service monitor select a n users home values is by default true but we have to set it to false so Prometheus can pick up the different metrics from service monitors from across your cluster that's really important these are not as important these are just the default but these ones are very important and you have to make sure they are nested the next thing is the rule selector the rule selector basically specifies how promethus knows where the promethus rule rules and we will get to that in more detail but it's really important that you have a key for the selector and then a value for the key a key in yaml is for example type and then a value is low key so key value so here we setting a key promethus and then values now the values in this case is an array you could also add um and of name for values and then you can use Prometheus and as a key and a value to three or example rules right um so but basically this part is just needed so that Prometheus knows where how to connect to Prometheus rules and that's done through labels in your kubernetes uh manifest so we will look at that in a second but basically this is really important that you have that otherwise you can't configure the alerts that promia should monitor so basically otherwise you can't configure these alerts um the promus rules that this is a crd this is a Yama manifest that we will install once the help chart has been installed and prus Rule basically specifies which alerts to track and it has a label um or labels and then within the labels key oops [Music] key value this is the key value promethus and an example one or what what it was set there and this is important this is needed so that Prometheus knows where to find this crd because this crd will be somewhere in your cluster but promethus doesn't automatically know how to connect to it so it will need to have a mechanism to know how to connect it and that's done through the labels in your crds but will show that in more detail in a second anyway going back grafana is the same configuration you don't need it if you're not using grafana as part of this tutorial you don't need it you can use your own custom grafana configuration whatever you like this is from a previous tutorial just because a lot of times I use low key in uh together with grafana and Prometheus as well then and the best part is the alert manager configuration now you can see here my slack API URL and you will have to add your own slack API URL because after this tutorial I will basically make sure that this web hook doesn't work so you can't copy it and then spam my slack Channel I don't want you to do that but for the sake of this tutorial I left this URL here so as you can see here's the URL the select API URL and we need that so alert manager can connect to slack now this is just some configuration I more or less copy past it from somewhere online and the receiver null is kind of needed I read somewhere because by default alert manager will have a receiver n which is here listed as part of the receivers so we have to list it here for some reason as well but basically we want to add the route for receiver Slack and the route for receiver slack is going to be this and we can customize the text and the message a lot more later on but basically it it's just there so aler manager knows which channel is allowed to post and what rout to use to post on that channel that's all this uh value manifest specifies if you have any questions post them below or send me a DM or comment on the video probably best way um it's also further explained in the blog post so check that out first before you ask questions but then ask questions if you have questions anyway um it's something that I found uh when while I was doing the research for this tutorial is that it's very poorly explained where you put this information and how you configure this information and then where you put the promethus alerts because you can also specify promethus alerts uh promus rules right through the Val manifest right here as [Music] well so basically in this Val y manifest you can also specify the promethus rules however that can get a lot more complicated and it makes the coupling between the hch installation in your cluster so the way that you've installed the hel chart in blue a lot more tightly coupled with the promethus rules and ideally I want to manage prus rules independently of the hel Tru because I assume that my promethus rules will change in configuration a lot more they should be a lot more Dynamic and have different life cycles than the promethus helmart installation that's why I've done it in such a way so now we want to go ahead and we want to actually install this Helm chart now you can find in the main gith repository you can find a way on how to add the promethus community Helm chart to your chart repository list for this you have to make sure that you have Helm installed so we have Helm installed we have Cube cutle installed if you extra cool you have K9s installed um and then if you have everything installed and you have access to a running q&as cluster you can first at the Prometheus Community charts and then you have to update the repositories home we update I think that's the command and you update your entire list of Home charts that you have access to locally and now you can go ahead and you can install this Helm chart for promethus with our custom configuration we are going to do this through the following command Helm upgrade D- install so we can reuse this command and basically once the hel chart is already installed it will be upgraded if it's not installed yet it will be installed that's is what this means this is our the release name of this Helm chart you can name it anything else that you like it's just important that you remember this information later on because it will be useful when you log in for example your girl a dashboard you need the relase name but that's a different topic and you can check out one of my different tutorials for that then you have here the Prometheus Community chart and a cute promethus stack chart from that that we want to install we install it in the monitoring name space and we provide the custom value sest but it's not going to be called prom values it's going to be called prom from alert manager values yaml and we're going to pass that in now before we do that we just make sure that everything is saved over here and then we can go ahead and we can install this Helm chart and fingers cross is going to install but I seem to have misnamed alert manager is this correct okay now it seems like it's going to install it and this is going to take a few seconds okay so now it's installed and we can look at our class set we can go into the monitoring name space and in the monitoring name space I will now have alert manager gra installed St operator and Matrix note exporter note exporter because I have two notes in my cluster and then the C promethus de Prometheus running um now you need to know the details on what the difference are between all of them uh that's okay but basically these are the parts and they relate to the deployments in your cluster so these are the three deployments in addition to the individual parts that you have running and then you have Services services and in the services you can p forward really nicely through um pressing shift F I believe um when you're on K9s and you can forward it to your local host so we're going to do that for alert manager and we're going to do that through the cube promethus St promethus and as you can see prom is the release name so whatever you named it if you named it pom pom it's going to be pom pom here instead of prom uh depending on what you named it so we're going to forward also Prometheus to our Local Host now and now we can access both locally okay so as you can see this is our aler manager at Local Host 1993 - alerts as you can do is nothing here because we haven't said any alerting rules yet um and nothing like you would only see here something I believe if any alerts are firing because Prometheus is tracking the alerts once they are triggered the Prometheus will forward it to alert manager alert manager than managing the alerts that's the name alert manager it's not actually managing the alerting rules just the alerts when they are triggered now in promethus we have a section called alerts as you can see I don't have any inactive pending or firing ones and that's something we want to change now we want to basically set up alerts uh for our security metrics in our cluster now to have security metrics in our cluster we first have to have security scans to produce security metrics and that's what we going to use the trivia operator for the trivia operator is using trivy an all-in one clown and security scanner under the hood it's a cuetes operator similar to the C stack operator right an operator basically in kubernetes alternates human behavior inside of your cluster so anything that you would do manually in your cluster instead an operator is going to automate that within your cluster so the trivia operator is installed in your cluster and once it's installed it automatically produces vulnerability reports config audit scans so basically basically scanning your configurations your kubernetes configurations from cluster resources from Mis configurations if anything is M configured it's can there any exposed secrets in your cluster your arburg rules and lots and lots more including compliance skins it also produces s bombs of your running workloads now the soft bill of materials of your kubernetes workloads is different to a k bomb a c a kubernetes bill of materials so Asom here relates to creating an AS for all of the deployments of your work clads of the application stack the K bomb that trvy the CLI can produce which is one of our new features just fym basically scans the core kubernetes cluster so anything that's within your Cube system namespace that makes your Cub this cluster run before I even touched anything else in this cluster since that uh for all of its libraries dependencies and so on that's the K bom that trivy separately can do and you can check that out the link if you're curious Below in description also up there now there is the video to Kum if you're curious about Kum but that's what sbom does so just a friendly FYI here but we're going to use trivia the trivia operator install the trivia operator inside of our cluster now if you head over to the documentation you will find lots and lots of information so we are on 0.17.1 and here you can find the different custom resource definitions that will be produced by this operator instead of your cluster once it's installed but we want to install the operator first also through the helm chart so again we have to add the uh chart repository Aqua registry chart registry hel chart registry to our hel chart registry list so we're going to do that now add that and then you have to always do a h repo update because otherwise you could run into installation issues uh if you've added a new hel chart if I don't use the H rer update and aqua has released updates to the charts to any of the charts within the registry I won't have those updates I can't install those updates that's why you would do Helm we update just FY now clear now we can go ahead and install the trivia operator Helm chart so this is the command but actually since I produced tutorial there was a new version released uh so thank you for the maintain us for that um so I hope fingers cross is going to be working the same way with the 0.9.1 version of the hand chart but this is basically how you install the TRV up over the hand chart so again this is the release name of the trvy operator this is the chart registry and this is the helm chart that we want to install in outside of our cluster we want to install in the trivy system names space we want to create that name space we want to set trivy ignore and fix to true because in our vulnerability reports of the vulnerabilities that are present in the libraries that are used within our container images here is a nice graphic right now so in those libraries they are scan for vulnerabilities and sometimes vulnerabilities are present but they don't have yet a fix available meaning we can't fix them so why should we look at them necessarily and then we want to also set the service monitor enabled to true why would you want to set the service monitor enabled to true that is so trivia is installed with a service Monitor and if there's a service monitor installed then we told Prometheus earlier remember that it can track all of the service monitors across our cluster meaning if the service monitor is in the triv system names space Prometheus will know about it and that's how you can install the hel chart now should be fairly quick okay so the trivia operator is installed in our cluster we can check it out if we head over to our cluster we know have a triy system Nam space in the triy system Nam space we have lots of vulnerability reports these are chops meaning they spin up and they die and they spin up and then once they're completed they're dead they're done right um now once they're done we have vulnerability reports crds in our cluster that tell us what are the vulnerabilities of the container image should have been scanned but basically all of these jobs relate to one deployment one replica set of uh container image that is being scanned right now now if you don't manage your pods through replica sets directly it's going to scan the individual pots um but this is what those vulnerability report jobs are there for and then we have our trivia operator pot running here as well and as you can see this name space is right now very active but soon it's going to be coming down a little bit we can check our vulnerability reports across our entire cluster if you press zero at K9s in K9s then you will automatically see uh the reports across all your different name spaces so we want to set up now a rule that basically notifies us if we have within a specific application names space new critical issues new critical vulnerabilities then we want to be notified of them um if there's any new critical issue then basically alert manager should receive that the alert is firing and shouldn't should notify us that there's a new critical issue new critical vulnerability detected in our application namespace so as you can see here we don't care necessarily about the critical issue in our metric server uh because that's within our Cube system name space and we usually don't have that much control over if you're using a managed kuus cluster now at this point we need to tell Prometheus where this alerts are right so going back to our nice graphic we haven't done this part the orange part yet we haven't set up the configuration for our Prometheus rules yet so we're going to do that next so Prometheus now how to when to notify us alert manager al already knows how to notify us but basically um promethus should know when to notify US ah also just FYI this API there's a link below in the description how you can generate the API for your slack Channel but you have to be an admin in that slack channel so you might have to create a new slack Channel but basically in the description of this video there is a how to guide and how you actually generate this web hook URL link anyway so the promus rules this is what the promethus rules looks like it's titled kind promethus room as you would expect and then it's the same API version version as most of your other monitoring related resources the metadata is again very important why is the metad datas are important you might ask a good question um so this relates to what we've set up here earlier to this we select a section remember we set up the key and then Prometheus and then the vales example rule well we find here Prometheus and then example rules is the key is it rules or rule rules yes and through this label in the metadata promethus will know where the promethus rulle is that what the pr rules and that it can connect to it that's the whole point of this metadata label and of this rule selector set up okay so this is the prome room next up we have the specification section specification section has a trivia group name before when I installed this this didn't show up properly which is a different issue which I might discuss in another video just f i I'm aware of it so you don't later point out something was misconfigured I'm aware that there might be an issue here but it all works and I'm going to show you in a second it all works anyway so we have the specification we have our groups the trivy group name and within the trivy group name we have three different types of alerts the first one is if there is a pot that's unhealthy within that trivy system namespace now it's in a trivy system name space and if the pot is unknown if the health of the pot is unknown or failed so for example if there's a vulnerability report that has failed or it nobody really knows what's going on with it that's when this alert should fire so here is that it has to fire for 30 seconds until we are notified the label of it is critical here are the annotations so basically different metad data relating to that Aller and then we have here our trivy missing alert so basically if the trivy PO itself is missing then this alert has to fire now the next thing the last thing is which I mentioned earlier if there's a new critic vulnerability then we want to fire this alert now it might take some experimentation to set up those alerts there are lots and lots of useful Guides Online on how to set up how to configure these Expressions that's not the issue the main issue actually was for me to know how to set up this crd because there's hardly any you wouldn't believe it but there's really hardly any tutorial that tells you once you've installed the pr operator with alert manager set up a promethus rule crd and install that separately to your cluster people tell you this part online and lots of different tutorials but they don't tell you that this part is within this resource would you believe that anyway so that's why it took me some time to set that up maybe it's intuitive for other people I don't know for me it wasn't intuitive that this section is supposed to be in uh crd within this section within this kind promethus rule so we have set up our promethus rule so we can go ahead and install the promethus rule to the trivia monit H not to to the monitoring namespace so the promethus rule is going to be installed to the monitoring namespace now here you can see all of our vulnerability reports and something I really like with in canus is that I see these metrics anyway so let's install our Prometheus rule so it's just a file in Cube cutle Cube cuddle apply and then file uh prom Rule and then we're going to say trivia alerts and that makes it easy because you can basically have a whole directory with lots of different types of alerts and you can cluster them in crds and manage them independently right so you can have trivy alerts you can have I don't know um CPU resource usage alerts you can have all those different crds all those different FS that relate to different types of alerts yes you could manage it through groups as well you can set up one manifest and I manage it through groups but what if people shouldn't have access to all of the alerts right I don't know then they can't have access to that trivia or to that crd so let me just tell you what I mean by this if you specify all of your alerts through a single crd right and you just basically the only Cate categorization you do is through the group section so you have let's say trivy group you have resource usage group and maybe several other groups then whenever somebody needs to modify anything within this promethus rule they will need access to the entire file right so maybe in some cases you want to restrict people in whether they have access to all of the different alerts or just to a specific promethus rule with the alerts that they are responsible for that they should manage anyway just a consideration to bear in mind hope that makes sense so we're going to apply this to the monitoring namespace monitoring C CLE apply file prom rule this is the directory and this is the crd that we specified now usod might have another name might be another path right uh and then we want to install it now you could have also cloned the repository just mentioning it maybe I should have mentioned it earlier you could have the link is in the description uh I will mentioned that earlier I will clip something in anyway we're going to install this we're going to apply it to our cluster see it has been created and now is the moment that I always dread because now I have to check in Prometheus if it actually has been created that alert so we head over to Prometheus and we refresh and this might take a few seconds before Prometheus actually realiz IES that there is there prous rules now so I will keep refreshing until it shows up okay awesome as you can see the time here if you're curious how long this took um I also didn't keep refreshing all the time but basically all of those alerts here right now are in green because they're not they're inactive they are healthy they're not been triggered basically there is no unhealthy pod in the trivy system namespace there is the trivy operator is not missing and there are no new critical vulnerabilities so let's change it right this is the exciting part we're going to create a new namespace called test because this is the alert that's related to the test name space and in that namespace create namespace test we're going to set up some anest deployment basically I have here a deployment an app manifest with a deployment with an outdated react example application that has critical vulnerabilities lots of critical vulnerabilities I'm sure there's lots more wrong with this application but that is wrong with that application so it's the app manifest that we want to install so Cube cuddle apply um app Money Fest in the test name space it's great yes okay so going back to the Nam spaces we have our test namespace and we have the trivia system namespace in the trivia system name space here is just the vulnerability report for our new deployment for the nasty react example application so this is just running scanning that container image for vulnerabilities really exciting we can see life life appointing um so here's our example application or container that we've installed inside of uban cluster and in here it just completed the vulnerability scan so we can head over to our other name space to our test name space here we will see two parts because these are part is our V cassette and that we cassette is defined in the deployment and here's our deployment with the two replica set both of them are running and we've just completed our vulnerability report our vulner vulnerability report is here and it has eight critical vulnerabilities in here it's a lot and then 58 higher ones 37 medium too low nice okay so now we should see in a few seconds Prometheus this alert go red because there are new critical vulnerabilities that are more than zero critical vulnerabilities detected in our test Nam space basically our application name space so you would want to have probably for each application component maybe each namespace you want to have separate alerts so you can manage them easier because if I would just say critical across the cluster I mean that's one Matrix but maybe I care more about some critical ones and some name spaces versus critical ones critical vulnerabilities and other name spaces so let's refresh and see it's firing okay so now that this critical alert is firing it should also appear in alert manager so refresh alert manager and here we can see our firing alert so it's of type C or D the trivy new critical vulnerability a trivy found a new critical vulnerability in deployed container image obviously when you use this in production you would make it more detailed here the different labels that we've added um to this alert you can modify the labels now we can go ahead and we can add links as well to this alert we can also go ahead and silence it we can also see set the basically to this alert we could also set links to specific refiner dashboards and much much more but basically now that this is firing this alert we should have also received a notification on slack Channel because that's the point that you want to ultimately forward this alerts once they are firing to your specific destinations to those receivers as they are called by alert manager and as we can see here oops I was so excited and as you can see here okay this is just my personal slack channel so please nobody join this one and as you can see here alert manager is firing here are all of my previous ones when I tested this when I set this up as you can see there's a lot of effort going into this tutorial so you better like it okay so anyway now that we've got that over to like and share this tutorial we can go back to the interesting stuff thank you um alert manager is firing we have the alert here we can then set here the link to I haven't done that yet to grafana and also link to runbook to tell for example more Junior people or people who have never really maybe never interact with critical vulnerabilities and don't know how to fix them don't know what to do about them but received the alert that they know through those run books what actually to do about them that's the whole point of this right that you basically provide information in quick links now once an allo is firing you would also maybe set up for example a connection to pag Duty so if that's something critical going on you actually notify people when they're sleeping as well anyway so we could go ahead and we could silence the alert we could also fix the alert um for example by just deleting the deployment deployment that should fix it uh we could obviously also fix the critical vulnerabilities but this is just for demonstration purposes so I'm not going to go to Great Lengths to fix a shitty application thank you and I'm allowed to swear on this channel um anyway so delete we're going to delete this application and with it it deletes automatically the vulnerability reports because the vulnerability reports are connected by trivia to the replica set to the deployment so the trivia operator uses the kubernetes garbage collector to clean up any resources that are no longer needed otherwise you would spin up applications you would have vulnerability reports you would delete applications and the vulnerability reports and other CDs from the security scans would just lie around in your cluster clut your cluster and we don't want that to happen because clutter is costly very costly so no no we've deleted it our alert went back to green yay anyway this was um that situation now what happens if somebody is like oh I want to install some very nasty in your classroom but I don't want you to be notified about it I'm just going to go ahead and delete the part that's supposed to run the scans in this case the triy operator part so we're going to go back to our cluster we're going to go to the trvy system Nam space and we're going to just delete this pod uh because this is the kind of alert I came up with on short notice so we have to trigger it um oh that was stupid uh we need to delete the deployment not just the pod um I hope I'm entertaining you so we delete the deployment because then the operator cannot spin up the part again so if the deployment doesn't exist we would have to reinstall the hel chart resources for this for the deployment and the Pod to be back so there are no pods now in this name space if we go to pod no not pod budget pod oh um maybe even balot should be firing now let's see no let's wait 30 seconds just to be sure so now we have one alert firing saying that there's an unhealthy um unhealthy pot which is a vulnerability report that cause an error now the trivia operator um here's an expression missing no actually the container trvy operator is not present this should be firing as well now so I'm going to check that but this should be firing as well but as you can see this is correctly firing because there's an error Po in the name space and this is it for this tutorial I hope you enjoyed it as always the link to the written tutorial the link to the blog post is below in the description of this video If you enjoyed this tutorial it would mean a lot to myself if you could give it a thumbs up and subscribe to my channel for upcoming videos these videos take a long time to produce both written version as well as the video tutorial and they for free just on my YouTube channel so it would be really really great to have your support also I have a more or less Weekly devops Newsletter so if you're curious about the things that I enjoy reading and writing about on a weekly basis then subscribe to my devops newsletter the link is also Below in the description I really hope to see you in one of my next videos have an amazing day bye-bye

Channel: Anais Urlichs

Views: 3,857

Keywords: kubernetes, devops, prometheus, introduction to kubernetes, kubernetes beginner tutorial, kubernetes tutorial for beginners, kubernetes tutorial, grafana dashboard tutorial, prometheus servicemonitor, prometheus monitoring, prometheus engineer, SRE, cloud security, cloud engineer, devops engineer, cloud native, kubernetes monitoring, kubernetes monitoring grafana prometheus, servicemonitor kubernetes, Prometheus AlertManager, AlertManager, Monitoring Alerts, Alerts

Id: HwB2oWUdoT4

Length: 42min 6sec (2526 seconds)

Published: Thu Dec 14 2023