TGI Kubernetes 107: pod logging and fluent-bit

hey hey hey everybody and welcome to episode 107 of CGI cake this week we're gonna be talking about pod longing container vlogging we're gonna do a little like you know dive back end a little bit of the detail of how all this stuff works and then we're gonna explore some of the ways to get those logs that come off of the off of your containers to some aggregated place and we'll probably talk about a little bit why and that kind of thing so as usual our notes are up at TGI K dot io / notes so feel free to you know play along at home if you're interested let's see who's with us today let's see here we got it's checking it with us let Maddie is with us this beautiful Friday he's wishing George have a wonderful day and I think that's awesome - yeah both are awesome we got Rory from Scotland Rory out your ears were burning earlier we were different I was I was getting to why I had the opportunity this morning to go and see Ian Coldwater and Brad G Simon present a incredible talk at RSA comp and they'll be presenting this talk again at cube Con this year in in Amsterdam and the talk was really great it was really about like kind of different advanced persistent threats and how they work inside of kubernetes and so if you're interested in that sort of stuff definitely check it out give them a follow on Twitter and you know it was it was really great but anyway your ears were burning Rory because we were you were mentioned a few times and it was really it was really great who else we got we have Mike Merrill signing it from New Jersey you've got Martin from the devil it's good to see Martin as always we have the Matty tell me thanks for picking this topic it's actually wasn't my original idea I think was it was it you George I think was George I came up with this one and I was like you know that's a really good point I was really surprised to be it never really hit that one so I was like well yeah I got to do that because that's a really good topic so shout out to Jordans Tim Downey from Santa Monica and Moses for Hondo from Dubai and I mean Santa Monica not where did you think Oh Santa Monica is that this place up in Santa Monica actually does exist on the planet but it's somewhere else probably Rudolfo Sanchez he is signing in from he is somebody I met last week in Palo Alto he was week before a Vemma conference IV bug leaders conference that I was speaking at which was really a pretty amazing experience it was taught it was people who are leaders of virtual VMware user groups around the world all coming together to talk about you know what's coming what's changing and that sort of stuff from the VMware ecosystem and that was that was pretty neat definitely enjoyed that we got Phillip human my Google Meyer from Bonn Germany I don't know if I'm saying the bond part but I feel pretty good about the rest of the words so yeah let's dig into it here see what the news of the week is so this week we get to see mr. Joe Vito we should recognize him oh you know what let me flip my camera here to a screen screen in face there we go yeah mr. Joe Beda talking about where we're headed with and some other notable people Pat Cal singer a CEO of VMware ragam Ryo Farrell and Kate Kohlberg Oh Coco bear we're all they're all gonna get together and record and talk about a launch event that kind of gives you a view into the direction that we're going with VMware and it's stuff so definitely worth checking it out I always like to see Joe speak a little bit when he's here to actually present TGI K but if you're interested in this sort of stuff definitely go check it out there is a new podcast out and it's put together by a friend rich Burroughs and and they are looking to start you know presenting really interesting information about kubernetes and the people who build and use it I've been asked to be on one of them I'm not sure which episode of the lead but it'll definitely be a cool one so definitely check this out if you're interested in more podcasting goodness for kubernetes type things we also have Keep Calm is right around the corner at the end of March beginning of April we will be doing oh it's realize I'm gonna have to change the date on something um that we will be doing a coupon in Amsterdam and I believe it's still unplaced alone they have a noir novel coronavirus update you know if you're meeting people at conferences and any conference doesn't have to be cubic on film you know feel free to bump elbows or that sort of thing rather than shake hands wash your hands a lot do that kind of thing it's a pretty big deal out there and we want to make sure everybody stays healthy I'm actually fine to Vegas next week and I'm like oh I'm just gonna like carry a backpack full of like Purell and that's my that's my plan but keep Kaunas coming up the schedule is posted I have a talk at the main conference talking about sec humph and security profiles and that sort of stuff that should be really fun I'm also co-presenting with Ian Coldwater again at security day and that is the next thing we have the schedules for our cognitive security day serverless practitioner summit and service meskada are up so if you're interested in these things these are day zero events they actually take place the day before keep convicts place that if you're interested in understanding more about what's happening there definitely go check them out that'll be really a fun some fun talks they're kind of folks in different areas the next one I found this week is the new dismiss that there's this new application manager which brings skin ops to Google Cooper and Cooper need attention and I think this is actually pretty awesome because I've been really kind of like trying to get to get up stuff out there the link is TGI K dot io / notes it's actually up above was look where you can find it up there if you go from Holland and we have Marco signing in from bedrock good to see you Marco has been a while since you were here live I'm glad you're here and we got Kristoff from düsseldorf Germany so this application manager is like a pretty neat thing because it is actually bringing kind of a get-ups flow to deploying applications inside of your few more news cluster I was really kind of disappointed though because it looks like you have to have you can only do this on a gke cluster so it's not something I can explore easily with like a kind cluster or any other kübra nice cluster have around and that's so I was kind of surprised by that you know um but it is what it is I guess and so I just I did want to call that out I was like I thought I thought it was great to see like more folks working on good apps but I am disappointed to see that this is a closed you know some appears to be a closed source thing and if I'm wrong about that feel free to throw a link in but just from what I can tell that's what's happening here is an example of a project that is not go source it could imply apply to any kubernetes cluster just like the other one I thought this was actually a pretty fun one so this is kind of like showing how we can how we can extend kubernetes to support our own understanding of a problem or our own our own capabilities and this is actually by there's a project is called Pangolin and it's an enhanced horizontal pod autoscaler for kubernetes and they go into why they want such a thing so horizontal pod auto scaling is where you have some metric that you're measuring against and when you see that metric come along like whether that is queue depth or whether that is IO or anything else other any other metric or even just some custom metric that you actually exposed from your application what you want is to make sure that you have a behavior where we can grow the number of instances so that we can support the load right and in this way we can have a thing where like you know our minimal deployment is like maybe two to four instances of a particular application and as we see that there is going to be a need for more of these instances than we can automatically or programmatically grow the number of instances so he digs into why they dig into why they have actually developed this new this new solution one of the neat things about it is that it uses rest the requirements a pretty recent though you have to have a Cooper is 117 cluster with our back and abled and the target deployments or pods would have to have Prometheus metrics available on port 90 90 so it looks like maybe it'll be scraping those endpoints directly which is kind of interesting they have a good demonstration here so if this is something that you're interested in definitely check that out but again I kind of really appreciate how this is open and you can anybody can go look at it and play with it and see if it resonates with them or you know maybe even contribute the next article up I have a really good article by new vector and new vector actually have been doing a lot of interesting articles in this space so their blog is actually pretty decent this one is actually kind of optimized for i/o intensive containers on kubernetes now what's interesting is that by default kubernetes doesn't really spend a lot of its time trying to police the i/o between containers right we have things we have a we have quota for memory and for CPU but what about IO and so definitely definitely check this out this is a very good very good article I thought and which gets into it and they even get down into some of the primitives I actually talked about like the different CPU schedulers that are scheduled you know that can be used on the underlying node and how those things that have an effect with relation to tasks that are running inside of kubernetes Thank You abc123 I appreciate it Oh selective information like see us maybe I misunderstood okay um so yeah like the game do they they get in some really good details they have some really good thoughts about how it works and they have a really good article here put together about how to approach this problem which I definitely worth a check out so go back to our notes here we've talked about penguin talked about IO intensive potaters oh okay so when you contribute the project decisions you have lots of great things happen including the ability for CN CF to host webinars on your project and actually kind of like help get the word out so what's fascinating about this talk is you could even tell already from like kind of the how-to or the information here on the right that this is a developer advocate hell maintainer some interesting ampersands and another deal of developer advocate from different companies all getting together to talk about an open source project called helm but I might sneeze no all right oh that's for ministries okay all right come back sorry okay the neat thing about this is that they're actually doing this webinar on kind of talking about helm they're verifying home installation is talking about signing and verifying health charts detecting and fixing vulnerabilities and container images which is probably the snick stuff and then we have kubernetes security in your charts and they're also digging in to kind of the security model of helm itself and so it's definitely worth checking out it has changed a lot introducing helm three really change this quite a lot so if you're interested in understanding more about how three and how it works we're gonna be playing with it in this episode but definitely worth checking it out now as I was looking at that one I also noticed that there was this article that came out on February 26 that was an overview of fluid D which is another project that has been donated to the CN CF and fluid D and fluid bit are kind of the two different means two different solutions that fit the same need and they even have a very consistent way between them of solving this problem so fluid D and fluid pit are very similar and we'll talk about the differences here in just a minute as we get into the actual episode here but what the what the differences are and that sort of stuff so it'll be really fun I think it'll be a fun talk but if you're interested in seeing that they're reading a little bit more about what the CN CF or the or this particular contributor brand Ruben's oft has to say definitely go check that out so you could talk or it's a good article I should say alright that was all the news that's fit to print from the cloud native landscape I'm sure there's more stuff out there but I wanted to make sure we save enough time to get through the content today and it'll be a fun one so let's get this let's get this kicked off here alright let's check the notes see how's everybody doing we got more Tessa from Tehran and Eduardo Silva who is one of the authors of fluent and fluent I believe I believe in both but I was working with Eduardo to understand a little bit more about fluid bit so they are awesome friend I'm glad they are here to chitchat about this stuff and answer come on well stir we have we got salt from Finland flip it on me so Eduardo works on fluid but only great we have Bradley from life Chester UK and alright well that's everybody so let's let's get started here let's see what we got alright so first I want to go back a little bit and before we start talking about pots and log aggregation and all of that stuff let's talk a little bit about like well how it works and why it works that way right so let's flip over to our terminal so containers really kind of changed the way that we think about logging and this is actually one of those truths about containers that I think you know bothers some people and doesn't bother others but I want to make sure that we let me think about it oh thanks Rory I like that one too it's like that it's almost like the Ironman logo for the Tom's ooh stuff but what containers do for us or the way that they actually kind of inherently solve logging is that they capture is that they expose standard out and standard error from the container and they throw this stuff into Oh Stephen Oh Stephen they throw their stuff into files so let's take a look how it works kill anybody so finally a docker so here's and I would do it again all right so see the inside of our live docker that's actually where we're going to see the output of the lot of all the containers that we might have running at any time right so if we look at these these logs these are just information about the container including the output of its logs right so this JSON dot log file that is associated with that particular container this guy here is actually the the capture of the standard out and standard error from this particular container right and so if we do this see docker ps4 PS - a I'm going to go ahead and clean up my docker environment here we'll just make sure we're clear so Ducker what is it system oh yes and then the command again yes clear ok so now if we go to violet docker containers we see no containers alright let's make one docker run bash okay so here we have we have duffer we have bass running right yeah we can we can do echo this to let it go to standard out and we'll do echo this alright so that's it now we see our container that we created so that's our containers that we just created and you can see that long string is actually related docker PS when this a it's related to the long string that represents the container IDs right so a 9 for d7 is actually that kind of that long hash that represents the the actual container ID and if we go into there we can see our log file again you don't get anything out of the law I would aware of it from there let's try this again docker run nginx what I should have done was yes bar log our live docker containers [Music] we see nginx instance running docker box something is wrong with my setup here I should be able to see the log output there and I'm not seeing it ok well that's odd I think maybe I messed up my configuration somehow yeah but I should also be able to see it like loading up right like it's a demon that's actually catching that stuff you know what I have a different idea for an image so you jump out of this but I can't get so pop it up put it in tab and we'll do Ducker rm- up this was actually a pretty cool thing I thought I found here grab - I are imaged [Music] where's my image name cat values bonsai block generator tagged oh three two okay so docker or uh this is a log generator we're gonna use later in the show so that's right let's go ahead and try it now because what this does is basically just generates logs for almost exactly this reason so now we go to docker PS I'm gonna do doctor logs we can see the output and this genom generate it's just generating logs constantly to the output the standard i/o centered error of the container so now let's see if we see that will be expected to see before it's this will be the interesting part lib docker containers okay there we go alright so it is wired up right I'm just doing something wrong if you send data out to the oh yeah sorry my bad so dr. PS and I've started this container image called bonsai clouds long log generator and what it's doing is it's literally just kicking out the the a steady stream of logs and to standard out and standard error so if we were to do docker logs - f busy the mirror we would see what the output of that log looks like this is just a dumb little process that is randomly generating logs kicking about to the output we're gonna play with this more later but effectively that is how the that is how containers handle locks so if you're running if you're used to running you know some other application that handles logs differently like maybe a java application or or one of the other applications that are or you know pretty much any other application that you might write if you're if your goal was instead - you know persist your logs to syslog or persist your lots to you know a file sitting on disk or those sorts of things the world is kind of changed when you start using containers you have to think about making you standard out and standard error instead of making a use of those those older mechanisms and the reason that's important is because it wires up to the way that containers block standardly it also gives us the ability then to think about handling the logging problem like how do we actually aggregate those logs how to get them off the machine how do we present them to our developers and that sort of stuff kind of an easier way or lower a less friction in a way that results in less friction I'm gonna go ahead and stop stop this container because I don't want to like fill up my um fill up my disk here yes okay cool so that is our test container it shows how containers work and pods work much the same way right like if I was gonna look for the log for a pod it would work the same way so let's go ahead and try that out so I've created this I created this environment just gonna play with it and before we actually get too far here I want to talk about like what is configured here so this is the configure of the cluster I'm using a kind cluster like usual you've all seen me do this a million times in this cluster I'm going to create one control plane node I'm gonna create three worker nodes and I'm going to use and I'm going to deploy an elk stack to this cluster and I'm also going to do some other interesting things before I get too far with it though I want to make sure that we understand oops cooing it is good I want to I want to make sure we understand what's happening here kind of in the background so one of the things I'm actually doing to this particular cluster is I'm also configuring it to handle audit logs right I want to make sure that the audit logs are coming out of the control clean node and being persisted to disk in the form of a file called var log Cuba API server auto log my discount a typo so let's fix this real quick let me I'll see that type of I do alright alright so my configuration I've actually I've got a few kind of interesting things happening here right I've got a inside of kind when I'm defining that node I'm describing an extra mount and that extra mount means that inside the container I'm going to put that if I want to mount that file at Etsy kubernetes policies at bada TMO and on the host my laptop I'm actually gonna put that file in temp just so it's easy to find and then I'm actually going to grab that file from the one from the EDC kubernetes policies directory and I'm going to load it into my API server as part of a policy file and have my API server output the API OP API server audit log to disk inside of the control plane node and I've actually also set audit log tree format JSON and we're gonna get to why I'm doing this but really what I'm doing but my goal here is to make sure that I can make I can make certain that the log the audit log is being kicked out in a way that I can then go and scrape it with flow indeed they're not mutually exclusive it's true a little bit and fill in D are not mutually exclusive that's a very good point is this better for containers to the fluid D we usually up second log stash I wouldn't say that what I mean they they each have their strengths and we'll talk about it we'll talk a little bit about what they are here in just a minute all right ok so let's get this let's get this kind cluster created create create cluster totally I could type config equals time we're gonna show logging for pause we're not there yet so we're gonna we're gonna come back to them but on the film admit documentation page which is here we can dig into about and talk about a little bit what it is right and here I think we actually do a fairly recent a fairly reasonable job of describing kind of the difference between the two they both actually are functionally very similar they're very flexible in the way that you can configure them they lay out a street and processing mechanism very similar right it is a log collector processor and aggregator actually fluid D is an aggregator it doesn't it doesn't have that feature inside of foot bit but the way that they lay these things out and we'll talk about them a little bit when we get into configuration is it's the same right you have inputs you have filters you have ways to manipulate the data that came from the input and then you have outputs ways to get that data out toward your your given destination now some of the stuff that's changing alright so fluid is written in C and in Ruby it takes quite a bit more memory it is high performance but but it does take you know depending on like the amount of data that you're moving in turn it in a second it can't really count to feel the burn so that D is built as a ruby gem and it requires a certain number of other gems one of the big secret superpowers of fluid D is that it has more than 100 or more than 650 plugins which means that very likely whatever that stream of data whatever the input is where the input is like related to logs coming off of a system D unit or whether that input is related to logs that need to be parsed as part of the kubernetes logging right or you know pretty much any other obscure thing you can think of right the the plugins that are here in fluid D give you the ability to really customize the different ways that the input is handled that the processing is handled and that the egress handle like how do you send it out right where as fluid bit has 35 plugins available it's not completely compatible all those plugins but it is also it would take significantly less memory it's written from the ground up in C and it's meant to be very very much more performant than the other right so the performance of Luud bit r is like pretty extreme but you're also constrained in what you can do with it and so as long as you're it's long as a little bit satisfies the requirements for what you're trying to achieve like in my case I'm just trying to achieve making sure that I get all the logs from all the containers inside of cabrini this I want to do some manipulation of that by making sure that I have enough metadata when those logs show up in my aggregation endpoint but I understand a little bit more contextually what's happened with those logs and then I don't want to kick them out too in my example here we're gonna use an elf stack to kick them up right so we're gonna output two elasticsearch so it's a little bit checks all those boxes for me and that's actually what I'm we're kind of exploring it they're both Apache 2 license they both have been around for a while now there's pretty major release coming up a bit Eduardo is working on but I think from my perspective that's kind of like the high level y one over the other kind of thing but yeah neat stuff so let's see how our cluster doing clusters up so docker or actually keep it all I'm gonna look for that image again what was it called lot generator history generator so this is our image so we're gonna go ahead and deploy that on our Cooper new cluster run log Jim anybody catch my typos yeah I'm always like going y'all are so patient with me alright so cute get all get pause so now I see my mic my pot is running it's a pot of only one container it is the lajjun container right and if I do cute catalogs we can see the logs from that container running yeah that's true in 2011 definitely and if we jump in here to our let's see cube kettle get pause that's a wide we are running on kind worker too so if we jump into our kind worker - well that's weird there's no varlet docker so I wonder how that works right but then I remember you know what in kind we use container D not docker so we thought to have some way of actually handling that does container T handle lobs in a similar way to docker let's take a look at that real quick so but if CR kettle PS I could see the containers that are running here there's my logged in container right if I do see Eric at all inspect the container the logged in container we can see the cgroups path we can see the output we can see the stuff the service account being mounted in resolved a compost name there is the termination log Etsy hosts [Music] there's our image and there's a log pass all right so it's going to send the output of this pod to this path on the underlying disk of our log pots so let's go look over there and see if we see our output right I mean also we can do see reckon all log stash F launched in oh no I'm sorry so we have our image named Jeff so we can see our logs just output just like we can with with dr. right just by using CI kettle instead of using docker because we don't have Ducker installed we only have container key but we saw the output of our you saw our path to the logs which is VAR log pods if we do if we cat that we can see that that is also persisting the output of logs from the container or from the pod from the container running inside of the pod into our underlying file system right it's running into var log pods we also see it out put into our log containers and if we do an LS minus L here we can see these are just symbols this is the symlink I believe is being done by my P cubed or might be done by container T itself but the actual containers you mean you had multiple containers per pod you would see the output of each containers up put here in this output right and you can see it and it's just linked into the actual via the VAR log pods path so this is basically where we store these logs on the underlying disk and it raises an important question of how long do these pods how long do these logs stay around right so if I do steer kid oh yes stee-rike it'll stop so now I just changed that right so now the logjam is a different ID so the question is how long does the old one stay round we still see both of them here we see how right we still we still see our a17 6e and we see our b88 ODB now what's interesting about this I wanted to show you this other cool thing if we do cubic it'll get pods you can see it's running we had it we had one restart one of the cool features that I really like about cube kettle is that you can do logs - P for previous and then point to that pod and it will show you the previous lock so if you have a pod that is actually failing you can actually use cube petal to see what the previous logs are so we'll keep those previous logs around for a bit but let's go ahead and do that same command that we did but let's do it again so see our kettle PS let's go ahead and stop this guy again er get all stop sure I can Oh PS change it again let's do one more time stop yes the VAR law containers and we can see that we only have two around right so what we just noticed here was that the actual logs it only keeps the previous one of the current one around it doesn't keep the ones before that around but what if they what if there was information in that log that was critical to me right like I actually noticed I didn't notice that it was failing over and over again until this recently but I want to be able to go back and see what the logs I won't understand why I feel earlier to be able to really dig into what's happening so let's take a look at our chat see how everybody's doing I know I'm talking a lot and I've covered a lot of deep detail but I hope it's useful to folks we got Eduardo so that the original author was is a strong C C++ ruby developer actually before fluent he created oh wow created message pack a binary serialization data format yeah totally he wrote it in Ruby because it was the initial PLC and but it did really well it sure did a lot of people are still using some of these and people started creating plugins for we've flown to you which is actually pretty killer right like I mean that's kind of the Bill of the really neat thing about the Ruby piece is that people feel comfortable contributing to it right getting those things out so there's not going to be a bar log docker in this one was it faux because there's no docker there's just a container D so in this case is actually it's gonna be in inside of a container DS output we got everybody being pretty cool so let's keep going alright so like I said these container locks don't stay around forever and so the important thing is that if you're gonna actually use these container logs to debug things that have happened over time then you're gonna have to make sure that you get these container locks off the system and that brings us around to how are we gonna solve that problem all right let's kick back over here we talked about logging for pods oh there's much it one more problem that I wanted to highlight and this is actually I mean we kind of did right the problem is that these pods are ephemeral they change a lot and when they change a lot that means that these logs are going to get deleted from the underlying host there is actually you know for the actual containers there are some there are some tweaks on the cubelet to manage the amount of logs that will keep around and that sort of stuff but but typically what you really want to make sure is that you actually get those Lots off the hit off the hosts we talked about where pod locks go we talked about how long they stay around we talked about lock you catalogs - P which is one of my favorite logs commands and then how do we get them off and choice of aggregator there's a ton of tools out there for this I really am a fan of fluid of the fluid stuff because it actually you know for a number of reasons though addy was contributing to see yeah there's a lot of really good stuff so cool so now before we get into the fluid bit docks I want to actually go ahead and deploy some stuff and I'm going to show you how I'm gonna go about deploying that stuff so let's go play with that so first things first I don't know if you all know this but there is a add-ons to kubernetes and let's go ahead and explore some of those real quick so before we go before we go there let's go check this out so if we go to the kubernetes hoovered it is cluster add-ons here we are on the kubernetes repo and inside a few hundred underneath Coover descubrió is cluster there is a directory called a das and add-ons are kind of the things that you have probably played with before there's lots of good stuff in here the dashboard is hosted here the device plugin for NVIDIA GPU the horizontal autoscaler for dns the dns plugin core DNS is actually goes to here auerbach are back in so see here you know some a bunch of the different stuff is actually hosted in this case but the one I really wanted to show was this one here the fluent D elasticsearch add-on and as you can tell it's like three months some changes two months ago those sorts of things it's actually continuously updated but what this elasticsearch one does and it has lots of big bright warnings about don't use this in production but what it does it actually is a maintained way of deploying an EF k stack where we where we deploy elastic fluid D and Gabbana and and then we can actually kind of explore the way lots work inside the cluster now the other thing is the is because as part of the kubernetes bundle it's part of that package that it's actually provided by a kubernetes release we can download this content by using this pretty neat trick for all monticello deal for cooper need for download ksi oh and then the version 17 0 and then the name of the file in this case it's going to be communities tar.gz creative I know now what this does is it pulls a tar file of those things including things like the add-ons right and so now if we knew tar - XV is X PDF tarball we get a bunch of inference we get a bunch of content including you know kind of all the add-ons and everything else that are related to the main upstream kind of piece mister hasn't been moved out yet so if we go into Cooper Diaz clusters add-ons there is all the manifests especially with that they're very elasticsearch now this setup will actually deploy fluid D it will float it'll deploy elastic search and it'll deploy kibana I'm gonna do a little trick here - IR image start out yeah mo dollar sign three bear with me for a second I'm gonna get a little happy here don't want that line so these are our image names and what the reason I'm doing this is because I want to be able to cache those images on my notes before I try and deploy them so this is actually kind of a kind trick so let's play with that so as you can image is xr- I give it the give it the object that we're going to change then we'll do kind load docker image Oh actually first we gotta do this let's pull these images first real quick somebody's are huge like the elasticsearch image is ridiculous which is why I'm pulling it locally here so that we don't have to wait for each individual one of my nodes to pull this image because we would be here for way longer than then I want to think about but because we're able to pre pull these things and then I'll be able to use kinds load image capability to push that image into the cache on each of my on each of my nodes we won't have to wait for that image pull to happen and that'll really cut down on the time it takes to get these things but up this is actually a really cool trick that kind of lets us do so I'm going to go ahead and finish pulling these and then we'll push them in there at our cabana downloading all right so now we got all of our images and their local on my machine if I do docker images which is I can see them here now I want to get them on all of my nose right so now if I do kind get notes right so cat images and then we'll do X arcs - I will give it the argument name right that'll do kind load docker image sets it makes it a bunch of assumptions here when that simple command right it's assuming that I want that file on all of my nodes and it goes in and it takes a look at the container T image list and sees if that image is present and if it isn't it copies it in weekend it can push to Kafka it can push to elasticsearch to get pushed to a bunch of different things Marko was good to see you I'm glad you were able to join you join us for as long as you could please enjoy the awesome weekend thank you very much for your time so now we're just pushing our images in and now when this completes we're gonna have all those images hosted on all of our nodes and then when we go ahead and deploy this stuff we will have all of that stuff you know kind of in in the cluster is that any better can you all hear me any better now but has made some changes I hope that's better okay good yeah all right cool sorry about that having some sound Dolph but anyway let's keep going so we've got that deployed let's play this let's take a look and see what we got pods - a and what we've got so far is we've got a elasticsearch elasticsearch logging daemon set running in the cube system namespace and we have the first one is up we're looking at bringing up a second one we have the fluent deep daemon set deployed we have cabana running but it's not completely running yet probably because it's waiting for elasticsearch to work it's to point all this into cube system waiting for things to come up here initializing man running system that's beginning so we got everything running cube kettle get pause shampoo system waiting for waiting for this one to come on happy q cuddle session open they're not happy all right so let's do cube kiddo proxy this is just how they've set their stuff up go kill feed rip proxy i parent li this one left this one around from before it can't possibly be proxy dating so let's do that again kick up our proxy there we go so now we're on one two seven zero zero eight thousand one it's not I want to actually connect to that service that is defining by the Cabana service so let's do QK I'll get SVC dash n cube system and we see our two important services we have our elasticsearch logging service we have our cabana service first let's actually see if the elasticsearch logging service is working correctly so we'll do cube kettle get SVC - and who system elasticsearch grip self mo the reason I'm grabbing this is because I'm going to use aq petal command to interact with that service using proxy so it's a cute kettle get raw proxy cat indices we can see that there is an index for cabana but I was kind of hoping to see an index for the locks - ah there we go all right so now what we're seeing is kind of what I was hoping to see earlier so these indexes are actually indexes that are hosted inside of elasticsearch and we're seeing oh yeah that's a really good question from that and I will answer it but not yet so so he said so now we see our two indexes we see a lot stash index and we see a Cabana index the lock stash one is actually being propagated by our fluids configuration up to elasticsearch and we can see it there so now we know that that's working go ahead and connect you the Cabana UI to kind of like play play around and look at the log sir so let's do this let's do the same command again get services logging but we're going to look for our cabana one that's dead right Gabanna logging that's our self link and symbol do xdg open HTTP is one call the 8001 that is our cue proxy port we're going to kick this on here I'm gonna proxy and what this is gonna do is it's actually going to proxy us it's going to open a browser because that's what XD g 1 x DG open that's going to do it's like a magic command that will open the file or the thing that you hand it because this is a URL it's gonna open it with my native browser let's try that out and then we'll kick over to that hey we got gimana alright and so what we're doing here is we're actually just connecting to the proxy of the actual application so this is the application has defined a port that has defined a port as part of its service keep/kill the proxy command can determine okay well there's only the one port exposed by this application so we will connect you to that application support and here we are looking at gabbana so now we're going to do the next part one part and we're going to look at the create index pattern page and we're going to see there's a couple of different indexes here there's one by date called blocks - and there's also cabana one but we're gonna go ahead and grab the logs - one will do next step our time filter will be time stamp and this is all of what's built into all the fields built into the the but I call it the facility configuration that is hosted on as part of the kubernetes upstream content right so we have fields like docker container ID doctor container keyword query this host whose labels all kinds of information all here we can actually see quite a lot of fields represented here and now if we go back to our inspects we can see the logs that are present on our other cluster right so this is all aggregated logs up from the cluster and so just as we saw before you know now we are actually aggregating these logs to an elastic search engine if we were to rotate that pot again a few times we wouldn't lose the stream of logs oh my gosh I hate audio someone is it still bad it's just high pitches mainly okay I could try it I could try to reset it again but let me know all right well may keep going so these logs are now off the node right these nodes are this this well sort of so these locks are now in elastic search the elastic search is actually hosted on the node so it's not completely answer and in fact it raises a very good point wherein if you actually did host elastic search on the cluster and you lose the cluster then you also lose the logs and so that's going to be hard to troubleshoot which is a really good argument for actually either hosting an elastic search on some centralized cluster or on a number of out clusters or using some other service to actually post that blog content don't post it locally right like that's the point is if if you're if you're if the tools you use to debug cluster are all hosted on that cluster and the cluster is misbehaving oh yeah you have discovered one of the many foot goes right make sure all right so cool so like now like I said now our lives are disconnected from the underlying host so we can actually see them and that's great so that is fluid bit but let's before we go too far let's go and take a look at how one bit is configured and then we'll jump over to Flint D or Flint let's take a look at it fluid let's take a look at how fluid D is configured and then won't think and then we'll take a look at how fluid bit will be deployed I'm gonna pull it D out put some beat MIT in so kick over here and then we'll do opt-ins like so this is octants octave is an open source project we work on here at VMware and it gives us kind of a visual tool for looking at how things are working I'm going to jump into the cube system namespace because that's where our fluid D configuration is hosted we can see we have a few things we have a daemon set which is flowing D and that's what I wanted to look at was like how that's configured right so if we look at how that one's configured we see a couple of different things we have bar log exposed up we have Varla Varla Farland docker containers for zooming XY's but that's interesting right because it's actually mounting a directory that doesn't exist on the underlying host we were just noticing that earlier if we look at the notes here there is no varlyn docker containers but there is a VAR log containers let's take a look at the configuration so then we have our fluent DES configuration and it's Health's posted any config map so let's go look at that config map and see what that looks like this is probably not the best way to read this so let's just jump back into our terminal press you know that's how we roll okay it's a cute kid I'll get config man - a new system all right good so that will actually parse better all right so here's our configuration for fluid bit and like I said before the way that bullet bit had fluid I keep saying foot a bit right now we're looking at fluid D in the future we're gonna look at fluid bit we're not to the foot bit part yet but much of this will look very similar when we get to the fluid bit part in this case we see our input off and it describes it's pretty well annotated file here for describing exactly how this is configured it has examples of what it's trying to parse and information about what it's actually looking for let's get down to the configuration so we've configured a source right and we're looking at VAR log containers start out log and that is interesting because it is exactly what we were looking at before right so there's a type tail which means it's going to tail all of the files that we described by this match and so on the underlying host on each node inside of our log containers I have all of those container outputs right and we can see the logs there and we can tag these things and we can have a position file so if our elastic so if our fluid process comes to an end or gets restarted or runs out of memory or what-have-you it can actually come back and continue on where it's aren't where it left off and I've had problems with that one before with fluid d where the position file gets corrupted and it had a hard time like picking it up again but you know that's an interesting thing we have a pattern that we're expecting the log output to look like we have exceptions that we're looking for we have and this is the filter the ability for us to manipulate that data so if we see if we see lot lines that don't look quite the right quite the right way we can actually just for them off on one single line we have filters where we're going to actually try and understand if it's a multi-line piece or a multi-line log message and if it is we're gonna try and again concatenate it into a single line and then this one is a really cool filter this filter annex works on fluid or fluid this filter actually gives us the kubernetes metadata as part of each log line so when it's aggregated up into elasticsearch we have that context right so before when we were looking at our Cabana output right look at all this metadata that we have for this just this one message right boom taken so look at all the metadata we have for this one message the fact that it is we have the container image name the image ID the container name the container host the labels that are associated with it the master URL that is in the environment the namespace IDs the namespace name the pot ID all of this information is actually added to the log line right when it pulls that out of the file it's actually adding that filter is actually adding all of the security to this metadata to the content before kicking it up to elasticsearch and that is killer because when you're debugging right you need that context and that's actually I think one of the coolest features of this whole fluent thing is that you can actually manipulate this data in a way that makes it more contextually relevant to you when it hits the log aggregation endpoint that is a killer feature we have things that are going to try and fix the JSON fields there's a bunch of other material here that is just manipulating the content looking for a Prometheus exporter looking for inputs the input here is pulling from Prometheus but a bunch of other content here that is actually pretty cool stuff right and there's even got a buffer and so it's actually so this is the part where we're sending the output this section here is output we're gonna send it to elasticsearch logging now this may not look super consistent with the way that we name things but if you think about it the service that was exposed that we interacted earlier with de-seeding indices right the service that we that we saw exposed was elasticsearch logging and because in this case fluid bit or fluid D is deployed within the same namespace it can route that traffic just to the service name because they're both in the same namespace if I put this in a different name space I would have to use a bit more context I would have hostname actually have to be the IP address of that service or possibly the more complete name elasticsearch - logging dot q - system dot SCC would do it what ports you listen on it's logstash format actually this is how it determines its name - right so that's that's our handle again we're also grabbing the system D of a system cough and we're grabbing the system input bunch other examples in here trying to pull logs from NCD log which is actually pretty interesting or I like a CD but I'm not sure if that's actually present we'd have to look at that but it's actually also trying to pull a log from the far log cubed log which I don't think it's actually even there let's take a look at that worker bar log you see yes cubed log so there's our there's our lot file for it but I don't think it's actually exposed in the way that they think it's exposed so we can't even really see it also we have this idea of journal that's cool alright so we see maybe this is actually worth getting it's probably giving the journal lock from the cubelet from the from the configuration of journal DS right so as part of the input here we are probably looking for system D yeah there it is so we're looking logs from a docker service which we will not see because there is no locker service we're looking for logs from the container front looking for logs from this systemd unit for Qibla service so we are actually aggregating these logs up and that's pretty great if we had no problem detector deployed we'd be grabbing that we're grabbing logs from the colonel we're getting a bunch of really good locks here and so again if we jump into our discover data let's see if we can find logs from a cubit mm does appear so and we'll talk about why that is but the reason I think that that's happening right we're not getting actually we're not getting the the content from the tag colonel we're not seeing that content [Music] zero documents zero documents so what's happening here is we're not getting that system heating up and I figured out why this is but it's kind of a wild thing and troubleshooting wise it's a fun one but we're going to get to that when we get to the float beep there's a fluid bit piece so let's go play with that how are y'all doing they're feeling good you having a good time I hope this has been pretty interesting so far let's go ahead and play with this stuff so I'm going to go ahead and uninstall the fluid bit stuff right so if I go into my Cooper Nita stuff for my manifests were cluster add-ons elasticsearch fluid elicenser and I'm gonna cube kettle delete stash F flow into D okay so now if I do cube kettle get pause session system I don't see any more fluid D install they're all they all have been deleted all right so let's let's go ahead and play with the fluid bit piece and then when we do that I want to dig it a little bit more into how awful it be and that kind of stuff works so let's play with that so here is our documentation for fluids and fluid bit and we're gonna go ahead and we're gonna play with like the installation method for how to do this this is one way to go about it but they also have a helmet art here is a way if you wanted to deploy this thing manually from a set of manifest you can see them here and I've and I highlighted this earlier playing with extensions v1 beta one has been expired and so if you're gonna deploy to a cluster newer cluster you have to make sure use apps v1 API version instead of the 1/16 stuff you've all seen me kind of hit that one a few times consume all container unlocks for the running node TL input would not depend more than five Meg into the engine until they're flushed to the elasticsearch back end this aims to provide a workaround for back pressure scenarios some details about how it works pretty cool cuneus filter will enrich the lawns with that kubernetes metadata that's what we talked about the default back in the configuration is the elastic search by the elastic search output now what I noticed when I was looking at this configure was looking at this stuff is that their air content is also available in the Helen chart but before we go there let's just walk through the getting started stuff and just better phrase like how this stuff works now a lot of this it's going to be very similar whether it's in float bit or float D right a lot of these things are very similar but this is actually kind of how its laid out and I want to talk through this real quick so as I said before you have you think about it like a stream processing you have your inputs we are getting the data and the one that we've seen so far was about getting data from lob files so we use the tail we use the tail mechanism to get that input and we have a parser how do we convert this unstructured data into structured data that we can kick it up into elastic search and make it a usable logline and some of the examples from the fluid bits died we were actually using a parser to you know make sure that if it was a multi-line log file we would try and catch it and turn it into a single line log file we were catching things like making sure that when we pull these things apart into fields we understand which field is the timestamp and which one is the severity and which one is informational and that sort of stuff and we have our filter which gives us the ability to alter that data this is like the really killer one is actually the to the Cooper Neda's option right like the ability to actually concatenate or to integrate all of the metadata for each of those log lines as you're kicking it up to elasticsearch with metadata about about that lot line like what container is from you know all that good stuff we have a buffer which is really killer like this is a really good idea and really definitely worth it because if you think about it like you know what if you had a couple misbehaving apps that were just spewing logs to the disk right now obviously we need some way of actually handling a buffer so for back Fisher scenarios and so we want to make sure that this buffer really helps us out it means that if you know if we are producing logs faster than elasticsearch can ingest them that's not going to be a situation where we over run elasticsearch or we lose content hopefully but it gives us the ability to manage it and then routing data ingested by an input surface is tagged that means a tag is assigned and this is used to determine where the data should be routed based on a match rule so right now we only have the one routing rule we're kicking it all to elasticsearch but what if we had a different problem and we'll talk about it here in just a minute and then the output rule the output basically defines the destination so this is the general layout of how fluid bit and fluid D work these are not different particularly in this way right like these this idea of multiple outputs and handling routing and and buffering and all the stuff we just saw in fluent D is actually very similar if not the same and they're a bunch of and they're a bunch of built-in options to handle these things right so if you click on any of the inputs you can see what some of the input plugins are right so for example we can pull from we pull input from collective we can listen to collect D CPU disk information we can exec and get and get content we listen to the kernel message lock buffer and then the tail one is actually the one that we're really that we found used for here and there's also a system D one which is actually pretty cool being able to interact with systemd leveraging journal e to pull lots and if you have an older learning system or one that doesn't use system D for those Lots you could also pull from syslog there's also a serial interface I mean like really it's a very it's I mean just this small list of plugins accounts for quite a huge capability of getting data from different sources into into this tool that allows you to handle that stream process and get it routed out right and the same thing for the parsers there's different parsers we have collect D you know different ways of actually parsing the content and again like you know just a ways of understanding what the fields will be and like in what order they come right so we have system D we have syslog just different ways of understanding how the how that data will map we have decoders parser name is dr. like and this is kind of the expected way that doctors gonna lay its data out then we have our filter plugins the one we talked about already was kubernetes but there's also quite a lot of things you can do with like different parsers and different Louis for the filter side of things including our crew Brina guests including grep I mean definitely pretty powerful combination of things and then the output plugins do you want to send it to the is your log analytics do it query to to flow counter to elasticsearch to a file to data blog like there's a lot of really great plugins here these are all the ones that are just built into fluid bid now as we said before if I were to pop over to the fluid d set of things the number of plugins over there is even crazier so fluid D plugins list of plugins by category right so so these are the plugins for for fluid D I'm not gonna spend any more time on this page but I wanted to but I wanted you know to kind of show that like there's a ton of really a ton of plugins here that are hosted and managed by different off as it relates to flu and D so if your case if the case you have is getting these logs processed in a more unique way where you know the tools that fluid bit exposed which are very powerful don't quite solve your need or maybe you're just taking the easier route and you see you know what somebody's already solved my problem there's already somebody who has an Aurora slow query log I just want to use that plugin to solve my problem right so it D might be a better solution for you and then fluid bit.if but if what you're trying to do is just handle like you know mass amounts of stream data into a kubernetes cluster and get it out into a place where you can actually manage it and do and do some metadata implementation like you know the kubernetes stuff then you know fluid bit might be more your mash and again me there's a performance piece to take into mind right so as you deploy fluid bit or flow into your flood bit you know evaluate the performance of those applications and others sure and understand like what you're looking at there to ensure that you're getting the best of what of what you can write like make sure that it's a reasonable implementation for what you're trying to achieve all right back to the ducts all really cool stuff what have we talked about we talked about that we did that part we also talked about what's a flood bit we are gonna get it as a helmet art piece now and play with that so all right so there is some repo lists there is a healthy bow from elastic where there is a help or there's a help chart stored for for fluent bit and I've gone ahead and added that to my configuration currently I didn't click on it hard enough ctrl-click um oh yeah that's not super helpful anyway they have contracts for all the stuff that we've got deployed here they would have held a chart for elastic surge they've got a helmet art for different versions of elastic search all kinds of interesting stuff I'll beat all the things from a lot from from there but they also have fluid bit as part of their as part of their chart and so if I move into the flat bit chart I can see the values here and also I read me let's go check out the little bit chart online so this is actually I'm using the hub cube apps calm way to look at this chart Oh actually did I pull it from I'm sorry yeah I'm actually pulling this truck from Kourou his charts not from are from hub Dutch charts not from elastic I was looking at elastic for something else so if you go to cube apps comm hub cube apps calm great this is a way to to see like a bunch of upstream charts that are that are hosted and these charts are generally maintained and you can see like who the maintainers of this chart are right so we have Kate Fox we have ed cipher who is with us today Eduardo Silva is the maintainer of this chart we have Hector and this is related to the fluid bit dot IO right now this chart is deploying I think 1.3 dot seven rather than 1.3 nine but we're gonna play with a little bit more now I'm gonna use the helm chart to deploy flat bit because it actually allows me to take a few runs and how this will be configured so let's go ahead and explore how that's gonna work so the way that I did this and the way that I typically work with sound charts is I pull the chart down the values file and it basically shows me what's here right so this particular helm shark gives us the ability to understand and configure an elastic search a forward back end to like a web hook an elastic search back-end an HTTP back-end a spunk back-end and the staff driver back-end and if you have the if you have the answers to these questions then you can go ahead and configure it however you want you can enable through the configuration here different parsers there are a bunch of general annotations like do you want to actually capture the audit logs this is actually why I added the auto logged on my cluster so that we could say yeah let's do that whether we want to capture the pot annotations or the labels check if those are on by default or not and then they it actually also configures some really interesting values like the ability to override things from a from inside the system right so being able to override particular features like you know that's this idea of Andy will exclude right so if I annotate a pod with fluid bit io / exclude true then that means that the logs from that pod will not show up in my log aggregator mid-nineteen anyway that might be super useful if we think about how much we pay for a log relation like Splunk or some of these other companies it gives us the ability to kind of like maybe manipulate that kind of directly and you can enable that or not enable that right so if you feel like it's gonna be used for for you know it's not going to be used for good and you know don't enable it we have a cube URL bunch of these things are abutting faults cube tags got a bunch of other configurations like what image bit tag is going to be used what the repository is if you're hosting it locally inside of your own image registry what the image post secret is if you care about that one and let me have our input tale parser our path you know what we what log files we want to grab those sorts of things here these are in I believe that they can be specified multiple times right and you have our back create this is really cool I haven't seen this happened very often but in this case the chart if you enable this will actually deploy a pop security policy that is locked down to just those things that the particular pods need which is actually pretty cool but if you want to get really flexible with it you can actually provide it a complete Rock and fig a bunch of other things in between an able metrics enable track offsets and that's the help chart so that's quite a lot of capability in the Alma chart let's go ahead and deploy this thing and play with it now so I was playing with this before and in the interest of time I'm gonna play with this I'm gonna do this some more but I'm gonna jump in here and kind of like forward the clock a little bit so what I've got is I'm using at the helm template command which is part of helm and l3 I'm giving the name of the deployment logging this is a humphrey thing before you had to do - - name i'm telling it to use the current directory as my as the chart directory and then i'm over writing some of these variables right I'm overriding the set input system D enabled the true because I do want to actually collect from system D the output of those things that we were watching before I'm telling you the multiple things I wanted to watch i only have container d inside my host I don't have Ducker so I want to watch the container D service and I also want to watch the cubelet service I'm telling it the input tale parser is the CRI parser instead of the docker parser because again I'm using container D not docker I'm telling you to set them back into the packing type to elasticsearch so little forward my things up to an elasticsearch end point and again like here in the output just like we saw in the configuration or I'm setting the back end to elasticsearch - logging dot cube system dot service dot cluster dot local and this is like an fqdn that can be used in any pod anywhere inside of this particular crew videos clustered to address the elasticsearch logging deployment that I have I told you go ahead and on it enable true and I've given it a different tag the 1:39 tag which is the more recent tag of fluid bit so when we kick all that off we can actually go ahead and take a look at the manifests that are gonna be created so we can see the release is called logging we can see the name of the service account alright this is the app foot bit app it's the name logging fluid bit we can see the secret that's being created we can see the config tile for this so here is our fluid be are our flood bit configuration it looks very similar to the Flint bit for the fluid configuration that we saw before in which we have our inputs we're catching inputs from tail we're watching VAR locked container startup log be told use the parser the CRI parser we're tagging it with cubed star we have some configuration here for refresh interval and for the memory buffer limit and to skip alum lines we have a different input for system D and we're in which we're watching for the container D service and the cube of service we're watching the input for tale again right this one is a system D piece but we're gonna use journal kettle data right to that log and this one is actually using the tail again to look for VAR log Q BPI server audit log we have a parser of docker here we'll see how that works out not sure if that will work out at all but we're gonna take a look at it really the key here the tag is audit and then key is Cooper dues audit and then we have our filter right so again we're using a filter of kubernetes we're matching on the cube the cube output and that match comes from the input right where we're actually setting the tag from that input we're matching it or we're tagging all this data is cube and here down on the filter layer we're doing a match on cube so anything coming from violet containers is actually match that here's actually you know we're going to authenticate to kubernetes itself to get that metadata we're merging the login we're grabbing the token from a service account and all that good stuff and then the output configuration right we're gonna output to elasticsearch are matching everything everything about everything that fluid D has at this point we want to kick it all up we're going to send that data to elasticsearch logging cube system service cluster local on port 9200 logstash prefix coober to this cluster that'll help us because we can actually see that's like if we see this new index right so this will actually give us a different index before we saw the log stash button now we're gonna see the coumarine cluster one here's the fluid bit configuration we're telling it like you know the service the input the filter the output and those are all broken up across those other files here's our parsers right we're gonna see the parsers that we have and here's our cluster all that's being defined we're allowing get pods this is actually the cluster all that fluid bit is going to use to pull that metadata down from kubernetes as it relates to these particular containers and our row binding there's a daemon set for fluid bit and now here's something that's been bothering me it didn't really do before but if you think about it this particular line is actually innocuous that doesn't do us anything because we're not using varlan docker container we're getting our data from Varla continued from our log containers in fact this mouse seems superfluous I actually just noticed this recently when I was playing with this log but if we go up here mainly where are we actually getting our logs our tale line and our input is Varla containers so we're not looking at Varla Varla of docker containers know we're in here in this entire file except for the mouth point are we actually addressing that mouth point interesting stuff right so what that means is that we're actually gathering content from this thing that we don't need all right so we're actually mounting a host path that we don't actually need here and so we should probably get rid of that but that was one thing I noticed about when I was looking through the manifest is that it's actually mounting something that we don't need and then he also has a test framework that allows us to understand whether we're actually seeing these things work so we've got almost configured let's go ahead and deploy a cute kettle up fly - Oh get along big NS log in spell it correctly all right so now I just applied that whole set of manifests what do we see up here at the that's enough pot level so if you don't get pods - and logging we're seeing the testing fire up we're seeing our fluid bit deploying something's not right right right here you cannot get it on SM logging the wine we only deployed this to workers bum-bum-bum that's interesting so the Toleration doesn't match every host it only matches those that are not masters so presumably if we go back to our command here Wow we go down to our diamond set that David said has no tolerance which means that it won't to apply to the master okay let's go back over here look at our configuration see if there's like something super easy to fix there those in toleration so we can't actually specify a toleration but I I'm not gonna worry about it for now so let's keep going actually I am gonna worry about it because I want to make sure this runs on my API server as well because right now it's not so let's take a look at the values file operator exists I don't think I can actually change being so here logging okay so long he's been cleaned up let's go ahead and edit our manifest best my best I am all go sit okay oops burner on it hey y'all I'm back I think open him back sorry about that that was a fun one machine just rebooted on me good times hey yeah that was Wow alright that was that was unfortunate so it's almost 240 but I want to actually finish this up I want to show this a little bit more so I'm gonna run through this real quick and we'll see how far we get so I appreciate you sticking around about this one I realized that that was like a pretty intense place to lose it lose it so the kind delete cluster so let's jump through this real quick see the kind kind creates cluster config I was a tough one we'll be back up in a jiff though so sorry about that that was pretty wild so we're good mister point again that was pretty well they don't think I've ever seen that happened before relatively new laptop not sure what actually caused the failure but like yeah I just basically rebooted on me where the little things might have been but I was actually manipulating the that I was manipulating like the file system to quickly across different namespaces in the same Linux kernel and I might have like found a little a little kernel exploit their doesn't work but hard to say oh shoot that's not going to work actually though the cluster CP cluster there we go that's better yeah probably could have brought another machine to do it but I'm glad you're all still with me we'll be back up here what just happened is I realize I didn't put the advanced audit log piece back in the file system so the control plane couldn't start so then I replaced it now we're back in play getting the worker knows brought up here okay we got a cluster up history crap coming in to load people need it is cluster here moo-coo vania's cluster what last a search here are yes okay images kind load Ducker image pop these on there Oh actually that's a great question let me pull it up ha ha Thank You mr. Josh Ross Oh taught me that trick what is it call objects Josh put up a post a while ago that was boom this thing the impurity is the animal support which I thought was really cool this is using the language server that can be used on a variety for I des I use them from my editor and so it was me I'm actually using this basically this same setup to do it so if you're interested in exploring that definitely check that out Eduardo it's very cool it's a yeah I really like it a lot it's actually the fanciest my vim setup has been in a very long time get off fly - cute kid I'll get pods - and cool system crap - running give that a sec to start up here yeah COC and you can do go pull you can use to go completion all kinds of cool stuff so yeah I'm actually planning on doing a session on this coming up in one of the new episodes I'm planning on doing like a development set up for kubernetes and that's part of what I want to show but yeah it's like it's pretty killer I'm really digging the new language server system once you get come up what you will do as soon as together elasticsearch and point is up okay we started at twice now get pod session coupe system in an elastic search to zero is up elastic search one is up Kabbalah is probably just about to be up and we're up all right so come on this back elastic stretches back puke it all proxy it'll proxy is back cat notes actually these are manifest you know from before we're good shape there so let's do it go ahead and reapply this you could alone fly that should benefit it'll get pods and fly chef manifests - - namespace equals lugging all right get pause - and lucky that's better okay so now you can all get pause - and login - why we can see that we're running on the master as well a control plane node as well and that's awesome but but I was about to show you before I was so rudely interrupted it was why that's working that way and so I want to share with you what this is this is actually a really cool thing so this Toleration I wish we were all a little bit more like this toleration this toleration means tolerate everything it means if it doesn't matter what it is with the toleration it means just it's a good it's a it's a orgasm it means tolerate everything so anywhere there's a node put this pod there doesn't matter what's on it doesn't matter how it's tainted just put it there there's a cubelet there and it's registered with the cluster put this pot on that cubelet that's what this means that's what Scoleri toleration operator exists me so pretty neat feature I'm pretty kind of kind of a superpower of toleration I see a lot of people trying to define toleration as that match like saw different configurations of clusters this is the simple and sometimes simplest is the best that's the simplest way to do that all right cool so now let's take a look at our indices for okay so before we saw a locks - one now we see the kubernetes cluster one and that is actually output coming from our new fluent did daemon set so let's go ahead and do our xdg open let's do our text eg open again and again we're going to jump into the cube on logging in point when I look at that output this is actually pretty cool because it means if I see the index it means I've got everything working right if I don't see the X it means that there's nothing working in it's not gonna give me a good time but I think we are past that so because it's a brand new install of elasticsearch so I have to again kind of pick my index name otherwise I would have had to go back and actually add a new index and that's fine next our time stamp is going to be time span we get so many fields 605 fields and here is our log coming from I want to look at tax [Music] that's mean all right so here is logs from the that's cool so this is these are actually lots from our grouping these cluster that are a part of our audit logs well it's actually is it's just probably part of the command stuff so so right now we only have like you know not very many positive Lloyd we didn't put our log generator back up there but we can see that there are you know bipods the content network pods the kind the kobata logging pod generating logs you got some source IP ease being captured by something I'm curious but that's well so let's go ahead and dig into that one this is kind of a fun part about logs you can dig into it a little bit more and forget what's happening so ah here so this is actually coming from our log of the audit decades - IO version right so that is awesome kind of Vince I'm looking for the tag though and I'm not seeing the tagging I must be missing something right like I see the log event which is awesome but I'm looking for a way to distinct disambiguate this type of log from all of the rest and so when we actually specified our tags I would have thought that that would have actually pulled that up but for some reason they don't see it but anyway this output is actually coming from the the audit log generated by the API server sitting on that control clean node and it's broken it up into things like this is an allow reason and like some of the content that are actually there's actually being captured by the kübra news a lot and yet another really great thing to get the heck off of your control flame node right like you don't you shouldn't have your audit log posted on your control play node because if it goes away then you no longer have an autumn log to describe what's happening but with that auto ID we can see who made what command and what they did like what the response was rather be allowed that in what the source IP address of the of that was lots of really good information to see right so these are basically just a get command and so this is all just a way of expressing the configuration of I mean this is actually just looking at the output of your current dip of fluid bit so that's I guess what I wanted to show you what I'm going to do is I'm going to make available everything that I've actually done here so this phone directory which includes the kind configuration the kubernetes manifests foot bit and fluent elasticsearch the log generator files and my notes are all going to be inside of the repository for this particular talk but yeah that's what I wanted to show you I guess there was actually it was one more thing that I noticed which was kind of a weird heads-up thing so let's take a look at that real quick if I look for cubelet logs tag [Music] so these logs from the Cuba's authentication log it's actually you know what let's do this let's do history grip blob Jim get a little scale deployment replicas 5 get pods so now we got our Bob ginge going let's go back to our aggregator and look for launch in so now we're gonna see stuff like the logs coming from the log gen output we're gonna be able to see when they get when that log gets created or we're gonna be able to see like the different lines long shell yeah struggling with Wagner output it's a different game but I'm not gonna play it right now anyway we're able to see like logs from all the things that are happening inside of here now and that's really cool the last thing I wanted to show you was that basically what I learned was that Debian by default doesn't persist logs to disk and so if I were that jump into the worker node bar log and Journal so this is actually where journal keeps all of its logs but only if they're made persistent by default they are on I or not and so when they noticed before was I wasn't able to actually get those outputs right so if I do journal fiddle - FL you fuel it right and I look for this I look for this line IO 28 it's kind of like reverse engineer our way into this right so this is a message coming from the cubelet and that's actually pretty powerful it gives us the ability to understand like a sexually standard area off of crime that that's not what I was looking for anyway we should be able to see the logs from the cubelet and from container D running this was clearly not specific enough this will probably be specific enough let's try that there we go all right so that is from the Keyblade it's just like identifier cubelet that's actually coming from the cubelet we can see that content so all of our things are actually showing up and so what I noticed before was I wasn't getting those outputs and so I needed to actually change the configuration to make sure that they're not in a different ethnics they're part of a different slot identifier so I think that's actually where I missed it so like in this case it's actually just like identifier cubelet but yeah so like this is the filter syslog yeah Oh anyway brutal that's what I wanted to show you I hope that was helpful I'm gonna sign off but yeah like right now they're not in a different index we could actually make it so that they were though by providing a different output at the moment they're all on the same output but yeah that's that's what I wanted to show you I hope that was helpful let me kick back over here to the big screen yeah all right thank you all very much there was a lotta soda I'm sorry that the reboot got us but we got back on our feet pretty quickly we've got things working again I hope that was really helpful and I look forward to seeing you next time thanks again for all of your help and I I hope all of this stuff isn't like - Wow but I look forward to seeing a bunch of you in Amsterdam if you have not met me in person please feel free to come by and say hello I'll be probably at the booth or I have a couple of presentations I love to meet people who are who have you were out there in the space trying to do good things and so it's been a pleasure I am Maui lying on Twitter I hope you all have a kicker and weekend and I look forward to presenting to you next time I'll see you all next time you

Channel: VMware Cloud Native Apps

Views: 3,096

Keywords:

Id: BmTCtLeuROU

Length: 123min 59sec (7439 seconds)

Published: Fri Feb 28 2020