Klustered Teams: Control Plane & Learnk8s | Rawkode Live

[Music] run hello and welcome to today's episode of clustered at the raw code academy i am your host raw code today we have two great teams picking up a clustered mantle and fighting each other to win this one's a bit weird because all the teams know each other and i think it's going to be a whole lot of fun all right now before we get started this is a little bit of housekeeping if you're not already subscribed to the channel i would encourage you to do so now also if you'd be so kind thumb up the video like it comment share all those other things it's just going to help youtube share this content with more people we also have membership options available at the academy so you can check them out and join for as little as 99 pence per month or cents and this this allows you to support it and i would appreciate that very much and if you want to continue the conversation or you're not watching live we do have a discord server available at rocco.chat there's only 600 cloud native technologists in there talking about all science cloud native kubernetes ebpf and everything in between so come and say hello and i look forward to meeting you i also want to thank teleport for sponsoring clustered this was the easiest decision ever had to made we've been using teleport since the very beginning is a fantastic tool you're going to see us using it today to pair and share terminal sessions on these rather broken clusters so check it out get it running in your cluster and that is how you get on you just used terry rockhold.live teleport uh it's a utm link but it keeps the sponsor happy so that keeps me happy all right so we had a little bit of technical troubles but we are back in action and i am going to introduce our two teams today so let's pop over here we have a slightly different layout we've got two teams of two which means we can have everybody on the screen at the same time it's going to be a whole lot of fun let's start at the top left jack do you want to say hello introduce yourself and then we'll just move around the windows please sure so i'm jack from control plane i'm an infrastructure engineer um usually i'm i'm doing more throwing stuff at clusters but yeah looking forward to fixing some as well awesome thank you hi everyone i'm james i'm also from control play um i'm also used to uh breaking things and pointing out broken things a bit more so looking forward to having to fix some stuff up today awesome thanks for sharing hello i'm chris uh i'm i guess stumbled into some bits of devops i spend most of my time kind of helping uh digital teams become kind of more efficient and more empowered and enabled and stuff um i'm a developer at heart so but yeah ended up kind of looking at devopsy and other buzzword nonsense um which made me stumbled into kubernetes um and i'm currently a teacher at uh learn carets awesome thank you hey my name is samanik bao and just like uh chris i am also an instructor for kubernetes and learn k8 and i also work as an amateur engineer for appvia i usually break stuff and don't bother fixing them that's what i do so but today is going to be a real test for me to try and fix the broken stuff all right thank you all for introducing yourselves so we're going to kick things off with our first cluster i have the learn kate's cluster in front of me so that means team control plane you are up first this is teleport i am going to click here to open a session on the control plane node please do not use that button yourself there is the activity panel here and active sessions so make sure you join my session i'm going to ask you to type echo hello or echo your name anything like that to let me know that you're there and then i will start the timer and we'll get the session underway just a moment so remember active sessions and then join okay this is the newest one all right i guess that's that we've got both of you in the chat and the chat in the session all right i'm going to start the timer feel free to configure your cube config and best of luck okie dokie yeah i guess you could uh you could use teleport and a session as the uh as a chat um i'm gonna get started by bringing some of my own tools just so that i can debug nice and quick it doesn't like that okay never mind um right let's uh let's have a look what we're working with so what kind of tools were downloading jack um it was going to be like bring my own uh cube ctl just in case i can't trust it oh come on now it's the first step you still know that's the first thing you've definitely watched us before if you don't trust control i'm just saying yeah okay let's get that going i mean let's just have a look at pods okey-dokey oh look at that maybe you should have trusted that oh yeah damn fixed it is it already fixed if i let's see what that error was i haven't used you can't do it you'd have to add a user okay i have a solution for this already oh yeah jack came prepared this stream is not gonna last too long now okay let's grab this instead copy okay cool i have nyx portable uh just make sure it works and then let me just grab my tools so i'm using nyx to grab a bunch of tools i could just write a script with wget or whatever um oh you're using netflix as well not this next show yes right so i'll go over the tools that are in there in a bit but um hopefully once this downloads a bunch of stuff i will be able to see it a bit clearer but um in theory if i didn't tr well in this i've got cube ctl i've got like busy box um you know requests uh i've got bats so that i can get some nice colors and a couple other bits and bobs do you appreciate some colors you can prepare yeah if you exactly if you're typically like debugging your own cluster you probably don't need to to treat it as such an adversary um but yeah i i've seen some pretty nasty stuff during clustered so i thought bring your own tools might help i like it um and yeah i mean hopefully that will fix this problem uh we can also look into kind of what happened to coop ctl there um i think messing with coop ctl might be a bit of a theme here um but we'll see uh i could have also slimmed this down a bit i did pull in stuff like docker as well which is a bit a bit too extra i think you did the right thing you will need that you'll do that later on so if i do which coupe ctl it's now uh in the next store um that doesn't actually exist on the host it's because i'm using next portable it's uh in a different location but we can uh we can skip over that so if i do that flake next we can see the tools i'm pulling in cube ctl dig requests node ctl yeah a bunch of other stuff and i thought busybox because why even trust ps when you can just bring your own um so if we look at user bin cubectl i guess yeah so it's not executable i guess we could make executable um the other thing we can do is we have a control plane controlled by an employee going rogue in this location we see lewis popping his head up ahead louis what are you doing man so we can see that they're they're roughly the same size so there might have been some meddling going on um but that they're they're going to be built slightly differently anyway um are you going to try modding your the yeah let's try it if it um yeah let's just see what it does i mean hopefully it doesn't like exit me out of this session or something okay it looks it looks all right uh no api's never like that let me just use my one just in case okay that's fine um right what should we do now should we should we check the kubelet quickly uh oh in a true okay let's let's pop out of this okay kubler what's it doing anything popping out here james so we've got some some pods that are broken are we dialing six four four three but i'm not sure that's particularly uh that's maybe expected since api server isn't up yes okay we could look at ptc cube manifests uh okay if we can spot anything a miss in uh the api server yammer maybe yeah that sounds like a good good uh thing to try out let me just uh grab my vim again david we're supposed to say hot recalled if they're getting closer or are we not no if they want to head they can give you a no just feel free to taunt and laugh if you wish right exactly where i'd look that's exactly what i'm doing i'll maybe save the taunts in the last for the last five minutes if we get anywhere near that [Laughter] sounds good sounds good jack we could uh have a little look at the time stamps on the files in that folder to see if a lot of people will it keeps what are you saying hot let's have a look okay 16 so yeah manifest has changed the second manifests we're looking at the coop controller coupe scheduler i'm not looking forward to this yeah i love it when people have time stamps from like 30 minutes before the show [Music] yeah can do um the other thing i was thinking is i have a i have a um a little tar file here that i can just pop in here okay wow you came prepared so let's see what the uh the diff is uh for the kubernetes uh backups kubernetes manifests so we can have a look so let's see is there anything odd that just looks pretty normal stuff um let's have a scroll through uh that that read-only i think looks pretty sauce yeah the read only is interesting for sure so which files that uh that's the lcd manifest okay very nice because yeah i noticed in a lot of these uh in a lot of these clustered um exercises it's just a case of what has changed and what has not okay controllers has has been changed are you feeling learned kate's seeing this being deconstructed right in front of you it's good there's a learning experience for everyone that's what you want learn from people to learn about kubernetes now the breakdown i'm not familiar with this format so does does disable replica sets yeah so the controllers you can do dash replica set dash replica replication controller and it will disable that individual controller and the controller manager yeah that's very cool i guess uh i guess if you know that you don't need something you can just remove it uh to reduce your attack surface uh anything else of interest you can see that the uh replicas that controller might be quite useful though yes yeah it probably is useful you might want that i guess right let's let's have a go so i think the main things there was the read-only and the rep cassette so in fact i one of the files was touched but i didn't i didn't notice any difference i wasn't paying massive amount of attention it's quite hard to notice these differences uh manifests so i think there was one in fcd wasn't there there was a read only yeah that was the read only you spotted okay also did it open on read only that's the other thing just does vim open where where the change happened so it was closed controller yeah so said vim has opened at where the issue is um it's always funny nice and helpful yeah i i was looking into that earlier and the file of interest is um info thanks for info yeah them info so if we go and bim info we can find some other interesting things edited we got um we got a jack the big question is whether jack and james have cleared their vehicle before shutting down their sister i don't think i'd be showing it off okay so i'm not sure why this file has been opened either i guess we'll find out a lot of laughter in the comments and people enjoying their forensic approach to this very good very good job right i think we uh there was also cube vip looked like it had been uh might have been edited that we didn't spot earlier yeah it turns out we broke kubevip without trying to break it um caused some issues um so great tip from russell there start editing all fails was said does the kubelet just restart itself or the cubelet won't restart itself no but it'll restart any static manifests that are modified when it notices the change ah got it okay or you can force it with the restart so i mean it might be a minute or two for the changes um i guess the other thing we could look into is that um other file to change so uh great there's a vet yeah or should we look at the um vip first actually kubernetes manifests so okay that one started at the top which is interesting i guess backups kubernetes manifests cube vip and then tc manifests cube vip and then i'll just pipeline back for some oh i don't have i'm not ignoring anyone escape to the shell uh what have we got here yeah i mean in theory it could have been changed but we'll find out i guess also if kubip was screwed up i guess we wouldn't be in teleport right now um so the other thing was um this fifth file so this is a file that i'm not familiar with and also i do not have backup of so i can't just cheat um i don't know if you're familiar with this one either james but yeah i'm not done that i would never touch this folder okay that looks very fun um i'm not sure we even need this file so i'm going to can you comment maybe you can comment in these phones i uh from the name would hazard a guess that the uh if up.d would run on interface being brought up so potentially we might need to undo that in ip tables yes i'm also not very familiar with ip tables so i'm not looking forward to this uh right okay so so my yeah some stuff don't know whether the uh state would have been saved in the uh save state file whether we can restore from that or whether we'll just have to target it manually so you can run ap tables dash capital l to list all the different rules and you want dash tnap properly to target than that table since that's where it is added uh would that just be the title uh let me just grab something with 8080 james is right with the dash t net you don't need to do the regret oh so dash is capital l base dash to that can i do them in any order yeah yes and you might want dash n as well which will give us the rule number sounds very good or maybe not actually no ends maybe you don't need the l of the dash t no you do okay this seems a bit noisy yes uh let me just do a clear um right working with him so i think about middle of the screen there's the offending rule in the output chain oh yeah my highlight isn't showing yeah below the second psyllium comment yeah so i guess we have two options you can either delete that rule or change the whole cluster so that it's not on that port just a quick bit of google foo to try and remind myself exactly the ip tables incantation sounds good the comments are spot on if you need them yeah i think dash dash line numbers is the exact flag i was looking for earlier there we go yes that's the okay we have a number but uh okay i'm not familiar with how i would target that i guess it's on output right nash i think you need the t-nut and then it's is it dash capital d for drop and then the line number if my very rusty memory serves me correctly 13 oh you could always take the sledgehammer approach which is the raw code approved way watch all the rules can i just wipe everything and then wait for it to be rebuilt you can i will also point out this is a psyllium cluster with no cube proxy using ebpf and xdp for traffic rating so well it seemed there's a couple psyllium things in here so is that just for compatibility so just not work together or i'm pretty confident you can make it and this this machine will be fine by lead suggesting the same push it yes yes could do um okay that would be dash capital f to flash everything okay sure should we just wing it if you i have the i think i have the command if you just want to delete the single one yeah let's let's go with the slightly safer option i don't want the cluster to implode dash capital d definitely run out of time uh do i need the the net thing uh yes output in caps and then the rule number which was two was it yeah yes okay moment of truth is it gone it is gone nice [Applause] okay cube ctl get pods we are in more business than we were earlier um i guess well while we're here um no this is a nice tool for for anyone doing their c-cad you can look at all the short names if you want to do the uh the shorthand for like pods and deployments and whatnot um cool so we've got the pods there uh i guess i should look and teleport to see if the service is running or in fact let's just get can i just curl this you are halfway through 20 minutes left okey dokey okay so it seems to be running fine oh another thing that's really good is remember the completion stuff cool do we want to preemptively check for any uh admission controllers potentially or should we just try and uh try and go ahead and edit the uh i think yeah going going ahead with an edit sounds uh sounds fine i can't i can't see it causing any issues that are non-recoverable cube ctl edit deploy and then cluster nice did i misspell it there you go um and then v1 v2 please um let me just have a scroll through this see if there's anything right you don't trust them i can't remember if it was health or health c that's something to yeah i can never remember this hell for help it's clustered first which was the nasty one from last time rather than it was on was it default i would never have caught that yeah matt tried to sneak that one past me but i wasn't letting him have that okay let's get the pods all right you want me to test it it looks like it recently if you could that would be handy moment of truth someone looks what it my word was over two minutes and as soon as you downloaded the tools i was like this is done this is done we gotta dance well done good job james good job that was the master master class in forensic debugging that was very good thanks yeah i i would have been completely stumped there on on ip tables i i avoid that like the plague no no you did well i like the approach you know you took it slow you just identified what you wanted to you wanted to check for history you wanted to check for vermentful prepare okay so this is our first stream of ultra ultra low latency on youtube in the chat i want this to run a test to see how low that latency is so put your fingers on the one key and then i'm gonna say go and you're all gonna press one and i'm gonna get the other cluster ready just now while i do that all right so control plane cool all right go got some more comments i've not seen any ones yet so they're still not all literacy enough i i can hear you on the stream uh they've got on the tv it's not that far yeah there's not much latency to be honest yeah it's not too bad all right thank you all all right let's jump back over to our screen share so i have the control plane cluster here now so if the learn kate's team salmon and chris if you can jump in and join this session remember to use active sessions and join type echo hello echo your name anything you wish let's make sure we're all on the same page and then we'll get this thing started thank you to everyone who typed a one i think it is a lot better definitely of course lewis comes in 12 minutes later nice one mate am i in the right one i'm not sure if i um i see nope just me and chris sessions i'm in active sessions and the only active session i see is learn cades hey you're in the wrong cluster join the control plane cluster teleport dot customer yeah oh wait there you go a second i'm gonna start your timer now salmon for being so unprepared that's supposed to happen it's not gonna make any difference i'll just blow it away i was just like jack you've got this i gotta do authenticator now real quick we're now about to demo how to absolutely not prepare for one of these and fail miserably no these are going to do great like i host the kubernetes office hours and we share your learn case debugging image a lot for people but we've got a lot of confidence in you we're looking forward to you fixing this cluster oh be prepared to be disappointed i'll put that warning out there now for some reason i can't log in control plane cluster.live different login it's your the other one did you register for both no i didn't register for both anyway chris is doing the taping you can do the navigating i'm going to start your timer now so set up your cube config and let's check for our control plane and uh best of luck cool uh where is it um [Music] uh cool okay um [Music] so it's like no less or no well that's weird because ls should be a built-in into the shell but that's that's yeah i've got no built-in things you have air cool uh vehicle i have a thing i can see that i'm maybe in bash come on uh interesting they know you know they're not going to get keep calling me that are you um [Music] it looks like you're missing a lot of things so how how does the shell find binaries uh bypass so yeah my path messed up thanks uh cool thank you yeah interesting okay so um you could echo path and fix it yeah um what would it normally be something like um just in case there's anything in there you trust what's in there [Laughter] cool i guess the other thing you could think about is um like how would we have removed the path oh okay fine um [Music] well i'll close to find uh ben uh cats uh it's bash isn't it um [Music] it's easiest way to restart the session or i can source this um yeah just sourcing it will work because we we put the original path at the top of the file as well oh okay cool we should add it back in ray fine cool thank you okay fine first stumbling block okay so i can now do you um uh if i still got my keep config yeah fine say more etcetera [Music] that looks reasonably unsuspect [Music] let's let's fix in peace louis come on so the pods currently in a state where it's got many ready wait it got elite readiness i guess terminating state [Music] so this is the binary i wouldn't trust i'm going to say that again i'd say this is the binary i wouldn't trust the ufctl binary interesting okay um so we had all sorts of aspirations of a bad keeping a bad cube coal um and never got around to doing it so kudos for actually doing the thing that we plan to uh we did want to do something like changing the code but it was way too difficult so it sounds very very crude yeah i had to look at changing the code i was like well not for not for now later on should we just get a new she was just getting i should just download a new one and uh stick then um well judging by what they just the clue that we just got i'm wondering whether that's yes any small so it's probably a shell script that's aliasing to it being somewhere else which is what i was gonna do for us and then we didn't even get to do that there you go there you go i'm not saying i can uh [Music] right now what does it look like so that's a real coop ctl this time interesting [Laughter] [Music] yeah should we just delete this deployment and carry our own first uh we could just do like a keep call run uh as an asa busybox yeah uh uh image oh yeah this this isn't meant to happen this is uh me not knowing how js policy works very well [Laughter] um but it would it would be blocking yeah i would have an admit there's an admission controller knocking about it's called js policy something so [Music] um [Music] what files look uh so if you've changed them you've moved the dates back um so was it js policy is somewhere doing something uh there is a docker right okay so js policy was the thing that you tried to put in but that's going to crash back yeah sometimes it works sometimes it doesn't we're not sure why i used it enough to judge um should be the same though yeah yeah the fix in in both scenarios is the same i can also brainstorm to look at the mutating web hooks um [Music] you can use api resources to get a list of the resources on the cluster [Music] [Music] i'll see the same source the oh what's up castle uh completion bash was that what was uh it might be a carrot instead of the dollar yeah less ensemble instead of a dollar okay yeah oh you're not current uh you're still going to use a dollar right right so validating the book um [Music] [Music] are you going to go for a splash hammer approach yeah delete it yeah yeah i mean why not uh what's the worst that can happen right you can give it a go okay and they didn't get recreated by some magic um i'm surprised that worked actually [Laughter] i'll take that better luck then um that's absolutely fine uh what was the other one and it was a web taking workbook uh if i can type which there isn't any uh and they're not making space fun okay um so now what we got we've now got a that's just come to life so what's the i'm guessing it's a status we're getting honk so is that something in writing an etcd the status is hunk we we haven't bothered with uh with it we haven't bothered with okay api server perhaps there's some modify the code in the api server to return home uh let's back off fail restart say you can have a look at the processes to see if there's anything suggestive but it is but it is actually right so uh [Music] right let's just optimistically change the thing and then get rid of the readiness probe just to make it so that it's ready want to do this then you wanna should we have a look at the api server config yeah sure i'm just gonna i get i'm just gonna pretend that the thing does work so i'm gonna get rid of the readiness and the liveliness uh because i'll assume that we've done an excellent job and that that's that's containers absolutely fine and it never needs to be checked or what's responsible for writing the status on a pod you might soon have better luck if you watch what's happening to the status um updating endpoint slices fancy snapdragon node not farm what's that yeah james set this one up and uh it doesn't show up in uh in events either which is quite brutal what he's looking for in a process list anything are there any anything that looks keywords you can think of that might be worth prepping for grab for a honk yeah lowercase might work uh what have you done and where is that oh it's in a container somewhere presumably what container is that going to be in similarly the api server right yeah probably or the cubelet maybe the cube on the nodes might be sending that back you can also see the uh the parent of the uh of the process um [Music] how did you see that remember it's like ps try eff yes and then it's the what was the thing it's uh [Music] well there's a funny name space somewhere [Music] let's look at that one and there was a the containers in an angry geese container deep running actually in there so what are you thinking just now uh get rid of that chess policy thing that's not helping us anyway but the um there's a container namespace of angry geese um [Music] which seems suspect i did that ps dash air ef but that's giving me like a load of things not just the tree um i think i did ps yesterday yeah uh what was the number that was the number of the process that we cared about right that opt uh honk the planet so i wanted to see the parent of that process it was this one right i think if you just do psef on its own you should get a tree i could be mistaken oh you need a [Music] so i think you've got two avenues right now um you're suspecting that this the file does not exist on the host so you're pretty much looking for a container that's running it so i mean you can run a queue control get pods all and look for random container accounts in a pod or you're going to have to use cry control to list containers that are running outside of kubernetes so if you're happy that that looks normal then you're going to want to start looking for containers that are not in the kubernetes namespace yeah thanks normal it was ps f a u x for the tree yeah f a u x is really good so [Applause] it might not help me at all but i'm just gonna delete all the pods and let everything come back in case it wasn't set up as something that was persistent um that sounds like a scary thing to do no i i don't know if you delete everything don't delete anything from cube system why not it's just a pod it'll come back yeah i mean no one's telling me not to that it'll be a really bad idea what's the what's the container run time on this container d so you can just correct should we should we check what containers are running i'm guessing they're just running i'm guessing you you've started that container using the the yeah so let me add the flag for you just because i don't know yeah i've never used the crowd control oh sorry oh sorry i'm talking all right right look at that okay cool okay so can we check what containers are running you can also use ctr um and do a ctrps as well so it depends whether you want to look at a kubernetes context or from a container d context you could do ls what is it pc ls let's continue cls okay in that angry geese namespace right so yes so you can add the dash n to that was it angry geese which we saw yeah you need to put the end before before that's the thing i want to get rid of um so what was the that's ls so what's the you can do a crm okay so like that [Music] c space rm and then the id okay you might have to build the task first left we're all showing our container d ignorance here you might have to um stop the task first so if you do tls okay sorry again so ctrl yeah t or task like that space ls yeah yeah i'm done okay now we can remove it so you need to rm both the task on the container okay okay fun and you might even need to um stop the task first sometimes i've just shut down the machine let's just turn off why uh it's much nicer if you have it but it's uh you know ttl is good all right [Music] [Music] rm that task i think the killer should hopefully have done that you'll now need to do a container ls to see if the container still exists and if so you'll need to remove that too yeah but my task is still running even though i've killed it yeah but now you should be able to remove the container oh wait it is yes you're running you sometimes have to do kill s sig kill uh then yeah all right okay and it is it's back running again yeah i would try a trm well we saw it stopped momentarily so can i just like string the commands along each other is there some delay between them there isn't an extra thing that we made you do here by the way you should just be able to kill rm and then rm the container it's always where it's starting yeah you've done some other accidental magic um we'll check the parameters on trm there must be a way to force remove this thing yeah ah um fine um [Music] would you like me to tape for a moment because i'm really curious yeah um i was gonna do that autumn i going to exec into it and just doing whatever it is and that's gone now i'm sure that's what i did at some point you need to check for the container uh cls and then i want to uh crm stop on that one i think because who needs consistent oh yeah we were doing crm earlier i did our eminence because it wasn't stopped are there any parameters to rm it doesn't look like that no it's just keep snapshots there yeah you're gonna have to do uh why is the container still there is the task still there let's do a tls because i'm curious if it's magically come back yes back right okay if i just exec onto the running thing and then i can do some other magic and just stop it doing the thing that it's doing maybe use nerd ctl it's it's way more comfortable to use it's more more darker style also not available on the machine right now though okay unless we downloaded it and put it somewhere sneaky on the machine do you think we're going to trust your tools i trust them more than mine you um you type nctl uh let's just attach onto the thing and quickly if i see if i can do that quickly and then um uh what was i saying unless this task exec like onto that presumably i'm gonna guess what the syntax is you need a primary dash dash exec dash pid sorry exec id for reasons that i understand and who says continue it's not easy to work with all right i'm gonna throw it why don't you try a task pause like we don't actually need to get rid of it we just want to stop it doing its naughty things right let's try a tea pause yeah oh and confirm with a yeah still running that's not fair now regret for maybe angry or geese or honk [Music] that's the one isn't it that doesn't help yeah the sham is always going to be recreated uh just an interesting time how did you just create this task because i'm really surprised task rn didn't get rid of it have you got a system d process running or a rogue script uh node ctl with always restart ah okay so if you type dot dot nctl you'll get nerd cto and it might be able to fix the problem you should be able to kill it from that a bit more easily yeah it looks like the always kind of thing isn't an official container d feature so i think it does some magic to keep itself alive sneaky sneaky good to know i don't think we even expected it to stay alive this long [Laughter] stay around for a bit all right is it gone is it actually gone chris is doing over here stop kill our rm just to make sure great right another time right here we've been donating for a while so i'm assuming your anger geese was constantly updating the status of those pods um so run a wall through luke oh you have that's not good considering you deleted that namespace js policy [Music] the original intention with the policy engine was to have the admission controller get in the way but also protect itself from removal but but that didn't quite work right the goal was that you'd have to remove the admission at all i think delete them both web hooks uh yeah maybe the dust actually didn't actually work in the back end it did appear to work yeah all right now just do your busy box again and just make sure that js policy is gone because it it might be recreating this oh yeah definitely will we but that doesn't matter right so now i've got a thing that looks like it's going to start yeah i think the creation is because your your tag change yes the cluster is up postgres is it coming up yeah uh now let's check it i think it's check is it up and running both of them and just to get buds all right yes cool uh i mean i could optimistically ask you to check it but i'm not i found that version right no we changed it we changed it already yeah let's try um let's check let's check yeah go on then different it's definitely not that there's gonna be something else oh fail to connect the database authentication failed okay but it that's it's got that far so the thing works so i just now can't connect to the database well the database credentials are all hard coded and configured in the postgres stateful set okay but it was an author that's what you saw it wasn't our failure yeah connectivity looked good so just the passwords then [Music] it's a stateful set yep the password has been changed okay uh where is the password i believe it's postgres one two three but i will confirm for you now i really should know this is it anywhere that i would have been able to find it in that text yeah so the password should be [Music] postgres well postgresql123 yeah okay so i just had an extra few characters thrown on the ends oh i'm not going to care about trying to fix them and change the thing that clicks obviously it lets me get away with that quickly before it respawns um you're going to have to delete js policy pod deployment sorry oh yeah yeah if i delete that name space didn't i yeah but there's probably a finalizer on the gs policy plot [Music] i would just delete the deployment and hope for the best rather looking at what it is all right and then i think three two one okay not healthy yet there we go good work good work five minutes to spare loads of time well done well done nice questions there well done everyone yeah good job guys good job yeah it's really good so that's a really i think the takeaway there for me i had no idea that nerd control offered that always restart like that's that's not behavior i expected to see from continuity there so like does nerd control have a long running process that was instigating that i'm not sure i was gonna say yeah when you when you add the dash dash always i'm not sure how exactly it works but we noticed like it injects itself in to the container to like capture logs and stuff so it must do do some injection where it like adds itself in so it can re-add containers or something that's a sneaky one and then basically magic that's what that's describing and i also love that the you never really got the gs policy thing doing exactly what you wanted but it was still causing absolute havoc through the entire thing as well yeah i'm not sure if it can cause kind of more hassle that way or not but yeah all right well uh yeah those those are great well done both teams good job breaking good job fixing i think we've seen a nice mixture of different stuff there which is always fun to see so uh thank you for for taking time out of your week you know it's it's not easy to break these clusters even if you leave it to the last minutes tomorrow just saying always but it was a pleasure it was good watching his work we learned a lot today so thank you for taking time out of your day to join us thank you to everyone that watched and thank you teleport for sponsoring remember to check out teleport rock live slash teleport all right any last words from anyone before i let you get back to your evening thank you very much for having us uh long time watcher first time streamer so thank you all thanks for having us all right thanks everyone have a great day and i'll see you all soon [Music] [Applause] thank you for watching [Music] you

Channel: Rawkode Academy

Views: 363

Rating: 5 out of 5

Keywords:

Id: FClIbQ8hdxY

Length: 68min 52sec (4132 seconds)

Published: Thu Sep 16 2021