Klustered: Community vs. Rawkode | Rawkode Live

[Music] hello and welcome to today's episode of clustered today's a bit different than normal uh today is clustered versus rock code versus the community not really sure how it's going to work but let me share some stuff with you first check out rocco.live that is the youtube channel please subscribe click the bell you'll get alerts and notifications for all new episodes of clustered as well as all new episodes where i explore cloud native products and technologies with their founders and make this all a little bit easier for us all there's also a discord server available at rockwell.chat there's nearly 500 of us in there now talking all things cloud native kubernetes and everything in between i started doing something a little bit different lately and i'm giving away a lot of swag um some people have asked what if i just want to buy a t-shirt and support it so i have included the link there so please feel free to check out store.rocco.com but also i will be giving away lots of swag over the next weeks and months so stay tuned enjoy the episodes and i hope you get to win something there's a couple of little links on this site now so custard.live is a thing i threw together literally in the last couple of days to try and make our streams more interactive and more engaging and give you an opportunity to win and it was a painful fighting with the twitter api but we do have something that works and i'll walk you through that in just a moment today's episode is a verses episode with myself so we have a cluster broken by some regulars you may know no or fresbo and russell w i had a third cluster which asked someone to break but unfortunately due to constraints time and well life it wasn't able to be broken in time for today's episode so you'll also notice here there is a rock dot live slash destroy dash it this is a public cube config to a kubernetes cluster running on equinix metal my challenge to you lot let's go with it uh i will be using well i will be blocking access to it very soon and with the next 30 minutes feel free to try and break it as you wish i also want to thank teleport for sponsoring uh sponsoring custard um you know i've been using teleport since the first episode i think it's an absolutely fantastic product it's just wonderful that i get to use it on the stream every week and get to show you all how cool it is so you should go check it out they support the show they're providing the swag they're the ones that are making all of this all of these giveaways possible so thank you to teleport um to support the show go to raw blooper rock dot live slash teleport it is a utm link but it just means that they have a little bit of feedback that shows you know supporting this channel works for them it's quite a lot of talk for the start of this episode um i'm also very nervous about this episode because i'm fixing things alone so i do have a beer okay let's take a look at today this is clustered.live let me remove our sponsor message for the time being where we can click saying and with twitter now there have been a few people that have mentioned that the permissions to access this application um are a little uh permissive but unfortunately the twitter api only really exposes read or read write there is nothing in between so i'm really sorry about that but it is the best that we can do i also ask for those permissions because when you enter the competition it will follow my account follow the teleport account and repeat retweet the tweet now i don't see it here because i've already entered but there is lots of warner messages i'm not doing this without your permission i hope you understand that it's really just to work around the twitter api and the fact that i can't check painfully cannot check if you follow an account or if you have retweeted a tweet i cannot stress it enough this was not the way that i wanted this to go but it was the best that i could come up with at least in this amount of time so i'm going to try and make this better but if you do want to win some swag today we have three rockwood swag packs we had three teleport swag packs and we have 10 copies of siam patak's sks training manual so enter the competition your odds are really really good of winning right we've got 10 13 16 16 prizes and we don't even have 16 entries yet so the chances are click that button and you can win something today raw code dot live destroy it will take you to this guest where there is a cube config download it attack the cluster i'll be getting to that in around 30 minutes time and also i know the permissions on my application are a bit wild they allow me to upgrade your profile they allowed me to delete your account i think there's a whole bunch of things again not by choice but the code for all of it is open source you can find it at github.com rocco dash academy slash clustered dot live cool today we are going to fix some clusters so i have three here we're going to start with russell w's cluster so thank you for taking the time out of your day to break this and what i'm going to do is start my timer there we go and zoom in oh hold on a minute why am i giving myself four five minutes all right we'll do i can i actually have much less than 45 minutes per cluster because i have three so really i've got like 20 ones per cluster and i can't really keep up with the chat and all the talking so i'll try and keep my eye out there who broke teleport come on kevin you know that's a pretty teleport uh you're also telling me it's a twitter page rubbish yeah it's absolute awful cannot stand it okay let's see what we've got the reason i'm starting with brussels because um i did look at nulls and notice we don't even have any worker nodes so i'm going to come back to that one instead we'll start with this one we'll log on to the admin account and i'll export my cube config alias k and i run get nodes all right i have no control plane thank you russell need a bigger drink so what do we do when we have no control plan first thing i'm going to do is check our cubelet and i can see the service is loaded but it is stopped i'm going to try just starting that back up and well that starts i'm going to jump into our kubernetes manifest thanks thanks russell um i hope you scripted that have your sip for each break yeah it might go that way no uh i don't have any static manifests they all appear to be the same number of baits i think they're all all right okay they're not all the same number base the letters are the same number of base so i think a b c d and e are the files i need and everything else is noise all right i am going to move a b c d e ammo to temp run ls yep and now i don't trust i don't need these so we'll call this dumping ground i'm gonna move everything to our dumping ground we're gonna move everything back to here where did that celium come from i don't think that's main so i'm going to push that back okay now we have a b c d e and e dot gamma i'm actually not fussed about the names i just want to make sure i've got the right components so that would just uh start oh yeah well so we've got our api server lcd controller manager scheduler cubezip okay hopefully that gets me a control plane online i guess what we want to see now is do we have an api server do we have a controller not quite do we have an actual working keyboard yet no okay so our couplet is broken we're going to take a look at our cubelet logs and it is misconfigured okay we shouldn't have an etsy cubelet fail so we're going to go into our start we do a system system control cat couplet and we're going to check to make sure there's nothing weird here or exact start here has been tampered with and that is in this drop in here so and so our cubelet should be in user then here's your bank cable maybe user ben local kiblet um user bank kublet the demon reload restart cubelet and we'll take a look at those logs and that looks a bit better hopefully those editors go away let's see where we got the chat so far yeah well i brought in the countdown during the team stuff just to make sure that we gave appropriate time to each of the clusters that we have so it's really more of a guide it's not always enforced but um yeah i try to keep it there uh some love for russell's break people laughing hahaha i'm not laughing uh that's a loading ammo all right okay so then was not you russell thank you um that helps i'm not sure why that's a little but yeah i'm always there ah your gentleman okay so we do have hints of we need some and slash root hands okay i'm curious if we have an api server we do not uh controller cube all right we've got a few things so now our api server is failing and we can go to var log containers and we can tail our cube api server and it's been shut down non-structural schema condition controller hmm interesting uh oh yeah they're called funny names all right let's see api server advertise address allow privileged authorization mode all fine time fine admission plugins no destruction is okay uh lcd should be all right you should check that it's running though and that looks all right i'm not sure about the probes hardcoded ip address is probably a private ipv4 of this machine not something i often look at i'm going to assume it's all right and i'm not sure if we should have a default set comp profile on api server but i'm thinking let's just run it in a web server yeah it should should be all right it's not scheduling anything okay that looks alright so i because i'm confused i'm going to take a look at cd being at the top not probably a good sign right uh yeah i think it looks okay okay uh let's see cube api look at this in a bit more detail when's this last star this is 1602 right that's a while ago so we're starting here oh we got a shutdown here how can it shut down before it starts yes that's funny okay could qbpi server control plane it looks looks fine i can never remember this stuff oh no wait it's there crystal uh runtime endpoint okay api is ever four minutes ago let me do logs container id all right okay so we can do logs but we need uh ps and then log on our api server's gone again okay strange i guess this is just the pod shows because the part is created even though the container doesn't exist i move fail just to try and encourage the cubelet to reschedule it okay um hmm so the cuba editors are because it kind of speaks to the api server but i'm not seeing one start i'm not getting any more logs i'm going to restart um i really want to see this cube api server available to you so i'm just going to remove it which i'm going to regret in five minutes and watch and we'll wait for that fail to come back see if we can get some more logs out of that api server i'm not sure what is happening here at the moment but i'm doing okay for time i think russell is just happy it took more than five minutes you know people always worry when i invite them on to break clusters that their effects is super too superficial and it just doesn't work that way like it's really hard debugging clusters i'll say that because i'm doing one but uh huh my cuba is running we're still almost again header messages here kind of want to be able to filter that out so let's remove the follow and search for api um okay so fail is creating another pod i finally got something that looks like it's useful and it's failing because the part already exists yeah so we did see that behavior with the cry control ps that's it was created a few minutes ago i suspect this is going to be even newer now no is it rmp so i've removed that pod um let's see if it comes back yep nope i don't know api server logs triplet what's happening just now um okay so okay so we got an api server crashlyt back off i understand component skipping field to start container follow the cube api server with a crashly back off okay um then why am i not getting logs that's a good idea kevin i think i will take a hint i'm going to take a look at our ips i'm going to take a look at this one more time because i feel uh why are we not getting logs okay is that a directory yeah okay so we're past the beach what's the second tent oh there's a readme okay welcome to uh you never told me there was a bag story okay so welcome to the crystal maze cluster you will need to travel through four zones in order to solve their puzzles good luck i hope it's challenging enough without being too frustrating well you failed it's frustrating let's begin picture zone if you want hints or just dive straight in beach sister via the beach yeah we done the beach so i don't think this is i don't think this is the industrial networking one something with the stairs and then there is a phrase you can't see the wood for the trees yeah i don't want to start opening all these hints russell so uh which one which hint should i look at for the api server break give me a heads up and i'll think really annoying there's no vlogs um i need to know why it's breaking must be something in here okay trees thank you russell the forest oh we've got a couple of hints here okay there's so many questions which files do you need to edit however this many fails and the static manifesto retrieval only a few containers being run where is the api server right well i think i fixed the first part of the forest ah sneaky very sneaky all right well i do not need any of your aliases hmm i should have paid more attention to them that i've just deleted them all because i wasn't actually sure what you were doing it looked like you were just intercepting my ls and cd which i meant i thought you had modified my manifest directory yeah i was far too quick to delete those aliases i should have paid attention okay it was intercepted in cd and ls i now know i've got the proper ls i'm going to take another hint just because i deleted that stuff too quickly i shouldn't have to login or nar log out so i did think you'd maybe change the manifest through that's why i did that the cat on a cubelet and i was gonna start poking around to see if you'd maybe uh told it to live in a different location um let's take a look there we go and use that i should have trusted my gap okay this is the actual static manifest directory api server looks good good looks good i i i don't think there's anything wrong with that api server don't be going around in circles here all right we got 20 minutes left might as well read all the forest hence what over here so this one was the best realities we found the real manifest so yes i am frustrated i can't try calling mr ben okay so you've basically broken everything is what you're telling me this is definitely this that's the theme and i still have uh qbpi server that i think is all right and you're telling me there's two problems in this api so yeah thank you russell okay i'm flying by lag and stop rushing so that's definitely the right command advertise address it should be okay i have ipv4 um okay these are fine i'm hoping i don't have to check all the stair paths secure port is okay there we go unless that is the port i've just never noticed that before i don't even know if that's a fixer i'm breaking it more um i need i wonder if i couldn't see logs because of that cd intercept maybe that was an old logan i still don't have any api okay well the probe supports the two issues in that spec oh this is wrong too that other parts are wrong okay let's try that log yet restarting the cubelet oh i hate you all right i'm just going to keep restarting the cubelet until i get an api server why is it not typing oh teleport is broken what have i done uh okay let me pull the bottom of that up we have an api server at least and i don't know if i broke teleport okay i don't know what happened there okay let's export here get pods i think they're being recorded no okay so russell says that's forrest and cryptography covered it's just the industrial stage now um so let's see these we're going to wait uh water one worker one worker two and worker one okay so i think your readme suggested that everything is on the control plane i've not touched the worker nodes at all okay let's see if we've got any 10x rules we don't we are these clusters are all 122. uh no i started provisioning them the day after release okay so uh we've got some networking problem and what we can do see if we can get any logs from this part here unable to reach the api server that may just need oh come down uh they just need to re-nudge and not okay let's see if that celine gets happy again and i wonder if that's the same reason the ambassador is potentially broken all right we'll just start rotating pods let's see okay that's definitely healthier so that means i can do keyget deployments and deployments clustered image two let's see if that works and it did let's see what happens and i've got the dance so i'm not sure what your industrial problem was russell but it did not stop me from deploying version two all right cluster fix thank you russell that was infuriating but very good um yeah the bachelor's is that you know what i just don't even i don't think you know there's there's nothing there's no feedback to let you know that you're working with an alias or anything like that so yeah maybe something i should start checking by default in the future uh yeah and that image sneaky okay let's jump on to knolls cluster which isn't looking very healthy at the moment we have no worker nodes here yes control all right well at least i'm getting real at ls cd it's a bubble in okay okay notes uh which keep control ctl looks all right version i've got an api server but something we have no apps we want our core v1 hmm okay so how would this be possible well we're going straight back to our static manifest we're taking a look at our api server configuration disappointed in my cursor did not start halfway down the file and do we have all our controllers running certainly looks like it didn't start my timer there you go [Music] i mean you could have reset the times on the files of course but it looks clean um double check the image i'm getting bit by that twice okay so either i don't have access to these types okay let's get cluster rules here hopefully it's not our back please don't be our back um so i have my admin conf let's take a look at the context for kubernetes admin speaking to the correct cluster we've got cluster admin here not our back that's good uh that was the only lead i had to know so let's check make sure this match is what we can expected to see from a static pod manifest no sneaky static admission controllers no controllers being disabled okay i should have brought two beers to this episode okay how are you removing pods how are you to move pots okay let's roll that out what do we have access to i'm able to retrieve the compute list of server apis i have to be one the server could not find how did you remove apps v1 and where's core v1 i've never seen that before i'm impressed um i mean it has to be it has to be api server unless you did something in the cluster and then out the cluster you know i think i'm i'm going to have to reach for henry okay so the apis have decided to play hide and seek i have the gun nasty ghosts um that's a hint this is sneaky okay so you could have modified cube control even though it's a binary may not be the correct one it does to get tree stick clean um let's make sure i'm calling this cube control i am calling this cube control do we have any other cube controls uh i'll shoot our pen cube this is too funny there it is run time config v1 equals false and ask v1 equal false okay so i can see it here run time but not there you've moved to stacked it like you've done the same right so let's take a look and our config here it's not been moved the static port manifest is the same how is that run time config again and to the api server what if it's not run as a static part all right it's definitely not in this fail right okay so it could be there's two static manifests and one of these fails no i like this i could see the problem do we trust a cubelet and those tell me to read the hints again okay the apis have to say play hide go our nasty goose don't worry i think he tries to use v i'm starting to think my cube is not a cubelet and i notice you've got a go directory here um can't really tell what you were building i should have hidden it from ps as well this is also a hint you all are cruel okay so we could be in a position where our api server scheduled by the cubelet just isn't running that may be the case because uh got it okay so that's nice our kubrick can't actually schedule the api server via container d because the port has already been bound so no has managed to i mean i did think systemd and i took a look in the systemd directory but i guess it's been a lot more sneaky than that which means in theory um i could probably just kill this process no it came straight back um i'm going to stop the cubelet i'm going to kill there and it's gone and start install reinstall it if you hacked my cubelet that's taking a little bit longer than i was kind of expecting it to i don't know if that's good or bad there we go and then i'm going to restart it but i'm going to assume you compile john cubelet to spin up his own api server before running a static micropods i think that's maybe what you've done uh no darn it okay i want to stop the koopa again i want to confirm that cubelet is definitely starting this process desperation on the linux command container d i think that has actually just powered ctrl okay that's a pretty big hen null says do you trust the manifest fails that you see i mean i think so um okay so you've not done something through a shell let's bare and i'll be ebpf do i trust the manifest fails i see okay so the api server does have an extra line in it it's kind of what i think you're telling me i can't see it i hate computers um let me if i stop my keeper the cubelet.com should just be the user to connect to it i don't think there would be anything in there seems normal woodworking is a great idea okay i have no idea i'm curious what happens if i add desktop i don't think it's adding let's kill the process to be sure i think this has to be bpf and there's no way to debug that have i broke it it's not getting my flags i added either you've stumped me no tell me what it is i just don't know where to start now so you're gonna have to i don't know how to use bpf tool near zbpf though and russell i did add a new param to the gamma it didn't show up um so i i don't know so nolan said he left bpf2 which means there is an ebf hack in somewhere all right no no more hints just tell us null said do i trust the off system into the node's teleport not particularly are you telling me i can sshn manually [Music] you're saying run system control cap teleport d [Music] okay so you're using a bar temp eppf kit with this source so there's a rogue cube api server manifest and var temp which is targeting a fail over there so you've got an ebpf probe that whenever the cubelet tries to access the qbpi server yaml inside of the manifest directory you're actually redirecting it to var temp was that supposed to happen all right i'm gonna just kill it it does not like that stop the couple kill the process oh yeah it does not like me doing that oh you have to oh okay got it oh no and the chat is saying i'll have to stop to teleport the process with the ebpf kit that's nice just of ebpf and really difficult to track down i think it's fair to say that eppf is the devil's work uh let's jump back over to here just now i really what i should have done at some point is run a status and take a look at everything in this list and i would have spotted our phony teleport d i think i think it would have spotted it so there we go a couple of a couple of earnings there always take a look at the systemd process table service list and and never let noel and russell near cluster again uh yeah i don't know what's happening here so i'm just going to leave and i'm going to assume that ebpf kit is not going to show up here i'm curious how you hid that but we'll just leave it that was tricky really really tricky good work now all right i gave you all access to this cluster i don't know if anyone has access to machine i don't know if anyone's wrong well no one's ssh i never give you access to uh all right i have removed yeah well you thanks for joining us uh we'll quickly check if anyone did anything to this cluster and if not we'll call it oh there has been a guess all right guess i'm fixing one more all right we have a cubelet i have an api server uh it's got secure port 663 ah i just i bet there's nothing wrong with this cluster except for me i didn't do the export properly oh maybe someone did break something okay let's see why our application is crashing there's no logs on that of course i should know that by now someone change the label sneaky and although maybe it's here so maybe it's a table in my automation that would be funny let's just bump it up to v2 always oh someone changed the probes i can't remember the ports on this application i think it's 8080 and here death by a thousand cuts i can see and resource limits click ok and what do we get here okay i can't change the label so maybe that is part of my automation we'll fix that back let's see what happens okay the probes aren't passing that's typically what the internal server error means it does look okay maybe i just better no it's broken okay let's see it should be a node port service maybe someone's modified that too uh selectors wrong okay so i think someone actually deleted that deployment and reapplied it with the broken labels so it's probably not my automation and we've got an endpoint now is that healthier i think that's healthier possibly going to tame it because postgres does have endpoints what happens if i curl okay what's wrong with my application and see i don't even know how to ask for help now i actually think if we pull up my automation uh i think it just runs in port 80 i don't think it is 8080 at all but it could be it could be wrong no 8080 okay or 8080 okay so what is wrong clustered so we've got an endpoint it's running uh but i'm not really able to do anything so let's try getting a shell let's see what happens you see they're going to timeout from the database or something else i failed to connect to this okay so all right can it resolve no okay so get pods all where's coordinates it's running um let's see if it is broken dns and everything i don't know if that's resolving no okay i have no networking and so it's probably not dns i've never said that before and possibly that i just can't query dns within the cluster okay host is fine selim appears to be okay um i'm kind of running out of time because i do have a hard stop but i'll see how far i can get through this uh if anybody made changes to this cluster and you're still watching drop a message in the in the chat and maybe we can speed this up a little bit okay so i wonder if someone's just applied cluster policies rather than modifying the conflict map because modifying the conflict map would be a bit mean [Music] there we go ah see sometimes you make it more difficult for yourself by looking into like oh they could have broken certainly when it can just be the really really really simple things okay so we've got some swag to give away and thank you to anyone who broke that cluster and that was a nice easy one which is uh appreciated after those two car crashes that we had and what do we need to do well we need to go to rockwood academy clustered live so like i said this repository is all online um we can run chicken dinner and we'll see we have 19 participants in today's competition i can stop my little timer now we don't need that right there we go and we have 10 winners so thank you to our sponsor teleport who are providing 10 copies of siam's cks book and they are providing three vouchers for swag i am also going to be giving away three rock code t-shirts so in total we are looking for 16 winners today the way this is going to work is the script just doesn't you know i it's it's i need to be able to run it like so it's just going to spout 16 names so what i'm gonna say is the first three names get the rocco t-shirts the next three names will get the teleport swag and the final ten names will get the cks book hopefully you all are happy with that i will make the code around this competition a bit more sophisticated over the next days and weeks but it just needs a bit more time this is what i could put on very quickly and i'm going to go back to fighting the twitter api very soon so python three chicken dinner sixteen winners please only three if you're not going to win i'm really sorry about that but that's the way the cookie crumbles okay rocco t-shirts go to waleed just me and open source and crux we have teleport swag for noel kevin and arishba and oh i won i'm gonna have to draw one more the merrick russell uh sassy yes i'm not sure um roberto jason phillip steve and avnish so i will save these to our winners i will draw one more because i obviously don't want to win let me remove the sponsor message just now and hopefully we don't get a repeat name said palace there you go sid devops directive you have also won a cks book so there is our winners uh thank you everyone awesome no worries uh i'll reach out i have all your twitter handles which are stored in firebase i have a drop user dm we will organize your swag and we will be doing this every week on clustered so make sure you come back we're also looking for more people to compete on clustered and more teams so if you want to come on and have some fun regardless of your level of skill or experience with kubernetes and we'll find a way to get you on and have some fun with this so drop me a dm on twitter say if you want to join solo or teams or both and we'll do our best to make that happen i'm going to give one final thank you to teleport for their support and for letting us give away some swag and i'm going to say goodbye and i will see you all later so thanks for joining us have a wonderful day and i'll speak to y'all soon bye so [Music] [Music] you

Channel: Rawkode Academy

Views: 220

Keywords:

Id: _BFbrrXKMOM

Length: 81min 31sec (4891 seconds)

Published: Thu Aug 12 2021