2021-04-03 - Building a HA k3s Cluster with embedded etcd and with HA Rancher

[Music] hey how's it going happy saturday how's everyone doing sorry for the delay right there as you can see probably in the uh in the chat logs i was checking my bot so um usually my bot's up well it's not right now and it's pretty it's pretty funny that i'm actually going to talk about like my kubernetes network and my kubernetes cluster and how i have it architected uh and yet something's not working but i think i know what it is so like that's why the intro took a little bit longer i ended up uh committing i ended up writing some code really quick and deploying it so i don't know i'll test in production so it's rolling out now hopefully my bot will be up here in a little bit i don't know maybe in about five minutes i think it is so i mean you you guys probably know if you've been around the channel for a while i have full ci cd working um within my kubernetes cluster i mean i use a combination of git lab and and a couple other things uh to build my code and deploy it and it deploys to my cluster that's in my basement my kubernetes cluster um hey real quick skill point thanks for the fall appreciate it reese dude thanks for the resub prime three months thank you one month streak dude thank you so much thank you appreciate it um thanks man appreciate it sorry no lights uh actually here i will manually flash the lights for you i think i can there we go manually flash the lights for a second thank you for the sub appreciate it thank you so much takano dude i'll keep the lights flashing why not takano dude thanks so much thanks for the resub dude 10 months 10 months resubscribe dude thank you so much appreciate it takana how you doing man how's it going rhys how you doing dude i missed that one too r-r ario mayo hey thanks for the sub prime's up dude thank you so much appreciate it thank you i'll just let's just keep these lights going let's keep these i might as well keep it going well i train too wow thank you so much man i i should my bot should break more often dude thank you so much uh cars car size thank you so much has subscribed uh prime sub thank you so much hype train level one complete already wow i just got started thank you thank you oh i need to iron this shirt ah man i'm like hype train and my shirt's all wrinkly but thank you so much i really appreciate it really appreciate it um all of you i just want to make sure reese ario mayo takano car size car seats i'm going to go with that thank you so much for the the sub and continued support for those that resubscribed really appreciate it um keep this going for a minute uh let me see can i get it back to non-flash all right see the manual way the manually so what i was saying is i i have a bot um that runs this channel or is as a participant in this channel i should say um and uh wasn't working and uh it's it's i figured it out it's only the twitch piece and so i was like okay i'm gonna let my intro screen roll one more time if i can write this code really quick to fix it um uh i'm gonna push it up and so that's what i did i i wrote the code committed it pushed it up tagged it it's rolling out now usually so i have ci and cd um between git lab and my kubernetes cluster and so now it's rolling out now so in about four or five minutes i think in about five minutes i can get code into production my home production uh which is my kubernetes service which is a brand new cluster so there's a lot of things that uh that need to go right right now uh that i've never tested like live like this and uh so like i was saying like today um you know i kind of wanted to talk through some of the changes i made to my kubernetes cluster a lot of folks have been asking you know about about my new new kind of architecture because i've been i've been hinting that i i've been working on a new architecture um and uh so i had some problems as you probably saw in the video today i had some problems with my kubernetes cluster it uh it's it's it's probably my own doing but i thought you know what i want h.a databases and so as you guys know or may or may not know i've used a k3s with a combination of mysql for my databases and so the my sequel way is fantastic don't get me wrong like nothing i say from here on out has anything to do with an external database version of k3s because it works fantastic um but it's it's not ha unless you have um a replication and you have an aj database if you get databases of service from somewhere else that provides a ha service done done easy if you're running it at home it's a little bit more challenging and so i'm you know i'm not a database administrator but uh i play one at home just like i play a lot of other things at home so i was like i got this i got this you know what i'll set up replication uh with my sequel uh i'll i'll join them to a cluster or at least replicate them and there's a couple ways you can replicate them and i was like i got this and so i set it up and it started working and i was like yeah that was easy that was easy well little did i know slowly over time my databases were drifting and like syncing the wrong things with each other you know on week one i was like hmm why do i see old pods that shouldn't be there anymore i was like ah maybe it's a fluke maybe it's a bug with rancher you know blame it on everything but yourself first so then week two i was like hold on there are nodes in my cluster according to rancher that aren't in my cluster anymore according to coug control that's when i started to realize that something was definitely wrong and so you know over the last like two three weeks i've just been just been hot fixing just like okay restore my database back to a good point fix some services and that worked okay but then all of a sudden like my ingresses started disappearing um because new ones weren't getting created when i would deploy new services with rancher and i was like yeah this isn't good this is not good at all and so so then i was like okay i i'm done hot fixing i'm done like patching and like doing all this monkey patching myself uh you know i'm i'm you know i i i want less ops at home you know less ups the least you know amount of operational stuff i have to do the better and so kubernetes you know is great at that it'll take care of stuff for you dead pods deployments roll outs scale up scale down horizontal scaling whatever and so i was like okay i'm done with this i'm i'm so done let me let me take a step back make sure my services are running and think about this and so you know over the last um last it was the last week i really took a step back and thought about and thought about okay what what what would you know a next version of my cluster look like you know what would it look like would it look the same you know and i i removed all all assumptions that i had and all preconceived ideas about what it should be like or even what i've even done in the past or what i've taught in the past hey pc geek dude how's it going man how you doing good to see you dude dude thank you so much resub tier one seven months spot's still not working uh we'll see i'm blaming it on uh i'm blaming it on uh the deployment just being a little bit slower but dude pz geek thanks tim for all the uh thanks tim for all that you do and for the community dude thank you so much i appreciate what you do man dude every day every day pc geeks in in in discord helping out lending a hand always has a positive attitude uh thank you so much dude really appreciate it uh appreciate all you do too man really appreciate it dude i load what's up man speaking of mods dude i load two man dude tier one seven months dude seven months straight dude thank you guys so much really appreciate i loot know the moderator discord uh fantastic wealth of knowledge um and and and docker and a lot of stuff uh great help to the community too but both of you guys dude thank you so much thanks for the continued support i appreciate it um so um so i so i i thought about like what does what would my kubernetes cluster be if i didn't really you know um already have ideas of what it should be and i was like okay do do i need rancher i even i even put rancher on the table on the chopping block i was like do i want rancher you know because i was starting all over from scratch i mean i had my services i knew how they should run so i i had a lot of manifest for um a lot of my deployments and i knew how that should work but it was like what's gonna orchestrate kubernetes do i need an orchestrator for kubernetes is it going to be portainer is it going to be rancher and i start from the beginning and then i then i started to think about okay well you know coupe control commands aren't that great oh oh yeah so i i'm i'm pretty familiar with coop control i can deploy my whole entire cluster with coop control commands for the most part i have manifest now for a lot of things so you know all i have to do is go into one folder and say cube control deploy you know all my secrets all my deployments all my pods all my uh persistent storage volumes you know a couple of helm commands and i can have my my cluster up and going i'm over simplifying but it's it's almost to that point probably take me 45 minutes you know of just you know verifying and uh so then i was like hmm you know i don't like um you know i i kind of want a little more visibility into my cluster i want to you know kind of be able i want a ui at some point i'm not going to run kube control commands from my phone when i'm on the run you know i could vpn in and sure get a get a client and you know run my kube control commands and make sure everything's working but i wanted a dashboard that's when i started looking at kubernetes dashboard and don't get me wrong that's fantastic it's fantastic um but you know i wanted something where i didn't have to uh uh the way it's set up right now i should say this is that it basically when you run the kubernetes dashboard you're running in god mode you're running it as basically root because you have to use a token that has access to everything in kubernetes so i was like i really don't want that hanging out all the time then i thought about well i could just spin it up and you know uh coupe control you know port forward the ports back to me but i'm like well i can't do that on mobile um and then that's when that's when orchestrators came back on the on the on the docket or on the table where i was like okay well they're not off the table i don't know the chopping block do they stay on the table when they're i don't know anyways then i was like okay um i think i'm gonna go with rancher again because you know i'm a huge fan of rancher and i built up this knowledge of rancher and i i enjoy it i really do but then it was like okay now i'm gonna you know what how am i gonna how am i gonna build my kubernetes cluster am i gonna am i gonna go and i'll show you some stuff here in a little bit we're gonna map some stuff out i'm gonna show you how it works um you know i started thinking about um you know uh do i do i use k3s again do i use rke do i use katy k3d you know how do i how do i build this you know if i'm i'm going to do it from scratch and then do i take the rancher recommended approach where rancher has its own cluster you know these are all things that were on the table and so um you know i these are things i had to seriously consider um and uh and and i i think i found something that's that's that's pretty stable um and and and to take it even a little bit further you know i you know my sequel is still on the chopping block too you know am i going to use an external database b you know versus um an internal database like uh you know hosted or internal at cd um because way back when i did this for the first time you know uh et cd wasn't even um on the it wasn't even available you know uh the mysql version was still alpha you know and now that's deprecated uh except for single nodes i think they say and and now it's you know now it's uh embedded at cd it's now no longer alpha it's it's fully supported so i'm like okay maybe i can do this so anyways before we get into that real quick i want to read comments i want to i want to read comments and i'll dive in i'll show you my cluster i'll show you how things how i have things arranged um and we'll kind of diagram you can help me diagram poke holes at my cluster i already know where you know pieces that aren't fault tolerant but i'd love to hear your feedback too so i'm going to go through these really quick um just because you wrote them so i want to make sure i address them takano you're here first how's it going how you doing man jaymo reese hey you were you were almost first you were almost first farhan how's it going yellow i like it yellow hey hey uh reese you came back yeah i did come back yeah so and a couple people mentioned that on my youtube videos today like hey he's back i'm like wow i didn't really notice that people noticed that i didn't post a video for a while so i appreciate you noticing i have been here on stream on saturdays um but videos had to kind of take a backseat for all the things i got going on and i mean we're all busy so no excuses um but i i did um i didn't have to put things on pause uh but they're not now they're back on play i guess is that how it works uh just had to take care a lot of stuff homework and then cluster stuff and a lot of other stuff going on plus it didn't feel well so i was like and you can kind of tell in my last video it's it's it's not the normal me but i was so drained but i told myself nope i i absolutely have to do this uh it's the only way to get back on the horse is just to do it and so yeah uh chicano got coin mech up usb timer board enjoy six nice dude hopefully gonna start finishing up this machine yeah nice so takano's building a little uh uh arcade game that he can plug money into when he wants to play video games he can plug money into it and it's a good way to kind of save uh dream traveling yo what's up how's it going uh it's kind of also thinking of buying a vending machine a nice man you're going all out you're going all out uh sly humans sup what's up man uh toffian how's it going hopefully you're not working on that hopefully you're not working on that that's not good that's not good tecano i already have a place where i'm thinking of putting it drinks only uh drinks only but it's only going to be uh at a location of people who work out in the sun nice oh you're gonna make money off this vending machine i like it i thought you meant just for home for fun nice uh all right uh jorge how's it going pasada uh i thought you might find that enjoyable didn't see the link but i'll i'll ping me in discord uh the timer increase yeah that's where i was trying to deploy my code i don't even know if it's out there now i don't even know if it's out there but um you can try it if not i'll have to debug it later so i do apologize um so you probably can't change the lights until i fix that but uh next time i should i should have checked a little bit earlier but that's the last thing i think about is my bot up because most of the time the answer is yes um intellectual play space how's it going good to see you uh sigour how's it going what's new yeah whole lot new whole lot new we're going to talk about it here in a little bit uh hype train subs thank you so much skill point uh hi i have no idea how long you've been trying to catch me streaming hey you you caught me uh saturday of three is almost it's almost always going to be this one's going to be available other ones will be available as as i become available it's just it's getting real tough to do them at night too and i think as soon as things start opening back up and you know work arrangement change it might be even a little more difficult i don't know how i did it a year ago um but yeah yo hey everyone good afternoon anywho uh holy mark how's it going dude uh good afternoon who need a bot fire yeah totally totally man uh you can you can razz me about my bot like my code should be up and working kubernetes especially like since this is my youtube job you know making sure services stay up but i'll figure it out this is nothing to do with kubernetes this is my own code and i think my token expired and i was like oh i didn't write um the best code to refresh my token so i tried to force it manually but anyways uh farhan how's it going starting fresh with kubernetes what do you recommend uh we're gonna talk about that so hang tight uh intellectual kate's uh intellectual play space kate's on a vm are on proxmox uh cts um i go with vms personally but uh if you can get it working in uh a container like lxc yeah absolutely go for it uh it's it's totally up to you i personally do vms because there i can move them around really easily um as you can on uh lxc containers yeah go go with what works i haven't done a container uh on an lxc container but uh i think some people have i yeah um but i i'm i could be wrong uh resect trying to add a new node to rke and every time our key up it doesn't work i haven't done our plain old rke stuff yeah because ncd needs quorum and i only have two server notes wow you should have quorum with at least one your first one the one you init with should be and that's we'll talk about that here in a little bit uh techno 2 why don't you use embedded ncd maybe i do hang tight we'll talk about it uh this username is this username is inexistent it is existent now because you grabbed it hey how's it going um pcgeek dude thanks again i love thanks again uh ariel mayo choo choo yeah you guys got the train piece of kitchen hey johnny enough hairdo day yeah i uh my hair is a little tall so my my wife cut it uh just a little bit ago i made it a little bit tall and curly um we we have something to do a little bit later today we're gonna hang out with friends it's like 60 something and sunny so we're gonna go to a little cookout here in a little bit um and so thought i'll go with hair uh it's it's funny because i last time i saw most people i didn't have hair i just always shaved it off so it's it's funny to see people's reaction when i walk in but yeah uh uh yeah uh see gore uh best discord around by far hey i agree i agree and it's it's the people who make it up it's the mods who make it happen it's the people who contribute content i totally agree i totally agree um yeah yeah i totally agree uh great discord full of ton of help ton of help for sure um farhan uh i have started with your channel on proxmox but uh all started starting fresh with kubernetes what do you recommend we're getting there farhan hold in hold in there uh ilud kube on proxmox is always a good way to go then you can spin up another cluster and replace the first if you want to yeah yeah totally agree uh pc geek juice ssh yeah i i assume that's a mobile ssh client i should check it out um reese i use coop control quite a bit should i say i've said coup control ever since i discovered it i know a lot of people say cube ctl and i've already talked about that many times but uh uh coop control a bit and i like having rancher for new deployments yeah i i totally agree i like rancher for the the the piece and stuff like that sig kill dude thanks for the sub appreciate it see sig yeah sig kill sig kill term all right all right i like it uh has subscribed prime sub thank you so much appreciate it we're gonna jump in here in here in a second um but yeah reese i i totally agree um i've uh i i've used tons of coop control commands and then i when i switched to rancher i was like i don't need control anymore now i kind of like it only because it helps me um deploy things really fast it helps me repeat it really fast and i can version control and and commit uh my manifest to source control which helps me do it again later or you know destroy it tear it down build it back up but uh i love rancher for the visibility i get and for the additional apps and stuff like that and just to be able to just navigate around my my cluster exec into pods look at logs really easily it's it's super fantastic and um anyways i i got some more stuff coming too uh so uh rhys etcd is great i i i agree i agree well i'm giving stuff away i'll show you uh i agree uh pc geek just got financed deal with my new rv done today nice dude bunch of stuff needed for uh needed for it came two including portable dump tank nice dude nice yeah you bought an rv it looks awesome dude i'm super jelly man when i see it i'm just like maybe i need one i mean i i i'm you know yeah i i think at some point i would love to because we have dogs too and so when we travel it's like i gotta leave the dog somewhere i'd love to bring him with uh and that would that would give us the ability to uh but yeah it looks awesome man i'm excited for you eromeo hi can you tell me which ingress do you prefer ranch or aggressive trafficking guest route i'm following your documentation for github two three yeah yeah which should i choose and look for a simple way to expose my services yeah so um real quick and i'll talk about this here in my diagram you know i i used um i i used to use rancher ingress um until i started using k3s and then um you know which ships with a different version of traffic so then i end up disabling the load balancer and the ingress from k3s and then bringing my own traffic and then on top of that just recently i switched to ingress route and it kind of seems like i'm going backwards um because you know uh traffic ingress if you create one it'll it'll um do some stuff for you automatically but with ingra ingress route it's more declarative like you have to define your routes per route um but i kind of like that i kind of like that because you know anytime i spin up a new service i have three pieces i need to deploy it's it's going to be my well at least three uh well not at least three if i have a service that needs to be exposed um or needs an ingress i'll say that i need at least three you know it's going to be uh my service uh my deployment and then my ingress and i kind of like that um and and maybe you know it it see that was one of the first features that they supported but i you know it's it's a lot easier for me to know that something is exposed if i deploy that too rather than playing the game of you know hey i deployed this thing with an ingress um is it is it alive i mean i could do the coop control stuff to go and check and stuff like that um but either way i i think the ingress route is not the more modern approach that traffic is going after just a plain old ingress is is the is the way they want you to do it or the way they support now i i've kind of you know i've done both i i'm settling on ingress rep for now um changing it later is is super easy um but i you know i i just wanted to be able to bundle all of my manifest together and say here's everything i need to deploy this website boom it's it's right there in my kubernetes folder um lots of comments i'm gonna i'm gonna get caught up real quick and then i'm going to show you my cluster and we're going to talk about some stuff and i'm going to talk about some of the i guess reasons why i made these choices and you feel free to poke holes at choices i made because you know like i said i thought about it for a long time but then i implemented it really fast so i rebuilt my whole entire cluster in about six hours seven hours i think one day it was uh it was monday actually um and so i did it i had the day off and i was like today's the day so anyways um sigour takana you just need a soft serve machine too to round out the mix yeah he does yeah an ice cream yeah ice cream machine for sure reese i wasted my money uh to test for tim oh dude yeah i'll figure out my bot i would drill into it now but that's boring stuff it's boring stuff and i know it's it's code i wrote or my tokens or something's going on uh so i'll figure it out so i do i do apologize i should check that stuff earlier um yeah so the arcade machine is a home project um which no money will be made on it the vending machine will be used for profit i like it i like it farhan i just spinned up truenas scale on via vm on proxmox minutes ago but i didn't have similar drives to make a zfs pool yeah you're going to want that you're going to want that uh restech but if you don't have three server nodes you don't have aj yeah that's absolutely it's one of the things it's one of the things uh that it's it's i guess one of the cons i don't know if that's a pro icon for me it's a it's a con it's just one more machine to worry about uh takano 96 degrees here yeah it's it's warm i think you're in the southwest so that's that's too warm for me i mean even like i think we're going to have mid 70s uh next week here in minnesota in minneapolis and i'm kind of like i'm not ready for that yet you know i enjoy 50s and 60s i i i thoroughly enjoy it i wish i had more 50s and 60s in my life because here it's negative 20 to 90 and humid you know it's um our our in-between seasons are are short so spring and and fall are really short here because it goes straight from it feels like almost winter to summer get a you know we get about a month of like the most perfect weather ever and then the other months are kind of like i don't know if i want to go outside um so let me see uh give your points back didn't work reese i'll see what i can do i'll see what i can do i i apologize uh my bots whatever it's not working i'll see what i can do i don't i don't know if i can refund points but if if i can in the in the twitch manager today i'll refund any points that people spend today i need to write a note to myself too because i've never had to do that because my bot is is definitely down um refund points uh remind me i just should say hey google remind me at 10 o'clock tonight because that's when i take care of business um so let's see uh tecano uh when we talked to the owner they wanted two machines one of them for the north side one for the south side dairy farm nice uh cube nav is apparently a mobile desktop case management app disclaimer i have not tried it yeah the hottest it's been here in arizona is 121 degrees that's hot um yeah so yeah i i yeah i i've tried a lot of other kubernetes i've tried a lot of other kubernetes like dashboards i started to look and this was part of my journey too um i'll just switch over to here now i'll show you my desktop you guys are going to see a little bit behind the scenes i'll try not to show things that like give away uh i mean i don't have anything that's super yeah here's me again um i i'm not used to pressing that button um but you know i don't have too many secrets so i you know i share a lot of stuff a lot of the stuff a lot of my videos too are i just do it in production because i don't have a ton of secrets um hey read write read write execute rob rating 45 dude thank you man dude thank you so much thanks for the uh thanks for the raid big raid dude thank you so much i appreciate it man how's it going how's it going good to see you dude uh appreciate it um hope you're doing well i hope your stream is good welcome everyone that just came from uh i think it's read write execute rob i'm gonna go with rob how's it going man uh hope you enjoyed the stream um today uh you know usually we we talk about all kinds of tech here a lot of home lab stuff a lot of home lab stuff people like to build things and self-host things at home and build servers and build all kinds of services at home one of those services is is some people enjoy or don't enjoy is kubernetes uh it's it's big it's complicated it's it's sometimes messy but uh when you get it right it's it's pretty awesome over the last couple of weeks i did not get it right well i got it right for about a year and then the last couple of weeks i tried to make one of my database aha which then i had a lot of drift and i destroyed my cluster uh for for for lack of a better term and so slowly over time my databases were getting out of sync and um which in fact was then affecting my cluster because my cluster depended on the external database um for its state and so that's that's one of the the the i don't know if it's a pro icon but one of the differences if you use k3s with an external database that becomes that becomes your source of state he's like your one he's your one for quorum if nothing's up your database is up as soon as one comes up it's like oh you're you're you know you're in charge right now the database always always maintains that state and so when you switch over uh to another uh internal database here's my here's this is my live cluster right now so when you switch over to a different um type of database like an internal etcd cluster now the database is hosted on each one of those nodes and so i'll show you really quick uh so uh which cluster is this you guys are really going behind the curtains now um so that's kind of what i decided i i kind of wanted in in my my future cluster like i i had to make a quick decision but i had molded this over uh for many weeks in my head and you know a lot of people have asked me and and should i do it in bed at ncd and i was like yeah why not but you know i i always appreciate when someone has you know personal experience or a personal recommendation or or has had experience with that and i i didn't i wasn't able to give people any kind of recommendation or thoughts or any kind of feedback on what that looks like so i was like you know what i'm going to do this and rancher seems to they do that by default now so if you don't specify anything you're getting it embedded at cd which is pretty cool so embedded at cd is a database that's housed on within every single node that you have right and so every single node has a copy of this database and i in discord i listed some pros and cons um but um wanna i'll list some of the i guess cons first and they're not really conscious just things to be aware of you know you need at least three nodes um well yeah i'll go with three i mean one one would win but you need at least three nodes you know those nodes um now that they're housing a database and they all have a copy of the database they need to you know need a little bit more ram now and now they need a little bit more disk space and they need to be on you know faster i'll say faster hard drives um than than what you would traditionally need for kubernetes um and then there's going to be a little bit more io and that go plays into the the disk drives while they need to be a little bit more performant and there's going to be a little bit more network traffic too because they're all talking to each other not that they didn't before but now they're not talking to the database and so if you think about it like at first i was like yeah embedded lcd sounds like the perfect solution for raspberry pi seems like a perfect solution yeah why edge edge cluster why would you not want this you know um but you actually don't want that on a raspberry pi unless you have something better than a micro sd there because it's it's they're super chatty they're doing diffs they're uh getting updates to their database they're sending updates to uh the the the master and this in this cluster whoever the the the whoever's leading i guess um quorum the elected leader i guess i should say is probably a better term so um so probably not a great idea for for raspberry pi's unless you have ssds unless you're like going all out like jeff gearling you know if you're on the jeff gearling boat and you're and you're totally putting ssds on your on your raspberry pi's absolutely this would be fantastic if you're a micro sd person in raspberry pi um the the the role of server for that is probably not a good idea role of agent totally fine role of agent totally fine hey uh etd is expensive even on big iron kate's clusters running into problems um with ncd perf there's a concern that crds will make it worse ooh interesting interesting good to know good to know uh dude thanks for the bits pear pear paired lund uh thank you for guiding me in the first step in the kubernetes world hey no problem at all thank you so much thanks for thanks for being here thanks for the bits i really appreciate it um yeah thanks for thanks for stopping in too what about nfs connected nas for the pies uh possibly possibly but you gotta think like you know those databases um i mean if those pies could communicate in a relatively um well let's think about this you know what's what's you know um the the those databases need to be able to perform lots of transactions really fast and so local write is going to be the fastest and then on top of that local right local read write um you know on ssd is going to be even faster and so you can you can you know you can bundle everything on local read write into any type of drive that's connected to there physically connected to there once you start going over the network i'm not sure i'm not sure you you might your mileage may vary so i haven't done it so anyways so then i started thinking about okay i'm i'm gonna do this i'm gonna go embed it at cd you know and uh uh a quick tip two i'll have all of this once i document this i'll have all of this documented and i will have all the commands so you can do it yourself too so i ended up switching to embedded cd i was like okay well you know what i'm going to go embed it at cd let's go for this let's do this because i don't want to have to think about how i'm going to make my database ha again and so what i ended up doing was creating my initial cluster so if you're using if you're using rancher you know rancher likes to be installed on an existing kubernetes cluster so if you're using a single node this does not apply um but if you're using you know anything more than anything other than a single node you need a kubernetes cluster first and so that's what i did i was like okay well let's do k3s let's spin up three servers and so if you look at uh wow i am am i on the right one i'm like wow i'm like am i looking at the right thing it's this is i i just made a video on the new dashboard and i you think i'd know this um so anyways um so this is this is my new cluster uh so this is my new i have two clusters if you didn't didn't notice there this is my new kubernetes cluster and so my new kubernetes cluster the first one i bootstrapped was based on k3s and so on k3s i initialized one with that cd and then i initialized two more with that cd pointed them back at the first one again i'll have all these commands later and then after that i joined additional agents to this cluster and because because rancher does recommend that your uh it runs on its own cluster and i know people have said to me this a lot of times like you know should we really install stuff on the local cluster or should we have its own separate cluster you know i've been in the camp for a long time and i still am that like i didn't want to have six additional servers just to run rancher you know i get it if you're in a if this is a production environment a high use production environment you you probably should um you know at home should you i'm you know that's that's debatable and so for a long time i ran it without but i was like you know what when people ask me this question like what's the difference um you know i want to have an answer for it so i was like i you know i i have some resources laying around i have a i have a couple of cores i could use to do this so i figured i'd go the best practice route as far as i could go with rancher and kubernetes so that like then you know i i can say that i've done it and i understand it so when people ask questions at least i'm a little bit informed hey thank you so much tech freak too dude thanks for the bits appreciate it uh hello tim just wanted to say hey and love your videos thank you uh keep up the awesomeness dude appreciate it thank you so much thank you thank you um let's see uh uh piece of geek you could run uh rancher and docker right and have it manage the k3s cluster yeah absolutely you can yeah yeah good call good call you could run the rancher the docker version of rancher and then spin up your existing cluster from there so yeah good call pc geek you could you could you could uh definitely so a lot of legos here and you can build you know a lot of building blocks and that's that's a really good one that's a really good call out is that if you didn't want um an h a rancher cluster um only the rancher cluster you could use the docker version to spin that up only run rancher on that cluster and then create a new cluster uh of that is h a and so and and and that's something i learned too and something i've i've kind of known is that like um to to to have rancher bha for the most part rancher itself the service all you need is a cluster that's h a and then rancher automatically becomes aj because it's running where it needs to run um you know it's it has ingresses it can bounce around in and do all the things that kubernetes is good at doing and rancher is pretty good at doing too um so yeah yeah absolutely good point good point uh pc geek you could have ran a docker version of rancher installed and then create a new cluster that's totally ha and then just have you know one node a single node rancher install and then have an h a cluster yeah totally yeah good good point um and so so my my cluster i ended up spinning up you know three servers three agents uh and then joining them and creating this cluster once i had this cluster created i ended up then um installing uh rancher on top of it and so you know i was it was a little bit confusing because they they don't really you know they there's not a ton of guidance uh on what you do after that you know uh because because because you can do a lot of different things uh uh with rancher and kubernetes so that's so so that's what i ended up doing was creating my cluster installing rancher on it and then i was like okay well where do where do all my user workloads go right because i want to be able to separate uh my rancho cluster from my user cluster so i'm gonna that's the terminology i'm gonna use rancher cluster is the one called local user cluster is my own stuff and what what you know and and everything that i'm gonna run that's has nothing to do with rancher and so if you see here the this is my my rancher cluster the one called local uh aku these are all three uh these are my ncd host and you can see it right here at cd um and then these ones called archie are my my agents so the three agents that are gonna run the rancher workloads what are the rancho workloads pretty much only rancher that's okay i i'm like you know i'm gonna go i'm gonna do what rancher says and create my own network for it and there's some benefits to it too um so once this was installed then i was like okay well how do i you know how do i create my next cluster let's switch over to the cluster manager i'm going to try to not to give too many secrets away but this is probably what you guys are are seeing so now you have your rancher cluster um you know how do you create a new cluster it's it's pretty easy it's pretty easy so i went into the new cluster area uh clusters oh global add a cluster and then uh you know so this is gonna be my user cluster all you know all i had to do this this took me a little bit to figure out and i i won't go into it i tried to reuse some nodes and yeah anyways um so i you end up basically just going in and creating your new cluster my new cluster you know whatever you want to name it uh if i can spell or ty i know how to spell it's typing that's the the tough one uh and i think these need to be hyphenated who knows anyways but then you have kind of the same choices for your new cluster right you have the same choices and you need the same uh you need the same roles for your new cluster and what are those roles well you're going to need you know you're going to need a kubernetes control plane you're going to need a database which will be at cd uh and then you're going to need some workers and so that's kind of what this cluster name i knew it i knew it and it has to be half in it i know why because this is yaml behind the scenes so i i should know that everything's ammo this is just a form to fill out yaml um but you know what what we end up doing then is um uh we end up oh i press back anyways uh we end up choosing uh roles for this hello from shakopee all right you're not too far away how's it going hello from minneapolis um on on your new cluster i just totally went outside of here um on here's the command on your new cluster you're going to have to choose these these same kind of roles and so the first thing i did was actually uh create my my my servers for my new cluster um all with fcd and control plane so i created three of those and then i did the same thing for the worker nodes and so this command right here this command right here all it's going to end up doing is um really uh spin up a docker container and spin up that cluster for you it does take some time even on the high performing server that i have it takes a little bit of time in in in and five minutes is is okay so if you're wondering like hey uh you know i spun up these clusters um it's taking five minutes what's going on that's totally fine and you know i did the same thing where i'm actually exacting in you know i'm remoting in those machines show me the docker logs looking at docker logs i'm like what is it doing um but it has a lot of built-in checks and so i i would recommend that when you do this um if you do spin up your second cluster create one server and one worker node and let them be let them be for about five minutes um and if you're really curious you could remote into those machines run docker logs on those and you'll see you'll see the container spinning up there's some that check there's some that wait then it'll spin up as soon as you see pre-polling pre-pulling kubernetes images you know you're in a good spot if you don't see that for 10 minutes then you know that something might be wrong um but anyways so then you end up let me delete this then you end up getting your new cluster i'm going to switch back to the the the new cluster manager because you know because i just made a video on it and uh it's pretty awesome i would that would really suck if i just deleted my existing cluster that's the problem of doing things in production um so anyways um so so then once you have your two clusters spun up uh let's see uh software development in 2020 everything is the ammo so so true yeah for sure uh kate's is turning into us all into yamo programmers yeah for sure is the ammo programmer equals true yeah totally totally yeah yamo uh you know there's a there's a yeah yeah i you know i think i think it was kubernetes it's the first time i i had to really you know enjoy yaml yeah you got to get out your t-square right you got to get out your t-square and then you're all good because you just put your t-square on your monitor you move it over every time you tab or space whatever you use won't talk about that you know then then then you're good um and and i highly recommend if you if you are doing a ton of yammel and you use vs code uh red hat has an awesome extension it's a yemo validator um as if you know as if you needed another validator but but uh red hat has a great extension on vs code it's it's a yaml validator it will tell you it'll actually inspect um those name spaces to see if you have the right properties which can uh work sometimes but the best part about it is it'll tell you if you have invalid ammo but but if it understands though if those manifests are public like the types for it'll actually validate it too so it's pretty cool um you have to use spaces in yaml no tabs all right thank you uh that's a good call i personally don't like the new cluster explorer but we'll probably get used to it yeah it's uh it's it's different it's different um the reason why i like it is because it's a it's a little more compact when i'm looking at stuff you know i don't have to scroll as far i don't know but i feel like i should be getting used to this mainly because uh eventually the old one's going to go you know ranchers making their investments into this ui they're supporting both how long will the old ui be around who knows um and i don't get me wrong i love the old ui it's the first thing that drew me to rancher because i was like oh finally you know one i get a ui to to drive kubernetes and two it looks pretty good and it has dark mode so so anyways so now i have two clusters you know i have my my local cluster that's only running rancher and all the rancho related stuff and then i have my new cluster which is all of my user workloads so this is where all of my user workloads stuff runs so uh websites web services um agents you name it all my stuff's running here all even my dynamic dns stuff you can see what i have 76 resources i think about 40 or 50 of those are services i have 36 deployments but some of my things are services too so you know and and at the end of the day it's not uh it's this is a lot more i i have a lot of headroom as you can see i have a ton of headroom you know i i could you know use up to 440 pods i think it could probably do a little bit more than that um but that's what you'll find with kubernetes too if you start to build out nodes like you know um a lot of them aren't really busy depending on the type of workload right if it's going to be something that's going to be uh crunching data processing data converting video anything like that obviously is going to use a lot but for you know a lot of like enterprise workloads if you think about those a lot of those are you know web services or middleware services or lightweight websites not going to use a ton and that's uh it's it's it becomes really apparent when you have a dashboard like this and so you know after i had this going and after i had it working um then came the challenge of okay now i have two clusters where do i install and configure anything you know and what do i mean by anything so um you guys you guys probably know um that you know i i i use traffic you know i use metal lb uh you know and and and before that was pretty easy when you have one cluster you install on the only cluster you have you know and and how do you coup control what does that look like you have two clusters now where how how do you make sure you don't do the right the wrong thing to the right wrong cluster or the right thing to the wrong cluster and so those are things you'll have to you know kind of figure out too um but i will say that if you're using kube control you have a you have a easy way to switch context and so now you'll have multiple contacts i would show you my coupe control but it has all of my token secrets and everything and you know i trust you all but i don't trust anyone else who would watch this video later i only trust the people watching this live but later on when it's a vod i don't trust them i'm kidding i'm kidding um but you know just in general i'm not going to show you my my coop config file because it has my real tokens but if you do inspect your coop config file hey matter matter hacks thanks for fall appreciate it if you do inspect your coop config file you'll see now you have multiple servers and you'll have multiple clusters and it's actually pretty cool and you'll see in there what your default context is what what's my default context is it my local cluster you know is is it the the rancher local cluster or is it my user cluster which i you know called it very creatively cluster one it's it's about as creative as it gets with cluster names some reason i give servers awesome names for clusters i don't i don't servers get great names um and so uh oh you have uh oh you're using kate's 20 plus could you do a dual stack ipv4 with a new api you you uh you could do a dual stack ipv4 ipv6 using the new api interesting yeah i even done i haven't done anything with ipv6 uh i i i i even honestly would have to look into what you're talking about because i'm not too familiar with it but yeah that was one thing um so while we're talking about the version of k3s yes i'm using the latest and greatest k3s so i think it's uh 120.4 120 something and um you know this this this all was kind of serendipitous that it happened because just recently rancher started supporting you know kubernetes uh uh 1.20 right prior to it only supported kubernetes 1.19 which is k3s.119 um and then k3s revved and then rancher realized it didn't work and then so it broke rancher and then so i you know i have things in documentation where i'm like use the previous version i should probably remove those now so if you see that in github feel free to open a poll request to remove those uh to to specify the exact version if not it's in my notes um github so i don't forget uh bot points those are the three things i need to do um and so and so yes i i'm running the latest the latest k3s and i'm running the latest rancher i think 257 it's it's all working really uh really good uh hey techno tim have you any more uh slack bot work i'm using your guide as we speak in looking into taking a bit further um nah i haven't worked on my slide box in a while slat slack bot in a while uh yeah feel free to yeah if you it's open source if you um see something there needs to be fixed or want to add some stuff to it yeah absolutely i think the only thing i do is patch it for security that's one thing i do do even on my older repos if i'm not adding features i usually update node modules and just make sure you know anytime there's a dependent bot alert from github which is their security scanner um i usually take care of those pretty quickly uh but yeah thank you peace keep it's still not working oh oh my boss is not working oh yeah love dependable i do too i do too i i wish i could spin up a demand bot locally i wish i could have a self-hosted dependable so it is so dependable yeah that's right so if you're not familiar with depend about i was talking about a little bit it's basically a bot that watches your github repos and will um see look at your manifest for all of your dependencies and then check that list of dependencies against a known database of uh security vulnerabilities and then automatically open a pull request on your github repo so all you need to do is say accept it's really cool and uh microsoft ended up buying it our github which microsoft owns you know like they're buying up everything like discord um ended up uh github bought dependabot so now it's you know part of github which is which is really awesome it works on both public i think and private repos too so it's it's really awesome um and so if you have a github repo yeah i highly recommend add dependent bot it'll scan your scan your repo and open up pull requests for you for security vulnerability so you don't have to keep track of them and you just click merge you know you'd want it to run your ci pipelines and stuff like that if you have those but yeah or you know work work i forget what they call them and get help um but yeah uh so so anyways back to my uh real quick dimetabot also tries to work out if the change will break your code uh based on other pro that's really awesome so they're using some machine learning i like it i like it i like it uh jumping mush do you have any thoughts on monitoring kubernetes clusters i've got my cluster up thanks to your videos but would like to know if it's healthy absolutely yeah i'll have some real soon on that real soon so we'll talk we'll talk about monitoring uh and alerting really really soon so yeah nailed it uh but soon yeah absolutely i'm in the same boat uh you know you know if a tree falls in the woods does it make a sound that's kind of what it used to be like with my kubernetes cluster because you know i you arguably you could say that about my bot right now uh but yeah you know it's if you're looking at kubernetes at a dashboard you can kind of identify when things go wrong but if you're not looking at it how do you know and so that's absolutely uh where something like that comes into play and i'll have something very soon very soon on it and sooner than you think uh but yeah monitoring and alerting coming real soon and i think i hinted at that in today's video so so absolutely um so this is kind of my newer i know it i know you know and maybe i built it up but this is kind of my newer infrastructure now i'll even show you my proxmox though so we're going totally behind the scenes on everything that i have running hopefully i don't have any ah they're all they're all obfuscated they're all weird names um so the way that i have my proxmox uh uh cluster set up so i do have a proxmox cluster um is is i have three nodes and so i have three nodes now um one node is just kind of temporarily it's my intel noc i was doing some kind of exploratory work on you know what is it what does it mean to have an intel noc has a kind of a low powered proxmox node works out pretty good um but as you can see um you know i'm starting to starting to to outgrow i guess um you know some of the uh some of the resources uh that i think it's demanding um wouldn't it be cool if there was an alert for bots not working yeah totally yeah hello yeah would be cool i i totally should i i know and this could be solved with a health check this could totally be solved without check i just i didn't write a health check for that for some of my personal stuff i totally should um but anyways uh so i'm i'm only talking about this piece real quick because it's it's kind of um it's kind of good to think about where your nodes live within your cluster if you just have one big server this does not apply uh but if you have multiple you know uh cl if you have multiple server nodes in your uh uh proxmox cluster or your hypervisor cluster you kind of want to have one of each type of these nodes uh living somewhere else and so that's kind of what i've ended up doing was you know i have you know one of my one of my uh um what do i have one node so i have three fcd nodes uh for for my first cluster one of those lives on each one of these servers and then i have three workers uh for that first cluster my rancho cluster one of those nodes live on each of these servers and the same goes for my user cluster as well so on my user cluster i have the same idea three at cd nodes with control plane so when i say cd i i use it generically now to talk about just basically kubernetes you know operations so ncd and control plane so i lump those two together i don't separate out uh etcd and control plane you can but then you need three of those as well so anyways um so so my my kubernetes servers uh then for my user cluster are spread out uh uh one of each on each of these servers and then the same goes for my workers for that cluster so i have one of each on each of those servers and for some reason i have a fourth one i have no idea why i created four or four nodes for my my user my user cluster but i did i have no idea why uh and i think it was maybe just so in case i took everything down and then there were two left on one i could still reboot one i don't know why i would want to do that um but but anyways i do and it it doesn't really matter you can you can create agents on the fly uh add more agents take them away as as much as you'd like to um and and so i can have as little as many as i want so that's kind of how my my my proxmox is arranged um and so you know i i wanted to kind of you know a lot of people have asked you know and i i just talked about it but a lot of people have asked like you know what does it look like and so i i gave you you know verbally what it looks like but i'm you know i'm personally i'm more of a visual person so i i kind of wanted to just kind of draw or map out um you know what this looks like because i wanted to share with you i had people in discord say what did your new cluster look like and i typed it out i'm like you know but but where do things live and i think you know it's it's easier for me to make pictures about where things live uh rather than you know talk about them because it gets it gets a little bit confusing you know i talked about that a little bit earlier we talked about helm and we talked about we talked about helm and we talked about or any helmet stall like a middle lb or any you know traffic install using helm it gets a little bit it gets a little bit confusing on where those things live and so um and how you deploy them i talked a little bit about you know how you change your context that's something you'll want to learn for sure absolutely uh you want to learn how to change your contacts because you don't want to do the wrong thing to you know the the the you want to don't want to this is confusing you don't want to do the right thing to the wrong cluster and so i would highly recommend if you do end up uh changing your kubeconfig to always point at your user cluster why because you're most likely not going to do anything to that rancher cluster anymore once you create it i mean there might be few things but it's going to be the exception versus the rule and so the rule in general for me is i set my in my coop config i set my default context or my currently selected context to my user cluster and so you you'll if not you might end up deploying some stuff to it easy to roll back but if you end up deleting stuff that's you know destructive things are hard to to to to fix i mean sometimes it's easy with the manifest but you know if you if you go and do a helm delete this you know deployment that might not be so good so anyways some people have asked so i figured you know why not start like documenting kind of what my cluster looks like now um this is me just kind of spitballing i don't know what shapes i should use um i don't know why nvidia is coming up i know why shortcut or run video um oh you stole my shortcut and also i was using a mac shortcut anyways i'm so used to using my mac during the day that i'm like doing mac shortcuts now which are obviously going to work um but i wanted to kind of document it and i want you to absolutely like if you see holes in this cluster i want to talk about i want to talk about like any you know deficiencies that you see um and so i'm just going to put i'm going to put some boxes here and then and then we can we can kind of react to i should probably copy these all right i'm not not on a mac i'm on windows so i can use the same thing do we do we use boxes for servers i think we should use boxes for servers we don't have to oh i'm going to get totally hung up on this and i think uh penguin airlines the other day was uh maybe his last stream when i said hey this is what my cluster looks like now and i scribbled down on a piece of paper i actually built what i scribbled down um you know he was like well you know scribbling it down is like you know almost uh almost like not having it at all and i was like ooh that was that's kind of that's kind of tough but i uh and my response to him was always like well sometimes i get too lost in the tooling and how the tooling works that i you know i lose uh the intention of diagramming something which is documenting something and getting these ideas out of my head so anyways um so anyways i'm i'm going to try to ignore spaces and and and tooling stuff and just just go forward yeah i don't want to i don't want to struggle with tools like you know tool how you create a box and how you insert text is does not help you uh get thoughts out of your head quick enough at least for me so anyways yeah like this like how do i put a label all right i'm all right google so we're gonna go with you we're gonna do this so anyways um i'm gonna say um should i should i use my real server names i kind of want to and then maybe i'll maybe i'll just uh maybe i'll just yeah i think it will use real server names and then i'll just label them later uh oh talking um uh creating a custom windows iso nice like your videos thanks thanks jaya uh cool sounds fun you could use draw.io yeah cool yeah i i i looked at lucidcharts and i looked at drawda io i does draw.i o auto correct stuff for you too i think it does because i think that's like uh almost like a machine learning type of uh drawing tool i think last time i looked at it i don't remember you could like draw something and it'll like create a banana for you or draw kind of a star and it'll create a really nice star for you i could play with that game all day it's like it's like drawing with friends except for drawing with this computer by myself very weird but yeah i should check it out um uh oh talking about ms buying up everything have you ever thought about open source alternatives to discord like zulip yeah for your community by the way uh uh die a lot for your heart oh thanks a lot for your work hard work yeah no problem yeah there's also rocket chat rocket chat is another uh popular one like like zulip i think i you know i thought about it i i've thought about it it's um you know it i thought about it absolutely and um you know i we've seen the history of microsoft buying things i mean it doesn't mean that this is going to you know go away or whatever like skype and yammer and all those other things who knows what's going to happen uh you know i i think they want to roll it into their you know into their gaming platform into xbox and so they you know they're they're they're taking their gloves off uh against uh uh sony and this is really a move against sony uh who knows i mean i'm sure i mean don't get me wrong great discord's a great product um but i absolutely believe that you know probably sony was kind of buying it too and it's so do do they buy this you know to prevent their competition from getting it this is all speculation but but it happens a lot it happens a lot i think that even happened with skype someone else was super interested in buying it microsoft's like and they're you're a competition so we're going to buy not that no it's not a great product but anyways um yeah i don't know i i still enjoy discord i mean the the the jury's out on how it's going to change it might not change at all you know i you know and uh if if microsoft continues to um you know use it primarily for gaming and gaming community and you know in just smaller niche communities in general not that gaming's niche but you know you can create a lot of communities on on on discord and super niche ones like like ours as long as the spirit of that is still there i'm totally fine if i start like opening up you know outlook in outlook is prompting me to create a live you know clubhouse type stream in an outlook meeting that's when i'm like okay this has gone too far if there if we ever catch wind of like discord integration and outlook yeah or linkedin that's when you know things have gone too far but you know being a chat client that um you know has a lot of great features and there's a lot of members too it's hard it's hard to you know it'd be hard to comm you know it's it's hard enough now to to convince someone to click on a discord link it means you got to have discord it means you got to be signed in you know it means all these things you know creating uh then self-hosting a chat uh client is as awesome as that sounds it's really hard i i feel like the adoption rate might be a lot lower i i i could try it i could try absolutely absolutely um but yeah uh for linkedin yeah totally is the deal final i don't think so i i don't think so uh i i think it's been accepted or the offer you know but then fcc like everyone's got to get involved to make sure it's you know and not perceived as a monopoly you know all the things that need to happen you know when when companies get bought especially a big company like microsoft buy something you know there's a lot of regulations in place i i don't fully understand it uh it might take a while for that deal to go through who knows i've been following that closely other than i know they offered they offered i don't know nine ten billion dollars and i think the offer has been accepted i think i don't know maybe i'm maybe not i don't remember the last i read um and i don't know what it means for discord um so i i hope discord continues to be discord you know and and focusing on smaller niche communities because you know that's that's where it's thriving you know it's not slack you know if microsoft picks it up and tries to turn it into slack where it's like hey i need 15 000 accounts to sign into slack you know and i only get 10 000 messages in my private you know puts all those limitations and paywalls in front you know that's where we're going to see we're going to see you know maybe something else come in you know and you know make way for something else to come in and grow and fill that gap um you know most of the time too we for the most of the most of my discord life you know and still for a good portion of people discord's pretty much free uh and sometimes that's a tactic of a private company who wants to get acquired make everything free to grow your user base uh so then you have a large user base and a large daily weekly monthly active users so that when the person acquires you they'll want to pay 10 billion for it but you know sometimes that's not sustainable and so you know these are all things that like you know small companies you know do and that when they get acquired have to figure out or the larger company has to figure out when they get acquired how they then kind of monetize this thing and i hope it's i hope it's you know nothing more than what it is today i'm sure there will be additional features but who knows i i mean i i could i you know i don't mean to go on discord forever but you know i could kind of see the writing on the wall how did i see the writing on the wall you know microsoft has been doing a a big partnership with discord lately you know who else have they been doing a partnership with spotify i mean who knows i mean my you know i i don't want to speculate but you know if you you know got xbox live or vice versa you got you know discord uh nitro you would get three months of you know xbox or vice versa you would get free nitro tokens to spend for your community and discord so it's uh whenever you see partnerships like that marketing campaigns that last a really long time that you know that's that's always kind of a a clue on who's teaming up uh you know and they've been doing the same thing with spotify i don't even know i mean spotify would probably be i don't know 100 billion dollars i have no clue but you know microsoft's done stuff with spotify for a long long time and um you know microsoft got out of the music game who knows what they're gonna do in the future anyways i'm totally speculating on all kinds of stuff but i you know i thought it was you know i i mean it was a good fit for discord because you know they're they're primarily they they house a lot of gaming uh a lot a lot of gaming uh uh um communities um but you know that i think that deal is still on on xbox so i'm always i i always you know i i always pay attention to those more than just like oh sweet you know nice i get a couple months of nitro or vice versa i get a couple months of games with gold because of discord nitro i'm always like all right and what's next what's next um uh rhys uh nitro is too expensive in my opinion i would buy it for five to seven dollars months yeah it is it is kind of expensive you know and i've even looked at like like can i buy level two for my whole entire community you know i'd be willing to pay that but then i was like oh my gosh how many nitro tokens do i need to buy five six and that's you know multiplying that out that's a little more expensive than i thought and so you know i maybe with microsoft something like that will come you know maybe it's hey you know let's totally speculate play this out you know microsoft has a gaming streaming service they already have games with gold or xbox live gold now it's xbox network um what are the things you get free from xbox network uh well depending on if you're gonna do you know the game streaming or you're just gonna do the you know the the gold uh you know the gold membership but i could see you know xbox ones or microsoft or xbox wants people to subscribe and they want to people describe that higher tiers and what do you get at higher tiers you get more rewards and so i could see discord nitro just being just a reward you get for you know for for being you know a microsoft uh xbox live or xbox network member i don't know what they're calling now but i get it um so i can see this being another tier and maybe if you're going to do you know the the unlimited gaming you know the game streaming um uh monthly uh which mine just ran out uh last month in march i had it for like a year and a half because i converted my gold to uh i forget what it's called i converted it and i got like a year and a half of it i played probably five games but i could see microsoft just bundling this in again so hopefully that's all it is and hopefully it just means we get the same old discord uh with the same great features maybe at a lower cost with some more features but yeah if we start to see linkedin integration or yammer stuff or or outlook meeting invites getting feathered in here enough so yeah um dude apologize about that uh dinner's ready catch everyone later on discord thank you so much i apologize my bot's not working right now um i gotta look at the time too okay so i'm gonna i'm gonna document this really quick we're gonna go through this really quick because i could talk about the disc sp speculate about microsoft and discord for a long time because it's it's fun it's fun so anyways um so here here's my here's my cluster now um i'm gonna just copy these out here okay oh yeah i'm not on mac there we go okay so this is my i'm gonna i'm gonna arrange these kind of uh see that i'm gonna get lost in the dueling um i'm gonna arrange these a little bit nicer so this is my first i don't know what that thing is what are you invisible thing text box oh there's another one see ya i'm gonna put circles around these two um can i zoom in and out of here i can oh no oh no i'm in chrome as you can or can't tell um so this is my first cluster i'm gonna where's text text here is this is my local cluster uh local cluster and i'll make this nicer before i share it out because a lot of people have asked like hey can you can you document this somewhere i'm gonna put it here for now this is my local cluster and what's running in my local cluster well it's pretty much rancher only so this is my management cluster so think this is my cluster that manages the rest of my clusters that i added so it's ranger only here let's let's not use two text boxes let's do one uh local cluster sorry my mic's right here uh rancher only um it's uh k3s uh yeah yeah bullet points why not let's let's go for it uh local cluster loca lemon la vida loca cluster okay uh local cluster rancher only uh k3s and uh there's a couple other things i installed here so i'm running metal lb here i'm actually running metal lb in two places but i'm running at metal lb here in this cluster um oh thanks for refunding pcg thank you dude you can refund points thank you so much that was something i was going to look into i appreciate it oh thanks man thank you i didn't know that that was a thing and you already figured out it was a thing and and and fixed it so i appreciate it thanks dude appreciate it um thank you thanks man appreciate it yeah awesome thank you so much huh um i didn't know that either thanks man he's on it he has he has all the answers if he ever in our discord he's all the answers and always has such a positive attitude it's you know so helpful uh the mods can do that thank you so much uh colluem uh uh kalu i aim i apologize hey thanks for fall i appreciate it and if i don't call out follows i'll try to at some point i just have a little bit you know i muted some stuff uh just a little bit so i can you know talk about this uh but i'll call them out too um so my local cluster rancher only it's running k3s and it's running metal lb so the reason why i installed metal lb here also is because i want to be able to expose here let's go another bullet point expose rancher to metal lb uh and so metal lobby if you're not familiar with it it's a it's a load balancer that actually runs inside of kubernetes and it's really cool because it has like it has this speaker uh speaker service that talks to all of these uh agents uh or nodes that run on every every node within here um all of the agents and can be a be a load balancer within kubernetes and and it's so it it's so cool because you know if if metal b dies on this one it can run here it can run here it can run here so it's really cool so anyways i use metal lb in my local cluster just to expose my rancher ui and that was purposeful um because i wanted to but then i have uh so i'm going to copy this uh let me let me here we're going to cd i'm just gonna uh oh no i'll put this here ranch only k3s at cd ncd uh middle lobby and then so these are my servers not on a mac stop stop hitting mac stuff oh yeah i can put labels in here all right and these are my agents these are my agents all right so these are my agents getting better at this and this is my local cluster all right and then so it's probably not important but all of these nodes uh all of these nodes can do are running on ubuntu it doesn't matter any os you want it to uh can't nginx do that yeah nginx can be uh can do load balancing too uh but you'd have to make it ha and so metal lb the way it kind of works is that metal ob service can run on any one of these agents so you kind of get h.a just by installing it within your kubernetes cluster and and so really so and and so really um men all be high level i'll put it here too in case you in case you don't know um uh it's a load balancer and it's really uh the the most practical way i can think about it if i can click on the right thing balancer balancer belancer row ads um is that um it's the way you get into your kubernetes cluster so um so i i know you can use host port node port you can expose a port on every node or expose a port directly from your cluster but what metal lb does is actually creates a a an endpoint within a bound to an ip within you know that that range that it's in and that's how you get in and out of of of kubernetes uh not how you get out necessarily but how you can get into your cluster i won't go into too much detail because you know i have traffic helps with some of that too um prefer layer 3 ip or 2 ethernet uh mode when possible yeah yeah awesome yeah uh do you use load balancer pi hole with metal lb uh love it for h8 um do i use it i i don't um i don't uh so i i temporarily move pie hole out of kubernetes and it's running on a virtual machine only and that was part of redoing my whole entire infrastructure because i needed dns there before i needed kubernetes and then so i haven't moved it back yet and i started thinking like you know dns is pretty critical i would love for it to go back in there at some point but if that ever goes down or my cluster ever goes down then my dns also goes down and then it's like the chicken or the egg deal like you know some of my uh all of these nodes refer to each other by dns name so i have dns everywhere now i don't know that's a good or bad thing but i i use you know cnames and uh dns entries so uh a records everywhere um and so if dns ever goes down then even these nodes uh all of kubernetes will totally collapse because i'm not doing it by ip address and that was purposeful because i you know i i wanted to be a little bit uh more flexible uh with stuff uh now check out looking fresh thanks yeah just my wife just did it this morning i have some other dns i have some dns stuff in my etsy house yeah i should do that too uh that's what happened to me yeah yeah so so i i mean it may go back into kubernetes at at some point but for now it's it's in a virtual machine and then i use gravity sync to sync the two and so i i technically don't have you know h a pi hole i have primary and secondary but they're in sync um and so so this is my local cluster only running rancher but yeah all great questions all great questions there is one consideration though and so there there's one thing um that um just because i have h a database availability with that cd um there's still something that happens in here that is maybe not as clear is that you need a load balancer still here the heck do you need a load balancer for oh you do need one uh can i draw can i is this this is text this is a shape just thinking out loud uh so you you still have a load balancer here lb all right yeah sure sure vertical lb why not and so this load balancer here um is really uh no this goes here let me let me redesign this a little bit i'm basically remaking the same the same diagram that rancher already has but i think it's important to know is that this load balancer here still needs to exist because this load balancer is how you communicate with these servers over coupe control and so um so uh that's something that i don't have ha yet i'm gonna work on it and so i'll probably use um you know keep alive d to keep up a load balancer point all of and then i'll get a so if i use keep a live d you know i'll end up having two lo two two nginx you know uh load balancers with a vip i'm not gonna draw it all and then that vip will be exposed uh and then all of these servers uh my key 3s servers will actually be able to to communicate with it uh on either ip and so that's something i'm going to end up doing um but what does that help with that just makes sure that you know your coupe control commands the thing you hit with kubecontrol is also h a and so it's pretty important i mean pretty important for you as a kubernetes operator to be able to issue coupe controlled commands and if this load balancer is down uh you have a vip or you have a you know a an ip that uh is actually floating um between two real physical um or two real load balancers to still be able to communicate with your k3s k3s service kuvip keep live d yeah adrian goins yeah absolutely i saw that too i saw it too uh and super interesting uh you know i'm on the fence yet do i do i do i do a traditional yeah absolutely adrienne going yeah awesome he is easy i think he works for rancher i think rancher you know he works for ranchers he's got a he has a wealth of knowledge in this stuff and so yeah i'd love to pick his brain sometime he asked me if we wanted to do something together sometime we might i don't know jury's still out um but he um he in his latest video about his lcd cluster he does talk about um using kubevip which then basically puts the floating virtual ip within your cluster and exposes a vip to you a virtual ip that's highly available uh just and it's just part of kubernetes which is super super interesting um and i i think in the first pass i might do that i think he ran into some bugs or had some problems or there was a problem with one of the repo of a coup vip he was using um in the meantime i might just spin up another nginx load balancer uh use keepa live d and put my you know have my my vip uh b right here uh oh we'll go yeah sure square uh basically vip here uh that these all communicate with which behind the scenes i'm now i'm like forecasting what my what my uh future infrastructure might look like but but the nice part about if you do this then you know these servers and you communicate with this i mean technically this is kind of like this uh it doesn't matter but kube control commands go to the vip which could go to either load balancer depending on if it's up or down if you use keep live d or a coupe bit um and then you'll be able to communicate with the servers and so and then also these servers are able to uh you know this load balancer knows if any any of these servers are up and down so we can take it in and out of that pool and not send traffic to those to those devices anyways yeah that's tbd i i'd love to do that at some point uh and i i think i might at some point uh i haven't done it yet so yeah that's a that's a good point so poking a hole right there you know my lb isn't aja yet um and so these these these are these are my services are uh middle b yeah pretty much is and so then i have you know this is where rancher is gonna run uh no cheap lb in this market uh i piehole running in lxc could start up another one and bundle them up h8 like you did uh in rancher a while ago i buy hole running in lxc could i start up another one and bundle them up very yeah um could you start yeah so you can have a mixed um topology of pi hole yeah absolutely uh if you're going to use something like gravity sync so so let's so can you have multiple pi hole servers running on different infrastructure absolutely you can you can have one that's running in docker one that's running in kubernetes one that's on a physical raspberry pi yeah so they're end up you know as long as you know your clients are going to look to them for dns requests yeah absolutely you can how do you keep it in sync that's a little bit harder there's this thing called gravity sync that you can use it's an awesome open source project that you can use to synchronize um synchronize uh those uh two servers so they're all of their uh your your block list as well as your um your block list as well as your cnames and your your local dns records your a records can be synchronized that way and you can have you know mixed mode one can be physical one can be containerized one can be in kubernetes there are some requirements though if you do containerize it i think there there's some script that has to run on the host so if you're gonna do it in kubernetes i think that script that has to run on the host has to be on every node that you know pihul could possibly run on but yeah yeah and absolutely you can use keepa id like uh pc mentions um and you know point to a vip with keep live d same same thing here point to a vip and then it'll just figure it out for you you know if that one's down it'll take it out of that pool of eyepiece uh and then send it only to the ones that are live yeah and so that's what i think i'm doing right here with my my coo my uh load balancer for sure um but yeah and then keep them in sync with gravity sync absolutely um and then i think for here you know i basically have this twice uh and so this is my user cluster here user cluster oh uh what do i have here uh my workloads i'm gonna say my because it makes it sound more personal uh and so we know i'm not gonna put rancher in there but my workloads only this is not running on k3s so my second my second uh my second cluster is not running k3s so this is what you saw there it's a rancher provisioned rancher provisioned provisioned uh cluster and that's what we saw earlier so this was actually when you go into add a new cluster create a custom cluster and then from that custom cluster it'll generate docker commands for you and those docker commands can then um those docker commands uh can then uh spin up uh the server or the agents of those types for you and join your cluster so that's what i ended up doing uh because it's it's one it is easy to do and two a lot of people do it that way so i figured why not there's a third reason why i did it that way too is because rancher has more support for for uh clusters that were provisioned with a rancher so there's this tiered matrix that they have of features that work from rancher on clusters and so you get a handful of cloth you get a handful of features if you just go and pour the k3s clusters you get a lot of you get a lot of stuff and so i'm not downplaying what what it can and can't do you get a lot of stuff but if you actually you know then import i think an rke cluster and you know one that was generated with rancher in the past you get a lot more but if you do it using you know their tooling you actually get a lot of features for backup for security scanning for logging and a lot of things so that's what i ended up doing um yeah absolutely yeah dude thank you thank you uh pc geek and so that's what i ended up doing uh farhan uh rke yes so this is actually our key i think that's how they that what they spin up i'll i'll look into it but i think it is an rke cluster and then so now when you spin up this new cluster um you know you're gonna need the same types of things so you're gonna need servers that run and when i say servers i'm lumping those two roles control plane and ncd into one um but your your servers are gonna be running control plane and fcd no user workloads and then your agents are going to be running only user workloads and so you need that same thing you need that same thing twice and so this is a big reason why i i haven't done this in the past is because i'm like man that's a whole lot of resources you know that's a whole lot of resources for this you know ain't nobody got time for that you know and um when i created my new one i figured you know what i'm just gonna do it i'm gonna delete this load balancer for now because it looks awkward as if it really means something um so yeah it's a whole lot of resources and i thought the same thing but i thought you know what if i'm going to do it this time i'm going to do it that way because when people ask me the question then i can say yeah i've done it here's here's how it works um so this rancher provision cluster now this is where it gets kind of interesting because i also install metal lb here and what do i install metal lb here for is because i want incoming traffic into this cluster you know i want a load balancer for my cluster and i said this a few times and maybe i didn't make it clear but but metal lb is basically a software version of a cloud load balancer that kubernetes expects to be there right and so when you think of cloud load balancers they're you know they're they're they're they're cl load balancers that are provided by cloud providers so you know azure uh you name it azure aws gcp lenod whoever um they're all there uh and they're and they're usually big expensive things and you rent them out and you pay for ips and stuff like that well metal i'll be acts like one of those a software version of one of those in your home or in your in your i should say in your you know on-prem or it doesn't even have to be on-prem on your self-provision kubernetes cluster that's not in a cloud provider could be hosted in cloud provider it gets confusing uh but anyways this acts like that and so metallob i actually do install here because why because i want to expose traffic uh try the the traffic i don't even know how to spell traffic anymore uh because i've been spelling traffic the other way the product name but i want to expose traffic a fix i think i spelled that right expose traffic um which is also living in here too i should i should draw some more boxes here too um but traffic actually then i can expose this is what layer seven l7 load balancer uh i i should just put reverse proxy so it doesn't get confusing it can do more than that i only use it for a reverse proxy uh but i ended up putting metal i'll be there too and i put in traffic there too and so i can have my reverse proxy there so that i can expose these these workloads uh to the public or even internally even internally here and i get ssl there so i'm going to draw a box because i think that's kind of an important takeaway here i should probably write draw two boxes i'm going to draw them here so i have metal i'll be here but i'll be i'm bad at i'm bad at boxes there we go metal llb i wish this was bigger i i should probably make this drawing board or whatever it's called bigger put it there metal b and then here i then also have traffic too so i'm going to draw another box here traffic um and so that's my reverse proxy and so here uh i'm going to expose i'm going to put an ip here so i'm going to just say 192 160 whatever you want it to be uh 168 dot whatever two whatever i wanted to make it look like it's not it's important so i won't put 100. whatever whatever your ip is here but this needs to be on the same you know subnet as everything else and so what what what this hap what happens here is is really cool so if you haven't used it you know um you have incoming requests that come in uh to this 192 168 oh if i could oh if i could type whatever 40 dot 230. i want it to look a little more important than a default ip that something gets uh anyways so the cool thing is you have incoming traffic here so let's say you have a request for example ah can i can i type today um example.com so example.com uh comes in here uh middle of b says hey you know you're you're um this is basically uh a load balancer um and so yep i can i can forward that on uh traffic then basically says hey i'm looking uh i will look up example.com um let me see if i if i can hit the right buttons i just want to click there we go i can uh i i know example.com example.com uh resolves to one of these these uh uh pods in here and then example.com uh let's say runs in one of these agents uh with a web server that's running in here so this is also gonna do ssl here too ssl ssl so this has let's encrypt because a lot of people are like yeah i use nginx and let's ecrypt yeah same same idea same idea different product same idea same capability so let's encrypt and uh let's encrypt uh is running here uh with certbot and reverse proxy i'll i'll put them all in here roxy and it can do a bunch of other things too they can do a bunch of other things too uh why not a kubernetes ingress uh yeah um yeah i'm not sure not sure i i don't know jerry bohr um i you know i i this is kind of the infrastructure that i settled on um you know i i use i use metal lb because um uh uh one it's easy and two i i need a load balancer somewhere to get traffic within my my kubernetes cluster um and you know and then from there i need i mean i could you could use any uh ingress controller you want here i i use traffic so yeah totally good uh you totally could uh but but traffic becomes your you know traffic isn't ingress too um why not everything on amazon yeah yeah totally yeah why not yeah i want to pay amazon not really to run all this uh no way uh but yeah for sure um so uh yeah that's a good question uh i mean i've chosen yeah yeah no thank you yeah i'm yeah not like this for sure um so yeah this is this is kind of how it's designed and this is how a lot of my videos talk and this is how this is where traffic plays really well so this can be in nginx and grass it can be anything you want it can be anything you want for sure uh but uh but i've i've settled on traffic because it ends up being super duper easy to do and uh you know i i run nginx a lot of other places and when i was looking for you know my ingress i thought hey let's let's go with let's go with something that seems new and easier to use um windower how many servers will be involved well if you look at all these boxes these are all servers so these are all kubernetes nodes and so when i when i when you hear node in general you could just think server one to one server and so how many nodes do i have well i i have two clusters right i'm gonna draw like a whole bunch of all kinds of stuff now and i'll fix this up later can i can i box this up into one oh killing me whatever google um here i can do a line there how do you like that i'll fix this all up later so how many servers do i have well i have two clusters um and each one of these boxes are a server so i have you know six servers uh per per cluster and that's because these have different roles and because i've separated all the roles except for a control plane control plane and ncd are running on the on the same server because i i don't see any need for me to separate those out huge maybe if you're running a gigantic cluster maybe that will work for you um but six so i have 12. i have 12 dedicated to it i actually have 13. i have an odd number uh because for some reason i decided to add another agent and i think that was because of my old infrastructure already had one so i'm like hey he's already provisioned he already has ip dns entries he has everything he needs why not let's let's go for broke uh is it required no it's not um so yeah uh but you could do this with much much less much less like i said like my my entire cluster was this the bottom half for a long time uh you know when i ran i ran rancher and kubernetes and all my so i ran my rancher workloads and my user workloads in the same spot and you could arguably you could still do that there's there's absolutely nothing wrong with that ranch will tell you not to and i i will tell you it works fine you should probably listen to them instead of me but you know i've i i understand why they say you don't um you know i i totally get it and there are there are a few you know there are a few uh uh things that you run into if you run it that way uh because there are some things with their their load balancer that you'll run into and it's not problems you'll just notice a blips every now and then uh if a rancher uh server goes down or pod moves or whatever the case may be your load balance or your ingress your ingress will blip for just a minute and uh so will some of your services that are connected to that same ingress so it's not a huge deal but i have seen why they say you're not going to um let's see uh sounds good uh dumb role i got to get this name dumb roll dally uh have i tried ansible yeah absolutely yeah i run ansible yep i have and so uh uh did i create playbooks to provision all of this no no but i have playbooks for my k-3s agents um certain things i need to do and certain things i need to uh maintenance types things um and i have playbooks for uh virtual guests for proxmox so i have a lot of different playbooks but i didn't provision all of this no no way i use you know i i semi old school proxmox uh templates clones and you know and uh uh then i ran ansible after that so so my my i don't run ansible against uh my proxmox server to actually provision the node but my templates and when i clone them as soon as they're up and going then i have playbooks on how to make them you know uh a node for a kubernetes server so everything from ssh keys to updates to additional libraries or packages that i need to install uh the whole nine yards so yeah i i have i have everything but the actual cloning uh uh working uh except for proxmox i'm sad you're not using xcloud for this diagram i know i know i know i know i should be dog food my own stuff right i just i you know my bots not working today and it's like okay is my my next cloud not gonna work today yeah totally call me out on that stuff i should be dog food my old stuff right uh just call me barack barak yeah yeah that is easier brock i will remember that it seems powerful all right uh yeah it is powerful and so and so a lot of a lot of these um so uh so your powerful comment was about ansible yeah absolutely it is powerful uh and i have a lot of my playbooks i guess i haven't contributed too much a lot of my playbooks are are um in my github repo too you can you feel free to pull them down and add to them uh i need to add to them too because i i've i've they've evolved and i haven't keep kept the ones i have in github up to date i probably should um dinner was delicious now i'm back all right lude yeah you're making me hungry um yeah and so uh speaking of yeah speaking of getting hungry yeah i'm kind of getting hungry but um so yeah this is kind of you know i i know i probably could have explained this in some other diagram but i figured we'd you know kind of talk about it and talk about why um and then so if someone you know has questions you know i can refer them back to this better looking diagram that i will create here soon um and then i will also create uh the same the same commands that i used to spin up all these clusters if you're on discord you can already kind of find them i think i think john and discord asked for him and so i i supplied him uh but i will i will also uh make better versions of that available i think i'm gonna sneeze maybe no okay all right i thought it wasn't then i was then it wasn't then it was but i did um so anyways yeah like uh absolutely share this out um i'll absolutely share this out and and make sure it gets into your hands and and maybe i'll do something a little more formal too to put it together to kind of talk about some of these things i don't i don't know i don't know maybe i'll edit the font who knows um i will say though that um uh if you're interested in this uh this is this is kind of my outro because i do have to wrap things up because i did promise my wife i would be done in about four minutes uh because we have somewhere to be um so i do have to wrap things up i just want to say absolutely if you're interested in any of this and you haven't had and you're not a member of a discord community we talked a lot about discord hopefully microsoft doesn't destroy it if they do we'll go somewhere else right uh if you're not a member of the discord uh community i absolutely encourage you to join um i can't run the discord command now but if you look down below there's a big discord link you can absolutely click on that button and join our community there's a lot of people that are in our community that talk about a lot of this stuff and it's not all kubernetes so if you're like ah this guy's talking about kubernetes he talks about kubernetes way too much it's not all about kubernetes it's there's a lot of hardware there's a lot of services there's a lot of different things in there so i absolutely encourage you to join if you're interested um and also i just want to say yeah absolutely thanks thanks so much for for coming out and for for for being here uh chatting poking holes in my diagram asking lots of great questions not a lot of tough questions i'm glad there weren't any tough questions like why the heck did you do that like have you ever run kubernetes before i was expecting some of those i knew you weren't going to do it but absolutely um i knew you weren't going to do that because you guys are you guys are nice you guys are way good um and so there were a lot of follows a lot of subs and resubs and bits so i greatly appreciate that so thank you thank you so much um and if you have any additional questions uh ping me and discord ping me on twitter um thank you yeah thanks have a great saturday sorry i'm cutting a little bit short have one minute and 30 seconds uh and then my to-do's i have to do's so i got to update some stuff in github pcgeek thank you so much for fixing the points i don't have to do that um and then make a better diagram that's shareable along with the command so i appreciate it yeah piece of geek found found the link thank you so much but yeah guys uh have a great saturday thanks for tuning in uh i will be back next saturday i'm gonna have something coming for you soon someone asked about hey can you tell me about how you do this well that video is coming soon too so i i mean i'll just put it out there monitoring alerting for kubernetes the whole nine yards coming soon um and uh yeah i hope you guys have a great day thanks thanks so much listen to your wife i know i'm like blabbing away and i actually have a minute and five seconds so i'm i'm ahead of time thanks so much uh have a great weekend and uh be good to each other take care folks see ya [Music] [Music] [Music] you

Channel: Techno Tim Talks

Views: 2,305

Keywords: twitch, games, techno tim, homelab, selfhosted, self, hosted, home lab, technotim, live stream, tech talk, tech chat, virtualiztion, containerization, rancher, portainer, k8s, kubernetes, docker, server, home server, networking, obs, discord, youtube, software developer, software engineering, bots, slack, gaming, traefik, k3s, etcd, k3s etcd, ha k3s, embedded etcd, network diagram, cluster, cluster diagram

Id: 3aNS4cdQmqI

Length: 102min 39sec (6159 seconds)

Published: Thu Apr 08 2021