Elastic Stack on Kubernetes, with Skaffold

hey josak here giving you a sneak preview of a talk that i did at the elasticsearch developer meetup in san diego virtual of course and i wanted to share it for you on youtube so the talk is about easy local development with elastic cloud on kubernetes and i'm going to be using a tool that i really like called scaffold to put it together so the reason that i want to do this talk is because i think it's really important to have a dev and production environment that are in sync the closer you can have these two things uh the better you avoid mistakes and life is just easier and then when it comes time to debug production then you're already familiar with it that's hard for really complex systems uh like for example you know elasticsearch where you might have several different services involved different kinds of nodes different types of environments and so what happens is a lot of times organizations will set up like a shared server somewhere that all the developers can use i'm just going to pretend we have a team of 20 developers and so you may have one developer that sets up and maintains this environment that the other 19 use and uh you know that works out okay but what if some developers are working on features that are going to roll out a couple months and part of that includes a newer version of elasticsearch or some other services or what if you've got developers working on a fix that requires an older version well then you need people to set up additional environments and that kind of stinks also sometimes developers make mistakes and do things like i don't know wipe out all the data and if anyone needs to do any sort of maintenance on one of these systems then they're going to take it down and break things for all the developers who are going to have a period where they're not going to be able to do their day jobs and that's a problem also that environment is often out of sync with production like i mentioned if you're trying to work on some new features that uh require a new version of elasticsearch that you're going to roll out in production there's going to be a period where your dev environments are ahead of your production environments which is problematic and so yeah it's complicated and it doesn't have to be and so what i want to show you today is a tool from google called scaffold that you can use to run kubernetes locally and it makes it easy to do that and you can use scaffold to actually deploy out to your environments so you can use basically one file to determine your architecture that you use in both dev and production keeping everything in sync which is really great uh so the tools involved here now we've got elastic this is for an elastic meetup so it's going to be focused on that heavily we're going to be running things on kubernetes that's kind of the backbone here and the tool that we're using to keep these things all running together and running well together is scaffold which is an open source tool from google that is the glue here all right and now i'm going to do a little demonstration where i'm going to create a scaffold.yaml file which is a special file that the tool i mentioned earlier scaffold can use in order to spit up an architecture and it's got a bunch of really nice stuff built in that i'm going to give you a tour of so i'm not going to go too deep into elasticsearch or the uh elastic cloud on kubernetes tool or kubernetes because i just kind of want to keep things light and show you what's possible but the source code for this presentation is available and unlike a lot of other presentations or demonstrations you might see for stuff this uh can serve as a suitable bedrock here so everything that i'm gonna show you uh is not like uh kind of illusory smoke mirrors demo type stuff this is like a a real foundation that you can use to go and kind of start building your dev workflow and even production build deployments off of which i think is really exciting so uh let's go ahead and just do this thing from scratch don't worry i got some notes as a backup in case i uh end up downloading to do so uh if you're not familiar with kubernetes uh you'll see that um i'm gonna be kind of typing in some weird nonsensical boilerplate here uh and if you are familiar with kubernetes and you know this is the kind of stuff that you end up doing with pretty much everything that you do in kubernetes so this is very consistent with kubernetes type stuff i'm just creating a file here and we don't have anything in it yet now let's make sure i have a blank cluster so this is a tool that i like you're seeing this a lot today that makes for really great presentations it's basically a ui for kubernetes and what we're seeing right now is that i'm hooked up to a local cluster running inside of docker i'm in the default namespace and i'm looking at pods which you can think of as being processes running in kubernetes so this kind of maps to things like an elasticsearch node for example or an instance of cabana and uh we'll see those things pop it up here in a minute now we don't have any artifacts to build so i don't have any custom docker files or anything like that we're just going to be working with the elastic cloud on kubernetes to keep things simple so we kind of focus on the meat here so i'm going to deploy and there's multiple options here uh helm is uh probably the most common thing you'll see if you're doing any like real work with kubernetes there's customize there's a can eco or something several different options for things you can deploy and uh let's go ahead and get the documentation up here it's a really nice and let's see references scaffold.yaml so everything you're going to see me typing is all just straight out of here like i'm going to be filling in different options and different things that i need in order to do what i want to do given this single file reference and i should mention too or just reiterate that scaffold is an open source tool from google it's a cross-platform so i'm going to be running out on windows i installed it with chocolaty if you're on osx you can install it with brew the only trick is a scaffold with a k as you know kubernetes back to the men are at hand so i'm going to be installing manifest now i mentioned eck a few times if you're not familiar eck is elastic cloud on kubernetes it's basically a set of custom resource definitions and an operator for elasticsearch and those are definitely kind of heavy kubernetes topics but all you need to know for this talk is that they make running elasticsearch super easy in kubernetes so let's just straight up google for this elastic eck if we go to the quick start guide and then deploy uck in your cluster there's a url here that contains everything you need to set up eck i'm just going to pop that in there and what i'm telling scaffold to do here is that i'm telling you that i want it to deploy via cube cuddle the files in this array and there's only one item now that's what this little dash means it's an item in a like a list or an array we're going to install this so i'll come over here say scaffold run and it's going to do that if i typed it correctly and i did not i put the file in the wrong spot so i'm going to move this up a level and we're live coding here should uh apologize in advance try this again there we go so there is a list of all the things that this operator or this eck just installed which does include an operator which is responsible for kind of managing uh my clusters you can kind of think of as like someone on your team that will set the stuff up for you after you tell it what to do unless there are custom resource definitions that uh kind of create some abstractions around the things that we want to do and just makes it just dead simple at least as simple as kubernetes can be so that's pretty cool let's go and actually deploy an elasticsearch cluster now so i'm just going to grab this file out of here the contents here and we're going to deploy an instance of an elastic search which is an easy eck concept it's got a name of quickstart it's got a version and it's going to have one node now uh eck is what elastic cloud actually runs in the background it's capable of doing uh pretty much anything you would want to do with elasticsearch so you can have masternodes and ingestion nodes and to find the storage and the storage classes uh ram limits request all sorts of stuff behind the scenes but we're just going to keep it simple so don't get bogged down those details now i have already downloaded this just in case you know the website was down or something but i'm going to go ahead and just paste over it looks like they upgraded the version on me so let's do that and let's go ahead and grab cabana as well and normally or at least if you were working through like a getting started kind of guide or if you're working through the elastic article i mentioned you're gonna be deploying this stuff by hand by typing cube cuddle apply dash f okay slash dev slash maybe star and deploy those files and then if you want to remove them then you're going to have to do some other stuff and you're going to have to kind of know how to unwind that now scaffold handles that for us so i'm just going to do scaffold run again this time i'm going to add my folder for kate's dev slash star now it's going to make sure that it's got the custom resources installed and it does and then it went and added my additional stuff you probably see up here at the top okay installed elastic and uh kibana so if we come over here i don't see any pods yet now this is pulling in the background but could still be coming up so let's go check on a custom resource type of elasticsearch remember that was the type that we had in the file so we've got one coming up it's got unknown health because it's still uh just getting started see it's got a phase of applying changes so uh things are going well just gonna take a second and there we go so it's not ready yet see this uh zero out of one means we have one node for elasticsearch one node for cabana and that's going to take a minute to actually be usable or or right now elasticsearch is already available view the logs for that make sure it looks okay any errors nope all right good and now kibana is up as well so now we can go check this out make sure it's working first i need to look up the password and this is an eck thing they will generate a password for you by default and you don't have to you don't have to do that you can use a hard-coded one if you want but again i'm just kind of trying to keep things simple so we come over here to localhost and let's come out of port 5601 oh no it doesn't seem to be working and that's because it's one last thing i need to do and that's i need to pour forward we'll see how to make this easier in a second so i'm going to go to kibana and uh in my case this is a you would you could type a cube cuddle command here to do this but this ui i'm just going to hit shift f and i'm going to hook it up so that on my local computer 5601 is pointing into this cluster and hooking up to uh kibana on port 5601 and the deal here is just the kubernetes uh maintains its own dns that's kind of like a cloud in the box in some ways and now i've kind of tunneled into that box and we click past the warnings because we're running under https and if you want to run this in your organization you're going to hook up some certificates and stuff in order to get rid of that warning oh and i lost my password let's go get that one more time all right okay and i'll go to our data tools and see what we got all right we got a couple indexes uh nothing exciting we've got a you know a couple documents from just kind of built-in type stuff but uh not very exciting i will uh we'll fix that here in a few minutes but uh first i want to kind of underline that uh what i'm trying to get across here is not hey look you can set this up in two seconds and then you would never take this production and there's a whole lot of stuff you have to do on top of it it's a a point here is really to show you this kind of workflow so what i'm trying to get across is that we can take this file and keep iterating on it to make life easier for every developer on our team so now every developer who has this file checked in and is using kubernetes locally has the ability to spin up the same architecture as you with a one-liner and they can show it down with a one-liner so we got that that looks good uh what if we wanted to make our life easier let's look at scaffold dev real quick scaffold dev gives us a way to watch for changes that's really nice when you're developing and what that means is that much like run it's going to do its thing it's going to set you all up but over here now if i come in and change a value like say set the number of nodes for my elastic search cluster up to three it noticed that i made a change in one of my files and now it's out deploying it so if we come back to our cluster then we can see that we're going to get two more notes split up which is really easy but uh that isn't fantastic for everybody right sometimes if i'm working locally i don't want to use so many resources so i'd rather just have like a single node maybe or maybe i need to run a different version than you and so i want to be able to do that stuff a little bit more dynamically and you don't want people going in and tweaking this file that everybody's sharing right that's just bad news so what we can do instead is set up profiles now the way i'm going to go ahead and bring it back down to one and then go ahead and stop this profiles are good for setting up some modularity in your environment so profiles give you a way to optionally spin up various pieces and not only is it kind of bolting on pieces it can also modify or delete pieces of your architecture that you've got defined this is not the right file that you've got defined over here so in our case what i'm going to do is i'm going to add dynamically an entry to this manifest manifests section here so we'll say profile i think it's got s on it it's a good chance i'll i'll mess this up uh name and tell you i'm not even going to try and remember this let's grab this from our notes and this is all up on github by the way for files okay profiles and let's go ahead and add file beat sure so uh name is file beat this is going to be what me the human types in in order to tell it to use this and i'm telling it to add the file is located here to deploy keep cuddle manifests in this file so again that's deploy cuddle manifest and you can see it here deploy cube cuddle manifests what that means is that anything you set this path you can actually change dynamically so for some crazy reason you wanted to change the api version uh you could probably do that you would just want to change you just do a replace if you wanted to add for example a helm deploy here in addition to your cube cuddle then you would say uh add path is deploy value is helm and you pop in your helm stuff here now let me just show you what this looks like and uh this file be here that i've got already on disk i'm going to go ahead and upgrade it to two this is just another file where i literally googled like you know elastic eck file beat and and there we go just standard kubernetes stuff with that let's go ahead and paste it in here so now we got scaffold we got a profile named filebeat and the way you do this is you can do scaffold run or div or there's actually several commands here we're not even going to look at all them today not by long strap and then we pass a p or dash dash profile and we can do file b alright so now if we come back over here we should see a file be coming up and it's already written and the way that filebeat works here is it creates another kubernetes concept called the daemon set which is something that can communicate with like the the know the actual like hardware underneath and it'll do things like uh you know could file beat stuff out of there so now if we come back to our uh kibana and run get cat indices now we should see something for file be in here i don't yet so let's check the logs do you see an error failed to connect to elastic unauthorized uh-oh security exception uh-oh all right tell you what maybe we'll just take this whole thing down or maybe we just wait okay hey good thing we waited i got a little nervous there uh that things are gonna go down a weird path but it looks like i just needed to be patient took a little while to figure thing out and so now we've got an index who was worried i wasn't worried name file beat yadda yadda yadda baby star and we'll just search it and see we get some events uh underscore count so we've got two nine five nine if i wait a little bit and refresh two nine six four so that's slowly going up and so that's a way that you might for example give some developers a way to dynamically choose what they want so maybe a normal use case you don't want to have data flowing in some cases you do another use case for something like this is you might want to say name i'll call prod and patches we'll do an operation this time we'll do a replace oops do a replace uh path is deploy keep cuddle manifests and what i'm saying is i'm going to replace what we've gotten here with values over here it's and i think i've already got it set up yeah i've got a product folder here with some items and i'm just going to do that thing where i copy the files over because the versions are slightly different so here for prod this is just a kind of example type thing you could do let's say we want five nodes now there's easier ways to do variables and there's better ways to do variables for sure but i just want to show you demonstrate that you can mix and match these profiles so now over here let's go ahead and do a scaffold delete so that's going to delete everything oh would i give something up on marshall error let's try getting rid of that profile looks like i had some sort of syntax error there so i just destroyed our cluster which is also pretty nice to be able to do that easily if you are working with the team developers you wouldn't want to do that and mess everybody else up and so to make sure i typed this right oh look i typed open here that's why it didn't work so now i am going to say that i want to scaffold run if i were to replace that means i also need to add this manifest back this was actually prod star scaffold run profile is prod and file beat that's going to run both of these let's see if this works and it doesn't uh it didn't like didn't like this in the interest of moving this along i think uh there's something i'm maybe doing wrong here is it uh it didn't it's like trying to create the items later in the array i don't know i'll take that offline uh that's the problem i saw the other day and i thought i had fixed it all right we do a scaffold run sorry for cheating prod and file beat all right now that worked so if we do take a look-see over here that pods is going to take a little bit to come up to do you should see that obviously several nodes end up spinning up we can also do what we did last time and watch this over here on elasticsearch quickstart created i do something wrong um all right well let's take a look at the docs here maybe we can figure this out this will be under scaffold uh profile there we go and there's a thing called op you can do all sorts of fancy cool stuff like in introspecting the type of environment in order to automatically figure out what kind of profiles to use there's all sorts of different operations like building and testing and deploying scaffold does have stuff built in for testing which is really nice override via patches so here they've got a name the office replace and they're replacing build artifacts the first item okay let's uh let's try that what could go wrong so i think this is where i've done goofed up so i'm saying replace cube pedal manifests the first item so i'm replacing this with this uh-oh cannot unmarshal a sequence to a string uh that's because it should just be a straight up value here not an array okay well hey good thing good thing this hopefully worked on game day i'm actually recording this before the talk but it'll be published after the talk so uh now we've hit something that is good to hit ahead of time you know what i'm saying all right did this work like we expected doesn't look like it manifest slash one let's try just doing prod and bad path keep all tabbing sorry habit habits okay so it did our normal stuff and it looks like it did our elastic search and our kubernetes oh my gosh oh my gosh you know what it is on windows you have to put quotes around it so this is a total user error what do you know so now if we do prod file beat we should see that same list of stuff and now also including file beat so what i had here uh was probably fine all right and so now it saw that file beat was the only thing that was missing here so whoo all right uh no take back these though we're going to leave that in the video uh so you can that's how you know this is legit so this used our prod profile and so the point of me showing you that was just to demonstrate that you can kind of slice and dice and mix and dat and match and replace things add delete and modify uh your type of environments and so that's really good for having things like you know i use prod as an example here we could use that to deploy to a product environment and then uh you know file b for like optional type work i think i did set up an additional thing here which i don't know if i'm going to get to this uh on the presentation that already happened but we also have the ability to do helm here and so i just wanted to kind of show what that looked like real quick so if we do helm and then i think it's releases and then i need a name for the chart in which case i'm going to do website here and the chart path it's the directory so we've got it up here so it's in charts slash website and here we've got a template that's going to set up a custom website and so my point here was just to kind of to reiterate and show that you can use other tools besides just helm so do a scaffold run it should deploy my website now this does include how it means you have to have helm installed on your computer which is really common tool used with kubernetes and uh that's probably more likely how you would do things like that the product example i set up where you'd want to use like different variables to pass the same files in order to set things like the number of replicas or whatever but just kind of want to demonstrate how like maybe a custom website would work here and if you had any custom images you need to use for docker instead of using uh for example see where am i getting my image from so this image comes from here okay so glad we looked down off the rails again uh this is a custom docker file that we wrote here so that does mean that i have to add a build section here and this is really easy uh i don't have it memorized though so let's go back over here to the documentation and see what kinds of stuff we would have to pop in for the build really glad to be recording this video now because i see uh that i'm going to have to cut some stuff because if i run into any problems i'm going to go over time aiming for 35 minutes so build next up is artifacts that we do image and context everything else had defaults so build artifacts it's an array it's got image which was website right and i'm fine with that name and context is going to be where i have that docker file living which is charts slash website yep and it's right there so now check this out this is never going to finish it's constantly waiting for website to be available now we can do scaffold build and it will go off and build all of our artifacts and we only have one and there is one catch here there's something special you have to do for helm and it's because of this generated tag right here every time we do a build we're going to re-tag and get a new version of this container and this would be something that normally your build server would go out and publish to would publish to like a container registry and then that would kind of set you up for your deploy so the way this works is that i need to tell it about my images and is that right i think it is actually let me uh double check my notes but what i'm saying here is basically there's an artifact in a variable over here there's an image named website and i need to tell it about this so it can override this variable something you can do at home in order to tack this this hatch on at the end and uh the value before the hash is website so i think that's it it's gonna check my notes real quick to make sure website charts website lisa's name charts website uh image overwrites it's actually uh the left hand value should be image which is the name of the variable not the value of the variable okay so let's try a scaffold run we should get our uh environment showing up we should get a new pod coming up any second oh i've uh i'm filtering for elasticsearch here take that off and we can see it so uh the point with that is just kind of show that you can do you can work with helm and also if i were to do a scaffold dev which means it's watching for changes uh then i can do stuff like actually let's do one more dev i'm gonna go ahead and tell it to port forward for me which is another option which remember we put forward into cabana this will just automatically go ahead and set that up for anything that scaffold knows how to see and uh this is red because it had uh some sort of race condition but this will actually end up getting uh hooked up here in a minute if we give it long enough but we can see that it hooked up port 80 so if i go over here we should be able to see port 80. hem 80. and hey oh we got some sort of problem with this docker image so if we come over here and take a look at our html file uh we're missing some stuff it just starts straight up with uh css so uh we need to fix this up we can do that by making html and then we need to start a head and then there's gonna be a style tag and then i think that'll do it so now i save that so scaffold because it's watching is going to go off and rebuild that and redeploy if i keep refreshing long enough hey it's fixed and so you can kind of imagine how developers could actually use this for a development tool so if they need to make quick changes then this is the way they can do that by just running that one command setting up whatever profiles they want getting their port forwarding that's basically everything i wanted out of doc compose if you're familiar with that it's another tool for kind of spinning up architectures except the big benefit of using scaffold over something like docker compose is that i can use this scaffold file to actually do my builds and deployments on my build server now i can tell that i went way over time so i'm probably going to go and tweak this talk a little bit so i'm not going to go into some things that i had originally planned on because now i can tell that it's probably going to be too much but if you leave a comment on this video let me know what you would like to see or if you have any feedback for me that would be great for whatever next version i do this talk later later this year next year but the way you would do this just kind of conceptually is you would come in here and say scaffold build and that is going to go out build and tag my website and if you look at the help for build you can see that it's got a ton of options for doing things like specifying your tags there's also ways to set up different profiles and different ways of of hooking up to different container registries so that after you build you can do a publish and actually if we look at uh just scaffold dash help you can see that you know we only looked at like uh run and dev the bug gives you a way to actually hook up to buggers by setting up special ports and kind of tunneling things up so you can do things like step through and debugger and like node or c-sharp for example python python specific are pipeline specific things i'm kind of mentioning now is like build for publishing and building artifacts deploy for deploying pre-built artifacts that means you've already done a build and now you want to deploy those things so build and deploy these two steps are basically the same thing as run which we've been running throughout so if you want to have those uh a single step for some reason that's what a way you would want to do it but typically in bigger ci cd environments you're going to want to separate these things because the stuff that you build might get deployed out to multiple environments and may get even promoted between various environments really good devs staging production as various test paths we talked about delete you can do render in order to just clean spit out the files that you need that need to be uh applied if you want to use something else to apply uh there's a couple nice things for um you know kind of updating or getting started with a new project we typed everything by hand but if you've already got a doctor compose file or a sorry rather a kubernetes infrastructure you can just use in it and they've got some other kind of random stuff that's it's pretty nice again we mentioned for a version or if you want to take a survey you've got that built in that's cool so i mean aside from that i'm definitely chopping off a little bit of the talk that i planned on going to but i think that's okay uh we just take a quick rule over here and we can kind of see what that looks like for setting up the container registry and i think uh you know you can configure repositories and pass it and then what's kind of important about that is that when you tag these images this might be getting too meta but when you tag images like here typically you're going to want the registry name in here so it would be like that i don't know dockerhub.com you know b1 dot whatever slash staging you know it'll have some sort of prefix on it and uh that's what setting up the repo will allow you to do so you don't have to have that stuff in your developers faces in fact you can configure it only on the build service if you want to keep that stuff private which is a good idea and so what you would do is basically just install uh scaffold on the build server and have it do your your uh scuffle build employ so with that uh let's go ahead and call it um didn't exactly go as i planned but that wasn't too far off so let me know what you think of this talk you know i'm always looking for feedback and always trying to make things better and always trying to keep things you know kind of real by even those mistakes in so you know let me know if you hate that too all right and a good one you

Channel: Coding Blocks

Views: 436

Keywords: elasticsearch, kubernetes, skaffold

Id: Fu5CpVaiE_E

Length: 41min 38sec (2498 seconds)

Published: Sat Jan 23 2021