Perparing Machine Images for qemu/KVM

hey everyone welcome to this video on preparing machine images for qmu kvm this is actually an extension from a previous post that i had done not too long ago about linux hypervisor setup using the libert qmu kvm stack now if you're not too familiar with the stack i recommend you stop now watch this video or read my blog post i'll have the link in the description of this video but we talked a bit about how it's a really robust minimal capable system for running virtual machines and after posting this video there's a really good comment that came up from this user lomka02sk saying do you know by chance how you can have multiple vms accessible over lan if i clone multiple vms they have the same ip address so i can only access one through ssh they can change them manually but would like to know if it could be automated this is a great question i think it's one that oftentimes causes us some pain when we're first learning about machine images this is why i wanted to do a video about preparing machine images for qmu kvm now what exactly are we getting at here i think it's kind of there's kind of two phases to it so in the previous post we went through and we set up all the different components for libvert qmu kvm and we installed a or multiple ubuntu vms so in this post quite similar and i'm going to have all these steps documented on my website if you want to check that out check the description of this video but needless to say we talked a bit about how you can download an iso and then run it so let's let's start with just that to kind of frame this problem here so i have the command in this post as well that is the command line version of installing this i've also put this in my machine here so if we look at it we run a command called vert install we set up the name for the domain inside of libvert we give it some ram we give it some cpu but we also specify the path that the image will live at along with the iso we downloaded for ubuntu and this is nothing unique to what we talked about last time we'll go ahead and run the vert sh script which in this case will open up a utility called vert viewer so basically it's set up a vnc connection to allow me to start the installation of this specific of the specific virtual machine so let's make that a little bit bigger okay we'll go to english down here install the server all right and we'll go ahead and let this get bootstrapped now and i'm just going to kind of start this installation off so as we talk more about this idea of preparing images and how to solve for unique ips and all that good stuff we at least have a base to to kind of start off with here so once this is up and running we should get the normal ubuntu graphical installation we'll give it just another second here and there it is all right so we're going to just blast right through this you can pause the video or go see my other one for more details but we choose the language we continue we keep going through now it does mention that it's going to give us an ip address and it says 122.17 is what it will give us so let's try to remember that for now it'll be a little bit relevant later we'll hit done no proxy default mirror use the entire disk let me get out of that okay we will do the operation of rewriting and then i'm just going to put in the information now the host name is going to be u20 so that's what i'm putting in here my username will be my name and then i'll go ahead and put in a password i'll also install openssh server because while this vnc works okay i'll be doing most of the configuration through ssh and while we're talking more we'll go ahead and let this initial installation of ubuntu start so let's talk a little bit more about how we're going to prepare this image and it kind of comes down to two things the first thing is once linux is installed i might want to go out and clone instances and instances and instances of this kind of base image if you will and this was i think what the question was really getting at now let's assume in this example i'll be kind of pulling through an example of needing to set up hosts to eventually run kubernetes clusters so in that case i'll typically need a container runtime i'll need a kubernetes agent called a cubelet and i might even need to do some configuration like in kubernetes you have to have the swap space disabled now if you're not a kubernetes user and couldn't care less about it think of these things as some package that you care about installing maybe it's some system utilities or application monitoring agents or whatever it might be there's just there's some things you want to install and you want them available at every instance of your virtual machine and then and this kind of hits closer to where the question was rooted there's questions about how do we clean up that base image to ensure when we clone it things we don't want are are not persisted so as an example this is where the ip dress was coming in how do we do the dhcp lease in other words how do we make sure that if one virtual machine got an ip address from dhcp when we clone it the next vm doesn't get the same ip then they'll overlap you won't be able to ssh in all that bad stuff will happen additionally what about the host name if we name one of our hosts dog there's a pretty good chance that when we clone it we won't want all of our instances to be named dog so in our case with u20 you know how do we make sure that not every single host is u20 all right so that's exactly what we're going to solve today and we're going to do it in talking a little bit about image setup so all of the things we'll do today are going to be completely manual because if your tinkering manual is fine and frankly you know it's a great learning tool to kind of go step by step and learn how these pieces work in you know more serious scenarios like production type environments you're of course going to automate all the things that we're talking about you might automate it through tools like ansible and then print out machine images from there there's a bunch of ways to do it but in this case we're going to do it all manual and maybe i'll do a post in the future about how to kind of automate this whole process if it interests you let me know in the comments now we've got the operating system installing and we've run the vert install command which is now going through the ubuntu install and hopefully ubuntu is setting itself up so right now it looks like the os has actually installed and it's just going through and doing some security updates so that's perfect now let's actually take a quick look at the dhcp leases that libvert knows about or reverse can tell us about now i'm using the default network through libvert if you're not like you're using a bridge network or relying on an external dhcp server this command might not give you any details if that's the case you might have to go you know to if in the case if i had my my router giving me the dhcp leases i might have to look there or just look through some different mechanism but this is a really cool output if we're doing it all through libvert so let's take a quick look here we'll go over to this bash window we'll do that and we can actually see two instances so before this video recording i was testing some stuff out and this is actually a lease from an original install 2294. now you'll notice that hostname is the same as the hostname right below it which is our current install this is the 122.17 we saw before now the reason i point this out is it'll become relevant later when we learn about how to ensure unique ip addresses the key things here are this ip address was given to this client id and what we need to make sure is when we clone the replicas don't inherit the same client id because if they do they may end up with the same ip address which is where you get those overlapping issues and inabilities to network correctly so we'll go back to view vert viewer here it looks like it's getting pretty darn close so let's just set ourselves up for a quick ssh session here we will clear out we will ssh i'll put in my name and what we're going to do once this install is complete is i will ssh into the server and then what i've done is i've taken all the common steps that i would need to do again normally you'd automate this through a more elegant tool like ansible but i just wrote a quick bash script that does the common things it upgrades the system it installs all of the container runtime bits so in this case container d with with docker on top and it's going to do some configuration of that container run time and then lastly specific to kubernetes i install the cubelet which is the cube agent the bootstrapping tool cube admin and then cube cuddle the command line utility and one interesting thing here too is i'm using apt which is the package manager aptitude to mark the cubelet cube adm and cubecuddle as a hold which basically means if i do a future system upgrade i don't want these three utilities to upgrade without me intervening since these are really core to the version of cube i'm running and so on i want to keep these static so i'll go ahead and copy all of this and this will automate some of our initial install on on our future future base image here we'll go back to vert viewer it looks like it has set itself up so i'm going to go ahead and hit the reboot button that it has mentioned and it will restart itself now it did open another viewer window but now we should be good to close out of the vnc and just do everything going forward through the command line so we'll go ahead and ssh in i'll say yes put in my password and then this install script will be done exclusively as sudo and i will go ahead and put in the install sh we will go ahead and make that executable as well so install.sh looks pretty good we'll do a set paste and let's go ahead and paste in all of that so again this is arbitrary packages this could be something totally different depending on what you care about but for now we'll just go ahead and run this install script and this will basically set up our base system for us so while that's installing updating and all that good stuff we'll talk just a little bit about uh the the swap space bit that we're gonna do next with this system so this is again something specific to kubernetes but in the ubuntu installer i'm not really sure how to disable swap space i'm sure there's some way to do it so what happened inside of the virtual machine is that inside of etsy file system table or fstab sounds much cooler right there is going to be a line for setting up the swap space and basically i'm just going to comment this out or or delete the line frankly and that will ensure that i persist in an effect that the swap space will not be there so in fact while this is installing what we could do super easily let's just grab that that lease one more time so this was it was actually the 17 one i think right and you can actually actually you can see the host name now that it uh restarted my uh my human is in the way down here u20 uh is now the host name associated with this ip because it restarted with the host name being instantiated so we'll go ahead and ssh in josh at here and we will put in our password so while that's installing up there we'll go in and edit the etsy f-stab folder or file i should say and then at the very bottom here you can see the swap space so we want to take the swap space and again either delete it or in this case just for reference i'll go ahead and comment it out disabled swap all right and then we'll save that up and now we should be pretty good on swap space so we have installed ubuntu we have run a script or are running a script to install all the packages that i care about for my clones and i have disabled the f-stab so we should be looking pretty good overall i think the top has just finished up so you can see that it has uh put cube let's cube admin and cube cuddle on hold and now we're looking pretty good now we've done the image setup i would say at this point we've got a pretty good base image at least as far as kubernetes is concerned but we need to talk about prepping it to actually be a base image now what does that mean so we're going to focus on the host name and we're also going to focus on the ip address piece that we talked about earlier the ip address piece is super interesting if you've ever run into this you might find how this works to be to be really intriguing so basically what happens is your linux host is going to have some type of dhcp client there's tools like dhclient on desktop linux boxes oftentimes folks use this thing called network manager which can use dhclient or use its own dhcp client but on a lot of servers we use systemd network d now what's interesting about systemd network d is inside of run systemd net if i'm trying to remember leases to okay there is a file that get that gets written here under run um that basically has all the details about the dhcp request so this actually tells me some of the key things where did we go out to get the ip address what ip address were we given what client id did we provide so if you recall this id ending in 84 is the same id if we do that net dhcp watch again that's the same ending here 9ad4 that we see so this is kind of correlating from the dhcp server standpoint this is on my host to the ubuntu box which is right here which is again kind of the client perspective now the question here is how does this client id get generated and how do we ensure that clones do not take on the same client id so to do this we need to figure out where it comes from now what's interesting about this is by default in systems like ubuntu and some other ones as well this actually gets generated from this file now the values are a bit different here but make no mistake this is in fact what is going to be used to generate this duid or client id so what's important for us to do here is to ensure that we understand where this value comes from and how we can ensure it is unique for every new clone that we use now i'll pause here and just let you know and if you do some researching you'll find this that many people will go in and they'll configure their host to use the mac address of their interface as the client id so if i do ip a i can look at my ethernet device here on the vm and you can see the mac address right here now if i went in in the case of ubuntu i can go into this thing called netplan and say use mac address as the dhcp client id that's fine that technically works but it's kind of good practice to make sure that this machine id is in fact flushed and appropriate to the machine and in effect we can also solve the dhcp issue by doing that so let's take a quick look at how exactly this machine id works so we know that machine id is set on this host now there's actually a utility that is called systemd machine id setup a little c program that when you run it it seems like it does nothing right and it actually isn't doing anything if we were to go ahead and just clear out this file for a moment so we're going to do echo n and we will clear out etsy machine id now if we run the systemd machine id setup you'll notice it outputs something differently it says initializing machine id from kvm uuid so this is rather curious right basically what's happening here is the the uh the machine id is figuring out whether that value is empty and needs to be set if it does need to be set what it goes ahead and does is it realizes that it's running virtualized in kvm and it actually pulls the value from sys class dmi id product uuid this is the same value as what we have up here in machine id now what it's getting this from is basically the kvm uuid for each domain that gets stood up if i open up a bottom buffer here and we look on my host for a moment using verse we can ask for the domain uuid for u20 and there is the exact same value so this is where it's pulling it from now the good news is every kvm instance is going to have or every machine id is going to have its own unique uuid but as we learned up in the top here if this file is not empty it will not rewrite what is inside of there and thus you'll end up with the same client id in effect the same ip address as well pretty interesting stuff and if you're interested in kind of the gory details here on my website i have the snippet from the system dc code that describes this process you can actually read it pretty easily showing that if it's empty it goes in and it will check to see if it's in a container if not it'll check to see if it's in a kvm and if it's in a kvm you can actually see where it reads from that specific uuid so i've got that in my site and a link to the the source on github as well if you want to check that out nonetheless this is a long way of saying that if we flush this file before we uh before we restart and start or not restart but stop and start using this as a base we're gonna be good to go so in effect what we need to do is we need to run the echo command that i did before get rid of that and then just make sure that machine id is in fact empty and now we can be sure that it will start back up with a new ip on all the clones so at this point the ip address is taken care of but another concern that i had mentioned which is conceptually simpler but a little bit more involved is that there is a host name set and in linux we set this in the etsy hostname file now how you do your hostname is really up to you in the past i've mimicked my ip address and wrote a little script that always wrote the ip address i've actually come to prefer setting a random alpha based or alphanumeric short sequence of characters to represent my hostname and what i prefer to do is when a new machine turns on the hostname gets written and then it's never altered for the rest of the life of that virtual machine that's just kind of the flow that i like and i like it to be completely random doesn't have to be crazy or fancy or configurable so how do i do this well what i start by doing is i set up a little script inside of user local bin that i call host name init dot shell and i'll make sure that this is executable too so i don't forget so we'll do a plus x here for host i guess i need to do the full path again right user local bin hostname init dot shell all right now on my site i have this script in here if you would like to copy and paste it or try it for yourself let's copy it over and i'll explain what it's got going on it's nothing nothing revolutionary it's pretty pretty mundane so i set this variable to the name of the script because i use it in log messages to be able to quickly find it inside of journalctl the first thing i do is i check to see if etsy hostname file exists if it does exist i do literally nothing at all i exit immediately i don't assume that the thing calling the script will check to see if the file exists i always check it in the script then if it doesn't exist i go ahead and set the hostname now you can go and google what each of these pieces do but suffice to say it prints out way too much data it grabs three characters a through z and it basically writes the hostname from there which it then puts in the hostname file so it's totally kind of ridiculous but it does the job really well so it sets the host name i log a message saying that that's there and then the one thing that i don't love about this script and there's probably a much better way to do it you know if you can think of a way let me know in the comments but once i'm done if the hostname file then exists which ideally it should unless it failed i run a reboot and the reason i run a reboot is when it goes and turns back on i like to be sure that when it goes and grabs the dh the address of the dhcp server it goes in with the host name being set which means that it fills in here i've played around with like different run levels in systemd to try to get it to set the host name early enough i never could make that work even when i put it before all the networking stuff so i don't know what the deal is this works really well it's certainly dangerous because if something went wrong here in theory i could end up in a reboot loop but this is experimentation this is home lab stuff it overall it works totally fine so let's go ahead and save that up we've now got that as an executable file and since hostname does exist just as a quick sanity check i should be able to run hostname init and you can see that it will tell me that it exists and it's a no op now i'm not going to delete hostname and run it right now because it would restart the server which would again redo my machine id stuff that's that's kind of an important thing once we get the base image all cleaned up and good to go it's important to note that if we were to start this server again removing the hostname file cleaning out the machine id we'd have to do all that again because on boot up it will even in the base image it will write that stuff again so you know this is something we're kind of trying to set up and then kind of lock down the the base image and only use it as a cloned thing so the next thing i need to do to make sure that this can work is set up a systemd unit so systemd is an init system um it is on most modern linux distros people have opinions about it that's neat but overall it's there and we've learned to live with it so if we do etsy systemd system hostname dash init.service this will be my unit file if you will and inside of this unit file i'm going to paste in some stuff and it's pretty self-explanatory i've got a description for the unit i do also check to see if etsy hostname exists here while this is redundant because the script checks this i don't like the idea of the script having to understand about things outside of it having to understand like oh is someone else going to check this for me i like them to all kind of only know about themselves this enables me to do it and frankly two checks is just you know sometimes better than one so it works well we call the host name init script and then we call the multi-user target as well for the install directive now i won't get too much into the multi-user target this has stuff to do if you're familiar with like sys5 init with like the init levels i think this covers like three through five or something but it's neither here nor there um you can check out the systemdocs to understand this the most important part is really the exact start in the condition path exists so if we save this file up we'll make sure that it has the right privileges it needs um etsy system d i'll just copy this over so i don't fat finger it here we'll copy this over okay great we've got the permission set up now the last thing i need to do to make systemd work well is i just need to say that i want to do system ctl and i want to enable hostname init service and this makes some sim links a very fancy way of saying that now when the system turns on this init unit is going to fire off which in effect should hit the init script so the very last step here if all goes well is to ensure that this thing can actually run and be successful i need to delete etsy the etsy host name so kind of as a quick sanity check here's an example so if we do system ctl start host name and net service okay then we do system ctl status host name init service you'll notice that condition failed that's exactly what we wanted so the script didn't even get touched because the condition failed nothing needed to occur that's exactly what we want so what we'll do here is we'll run the removal of the etsy hostname file now again i'm not going to start the systemd unit or run the script because it will trigger a restart on this host but now we can see i no longer have a etsy hostname file which means it should be generated on startup and now we've set this up as a pretty good base image to start cloning now you might have some more stuff you want to clean up right you may say oh you know i really you know let's look in here i really don't like the install script sitting around totally reasonable perhaps you look around and notice that there is a bash history of all or maybe is it bash history huh maybe there isn't a bash history on this host um well if there was a bash history of sorts then perhaps you'd want to go in and actually delete that bash history because you don't want all your commands stored in a file and you know so on and so forth but basically sky's the limit we can clean up whatever we want to and once we feel good and i feel good about this now we can go ahead and um we can go ahead and basically power it off now i guess as kind of a final step in this video i'll spin up a kubernetes cluster on the vms just because it's kind of the point so one of the things i'll do to include in this base image is i'll run cube adm config images pool and what this command is going to do is it's actually going to pull down the kubernetes components they come in as as docker images or not docker images container images and just pre-loading them here and having them in the base image will enable me to start all my other machines quicker so this is just you know one extra step i'm doing here for uh for extra credit if you will all right looks pretty good let's make sure etsy hostname is gone is a sanity check great and then to make sure dhcp works we'll just check etsy machine id one more time looks great let's go ahead and power off this host now and now we've got a base image so i'm back on my machine if i do verse list all i'm now considering u20 to be my base image and we are now ready to start cloning these images and see if we can get these working so to reiterate one more time if i start u20 in the future i will have to do my cleanup steps again unless i you know maybe it's best to like quarantine this off and make sure i never mistake it for a vm but again i'm going to keep this off i'm only going to clone from it going forward to clone from it is pretty simple as well there is on the website a quick command that you can run using a utility that we covered in the last video which is vert clone so if we go over here and we'll see if i have a clone sh already set up i do perfect so here's what we're going to do we're going to run vert clone where the original is u20 we're going to name one of the nodes cp 0 or control plane 0. so in kubernetes there's control plane nodes and worker nodes the file is going to be stored in cp0 and then for uh worker zero we'll do the same command we'll do worker zero and we'll set it in the file w0 so let's go ahead and run clone sh so this is not going to start the virtual machines it is only going to clone the images i can still start them as a separate process and what's great here is we'll be able to watch this uh this uh verse net dhcp list as the host come up the ideal state here is they're going to start up and initially they won't have a host name right because when they first start the host name script is going to run all that jazz but then they should restart and we should actually see the host name show up and again we should see each new vm has its own unique ip address because that that machine id was empty thus we expect the systemd you know machine id setup command to run and to gather the uuid unique to the kvm so these have now been created let's go ahead and do a quick verse list all and then we'll start them up so we'll do a verse start for cp0 we'll do the same thing for m0 if i type it correctly that is okay and oh not m0 cp0 and w0 is what i should have said and now we'll come back here and we can see them starting already so here's our first machine you can see it's missing a hostname but this is 104. so it's likely that one zero four is my control plane and just like that it's really that fast it set up its hostname and restarted it called itself beth or beef uh and then the second node started txb it's got a unique ip address just like that we have instantiated two good to go machine images ready to run kubernetes done deal this this is what's so cool about this tool set is it's just so simple to put the pieces together so now we can grab 104 and 15 as an example i can i'll open up a third buffer here all right so we'll do an ssh and up here we'll do 104 yep okay so now we're at 104 and then we'll go back and let's go ahead and grab 15 as well or txb so we'll ssh into that so ssh okay and we'll do this ip right here we'll do yes we'll do that now check this out if we go into bef and we do a quick sudo all right so we'll do that and we will not cube cuddle getting a step ahead of myself we'll do journal ctl for the unit host name service and check this out you can see where it called itself or it got called and it said hostname hostname bef created set a random hostname so on and so forth resulted in the host name and uh in in the in the reboot you can see where after the host name was set the condition check resulted in it not needing to set the host name so it did nothing it's exactly what we want so this will be called bef until i either delete the etsy host name or uh this virtual machine is no longer just like that so i'll go into sudo for both of them here all right and then for kubernetes we can run a command called cube adm init this will actually initialize a kubernetes cluster so this top one like i had mentioned is going to be my control plane cluster and then this bottom one here is going to be one of my worker nodes so piece of cake to set this up and this isn't like kubernetes running in containers on containers these are real virtual machines you know completely configurable i've got full control of them i can then grab the join command right here from the init i can throw that down here okay and the swap space is turned off and all that good stuff otherwise it would yell at me and if we grab this output here and exit out so these are all kubernetes specific things at this point but if we do cube cuddle uh get nodes watch cube cuddle get nodes here we can see our two host names are set up just like that we've got bef we've got txb bef is the master txb is the uh is the worker node and likely the only thing that's holding it back from being ready is that there is no uh there is no cni or networking installed so yep this is a perfectly healthy cluster scheduler controller manager api server all turning on and all this stuff will go to ready the core dns bits especially if a container networking plugin was set up which just would mean me applying a yaml file and then boom i've got a working kubernetes cluster just like that so pretty freaking cool i i hope you i hope you found this video interesting um like i had said i think this really is kind of a natural extension of sort of what we had talked about last time which was setting up the whole libvert stack and now we've taken that a step further and basically set ourselves up to create base images for cube clusters and if your use case is different you're just going to install different packages but now you understand the cleanup mechanics you understand where that dhcp id comes from and all that good stuff so if you like this video i'd love to hear from you shoot a comment if you've got ideas for new videos this video came from the idea of a commenter give me a comment give me a like let me know what you think good or bad and with that said i will see you next time thanks again for watching later all

Channel: octetz

Views: 4,635

Keywords: kvm, linux, qemu, virtual machine, vm, vm image, base image

Id: 6ccpDwT1qnw

Length: 34min 19sec (2059 seconds)

Published: Mon Oct 19 2020