Proxmox Recovery: Getting VMs running after a host failure

today I'm going to be taking a look at proxmox recovery how to get your VMS up and running again after your host encounters a problem I'm going to take a look at recovering VMS from backups including proxmox backup server and the VZ dump backups and also if you just have the disk that had the VM files on it so I'm going to be taking a look at trying to get them from ZFS storage file storage or lvm storage back into running VMS to demonstrate this I created a test proxmox system with VMS and containers stored in many different ways I have my VMS backed up via VZ dump backups and proxmox PBS server and I also have VMS stored on many different types of disks including ZFS lvm and directory storage and now I'm going to break that VM so it can't boot anymore and then I'm going to try to recover all of the VMS on that system so I just went in Via ipmi and deleted a ton of stuff you shouldn't delete on that system my proxmox server won't boot anymore it just gives me a kernel panic upon Boot and I want to get all the data from this system I have a separate boot drive in this system that doesn't have any VM files I care about and that means I could overwrite that drive with a new copy of the operating system if you have your proxbox operating system installed on a drive that has VM data reinstalling it will format the drive and remove that data so you'll want to get a separate drive to install on or make a clone of your previous disk making a clone of all the disks with data is a good idea anyways because if you make a mistake you still have another copy of that data I will need to reinstall proxmox on the system to have a OS to run the VMS on I can recreate most of the configuration while from the setup or from the backup files but the rest of the data I will have in the VM images which I will be able to restore the one thing to note when reinstalling it after a recovery is to make sure you don't touch any data on the drives you don't want to lose so I'd be extra careful and unplug any drives from the system that has data you don't want to be used just so you can't accidentally install it on that drive when you're installing proxmox now the proximox installer has finished and I have this blank proximox system right here the boot drive has no data and almost no configuration on it and I'm ready to store the VMS from the other drives in the system depending on your setup you might have to do some configuration first as the configuration is not backed up by default so you may need to do things like set up your network users permissions and other things but I have a relatively simple configuration set up here so I just installed updates and I'm ready to go a lot of what I'm doing requires the terminal so I've SSH into the system's root account and I'm going to start running some commands to try to get some of that data back and find out what the drives have so the First Command I like to run is LS Blk and this lists the different block devices on the system so this SDA Drive is my new boot device that I've just set up so it doesn't have any VMS on it as an sdb and SDC are where I've set up all of my VMS earlier that I want to restore SDC has all this lvm data that it's automatically detected so this has my ovm VMS that I'm going to restore as an sdb has this big partition and a little partition I set it up as ZFS so if I wanted to detect the ZFS drive I can do Z Pool import and it's going to see that z-pool that I have now I want Z Pool import and the name of the pool and it will import the pool and mount it and make it so I can see all the data on it it says I have to run Dash F to force it because this system's never seen that pool before now we're going to ZFS list I can see my ZFS data so all of these story ones these are all ZFS datas that I've set up I set up a few different ways of storing data to demonstrate how it can be restored so I created a few data sets here one of them is PBS that I'm going to be using PBS to install it from VM file is just a file store this is the same as if you used ext4 xfs or another file storage solution to store it and then I have ZFS VM stored on here too I'm going to restore is my bz dump backups and in order to do that I'm going to remount my directory file that I mounted earlier on this new install and it'll detect all the files that were previously there so if I look at CD slash store slash a VM file what I'm going to notice is it's going to have all the proxmox data so dump images private Snippets template and then if I add this in proxmox to a new data store it will detect this data and it'll just show it up so what I'm going to do is I'm going to go to Data Center storage add directory I'm going to say um new story file and I'm going to do slash storage VM file and it's going to start detecting the data once I set it up I'm going to make sure to check all of these because I had many different types of data and I'm going to click add and in a second or two it's going to show up here shows a little bit of usage and I can see all of the data on this I can see my backups I can see my VM disks I can see my ISO images I put on here my templates I'm lots of different other data that I have on this system now it's not perfectly correct it's confused about some things like these VM raw images are actually containers not VMS but I'm not going to look at these VM disks because I have backups and backups are really easy to restore on proxbox and another thing I didn't do is I'm going to create a new data set in ZFS to store my restored data on because I don't want to store the data that I've restored on the same ones that has existing data because you can have some possible issues because it might be using VM numbers or something else now that I've created a spot to put my VMS on once I will store them I'm going to click on a VM that I want to restore I'm going to click restore I can see like notes of it like the name that I had it earlier I can see like the date and time I made the most recent backup I got to click this new directory that I just created this empty spot to put it on I can select the number and by default it'll just keep everything the same and restore it and now my VM has finished restoring for backups I can click Start it's going to run the VM it has the same name has all the same Hardware configuration and it's just ready to go one issue you may run into is a VM won't be able to start because the CD drive has a CD that it isn't able to access anymore and if this is just a boot CD you won't need it anymore so I'm just going to edit and say don't use any media and now it won't be using that old CD Virtual Drive so I can start the VM up right now so if I take a look at this this is a fedora VM and it's just going to start booting up and I have a fully working Fedora system right here and this is the simplest way to do restoring files in proxbox you just put them in a backup storage directory on a new system you click restore and it'll return it just to the configuration you had earlier I'm also going to do this with a container just to show how a container can be done too then I'm going to set it up really easily and just click restore and since containers have less information on them that's super fast I just finished right there on my little container can start right now and I just have a working little container running duvian Linux now I'm going to restore data from PDFs so PBS has its own way of storing data the that's not really accessible to a normal user just like a basic image file so you're going to need to install proxmox backup so to read those backups you have created so now I'm in the control panel for PBS and I need to get that data back again I will need to add it as a data start and what I'm going to do is I'm going to call it like PBS data and I'm going to put that path in of the system but unfortunately I can't just do this right here because it's going to say that I can't create it because the file exists so I'm going to actually have to go into the config files in the system and edit those to add it so what I'm going to do is I'm going to go over to my proxmox system and I'm going to edit the file and slash Etc proxmox backup and I'm going to run vim datastore.comfig and this is what keeps all the information about those data stores so I'm going to edit some text here add data store um colon the name I want so like PBS store and then I'm going to add a tab and then the path that it wants so it's like Story PBS is what I had the data in so now if I refresh this I see my PBS storing data and nay those are all my VMS from it I can see all these VMS here and these are all my backups I can do if I want I can just download these backups and put them on another system right now but if I want to be able to restore them on my proximox server I'm going to go to Data Center storage and add this proxmox backup server repository and it'll detect all the data that's already there and now I've added the PBS data store and I can see all these backups so it's the same as the other backups I can just click on Restore for these backups select the storage that I want it to be restored onto it has all the settings for the VM I click restore it'll start copying files and within a minute or two it'll be ready to start that VM up again unfortunately though you don't always have access to backups to restore the VM from sometimes the only thing you have is the VM disk that it's booted from so this would be something like an IMG file or ZFS something like the ZFS V devs that it uses or an lvm I'm going to be taking a look at all these different types and how you can get it going so the general process for all of these though is to take that virtual image file create a new VM and add that image file that is already being used so let's take a look at how we can do that so I'm in proxbox right now I'm going to say create a VM and I want to emulate let's say I think I have a Windows 10 one so I'm going to call it like when pen restore unfortunately this configuration isn't stored with the image and it's stored on the proximox server and if that data gets lost you don't have access to it unlike the backups so you might have to do some trial and AO to make sure you get everything right one thing to make sure you get right is the ubfi versus Legacy BIOS I believe this was a Legacy BIOS VM but if it doesn't show up as a boot device at all try changing that to the other one and see if that will work discs I'm going to actually delete all of these right now because I'm going to be adding that later on CPU and stuff I'm going to be adding about what I thought I gave the VM I'm not sure exactly what config this had but I can tweak this later on once I get it booted so now I have a Windows 10 restore one which doesn't have any drives to boot off of and I'm gonna have to get that drive from the system so I need to look around and see where it's actually is and I'm going to first take a look at if you're using file stored so I'm going to take a look at that new story file which was the one that used to have all these VMS and under VM disks I have this vm106 and that's the VM that had my windows VM I want to restore but it's 106 instead of 105. so I'm going to do a little tweaking to make sure I can actually get added to that VM so what I'm going to do is I'm going to find the configuration file for that VM and then I can edit it so it has kind of custom settings that you can't normally add in a web page and all this configuration data is an under slash Etc PVE commute server and if I run LS I can see the names of all my different VMS and no config files see I'm trying to get this onto 105 so I'm going to edit this with a text editor in this case to vim and then I'm going to edit this here and in order to get the data I want I'm going to use SATA one right Theta zero so this would mean the first SATA drive connected to the system if you don't know exactly what to do you can read like the other configs to get an example so for example if I looked at this 103 config I could see it's using io1 the other thing you could do is you could create a new virtual disk and see like what type of name it uses so I could also go create like a hard disk with SATA on it I could say I want it to be this empty one so now if I edit this um 105 configuration I will see that it's added that file right here and now I'm going to add the same name of that storage repository here and then I'm going to put a colon the VM number that I want it to use it from so in this case that's one zero six one zero six I then slash the name of the full name here so that vm106-z disk 0.q cal2 and then I'm going to save my text file and go back to the GUI and see if it's shown up there so I can see I now have a hard disk here it's showing up as something let's just see what happened it's going to try to start it and it says it's okay and if I open the console it's having an issue here and my issue here is because it's not set as the correct boot device because when I add this later it doesn't adjust the boot device so if you have that issue you're gonna have to go under the options boot order and then add that say the device you just added so then it can actually boot from that SATA drive so now I'm going to do start and hopefully it can boot from the SATA drive that I've added to it and yeah that looks like like a Windows install that's starting up right now and now I can see my Windows 10 VM is booted to the desktop and it's on the new storage so I know it's all running there and it's all set up correctly so that the whole system is going to be working fine after this restore now let's take a look at getting it from different types of storage so I'm going to take a look at lvm now and lvm is a little bit different than file because you can't just CD into a directory it has the same idea of kind of like a file container but it's a block device in Linux it shows up instead of a file so in order to do that we're gonna have to mount the storage and see if we can get it mounted so I'm going to just do lvm and see if it will find the existing volume groups and we're going to just show it that and we're going to say oh old snoo which is the name of my second one so I'm going to add it and then it's going to show up here and I'm going to see if it shows up with the VM disks here and yep I see all these VM disks they have all the data on it and I can start another one with it and let's actually start a container with it because I haven't done much with containers here one thing that's a bit annoying right now is two of these are containers and three of these are VMS and all showing up as raw data and nothing shows up under container volumes so I'm gonna have to kind of take a peek and try to figure out which ones are containers and which ones aren't so if I run LS BLK again I can see all these devices these ones here and the amount of size and this eight gig one is definitely a container because you cannot only make a VM that small and that's the default for a container but one of these also is a container and I'm going to actually run some tools on it to see what type of data these containers have so I can try to make a good guess so I'm going to run like part X and this lists the partition on the drive or the file that I'm giving it so I'm going to essentially give it the lvm file and ask what petitions it has on there so now I've ran part X on all of these and it shows something interesting so on three of these it shows these partitions that it has as like an EFI system partition and then a main data partition and this is kind of the default for most operating systems how they partition the drives but these two said it can't read the partition table at all and one of these is the one I already know is a container so it looks like 110 and 111 are both containers sometimes you have to do a little bit of detective work to kind of find out which one of these or which so now I'm gonna have to create a new container and then I'm going to map it to that data that I've already created so I'm going to do like fixed 110. let's call it that and I'm going to set up a basic password it's going to set up a lot of data but I'm going to replace a lot of that data later on I'm going to pull up one of the templates I had in the past all the state is going to get replaced so I don't mind it but I have to keep this here for now so I'm going to do like CPU memory Network just kind of set up all the defaults for these because I just want a basic container for me to replace data in later so it's going to create this one and then I'm now going to have this little container right here that I'm going to want to recover data from so this is the CT 106 that I have and under resources this is that root disk that I want to replace so now I'm going to edit the container config file the container config file is a lot like the VM config file it's in a slightly different spot it's in slash Etc PVE lxc if I want LS I can see the config file for both of the containers my system has right now I'm going to go edit that root disk for 106 that I've just created so I'm going to do vim106.conf and then there's root FS I'm going to go change that to the one that I knew was a root FS earlier so I'm going to actually go to the one right here so it's old snow is the old lvm one that had the data I wanted so then I'm going to take this 110 disk and start cocking now that I know that my old VM disk is this 110 desk on Old snow I can add it here and it's going to be about the same as the VM where you have like the name of it and then you have the full one um 110 and then slash VM dash one one zero Dash disk Dash zero and hopefully this will just boot right up once I edit it so now if I go to fixed 110 on my new one it says it has a different root disk let's see how it boots on the different root disc so it actually couldn't find that name and was saying it as an lvm name wasn't valid so I wanted to figure out how I could make a valid name so what I did is actually created a new container to find out how it names it so I went on a root disk and I can see this root disk here and then if I look at the config file for this one I see on the other one if I go to vim107 the new one I just created the root FS doesn't have like a 101 slash or something it just has that name so now I'm gonna go edit my previous one with that new data so I'm going to just remove this 110 slash save the config file and then hopefully it can boot from that drive again so I'm going to hit start and now I have my container running with the previous data from the old container copied to it so this has all my old data but it's a new container running it with a new configuration let's take another look at ZFS because it's a different way of storing data and I just want to see how it was storing onto ZFS works so I'm going to add my ZFS one so I got my ZFS mounted and it's showing some data but it seems to be weird showing subvoll on some of these it shows four of these and one container but I know I have two containers I know this one with eight gigs of storage is also a container and a lot of restoring from ZFS is like restoring from lvm because ZFS and proxmox uses the Z Vol which is essentially a virtual block device that ZFS creates and it stores the VM in it and if I want to see where those are I can run lsbo K again and all these ZD devices are what I have another way to see this is slash Dev slash zfall story and a name and then I can see under like VMS and I can see all my different ones so I believe I was using 102 as a previous windows VM so let's restore that one again so I'm going to say like win 10 restore 2 and I'm going to just create another simple VM Without Really any data so I can add my disk later on now with this VM I just created I found the VM disk name it's in my old ZFS one that I pulled it from I created an empty VM to copy the data to and now I'm going to go edit the config file for this VM and add that SATA device with the ZFS pool and the ZFS block device that it's named after well it turns out I actually rescued the wrong VM because this is an Ubuntu VM not a Windows VM that's one of the annoyances I just know the number that I previously used not the name this method can get relatively complicated and a lot of Linux knowledge of knowing where those files are and a lot of just tweaking and playing with the config files it can definitely be a challenge to restore this way which is a great reason to make backups using PBS or those VZ dump backup files because it's essentially a one click restore versus hunting for files and editing config files but it is nice that you can do this if you need to restore and if you have a image of that VM you can get it running from just that image and no other config data needed let me know if you've had to do any of these recovery methods before and how it worked for you I actually had to do this on one of my personal servers because I made a mistake with the boot configuration and all I had was backups and a few disk files luckily I had PBS backups for most of the VMS but some of them I just had disk files and had to use this annoying method to try to hunt those files down make a new VM and configuration it's really worth it to make VMS because restoration is so much easier thanks for watching this little proxbox video And subscribe for more prox monks and other computery videos in the future

Channel: ElectronicsWizardry

Views: 13,550

Keywords:

Id: xgLr9uaMqro

Length: 20min 9sec (1209 seconds)

Published: Tue Jan 17 2023