Backup Proxmox VE to the CLOUD! Backup Hook Scripts to push backups to S3-compliant cloud storage

hello everyone welcome back to the home lab today we have a very important topic to discover backups so most people try to follow a three two one rule of backups three copies of data two different mediums one of which is off-site most of us aren't quite there we'd like to be but we're not so what's one of the easiest ways you can get the different media and the off-site backup it's pay someone else to host your data it's a fancy name for using the cloud because the cloud is basically someone else's computer so if we already have good backups on site let's take the next step and make proxmox push our backups off-site say it with me raid is not a backup so today in this video we are going to learn about hook scripts in the proxmox backup scheduler and how we can use them to push backup files off-site automatically when a scheduled backup runs this video is pretty soft for heavy so I'm going to assume you already have a running proxmox setup this doesn't use proxmox backup server this is purely proximox ve the virtualization environment so let's get started shall we so this is my proxbox instance I have a job scheduled to run at 3am every morning to do a backup to my Nas and I want to do a backup every week to cloud storage so the feature I'm going to use to do this is called a hook script so the documentation on this is really sparse it just says you may specify a hook script and there is an example and I had a really hard time finding this example because it's not linked here but I did end up finding a version of it and I've put that on my website if you want to see the original so going back to proxbox we need to add a new job but we can do this to the GUI so I'm going to say I wanted to run every Sunday at 1am it's reasonable once a week I don't need to back up everything quite so frequently to the cloud just for the purpose of this video I'm just going to back up one VM I know this VM is pretty small so it shouldn't take a long time to upload some of these VMS can get quite large and if you're uploading all of them to Cloud you need a pretty good internet connection I'm going to use these standard compression snapshot mode and choose your email settings however you want and retention and retention isn't exactly going to matter because the backup script you're going to write is actually going to delete it anyway also note that I've put it on my Nas and that is because we need to initially do the backup to local storage then we can upload it to the cloud so in this case I've set mine ask the target I could set a local storage as the Target and then every time it does the backup it backs up to local storage and then it copies that file from local storage up to the cloud but you need some place on the local system or local network to temporarily put the backup while you're uploading it to the cloud so now I have this created here so let's go into the console and see what that looks like in the actual VZ dump config so that configuration is stored in Etsy PVE and it's called jobs.cfg so in this case this is the backup we just created so you can see it has options and values so if we want to add a script we could add script and we need to think of what our name for our script is going to be so I'm going to call it vzdump.pearl I'm going to put it in the root directory save that if we go back and we edit the job through the GUI we won't see that it has anything has changed but the script option is still retained in the background it's not going to delete just because we modified it so now we need to create that file and we're already in slash root because that's the root uses home directory so we're going to make a new file called vzdump.pl I'm going to paste this in for my website and we can review it so this example script uses our clone to copy something somewhere else so essentially what happens is the script is passed the number of arguments and environment variables to tell it what phase of the backup job we're in so if we're in jobstart job end your job abort that's the backup job as a whole so if it's backing up more than one VM these only get called once and we get some options of what's going on and then we have other phases here backup start backup end Etc and these are done per backup so for each virtual machine you're going to have a backup start a backup end Etc and those come with some additional datas so we can get the VM ID and the mode as arguments and we get some other things like the VM type the directory we're putting the file in Etc as environment variables and one of them is called Target and that is the absolute path on the local system to the backup file name and so this includes the mount points if we use remote storage this gets the remote storage gets mounted somewhere on the local system and this file is that file so I'm going to comment out VR Saint calls just so we can see what the script would do such as system or die because I don't want to actually call our sync and let's see what happens when we run this so we get the script is not executable oops so that's one of the things that prevents us from putting the script in the Etsy PVE directory because Etsy PVE is synced with all nodes in the cluster so ideally we just like to put all our files in there but permissions in Etsy PVE are defined by the file system we can't modify them so we can't make them executable so CH not plus X so it adds the executable flag so we wouldn't be able to do this if we put the file in B in Etsy PVE and that's why I put it in the user's home directory so if I look at what's going on you can see that the hook script is printing information and it's going in this info line so hook job start it's I'm going to mount PV iridium dump sorry is iridium when we get down here to backup and you can see our tar file is Mount PVE iridium dump VZ dump qmu 111 date time vma.zst so this is the file name we need to copy up to our cloud provider and we need to do it in the back of the end phase and we need to do it for each VM so what the hook script will be called in backup n for every single VM we're backing up so this will get called several times if you have a lot of VMS to back up and then we also have log end and that um in that case we use the log file so we need to write some code in the backup end phase and the log n phase that copies this up to our cloud service and at this current point we really have no way of doing that so we need to install some tools that let us send data to our cloud service so the tool we're going to use for this is called S3 command and it's a command line utility to interact with the Amazon S3 protocol and just because I've said Amazon doesn't mean we have to use Amazon object storage the S3 protocol has become a de facto standard for a lot of cloud providers that provide object storage and so if you use backblaze or the note or whatever they're going to support S3 protocols other object storage and S3 command is going to work with them so it's not a particularly difficult program to use and it's really tiny so let's go ahead and install that in our proximox system it's already in the package repos so we can just install it look at that less than a megabyte so while this installs you're going to want to get your access credentials set up on your cloud provider if I use lenode and I'm not going to share my access credentials here but I will walk you through the process of how to set them up so now that we're done installing it we're going to run S3 command configure and this will configure a file in the root user's home directory that includes the access key to access S3 so it's going to ask us a few questions and we are going to need to find this information from our hosting provider so when it asks you for the region this depends a little bit on your hosting provider or Leno they've specified to always leave the region as us and use the host field instead of the region field so here is the end point and this is where we put in our provider in my case I am using lenode and my post is a new work host so it's U.S east one Dot lanodeobjects.com so in this case we need to give it the F to the bucket which in this case for me is percent bucket dot Us East one Dot linodeobjects.com if you want to use client-side encryption you can add a gpg key I'm going to leave that out use https yes we want to do that we have no proxy server if you have a proxy server for HTTP you put it here so we create a file called s3cfg that includes the configuration for S3 that means we can run S3 command in the future and it will just use these settings without a second attachment every time if you'd like you can put this file in Etsy PVE and Create a Sim link from slash root that'll automatically replicate this file across your cluster so I have a bucket here that I just created called Apple proxmox that's what I'm going to be using to test and you can see that S3 command was able to connect one thing that's important to know about the S3 protocol is you can't modify files you can only put them delete them and read them so even if proxmox supported incremental backups we'd have to store each increment as a separate file or we'd have to download modify and re-upload the file which is not ideal so the fact that proximox doesn't do incremental backups aside from proxbox backup server doesn't really impact this too much we're still going to be uploading the entire VM image to the cloud service every time which is why I'm not doing it super frequently so now that we know how to use S3 command and we know our vzdem script let's edit that script to use S3 command instead excuse what we got so if we're at the backup end phase we got S3 Command put the backup file which is the tar file you include the bucket else upload backup fail and in the end of the log will be the same for the log file so took a snapshot of the VM which it does using ZFS if you're using ZFS then copies that snapshot so now we've called the backup and hook it is 865 megabytes to work with so it could take a while yep here we go so we're getting progress from S3 command let's put into 15 Meg chunks and it has 58 of them so this will be a while there we go backup job finish successfully so now we should have our VMA file and our log file on the S3 bucket but they're also still going to be a local system so we need to delete those so if I do a S3 S3 LS it should tell me that there's the yeah so we got the log file and the Z standard dma file so these are our backup and our logs so they're there but they're also on the local system so the next trick we want to do is we want to clean up these log files after some amount of time because we don't just want our S3 bucket to keep getting bigger and bigger and bigger if you pay for infinite storage then that's okay for you but I don't so searching around I found a nice script that someone wrote in Bash to use S3 command to delete all files so we're gonna create a script called S3 cleanup and we'll paste in our script and make it executable and let's try to run it just to see what happens and I'm going to tell anything more than 10 minutes old okay so Sims only works on days not minutes um it just deleted everything so I'm going to add this to our vzdom file don't forget the final versions of all of these will be available on my blog so let's run that again and see if it does the deletion so as soon as it's done uploading the next part it should then upload the log file and then it should delete everything because we told it negative one day is old there it goes so the job and took it decided to delete the files because they were more than negative one days old so obviously if you change that negative one to be some time in the past instead of the future it would delete old files so hopefully this tutorial helps you improve your backup workflow comment down below if you like this you can like And subscribe I've got a Discord server down below if you want to chat with me later I'll see you on the next adventure and don't forget raid is not a backup

Channel: apalrd's adventures

Views: 14,951

Keywords:

Id: aFDX2CDTJCE

Length: 13min 57sec (837 seconds)

Published: Thu Oct 27 2022