Today I'm going to be going over how to use Proxmox Backup Server or PBS to manage all of your Proxmox virtual machine and container backups. The advantages to using Proxmox Backup Server compared to the built-in VZDump backups is it's a nice interface for managing all your backups from different sources, and it does incremental backups very well. So it uses much less disk space than VZDump backups, and it also is able to only backup changes from the last backup, meaning that backups can happen much quicker, and you can do them on a much more frequent basis with a minimal performance hit on your hypervisor. But before I get into actually how to set up and use Proxmox Backup Server, I want to talk a little bit about where you should run Proxmox Backup Server and the hardware requirements for it. I like to keep my backup software and hardware separate from my actual virtualization. And I think it's best if possible to have a separate system running PBS than your main Proxmox virtualization environment and all your VMs. That way, for whatever reason, if your main hypervisor fails and everything is gone, you can replace that hypervisor and restore all your VMs on a newly installed hypervisor. But I realize in a lot of cases it's not practical, especially in small environments, to be able to set up a fully separate backup node. And in that case, you can run Proxmox Backup Server on a virtual machine or on the same host as Proxmox. Now, I do want you to be warned that restoring backups is going to be more of a pain if it's on that same host, or if it is in a virtual machine, and be ready for a situation like that if it was to come up. But it is less hardware, less power, and less cost if you're in that environment. And running PBS on the same host as your virtualization engine still gives you a good amount of advantages, like being able to use significantly less disk space than VZDump, and it also completes the backups faster. Also, in the case of a full disk failure, if you're still able to mount the disk that you have all your backups on, you can still restore from those files. Now let's talk about what system you should run PBS on. On the PBS wiki, they go over minimum system requirements and recommended system requirements. And I generally agree with these. The minimum requirements are essentially a 64-bit system with 2GB RAM and 8GB of disk space for the programs itself. You're going to need a lot more disk space for your VMs. I generally recommend about 2 or 3x the amount of space that your VMs are currently using, because you're going to want to have space for incremental copies of that data, and you're also going to want to have space in case your VMs grow. Because likely the VMs are going to be getting bigger over time. On the recommended system requirements, they recommend a faster processor with at least 4 cores and being modern. They recommend a minimum of 4GB of memory, and then additional memory is going to be used as a cache. They recommend an additional GB of RAM for a terabyte of hard drive space, which is a reasonable recommendation. And when it comes to actually storing the backups, they recommend using only SSDs for best results. And if you're using HDDs, they recommend using a ZFS special device. This gets into a little bit about how PBS works under the hood, but essentially it is very IO intensive and does a lot of file system operations. Which is why a ZFS special device, which stores all of these system metadata, can significantly speed this up. But if you're in a home environment with a single server, not huge amounts of changes, and when it comes to restoring backups, you don't need the most minimal amount of downtime, I've ran it just fine on a single hard drive without any major issues. Things like backups, restores, garbage collections, and other operations will take significantly longer, but it still works just fine. If you're in a production environment, especially if you have multiple nodes, going all SSD if you can afford it for disk space is great. Otherwise, try to get an HDD array with some SSDs working as a ZFS special device. Proxmox Backup Server can be installed two main ways. The first way is downloading their ISO file here on their website. Once downloaded, you can boot off this ISO file in a virtual machine or on physical hardware and install it, and you'll get a Debian OS with Proxmox Backup Server and all of that software all set up and ready for use. The other way to install it is using an existing Debian or Proxmox installation, installing the correct repositories, and then adding it on there. Since this is kind of a simple tutorial, I'm going to go over the ISO method, but if you already have an existing Proxmox server, you have to set up the package repositories for PBS, and then you can install this little command here, and PBS will be installed alongside Proxmox Virtualization in a separate web UI. PBS installed this way will have all the same features and functionality as if you installed it from the ISO. The difference is it's running alongside whatever else you had on that host, like Proxmox Virtualization. For this demonstration, I'm going to be installing PBS on a virtual machine inside of Proxmox. If you're going to do this in your actual setup, just be aware that if your main hypervisor isn't booting, it's going to be a pain to access all your backups that are inside a VM on that hypervisor. But let's take a look at this VM that I've created for PBS. So I have 16 gigs of RAM, I have 12 cores on this system because my host has plenty of cores to work with, and other than that, I just have the ISO image that's ready to be booted off of. So let's turn on this virtual machine and start the installation. And now I'm in the PBS installer. I'm going to agree to the EULA, and then I can configure my disks. I have two disks on the system. One is a 32 gig disk that I'm going to use for storing the boot files, and then the other disk I'm going to use for storing all of my backups. I like to keep it separate, but you can store both on one disk if you want to. Under options, you can see the exact same options you see in Proxmox Virtualization for how you want it to be installed. I'm a big ZFS fan, so I'm going to use a ZFS RAID 0 with just my single 32 gigabyte drive. So I'm going to click OK, and then click Next to continue through the installer. And this should be all about the same amount of stuff that you see in your Proxmox Virtualization, so I'm just going to select all the info here that's applicable to me right now. And now that PBS is finished installing, I see a little URL here that I can go to in a web browser to access the interface. And for a lot of users, that's probably more than all you need, and you never actually need to log into the local system itself. I'm going to click Accept the Risk because it's not a signed certificate, and then I can log in using the root account and whatever password I set up earlier. And now that I've logged in, here's the web interface for PBS. I'm by default on the dashboard, which shows me a lot of different information. There's nothing here yet because it hasn't done anything, but it will show a lot of cool information about what's happening on the system and what you might want to take a closer look at. But before I can start putting backups on this system, I'm going to have to configure a few more things. The first thing I'm going to want to do is go under Administration, and then under Repositories, and I'm going to go change some repositories. Either I'm going to go set up a subscription that I have for this system, or I want to set up the No Subscription repository. If you don't do this, it'll update Debian when you update the system, but it won't update Proxmox itself, so you want to make sure you set that up. So I'm going to go under Add, and then it's going to prompt me that I don't have a subscription. I'm going to go under the No Subscription and click Add. And then I'm going to remove this Enterprise Depository if I don't have a subscription because otherwise it will throw a warning every time I try to update it. Now I'm going to go under Updates, and I'm going to go Refresh, which is going to refresh the list of available updates that I can get on this system. And now that this task is done, I'm going to click Upgrade to upgrade all the packages to the latest version. This is just using App Update and App Upgrade if you're used to using Debian or Ubuntu or similar distributions. I do this first before doing anything else on the system just to make sure you have the latest version. After it's finished the updates, I reboot the system because often you can get a new kernel which requires a reboot to apply, and there's no reason really not to reboot at this point. And now my PBS system is finished rebooting after the updates. Some other settings that you might want to work with before you start setting up the actual backups on this PBS system include two-factor authentication and user management to make it so your system is more secure and has the users you might need. Setting up certificates might also be a good idea if you have a certificate authority so that way it doesn't have that HTTPS warning every time you open the website. But with that out of the way, let's start setting up these disks on here. Under the disks here, I can see this one terabyte drive that I set the system up in the virtual machine that I want to store my backups on. I'm going to have to put a file system on this disk so that way it can store backups. I can run whatever file system I want realistically, but I'm going to choose to use ZFS because I like using ZFS. Other file systems like EX-T4 and XFS will also work just fine here if you have a single disk, but the main reason I see to use ZFS is you have a mixed combination of hard drives and SSDs, and you want to use SSDs as a special device to speed up some PBS operations. So I'm just going to call this backup on here, add it as a datastore, have compression on, A-shift set to 12. All of these are default and they make a lot of sense for what I want to do here. So I'm going to click on here, and in a second or two, this is going to be finished creating. And since I said to make a datastore, I'm going to have a datastore added right now that I can use to store all of my data on. But if I don't create it that way and I want to make a datastore on another directory that I might already have or mount it a different way, I can click add datastore, I can call it test whatever I want it to, and then maybe I can just give it any arbitrary path that it might be mounted to. And now that my backup datastore has been made, I can take a look at it here and I can see this dashboard with information. Right now, it's pretty much all blank because there's no data on it. And there's a lot more tabs that show more information, but I'm going to talk about that information in a little bit when I actually have backups put on this system. So let's get into how you actually put backups on this system. So in Proxmox Virtualization Engine, if you go into a data center and then storage and then add and then P Proxmox Backup Server, you can add Proxmox Backup Server as a storage method. And then when you go to backup your virtual machine or something, it'll show up as a repository to put that data in. So under ID is just what you want to be called. So I'm going to call it like backup VM in this instance. Server is going to be the IP address of that server that you want it to connect to, the PBS server. So that's going to be this 192.168.1.206 in my case right now. Username is going to be root, but you also have to put the app PAM at it if it's going to be PAM or something else. But by default, it will be at PAM. The password is the password on that root. It did auto fill it for me because it's the same password there. The content is going to be what it's storing, which is only backups on PBS. I'm going to say being able to use by all nodes. It's enabled. And then on the data store, it's going to be the data store name in PBS. So that's going to be backup in this case. And then the namespace would be left by default because I'm not using any namespaces right now. And then on the very bottom is the fingerprint. The easy way to get that is to go under dashboard and then show fingerprint, copy, tab back over, paste it, click add, and now it's added. And then if I click on backup VM right now, I can see it has that one terabyte of space available for me to store all of my backups on. And then if I go to a different virtual machine, I can take a look at it. And then if I go into backup, I can say backup now I can point it to the correct storage that I want, which is backup VM. I can say use the snapshot method. So that way it'll take a snapshot of a running VM and doesn't stop the VM. I can set notes or any other information that I want. I think like backup and I can see this information about my backup job that is starting right now. And because this is all done on SSDs, it should be pretty fast. So it looks like it's going at over 100 megabytes per second, depending on what's going on. But you likely want to be able to automate your backup. So that way it runs them automatically every given period of time. And in order to do that, go into data center, backup, add a backup job, click under schedule, you can set to whatever you want. So maybe every 30 minutes I want to be super aggressive with my backups. And then you can say include selected VM. So I'm going to select all my VMs in this case, except for my PBS demo, because I don't want to be backing up my backup machine. And then I'm going to say, OK, and now every 30 minutes, it's going to back up all of my VMs to the PBS demo VM. So I'm going to click run now and it's going to start running all of those backups, actually to run it once my existing backup job finishes. Proxmox and PBS are back up the VM differently depending on the storage type you're using and the VM state at the time that it is being backed up. I want to make a note about snapshots. If you can make a snapshot of your VM, that means the storage underneath supports snapshots and making backups work a lot better. So as a demonstration for how a few different configurations work, I have a couple of different VMs and containers to try to explain it with. When it comes to containers, if the container isn't running, it doesn't really matter if you can make a snapshot or not. It'll just do a backup. If the container is running and you can make snapshots, it'll make a snapshot of that container and backup the container. If it is running and it can't make snapshots, it will suspend or pause the container, make a backup and then resume it, which means there will be some downtime in that container. So make sure your container can have snapshots taken, otherwise there will be a little bit of downtime. When it comes to virtual machines, it'll do a full backup every time and have to read all of the data in the VM if the VM is shut down. If the VM is running and can take snapshots, it'll only backup the difference since it started running. So generally, I'd recommend that you don't run super frequent backups of nonrunning VMs because that means you have to do a lot of additional reads. And likely if the VM isn't running, you don't need to back it up as often. So if you're doing something like backups every 30 minutes or every two hours, I'd only backup running VMs and that'll save a lot of backup time. And then have a separate backup job for VMs that aren't running and don't need to be backed up as frequently. And just like containers, if the VM can't have snapshots made, it'll have to suspend the VM, make the backup and then resume it. I would recommend if you want to use PBS to try to store all your VMs on storage types that can have snapshots made, like ZFS, LVM or QCOW2 for VMs. So that way you can take a backup of a running VM in container without any effect to the operation of that VM or container. One other thing to note is if your VM is hibernated or suspended, it is locked and it will backup will not be taken and it will throw an error. And if you do get job errors like I do here, you can double click on this task and generally look through it. If you scroll at the bottom, it'll take task error, but sometimes this can be a pain to look at. One thing I like to do is put this in a text editor. I like using Notepad++, so I'm going to open Notepad++ on this system, paste in all the information from this task, and then I'm going to search for like error on here and I can see that backup VM 102 failed because VM is locked suspended. So there I can see that a suspended VM can't be backed up using that information. Now that I've done a couple backups, let's take another look at PBS and see how things are going there. If I go back to my data store, all this information seems a little more filled in. I can see I'm starting to use a little bit of storage information on here. I can have some transfer rates when I'm moving data around and I see my IOPS peak when I'm doing lots of different backups. When I switch to content, I can see all the VMs and containers that I've backed up on here. If I click the little plus symbol, I can see all the different versions for that VM here. And then under options here, I can rename or edit this so I can change the names and notes for it. I can verify it, which means it'll read all the data and compare it to known good checksums of that data. I can protect it so it can't easily be deleted. And I can also just delete that specific snapshot if I want that one to be gone. I also in here have an option to add a namespace and namespace can kind of be thought of as a fancy folder. So I might be able to call this like extra VMs or something. And then if I wanted to put backups in this namespace, what I do is I go back to my Proxmox virtualization, go into data center, and then under storage, add Proxmox backup server, and I can put that namespace in and it'll be in that folder. Some things happen specific to a namespace, so it can be nice to have different folders for different types of VMs, for example. Also at the top, I can reload if I want to check it again. I can verify all, which means it'll read all the data to make sure it is correct. I also have an option to prune all of them. Let's talk about pruning now. Pruning is what PBS refers to as removing old backups that you don't want. I can add a prune drop here, which I generally recommend you do, that keeps the last daily, monthly, hourly, weekly, and yearly backups that you want of your VMs or containers. So I might be able to say I want the last like four, keep them from the last maybe 14 days, keep them from the last three months, and that's good. And it'll do that. One cool way if you want to see what actually exactly does this mean is if I go under one of these and say prune all, I can do a drive run. And that means it'll just show me what it will delete. So if I say keep last two and do a dry run, it'll tell me it will prune this group and it would keep these and delete these. So that's a good way of knowing what's going to happen when you delete backups and how to keep it. And because of how Proxmox Backup Server works under the hood, it actually is pretty darn effective and doesn't have a lot of the normal compromises and issues when it comes to full and incremental backups because it uses everything in chunks. And what that means to you as a user is you can keep a lot of old backups without having the whole backup chain that you have to keep. It generally handles keeping very arbitrary backups that you'd want to keep quite well. The other thing you're going to have to do is garbage collection. Garbage collection will run this command right here. And what essentially it's doing is it's looking through all the different backups and taking a look at what files on disk do those backups need. And then it says I'm going to remove all the files that it doesn't actually need. This actually deletes the data, whereas pruning just marks it as unneeded and the next garbage collection will delete it. By default, it says it has none right here. So I'm going to edit this and say maybe doing it once every week makes a good amount of sense to me. Sync jobs can be done to sync this server with another server. It is done in a pull matter. So you want to set up the sync job on the server that is pulling the data to it. So if you have an onsite PBS server that all your backups are going to and you have an offsite one, you set up the sync job on the offsite server. Verified jobs are where it reads all the data to make sure it's read correctly. I can add one to maybe do it every month or something. I wouldn't do it too often. Generally, PBS seems to be written well, so it doesn't corrupt backups in any way. ZFS is also very good at keeping backups well. And if you do this too often, you end up reading a lot of data under options. Under options, you get a few extra options for tuning, maintenance modes, verifying new snapshots, and notifications. Generally, I wouldn't touch those unless you need to for a specific reason. And now that your backups are made, let's get into the most important thing with backups, which is being able to restore those backups. So for example, if you have a virtual machine that you want to restore to a previous version, let's take a look at 101 in this example. I can click on it, go under backup, make sure my storage is set correctly, and then I can take a look at one of these. If I want to restore back to this VM, I can click on restore right now, and it's going to say restore from backup configuration, and it'll just essentially restore the exact same configuration as this VM was when it was backed up. So I'm going to click restore, and it will overwrite my VM. So anything after this backup will be lost, so be very careful in doing that. I can also have it restore as a separate VM, a new one. So maybe if I had a I screwed up in a now at this state, and then I had a VM for the previous state that now I've edited and working from. That can be a good idea if you might want to copy data from the previous bad state. The other way I can restore VMs is I can go under the storage right here. So under backup VM, under backups, I can see all the backups that are stored by here. One thing to know of is I can sort by something. So maybe I only want to backup 101, for example, and these are all the ones with 101 in it, and I can restore it. The other thing is I can also mount this backup VM PBS storage on any Proxmox host, and this will all appear. So this is an easy way to migrate or restore to a new host if something happened to your host and you want to be able to store all the data back onto it. The nice thing is restoring backups and PBS is almost identical to restoring VMs with VZ dump. So it's a relatively easy and painless process to use if you're used to using Proxmox before. That should be most of what you need to keep a basic PBS server up and running. What I'd keep an eye on is disk space usage to make sure you don't use up too much disk space. I'd also make a look at all these backup jobs to make sure if they have errors, what the errors are and why they're having them. The other thing I'd keep an eye on if you're having a PBS system is if you go into backup, take a look at guests without a backup job and make sure that it doesn't have a backup job that you actually want them to not have a backup job because otherwise it's pretty easy to miss a new VM you created and didn't get added to the backup configuration. Another thing that can be nice to look at is just a dashboard here. Lots of useful information, especially these longest tasks. If these tasks get into the tens of hours, you might want to look at why it's taking so long. For example, if garbage collection tasks are taking forever, you might want to lower the amount of objects on there so it can do those quicker or get something like a more SSD storage to speed those up. The other cool thing PBS does is has an estimated fold aisle so that way you know about how long it's going to be until it's full at the current rate. Keep an eye on that if that number is super low, especially if it's been in use for a while, you probably want to change your backup tasks. Otherwise, just make sure the backup doesn't fill all the way up. The other thing I keep an eye on is system load utilization. You can look at graphs here in administration and also look back at longer periods of times if you want and just make sure that CPU usage isn't sitting too high for any extended period of time. Server load can also be a good idea to keep an eye on because if server load is too high, that's another way of seeing the system is possibly being overloaded and you might want to get faster CPUs, disks or more RAM. Hopefully, this video is useful for helping you set up your first Proxmox backup server system and make sure your backups are running correctly. Let me know if you have any questions about this video and any topics that you'd like me to go in more in depth with. Thanks for watching.
