Whats the faster VM storage on Proxmox

today i'm going to be taking a look at proxmox storage and the different methods of storing data on a drive i'm going to be comparing the performance and features of different ways to store data on the drive including directory lvm lvm thin zfs and btrfs the overall summary is lvm is the fastest and zfs has the best features and generally i'd recommend using zfs because the lost performance when using zfs compelled to lvm is well made up for in my opinion by the extra features it gives you which i think overall is a better experience for storing vms now i'm going to go over each of the storage types individually so let's take a look at lvm first lvm is the linux volume manager and it takes a group of physical disks on the system merges them into a volume group and then in that volume group you can make logical volumes on top of it so it's kind of an abstraction layer between a physical disk and the logical volumes on top of it and then proxmox takes these logical volumes and gives it to the vm to store data on the disadvantage to this is it doesn't allow for thin allocation or snapshots but it is really fast because it's fairly simple one thing to be aware of is snapshots are used for more proxmox than just the snapshot section in the vm they're also used for backups as it allows a snapshot of the vm to be taken where it's running and back up that snapshot you can't just back up the disk of the whole vm when it's running because the disk will change and you have kind of a mismatched disk that's been backed up and will not allow it if you do not have snapshots on your volume you can only backup the vm by either suspending the vm or shutting down the vm both of which will cause downtime on the guest to work past the limitations created by lvm lvm thin exists and lvm thin uses the same linux volume manager on the top but adds some extra features that lvm has that allows for snapshotting and thin allocation thin allocation means if you have a vm and you're assigned it for example one terabyte of storage but it's only used 100 gigs of that one terabyte virtual disk it only uses 100 gigs on the physical disks on the host and that can save a lot of space if you have lots of vms with kind of empty drives and near empty drives snapchatting is also super nice but it does have a bit of a performance penalty it's slower than lvm with these extra features added as there's a bit more overhead to add these features the next fastest method of storing data is directory set up correctly directory isn't a specific way of storing data on the disk it's just kind of a way to allow you to store it on any available file system in linux and you can use things like ext4 xfs zfs btrfs or any other file system you'd like some of these file systems i mentioned like zfs and btrfs also have a dedicated mode that uses the features of those file systems so i'd normally recommend using that dedicated mode and not directory but directory will work and often makes sense if you want to store things like iso images or backups or templates on zfs storage because the zfs in proxmox does not allow the storage of that type of media directory also allows for two ways for the vm virtual disk to be stored one is via raw image where it just takes the bytes that are stored by that guest operating system and puts it into a file there's also qcad 2 which allows for snapshotting and qcad 2 will keep track of which blocks are being used by which snapshots so you can create snapshots delete snapshots and roll back to snapshots easily qcal 2 does have a bit more overhead and means it's a bit slower so you take a performance hit for creating snapshots performance wise raw on a high speed file system like ext4 or xfs performs a little bit slower than lvm but very small generally and q cow 2 on those high speed file systems is a little bit slower than lvm thin these are still very fast ways of storing data in proxmox though the next method of storing data in proxmox i'm going to take a look at is zfs zfs is one of my favorite ways to store data in proxmox because it's a very flexible file system with lots of features and some of these features that other ones don't offer include thin provisioning snapshotting compression check sums of data and built-in raid functionality zfs is also great because it's one tool that connects to the physical disks and goes all the way up to the virtual disks that are given to the vm this can be a lot simpler than managing like a raid controller with lvm running on top of it with like a file system on top of that where if you want to expand a disk for example you'd have to expand the radar right in the raid controller and then lvm and then the disk on top of it it's a lot simpler just use zfs zfs is a bit slower than the other qcad 2 file systems and lvm on a higher speed file system like ext4 and this is likely due to the extra overhead and it being a copy on write file system it is still pretty fast for vms and for most users it'll be perfectly fine and i think the extra features that zfs provides is well worth the performance hit btrfs storage was recently added to proxmox as a technical preview so there is likely still things that are going to be changed but it doesn't seem great from what i've seen with it btrfs has kind of been an instability level for quite a while now so it still has a lot of issues with many features especially raid 5 and 6 which would be great because it allows for raid with mismatched disk sizes and still using all the space it also allows for easy raid expansion by adding a single disk and you can also shrink it by removing disks btfs is also a copy and write file system and offers many of the other features that zfs offers except the stability one really it just isn't super stable even when i was doing a little bit of testing on btfs i ran into some weird issues that forced me to reboot the system and i generally just stay away from btrfs because i can't think of a good reason to use it for most use cases and while btrfs's raid flexibility is great it doesn't make up for the poor performance and poor stability that btrfest has this might change in the future but btfs has been in development for a very long time and some of the issues that i'd like fixed like raid 5 and 6 still have not been addressed i also tried some other methods of passing a disk to a vm and proxmox that aren't the traditional methods one of these is just a full disk pasteur so when you set up a disk in proxmox you say like the storage volume type and then colon and name but you can also just put a path in there in linux so for example i could do slash dev slash nvme zero to put my first nvme drive to the vm and this is a little bit faster than using lvm but not very much and i think the flexibility of putting multiple disks on one system and the ability to do backups and stuff which you can't do with a full pass-through disk makes a lot of sense and is definitely worth that slight performance head the other way to pass a single disk to a vm is pcie pass-through so instead of it showing a virtual storage disk to the vm it just gives it the nvme device this is the fastest way to pass a disk to a vm but also requires more setup as proxmox isn't actually controlling the storage at all in this use case you can run multiple vms on one disk doing this if you want to use like nvme name spaces but that's not a super common feature and i think for most use cases the compromises for pcie passthrough are not worth the slight performance advantages though passing through an nvme disk to a specific vm can be great if you need like a high speed cache or scratch disk or something for that vm to use i also created a quick test to test the performance impact of lots of snapshots on a vm i created a little script that would make a snapshot of the vm every five seconds and then had the vm continuously writing data and created roughly 400 snapshots this was done using the qcow 2 data storage method on a directory and noticed no performance hit in proxmox i left the cache mode to none for a couple of reasons first of all it's the default mode on virtual machine images and proxmox so it's probably what people most likely going to be using if they don't go in and play with it explicitly by themselves the next thing is proxmox themselves on their guide recommends it for most use cases so i'm guessing most people will be using it the next thing is dealing with the different layers of cache and trying to benchmark because if you run a lot of benchmarks you'll kind of notice that you're caching ram and another thing that i noticed is different file systems treat the caching a little bit differently especially zfs zfs doesn't really use the linux native caching and rather uses its arc system and even though proxmox tells it with the none type caching to not use the ram cache zfs still uses the ram cache so you get performance in crystal disk mic with zfs that is significantly better than the disk is rated to do or any other file system to do but that's just because you're reading ram and zfs is faking that while other file systems are doing what proxmox is telling it to do so zfs can be a little quirky in that way but all the other os's pretty much handle it like you'd expect and how proxmox says it is handling it let me know about the experiences you've had with storage on proxmox and other kvm based hypervisors and thanks for watching this video

Channel: ElectronicsWizardry

Views: 29,815

Keywords:

Id: eRqsbS3s554

Length: 8min 56sec (536 seconds)

Published: Sat Sep 10 2022