UNRAID ZFS Pools and Shares Performance Optimizing

if you've been using unraid then it's very likely you've got some questions about ZFS maybe you're even using it so today we're going to do a deep dive on some setup some configuration some benchmarking and we're going to see what kind of performance we can get out of ZFS arrays on unraid this is the latest 6.12.3 release that we'll be using and we're going to be benchmarking this over some SMB shares to see what kind of real world performance you might be able to get we're going to use both nvme storage and HDD based storage we've got our three Intel optane 900p these are 480 gigabyte each but in a RAID 0 array they should be wicked fast and we're also going to try them out and a raid C1 array which should give us some other interesting results I'm very excited for that one we've got our eight hard drives these are 22 terabyte Seagate white labels and these are available on shop.digitalspaceport.com for great prices and channel members get a three percent or a five percent discount and these are sat at base drives we're going to be putting these into Ardell r520 this is the r520 that we've used in a couple of other videos and this might be one of the last times that we're using this because we will be updating this too this Red River Pro system this is going to become a server for us and we'll probably be hosting quite a few things on This Server so that Hardware swap is coming up soon as we move over to pcie board generation systems and the threadripper pro is actually a really great workstation that is configured with a 5965 WX chipset in it however that's about to be a second class chip because the new threadripper Pros are on the horizon it's got 256 gigabytes of ddr4 2400 needs to see in it it does have two 40 gigabit nics in it right now although link one of those is in use here for the transfer over to the unraid machine and it's got three sx600 2.6 Fusion i o drives which again in a RAID 0 configuration offers some pretty amazing storage the amount of ram in here we have 192 gigabytes and that is at 1600 speed that is DDR3 and that is ECC to 2470 V2 Xeon processors we also have some high-speed networking we're going to be having one 40 gigabit Nick this is a melanox connect X3 Nick and this will definitely be able to push quite a bit of performance so I hope you're as excited for this testing as I am make sure that you hit like And subscribe and also check out some of the links in the description below to our rating guide on this let's talk about what we're going to cover in this video and I think these are some of the things that are interesting to me that I'm looking at but sound off in the comments below and let me know what kind of use cases you have in particular for unraid I think this is a fantastic platform myself I love it for all the fun things I think it's one of the cleanest Docker interfaces out there and it makes a great home media server for me so some of the things that we're going to look at is we'll be looking at the performance of raid Z1 and raid 0. however we will not be diving into all of the raid Z configuration options next I'm going to cover some of the things that I'm using for my 40 gigabit Nix to make sure that they're able to get the best speed that they can if you're using 10 gigabit or faster NYX you probably want to consider because that is potentially a performance bottleneck that you could hit on a unraised system over SMB or NFS and we're going to be looking at the impacts that fuse has on a user's experience when you're using SMB especially and some ways to mitigate that and get better speed if you've checked out some of the other unread videos that I've released recently you've seen some of those performance bottlenecks manifest we've got some new information around this storage performance and I think I'm pretty excited for it and I think you will be too at any time you can use the chapters below to zoom to any one of these topics so before we get into the configuration and setup of our ZFS pools in unraid let's take a look really quick at some of what we're going to be using locally for our benchmarking Story 3 in vmes and that is in our e Drive here these are actually some high performance Fusion i o devices that I've had around for quite some time and they do get really good performance and we'll run these with an 8 gigabyte profile we do have our settings set down here to nvme ssds for these and let's give this a test and as you can see these three ssds are working together in a RAID 0 here and they are capable of quite good read speed so the reads performance will not be a bottleneck I do not think when we're looking at our 40 gigabit Network and the ability to saturate it that these will be able to provide and as you can see they have some really great performance so those numbers certainly nothing that you would expect out of something that old but these have tremendous bandwidth on them when you put them in a raid zero array together so let's go ahead and set up our unrage here take really quick another peek at this that these are the E5 2470 V2 processors and we've got our 192 gigabytes of RAM here and we've got our interfaces here which if we go over here to general information you can see we've got a 40 gigabit Nic connection let's go ahead to the main here and set up our pool devices in here we've got one also for our array since you have to have at least one device we have a wd easy store this is just a USB drive and this really won't be part of the testing today but you do have to have one in there when you're doing this kind of setup so we're going to go ahead and create our first pool and we'll call this uh opte and we're going to create our second pool here and call this spinners and we'll add those eight discs to this as well so let's get these octanes added in here then after that we're going to select the Raid 0 profile changing that from Auto to ZFS with RAID 0 from the drop down and three devices we'll leave Auto Trim on but we're going to leave compression off hit apply and done next we're going to do the same with our hdds we're going to go ahead and set this raid ZFS profile to be mirrors and we can leave this as two groups of four devices or four groups of two devices we're gonna go with four groups of two devices and for compression we will leave that off and for Auto Trim we'll turn that off also and we'll hit apply and now when we go down here and hit start we're going to need to format these disks and by checking the box here that is of course a destructive process that we just enabled we're going to go ahead and format those so the next thing that we're going to do is look at some of the plugins that I have installed here and go over some of the settings that I have added in here which I think are important to cover just so you can see what I have done so you can if you're doing this a lot also follow along and have some sort of a baseline so the melanox firmware tools that we have you can see are installed here that's something that I recommend if you're using any connect X Card it'll give you a way to flash it also if you need to flash it I've already flashed mine more guides on those kind of things coming up soon as well and if we go back to the plugins page here there's another very important one that we have and this is the tips and tweaks one and in the tips and tweaks one you can see that we have disabled our Nic flow control because these are two melodox connect X3 cards that are connecting to a melanox 6036sx switch and flow control on these does not speed things up whatsoever it is a matter of fact slows things down hand Paces it as well on the window side of this I do have that turned off and now on the unraid side of this I have that turned off as well our ethernet net Nick offers for sending and receiving are set to 40.96 as they are on the Windows side as well and you can see we just have the one network card enabled here I have my VM dirty background cache sets 40 and my dirty ratio set to 41. this gives me a very decent amount of ram to use as far as caching and writing the rest of the files over I've set the inter Intel performance mode here to yes and I have the performance mode enabled on the governor scaling and On Demand so that it can peek up there really quick whenever you make those settings go ahead and hit apply that certainly is going to use more watts and I also have a performance mode turned on in the bias these are things just so we can max out and see what the upper Edge looks like you should definitely have a bias towards performance but what because the savings and electricity is not at all insignificant on either your desktop machine or on a server next let's take a look at some of the settings that I've got over here in the global shares setting I have permit exclusive access two shares this allows us to bypass the fuse and this can give us significant performance benefits when we're writing to our pool and also do make sure to set your disk shares to enabled and let's take a look really quick at a few things that I did change here that can help you if you have high performance networking do not apply these changes if you do not have high performance networking however because these could actually firm you and that would be counter to what you want to have happen if you have a one gigabit Network for instance having a up to 128 megabyte buffer would probably not be something that would be beneficial to you so we have our window tuning we have it also utilized the bbr instead of Reno uh congestion control pattern and I think this was one of the things that I saw gave me the most consistency so instead of massive spikes up and down like you sometimes see typically this immobilize the curve a lot and while there's not huge Peaks up there's not huge Peaks down so that's one thing you should take a look at and for that to also work with bbr as your congestion control you would also have to to set your Q disc to Fair queuing which is fq and because we have jumbo frames enabled I went ahead and set MTU probing to true these were the settings that I found that gave me the best performance let's check really quickly on the network side and look at these settings so I have a static IP address configured here as well I have my mtus set to 9000. if you are using a melanox connect X two or three and you have 10 gigabit or greater do consider that you should on the pathway if you have everything else set to jumbo frames which is the MTU 9000 do that as well on this system settings that we have for our SMB over here I want to show you a couple of things that I've enabled on mine that did speed things up a little bit while you don't need to do these things right now for this particular setting this is something that you might want to consider if you're running Windows 11 or 10 workstation and you do want to set SMB multi-channel support this is definitely some things you would want to have for your settings and if you look here I did have this configured for tunics however we only have one Nick plugged in right now so that we would be able to get that extra nvme in there for that really really critical speed the EA support equaling no and the store dos attributes equaling no when you see enable wsd here this is the more advanced SMB feature set and we have disabled netbios because of course that is a security concern when you do this and and when you have enabled the exclusive access mode you're going to see disk shares show up down here and if you click on your disk shares you'll be able to set this to yes and that is essentially the only thing you're going to have to do to create a share so we'll go ahead and click done on this and then go over to our Spinners and select yes from here as well and this will allow us to bypass the fuse system and let me know in the comments below if you like benchmarks because we can do more benchmarks oh boy can we do more benchmarks showing up inside our PC under our network configuration here yes we've got the guac server here and we've got our optanes and we also have our Spinners showing up as well so let's go ahead and run some benchmarks on those by using the select folder and we'll start with the obtains pretty excited to see what kind of performance we can get over this SMB share here and let me go over here to our network card so you can watch along with that as well and we're going to run the same test here and it looks like we're hitting right around 15 to 18 gigabits per second and our write speed is looking pretty darn nice also at 2.2 gigabytes there as well and so you can see that we have pretty good s b performance to these Intel optane and if we check this out in gigabytes per second you can see that those numbers look really good we check out our iops that we were able to achieve to these those are some pretty respectable numbers and checking out our latency now let's go ahead and select our next folder and that is going to be the Spinners and this one I'm very excited for because we have ink Spinners and that is quite a bit of uh possibly high performance let's take a look and that is actually really good performance that we're seeing there also at 2.3 again clocking in here and so there's some synthetics that we can look at but let's actually go ahead and transfer some files so we can see what the actual real world performance looks like here so over on our previously tested Speedy drives this is once again the SX 600 array of three Raid 0 2.6 terabyte in vmes we're going to go ahead and copy this and put one of these files on the spinner here and these are quite large files and the reason you want to do that is so you can get an idea whether or not there's some sort of a caching mechanism like Ram that's being able to be utilized you can see that impact up front here which that was a lot of RAM for most smaller files you're going to actually probably be just A-Okay with that and if you notice the difference between the write speed here being about half of what you see represented here I'm guessing that has to do with the fact that we're using mirrors for this configuration and next let's go ahead and look at that same right performance to our Intel nvme array and I'll copy a different file just to make sure that we're not getting any benefit from any caches and that is some rather good sustained performance that we're seeing for these nvme devices now going up until the about 2.2 ish gigabytes per second and so I noticed from some testing I did when I had just two devices in there that I was actually hitting the same speed so I think that has to do with me having a new span thing going on when I moved that extra nvme drive onto the other sockets location that probably impacted the ability for things to stay within the same pneumo cluster so I'm guessing that we saw some performance impacts there this probably just has to do with the fact that if you are looking at three additional slots that are 8X on one side and you also have a 16x slot on the other side on the r520 it may not be an optimal configuration for making sure that you have the maximum speed across those devices because they're going to be split on two different CPUs so let's change this configuration instead of having it be a ZFS RAID 0 of 3 devices nice this and we'll set this up as a rain Z with one group of three devices click apply on that and done here let's go ahead and erase the Spinners also and try a different configuration on that and we're going to go with the raid Z on this one as well with one group of eight devices and we'll leave the rest of these settings as they are and when we come back here we're going to need to a format and you can see that we have quite a bit more space available on those 22 terabyte drives eight of them was a depressingly small size when we were using the mirrors Arrangement before but we now have 148 terabytes of space let's check out that performance comparison because I think they will see something pretty interesting here and hopefully that can help you out if you're doing an evaluation in your own hard drive layout design process but for sure you should always Benchmark and so let's roll the synthetics again on this and we'll start off with those octanes again and you can see that for most purposes you're going to have really decent performance with some really nice nvme especially if you're looking at a raid Z1 Arrangement on your pools let's check out that next Spinners and see what it looks like on those this one I think is going to be interesting I don't know that but I just have a feeling that there will be some interesting results speeds are good I almost crusted it 20 gigabits and at 1.95 that is a pretty decent trade-off versus the mirrors Arrangement and the lack of space that we had with that setup just a little bit of difference on the performance on that re and we can see that our Z1 performance here still looks really good so for sure being able to reclaim a not insignificant amount of space would be something that anybody would probably want to take into a serious consideration versus mirrors if you're looking at your Arrangement here of course there are some security concerns but if you have properly tiered backups that should minimize those impacts and favor in definitely my case with going with more sides and now let's do again some real world moves so that we can actually see what it looks like when we move some files from one to the other that is a great speed for a file that is about 80 gigabytes in size and now let's take a look at our Spinners here and while rape performance is certainly one of the things you want to consider at all points in time let's take a look at what the read performance looks like on these when we read these two really large files back to our local nvme we'll start with the Spinners here and cut this and bring it back on and so I think we're seeing the real world results match up pretty close with what we would expect to see from the performance uh to the synthetics that we just did now let's go ahead and check the performance coming back off of the obtains again really great performance in my opinion so I think that we've seen that the Z1 option is not a bad option if your Libby will set up and maximizing your space as much as possible while giving you a little bit of redundancy if you're not yet using a array device for your ZFS that might be something you want to consider which might even provide you a pathway for some data migration which I know a lot of people are talking about how to move data from one to the other you can also do an In-Place upgrade a little bit risky in my opinion but uh if you got to back up somewhere then all those risks kind of Fall by the way say you have a backup right should always have a backup probably a couple of backups actually and so let's look at the performance on this file share if we have a folder full of definitely a lot of ducks geese and cats and so I'm going to move this over to our optane drive here where's that right speed is pretty decent here and let's see what the thumbnail performance is like with extra large icons so I would call that actually pretty darn good and these are rather large these ones are about 23 24 megabytes and so that's a pretty big picture of course these much smaller so this is only like six and a half so you saw those all kind of pop in at once there so this is actually not bad at all as far as the performance that you're seeing if you were going to have remote photo storage something like dark table might be able to access this really nicely now let's go ahead and do the same thing and pop it over to the Spinners The Spinners wonderful right performance there and let's check out what it looks like as far as the icons so generating them certainly slower a little bit it looks like than what we saw on the optanes but still pretty decent performance uh I think I'm going to copy over uh some videos and see what remote video editing would look like on these because I think that can help you make some decisions possibly about warm storage hot storage whether you would be able to utilize this for things like 4K video editing at the same time so real world testing definitely the optanes but those octanes are built for super low latency so really impressed whenever I see those come to light and man they shine when you do things like photo editing or video editing wrong so I've moved over here from our true Nas array our heat and home Labs video where we talked about things about heat mitigation this is a 4K video and it was done with Kaden the editing on it so we should be able to open that back up with Kaden and get some idea as far as what kind of performance we would be able to see we're definitely going to do this on the optis drive because I think that it just makes sense a probable workflow would be to use your fastest nvme based storage with the lowest latency for doing things like video editing or photo editing and then move that over to other pools or your array for longer term storage I think there's a really nice potential tiered option in unraid coming up could be really exciting now let's move this on over here and see what kind of write speed we get and I'm very pleased to see that kind of write speed there 1.5 to 1.6 gigabytes per second okay let's go ahead and open up this particular project and so this should load it up here remotely and we're going to run a stopwatch really quick so we can see how long it takes to actually load and go and Caden certainly not as fast as every other video editing product out there but it does actually a great job I've come to really enjoy using it let's do some things that you might typically do like scrolling around and playing and it is fast it is actually very fast not bad the performance on this actually I think is totally something that is viable for you to be able to use for editing if I look at some of the fine scrobbling like that's very responsive to the mouse movements let's see what that performance looks like if we look if we tried the same thing locally on this machine and while it was faster to load the interface it still looks like it's actually loading the clips end here so what the total time impact looks like will be something that we'll just have to measure out and of course local performance on this is as I would expect very Snappy also and let's try one more thing we'll actually load this up to the Spinners and just see what the performance looks like there I get out of the way you can actually see the r520 right there open at that data all right now let's give it a shot opening that up and certainly you can see those discs blinking them pretty fast now you know what I'm thinking that there was some sort of a bias and it'd be loading of it the first time because it seems to really be popping back really fast here so I'm going to redo the optane's test after this because the interface does look like it's loading up way quicker than what we saw for the load up uh with the optane so maybe that actually is caching some of the program stuff locally I think it's only fair if we actually evaluate that as well yeah and that load up time is quite good there and this is of course all 4K stuff so very very happy with the the performance even off of the spindles this is great let's take a look at those opt-ins one more time and I think one of the things that I can say already and this is I think pretty impressive and also just kind of fun I can't tell that much of a difference here I would be uh shocked if you could actually so I think this is actually fairly viable for any of these Arrangements even for 4K video editing now Granite that's a 40 gigabyte connection up 40 gigabit networking cards are actually pretty cheap and you don't have to have a networking switch that supports 40 gigabit you can do direct connection with 40 gigabit and get the same performance so that is even a little bit better if you remove the switch from it so in conclusion some of the takeaways here are that the performance that you can get by bypassing the fuse file system does seem quite impressive as well examining some use cases for these uh sites from basically the photos it was very responsive and most people probably would use some sort of a cache device for that instead of having it just directly coming off of their spindles so there is some impressive things that I've got to say I do like to see and definitely raid Z1 does look like an attractive option to me for pools especially if you're looking at really fast nbmes and also large arrays of storage because this one-two combo could provide you with some really great performance for really hot things that you're doing and editing like cash directory basically for all of your apps that would be running on unrate as well if you add photos that you were keeping and storing off of that then you could also move them over to the ZFS array before you move them over to your colder storage which could be your true Nas array so there is some really cool stuff here the concepting of tiered storage has come a long way as a result of some of the things that we've seen here I think there does need to be some uh effort that you would put in around your mover scripts to make sure that you have things uh accurately Landing where you want them to land but there is a lot of possibilities that I'm seeing here I think I'm going to go ahead and build out and scale out a pretty large system with this just to see what kind of performance we might be able to get and certainly thank you to all the channel members you guys make this all possible and I thank you all for hitting that like And subscribe button and showing up for these videos and definitely leaving comments is one of the things that I do read almost every single one of them try to respond to as many as I can and that gives me a great insight into what you find valuable and that helps me create content that you're interested in so make sure you let me know anything in the comments below that you would like to see especially if it's related to Henry true dance and we've got a ton of stuff coming up regarding proxbox and we will be having a really really really fat array petabyte definitely another petabyte project is in the works so I have a great rest of your day keep playing and keep learning and definitely these are some amazing systems when you look at your ability to do things with unrate and having been an unread user for quite some time I can say the advancements that we've seen in it have come a long way it is definitely one of my favorite fun enjoyable operating systems for me to go into play with and utilize and so thanks for joining in today and I hope you have a great I'll see your day

Channel: Digital Spaceport

Views: 13,776

Keywords: unraid, zfs, unraid setup, unraid zfs 6.12, unraid zfs cache pool, unraid pool, 40gbe nas, 40 gbit network, 40 gigabit ethernet, ioFusion Drives, 22TB hard drive, Raidz, smb share unraid, samba share unraid, unraid shares, unraid share setup, unraid performance tweaks, unraid performance vs truenas, unraid performance test, 10gbe, 40gbe, raid 0, ZFS raid 0, ZFS performance, zfs network share, nas, zfs nvme, zfs nas, zfs nvme performance, unraid tutorial, unraid setup 2023

Id: 36moTWdxjnI

Length: 33min 5sec (1985 seconds)

Published: Wed Aug 23 2023