I had VDEV Layouts all WRONG! ...and you probably do too!

I've got a confession to make I've been using Tunas core and scale for a while now but if you ask me exactly which VW layout would produce the best pool performance I honestly wouldn't know and the thing is I'm betting you couldn't tell me either why because it's hard that's why I've built different pools using different layouts and haven't seen much of a difference in performance which is left me with more questions and answers and I think it's time we fix that welcome home lovers and self-hosters Rich here and in this video we're going on a journey I'm going to build a bunch of different VW layouts test them using Crystal dismark over SMB see what the performance differences are between them and hopefully once and for all discover which speed Dev layouts are the best for your Nas let's get to it open ZFS has a variety of different vdf types that you can combine to create your storage pool for many of you watching this video you might not be totally familiar with the different types so let's run through them first let's talk about data V devs datavidevs are where you store your data within a data V Dev you can specify no less than five different layouts depending on the number of disks you assign to it you can create Stripes mirrors a raid Z1 which is akin to a raid 5 array a raid Z2 which is akin to a raid 6 array and a raids E3 which really doesn't have a traditional raid equivalent all of the raid Z numbers denote how many discs you can lose at a time and have your day to be safe a single disk for raid Z1 two discs for a raid Z2 and you guessed it three discs for raids E3 and if that wasn't complicated enough a pool can have multiple data V devs in it next let's talk about cash fee devs a cachevidev also known as an L2 Arc read cache is used to accelerate read operations conceptually a read cache holds onto frequently accessed data to help speed up access to bits stored on slower spinning disks you can only have one recash per pool now let's talk about log vdevs log vdevs also known as a write cache are meant to Cache synchronous rights coming into your Nas and commit them back to the slower disks over time the idea is pretty simple if you have big slow discs in your storage having a right cache would increase the speed at which you can write data to your storage system by caching those rights and then in the background the system can write the data back to slower storage and log vdevs have two different disk layout options there are more types of v-devs as well like metadata V devs and dedupe vdevs but we're going to focus on the big three we have in our possession the newest triness mini R from IX which we reviewed recently so check that out if you haven't seen it yet it's the perfect platform for me to test different Dev layouts to see which one produces the best storage performance let's talk about the hardware first as mentioned we'll be using the brand new trunass Mini R from IX systems this awesome Nas rocks 12 3.5 inch 6 gig hot swap drive base 64 gigs of RAM and has dual 10 gig ethernet our mini R is running true Nas scale and is the perfect test bench for this exercise Let's Talk About Storage our storage consists of 8 10 terabyte Western Digital red plus NAS Drives and four 1.9 terabyte IX branded ssds we'll be combining these drives in a variety of different vdev combinations to see how well each configuration performs and how much effective storage we'll get from each configuration there are so many possible combinations to test they decide to break it down into four data Vita layouts each with three different groups of tests one with just data V devs alone then a test with just a mirrored right cache and lastly a test with a mirrored right cache and read cache testing these different layouts should show us what the effects different layouts have on SMB performance you ready to look at some graphs let's get to the data be before I show you the graphs and talk about the details I want to talk about the sheer amount of testing I did prior to making the video this video would be super long and boring if I went into every single test result so I'm only going to summarize the best performing test here for the sake of your sanity and mine if you want to see the Raw results of my testing let me know in the comments and I can put the spreadsheets up on Google Sheets for you hardcore data people okay first let's compare the four pool layouts that only consist of data V devs to each other to see if there's a performance difference between data V Dev layouts as a reminder the first pool layout is four sets of mirrored 10 terabyte disks with the other pool layouts being variations of a single raid Z1 two and three that are comprised of a single eight disk V Dev right away we can see that for the 4X mirrored data V Dev layout there is a benefit to disk rights but surprisingly there is absolutely no difference in read performance for any data V Dev pool layout regardless of type and for the most part very little to no performance difference for write operations outside of the the Forex mirrored layout alright so we have our baseline from this point we should be able to see what impact having read and write caches does to our performance let's take a look in these tests we've added a 4X mirror write cache to each data V Dev layout why a 4X mirror you ask one because we have the discs and two that should provide the best possible performance for a ride cache these results are kind of shocking to me I expected to see a massive boost in write speeds for all data VW layouts because we added a right cache but in reality having a right cache seems to have hurt every layout's right performance to the tune of around 30 to 60 megabytes a second now let's move on to the read and write cache test results with the addition of a read cache and write cache to our data V devs we essentially see the same results as we saw with the additional write cache only all these numbers are basically within the same margin of error throughout all tests what is going on here those numbers are basically all the same regardless of this layout or whether they have read or write caches and that just doesn't make any sense to me I would expect that when you add caches that at least we'd see some performance improvements over pools without them or what would be the value of having them in the first place clearly I don't fully grasp what's going on here and what do you do when you don't know the answer to something you get a hold of an expert this is Chris paradon technical marketing engineer at IX systems Chris thank you so much for taking the time to answer my questions thanks for having me on the show Rich anytime so I'll get right to it I sent you the results ahead of time in the The Burning question I have to ask right off the top here is why are my results so similar so your results were similar because what you're seeing happen here is ZFS caching is kicking in both on your reads and your right tests so uh when you've got the reads going on you've got what how What's called the Adaptive read cache in ZFS so that's storing a mix of the recently and the frequently used data in Ram since your benchmark for a lot of those cases was under the 64 gigs of RAM that you have in your mini R most of that file was coming straight off of ram speeds so when you were doing those one Meg size retests it was basically benchmarking just how fast we can reach a ram over the network when we got to the smaller 4K block sizes we were looking at more of a network latency Benchmark there basically how fast you can send from your client back to the true Nas machine and it would give you an accurate reply so that covers your read side of the benchmarks on the right side ZFS is actually Gathering up pending rights into RAM so there's been a lot of discussion that goes on about how ZFS buffers up to five seconds before it writes it's not 100 accurate it will wait for up to five seconds to write unless something makes it right sooner so there's a limit to how much pending data can pile up before right happens in the way your mini R was set up using scale out of the box that number's a little under a gig so that's why you saw sort of that dip happen right around there if we re-ran those tests with a larger data set if you push it up to 128 to 56 gigs or even like a terabyte then we'd start to flesh out some of the differences in those underlying vdev layouts and the difference between mirrors and rates so then I guess my next national question would be so where do caching V devs actually make sense to deploy so when we're talking about a cache and v-dev if we talk about the read cache vdev that's useful when you have more hot data so data that you're reading more than once or twice then you have Ram in the system so in this case if we raised your test data set up to 256 gigabytes that's more than you have Ram in your mini R so you would start to see some benefits of having the read cache there good examples of that would be if you're using it for virtualization if you've got it as a corporate file server and you don't necessarily have all of the RAM available to Cache it but your your second level would be it's coming off of SSD instead of reaching down to your hard drives for somebody's home media server for example if they're just running Plex they've got a few of their CDs or DVDs stored on their true Nas machine they're not going to see a big benefit from L2 because you don't typically watch the same movie over and over unless like me maybe you've got some small children in the house and they just love watching the same shows or listening to the same music the right side of things which is called the the log device sometimes called the right cache it's also a little bit misunderstood so for SMB by default the right cache isn't going to be in that data path at all windows writes to SMB in an asynchronous fashion it's going to hand over that data and basically tell true now Hey whenever you've got a chance to put this onto disks go right ahead so ZFS will happily cue that up in Ram and spin it out to your back end drives as it gets a chance a log device is going to benefit you if you're doing an NFS share if you're doing virtualization something like that that says I need you to Commit This to stable storage and I will not proceed any further in my side until you tell me with 100 assurance it's there that's what the log device is for because when you do a write like that if it's a database if you're doing a financial transaction or if you're running a virtual machine where it says I need to be 100 positive this data landed on a stable disk that's where it's going to be written to RAM and it's going to be copied into that separate log device the slog that's that's what's going to hit you the hardest for random i o because random i o on the hard drive is slow we all know that um anybody who's put a solid state drive in their desktop computer has seen the massive difference between random i o on a hard drive in a solid state so for a slog device you're going to want to add a really fast SSD that's going to benefit you for your virtualization and your NFS use cases awesome okay then that makes a lot of sense so naturally then I guess the next question would be since it's clear that RAM and ZFS are very important and CFS prefers Ram or uses Ram first before using any sort of caching what's the sweet spot for Ram and a system then uh how many slots do you have and how big can you fill them there isn't generally a maximum amount that's the first thing when somebody says I want my system to go faster I will say can you fit more RAM in it and that's often the first question ZFS is going to use all the ram that you throw at it there's been a lot of minimum guidelines that have come up over the years there's been talk of eight gigs for minimum a one gig for one terabyte yeah mostly it's coming down to the performance you're after for your workload um if you're just using it for backup of the occasional files uh you're not reading too heavily for it or you're okay with the level of rigs that you get from the disk such as again in that sort of media streaming situation you could have you know 16 gigs of RAM for a couple terabytes of usable space even a few dozen and you wouldn't feel any real perceptible difference when you're doing latency sensitive workloads again if we're going back to VMS or NFS or transactional databases that's where you're going to be looking at the the other side of that where you're going to have hundreds of gigabytes of RAM you might just have a couple terabytes of usable space but you want to serve as much of those reads as possible out of that Ram cache so that's where you get into numbers like you'll see in our true now's Enterprise Hardware where we're putting half a terabyte or more of ram into some of these systems so let me ask you a couple scenarios here because it sounds like there's uh like in my case the way I was perceiving caching is really dependent on the type of workloads I'm feeding the uh my my Nas so um we'll start with kind of a block of three so first would be what sort of VW layout would you recommend for S and B solely just like straight SMB Network shares followed by say um just solely NFS Network shares and then you know followed by let's say a mixture of both SMB and NFS so for this one the protocol isn't as important uh for NFS we go back to the question about the the log devices we did before but in general what's going to determine your vdev layout here will be more the number of clients that are going to access it at the same time and the size of the files you're going to be accessing if you have a small number of users maybe five to ten and if you're accessing large files uh sequentially for Content creation like if you're pulling in uh large Banks of photos or doing video editing um locally and not doing it directly off the nas uh Ray Z2 is going to be a fit there because you're going to get good sequential read and write performance you have that double disc failure tolerance and you got a good level of space efficiency when you go for large files like that but if you're looking at having more than a dozen users if you're dealing with lots of really small files let's say you have a GitHub repo for a small development Studio Game Dev they're dealing with lots of files that are only a couple kilobytes that's where you're going to be looking at mirrors because you're going to get the random access benefits each of those mirrors can be serving a different read or a write request at a time so when you're serving those tiny little files that are a couple of k each you could have all of those dozen users hitting a different file and each of your mirrors are going to be able to serve a portion of that so then if you were to what would you recommend for a high random read write deployment then I would definitely say mirrors in this case you might be able to get away with a really narrow raid but generally speaking for most users if you can sort of I say stomach the 50 space efficiency overhead of mirrors then mirrors are definitely the winner here because you get those if you're looking at say a 12 drive system like our mini R if you arrange it in six two Drive mirrors then you're gonna have six sort of concurrent read write requests being able to be queued up versus if you did it as raid Z2 with two six driver eight Z's you're only looking at two concurrent read write requests okay and then so what would you say for you know iSCSI sand style virtualization back-end kind of approach yeah sand style virtualization is going to fall under a lot of those same things with random i o um if you only had one particular Target using it if you're doing uh iSCSI as a veeam or some other backup Target then maybe you get away with raid Z2 but in general as soon as you put more than one or two VMS on there they start hitting What's called the i o blender basically concurrent sequential access concurrent sequential access turns into random i o so you're going to want to use mirrors for your vdevs for the most part and you're definitely going to want a dedicated high performance log device if you're doing virtualization whether it's over iSCSI or NFS because you want to flip on synchronous rights and say I'm going to guarantee that every single bit that comes from one of my guest machines is going to be committed and stored safely so that's where you need your high performance log device and you want probably pool disks and mirrors that does it for my questions I think the next thing is do you have any anything you'd like to add to any of these uh any of this nothing really outstanding just that we we kind of talked specifically about your performance here uh specifically on the mini R and a lot of like I said the performance is going to be limited by the amount of memory in the machine so somebody who has 16 gigabytes of RAM they might hit a uh sort of the the fact where they're reaching to disks sooner than you would in your mini yard with 64 gigs uh conversely somebody who's built their own system running on some old data center here with half a terabyte of ram they won't hit that wall until much further down the road but much of this applies the same to anybody who's running scale or core on compatible Hardware um if anybody wants to try it out again it's always free you can go to truenows.com grab your copy of scale or core for yourself and keep an eye out for an upcoming evaluation guide it's going to help you get some hands-on experience in not just scale and core themselves but a lot of the deeper features and functionality that's awesome Chris thank you again so much I appreciate you helping set me straight because obviously I had a lot of uh just assumptions and preconceived notions that were clearly not correct when it came to caching v-depth so I appreciate that and I think that uh these little nuggets of of knowledge you've provided are definitely going to be helpful for others of our viewers who are kind of like trying to figure out what's the right VW layout for them because it's it's complicated right and if you look at it from a top down it's uh it can be a little confusing so I appreciate that very much no problem Rich I'm always happy to talk about ZFS and true as and thanks for having me on the show again so there you have it my blanket assumption that just adding read and write caches would increase the performance of my transfers was wrong and that their value really depends on the workloads you're using your Nas for it's also interesting to learn that for basic SMB Network file sharing for a home user adding read and write caches don't really add any value and instead of spending money on ssds for caching you should spend that money on more RAM for your system regarding the best pool layout generally speaking for most people it's going to be just a simple raisy 2 layout unless you have workloads that are high in random read rights or you have a really high user account and that friends will do it for this video if you found it useful give it a like or a sub or if you have a beef with something you said or did get down those comments and let us know and now she finished watching this video how about checking out some of the other great home Lab videos we've done in the past [Music] foreign [Music]

Channel: 2GuysTek

Views: 53,396

Keywords: best VDEV layout for your NAS, TrueNAS VDEV layout, best VDEV layout, Do caching VDEVs help, What is the best VDEV layout?, TrueNAS CORE, TrueNAS SCALE, Interview with iXsystems, nas, zfs, OpenZFS, Performance testing of OpenZFS, Performance testing of TrueNAS SCALE, TrueNAS Mini-R Performance testing, TrueNAS Mini-R, which VDEV layout is the best?, What are data VDEVs?, What is an L2ARC VDEV?, What is a SLOG VDEV?, 2GuysTek, zfs pool, zfs file system

Id: _aACgNm8UCw

Length: 17min 42sec (1062 seconds)

Published: Thu Mar 30 2023