FreeNAS Spare Parts Build: Testing ZFS With Imbalanced VDEVs and Mismatched Drives

Tommy are from water systems and this is my spare parts FreeNAS build that we kind of put together to kind of demonstrate setting up FreeNAS with spare parts and less than ideal but still we'll be able to get some performance out of it and just kind of show the setup process if you own learn more about me and my company head over to lawrence systems comm if you like to hire short project there's a hires button up at the top if you want to support this channel in other ways from affiliate links down below to get your deals and discounts on products and services we talked about on this channel and i've been talking a lot about FreeNAS and this is just like I said spare parts I will start with what these drives are in front of me by showing you the login to the FreeNAS here we've got nothing built but let's talk about some of the parts so I have a couple of drives here like this one and that's in this pile here of three and a half inch hard drives is this Toshiba desktop 7200 rpm drive that's 500 gigs so nothing real in performance here let's see what do we have at 88 0 that's a Seagate constellation that's some of the two and a half inch drives let's look at 3 another see a consolation and I hodgepodge these together what's at 9 oh we got a Seagate Barracuda it's mostly Seagate's but there's at least one Toshiba on the mix here let's try this one 6 and another siegy 1 terabyte constellation in here so kind of got a mix of some of the different parts in here as far as hard drives so we got some three and a half inch hard drives here and all the two and a half inch hard drives are over here the Seagate constellations are on the bottom this fan is because these little Seagate constellation drives which have plenty of hours on them from old server that was decommissioned but they won't cool themselves very well they kind of expect to have the tray cooling them so when we printed a little 3d holder for all the drives it turns out not to work very well in terms of cooling so we got a fan sitting here but what I'm gonna demonstrate here is we have 5 one gig drives and 5 500 drives and yes you can put mismatched drives together for free dance and I've done a video before showing how yes you can expand your FreeNAS volume with another video of different size drives and we're going to do this from the beginning but yes this is something you do it I have another video about that now this is not the same as expanding a single video but B dubs are what make up a pool inside of FreeNAS I'll link to a bigger explainer I have on all the details and diving into that but let's start with this here we're gonna go and we have 8 gigs of ram we do have we move it slightly out of the way if you can really see we get the overhead here this is a Mellanox 10 gig single deck connector so this will have 10 gig speed and that is uh you know adequate for what we're doing here 8 gigs ram is adequate for we're doing here and let's take a look at the system itself this is a for the dashboard we've already got FreeNAS loaded this is a Intel Core i7 4770k so it's an i7 which is nice as liquid-cooled as you can see by the little rigging we have over here but it's nothing it's an older board it's a probably 5 or 6 years old so it is a Republican gamer board but it's not ideal so so to speak for freenas this is all the parts being laying around now let's run through just what the freenas installer looks like that's what I have set up here's to refresh this and I'm not gonna go through the install process completely because it's kind of boring but at least I'll show you for those wondering what it looks like now FreeNAS does have the ability when you run the installer which is download it put it to a boot drive and then you want to install somewhere we're not installing FreeNAS on any of the data drives now this is once again not ideal but based on the spare parts build we put this on a little sandesh cruiser USB you can install free nasty USB if you're in a pinch it will work it doesn't have to be much in terms of the boot drive because not a lots going on and if you have a pair of USBs you could even choose reading them together then we hit okay would you like to proceed with the installation installing flash media is preferred and sawing on a hard drive it does have that option and I see in a pinch you can do the FreeNAS onto a USB freeness does you know completely support that but a USB one sometimes we found those not to be a hundred percent reliable they work overall pretty good we've got some installed years that have never had a problem and the reason for that is once you get this set up you take your syslog and you don't point it at the USB you point your syslog or any type of logging over here because once it reads from USB it doesn't right back to it unless you're making changes which aren't very often all the data sets handle all the rest of it I think you set your freenas password who'd be a bios or boot via UEFI kind of depends on the motherboard you're using but in this case base you set it up to boot via BIOS it's going to run and do the setup it's going to copy the files and reboot so not much to this to getting free NASA set up at least I just want to show it in this little VM here so we can get you an idea and I'm not actually gonna go ahead and force shutdown this I don't need this running and we're gonna cover a couple quick topics right here I'll leave a link to this is a good read up on here so ZF s allocates writes according to free space / video of not latency for v2 why am i bringing it up we're gonna be creating a video with the 500 gig drives and we're going to create creating AV dev with the one terabyte drives and then we're gonna merge both of these into one single pool well actually you create the pool and you just set the V dubs we'll get into the actual creation process just in a second so this write up is to break down the fact that some people have commented this I think these comments are still on my video from a while ago of how to expand a pool with adding B dubs that are different sizes and there's this myth that it allocates rights based on latency and that's not true and what happens when you have in bailant speed halves because we're gonna be creating some of these on the 500 gig drives and some of these on the 1 terabyte drives well you got double the space that means for every file written it's gonna put let's say call them blocks per say a couple blocks over here in one block over here a couple blocks over here and one block over here and this is how it's going to balance it so it doesn't wait and right here fill this up and then flow over to here or fill this one up and flow over to here it distributes them evenly based on their size so it kind of creates a calculation going well this is only 500 gigs so it has half the capacity of the one terabyte ones and will balance them by putting for every two writes that does over on the one terabyte series it puts half of that over on here so this article actually breaks it down this person has a lot of good demos and testing and of course someone always says what if I already had data and then I expanded it there's a whole how the drive system rebalances it's a fun read and something definitely worth looking into and I'll leave those links down below and in a forum post where all has to be done so now that we've kind of got the concept of what we're doing let's actually do it and show it working so we're gonna go over here to pools and we're gonna add now this only had 10 SATA ports so that only gives me the potential for 10 drives and people always ask what about a caching drive what about a Zil a log drive on there well those are more for performance systems with specific you cases let's say you're having NFS writes and you really want a Zil drive on there to help buffer all that that can help but you have to remember when you have 10 drives in a pool you actually have a lot of this buffering going on where it's going to distribute the rates across here and of course distribute the reads what I mean by that is you're gonna get a pretty reasonable lot of performance even though we don't have a cache drive so I'm more likely in this particular scenario with these are all 7200 rpm drives to get a better performance out of these than if I would have sacrificed one of these drives sacrifice some of the storage for that Zil log on there but that's I don't want to get too deep in technical on there also if you are using something like a Zil you really should be using it as an MBI me because if it's not performance and it doesn't have a really fast ability to read and write the data it's not going to help these either same thing goes for a cache drive cache drives will do caching for frequently assets files there are you case use cases specific for that but once again if you have the ability to put more RAM in there that is faster because your first cache hits are going to come from RAM and then it will start looking at the Year level 2 cache which could be a hard drive but if you can put more RAM and obviously that would be much better because the more memory you can have the more I can cache those files also the other myth that you have to have on a massive amount of memory to get any real performance out of free now it will perform better with memory because yes you'll be able to cache things in memory but it's not imperative that you have memory you can still create this drive setup we're doing here without that much RAM so we do have 8 gigs of ram in here and as purposely I actually am or am I could have put in but we're gonna show the performance and setup of our spare parts build with only 8 gigs of ram now if you try to do things like run VMs and things like that in here which we're not for this demo obviously that would have definitely affect your amount of RAM you need so let's go back to creating a pool and let's see we'll suggest the layout there we go it grabbed all these and said let's put all these into the first data V dev and we're just gonna do them at a rate z1 which means we can lose one drive raid-z 2 means you glued up to 2 drives for parity so it kind of depends on what's your risk factor on doing it raise e to less risk at the sacrifice to lose a little more space but raid-z - you know I I feel as though I can just rebuild a drive in here and it shouldn't be too big of a deal so what will read z1 which means if I lose 2 drives it's done also if I lose 2 drives out of here and 2 drives out of here it's completely done it doesn't matter which side you lose them on it needs both beat halves to be there so if I lose 2 drives on this video it's not like the date over here safe the data is written across all the devs so that that is definitely a factor when you're doing it so now if I try to add these to this video that's not going to work that's going to go nope it will say these are mismatched drives so we're gonna go ahead and click down here and say add data what that does is now I can select these drives and that's how we build the second beat if so this is the first feed up here and the second beat of here so we have an estimated capacity 3.6 3 terabytes and 1.3 6 terabytes down here so I did see this one like I said it's gonna be a lot bigger so once we've done that we got to give it a name we'll call it Orion I do like doing encryption on these that's because if ever you have a drive you have to send back you know because the drive quits reading obviously there's some worry that maybe some data might be on there if you have it encrypted that worry just goes away the downside of if you set this up with an option of password and you have lost set encryption and password or you don't back up the encryption key that's gonna get written to the drive over here the boot drive of the freidy's itself then you also have lost all your data so please it reminds you to all to back that up but it's just something really important well switch this one down here to see your raid Z so now they match any because you can set it right here but they do have each beat up has to match and we'll hit create create pool this is where it's gonna warn you download the encryption key please back this up this is the key file you need in what this means is if the boot drive the little SanDisk USB that I have plugged in the back of this motherboard it were to ever fail then you definitely need to have that key with your restore so that's just what that's doing right there is I think that they put more emphasis on this with older versions of FreeNAS I just have to say it now any time you build an encrypted pool it definitely prompts you for it so now we have this all created but how fast is it how well does it work well let's go ahead and cover that real quick let's run it over here and let's create a share on it now as far as this set up even though I didn't walk the install do anything different from this and saw from like the absolute out of the box so over here to system general and I check the little box right here because actually by default it doesn't have HTTPS turned on and if we go to the services I do turn on SSH so SSH is turned on right now those are the only out-of-the-box configs from default that this is configured on so let's go back over here and create a share so sharing actually first getting ahead of myself here create a dataset I could just share the raw data set itself we actually want to create one demo share there we go comments we don't really need any let's call it demo save now we've got the demo share created then write it over here to sharing window share Add there's the demo share allow guest access advanced and we're gonna say only allow guests access now what that's doing and I've got another video where I break down ACLs and permissions you are going to when you say only guests you're allowing guests they don't need a password I can just log into the system to open share and we're actually gonna change here to open and like I said I have another video on how to get more fine-grained NaCl permissions and groups but for the purposes of this I just wanted to have a wide open share and then you set the ACL to open we're gonna go down here and hit save and now we have this demo share and now from this point we're gonna jump over to my desktop and start copying files to it to kind of see what kind of performance we get on it but for the most part FreeNAS as long as you have a reasonably modern system it runs pretty well on here it will boot off of most any system the only thing before we leave this little set up here and go to my office I want to comment on is I have the monitor up here because because it created all of these the BIOS tries to outsmart me each time and it sees partition changes on these so the next time I reboot it's going to mix up the order of the drives I've actually noticed this on a couple of the motherboards once you've written it it's fine but then you have to go change the boot order again to be the boot order you want the good news is there's no danger after you build these drives it'll tell you hey this is a free nasty to drive you cannot boot off it that's all you'll end up with on the screen and I've seen people kind of panic on that a simple workaround like I said you go back in the BIOS and we've observed this on several consumer motherboards where it just kind of goes hey you wrote partitions we think that means you must have loaded an operating system and want to change the boot order again you could probably get around that with turning UEFI on and installing it in a UEFI or maybe if we weren't using USB it would do it but I've seen them do it even when I was selling it to a hard drive it sees the partition changes so I'll just leave that as a note but now we'll go in my office and let's see how fast this is when we dump some data to it all right so I'm sitting back on my desk in let's take a look at the shares here so mount to ryan demo shares the path demo shares the name pretty simple we're gonna go here you go to impress control if you're using Linux Windows you know backside smash the IP address and when Linux its SMB colon slash slash and we're using an IP address not the full like name of the system but that is a possibility as well 192 168 3.2 a nine slash demo share and we have this folder now Before we jump all the way over there I mentioned that this was connected to ten gigs so let's SSH into that server we're just gonna do a quick test and this will help you observe and stash s this will help you a lot because the first thing you want to know is can your system talk to the pre nan system in this case at full-tang gigs because if it can't well then your files troubleshooting woes don't begin there I've had people say oh I'm not getting the file transfer speeds they want let's first look if you can connect to that system at full speed then you can start whittling down where the problem is so this system fully connected at nine point four okay ten gigs so that's what we want we are seeing definitely good connection there I'm going to exit out and do this and while we're waiting we'll type zpool IO stat Rin - be one that should work all right this will show the rights going on to the drive when we copy some files over go here and I grabbed a couple folders and we'll just happy um some backup holders in my system and paste them over let's see what kind of performance we get out of there now it's going to start out sometimes like a really high number and as you know when file transfers it takes a second to either ramp up or slow down to the speed that it actually transfers at not bad at about 300 mega second right now so that's actually pretty reasonable 299 megabytes a second going back up to 90 to 97 well let this run for a second and look at what's going on in the background over here but you can see it committing all the rights to this and slowly as it's going to watch the allocation here there we go we with it - be over there so you can see it actually reading and writing to each of the pools and as I'd said it's gonna start distributing this data across these pools and across the writes something to note here if you have slower drives in one of the beat up so one beat up has faster drives than the other it doesn't mean you're going to aggregate get a better performance overall you'll actually slow it down it does have to wait for these some of these rights to commit to the other drive so if you do have a couple slow drives in your pool especially because we use such a mismatch so if not even the same brand drives it can't operate really more than the slowest drive in your pool I mean there's still some factors it may still be a little bit faster but it may pause when it gets that particular Drive so something to consider when you are adding these when it comes to performance but generally when you're building a bunch of hodgepodge mismatched stuff together it's not to build a performance system is building on what you have or in many cases sometimes just a backup system and we've built backup pre NASA systems for clients or you know you take an older server like hey this server has some of these drives and some of these drives we just need a big storage repository for all of your backup stuff and that can go over here and it's usually like the you know the last generations the servers they can be repurposed for that and then your new generation servers run all of your primary but it's nice to have any one extra on-site backup with hardware that it will still perform pretty well and despite this being a little bit older of a system let's see what kind of transfers it's giving now that it's run for a while we're up to like two three hundred and seventeen Meg's a second and I grabbed this because there's a big wide variety of files mixed in here so fifteen thousand files in here some are small some are big if you did something and we could probably cancel this real quick and started doing it as this isn't X in front of it if we just did ISOs which is going to be how big is this about 25 gigs which the ISOs and we drop them over here they'll probably transfer faster because it's just large files being transferred out a bunch of small files where it has to be as much activity on the SMB so let's see what this transfers am now still only about 300 it looks like some of the other ones we're hitting the 320 mark so it gives you a good idea though that Oh 300 megabytes a second out of a handful of old hard drives in here it is still pretty reasonable now if we want to do the z2 I figured we'll do one more set up and do it z2 it just run run through the same set up and see how much z2 hurts us in performance we know it's going to lose a little bit space but then we do gain that extra redundancy so if we're getting 300 mega second on a z1 with these pairs let's just tear them down and rebuild them in a z2 format so go ahead and cancel this control-c we'll just do that and let's go ahead and kill these drives off real quick so the pools disconnect destroy the data by the way if you do not destroy the data you will have probably leather they have to re-import them or blink the drives because what happens is if you don't destroy the data on the pool it's expecting you to bring them back in and France will give some warnings that you didn't erase these drives and it's to protect you from accidentally overwriting them that it it's just one of those things that you want to destroy them if you plan on exporting them and destroying them out to be reused in a pool again like we're going to do but if you want to just to not destroy them so you just export the pool you could do that but we actually want to fully destroy these because we're gonna really pull them back in also SMB demo share it does recognize this as a share in there and it'll say hey you want to get rid of this too and restart service no problem reconfiguring data set to none and then we'll go ahead and restart the process yeah all the data on the pool was destroyed so we can create a new pool do the same setup again call this one Orion z2 consistent naming here at another data so Z 2 0 z 2 there are right and keep the encryption understand confirm great great pool download encryption key done cool a dataset demo z2 see to share cool save Zomba you you demo share bands same thing we're just going to allow guest access only allow guests save figure yells save and confirm that I have rewrite access over here well I definitely can rewrite data to it and let's see what kind of transfers we get now we'll do the same thing we'll grab both of these and give it a few minutes to ramp up and write some data let's see if the z2 has much of a performance hit you so I let this run for a while and even with z2 we're still seeing over three hundred megabytes a second and transfer that being said what does the cache look like I went as a performance assistant look like well I predict pretty safely here that we've used all of the memory towards cache so free memory point two gigs not much ZFS cache allocated six point eight gigs services running on the system only account for one gig like I said free dance does not need a massive amount of remedy memory to get high performance out of a system in terms of not super high performance obviously but a reasonably fast system right here being able to transfer is down with the 10 gig cart so over that breaks the myth of the memory and even having mismatched drives in here that are a random assortment is not even the same brands all of them there's a handful that are the same but not all of them are the same mismatch sizes of course to go with that and yeah we're still able to transfer at a pretty reasonable rate on this particular system so nice thing we'll do is go over to the reporting we can see how much CPU is being used which there's definitely some it's you know hitting probably some limitations with SMB going ok I'm gonna pin out this particular process we can go over the reporting and also look it's like the ZFS we can see the arc size hit ratio so it is you know pulling some data and definitely lining it up when I'm pulling it back from there some things are getting loaded in and ran to make a little faster and then we can look at the disks as well so we look at the disks and let's actually select on all the devices and metric wise IO latency and went out let's just select all to get warmer I'll scroll down here io activity which you're seeing it very evenly spread across all the drives so now we get to disk IO itself which by the way this is the da zero is the little USB drive that's the boot pool these are the all the data drives on there you can see there's very little at all the kits written there all the activity happens on all of those stripes here so you can see the disk busy once again kind of lines up much the same scroll down here and now we have the latency to look at and you can see the different links you know this is gonna change a little bit from the dryest because some of the latency is gonna be vary based on the ability of these tribes to handle something so while they're waiting for another drive or not so they're probably a little bit of variation in this it's kind of interesting to see you know all the data going across so that's it for this FreeNAS build at least said you can get an idea now that yes it'll work and hopefully this broke a couple of them s yes you can have mismatched D devs and yes you can have only a cakes Ram but still have a system I don't it's probably done transferring yeah it's done transferring but you know was able to transfer it and a pretty reasonable speed there at between three and four hundred Meg's can I bounce around depending on the type of file transfers on there now the highest-performing team is still not bad for a bunch of spare parts thanks and thank you for making it to the end of the video if you like this video please give it a thumbs up if you like to see more content from the channel hit the subscribe button and hit the bell icon if you like youtube to notify you when new videos come out if you'd like to hire us head over to Lauren systems comm fill out our contact page and let us know what we can help you with and what projects you like us to work together on if you want to carry on the discussion how to row to forum style Lauren systems comm where we can carry on the discussion about this video other videos or other tech topics and general even suggestions for new videos that are accepted right there on our forums which are free also if you like to help the channel on other ways head over to our affiliate page we have a lot of great tech offers for you and once again thanks for watching and see you next time

Channel: Lawrence Systems

Views: 44,208

Keywords: lawrencesystems, Imbalanced, Imbalanced zfs, zfs, zfs (software), zpool, storage, freenas, network attached storage, raid, nas, Freenas Build, freenas build 2020, freenas build your own, freenas build guide, home server, diy nas, build, tutorial, freenas (software), nas build, nas server

Id: EFrlG3CUKFQ

Length: 27min 32sec (1652 seconds)

Published: Fri Apr 24 2020