36 TB FreeNAS Build: Hardware & Setup. Kingwin Tray-Less Hot-Swap & Western Digital Red NAS Drives

so we're getting right here to do a FreeNAS build I have here six westrom digital read master eyes the six terabyte variety six of those I have 32 gigs of ram gigabyte motherboard an AMD FX processor and an e EVGA power supply now all this is going into a custom freenas build for a client that has some large storage needs and we're going to kind of walk you through the process now some of you are already going to start the debate and you can find plenty of articles on mystical but Tom I think that's not ECC memory you are correct it is not ECC memory I have been using non-ecc for a long time there is plenty of information including from the guys from the FreeNAS project and from other people in the industry that you do not have to use ECC memory also this is going to be running FreeNAS which is based on ZFS freebsd so we'll be covering some of that how we can figure out how we set it up and how we integrate it into their network and so I'm going to give you the whole build process here and then some of the software setup rosters kind of walk you through building a nice large raid array to store your stuff because we're producing data just an amazing amount now and if you don't have it stored somewhere redundantly you know even if it's just on a local server you do not have a proper backup and the problem is just storing in the cloud can be rather difficult so even for this client this server once it's built we're taking their older rate server which is going to be a ready gear and it ready here the net gear nass box and we're going to be upgrading that because we're going to have the files copying back and forth between them so they have two copies of all the data on site and then their third backup is going to be off-site now the problem with that is it's downtime so in worst case scenario if one of these servers goes down we're going to be able to pull the backup ideally this is going to be their primary server we will have a local copy to backup in the event two folks go down we have a call backup the downside of that is pulling that much data takes a little while so it'll either be mailing back hard drive but everything's just creating layers of redundancy kind of what this project is about in terms of the client in terms of for you guys this is about building and everything for a freenas box so if you want to build one at home you want to build one for your company this will get you started and kind of gets you the idea of how to work with freeness so let's get started so the box has been built it's assembled it's tested and it works really well so we went with a raid z2 with all these six terabyte hard drives we wouldn't be as I showed the beginning the Western Digital Red Nass drives they're really low wattage and letting this thing run for 24 hours and dumping over a terabyte of half a data deleting it and putting it back on here a couple times to really make sure there wasn't any problems it didn't have any issues this worked really well the drives really never got above about thirty-four thirty-five at their hottest under full load now that's important out these docks right here and so I got this here from kinguin this dock system is really cool it's compatible with SAS and Sita and these are just a pretty slick system so it almost raised you a Nintendo cartridge when you pull these in and out and I'll show you that in a second here we went a gigabyte motherboard and this is one of their ultra durable motherboards now this system is really for storage for our client it's about having readily available archival videos that they have they do civil engineering work so they have lots of things they need access lots of little files pictures when they had things taken apart like sewer systems or other picture large scale city projects so all the documentation if there's an emergency they need be able to get to it pretty easily and they wanted to have something like you know online all the time they can't archive and try to pull it off somewhere else and that's where this system comes in so all six drives you know as I said 36 terabytes worth of drives when you put that in a raid Z 2 configuration you come down to about 22 terabytes of usable drive space on FreeNAS and the reason it was Z 2 is you have an issue with raid where okay I have at least one redundant drive in seems to typically like you to try to fail except this is the part share to think about if one drive fails you have to rebuild it off of the data on your drives the stress of the rebuild could cause other trying to fail and then the whole raid array collapses at that point raid-z - it requires two drives to fail before the raid array collapses so while one's being rebuilt if another one were to fail there's enough data put there now you can go even further with raid z3 and reduces the amount of usable but we didn't need to go all the way that raid z2 to drive failures and we're good you know we can we can survive that on here now the processor below is the AMD FX e 320 that's the low wattage model because it just doesn't need a lot of horsepower now we did go with 32 gigs of ram and the reason for that is with the ram you want enough ram roughly you have a gigabyte of ram for each terabyte of hard drive space and that it's a rough rule of thumb you can get away with it being a little bit lower but what it creates is it loads all the tables in France the entirety of the drive and has it more accessible so it's not just like caching but it's more of hey here's an index of where all the things are and you start talking about 50 60 thousand files on the drive you want good fast access to that and FreeNAS can facilitate that so it will suck up as much RAM as needed when you first turn it on just really nothing going on you'll see the RAM use as low soon as you start copying files to it or from it it starts rising right up and it'll just eat up all the memory available to it the FreeNAS uses ZFS z FS is a co w copy-on-write file system or calles they call it that is really really fault tolerant and the ways the FS design it's just an amazing system you can buy a matter of fact I'm going to plug Michael Lucas I've met the guy offend his talks he wrote the book matter of fact two books on how ZFS works it great in depth you can learn about the resilient nature of the ZFS filesystem and I've even talked to engineers that worked at a pond tuner like they like it some other back-end stuff does run on it because you can literally just shut it off like they don't have to worry about it ranking they've actually pulled them out and just seven yell boom and it's amazing it recovers perfectly finding the we've had power failures at other places in our own ZFS and it's I've never once had it have a problem and we've replaced drives and rebuilt the rate arrays in them it's just can't say enough good things about it and that's really what matters is having really good reliability on this now it's also with free mass if you wanted to build something high-performance oh maybe sometime I'll get around the building one like this you can set SSDs in front of these because these are only 5400 rpm drives you can put and you highly recommend putting these in a raid rate you can put basically caching drives in front of the other drives so that gives it a high speed spot to write you can actually build a separate array of them and those ones you would have that would write back there now the risk of course would be if in the power failure and your cache drives haven't flushed out that's also why you would want them to be rate but those are real specific high-end configurations for this one like I said we just went with those in the storage now the other thing we did and I'm going to show you the back here is we went with dual USB so these USB thumb drives when you install them when you're running through the install you select both of the thumb drives for the install and then automatically goes okay you must want these in a mirror and it mirrors them for you so I can remove either one of them and it will still boot now if one of them fails same thing it still boots up now I always recommend backup and download the configuration file which France is really easy to set backup so if we ever had a problem we could just replace the board it doesn't care it's not particular about hardware like Windows is you can take the drives out as long as you have all the drives or at least enough of them you can rebuild the pool in another FreeNAS machine and it'll bring them all back in but also not having to worry about the thumb drives if one goes bad alert comes up it says hey I can't find one of them we've tested this it blooms with either one individually but then it tells you that there's a resyncing issue but then you plug them both back in and a tree syncs them and away you go so it works really well for having that for redundancy and there's so many expensive Amazings are ten bucks apiece so for ten extra dollars you can have redundant rated you know mirrored raid one thumb drives in there so let's get into the dock and kind of show you how that works too now they have this tiny little key and I know it's really small and it allows you to lock the drives so you can keep people from pulling it out but it's just mostly easy to keep you from accidentally ejecting the drive it doesn't really provide a lot of security in there says twist you know turn this way but you can probably put a pair of pliers and just you know put it in there's nothing real special about it so that comes there now the dryest themselves slide out and it's really slight now you have when you set them here you feel like I should be able to push it in but you don't have to you slowly close the dock and they stamp in now I also feel like if there's a problem with Drive you can blow on it like a Nintendo cartridge because that's what it feels like to be put Nick open it out it's like a look Nintendo cartridge but this dock is nice I mean this is completely tool-less really it's tooless I can take the drives in and out I can just slide over I've out here another one and yeah you open a box anyway you go now the way we lever the drives to keep it simple there's six SATA ports and they referred to as ata zero through 88 five so we just started 2-0 at the top put the five at the bottom and we know which one so free Nancy identifies a drive is bad we know exactly which one it is and we can find it in the dock now these docks are hot swappable as well and you can do that without restarting the machine now another neat thing about the way FreeNAS works is this doesn't matter we can take the drives and we're going to do a little rearrangement here shuffle them and renounce just goes okay they're all part of the same array so it doesn't have any issues it's not even even the error when it comes up with this it finds all the partitions it has the configuration file so swapping the drives no big deal and if you've got to swap one out as long as it's the same equivalent size you can add it right back in the pool it rebuilds now legs I'm going to prom with that if people the drive smaller not going to rebuild and I highly recommend going with all the same drives now in our consideration with freeness when you're ordering these drives don't order them from all the same vendor at the same time the problem you run into is let's say there was a bad batch of drives and the vendors large scale you know Newegg Amazon wherever you're getting them from Micro Center they order batches at a time so if you have an entire large batch of drives and they were all bad that vendor may have an entire pile of them all bad but when you order from different vendors and different pools you'll find that the drives take up their different slightly different build dates so statistically this is just a guessing game of course statistically less likely to fail so maybe on a certain day they had a bad run but that run went to one company and another run that was good went to another company from a different day and by mixing them up from different vendors and firstly and always get the best price but with all that price matching we really get a good deal and now we have a variety of manufacture dates on the drive speaking of which something we found interesting when looking at the drive even with given from different vendors most of these drives were let's see 2016 November and I think some of them were in October so these kind of sit around for a little while because this is June and like we literally I mean they just came in the other day so this is June 2017 so these guys have had sitting around for four or five months so we've been noticing that in some of the drives we think just some of the larger ones like the six terabyte they take a little while before they you know ship them out so that was kind of it for the bill I can go in briefly cover to software and just kind of show you how you configure the rate in there it's pretty straightforward I've done I think some videos on this before but I'll give you the rundown of it real quick and show you how you set up the array for raizy - it's really easy alright so I'm in VirtualBox and the reason for that is to show you the install and do a clean clear screen capture we're going to just run through the installer process here it's exactly the same on the desktop so anything different we did on the desktop was we copied it to a USB Drive to install it to the dual USB Drive we could have just as easily rip the ISO into a CD and booted off a CD and do either way it works VirtualBox doesn't support USB bootable so we just took the ISO and attached the ISO to one of the controllers and these are my tree and ass boot one boot - we're going to show you how to set up a raid during install I can make my cells really clean and capturing it this way looks way better than a camera or my shoulder and you try to read what's on the screen but once I go through the install then we'll jump over to the physical actual build and I'll walk you through how so the radar inside of there because that's all done to the web interface let's get started now you never really want to install FreeNAS other than for testing purposes inside of a virtual machine like this it really wants direct access to the hardware just a little side note there for people wondering if hey should I run this in a virtualized environment not the most idea you want FreeNAS to have folds wrecked access now this is actually nice because you can install or upgrade reboot system shut down this is your standard menu installer but this is also for recovering a FreeNAS never had to do it I've never had one crash so bad that I couldn't just do this but you know select one or more drives where FreeNAS should be installed use the arrow keys now this is let us talk about the Select rate there's no real configuration you need to do if you select two devices those devices become the rated devices as simple as that now they're all labeled in here ata zero and eighty eight one I believe in free and a sturdy a0 and DA one which that is referring to the USB in this case because they're all connected to the PCI bus now you can install FreeNAS to a hard drive but why would you that's kind of a waste of a hard drive you could use an over hard drive because you have it but then that's not as reliable go spend a few dollars on the USB and get two of them because well we've got these on sale for six six dollars each it just makes a lot more sense this will cause all eighty-eight to be erased it's just letting you know yep we get to set a password I love when companies don't have default passwords on things when uninstall is because well people go I'll just leave it at default and not a problem and then someone hacks you later because you've left everything at default you have to use a password you can boot via ufi or bios depending on the motherboard setup I'm going to do BIOS I don't think VirtualBox does UEFI very well not real sure on that never really trying now it's going to run through the installer I'll fast-forward through all this and that's it completed the installs relative our class we're going to shut down so we can remove the CD and then you reboot Prine asks just going to do normal boot up and waiting for the seconds to countdown and now that's it so from here you can configure the network interfaces if you need to set it as static and things like that set up default routes VLAN interface link aggregation you can start getting into all the features here or do a you know reset the root password because somehow you figure out what you set it to and or just go right to the HTTP the address and get started with FreeNAS so it's kind of nice thing as it tells you what IP address accounts you don't got to go digging for it but pretty much after this if you remembered the password you set you can simply go to the AP address login as root and the password you set and get started with FreeNAS and that's where we're going to start now as we actually go into the production machine and show you how we have it configured now before we switch over to the production machine now that we've run through the install I'm going to show you what it looks like to run the wizard now I don't usually run it but if you're just getting started it's a quick way to get a couple shares enabled in France so we'll go ahead and set it to Detroit next now koala name we'll call it test tank now automatic pick reasonable defaults good reliability under virtualization backups media raid-z one or logs raid zero which means just put all them together now the wizard does a pretty good job I believe automatically it will probably pick a standard raid z2 for this one or a raid Z one for medium reliability in more storage now this is all about how many drives now the pool can fail before there's problems so we'll go ahead and just hit next in essence to show you what the wizard does or skip an active directory for this call your name is our share test we're going to hit allow guest ownership if I added users will see they can have all the permissions and add next council messages I do like this on this is a shown in the footer down here so ahead now we're not going to setup email right now confirm and with that it created the test tank the share test and it's already turned on things like the if you go to services it already turned on SMB it also creates a window share with the same name alright so we're on the production machine and it's all set up and configured so the first thing we want to do is switch it HTTP I don't care if it's your own internal network you feel it's a trusted network you I really recommend encrypting everything just have it I'll be in the habit of doing it it's a good security habit we're over here to a CA we have to create an internal CA so it's going to give us you know the error but no big deal there we'll run through a real quick all right so now we have an internal CA now we can go create a certificate create internal search all right now we have our own internal certificate we go back over here HTTPS and choose internal cert it's save all right so now we have our you know course error existent early sound certificate but you know like I said security that's important so having a certain error keeps it from passing everything through clear text so let's go to the boot and this is about us talking about how you can see the default install reboot boot volume condition healthy now this is where you'll get an error if you have a problem like say when I ensure there was two thumb drives in this I keeps the status on there and it'll do a little alert and tell you the status of these go to advanced the things I like to turn on is you can turn off actually by default the time this council menu like so if you didn't want you want to walk up to the machine generally when we're putting these machines and there's no keyboard or mouse attached from only an emergency eases show council messages in the footer I like this feature quite a bit it's just going to put everything at the bottom there in able auto-tune this is a cool little feature not the auto-tune music but what it does is is going to have all the system tunable z' which are not populated by default it'll choose all the optimal configurations for this now i've never had a problem with it i'm always turning it on all the system tunable z' but if you do have problems turn that off there's not much it found in here it says generated by auto-tune exam prefer generates a couple more in reboot or not I always just turn it on in case there's anything I need to do or it needs to do in there updates by default already check for being able to check there leave it at the stable don't accidentally choose this because if you choose a higher up it may not roll back properly better effect very likely it won't so there's kind of the basics for getting it set up now let's get to the important part the storage array so there's nothing in here know that nothing in divine management you can see some disks I don't know entry says we have none set up yet so we better in we could import a volume if you were doing this on a load and bringing in other ones or we know the volume manager and let's set this up so this is here's our six drives and this is the system right here where you can simply choose how you want the drives to be laid out so if we go to a z3 we have a capacity of only 16 gigs see I'm sorry 16 terabytes raid-z would give us 27 terabytes but if any one drive failed that's it second drive fails you're lost raid-z 2 gives us the 21.8 2 terabytes so we'll actually just call this drive on creatively 21 terabytes all right there's an encryption option they give you a warning please read this you must backup the key you don't have backup the key and your thumb drives fail you are not getting back whatever's on that valium now I highly recommend encrypting the volumes and one of the real reasons you encrypt it it's not you know just because you have personal stuff on there but a lot of people do but it's if you send those drives back where's the data going I know granted one drive fails they've only got enough data so maybe reassemble something but it's not worth the risk this way if a drive fails and you send it back there's no work the drives encrypted and without that key they're not getting it so I highly recommend encryption but we're going to show you how to back it up I highly recommend backing up the encryption key it's not hard to do but please store it in a safe place stored offline keep it somewhere wherever you need to keep it to make sure that you know if something happens you can get those drives back up and running if not you lose the key that's it all right now the volumes created these are some of the council notifications that show up these are some of these show up on the council itself as well whenever you create a volume so if there's any errors you can look for me here this is also I like the council being and really accessible you just click this at the bottom and away you go so we can see it's 21 terabytes the raid-z 2 configuration so they're all in one large pool now designer FreeNAS I can't just add to this pool I can't just say let me throw another Drive in and rebuild it it'll just add more magically there are a couple different raid systems out there I've seen that have different options like that I think Drobo has some specialized raid system I'm very confident in the ZFS file system a very confident in how it handles everything and so I'm fine with the fact that I can't expand it generally when I'm building I'd build out all the drives I'm going to put into a system and it's going to run for years until it's time to replace your operate that system now you can add this is just one set if you have like three more and it's what we did in our machine we put in the four drives later we added a few more we have more expansion or you can get a card like a PCIe card and add more drives so you can add another volume you just can't change the existing volume but easy solution copy everything from the original volume to the new one then remove it and now you can have the same system later on swap out big or hard drives to it now this also is where you would replace a disk at a just take a disk out of the pool pretty straightforward if you had an extra disk or you'll actually see an error on it we'd be offline you hit replace and you would choose from the free pool I've got a separate video on how to swap out one of those drives if they go bad so back over here is storage now the next thing we're going to do here is this is where we start setting up each one of the volumes now the way this company set up they have just one big shared drive for the engineering team and that's who's going to be doing this so we'll just create one data set now we don't want to create as eval that's different in the kind of special use which I will get into for the most part you just want to be able to create data sets and roll let's call it shared drive shared drive and we're going to go ahead and add the data set now not going to be included in this video I'll do this as a separate video altogether they're going to be doing this with Active Directory Integration so I'll do a whole separate videos though because I think there - it might be worthy of just a separate video in plus I don't have an active directory to hook it up to it's still at my office and testing here part of the testing we do this is important to us because you don't realize there's problems until you start loading data I don't want to load the customer data and then learn as a problem so we load test these and just find all kinds of stuff to put on there I've actually already went through one load test and then we actually just reset it again for the video to make sure everything was good on this but load testing the machine is really really important so we dumped about four terabytes of data over to the sir I've destroyed it as in detach the array and reset it then went and did it again and did it again so it went through several that since we built it it's actually now been through a couple load tests and there was an error here we think we had a CD cable is one plug to plug it back in the error went away I did do some research and someone else had pointed out an error with these tries but they're in error days it was one CRC error so but we still keep an eye on everything to make sure it works all the discs still show healthy and no issues at all all right so we have the drive set up they're configured and I had brought up making sure that you back everything up this is where you can get the keys now you can rekey it and I think every time you add a drive to the pool I think it automatically wants to rekey from not mistaken is another video on that but we're going to do right now is download the key because the key itself is not stored in the backup config so you have the backup config but you want to get this GE lig file it's really important to keep this key file there's really not much in this file either it's a very small little file keep it for the client because it will save you if ever everything happens this G il GLI key is stored within the USB drives if for some reason even though they're mirrored they fry the USB bus fries you're like oh cool I'm just going to rebuild the box and import the volume you won't import the volume at all if you don't have that key even if you have the backup config so if we go here to system general and we save the config here that key is not in here just all the settings for free master in here so you could end up in a lot of trouble you're like oh I don't have it well you have 21 terabytes of encrypted information at that point if you didn't have it so just a side note make sure you backup that key and the backup of this storm in a safe place once you have this thing up and running okay I got it configured I'm going to put my important things on it do that configuration on there maybe another time I'll get in more in-depth if there's some interest in how to set the shares and things like that it's not too hard to do but thanks for watching if you like the content here like like and subscribe if you have questions about the FreeNAS build reach out to me and we'll cover any more questions or gives you a more in depth video on certain things on preamps thanks for watching

Channel: Lawrence Systems

Views: 149,224

Keywords: nas, zfs, storage, network attached storage, freenas (software), server, open source, install freenas, freenas tutorial, freebsd, freenas setup guide, freenas, network-attached storage, bsd

Id: Tm4Iu6fGZe0

Length: 28min 0sec (1680 seconds)

Published: Wed Jun 14 2017

Reddit Comments

[u/charliebrownau]

• Gday Interesting build

• Great to see someone actually using WD red (non pro) drives

• Would be nice to hear about this build how it pans out 12 and 14 months down the track

• Seems odd considering your paying that much money you didn't build this is an Server mobo and ECC ram and using 4 port gig nic or 10GE nic

[u/charliebrownau]

Whats the file transfer speed like ?

Would it not be a good idea to use multiple nics with intels team load balance or linux's bonding to increase LAN speed thouput ?