TrueNAS Cache Disks with FusionIO Drives

i am always looking for new ways to speed up my storage server and installing a cache drive is the most recommended way to do that but what if i told you that adding a cache disk to your truenav storage pool doesn't do what you think it does welcome back to craft computing everyone as always i'm jeff a little while back i took a look at a couple different types of pci express cache drives first there was the sun f80 an 800 gigabyte drive on a pci express card which more accurately was actually four 200 gigabyte sas ssds while you could address each of these drives individually if you wanted to they were still limited to the bus speed of sas which is around 550 megabytes per second flashing the firmware allowed for striping the drives together into a massive raid 0 array increasing the read and write speeds to around 2000 megabytes per second and giving you a single 800 gigabyte volume a short time after that video went live a viewer reached out and sent me a full set of fusion i o i o memory drives ranging from a lowly 160 gigabyte disk all the way up to a 2.6 terabyte monster usability was let's just call it not great with the only drivers available being for windows even then that only gave you the ability to use them as a glorified nvme drive not to accelerate slower storage so i have a great set of high endurance ssds but no way of getting them working on my file server that's full of spinning disks but when have no drivers or no official support ever stopped me before surely there's a way to get these working inside true nas and put a little pep into that 10 gigabit backbone on my network luckily there are people out there who think exactly like me and they post on github today i'm going to show you both how to get these fusion io drives working inside of trunas as well as talk about two different methods of setting up a cache disk so you can choose the one that works for you for starters if you haven't figured it out already the drivers we'll be installing today are not officially supported so your mileage may vary both for stability and for speed just keep in mind that i'm setting these up in my home server and not in a production environment i'll also be setting this up in the truenas scale beta version 21.08 as it's based on debian rather than freebsd while the source code is available for this driver pack i have not tested it on the bsd based truenas core for my main storage pool i'm going to be using a 1.2 terabyte fusion i o i o drive 2. while it's not the fastest i o drive in my collection at around 2000 megabytes per second it more than exceeds the speeds of a 10 gigabit network so anything faster installed in the server would just be a waste to start off as i am virtualizing my true nas scale install i simply need to pass through the i o drive to my trunas vm in proxmox this is as easy as selecting the vm going to the hardware tab and clicking on add pci device then just select the i o drive from the pull-down menu and click on add once truenas has booted up i'm going to download the remix vsl repo from github there are two different versions depending on your model of fusion i o drive so make sure to check the documentation in my case the i o drive 2 is going to use the vsl3 driver package to download type in git clone followed by the url of the remix vsl repo and in less time than it takes to take a sip from your craft computing pint glass that you got over at craft computing.store we're ready to install navigate into the newly created i o memory vsl directory and then type in make dkms this will create a kernel module with the required driver package for your fusion i o drive and once installed go ahead and reboot truenas getting the fusion drive working really is as simple as that the disk should now be showing up as a storage disk and ready to configure inside your storage pools but now we get into the question of the day and that is what do you want out of your cash drive if you're like me you probably assumed that setting up a cash drive on a cfs pool would enable write caching where anything you wanted to write to storage would be sent to your ssd before being written over to the slower spinning storage unfortunately that's not how caching works on a zfs pool cache drives are actually there to ensure data integrity during file transfers not to speed up the transfer itself so by default adding an ssd cache drive to an existing storage pool will better guarantee that all data actually makes it onto your hard drives but it won't help speed things up when writing to a zfs pool data is first cached in your ram and then written onto your spinning disks a cache drive doesn't speed up that transfer rather it's there as a second destination when transferring data out of memory making sure the data is intact in case of a power outage or some other type of failure but what if i wanted to saturate my 10 gigabit network with every single file transfer that's where zfs intent log and separate intent log caching come into play that's what allows you to set up a high speed ssd as a write cache disk where the ssd serves as an intermediate destination between memory and your spinning storage drastically increasing transfer speeds at least in theory and yet again the process for installing a cache drive could not be easier open up your truenast management page then go to storage click on the settings cog for the storage pool you'd like to modify and click on add v-devs inside the pool manager click on the add v-dev pull-down menu and select log from the list then just choose your ssd and add it to the log v-dev and there's one last thing you need to do before you can start swinging files around and that's enable real-time sync on your data sets this will write all new data first from memory to your cache drive and then to your other hard drives under storage click on the three dots to the right of the storage pool you'd like to enable write caching on and then click on edit options in the options menu set sync to always and then save your changes and that should about do it your right cache is now up and running but let's not celebrate too much as i haven't talked about my results yet before adding in the cache drive i would see a burst at the beginning of a long file transfer usually around 400 to 450 megabytes per second after around 5 gigabytes had been copied over the speed would drop to around 230 megabytes per second where it would stay until the copy was completed in the display over here you can see that each disk is only capable of riding at around 50 megabytes per second even spread across eight disks that still leaves quite a bit of headroom on my 10 gigabit network so by adding a cache disk that's easily capable of 2 000 megabytes per second my problem should all be solved right actually the opposite happened even working properly by utilizing my cache disk for writing the entire time my transfer speed actually dropped from a sustained 230 megabytes per second to around 110 megabytes per second that's basically gigabit speeds and completely counter intuitive to what should have happened so what gives well remember this is an amd epic 7601 first generation cpu and while it may have 32 cores and 64 threads it also has a base clock of 2.2 gigahertz and doesn't boost inside of proxmox meaning my single threaded performance is around half that of a ryzen 1700 and in my cpu monitor i can regularly see threads hitting 100 and holding there during file transfers so the issue at this point isn't my storage bandwidth it's actually my cpu single threaded performance holding me back and there may not be much that i can do to combat at this time short of switching platforms again but at this state i am pretty invested in my two epic servers and would really prefer to keep them as they are at least for the next couple months anyway remember i'm still me some disappointing results but that's the way it goes sometimes at least i was able to successfully get the fusion i o drives working inside true nas and we learned a little something about cache functionality in zfs on the way it requires a crap ton of single threaded speed if you have any suggestions on improving either my right or read speeds please let me know down in the comments below on your way down there make sure to drop this video a like and subscribe to craft computing if you haven't done so already follow me on twitter craft computing to keep up with daily shenanigans like this and if you like the content you see on this channel and want to help support me in what i do head on over to craftcomputing.store and pick yourself up a matching pint glass if you want to directly contribute to the channel and jump onto my exclusive discord server head on over to patreon or floatplane and uh look me up and that is all i have for you in this one everyone thank you all so much for watching and as always i will see you in the next video cheers guys here for today is from adroit theory and it is the death of civilization 1 hazy imperial ipa advert theory is located in vint hill virginia and this clocks in at 8.0 percent a lot of pineapple right up front a little bit of an orange peel on the nose of this one and like i said maybe a little lemon and a little pineapple kind of poking through but the flavor on this one it is all pineapple right at the front of it now this is gonna sound a little odd but it's the second time i've smelled it there's like a hint of baby powder there's no other way to describe that um as soon as i cracked the can i got a little bit of a whiff i'm like i know that smell like i i know what that smell is uh yeah it's orange peel and baby powder and if you like the channel and want to help support me in what i do head on over to craftycomputing.store get yourself a pint glass or a swaggy t-shirt the hell did i say swaggy i've never said swaggy before in my life

Channel: Craft Computing

Views: 32,559

Keywords:

Id: eRL7vGHewRE

Length: 9min 54sec (594 seconds)

Published: Sun Sep 12 2021