4 NVMe Drives on a Single PCI Card ($25) - My New TrueNAS Cache Solution

over the years there's been many technological advancements in the world of personal computers but not more significant than the hard drive now when i was first building computers it wasn't uncommon to have something like this as your boot drive now this is a 7200 regular hard disk drive over the years we've kind of evolved into a place where now many of your computers will often boot from a drive the size of a stick of gum this is an nvme drive and it's about 30 times faster than its bulkier grandfather so you can see the size comparison it's pretty easy to see the difference there in terms of speed though you're looking at about 100 megabytes per second reading writes on a hard disk like this but with current nvme technology now this is a gen 3 drive we're currently up to gen 4. but this drive will get you around 3 400 and 2300 megabytes per second reading writes respectively but what happens if we take three more of those and combine that with a pcie adapter so the idea here is to see if we can actually get four times the performance of a single drive by using this adapter let's try it out [Music] let's put these aside for now and we are definitely done with this so what this is is a four times nvme to achi pci adapter this comes from a brand called gy maybe zyn zy i clearly don't speak chinese but i will leave a link to the product page below i got it off of aliexpress for only 25 and i couldn't really pass that up because i've been looking for a reason to utilize some of the pci lanes on my server motherboard and this felt like a good opportunity to do that the idea is that it can take four different nvme drives two on one side two on the other and utilize a pci lane to give you four drives now there are a lot of different cards out there that take your drives and let you adapt them to a pci lane and most of the time they'll be in one of two modes it mode or ir mode now this card is in what we would call it mode or initiated target mode meaning that every drive you plug into this card will be seen independently by your system on the other side you have ir mode for integrated raid so what that does is actually does a hardware raid on the actual device of every drive that you have on the card so a benefit of that is that if you wanted to use um these drives as a boot drive you wouldn't have to do any software configuration all of your drives would be rated before it is even seen by the system now personally i'd rather i t mode because i'd rather the flexibility of being able to do what i want to do with each individual drive and if i want to rate them all together i can definitely do that so the idea here is to first install all the drives uh pass them through to a windows vm and raid them all together in raid 0. let's see what we can get for 25 bucks okay we should probably install the drives first right does this look a little bent to you guys this is uh i'm gonna stop focusing on my face it seems a little bad but i probably should have mentioned before these are 512 megabyte megabytes these are 512 gigabyte drives from silicon power they are pci gen 3 just because that's more compatible if i wanted to use it on a different server or a different computer and they're just cheaper so let's finish installing them okay here you have it two on one side two on the other and all that's kind of left is to install it in the server one thing to note though i mentioned earlier that this is essentially an it mode meaning that the motherboard will see each drive individually now the only way that will actually work is if your motherboard supports bifurcation on the pci slot that you install this on now bifurcation is taking a pci 16x slot and turning that into four by 4x slots so if you plug this into a regular pci by 16 slot that doesn't support bifurcation it's possible that your motherboard will only see one drive so check your motherboard manual and motherboard sites to ensure that your motherboard actually supports bifurcation and how to enable that in the bios before trying something like this okay let's go install it okay so we are back i've installed the card into the server now immediately one of my favorite parts about the server motherboard that i'm using is that it has ipmi so what this does is allows me to interact with the server over a network connection so i can basically do anything i want with the server in a standard web browser here you can see my ipmi dashboard now the server is off now but i can easily go to this remote control and launch the kvm so i can actually control my server and see what's going on through the video output directly in the browser so i've already launched it over here and we are going to send a power signal okay we are in our bios setup screen and from here we can basically do whatever it is we need to do so the reason i'm in here now is because before i mentioned bifurcation on the pci slot so i'm going to go ahead and do that now i just have to remember where it is so under chipset i actually put this in slot 5 so if i go to pci slot five link width i can change this to four by four by four by four essentially breaking down that uh 16x slot into four separate uh 4x slots and it's going to do all its rebooting but i will see you back when i am in my proxmox web gui okay here we are in my proxmox dashboard so what we want to do is actually pass through all those nvme drives now if we did this correctly we should see four separate nvme drives or pci controllers we can pass through so if i go to hardware add pci device i should see some kind of huh i don't know what 51 is this could be it so let's try it okay click start and it looks like it is not going to start up okay and yeah we are getting io error eyewear while writing super block okay after doing some troubleshooting um i can't seem to get the actual pci card passed through i can't seem to get past this failed to set apst feature error that i keep getting so we're gonna go a different route and it shouldn't be an issue but instead of trying to pass through the actual pci card to the vm we're just gonna pass through the drives themselves directly to the vm so what i've done is i actually went in and used proxmox's official guide for passing through a physical disk to a virtual machine and it's extremely simple you just go in and specify which device or which disk id you want to pass through and one line of code essentially that you have to run it's fairly straightforward and then that device will be seen by the vm so luckily if we go into our server and down to disks you'll see that at least the host is seeing all of my drives here you can see all four of the 500 gigabyte nvme drives now i've already run the configuration for all four drives to pass them through to my windows vm so you can see here one two three four uh scuzzy pass-throughs and now if we go to start the vm we should not see any errors perfect okay so that will boot up and i will remote into that server okay so here we are in windows and by default i don't believe they will be set up and formatted now so if you look in here you'll see that only my local drive exists so what you're going to want to do in windows is actually create a striped volume and it's pretty simple to do you're going to go back to your disk management and what you're going to want to do is click on one of your disks and right click in the gray area and go to new striped volume and click next and it's going to ask which disks you want to use for that striped volume so we're going to want to add all those disks next i'm going to click next we're going to assign that to e we're going to call this red zero next finish so right now saying it's going to convert them all to dynamic disks which is fine you could have done that before but okay so we wait for it to format okay so it's finally done formatting and that took way longer than i thought it would but here you can see it looks a little bit different but here all of them are dynamic now which is why they're green and they are finished formatting and marked as healthy so if we go into our windows explorer you will see it looks pretty similar so let's go ahead and run the test and right off the bat we should see an increased speed okay much better all right we're gonna go ahead and let this finish and when we come back we will look at the results okay it is done running and we have some interesting results you can see clearly that our read speeds on the random reads went way up and you can tell definitely from the write speeds that we didn't really increase that much now whether this is a factor based on the pci card itself or with how windows handles the striping of the drive or the way crystal disk mark actually writes the data there's many factors that can go into it but the fact that we're seeing an increase in performance after striping the drives together is definitely a positive but i don't really care to use those four drives in raid zero my use case for this card was actually to use all four of the drives independently the fact that i can add four more nvme drives off of a single pci slot is extremely beneficial so what i want to do now is actually pass all four of them directly through to my true nas server and separate those drives into separate caching disks so let's go ahead and do that okay we are back and you can see that i am now in my truenast vm and we have done the same thing we have taken those drives away from my windows vm and now we are passing them directly through to my true nas vm so i'm gonna go ahead and start my trunas vm up here we are in my true nas dashboard and you can see we've allocated 60 gigs of ram we have two main hard disk pools so here is one this is four 12 terabyte drives that is my main one and then we also have my one that i use for plex which is two six terabyte ones now right now neither of them have cash drives but if i go into storage and disks you can now see that we have one two three where's the fourth one uh oh four so we can now use those as cache drives and the cool thing about cache drives is that um you can add them and take them away from a pool without having to do any moving of actual data it's non-destructive so to do that we can simply pick a pool i'll do my plex one first and then you go down to add v devs so right now it's going to ask what kind of vw do you want to do and there's a lot that goes into caching and logging the two separate types of kind of caching you can do with high-speed drives now there's a lot of forums and a lot smarter people that know a lot more about zfs that can answer this information better about the differences and when to use what unless you have a low amount of ram you probably don't want to use an l2 arc cache drive the log file on the other hand can definitely speed up writing to the drives so that's what we're going to do we're going to choose a log and we're going to pick da1 da2 and down here you can choose whether you want to stripe your disks or if you want to mirror them so you can see when we have it selected to stripe it's saying a stripe log v dev is highly discouraged and will result in data loss if it fails but to play it safe we will change it to mirror and that error goes away and everything is happy okay so next we are going to want to add a v dev to my other pool so i go back to dashboard you'll see the pools on here here's our four by 12 terabyte you can see two data v-devs and one log v-dev which is a mirror of two of our nvme drives and you'll see very similarly on our two times six terabyte pool one v dev and one log vita which is again a mirror of two nvme drives so overall i am pretty happy with my purchase it was 25 and the main thing was that it allowed me to take four high-speed nvme disks and pass them directly through to my truenass virtual machine while only taking up a single pci slot but for 25.00 i really don't think you can beat that i'd strongly recommend this if you're looking to do something similar like passing through large amounts of nvme drives to virtual machines that card is pretty solid for that purpose that's all i will leave a link in the description below to the card that i got off of aliexpress if you're interested go pick one up if you like this video be sure to drop a like comment below if you have any different pci nvme cards that you're using and what kind of performance you're getting off of that i'd be interested to see what you guys are doing at home with that that's it uh until next time i will see you guys in the next video

Channel: Raid Owl

Views: 13,989

Keywords:

Id: cOcR5OGbNBI

Length: 17min 31sec (1051 seconds)

Published: Mon Apr 05 2021