Dell PowerEdge R820 R720 storage upgrade Part 3 | How to upgrade with NVMe U.2 SSDs!

hi guys welcome to the art of server and welcome back to part three of the r810 storage upgrade series this is the final video in this series and as i alluded to in the last video i'm going to show you another upgrade path for storage on the r810 and this also happens to be an option that will apply to the r720 as well so long as you have the two and a half inch base and today we're going to show you how to upgrade these servers with these ssds now these are at first glance look like your normal sas or sata ssd but in fact they are u.2 nvme ssds and so i have here an intel dcp 3600 series 400 gigabyte u.2 ssd now if you look at it initially it looks like it has a normal sas connector but it's actually quite different let me show you in in comparison to the sas ssds that we tested out in the last video if you kind of look at the connectors you'll notice that yes physically they look uh very similar but electrically there's some differences here you'll notice for example that on the assass 3 on the left here there are no pins on the bottom side of this bridge between the power and the data connection whereas on the u.2 there are it looks like appears to be six pins on that bridge area also it's probably a little bit hard to see in the camera here but on the back side of the sas connector there are no pins other than on the bridge for the secondary data channel and that's all there is while on the u.2 there are actually a whole array of pins on the back side and so these are ssds that are meant to be connected directly to the pcie bus there is no sas control or anything involved there and whereas the sas 3 does require a sas 3 controller like we showed in the last video so although mechanically they look very similar electrically they are quite different and so what i have here is uh well this is just a sample but i have four of these in this drive cage and so what i have here is a actual nvme an official dell nvme upgrade kit for the 12th generation dell servers this kit does fit the r720 as well as the r820 and in order for this kit to work obviously you need the ssds but you also need the drive cage with the back plane now it might be possible to fit the backplane into a drive cage that's not an official one for the nvme ssds i haven't tried that but you know looking at it it kind of looks like it might actually work that way so if you already have an expert drive cage and you're just looking for the back plane you might be able to get away with just getting the back plane all right but anyway uh this is the official nvme upgrade kit drive cage and so you will notice on top here it says pcie ssd and you'll notice that we only have four bays and that's all we can afford in terms of pcie lanes for this back plane and so this other half of this drive cage is basically has just a blank there's nothing in it on the back here we have the nvme backplane and so there are four u.2 slots on this side here and they are connected to these four sff 8087 ports so at first glance you might think that looks like a sas port but these uh basically sas cables they're equivalent to sas cables are being used to transmit the pcie lanes directly from the pci bus of the system in addition to this if you are going to be buying this upgrade kit and by the way there are a lot of different parts here i'm not going to read all the part numbers just because there's just quite a few of them but i will leave the part number details down in the video description below with affiliate links for you to find these parts on ebay and so if you are going to pursue this upgrade path go check that out for the details so anyway in addition to this back plane you do need these two additional cables so if you're going to buy this kit make sure that you get these cables the one of these cables is for power the power of the backplane which powers the ssds and the other one is a data cable that communicates to the motherboard and so i'm not exactly sure what the signaling is exactly for but i suspect it's probably to for example detect presence of an ssd on the back plane and that type of stuff so you will need these two cables with this back plane sometimes if you're buying these kits used they pull off these cables and throw them away or do something else make sure you get a kit with the cables or make sure you get the cables in addition to the back plane if you don't if the back plane you get doesn't come with the cables okay so like i said this connects to the pcie bus so you have the ssds to the back plane the back plane has those sff 8087 connectors and you need to connect those to the pcie bus and that's done through this cable set here so this is the official uh cable set that goes with this kit and again i will leave the part number two these cables down in the video description they look like ordinary sas cables with sff 8087 connectors on both ends i don't actually know if they are wired just like the sas cable so i don't know if you can substitute normal sas cables um instead of using this official cable kit but if you get the official cable kit uh it it is nicely labeled it has controller pcie d c b and a labeled on this side nothing labeled on this other side but anyway you know probably safest bet to get this cable it is almost just the right length for the installation so might as well do that all right so once you have the cables connected to the back plane that has to connect to the pcie bus of the system and you do that with this card here so this card is a by 16 pcie card that splits those pcie lanes into four by four sff 8087 connectors and like i said this is not a sas controller you might look at this and say oh this looks like some sort of sas expander or something like that but it is not it is for transmitting these pcie lanes over those sff 8087 ports to this back plane to enable you to connect these nvme drives into the server all right so again i'm going to leave all the part numbers to this kit all the details including the blanks the cage the back plane the cables the sff 8087 cables and as well as this pcie bridge chip or whatever it is this card is so anyway if you are going to pursue this yeah definitely go check out all the details uh in the video description down below i it took me some time to kind of research and research this and find all the uh correct part numbers and so anyway today we're going to go ahead and upgrade my r820 with this kit and then we're going to boot it up and log into a os and test out the performance of these four nvme ssds with this upgrade all right guys so here we are inside the ra20 again we're gonna have to remove the top motherboard we're gonna remove the fans because we need to connect the backplane uh cables to the motherboard under here so let's go ahead and just remove all that and clear some room all right so here's the plan there are connectors on the motherboard right here that says bp2 bp3 and bp63 bp sig 2 so i think i'm going to use the bp2 it's interesting that there is a bp3 i'm not really sure what that's for because i can't imagine that the third backplane can fit in this system so but i don't know maybe there is some such a configuration so we're going to put the drive cage over here we're going to replace this blank with that drive cage that i just showed you we're going to connect that to bp2 on the motherboard and we're going to put the pcie bridge card over here now that bridge card is a full profile card so in the r820 it has to fit here on riser one on the r720 i believe uh the riser one area is going to be a low profile so you won't be able to fit it there it'll probably have to fit in the riser two area which is i believe a full profile on the r720 so just make note of that difference if you're installing this upgrade kit on a r720 all right so let me get some screwdrivers let's get this cage out and put the drive cage in first all right so there are two screws holding the let me just move this cable out of the way a little bit this is the data cable to that first backplane there are two screws here [Music] one more on this side let me just kind of get that out of the way all right so with the two screws removed i believe this drive cage i can just push it right out yep there we go all right it's getting a little bit stuck here all right just a little bit more force and it looks like it can come right out all right let me change the camera angle so you can kind of see how that works all right as you can see the drive blank or the drive cage blank is coming out so all i have to do i believe just pull the rest of it out and i'll put this aside all right so now we're just going to install the nvme drive cage with the back plane i'll just kind of guide these cables in first and then i should be able to slide the entire cage right in there okay it's a little bit of a tight fit all right let's try this again all right guys uh sorry i had to cut the video there because i was having a hard time getting this drive cage into the server here and i tried a couple different things and nothing was really working and so i ended up figuring out that it's a lot easier to install this when you have everything taken apart so i've taken out the drives i've disconnected the two cables and i'm going to take out this blank as well as well as the back plane itself and so this might just kind of show you the process we basically press on this blue thing here and the back plane itself comes right out okay so i'm going to put those aside for a second here and now i'm going to install the the bare cage by itself all right so for whatever reason with everything inside this just seems to have a really hard time fitting in and i found that it's easier if you just take the whole thing apart install the cage by itself first and then we can put all the pieces back together so i just want to share that with you guys in case you're following along this video and you're having a hard time installing this kit uh try taking it all apart and just install the cage first all right so i've got the blank in there let's go ahead and put the uh backplane on the backside okay so to get the backplane here there are two little notches right here that fit in there and over here so you want to kind of angle it in a little bit first and get it into those notches and then these two little hooks will also hook onto the top and then you just press it in and then that will snap in and so it should be pretty secure once that's in and then all i need next are the two cables so let me deal with the power cable first so this is to the back plane okay so the power connector here i guess i can put these other well you know what hold on before i do that let me go ahead and secure the the drive cage and everything with the screws here just so i don't forget that once i get the cables in there it might be a little bit hard to get the screws into those holes so let me do that first okay all right now let's get back to the cables so this is to the back plane that goes down here all right and let me put some of these cables back into the little cable holders and this goes in here so there's a backplane power connector here on the motherboard i'll plug it right in there and then see if i can tuck this cable also into the little cable holder over there all right and then we have the data cable and so that size says mb so that's for the motherboard side and then this side must be for the back plane so this is hopefully you guys can see that a little bit but move my hand out of the way basically i'll just press this in there oops oh well that's interesting the back plane is still loose oh you know what i put the screws in the wrong hole so let me [Music] take that out sorry about that guys this is the first time i've been actually installing this kit so i gotta do it [Music] all right so that should get it in there now let me fix the other screw as well i definitely do the screws before the cables so you can see the cables are kind of getting in the way here let me see if i can all right so hopefully that does it you know i might need another screw here because this screw actually holds onto the blank as you can see if i move the blank so i'll have to go find another screw that might fit in there i don't have a spare screw but i don't want the blank to kind of fall out so i'll need another screw to secure that blank later on but anyway we got the data cable let's get back to getting all the cabling down the data cable then goes into this connector here all right okay so let me see if i can wrap this cable into that cable holder right there all right so hopefully that's good enough well let's see that doesn't seem ideal but maybe like this all right that seems a little bit better so make sure the connections are good all right so let me go put the ssds back into the okay so i'm going gonna just install the ssds back in here yeah definitely i was having a hard time getting this cage in with all the stuff uh already inside the drive cage so it's definitely a lot easier to do this with the all that stuff removed so just a little tip for you guys who are following along all right so drive cage is installed back plane cabling all that is done let's go ahead and get back to the inside of the server and get the rest installed all right so we got the backplane in there let's see if i can get this cabling in let's remove all the caps on the cable here okay i didn't notice this earlier but see if i can show you these connectors are actually labeled they were just labeled on the head not on the cable itself so that goes to a b c and d all right and also on the back plane they are labeled a b c and d okay that d connector is a little bit tight there all right so got the cables connected see if i can run this into the cable channel here there's quite a few cables here so i don't know how well this is going to fit it should fit all right that looks like it fits now these cables are tied up over here but i don't really think that's helping me so i'm going to cut that tape i'm just going to take this tape off here let me just route this independent independently might make it a little bit easier now for the second cable yeah that is quite a lot of cable to fit in here and i'm going to fit the yeah this is definitely a very tight fit with all this thick cabling like i said i've never installed this kit before all right so i guess it can be made to fit but it is a very tight fit now i do need to put the pcie bridge card in this riser so let me pull this out i'm not really sure which slot would be ideal at this point but i'm going to try okay so reading the print on the riser board here this is a by eight connected cpu two and this is by 16 so i think we're going to have to use the top slot since that's an actual by 16 electrically by 16 slot so [Music] that choice i think is already determined by that uh let's see i think this comes out doesn't it okay make some room and this little blue bracket has these little notches that hold the board so i guess as i slide it back in i want to make sure that that engages the board now i can't really see the board anymore but i know that this port is labeled a just kind of looking at it from the side here this is a b c and d so keeping that in mind let's see if we can get these cables in there in the right order all right let's get the caps off these all right so i know a let me start with that a goes in here b goes in the next one you know what there's a little cable holder thing right here on this black bracket i'm going to take advantage of that to help me kind of hold the cables in place okay and then c next one and then d all right so that looks like how it's going to go yeah definitely i can't see how you can install this with these two cables taped together so definitely cut the cable maybe this cable is more designed for the r720 where this board would have been installed in the middle but in any case it should still work here alright so everything looks looks good so far let me go ahead and put the fan wall and the second motherboard back in and just make sure everything is still going to fit alright so here is the fan wall [Music] okay that looks like it's fitting just right all right let's get the motherboard all right so everything looks like it actually does fit and these cables are just at the edge of the motherboard there so this is actually it's very tight fit but it does fit all right so that's the end of the actual physical hardware upgrade let me close this guy up and then we'll turn it on boot up into centos 7 and see if we see the nvme drives we'll run a couple benchmarks and see how they perform all right guys so i'm logged into the server by ssh right now and first thing i'd like to just kind of verify that all that hardware that we just put in is actually visible to the operating system so i'm going to run ls pci and i'll do the tree view let's take a look here so lots of connected devices here all right so right here we see intel data center ssds and so those are the only four intel ssds in that system so this must be it and we can see the tree uh of connected devices here so these four ssds are showing up as pci devices 464 through 49 that is connected to this these devices here and which is like 45 through 49 i guess and then 44 through 49 which is then connected to 40. so let's just kind of look through this pci device tree just from the top and see if we can identify what we're looking at here all right so we know the top device here is 40 and it's connected to this branch here 03.3 so let's take a look at that so that's 40 03.3 or sorry 03.0 all right so this looks like it's the pci root port from the processor i think and so let's go further down the tree here so this is then connected to device 44 through 49 let's take a look at those two so that's 44 all right so here we have plx technology so that is a pci switch company that is now owned by broadcom or what a lot of you guys might know as lsi okay so this is definitely the that card with the four ports that is basically a pci switch so let's take a closer look at this we can see the vendor id there and we can see this uh product code here so let's let's take a look at this by the vendor and product 8632 all right so we can see that we have four devices here from that pci switch oh hmm well this is interesting um something i wasn't really expecting but if you take a look here it says it's a 12 poor pci express gen 2 at five giga transfers per second so this system and those ssds are actually pci gen 3 uh or pci 3.0 but it looks like this pci switch is a 2.0 switch only we'll have to investigate well you know what let's take a look at what the link um all right let's all right so i'm going to look at all those devices again and i'm going to get some device properties out of that and we'll look for specifically link status yeah okay so definitely this is pci 2.0 you can see here that we have a by 16 connection so that's that's the connection on the pci switch that's connected directly to that riser because remember we put it in a by 16 slot and there are four by four connections so so those are the sff 887 ports that are connected to that backplane each one of them carrying four pcie lanes but you'll notice the speed here is they're all at five giga transfers per second so that's definitely pci 2.0 speeds and not pci 3.0 speeds uh yeah so that is kind of a surprise i wonder why intel or not intel but dell used a pci 2.0 switch for this kit i'm not really well you know let's take a look here i'm just kind of curious uh let's go to google and what was that it is a plx 8632 so let's let's do a quick search for this plx 8632 and there's a link from broadcom so i'm going to go with that because that's basically the company all right so the pex 8632 device offers 32 pci express gen 2 lanes capable of configuring up to 12 flexible ports yeah so this is definitely definitely a a pcie gen 2 uh pci switch on that card and i was actually hoping that would be a gen 3 considering that this generation of hardware is all gen 3 anyway that's really surprising to me so well okay so that means that at four lanes we're looking at about 500 megabytes so gen two speeds will give you about 500 megabytes per second per pci lane so at by four uh that means we're going to be seeing about 2 000 megabytes per second theoretical max per connection to the ssd so so that is still better than sas three for example which i think maxes out at around one point something gigabytes per second this is at least going to be about almost 2 gigabytes per second so there is a little bit more bandwidth room there than the sas 3 upgrade that we did in the last video but it's definitely not uh gen 3 that i was hoping for which would give us about 7.8 uh gigabytes per second bandwidth per or not per but well per connection to the ssd with the by 4 pci lanes but that said the ssds that i have in here so let's take a look at those so these are let's just take a really quick look at this i don't think the bandwidth limitation is going to be horrible for these particular ssds because they're just not the highest performing ones but let's take a look at what they're marketed as it okay so they're marketed as uh capable sequential raise of uh 2100 megabytes per second so we're a little bit under that at 2000 megabytes per second we're just a little bit under that so we might not see the full read performance of these ssds with the pci 2.0 switch in there but as far as writes are concerned it really doesn't even approach anything close to 2000 megabytes per second so you know we're not going to really lose out on any right performance here with these particular ssds but if you were to use a higher performing ssds then i think this pci 2.0 restriction on this plx switch would be more of a bottleneck or more of a hindrance so with that in mind if you do pursue this upgrade path and i'm just kind of learning this as i go here but uh the the thing that's coming to mind now is that okay i'm kind of glad that i got the piece 3600 um u.2 ssds because if i got anything more performant than this i would probably bottleneck by that pci switch anyway now of course these ssds are not the most high performing in terms of writes so if you got something with if you needed more right bandwidth more right performance then getting a faster ssd would be okay you still there's a plenty of headroom compared to this 550 up to 2000 right so there is headroom for that to add a more uh right performant ssd in here to take advantage of what is available which is the you know pci 2.0 by four but you know anything that goes beyond two thousand megabytes per second on either reads or rights would essentially be pointless in this case because this particular plx pci switch isn't capable of more than that all right so that's something to keep in mind that uh is news to me i wasn't really expecting this but you know i'm just sharing with you guys as we go along here okay so anyway let's get back to business that was uh certainly an interesting discovery here uh you know what i want to check one other thing here let's take a look at so looking at the tree that we looked at earlier oops oh forgot hey dash okay so we're looking at these four ssds right there right so let's actually look at those specific devices let me see if i can just pull those out so they are 8086 for intel's uh vendor code and the product code is zero nine five three okay so yeah so they're showing up as non-volatile memory controller by intel pci data center ssd and product is dcp 3600 ssd okay yeah so this is definitely those four devices now what i want to do here is i just want to confirm that they are linking at so we'll look at link status on the ssds themselves so remember these are pci devices and so they should show yep okay so definitely they're also linking uh as expected but i just kind of wanted to verify at uh five giga trans per second so that's definitely pci 2.0 speeds they are getting the full uh by four uh pci lanes on each of those ssds but you know here's the thing i bet that the capacity or the capability is going to be 3.0 right so yeah so you see these guys are actually capable of uh 3.0 speeds you can see the speed capability is eight giga transfers per second so that's pci 3.0 speeds but because of the pci 2.0 switch that we have in here it's negotiating down to pci 2.0 speeds all right so i just want to kind of confirm that everything makes sense although it was a little unexpected but at least everything is connecting the way that makes sense all right so let's get going just as with last time all right let me clear this all right so i'm going to switch over to this screen here and just as i've done in the last videos or last couple of videos i'm going to be running a benchmark on oh so i'm going to run a couple different benchmarks i'm going to do a dd sequential ride dd sequential read with one megabyte uh record sizes so this is just to kind of get a ballpark figure you know are we in the right ballpark or not then we're going to then run a real uh benchmark with io zone so we'll do sequential writes and reads with one megabyte uh one megabyte blocks random writes and reads also with one megabyte blocks and then we will do the 4k over here with sequential right sequential read 4k and then random writes and random reads at 4k as well i'm going to run these benchmarks across four different configurations first we'll do it on a single ssd just to get a baseline of what each ssd can do and we just saw earlier from intel's marketing page uh that it's you know about two gigabytes reads and about half a gigabyte uh rights at least according to intel we'll see what our benchmarks actually demonstrate and then i'm going to take those four ssds i'm going to make them into a raid 0 with software raid and then and then run all those tests again and then finally i'm going to configure those four ssds with zfs in a raid z1 and so this is a little bit different than last time because you know in the previous videos we've had eight devices so it makes more sense to use raid z2 in when you have enough devices but with raid z1 going or sorry with the four ssds going with raid z2 we really wouldn't gain anything you know at that point then you might as well use like like a raid 10 type of scenario you know we're gonna do a raid z1 instead so we'll have one parity disk and three of those ssds is for data in the stripe okay so we'll rerun all those benchmarks uh with this raid z1 configuration across those four nvme ssds all right so that's what we're going to do today and let's go ahead and get started with at least one of them all right so we have uh you know what let's just pick the i'll just pick the first one so let's all right so i'm just going to go ahead and make a file system on that now okay and let's go ahead and mount that all right so we have a roughly 400 gigabyte ssd now mounted under slash data okay so first thing and well by the way in case you didn't watch the second part this machine has 1.5 terabytes of ram but i've booted up with a kernel option where mem equals hg so that there's only eight gigabytes of addressable memory to the kernel so i'm basically limiting the amount of available memory artificially so that these benchmarks when we run them we're not seeing the effects of uh caching and ram and stuff like that so just want to mention that in case you haven't watched the previous videos where i've explained that i didn't really explain that much in this video so i just want to kind of mention that before we get started alright so we have it mounted under slash data let's go ahead progress we'll get a bunch of zeros write it to that file in under slash data one megabyte block sizes and we'll do 10 gigabytes of writes all right so we got about 514 megabytes per second which is consistent with what we saw in intel's marketing material for these ssds alright so at this point i'm going to drop any caches and we'll do a sequential read and we'll send this to dev null again with one megabyte block sizes all right okay so we're seeing about 699 megabytes per second almost 700 so that is quite a bit less than what the intel marketing material is saying so that's kind of interesting because remember the the intel arc sites said that these can do a sequential read of about 2 000 megabytes per second we know we're connecting at pci 2.0 speeds but even at 2.0 speeds with four lanes we should be able to get over a gigabyte for sure and uh probably closer to two gigabytes um maybe with some overhead and whatnot we might not exactly hit two gigabytes but yeah that is definitely quite a bit less than i was expecting for the sequential reads but again this is dd so when we run i o zone let's see what that actually tells us because this is not always completely accurate but it does kind of give us a ballpark figure all right let's go ahead and record this all right so for the i o zone test we're going to slash data and in here we'll run i o zone so i'm going to use 256 megabyte file sizes and because we're going to have 32 threads so that will give us about a total data set of about 8 gigabytes which will exceed the amount of ram that we have here for sure so we're going to use one megabyte records we're going to do dash i zero which is the initial sequential write test dash i1 which is the sequential read test and then dash i2 which will give us the random read and write test and we're going to use 32 threads because i have 32 cpus in the system we'll have a stride size of 64. and i want to record cpu utilization and i will use a direct i o option here so all right let's go ahead and see how this turns out all right guys it looks like the benchmark completed and we're definitely seeing some higher numbers in terms of sequential reads or even random reads here so uh yeah definitely that dd test in the beginning you know was probably bottlenecking maybe at the single thread or something like that but anyway let's go ahead and record the results here all right next up we're going to rerun that test with 4k record sizes all right let's go all right looks like the 4k iozone test is done let's go ahead and record these results all right i just want to take a quick moment to look at the results here so basically the dd test told us we're kind of in the ballpark as far as write speeds the sequential read was not quite what we were expecting based on what we saw in the intel marketing uh or the arc website basically so that could have been a result of a single thread uh being uh the bottleneck here i'm not really sure but anyway like i've always said dd is not a great benchmark but it does kind of tell you whether you're in the ballpark or not but i o zone definitely definitely is is more consistent with what the marketing material was telling us that these ssds could perform so sequential writes on one megabyte record sizes was about 461 megabytes a second these sequential reads was about 1.6 gigabytes per second which is getting closer to that to 2100 megabytes that was mentioned on the arc uh intel website but again keep in mind we may be seeing a bottleneck at the pcie switch here because it only does pci 2.0 okay and then as far as one megabyte random reads and writes we're basically seeing more or less the same performance and that's uh sort of expected because we're just dealing with the same record sizes and ssds you know don't really have a penalty for random rights or random i o basically uh like hard drives do because there's no arm to actuate and stuff like that well sequential reads or at least when we go to the 4k test sorry when we go to the 4k test we see that the performance is for sequential rights is about the same and but sequential reads is we're starting to see about half of that performance so definitely with these much much smaller record sizes i think we're hitting an iops limit on the ssds and so we're hitting about 850 megabytes per second there and likewise on the random 4k uh read and writes we're basically seeing uh similar numbers to the sequential region rights at 4k all right so that gives us a baseline of what the single ssd can do so the next step here is to combine all four of those ssds and put them into raid 0 just to get a kind of theoretical maximum performance of all that we have or basically what we installed today all right so let's go ahead and do that so i'm going to go ahead and create a single partition for all the other nvme drives and then we're going to combine that together with md adm into a raid 0. all right so i've got the partitions created let's go ahead and create that raid 0 as devmd0 we have four devices this will be raid zero zero through three all right so it looks like we have our raid 0 set up and let's go ahead and make a file system on that okay and mount it all right so here we have the 1.5 terabyte raid 0 combination of all four of those ssds under slash data all right so we'll start out with the dd test guys i just want to point out here if you were looking on the screen on this side you'll see that dd was maxing out almost 100 cpu so we're definitely seeing the limitations of a single threaded dd run so uh these numbers obviously are going to be limited compared to what the actual hardware will be able to do so but anyway it's just a ballpark figure let's just go ahead and record this and move on to the actual real benchmark with the i o zone all right so let's actually run the i o zone benchmarks again with one megabyte record sizes and for this test i'm going to use one gigabyte files since we have the entire array of four one megabyte blocks or record sizes and so with 32 threads that's a total working data set of 32 gigabytes we're again we're going to run the sequential write read and random read and write tests all right here we go all right the idle zone test is complete so let's go ahead and record these results but we've got some pretty awesome numbers here and i'll get to it once i record and we'll kind of discuss a little bit all right guys so i got the numbers recorded here and just as i suspected this number with the dd the single thread dd was bottlenecked at the cpu because you can see that with the i o zone and 32 threads uh basically one thread per cpu on the system we are hitting about 6.5 uh almost 6.5 gigabytes per second and so if you look at that compared to the performance of a single ssd it is roughly about four times and so we're scaling almost linearly from the performance of a single uh nvme drive to four nvme drives in raid zero and likewise you can see this similar result in the sequential uh right test so we're looking at this is the one megabyte test four went from 461 to about 1850 so that's again almost quadruple the performance of a single drive so yeah with raid 0 and these ssds we're really seeing a a very linear scale here so that's kind of awesome and as we have seen in the past the random i o on one megabyte blocks basically performs almost the same as the sequential i o on one megabyte blocks we see a slight decrease here on the random reads by about about 100 megabytes per second there but you know that's a really insignificant difference there so all right so let's go ahead and rerun the test with 4k block sizes and see what numbers we get all right here we go [Music] alright guys so we just finished the i o zone test with 4k uh record sizes let's go ahead and record these all right guys done recording all the benchmark results for the raid 0 set and so just want to quickly discuss this like i've said earlier we're seeing a linear scale up when we're talking about one meg by record sizes uh across the board with right and reads and random i o but when we get to the 4k tests these last four here we're seeing that the scale up uh is a little bit hindered and that's probably because we're demanding a lot more iops with those small record sizes and so we're not seeing as great of a throughput as we did with the larger larger one megabyte record sizes and so you'll see in fact uh sequential read is underperforming sequential writes even though the ssd itself a single ssd usually outperforms the uh the write speeds so you'll see for example here on a single ssd we have about uh for 4k sequential reads we have about 854 megabytes per second throughput uh whereas the sequential writes at 4k was 459 so it's a little bit under double in terms of read uh versus right performance but uh we're actually seeing a reverse of that in the raid 0 here for the 4k we're seeing this coming out to about 1234 megabytes per second versus the sequential rights at 4k at 17.71 and if we go over to the random uh rights and random reads at 4k we're seeing basically roughly the same numbers uh that we saw with the sequential rights and sequential reads so again i'm not really sure why the sequential reads with these small record sizes is having such a hit on these ssds in raid 0. not really sure why that is but that's kind of interesting okay so next test is the raid z1 of these four ssds so let's go ahead and do that setup all right so let's do zpool create data raid z1 and we're going to do nvme 0 through 3 and 1. let's see what do we have here okay that's strange looks like it already created something oh okay okay yeah no i totally forgot that the system still has that whole sas three ssd array and that's showing up under slash data so there's already that pool i think yeah so okay so i need to put this somewhere else let's create a slash data too all right let's try this again okay so we have slash data2 where we have the 1.1 terabytes of the four 400 gigabyte ssds in a raid z1 configuration so let's take a look at that yep so now we are ready to go let's go into the slash data tube and we'll start out with the dd test all right let's go well hold on a second that was not right i'm writing to the wrong place here so let me revise that all right let's try again okay so over here again i saw dd maxing out at 100 cpu so um i don't think we were able to fully exploit the capabilities of these ssds even in this configuration so anyway i will go ahead and record this all right now on to i o zone all right so we're gonna re-run the i o zone with a one gigabyte file per thread with a total of 32 gigabytes of data we're using one megabyte record sizes all right let's go all right i'll zone is done let's go ahead and record the results all right next up let's rerun that test with 4k record sizes all right here you go [Music] all right guys i just remembered that we're not going to run the random i o at 4k uh because zfs is you know we're just using default settings so it's 128 kilobyte record sizes and with 4k we're just going to be putting a lot of wear on the ssd so i'm going to stop the i o zone uh run for the random i o test because we're just not going to do that but i'm going to go ahead and record the sequential read and writes for 4k all right all right guys so so that's the final benchmark that we're gonna run today and just to take a quick look under the one megabyte record size tests we're seeing that the scale up is obviously not going to be linear first we have parity and second i think there's just overhead with zfs as before i find that zfs uh sequential reads or even random reads don't perform very well in terms of scaling up we know that these four ssds can scale up on sequential reads up to over six gigabytes per second almost 6.5 gigabytes per second uh with zfs raid z1 even though we're dedicating you know one drive to parity we're seeing basically the performance level of a single ssd so yeah so that's that doesn't seem great the sequential writes scale up a little bit more a little bit better i think almost linearly actually if you consider that uh these are four ssds one of them is parity so let's just say times three so if you take the 461 times three it's you know pretty close to that 1308 number and similarly yeah so on rights we seem to be scaling up fairly well on the reads uh zfs is just not scaling very well at all here anyway perhaps some tuning might help there i'm not really sure but let if we look at these sequential writes on the 4k read and write test we're seeing a better performance on the sequential read 4k that is compared to the sequential read uh with one megabyte yeah so with 4k somehow that's performing better uh with the sequential writes at 4k records it's yeah it's a little bit slower than you know this but uh but that's to be expected since we're using much smaller block sizes so there's a higher iops load on those ssds but in general um yeah the reads are not scaling very well but you know this is so let's actually compare this to it's not a really great comparison because the other uh setup in the previous videos we had eight ssds involved but if you look at this uh yeah let's look at 4k first so we're looking about 900 to about 2000 and this is uh a little bit better just a tiny little bit better so yeah unfortunately i don't think we're really able to exploit the full capability of these nvme ssds with zfs here i mean if this was if this was the targeted performance i o pattern that i'm going for like 4k sequential region rights i think i would get more out of my money by going with the sas 3 route even though it's a slightly less performance here i think that setup probably well requires a little bit less components you know so uh let's look at the large record size benchmarks here so sequential rights and reads and one megabytes is looking about 1600 to 1700 so 1600 to 1700 uh yeah so with the sas 3 setup we actually got better performance and then we we're doing here now granted again there's not a completely fair comparison because with the sas 3 setup we have eight ssds in play here with raid z2 uh versus this is four ssds and this is you know that's the most we can put in uh with this upgrade kit so we're kind of limited to four ssds nvme ssds but you know put another way i guess you could also compare look at it as you're getting roughly the same performance level i mean this is actually a little bit higher than the nv nvme scenario but you're getting roughly the same performance level with half the number of ssds so you know maybe that's a consideration maybe that's a factor in there i don't know i haven't looked at the price comparison between those sas three and and versus the nvme um but you know we're not seeing anything spectacular uh at least in the raid um the zfs scenarios here now if we look at just the pure uh you know raid 0 scenario we're definitely seeing some great performance boosts here now also keep in mind these p3600s are not great for uh writes right performance so there is that whereas the hitachi sas 3 ssds that we use tend to perform fairly well with writes so if you go back here you'll see that we're getting about almost 650 megabytes per second right per uh ssd whereas these nvme ssds are limited to somewhere under 500 okay so again you know these we're not really trying to do an apple's apple comparison but i'm just kind of showing you the different types of scenarios uh the different types of upgrade paths that are available in the r820 and ra 720 servers and you know what type of performance you can kind of expect now the sequential rights here obviously if you use a better ssd that has better uh sequential write performance like again we're up to about two gigabytes per second uh you know this actually might scale up a lot better so i think the p 3700 is a better um uh nvme ssd for sequential rights or just overall write performance you know so if we were using the p 3700 instead we'd probably see much better numbers but uh just looking at the sequential read number you can see this is 6000 over 6400 with four ssds for nvmes whereas this is 5 300 with eight sas three ssds so obviously the nvme completely outperforms the sas three setup uh with half the number of ssds okay so uh so there's definitely some headroom here depending on the ssd you choose to use now keep in mind again with the pci 2.0 limit you don't want to get something that's going to perform uh greater than the available pcie bandwidth but if you got something that's a little bit more performant than this p3600 especially on the right uh performance side i think you could outperform the sas 3 configuration that we set up from last time all right so you know somebody in the comments um in the last video i think mentioned you know what if you just use software raid 5 or raid 6. as i'm kind of discussing these results here i'm a little bit curious because obviously we're seeing some sort of bottleneck with red uh zfs red z here and i'm just kind of wondering also in my mind what kind of performance could we get if we were to use instead of zfs raid z just md adm software raid uh five with these four ssd so you know what okay this was a little bit unplanned but um let's go ahead and try that let's go ahead and tear down the zfs pool there and uh set it up we'll set up these four ssds um we're gonna go a raid five okay so kind of the equivalent of rain z1 yeah let's go ahead and try that out so sorry if this makes this video a little bit longer than i uh anticipated but i'm just really curious you know um i by the way i do read all your comments and and i thought that comment was pretty uh interesting i i do think it would be kind of interesting to compare how this would perform with with parody raid but not in the zfs variety of it okay so let's go ahead and try that out so first thing is i need to destroy if i can spell destroy correctly this pool here all right so i'm going to go ahead and just blow away all of this stuff here so all the partitions that zfs created i'll do that with just i'll just recreate the gpd label all right i've got that done now let's go ahead now and create a single partition across each of those uh ssds all right so with that i'm going to then create the raid 5. so let's see if i remember how to do this create dev md 0 uh raid devices is going to be four devices and we're going to do raid 5 this time zero through three and one p1 why do i have resource busy here oh i never stopped okay um i don't know i forgot to stop the software raid zero and i'm already using yeah these drives in other things so i don't know let's see what happens there okay let's see if this works yeah i should be a little bit more careful and actually stop the software raids before i reutilize these things okay yeah looks like we have a raid 5 running under devmd0 so let's go ahead and make that file system all right and then i will mount it all right so now we have 1.1 terabytes here uh of a raid 5 software raid 5 of those four ssds under slash data 2. all right so let's go ahead and run that dd test all right so let's go ahead and record this okay so we're seeing that 2150 number again i think that's the single thread limitation of dd uh i did see the seep utilization for dd max out again at 100 so uh but anyway let's go ahead and now run the i o zone test on this all right so this is going to be the same idle zone one gigabyte file sizes 32 threads for a total of 32 gigabytes of uh data set and a record size of one megabytes we're going to do the sequential write read and random read and writes with 32 threads all right let's go hey guys so normally i would just uh fast forward through this section of the video but i wanted to point out something here we're looking at the single process here md0 raid 5 maxing out the cpu 100 percent and if you look at the iostat here we're really not able to push the performance uh so this is i believe initially it's a sequential write so there's some reading going on here i'm wondering if there's some sort of uh read modified right uh penalty that's going on like i i only used the default settings so yeah i'm not really sure i don't know why there's a lot of reads going on here but it looks like this might be a read modified right type of bottleneck happening and we're really not seeing the the great performance out of these ssds we know the ssds can do much more than about 100 megabytes per second here but i think that's because this process that is probably involved in calculating the raid parity is maxing out at 100 and so so there might be that involved there might be some read modified right uh amplification happening here too and so this is yeah this is not going to turn out well i can already tell all right guys so this has been running for a little while now and i'm just gonna break it and stop it i'm not exactly sure what's going on uh perhaps some tweaking isn't uh is required here to optimize this raid 5 to work better with these ssds but as you saw earlier the performance was pretty bad so i'm just not going to continue running this and put extra wear on my ssds just to you know get a final number here because the final number is going to be worse than what we had with raid z5 or sorry with red z1 anyway and so i'm just going to stop this sorry i've diverted it to this i thought it was you know be kind of curious to see uh how raid 5 software raid 5 would perform here but you know i there was something strange going on as i mentioned earlier there was a lot of reads going on versus writes so that's unexpected when we're just doing a sequential uh right here so i'm not really sure why that was happening and there might be more involved here but i don't want to make this video you know super long here this was this part was already kind of unplanned but i just thought you know i'd try it out but anyway i think we're going to end it here all right guys uh just going back to looking at the benchmarks yeah that raid raid 5 thing somehow just wasn't working out and so i'm not going to continue running that and putting extra wear on my ssds just you know to get a number that obviously is not really going to be very valuable you know if if we can figure out uh what tweaks need to be done with that software raid 5 to make that work a little bit better i might revisit this but definitely we were already looking at you know less than four well yeah no actually it was worse than that i was like about 180 190 so less than 200 megabytes per second uh in sequential writes which is uh you know four for four ssds and raid five it's performing worse than a single ssd so yeah anyway i'm just not gonna do that but anywho uh guys so this is the last video of the series hopefully these videos have helped you kind of open your eyes to some new upgrade options for your r820 or r720 server i wanted to kind of demonstrate to you guys what all the different possibilities are some of them obviously not officially uh dell supported like the first two videos this nvme upgrade kit is an official dell supported uh upgrade kit that you can install in the r820 or r720 now you do have to have the models that have the two and a half inch bays in order to do this upgrade the models with the three and a half inch bays will not be able to take this type of upgrade i hope you guys found this useful uh sorry about that raid 5 benchmark that was just a little diversion unplanned and un totally unsuccessful but anyway just wanted to share that with you guys if you enjoyed these videos make sure to give me that thumbs up and if you like this type of content be sure to subscribe to my channel if you're new to my channel also if you'd like to support my channel check out my ebay store i've got all sorts of server goodies for you there and i'll leave a link to my ebay store down in the video description below so go ahead and check those out alright guys thank you very much for watching and have a great day bye bye

Channel: Art of Server

Views: 18,929

Keywords: PowerEdge R820, PowerEdge R720, storage upgrade, NVMe upgrade, U.2 upgrade

Id: mtC1_YKbs3M

Length: 76min 41sec (4601 seconds)

Published: Fri Nov 06 2020