ISCSI vs NFS Performance Comparison Using FreeNAS and XCP-NG Xenserver

all right so we're gonna cover a topic here that seems to be a lot of debate and it related to storage planning so we have a freenas server here 11 to rc1 running an older AMD Opteron 61 72 12 core processor now we're gonna do I scuzzy versus NFS and I know there's better machines out there there's Enterprise machines Tom did you test on this machine I wish I had time maybe if someone throws enough money at me I will test it on more machines or if they throw machines at me and money so I have all of it together more than happy to do it my important part about this test is for consistency we're gonna be testing it on exactly this system so the NFS on this system and the ice cuz the end this system so we have at least based on the same pieces of hardware and it should scale if you have better hardware the performance results should scale upwards for both devices or both different formats I mean of NFS or I scuzzy for how you want to present to the storage to a hypervisor and you should be able to kind of extrapolate some results a couple things we want to cover here is all the details this has 20 gigs of ram in it it's running the release candidate which I found very stable I know the new build comes out in only a few weeks but I decided to run it with this one I've tested it with the 11.1 that we have and my results overall are the same generally speaking guys because he's faster but we'll get into the details shortly here so let's talk about first how are the disk setup because that's an important aspect of this NFS benefits greatly from having a Zil drive so we go over here and we have these scuzzy and the NFS discusses as evil block storage data set so standard files we're gonna look at the status it's just a raid Z with three Western digital black one terabytes it's what I had handy had three of them I said four one was bad these are some used tries but they all tested good except for the one that's not in here no more then we have our Zil here which is an SSD no it's not an Intel obtained or anything amazing it's just a standard SanDisk SSD I had laying around so that's on there for these ills so if you're not familiar with how Zil works there's a great write-up that I'll leave a link to on the FreeNAS page below that has an explanation on how Zil works with CFS it's really interesting it's not exactly a right cache like people would call it it's the intention log cache and it does help a lot especially with ZFS and we're gonna show some of that in the testing so now that you kind of get an idea there we're gonna look at the services we have running now I have it set up with NFS there was 12 cores to this machine so we have 12 cores 12 servers set up with NFS what this is is it's allowed to spawn 12 of them were using NSF v4 is terms as the mount down here I didn't really see any test speed differences or apparently there's some minor deviations between NS nfsv4 and NFS v3 back to services for the I scuzzy it's pretty much default all the way across there's no passwords on it it's just basically set up pretty simple one extent the only checkbox that a default would be the fact that this is Xen and this year compatibility mode has been checked other than that everything else is just the defaults on this and let's talk about network connectivity and interfaces there is eight head and gig fiber connect between these so this is statically set to 192 one six eight ten with no Gateway because this is set up to be a direct connect between the server running Zen the latest version and the freenas so these are 10 gigabit in between so we don't have a uh bottlenecks in terms of networking when we're presenting this so let's go back over here to look at the XenServer we have debian on scuzzy and debian FNS an F s look at the host storage I just kept them really simple this one's called scuzzy this one's called NFS once mounted and a fast once mounted VI scuzzy so really straightforward here and there like I said they're mounted at 10 gigabit between them this is the storage network 192.168.1.0 switch in between it doesn't even have any problem so I've done everything I can to optimize the performance between my head and unit of running XC PNG server and the freeness box in shredder them in any type of i/o bottlenecks so they're on equal platform both of them are mapped over this both the NFS mount and the I scuzzy mount our mount across this ten gate link so that is an important distinction just so we're all on the same page on these and lastly I'm using the four Onix test suite the open sourced wet test suite for benchmarking so this is what we'll be using to benchmark this so all the results are completely reproducible by you this is a free download if you wanted to use it all right and then we have here net data which is built into FreeNAS since 11.1 it's a great visual way and it's really pretty way to look at all the results kind of in real time so we're looking at some of the services running on this and what it looks like from a performance standpoint and we'll also be looking at some of the ZFS backend on here all right let's get started on the testings that's where the fun part is all right I have this one called scuzzy on debian and a fest debian I mean the naming as clear as possible so tests on this one of course sure on the scuzzy one that's on the other one or any NFS one other thing to note because someone always says hey Tom how'd you get your shell to look like that they'll be linked below to my github where you can get the shell to look just like this that the parameters I have put in here for the shell they're free okay let's go ahead and run the test so it's the for our next to suite benchmark and I dispel benchmark properly there we go we'll go ahead and test all options for so we're gonna test for kilobyte 64 kilobyte and 1 Meg will test with a 512 megabyte test file if I had time I'd do all of them if there's interest maybe I'll run more tests later that are more extensive and then we're gonna choose 3 tests all options will save the results because I'll leave links so you can view these results and stare at them in detail if that pleases ya and we're gonna give it this name here as you can tell I've been doing some of the tests on what I use tests things before I do the videos to make sure that there we get the idea of how it's going so we're going ahead and do this as the test this is just I scuzzy all the defaults so I guess I could put defaults so no optimization in here we'll leave that as the same and we'll go ahead and let this run now while it's running a couple things we'll pull up is we'll look at how the rights are actually being written to the drive so I just village in cz pool IO stat - - v1 was in refresh every one second and what does show you real quick is the data being written if there is any to the log file which there's none because it's doing the read test if I had to guess right now yep still running the read test so there's not going to be a lot written to the Zil because it's not a write test we're gonna see this flip around when it comes to writing you'll see a lot more in there now while this is running over here so let's move it over here let it do its magic and we can see right away here's the CPU usage so you can see this bikes in CPU usage what we're doing this now we've jumped up to the right performance testing so same thing kind of peach between each one of them while it rates out the files now this is also when we go over here there's still not too much going on in the Zil and it's part of the way and I don't have the best not working knowledge of exactly how the protocol is written but it can it's the way it synchronizes the writes with I scuzzy it doesn't seem to be as ill intensive so when we do these tests we're going to do the test because NFS is Zil intensive on a ZFS filesystem we'll do the testing on that and it does make a performance difference but we're not going to bother testing the I scuzzy again with and without the Zil because it dilly doesn't seem to make much of a difference on there at least from the testing I've done I'm also always open to doing these tests over again if someone tells me I did something wrong or has a twist on it that I completely missed or some optimization setting so please let me know in the comments below I do you read all the comments so I can learn more and we can all learn more together cuz I like doing follow-up videos on this ok it is done let's go ahead and save the testing results so yes yes upload it all you can have all the logs and everything on there and this this link you don't have to try and copy it I will paste it below as well this is where the results page will end up so from a reading standpoint until the files start exhausting the cache from a read standpoint you get some really crazy high reads that are really really fast touchy with the small file sizes and then the writes are a little bit more sane when you do the rate performance testing 512 so about 409 here so okay this is the base test for this one here let's go ahead and move over to the NFS world so we ran this right here so that's the test that was run copy paced we're gonna do four just like you did last time test all options one five twelve so and then three test all options yes we give the same results file name and FS default options alright it's gonna take a little longer to run and you're gonna notice right away that's because it just doesn't have the rewrite performance so let's look at what it's doing though so we look over here while these tests are running so we see our CPU is peaking up here in the twenty six twenty eight twenty nine percent of CPU 13% CPU so it's just not hitting the CPUs hard look over here at the ZFS whoops ZFS system here was those tests run over here so ZFS reads versus just not peeking as much when you're over here you're not getting the thorough put that you were getting with their I try say that word right someone said I said it wrong I don't want to say things wrong but that does happen Thoreau put through through throughput throughput I think is as you can tell you if you didn't know I read more than I talk although I do talk a lot in videos ZFS I ops so we seem to actually have more on AI ops here alright let's go look at the results so we have really good so far read results are up here at five thousand five nine five Oh which obviously we all compare more but we're we're looking up here at six oh four seven and 362 so it's still a lot higher but not substantially but when we get to the right test that's where you're really going to see the differences because of the way the synchronization is from reading and writing with NFS being a lot different we're going to show the optimizations on how to change that so let's go back up here to the top so here's what the retest now we're into the right testing and we're using even less CPU now what's happening here - let's look over here now please note the log file which was barely in use before is really being used quite a bit so we constantly have all this data flushing in and out of the log the Zil system because that's NFS is very dependent and that's why we have this in here didn't trust me when we remove till the performance just goes down to nothing when you're doing it especially when you're doing rate testing it just it goes away so Zil is an important aspect to this so see how the tests are going almost done oops oh god I thought almost Suns got a few more to go it does take that much longer compared to that so the the i/o performance you can already probably summize what we're going with this I'm going to fast forward for a second here but you can tell that we're just not seeing the same i/o performance all right in our testing is complete and it's sad yep 37 Meg's so yes we're gonna save the results all right let's look at them real quick here copy link NFS default options so yeah they're bad so we're sitting here looking at 37 verses let's scroll on the bottom of this one here four hundred Meg's 37 Meg's okay so obviously these are really bad and let's talk about why they're really bad and what can be done about this so NFS because of the sinking issue on writes ZFS tries to sink every single right so how do you get around that well the solution is not amazing but it is kind of a workaround so let's talk about what that is so we're gonna go here and close this and we're gonna go ahead and set so it is we're gonna set these sinking to be off now corruption and missing data are two different things so you may see people say don't set sinking off because you have the potential to lose data but ZFS is a copy-on-write file system so that means it never puts another piece of data on there until their copy is made so you can't end up with corrupted data but you will lose data in flight so data loss is definitely a potential with this so by not committing the sinks in the same way and you had a power loss you could potentially lose some of that data in flight more so than if this sinking was enabled so right now syncing is enabled and we're gonna go ahead and set it to off it's pretty easy to do CFS set sink equals disabled tank /nf oops header a first time the NFS that turns off the sinking now you don't need to redo anything matter of fact the VMS are still running you can do this while the VMS are running and we're gonna run the same exact test again with syncing turned off so first ones are not default now we'll run the same test again but sinkings turned off so it's gonna go way faster so option four one three yes and I fess with z-pass sync off I disabled is actually correct term so call to save do everything else could stay the same and we'll just go ahead and run it to test again now go here and look at the i/o stat and you know we don't see it hand well we got to get to the right test and we'll see if it hammers the Zil port partition as much that went fast we're still in the read performance because the sinking is a write problem not a read problem all right so we're not committing all these in the same interval so we're not seeing as much high usage we did before of the log with the benchmark keep running benchmarks already running way faster and we're seeing higher CPU usage because of high or third put of it so we do see right away some results and did it finish running now it's still running a couple more still running some more tests but they're going so much faster you can already start seeing these high results on this sure on to the last test now all right testings done we'll go ahead and see the results yes yes yes copy link and here's the results NFS default options NFS with ZFS sync disabled so when it comes to the read performance we're here not a big difference right performance it's like two completely different systems here so we're getting up here about 360 375 read performance again and final write performance on here so dramatically all we did was NFS with sync disabled to be able to do that now let's go over here and compare it to we're still not seeing the same performance we're at 400 here 404 14 versus the best rate we got was 384 here yeah 375 361 the right performance just isn't the same between us was really close so we are we are approaching close on here now the last test I'm going to do relate this is still on the NFS I like I said we've just left ice cuz he had all the defaults and not seen a big variation so we're gonna go over here to our free Ness box go over to the pools storage pools status and one of the magical things with ZFS is let's go ahead and make the Zil go away and by the way you can do this without rebooting the vm so I'm leaving them running and without rebooting the servers or anything so if you want to add or remove a log or a cache you can discover here and remove or replace it if one's going bad as part of the beauty of systems you have to take them down because taking system's down is oh that's not fun alright we no longer have a log so if we go over here see Pouliot stat there's just the tank raid z1 there's no more log on there I don't leave it running VM is still running so we're gonna just run this test again the only thing we have we still have sink disabled but we're also knows ill drive on this right now so for one three yes this is NFS sink and no Zil alright and we'll let this test friend know and see what the differences are without the SSDs LLL drive in the mix Katie tests completed yes yes yes and not having a Zil not much of difference I see here about the right reading performance but writing performance puzzling me because everything I read says definitely have a Zil went up a little bit so I found that actually kind of puzzling here so NFC's knows ill so we've seen a little bit of a performance bump in there so we're actually now just about on par with our I scuzzy implementation so it's really close here so now that we have I scuzzy maybe this is a problem with my zeal drive that I'm using and this is all in the same array so now we're gonna run the I scuzzy one again we're not changing a sync option the only thing different between us is that we remove the Zil drive from us this is the final retest one we're gonna do here so we're gonna head and and do a test all options one three yes no Zil drive just to see if maybe there's something up on my system see if it makes any difference to the I scuzzy at the zil's missing all right results are done let's save them and see the comparison so reads the same writes the same I scuzzy seems not to care if there's a cell we heavy actually the slightest drop but I'm gonna call that enough with some deviation cuz we're seeing if I run these results a couple percent deviation and it's noted in the benchmark that sometimes it does that but we do see an interesting result difference with these ill there or not there when it comes to the testing with NFS so I find it kind of interesting we do seem to get a little bit performance this goes against my understanding of having as L on there but this is done with ZFS having these synching disabled because NFS and ZFS fight with each other when it comes to sink sinking now like I said there is some data risks that come with this as in terms of you lose data in flight that's uncommitted because it's asynchronous that's a deciding factor up to you it's not the same as data corruption but it is a result that you'll will have to contend with if there's a problem and I don't have an Enterprise Zell to test if that would make a difference you know Intel obtains and some of the other Zeus cache and a few other companies make devices that are designed exclusively for this for really high end but this has been my results for NFS versus I scuzzy I'm still leaning towards and this is why we provision I scuzzy it performs consistently better I do know that the thin provisioning thing is a big deal of some people and I can very briefly touch on that because that'll be the final little things that's the other deciding factors but you can't thin provision it's interesting because and we'll show this here let me pull up the ZFS information so if we go here and just so you know the command ZFS get all for the tank the NFS and we look through here used available for a compression ratio 4.5 X so I'm showing this because these are the same VMs copied on here so we see that ZFS is able to very much compress this because well there's a lot of just dead space inside of here so you have first thin provisioning handled by the hypervisor and then you also have the compression on ZFS you can get some storage savings which of course can in translate into performance and storage savings so used by data set is only two point three two gigs now then server does not support thin provisioning over I scuzzy so we're gonna get all tank and scuzzy and we see something different we see the same high compression ratio that we had on the other one so we're definitely compressing a lot of it and the same thing it's you're not using as much data on here so when you look at the compress used by data set factor so we'll do this one right here a matter of fact let's just do this we are seeing that scuzzy uses just a little bit more verses here about the way it provisions inside of the data set now you get members eat though easy if s works because it's providing the I scuzzy targets as AZ vial block storage versus a NFS which is file level storage so it's going to have a different understanding because you're looking at files so if we do an LS slash mount tanks then FS whoops we can see the VHD file here and what it thinks it's provisioned is 11 gigs even though we've seen it's only using 2.3 to because of the compression of what's actually in there so it gets kind of interesting now the last thing we'll cover on this directly it it is let's duplicate a few things so let's go ahead and we'll stop each of these so stop stop we're stopping them so we can fast clone them so we'll go ahead and fast clone let's make one too so now we should have a couple of them so we've got a couple of these and let's go ahead and fire these up whoops so they've actually done something yep start these three vm's and we're going to go over here and we'll clone a couple of these all right so now we're spinning up all these ones here so they're all booting and doing something and when we look at the storage here's those drives not thin provisioned so there's no compression anything on there and we're gonna go ahead look at the storage look at the NFS ones and because it's thin provisioning with a base copy and then showing this here so it's a little bit works a bit different but when you look at the disks here's all the different disks but they're all forked off each other but let's talk about what happens behind the scenes inside the ZFS while these are all running so here's all of them there's one two three here well you know what we need to clone one more so let's go ahead and stop and start and clone there we go now we have the equal number of them all the way around all booted up and running look at the storage okay everything's up and running look at the stats yep we pinned the CPU spinning up all those VMs and it seems to clone the same when you're doing this I may be slightly faster in NFS I'm not sure but let's take a look at what's going on behind the scenes now here's all those VH DS and it's only tracking the differentials between them so whenever you do either a clone or create a snapshot you have your base which is this one here and then we have all these individual ones here which is only the differential between the other ones hence the thin provisioning so they're not taken up even though each of these hard drives are potentially 16 gigs they're not really taking up much space on the actual hard drive so get by use by day sets we still we're up here we were at 2.3 to we're barely using anymore as we got what fifty hundred two hundred more Meg's of spinning those up but that's also the same here used by data set 3.09 gigs so if I spun them all up and it's not then provisioning but it's still only using all of that well that's comes back to this all whoops and it's the compression ratios so we keep getting more and more compression going because it's really just seeing a bunch of duplications so you're still getting a lot of efficiencies because ZFS looks at it and goes this is all a bunch of duplicates but XenServer because it doesn't support thin provisioning isn't seeing it that way so it's gonna report differently so just take for this knowledge which you will I just wanted to show how that works so we are going to run out of space quickly here because we have you know the base copies and each one not thin provisioned all the way out here but when ZFS handles it on the back end we get these really high compression ratios and so it still works you still have a lot of efficiencies and yeah it's kind of novel I just want to point that out for those of you that say but I need the thin person maybe you do maybe that's a reason you absolutely want to do this with NFS so it's our compression ratio here we actually have worse compression now so I think let me show you looking at the right numbers here okay let's just get the compression ratio and compare them that one's at that and then scuzzy is at 5 2 X interesting Lee is compressing scuzzy more so I don't know why I mean like I said these are the same VM we duplicated the same number of VMs and we are some reason saying there so these are some of the underlying things I don't have in ton of answers but I'm sure someone a lot smarter me maybe has a link to a great article that can tell me some of the other things I may have missed or if there's other things that I did completely wrong but I will leave a link to the test results just so they're all laid out here for you and how the system is set up and like I said I covered all that but if there's something I missed that will be a part to this video with something blatant that I just setup wrong and you're like and you find a reason to NFS should perform substantially better than I scuzzy provisioning because I didn't do a setting let me know and I will make a follow-up video to this Thanks and hopefully this was insightful I know I did some learning today thanks for watching if you like this video go ahead and click the thumbs up leave us some feedback below to let us know any details what you liked and didn't like as well because we love hearing a feedback or if you just want to say thanks leave a comment if you wanted to be notified of new videos as they come out go ahead and to subscribe and the bell icon that lets YouTube know that you're interested in notifications hopefully they send them as we've learned with YouTube anyways if you want to contract us for consulting services you go ahead and hit Lauren systems comm and you can reach out to us for all the projects that we can do and help you we work with a lot of small businesses IT companies even some large companies and you can farm different work out to us or just hire us as a consultant to help design your network also if you want to help the channel other ways we have a patreon we have affiliate links you'll find them in a description you'll also find recommendations to other affiliate links and things you can sign up for on Laurens systems comm once again thanks for watching and I'll see you in the next video

Channel: Lawrence Systems

Views: 42,531

Rating: 4.9459457 out of 5

Keywords: xen server, ISCSI vs NFS Performance, iscsi, nas, storage, san, iscsi vs nfs, nfs, freenas performance, freenas iscsi, xen server setup, xen orchestra, xenserver 7.5, network attached storage review, network attached storage setup

Id: OAe3_8qoCWo

Length: 33min 0sec (1980 seconds)

Published: Tue Oct 30 2018