FreeNAS Virtual Machine Storage Performance Comparison using, SLOG/ ZIL Sync Writes with NFS & iSCSI

so let's dive into talking about storage I scuzzy NFS in virtualization so getting performance out of this there's a lot of factors involved so I wanted to talk about a few of them because one of the first things people run into when they start doing things like setting up an NFS share and generally NFS becomes a very popular and is not a bad way to go but you right away run into some of these issues and questions and test configurations and diving deep into this so we'll start with this article here and by the way I'm going to leave the link to all three these articles it's up to you if you want to read them before or after you watch this video but they're going to have more depth on some of the really specific details inside of here and I may or may not cover some of these I want to show you some of the practical side of the knowledge I gained from reading all of these articles so we'll jump right to the summary NFS by default will implement sync rights as requested by ESXi clients now you can insert ESXi proxmox whatever your hypervisor is and my case is going to be XC PNG but citrix you could fit that one in there as well it's the way the synchronize rights hack with that happen with NFS this is one of the reasons that I scuzzy can appear faster well it technically is faster but just comes back down to that whole integrity versus speed and what is an acceptable amount of risk when it comes to your data so the options are doing NFS with sync disabled we're going to test out that some people use sync disabled none investigate speed this causes asynchronous rates your VM and yes it is lightning fast however in addition to turning off sync rights for the VM data it turns off sync rights for the V ZFS metadata so this actually doesn't just affect the VMS as in the connection between the VM and where you're storing it and it's also is turning off syncing for the ZFS metadata or the Zil so this may be hazard wealth your VMs and the integrity of that particular CFS pool so those are risks and we're talking about catastrophic failure like you know absolute power loss or drives failing while data is in flight I scuzzy by default does not implement sync writes as such it often appears to be much faster therefore much better choice in NFS however your VM data is being written asynchronously which is hazards to your VMs on the other hand ZFS filesystem pool metadata is being written synchronously which is a good thing so we're gonna explore how that works in practice this also means you probably know where to go if you refers to by an SSD slogged free device and are ok with some rush to your VMs so it is probably a good way to go but you do require a slog device now I'm not going to jump into the entire article of this but this is the Xia fazil and slog demystified it is not a cache it is the ZFS intent log or slog is where they call it and while using a spinning hard disk force log will yield performance benefits by reducing duplicate writes to the same disk it is poor use of a hard drive given the small size but high frequency of the incoming data and what they're saying is you really want to use SSDs for the slog or in bring this up to 2019 terms because this is from 2015 it's not just solid-state let's go all the way to say things like m vm e and of course when you get into the higher end enterprise equipment this goes even further they make special eyes devices that essentially operate more of like ram speed in order to do this intent log and this is where there's a lot of reading writing going on to get the right synchronous and this is very relevant because we're talking about how vm sync so how the data stored in a virtual machine is written there this adds a lot of complexity and because of the way a file system that has been virtualized and then running instead of vm changes there's a ton of little writes going on it's not like normal file writes where I'm just copying a file over to a server I changed one file out of 10,000 in the OS is handling it when you use a share for example on pre-announced and ZFS alright so let's jump into the hardware setup and how this is actually configured first this was a freelance mini xl+ that is on loan from the folks at I act systems that we're using for demonstration it is hooked up at 10 gig this server is going to be this box here so we'll go over here to hosts for Polar's wondering this is a Dell PowerEdge are 720 XD provided by the folks over at tech supplied direct we have an offer code with them if you want to check it out below they get you 10% off purchases of servers workstations or anything you want to buy from their website they have some excellent deals we've bought servers from them and this one was provided kind of a sponsorship for the channel also I really want to thank them and if you likes if you want 10% off we have an affiliate link below they get you 10% off any purchases on their server equipment alright let's jump into how this system is configured in more detail so this free dance mini XL is connected via rj45 at 10 gig I have a little review of this box and I'll leave review link to it that shows how it works and all the little you know hardware details of it it's running an Intel Atom C 37:58 at 2.2 gigahertz and these are all SSDs that we have for the discs in use so everything we're gonna be doing is 100% test with SSD drives they're just some sand disk drives or 240s or nothing high-performance or nothing really high-end their consumer drives like C P and G also connected at 10 gigs so the storage is connected at 10 gigs the EC rows connected at 10 gigs were actually running everything over just the East 0 but it is a tanky connection and I set it up here for demonstration just so people know no magic here I'm not that good at editing to create a bunch of lies you just have to take my word for it yes it talks at full 10 gigs 9.41 gigs a second no problems there but that's all that for those wondering how I'm splitting the screen because that always comes up first and you can see already been doing some testing here I will clear all this this is T MUX and I've just split a screen so we can have each one of the VMS I log into here and up top so we know we're running here this is low on the FreeNAS and we're gonna this file right here all we're doing is running zpool io stat - v4 SSD tank that's the name of the tank on there update every 1 second so that's just so we have some information rolling by so we can see these reads and writes happening go back already here and we have and we'll go ahead and both of these are started up so we'll go ahead and log into them like ice had already been running a lot of tests will jump into all the details of the results and everything else here so this one the Debian lab on NFS pre-announcement EXL plus log in to that one here and then we'll go over here get the IP address of this one which is one five eight and we'll log into that one here oops okay both of these ready and to make things as clear as possible when I'm doing these demonstrations this one says Debian free nasty XL plus NFS this one says Debbie and free and ask mini XL plus I scuzzy and to show you the pools the way the pools are set up here is the scuzzy one set up his eval so it set up his block storage through ice cozy here is the SSD test right here this is set up as an NFS so this one's I scuzzy this is the NSS test I labeled SSD test and forgot to add it I guess I can always edit the comments on there but it you get the idea this is this guy Z 1 this is the NFS one they're both running on the same disk pool so we're eliminating all the different variables we're seeing directly because it's the same FreeNAS box it's the same disk pool we're seeing how this is all set up other details that are noteworthy so we're gonna go here and edit options and like we mentioned we have sync disabled this is the higher risk but better performance when you're running in an effect share now with the I scuzzy here we're gonna go ahead and whoops edit the XIV ow it's standard sink so nothing special that's just the default because by default with ice cos you created as eval it's gonna sink inherited based on the data set that date is set by default ones to sink so that's all set to be normal as far as that goes so that should give us the best performance in a fast nice what we're going to start what kind of performance do we get out of this NFS system so you over here what you're seeing at the top here is the four drives and I have a cache drive in here I love the cache drive in because a lot of people think you can get a lot more performance when you're dealing with virtual machines out of a caching drive this is not the same as a Zil but the read cache and that's kind of limited because it's better to put more memory in a system because it will pull read it from the memory this is one of the things that ZFS does really well is push a lot of people think ZFS is memory hog but it's actually pushing all the constantly read data into memory so it does benefit in performance for them having a lot of memory but I went ahead and threw this in here and you can see how much it's being used even with these running use these little writes happening here in Aaron it's just not caching up but I'll leave it up there so you can see what's going on first thing we're gonna do and I'll show you of course how this works speed tests we're just gonna create a file and delete it I think that create a two gig file so what kind of speed do we get out of this 949 Meg's a second so run it again 975 so you can pretty fast speed out of these four drives now this is raid-z one if you didn't notice up here so in case you're wondering and like I said in those articles that I'm gonna leave links to there's breakdowns of the performance difference in z1 z2 etc and spreading the rights across more drives so now we're going to run the speed test over on the I scuzzy here what kind of performance are we going to get out of this 731 now pay I will run it again he run him a few times and it's pushing the data writing back 787 so cool once again cash had nothing to do with all this doing is read and writing a simple drive we're gonna get something more complicated here next and we're gonna run the veronik styles own test we're gonna do I will say 64 kilobit alright so option two option two for writing a two gig of data and one and let's see what the right performance is on this so you're gonna watch this up here happen so we were seeing all the drives spin up and all the writes occurring over here and it's running the tests let this testcomplete all right this was able to complete it looks like it's uh 694 alright so in the same test over here options option two option two great performance and the reason we're doing right performance is cuz read performance gives you ridiculously big numbers because of the caching so this tool doesn't do a good job of testing read performance because it comes up with some really artificial numbers that are unrealistic for the actual speed of the drive all right now that completed so we have 694 over here with NFS and we have 530 here with iceskating so you're probably thinking Oh NFS all the way definitely a much faster system etc etc but there's more to the story so the first part we're going to do is this was without integrity this is like the high performance but higher risk and for my lab I think this is perfectly fine setup for production I don't accept that as an option because you you know occasionally things happen even with a UPS even with proper shutdowns something could happen you want the most integrity so let's see what happens when we turn on CFS synching Riggin over here and we'll to go to this here and we're going to edit options Cinque always save you can do these without shutting down the VMS this is perfectly fine and let's just run we don't even need to much more than run the raw speed test so it'll speed tests here have the number pulled up 776 so run it here ok this is where things slow down it's going to take a while to even complete the speed test this is we went from if you remember the original speed test before the right here and well actually scroll back while we're waiting so if we scroll all the way up to the top here we're hitting 975 949 and you can see all the writes happening but when you tell it to sync every change as it's occurring we lose 90% of the speed without a slog device even though we're running SSDs so if you're wondering if SSDs still need a slog device absolutely so we're down to 806 so almost 90% of our speeds it's just gone so it's substantially and I'm not gonna bother running this test it's gonna be the same it's gonna be really really poor performance so right here is telling you that you really need the sinks turned on so now what happens if we go and turn the sinking on and add a slog device so we're going to do both here swear to God sinking turned on so now let's go ahead and once again you don't need to shut down the VMS for this because the devices like slog and cache can be dynamically removed or added to a pool we can go here we're going to go add the log device and this is another SSD we have in here so we're gonna go ahead and though this over here and extend confirm we're extending the pool and watch what happens over here this is going to refresh in a second it'll show up there we go now let's just run the same test now we have this slog device and all that people ask how big it needs to be on this is way bigger than it needs to be by the way a really small drive because you're only putting as much data as it can cumulate over a short period of time they detail that there's a formula you can calculate but they can be very very small drives and I'll demonstrate that here run the same speed test and the only thing we did was add a slog drive so here we actually see quite a bit of data being written to this log drive now like I said these are all SSDs and this is not any faster than these so we went from 106 to 426 now let's run this test again to see what type of performance changes there are here whoop so we'll run it as a seam here so 2 2 1 now the reason I'm running this test again is because it's not going to be as fast as it was once you turn on synchronization even with this device here there are still slow down source still performance issues that happen here it's just not going to be quite as fast but you have very high integrity because it's absolutely synchronizing and writing to the ZFS metadata every single minor changed it happens to the system so it's not that ZFS with NFS is bad but if you want 100% top-notch integrity this is the way to get it but that does come still at the expense of not being quite as fast running here and we'll see what the results were yeah down about 300 on this particular test so that is a far cry from if we scroll up here the 694 that we had before so you can see there's quite a bit of a difference but that's still not the full story matter of fact when we run as a couple times it may even improve a little bit as it may expand the VM but it's gonna be will write one more time while the waiting is so two to one and see if it's a little bit better the second time but probably not much sometimes you get small variations on there after you add something I don't know if it's because it's caching or expanding and drive a little bit but you'll see some small variations on it but you can see the integrity comes at the expense of speed but we're gonna dive deeper into this because this is still just one thing happening at a time the idea of entire virtualization stack is so many things can happen at a time and you actually knows we're using a little bit of the cash like kilobits of the cash drive here so not much that's one reason I left in there it's not going to even with the ice because the over here it's not going to create a big performance difference so this is running and let's see where we end up should be just about finished there we go yeah 292 pretty much the same as before but here we are over at the ice cozy and we're on the same test again two two one and while that's running I want to point out something else so all this is fine and we're running each one of these individually so let's take a look though at the stats that we're getting from the storage devices so storages and we'll open up each one separately there's a scuzzy on free dance mini XL plus and here's the NFS on there and we'll go to the stats page so we see they're pretty impressive I ops that we're getting out there and if you're not familiar with AI ops this is an important number when it comes to determining virtual machine performance it's not just about like raw performance of how fast can we transfer data back and forth it's you know there's a lot of operations going on especially with these sync operations especially with a lot of vm's ideally maybe 20 30 VMs you have running on they're all doing things you need I up says and how much kind of performance on there then we have the i/o wait time which is how long are we waiting for the i/o and then we have the latency here so these are all the different factors that's why these stats are here and then here on the I scuzzy one we're seeing what you just ran that test here so we've seen about 450 493 oriented tests before but we only have a two millisecond latency and our i/o wait time hit 23% versus over here we were hitting in the 31% and four millisecond but we did see some bursting that was a lot faster so that's going to be some of the factors that play into it now why am i bringing all this up so let's go ahead and we're going to do this side by side we're gonna start running the test and there we go this lined up to better so now we have these two lined up so now we can run this here and let's push it quite a bit further so we're still getting 520 out of here but still 290 so it's not as much but these are of course benchmarks which sometimes don't always get things right so I want to show some more real world here one of the things you're gonna see right away is when you start creating snapshots because snapshots which we love and this is the wonderful reason for running any of these so we're gonna hit AFS right now - at snapshot here is you want to be able to snapshot your VMs it's one of the huge things before you make a change like a snapshot you can revert back to snapshots that makes my backups easier and things like that snapshots come at a performance price this is one of the reasons when you're doing storage planning and designing I've never understood people who want to store it everything inside the VM for example if you have a file server don't store it all in the VM because if you want to create any snapshots you're trying to keep differentials and we have clients you know liquid a few terabytes of data and thousands of little documents for course they're Enterprise if you store them all inside of um that we very difficult because now your snapchatting at this level and big performance pencil you at some point you want the file system to be directly attached to some other storage device for storage as a talk for another day about storage planning outside of here we're gonna focus on the BMS but this differential that has to be created is we have to keep track of the changes between the current status of the operating system that's running in a VM and the difference between here so I can at any given time you know revert this back to the snapshot so what does that look like in terms of this so if we run here you know we'll just actually do a speed test here's the first time we run it it's gonna be even slower than the next time we run it so we'll run it once here and you'll start seeing we lost we went from 400 down to 373 we run it again and probably run a little bit faster because it's now created at least one Delta that's about that big between the two yeah about the same and I was running over here and we're still getting about the same speed now why do we lose speed over here and over here well this is the way you treat those deltas the deltas are separate individual files that the underlying operating system is handling so if we go here and CD slash Mount some lab here are the files we just created and what these files are are the Delta differences so we have the main VM and then we have the Delta differences between those and that is what's occurring here so we wrote a two gig file but it seems to that we have 3.3 gigs here so it's telling you go create this other file create this Delta and create separation between it and any changes that occur in this so we can always revert back to them log those changes in there and it's basically XZ PNG or whatever your hypervisor is talking back to the system on the back end and create these files and put this transaction data here so for any time you create transaction data in this it has to then be replicated over here or the differential between at the Delta and this obviously comes at the price of performance because when you have a bunch of snapshots and we have to keep track of all the snapshots and then you compound this problem by having a whole lot of VMs running and a whole lot of snapshots and you can see how this can add up really quickly and then there's some management overhead that comes with all that I scuzzy is presented as a block device to the hypervisor so exci PNG handles that by talking directly to it as if it's a hard drive attached to it and then that's why we can't see if even from the command line when you look at block devices inside a freenas it just goes it's a block device it's as eval we presented it as if it was a hard drive so all those transactions occur by just talking to it as if it's a hard drive so it's a different fundamental two-way ice because he works which means we lose less performance out of it so we were doing this b test with or without those snapshots we still get reasonable performance so let's look at the performances now since we have this so the last one before the snapshot was 292 what are we looking at with the snapshot two to one let that run like she will put at the top here so pull the stats again we see all the read rights happening the Zil cash doing its thing and once again it's not just doing its thing for that one VHD file it has to to keep track of so we're at 291 and let's scroll back again to scroll back a little further not too bad we're keeping up pretty decent with it but it's not near as what it was before so here this same thing we got a snapshot look over here two to one so what kind of performance we get on the ice cozy are you d completed and interestingly because it had one run that apparently was 376 maybe it was some left over rights happening so one time slow but the rest of time consistent we ended up with 508 so even with these snapshots we didn't kill the performance over here but for the hell but let's go create another snapshot so there's more things keep track of so a new snapshot again it was the same over here now let's look at this I want to look at the performance real quick so here's the NFS 1 here's here io speed I ops we still get more speed out of here because of the way the transactions are working so we actually still got better performance out of the ice cuz he it was able to perform at 11,000 I ops versus only 7,000 over here let's go back and now that we created more snapshots let's just do the raw speed test cuz it's faster and like I said there's always I've commented many times lies damn lies and benchmarks benchmarks are only so much you can do but they at least give us an idea of the performance difference and even with more snapshots we still don't lose over here because it's all handling it within the system directly now the other thing we're gonna do is go ahead and purge out these snapshots so purge that one urge this one you this I'll also mention when you do the purging or I should go here CD slash amount and let's see how fast this happens depending on the amount of i/o activity there is a time it takes to coalesce basically and do a cleanup of all those files that got created on there so that's worth mentioning that when you do these sometimes you have to wait a minute for the i/o to settle down so it takes time to coalesce so the speed performance gets back to where it should be looks like those things may have coalesced fast enough and you can look up coalescence between them it's not completely coalesced because it's still performing a bit slower and this one's perform well now the last thing I want to test here to kind of show this we're going to do for all options five all options just go for rate performance we'll do the same thing over here so for I've won so now both of these are just beating up the same pool of drives intensely fighting for resources so let's see how that's being handled on the back end here so we're gonna look at this here and let this ramp up so we're watching all the drives we'll let this run for a second and see what kind of AI Ops we get like I said they're competing for resources on here and we're gonna go over here inside our FreeNAS and go to sharing of not sharing services start and stop this because if nut date is running by the way and you add and remove cash drives sometimes it decides to not show them so by doing this there we go but you can see that the disk rate is being pushed pretty hard these CPUs being pushed pretty hard but not 100% it's not completely used as matter of fact our comments as well France is very responsive despite all this disk reading going on ZFS system so you can see what's going on here there's a little gap I just turned it on and off so make sure had all the drives in here so we can see the demand prefetch made of data read and write performance going on right here not much reading a whole lot of writing efficiency and this is a data demand efficiency we're gonna see this red right here this is where it's just you know actually writing and exhausting caches and information where it needs to be so go back up here the system overview and let's look at these now so competing we're back up to better performance over here on the I scuzzy side so we're seeing 496 peak with 11,000 on the I ops versus 275 peak and 6090 I ops in our I oh wait time two milliseconds of latency here four milliseconds latency here I oh wait is around 22% io8 just 20 so really close I mean I'm seeing some peak here look we have 29 percent on there so pretty reasonable but you're still seeing that I scuzzy edging it out now I'll come back to the article one more times everyone I do want to mention something here just to reiterate and as I said the beginning I scuzzy by default does not implement sync rates as such it often appears users much faster therefore much better choice in NFS however your VM did is being written async so the VM data itself is being written async but the integrity of ZFS behind here isn't sync so potentially because and this actually is true of really any time you're dealing with block level device being presented to another operating system from another operating system which is what I scuzzy is doing is presenting the Z Val as a hard drive to whatever you're going to save it on there is an inherent risk because you're taking and changing blocks on the hard drive it's just like if you unplugged a hard drive mid-session there is the potential for a corruption in there in one risk that you may face is if that corruption crops the entire Z Val all the VMS within has eval well they could have a problem and this is one of the reasons backup is so critical why having a UPS and proper shutdown management so you don't have sudden power loss is critical so there is that potential for it yes there's some checking that goes on but think about this just in a perspective of if you unplug this AV cable to a hard drive that's doing an internal write process that's you know scattering on the files or let's say something like a defrag or Co lessons that may occur with vm's these are things that are happening all the time especially in a busy VM environment so there is an inherent risk of data corruption we really with any system your rate array going bad and writing things bad so this is just something to think about reasons you should have snapshots for any of these and it's one of the things that I scuzzy is really good on performance but because you're not writing solid individual block files it's all one block presented to the operating system there is always the chance for corruption in a catastrophic failure versus well any v8 VHD files that are saved to the hard drive in a case of using NFS those VHD files are all individual files so it may not corrupt all the VMS that may corrupt the one that was doing the writing but other ones and especially any of them that are turned off there's no risk of corruption it knows because well they're not doing anything they're off so there's not active writing going on so this throwing the options out there for a lab environment I think NFS and this is actually what we use and why so much for ours and if s our lab environment is NFS with sync turned off I find it to be an acceptable risk and I get the absolute best performance out of it but as always backup backup backup i backup everything any time I think there's gonna be a significant change even to our lab environment we have this dedicated just to do all these different Delta backups for lab that way they can be back to very very fast and anytime they make any large updates or changes to them that way in case there was ever some catastrophic failure no big deal not to mention your lab machines I used for YouTube demos or testing a theory I have or something that a client may want set up they're not mission-critical and they can be reloaded I back them up more just for the convenience of not having to reload them as I do things but like I said these are some of the options out there those articles will let you dive deeper into ZFS and for those you really wanted to dive into performance I do I've mentioned this article many times this is such a good article to all the different write performances and read performances of all the different rate options in CFS and what they mean and it's a good dive into it if you want to really get a grasp on how all that works all right and thanks and thank you for making it to the end of the video if you like this video please give it a thumbs up if you'd like to see more content from the channel hit the subscribe button and hit the bell icon if you like youtube to notify you when new videos come out if you'd like to hire us head over to lawrence systems comm fill out our contact page and let us know what we can help you with in what projects you like us to work together on if you want to carry on the discussion hetero to forum style or insistence calm where we can carry on the discussion about this video other videos or other tech topics in general even suggestions for new videos that are accepted right there on our forums which are free also if you'd like to help the channel on other ways head over to our affiliate page we have a lot of great tech offers for you and once again thanks for watching and see you next time

Channel: Lawrence Systems

Views: 29,075

Keywords: freenas (software), iscsi initiator, virtual machine, san storage, xen server, network attached storage 2019, xen orchestra, freenas iscsi, network attached storage explained, open source, esxi nfs, vmware, nfs, storage, nfs slow iscsi, iscsi, nas, network attached storage, why is NFS slow

Id: fo3EGuCszpE

Length: 34min 8sec (2048 seconds)

Published: Fri Oct 04 2019