ZFS for Newbies

I like this disclaimer because it absolves me of all responsibility so those of you familiar with edifice may feel that I'm not fully covering some issues and that's but on purpose because this is for newbies I can't I can't go into detail so you'll hear me say something you'll say that don't because it's not relevant to newbies it'll be relevant to the months they've been using it for well but if you're a newbie those little esoteric things that I've not mentioned or glossed over aren't relevant because I am grossly simplifying so I will admit stuff and there will be options skipped because this is a talk for newbies mm-hm so what we're gonna cover are gonna cover all that that's just a copy of what was in that in the program so started as Sun Microsystems 2008 that became a part of FreeBSD and I don't really know when I started it sometime between 2008 and March 2010 I don't know where in there but I just remember what I was doing at the time not listed in this list is the Sun is the Oracle acquisition of Sun and ZFS is still developed in Hibbs there and most of the open zetas festive element now occurs into the open ZFS project this is stuff you can look up basically you're not gonna run out you're gonna run out of hardware before you run out of address space in ZFS um ZFS is not raid don't think of it as raid think of it as redundancy basically you put your drives together into a pool which we call a zpool you can create a mirror consisting of two to n drives you can create a raid Z which mean which usually has a number 1 2 3 after it that's the number of drives you can lose from that device without losing any data I like to think about that way instead of traditional raid because if I have a raid C 3 sitting here I can lose three of those 8 drives and the system still performs think of raid Z is buying your time to replace the failed drive it's not it's not keeping your data it's just buying you time to replace the drive before you lose your data all the above commands to create a pool come from Z pull create a file system is part of a zpool so you put your drives in the box you say ok these drives are going to be this Z pull and then in that zpool you create a fall system because you're drawing from your pool of disks to create a file system a file system can have inherited properties so if you say ok this is equal I'm gonna put compression on and you can say everything below it is going to have compression on such as you're gonna have say slash of R is gonna have compression then VAR temp VAR DB whatever else is created in there by default will have compression on you can override it in any let at any step along the way the big advantage of pooling your drives instead of giving a partition for each little part of the system that you used to do is that you don't wind up having no space at all on VAR DB but 200 Meghan slash user isn't Watlington you don't have to do assembling from VAR DB MySQL to slash user local minus QL for example this is a zpool this is equal taken from a did you loosen droplet that I have that does an egg is installation this is the name of the 0 how much drive space I have was allocated used fragmentation is not the fragmentation that you think of when you think fragmentation so forget it capacity is how much of this I've used and yeah it's online and dee doop don't use D be friends don't let friends use dee doop you'll hear about it and you'll want to use it but don't do it use compression instead so this is this whole file system based on that this is set up for be e admin sorry boot environment we'll talk about that later but if you just ignore the Z root bit you can basically see the fall system which is more or less mirrored over here you don't have to do it that way I could create Z root Dan and mount it over here under user home Dan so your mount points are separate from your filesystem names either your filesystem names all the way through here and these are the mount points so let me just make sure I kept everything Z root is the name of the zpool 0 slash root is a data set file system within that pool Z root root default this one here o 0 root default is a descendant of root and is also a filesystem so yeah this is a filesystem lots of file systems as a Soza it does not have to be named root by the way it can be named whatever you want for this boot environment we'll get into boot environments later legacy this legacy bit here it means that the bootloader VFS root man from equals gets set from 0 : 0 default basically in here that's what the legacy means and it's derived from the boot FS property of the pool so each Z pool you can say ok when you want to boot from here boot from there which means you can turn the bios off and your HBA so it doesn't iterate all the drives and it boots up faster because I generally boot from a separate Drive connected directly to the motherboard rather than off the HBA ok what's a v-dub I again grossly grossly simplified because it's skipping over spares logs cache and files which you don't want to know about because worker this is a newbie talk what savvy death a V dev is a single disk that V dev can be a mirror of two or more disks or V dev can be a raid Z which we've already covered all these terms so why do you need to know about a V dev because you create a zpool from a V dev do I get that right so some of the terms I'm going to use I'm gonna constantly refer to fall systems on I'm gonna refer to data sets they are the same thing don't get confused by that so a data set can be a filesystem a volume which is not commonly used or it can be a snapshot and we'll cover snapshots later but that's a read-only version of a file system these are some interesting properties you can set on a filesystem and set a time off if you don't set a time off what happens is when you read something it's going to write something every time so W here IO and a time isn't really as relevant as it used to be exact equals no is useful especially if you put it on something slack like slash temp because it means someone can't write a file there and then run it which breaks IO cage reservation means hey listen always keep 10 gigs spare on this file system or don't let anyone go over 5 gig on this file system so you give space to your users they have a writable directory and you set a limit on it when it comes time to replace a failed Drive step one is the important bit you don't want to remove the wrong Drive some people will do a very clever thing which involves putting the serial number in the device name which can be useful because then you can see which drive you're supposed to pull out so add the new Drive in remember I didn't save her remove the failing drive because you your drive may be wonky but your system is still perfectly intact the zpool is not degraded and if you pull a drive out you Z pull is degraded and if another drive fails then you've got an even bigger problem so if you can keep your existing drive in there until this process finishes at which case the failing drive is no longer part of the Z poling just pull it out so this is a system in which I did exactly what I'm talking about this is an old case I'm not sure if I still have it but I might have a is uh I think there's three HBAs up on the top here and so I had to replace a drive this was actually me replacing three terabyte drives with five terabyte drives and yes that is just sitting loose in the case and there is actually this may have been before I attach the power cords and so but I put the I took one of my drives out of the drive bays down here and put it in there and then put a new drive in here and then started replacing the 3 terabyte with five terabyte one at a time don't use regen cards just like I said friends don't let friends do something what was it before do you do friends don't let friends do use red curtains LFS raid hides stuff so the rate card will try to fix something and fix something and fix something and say oh no this drive is Jed I can't do anything with it and then zetas fesses what where'd my Drive go where as NFS will try and fix the drive for you it'll say ok this Drive is having trouble reading from over here let me go over and check this drive because it's mirrored cuz I know it's gonna be over here oh it's there it's just fine oh I'll put this somewhere else over here but it works so zetta thoughts are trying to fix something if it can and if it can't it'll tell you that it failed it won't silently give you a corrupted file it'll tell you that it failed so use HPA not right scaling if you need more space you can do what I did earlier and upgrade all the drives replace three terabyte with five terabyte or you can add a new video we talked about vide Ed's earlier so you've got a mirror here of drives you can just add another mirror what you can't do is just take a raid Z something and add new drives so you can't go from an eighth drive raid z2 to a nine drive raid-z to because math is hard like really hard to do that so but they are working on changing that so you may have heard the ZFS is not not expandable that's the bit that's not expandable it is expandable where you can say either Z pool with 8 drives it's a raid Z 2 here's another 8 drives rigsy true is it raid-z to strike them and it's done its stripes you'd not doubled your space this is a single most important reason to use ZFS even if it's only one drive it checks sums everything it puts it in metadata and it's hierarchical so basically you write data down here in a file and it puts a checksum here but then it checks on it with the other data that matches up and all the way up to the top of the tree it's check sums all the way down so when it goes to read something it checks sums data and looks at the checksum that's already on disk and if it doesn't match it goes to try and fix it but if it can't fix it it tells you but it'll tell you about the errors and it'll look for them and can correct it if it can so the difference between that and a regular file system will non check something file system is that you'll find out about the data being wrong instead of just being given this here's this photograph but one bit somewhere is wrong instead you'll be told here's this photograph and the data is bad not so much important in a file but sorry in a photograph but maybe in a file that has a bunch of accounting figures in it and one bit is changed so scrubs what scrubs does is it reads all the data that you've written to the disk and it reads it all and compares checksums and reports any errors that it finds and attempts to fix any errors that you find always turn this on and it's very easy to turn on I do it every 7 days you can do it every month however frequent you want to do it just make sure you don't do it more frequent than it takes to run I've got some scrubs that take about 24 hours run but they won't overlap snapshots are the most fantastic thing for backups one of the biggest problems with backups is that you will start your backup let's say at midnight and you'll be backing up all these files and by the time you finish at 2:30 a.m. this log file over here doesn't match this log file over here because this log file stops at midnight and this log file stops at 2:30 because they're constantly being written what you can do in the snapshot because it's read-only is you can take a snapshot at midnight and every single file as of midnight is in that snapshot it does not change for the lifetime of the backup so you take a snapshot you backup the snapshot you don't back up a lot of file system you backup the snapshot and it's an instant in time snapshots are usually very quick so you do a snapshot it's done yesterday ever this I discovered a snapshot that started running on Friday the 13th I had to reboot the system nobody could figure out what was going on snapshots cannot be modified so they're good in terms of I don't know if they would pass say some legal requirements for for images of files but they can't be changed that's why I like them for backups and but keep in mind snapshots then the same hosts who not backups but you can send a snapshot to another host so you've got a host on these coasts and you get a hosts on the west coast take a snapshot and you send it to the other host you can share a snapshot from any system to another system or even in a system to within the same system it's just an ability to say okay ZFS send type ZFS receive it's a little more complex than that but it is a useful way to do backups send them wherever you want because snapshots on another host are read-only and are a copy of the data offline well offline relative the first system we talked briefly about mirrors before two more drives with duplicate content but you can also stripe over mirrors so basically you have a set of mirrors here and a set of mirrors air and you stripe over both of them it's sometimes referred to as raid 10 because the raid 1 is the mirror in that 10 is the striping over the two raid 0 is usually referred to as striping but in in a stripe over two mirrors you can lose two drives so long as they're not in the same mirror before you die so briefly talked about raid Z so you need at least four drives to do raid Z and that'll get your raid Z one but the more drives you have the higher this number can come and this is the beauty you can lose n drives and still be operational and by operational I mean the system would continue to run it's not like oh my god this is some Cameron because I'm missing two it now it just runs one of the most neat things I use on ZFS is mounting within mounts so I have a bunch of slow drives for the main system two old spinning disks and that's what I boot from there directly connected to the motherboard but then I have a database that I run run on the system so add in a couple of SSDs for that database and I create the Z pull on the SSDs but then mount them in VAR DB Postgres which is just a mount point on the main system and your your code doesn't care it's on a different file system it's completely hidden and it just works you can also do this with slash temp but slash temp on the fast drives as well so here's an example of me doing exactly that for podría or have my Z root with my smaller drives here but then I have my fast tank with a lot of space here and you can see that pudiera is mounted at user local food rare and that's just part of user and it's completely transparent to anything else that I'm doing this is also another way to expand the system if you don't want to touch your original raid-z you can just put the space where you want up you can do a whole bunch of mount points like this boot environments a boot environment is a ZFS file system designed for booting and there's a few requirements for that but we're not gonna get into what the actual requirements are for a boot environment we're just gonna show you what you can do with it because you're gonna like it basically you can manage boot environments with b/e admin or be ECT LBE admin it's been around a bit longer Bea's CTL is now in base on FreeBSD I think be admin is imports basically what it does is it saves your current boot environment and the use case that we're going to use is we're going to upgrade from FreeBSD 11.3 to FreeBSD 12 so what you do is you save your current boot environment and what that does basically is it clones the environment that you're booting from now after you've done that you upgrade your current environment and then you reboot and if so all okay great if it's not you reboot and during the reboot process you choose the environment that you saved away here's an example see this magic option here number seven I press number seven here and I got that which is my default and I change it to that which is really difficult to read but that's the other bit environment and then I put it back in to eleven point three so anytime you're doing updates anytime you're doing major changes you can use boot environments for something like that you can also use it for booting new kernels if you're doing a lot of kernel work next boot is the neat thing that you can use for that because all you do just say on my next boot boot from this and then the next boot after that is what you're using before so it is truly just boot once from this other thing so we're gonna go through some very simple configuration things here we're creating a partition on da0 we add a ZFS partition this is just for K alignment this is a label which is the serial number of the drive and boom that's what looks like there and you've got the ZFS partition and then what I do is I've done the same thing with my other drive and I saves you pool create this is the name of the Z pool I'm call it it's going to be a mirror and it's gonna be on those two drives and bang there's my Z pool now this is actually an older Z pool because I actually did a scrub on this but you can see that it is it is called env D it doesn't match up with my data because this is actually taken from one system and then modified to suit the example ignore that part basically here's mirror zero and if you have a stripe of mirrors you could duplicate these three lines just down below here we'll see that in the next example so here we've got four drives and we're going to create a raid z1 again it's going to be called my data and it's raid-z one and it's four drives what does that look raid-z - we just had another drive crazies we'll get to raise the three nets but there's raid-z - you can see the four the five six drive one of them has a serial number in it so there's a red XIII with six Drive there's a raid z1 again you basically give it a name say I'm gonna mirror those two drives and I'm gonna mirror those two drives and you can repeat that process as far down the pages you have drives and there's a red Z raid10 a mirror a mirror and it's striped over the to talk briefly about quotas it's on the data set its limit on how much space you can use and it includes the descendants so if you let you use your create new file systems that's all going to be included in there the other thing is it includes snapshots because although you take a snap if you take a snapshot and then delete a five gig file it doesn't free up the space because it's still involved in the snapshot so you got to keep that in mind that snapshots when you make them that space will stay around until you delete the snapshot yeah and I don't want to get into this stuff because even I have trouble figuring out what it means remember to scrub you can create an egg as script for doing that also runs evil status because that tells you if there's any problems monitor quotas and monitored the capacity and there are scripts for during all that easily available now we're gonna go through some myth-busting because there's a lot of bad advice that says that there so I said this before a single Drive with set if s is better than those edifice at all so just try it and you will enjoy it ECC is not required people will say you got to use ECC with ZFS but ZFS what the ECC is better than those edifice at all you need high-end our hardware no you don't most of my systems are consumer grade they're definitely not enterprise you can get an HPA for about $100 off ebay I do have some super micro chess ease because I bought myself birthday presents and they were very much nicer than what I had before when you're looking for hardware look to the FreeNAS community because they figured this all out already especially for consumer-grade stuff lots of RAM no that's not true either I have a system running with one gig of ram and that one system runs with 250 gig free 250 mega free sorry and that's the digitalocean droplet that I have it's very slow when I go to the web page but it's just for monitoring I don't I don't need speed I just needed to monitor oh so this isn't a tip from last night this is more like a tip from sometime in August but put your OS on a zetas fit for your operating system on an on a ZFS mirror and put your data on the rest because then you can actually you could actually use you FS on the OS instead but whatever you do don't move from the HBA and the reason I say that is if you have your boot drives spread across seven or eight drives in your raid Z the HP is going to choose one of them to boot from and that means you have to iterate through all those drives during the boot process you can speed up your boot process by turning off the BIOS on your drive if you're booting from things directly connected to the motherboard and the the main reason I like booting from the motherboard is it eliminates any problems that you might have with the HBA if your raid Z gives out or the HP ink is out you can probably still boot off the motherboard I just think it removes complexity from your breeding I got some tips from savage late she's here I don't know if she's in the room but she is at the conference oh oh that's not good there's this one button down here that takes me out of it I already mentioned this tell your BIOS to ignore the HP so it's fewer drives to scan that goes faster you can safely partition SSDs using the OS so basically you get your two SSDs you get the OS mounted on you don't need all that space for the sf4 for the OS you only need what to three gig for the operating system you can partition other bits of that SSD and use it for other things create another Z pool so basically the same Drive is involved in multiple Z pools and I do that I do that at home set your record size very big for files that have large data you can go you can get another 5 to 10 it was a significant percentage I don't want to say how much it was but it was a significant percentage we change the record size on on some drives when I restarted this it lost my time so basically I've been going about a half hour now right we started at 10:30 so we covered a lot of stuff I went through that way faster than I thought I would coffee helps there must be questions because as well you I bet you this can move I just so it gets recorded so two questions from newbie you said that snapshots are good for backup but they are read-only so how do you restore you just restore as you would and you restore it over the main file system you don't restore into the backup yeah but if you send your snapshot back it's still read-only right yeah so you remove the read-only flag and you're done or I mean no no how do you make it readwrite again what you can do is you there's a hidden directory called dot ZFS at the top of every file system and if you just want one file you can do CDs NFS slash snapshot slash whatever you name the snapshot down to where you have the file and you can copy it over or you can do a ZFS roll back but that rolls back everything now what I mean is suppose you make a snapshot of your dataset oh yeah yeah send it to another data center yeah and in your original data center there was a disaster so you lost everything now you want to restore so you have read-only it's not shot waiting for you to that's a good point is the thing that you sent actually read-only no I don't think it is because it's not a snapshot anyone done this already what you've received is not read-only right yeah yeah so you just do is that a first send of your snapshot into the first receive closely person right you know is that if I send over to your other system so the first receive which unpacks it as a user FS in the system and it's rewrite on the other says do everything yep thank you Dan I have a question regarding terminology you said ZFS but zebu mm-hmm why why is the Rossetti havoc habit because that pool doesn't sound very good and yeah Canadian so it is if is it I don't know how my vocabulary changes I don't know what what do Americans say another question so to keep the flame wars and III I mean the religious wars away very briefly your comments or your comparison of is it the fest with but Rufus oh that's easy I've never used better FS when you are asking about the zpool why is that VF o Z pullers consists of V devs right right I always have trouble remembering this myself I'm figuring out I just know that I create busy people a lot of travelers but like the Zetas okay oh virtual device I think is what it means but I have question about the partitions you but before you add a drive to a Zippo you partition a with GP t yes and I don't know why because the drives may have slightly different sizes or something Linux tends to just give it a whole drive FreeBSD tends to partitioner and give it the partition that allows you to do a lot of things I think the main reason is because you can do a lot of magic with giome if you have the partitions but if you have the raw drive you can't the other reason is give it our partition don't use all the drive so then if you have to replace it with another drive which may be slightly smaller even though it's advertises the same size you can just produce a partition a little bit and you haven't run into a bad place more questions so who hasn't used that FS yet all right so it's been around a long time it worked it's pretty solid once you get into the bigger jobs you don't have to worry about the F sack at the beginning it's no more questions we're done thank you [Applause] you

Channel: EuroBSDcon

Views: 40,897

Keywords:

Id: 3oG-1U5AI9A

Length: 34min 35sec (2075 seconds)

Published: Sun Oct 27 2019

Reddit Comments

[u/Maltz42]

Yeah, not sure why some people are so hung up on ECC RAM as a requirement for ZFS, however it's not like ECC has no impact at all. I would tweak that slide:

ECC > Non-ECC >> No ZFS at all

My personal preference is to use ECC, but I do have a ZFS system that doesn't have it. I do burn-in new RAM with a lot of testing with memtest86 first, though.

And yeah, friends don't let friends dedup. lol That is where the gobs of RAM myth comes from, btw. You do need a LOT of RAM if you're doing dedup, but then you're saving a tiny bit of dirt-cheap storage by adding a ton of really expensive RAM. It's counterproductive, especially for consumer use.

[u/pelicSinsin]

Thanks for the post—can’t wait to watch

[u/dlangille]

Cool t-shirt.