How To Backup Your FreeNAS 11.3 Using ZFS Replication

I'm here from Lawrence Systems we're going to talk about ZFS replication tasks in FreeNAS 11.3 series which is the latest one as of May of 2020 they've really streamlined a process so I want to do a new video on it and let's get started but first if you like to learn more about me and my company head over to Lawrence Systems calm if you'd like to hire short project there's a higher button right at the top if you want to support this channel in other ways there's affiliate links down below to get your deals and discounts on products and services we talked about on this channel including a link to our patreon if you like become a patreon supporter we also have a swag store where you can get shirts and other items that are for sale and that changes from time to time what's available and what's not so go ahead and check that out frequently and finally our forums if you'd like to have a more in-depth discussion about this video suggestions for new videos or just reach out say hi and talk tech our forums are a great place for that all right now back to the content now before we go all the way deep into ZFS replication if our sink is so great why is ZFS replication maybe a thing I'll leave you the link to this article over at ARS technica they dive into some of the technical details what's really important is ZFS actually duplicates at the block level of the ZFS system the ZFS send is a function built into ZFS to replicate a data set to another ZFS pool so it's really fast it's not just looking at files like a lot of other tools are it's working at the level below that so it's working as in sending the ZFS blocks over there but there's a lot more technical to it and I'll leave this Ars Technica article so you can read and see some of the advantages of how fast and efficient it is at replicating data now the one drawback is yes you do need ZFS for this to work so it has to be doesn't necessarily FreeNAS but we're gonna do all this all up renounce to ZFS replication replicates not to other locations not to a file but to another ZFS pool that is an important thing if you're doing your free necessary - no problem if you're doing it FreeNAS to something else running ZFS provided it's a compatible version that does support the same functionality it should work but we're going to do this from freenas to FreeNAS replication is the process of copying ZFS dataset snapshot from one storage pool to another I wanted to make sure that this was highlighted that it's a ZFS data snapshot I've done videos before on how snapshots work and what they are is a differential of the data set at any given moment from another spot in time and it's a snapshot of the data and the only space taken up by a snapshot and of course this also ties in to how it replicates is the difference in any files added or deleted and he changes to the file system after that snapshot so think of it as a snapshot in time that's exactly what it is now they're very important to the replication process because you can't replicate data in flight so you have to have a snapshot a point in time at which you reference to send it so the replication task itself doesn't run on a schedule of its own it ties to this schedule of a data snapshot so once we build this any adjustments we want to make to the retention policies or how long we hold the data essentially or when we snapshot the data or how frequently snapshot the data is all controlled by the ZFS data snapshot task and we'll cover that in when we get in here now I've set up two FreeNAS boxes and this ugly theme I have loaded which good news is we don't have to look at it much as the high contrast theme inside a year but as did that this is to show this is our destination server this is a really lower end it's got the I just did the this is my partially post spare-parts freenas build I did a video on I created a empty pool right here just called backup Nass really simple pool name really simple lower end servers usually you know rain is not a backup but you can't back up to another raid system and maybe the main raid system has all the heavy lifting to do with all the users but you see somewhere else to store the data that's no not the main system because rates not a backup and we chose this system for our destination this system right here is the FreeNAS our 730 XD I did a review of this thank you for tech supply thank you to sect X apply direct for sending us out to me if you're interested I have the full review I'll leave a link to on this and if you're innocent and buying a serve from tech supply direct we do have an offer they get you a 10% discount on any server equipment you'd like to purchase from them we've bought quite a few servers from their great company to deal with easy to work with I'm not just saying it because he sent me the server we actually started buying servers from them prior to working out the affiliate deal that we have for that we dislike them as a company they've been great to us all right back over to here we have this storage pool and we have a data set we want to back up called SMB demo share I was doing this demo and it does say delete me later but we're gonna pretend the important there's really important data in here that we want back to up snapshots there's none we go over here to TAS vacation tasks non-periodic snapshot none and we're gonna hear the system and we're gonna go ssh connection ssh key pair all blank these servers don't know each other we're gonna make an introduction and the way they've done the wizard is really slick for doing this so right here is one 92168 three two oh nine and this is three two one two we're gonna start all through the wizard go over here to task replication task we're gonna add one source location this system now this is a push-pull system you can actually say the source is somewhere else and I want to pull it over here or I can push the data over there you can have it on either system go either way I'm choosing to do it from the main server pushing that way but this does work in both directions so it can be done either way I just this is the way I'm gonna choose to do it choose here she was here and we're gonna say SMB demo share back it up so right there's that SMB demo share how do we know how to get to the other server on this system I will note if you had multiple pools on a single system this replication will work internally on a single FreeNAS but then you still have all the data physically in one box maybe you do have a reason you want to duplicate it over to another pool that is a possibility but ideally you're backing up to a physically different box maybe even at a physically different location and this does use a secure transport layer but I'd still put it over a VPN if you wanted to replicate across sites but this these two servers are in the same room with each other but it's still a backup because it's in two locations at that point just not geographically backed up so on a different system that's the sage connection if we already had set this up previously we could easily do that and say all right we're going to create an ssh connection and use an existing one but the wizard is really handy because it says create new one this is where we get to make an introduction so we'll call this FreeNAS back up server we'll use the set up method of semi-automatic and then we're going to take the copy that and paste it here that's it you give it the URL for the other freenas username root password I don't know the password but it worden does we're gonna go here and happy the password paste it private key if you haven't generated one or not even sure what that means it'll take care of it this uses SSH as a transport to encapsulate everything and send it across it's a really secure way to do it you can look up how SSH secure key connections work and this Creech went for you automatically this is really nice the way this is built-in now the two systems have been introduced to each other which means I can see the data set like this that says backup masked now you don't want to just send it to the backup now if you want to give it a destination and we'll give it the destination of SMB demo what was it called demo share oh de mo demo share all right now recursive if you're backing up and let's say we started here where it says this pool you can recursively backup everything under that pools not always how you're doing it but maybe that's how you want to do it that is a possibility and you could do that and get everything except that way we're not gonna bother with that we're not gonna worry about any custom manual snapshots we're gonna leave that it says they're not found anyways because I don't have any encryption more secure but slower I always go with encryption it's always better to encrypt and err on the side of caution but maybe if you have some secure place you have an excuse not to encrypt it because you just want to send across I'm always encrypt this my attitude unless there's some really good reason not to destination snapshot lifetime and run on schedule default is running on schedule and the defaults of statement source I always leave the statement source can be adjusted later what this means is how long do you keep those snapshots for whatever your data retention policies are and when you adjust them you adjust them on the system doing the replication tasks and a periodic snapshot task which is this main system and like I said they can be adjusted later just into whatever your data retention policies are scheduling daily at midnight is the default we could go hourly we can go custom maybe you want them only backed up at certain minute certain hours certain days they have some presets we'll leave it at daily here because that's perfectly fine and we hit start replication which actually doesn't start the replication it starts as pending because there's no snapshots there's no anything yet so this built it all out but it's not actually replicating yet it'll replicate on the schedule like it's supposed to or we're just going to go ahead and force its hand and say run now demo share has started now so as we refresh this this is starting snapshots refresh says hey there's that SMB demo share and there's already 313 megabytes of data let's refresh it again there's a gig of data one point two gigs every time we refresh it a little bit more data is moving across there it's moving pretty fast application.run about 21% done is to Ana verification it goes making sure all the data sent and 35 will just pause right here for a second and pass forward you and done so now we have one snapshot in the snapshot if you dive into or my previous IDO snapshots you know there's only going to reference the data that is sent so the reference means how much data there is versus how much two snapshots using snapshots not using any because there's not any differences and there won't unless we change any data on this particular folder it'll just keep synchronizing and verifying that everything's in sync so we look over here at the replication tasks and we look over here at the snapshot tasks what if we go well I want to change that let's uh let's synchronize this faster so we go to edit and snapshot lifetime two weeks seems good but why not have it custom hourly but let's go minutes because I only want to wait minutes so I can finish my video here so if I put an asterisk here here and here every minute it's going to replicate in sync now the first time is synchronized they had to get all the data every one after that is just a differential of any data that has changed so it actually ends up going really really fast all the other ones that occur after this occur every minute really really fast and no big deal they're going to only look at any block level differences and I said Blok not file level which means even if you do some renaming or things like that you're barely changing any blocks that you're just changing names of files it says oh there's not much of a block level difference between what you did in this particular folder so we're only gonna send over little changes and these happen really really fast I've even seen some companies when they have massive data sets to move where they'll start with a FreeNAS and the local building get the first synchronization done but they don't produce that much data on a daily basis when they're doing a synchronization to then physically take the server somewhere else to replicate to it on a more ongoing basis but however you start your replication process the first one does have to send all the data and every one after that only sends the differential data after it will go over here and yep another one completed vacation tasks all right we'll go over at the pool so storage there's two so every time a new snapshot gets created it sends it over there and we look over here in snapshots now there's two of them here it's as simple as that for each one for each task that's occurring there's another one now let's go ahead and create one more task we go over here go to task I shot and we'll add one more tasks add location this system and this time we'll just grab the IO cage folder this we do want to be recursively there's like a whole set there's a bunch of datasets underneath there so each each of these are data sets now the difference between a data set in a folder two data sets an actual functional part of the ZFS versus all the folders that are within a data set are automatically so you don't actually have to do the recursive to get folders underneath but if there's data sets underneath those are different types but we want to backup everything it's under the i/o cage recursively make sense all right and where we want to send it as a destination on a different system SSH connections we can create a new one again but we already have the server so now lets me choose it choose this choose this and we'll put it right here and we'll put it I oh cage back up it next daily same as source we'll let this one be daily it's just IO cage maybe I want to back all that up one day well for purposes of the video I jump back into the snapshot task let's actually set this to be custom like the other one where we're just gonna run it with every minute that'll give us some data to work with here so I've set it to a minute and skipped ahead a few minutes so we have a few things going on there there's one of the challenges is if you have the replication tasks but no snapshots have been created so you try to run now with a replication task but there's not been at least one cycle of the staff shot tasks running then it doesn't really have any data sense so for purposes this video I went it set both of these to run every minute and I waited a little while to build a whole lot of data over here so if we go back over to the system I'm gonna look at these services I'm sorry to storage snapshots here's all those different snap shots now because I said recursive this is what it meant so back up IOH back up download images the fir Onix one I have on here all the jails all the logs releases it recursively grabbed every data set under there now that same thing applies because it's working off of the snapshot task that is recursive right here for the replication so this is recursive and so let's go back over here to task yes and these are set to recursive as well with the two-week retention and setup to run every minute and then we look on the storage of this one and we see this the same thing here's all those different shares that are on here all the i/o K H ones etc set our recursive ly being copied over and being copied over at speed because it's only looking at any differential so even though you do see what looks like a lot going on here it's only differential therefore if there's not much going on in any of these it's not much data now a couple little housekeeping things and we'll go over here system SSH connections there's the SSH connection that was generated for this FreeNAS backup server we can edit this you can see what the hosts IP address is if you have some special use case where we put the host IP address but we actually have this other card in here and if each of these freelances had make maybe a storage network you could actually change provided they can talk to each other over that IP they were ping Abell and on the same network you could change it to that IP that is the IP it's going to be destination to to assuming over port 22 and how it's going to get to that particular server so if you have some special use case you can change the host and once you edit it here because this is what SSH key was used or SSH setup was used to talk to that server it is changeable and over here SSH key pairs here is the key pair and where it was saved and if you edit you can see the private key and the public key that was sent over I never share your private key I'm destroying this one after this video is done but these are where those things are kept if you ever have to go back edit change or create a new key pair and start it up but that's it replication once it's up and running it's pretty straightforward and the last thing I'll comment is like well what about all the data I am SSH into that particular machine the one that's at 192 168 3 209 well back here mount backup Nass and there's our SMB demo share all the files are just copied over as far as what it looks like so in the event of a catastrophic failure this is that folder is replicated to this machine and we'll split the screen right here you mine too one six eight dot 3.2 one two one two is the other server you can see the same actually gotta go in the same folder there's the same folders across there so if I delete something here it'll get replicated over and deleted over here and if there was a catastrophic failure of our main system right here if something horrible happened to it all the data from the last snapshot is over here one of two things well this is FreeNAS and if I wanted to I could just create a share and share data back from this server out now the synchronization is not both ways so it's pushing over to this server so if you do open up a share on this server and so on edit sort deletes a file and we can do that real quick you may get some weird errors out of FreeNAS it's not expecting that but ideally it should just be pushing the data back over and what came from the main server should overwrite back on this one that's not ideal you probably don't want to be messing with the data on here that's the destination for it but in a catastrophic failure the data is here and once there's no more data being sent you can just well turn these on and share them out and be able to be back them up and running relatively quick while you rebuild your main FreeNAS and of course you can just reverse the process and pull all the data back from this server to your better faster nicer server but hopefully it is clarified the replication process it's wonderful inside of FreeNAS it's really solid it's really really fast because it's working at the base level yes it will also backup like things like I scuzzy easy vowels anything that's within the data sets that you have will be copied over to the other side so that's an important aspect of it it's really nice the wizard has made it just as you see here really quick to get it up and running but remember all the scheduling and retention and all the policies are really the same as a snapshot so it's all based on snapshots it's just the synchronization is an extension to the snapshot it's not really a separate thing once you have that concept you can kind of mess around with it and go from there all right thanks and thank you for making it to the end of the video if you like this video please give it a thumbs up if you like to see more content from the channel hit the subscribe button and hit the bell icon if you like youtube to notify you when new videos come out if you'd like to hire us head over to Lauren systems comm fill out our contact page and let us know what we can help you with and what projects you like us to work together on if you want to carry on the discussion hetero two forums that Lawrence Systems calm where we can carry on the discussion about this video other videos or other tech topics in general even suggestions for new videos they're accepted right there on our forums which are free also if you'd like to help the channel on other ways head over to our affiliate page we have a lot of great tech offers for you and once again thanks for watching and see you next time

Channel: Lawrence Systems

Views: 44,425

Keywords: lawrencesystems, freenas replication, freenas replication to another freenas, freenas replication task, freenas replication vs rsync, freenas replication 11.3, freenas replication snapshot, freenas replication to local disk, zfs, network attached storage, freenas, storage, zfs replication freenas, freenas (software), freebsd, open source, home server

Id: et7JyacV_hA

Length: 21min 14sec (1274 seconds)

Published: Mon May 04 2020