Mastering ZFS: Easily Auto Snapshot & Replicate a ZFS Dataset - Part A

[Music] hi there guys and welcome to another video so in this video we're going to be continuing our journey delving into ZFS on un raid in the last videos we looked at a few things we saw how to convert our cash drive into a ZFS pool and from there we saw how we could easily use a script to autoc convert folders into data sets and we did that on our app data so each of our containers has its own individual data set now this was always with the view that we wanted to Auto snapshot these as well as being able to replicate that data giving us a backup somewhere else so this combination can pretty much give us a bulletproof server now I run home assistant as a VM on the server and many times I've had an update only to find that my home assistant doesn't work properly well that's a thing of the past worrying about that because I just hit yesterday's snapshot roll the VM back and everything continues and because of replicating this data as well should the worst happen and the drives that make up my domains pull they all happen to fail well I haven't lost my VMS because the data and those snapshots are replicated somewhere else so in no time I can be back up and running with everything restored how it was so having your most important data replicated and snapshotted is a really good idea now a long time ago before unraid officially supported ZFS I even made a video showing how snapshots can beat ransomware and if you're interested in watching that video then you can see that here now if we want to do ZFS replication we've got to have another data set we can replicate at too and that data set should be on a different Zol and so this is why in one of the earlier videos I showed you how you can convert an array drive into a ZFS drive so if you followed along the videos so far you should have a ZFS cache and you should have one drive in the array that's formatted as ZFS so that means if even if our cach drive is just a single drive we can give it redundancy by having the replication back up to the the ZFS formatted drive we got on the array now I'm sure many of you out there by now have set up multi-drive ZFS pools in your server with the data striped across them giving excellent speed and also redundancy should a drive fail in the pool but even for you guys snapshotting and replication is still a great idea because what would happen if you deleted some data accidentally well if you're using snapshots that really doesn't matter you can get it back in seconds now I can almost hear you guys out there saying yeah yeah space invader you don't need to preach to the choir we know ZFS is good so what this video is all about really is making it really easy to use some of the advanced features of CFS because we all want to use snapshots but we don't want to have to do it manually and we also don't want to have to be writing configuration files for things like sanid and we want to have some way of managing our snapshots pruning old ones and just keeping a certain amount well I've been working on this script for a long long time I can't count how many times I've actually Rewritten it I'm not sure if I've drun more cups of coffee while writing it or I've Rewritten it more times than I've drunk the coffee but I've had the script on my GitHub for a while and from feedback I've had from people they asked me to put in things like incremental rsync backup as well as just the CFS replication now what this allows us to do is to still snapshot our source data set but we can replicate it to anywhere we like no matter what the file format okay so that's the end of the introduction let's jump in now and get started so to get started just do a search for space invader 1 GitHub then just click on the link to go to the page click onto repositories here and the script we're looking for is the unraid CFS data set snapshot and replication script and if we click onto it here you can see various information about the script now I suggest you read through all of this and get yourself familiar with the script but for now let's just talk about its key features so what the script will do is it will take automatic snapshots of any ZFS data set that you've got and it will manage those snapshots using a retention policy that you choose now as well as doing the automatic snapshots the script can also replicate the data to somewhere else and it can do it in two different ways it can use CFS replication which is great because this replication is done at the Block Level and why that's important for example say you had a v dis for a VM and that was 1 tbte in size normally if you replicate that or just copy it somewhere else you've got to copy that whole one tbte to another location and that's going to take quite a long time and so what happens then if you fire up the VM and you maybe update say Firefox to the latest dis verion and then shut down the VM well when replication is not done on the actual Block Level that whole 1 tbte file would have to be sent again to the backup location just because something inside it's changed but the great thing about ZFS is because it works on the Block Level it can just see the blocks that have changed and it will replicate that probably just in a matter of seconds not having to transfer the whole of that large file across it will just do the changes in inside the file so in my opinion that's a really good benefit of using CFS replication and as well as that the CFS replication will also replicate any snapshots that you may have taken of the source data set but what happens if you want to replicate to somewhere which isn't a CFS location well the script then if we set it to rsync replication will allow us to take a snapshot of the source data and then replicate that somewhere else and basically this can be any file system at all if your server's got xfs butter FS Riser FS basically any file system at all our sync will allow you to replicate across to there and there's two different modes of our sync there's a mirror copy that will basically just keep both locations in total sync with each other and whatever changes happen on the source will happen on the destination and it'll always be the most up to date but also there's an option to do an incremental copy here so say you have it run every day only the files that have changed since the last copy will be replicated across and it will use something called hard links to create space efficient backups because hard links mean when the data is sent across the original data is hard linked to the new data so it's very space efficient and you can have incremental backups without taking up any more space than just the changes from one backup to the next now the script also gives you the option to back up to the same local server or to a remote server and that remote server can be either on the same network or even across the internet but for that to work you're going to have to have SSH Keys exchanged between both servers now another great thing about this is we can do live data replication so say all of our Docker containers are all running and we want to back up our app data now normally if we're not using a snapshot then we'd have to shut all of the containers down make the backup and then restart all the containers now I'm sure some of you may be running Plex or MB you might be using something like the appd backup plugin you're watching a movie and you find suddenly Plex or MB shuts down because it's late and you've just forgotten that the plugin's running and it's going to make a backup it shuts everything down makes the backup and then restarts everything but with snapshots what we can do is just before the replication the script will basically take a snapshot which is a frozen moment in time of all of the data and then this moment in time snapshot the data in that snapshot is then replicated either by ZFS replication or by the rsync replication across to the destination and that means none of the containers actually need to be shut down they can all continue running because they're not actually writing to the snapshot the snapshot's totally independent of the actual live data so I find this really useful being able to snapshot running systems and then replicate the data afterwards without having to stop anything such as Docker containers or VMS okay so obviously the script's got various safety checks built in and when the script runs it can send notifications to the unraid guey or if you've got notification Agents set up in unraid then those notifications will be sent through that as well now just for fun I put in some audio notifications into the script so the server can play a little tune if there's a success or failure now this does need what's called a beep speaker to be present in the server and if you haven't got one they're just a couple of you can buy them on Amazon I'll put a link in the description in case any of you want to do that but the tune notifications when a successful snapshots done it will play the old Nokia ringtone when the successful replication it will play the Mario achievement [Music] tune and for any errors it will play the Star Wars Imperial March [Music] now obviously I know this isn't for everyone so of course this can be turned on and off now there's some prerequisites that we're going to need obviously we're going to need an unraid version that supports CFS so 6.12 or higher we're going to need Squid's excellent user scripts plugin to be installed thank you Andrew for making that and we're going to need the sanid plugin to be installed because the script it uses sanid and scoid to do its snapshotting and replication so big thanks for steiny 84 for making the plugin so we can install sanid on our servers now this is optional but I really recommend installing the CFS Master plugin and again a big thanks to the author IA saint for bringing us that great plugin as well okay so that's a brief overview of the script so now let's go across onto a server and I'm going to explain what we're going to do and how we're going to set this all up so I'm going to go across to my friend and patreon server here and what I'm planning to do is I'm going to Auto snapshot the app data and all of its child data sets and then replicate that across to another Z Pool which is actually on the main array now let's go back to the main tab here and we can see the ZFS Master plug-in it's already installed here and these are the three Z pools that are currently present on This Server so the cash drive this is a zpool and this is where it houses the app data and we can see all of the child data sets for the different containers are contained within this one data set now if you want to be able to convert all of your app data into individual data sets then please see my other video here so after the snapshotting I'm going to want to make a backup of the app data and I'm going to use ZFS replication to put it here into this data set which you can see is called dis 13 and that's because this Zol which which we can see currently doesn't have any data sets is an individual Drive in the unraid array which we can see here now I recently converted this to ZFS and if you want to know how to do that well see the other video here okay so on dis 13 there's no data sets here and we can confirm this when we click show data sets in ZFS Master there's nothing here so the first thing I need to do is to create a destination data set for where I want the app data to replicate too so for that I'm going to go to the shares tab here and click add share and I'm going to call it ZFS now normally I'd do an underscore as I don't really like spaces in the names of folders and files but CFS does in fact actually support data sets in Linux with a space so for the sake of this video I'm going to put a space and I'm going to put backups space array now another reason for doing this is because the CFS Master plug-in now also support spaces in the names of data sets but there's one caveat to this which we'll come to in a moment so I need to choose the primary storage for this share so as we were discussing earlier this is going to be on the array and it's going to need to be only on disk 13 so I have to select this otherwise it's not going to create a data set so I'm saying the only dis here is going to be dis 13 okay so with that done I can click on to add share and if I go to the main tab now if we have a look on dis 13 we can see this folder which in fact is a data set called ZFS backups array so just to double check it is in fact a data set CU that's very important it must be if we look at the CFS Master plugin here and now click on dis 13 we can see it's listed here as a ZFS data set so that's perfect right so we got our destination sorted out now we want to make sure that we've got all of the plugins needed installed so let's go across to to the apps tab well we know we've got the ZFS Master plug-in installed because we were just using that but if you haven't got it I really recommend installing it so next obviously because we're going to be using a script we need to have the user scripts and again you can see on This Server that's already installed and because the script uses sanid and syid we need to make sure that's installed and we can see on This Server it isn't so I'm going to go ahead and install that now now make sure not to click close wait until it says done otherwise the plugin won't be installed correctly okay great so that's done so we've got what we need installed all of our plugins are present so now I'm going to go back across to the GitHub page here and I'm going to click on here onto the script and I'm going to click raw here and I'm going to select all of this text and copy it I'm going to go back to the server to settings then user scripts and I'm going to add a new script here and just to make it go to the top I'm going to put 00001 and that's go going to go to the top and now I can actually rename it and it will still be at the top and I'm going to call it app data snap and replication and so with the script named hovering over the Cog I'm going to go to edit script and then what we copied onto the clipboard I'm going to get rid of the bin bash here cuz it is already present in the script so now I'm going to paste what's on the clipboard into here and just to stop anything going wrong I'm going to click save changes and I'm going to come back in and click on to edit again right so let's start going through the user variables which we can set so this section here is all about notifications when the script runs by default the notification type is set to all and that will give us an unraid notification that comes up here when there's either a success or failure now some of you out there you might not want to have every time the script completes successfully any notifications so if you only wanted notifications if something goes wrong then you just change this to errors and if you don't want any notifications at all just change it to non but for this video I'm going to leave it set to all now next under here this is the tune notify so if you got a beep speaker set this to yes if you want audiable notifications set it to no if you don't again I'm going to leave it set to yes now now next here this is the source where we're going to do our snapshots and or do our replication from so first where it says Source pool here we need to put in the pool name of our data set so I'm going to open up a new tab here and I'm going to go to ZFS master and we can see the name of the pool here where the app data is and the pool is just called cache now it's always best to look at it through ZFS Master because if I look on pool devices here I'm going to see cachee with a Capital C and the pool name is case sensitive so we want to look at it from here and just to be sure I'm going to copy it to the clipboard and go back across to the script and paste that in as the pool name so next we need the data set name and that's the app data here so I'm going to pop that in here now if you remember earlier I was talking about a caveat with ZFS and data sets that have spaces in the name now now some tools just don't like it and sanid is one of those tools that don't like it because the configuration file for sanid it doesn't like it when you've got a space in the name so if you want to do the automatic snapshotting you need to make sure that your Source data set doesn't have a space in so if it was something like update a new if you've got automatic snapshotting enabled well sanid it will have an error with this because it won't be able to interpret the source data set with a space in the name however if you weren't doing the auto snapshotting and you just wanted to be doing replication then the space in the data set name wouldn't matter and also for our destination data set it doesn't matter if we have a space in the name for that it's only the source one and only for one that we're using Auto snapshotting on okay so going down to next here's the CFS snapshot settings by default it's set to yes but like I said if we didn't want it to do auto snap shots we're only interested in replication then we would set this to no but I think it's really good to have snapshots so I'm going to leave it on yes and then underneath here this is the snapshot retention policy so this is how many snapshots sanid will keep on your server so your server doesn't just get filled up with loads and loads of snapshots over a period of time so I've got this set here to no hourly snapshots s Daily Four weekly and three monthly so basically it won't ever have more than seven daily snapshots it won't have more than four weekly snapshots and it won't have more than three monthly snapshots so I'll be able to roll back or clone a snapshot from any of the last 7 days if I want to go older than that I can choose from any of the last 4 weeks and older than that it would be just from one monthly Snapshot from the last 3 months now you can change this to however you want it to be but I'm going to leave it as the default here and the script it will actually build your sanid configuration file so you don't have to worry about actually building the actual sanid config file it will be done for you now next this section here this is where you're backing up to a local server basically the same server your unraid server that you're already on or to a remote server somewhere else and if it was a remote server obviously you would change this to yes you would need to put in the usern name of the server for an unraid server that would be root if it was a different server that wasn't unraid you would just put the username of the account that you want to be able to log in with and then here you just put the IP address of the server which you want to connect to like I said SSH Keys would need to be paired between the two servers for the script to actually be able to connect to the remote server but for now on this server we are not going to be connecting to a remote server so I'm going to put that back to no but we will come back and do another test on another server where we will be doing a remote destination but for now let's leave it as is okay so the next block of settings here this is the replication type so whether you want to do ZFS replication you'd leave that as ZFS if you wanted rsync replication You' change that to rsync and if you didn't want any replication at all you and wanted just snapshotting well then you just set this to none but we want ZFS replication here here so I'm going to leave that as ZFS and so because we're using CFS replication then we're going to need to fill in here the ZFS replication variables so we can see here we've got the destination pool so just like the source pool we need to fill in the name of the pool which we want to replicate too so let's go back across to ZFS master and this is the pool and the data set which we want to back up to so the pool name here is dis 13 so let's copy that and paste that in for the destination pool and now the destination data set we need to put the name of that in so that was the name of the share we created earlier I'm going to copy it just to not make any mistakes and paste that in so we've got the destination pool here and the destination data set now as you'll see when we run the script inside of this data set the script will create a child data set where the backup will go and that will be based off the name of the source Source pool and the source data set making it unique now next here we've got the syroid mode you can set that to strict mirror or basic for most people you're probably going to want to have it on strict mirror now the difference here is the flag that it uses with Synco strict mirror uses the force delete flag which basically deletes snapshots in the destination which are not in the source so it basically keeps the destination and the source identical having the same and the same snapshots now for some people you might worry about you don't want to delete snapshots that possibly might be in the destination that not in the source so if that was the case you would just change the syroid mode to be basic okay so next this part here we don't need to bother about this at the moment because this is only if we choose rsync replication which we're not going to do at the moment so we'll come back to that on another server when we do some rsync backups so for now now that's everything set up always best just to go through and check everything so Source pool is cache Source data set is app data remote destination no CFS replication destination pool dis 13 and the parent destination data set CFS backups array all looks good to me so I'm going to click on to save changes here so now we could just click run script and run it in the foreground and wait for it to finish but when we do that if we close the window then the script will stop running and the first time you run it it can take a while so I always like to just click run in the background so with it running in the background and we can see here the notification saying snapshots were successfully created now if you want to see what's actually happening in the script while it's running in the background just click onto the show log button here and we can see it's going through and replicating data at the moment so whilst it does that let's go back to the main Tab and have a look at ZFS master and you can see now there's 144 snapshots so why so many you might ask well if I click on to show data sets here we can see basically there's four snapshots for each data set so if we just look at the first one here going across to actions and then snapshots admin we can see here there's the monthly weekly and daily snapshot and this snapshot here is what syco has used to be able to make its replication to the other Zol so let's give it a while and we'll come back when everything's finished okay so now let's have a look at the script and see if it's finished running and we can see it's all finished everything's been replicated across to the other data set and if we click on the notification icon here and look at notices we can see here the CFS replication was successful from Source cache app data to local destination dis 13 ZFS backups array to the data set cashore appdata so it's built that name basically from The Source data set and pool name so let's go across and have a look at that so looking here on the pools the snapshot numbers are exactly the same that's because we've only got one data set that's been snapshotted but if I have a look here we can see each data set has got its four snapshots and that will be exactly the same in our backup destination each each one's got four snapshots now this is zero here because we backed up into a child data set which we created here from the name of the pool and the data set name so now if I go to the shares tab here and I look inside the app data well we can see everything here as it should be and if I go to our ZFS backups array here we can see inside of here we've got that backup that's being created and inside there there's a total duplicate of everything so the reason I've got the script like this is because we can have our backup location and you could have multiple scripts that are doing different backups and it can put it all in the same parent data set it will just create child data sets based off the name of whatever you want to replicate okay so we know the scripts doing what we want so let's go to settings now and back across to user scripts and now we want to set this to run on our desired schedule now I'm going to scroll down through here and just see if there's anything else set to auto run well this one here is running at 3:00 a.m. here so what I'm going to do is I'm going to click onto this drop down box here and go to custom and this is where we put in our custom schedule now the easiest way to fill this in is open a search engine and do a search for Chrome tab Guru and click on this website here and we can see here 54 and then triple asteris is will run every day at 0405 hours now this is a 24-hour clock so I had something running around 3:00 a.m. so I'm just going to going to make this run at two I think so for the minutes I'm going to set that to zero and then for the hours obviously two so now 02 and then triple asx's is going to run at 2: a.m. every day so let's copy that go back to the user scripts page here and I'm going to paste that in here so with that done I'm going to scroll down to the bottom and click on to apply and so now every day at 2 a.m. on this server is going to Auto snapshot and then back up everything that's changed now future backups will probably just take a matter of seconds so it's very efficient and should anything happen not only we can roll back individual snapshots of the containers through the CFS Master plug-in here but should this whole Drive fail well we've got a complete backup including all of our snapshots on dis 13 okay so now let's jump forward 48 hours and if we go to ZFS Master now we can see our snapshots are up to 216 and if if we look in dis 13 here which is our replication if we look at the snapshots admin here we can see now because it's been running for 3 days we've got three daily snapshots now this will go up to seven and then the retention policy will start pruning snapshots okay so really that brings us to the end of this part of the video where we've done ZFS replication and in Part B of this video we're going to be looking at replicating to a remote server and as well as that we're going to be taking look at the rsync replication and using incremental backups well guys I really hope you find this script useful and I really hope you enjoyed the video now if you did then please hit that like button and share the video with anyone else who you think might find it useful now if you haven't already subscribed to the channel then please subscribe now as always I want to give a really big thank you to all of my patrons and supporters out there it really does mean the world to me because without you guys guys I wouldn't be able to make these videos anyway guys it's getting late here so it's time for me to go but whatever you're up to for the rest of the day I hope it's good and I'll catch you in the next video

Channel: Spaceinvader One

Views: 19,278

Keywords:

Id: RTMMPHc9OoU

Length: 29min 13sec (1753 seconds)

Published: Wed Aug 23 2023