How To Use TrueNAS ZFS Snapshots For Ransomware Protection & VSS Shadow Copies

tommy here from lawrence systems and ransomware was a big topic in 2018 2019 right here in 2020 and i don't really see it going away anytime soon these attacks only get better more sophisticated and harder to defend against now i'm not going to talk too much about the cyber security side of it in terms of defense we know that needs to be done but that is still a cat and mouse game because as we set up better defenses the offense gets a little bit more skilled and away we go we're going to talk about is how you can protect your files using zfs snapshots here in true nas now i've done this before and i want to do an updated version of it i was just on two ransomware investigations just a week ago unrelated to each other and both times all the volume shadow copies were completely erased as i said these attacks only get better so they go through and they methodically destroy their backups now there becomes a business question of whether or not you should pay the ransom and i'm not going to debate the merits of it but this is how the calculation goes if you have let's say a hundred terabytes of data and you did proper offsite backups how quickly can you get that data back to where it belongs this is a big challenge so while a lot of people just say off-site backups said and done you don't even need to do this video tom i deal with the real world where we have to try to figure out how to put data back and put it back quickly because if the loss of time let's say it takes x amount of days that is the loss of that business will not be up and running so off-site backups may be a way to mitigate things but it may take too long to restore so it would be less expensive actually to pay the ransom and get the files restored faster but i want to show you here in trunance with the zfs snapshot is one how to make the shadow copies immutable as in not accessible by any type of ransomware attack and why you need to keep your password separate when you tie true nas to your domain you know i'm assuming a lot of these now i'm not going to have a domain tied to it in this particular one but if you do the same exact rules for the same setup apply if you have this tied to a domain the one thing i'm going to say is that you do not want to use the same password for your trunas server as you do for your domain controller this is just a bad idea so you should have a separate login that's at least one part that we're gonna talk about again in the beginning before we dive into all these little details let's first if you'd like to learn more about me or my company head over to lawrences.com if you like to hire a sharp project there's a hires button right at the top if you'd like to help keep this channel sponsor free and thank you to everyone who already has there is a join button here for youtube and a patreon page your support is greatly appreciated if you're looking for deals or discounts on products and services we offer on this channel check out the affiliate links down below they're in the description of all of our videos including a link to our shirt store we have a wide variety of shirts that we sell and new designs come out well randomly so check back frequently and finally our forums forums.laurensystems.com is where you can have a more in-depth discussion about this video and other tech topics you've seen on this channel now back to our content now let's break down how this project is set up what i have right here is just a ton of files being dumped between different folders so i'm actively dumping data to our demo that we have this way it's actually doing something and not just sitting idle kind of give you a little bit more of a real world scenario the device of choice here is a true nas mini 3.0 x provided by ix systems and i've been doing a long term review of this it's been working great including under heavy load like it is right now dumping lots of data now one thing i'd mention in the beginning and we're going to talk about a couple features here when you look at like directory services and active directory for this demo i did not build out an active directory controller to this and tie it all together you on many corporate setups this is the type of setup that you have where you have the files may live on the trunas itself and then all the active directory tie into the main domain controller is done and that's perfectly fine and it won't change this demo any what i do want to make sure is when you're looking at these systems and this is where people frequently do this out of convenience and i get it when you're setting up the system accounts and we look at the users specifically user root they'll just use the same root password as they do for their active directory administrator password do not do this this is something that is completely separate and should be separate just because you tie a system over to active directory will not override the root password so it is very important that you have a separate password and don't say well i'll just use the same because that's convenient keep this locked down because the ransomware and threat actors will look through and figure out where things are going and if they have access to this we completely negate all the things i tell you because they'll go in here and they will delete the things they find inside of here and of course everything is the goal of getting you to pay the ransom the way to get you to pay it is for you not to have data and even if you have those off-site backups they know how difficult they are to put back in place now with that being qualified i just did a video the other day on permissions and i'll leave a link to that and we built out just a basic shared drive with a file system here we i'm dumping data actively into it there's just under a terabyte there now and this is just a standard data set created just like the video i did in youtube permissions nothing special about it we'll go here and take a look at it real quick like edit permissions tom lts owner all really basic stuff nothing special here now the share itself there's nothing too special about that either so we're gonna go over here to sharing we're gonna go to window share and here is the shared drive i'm gonna go ahead and edit advanced and this was all set up with default the one thing to make sure though is that enable shadow copies is turned on and what this does is we're going to be using snapshots to pretend they're shadow copies shadow copies are a feature of the microsoft and tfs system to basically have a way to restore a previous version of a file samba which is the underlying tool inside of freenas the underlying service inside of freenas that presents windows shares as if it's a windows server is going to emulate those shadows copies with the snapshots now snapshots and zfs are very clever they only take up the differential of what data was changed it does this at a block level and being able to have zfs snapshots has been around for a long time and been a great feature when you need to restore data now enabling it in shadow copies makes it really convenient because now this will even allow users to do this and obviously this protection not just from ransomware but in general when people delete files is a great service to have that way they can restore things now there's the share and now let's look at the tasks i set up for the snapshots we're going to go over here periodic snapshot tests and here it is and we'll edit it to look at what we did so here is the data set that we're specifically doing the snapshots snapshot lifetime that is how long until they expire you have to make a decision based on how much storage you have or should i keep them for a week two weeks a month however long your data retention will allow and of course your storage budget will allow is probably the bigger determining factor you can come up with an aiming scheme for them i usually leave the default one on there and you have a schedule begin and end when do you want these snapshot services to run you can say only run them during business hours or run them all the time and that's essentially what this is doing here is just basically running them all the time allow empty snapshots well do you need them if there's no data change and this is a option you may need to enable for compatibility with older versions of free and ask truenast if you're going back and forth between them but what they do is if there's no data change it still creates a snapshot now this can be confusing when you're looking at the previous versions because if you are looking at the way and we'll switch to it real quick to give you an idea you go over here and you're inside of windows and you go i want to see the previous versions and it'll list every snapshot even if there wasn't any changes so you're not really restoring your previous version you're just seeing all the different snapshots so it's kind of up to you whether or not you want that in there the reason are so many in my demo here is because i was moving data and have the snapshot set up now specifically the way i have this scheduled is custom and i have it running every minute every hour every day the reason i'm doing that is because well it's essentially a demo that i wanted to show one you can do it by the minute may not be effective it may over task your server but for purpose of this demo so we can move things around every minute means i don't have to wait as long or cut out any part of the video when i show you how we delete and restore things but let's go ahead and look at creating a new snapshot task from scratch just to give you an idea when you're setting these up what it should look like here's the snapshot lifetime we have it set in weeks it's pretty easy they have how you define them so we can even say like keep 12 weeks of it go here 12 weeks auto daily weekly maybe you only want one a week and you can have different types of snapshot shovels you can have like an hourly one and a daily one or a weekly one and then you decide these retention times for maybe you want to keep a monthly snapshot that lasts maybe a year so you end up with only 12 of those but then the more critical things because it depends on how long you need for data retention maybe you want these running every hour that way you never lost an hour's worth of data if someone makes a big mistake or if ransomware attack does occur so it's pretty straightforward when you're setting these up now once this is all done and this is actively dumping data right here let's look at what those snapshots look look like so we go over here to storage and we go over here snapshots and there's all the different snapshots and the reference point you see is the amount of data that the snapshot's using it's only going to reference the change differential and that change differential because i started these snapshots before there was any data of course it's going to be quite big because i've dumped that much data we've currently dumped 978 gigs of data in there so when you're going through these yes they cumulatively get bigger and bigger as the reference data that being said it's still at the block level so there's still a great level of efficiency with the way this is stored so it's not like it's going to just automatically take up a massive amount of dry space because snapshots as i said are working at the block level and only seeing the differential now let's actually break something delete something and go through and restore it and we'll do it first through the shadow copies so let's go here and uh let's see we'll delete all this this is a bunch of the tesla videos that are recorded and one day i'll record more but that's not today so we'll go ahead and whoops these away we'll just permanently delete these shift delete and now i've broken the tesla folder there's no more folder data it's all gone how do you restore it pretty easy go here restore previous versions and uh let's go just before tom deleted something so we'll restore that one there yes from 12 minutes ago and it's going to copy them in there we can actually open these but it's not going to do it while it's doing this hold on we'll just skip ahead while this is finishing files copy it popped open a folder now go ahead and close this successfully restored the previous version so now we can actually go back so i copied everything back in there and you can see any of these videos if i wanted to play them that are about my tesla and there's me talking about tesla pretty straightforward in terms of how you do that and this of course didn't require any non-user interaction or administrator interaction i should say so it's not like the user needed to contact the admin to be able to restore previous version it's the same ability that you have on if this was on a windows server now what about a faster type of restore well this is where the snapshots have a couple different options let's go over here and we're going to look at the snapshots again we look at these and all these different snapshots are from different time stamps so we can pick which one we want and let's say we want one before i destroyed all the data there's two different ways and i haven't destroyed the data just yet but we can clone a data set from here so we're going to click on the little three dots clone to new data set now when you do this it creates a separate data set that is not shared this is essentially for administrator purposes going i need to know of that snapshot what that folder structure and everything look like so i need a point in time that i can restore to and we're just going to go restore to this one right here and here it is this is what's going to be inside of it we can dig around through it we can create a separate share if i need to or i can just go to the command line and i can go through and look at things exactly as they are snapshows are not a pick and choose type thing when you say give me that snapshot of that particular instance it grabs everything within there the snapshot itself is not particularly granular but sometimes you need to restore a snapshot to a cloned data set so i can grab the one thing i needed out of there because sometimes this is a challenge users create for us administrators they delete something or overwrite something and you got to try and do some differentials and figure it out and of course this all being based on command line like this makes it actually easy so you can start doing some scripting to do differentials or just open up a share and use the standard file management tools to manage it however you want but pretty straightforward on how to do that and we'll go ahead and get out of this directory as when we're done we don't really need this wasting any space so we're going to go here and we're just going to go ahead and delete that data set before you delete a data set i actually recommend doing this you take this copy delete data set you have to type it all out so that's why i did the copy confirm that data set's deleted the snapshot itself is not deleted we restored it temporarily to a data set that was able to go through look at it do what we want with it and then we deleted the data set but the snapshot itself is exactly as it was and it's untouched so those snapshots are still there and we didn't break anything in the snapshot system now let's go back over here to windows and look at this shared drive let's go ahead and delete everything in there and i'm going to do it from the command line just because it's going to take too long in windows but we'll just go ahead and purge everything so if we go in uh we're just going to rm rf star just nuke everything in there everything's gone it's all gone and we're gonna wait one minute for another snapshot to occur and we're gonna take a look at things and show you how to do a master store now deleting it or encrypting it it really doesn't matter when we do this restore we're going to revert this time in the entirety of it and if you were to suffer a ransomware attack or some type of mass deletion of files like this like someone rmrf and you wanted to restore it we're going to show you how to very quickly restore that entire point in time right back to where it was and that's what is the next step so after one minute goes by all right i allowed a couple snapshots to occur and i wanted to show something here shared drive file system there's still only one terabyte used because these snapshots all belong to this particular shared drive and they're still referencing data even though there's technically no data we erased all of it and as you see here everything's blank in windows go back over here we're still showing a terabyte of data so we go over to snapshots because we waited a few minutes we can see there's very little just a couple bytes used but we go down here and there are the well actually got to sort by date created there we go there's our one terabyte reference that we're going to restore to so let's go and i know tom nuked it all right about here a couple minutes ago so we're going to restore this one we're just going to roll back and you will get a warning dataset rollback from snapshot please think about this whether or not you want to do it because this can also destroy the other snapshots so we're gonna just go no safety check i know this is when it was good we're gonna do rollback and we'll look over here ls all the files are back and let's go over here windows refresh all the files are back that quick that is a terabyte of data restored in seconds now granted yes these are fast drives and this is a fast system and there are limitations the slower the system it would take a little bit longer but you're not talking about forever uh or hours and usually it rolls back in minutes and the reason why is because zfs snapshots are done at the block file level they actually don't care what exactly is going on in terms of which files or directories are in there what it does is taking advantage of the underlying zfs copy on write transaction model and this is why zfs is such a great file system is this works so well because all it's doing is going back and the data is not necessarily destroyed with a copy on write file system there is a series when you have snapshots turned on there is a series of block pointers within the file system now i'm not going to get too much into the complexities of it you can spend some time reading on all the beautiful things of the way zfs handles it the bottom line it is able to restore things very very fast because of this design so it's not like you're actually copying the files from one piece of it to another you're doing it at the transaction level and we're essentially rolling back the log of transactions which are the snapshots so because we have this turned on when we hit roll back it immediately puts it back to the reference it had for this it's a lot to learn if you want to dive in deep of it but it's also one of the popularity of zfs is based on these type of transaction models that are underlying it that allow for us to be able to restore things very fast so go ahead and do some more reading if you want to understand it better if you just want to trust in the science of it it does work you see me delete and remove files rather quickly you can see how it works with shadow copies it's a really solid system for doing this and as long as you keep your route access somewhere else other than shared password with your administrator it is not very likely that if a threat actor gets inside of your network that they'll have access to it they will do their best to destroy all the data they're going to do their best to destroy everything they find on the share provided you have snapshots and you didn't let the snapshots expire prior to their deletion so if you only have one hour of snapshots and well you waited two hours to go back then you may have a problem but as long as you have a sane and rational way that you keep the data enough retention and that you notice this happen this is a great way to get you back up and running very quickly hopefully this was helpful and hopefully this saves someone from having some of these problems appreciate it thanks and thank you for making it to the end of the video if you like this video please give it a thumbs up if you like to see more content from the channel hit the subscribe button and hit the bell icon if you like youtube to notify you when new videos come out if you'd like to hire us head over to laurensystems.com fill out our contact page and let us know what we can help you with and what projects you'd like us to work together on if you want to carry on the discussion head over to forums.lawrences.com where we can carry on the discussion about this video other videos or other tech topics in general even suggestions for new videos they're accepted right there on our forums which are free also if you'd like to help the channel in other ways head over to our affiliate page we have a lot of great tech offers for you and once again thanks for watching and see you next time

Channel: Lawrence Systems

Views: 24,768

Keywords: lawrencesystems, freenas ransomware protection, network attached storage, nas, freenas, zfs, storage, truenas ransomware protection, truenas, freenas volume shadow copy, TrueNAS volume shadow copy, Truenas VSS, FreeNAS VSS

Id: 8AjuEG_Ug4g

Length: 20min 31sec (1231 seconds)

Published: Wed Nov 18 2020

Reddit Comments

[u/MudKing1234]

love it!

[u/tobimai]

Snapshots are awesome, especially as they are integrated with smb Shadow copies

[u/ThisIsTenou]

!remindme 9h