How to install and setup Snapraid

in today's video I'm going to introduce you to an incredible software called Snap rate if you're looking to safeguard your precious data and add an extra layer of protection to your home server then you're in the right place stick around as I'll guide you through the step-by-step process on ensuring that you can Implement snapbreed on your Home Server what is snap rate and why do you need it let's start by understanding what snap rate is and why it's crucial for any Home Server enthusiast snap rate is an open source software solution that provides data protection by implementing a rate like system called parity raid it allows you to recover data even if multiple drives fail simultaneously ensuring the integrity and availability of your files this means that even if a drive fails your data remains intact now key features and benefits of snap rate snap rate offers several features that make it a standout choice for Home Server setups here are a few notable features and benefits it's non-destructive nature a snap rate operates in a non-destructive manner meaning it doesn't strip your data across drives its Drive can be accessed independently making it easier to retrieve individual files scalability with snapbridge you can easily expand your storage capacity by adding drives as needed without reconfiguring your entire system you can use disk already filled with files without needing to reformat or reconfigure them all your data is hashed to ensure data integrity and avoid corruption if a failed disk are too many to recover then you only lose the data of that disk if you accidentally delete some files you can recover them that this can be different sizes it can be a 8 terabyte combined with a four terabyte combined with a 12 terabyte so it's very flexible it doesn't lock in your data you can stop using a snap rate at any point without needing to reformat move data or reconfigure anything so what are the limitations of snap raid well Snapper combines features of raid and backup programs to provide optimal benefits however it has its limitations to consider if a disk fails and you haven't recently synced a complete recovery may not be possible specifically you may not be able to recover files change see or delete it since the last sync even if they're not on the failed disk Snapper is best for data that rarely changes that's why it's perfect for a home server or like a media server so here are a couple of other limitations no real-time recovery snappery does not support real-time recovery meaning you may need to temporarily stop working when a disk fails so that you can start the process of recovering that failed disk limited recovery snap break can recover data only from a limited number of disks whereas with the backups system you can recover from a complete failure of the entire disk array so if your entire Discovery fails let's say eight or like half of your disks fails at the same time you won't be able to recover data that's why I suggest you always have a backup of all your data in another server in another computer or in a or in an external drive limited data saved snapdrit only saves file data timestamps Sim links and hard links permissions ownership and extended attributes are not preserved despite these limitations snap rate is a valuable option particularly for data that changes very rarely before diving into the setup process it's essential to plan your sniper configuration determine the number of data in priority drives you'll need based on the storage requirements and the level of redundancy that you desire we recommend having at least one priority drive for your array this will be equivalent of a conventional raid 5. if you want more protection you can choose two parity drives this will be the equivalent of raid 6. now it's time to roll up our sleeves and get into the setup process we're going to go to our previous VM where we left off in the last video so we have merger Fest already installed and setup on that VM this combination of merger fs and snap rate is perfect for a Home Server so let's dive in first thing that we're going to do is we're going to connect with SSH to our VM we put in the password so let's look at our drives for that we're going to use lsblk command all right so we can see that we have three drives with three partitions here one two and three it's important to know this because when we're going to set up the snap rate configuration we're going to need the paths to this drives we're going to clone the git repository of the author of snap rate this is the command okay as you can see git is not installed on our VM so let's go ahead and install git we say yes to this since snap rate is not an application that is already compiled we need to install some development tools on our VM so that we can compile and install snap rate one of them is git which we just installed so the next one is a development tool group of packages and to do that we are going to use the following command okay so we say yes to this it's going to install compilers libraries and many other packages that we're going to need to be able to compile snap rate okay so all of the packages are installed now let's try to clone again okay perfect so now if we list here we can see that we have our snap rate folder and to make life easier I'm just going to paste the command that is going to do everything for you this command will be in the description of the video so I'll explain the command we're going to CD or change directory into the snap right folder we're going to execute the autogen script to prepare the compilation and then we're going to execute the configure command which is going to configure all the settings for the compilation then we're going to execute the make Command which is actually going to go ahead and compile a snap rate and then we're going to use the make install command to actually install snap rate on our server okay so it's going to run through everything that it needs now it's compiling this might take a little bit longer depending on how powerful your server is but this is a small VM with four cores and as you can see it was pretty fast to compile so it looks like everything finished properly let's see if a snap rate is installed and let's check the version just to make sure that everything is correct correctly installed for that we're going to type snap rate as you can see it's in my path already that's a good sign and then we're going to do version and as you can see it's not ready is installed perfect let's clear our terminal now okay so there's a couple of things that we need to set up before we can start using snap raid the first thing that we need to set up is the configuration file I like to put it just on the Etsy directory it's just easy to find and easy to remember so that I don't have to dive into like many folders and try to find the configuration file so we're going to create a new file called snapgrade.com for that we're going to use Vim again we're gonna go to the Etsy directory and we're gonna call it snap raid.com all right we hit I to insert new content and I'm going to copy the configuration file that I already use on my production server and I'm going to put this on the description of the video too so that you can copy it and then use it as a template for your server at home so let's dive into this configuration file okay so the first section of the file actually let's save first just as it as it is let's open it again okay that's a better format it looks better okay so the first part is where we're going to configure our parity drives as you can see here I have one dash parity two dash parity meaning that in my production server I have two priority drives now so we have to give it the path to the Mount point where our priority Drive is mounted so let's go and find that out first let's exit from here and if we do the command d f Dash h okay so we can see that our priority drive one is mounted into slash MNT Priority One so let's copy that path let's open a configuration file again since we're only going to have one parody we're just going to delete the second one and we're going to replace the path here I think it's very similar to my production server so that's how you want the configuration file to look like for your parity basically this parity this snap rate dot parity file is all the parity data of your array so we're just telling with this configuration file we're telling uh please store it into this drive now remember that this drive is not part of your merger FS um array so only this file the snap rate.parity should live in your priority Drive no other file should be there so now the next thing that we need to do is we need to configure the snap rate dot content file so as you can see as you can see here you can read the description of the configuration it says Define the files used as content list you can use multiple specifications to store more copies you must have at least at least one copy for each priority file plus one they can be in disk used for data priority or boot but each file must be in a different disk format content file path so basically the snap rate.content file is the file that is it's kind of like a database of all uh where are all your files located so this file is very important too like the snap rate.parity file so what I do is I usually choose I usually choose a location in my in my boot drive and I also choose other two other locations in a random disks on my array so this way I have like a triple uh fail safe so if two drives is if disk one and disc two they both fail and I lose the snap rate dot content file I'm gonna have a copy in my boot drive and the same thing the other way around if my boot Drive fails then well you're in more trouble but you will also have a copy on your other drives and since we're using merger face and snap raid we have the ability to just take one drive out of the array plug it into a USB converter and we can actually read the file system and the files that are that are in there so that's uh also very flexible and a great uh feature of snap rate and merger FS so for this if we go back to our Drive let's save our configuration file if we go back to our regular data drives we have it here in slash Mount disk one and slash Mount MNT disk 2. so those two already match but you would need to put what you would need to put the equivalent on your server if you configure your Mount points differently you need to you need to change the configuration file so let's go back to the configuration file and here we can see that slash varsnap rate.content MNT disk one MNT disk two so for this example we don't need to change anything but you will have to change this if your home if your Mount points are different so now the third part of the configuration file is you need to Define your data disks so here is where we are going to Define your data disk if you have nine disks like in my production server you will have to put all nine of them for this example we only have two drives so we're only going to define the two drives that we have we're going to erase all the drives and just leave disk 1 and disk 2. all right now the third part of the configuration file is exclude hidden files and directories if there are some files and directories on your array or your data disk that you would like to exclude from the priority calculation and from the protection that snap rate brings here's where you would put those directories in this case I exclude the lost and found just because it's um it's information that I don't need to be synced this is the lost and found is um a feature of the Linux file system xfs and I don't really care about that data so I just exclude it so with this now we have good configuration file we configured our parity we configured our content files and we configured our two disks that we have and we also excluded some files and folders that we don't really care about syncing or protecting all right so let's save this so with the configuration file set now we can start using snap rate so the first thing that we need to do is to make sure that our configuration file is good and to do that we're going to do a sync with empty drives to do that we're going to type the command snap raid sync so if we analyze the output a little bit we can see that there's a couple of warnings there saying that warning content file not found trying another copy and it's going to look for those files since this is a new array those files don't exist so this warning is completely normal by doing a sync we're basically protecting our drives it's going to check all the data that changed in your drives and it's going to recalculate the parity for you so it's very important that you sync your array every day if you want to protect your your server for everyday changes or every week if you really don't add much stuff to your server and the files change very rarely in my case I do it every week because I don't add or remove files very often from my server and it's perfect it's been working perfectly fine for me but you can schedule it to run every day we will see that in the next part of the video where I will show you how to automate the process so let's add some files to our drives and let's see how the calculation and how snap rate works now let's go let's go ahead and add some files to our drives I'm going to navigate to the slash MNT storage remember this is our merger FS bound point and as you can see we have our snap rate.cont file and I already have a couple files here but let's create more files so we're just going to do vim and then we're going to do file two we're going to add some text here and just so that it can have some data it doesn't matter what it is it's just an empty file so whatever we can now maybe copy that file create a new file three let's say okay so before we do a sync we can do another command called Snap rate diff okay basically what this command is going to do is going is going to scan your disks very quickly and it's going to see if there's any changes so if you added or removed or deleted or moved one file from one disk to another or whatever this command will give you the output and you'll you'll know how many files were changed in your array so it's just handy information if you want to use it now let's do a snap rate sync to sync all the new changes to the array snap rate sync okay so that's great this is an example of the output that you would see every time that you sync your array so let's see what it says so it says everything okay okay saving the state of your array to the snap rate.content file great verifying the snap rate.content files that's perfect so it wrote some data there and let's see what it says here so it says loading state state from VAR snapper.content scanning data 1 and data two resizing saving State verifying okay so it goes everything it says syncing 100 completed in zero seconds so this was actually pretty fast because the files that we have in there are very small but if you have large movie files there are like eight gigs 12 gig files and like there's many changes to the disk array this process of syncing can take up to a couple of hours depending on how many files and how big your array is so this is why I like to automate this process so that it just it runs in the background once a day or once a week and I receive an email notification telling me that the sync was successful or that this thing failed so now that we have snap rate up and running and set up and we did a sync and everything is working so now it's time to automate for the automation part we're going to use the script already created by someone and this script is called sniperate Runner we're going to clone the git repository for this script too and this is the command for it I usually like to store this in the root directory of the root user so we're here if you do PWD is slash root I have a couple files here already we have our snap right folder that we used previously to complete and now we have our snap rate Runner automation script so this snap braid Runner automation script also has a configuration file that we need to configure so let's go ahead and do that so the first thing that we need to do is we need to rename the snapgrade.com.example to just snap rate runner.com so we're going to copy that and we're going to do a move and just leave it like that snap rate.com perfect so if we list again we see that now we have our snapgrade.com and our snap rate Runner dot py script so let's open the configuration file so this is the snap rate automation configuration file first thing that we need to do is path to the snap rate executable where is the executable file from a snap rate in this case it just says executable equals snap rate but we actually have to give it the full path so we're gonna exit out of this configuration file and to know where snap rate was installed on your system this will be different in any system so if you're following up with the tutorial and you're using Rocky 9 it will be easy to find out but if you're using Ubuntu or um Santos or other or Debian or any other distribution and there's a handy command that you can use to find out where the executable is actually located this command is called which so we're going to do that which and we type the command snap raid okay so it's telling us that the snap rate is installed in USR local bin snap raid so we're gonna go ahead and copy that we're going to open our configuration file again conf and we're gonna paste that here USR local Bean snap rate perfect remember where we put our configuration file for SNAP rate yes we put it in slash Etsy snapbreed.com so we need to also give that path here slash Etsy snapgrade.com perfect next thing to configure is the deletion threshold this threshold is set so if there is an amount of files that get deleted randomly is going to prevent the automation script from creating a sync this is a fail safe in in the case that you lose the drive because if you lose a drive and the automation kicks in and then it creates a new sync basically what's going to happen is it's going to create the new parity data without the drive that failed meaning that after that if that happens you won't be able to execute a recovery for this drive so it's very important for you to set up this threshold properly I scanned my drives in the past it's there are four terabyte drives and in average they have around 28 to 30 000 files each so the threshold that I chose for my disk array and for my drives is 25 000 files this means that if I lose a drive the snap rate automation is going to detect that there are more than 25 000 files that are missing and is going to stop automatically it's going to send me an email to alert me that there are more than 25 000 files missing from your drive array and it's more likely if if this happens it's very likely that you lost a drive it's very important for you to set this threshold properly okay so the next part is the log so every time this automation script is going to be executed we want to log all the output of the script so that we can have a trace if anything goes wrong I like to store this log just in the root directory just it's easy to find and quick quick to find so I'm going to set that up now I'm going to go here and we're going to set this for root snap rate.log perfect Max size 5 megabytes I don't think it's uh you can set this up for more but five megabytes the default is pretty it's pretty good it's going to do a log rotate so you're always going to have the latest logs from the latest syncs so now we have the configuration for the email for this you I use Gmail as my sntp provider but if you have any other sntb for Rider then you have to set this configuration here if you want to set it up with Gmail the same way I did you need to create an application password this is created on your Google account and you're going to have an a specific password that will allow this script to use the sntp and use your Gmail account to send yourself an email I'm going to show you how I set it up here so that you can follow the same way with Gmail so the first configuration for here is send on success and send on error and this is going to send an email if the sync is a success it's going to send you an email telling you that it was successful if it's an if the Sync has errors it's going to also send you an email telling you that there were some errors short true here you can change this if you want to have the if you want to have all the output like the entire output of the sync command you can set this to false but it's going to be a really big email that you're going to receive so I just leave it to short I just really need to know if it was successful or not and it's going to tell you how many files changed and basically the important information so I just leave this as true and then from I you just need to put your email at Gmail and two you want to send yourself an email so from your email to your email host is SMTP smtp.gmail.com the port is 465-465 here I leave it um TLS I leave it false and then SSL I leave it true Google requires this to be able to send and receive emails and then the user is going to be your email and the password is going to be the password of your application so Gmail application password leave a comment down below if you want me to show you how to create a Google application password for this so now the next part is scrubbing so scrubbing is basically a feature of snap rate that from time to time or whenever you want it's going to read all the data and compare the data of the array with your parity data this is kind of like a sanity check to make sure that all the data in the parity file is correct and there hasn't been any corruption here is disabled by default but I usually run it from time to time and I run it every time I sync so every week to make sure that the data that it's there is not corrupted and it's actually correct if you lose a drive you're going to actually be able to restore that information and it's not going to be corrupted information so I set this to true here I usually do 25 and what this means is a percentage of the drive array I want to check 25 of the array but I want to limit the the scrub to only the files that are older than 10 days this is because I don't really need to scan every single file across all the drives that takes very very long I tried it once and for my array it took around eight to nine hours just to be able to read all the information from the arrays so it's not really necessary but if you really want to make sure that every single file on your array is not corrupted then you would set this to you will set the plan to full and then this you would just comment this file out this way with this configuration you're telling the automation script to every time that you're going to do a sync it's going to scan every single file and compare it to the parity data and to make sure that the the data is not corrupted for this tutorial I'm just going to set this to 25 of the disk array or not not all older than 10 days I think that's a very healthy setting for this okay so now that we have this all set up we're just going to save so before we set up the automation let's execute the script to make sure that everything is working I set up the email I just put a dummy email so if I execute the script right now it's going to fail so I'm just going to go back and change the the email configuration to false so that I don't get any errors in the execution of the script so to disable the email you just need to comment out by putting a semicolon on all the email and SMTP configuration and then we save so let's execute the script and check out the output okay as you can see the script runs a div command first it checks if there are differences as you can see here it says that there's no differences because we didn't add or remove any files from the array and in this case it's not going to sync it says no changes no sync required and then it's going to do a Scrub but since no files are older than 10 days there's nothing to scrub now their script is working now let's automate it to automate it we're going to configure a Chrome tab if you don't know what a cron tab is is a built-in automation scheduling task tool from Linux it's very handy and it's great so we're basically just going to schedule the execution of this script to run every day or every week so I'm going to open the Chrome configuration and we're going to set up our Chrome job for that we're going to go to slash Etsy Chrome tab okay so we open this file and I'm going to paste here my configuration that I have for my production server I do it once a week so if you want to do it once a week this is the configuration that you need to put so this is the configuration basically this is where we're scheduling it I'll explain how this works a little bit better if you want to do this every day we're going to use something called a cront tab calculator it's going to help you understand how this works so the next part is we're going to tell it to use a root user we're going to use Python 3 because the script is run it's written in Python we're gonna tell it what is the python script that we want to run and then the dash C is to tell it what configuration file do we want to use when running this script basically our snap braid Runner configuration file this is set up to opt but as you remember our snap rate Runner script is not in opt it's in root so we need to change this to root if you want to store this Snapper Runner folder somewhere else you can put it anywhere you want but you would have to change this configuration here okay so let's save the changes okay we save that and as easy as that we configured a scheduled task that is going to run the automation script and it's going to check everything on your array using snap rate and it's going to create a sync if it needs it's going to scrub and check for the data if it needs and it's going to send you an email if it fails or if it's successful now if you want to set up this uh every day let's open our browser and we're going to look for something called Cron tab calculator account calculator the first one is fine so this is a handy application that will help you create your scheduled jobs let's put this back to normal and let's grab the current one that we have if we open our Chrome tab again say that it's zero one okay we just copy that and paste that here and as you can see from the output of this it says that it's going to run at the 1 am on Friday so every Friday is going to run it at 1am I put it at 1am because my disk array is quite big so sometimes it takes very long to complete so it starts at 1am on Friday Saturday morning I wake up and I see an email from my automation telling me whether it was successful or if there was any errors now if you want to run this every day the only thing that you need to change is you need to change this five for nothing just a asterisk and basically what this means is it's saying that at one meaning that it's going to run every day at 1am and that's pretty much it so if we go back to our configuration file our current configuration here we can just come here remove these five put a star save our configuration and now the automation is going to execute every day at 1am congratulations you've now learned about the power of snap rate and how to set it up in your Home Server thank you for watching if you find this video useful don't forget to leave a thumbs up and subscribe to the channel for more content like this if you have any questions or suggestions feel free to leave them in the comment section below and I'll see you on the next one

Channel: Distro Domain

Views: 2,375

Keywords: how to install snapraid, snapraid 2023, how to setup snapraid, home media server raid, raid vs snapraid, snapraid vs raid, rocky linux snapraid, how to configure cron job, how raid works

Id: QFGEKh1A90I

Length: 31min 51sec (1911 seconds)

Published: Mon Jul 17 2023