rsync | The most powerful backup tool you're not using

foreign [Music] welcome to another Nielsen networking video in this video we're going to talk about one of my favorite Linux command line tools and in case the title didn't give it away the name of that tool is rsync and rsync is freaking amazing and you will love it I promise it will change your life now how is it going to do that RC can be used to efficiently transfer and synchronize files between let's say the local just directories you have a machine you want to synchronize some directories or let's say you have a remote server let's have a Windows server that you need to back up to a Linux server no problem arson got you covered you need to back up you know the three different servers three different directors on three different servers RC has you covered uh it doesn't care if it's local file system remote file system um et cetera et cetera there is some configuration I'm gonna let you and say you just push a button to make this work but I'm going to show you that in this video how you can utilize rsync for those things um and I hope to help you unlock the potential of our sync and that's kind of the point of this video to show you the most I guess you could say essential ways I believe you can be using rsync uh just to name a few we're going to talk about how you can synchronize you know those remote file shares whether they're whatever operating system uh how to automate the process of synchronizing those you know leveraging some scripting and some cron jobs uh and much more I don't want to go all into it on the introduction but that's where we're gonna head we're going to start with simple and by the end we're going to show you the full-blown how to automate it so without further Ado let's get started all right to start we need to get to the command line surprise surprise and just to give you a quick lay of the land here for this video I will be using two machines I'll be using my Kali Linux machine right here uh and I will be using a Windows 7 machine as the remote Windows machine again you don't have to use Kali Linux you don't have to use Windows 7 you just use anything that supports rsync and they actually even believe it or not they ported it to Windows so you can use it on anything it's just a lot it's in its native environment which is UNIX it works much more efficient efficiently and that's kind of the whole point so that said I have gone ahead and set up a mount point for that Windows machine it is now mounted here you can see that but I go in here and doing an LS and there you go there it is we can even go a little bit further to see that there is stuff down within those directories because we will be syncing it and then for the local backup we have a directory known as backup inside it I have a Kelly local folder which has nothing in it right now and I have a Windows folder in it which has a lot in right now so let's just get rid of it and there you go so both are and and that's what we want should have done that out of the video but anyways that's what we wanted and we're going to start off with a very basic uh and we're just gonna go to our sink and then what we're going to do is you can do a dash AV and what a stands for in this instance is archive and then we're going to do V which is a verbose so that way it will display what's happening to our screen so we'll go ahead and do that and then we have to pick the source directory which I'm going to go ahead and do home and then I'm going to tell it where I want the destination directory to end up so I'm going to do backup and I'm going to do Cali local so that should take well let's just run it and then I'll show you what it should have done and the first time you run the sync remember it's gonna have to copy everything over so it will take a little while now the second and third and you know on and on and on copies will be much quicker because it's only going to do the Delta changes or the incremental changes so we'll let this thing finish all right and it's finished so let's go ahead and go into it take a look um and let's just go in there and as you can see everything's in there we can check the size so we're showing 1.8 gigs let's just go over here to home um and then okay and do it the ush 1.8 gigs so as you can see everything's synchronized so now what we'll do is over here in the home directory why don't we just go ahead and do uh let's touch test one test two and then we'll go back over here and you can see it's not in there there's nothing under touch and we'll just run that RC command again real quick if I can get to it and let's just run it and as you can see it ran and then if we go back and do a there they are so you can see that's how quick it is all right so now let's run our sync on the remote Windows server to see how quickly it will synchronize the data over there so to do that we're going to need to and remember this works because I have mounted that server under MNT windows so in order to make this work on a remote server you're obviously going to have to take the step of mounting um the other server or technically you could go over SSH but that's a little more advanced so for this one we're going to stick with the mount point so we would do our sync and we would go mmt windows so again this is the source and this is going to be the destination so we'll go ahead and run it and you can see it copied all the files over there we can verify this by going to Windows and windows and the reason I put the double directories here is because I didn't put the trailing slash which I'll show you in a minute which actually is a little cleaner because then you don't get the duplicate Windows slash Windows directory that you're seeing here you would just get windows with the contents inside that you're going to see here so all right so there's the data that was copied over and so what we can do now we can go to our Windows 7 machine and let's make a new directory and let's make a text file just for Google's two and then we'll rerun the sync again so we have those there and if I can remember what I'm doing so then we go back over here and we just run that sync again and it tells you you know the changes it made and then you go over here and there they are so as you can see obviously there's not a lot of sizable data in that Windows volume I I'm copying over but you can see how much it did move over so we'll actually put some more data there in a minute here just to show you the speed of going across the Remote Link so alrighty let's try something else and what I think we'll try is I'll show you the um trailing slash so we don't have this directory anymore so what we'll do is we'll go up and we'll actually we need to go down one more and we're going to get rid of that so yeah all right so we're clean there we're gonna go up and now we're going to do is super simple so you're just gonna go here and what all you need to do is add a trailing slash oops a challenge to the source so let's remember to put that on the source directory if you want it to work so now we'll go ahead and run it and if we go back over here we got our windows again so we go into windows and you go in here and look everything's right in that single Windows directory and remember all we did different was add this little slash there so you go back to original one it's not there and if we run it again check it out now there's another directory so we don't want either of those at this point but you get the point um trailing slash if you want it to be just what's actually in that folder you're syncing don't put one there if you wanted to actually copy the folder and all its contents so that's kind of the rule of thumb for that all right now let's copy something from the Kali Linux box over to the Windows 7 machine and I'm going to go ahead and cheat and paste this in here because I'm too lazy to type that's what we're going to do is we're going to go and we're going to use the same AV archive and verbose and we're going to copy from The Source the home user NN Cali user so this is their home directory and we're going to copy it to the mmt windows which remember is the Mount point for the Windows 7 machines we're going to go ahead and run it all right and we're finally done so let's go ahead and check our Windows machine and see what we got and here you can see here's our directory hidden files and everything um so now you know let's say like I said earlier in the video you only had a backup drive or whatever on the um Windows Server now you've copied over something you wanted from that Linux server boom it gets backed up you're good to go all right so let's do something a little bit different here so let's say now we want a copy but we don't want to include all these hidden files because these took a long time to copy over because a lot of them have cash in them uh and I don't really need them let's say so what I'm going to do is I'm going to get rid of this and we're going to rerun the scan for the scan right I have nmap on my brand I'm going to rerun the sync I love that map too that's another awesome Tool uh well we're gonna do it but what we're gonna do is we're actually gonna exclude so we're gonna do exclude and I'm gonna tell it to exclude anything with a dot in front of it so that way um and then I'm going to rerun it and we're going to go quick because watch there's not going to be any need to scan all that cash scan and copy I should say I'm just going to say scan you'll know what I mean by the end of the video so scan means sync in this so there we go again it's over here and there are there it is with none of the other uh junk we don't want so one last thing we'll do delete it again we'll go back over here and we'll run another sync so I said it right and let's say you wanted to exclude a second you would then do exclude and you would then put in let's just do the archive folder because that's a big one put it in there again and run it again and if we go back over here we'll see it here no hidden files and no archive files or folder I should say so that is great now one last thing you can do you can actually [Music] um use what's called an exclude list or an exclude file I should say I think it's the correct term so to do that we're going to do the following and let me just clean this up real quick and then we'll do it so what I'm going to do is I had already created this exclude file and I decided it made sense to have a rsync folder so here I have this and we'll just go into it we'll use um Nano and as you can see I excluded the archive and I excluded anything that starts with a period so hidden files so we can get out of here now and what we're going to do is we're going to run this command and instead of EX saying exclude and then telling what to exclude we're going to say exclude from and then point to the um exclude list we're going to go ahead and then copy home and then kill a user to windows again and let's see what we got and as you can see no hidden files and no archive but to verify that we'll go over and take a look here so as you can see it's reading that file now just to verify that what we'll do is we'll go ahead and delete this and we'll add the desktop to it so then we can go in here again and we'll add desktop and we'll get out of there and run it again and we'll go and look and no desktop pretty cool huh and you can actually use this command well actually well I mean let me just clean this up you can use the exclude from and the exclude in the same command so you could say exclude from and then in the next oh here let me just show you right after it you will see what you would do is you would do that and then you would just do exclude and put in what you want so that's how you would do it all right now let's say you don't want something that is in the source meaning you know what you're copying to be in the destination so you want it when whatever's in that Source directory if something gets deleted from that you want it to also be deleted from the destination because by default it won't you have to tell it to so first thing we're going to do is we're going to need to run a sync on something so we're just going to sync up um over to our Windows machine again and then what we're going to do is we're gonna go into oops we're going to go in here and we'll head off to get rid of something so let's just get rid of test one and test two okay so those are not there so let's go over here and look on our Windows 7 machine and you can clearly see test one and test two are there so we're going to need to do is go over here oops not there go here and we're going to do is we are going to add if I can find what we're going to do is excuse me we're going to add the delete option in there and we're going to rerun it and then we're going to go back over here and voila test one and test two are gone so that is how you would tell it anything that gets deleted from the source needs to get deleted from the destination clear enough all right so let's say you wanted to have the option to see what was going to happen when you actually ran this command you were a little worried that you might blow something up so there's something called a dry run that you can actually run that will show you exactly that so what I first want to do is I want to run this without the dryer on command and I want to verify it came over I want to verify everything's there because I added a couple extra files for this test so we have dry one two and three in there right so we're going to do now oops didn't mean to do that what we're going to do now we're gonna go back to our next box here and what we're going to do is we're going to go in here we're going to get rid of all three of those files so then what we're going to do is we're going to come back over here we're going to do the same exact thing as a normal sink but we're going to add dry what and then you go ahead and hit enter and as you can see it's telling you what it's going to delete and it will even tell you down here that this is a dry run now if we go to Windows it shouldn't have deleted anything yet which it did it and we can refresh to verify we'll go out and come back in and they're still there so they're safe right we didn't lose anything now let's say we're happy we say you know what we want to get rid of those so you would just go back and you would just simply get rid of the dry run it again go back over there and it is gone so that is how dry run works it's kind of like a check out what you're doing before you blow it up um type of thing and it's a it's a really good eye to do I don't know a really good eye idea to do Sorry video is getting a little long here my tongue's getting tight okay let's check something else out all right so let's say you have automated this and you're not there to watch the screen all the time you are going to want some type of logging at least that's what I would want and guess what RC cast logging so we're going to do is we're going to kind of run the same command and what we're going to do is we're going to actually add in and we'll do it right here space and we're going to add in log Dash file equals and I'm going to put this in the rsync folder and I'm just going to call it test.log all right so we're gonna let it do its thing and it should be done and then if we go over here and go to our sync we should have a test.log and it shows you everything you need to see there very good so if we go over here we have everything that was copied again so very cool that is how you would do a log file pretty straightforward we don't need to spend too much time on that but it is an option you will want to use if you're going to set up you know a Cron job to have this run so you can go back and check to make sure it's actually working all right let's try something else all right and one of the last things I want to show you is how RC handles uh links because this can kind of come back to haunt you uh if you don't do it the right way so normally when you uh let's just look in our user directory here for NN Cali and then user so you look in there and we have a couple symbolic links right one is going to within the folder itself which is cool because rsync should back that data up the other one is pointing to a different directory so let's say you created this symbolic link to point to a different directory that might be on a different volume that you're low on space so you needed to move the data off to another location and then just create a symbolic link to point to it to save space the problem is when you just do a default okay you know I'm being lazy it's not going to back up that data to the location that it's pointing to it's just going to do the link so you're going to get like for like if you go back here and get it back up you go in there and see you're getting a link to um that folder now the problem is what if that folder gets renamed or it's gone right like let's just say a link test is now test so over here your backup copy is hosed because for whatever reason that's not available and that's not good so what you're going to want to do is first you're going to want to rename what you just changed and what you're going to want to do is also go up here and we're just gonna do a fresh fresh run here and we're going to rerun the RC but we're going to add an uppercase l just like that I'm gonna hit enter and then we're going to let it do its thing and then we're going to go and look and now as you can see a link test is showing up as an actual folder go in there and the contents are in there so now if we were to go over here and we even know let's just we're going to rename it right doesn't matter it copied over the contents of that symbolic link as an actual directory so now whether let's say that was pointing to an external hard drive that volume wherever it was it's gone you now have an actual copy of that data so that's kind of critical to know if you don't do the uppercase L and you just do the normal thing it will work 99 all the time as long as that symbolic link is pointing to data that you've actually synced but if it hasn't you're probably up up a creek without a paddle so all right now that we have seen pretty much everything I wanted to show you on rsync and how the uh you know the delete the log files the exclude trailing slash all that good stuff I now want to show you how you can automate this so let's go ahead and go over to our rsync folder and what you would want to do is you would want to create a script and I already have one here so we're just going to go ahead and use that because I don't want to recreate the will so we're gonna go in there and check out what we would do and what I've done here is I've created a super basic RC script what I'm telling it to do is I'm telling it to run you know using the standard AV archive and verbose I'm telling it to put a log file here and I'm telling it to run a sync from Windows to backup windows and then I'm also telling it to run a sync from the home directory to the Cali um backup directory so we're going to do is we're going to go out of here and what I want to do is I want to make sure we get rid of this so I want this to be clean and then I want to go back over here and I want to go to and then I'm going to go to and see what's in here okay so I want to get this empty out as well and yep all right now we have clean directors so what I want to do we'll just verify and what I want to do is go back to our sync and I want to cut the basic just to make sure I have that correctly so source is going to be mmt windows and we're going to put that there and home two okay so that is cool so we're going to do now is we're going to run it and we're going to just run that it's going to go ahead and do its thing we'll give it a little bit here and that went pretty quick so now we should have two log files in here which we do and then what we can do is we can just count one of them we can just count both of them if we want and you could go back and check out everything that just happened okay so we know that worked and let's just verify that actually that's not where we put it we put them both here so if we do LS and we do an LTR we can see here's all the window stuff and then here is all the and then Cal user stuff so it worked perfectly that's beautiful all right so that is how you can create a script now let's say you wanted to take this script one level further and keep in mind you could add as many lines to the rsync script as you wanted you know I only did two things here but you could also back up different directories you could have a multiple Mount points you could do anything you wanted you could have some of them do delete some of them don't delete if the source changes whatever you want to do you could take all that stuff I just showed you throughout this video and add it to this file and then just run it and you know what I would do I would have several scripts that are going to do if you want to do these manually even rename them just for if you're going to run one specific directories like sync create a script that way you don't have to remember the whole code and what you need to put in to get the uh our sync and the switches to work so what I want to do now is we're going to look at the cron Tab and if you don't have a cron tab entry first time you run this it's going to deny you I do um actually here I forgot you need to do an dashl I have a Chrome tab entry here that I created if you didn't you'll get an error the first time you do it what you'll need to do is you'll need to do a Quant tab Dash key and this is set to run every minute of every day forever you could actually change the variable it explains this up to you it explains this to you up here but if you wanted to know more you could obviously go and Google or maybe I'll do a short little video on how cron tab works so what we're going to want to do is you're going to want to exit and save this and now your cron tab is working so if we go to Chrome tab and we do a dash L it's going to list it as this is what's going to work so now if we go here and let's just we'll just go into Windows one actually here while we're waiting we'll go into oops and we'll delete the logs that way when we get the logs we'll know the file should see okay it's been about a minute here and there they are and let's look at them and as you can see they ran at 11 it says it's 11 so it just ran here so now just for double verification we're going to go to backup and let's just do an LT LT R and there's all our stuff and that's how you would automate the current Tab and again you could have multiple entries in the cron tab for different scripts there's no reason you can't have you know basic script number two three four whatever you want set up different increments you wanted it to one run once a week or you wanted a script that's gonna rsync to another server but you only want it to happen once a month you know you can do whatever you want fully customizable and it just works great um and with that everyone I think I'm going to call an end to this video so if you've enjoyed the video uh please do me a favor it doesn't cost you anything and smash that like button um if you'd like to see future content which I hopefully will be updating a little more frequently than I've been able to do subscribe and you'll get to see those videos so other than that thank you for taking the time to watch it I hope you've enjoyed it and you have a great rest of your day thank you

Channel: Nielsen Networking

Views: 4,783

Keywords: rsync, rsync tutorial, linux tutorial, rsync command, rsync (software), how to, rsync backup, linux rsync, rsync command in linux, rsync on linux, how to use rsync, rsync backup linux, rsync linux, rsync tutorial linux, rsync examples, linux rsync server, move files with linux, ssh backup, alternative cloud, linux server move files, cloud computing basics, how to basic, linux server, rsync tutorial for beginners, rsync command options, how to rsync, linux backup

Id: _D7sTx93gZ4

Length: 27min 59sec (1679 seconds)

Published: Fri Aug 04 2023