RoboCopy: Your Data Copying Ally for IT Admins

[Music] what is robocopy well robocopy is the swiss army knife of every admin who needs to copy move or sync large volumes of data from one location to another when any of us went to our command prompt and we typed in robocopy forward slash question mark and we pulled down the list of parameters for this utility we are like overwhelmed it's like you've got to be kidding me all of these switches and arguments parameters for one utility but when you think about the metadata in your file system you've got time stamps you've got attributes for files you've got attributes for directories you've got access control list auditing information all those parameters simply give you the ability to choose what is copied what is moved and what is synced it's not complex at all on top of that robocopy has great features if i'm moving large amount of data from kansas city to new york data center and i've got a lousy wan link between the two i want reliability and robocopy provides all kinds of great options so that you know and are assured that all of that data in kansas city gets moved to new york reliably on top of that it has a rich array of logging so that when you are moving or copying or syncing data you can go back after that operation and look at your log files and be assured what you intended to happen took place now in order to demonstrate robocopy i have a source directory and i have five files in here and i have one subfolder in my destination folder i have basically empty the reason i'm using five files is so that you can clearly see the behavior of robocopy with its various parameters and see the flexibility and the power that this utility offers now you're not going to use robocopy with five files you're going to use it with terabytes of data in my source folder i have a subdirectory called subfolder and if you open up it's empty in my list of five files i have a powerpoint and if i right mouse collect and go to properties you see i've got it locked down got denied for system and denied for administrators this file itself is going to be a problem with any effort to work with it but we'll see that robocopy gives you the parameters and flexibility even files that you don't have rights you still can manage and work with them so robocopy is pretty straightforward we type in the utility robocopy and we give it a source folder and a destination folder now that could be a unc to a network share that's fine it works across the network just as well as it does with local drives let's go ahead and execute this one and notice it gives us almost a real-time view as the files are copied from the source to the destination down in our statistics we can see directories a total of one directory one was skipped because by default robocopy does not move subdirectories we'll see how to do that very quickly files it had five files in the source four were copied one failed and that was our powerpoint that we have no rights to now the information on the screen is known as the robocopy summary so let's take a look at what happened it noticed the subfolder was skipped so we didn't move our subfolder it did not move our powerpoint out of five files it moved four let's take a look at the behavior robocopy as we execute this again right now we have a number of files in the destination that match files in the source let's execute this again and see what happens notice all of the source files that were exactly the same in the destination were skipped we see four files were skipped so nothing was moved robocopy understood that what was in the source was also in the destination and did not move anything that's exactly the behavior we want keep in mind i'm only showing you five files but we're typically talking about terabytes of data let's drill down into the default behavior of robocopy if we come down to the options we see it simply says star.star all files in the source can be copied to the destination we see our first parameter forward slash d copy now the d copy parameter with its flags alerts robocopy as to what to do with directories we can see the default is d copy colon d a or data and attributes we could also add timestamps but the default is simply d a now the next parameter we're going to look at is forward slash copy and it's got many flags that we can attach to that parameter the default for robocopy is data attributes and timestamps but we can see we could add s for security o for ownership and u for auditing so we can definitely add more metadata attributes in our copying of files from source to destination let's look at my explorer view of the source and the destination notice i've added attribute column date created and ownership and i've done the same in my destination notice the ownership in the source the owner is home bus down in the destination the owner is administration i want to sync that ownership down to my destination so look at my command line i've added forward slash copy and i've added the flags data attribute timestamps and oh ownership let's go ahead and execute it and notice it skipped those files it still recognizes the files in the source are the same as the files in the destination so let's take a look at our destination and notice none of that ownership was updated we don't want to copy a terabyte of data again just to update a metadata or ownership or an attribute so how do we get that information updated without copying the entire file especially when we're talking across the lan and terabytes of data so there's a special parameter called sec fix and it allows security and a lot of other attributes to be updated even though the files are skipped so notice my copy parameter includes ownership i've added the forward slash sec fix parameter which is going to allow this operation to happen even against skipped files let's go ahead and execute it notice the files were not moved the four files that were existing in the destination were skipped let's see if the attributes were updated and i can see down here my destination home boss is now the owner of my destination files no files were copied just the attributes my source directory has a subfolder which i want to get down into my destination so let's look at getting subdirectories from source to destination now there are two parameters that allow me to include subdirectories one is forward slash s which will copy all subdirectories except empty ones and forward slash e which will copy all subdirectories including empty ones mine is empty so i have to use e let's go ahead and execute it and notice files were skipped two directories were in my source one was copied one was skip if we look at our destination we can see that our subdirectory was moved down into our destination worked perfect if you're saying wait a minute mr vanderpool why do we have two directories in the source when we really only have one subdirectory why does it indicate two one copied one skip well let's take a look at the directory so i'm going to pull up the directory information of our source if you'll notice there are two directories one is indicated by the parent or the directory called source the subdirectory is called subfolder robocopy sees these as two directories the subdirectory was copied the source directory was not now we still have the issue of our powerpoint that is locked down and we don't have the rights to do anything with it so we're going to tell robocopy to shift into backup mode this is a special set of apis that it actually will take advantage of the backup api to read and write the file make sure that the credentials you're running on are as either administrator or you have rights as a backup operator there are two parameters for this type of situation one is the forward slash b which moves robocopy into a backup mode and there's also forward slash z that will restart should something happen in the copy process we're going to go ahead and execute this and notice four files were skipped one additional file was copied and if we look down here sure enough our powerpoint is down in the destination as in most source folders our users are going to begin to open these files and modify them the size of the file is going to change maybe the name of the file will stay the same but we're also going to see timestamp information like date modified changed how will robocopy behave let's take a look i'm going to open up my textfile.txt i'm just going to open it up with notepad and i'm going to put and i'm just going to save that so i'm i've got a new size for the file and a new date modified so you can see it's 725. i'm going to go ahead and run robocopy again with no parameters now notice in our summary it shows us a newer indicates a file is newer it copied textfile.txt which we expected and we can see in our statistics five files were in the source one was copied four were skipped because nothing had changed so the behavior that we expected is happened if we go to our destination folder we can see that textfile.txt now has 725 as its date [Music] using robocopy and having a source and destination folder there are times that you may find that your destination folder has files that are not in the source let's take a look at how robocopy deals with that i've created a file called extra bmp and it's in the destination but not in the source let's run robocopy again and see what happens with no parameters just the default run row will copy and if you'll notice i now have in my column called extras i've got one file that's labeled extra so when you see a file under the column extra that means it simply doesn't exist in the source it's in the destination but not in the source now they have a parameter called purge forward slash purge and it allows you to fix this problem of my destination doesn't match my source let's try it so i'm using forward slash purge and notice now my source and destination any extra files in my destination are removed [Music] now another powerful parameter in robocopy is the mirror parameter forward slash mir take a look at my source it's going to mirror the source to a destination if you notice my destination has a lot of different named files differently including subdirectories than my source but i'm going to use the mirror command to just say sync them so they're the same so let's go ahead and execute that and give it a minute and if you notice everything in my destination that was not in the source was changed deleted removed and now i have a perfect mirror between my source and my destination this is really cool now if you like that we can get much better what if we could monitor we're mirroring source to destination and we'd like to add a monitoring capability so after a few seconds it would check the source and destination if it saw changes it would fix that it would mirror those again or after x amount of changes it would then mirror the source and destination well we do we have both of those the parameter forward slash m-o-n colon n n is just a variable you could put i want you to monitor the source and destination and if you see a change two changes three changes four changes whatever n value you want to select it will look for that many changes after one minute it will check again look for that amount of changes and sync the two folders or you could do mot colon and then a variable where after so many seconds it will again look the source and destination and sync the two folders automatically let's start with monitoring two changes every minute so i'm going to go ahead and say and right now if you look at my robocopy summary it's looking waiting for one minute for two changes so it's now monitoring these mirrored folders so i'm going to come up here to my source and just change the name just a jpeg and rather than traffic mov i'm just going to call it t mov and we'll just sit and wait for the time to elapse and then it will monitor the source and destination and sync them up and here you can see we have reached our time boundary it has recognized two changes and went ahead and updated our destination to ajpeg tmov and now they are synced again and again you can see from my summary it's waiting to look at this again and again and again now if you are using this on terabytes of data you're going to have a performance hit on the server so just keep in mind there's no free lunch here it has to monitor every file has to monitor depending on the size and the magnitude of files you're asking it to monitor that could be a performance hit on that server [Music] now let's take a look at robocopy and moving files this is where it gets a little bit frustrating because microsoft's syntax and its descriptions are not very clear and the results are not what we expect first we have the forward slash mov and it says move files deletes them from the source after they've been copied then we have another parameter forward slash m-o-v-e moves files and directories deletes them from the source after they've been copied doesn't work exactly like that so let's take a look at the mov moving files and i'm going to add the slash e for subdirectories now i've added the mov parameter and forward slash e so it will pull hopefully my empty subdirectory let's take a look and if we look at the results my source still has that subfolder subdirectory i thought that was going to be deleted and i come down here and it did move everything it left a subdirectory but it did move all the files and the subdirectories into the destination wasn't what i was expecting at all now let's use the parameter forward slash mov e that's supposedly to delete the subdirectory or move subdirectories now what's going to happen i'm going to come up to my command line and i'm going to leave the forward slash e and add i'm going to set up the parameter forward slash move which is supposed to move all files and directories and i went ahead and left my forward slash e which should move subdirectories now when i execute this it wants to delete the source directory which is called source so what i'm going to do is close out my explorer view get rid of that it's still there i just not showing it and then we'll execute the command line now if i look at my results everything is in my destination if i look at my statistics everything looked like it went let's go back to our source now i was using my drive j and if you look in drive j there is no source directory so when i use the m-o-v-e it goes back and deletes the ex the source directory as well as all the files so my source directory is gone that behavior is not clearly explained at all in the description of the syntax [Music] now let's take a look at the behavior and the parameters available for robocopy one i have my source in one host and i have my destination on another host and a network in between when i start copying lots of files what happens when i run into a congested switch or a router that was rebooted in the middle of a file transfer well robocopy can deal with that very very well let's take a look now in my example my source has got two large ubuntu iso files so they're almost four gigs each so we've got a lot of large files to copy my destination is on a host with a share that has a lot of network in between they're both virtual machines i'm going to unplug the network as we start this process of copying so you can see the behavior of robocopy two parameters that are built into robocopy by default one is forward slash r which is the number of retries on a failed copy the default is a million that's probably way too high something in the order of 10 to 20 retries the second parameter that's by default is the weights per second before you retry forward slash w that is by default 30 again that's way too high in most cases 5 to 10 is probably a reasonable wait time before you retry again what i did on my command line i've got my source my destination retry is not a million but 10 weight per each retry is rather than 30 seconds is 5 seconds now i'm going to execute this and i'm going to at some point pull the plug on my switch to let you see what's going on let's go so here you see i've pulled the switch port out and you can see it's just sitting there waiting and we're waiting for those retries and the weights in seconds between each retry now i'm going to go ahead and plug it back in and you can see robocopy then begins to pick up where it left off even though i literally pulled the jack out of the switch port killing the network after plugging it back in everything's all the network elements settle down robocopy sits simply picks up where it lefts off and it's off and running even as drastic is pulling the jack out of the switch pretty nice here i took the destination host and jerked it out of the switch so it's no longer on the network and again you can see it just stopped it's trying to figure out what to do it's got some wait times so there you can see how so it killed my art remote desktop connection let's see if it pops back up so my remote desktop connection came back and again you can see remote you can see robocopy picks right back up and continues on as if nothing happened this is very very for a network administrator this is essential it works very well so here's my results my summary is all the files were copied except one file which i don't have rights to it was called no access.exe all other files moved even though i pulled hosts completely out of the switch and plugged them back in once the network settled so we had 12 files on the source 11 were copied one was one failed because of ntfs security everything else went fine you can drop down and look at my speed 36 million bytes per second 2 million megabytes per minute now there is additional features that can improve performance there are two additional parameters that can be used across the network one is the forward slash j which is copy using unbuffered io if you're copying large files like isos that can improve performance another parameter is forward slash no offload and it disables the windows off copy offload function now there's so much to cover in robocopy things like forward slash ipg which allows you to tweak the rate based on a slow lan link which can improve that transfer rate or forward slash mt which allows you to actually adjust the multi-threading capability of robocopy the default is eight threads you can include changed files you can exclude change files you've got all kinds of file selection options no end you can copy files based on size you can copy files based on age and it has arguments and parameters to deal with those pesky junction points it even has some parameters to deal with your destination disk if it has insufficient disk space great logging options so you can log to files you can append your logs you can overwrite your logs you can control what is put in the logs you can actually have your summary displayed as well as logged simultaneously another feature is if you have complex robocopy operations you can save that whole operation as a job file and it gets an extension rcj in our video notes we have lots of links to some great online guides tutorials that we felt were just excellent concerning robocopy take a look at the video notes they're free just download them you'll see some great resources on robocopy [Music] [Applause] [Music] you

Channel: TechsavvyProductions

Views: 18,661

Keywords:

Id: 099Df3jCPBI

Length: 24min 28sec (1468 seconds)

Published: Tue Jul 26 2022