Backing up your Docker Configurations and Data.

[Music] foreign [Music] Source Advocate and I'm back with another video and today I wanted to talk about backing up and saving all of your Docker and Docker volume information I get this question quite a bit people get things going in Docker and they start realizing okay backing things up isn't quite as easy as I thought it would be there's not really a great utility out there for doing this there are some open source utilities out there that are kind of made to do this but none of them really do exactly what people want I think so I wanted to talk about my process and kind of what I do and I cover this in a lot of the videos that I make so it's really not a lot too and I'm going to go through a quick presentation just because it was easier for me to put it together this way and put my thoughts together I want to say thank you to all of my subscribers and all of my patrons over at patreon seriously you guys make this so worth it for me to do these videos every week I really truly enjoy it and I just can't say thank you enough if you're enjoying these videos subscribe let YouTube know that I'm doing a good job by subscribing to the channel plus you'll get notified when I have new videos coming out and finally if you're enjoying what I'm doing give it a like just click on that thumbs up and that way YouTube knows that you like it and they'll pass it along to other people that might enjoy my content as well I really appreciate it thank you again let's get started so really there's two things you want to worry about one the container itself and then two the volumes or the persistent data that you're trying to keep constantly and I think really that second was the more important of the two because you can always pull down the image and restart the container but having the data and the volumes in place is is kind of the hard part and then sometimes the environment variables and some of the special configuration you have in the docker compose file is also useful or makes it easy to bring things back up really fast so really making you know the container itself backing it up isn't necessary unless you want to have that specific version of that container and replicate that across a lot of different instances for some reason in which case there's instructions on the docker side of how to create a container and then create and then turn that into a new image that you can then just kind of replicate so if you had a special way that you set things up you can then create that into a new image and run it that way but really if you have your Docker compose and your and your Docker run command saved and you have all your environment variables set up and you can set up your your new machine the same way you had before it's really easy to bring things back from from the dead basically now volumes that's the big one that you really want to be able to back up so whenever whenever you've got that persistent data you want to set that up as volumes or binds where you're binding that volume to something on the host system and then once you've got that you can take that and just create a backup like you normally would of that data and then name it so that you know what it is and whenever some if if something ever happens which hopefully it doesn't but if it ever does you've got that you can bring it back and if you're doing that on a daily basis that just makes it even better if you have if you have data that's changing a lot maybe an hourly basis is even better but there's some some got just to backing up the volumes so I have a top level Docker folder and then under that I have a folder for each application or service that I'm running and then inside of those I have their different volumes that I'm mounting a lot of times so you can see these different kind of folders in this different kind of setup that I have for each of these applications that's really how I do it because then all I have to do is take this top level Docker container zip up everything inside of it and then back up that zip file to some other off device kind of storage medium that could be USB that could be another computer on your network that could be a Nas that could be cloud storage it's completely up to you how you do that and where you do that but getting it off that machine in case there's a machine level failure is kind of your best bet so I've talked about kind of what I'm doing and that I'm zipping that up and just moving it off that's really what that's about so there's some things to keep in mind it's always best to stop a container before you back up its data it's very important that you stop that container before you back it up um now you can do this and at night so at night nobody's using my my containers these are just for my home use for my family everybody's asleep in the middle of the night so I can run this in the middle of night and feel fairly confident that nothing was being changed or written and not have to stop those containers and and usually there's no problems but I definitely need to check those things to make sure there's not corruption once in a while um but the best thing you can do is stop the containers because whenever you have databases like things with mySQL mongodb postgres SQL any of those things which a lot of these applications have if they're writing at the time that you run your backup you could get corrupted data in that backup so that's not what you want so if you stop those containers nothing's going to be writing that stuff that volume is going to be sitting there static and you can go ahead and back it up with pretty good confidence and it's going to be a good backup um this could happen to anything actually just writing of the file system this could also happen but file systems tend to have pretty good Protections in place to keep those kind of things from happening but it still could happen so just be aware of that foreign so the good news is that if your containers like mine are dead in the middle of the night you can probably just run the backups without having to stop everything and it's going to be fine but if you have any reason to believe that it's a time you're running your backups it could be in use or something could be writing that data it's always better to stop it and and that could be scripted it's not hard so I'm going to go through how to do that so we're done with the presentation yay that's the part that everybody doesn't want to go through so I've kind of run the tree command here just so you guys can see what I've got so you see I've got this Docker folder and then below that I've got Dashie and there you go so I've got my icons I've got my public and then I've got home box and then I've got the homebox data so you can see each application that I'm running is all inside of my docker and then I've got those volumes mounted if they need one to be mounted so some of these are just testing things so I don't have anything mounted right now but the ones that I'm really using I've got the volumes mounted so when I've got that done I can go back here and I've got this uh let me let me clear this out in fact so that you guys can see this um I've got this little bash script that I've written so we're going to look at that guy you always want to start with this shebang slash bin slash bash okay that just tells the system hey this is going to be a bash script the next thing I do is I grab the date from the system and I format it and it's just the year hyphen month hyphen day format that this percent F does but writing this out correctly is important so I have this in the show notes it's dollar sign open parentheses and then no space date one space and then plus and then the single quote percent F single quote and close parentheses so there's no space between the plus and the and the first uh single quote there so just be aware of that if you put spaces in it it will not format correctly the next thing I'm doing is just echoing out that backup date for when I run it manually just to make sure I'm getting the date the way I expect that's just a testing thing for me you can get rid of that when you're ready and then I move into the folder that I'm wanting to do some things in so remember I said you want to stop these different containers from running and then start them back up so what we're going to do is move into each containers or each applications folder we're going to run the docker compose stop command on each one so the nice thing is if like you have if you have a application which a lot of applications have a lot of containers that run so if you have an application with a lot of containers that are running when you run that Docker compose stop it stops all of the containers with that one command instead of having to write Docker stop you know something Dash app Docker stop something Dash DB Docker stop something Dash you know nginx or whatever it may be running that just helps you this just does it all at once so it makes it a little bit simpler so all I'm doing is I move it to the Dashie I stop it and then the CD dot dot slash means go back one level and then move into mesh Central and then I stop it and then CD back one level dot dot slash into navigrome and stop it in the same way CD dot slash speed trash speed test tracker and stop it now I'm not doing all of them for this video I'm just giving you an idea but for every folder I had I would actually want to go do this and run the stop command just to stop them real quick and then I've got this uh command that says now go back to my home directory because I want to run my zip up file my zip file into that home directory now you can put that anywhere you want but in my case I'm using home and I'm going to run the tar command Dash C which is compress Z which is zip and then V is verbose and F is four so I just leave it it's fine it doesn't really hurt anything you can take out the V if you don't want to see it on the screen or if it's running in a Cron job which is what I'll end up doing eventually and then I'm going to name it Docker backup and then I'm going to use that variable that we created up here for the date so that I know which day this was from and so it's going to have the backup date there dot tar.gz and I'm going to save the docker folder so I'm zipping up the docker folder so first I tell it what I want to name it then I tell it what I want it to have in it so you can put more than one folder you could put a space right here and you could do slash home slash Brian slash pictures for instance and that would get my Docker folder and my pictures and zip them all up I don't want to do that but that's something that you can do with the zip that's why you put what you're zipping up after you put the name of the file so once that's finished zipping up I want to go back into my speed test tracker so I'm going to CD into Docker because I'm in the home directory slash speed test tracker I'm gonna do Docker post start to start it back up and then I'm going to go back one level so dot slash navigrome and start and the same thing for mesh Central and the same thing for Dashi and once those are finished that's great I've got everything started back up now I want to go back into my my main home directory I'm just echoing out a space and I'm just putting something on the screen so I know it when I run it manually that it's running the rsync and then I'm going to rsync and it's Dash AZ so this a says create an archive of it and Z means compress it and I'm going to run this this R sync that says run this Docker backup that I just created this file and R sync it over to my other machine that I want it in and I have a backups folder over there where I'm backing up my Docker stuff so this this little script does everything that we've talked about we're gonna basically compress our Docker image and we're going to back it up so really simple we're going to back it up to another machine now you could set this up so it's got a cloud backup you know running off of this this folder on this other machine there's all kinds of different ways that you could do this you could run this to a USB drive you could run this in lots of ways I'm just moving it to a different machine on my network and in this case for the example this is actually going to a different virtual machine on the same physical server so it's not the best way to back it up but I could do this to anything it's just an example for you guys to kind of see how this can be done and things that you can do now there are tools out there for running backups as well so you could use this script to just create the tar file and then use Something Like Ur backup to have that backup moved over to your backup server you could use something like duplicate to move it to the cloud storage or duplicacy to move it to cloud storage so there's a lot of different options and different things out there that can do backups and there's all kinds of great tools out there for doing backups it's really up to you how you kind of work through that I think the most important part honestly is to get to this point that you've actually created your gz file and then you restart all your containers and then how you move the backup over is really up to you I'm just giving you one one way to do it which is our sync which is great our sync is really fast and it does a good job of doing things so I've kind of walked through this I'll have this in the show notes so you can go look at it and kind of take it you know kind of see what I'm doing now you wouldn't want to copy it directly you'd want to modify it for your setup of course and then I'm just going to save that just in case I made any changes I'm going to exit and we will actually run this thing so I'm going to do sudo backup maybe uh dot slash backup there we go and it's going to prompt me for my super user password and I do sudo because a lot of the docker volumes themselves that are created running root inside the container so they have some weird permissions so it's just better to do it this way to keep you from having any permissions issues so it's going to go through and it's going to stop all of my different containers mesh Central takes a little longer to stop than some of the others but um once they stop then it's going to create that archive file and it's going to start them back up and then it'll rsync all right so if I jump over to my other machine I'm already in the folder that we talked about where those backups are going so if I do an LS you'll see I've got one from the 20th and then I've got one from today so you can tell that I've got two different backups of my of my archives so as you go this older backup may not be useful but it's usually good to keep like two or three maybe four versions just in case something's wrong with some of the ones that you've made recently maybe you didn't catch a problem that you had for the last week and and you want to keep some of those these aren't super huge um they're they're a few hundred megabytes um let's see yeah so 305 megabytes and then this one's 317 megabytes so you can see that I've added data since two days ago so you know a dozen you know 10 megabytes or so so yeah I mean it's worth doing these backups on a regular basis because you can see I'm getting more data now a lot of this could be log data uh there's ways to kind of say ignore logs and things like that and different tools that are out there so again other ways of doing things are sometimes better sometimes harder sometimes easier just depends but I'm just giving you the way that I do this to kind of feel like hey I've got things backed up and I can bring it back if I need to pretty easily without having to go through a ton of trouble and losing all of my data so I'm going to clear this out now the other part we talked about so what I'll do is I'll go in and I'm going to modify that script real quick so I'm going to say Nano backup and I'm going to scroll down to the bottom where it does the rsync and I'm going to switch this to SCP I don't need to do sudo anymore and I'm going to say yeah that just that and that's going to try to send it over without having to have the r sync in place so I'm going to just save that and exit and then I'm going to clear this out and I'm going to do uh cron I'm going to do sudo cron tab because I have to actually set this up as a Cron job so if you run cron tab like this with Just cront Tab Dash e it's going to open it up as a regular user but if you need to run something that has pseudo privileges necessary and things like that to run you need to do pseudo cron tab Dash e the first time your order it's going to ask you which text editor you want to use just type in the number of the text editor you want and it'll open it up in that text editor from then on so for me I use to Nano it's fine if you scroll down they give you information about how to set up the cron and they explain what the different things mean but it's it's minute and then hour and then day of month month and then day of week so this is in this case up here you've got zero and five so that's 5 AM and it's every day of the month in every month of the year but only on the first day of the week now it starts as zero is Sunday so one is Monday two Tuesday three Wednesday four Thursday five Friday six Saturday this is how it works so you can set this to go off at any time you want on any day of the week that you want now if you put a star here it would do it every single day of the week and it would do it every every day at 5am so if you want this to be a daily backup you would change this to star so if we go down here and we say we want to run ours every day and I want to run it at zero and three three a.m everybody should be asleep star space star space star that's going to run every day at 3am and what do I want it to run I want it to run the slash home slash Brian slash backup Docker dot sh I want it to run that command so I'm going to say Bash and it's going to run that for me so this is what it's going to do it's going to run that script every day I'm going to save it and I'm going to exit and now it'll start running that thing and it'll SCP things over and it'll make everything move into that folder that I said and every day it's going to have the correct date for when it ran and I'll know which date it was now eventually I need to go and look and make sure I'm not filling up that other drive so there's things in that script that you can do up at the top to tell it get rid of the you know ones that are older than so many days and things like that and I can go into that in another video if you guys are interested but people have asked me about how to do this and how I do this this is the way I do it it's kind of manual but it really works it's pretty simple and straightforward so I hope you enjoyed this if you did like subscribe tell your friends about it so they can come along on the journey with us and I'll talk to you next time foreign [Music] Source Advocate and I'm back and I've set up a store with a little bit of merchandise I love being your open source Advocate but I want you guys to be the open source Advocates with me so if you want to get out there and get some of this stuff and if you do let me know what you think of it thank you for subscribing

Channel: Awesome Open Source

Views: 27,182

Keywords: open, source, opensource, open-source, self, hosted, selfhosted, self-hosted, free, libre, software, server, web, internet, browser, linux, mac, macos, os x, windows, microsoft, unix, bsd, ios, android, pi, raspberry, desktop, vps, tutorial, how to, setup, installation, instructions, cli, command line, terminal, interface, network, networking, news, projects, backup, duplicati, duplicacy, urbackup, sync, easy, vm, virtual, machine, podman, kubernetes, script, powershell, bash, sh, zsh, volumes, container, contaainers, data, database

Id: wUXSLmGAtgQ

Length: 17min 40sec (1060 seconds)

Published: Tue May 23 2023