Linux Server Storage management with MD/RAID and LVM

okay so today we're gonna go over some linux disk and filesystem maintenance tools that doe i have found very helpful over the years for various events that come up and just kind of how to manage and go through the various levels of storage and file systems that we have on our linux based servers so one of the first tools will start at the very lowest level on the actual physical drives one of the very first tools that I like to use is smart Mon tools this allows us to actually access the smart data on the hard drive as well as getting various information from your hard drives and solid state drives so we can go ahead and install that it doesn't come installed by default but we just say ask installed smart on tools and it will install it there on the server so the smart the tool we use is smart control some a RT CTL smart control and we would just need to give it a drive so if we want to get information from a drive will just be smart CTL - I - just get information and let's pull out information from dev SDF on this particular machine I've connected a couple really old drives actually BIA a USB connection sometimes this tool has a little trouble with USB connections but it works fine today so we can see there's a 160 gig Seagate Drive really old laptop drive and SDG for example and there's the the drive it just gives us the serial number capacity and lots of useful information from the drive like that now let's suppose we wanted to make an inquiry into the health of the drive instead of - I we could do - capital H and it's gonna give us just one line opinion of whether that drive is healthy or not so same thing for G etc sometimes you might want to do a little troubleshooting on a drive and actually get some information from that we could go ahead and be instead of - H - lowercase a and that's going to give us a quite a large print out from this on this drive and we can scroll back here through this printout and kind of go through there's the the information there's the health assessment there's information about self tests that the drive has run and here's just some raw stats from that drive one I like to look at is this one here the power on hours this drive it even though it's old it's not been run much at all so a hundred hours on the drive I've seen drives with as much as 50,000 hours on the drive other things that you might want to look at is certain things like read error rates solid-state drives have information down here that gives you information on what the wear is on the drive sometimes those attributes don't always come through with this tool but you get lots of information on the drive from here you can also use this tool to start a self-test so we would just do smart control - T we'll just do a short self-test on SDG and also on SDF and it'll instruct that drive to perform its internal self-test routines and here in a couple minutes we'll be able to go and query that and get the results of those self tests alright we'll go ahead and see if those self tests have completed we'll go ahead and just print out all that information for the SDF and SDG here there's SDG and sure enough there's the short offline self-test complete without error as you'll notice on most of our systems we use software raid and I've demonstrated this on a previous video but I'll go ahead and demonstrate very briefly setting up a raid array and then dealing with various sorts of failure events on that array so I've already actually created a partition on each of these drives that you can see dev SDF there's a 5k partition SD f1 and SDG there's also a 5 gig partition there I'm gonna go ahead and create a small raid array out of these two small partitions let that sink up and then see how it deals with the various failure events so to do that we'll go ahead and MD admin create we'll go ahead and make this a mirrored raid 1 raid devices - we'll call this dev in the - in this instance and then at dev SDF 1 and dev SDG 1 and it's gonna ask us yeah we'll go ahead and continue creating the array and it started so now I can cat frock and be stat and I can see that it's there it's already syncing up it's gonna take a couple minutes to finish that we can use the watch tool to watch and see that go while that's syncing up we'll go ahead and create a file system on the array so we can see how it behaves under various failure events so actually pretty simple make mkfs at ext4 and we'll call that dev md - is what we call that okay put the filing system on there and now we'll go ahead and make a mount point for it MTM d2 and then mount dev - mm tem d2 and then we'll go ahead and CD into that directory so we'll go ahead and create some files in here 5 there we can see there's several files there and we'll make a big file tab 0 big fun will make that trigger megabytes and we'll just go about with zeros and so there we've got some files there let's see if our raid array is finished syncing out okay now we have mb - and you can see there s dg1 SDF 1 you can see that it's mirrored we can go ahead and get some details on that by doing in the ADM Josh - detail slash dev slash mb - and there it's gonna show us it's its claim to active working devices etc and gives us the fault stats like that now whenever we create a new raid array on a running system it's a good idea to go ahead and update the running great configuration so that when you reboot the system it comes up with the same device name otherwise as you can see here in that previous command we've got some inactive raid arrays here with really kind of some odd names so in order to update that there's a tool it's a rather obscure tool but it's under users share in the ADM in case conf and that will generate a configuration file which we can then spit out to overwrite our existing configuration file and Etsy and medium India TM calm so we can go ahead and do that and then we can use nano to go ahead and edit that file and name those raid devices how we like them so you can see here we have that MV 0 I usually take these slashes out of the necessary budget in D 0 and B 1 and B 2 so now our rate configuration will be persistent across reboots one thing I do have to do after this because we want to have that loaded into the initial RAM disk is to D package reconfigure and reconfigure the running kernel and that will go ahead and copy that rate configuration into the Anette ram disk of the kernel so that well load as soon as the system boots now let's go ahead and simulate some different failure events on this raid array so again we can kind of watch to see what's happening here - I'm going to watch this and here we have a detail it shows a claim and everything and I'm just going to go ahead and pull the plug on one of these hard drives nanus now suddenly the rate array becomes degraded it's still running but it shows that one of the drives is removed however because it was mirrored we can still access all of these files you can see I could copy this big file call it big 2 dot file and I can still work with these but we are missing of course one of the drives so we'll have to go ahead and add that drive and back in so we'll connect that drive back to the system and it might come up under a different drive letter we'll see what it does Ami's D message to get some idea of what's going on I noticed there at the top here it's showing the disk failure on SDG and it shows that the rate is continuing on one device is printing out this configuration now we got SDG attached again so if we look at our MD stat of course the drive is not automatically re add it to the array I can look at that drive and make sure the configuration is still the same yes yeah of course that's the one that's that we didn't then plug s DG is the one we had actually unplug the piece is still there but we need to re add back to the array so this is the important command empty ATM - - let's see dev MV - - - ad dev s DG one that's going to re add that back into the array and then if we want Rock MV step we can see but now it's actually resyncing the array again alright at this point I can see that the raid has rebuilt again so I'll show you another way we can do the same thing by removing a drive from the array without physically disconnecting it from the system so we would just again we'll take a look at our configuration SD f 1 and f DG 1 in this time we'll go ahead and remove SD f 1 so we'll do Indian first of all we'll market is faulty fail sorry Canadian dev MBT - fail deb SD f 1 okay now it said it's faulty now I run the same command but this time we will simply remove the SDF one all right now if I take a look at our configuration again notice SDF one is no longer there even though we've disconnected one drive reconnected it and now disconnected the second notice that our rate is still there we can still do plenty of operations in our files we've never unmounted this file system so the system continues running all the files are still there we can't obviously remove both drives at the same time we really would lose our files in that case but if I need to re add that device back in just simply md80 em dead in the to add dev SDF one and then again we can we're curious watch in the ATM that detail dead in d2 there we can see that it is resyncing we can see the rebuild status down here how quickly it's recovering that array now that we've gone over some basic operations with the raid arrays let's move it to logical volumes we'll go ahead and unmount this particular file system and in fact we'll blow it away in our next operation usually when I create a raid array I don't create a filing system directly on that array I add it to a volume group and then create the filing systems and logical volumes in that volume group the first step in creating a volume group is to format the physical volume for use with LVM so we do that by using pv create and then we'll go ahead and give it the device name which we just created the dev md2 now I detects of course that we formatted this as an XT for file system so we will go ahead and wipe that and now it is set up as a physical volume and we can actually see this by typing PBS and it will show those physical volumes and that is on this system we already have a volume group called bg1 if we didn't we could use something like VG create and bg1 and then give it the names of our physical devices but we don't have to do that this time because we already have a volume group so we're going to use VG extend to extend that volume for bg1 and add our dev mv2 into that volume group so VG extend now that apbs again it's going to show that devon d2 is now a part of the VG one Vaughn group now we've done that we can go ahead and create logical volumes with LV create again we're gonna create them in the volume group eg one I'm just going to name this particular one test one and we'll make this say a three gigabyte volume and we'll put that on our newly provisioned partition and EV MB - alright well family had typed that wrong I'll be create PG one yes and that's created and now we can actually create a filing system on that logical volume okay that's that the XT for dev mapper vg1 - test one and it's creating that filing system for us now I can make them out point I'd say mmm t-test one and we can actually mount our filing system not deaf mapper VT one - test one at mount point M&T test one up the one their dev mapper VT one test one at M&T test line all right and then I can actually see the into there and we can create files like we did before sure now we might want to do some maintenance occasionally on our logical volumes well let's go over some commands to get some information VT display will show the entire volume group and some information about that for short vgs we can use something like pv display to show information about each of our physical volumes or for short pv s I like the PBS version because it does give us down here a nice assessment of how much space is used and how much space is free on each of the different physical volumes you can see this is basically five gigabytes here we have basically two gigabytes free and then for the actual logical volumes themselves we can use LV display it gives us detailed output or for short obvious and now we can see on the system already we've created route and swap and this is the one we just created test one we can do LVS - o+ devices and that will add one more column here to show us which device the individual logical volumes are on we can even add a dash a to show us any hidden volumes that might exist on the system as well which we don't have any at this point we will before this demonstration is now suppose that we're working along here and for whatever reason we run out of space maybe we made some big files list copy lots of zeros we'll call it zero stock file one megabyte and count equals one thousand okay and we'll call this in another file zeros - no ok so now we have a couple big files here if we show the free disk space you can see oh we've used 73% once it starts getting there I start having to think about enlarging our partition because we do it certainly don't want this to run out of space in the middle of a busy day so let's go ahead and use LV it's extend to grow that particular logical volume and we'll go ahead and add ten gigs to that and obviously it's gonna have to allocate extends from more than one physical volume in order to do that but it should be able to do that happily and sure enough and if I do LVS again now it's going to show that just out of curiosity if we do PVS it's gonna show us obviously that MV 2 is full and it's taken some more extents from it easier looks like but anyhow so we've extended our physical volume but our filing system isn't extended so if we do disk free at this point it's still gonna show that the device is only 2.9 gigabytes and at 73% full so once we extend the logical volume we have to use the resize 2fs or some similar command to resize that filing system to take up this full space of device so we're looking yeah mapper eb1 - test one and it's gonna go ahead and resize that online show the free disk space and now you can see the device is enough 13 gigabytes and it's only 17% used now suppose I wanted to remove this raid array that we've just created with these external USB drives for example that's probably not a good idea to run with these external drives in production but if I show the configuration of my logical volumes I can do LVS - o + + devices and I can see that that test 1 volume group now resides on both dev mb0 and dev MD - and that Devon B 2 is the one that is on those USB drives so I can use something like pv move and tell the logical volume manager to move all the physical extents from dead and e2 over to death in the one well family I didn't type that right evey move dev and visit - dev MD 1 that'll take a little while we might come back to this in a minute ok at this point you can see that it says it has completed the pv move so if I do the LVS command again it shows down here the devices are NB 0 and m v1 but NB 2 is out of the equation and I can confirm that by doing PDS and it shows that the total size and three size on MB - like a DP v display good MV - it's another way of doing it sure there's no allocated physical extents so at this point we can nicely take this mv2 down just reversing the process so I'm going to VG reduce VG 1 dev MB 2 and that's going to remove it so if I show that physical volumes now dev MD 2 is now stopped now we could remove the formatting from that so TV removed dev NB 2 and that's gonna just wipe that off so that it doesn't even show in our list of physical volumes then we can use MD admin to stop that raid array if you see here the raid is still running Devon D 2 so if we do in dat M - have to stop dev MV - it's gonna stop that and now it doesn't even show so at this point we can safely unplug those USB devices and nothing at the higher levels will complain about it okay we've removed our USB devices entirely as you can see if I can f disc - now it's gonna show the various devices at this Delta s DF SD SD g those devices are gone but of course our new volume group turn a logical volume is still running and mounted on the emiti test 1 now we can of course unmount that if we don't need that mounted anymore and here's a couple other commands this you might find useful sometimes we display bg1 test one sometimes you may have a need to mark a logical volume as not available that'll take it out of the system as a block device that can be scanned for by other things on the system so in order to do that we would simply do LV change - a which is available in and VG 1 slash test 1 if we do I'll be display again notice it shows LV status not available sometimes like if we boot up to rescue mode all of the logical volumes will simply come up automatically in this status not available and at that point if we want to work with those volumes would you need to use LV change - 8 Y and then give it the device and leave you 1 slash test and then that becomes available again I'll be display and it shows that that's available now if you're really done with a logical volume you don't need any data on it anymore and you really want to destroy it sometimes we do this we can just simply LV remove eg one test one and it's gonna confirm do you really want to remove it and yes and there it's gone we do LVS that you can see it's no longer in the list so that's just a brief overview of some of the different operations you can do with logical volumes I haven't gone into snapshotting or anything like that but I will in the next video make a short demo on maintaining the LVM cache devices

Channel: Daniel McFeeters

Views: 6,153

Rating: 4.9720278 out of 5

Keywords:

Id: mKIrK-iifao

Length: 27min 55sec (1675 seconds)

Published: Fri Aug 24 2018