Linux Talk | Partition Schemes, Swap Space and File Fragmentation

greetings and salutations and thank you for clicking on the video in this video we're going to talk about drive partitioning swap space and we'll get into file fragmentation at the end of the video so this video is definitely for more advanced users but even if you're starting out hopefully there'll be some good information in here I was planning on doing a video about partitioning and swap space and then somebody asked about file fragmentation on the easy Linux Facebook page so we're going to put that in here as well and hopefully it will be an informative video anytime that I comment on these subjects I always get a lot of questions and people express a lot of opinions which is cool because Linux gives you lots of options when it comes to how you set up your hard disks and how you get your system spread out over whatever storage you have installed in your computer so let's get started by talking about partitioning schemes there are many options and each one has its advantages I have written in my notes here and let's start out with a couple of examples the default partitioning scheme traditionally for most aboon to type Linux has been to create a large partition at the beginning of the drive which holds the system all of the data folders everything and then it will generate a swap partition that it will put at the end of the drive in our examples today we're looking at MBR but it's really not that different if you're using a modern system with UEFI and GPT for the partitioning scheme because the only difference there is that with UEFI there will always be a efi partition up front and that is your boot partition so the basic old way of doing it with MBR or if you have a system today that doesn't use UEFI and turn that off and you're going in legacy mode if you just install Ubuntu everything up to 1610 what it's going to do is it will create that large partition up front and it will format at ext4 it will mount it at root and then it will create a swap space or swap partition and it will usually go ahead and put that in a logical partition because with MBR you only get four primary partitions so therefore you can create a logical partition and then you can have as many logical drives in there as you want so that's how they do it and then a more advanced method and the one that I have done a lot myself and done for clients over the years is to use a separate partition for your home folder and there's a lot of advantages to doing that first of all it separates your personal data away from the system so if you have to reinstall the system you can leave your home partition unformatted when you do that and then all of your data will be there if you need to upgrade something goes wrong you need to reinstall whatever that's pretty cool also having a separate home partition will allow you to move the swap partition up on the drive and put it between the root partition where the system lives and the home partition where your personal files live and it makes more sense if the machine is going to be using swap then having that placed between the two clusters of data means it will be right in the middle instead of having to go all the way to the end of the drive if it starts swapping as in the traditional default scheme there I came up with a scheme that I experimented around with about a week or two ago and been playing with I did a video about this talked about it not too long ago and what I did was I did the traditional separating the routes and the home there and then I added another partition and I formatted that with the XFS filesystem and then I mounted it at VM and that was for machines only ext4 is pretty much the default file system for most boon to based linux Arch Linux if you just install it that's what you're going to get other distributions have moved on to more advanced systems faster systems for instance if you install openSUSE they will use butter FS for the route and then they will give you X FS for the home partition butter FS and x FS are like I said more advanced systems they have their advantages there faster butter FS there's still questions about its stability there's some problems with butter FS x FS is relatively stable and it's been around for a long time and it is really optimal on servers and raid arrays with large spinning drives and lots of files that's what it's good at and it also is very good at large files it uses a multi-threaded system to access a drive so it can actually be reading from two parts of a drive at the same time which tends to speed things up that was the reason why I tried x FS on the set-aside partition just for virtual machines and that was to see if it would run any faster it did but not by much there was there's always a trade-off with that because xf s actually uses more cpu so in the case of the virtual machines yeah the hard drive access was faster on the large disk files but what happened was is that the CPU usage went up so it's kind of half-dozen one six to the other and it would actually slow down the Machine a little bit so yeah x FS is cool at this point in time it probably would not be a great idea to use X FS as a boot partition with MBR simply because x FS unlike ext4 is not automatically checked at boot time with ext for the kernel actually runs a program called fsck just to make sure everything is okay and the filesystem checks out at boot time does a real quick check to make sure that everything is okay then it mounts the file system and then it starts reading and writing to it whereas with XFS if you want to do those things you have to install separate tools and then you have to know how to use them so the only way to actually check and XFS drive is to have it unmounted now if it's a data drive that's no big deal just unmount it and run the XFS repair tool you're good to go but if XFS is on your roof partition then you're going to have to boot the machine from like a DVD or USB and then you're going to have to do it from there it's a bit of a pain and the other thing about XFS is it is not as robust as ext4 it doesn't like crashes it doesn't like the system to halt without unmounting XFS and sometimes that can cause lots of errors whereas ext4 at this point in time it is very robust you may get an error or two if the system crashes but the fsck program will fix that at boot time most of the time you will never have an issue with ext4 ext4 is also simpler as far as how it accesses the system it's not multi-threaded makes it a little bit slower of course but it's extremely stable and very steady so which one you use is up to you but anyway in my case I gave it a shot and it really wasn't worth all the effort so I ended up going to a different partitioning scheme so let us look at some of my machines on the network here and I'll talk about the different partitioning schemes that I'm using and why they are here my thoughts on this have evolved quite a bit through the years I don't want that fullscreen thank you my thoughts on this have evolved quite a bit so the first machine we're going to look at is my HP laptop that's been kind of converted into a desktop and mainly Cindy uses that machine so to figure out the partitioning scheme of any computer just use LS blocks so you'll see on this system it's really simple we have one large partition that has the root in it and then we have at the end of the drive we have a swap partition and that swap partition is just a little bit bigger than the physical memory in the machine this machine has a spinning 5200 rpm 320 gigabyte drive in it that drive is about half full and if you take your system and you split off all your partitions when you're dealing with one drive you sometimes can run into a situation where it's not a very efficient use of space you end up with two smaller partitions and you can't fit all the data you want to in any individual partition and you kind of wish okay I wish I could get some of the data from over on the other partition ZFS is really cool that is a really new advanced system that's coming into Linux now and very popular with BSD and UNIX that system actually just creates pools and it can steal from other pools which is pretty awesome and all these snapshots and other stuff as well it's not quite ready for the average Linux desktop or laptop yet but it's on its way so it's pretty cool but if you set these partitions where they're you know pretty much in stone then you don't get efficient use of the space also this machine has it's not really working that hard okay what she does with the machine is she does a lot of internet does a lot of email she might play a little bit of music here and there that's about it that's all this machine is doing so having everything separated and all that stuff it's not that big of a deal I've got really good backups so therefore I'm not really worried about keeping the home partition separate I probably will not be upgrading this machine at any time if I have to reinstall I can just rebuild the whole thing so that's why this is done the way and it's done so let's look at another machine and this one is a laptop this is the one that is kind of like the family computer that everybody goes to alright and for a hard drive this machine has a hybrid drive it's as sshd which means that there's a spinning drive but there's also a cache huge cache in the drive which is like 8 gigabytes so the drive tries to remember stuff that people access all the time and it throws it in the solid state portion and will access that before accessing the spinning drive and on this machine there's a lot of accounts there's actually five user accounts created here so in this case it was actually probably a good idea to separate that home partition because rebuilding all of those accounts even with a good backup is going to take time you're going to have to go through and reset everything that needs to be reset and all that stuff whereas if something does go wrong with the operating system on this machine then all I got to do is reinstall the operating system recreate the user accounts and all the settings and everything remain the same that's assuming that the hard drive itself doesn't crash if the hard drive dies well you got to start from scratch and that's just sort of the way it is but this is the scheme that the advanced scheme that I pretty much told people who you know have the savvy to be able to set this up to do four years at one point in time I was talking about using a boot partition even on the MBR partitioning scheme I've had that in a couple of videos but since I since then I have changed my mind on it I don't really care about boot partitions anymore on the old MBR and of course with UEFI and having an efi partition on there that contains the bootloader you don't have to worry about that it takes care of itself as long as that set up properly so I told you that I had been experimenting with different partitions at the beginning of the video so what partitioning scheme do I have running on this machine that I'm doing the video on which is the one I did all the experiments so I'll show you and you will probably be a little surprised that's it gang one partition that's it one great big one terabyte drive one great big partition and by the way all my machines at this point in time they all have spending drives we're not talking about SSDs at this point we're talking about good old spindle driven hard drives with mechanics and stuff in them and this drive is a caviar black one terabyte 7200 rpm and so I wanted to see one of the reasons I was doing all these different things was just to see what would make it go faster I mean I've got virtual machines I've got little files I've got my whole system on here and the reason why I went to this was because if we go down here and we do the same thing with this boon to 1604 virtual machine you're going to see that they have kind of the same deal as the default this is actually the default so what you get here is that you have a the partition scheme there that we talked about in the beginning of the video with one big partition for the system and then we have ext two is a logical partition and then ext five is the first logical drive so that's what that looks like it just looked a little different then it kind of confused me a little bit there so if we jump over here and we look at a boom to 1704 which is actually in beta and will be coming out very soon and I'm running it to take a look at it you're going to see that now what a boon to does is they have one great big partition that's it see that's all and the reason why is because Ubuntu has decided that they are no longer going to create a separate swap partition on the drive and they will be using swap files so therefore I decided that I was going to give that a shot even though I'm not running a boom to 1704 on this machine and I found out something kind of interesting after all of those different partition schemes this one seems to be the zip iasts I'm using ext4 for the file system so I'm not using X FS and everything is in one partition I don't care I do really good backups so I don't feel the need to separate my data away from the system I wanted to see how this worked anyway because this is the new default scheme for a boom - Arch Linux has been using this as a default for a while guess what gang keeping it simple makes it faster even with ext4 which is slower than other file systems so that's what I came up with so what should you do when you're partitioning your system I'll tell you what if you're new to Linux and you're just starting out then what I would tell you is just let the dang installer partition the drive and I'm assuming here that you are not going to try and dual boots and you don't have an extra data partition you're trying to get around you just don't care you've got a hard drive in the computer and you just want to format it and get it going that's probably the easiest way to do it it's just a let it partition itself if you have more than one drive if you want to separate your home partition for whatever reason if you want to do some funny funky things there then do whatever you want to but now with most clients these days I don't even bother going through setting up a separate home partition and all that stuff I just say you know erase and install so that's where I'm at on that today let's move on to the next part of what I wanted to talk about here and that is I want to talk a bit about swap space because anytime I mention swap space I get all kinds of comments about it I also wanted to talk about swap files versus swap partitions swap space provides memory for the kernel to use at times when physical RAM is running out which means that let's say that you have a low resource machine it's only got like 1 gig of ram in it you can set aside swap space on your hard drive and let's say you gave it 3 gigabytes of swap space you could run that machine pretty hard and what the colonel will do is as it's getting to a place where it doesn't have new physical RAM to stick data into it can actually swap that out to the hard drive and it does it in pages that's why in the Windows system you'll hear about a page file it's exactly the same thing all operating systems have some way of taking advantage of virtual memory that's using much slower block storage which would be a hard drive in place of memory and what the kernel tries to do is it tries to figure out what is the most active right now so it may have stuff that's stored in memory that the user is not actively using so it goes okay I can page that out and then it puts it over on the drive in virtual memory and if it needs it back it can go get it back in the days when memory was expensive and it was very typical to be on a machine that if you're really lucky it would have you know half a gigabyte like 512 swap space was really important my brother had an old machine that we installed Linux on and it only had 512 megabytes of memory in it but he was actually able to do a lot because of swapping the machine was slow and if he pushed it too hard he would get to a place where this thing called thrashing would happen and thrashing is when the kernel has swapped too much stuff out to the hard drive and it's having to read and write to it very quickly to try and access the memory that is on that drive so when you get to that stage and that's when you need to go buy more physical RAM so even today when we have computers it's nothing for a machine to have eight gigs that's almost introductory a lot of machines have 16 32 you can get 64 you don't know how I don't know where the limit is on memory you can have as much as you want it's dirt cheap these days and the motherboard manufacturers are making it easy to really stuff them so do you need swap space on a machine like that well some programs use swap space independently of the kernel they actually will access that part of the system for whatever reason that they're using it and so therefore it's kind of nice to have it around because you do have a program that wants to do that running then it and you don't create some sort of swap space for the system it may run into a problem it may become very unstable swap is also used to dump the contents of RAM and I only have RM there I'll fix that when I post it on YouTube when there is a crash so if the system has a hard crash it'll take everything in the memory and throw it in swap and if you are one of the six people on the planet you know how to read a swap file or a swap partition and extract what information that is you may be able to debug it maybe there also must be a separate swap partition in Linux that is slightly larger than the total ram space in order to use hibernation hibernation is when you can shut down the machine but save its state so what the system does is it takes everything that's currently in memory and it dumps it into the swap area and then when the machine boots up next time instead of reloading the system fresh it actually reads from that swap space and then it restores it to exactly where it was hibernation is problematic on a lot of machines and a lot of systems these days don't even offer it as an option so therefore if you don't need to hibernate then you don't need swap space that's the size of the memory and if you're not really worried about reading dumps when the system crashes because you don't know how to do that then you don't need swap space that's the size of the memory an old rule of thumb was that swap space should be twice the size of your memory well yeah back when we had 512 that made perfect sense even if you had a gigabyte that made perfect sense nowadays doesn't make sense at all because if you have more than 8 gigabytes of memory on the computer chances are you ain't never going to use swap space and if you do you're running that machine really really really hard you have a lot of programs open if you are gamer if you work with virtual machines if you edit video and audio on your computer then you might actually run out of memory with eight gigs you could get close what happens if the machine runs out of memory the kernel stops it just stops everything freezes so that is why you really need to have a little bit of swap space available even if you've got scabs of memory and these days a good option for doing that is creating a swap file so the system in Linux can either take a partition on a disk and it can make that a swap space and then what it can do after that is that's where the virtual memory lives and if you had a situation where you were booting more than one Linux system you don't need to create separate swap space for all of those different systems that might be living on that computer you can have one swap area and you can point all of those systems to that area when they boot so that's kind of cool but if you're going to have one system having a partition these days really isn't necessary you can use a swap file and this is what Windows has done traditionally for years and some people will go well I thought it was cool that Linux had a swap partition man it was really nice I know Windows uses page files and I don't really like that it makes absolutely positively no difference to the kernel where it the virtual memory is none has no effect on performance I have read that in I don't know how many different places now and pretty much all the system engineers that work on this sort of thing say eh you do need a little bit of swap space and B it doesn't matter where it is could be in a partition could be in a file so a boon to and Arch Linux now offer to create those swap files for you you can create a swap file yourself on your system by following these commands really quite simple the first thing you will need to do is create a file and dump a bunch of zeros into it you'll use the DD command to do that and then the next thing that you will do is make that file swap in other words you're going to tell the system turn this into a swap file system and it'll go do it and then the next thing that you want to do is to turn that on to make sure it works and the command to do that is swap on all of this needs to be done as root to make that happen once you create a swap file you will then probably want the machine to use it at boot time and to do that all you do is add this line to your FS tab which is a file that lives in the NC directory that is the file system table and it tells the system where all of the partitions on your drive are so if you stick this line at the end and you in this case I'm using swap file name as an example you're probably going to name it just plain old swap file works just fine you could call it anything you wanted it to just as long as the system knows where it is and what its name is you can use this line of code here to actually mount it at startup so if you created a system that doesn't have any swap space and you have just saw my video and now you're all paranoid because you're like well you'd swap space I've got to have it I didn't know I thought I had 32 gigs of RAM this has been a little unstable maybe this will help here's how you can fix that right there at install with any Linux system you can just do a manual partition and then not include the swap area the boon to will complain older versions will like 1604 it will complain and go I don't why don't why aren't you putting in swap space we need swap space but then once you get the system up you can go through this process here and you can create a swap file of any size that you want to and then the system will be fine and so therefore that's what I got to say this seems to be the trend now is to not have those separate partitions but just to have a file we're heading in that direction it doesn't matter to me I've done it on this system it work just fine I have been playing around with a boon to 1704 where they set that up by default and it works just fine I got no problem with it at all now one of the things that people will go well I had one guy say when I was talking about this in past video was that it caused this file system fragmentation no it doesn't it doesn't matter the swap file in Windows yes it would because the swap files in Windows were dynamic and they could grow and shrink and they would cut themselves I was really not that great of a system with the old page file fist they called it that's what the swap file in Windows is called page file dot sys and those files they had it where it could grow and shrink and I think you can do that as well in Linux but in this day and age you don't even need to worry about that just give it like 2 gigabytes worth and go on with your life it's no big deal we don't need to go in the hole and hassle so once you create the file on your system it's going to be relatively static anyway it's not going to fragment anything and that leads me into the last part that I wanted to talk about and that is ext4 and XFS file fragmentation now this applies only to folks who have spinning hard drives if you have an SSD then there is no need to ever defragment that drive and you're probably saying well if I tried to do that the system would say it's an SSD and I don't need to anyway guess what gang if you use SSH and you remote into a system it doesn't catch that and you can actually run the defrag program on an SSD it won't hurt it but it's totally unnecessary one and number two it causes a lot of write cycles on your SSD because it's reorganizing all that data so there's no need to do it it makes no difference so what did I write here I said do you need to defrag your drives the answer is probably not and the reason why is because the ext4 file system and XSS and many of the other file systems like JFS that you'll find available for Linux they don't tend to fragment as they add data to the drive as a de is data is added and taken away first of all what ext4 does is that it looks for free space on the drive and it also groups like files together so back in the days of NTFS for Windows if you go back long enough you'll remember fat16 and fat32 all it did was just if you sent data to the drive and you wanted it written it found the first available space it started writing and if it ran into a used block of space on the drive it would just pick up and it would just skip it and then go to the next available and it would fragment itself this is why NTFS even to this day D fragments itself every week by default it's usually set up in Windows to do that so it we in Linux when they came up with the ext4 system they were trying to work against that mainly because you have servers that run unattended they really can't be brought offline to defrag they're too busy to defrag so the file systems really need to be able to do it for themselves ok so why would you ever want to defragment a Linux system well there's a couple of reasons for that if you have very large files if you're working with virtual machines if you are working with big audio and video files and also if you're working with drives that are almost full we're talking like more than 75% full then what will happen is is that you may actually get a little file fragmentation because now ext4 or XFS can't find out you know they're not finding the blocks of space it needs to put a whole file so it's going to have to resort to fragmentation with large files they end up being a little bit fragmented anyway it's just the way the system works so you can defrag those if you want to to check whether your drive needs to be defragmented you can use the pseudo efore defrag program with the option C that's just check and then give it the path name or the device name to the drive so let me show you how that works so let's just jump down here to Ubuntu 1604 because it this is totally distro non-dependent it makes no difference its distro agnostic okay so let's do a sudo and we'll do a 4d fragged and then give it the option see we won't leave an equal sign in there because that won't work and we can either give it a device or we can give it the actual folder that we want to look at in this case we're just going to do it for the root partition since everything is in root this has to be run with administrator privileges so we'll just let that do that it's got a crunch for a little while it's going through it's going to read the entire file system table it's going to go through every inode and then it's going to go through the entire bitmap everything all the data table plus all the I nodes it's going to go through all that stuff to can look at the journal it's what it's doing right now and to gather this information and to see the fragmentation level of the drive so this actually does take a few seconds even on a really fast drive this will take a few seconds so we'll just be patient and wait actually there we go so it gives us a list of the fragmented files and it kind of rendered funny but here we go so we got the fragmented files it says we got a few in here that's to be expected and then down here it gives us a total it tells us what the average size of a data block is on this systems 32 kilobytes that's called an extent that's where ext4 comes from it's an extended file system and then the fragmentation score in this case is zero and I bet you that if you've never done this on your Linux system and you're using ext4 and it's an average desktop where you do a verage things that people do with desktop computers even if you've never done it and the machine has been up and running for five years that score will be zero that's how good a xt4 is doing that so let's just run that command one more time and we're going to do that this time with the device name so you can do a whole device actually a whole partition as long as that partition is the xp for so how would you do that well in this case we've got SDA that's the only ext4 partition we have available to us so let's just do dev SDA and now get a password in there look file system not mounted oh um that's my fault see I make mistakes too game it's got to be SDA one that's where that's where it actually lives right there it's not SDA is the drive now those are device names and you can see those listed in your slash dev folder on the Linux system near you so it's actually just going to do the same thing we don't have to sit around and wait for this to happen to did a little faster over here and we have a fragmentation score of zero and the average size per extent is 41 kilobytes so some useful information there so there you go so that's pretty much all you need to know about partitioning and fragmentation I hope this helped I hope that you got something out of the video and I'm going to post these notes and while you guys are watching I am actually going to go ahead and fix that typo where is it where is it where is it I'm going to find it there it is right there let's put them a in there because now it makes sense and we'll save that file so before I do it post it you know what's going on all right gang thanks for watching love to hear your comments and suggestions please check out easy Linux on Facebook and if you do give it a like check out easy Linux on the web at easy Linux comm and also check out freedom penguin comm I got an article that's going to be coming up there soon that ties in with the video I posted about podcasting not too long ago and also while I'm on the subject of freedom penguin check out the new series of videos that Matt Hartley is going to be doing I think they're really going to be interesting he's going to throw a subject out to the community get people's thoughts on it and then comment on it's going to be pretty cool how he's going to do that and Matt was on Brian London London Tower show yesterday so last night while I was working on the computer here before I went to bed I had that plane in the background and I was listening to him and Matt gave me a big shout out on Brian Brian's show there so thank you Matt and thank you for freedom penguin I do urge you to check that out that's it gang talk to you again soon

Channel: Joe Collins

Views: 49,108

Rating: 4.9390683 out of 5

Keywords: Linux, OS, Software, Hard Drive, Computer, Laptop, Desktop, File Fragmentation, Ext4, XFS

Id: 1OEidskpcC0

Length: 36min 56sec (2216 seconds)

Published: Thu Mar 23 2017