What Does Formatting Actually Do, Anyway?

have you ever wondered what actually is going on when you format a hard drive or a flash drive or whatever and what all the different settings are for when you go to do so such as unit allocation size and quick format well that's exactly what we're going to be going over today and more including terms like high-level formatting and low-level formatting as well as what goes on at the bit level when you format a hard drive as well as ssds and how they're a little bit different so you should be able to learn a lot and of course as usual the time stamps are in the description now the term formatting actually can mean several different things depending on the context for most people that just means the thing you do that erases the whole hard drive but that's not exactly correct but to explain everything we can first start out with what everyone is familiar with which is the format function in an operating system such as windows you might already know that there are different possible file systems that can be on a drive and this includes ntfs or ext4 or fat32 and it's basically just a different way of organizing and storing and accounting for files on the drive and usually when you go to format a drive you're basically adding file system files to the drive called metadata files which are needed by the operating system to actually kind of keep track using whatever file system method of all the files like i said before but i'm not talking about hidden files like you would see by enabling the checkbox in windows explorer note these are truly hidden files that you cannot view in windows explorer or anything else no matter what it's invisible to the windows api except if you're using very special like forensic software one example is os forensics that i'm going to use in a demonstration so using this program i can go into a drive and actually see the ntfs metadata files in the top directory and they all begin with a dollar sign and these are reserved names that you can't really create in the regular operating system at the top level because they already exist but you just can't ever see them one example is the master file table that mft and this contains info about the locations of all files and folders on the entire drive this is where the map basically of your whole hard drive is so now i can tell you that when you go to format a drive for a file system or at least using quick format i'll get to the other one in a second you're basically just writing or rewriting the file system metadata files and that's pretty much it so you might actually know i hope you do by this point when you actually delete a file in an operating system or windows you're not actually deleting the file on data on the disk literally all you're doing is telling the drive or the file system this file is no longer in use and it frees up the space on the drive to be overwritten but until it is over overwritten that all data is still there even if you deleted it and doing a quick format is basically like the same thing except being applied to the whole drive so you're rewriting the file system metadata files and then basically saying yep the whole disk is now open to be available so when you go and look at it it looks like the whole drive is empty but really everything that was already in there is still in there it's just marked as being invisible you'll also notice when you go to format you'll see an option called allocation unit size but i'll get to that later so if that's what a quick format does what if you don't do quick format that actually does quite a lot more so with quick format it rewrote the file system metadata files and not doing a quick format also does that however it also goes through the entire drive and writes zeros and simultaneously checks for drive errors while it's doing that because as you're writing the zeros to the drive then it can detect oh this one didn't let me write a zero here so then it obviously is marked as an error so it's kind of like two and one but basically writing zeros means that it literally goes through every single possible bit on the entire hard drive and writes a zero there so doing a non-quick format actually does truly erase the entire hard drive and the data on it now sidenotes theoretically some people say even if you write zeros you can theoretically go in and figure out what the data used to be based on magnetic levels and stuff i'm not going to get into that just know that's a possible thing so that's why it's so much faster to a quick format because it doesn't have to go through the entire drive writing zeros to it and you could probably calculate based on how fast a hard drive can write how long it would take depending on the size of the drive so i could actually prove this is what happens using a demonstration i'll just use an old 256 megabyte flash drive that i have and it's super small so it won't take too long now this is already formatted as ntfs and you can see the metadata files in there and what i'll do is add a file in there that pretty much takes up the whole drive so now there's a whole bunch of data pretty much throughout the entire flash drive so next what i'll do is do a quick format so of course this is going to make it look like it's completely empty but then if we use os forensics again to look at the raw data on the drive we can see that all the old data is still on the whole thing next though i did a not quick format normal format it obviously does take a lot longer but now you can see that literally looking at the raw data the whole drive is just all zero so went through and wrote zeros to the whole drive so the old stuff is not on there at all now that kind of formatting we just talked about writing a new file system to drive is known as high level formatting low level formatting on the other hand is very very different and you cannot actually low level format your own drive it's something that is done at the factory now to fully understand low level formatting you're first gonna have to understand the basics of hard drive architecture specifically that hard drives are organized into platters tracks and sectors and there's also cylinders but i'm not going to get into those and you might be thinking oh my god what is he saying but it's not as complicated as it sounds i'll try to make it as simple as possible so platters are literally the physical stacked disks in a hard drive and there's going to be several of them and on each platter there are many many circular tracks they're not a spiral like on a cd or anything but concentric circles then the tracks are further divided up into sectors and again there are very many many of these way more than this diagram is suggesting basically the locations of files on a disk are stored like coordinates or an address so to look up where a file is you might say it's on platter1 track 4 sector 500. and sectors can either refer to just an individual part of a track or an entire section on a disk but usually when someone says a sector they're basically referring to an individual sector meaning the only part of that whole sector that is part of a specific track if that makes sense and individual sectors for one track are not big at all they're usually either between 512 bytes to more modern drives being 4096 bytes so not very big at all so you can probably imagine how many there are on like a four terabyte hard drive or something and actually one sector is the minimum right size for anything if you want to write something to the drive the minimum you're going to have to write is one sector's worth so for example again 40 96 bytes what this means is for example say you are using a hard drive with a four kilobyte sector size even if you just want to make a one kilobyte text file then it's still going to just write the one kilobyte and still take up the rest of the three kilobytes on that sector it'll just be like blank also there can be what are called blocks which are groups of sectors or it could just be one sector but the thing is blocks are actually defined at the file system level and they can change you can set what you want the block size to be and blocks are also kind of like sectors because they also define the minimum amount you have to write for something but again remember because blocks are defined by the file system whereas the sectors are the minimum right at the disk level you can't make a block size that is smaller than a sector size basically it's going to have to be a multiple of the sector sizes and actually when you go to format a drive or something and it asks you for the allocation unit size this is the block size that it's referring to now why would you want to use a larger block size well basically a larger block size is better if the drive is predominantly going to store very large files whereas a smaller block size is better if it's going to be a lot of very small files because if you think about it if you have a very large minimum block size but you put on a bunch of tiny little files then each tiny little file is still going to take up the block size so you're going to be wasting a ton of space but if you have only a bunch of big files then the operating system in the file system is going to have fewer blocks that it has to search through because there are fewer of them because each one is bigger also as a side note to all this i should point out that these days modern hard drives actually use what is called zone bit recording for defining sectors and that basically just means that the outer tracks on the hard drive have more sectors divided into them because the outer tracks are bigger so it would definitely be a waste to only store the same amount of data on the outside as on the inside if the outer one can store a whole bunch more alright so now that we understand that we can get back to what low level formatting is so you have to imagine when you first manufacture a hard drive the platter is just blank it's completely smooth it's not like there's magnetic tracks actually etched into it or anything those magnetic tracks are just magnetically created so that the head can then follow them later but it's not like a record or something where something is physically holding it in there but those tracks do have to be added at some point before even an operating system could format it for a certain file system otherwise it wouldn't even know where to write on the platter itself so that's what low level formatting does it actually creates all the most basic marks on the surface magnetically to indicate where the sectors are where the tracks are that sort of thing so that the magnetic head can then follow them whenever it needs to be used and from my understanding these magnetic marks from the low level formatting are actually quite a bit stronger than what is used from the magnetic fields of just data being written now fun fact apparently back in the day for older hard drives you actually could low level format your own drive you can't anymore and i'll explain why in a second but you used to be able to and actually manufacturers would ship drives with information about probably how many tracks it can technically hold and stuff like that and you could actually do a low-level format using software to write the tracks on there the reason you used to be able to do that was apparently because the low-level formatting markings magnetically and stuff would actually fade over time whereas they don't anymore so it kind of was helpful to let users re low level format their drives to re-add all that really important basic data onto the drive but these days because of improvements in the manufacturing process we don't have to worry about the magnetization fading over time it's going to survive the whole life of the hard drive and actually also what i read is you wouldn't really want the ability to low level format your hard drive because that means that they would have to actually add additional hardware on the drive head to be able to produce the stronger magnetic field which means that the drive head has to be bigger which means that it could potentially not be able to read and store as much data compactly as modern hard drives do so actually the fact that they didn't have to include the extra bigger stuff in the drive means that they could pack more data in for the end user so it's not like you should be lamenting the fact that you can't low level format hard drive anymore because it would just mean that you would have to have a smaller drive now you might be wondering how ssds work with formatting and the storage and all that and they do act a lot differently obviously because it's a completely different technology so ssds consist of multiple physical flash chips and they're all kind of combined almost like platters except they're just physical chips and each ssd is also divided and subdivided like an address of course again just differently than a hard drive so instead of platters tracks and sectors each ssd has multiple physical chips which are dies and each die is divided into planes then blocks and then pages and there's also subdivisions for word lines bit lines and cells which store one bit but for the purpose of this video i'm just going to kind of simplify it as much as i can so you just have to know down to what pages are and pages are a basically string of individual cells and again cells are what just store a one or a zero just a bit now what's important to know is that pages are the minimum unit think that can be read or written at a time so if you want to write only one kilobyte but a page is four kilobytes long you have to write an entire four kilobytes worth to the whole page kind of like a sector on a hard drive but it's for a different reason with flash memory it's actually because of the way it's constructed you physically can't write fewer than the page size but i'm not going to get into that again each page is probably going to be around 4 kilobytes or so but it could be less down to 2 or even up to 16 or so now pages are then combined into blocks which are actually the smallest unit that can be erased so again if you wanted to erase just one kilobyte of data you would have to erase a whole block's worth of it and blocks are usually going to consist of around 128 or 256 pages but take note the term block here is different from the term of a file system block size that we were talking about before is completely different thing okay then after blocks the blocks are all organized into what are called planes and basically a plane is just a part of the die that can kind of do its own thing so a lot of times they're not too many of these maybe one it could just be one plane on a die or maybe two and that just means say there's two planes on a die one of the planes could be reading something in there and the other one could be reading something else or writing something like that anyway all you have to understand is basically that yes flash memory does have arrangements of physical parts that are organized in a hierarchy just differently than a hard drive but it's also important to know here is that because each flash memory cell has a limited number of write cycles this is going to add another level of complexity because this means that you have to add in some logic to the controller which is what determines what gets written where you have to add logic in there that is going to spread out the writes across the whole thing and not just concentrate them at the first ones like you might on a hard drive and this is called wear leveling so this means that unlike a hard drive where everything could be kind of defragmented and put all nice close to each other so it's faster to read you don't really do that with an ssd because you need it to be spread out so it's just kind of all over the place but of course remember because an ssd doesn't have any moving parts it's still going to be way faster than the seek time of the hard drive but here's where all this comes together where we're talking about formatting so we already talked about how a file system is going to store the location on a disk of basically all the files on the whole disk but because an ssd wants to put all the data all spread out it has what is called a flash translation layer and this is software or hardware built into the ssd that takes the location that wants to be written from the os and actually puts it somewhere else on the ssd that is to optimize its lifetime endurance and it has what's called a mapping table to translate basically where the os thinks it is to where it actually goes physically so the os can still use the whole sector notation where it says hey i need this file which is in this sector well then the flash translation layer can look up where that sector is physically and then give it back to the os so anyway my point is with all this when you go to format an ssd it still is going to be doing the same thing logically as a hard drive but the ssd physically is going to be doing something much different but the operating system still thinks it's basically doing the same thing as a hard drive but it's not actually but actually also fun fact there are what are called open channel ssds where the translation layer is not done on the ssd but the actual operating system does all that that's actually what nvme drives do but i'm not going to get into that so i guess i went a bit extra in this video but now you know what goes on when you format a drive and why it formats it what kind of goes into the file system information so basically formatting just kind of creates all the necessary stuff needed for the file system and be able to store where the data is on the drive and then it's taken from there so let me know what you think down in the comments if you liked the video give it a thumbs up and if you want to subscribe i make a couple videos every week so it should be worth it if you guys want to keep watching the next video i recommend is where i was talking about hidden and forbidden file names in windows so that should be pretty interesting you just click that right there so thanks for watching guys i'll see in the next one

Channel: ThioJoe

Views: 431,924

Keywords: technology, tech, computers, windows, formatting, computer formatting, hard drive, ssd, hdd, hard drive formatting, format hard drive, format ssd, quick format, what is quick format

Id: NiefQwuNtjg

Length: 15min 59sec (959 seconds)

Published: Sun Apr 04 2021