All about the unRAID Array, how data is written and how the parity works

hi guys and welcome to another video in today's video are going to take a close look at the unread array and what it's composed of and how it works we will see how data is written and how the unread parity works so let's get to it [Music] so in this video we're going to discuss all about the Unrated array and we're going to look at the following things one the components of the array - how data is stored on the hard disk 3 the difference between a none raid array and a traditional array for how parity works in both single and dual configurations and five other cache drivers used in unranked so firstly let's look at what makes up an array non-raid a none raid array can have up to three different parts one data disks now this is where all of the data is stored things such as movies music work data you know whatever you store on the array now the second part of an unrated array are the parity bit now these so it's still only data as such what they store is the parity information and this information gives the data redundancy and it can be used to calculate what the data was on the hard drive should it ever happen to fail and then this data can be used to then rebuild the disk with all the data on it in unraised you can have either one or two parity Drive and one parity Drive and allow you to recover from the failure of one hard drive while to power two drives will allow you to recover from the failure of up to two days to drive and the third type of hard drive you'll find is a nun raid array is a cache drive and this can be used to speed up right to the main array data can be written to a cache drive first for speed then later automatically move to the protected array so that is the three things that make up the unmade array but what is an array now I'm sure most of you watching this will have heard of a raid array before so for those of you that haven't it's been around for a long time it's roots going back as far as 25 years and raid stands for redundant array of independent disks and whose purpose is about encoding the user's data in such a way that it can tolerate the failure of Hardware without losing data now many people incorrectly consider the unveil array to be the same as either a raid 5 or a race 6 array well this is not true there are of course similarities only in the fact of how many drives the needs of full parity and the parity calculations were to use the main difference seeing how the data and the parity information is actually stored on those bits but before we get into that let's just go right back to basics so we can more easily understand how everything works let's talk about how the hard drives actually store this data ok so we all know that computers use binary either a 0 or a 1 this is also true how data is written to a disk the data is written in a series of ones and zeros and yes this is true whether it's a movie or a bank statement it's made of ones and zeros well if it's my bank statement it would probably only be made in serious well now anyway seriously you get the idea the data stored on the hard drive is ones and zeros no matter what the file contains and so there are two types of hard drives one the traditional mechanical hard drives and HDD and this type will normally be used in an unrated or a 4 beta and parity drive and the second type is a solid-state drive an SSD and this type is normally used in the array for a cache drive in a mechanical Drive the part that stores the data is called a platter and this is basically a round disc that's made of a nonmetallic substance and then this is covered with a really really thin layer of a magnetic material so we've all heard that hard drives are magnetic media but how does a magnet store data well the data is stored digitally in the form of tiny magnet Iberia's on that platter my each area represents a bit remember at school when you're told magnets have two poles a north and a south well it's true they do and using this fact is how the hard drive can represent a one or a zero for instance as the platter is rotating if the north pole arrives before the south which is a1 and if the South Pole arrives before the law and it's a zero okay so you get that magnet fit now for SSD so use magnets so how do they store their ones in zeros well SSDs work in a similar way and the fact that lists all ones and zeros they don't do it magnetically but using transistors arranged in the grid with columns and rows if a chain of these resistors conducts currents that it has the value of one and if it doesn't conduct current it's a zero so whether the current is blocked or not to some transistors determines whether they are ones or zeros so now that we know how data is stored on the drive let's have a look at the differences between how a none raid array distributes its data compared to the standard ray config well with a single disk all the ones and zeros that make up that file being written are written to the disk in its entirety as as the only place to read data to go however with a traditional raid array it's normally different the ones and zeros that make up the file which is being written are striked across the hard drive by placing a part of the file on each Drive of the array this has both advantages and disadvantages the advantages is it's far faster to read data as it's coming from multiple disks all at once the disadvantage is if one of the drives should fail all data is lost on all Drive this is because for every file written on the array if a disk is missing then a part of the data for each file is also missing for this reason raid adds parenting to the mix and this allows the data redundancy so should if I fail it will allow for the data to be rebuilt in base five it needs one extra Drive the parity and in raid 6 it requires 2 this is why people compare the unread array to these raid levels due to the parity drives however again just like the data being striped in raid 5 or 6 parity informations also stripes across the drive raid 5 and 6 is fine so long as only one or two drives fail because it can be rebuilt it should another Drive fail or fail while C array is being rebuilt again all data is locked from the whole array so with this in mind the Unrated ray goes about its data storage slightly differently a nun raid array doesn't strike the data across the disks it writes complete files to only one disk a file is never split or over multiple disks so should a drive fail the most data that can be lost is minimized to only that disk we in most cases as I made has up to two parity drives the failed disk can also be rebuilt so the advantages of unread not striping data and parity on the following one should rebuild proved to be impossible for the failed drive we don't lose any data on the other drives which haven't failed to unlike raid 5 or raids mix we can add extra drives to the array expanding its capacity as and when we need it and three we can use different size drives in our array they don't have to be all the same size now the disadvantages for not striping is we do need speed in our array compared to a special array the unread combat miss with the use of a cache drive which we'll discuss later as well even if the parity drive sales that drive can be replaced and parity recalculating references are nicely to the next topic to explain how parity works let's talk about single parity first now let's take an array of three data discs and one parity disk now parity is calculated from a calculation that's made across all of the sectors of all of the discs and obviously discs have many sectors but for simplicity we're just going to imagine one sector on each disk for this example so obviously the parity calculation is done for each and every sector on all of the discs now remember earlier when we're talking about ones and zeros on the disk well what unread does is it zeros all of the disk would it formats them before it goes into the array so it knows that all of the discs have a zero value on all of their sectors so we'll represent this with a zero on the top of each hard drive so now let's choose which of these drives will be the data drives and which one will be the parity drive so let's have this one as a day to drive this one as a day to drive and this one is a day to drive and the final one here we'll use this as a parity drive so let's swap this zero and put this zero on which says even on the top now we know we have either ones or zeros so one is an odd number and zero is considered an even number so I've marked the zero on the parity as even single drive parity is done using a calculation called XOR or XOR how this works is the number of ones across the axis of the drives is calculated if this number is even then it has even parity it has an odd parity if the numbers of ones is odd so let's demonstrate this simply with our one sector drive example so the number of ones across our data disks is zero and zero is an even number so it means that our parity is also a zero and also even so should the data on this disk here be a one they found that the ones on our data we only have one one so that means it's old and our parity is odd okay so let's make the data on this driver 1 as well so now the data drives we have a 1 1 and a 0 so how many ones do we have we have two ones so that's even so we have even parity now okay so now let's change the last drive to also be a 1 now our date drives the three ones so obviously three ones is also odd so that means we're now going to have odd parity and again if we change this to a zero we've got two ones so we've got even parity again so you can see how the parity date is written but what happens if this drive here fails how do we know what used to be on the drive so once the new drivers put in we can actually calculate what should be on here so we know when the drive fails the parity was even so let's add up how many ones are in our data disk we've got 1 2 so 2 is even so the data on this disk it had to be a zero otherwise it wouldn't have had even parity ok so that's how the data calculation works in single parity for rebuilding a data drive but if we want to add an extra drive to the array that's really simple as well before a drive is added into the array in an raid and raid will zero the drive and so therefore the zero on the drive doesn't affect the parity at all so the parity doesn't change when an extra drive is added so in this case here the parity would still be even so now let's look at the same but starting from an odd parity again adding a zero drive makes absolutely no difference because the parity stays the same and it's still odd okay so I think that demonstration looked a little bit like a dodgy car dealer in Las Vegas but hopefully it got my point across so that sell single parity works now a nun raid we can actually have to parity drives I've heard some people say that the second parity drive is just a mirrored copy of the first so it's only good if the parity drive fails now this is not true dual parity will actually allow for two days drives to fail in the array and trip to know to rebuild that now the second parity drive doesn't use X or parity calculation it uses another more complex calculation based on the lowest field algebra so with this additional power to calculation on top of the first X or parity calculation to drive failures can be tolerated having this dual parity is especially useful for large arrays with multiple drives in it as there is more chance of two drives failing one critical time the drives can fail is during the rebuild process so dual parity protects against this the dual parity has another advantage as well when it comes to rebuild encountering an unrecoverable read error or aur e now this type of error happens when reading data on modern satellites about once every 12 terabyte to say to read one of these errors can theoretically occur with discs getting larger and larger and up to the size of 8 terabytes becoming more common this is becoming more likely to happen during the rebuild so what happens it unraised you're in the rebuild if this error happens well luckily the rebuild doesn't just fail as it can in raid 5 what happens in unread is in single parity configuration if an unrecoverable read error occurs preventing reconstruct of the target vlog the target blockers not written to at all and an error message is written to the system log so this would result in a missing part of the data a hole so to speak so the file at that position would be damaged in dual parity configuration however there would have to be two or more disks with unrecoverable errors for this to happen also in the dual parity if just one other disk reports an unrecoverable read error unlaid rebuilds both the disk and the target disk the one being reconstructed and writes to them both so dual parity has been a help to not having any data gaps in a rebuild should you experience a ura join the program okay so that's how a single and dual parity work in the unreal array we know we can recover for up to two disks failing that we also know write speeds to any party protecting array of slower so let's get back to the cache Drive the third component of the Unrated array it is recommended with a cache drive with an SSD this has the obvious speed advantage but also SSDs have no moving parts and these theoretically should not be so prone to failure so it makes sense to have an SSD for the drive that will probably have some of the most use on the array now the cache drive isn't only used to speed up right it's also used by a lot of people for storing VM images and documents so should your cash flow fail you would lose any VMs or doctors that you have stored there so to protect against this it makes sense to use a raid 1 btrs mirrored configuration more information on how to set this up please see my video on installing a cache drive ok so that's how our arrays work in an raid so guys I hope you enjoyed the video and found it interesting and if you did then please hit the like button and if you want to see more videos from me then please subscribe to the channel anyway guys whatever you're up to for the rest of the day I hope it's good and I'll catch you in the next video

Channel: Spaceinvader One

Views: 82,265

Keywords: linux, unraid, unraid array, raid 5, raid 6, parity, single parity, dual parity, xor, reed solomon, lime technology, lime tech, unRAID, nas, data, how data is stored on a harddrive, ssd, hdd, hard drive, how a hard drive works, home server, server, kvm, docker, qemu, All about the Array, how data is written and how the parity works in unRAID, tutorial, guide, how raid works, how unraid works, difference between raid and unraid, double parity, what is parity

Id: HybwCOVDg9k

Length: 17min 39sec (1059 seconds)

Published: Fri Feb 03 2017

Reddit Comments

[u/spaceinvaderone]

This video is all about the array and how it works in unRAID. It discusses the 3 different types of drives that make up the array. You will learn how data is both written and stored on a harddisk and an array. You will see how both single and dual parity work to protect your server from hard drive failures. You will see the differences between raid5/6 and the unRAID array. Hope you find it interesting !!

[u/canyouhearme]

That's OK, but misses out on one of the things UNRAID is moving towards.

At present, by default, when it writes to the array UNRAID first reads the disk it's going to put the data on, and the parity. If the data bit is going to change it then writes the reversed bit to the data and parity drives (Read/Read:Write/Write). The advantage here is that only two disks are spun up.

With 'turbo' mode it instead reads all the data disks (except the one where the data is going, calculates the XOR, then writes the data and parity disks (Read/Read/Read:Write/Write). All drives are spun up.

Why does this matter?

Well, in the first default case the drives have to continually swap between read and write modes - and it turns out there is a latency in doing this. That slows down the overall write speed as the drives are continually swapping modes. In the 'turbo' mode each disk is either in read or write mode; speeding up the overall operation at the cost of more power usage and more disk usage.

The aim is 'auto' which hides all this from the user - but currently it's an option to set.

See http://lime-technology.com/forum/index.php?topic=52122.msg500304#msg500304

Someone did some benchmarking, and in a best (though quite typical blu ray copy) case got :

Normal        Time  7:20         Ave  49.75MB/s       Max  122.01MB/s
Turbo         Time  4.01         Ave  90.83MB/s       Max  124.14MB/s 

although less for more disjoint cases.

[u/mcai8rw2]

Oh my very days!

Thank you for explaining how parity works. I always thought it was some kind of ancient magic that allowed parity to work. I'm still not sure i get it 100% ... but your vid has helped.

In your example you use a one bit area on the disk to demonstrate parity.

Might be a silly question... but is parity calculated bit-by-bit in Unraid... if so there has to be some kind of exact and 100% accurate filesystem syncing between read heads and portions of the disc. Afterall even the tiniest inaccuracy in syncronisation between the bits on your HDD and its completely useless.

You know what... i'm not sure anymore... I still think there's a little bit of goblin magic involved.

[u/nogami]

One of the things that just threw me a bit is when one of my drives went "unmountable" (probably due to a power bump that I didn't catch before my ups battery failed), and couldn't be repaired by mounting the drive offline to replay the Log file.

No physical damage to drive, just corruption of file system.

Only zeroing the log allowed it to come back, but still don't entirely trust it.

Scared the heck out of me. Fortunately I have everything backed up on an offline drobo nas, but should never have happened. Seems that xfs just isn't that robust when mounted in the unraid style.