An Overview of GlusterFS Architecture Part 1 - Distributed Cluster

hi there welcome to 45 drives recording studio here and gonna sit in here with Brett Kelley a head of research and development and we're here today what we want to talk about is clustering and specifically we want to talk about Gluster FS which is one of the big open source clustering projects a very mature primetime ready massive user base okay and as we said we brought you and I did a video very recently just talk about the difference between Gluster and SEF FS mm-hmm and and what brought up the feedback it brought up and everything we really got to the point where we said we should do another video but we should concentrate and Gluster this time so this is for anybody that's seriously considering cluster FS and and do an installation or somebody that just wants to know more about cluster FS so what we want to do in this is we want to go over the basics of how you would the architecture of your cluster and how you would think it out at a conceptual level and what all the pieces are and what you have to go through to set the thing up so we're gonna go right down to the basics in this video what we want to focus in on is three levels and I guess the three feature levels in gloucestershire we want to go over first of all a simple just a distributed storage cluster and got it and we're use the word non replicated although you'd call it a distributed distributed is the technical Gluster term that they call that volume but you got it it's just a non replicated big cluster okay and that's your most basic scalable single storage space that you can you know virtual storage pool or whatever want to call it platter okay number two we want to talk about a replicated cluster so we replicate clusters the whole purpose replicating clusters so that any of you can have one or more server in your cluster can go down so it's about high availability so keeping that uptime near that 99 9% and last we're gonna talk about something that bread always says is just the crown jewel of Gloucester yeah cheese Geo replication yep and just the and really what makes it so nice is just how it's the word in integrated it is into the cluster and just what will go into the details of what really makes it yeah and so at a 10,000 foot level geo replication is just very simply we put another storage cluster at a remote site and it a sync receipts you got it the replicated cluster is synchronous replication is always there so the mini of downtime you're good the geo replication is an off-site backup but just a routine recovery is what asked recover can and and dead easy to set up yet just works in the background absolutely wonderful so yeah for those who are may be familiar with using our sink or kind of spinning their own little sync tool to back up their file systems would be greatly in love what's cluster fest is to your replication as you go oh I just set it up and it just always works so yeah it's it's uh pretty nice good so so here we go for a dive into Gloucester yeah okay let's get into it okay first thing is use case talk about computer systems of any sort now you can try to talk about in general it gets abstract we really want to bring this down to the ground floor and talk with the way we talk about our store Nader customers who come to us and for help with clustering and you know it comes down to what they're trying to do with it right you got it so so so to make this simple I'm gonna make up a hypothetical use case for you okay sounds good okay so base case we're talking about somebody with 500 terabytes of data okay okay ballpark I'll let you twiddle it around if you want when you do the math just make it up or down a little bit cool let's go for eighty and I don't make it easy so for many terabytes of it so we'll call that usable capacity cool write down a couple of notes here for you Brett sure so so we're talking about 500 terabytes usable okay and we are going to another thing comes in that I looked at hard drives the other day and 8 terabytes is my best bang for the buck when I'm buying enterprise-grade hard drives so I want you to build this on 8 terabyte hard drives thankful it's ranked number one and you're going to use Thor Nader's of course we are 45 drives and we have these lovely enterprise-grade super performance servers that we can use so it's going to get billed out of there and we do have 15 30 45 and 60 available and so it may be what we'll get to talk about the choice you know what what how we make the choice on that when we get there yeah so so ok so there's a use case we want a cluster it's a distributed cluster made 8 terabyte hard drives and you're gonna make it Otis Thor Nader's and somehow you're gonna show us how your team would make a choice as you do the architecture for that ok let's get into it ok so let me grab my notes here now and let's go over some of the some of the items ok sounds good and bright you've told me a hundred times you know if you think about your cluster architecture a little bit before me and and and you make the right decisions then you get an architecture that is straightforward to set up and straightforward to maintain you got it ok especially that's a very important thing like you mentioned with cluster their cluster is a wonderful tool but it definitely plan it out before you get started because we're assuming that like everybody you're gonna put your cluster in if you had a static cluster that never requirements never grew you'd be pretty interesting organization hmm exactly growth is good good for everyone and that is the advantage of it is that it grows and if you think ahead on your architecture it's easier to go off and you got it ok you know on things so it's gonna be made up of servers that are storing aiders notes I like to call them nodes and especially the Gloucester's language as many people know and wonderful world of IT you'll get many synonyms thrown right around for the same word but if you hear me say note during this long during our video Here I am talking about our store Nader's physical store native so the situation no wouldn't server are interchangeable you got it okay which will be raid pooled you got it which we do why do we rate pools the advantages are two reasons data protection number two performance you you put a bunch of drives in big raid volume and you get your straight performance out of that okay okay and next thing we do is we take our our raid volumes and we typically pull them together yep and we create something called a brick that is correct yeah so when we pull these hard drives together and we have our storage space our protected data volume on top of that we all make bricks and you can kind of think in the same way that we made the hard drive pool a storage pool will make the bricks the cluster volume will be a pool of these bricks okay nice symmetry yeah in regular storage we can have hard drives into raid volumes we have raid volumes into pools mm-hm and in this case the pool we're going to call it a brick and so a cluster is simply a pool of bricks you got it right on and that's it that's really the basic concept the clustering isn't it pretty much today and just like a raid volume I can go back like that I got to keep a job here does yeah I'm just kidding it is that simple it really is it really is easy one yeah once you get used to it it so understanding those those structural basics raid type mhm great question to to to practical options for us yeah in Gloucester and really with raid raid type but like rate implementation because there's really at the end of the day you want to raid and then you got to put your filesystem on top of it so there's two options here and ZFS raid and then your standard hardware or software Linux raid LLVM layer on top and then the XFS filesystem those are our two options for okay so so ZFS copy-on-write big fan ZFS yeah you but forever why do you love ZFS well number one is the East ease of administration it is very easy to replace a drive for those who maybe played with the Linux stack of raid LVM XFS while they know ridiculously performant and stable but can be a little time consuming to set up and fix CFS is very very very very very easy to administer so that's my number one reason all kinds of other things copy-on-write making snapshots is instantaneous and I could honestly go on for days but GFS is but the big thing from ZFS in when you're using it you're building bricks of ZFS ZFS arrays it's really what a hard drive dies right yeah that's that's really it and and what you'll see sometimes too is so you might be going well when do I choose one or the other yeah cuz that's really what it all comes down to you can listen to me talk which one I like better but when do you choose which one and really it comes down to feature differences so like I said ZFS is ridiculously easy to set up and administer drive fails zpool replace slot slot like you're done that's that's the driver policeman clusters unaware of all that everything ticking along x vests with LVM underneath and then with the software hardware RAID underneath sometimes well it's just not near that fast so with that simplicity in that case if you're using X x FS you really have to go back right down to your MVA medium yeah and then you go to work at each layer and turn fix that turn the layer back on above and above like that so and is there disruption up in the function your cluster the cluster still works it's just the administrator will have to spend 45 minutes to fix it back to its working state rather than however long it takes to so for all your people out there who are assisted mins you can think you know think forward and that one you end up with a big cluster and over time it's yeah that's just this is a simplicity to that exam pass so when you know XFS I have to do more work on them what are the advantages to play use XFS well your standard that's the one thing right so we're red red Hattie's the the the publisher of Gluster FS and they have set the standard that EXIF se is the supported file system underneath it by all means that if that is completely works to but but exif s you're allowed to do a couple more nice cluster features that you couldn't do with CFS particularly tearing and cluster wide snapshots so if those are two things that you really would like out of your cluster then go the XFS route and luckily I know I just sat here said how annoying it all set up that's where I can plug my wonderful support team here we're here to help you do that as well so I have a question they never popped my mind before if you don't mind me asking there could you mix it could you have a ZFS bricks and XFS bricks so the R&D engineer in me is telling ya I'd be a fun little experiment production no do not do that regret just just it's your it's a recipe for disaster so your answer is maybe technically yes but don't do it yeah if you value your sanity okay yeah fair enough home lab kicking around for fun go to town let's see what happens there so once you decide on which rate implementation you want to do then you can figure out whether your z1 z2 is there your whether your are five raid 5 raid 6 would be the standard ones you you got it I we typically like to stick with raid 6 specially with the size of the drives and the amount of drives we have in our system it's it's you get adequate performance you get good good balance sorry good balance of adequate the wrong word what I meant to say was good balanced performance protection and storage efficiency as and you don't want to go in bigger than two parity and raid 5 well sometimes you need the performance the protection it raid 5 makes me cringe sometimes just with the size of data that we're having now yeah so yeah yeah and and that of course is yeah ok so you know and that brings up a great topic Brett which is how many drives of course for the raid array I mean if I we're gonna be here all day with that question if I have but yeah let's touch on it cuz it is ok when I'm building bricks right and so I mean if I had you know the low density storage I have three hard drives you're not gonna put that in raid 6 it's rebellious right you physically can't yeah yeah that's right yeah and no pointless anyway raid5 at that point you'd have the same efficiency of like a rate it just doesn't make it yeah it doesn't make any sense whatsoever so yeah so when you have high density let's say you have a sixty drive chassis and then this goes for for both clustering and standalone service you know I got a great big high-density servers what's like practically speaking and make it simple for me what how many drives should I put in a raid array I mean I just what 60 you know if I wanted right you got it but don't why wouldn't it 16 ok well that's an absolute no I know I but but why is that enough 62 no because you can just kind of genuinely think but you've got 60 disks in there you've got raid 6 meaning that you've got protection against two of those failing your odds at that case are more likely that you might have too many drives fail before you bring it all back up so there's a protection issue what's my other issue rebuild times do you build time to rebuild times rebuild times rebuild times definitely vary a bit depending on the implementation like we mentioned early GFS hardware RAID MB MDM but particularly with CFS it'll rebuild based on the raid group underneath so if we have one huge rate group it's gonna take really really that's not linear right yeah more put in it yes well you know is really to sum all what I just said up there the rebuild times are gross your chances of data failure go up so just stay away so what's my maximum size if I'm in let's say raid 6 what's the biggest that you'd wanted to see good rule of thumb is no more than 15 ok sometimes you'll see people less but we definitely hard no on anything over than 15 and that's a little mix of like what I said you don't want to go too big but also is a good sweet spot for our as you can see 15 30 45 60 you can get the maximum usage out of our servers okay so so Brett let's say that system in you're working with says you know I really want to use snapshots I want to use Gluster snapshots perfect and so I got to use the XFS not a problem so we're into conventional what anybody who knows Linux knows and the M and L VM yep and and and building up ahead Bill volumes up and pull them using that so let's go on that world and let's say based on what we talked about that 15 maximum it's a nice number fits into our servers nicely let's say we're gonna do raid six arrays okay and and and 15 perfectly between if we nail that down good decision-making process we got through to that maybe you came to ZFS and you come out different sized arrays but let's do that just for this example so we have yeah four and eighty terabytes divided by eight I'll get sixty hard drive so you got it that's correct yeah okay so now if I just built my cluster at sixty would be short capacity but we're need more than that we're now elf word adjust that number right you got it because we're gonna lose some we're gonna lose some space here to well there's three considerations we have to get into parity we're gonna lose some space to the redundancy induced by our raid arrays second we instill a 90 percent quota on each raid volume such that you don't fill it up all the way this is what happens if my drugs fill up if my radar fills up full I Oh pause of the clip of the raid array therefore your cluster so it's one of those little added protections of oh I just bumped my quota rather than oh my clusters at a hundred percent and I need to go in and do some manual deletions of things no one wants to do that so that's where that ninety it's kind of like a little fudge factor to keep you safe keep the end-user safe and now we realize that where everybody that sets up the cluster you got it and a rate array for that matter right yeah really just just never run anything to a hundred percent really and then the third one is leave room for expansion okay as in like you said earlier you're gonna be growing your cluster is gonna be growing your company is growing so leave a little extra room there maybe you did better the shielding factor we've ever had a complaint from somebody said during I put too much space in my storage no but they always get excited like oh my god I've never seen that many terabytes in one spot before alright so yeah that's pretty fun all right so let's do this weed sixty hard drives you got it and so that's four sets of fifteen mm-hmm right four sets of fifteen okay so we have four sets of fifteen and we're gonna lose two OT to those so we need add so we needed count for who's 8 ya 4 2 s 8 so 8 8 terabytes so 64 terabytes so 64 for more terabytes so when in terms of number of hard drives so it now becomes 68 hard drives minimum to get that usable capacity you got it okay so let me write this down Brad I got to go to it and write this down again I just make this little more tangible to myself okay so we can we needed 60 hard drives mm-hmm number one we have to add at a minimum we have to add eight hard drives to account for a redundancy to keep our well 480 usable yeah and that's just roughing it in yep just more refine what we have to do to that okay so I got 68 mm-hmm now we need to do some real math here we need to up that by 10% but up and consider that's 90% so we'd have to multiply that by 100 divided by 90 and we're just sitting here roughing this out for people so we're instead we're gonna take a little more than 10% of that dough let's say seven drives so we got to add on about seven drives which that works great too because we're gonna be building in groups of 15 so that puts us at 75 so now we have 75 run out of board space here Brett yep so now we got 75 we've accounted for the 90% and a little more than that too so we've got a little growth room too and we can still cleanly expand our cluster okay so okay cool so 75 I need five groups of 15 will do this roughly to hit me about where I wanted not that 480 and then you're really faced in a situation so I'm looking at that and I'm going I need five sets of 15 if I bought three 30s that would work I now have the ability to put six groups of 15 in there that's correct but that's great because I'm expanding that or maybe right off the bat I want to just jump up a little bit bigger and you got it okay so 45 I could do two 45-second have 90 mm-hmm and or it could it would it be possible if I wanted to do for example I wanted to do a a 30 and a 45 it's possible that's there that can be done okay we'd recommend keeping it somewhat balanced as in it's easier to expand that way okay so I should balance them better to use things identical but not essential right not essential and that's that's the thing like sometimes a use case paint see in a weird corner and you got to go off the off the beaten trail and that's fine we can help you do that but typically we would like to keep things symmetrical Minister exactly exactly you set up easier to expand so well let's look at that and say if we want to keep it symmetrical so shall we choose 330s let's do three thirties that actually really helps when we talk about a replicator volume but we'll touch on that point as we get there [Music]

Channel: 45Drives

Views: 5,472

Rating: 4.8350515 out of 5

Keywords: storage, clustering, storage clustering, big data, 45 drives, storinator, glusterfs, distributed storage cluster, replicated storage cluster, geo replication, cluster volume, raid volume

Id: D6lglo1cvEA

Length: 22min 2sec (1322 seconds)

Published: Tue Jan 15 2019