Architecture and Build - Building a ZFS Storage Server for a Mixed Workload

hello everyone welcome back to uh episode two of our uh architect day storage server video series uh where we do a little bit of whiteboard we do a little bit of lab building and uh well we just do a real-life kind of scenario and talk about all the details so uh you're joined by familiar faces you've got me brett and uh mitch hall over here and uh well today we're gonna talk about seth again fresh buddy what all we do is talk about seth no no no no we don't are you sure about that well everything more about seth we've talked about this already but ceph clustering we love seth for those who have mounted a cefs share before what are my options for tools to deploy a ceph cluster it is called ceph georep this week i am talking about sapphire scuzzy chef clusters i guess so what do you want to talk about well i think it's time to give a little bit a lot of zfs we haven't really talked about it very much it's a big part of our business i'm a big fan of zfs sold all right we're doing a dfs one [Music] [Applause] [Music] okay so we're gonna do a zfs architecture video and well uh mitch you got your wish what are we talking about today oh god put me on the spot here okay so well through the run of a day as an architect here i talk to businesses all the time big small medium sized and for the smaller and the medium-sized businesses what i'll see a lot of times is they have a lot of storage needs you know they have a need for virtual machines they might have a need for even a database or two as well as file storage right everyone needs file shares as well you need a little bit of everything a little bit of everything but they may not have a budget to have a dedicated sand for their vms and a dedicated uh high performance legacy way of thinking you have a box that does one thing and just that just does it so i mean zfs is a really unique architecture and that that comes with a lot of really cool tools and i think what we can do today is really show off the way that you can leverage everything that zfs has to offer to really make a compelling solution that can do all of those things okay so we're going to architect a single server zfs file database vm store exactly okay well here i'm going to give you some rules okay let's get to the rules all right so this thing's got to be like here i'll just give this a little like let's just call it workload okay sorry guys i have the handwriting of a doctor but and i before we even start i want to preface this by saying um that in no way am i gonna say that a dedicated cfs box even if we take all the tricks and everything that zfs gives us is it not going to be better than a dedicated high performance san a dedicated file server a dedicated storage for your database but what this will do is give you something that is fantastic at serving everything on a good good budget and it'll be good enough for most people absolutely yeah yeah you don't need yeah okay i see what you're saying yeah all right okay so this thing is going to have some db storage let's say uh you're going to want to start with 10 terabytes on there um and uh what's 64 64 kilowatt record size that's a common one we can always tune for other record sizes as well but we'll just we'll have to have a target that we want to hit so let's do that right to small and i did so then we're going to have some vm store here damn storage so i'm thinking like infrastructure right people need to store their dns server their active directory potentially a mail server uh things like that five to ten so we're not actually putting like major data here it's literally the operating system is that anything else we can put on the yup okay cool uh nothing special there and the rest let's just do some good old general file sharing yeah um where we do some smb nfs yeah so just just general storage for everyone and we'll just use the rest of the space okay and as a general rule your virtual machine infrastructure is going to be more i o intensive or random i o intensive than say a file share which may be more streaming reads and writes and so this is why we're kind of defining all these to make sure that we have a z pool that can handle all of these simultaneously on the same pool yeah because i you'll get into it but like this is one of the coolest parts of cfs how you can segment certain parts of the pool exactly tune accordingly into recording exactly you got it okay so one other uh one other rule that you have oh i literally just crossed that out whatever yeah um we'll keep that on the board to get something to reference okay but um oh we're back to the squares again are we oh it's a rectangle well yeah true enough and it doesn't look like a trapezoid slash parallelogram or whatever whatever you do last time yeah oh yeah i'm sorry no i already got a hardware and some hardware in the lab for okay we have our q3816 so what that means is there's the mobile chamber you've got 15 hard drives okay available to you you've got seven hard drives in this row and the rest is 16 ssds okay and then up here i have a single nvme for you one big 1.9 terabyte version i can work with that and that's actually great because you know in a traditional z pool you would imagine there's not really any good way without taking advantage of everything zfs to off has to offer to build a pool that uses all of these in just regular v devs so we're going to have to really kind of know what we're doing with zfs to make use of all these in a single z pool which reminds me too because you said the magic word vdf disclaimer out to everyone we're going to dive right into some zfs terms this isn't really a zfs 101 we have a lot of stuff whether it's videos actually the latest blog we put out is a great primer on on all the cfs terminology and what that means to um uh traditional raid stuff and then and just more of how it breaks the mold yeah so for any newcomers and if you say we're newcomers to cfs and we say in words you might not understand go check out some of that material and come on back yeah that is a great point because we're not going to spend the time everything we talk about we're not going to break down what it is we'll break down the important bits but yeah definitely reach out and uh sorry check out our other content that we have for sure reach out too yeah yeah anyway we'll leave this up but i'll get i'll get rid of this picture cool and um so that's the hardware you got we won't talk network here let's just say it's 10 gig yeah just 10 gig we don't have to get any fancy stuff there um uh i'll turn the marker over to you and get started just what are we building well i think what we're going to do is the the vast majority of our storage is going to be the file sharing right so i think that's where our v devs are first going to be kind of what we're going to focus on at first and so with a file sharing or any solution really well raid z1 is always an option it's not an option that we'd ever really recommend for productivity one just one period one parity one drive a parody right you lose that drive you're essentially the pucker factor kicks in and and you're gonna help yeah i like that you're waiting for that drive to re-silver and if it doesn't well you have some data loss so we're going to definitely build on a raid z2 so um let's go with i won't make fun of yours no no this is big buddy so a v dev is like a rate group right yeah not specifically striped but it's taking two raid z2s and and putting them together so let's say we've got is that eight nope there we go so we'll do raid groups of eight yeah and we're just spinning storage just for everyone i gave you that hybrid earlier but we're talking spinning storage we have 15 and then we got seven we have 22 hard drives available too okay so um right 15 plus 7 is 22. good so it'd be not yeah so with the eight grade group i guess it doesn't really work but let's just say we only need 16 for now so let's do the same thing we'll go we'll have eight oh okay we're only going to utilize yeah we'll use 16 drives for now just just to uh kind of make this work okay um all right so you've got two drives of parity in each one so you can lose two discs from either raid group obviously if you lose more than two from a single one you're in trouble but that's why you stay on top of your failures and rebuild and re-silver when you have to okay okay so that's going to take care of this right it's going to take care of our file sharing we're going to have and this is i guess we'll call this our mark level adaptive replacement cache this is what we're going to have to read back out to our clients and we'll also even call this our ram as well because what i want to show off here is a lot of our file sharing data is going to be what we call async asynchronous rights what asynchronous rights are is they come from the client they come into the memory they get written here as soon as they get written here they get acknowledged back to that client so they go right back without ever committing into the disk long term okay so cfs uses something called transaction groups so every five seconds what dfs will do is it will flush these transaction groups back to the long term storage makes sense okay kind of like a right back cache in a way yeah it really is like and steph or seth here we go we're talking staff already um zfs does autonomous rights so if any of these writes that it's doing is is actually changing data that already exists it unlinks the old block and links the new one in one single transaction so you don't have to put data corruption or anything like that copy on right copy on right you've got it now there's another part here that's built into a z pool by default called a zil zfs intent log and for anyone that really understands zfs i'm not drawing to spec like this isn't what it actually looks like but it really helps you visualize so a zfs intent log what that is for is for sink right so if we have async there we've got sync here so sync writes can be anything well it really depends whatever the client if the client write has a sync flag well it's going to go sync so things like virtual machines by default are going to be sync rights databases are going to be sync rights these are rights that you want to make sure commits to your long-term storage before sending that acknowledgement back out right because you want to make sure that that's there on disk before it sends it back to clients so you can think of like let's say if you're a business and you do transactions with money every day some of those transactions you want to make sure they're committed there and you have that on file before it goes back out because if the server fails or if power goes out things like that okay so sync writes are going to be the vm and the db so this type of solution right here what would happen is the write comes into the client it's going to go to the arc as normal because it still uses those transaction groups so it comes in but it also at the exact same time goes to the zil and it writes there as well and its intent it's intent of what it's going to eventually write and so as soon as it gets hit there and gets committed it immediately goes back out to the client now eventually the transaction group flushes and it gets written to disk as normal but this is to make sure that if power ever did go out and when you come back on you could re-read this zill and say we were going to write block a b c etc and this is what one of the things that makes cfs really unique compared to other file systems absolutely is that it's atomic in its way that it is acid compliant yeah just the database word right and it's that idea that it needs to guarantee it functions more as a database than exactly streaming and that's what it is yeah yeah cool and this zeal is always present always present it even has to be there it has to be there exactly so by default it lives on the spinners exactly and so what you're getting there is you're competing i o right because you're you're sync right going into ram and it's getting committed here but it's also being written here and it's got it it's got to do this first it does that first exactly so you're telling me that it wants to do a small somewhat random database transaction yeah on spinners before it can write it before i can actually write this number exactly there is evidence of the zfs intent log being able to do like right at once and then instead of committing it doing pointers but every research that i've come up against is that that's a really bad idea because it makes every right in your zill essentially an uh a random right so it's not a good idea so what we talked about here is the way it works so there's a way to move this deal we said it's on the display funnel but we can put it somewhere else exactly and that's how we're going to make use of these extra ssds and these nvmes that we have okay um okay so like we touched on right the file sharing on its own would have done great with that but now we throw these in the mix and you've got everything competing for the same disks so what we can now do is we can say we want a slog device all these funny terms right if you're not familiar with zfs so what we can do is we can create a new v dev in our system called a slog and what that is is a separate zfs intent lock so typically we like to mirror that right so um yeah because you don't want to lose it right if it dies yeah that's a few seconds of dirty data that you're going to lose or sorry that's clean data but it's it doesn't have to be very big right no actually very very small because at the worst it's like 20 30 seconds of data that that is ever going to stay on there so these transactions so rule of thumb should be then like whatever your like max network bandwidth is yes times 30 seconds exactly you've got size for that because it can't ever ingest more you can't ever ingest more than what it can do over that many seconds you got it exactly okay actually i wasn't gonna bring that up so thank you so what i've got here is a new v dev and and well it is part of this pool i'm just making it separate so we can kind of is this the slog you're showing us here yes okay we're doing this vlog okay so all right and so now what you can see is when that sync write is coming in rather than having to come in here and compete with the rest of io it goes to these and i'll remind you much faster ssds than the spinning storage as soon as it gets here it can commit that back to the client and say hey i've committed that you're good to go keep keep feeding me data so now this is unburdened and can still do this and this is doing just a dedicated function of handling your vms and your database so the zil's living on these now right yes you drew four i really only need do you really only need two i was like i just want to show you how good you can draw squares rectangles yeah exactly um we've got like i think eight of them in there so i was like hey i'll just take four of them but yeah two two's good if you want to double them up and do two mirrors you can but yeah so we got four ssds okay so that's great that is going to help your vms any rights in your vms is going to help we talked a little bit about the arc but i'm going to explain a little more so where the arc is an adapter replacement cache so every file system has a caching mechanism of some kind most of them or all of them that i know use the actual operating system cache like the page cache of the os cfs is very different where it can take a whole lot of your ram your memory and use it as a as a much better type of cache so it's not the os cache it has its own it's called the arc yeah the os cache is optimized for um being a good general purpose general purpose you've got the arc is optimized for storage just for storage system just for zfs so what it does is it has the least recently used like the regular page cache where the oldest stuff gets pushed out but what also has is data that gets read more often gets a higher weight in your arc so if you have some vms that get red very very often you're going to see that a lot of that is going to sit in your high performance arc and you're going to get some great performance vfs has some cool tools for like scenes and seeing that and seeing exactly seeing like how much how often your arc is being hit and if it's very low well then you can do some tuning because that's the hard part about cache it's hard to know if you're using it very very so yeah cfs makes that easy yeah they very much do um okay so that that covers like i said um that's going to help with both of these um the arc is going to help with all of them and then the last one that we want to talk about and it's it's actually not talked about very often from what i see so a relatively new feature it is quite new yeah uh is the special v dev and the special v dev's main purpose is to offload the metadata of your file system onto a new v dev so by default zfs will hold a lot of your your active metadata in this memory pool so a lot of your metadata will stay here but again it can't hold everything and there's going to be some that it has to go back to disk for so let's just call that your metadata no one's just scribble that man okay so what we can now do with our nvmes that we have left over and i'm gonna just remove that for a second is build one more v dev and this time it's we're going to do two uh we only have one in our system down the lab but that's just because that's all we had you're going to want to mirror this up as well because it is very important and more than anything maybe even potentially a three-way mirror because if you ever lose your your metadata your file system everything is gone then right so you may even want to do like a triple mirror okay but we're going to play fast and low yeah just pretend just for tonight yeah yeah so what this does is like i said it's special because it's the special edit and so all of your metadata now is being removed off of this and in a file system uh when you say this you mean like this video stores the actual regular it's the raid z2 yes now in a file system you can think uh file names the size of the files the permission bits all that stuff are reads and writes that your array has to do alongside the actual file data itself and they're very small very very small exactly and so with spinning storage which isn't fantastic at small reads and writes that can bog that down like i've heard terms of like up to 50 of your file system workload is metadata and that's very very positive so like just for those it's like when you would be searching metadata it's like literally if you go you've got this thing through a song by smb share you go to file explorer and you search a file yes and it's doing the displaying as it loads it's because it's a metadata workload and it's reading it all off the disks so what you're telling me here is where we move that over to the fast special device should just be able to see much faster so then the spinners can do its job can you do it it serves the data and let the fast things do the exactly and that's really really where it's fantastic right you're letting your hdds do what they do best and you're letting things do things better [Laughter] and so yeah like like you said the search or even like an ls of a directory like just so fast yeah um okay and then the one last thing i want to talk about the special dev two is the fact that you can also do a small block workload on there and what i mean by that is in our z pool for our data sets we may use different record sizes for different use cases so in the case of our db workload we may we would definitely want to match our zfs data set record size to our 64k record size of the database matching block sizes really helps performance um but what you can do is so let's say you have a 64k record size for a dataset or a 128k for file sharing what you can do is you can set smaller blocks so let's just say in the case of this we want to do 32 that's a 2. that's a 2 32 k block sizes and smaller instead of going here and being written here it goes directly to the special v dead so cool so what that really comes with is you end up with kind of a hybrid tier at that point right yeah any small writes bypass your htd and be written to your nvme high performance storage and with all that said we still have the arc cache and everything it came out too yeah yeah you said we got tears on tears we got cheers everywhere and safe tiers of that because you know what i mean many times when you start putting all these caching mechanisms place it's kind of sketchy right but uh because we're playing in zfs rule set we're actually doing something that's very very safe everything's acid everything goes through this slog that's right oh that's you've got it and everything is transaction and atomic and so we're not you know deleting a block before we write another one everything happens at the same time so there's very very low risk of corruption and things like that um fantastic yeah so i think that right there gives us a fantastic single server which doesn't break the bank rate because you're only using in comparison to the rest very small amounts of flash compared to your spinners and you're getting a really good balanced uh cfs array cool and when we we're going to have a data set for each of these use cases right yeah absolutely okay so we'll get into like what what parameters and stuff we want to change and how we're going to cut that up when we get in the lab yeah i think so people may be surprised because uh for the vms they may expect that like if we're going iscsi we may be using z vols uh many times we'll actually just use a data set and use a file i o over iscsi if we're using proxmox as the onboard zfs array that will use evolves but i think for us because we're using iscsi we'll be using file io but yeah we'll show that stuff off once we get in the lab okay sounds good all right so that's a good whiteboard partner yeah like you said this may not be as clean as yours i don't know man the last time we did this we had squares look like circles we're getting a little better yeah so we're going to build we're going to take our q30 our a16 we're gonna build two v devs of raid z2 on the thing we're going to use the say it ssds in it as our slog devices um and we're going to use our nvme as the special perfect and then we'll hook it into everything sounds good to me let's go to lab let's go to the lab all right so here we are back in the lab don't mind the echo um we got one chirp at least from high ceilings it's a nice spot yeah we like our echo no uh we're not here to talk about that we're here to finish our architecture video so we're going to take all the good stuff that we talked about in the other room that we drew up and we're going to do it on our q3816 here sorry mitch i pushed you to frame a little bit oh good but uh let's let's take a close-up look at this thing yeah come on in um first of all i want to say i tricked all you guys this he's a seth video oh you've got to be one of these nodes i pulled apart stuff you can never quite get away from that uh follows everywhere let's take a look so what are we working with here mitch all right so we got our 16 drives that we talked about 16 hdd drives that is uh since this is a hybrid 16 we've got our four ssds we've got lots of room available for expansion on both sides and then also we've got our nvme so this is our nvme carrier card this can fit up to four nvmes on there um so yeah cool the long short of it which i was thinking after we finished that first part like sizing things are relatively easy but i was thinking about it the if we are going to put small like the small metadata or small block reads or rights or geez i'm screwing that up right like the small part of it that we can serve from the special vdef it's kind of variable on it's variable on how heavy your workload is for how like how much you're writing data yeah yeah so i guess what i'm saying is sizing that special v dev can kind of could get difficult to get a little difficult and that is very true there's a few things that zfs does to combat though one is if you set that let's say that that cfs small block size to 32k um and all your 32k rights will go there until it hits to about 15 percent left of space let's say 85 85 zfs will be like okay you're pretty full and it'll start moving those back to your spinning discs to reserve that last space for your metadata gotcha um but you totally don't even know how much space in general they're going to exactly i don't know how much of that will be metadata yeah so that's nice protection it is great because i guess that's the flexibility of this we are showing this to show spinners state ssds and nvme but we could very well populate this whole thing have a huge special v dev absolutely yeah rather than being limited by these four we could have many more we're flexible yeah uh all right so let's quick toilet hardware this is our beautiful hybrid unit um let's uh let's jump to houston and build the thing let's do it okay cool all right so this is our landing page i think most people have probably seen this before so let's hop right into the cfs tab so no pools have been found make sense because they're not there yep so i guess the starting point is creating those raid z2 v dips uh i'm laughing at my comment uh there's no pool there because it's not there it's not there because it's not there it's like that uh you can tell this is a birch street because the way it is you want to see that video no oh no okay well we're watching we'll find that later yeah so what are we calling this thing uh storage i've decided that makes sense okay it is storage all right so this is this is uh grade z2 that's what we want correct um and the way this works is we add one v dev at a time so we're going to create a tool with one v dev and then we'll add the second one then we'll add this log then we'll add the devices we don't want to add all 16 at once or it'll just show up as a single big old v dev cool that's not good and for anyone that doesn't really know our ui this is really cool because we have our device aliasing that gives us the actual id right to the actual slot number and we'll probably show more of that with our discs later yes yes yes yes exactly so when you make this pool you gotta address the disc there's multiple ways to do it you can do it by its regular old sd name but everyone knows it's not best because it will change change on reboot yeah and the other thing is where is dev sdc i don't know yeah i don't know either so device alias is a way that we can address disks by their slot yeah anyway i just repeated what you already said but hey we headed home okay there we go okay so we'll make our first v dip so we want to add how many to this uh eight of them okay so one one one two one three one four one five one six one seven and one eight there we go okay sector size the default 4k that's normal so that is the sector size of the underlying disks that's your advanced format disk 4k record size so this is cool and i wanted to mention this in the whiteboard part but quickly i'll just talk about it zfs chose the 128k record size as the default record size um for the reason of it's it's not particularly amazing for any workload but it's also not terrible for any workload either the jack of all it's the nice jack of all trades so we'll leave our pool with that but we're definitely going to want to play with that on our data sets so when we cut data sets like i can choose different records that's why zfs is good for mixed workloads because you don't have to have one storage pool that works one way exactly you can change that as as needed for different datasets gotcha dedupe uh no let's not do that okay so reservation yep let's just pull that down for now yeah we just want to use all the space we have yep lz4 you get compression for free pretty much for free yep actually with zfs you end up getting a little more better performance yeah absolutely i know it's crazy um i don't want to automatically expand right now automatic trim i don't have any ssds so that's fine and we'll leave the rest of it for now forcefully create pool we don't need to do that that can be useful if you're doing things you necessarily maybe shouldn't be doing yeah different z different drive sizes or something no one does that here we always follow the rules yes always all right so let's spin for a minute as it does okay and there we go we got a pool anyone who's used to uh other raid that's not zfs yep uh either that software raid with md raid or hardware raid yeah when you make your pool you make your pool and you go home and come back two days later yeah zfs because the copy on right does not do that no sync time so it it instantly is ready to go it's actually funny because a lot of customers that have used raids in the past and come to us they're they're ready for that big two day or one day sync time and we're like no no you could start using this is gonna take you three days exactly sorry all right so we're good we got our first storage pool so what are we working with here we have 58.22 terabytes available to us there we go cool didn't want to shrink that so what did we do we wanted to say we were going to do about 10 terabytes for the db about 5 to 10 and then everything else so and then yeah it seems about right so yeah 20 something terabytes for a file sheet we're going to double this up here in a second that'll work yep um so it's mounted no replication not doing anything so what's the next steps here next step let's add our psychovita then oh thank you i forgot all about that that's why we're missing that's where doesn't have the space i was just questioning that a little bit you know the show must go on uh okay so what we'll do over here is we're on files we'll jump over to the status tab yep that's right so here's the pool we made in storage and our first v dev which is just raid z2 and then the the first zero is the first veto exactly so our next one will be raised yep all right um next we're gonna go not configure sorry we're gonna get to this top level yeah we're gonna add a virtual device same thing again careful it defaults to mirror we don't wanna do that we wanna do a z2 you got it so i'm gonna pick one 9 110 111 112 113 14 15. and then we're going to jump all the way over to 217. you got it yeah same idea you can forcefully add it if you want if you're adding things of the wrong size again you probably shouldn't be checking that unless you know testing or playing around yep okay now we got our storage okay cool so we'll go back to file systems we'll take a look at that so now we got our 116 terabytes yeah that seems a little more what i was expecting it's much better okay so uh we're not done on vw's yet though no we're definitely all we did was added our we added our data v-days i guess you could call them now we're gonna add our support v devs gotcha gotcha so same thing again we're going to the top levels idea yep we're going to add a virtual device this time we had our log so we're doing slog this time yep it's called log in here slog zil you'll hear you'll hear different things yeah slogan log is usually what we're talking about we want to click mirror because it wants to mirror the discs still device alias so we got four ssds so we might as well throw four of them on there okay sounds good yup i won't make a comment about forcing it this time but i guess i just did so sorry uh all right that was quick that was quick so now we've got our logs our mirror great what's next we need another video so our last vdab we're gonna go and get our special v dev now on our nvme so again we're gonna add our virtual device yep we're going to jump right to special yep typically in real life you should mirror this we only have one installed so we're not going to mirror it yeah real world we would be mirroring it oh no and it couldn't let's check logs why i can't do that i'll do it again while this is up yep it's not a video if something didn't fail right that's right just like the demos that i do every week you practice and practice and nothing will ever go exactly to plan it may actually require you to force if you're only adding one but i'm not sure on that yes it does actually now i remembering it because when i do it in the command line with a single it will require you to force it because it doesn't want you to be doing just a single uh but we don't want a mirror either we don't have enough so i should have made a comment you should have made the force well okay now it's not going all right we can add it from the command line but i guess yeah that's a weird one that is weird um i could just keep beating my head against the wall here or we could just do it from the command line yeah let's try everything let's do it from the command line all right so we're going to pull cool storage yeah this is just special yep dev nvme zero ends it's very possible there's something on this yeah dash f you're gonna yep so you need the dash f mismatch replication pool uses raid z and new v dev as a z yeah which is fine there okay so for some reason the uh the ui doesn't like it unless you have a mirror which i mean there's not going to be any real yeah exactly good there's not going to be any real workload where you really want to do that anyway so now we've got to edit that no we won't all right so there you go so it looks like the zfs tab really wants you to do it right which is what we'd expect right um but we we kind of fudge it just to get through the video because we only have one nvme here but we can now see we have our log cool we have our special let's make sure that shows up in zfs though okay cool so let's go to status here trade z1 ranges yep there we go there's our log there's our special everyone's happy so our z pool is ready now what do we got we've got to create our data sets for one for our uh database okay and then one for our file share let's do the database first then so we'll just simply call it db sounds great easy peasy um access time that's usually one that you can turn off to save some performance uh this is just one less io you have exactly it's just and it's we already have modified time we have all these other ones that are enabled so um yeah we'll let it do everything else yep everything the only thing we want to change is the record size to match our database so we don't want to inherit that that's really cool too we'll get to that the special blocks so what do we said 64k yes we did yep so and then the idea again here is your storage point your data sets can be variable right exactly yep um all right we'll hit create then there we go boom file system created we got our db and that's where our db is going to write to uh so now let's create our file server or file share okay so really it is going to depend on what you're using your file server for but most business file servers it's a lot of documents there's a lot of smaller files so you could go with a smaller than a 128k record size but at the same time if you also have big videos and big files you want those to be good as well so i think the 128k record side is a very sane one to pick for a mixed workload file server that's great um access time yeah we could pull that off as well okay and again we want to keep our compression and all that because it's nice free save space and free performance cool so i'm going to make it then you want to put a quote on anything yeah you know what let's go back and put it yeah let's do that let's set a quote on our database oh edit permissions wrong button i want to go configure file system yep or a quota yes there we go all right so the way it stands our metadata is going to our special v dev our sink rights are going to our slog the only thing left that we talked about that we didn't set up and it's something that you have to do from the command line is to set those small blocks oh yeah did i forget one oh nvms of course yes we'll just we'll just call it vm yep simple uh what do you want to do bracket size here so if we're using z-volts it'll it'll have its own record size for these ones yeah the 128k is fine for now but if we've got some small i o we may want to have separate data sets for separate vms so like if we've got some small i o we'll go all the way down to like 16 or 32k for our record size so let's do 32 yep perfect um great there we go okay now i cut you off in the fun part very small blocks a small block size exactly so where we can also use that special v dev for not only metadata but also any small block writes so you kind of touched on the fact that doing that you got to make sure that you have enough storage in your special v dev to handle all those small blocks it is great there is a fail safe in place that the special v dev will say hey i'm 85 full i'm putting these small blocks back onto your spinners um so let's go in and let's set the small block size for our file share so are we going to do that for every data set no i think for right off the hop we'll just do it for the for the file share okay but you can set that as the true true true true because the database and all that like that's using the 64k record sizes already yeah um so we're going to configure file system actually i'm pretty sure this one you have to oh you oh yeah we already said that we found it here yeah that's really cool i didn't even know this was in there so now we can love the command yeah all right so let's say we want yeah 32 and smaller in our file share we really want that to go to our special lead app okay um and we're putting no quota on that that's no that's right there we go um just to be complete we said we were gonna put a quote on this too yes yep i'll just say 10 terabytes for that as well that makes sense all right so we have literally just set up an incredible mixed-use z-pool that can do a lot of things really really well just by making use of what zfs has to offer in its architecture or if it's in its programming so from here we would hook this storage into the db exactly we probably like maybe you want to run the db right anyway cool yep file sharing would go over the phone will do async nfs will do sync but you can enable async so you don't have to use that zil most people where we're using this kind of as like a general storage share we'll use async yeah don't use smb yeah smb exactly uh vm storage we'll uh we can hook that into products box we hook that into proxmox via iscsi you got it okay um one other thing i just want to cover real too is like okay you've got all this data and it's running and um you're worried well their raid's safe everything's pretty good but you know disasters happen yes they do uh you just go buy another one of these stick it somewhere else and you see this replication tab here cfs send and receive baby can go and configure replication tasks that is really so what you would do is just point it to the ip and data set of that other box yeah and you'll get a full copy of everything you see here on the other one so this isn't like a file based copy right like this is a real it's blocked yeah we use zfs send and receive it streams that across and it's intelligent as in it it doesn't send the same stuff over and over exactly it doesn't change incrementally incremental yeah and it's not like an rsync where you have to crawl the file system every time it just essentially knows what has changed wonderful very cool so this box is ready to consume storage yeah you should call it here yeah i think we can call it here obviously if anyone else wants to see anything more in depth of hooking some of these things into the end use case let us know we can absolutely go a little further for you but i think everything we've done here was it was a really good job and we can call it cool all right so this is a good covering of uh the flexibility of zfs our q3816 and um well let us know what you want to see next thanks for watching guys bye

Channel: 45Drives

Views: 27,715

Keywords: 45drives, storinator, stornado, storage server, server storage, nas storage, storage nas, network attached storage, zfs storage, zfs, single server, storage clustering, petabyte storage

Id: T0frbOzBlHI

Length: 39min 30sec (2370 seconds)

Published: Tue May 31 2022