TrueNas Scale: State of the Beta Q4 2021

today today we embark on a journey that's probably going to take us several videos i'm going to start by talking about trueness scale truenas you can run a server in your small or medium-sized business or even at home if you're an enthusiast and it's going to do a lot of really awesome things for you [Music] pretty much anybody that is a computing enthusiast eventually ends up with some kind of a something at home and check it out this is what we're rocking it's enterprise grade storage when you talk server usually talk storage our storage is based around lsi old netapp uh and lsi uh disk shelves originally bundled was part of a netapp solution which is like ibm's fancy patch storage thing this cost hundreds of thousands of dollars you know more than like five years ago but for home usage other than the electricity usage which stinks a little bit something like this is really incredible and or small business or you know random youtuber usage this kind of thing is pretty good cast off enterprise equipment you can save it from the dumpster you can save it from the landfill i definitely saved my fair share of hardware from the landfill and you can run something on it like trueness and have access to a lot more functionality than you used to if i wanted to keep running you know the lsi stuff the netapp stuff from ibm there'd be licenses and questions and it wouldn't run quite as well i can put faster hardware in front of the older disk shelves and the whole thing actually speeds up and so our server platform here is based around our xeon 8124 ms we did a separate video on this a while back you can pick up some of these old skylake xeons on ebay for practically nothing we've also got another system here that has a bunch of uh it's a 2u server instead of the 1u server it has a bunch of nvme we're going to come back to that one but i'm going to be running trueness scale on both of them but i also wanted to point out you know if you're in the home server use case you could get a nook or you can get something like the desk mini x300 this has an eight core 4750 g in it this is crazy a crazy amount of horsepower it has 32 gigabytes of memory and it has two four terabyte two and a half inch hard drives yeah check it out this is the samsung cuvo four terabyte two of these in here in raid one it's a mirror so if either one of these dies this thing will keep on trucking we also got nvme storage optional wireless uh i wish the integrated nick was just a little bit better like you know two and a half gig something like that but hey i've got a couple of uh usb ports and all that this for my home server and if you need more than four terabytes of storage i can add an external usb 3 like a 20 terabyte mechanical hard drive for media storage but this thing is tiny look at this this is the even like a studio apartment you could use this as a server and run a ton of containers but why would you ever want to do that well let's take a look at the uh applications section of trueness scale all right so we just click on apps on the left side and we go to available applications and out of the box it doesn't look like there's a ton here but there's you can actually add more this is actually really cool ipfs mineo tons of chia mining wait what we can plot chia on this next cloud caller bora and plex plex media server these are all official apps now you can definitely install unofficial apps and you can go to manage docker images so if you want to run something like home assistant for home automation stuff as a docker image or even as a full-fledged virtual machine you can do that you want to use something like zfs with snapshots for ransomware protection and backup your home computer or home computers onto you know your documents and stuff like that onto something like this you can do that this thing also has a built-in vpn so if you want to run openvpn and have openvpn running on your phone which has really good support for both android and ios you can have a vpn on your phone all the way back to your trunas appliance and then all of your stuff is encrypted but you can access your your files locally anything that you get sorted next cloud and the same is true with like your laptop or your remote computers or anything else that you might be running it's basically access to your stuff anywhere but without necessarily relying on the cloud if your home internet connection is good enough your upload speed specifically is good enough you won't really be able to tell much difference between running it off of a system like this and running it in a data center so you can avoid a subscription or something like that some isps don't even give you a real public internet address so you can use the node or something like that to host the internet connection and how you get there is your your your true nas machine will connect to the node and you'll connect to the node through the vpn and then you can talk to your tree last machine even though your isp might firewall your internet connection or not give you a public ip address all of those things are true so truenas is from a company called ix systems and trunas this user interface and things like that is really uh the result of a bunch of years of work you may have heard of freenas in the past well freeness and uh truenas true as well but freenas was a really really popular product from ix systems because you could if you're a reasonably advanced user you could download it and start using all that kind of stuff so it's not super noob friendly but once you figure it out and do the stuff it was really awesome well true nas there's trueness core and truenet scale they have a very very similar almost identical user interface but the underpinnings are or what's different or what separate them so trueness scale is linux based and trueness core is freebsd based and there's lots of videos on on both of them already you can definitely check them out but because truenas scale is linux based i wanted to look at it i wanted to push it to the breaking point i wanted to do things that they definitely don't intend for you to do which is awesome we're gonna let's get a foreshadowing for some videos that are coming so this system as configured uh has two 10 gigabit interfaces but i'm going to be upgrading that to melanox connections that are 100 gigabit and for the switch i'm going to be using a dell s5212 f-o-n and the reason is this is a relatively affordable 100 gigabit switch it has 300 gig ports and 12 25 gig ports and the street price of around 1200 give or take um the switch actually runs debian as a control operating system actually that's a whole other set of conversations that i want to get into because on the networking side we have the open network initiative stuff and so out of the box the dell switch doesn't come with anything installed in the operating system it actually has uefi and you can configure stuff and that's just there's a forum thread on that i can get my hands on some cheap oney equipment and uh talk about open switching stuff and things like that but this is a dell product for the switch side of it and that's what we're going to be using for our connection for this kind of thing but i wanted to give you my sort of introduction to true nas scale as something that should definitely be on your radar and something that will scale from something like this to something like our massive storage server that currently has 48 mechanical hard drives and 16 2 terabyte nvme drives attached to it yeah that's pretty much the limit of what you're going to be able to do on skylake we're going to do some other stuff on our uh on our epic platform our amd epic platform now both of these servers that we're using the enterprise great servers here they are from gigabyte and we have done videos on both of these gigabyte enterprise servers in the past so if that's something you're interested in you should definitely check out our past videos on that kind of stuff but both of these platforms are dual platform sockets they've got tons of memory capacity uh right now i've got 192 gigabytes in the xeon system i'll probably move most of that over to the amd system when we switch to testing on the amd platform i actually don't have enough memory to keep all of the servers that i have running because the memory storage memory is expensive and experiments gotta always run experiments there's a lot of experiments that are coming i've also got an 8380z on that we're going to test the dual socket 8380z on i'm working out some some weird performance anomalies with nvme on that platform i think we got them fixed turns out it's a microcode platform update kind of thing from super micro so that'll be another future video down the road but true last scale can it scale to 100 gigabits i tell you right now yes it absolutely can scale to 100 gigabits uh the thing that you run into with scaling 200 gigabits and beyond is more on the zfs side of things because zfs loves to make copies of things in memory even when you tell it not to and so you know you need two to three times the amount of main memory bandwidth as your storage bandwidth and if you've got 16 you know nvme drives that can do eight gigabytes per second you really quickly start to be within striking distance of the total memory bandwidth of the entire platform and this is a problem that cascades down to a lot of different things to the point that now network cards are doing more and more offloading because it's just too much work to move the data like the computer the processor is fast enough the bus is fast enough but it's just too much work with everything else to shuttle the data from the network card over the bus to the cpu let the cpu deal with it maybe it's going to juggle some stuff in memory and send it on back out again i'm not really ready to talk about that part of it yet but to give you a little preview i want to get rdma remote dma working in a server capacity on truenast scale and uh one of the first things that i've run into with trueness scale is that it really wants to be an appliance even more so than than free nas in the past freeness in the past really wanted to be an appliance too but you could sort of uh take the system by the reins and use freebsd package management and then be able to customize your system and the portable nature of zfs means that if you just completely screwed up your free nas system your zfs pool where all of your important data was was completely protected and safe and that carries over into trunas trueness has uh trueness scale has some incredible stuff built into the gui to help you set up redundancy so i've initially just set up a a four drive raid z1 for experiment out of two terabyte keyoxia nvme drives and it's fast it's responsive it's very stable it's a very polished user interface i like it a lot i like what what trueness scale has done this is easier and more accessible than it's ever been in the past when it was you know a free nas type product and truna scale being a linux underbelly maybe gives you uh some better driver support at least more wide driver support than you would get with freebsd and maybe some different performance characteristics you know over at lawrence systems tom lawrence has been doing testing of iscsi so like if you run a vmware server you might have virtual machine storage available on your your esxi hosts as iscsi targets and depending on if you're going to do that you know the ics performance actually is a little bit better right now on freebsd than it is on linux iops and other performance so if i were going to do that with my storage then freebsd trueness core instead of trueness scale or one of the paid eye assistance products might be a better choice but my ultimate ambition is to be able to use some of the features in the kvm virtual machine stuff in linux to do some advanced stuff with virtualization and trunas scale doesn't provide any facility for that yet and actually go so far as to discourage anybody from messing around with the linux underbelly because they store the configuration for trueness scale and its own sort of configuration management thing and so if you go off the beaten path and start customizing stuff then that'll break things and then you'll go to their forum for support and do all that stuff so if you do anything that i'm even suggesting or hinting don't go to ie systems for help they will send the hit squad and it will be bad it will be very bad very very bad but you could come to the level one forums and let's talk about it figure it out maybe we can actually get it going so one of the first things that i ran into is i wanted to customize virtual machine configuration you can do that from the command line on linux using verse vrsa virtual machine shell and you export an xml file and you can manipulate the xml file and you can do a whole bunch of other stuff zero support for that in trueness scale and that's by design they don't want you to mess with that because it enters unsupportable territory really quickly it's easy to break your system in unexpected ways if you're messing with that so what did i do i installed cockpit i just wanted to see what would happen cockpit is a web gui management thing actually from red hat but you can install it on the underlying linux operating system of true nas scale and it mostly is functional and so you can have a separate web gui on the same machine for managing virtual machines and it can see the virtual machine that trueness creates if it manipulates it truenass will reset that manipulation back to whatever it thinks the configuration should be so they fight with one another definitely don't recommend this but i just thought it was funny that there was at least a capability of uh of doing that kind of thing uh even though even though that's very off-script and very unsupportable and very i don't recommend it and very if you're gonna do this at home just you know stick to the docker container stick to the stuff that you can do in the gui because that's the choice that they've made with their platform and so but me i like to kill her outside the lines so here's what i want to work on here's the here's what i've got got for you to look forward to 100 gigabit ethernet can we saturate 100 gigabit ethernet and also minimize the cpu utilization you know of course of course with two 18 core sockets or 264 core sockets depending on which platform we're using or even two 40 core sockets on our xeon 8380 that is not going to be a problem to saturate 100 gigabit ethernet with our melanox connect x5 network cards i need more of those by the way if you does your company have connectx four and five cards just laying around you want to get rid of uh call me let's uh let's chat let's do some stuff so yes you can do it if you throw a bunch of cpu at it but you don't really need to throw a bunch of cpu at it because those melodic cards have excellent offload capabilities one of the one of the really frustrating things is in windows storage spaces and their storage stuff sucks it's bad even in server 2022 i don't think anybody's really testing that outside of the azure use case on azure it's glorious but nobody's building the hardware that's exactly like azure so if your performance is going to be relatively garbage now in terms of like the the bubble that is windows server 2022 storage performance is amazing so much better than server 2019 but when you step just a little tiny bit outside of that the current state of the art with linux is the ability to run 10 million iops on two threads in one core that is where we are with the bleedingest edge version of the linux kernel that version of the linux kernel is on a trueness scale yet we're about a year or two away from that version of the kernel being widespread again if you're like me and you can you know do some stuff and customize it you can have that today uh but yeah 10 million high ops with two threads is completely completely insane and you can do that with a couple of uh well a set of intel p5800 obtain drives um because the very low latency and very high throughput i've got two p 5800 x's and they are screaming screaming monsters on any platform that you use it on team red or team blue looks like team blue is going to do some crazy stuff with sapphire rapids but that level of performance with that relatively low cpu overhead is not currently possible in trueness scale so my question is is can i get there and can i get there with rdma remote dma because remember i mentioned the whole like offloading thing remote dma is something that windows server actually does well and has been really well supported and interestingly enough on the smb client our dma is also supported so if you have a windows server and a linux client you can offload a lot of the processing and and things like with multi-channel and the network card does most of the heavy lifting and so it's very low overhead to get many many multiples of gigabytes per second from windows server to linux client but no one has ever implemented rdma on the server side for samba at least not that i can find a knot that's paid for i may be wrong about that so if you know something about that you want to share you want to do some homework with me by all means let's work together let's do something but i think i'm going to try some some out of the box stuff and try to uh frankenstein some of the stuff from windows into this platform and see if i can't get rdma to operate anyway and so i'm setting up this platform i'm setting up trueness scale and i'm setting up for some other videos so i thought this first video is an introduction to to say hey 100 gigabit platform dual xeon 8380 dual epic 64 core 7763 dual skylake xeons and all of the memory and all of the fixins and hundred gigabit malinox connections what do you want to see with this let's do something really amazing and exciting and interesting and something like trueness scale is the minimum that you need to take advantage of that i mean yes yes you can run something pedestrian on this but truenass scale has gotten so easy that you should at least check it out because even though it has a bunch more enterprise features than other storage options that you have for you know deploying something on this to to have your home server i like the way the true nas scale and trueness core both of them do it and you can run either one of them on this you can run either one of them on the other stuff but for the stuff that i plan to do with virtual virtualization and virtual machines i can only do what i plan to do on trueness scale it's not going to work on truenet's core i know that much but i've still got some homework to do still got some some stuff that i want to do and uh yeah i just wanted to just wanted to share just wanted to see how this how this uh this video goes and see what questions you have if you have any questions about building something like this for your home you know it's like hey i thought you had the x300 with dual four terabyte hard drives how does that work well come to the forum let's chat or if you already have something like that you know let's let's take a look at it a lot of you are rocking you know in xeon nux and uh or other intel nuts including the xeon nook and some other stuff at home and that's always a lot of fun it's a little bigger than this i also reviewed that but yeah trueness as your home and small medium business server with all the apps and the virtualization stuff is extremely powerful but overall take away my impression of this kind of software is that it's come a long way in terms of features and flexibility but it's also gotten more user friendly than other ever so if you've tried freenas in the past and had a bad experience you really should give trueness scale or tunas core a try i slightly recommend this release candidate for trueness scale i'm gonna have to do you know over core but i'm gonna have to do some more testing and some more other stuff um and that's mainly because i want to use this as a system to build on to which ic system kind of seems like they frown on just a little bit uh which i get i get you know it's like they're overworked as it is come to the level one forms if you need help with the the color outside the lines don't don't burden the people that work on this for a day job because it'll suck up all their time and we don't we don't need that that's not it's not good so i'm wondering this is level one this has been a quick vlog-ish ramble about what i'm doing with trueness scale on the ridiculous server hardware so yeah stay tuned to subscribe let's talk about stuff let's talk about your home server use case because being able to run next cloud at home and vpn into that it's kind of transformative it's really good stuff maybe i should do a separate video on that a model is level one i'm signing out you can find me in the level one forums [Music] you

Channel: Level1Techs

Views: 34,218

Keywords: technology, science, design, ux, computers, hardware, software, programming, level1, l1, level one

Id: tBZknK6Yktg

Length: 20min 1sec (1201 seconds)

Published: Thu Dec 02 2021