Homelab and Server Rack Tour - 2024 Edition

it's been a little while since I've given a full rack tour top to bottom and I figured with the start of 2024 there's no time like the present to see what changes I've made over the last year and what lies ahead for the future of my home lab let's dive in welcome back to craft Computing everyone as always I'm Jeff and this is my home lab this is the server rack that I keep out here in my G and you've all seen it in various iterations since I started the Channel all the way back in 2017 but what changes hasn't gone through to get to this point and what are my plans for in the very near future that's what we're going to talk about today one thing I think I need to clarify just a little bit is what is a home lab and how do I use mine for me there's really two definitions of Home Labs or two main use cases for home Labs one is running production servers to make your life easier with Services you want at home and that's pretty much where I fall into my use case This Server rack is where I run my business it's what stores my video files and current projects important documents it makes sure that I have things backed up to the cloud and it makes my life easier by running services like Plex and py hole and vpns and all that kind of thing that's what runs in my rack and allows me to do my day-to-day tasks there's another classification of H lab and that's really where I started my journey with home lab and that is building a server or set of servers that you can use for professional development that is learning on hardware and tinkering with things and giving yourself an environment that you can Tinker with and break and fix and learn from while mine still does serve both those purposes its primary function is to basically host my business and all of my operations and so that's where my primary focus of This Server rack has always been one question I get all the time about my Home Server rack is how how much does it actually cost to run this thing specifically around electricity costs because power is not cheap for most of the world in my particular situation power is incredibly affordable I am very benefited to live in an area where power is about 7.4 cents per kilowatt hour and 98% renewable that's not something a lot of people can say one of the focuses this last year was actually bring my power consumption way down my server rack at one point early this year was idling at about 900 watts that's 900 watts of power not doing anything just powering the servers on and running its basic Services as of right now the entire server rack top to bottom idles at about 350 Watts which is absolutely insane and it's why I can actually stand next to it and have a conversation now it's not incredibly loud and it doesn't require nearly as many fans or nearly as much air flow to keep 350 wats of power cooled right now at 3 50 Watts at 7.4 cents per kilowatt hour my rack runs about $225 to keep powered on 247365 so $225 for the entire year again it's incredibly affordable and I'm very blessed and benefited to have such affordable power here but speaking of keeping the server rack cool there's one thing that I haven't mentioned that I also run inside this rack during the summer months and that is my air conditioner and I think that's where we'll start this rack tour at down here at the bottom is my triplite 7 krm it is a 7,000 B air conditioner designed specifically for rack mount installations I did a full video on the install of this as well as its efficiency a couple of years ago and I will have links to everything in this rack down in the video description if you're interested in learning a little bit more in short this is a rack mount air conditioner again 7,000 BTUs of heat displacement which is good for about 1,000 watts of heat generated by the server so even when I was generating 900 watts of heat inside the server rack this was able to keep it cool during those summer months now again I only operate this when the temperature outside gets above about 85° fhe or roughly 30° C and I have environmental monitors to tell me when those temperatures start exceeding those limits this when it's running at its peak efficiency can actually create about a 17° fah Delta or about 92° C between the ambient air in my garage and the air inside of the server rack to keep all that cold air inside of the server rack there's actually a vinyl sheet very similar to what your grandma would put over the carpet in her house back in the80s uh that keeps all of that cold air right in front of the servers creating essentially a plenum space here in the front of the rack now because this is essentially a heat exchanger that's built into my rack I've got to do something with all the hot air that it's taking out of the environment so I actually have this plumbed and directing all the hot air directly out of the back of the garage it doesn't recirculate within the environment that would be kind of pointless and uh it actually does a phenomenal job I've been running this about two at least the last two summers and it has done fantastically well in that time but because it's a heat exchanger it itself does get quite warm so I've got a 2u spacer in here before any of my other equipment starts this in total takes up what is this eight racku so 10 racku of space just for my air conditioner up next you might notice a hole in one of my components here and that is the battery out of my UPS unit uh this is an APC uh smart 1500 VA to you rackm mat APS and it is a smart UPS unit with actual network connection and environmental monitoring so not only does this provide ancillary or backup power for my server rack to keep everything on when there's a power outage it also uh is able to report all of the environmental data from around my garage and inside my server rack both internal and external temperatures it's a fairly old unit and still uses lead acid batteries uh but replacing the batteries is still much more cost effective than buying a brand new unit with lithium ion chemistry in it it's not quite as efficient the servers don't run as long but at 330 Watts I can actually run my servers for upwards of 2 hours on this battery backup unit so not terrible at all and it still totally fulfills all the needs that I have for it so I are the batteries outside of my UPS right now well the most recent power outage that I had uh I think did them in finally uh these batteries have been in my rack for gosh 5 years now uh I replaced the batteries when I first built my server rack all the way back in my last house I think that was in 2018 is how long this has been in service so going on 6 years now it was definitely time to replace them now a new battery unit like this uh was about $240 so again very cost effective to keep this one going rather than switching to new chemistry which would only give me more runtime which is something I really don't need here moving up the chain we come to the first server in the rack and this one is truly what I use for my home lab uh this is a server that I can Tinker with make break destroy and uh basically do it over again and keep on going this was my cloud gaming server and it will be again but with some very significant upgrades stay tuned to the channel so you don't miss that a quick rundown of the internals that are in here right now this is running an epic Rome 7742 64 core CPU with 256 GB of ddr4 registered ECC memory running at 3200 MTS we've also got a dark Power Pro from be quiet 1200 wat power supply I've had a number of different video cards in this server for doing virtualized gaming or cloud gaming as I like to call it I've had anything from three 1070 TI to three Titan XPS all the way up to a pair of RTX a5000 now I didn't quite get to publish that video I was waiting for some other things to work themselves out I'm hoping to actually swap the a5000 back into here and do that video before I go and upgrade all the components to we'll just say the next generation of cloud gaming the next time you see this server it will probably have those a5000 into it and then we're going to replace every last thing inside next up is a pair of servers that I built earlier this year and they might be a little controversial with some people but these two servers are what allowed me to get that power draw in my server rack down to a somewhat acceptable level each of these servers is rocking an ear Ying tiger Lake 11th gen motherboard with a 45 wat Intel I9 11 900h engineering sample mobile CPU what that essentially means is each server has an 8 core 16 thread chip with 64 GB of ddr4 running at a TDP of just 45 Watts there are two schools of thoughts when it comes to virtualization one is get as many cores and threads as you possibly can and build yourself something like an ivy Bridge with 16 cores and 32 threads and even double that up to 64 cores and 32 threads per board uh or you get yourself much faster single-threaded ships that can do twice the number of extractions per clock as the older counterparts and that's what I went with here while these may only have eight cores and 16 threads they are literally 2: one per core performance of an ivybridge CPU each of these servers also has a pair of silicon power a60 1 tbte nvme drives in a ZFS mirror for redundancy on the system booting the OS as well as storing all of the VM hard drives I do have a couple of Nvidia cards in these as well the bottom one has a Tesla P4 while the upper one has a Tesla T4 although I've never really done any configuration with them I was planning on doing some virtualized gaming so my daughters would have some systems to be able to play Minecraft with but we ended up not needing those as we have a couple of gaming desktops in the living room now we may end up still enabling those cards for full virtualized gaming in the future but there just hasn't been the need to set it up at this point I also went ahead and added some SSD cages to each of these and fully loaded them out from the SSD server that I took apart earlier this here as well so each server also has four 1.92 tbte Patriot ssds installed in there again I was planning on building these out into either a fully or highly available set of SSD storage but I opted for some nvme storage in my Nas which we'll get to momentarily overall these have been fantastic for Home Server use again only 45 watt TDP on the CPUs themselves you can tune them up to 60 or even 65 Watts if if you want if you want a little bit extra performance out of them but the systems as a whole draw less than 100 Watts from the wall at Full Tilt that is absolutely insane considering how many VMS you can actually load up on these systems and we're almost back to a comfortable height so you'll excuse me if my head is out of frame for this shot but the next item in my server rack is my KVM console or essentially what is a keyboard trackpad and monitor that only takes up a single U inside my server rack while I don't need this all the time it is fantastic to have bare metal access to all of the servers in my rack at a moment's notice not all of my servers run ipmi and so sometimes you just need a physical keyboard mouse and monitor attached to a system and this allows me to do that unfortunately the actual display driver for this LCD doesn't like to always negotiate the resolutions properly and so sometimes I end up with some garbled displays uh offset images so you can see right here that my console is actually shifted all the way to the right by a fair margin and in fact it's probably cut off on the other side as well while it does work I'm not thrilled with this as a unit and I will probably end up trying to Source a newer one maybe even with a 1080p display sometime in the near future but overall I do like the form factor it folds up nice and small and slides into a one U rack bace now one thing that might be a little bit difficult to see is this guy right here this is actually a temperature and humidity gauge that I added into a blank panel so I can check uh system specs at a glance without having to dial into my UPS unit to get environmental data right now we are rocking at 56° fahit and 62% humidity again it is winter time here in Oregon all right and now I can finally stand up fully and talk you through the rest of this up next is my nass this is the Crafton nator uh very generously provided by 45 drives for my Home Server rack and again this is essentially what operates my business this is where all of my video storage goes this is where all of my current projects all of my important documents that I need to keep records of goes this is the primary server inside my server rack now those who watch the channel know it's been through a couple iterations already and a couple of those aren't even on camera uh I famously put one terabyte of memory into my craft anator a couple of months after I got it however that system has been swapped out and it is currently running an AMD Naples platform with a 7601 32 core CPU and a super micro h11 SSL motherboard a very common combo that is actually very readily available on eBay right now for somewhere around $300 to $400 if you find the right seller as far as drives go this thing is fully kitted out I have seven Seagate exos 18 terab Enterprise drives as well as eight hgst h8 helium filed Enterprise priz discs in the front of this for a total of 15 on an add-in card I also have four Western Digital SN 770 2 TB gen 4x4 nvme drives and that is where all of my current projects go so I have incredibly fast network access to some high-speed storage on this Nas device I've also added in a 100 Gig Network Port which we'll get to again momentarily now all three of those disc sets are running in a raid Z2 for a two disk parody and for those playing the home game that gives my seate exos drives about an 85 tbyte capacity after raid uh configuration my 8 TB hgst drives about 42 terab of space and my ssds have about 5 and a/4 terabyt of space so plenty of usable space and in fact each of those pools is sitting right around 50% utilization which is exactly where I like to see them as far as what operating system I'm running I have gone through a couple iterations over the last year but I think I finally settled on running true Nas scale on bare metal this is only running as a Nas it's not running any other virtualization systems it's not running any applications like Plex or MB or VPN all it's doing is hosting my files and what better to do that than an ass that runs on bare metal up next is the newest member on the Block and that is the hl5 again from 45 drives this is their first crack at a home lab server and I'm actually hoping this will fulfill that purpose in my server r giving me a true playground to play with whether it comes to virtualization or uh file servers and give me an environment that I can Tinker with and demonstrate different use cases to you at home because the majority of my server rack has gone towards production or business use or required services I don't really have any systems that I can Tinker with right now and I'm hoping this one will fit that bill now it did come preloaded with a super micro X11 motherboard and a Zeon 3204 core CPU now that's going to be just fine for a lot of file server uses in fact later this week you'll see a use case in which I deployed one of these to another YouTuber again make sure you're subscribed you're not going to want to miss that collaboration but I want a system that I can Tinker with that I can run 30 different virtual machines on or I can play with different file system arrangements or high availability or something like that that's going to be what this server does inside my server rack moving forward next up let's get into networking and it's really funny looking at the progression of my home network because what is my Edge switch right now started as my core switch and every switch has gone kind of through that same iteration Through The Years this right here is the microtic CRS 328 24p it is a 24 Port gbit poe+ switch with four 10 GB SFP plus ports on board can see I only have around eight of the gigabit ports plugged in right now as well as the 10 gig Port right here for Uplink to the rest of my network while it's not being fully utilized at this moment it is still a crucial component as it is the only power over ethernet that I have in my server rack that allows me to run my access points a couple of security cameras that I've got around the house as well as a couple of ipmi devices down here in my server rack right above that is a 24 for patch panel keeping everything nice and clean and uniform inside the rack at least when it comes to my RJ45 connections over on this side I do have some LC fiber pass through ports which unfortunately you do have to get essentially 1 M cables at their shortest so it leaves some of this stuff hanging out and about but overall it still keeps everything much cleaner than it would be if I was just routing those cables through uh you know a oneu grommet or something I wish it was a little bit cleaner than it is but overall it's hard to get better than this now I mentioned my KVM console down lower in the rack but that was only partially true the keyboard Monitor and mouse only do dis display and connectivity they actually don't handle the KVM switching and that's why I have this unit right here this is an 8 Port VGA and USB KVM and it has essentially a pigtail that Jets off to every server in the rack allowing the keyboard video and mouse to pass through to that console next up we get to the switch that was my core switch last time I did a server rack walkth through however now it is serving more as a top of rack switch with only gosh am I down to 2 10 gig ports in my server rack I think I am this is the microtic CRS 3171 16s plus and it is a 16 Port 10 GB switch as you can see while it does have a couple of cables still plugged in here only two of those are active so it is essentially acting as a pass through switch between my 100 Gig Network and my 1 gig Network that's its only purpose right now because I've eliminated most of the 10 gig Network out of my rack entirely those two eering virtual ation boxes those are still running on 1 gig Network because not a lot of services that I have that are just serving applications need more than 1 GB connectivity my Plex server certainly doesn't need 10 gig my pie hole certainly doesn't need 10 gig why go through the trouble of wiring up 10 Gig if you're not going to use it overall this is a fantastic switch in the CRS 317 even though it's barely being utilized right now at one point I did have 10 of the 10 gig ports filled inside this switch and it really did run Dynamite for the 2 or 3 years that I ran it like that it probably will be doing a little bit more here in the near future as I roll out a couple new servers for the 2024 year but right now yeah essentially it's working as a media converter and now we come to the switch that answers the question why is my 10 gig switch being underutilized and that is the CRS 504 4X Qin again from mcrock this switch I reviewed uh earlier this year sometime this is a four port qsfp28 switch that is four 100 Gig ports inside this tiny little oneu switch and finally we come to the most recent switch in my server rack and what is now serving as my core switch and it is the microtic CRS 504 4xq it is a four port qsfp28 that is for 100 gig ports in this tiny little oneu switch at one point I did have two of my servers with full 100 Gig links I've actually decommissioned my 2u uh dl80 gen 8 HP server it's no longer serving a purpose and so uh Port two is just kind of sitting here idle however the craft anator Nas is still running at 100 Gig links Port three is running single mode fiber all the way back to my office to another CRS 504 switch which has full 100 Gig to both of my desktop PCS which means either of my PCS has full 100 Gig access to the craf anator nas so was it worth setting that Network up I can fully say after more than 6 months of testing no uh running 100 Gig in my network is completely stupid I don't recommend doing it at most right now no matter what setup you have in a server rack no matter what desktops you're running 25 gig is probably the theoretical limit of file transfers as long as you're running Windows SMB or a standard consumer file share even with a ZFS array with four Gen 4 nvme drives inside craft anator and 100 Gig direct point-to-point connections to both of my desktops the most I've been able to get out of a SMB file share is about a 20 sometimes trending 25 gbit connection and that's under ideal circumstances while the network may have the bandwidth to carry pretty much any amount of traffic that I want to sling between those two devices the problem is I'm starting to run into physical limitations of the media that I'm writing or reading from that is if I'm copying from an SD card or a SATA SSD which runs on my ninja Atmos here and Records all of my videos I've still only got 550 megabytes per second of read speed out of that that's not going to translate into 2 GB per second to be able to feed into my server as far as reading data back off of the server even though I've got four Gen 4 nvmes in here I'm still running into limitations of SMB as a file service share itself it seems to be limited again to between that 20 and 25 gabit per second at least as far as single thread operations when it comes to current generation CPUs so 100 Gig it's cool to talk about it I love that I have it I love that I actually got it to work within my environment the problem is it's completely pointless even with the highest and Hardware that I'm throwing at this with a number of Gen 4 nvmes and a 13900 k or a 550x on the other end of things it just doesn't translate into real world performance storage gains and finally we come to the last couple of items in my server rack and they really haven't changed from the last time we went through this tour my firewall is still a UniFi udm Pro running stock firmware and basically only being used as a router and a controller for the other UniFi devices that I have in my network I have have three access points that I run here in my house uh it runs perfectly fine with gigabit connectivity to my Wan I have two isps here at my house one of them is Comcast I get a 1 gig by 35 uh W connection as my primary Network and then I also have a starlink connection serving as a backup as well as an internet connection that I can take when I'm off camping and I can still do live shows and edit videos and that kind of thing remotely utilizing that when I'm on the road at this point if you have terrestrial internet that is wired Comcast Cox FiOS whatever I would definitely recommend that over a starlink connection as starlink has fuzzed out on me a couple of times however I've also had Comcast fuzz out on me and having starlink as a backup connection has been a fantastic little benefit of living where I do if starlink is your only other high-speed option besides hughet or getting something that's maybe a 5G connection starlink is probably the best option that you have and I'd fully support it if you need it in your environment in fact I know of a couple people who have deployed those to remote work sites so they can have high-speed internet when they're up in the mountains I've taken mine and done a couple of different live streams uh off my starlink Dish as well and it has worked fantastically well for that purpose last but not least I did rack mount my cable modem a couple years ago it is an RS surfboard sb8200 it technically will run Beyond gigabit speeds but Comcast won't give me Beyond gigabit speeds it technically will do uploads of up to 40000 megabit but again I'm stuck at 35 and finally if you are a member of my patreon on the premium Services you saw a video in which I built this this is just a two you blank plate that I thought I'd have a little bit of fun with my laser engravers and it says home lab Sweet Home lab if you're interested in buying something like this I might consider putting these on the store so let me know if this is something you'd be willing to pay for uh I've been toy with different things that I can add to craft computing. store to kind of spice things up particularly for home lab use case or maybe like Boutique system builds uh let me know if you'd like some vanity rack plates might be something that I can carry here in the very near future and that's going to do it for my rack tour here at the beginning of 2024 it's literally New Year's day that I'm filming this on let me know if you found this at all interesting and let me know if you guys have any ideas for things that you'd like to see done with my rack in the New Year I'm always looking for new ideas and content that I can bring to you as always make sure to drop this video a like And subscribe to craft Computing if you haven't done so already follow me on the social medias at craft Computing and if you like the content you see on this channel I want to help support me in what I do again visit craft computing. store for my latest in Pint glasses uh drinking accessories coasters and in fact I'm going to add some coffee tumblers here probably within the next week so make sure you check back often for new merchandise that's going to do for me in this one thank you all so much for watching and as always I will see you in the next video cheers everyone you didn't think i' make it through the video without opening a beer did you uh this is a beer that I reviewed on the channel before this is from Breakside Brewing it is the rainbows and unicorns uh tropical summery goodness you're in for a magical ride it's just a very simple IPA I love the can art on this can uh it's just fantastic I I love the the rainbow unicorn um just a good simple drinkable IPA 5 and a half 6% very bright without getting overly citrusy there's a reason domestic beers are popular and that's because ice cooled out of the fridge out of the chiller uh they're very refreshing they're very pallet cleansing this has a lot of that going on without getting into like the IPA bro kind of flavors it's bright it's refreshing it's a fantastic summertime beer but even here on January 1st I find it quite good cheers everyone around the house as well as a couple of ipmi devices down here in my server rack ow that hurt Dam I just hit my arm on the latch for the server rack I don't know if you know those are made out of solid steel oh there's already a KN forming in it oh that's going to hurt

Channel: Craft Computing

Views: 65,616

Keywords:

Id: rRXC8CzMpos

Length: 28min 31sec (1711 seconds)

Published: Tue Jan 02 2024