INSANE Homelab Networking!

this is Jerod with Digital Spaceport and if you have a home lab that's trying to come anywhere near this you're going to need some serious networking so join along today as we go step by step through redoing all the data networking at any point in time you can use the chapter guide below to skip to any part of this guide so first off let's start talking about the plan just a little bit I think people are after the action so we're going to try to get you to that as fast as possible here so one of the ways that you can start off is just doing a written guide this is probably better before you jump into anything like Vizio or draw.io and start trying to map things out outline in each one of these is my rack I've got rack one rack two and rack 3 up here and I've also got the networking stack highlighted here now this is basically so I can get the count of number of cables and types that I need for each one of these compute clusters the networking stack is one of the things that we will be changing first off here because the placement of some of the gear is just not right so one of the things you need to have on hand is all the cables before you start I've got some qsfp cables here I've got tons of SAS cables here I've also got some om2 I've also got some om3 and also some single mode which we're not going to be using I've got a variety of network cards here these are 40 gigabit melanox connect X3 cards these are just excellent and they're a great price and I've also got some melanox connect X2 cards here that we'll be using for some of the 10 gigabit networking I'm going to be additionally doing some future guides on storage so make sure that you hit like And subscribe so that you can get involved in those videos as well as we build out multi-pepa byte high performance compute clusters we also have some active cables that are qsfp to SFP plus these are from fs.com and we also have an active active cable that is a three meter fs.com qsfp plus and this will be used for some of our interconnects I also have a variety of different colored cables here I'm going to go over why I chose some of the cable coloring that I did I love to use pre-booted cables that are long and then just cut them to length and then punch them down that saves a lot of time on having to fit and also getting a good quality booted fit on a future cable we also have some support rails and some ties and we also have all the punching gear we would need a toner for making sure that we've got good adequate connections on our cat cables and also a crimper in case we do need to actually terminate anything I've also got a pile of velcro here and we've also got a bunch of zip ties and that should just be fun to see what people say in the comments below do you like zip ties or do you like velcro I also have a patch panel here that we're going to be installing as you can see I've got most of these pre-terminated already with the Cat6 cabling I've also got some wall mounted stuff here that I'm using to organize cables this is a freakishly large amount of Dax that I've got here and we've also got a couple of qsfp cables here these are good in case I need to add on some one gigabit connectors on the Fly for any reason and I use this here for connecting and spanning so that I can plug things into the toner wherever it is at and then just test them that way we also have a variety of our cabling that we'll need for connecting the power I like to connect the power first and keep that on one side of the rack and the networking and all the other s under cables on the other side of the rack I put the powering on the right hand side of the rack and the networking on the left hand side of the rack there's multiple ways to do that I'll talk about why I do that in particular since I have mainly Dell servers and so one of the first things that I'm going to do is go ahead and remove the existing cabling that's really just been put in piecemeal here if you see you can see it spanning some places from the outside that's just totally not acceptable we're also going to be installing the race track that we've got setting over here and we'll be using that to jump from here up to this cabling racetrack Runway Ingress point and that will span all the way over into the structured media panel which is the d-mark between the workbench and also the house is networking and we're also going to be doing the arrangement of the networking stack that I've got here if you look here I've got the icx this is a brocade 6610 hyphen 48p this is a Powerhouse of a switch and it also unfortunately eats quite a bit of power I'm contemplating removing this from the stack I'll probably still install it but hopefully I can bypass this and not need to power it on I would urge you to think about the power usage that this will incur it is about 140 Watts no matter what so that's pretty much a very high power usage switch we also have the melanox 6036 up here this is a 36 Port qsfp this does eth and infiniband if you are looking at either of these two switches on eBay links below to those do contact the seller and have them validate the licenses are in place for the advanced functionalities on these I also have some fingers here that are in the wrong position and I've got up here a Dell 55 48p which is a wonderful starter switch which can be had for pretty cheap and it has two SFP plus ports over here that are great for connecting at 10 gigabit uplinks to your network if you have a small Network this is a good way to go and these are usually pretty cheap at about 50 bucks and here we've got the Dell poweredge r2102 we've got a couple of low power usage processors we're going to be testing out in this this is going to be a proximox node it'll be housing a VM that VM of pfSense will be used as failover in case our primary VM physical machine actually fails that's inside the house and that networking video will be coming up shortly and so I'm going to go ahead by finishing up the removal of some of the networking cables that I've got here and this way we can actually go ahead and disconnect everything from the back side next and so since we're working on the back of the rack I'm going to go ahead switch this to bypass mode on both of the UPS's that we've got down here these are Eaton 9px 11 kilowatt units and the next thing I'm going to do is go to the front and go ahead and power down the UPS's now that they're in bypass mode and then we're going to go ahead and kill the power to the circuit breakers that feed the pdus and so all of the servers and all of the storage are shut down right now while I go ahead and power this down by holding the power button for a few seconds and then letting it know that we want to go into the shutdown mode we're going to do the same for the other one over here as well and now we're going to go ahead and hit the electrical 50 amp disconnects here and as I do that I'm going to come over here and switch off the pdus and we can see they're now powered off and the fans of now stopped okay so I'm going to go ahead and remove and place these cables in their appropriate homes here so that I can find them again when I need them here shortly and this is one of the reasons why it's actually pretty handy if you have some storage space behind here one of the things that I added after the build out of the server racks and the partition walls which are very well insulated you can find links to all of that historically that we've done under the live streams in this channel is I built this nice little pegboard wall and this is a great way to hang a bunch of cables which are exactly what you need behind a bunch of server racks and as I go down here I can see that some of the electrical cabling is done on these pretty good good enough for me so I'm not going to redo all of the Power cabling on all of them let's go over to this one here and it looks like yeah I've got some SAS cables here that are just and I would recommend if you can avoid doing one of the one to four breakout cables to just not buy one of those SAS cables oh most hit some hard drives with my elbow there I decided on the Fly that I was going to go ahead and not remove the 6610 from the stack yet I think it's safer if I just go ahead put it in hopefully I can actually get to the point where the 5548p is again the main one gigabit switch but for right now the 6610 has been something that I kind of engineered in a lot of the settings on it already configured for what I've got and I don't think I want to spend the time re-engineering that at this exact moment so I am going to go ahead I redid some of the layout here to reflect that so we're going to have our patch panel at the bottom the icx 6610 the 5548p the melanox 6036 the r210 II we're going to have a spacer and then we're going to have two user fingers up above that and removing everything from the rack is a good thing it just feels good whenever you take stuff out and then get to put it all back in the cabling a little bit less fun but definitely dealing with a bunch of networking switches is a lot of enjoyment so we're going to start with the icx 6610 and I'm going to go ahead and slide this into the rail setup that I've got here and if you notice this is actually setting on top of some old rails that I've just repurposed they have a fairly decent lip and this will keep it very flat and straight I'm still going to go ahead and attach the two screws into the front here to Anchor it back now we're ready to put on the 5548p and also the melanox can 6 SX 6036 all right and finally we've got our r2102 and I'll use some zip ties to secure these in place and maybe one day I'll buy actual ears for the r2102 and rails but I would imagine I actually just upgrade to a different system before I do that and well I can't say that I think that zip ties have a use for everything certainly uh they can work out pretty well if you need to improvise something to hold something in place really quick and you know you can buy some pretty strong industrial strength ones check the link in the description below for a large pack like I'm using from Amazon and so you'll notice that I'm using a magnet here this will catch all of the little shavings that are coming off of this this magnet came out of an old hard drive if you've got any old hard drives always save those magnets they are some very powerful magnets all right so just using some more zip ties here to secure the switches again these are not coming in or out so I think it zip ties fine send off and let me know if that's upsetting you in the comments below all right this gear is in place it is not moving and I like that a lot all right now we're going to put our fingers up here and so I'm going to go ahead and put the nuts in here and after we've done with that should be able to attach it up here and I'll just hand tighten one up on the top and one on the other side wrong size and so I had a lot of extra long cables that are going to be needed to reach the second rack there's a trick to doing this if you want to do it easy instead of trying to measure each one cut each one and get it right and that is to hook them all up in the back and then to pull them all forward and then to go ahead and cut them to the links you need and then punch them down I'm going to show you how to do that let's get this done thank you you want to make sure they're not going to just pull back out okay so we already have wired and punched up the networking for this rack but we don't have it over here on this rack so what we're going to do is take these that I just stuck through here and run them through the intended pathway and so we're going to be using these 1u Cable Management little units and these are great I'm going to link you to them and these are available from monoprize for very cheap if you put them at the very top of your back side of your rack very likely unless you've got a switch up there you're going to have that space free and they can really help you route the cables nicely and then go ahead and pull them on down and so we see that their heads on this end are looking about the same I'm actually going to go back to the other side find the ones that are the shortest because this one and this one can have a shorter one and these guys need the longest ones because of course they've got the cable management arm and that's going to take quite a bit of span okay and so we've got three super long ones over here we're gonna make sure those ones go all the way down in the bottom so one of the things you don't want to do is leave tremendous amounts of slack on the server side so you want to go ahead and pull this back over to the rack side whenever you have a whole lot of Slack happening and I like to make sure that I leave a little bit of slack on the computer side but once it enters the cable management arm I don't like to leave a lot of slack inside there so coming up through here is going to be the qsfp cables they're quite large and chunky so I'm actually going to take this and this seems like a pretty good placement for it I might actually consider yeah this looks like a good spot for it I'm going to go ahead and put this brace up here and that will support the cables as they come in from the server front side foreign we're going to terminate one here for slot number 32. and we're using B wiring which if you are doing anything it should definitely be be wiring and that is going to be outlined usually on a sticker on the back of your punch plate so just adhere to that almost all of your wiring will be be wiring and so for us that's going to be blue white blue in the first tab and then I like to actually go ahead and stretch out to the last tab which is going to be brown white brown foreign make sure your punch is facing the right direction and you can kind of just get it set down there so it won't be pulling out and then after that we've got green white green and orange white orange and you're going to do this for each one of the remainders that you've got left we'll pick back up as soon as I get this finished okay and so once you've got the rough sizing that you're going to be using for your various cables go ahead and make sure that there's none tangled up here like this one that's trying to live underneath the other cables this way you can keep a good separation between them I'm using two different color schemes blue is for communication data and also red is for connections that are going to power infrastructure so the next thing that we're going to do here is there's one of these that I need to make sure does not have okay it's this one here I need to make sure this one doesn't get done until last because I need to account for the extra space since I don't have a cable management arm for the r520 these over here are all amply long and they're already pulled and tensioned and set in their actual locations over there so one of the things that I was able to do which really helps out whenever you're looking at doing this is go ahead put the cables in go ahead fit them in the locations pull them up have them all actually ready to go this way when you go to terminate you know how much space you need to leave on the other end all right now we're going to go ahead and install the patch panel here that we got it terminated on the back side I've also got the top support bar in I'm going to go ahead and put the bottom support bar in as well that way we can have the zip ties attached to this and holding the cables secure as we pull it into place and this side is a little bit tighter but we got that hooked in there now all right now let's go ahead and get this setted in place and that'll clean off with a little bit of water I hope [Applause] foreign so now that we've got the patch panel in place I'm going to go ahead and put in the patch cables here these are the Slimline mono price and they are actually really good for what they are although some of these were actually various sizes they were supposed to be half a foot some of them are shorter than that some of them are a little bit longer than that so I had to sort them out ahead of time just keep that in mind if you're looking for any patch cables all right and now we've got our patch panel wired up with all the connects that we have for the back side so we've got the race track here that we're going to install up here I've got a couple of actually ladder holders that we're going to use as supports and some two by fours here and then it'll actually angle out and come over this direction allowing us to have the rack Cable Management on the top as well so I think I've got a plan here it looks like I can actually attach this I'm going to do some heavy duty zip tying of these two together and then I'm going to put a support here where there's a stud and then here we're actually going to have the slide down and it'll go to that transition piece there it should work out we're gonna find out have it just sitting right like this and I'm going to put one of these in here and I may even put a little bit of an angle on it so that it can slope down somewhat and let me know in the comments below if you have server racks and if so how many server racks what kind and what kind of cable management you're using for your system and I like to improvise whenever I see the opportunity too so sometimes you gotta be willing to make things Bend to your nature and it looks like I'm going to need to do just a little bit of fabrication here on this as well let's go cut some metal [Music] and it should be able to lock into place on the other one with two zip ties so now I'm trying to get this to a good angle that is going to run perfectly with this so that it can set the last piece of rack railing that we've got there and also get the standoff see up there installed as well and check the links below for some not cut up Raceway that you can find at Amazon and this is actually a link that should get you all of the different Cable Management things that I've talked about and showcase during this video this is a freakish amount of zip ties it is actually really really solid and this now curves here we're going to extend it with that last final piece and we should be able to put on the stands here at that point and then have our Ingress point for top managed cable any drops that we need to come out here and meet up with the dmarc or go further into the house there's one 40 gigabit span that we're going to be doing at some point in the future and that is definitely going to require a this so this was one of the reasons that this had to happen all right let's clip these off now we're going to go ahead and install the standoffs here and so once you've got that in there you can see it provides pretty darn good support and this one we're just going to have down here a little bit more this is another good reason that you need to have liquid tight for your whips or some sort of fixed conduit that is rigid and very protective to ensure that you don't actually Nick it and cause an electrical issue all right now we're going to go ahead and tie up the power cable here and now we're done with the Raceway installation up here this has taken just a few minutes to get done but the end result of this is actually pretty darn cool I think this will definitely do as well and now the best part of this we're going to be installing the melanox connect X3 networking cards these are 40 gigabit capable nics and they also do infiniband we're going to be using them for 40 gigabit Etho into each one of these servers and we're also going to use that to connect up here to the melanox 6036 network switch that has a whopping four terabits of switching power in this unit and we're getting this installed in one of the first r 9 30s here next up let's get the t620 here foreign and as you can see I've got a GTX 3080 TI inside here this is an absolute Powerhouse of a system all right and now we're going to go ahead and get this slotted in here and now we're going to go ahead and install it in the r920s and now we've got one last connect X3 that we're going to get installed in the r720 XD yeah foreign so now we're going to go ahead and actually route the cables through the back Cable Management I'm going to actually detach them now that we've got the sizing of the cables adequate the r720xd we have an eBay order in so hopefully we get the cable management arm for that pretty quickly we also have an order in for the r520 cable management arm but I did go ahead and create what I think is probably not going to be enough slack here for this but we will actually adjust this whenever we get these in and at least for now it will be working so I'm going to go ahead put the panel right here this is another finger panel we're going to feed these through the finger panel and then route them up and into the Cable Management areas then we're going to go ahead and connect the qsfp chunky cables up here they're going to come from the front from the melanox 6036 up here we're going to tie them to this bar and then route them this direction and down this direction and down and then we're going to connect the spans that need to go over the racetrack up into the remainder of the house and that will be the wrap up of the cabling of the networking okay so I'm going to go ahead and undo the cables here and the ties that we've got up here is held them all in place so that I could get the right sizes on everything and now I'm actually going to go ahead and detach the cables from all of this pull them back but I'm first going to label which ones go where and now the most important part of any project making labels that's all there is uh a fresh roll of labels perfect now let's go ahead and put our labels on the server's gigabit connections foreign [Applause] Ty one and as I kind of get these done I'll push them over to the side over there if you'll also note I've been using the label maker to actually label which Port this is connected to on the patch panel so I've got number 27 and number 29 here on this one for the second r930 let me know in the comments below if you love label makers also and don't worry I'll be running the shop back over this entire area when we're done okay so this side of the rack is done now I'm going to go ahead and pull them from this one over here so now I'm going to go ahead and install these in here okay so the first ones that I'm going to feed up are this cluster of blue ones here all right and we've got the 520 here this needs to actually stay over on this side so I'll kind of separate that now we've got the 720 620 and the 4 R9 20s and these are tied back there very nicely so they're already securely in place held with some zip ties to the retaining bar on the back side of the patch panel and while this may not be exactly the intended use of this it does work and I'm not going too tight here on this I'm not trying to cause any harm or damage to the cables so now I'm going to go ahead and Route them on up through here I'm started on that first slot in a second and then the Third see how that snagged on that cable management arm there this is exactly why you need to have cable routing in mind whenever you have a server rack and now before I bundle those up there I'm going to go ahead and connect these down here how did that happen all right now we've got the CDU cables pulled up here glad that I labeled them so I can see that this is going to cdu2 over here so we can pull this one up and get it fed over to the network connection for that port and there should be another really short one here this one yep this one actually is going to go over here to cdu1 so that's going to rest inside this cable tray one of the reasons why we actually are using a cable tray here and we've got cdu3 here so this takes care of the cdus once I get this one wired in and I am not loving the left hand side Cable Management little clamps that you can see right there on these racks but it's what they came with make sure when you're checking out your racks that you evaluate them try to find something with zero U trays and I already know this long one is going all the way over to the second eaten 11 kilowatt UPS and every now and then folks ask me how I'm liking the Eaton 9px 11ks love those units they are really good highly recommend them if you can afford them and these two shorties are going over here to the r720 and that'll pull out and up whenever I pull it forward still not ideal but probably not going to be messing and de-racking any of this until we get the actual uh Cable Management arm in and we got the final one here and it is going down to our eaten first 11 kilowatt pdu and now I'm going to do a little bit of cable bundling of the various groups that I've got here the blues again standard traffic the Reds communicating to the power units so like I mentioned I'm using the red for the power Communications and the blue for the standard traditional TCP Communications but that doesn't mean that you can't make up something of your own and it really doesn't start to take appearance in my opinion until you start bundling up the cable groups and then you start actually Distributing them then it starts to look pretty clean then you actually have a really decent Network that's not cluttered and now we're dealing with our last bundle of cables here this bundle is going up here and these guys down here are for these r930s and now we're connecting up to the brocades management port and the idrac on the r2102 and when you start a rack wide project like this whether it's cabling whether it's redoing servers whether it's bang together racks make sure that you allocate plenty of time this entire process actually took place over a week so do keep in mind that this is something that you're going to take your time you're going to plan it out you're going to execute it and then you're going to evaluate it so you're unlikely to be done in just one day with this level of a project before we go further with the cabling let's go over what we've just accomplished here with some of the cat cabling some of the things that you can do if you're looking for ingenious little ways to wrap some cables on the cheap in your racks these little one you guys are amazing for this just toss them up here probably nothing going up here at any rate great for running spans across the top of your rack we're going to use those also for the 40 gigabit qsfp cables and this here these fingers also pretty cheap and you can really do some coil cabling inside them here routing some down routing some up and it's got a nice Snap-on cover that's going to go on the face now it's time to go ahead and install our qsfp giant chonkin 40 gigabit cables check the link below for 40 gigabit cables on sale and so we're going to start by lining these up and skipping the first plug and routing these through the back after we straighten them out these are gigantic chunking cables active cables are probably a great idea for a lot of people out there they are more expensive these are significantly less expensive so that's why we're going with these for our home lab foreign cables are super inflexible and I was able to fit two in between each one of the fingers but that's really all the space that I had there so if you are putting a whole bunch of them in place keep that in mind also make sure that when you're tying everything up you start routing things from the bottom if you notice on the left hand side I've got some nut super great cable ties but I'm using those all the same trying to keep things as clean as possible these racks are not ideal racks I would recommend checking the links below for some better racks or if you can find some APC net shelter racks those always are really good those would probably be something that you would want to look for locally but these are some older Dell racks and while they are functional they are definitely a little bit on the short side and the cable management does not have the nice zero you trays that you would hope for that is something that when you look for a rack make sure you try to find zero you tray support now we're on one of the last parts of this installation we are going to be pulling the dmarc wires over into the rack area connecting those up and that will be the interconnects between the household and also the racks for communications so we have already over 300 gigabits of installed capacity on the rack which is insane and we're going to up that by just a little bit more here so let's get started and this white wire is actually a Poe device so we're going to be doing something in a future video about the poe stuff we're actually at after this fiber cable here this is the interconnect between the two so I'm going to go underneath here just kind of lay this gingerly along and so you want to be as careful as possible when you're pulling any om fiber optic cable through a potentially Jagged edged or sharp environment because one Nick could be all that it gets and that could actually end the life of the cable so as you notice I'm dropping this down here before I have some cuts that are placed and I actually did some coils later on that are really much better as far as how I am managing the cable that all happens before those Jagged edges that are there and you can see them kind of coming close really glad that I didn't Nick that cable but definitely if you are using some fabrication stuff on some Raceway be really careful because you might end up clipping one of your cables in like I said it doesn't take very much to destroy them so here we go we are connecting up this final cable to the house here and so we've only got a couple of active cables that are fiber based but this one here is the dmarcs connection to the house so that I can access everything on the rack at 10 gigabits per second and now we have the current interconnection for our house wired up we're going to put the blade on and this is going to be a wrap nice and clean so now that we've got it all hooked up let's power it on and I want you to hear what it sounds like when we get this home lab going are you ready for the power on let's get these two 50 amp Breakers turned on here now we've got the UPS's that have started their turn on process you can hear them over here so let's go on over here to the two Eaton 9px 11s and they're in bypass mode right now so it's going to take a second for them to actually start energizing their battery and it's telling us the load is unprotected because it is in bypass mode we're going to go to the back side to fix that [Music] and we can see we're in maintenance bypass here also okay and let's take a look at just all of that networking all of that comms that's hooked up now all that power beautifully routed yeah I do like it I've got to say I do like it a lot okay so down here on this we have a disconnect that we're going to turn this way and that's going to allow us to go ahead and flip on the pdus I'm going to actually try to time it when I do that so I'm going to do the one over here also so that we can get a really cool count up on them when they power on because they'll actually as switched pdus click click click [Music] all right foreign [Music] [Music] [Music] those fans are the sounds of 920s and 930s purring to life and they have a really dive bomber kind of sound whenever they're actually cranking it's a lot of fun whenever you actually are pumping out a lot of data through this to listen to them were Okay so we've gone through the network installation here we have a tremendous amount of SAS connectivity and we've redone our comms wiring so this is really the ultimate home lab networking guide if you have Bunches of racks and you've got lots of gear so we went through we installed a new patch panel I showed you how to do that we also redid the cabling for the switches rearranged the switches we installed Network race track and we also pulled all the wires and meticulously wired everything together as it should be so I hope you've had fun hit the like And subscribe down below if you have and thank you for tuning in be sure to check out some of the other videos in this channel in the home lab category where we do everything from building this entire data center during live streams to testing out various servers and other forms of high performance compute alright everybody have a great rest of your day

Channel: Digital Spaceport

Views: 131,442

Keywords: networking, sfp+, homelab, datacenter, data center, 10g, 1g, 40g, 10g network, 40g network, rack network cabling, rack network cable management, rack network switch, homelab tour, homelab network, homelab network setup, networking 2023, server rack, server rack cable management, mellanox, mellanox connectx-3, mellanox connectx-2, homelab rack, 40gbe home network, 40gb network, 10gb switch, 10tb network, homelab guide, home server, home server rack, networking guide

Id: nlC3euvNwXk

Length: 48min 59sec (2939 seconds)

Published: Sun Jun 04 2023