I'm finally listening to you - Server Room Update 2023

you guys have been telling us for years that our infrastructure the servers and the networking gear that our 20 video editors not to mention the other 90 employees rely on is a little Jake you weren't wrong then and realistically you're probably not going to be wrong a few years from now but we haven't been standing still and in the last year alone we've effectively replaced or rebuilt every single server in this building making them faster better more reliable cooler far less power hungry in fact we've done so much that most of the servers you've ever seen on camera are in this graveyard right in front of me which means it's high time to give you guys a full update on all of it including some brand new gear that is unlike anything we've ever shown you before first up is the main server room at the OG office where you can see there's been some serious server shrinkage there was one point during the Vault recovery process where we literally only had three rack units left and now this is only about half full but the important stuff is actually at the back where our old PF sense router has been replaced or rather it's pulling Duty as a steam cache now our steam cache used to be in a VM but we were having issues with it it'd be like weird performance things would be like 10 gig download and be like one gig I mean 10 gig again and one gig so we're back on bare metal now which is fine because we upgraded this thing anyway only to discover that PF sense didn't run on the new box this super micro one you is super cool it's got 25 gig networking on board except no drivers and PF sense yeah it's running a really old FreeBSD version fortunately opensense is a fork of PF sense that gets updated way more frequently now that could be a double-edged sword your organization might prefer the slow plotting approach to updates but we live on the bleeding edge where we love that we've got support for our 25 gig networking in here and you update it like a couple weeks after the news yeah I usually wait like a month after a big release before updating and it's been Rock Solid for us zero issues if we wanted to go back to PF sense though we totally could is what I would say except they still don't support this thing no they don't the other changes back here are not so much Hardware but rather topology we've still got that awesome Dell 100 Gig switch but now we're wired directly into all of our editors and ingest stations via fiber running 25 gig and we're using the 100 Gig ports to connect to each of our Downstream switches where we used to have it going from one to the other to the other to the other so that if we had to update one or something uh it would knock out everything Downstream of it which to their credit these Dell 48 part 10 gig switches other than updates have they gone down at all I've never had an issue with any of these three Dell switches ever they would have 100 up time if we didn't update them shout out ubiquity as well though we've been really happy with their Poe switches this one right here runs all of our cameras all of our Wi-Fi access points and then this guy up here was actually added when we acquired the unit on I guess this runs to both other sides right 101 and 105. so that's the shop and like where the WAN show set is and this one's got a 25 gig link to our 100 Gig switch now oh there is one other new piece of Hardware we have an NVR back here now yeah we have two actually we previously only had one but we had so many cameras that it was like starting to like cause weird performance issues so we split them out into two ubiquity keeps telling me that they're gonna add NVR stacking or you can combine them together but for now I have to like click between the we're really happy overall though yeah yeah yeah big brother's watching yeah yeah yeah yeah wait I missed something else what's this Cloud key doing up here well it used to be on this side let me remember don't we have a dream machine here I have one in my house no oh open sense right I can't remember why I thought you can run this in a VM now okay all right it was and then the database corrupted so we couldn't adopt anything properly so instead of just re-adopting all of the like 40 switches we had I was like what if we just transferred to a cloud key and it fixed it well it's not a cloud key now again like it was before love doing work twice the other thing that particularly the Europeans have tripped us about in here is our use of 120 volt power in a server room and while the output on this does stay only 110 and 120 volt is the options it's got two 120 volt phases so if you pair it with the right pdus or power distribution units you can totally do 208 volt which we're now doing thanks to these super cool pdus from triplite they're networked so you can not only monitor the total power usage but you can actually switch off individual Outlets say if a network device became unresponsive you can even create automations for it the main benefit of switching to 208 volt is efficiency you've probably seen in power supply marketing materials that there will be two lines for the efficiency curve one for 120 volt and one for 200 plus and the reason for that is the higher voltage allows your power supply to operate more efficiently it's exactly the same for servers so we dropped our power consumption in here by about 10 percent just from that changeover that means that combined with the server consolidation we've done and the additional battery units that we've added to our 9pxm this Eaton UPS can run for almost an hour with the power out completely which seems to come up around here a lot now let's talk about the great consolidation after we talk about our sponsor the ridge looking for a stylish and practical wallet the ridge wallet has got you covered it's minimalist design allows you to carry your essential cards and cash without the bulk check out the link below and use code Linus to save 10 off your purchase and get free shipping where we used to have five 45 Drive storinators taking up half our rack holding about 60 drives each we now have three of these super micro J bods and what we call the mother Vault each of these holds up to 90 drives and then we've got this single AMD epic Milan head unit that uses two epic Milan 32 core CPUs and then it uses external SAS connectors to attach to the j-bods right now we've only got a single j-bot attached to each of the SAS hpas in here but theoretically nothing would prevent us from adding another I don't know like three of these once we clear out Clover server this is running ancient Hardware it says Xeon 2699v4 I mean that's 28 cores but they are flipping slow and we're using this for all of our virtual machines which do we even have that many virtual machines what else we got going on in here not really this one's being deprecated soon in place of Clayton's server up here which is running proxmox instead that will be where all of our VMS move in the long term does that mean I can take this CPU from my home server sure our Clayton proxmox Server doesn't really run a lot of VMS to be honest we have a parsec relay which makes Park SEC run a little bit faster when you have a bunch of different users we have a couple VMS for the labs team and then the scene cache was on it but then like I said it kind of had issues so yeah 64 core epic 24 Bays for four drives in like three VMS wait we didn't mention the mother vault is running true Nas yeah I mean basically all of our servers run true now shout out trunas shout out ZFS and Shout Out Wendell from level one text hey it's not like the Vault data is essential that's why it wasn't redundant then and it's still not redundant now but it's a super nice to have and thanks to Wendell we managed to get back what like 98 and a half percent of what we lost most of it and to be fair I haven't run into a situation yet where someone's been like hey I need this file and I can't find it but I think it's freaking awesome and it's wild being in Windows Explorer seeing this single folder this single machine that now has everything 2.4 petabytes or whatever unbelievable speaking of unbelievable we got hated on a fair bit for this thing and you were right but not about the performance yeah this thing has been crazy reliable crazy fast it's basically the bee's knees I've only really went into one issue and this was recent the day of the heckoning one of the drives decided to shut off and if you want to change out an m.2 drive that's loaded onto a carrier card that's loaded into a module thing that's loaded into a server it's a bit of a pain in the butt especially because we didn't have the foresight to write down the serials and which carrier card they were installed in absolutely zero idea which drive it is oh my God that's what you did so what we ended up doing to restore the array's Integrity is putting a mismatched seven terabyte kyocsia drive into one of the front nvme Bays Jake yeah and the plan is apparently to do something about the serviceability of it soon TM well it hasn't needed any servicing right so to be fair when I restarted the server the drive came back online it has been online since we do have some ideas for improving the serviceability we found out that you can get u.2 to m.2 carrier cards so we could actually just take all those internal ssds load them into the u.2 slots in the front I guess we'll have to add more modules for that maybe use a different server or maybe use a difference oh okay so new new new monik is coming soon then wanted four and we could use those new sabrant drives they're sending over 22 of their wild eight terabyte rocket 4 plus drives so that we can try them out with that crazy 21 by m.2 carrier card that you probably saw in the news so wait no I am not thinking we use those carrier cards hey you were the one that brought that up no we're not we just put one it's the one one card server it's got 160 terabytes on one card yeah entire One X server and an Intel Nook yeah no no I mean you made it yeah it's not that way I'm the one that's supposed to say your ideas are stupid that's stupid it's not that stupid you like it I don't not after that meanwhile in Unit 101 where all the inventory and Logistics Department used to be but now you'll find only our makeshift short circuit set we've removed the server rack outright because we can't have servers sitting here next to where we're trying to film instead we've got this simple six rack unit wall mount here that's going to hold our two Network switches that handle all the network connectivity down here and upstairs we've also got a udm pro down here that's the router for our secondary backup internet connection it's not nearly as fast it's one gig instead of 10 gig but realistically most of what we're doing we don't need most of the time we just use it for like testing a router on short circuit so yeah it doesn't really matter now you might have noticed that this looks pretty okay this looks extremely not okay this is very much a work in progress these patch panels are going to come down here and then presumably at some point we're going to shorten these runs oh yeah oh yeah yeah because they're long enough to go into a rack well these will get replaced with little like those little ultra thin six inch cables from infinite cable Shadow infinite cables thanks for the thanks for these things and uh speaking of you know shout out shout out ubiquity for this access point yeah definitely oh my God is this the actual access point no no it's just like the backup connection one the only other noteworthy thing here is our ups this is to ensure that not only is our main internet connection battery backed up but also our secondary one that way in the event that we lose power to the building outright the WAN show can continue as you guys actually saw not that long ago that time we had to run it on jackeries because we actually ran out of backup battery so we bought a second battery pack that just needs to get put on here it's sitting over there it'll it'll get on there we gotta be able to do Wan show for like at least two hours yeah yeah on battery five moving over to the lab the biggest change is probably the complete removal of Windows from our infrastructure get it there used to be a window here yeah but because of all the sound bleed from the servers next door the poor social media team who's moving in here was going to develop tinnitus social media theme isn't that who's coming in here I hope this is my server room no what it what the server room's over here and now it is properly sealed so let's head in this room this is this no Honestly though removing the window and finishing up the drywall it's not really enough and it's especially not going to be enough with some of the gear that we're putting in here so we're doing it again but this time better way better hanging insulation on the wall of the server room yeah this time we're going to use this cool product called Sona par which like is a recycled wood panel they're like 30 bucks for a four by eight sheet which is really cheap for sand isolation and then we're gonna put rock wall on top of it with these cool adapters that Dan Made In fairness to me the original solution was messy but darn it did it ever work yeah it does work really well I took so much Flack for that I don't even know if sonopar existed then yeah and to be fair Nails is way easier than 3D printing hundreds of little brackets yeah yeah I did that in a weekend yeah we're gonna put the duvetine on with magnets it probably took like three months off my life with the amount of fibers I inhaled that yeah that's okay he got good life insurance man this is where you can see the difference between us eight and a half years ago architecting this stuff and then kind of upgrading piecemeal as we go and when we build something from scratch with all the knowledge we've gained since then here's where our internet connection comes in we did end up having to go for the expensive link over to the other building but hey now that the whole writing team's here and the lab is here we definitely do need it and then we've got these 100 Gig links that break out into 425s that go to these 10 gig switches one two three four five of them we might need one more too what the intention is for every single Connection in this building to be 10 gig obviously not cameras and like APS and stuff everything else is running off of these Enterprise 48 switches from ubiquity these are super cool they're 48 ports two and a half gig and they do Poe so these will do all the cameras and access points here which access points in particular now actually can use two and a half gig uplinks yeah so freaking fast and we put those Wi-Fi 60 Enterprise APS everywhere in this building at least with what's done so far we have ripping fast Wi-Fi here there's more than just networking over here though we've got the NVR Pro from ubiquity this is handling all of the cameras in this building and you may recognize this as the storeinator that 45 drives sent over for our vault recovery well now it's pulling completely different Duty as our Annie Dale server so this is our footage Dumping Ground and is one of the backup targets for our main video editing server wanick it's over at the lab now for more physical separation so that way if one of the buildings were to burn down in the past they were just two separate units of the same building now they are physically separate buildings probably the coolest thing over here though is this our first attempt at a high availability server that if all goes well is going to be the new wanek editing Nas you mean new new new high availability wanted yes yes that's the name we're gonna go for J wanted four um how do I get these out uh there's like a little [Music] each of these four units in here has a 64 core epic melon season these are Genoa yeah yeah so it's like 384 gigs of RAM 64 core Genoa six UDOT three slots in the front for pcie Gen 5 storage if you want and then there's two little m.2s for boot and there's four of them in this to you holy crap so there's 256 cores the idea here is that in the same way that you would run multiple hard drives so that if one fails the rest of them can just take over you run multiple servers so that if one fails for whatever reason the other three will just keep on trucking now there's a number of different ways that we can combine the storage of these servers whether we use something like Gloucester FS on trunes scale or Seth or even weka FS yeah we were going to use that weren't we we're working on it okay maybe yeah maybe we might need a second one of these because weka apparently needs eight nodes minimum now but still then we could have like three of them fail I think and still have storage which is crazy the other cool thing is you can update it in the middle of the workday like it's nothing oh that's super cool yeah bios update stay tuned for that yeah shout out Eaton by the way for this 9pxm UPS I think is the same one that's on the other side yeah did they just send this yeah they also sent us another two pdus which is super sick dude the pdus those aren't even the final ones they didn't have stock of the ones I wanted that have individual per Port energy monitoring no way so they were like oh yeah you can just have these ones for now and we'll send those ones when you want it and they're also providing full cable tray for the entire Warehouse so we can actually run all the ethernet in like those fancy trays so eaten our just complete Bros now yeah man their stuff is like aced here that's the that's this is what you want speaking of cable tray I guess that's the plan for this yeah the reason that that hasn't been cleaned up is we have to run like another 30 or 40 wires you might as well just do it all at once got it we didn't mention this on the other side but you might have noticed we have these color-coded power cables coming off of our pdus that's so that we can ensure that each of our servers is powered by two separate pdus so that in the event of a pdu failure it'll still stay running thanks to the Redundant power supply the last part of the infrastructure changes are on the client side specifically for our writers ubiquity hooked up each of them with one of these cool I think they're called the switchflex XG they're a Poe powered 10 gig switch but in our case we're just using wall power it gives each of our Riders four 10 gig ports and a gigabit Port obviously we have test benches and other devices that we're working on so just having one 10 gig connection to each workstation was a bit of a bottleneck this way they can plug in a bunch of different devices and still get that 10 gig speed which is obviously very important if we need to review footage off this server or what have you it's pretty sick and they're not that expensive either I think it's 2 99 for one of those we almost forgot about the network closet that was here when we moved into the lab the intention for this is for all of these cables to get moved over to the new server room the reason it's still here is these are all the old Connections in the warehouse that haven't been redone yet because of that cable tray so once that's all done these will probably just get cut out of the wall and recycled and yeah there won't really be anything in this closet we could probably just use it for cleaning products or something like that grammarly is throwing their hat into the AI ring with grammarly go which offers generative AI to accelerate your productivity while staying true to your personal style what makes grammarly go so unique is that you can customize your preferred communication Style by setting your voice to personalize your tone the business team uses grammarly go to rewrite emails and talking points to be more concise all while keeping it authentic to our voice it's me need help with a video title just input a prompt and grammarly go will provide innovative ideas to unlock your creative potential our favorite part about grammarly go is that it's there to help you write not replace you and there's no need to leave the platform you're on just download sign up and start typing you'll be amazed at what you can do with grammarly go so sign up at grammarly.com LTT and get 20 off grammarly premium and that's pretty much it big shout out to you ubiquity for all the networking gear micro tick as well super micro for all the servers infinite cables for the patch panels the networking they sent us like 20 boxes of a thousand feet of cat6a they got really sick patch cables 45 are you pimping infinite cases again yes yes 45 drives also makes awesome servers we're just not really using them aside from this one but definitely check them out God what's up what else is there oh I guess a video to watch probably the one where we put all this in this rack because that was just like pure distilled chaos or you could go watch the one on the mother Vault yeah that was a big change for us oh my gosh it's so nice though bye

Channel: Linus Tech Tips

Views: 2,300,833

Keywords: server room update 2023, ltt servers, ltt server update, we fixed our servers, networking ltt

Id: lK1gPoGxgAA

Length: 20min 57sec (1257 seconds)

Published: Thu Apr 27 2023