My server is broken :( - Fixing Linus' Home UNRAID Server

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments
Captions
One of the hard drives in my home server, it just died with my data on it. And that's totally okay because you see, I'm smart and always use protection. Look, device is disabled, but the contents are emulated because I use not one, but two parity drives. Meaning I can lose up to two drives in this bad boy before I risk the rest of my data. So all I gotta do is plop in a replacement disc, let it rebuild from parity and Bob's your uncle, server fixed, except I do actually have a problem today. When I built this machine, it was still the early days of the company when you could borrow stuff that nobody was using. Stolen. Whoa, whoa, whoa, whoa, hey. So my choice of hardware was dictated less by what actually made sense and more by what nobody else could possibly want, which is how I ended up with these eight terabyte Seagate archival drives. To be clear, I'm not saying they're low quality. I've been running these since 2016 when they were manufactured. I'm just saying that shingled magnetic drives are not great for RAID or even unRAID for that matter due to their abominably slow write speeds. And what really sucks is that while these drives are still available on Newegg, cost per gigabyte is awful at about $27. For some context, I could get my hands on Seagate Ironwolves in the same capacity, so proper actual mass drives, for just around $20 a terabyte. That is also a terrible deal. So here's the thing. There's a fixed cost to manufacture a hard drive. You've gotta build a chassis. You've gotta have a motor. You've gotta have platters and time on the production line. So as newer, higher capacity drives come out, these lower capacity drives can only reach a certain point in terms of price, which means that there's often a sweet spot for dollars per gig. For example, a 16 terabyte Exos, that's an enterprise grade drive, is just $15 per terabyte. That is only a 50% increase in cost for double the capacity. Now, if this was for work, I might be fussy and demand matching drives, but this is for my house where I'm running unRAID, one of the key benefits of which is the ability to mix and match. So I'm just gonna pick up whatever's the best bang for the buck and stuff it in there. Unfortunately, with that flexibility comes some trade-offs. Like a regular RAID array, I could replace my dead eight terabyte drive with a larger one, say a 22 terabyte IronWolf Pro, but I would only get to use a fraction of that capacity, eight terabytes to be precise, so it matches the rest of my drives. Then unRAID's flexibility shows up again, and as long as my parity drives match the capacity of my replacement drive, I get the full juice. So I made the call today to replace two drives, neither of which is my dead one. Then I'm gonna take my two parity drives and throw them into the array to replace the dead one, to give me a little extra capacity today, and with these juiced up parity drives, to give me an easy, cost-effective path to even more storage. The bad news is, as clever as our devious plan is, it is a little bit more complicated to execute. A straight drive replacement is pretty simple, because it uses the parity drives to rebuild the data, but if we wanna replace parity drives while we have a dead drive, is there a safe way to do this? Do we need a third drive to sit and take that spot for now? I don't know if you can do that in unRAID. I do happen to have a third drive, so we could just replace the dead one, then do the parity swap. The way to do it would be to replace the dead drive and then replace the parity drives. I think we should do that. I have a third drive. We're gonna cheat. I mean, in theory, we could replace one parity drive at a time, I think. But here's the thing. The most likely time for a hard drive to die is during an array rebuild, because you're putting strain on all the drives in the array. So having two parity drives means that even if one more drive does fail while we're doing a rebuild, we're still golden. If we were to intentionally take out a parity drive, then lose a drive during a rebuild, we would be kicking Jake, because that was his suggestion. I'd also be kicking myself, because I knew better. I knew better. Let's just use a third drive. Yeah, and then that way you have even more capacity. Do you happen to know the serial of the dead one? Does it happen to be F4SB? What are my odds here? No. Rats. E-W-D-E. E-W-D-E, got him. That's not so bad. Are they screwed in? Uh, I mean, they're supposed to be screwed in. I don't see any screw. Oh, we got one screw, boys. Yep, this server was built by me with this Noctua edition LTT screwdriver. How freaking awesome does that look? You can sign up for a notification when it gets in stock, lttstore.com. Two screws is enough, right? Enough screwing up the segue to our sponsor. Sorry, I was supposed to do that a while ago. MSI. You can game, stream, and create to your fullest potential with MSI's RTX 4060 Gaming X Trio graphics card. With their Trifrozer III thermal design and support for MSI Afterburner, the sky is the limit, but don't drop it. It can't actually fly. You can check it out at the link in the video description. Did you have the other drives yet? Well, not yet. Do you want me to add the other ones right now? I might as well. See, I could, but all the hardware for this computer is inside it. Why don't you just take one screw from that drive that had two screws and use that? These are really shoddy trays. I don't know if one screw's enough. I mean, it's just gotta hold the drive attached to the sled when you pull it in and out. I think it's probably fine. Oh my God. Against my better judgment. What are you talking about? You were just saying your sleds have no screws in them anyways. Better judgment, my ass. I just put that bloody archive drive back in. Where did I put it? Oh my gosh. EWDE. EWDE, yeah. It wouldn't have actually screwed anything up, but it would have confused me really good. Oh no, another drive died. Got my super convenient radiator on the back of my rack here, which makes it really easy to get at everything. I'm gonna get a little bit older and I'm gonna start really caring a lot more about the ergonomics in my setups. This procedure is gonna vary a bit depending on what NAS software you're running, but fundamentally the idea is the same. You've got a disk that is no longer present. You can see it's showing not installed. And we wanna take a combination of the data that's on our other disks and the data that's on our parity disks and use that to rebuild the data that used to be on it. I have never actually done this in Unraid before. Yeah, that was a really fun way to say it's a pain in the ass to do this in Unraid. I've just happened to be wearing my TrueNAS scale shirt today. It's very easy to do in TrueNAS. You don't even have to shut down the entire array like I'm about to do. TrueNAS is way better in a lot of ways, but the way that it's not way better is the ability to just willy-nilly, oh, I'll throw one drive in, or I'll throw another drive in. Oh, by the way, they don't match at all. With that said, there are plugins for TrueNAS that allow you to get kind of similar functionality. For personal use, I still think Unraid is pretty strong. We just pick whichever one we want. Yeah, so this is going to replace our broken drive. How spicy do you wanna get today? Because if I do that. This is my secondary copy of this data. Oh, so then why don't we just full send? Well, because then I will only have one copy of my data. Besides, you can't full send, you can't do one drive and both parodies at once anyway. Well, a full send in this situation would be one parody and one drive. Yeah. I don't know if it'll let us do that though. I don't really wanna find out. Let's not find out. Okay, disk in parity slot is not biggest. Wait, you can't even do this? That's stupid. What? I should be allowed to at least temporarily do this. If you're adding a new disk or replacing a disabled disk, try Parity Swap. Why wouldn't it just be like, no, you only get eight terabytes, like ZFS? I could have sworn it did. Parity Drive always has to be the largest drive or the same size as the largest. Okay, so we have to do the Parity Drives first. Can we just wipe this? Oh, no, no, no. Just copy it all. No, then I'll only have one copy of the data. That is not 321, Jake. This isn't 321. This is my precious memories. Oh my gosh. Put them on Google Drive or something. No. Why don't you use cloud storage? Well, a number of reasons. One is any of the data that is precious to me, I don't necessarily think should be on a cloud server somewhere, like pictures of my kids. Number two. You could encrypt it. Is any of the data that isn't particularly precious, like Linux ISOs, for example. What, I'm gonna pay a monthly fee to have that stored in the cloud somewhere? No, I'm not. You could just upload just the precious stuff. Yeah, but then it's also super inconvenient to have to go and get that stuff if I get it again. I can just have two copies of it. That's what you're supposed to do. You make that face, but that's what you are supposed to do. Well, the second copy could be online. Yeah, but it isn't. It's here. Okay, anyways. Replacing a parity drive is a lot more annoying, and I'm kind of worried because this array drive. Can I just like, what happens if I remove this? Can I start it with only one parity drive? Disable the missing disk and bring the array online. Install a replacement disk as soon as possible. But what is that gonna do? That doesn't help us. Unless I can use this now as the array disk. Yes, you probably could. Okay, so that's what we should do then. Let's try this. Hopefully this doesn't break anything. I mean, if it does, it is the second copy. YOLO! Don't you love it when we do a guide and we're figuring it out as we go? Well, when you replace a parity drive normally, you reset the config, which is like which slot each disk is assigned to. You add it, and then it formats it and does everything. Right. But because we're missing that disk, I was not sure about doing that. I'm less sure about this even, but this is the route we took. There's no going back now. Parity two, return to normal operation. I don't think so, homie. It's not there. Just leave it, buddy, just leave it. It's gonna be okay. It's dismissed now. You are dismissed. So about TrueNAS. Hey, it's fine. Okay, so now we shut it off again. This was surprisingly not that difficult, and it's not forcing us to recalculate the parity, which makes sense, because in theory, those two parity drives are exactly the same. Which is a cool thing about Unrate as well, because in a regular RAID array, your parity drives are not containing the same data as each other, necessarily. We chuck that in there. What happens if I do this? Can I do these both at the same time? No way! This is actually kinda okay. Very. You can't do that. It wasn't very obvious to the user, but if this works, then that's great. Yeah, okay. What I'm concerned about is if I do this, when I go to add the other 22, replace this one, is it gonna freak out? No, it shouldn't. Okay, so let's. Okay. I mean, theories. Ah, ah! This is a spicy salami. I'm really glad that we are experimenting on my secondary data right now. I will say, moving to a 22 terabyte drive is gonna kinda suck, because now when you go to rebuild your parity, it's rebuilding 22 terabytes of parity instead of eight. But at least it's not a shingled drive. Hey look, parity two is being reconstructed. Just give it some time. You're so impatient. I just. Why are you even worried? It's the second copy. Yeah, well, you know, if you lose your second copy, then you only have one copy. I only have one copy. No problem. One day and eight hours, total size. Yeah, because now it has to do 22 terabytes. So does that mean that that other parity drive, once this is done, is just gonna be doing nothing? How does that work? I have no idea. Because if it's thinking that the parity now is 22 terabytes, this is very. No, that's fine, because parity drives can be whatever you want. So you have this parity drive is as big or larger than all of the array disks. Right, for now. So I wonder what, I guess once, if we were to try to add a 22 after this, it would complain. Correct. Unless you remove the eight. Unless I also upgrade the other parity drive. At the same time. Then I can do whatever I want. Which you should be able to do. Oh yeah, right, yeah. So the next operation, you swap the eight to a 22, you add another 22. Or don't, even, I guess. Yeah. That's funny, we are using the parity drive to replace the other one. Oh, hey, are you out? She's out. Wow, these, oh, they're those quality rails that are all plastic. Why does this have to go help Alex for a bit? So in the meantime, I'm gonna take his old server and put it into the new chassis without him even knowing. Fun fact, this is actually like the original Whonnock server. Like look at this, accessory box. See that? Whonnock server. And even cooler, there's a Windows license sticker stuck to this. Totally normal, LTT screwdriver. I remember this computer. This thing is such a piece of crap. It doesn't have a graphics card in it, so there's no video output. And because it's a workstation X299 board, there's no onboard graphics. So this system has no video output. Earlier, when it wasn't turning on, I don't know how to see if it's working or not because there's no video output. You press caps lock on the keyboard, but that's all you're getting. It's so dusty too, look at that. That's Linus skin. Somebody on the internet would probably pay money for that. Man, he never even peeled it. It still has the plastic on it, hold on. Oh, brand new underneath. The nicest part of this machine. This Storinator over here is already pretty much ready to go. I think I even cleaned it. Yeah, look at that, it's so clean in here. Doesn't need the boot drives that are in here right now. We're gonna take those out and put the Unraid cache drives in here. Unraid uses a USB stick to boot off of, which is really not great because USB drives are not very durable and they can fail, which causes you a lot of headaches. But that's not my problem, that's Linus's problem. And it makes our life a little bit easier. So let's start moving some stuff over. I think I've chirped him for this in the past, but this honestly is kind of a legitimate strat. You take all the accessories for your NAS or your computer and you put them in a baggie and just tuck them in there. I don't need them, so I think I'm just gonna stop touching things I don't need. Tastes like five foot one. These are the cache SSDs from Linus's Unraid pool. And we do need these because they store the virtual machine disks and those are important. You know, I'm grateful you only put one screw in all these drives. I know, right? It's gonna make this a lot easier. I dropped a screw. I don't think they're magnetic. They're not really magnetic, but the LTT screwdriver is so magnetic that even the not really magnetic screws still stick to it. I'm gonna make a slight modification to this here server. By default, there's two separate brackets that hold the SSDs at the back. And this is really annoying because you can't take the SSDs out without unscrewing four screws at the back that hold these two brackets and then all eight screws on the SSDs. What you can do is forget about that piece and just screw the SSDs into the top because that's already plenty. And then if you need to swap them out, you just take those two screws out and it comes out. Hey, look at that, beautiful. We got SSD and other more different SSD. Now, if you ever wants to move these SSDs or replace them, you just take out those two screws and the SSD comes right out. Last but not least, the hard drives. It doesn't have anywhere near the 60 drives that this chassis can hold, so I'm gonna space them out just to make the air flow a little bit better. It actually does make a pretty substantial difference to the drive temps. There's dust on the drives, too. I'm gonna plug in that IPMI, which is something this server has that the other one did not. That allows you to fully access this machine like you have a monitor, keyboard, and mouse plugged in anywhere on the network, which is awesome. But then we need, wait, why is there water all over the floor? Is that supposed to be like that? Ah, it sounds like not our problem. One of the kind of annoying things about booting from a USB is sometimes certain systems will just be like, that's not a boot drive, that's a USB. And then you have to manually configure it to boot to that as the first option. So I don't know, it might be turning on. Oh my God, Linus. How many things could you possibly tangle around this? What are the odds of just boot? Oh my God, look at that. It's booting. Slow clap. Do you love Linux or what? Just no dicking around Windows update, blah, blah, blah, driver, blah, blah, blah, blah, blah. Just load all the drivers, it'll be fine. Exactly, though. The funny thing is we were just memeing that old server has a Windows 8.1 Pro, like OEM license stuck to the side of it. Oh yeah, yeah. Basically Windows server, right? Yeah. Yeah. Pretty much. Yeah. But wow. That's what I ran on it. Yeah, exactly. As our file server. Yeah, exactly. How old that chassis is. I actually had forgotten that we ever used Windows. Windows based SMB server, not actually that stupid, not genius. Look, you said it, not me. Are you ever gonna use this second network port? Nope. I'll just add it to the bond anyways. What's the point of me even answering questions when you ask them? Well, you know, I just try to ignore you. You know what they say, happy Linus, happy life. Happy wife's boyfriend, happy life. Oh my God. Happy. Happy boyfriend's wife. Boyfriend's wife. This server is finally being put to sleep. You know what? It's earned its sleep. Did it go from 1X slash office stuff right to your house? Yeah. Because I think this was the security server, wasn't it, for a little bit? Yeah, I think it was decommissioned for maybe a few months and then I went, oh, hey, nobody's using this and Jake hates these things, so we're never gonna use it again. I took it home and it's been working perfectly since then. Speaking of operating flawlessly, I plugged into the 10 gig NIC that was already in there. Did this one have 10 gig? It must have. So in theory, my downgrade is complete. Graded my storage, but my processor is actually slower. It's substantially slower. This is like a Xeon V4 eight core. I had a really weird chip in here, the 9990 XE. Intel only ever made a handful of them. It was 14 cores at five gigahertz on a weird workstation board, but there's real benefits from moving to server hardware, like the fact that I can remote in with IPMI, even when the system's powered off, so I can power cycle it, or if I'm away and I get a notification that the power's gone off at my house from my UPS, I can remote it and turn off my server, that kind of thing. Yeah, this one didn't even have a video output. Like I was saying earlier, there was no way to see what it was doing. I took out my one GPU because I put it in the TrueNAS machine so that I could try and use GPU acceleration and Plex, and then it turned out that GPU was defective anyway. Hey, look at that, the parity sync is running. So it's rebuilding this disk and then building the parity on this disk. So when this is done, I guess you shut down the array, remove this disk, start it again so it thinks it's not there anymore, stop it, add the 20, you get the deal, and it will still be slow. Slow, like this segue to our sponsor. Keeper, one of the most important steps to take with personal cybersecurity is proper password safety. The majority of security breaches are the result of human error and weak or stolen passwords are big contributors. Keeper security lets you manage and track all of your passwords and private info. The platform autofills your usernames, your passwords, and your 2FA codes on both mobile and desktop on any device. All you need to remember is your strong master password to access Keeper. Since they use a zero trust and zero knowledge security model, only you have access and control over who can see your passwords. And Keeper is hooking our audience up with a huge 50% off both family and personal plans with code LTT50. Plus, there's an exclusive 30-day free trial and feel free to use that code LTT50 after if you wanna keep using Keeper. Keep your data locked up by clicking the link below. You guys enjoy this kind of server maintenance? Boy, do we ever have some video suggestions for you. I don't think I can suggest just one. You know what? Why don't we do the vlog where we cleaned out the server room with compressed air?
Info
Channel: Linus Tech Tips
Views: 1,698,291
Rating: undefined out of 5
Keywords: unraid, server, backup, server room, array rebuild, Linus' house, home lab, nas, storage server, data protection, data hording, truenas, home server
Id: geb_gk1XXyA
Channel Id: undefined
Length: 20min 7sec (1207 seconds)
Published: Sat Aug 05 2023
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.