Well hello, YouTube Um..we have a bit of a problem today on this section of the moving, but we've been moved in for months moving vlog um Something is going on with the server and we haven't been able to work on a project for more than [like] 30 minutes at a time before the server crashes and Our computers all freeze and then we're all sad because we can't work on them. Most of the **** we do is on the server well I do it I'm actually doing some writing right now because I can do that. Can't you'd see I'd love to be working - anything that doesn't [see] the server We can do now well that's not much, but it always seems to go down Yeah, but the worst when there's nothing else to do you're kidding, but you know yolo, you only Liao once Let's see let's see the stress. That linus is going through It's probably gonna crash like while you're doing this How do we fix servers? How do you fix it? Tell Linus, he will fix it That's your go-to. Yeah Whether you're running a small [business] or a 1 billion dollar even multi-billion Dollar Enterprise Rackspace has your dedicated storage environment covered check out the link in the video [description] to learn more [alright], so here's the situation, Whonnock server over the course of the last couple of days has been spontaneously going offline and turning off in spite of my best efforts to turn it back on and Complete the backup that [I'm] in the middle of trying to do to our new unraid vault while I was standing in front of it, I observed One of my raid controllers giving up the ghost, so as a reminder This server is running 3 raid 5s striped together in Windows of a total of 24 SSDs so if one of the raid fives drops out entirely [all] The Data is gone only about 10% of the backup that I was just in the middle of remaking is saved So it's time to investigate All right, so here's Whonnock server so far troubleshooting steps that I've tried... you never wanna you never wanna rebuild something like this if there's a possibility you're going to have to go to a data recovery service because that Can make it much much worse so in terms of [troubleshooting] steps I tried transplanting it into another case so that [I] could use a different [Sata] backplane, that didn't work. I tried a different power supply a better power supply That didn't work none of that fixed it And I even went as far as to put in another LSI raid card that I had and try to import the array But while it did detect my drives all of them as unconfigured good and a foreign array, It was not able to import it So there is some good news there though while I'm getting firmware errors kicked out by this card right here when I try to boot All the drives are detected by another card, so hopefully it didn't write too much garbled Data to the array Okay, so this is basically what I've been doing for [the] last 14 hours is sending emails contacting different Data recovery services There's some that are local there's some that are not local But I'm not ruling those ones out in fact Right now the most likely solution looks like werecoverdata.com, they've got some Some custom tools that they've created that will allow them to SSH into the system and actually potentially recover the raid the missing raid 5 and then the other [two] raid fives and Assemble them all together and export them as one gigantic Pile of Data [too] to one of our servers, all without me actually having to send the drives away Which would be pretty ideal Now obviously an approach like that would not work in the case of physical damage to the drives But because it is only a raid controller issue. They're saying hey. There's a shot at this, so let's give it a try So I downloaded their remote recovery client thing here So right now. I'm waiting for their custom Linux based tool to load on a bootable external SSD. So that they can SSH into the server I'm not having a good couple days... Right now I can't get the utility to detect my... my USB drive Called Tech Support! Oh bloody hell... [uh], I just figured it out. This is linus calling by the way [I] just realized I have to run the WRD disk image as administrator You guys should put that in the documentation Never fails, this is why it's a good idea to call Tech support Because the second you call them You'll solve the problem on your own Alright, so in theory. We now have a bootable drive! Okay, so step one is getting this Unnecessary extraneous server out of the way, I'm grabbing a UPS to use now in a perfect world I Would have a motherboard or an HPA card a host bus adapter card that could allow me to plug in all 24 [of] the drives for them to do their data recovery Magic on but what we're going to have to settle for is having eight of them plugged into the motherboard [at] a time as Each raid five is rebuilt. Raid cards do all kinds of funny funny nonsense And they don't give other software direct access to the drives at a bit level in some cases So we need to remove all these drives and plug them in directly to the motherboard Okay Now it should boot to that in theory Let's just make sure all the [drives] are detected in the bios and everything oh That's a problem. I Need to pull up the other raid controllers as well otherwise They're going to be looking for their drives, and I don't have them powered up right now Okay [oh] actually that doesn't look very good pcieport errors What? So let's make [sure] all of our drives are even showing up. um.. uh oh... That is extremely worrying. [oh] that is extremely worrying... [oh] That's not good That means potentially three of the drives in a raid five are dead which means this data is not coming back... None of it If we lost three drives from one of the raid fives So let's try moving things around I've never tested the SATA Ports on this board It's possible three of them are dead, however extraordinarily unlikely that would be Five six, okay, we've got six now Okay, that gives some hope I'm just moving Sata ports around. I was just gonna say right now I'm even having difficulty getting this system to post, so I'm I'm at a whole new [level] [of] concern Yeah, I'll let you [know] I'll send an email Okay So I'm at the point now where I don't want to put the drives on this system Let's go with this one. I didn't like all those pcie Express errors that the bootable USB was kicking out I don't like that some of these drives aren't showing up. I'm starting to wonder if that's what corrupted the LSI card [will] pain in the ass so thousand things I can think of that. I would rather be doing right now Okay, I got seven right now really yeah, wow maybe it was bad PCI you line is speaking well, I can tell you what's happening so far is I put it on the system [that] I was running the raid card and the drives in and that baby started spitting out pcie errors all over the place, so I got everything pulled out of that system and I have plugged all the drives now into a different test bench And I'm trying to see [cuz] I wasn't even getting all the drive showing up two three four five six seven eight okay Hey, you guys are on the phone for the magic moment where all eight drives are detected by a new motherboard So I'm beginning to think there may be a motherboard issue that caused the firmware on that LSI controller to corrupt in the first place So I'm actually pressing enter and I am booting up to your usB and it is loading correctly on this I think that board is dead um **** Okay, well that's I don't know if that makes it worse Probably does doesn't it you can tell me I can take it that says waiting up to 60 more seconds for Network configuration Good timing with the follow up call If [you'd] called one minute earlier, I've been cursing you for dialing my number So I have to answer my phone because I was in the middle of thinking only five of my drives were being detected You guys know what you're doing right? [I] had a local shop that I was going to physically physically take it to and They basically they were like they kind of freaked out when I said [I] was having someone use their remote tool They were like well unless these guys really know what they're doing. That's pretty dangerous, [and] I was like okay Well, I'll check with them. I'll make sure they know what they're doing so so you guys know what you're doing, [so] [basically] they're explaining that they're going to virtually mount the volumes so we're not doing any writing to my disks [so] even in the event that we cannot recover it by this means that still leaves other options open to us okay, there is Okay, well that's that's very wonderful to hear but I'm trying not to get too excited right now. They sound pretty confident This is like the ultimate Roller coaster right now, so to put this in the appropriate context for the viewers right now Whonnock Server has several [in] [progress] projects Stored on it that do not have a backup anywhere else including a fully shot Channel Super Fun that involved a thousand dollars in equipment rental Including lots of footage from Linus Tech Tips videos like it is going to be a huge huge Devastating loss for us to lose this I mean It's got a lot of art our templates for editing editing videos For for scripting templates for employee reviews like all kinds of stuff like this is our main working server Know the off-site backup server has not ever been built yet the vault was the backup server If I ever have to like act in a in a video where I have to look for Lauren I'm going to think back to this moment Okay, so I do need to reboot it once in order to add that [8] terabyte drive [so] anyway right, okay? Thank you very much. Gentlemen. I hope to speak too with good news very soon. Ah ah ah Twelve SATA Connectors ah ah ah Okay, so let's make [sure] everything is showing up one two three four five six seven eight of those our boot Device and our eight Terabyte recovery Media. We are ready to rock Okay, so within about ten minutes. They contact me back. They said okay. Yeah, we're ready to have all the drives in there, so That means we are Gonna have to do some Janky-ass **** right now So basically what you're looking at [here] is These drives are going to be powered off this power supply With a jumper these cards and all these drives are going to be powered off of this rig So now we can power all those drives without powering up this motherboard or any of that nonsense, [okay]. So we have a grand total of twenty five drives connected to this test bench [sixteen] of which are running in bay's of this enclosure and eight of which are spread out here So there we go. There's one of my LSI controllers. There's it's eight drives There's my other one, whoa Where are your virtual drives? I think I remember a troubleshooting step where I tried Swapping all the drives to different bays where I was trying to see if it was the back plane or the cables Wow, panic moment there But it's okay So now we should see two virtual drives here On the screen that's one set Show me the second set Thought we lost another raid5 there, okay... It's good feeling good Reality TV ain't got nothing on IT work Show me all the devices! Two raid adapters one eight Terabyte drive one two three four five six seven eight Kingston drives boom Okay, I will let the data recovery specialists know All right, so all that's left now, then I think I got in touch with them And I asked if we can put a 10 gigabit network card in so we can accelerate the offloading of all of our files so while we're waiting [for] now is for them to reboot the machine and set me up with a network share that I can access and Fingers crossed just start pulling off all the footage and while everything else [alright], so check [this] out here [around] [here] So I've got the instructions for how [to] access our data So I go into the tritium Share and There it is Bunch of Eula's a bunch of stuff. We've got What looks like the file structure? Let's test it How about one that I? That we haven't finished releasing yet like something from the the mail a battle over here. Let's just Let's watch it! hmm... Okay, exhale it is for sure uh oh Okay, here. We [can] copy one of these let's copy one of these clips to my local machine and see if that helps Nope, oh boy Let's have a look I wonder if everything's even here like I [don't] even see the Linus Tech Tips folder 1 & 2 are missing Useful applications is empty that should be full of things okay. Well, I guess we'd better get in touch [with] them So thanks for watching subscribe follow all that good [stuff] and see you next time [I] just got an email back from them please stand by addressing your emails with the engineer It's possible the raid definitions were changed from one reboot to the next because of drive letter assignments and the Linux 26 was 26 drive limit we have [I] think we have more than 26 Devices in there, I got another email back - we recreated the block devices Please go ahead and check the videos and content that you are [looking] at again. No way. [oh] The suspense... I'm not enjoying this Nothing there. I'll let him know [the] [shares] empty This is just like the wildest Roller coaster ever I I don't even yeah. I don't even have anything else to say actually it's almost funny. [I] hope to some of you. It's funny Here's the email. I got 17 minutes ago. I was stuck on a call We got everything remounted so you can now go ahead and do the data once more. This is the [moment] do we not have a 2 I don't think we ever had a 2 oh, okay? um Okay, so let's go to that channel super fun video that. We were looking at yes Yes way way justing okay? And it was the delivery? I'm getting like tingling Okay, okay, you got to check the source file. Oh cuz that was what we know was corrupted, okay, okay, okay? Okay, so Untranscoded footage X-Ray They did it it's back. Oh my God It's back. Here's another file its back. Do we have everything? In theory yes, really? Holy, **** Oh I think it's finally over! Woo hu! [that] is fantastic Hey guys, guess what happened with the recovery. What?... We got everything back! Thank goodness Like it's just been such a stressful few weeks, and that was just such a huge Huge thing all of our pitches and proposals and stuff are on that server, [okay]? Whonnock is back Really? Really? Everything is back as far as I can tell I have played back video files They are not corrupted Whonnock is officially back! Yeah! Swegway jousting footage for the Melee Battle should be fine And we'll have a better backup Scheme in the future I promise That is all Yeah, to save video shots. We should physically print all of our videos every frame I don't think so... on paper [you're] the good news? That Whonnock is back? Whonnock is back! Yay for hiring other people to do our jobs! [I] know, I don't think it's back, but it is! It's back What are you doing? I'm just letting everyone know that Taran [it's] back the server's back Okay So it's not always that our sponsors tie so perfectly into our content But today it happened Today's sponsor is rackspace the top tier manage cloud computing company They pride themselves on best-in-class service across all platforms They've got over 300,000 customers in 120 countries with 10 worldwide Data centers. They've got Red Hat, Cisco, Microsoft, VMWare certifications and all of that Amounts to whether you're running a small business or a billion dollar Enterprise Whatever your needs are in terms of capacity storage or flash-based high performance storage they can take care of it without you going [grey] or bald because you did something wrong and Screwed up a backup and went and lost a raid array or whatever else they've got dedicated storage to meet your performance security network capacity and compliance needs everything from direct-attached storage so that gives you the flexibility and scalability of Just a simple [ok] let's attach some drives to this thing. It's redundant cool off you go to the races to [san] So that's high availability and reliability fully redundant for business to even mass with support for demanding workloads like virtualization file sharing and Rich media in fact [their] NAS stuff can scale [up] to 20 petabytes of Capacity they've got [private] public cloud and private cloud options as well And you can get your own server a scaleable private cloud in the data center in your data center in their data center And it's all supported by rackspace and Vmware they call their support Fanatical because these people are available 24 hours a day seven days a week 365 days a year, [and] they've got industry-leading service level agreements both managed and intensive, so if you need dedicated storage they got you covered! Go to rackspace.com com/ltt to learn more and Not go through what I did I Just yeah, I don't really have anything to say So thanks for watching guys if you disliked this video then Hit the dislike button. I guess but come on This is about as real [as] it [gets] around here if you [liked] the video hit that like [button] get subscribed maybe even consider Checking out our [Amazon] affiliate instructions for how to use it whenever you buy stuff there Instructions are up there Buying a cool shirt like this one or even giving us [some] of the contribution through our forum which gives you a little contributor badge Now that you're done doing all that stuff. Hey. Maybe you want to check out that Channel Super Fun video that we just discovered is back and It'll definitely be worth your while. We joust with each other on Swegways. It's pretty awesome, so See you next time
This was sincerely the scariest horror movie I've seen in a while.
Sure, aliens and zombies can be somewhat scary, but it does not compare to the feeling of complete terror of realizing that a whole "The One Server" of data is completely gone.
It's something I hadn't felt in a while, but years ago, while still merely dabbling, when helping out a student org with their stuff, I felt that feeling. I know what that's like.
I'm glad it worked out in the end for him.
And let's remember, he's not a sysadmin, he doesn't claim to be a server expert, he's gaming end-user who likes to play with hardware, who is stubborn enough to also try his hand at server hardware. It's entertaining.
The thing I like best is to see him try his hand at things I'd never do. I'd never run a server at RAID50 with that many disks, but I am interested in what such a hypothetical machine would do. I would never build together a machine with $30K of gaming hardware, to run 7 gamers off of 1 machine, but I do find it fascinating to watch him build it.
Instead of being angry or condescending, be glad that this is (besides entertainment) a kind of PSA to gamers who think that automatically makes them sysadmin-qualified to get (advice from) an expert in as well, to help them do things properly, instead of improvising until something blows up in their face.
Edit: corrected "while" to "whole"
They should get Veeam as a sponsor.
What the fuck. Striping across 3 raid 5's? Whats the point of that?
Why not put one raid card in that server and use RAID10?
I'm reading these comments and I'm rather saddened. Linus is not an IT guy, he does not have a full time IT dept. They are a media company, they work off of YouTube and sponsor money.
I get where you're all coming from, but let's not circle jerk about best practices when we all know that some where we all have some flaw. Or just lets not circle jerk around someone's failure, we could provide great solutions to him if we took 20 minutes to come up with some.
Even watch the first 1 minute, white box server, constantly crashing, no one can work. The problem has been going on for days. Also, his backup is failing (well, sync to a NAS).
He also, from what he seems to say anyhow, deleted his backup (sync to NAS) to start it again... 10% was completed.
The sponsor is rackspace for the video which is hilarious.
EDIT: https://youtu.be/gSrnXgAmK8k?t=1125 "Its been a stressful FEW WEEKS".
Jesus Chris
They really should have given a bigger shout out to the werecoverdata guys. The big add for rackspace made it sound like rackspace did all the work. Horrible set up though, it never surprises me the lengths some people will go to have one big drive/lun to dump stuff on, waste of ssd's.
The way his RAID failed is... odd and unique. Apparently the motherboard went crazy and fucked itself up, and the RAID card along with it? Weird. Bad luck, really... when RAID goes wrong, you better pray it's just a replaceable disk, otherwise you better have a goddamn backup.