Petabyte Project is FULL! Time to upgrade...

back when we installed a petabyte worth of hard drives in our server closet we were sure that with that much storage we'd be good for a long time and in fairness I guess two years worth of red footage is pretty good but it finally happened we are critically low on space I've got less than 5% available on our main editing server but with only 20 terabytes available on the vault there is nowhere to dump it too fortunately I've got a band-aid solution the very best kind of solution Seagate actually sent over these 15 12 terabyte iron wolf Pro drives for a totally unrelated project that well didn't actually go very well so instead of using them for that we're gonna use them to add more capacity to the vault speaking of the vault I keep all my segues in a vault reg wallet is the sleek way to keep wallet bulge down with its compact frame and RFID blocking inner plates use the offer code LTT September to save 10% and get free worldwide shipping [Music] all right so while I wait for Anthony to come down I'm gonna run through how this whole thing is gonna work with some really crude diagrams here our petabyte cluster uses a file system it's open source and it's called cluster FS that's what allows these two independent servers here to present to the rest of the network as a single large share now we could increase our capacity by adding more server boxes but that's a project for another day literally that's a project for another day we've got a video coming with a full petabyte of storage in a single server rather than two so make sure you're subscribed so you don't miss that today we're gonna take the NC 15 base in server Delta 2 down here and we're gonna expand our storage with another 180 terabytes of raw capacity we don't get to use all of this capacity though our Gluster FS implementation is geared towards raw capacity rather than redundancy so the only failsafe built into our local network here is the rain z2v deaths in each of our machines what that means is that out of our 15 drives only 13 of them count towards our capacity with the rest these two taken up by parity data that protects us from data loss in the event of a physical drive failure or a cable failure speaking of which how about replace that bad cable and Delta 1 oh that's gonna be quite a project guys the vault is going offline for probably about 2 hours p.m. just vault so Wanek is up and these servers are so heavy so we're gonna have to empty all the drives out of them onto one of these carts then take the server out put it on the cart and wheel it over to the island where we can work on it I'm still super proud of this cabinet door holding open innovation we drill the hole on the side of the keyboard tray and now we can get the servers out super easily you know you can just take the door off right no I let the door has a filter like that's the purpose of the door for us anyway yes I know I could take the door off I mean with the sides off the backs of off apparently we figured that out thank you YouTube comments it's quick to label a drive it's slow to label 60 drives and this actually has to be legible that's important why don't you actually work your way from the front and I'll work my way from the back okay and we'll do this like you know lady in the eating the spaghetti style that's a strange mental image but okay speaking of terrible segues sure is cold in this server room good thing I'm wearing my LTT swagat it's a sweater it's a jacket LTTE strong calm don't worry about it I don't think most people feel bad for me struggling to stack all my hard drives that are very time consuming to stack why can't I stack all these hard drives if you could grab this end though and help me up onto the cart that would be swell whoa oh it sucked on my fly okay I got it that's fine one thing we want to do as recording this over to the kitchen there is be really really gentle with the way that we're moving this this is about what 350 terabytes of our company's valuable data on here right now and I remember Patrick from served the home telling me that Yahoo had an incident where they moved their data center like across the parking lot rolling hard drives on carts not unlike this one and all the vibration killed like a significant portion of their drives so we could lose up to eight of them depending on oh seven because one of them's already degraded but but we don't want to do that so I guess this is the part of the video where we explain what happens when a cable fails in a store inator now most bulk storage servers with like lots and lots of hard drives use what's called a backplane so they'll take fewer connections off of your SATA or your SAS adapter or your raid card or whatever the case may be and then they will take those and they will split that bandwidth across multiple drives 45 drives takes a bit of a different approach so they wired every single Drive up individually across lots of back plane and more of an under plane the advantage is you get the full bandwidth another advantage is that in the event that you a connection fails you're not replacing an entire costly back plane but the disadvantage is that if a cable fails you are digging this entire apparatus out to replace one flaky cable so there were two drives that were dead one of them it was finally replaced it was it came back up everything rebuilt and everything was rosy but then there was the other one we replaced it a couple times we actually replaced the controller cards itself and tried different ports on different controllers that we knew worked but it still had the same issue and what's weird is sometimes it would kind of work and we could start rebuilding the data on it but like what kind of data speeds were we getting it would start out at like you know 300 400 megabytes per second which is kind of low but fine but then I would go down to like 10 yeah and the ETA was like a year yeah come on by the way evidence that my dust filter works just great yeah it looks brand-new there is one little bit here that I noticed but that's it okay so let's not hate on my filtered front cabinet door there okay okay so this guy needs to come out now this is a little tricky does that entire plane need to come out I hope not yeah cuz like I'm looking at this and this needs to go up under there in order to get screwed in there and in order to do that this like straight sucks this is exactly why I haven't had time to do it until now how long did we tell the editors this was gonna be down for two hours one thing I did notice though is that with how tightly integrated it is into the bottom of the case I like I don't know how I can't really get it very well okay they're shooting tech link now so we're gonna have to do a SMR server upgrade so I pulled these out and I can see where the cable goes so yes I will in fact have to pull out this and this I don't see anything obviously defective about it that's an angry episode of tech late someone must have removed some headphone jacks I hope it wasn't Samsung okay so this is it we begin the funeral procession again and this way it's an open casket whew oh yeah should we close this server maybe like now is as good a time as any to do that if we're gonna do full Yolo and assume that we have everything right then yes otherwise no otherwise no let's go no let's put some hard drives in yeah having to keep track of everything sucks so there just got a jellyfish so that drive with the rubbed off label that I wasn't 100% sure about I got to the end and the only thing left is 131 which is clearly not that unless I have a wicked case of dyslexia and I put 131 over here so I think it's time for a sanity check here we go ladies and gentlemen okay Anthony you got the drives do you want me to turn it on first should we do it yep is the hot spot on all right well hot pocket well hot swap I'm turning it on okay here we go Anthony you want to take the wheel here sure let's see a zpool status okay so guys you can actually see here what was going on with one of our raids e2s so each of the 15 drives is a raid z2v dent so this raid z2 reads to zero is online you can see the whole zpool is degraded though that's because raid z2 one here drive 117 the one that we just replaced the cable for is unavailable and then these are all the previous attempts at rebuilding it with different drives now we're gonna try again but with a new cable so we fixed 117 but now we've got four drives offline Oh balls that's like way down there you want to let me know if anything changed five six seven and eight disappeared they're unavailable so that's the wrong yep damn it okay so we're back and all the drives are here but they're not showing up with their like there are 45 drives tornado friendly IDs here and also five of them are riesling that seems bad how do you resilvered five drives these are Riesling because they got cut without being offline first in the meantime should we have the other 15 grads - Delta - what could go wrong what could go wrong so do we need to make a brick cuz that's blusters crap no we don't need to do that with buster that's all in number of /z pol pretty sure okay I don't see anything here that looks like just making a video yeah me neither okay so we just do zpool create zpool raid z2 and then the paths to the disks but if our disks don't show up okay hold on so let's do this and it is online so you might need to restart okay so maybe they're not how topical it should be but maybe not configured the server tonight finally yeah they're special driver might actually not do that okay well let's see in the meantime we can check in on Delta one and see if it's free silvering a little faster now it is not so now what we've replaced literally everything the drive the cable the controller I mean do we want to just pop the drive out and pop it in one more time and see what happens we can try it okay brand-new drive okay so Delta 2 is rebooted now and we've got 1.1 okay so I guess throw them all in yeah and see if we get them all and if so will create the raids u2 then at least it may be degraded but it's bigger so all fifteen drives for the expansion are in Delta 2 now I switch back over to Delta 1 and I have good news our reef silvering is going at 1.91 gigabytes a second which is pretty sweet that means it should only take a few days so 117 it's not there is it it's there ok but but but I try to replace it new device is a different optimal sector size what the crap so I need to figure out what that is well hold on hold on that's not like a 4k sector drive is it it might be that look probably I doubt I don't know this for a fact when I suspect mixing four K sector drives with five twelve sector drives in some kind of an array is probably super terrible so if you wanted a software it's offline it's already up one yeah okay did we accidentally buy the wrong drives Anthony oh my god that couldn't be like the problem could it they're not are these advanced format yeah yeah wait you have got to be kidding me is that why these replace operations haven't been working possibly you know what we can check the original video unboxing the petabyte project it's a good thing our entire life exists on YouTube at least mine does okay where's some b-roll of a hard drive here oh no they are advanced format so they're all there but they're not there there we go new raid-z to rate c-23 includes 15 drives all online 117 is now back online resell burring and it's doing it at almost 500 megabytes per second so we're good so I did notice curiously that the vault is still not any bigger right I believe the glossary best service either needs to be restarted or I might need to adjust the config real quick okay so let me just quickly look at this in Delta wait is it one of these disk doc config is it this docked in figure there it is whew hey I helped did I help on a Linux thing yes or no I did usually I'm not nearly as helpful when it comes to server stuff okay so final update guys it was just a matter of getting volume fours brick integrated into Gluster so you can see everything's here volume 1 2 3 1 2 3 4 for sure whatever doesn't matter no big deal so now we're gonna fire over to our other server this is Wanek server it runs Windows whatever want to fight about it we've got our Z Drive which is actually even tighter for storage than it was before we have less than a terabyte of space left but that's no problem the vault a hundred and sixty three terabytes ready to frickin go and as for Delta 1 it is riesling at 375 megabytes a second I can push this back in very ever so slowly and gently and we're done bud by the way if you guys liked this video we actually did a video rebuilding our water-cooled render server down here you can check that out up there and if you're not into that you can check out our sponsor for today's video I fix it the I fix an essential electronics toolkit is compact so it can go anywhere you can and help you fix almost anything it includes their most popular precision bits and they're held in place with high density foam so you can throw it around without any of the bits falling out and of course it comes with I fix its lifetime warranty it's just $24.99 at ifixit.com forward slash Linus so go check it out today that's it for this video I will see you guys next time when we're installing the single box petabyte project

Channel: Linus Tech Tips

Views: 2,836,848

Rating: 4.9373941 out of 5

Keywords: petabyte, storage, server, hard drives, raid, zfs, linux, storinator, 45drives, upgrade, repair, vlog, ironwolf, seagate, hotplug, nas, archive, vault, maintenance

Id: DsZtTpBk7s0

Length: 15min 44sec (944 seconds)

Published: Thu Sep 05 2019

Reddit Comments

[u/[deleted]]

I wish a company would send me a box full of drives lol

[u/GODDZILLA24]

I don't understand why I keep seeing comments questioning why they store all their footage. Have you forgotten which subreddit you are on? It's literally in the name.

[u/ktnr74]

Does anyone else want to comment on how this video highlights all the design flaws of the Storinator chassis?

[u/EposVox]

So many people questioning the practicality without realizing the obvious "fun, boundary-pushing projects is good content" aspect, on top of the "because they can" on top of.... did you see what subreddit you're on?!

[u/Kiwi_birds]

I literally watched a video about them backing up everything to gdrive yesterday and wondered how long it would take them to hit 1pb...

​

Found out it's today.

[u/milennium972]

Datahoader complaining about data hoarding.

[u/Genesis2nd]

Linus mentions an upcoming project with a petabyte in a single server. Storinators goes to 60 drives. 60 16TB drives adds up to 960TB. Is he rounding up or are there cases with higher drive count?

Also, I get that making tech-related videos are their thing, but after the Whonnock server going tits up a few years ago, you'd think they bring in someone experienced, when it comes to messing with their own data.

[u/adobeamd]

I know they probably get the storinator for free but man a 90 bay supermicro chassis would be such a better option. More reliable, better build quality, can expand without having another computer.

[u/[deleted]]

I love all these bitter comments from folks with dozens or hundreds of TB in their flair. There is always a bigger fish. Get over it.