Behind me right now is the biggest supercomputer in the country! It will be serving researchers across Canada studying the human genome and bioinformatics, particle physics, materials research even humanity's research! It's called Cedar, It costed the federal government through the Canadian Foundation for Innovation over 16 million dollars, and we get to be the first to unbox this beast! Savage Jerky is created without the use of nitrates or preservatives. Use offer code LTT to save 10% at the link in the video description. So Cedar is a big data machine! It takes up a quarter of the 5000 square foot datacenter it occupies, meaning actually that there's room for it to grow, but right now it has 27,000 Intel Xeon processing cores, 190 terabytes of RAM, 64 petabytes of storage 584 GPUs and a total power draw of 560 thousand watts! Though, with that said, its efficiency is a shocking 1.07 on the PUE scale, where 1 would be perfect and a typical data center would be 1.5 to 2. We'll get into how they did that a little bit later though. So our tour starts right here. Behind me are what they call the "high availability racks". So everything back there has dual power supplies for redundancy with a battery backup for that, and a diesel generator, backing up that. Everything back here is mission-critical. Things like networking, login servers and management servers are all here, and this is also where you'll find the bulk of Cedar's storage. Let's get in for a closer look at Cedar's connection to the outside world. This networking appliance from Huawei has a street price of around a million dollars! Woah! And right here, this is where it gets really bananas. These guys are Cedar's dual 100 gigabit connections through Vancouver, And then as if that wasn't enough, these orange ones here are dual 40 gigabit connections, through nearby Surrey, just in case somebody puts a backhoe through one of these other fiber lines and they would have otherwise lost their Internet connectivity. I mean... that's their backup! Backup! But Ethernet is not really the way you want to connect high-performance computing nodes. This. This right here is the true networking heart of Cedar. These are 48 port omni-path switches, and they're configured in what's called an "island topology". So the island is in almost all cases 32 compute nodes. Each of those compute nodes is connected to 32 ports on one of these switches in its rack. Then, the remaining 16 ports come back to here. That means that every island gets a dedicated line to each of the core switches, giving you failover and massive bandwith. Each one of these fiber links right here is capable of 100 gigabit per second. So even though between islands, we are - let's say - bottlenecked by our 16 connections. That's only half the total theoretical speed within an island, we're still talking about a hundred gigabyte per second. So it's not really an issue. OK, now let's move on to SFU and Compute Canada's version of Petabyte Project. Spoiler alert: theirs is better in every conceivable way! So in the five cabinets behind me, we've got Cedar's 50 petabyte IBM tape library system. They have a 40 gigabit link to the rest of the supercomputer, and each of the five thousand, ten terabyte magnetic tapes inside can be grabbed out of storage, moved with like a robotic arm into a reader and the data can be accessed when needed and this is done automatically. Cool, right? Okay, yeah, but due to the slowness of that swapping process, this is still what we would consider to be cold or archival storage. Next up here is general purpose storage land, where any data that's being used for any current research project would be housed. So here, they're using off-the-shelf 5U racks, each of which contains-let's see if we can crack one open here-a total of two kind of trays here, and 84 8 terabytes of -let's have a look here- enterprise capacity SAS drives from Seagate. But there's actually more to this system than meets the eye. Every 4 of these storage nodes requires 2 nodes of what they're calling "object storage servers". These act as a high-speed cache with their SAS 10,000 RPM drives as well as as kind of like a... ..."a traffic cop" for everything behind it. So every single read or write to these hard drives actually goes through these nodes. So right now, general storage land is 10 petabytes, but in the near to mid future, it will be expanding to 20. Twenty! Now that DIY approach to storage is great for scaling up at a low cost. But when it comes to performance, they went for this Data Direct Network storage appliance, because it has got the real goods. Now in the rack next to this brain, you'll find a mere four petabytes of actual storage due to its higher cost. But thanks to its proprietary hardware, custom software, and solid-state burst buffers, this thing can handle up to 40 gigabytes per second of sustained throughput, making it perfect for data intensive applications that rely on humongous data sets. Now let's get into compute. There are about half a dozen different types of compute nodes all connected to the same high speed on these half network backbone that are optimized for different types of research. We'll begin with the base compute node. A very whopping 576 of these each of these is a computer, so there's actually four in a single 2U shell each of which contains: 2 Xeon E5 2683 16 core processors, 128 gigs of RAM and about a Terabyte of raid 0 SSD storage for scratch. So, each rack here contains two islands, so that's a total of 64 compute node giving us a whopping 2048 compute units per rack. So these nodes are the basic workhorse of Cedar, handling everything, from Monte Carlo simulations for material science, to simulating dynamic processes in nature with a high degree of randomness, like snowfall or rainfall. They would also be used in any highly parallelized workload because, if you need - you know - 10,000 CPU cores for one job, there aren't enough cores in any other class of servers to handle that kind of load. Moving right on up. We've got the big memory nodes. There are 48 of these and half of them are just like the basic node except with 512 gigs of Ram, while the other half of them, these puppies, have one and a half terabytes of system memory. These ones take up twice as much rack space though, each of these one use is a single dual socket system. Because... you know what? There just wasn't enough gosh darned room for all 24 64 gig memory modules that are required for that much RAM. First world problem? Yes. These guys are really special. These are the aptly named "three terabyte nodes" There are only a handful of them, but these are quad socket machines with Xeon 4809-v4. Four of them! But wait a tick! Those are only 8 core processors! These don't even have more processing cores, then those little tiny ones that take half a U. What's the deal here? Well, it turns out that some vital informatics workloads, like genome sequencing, don't actually scale very well with more processors. They just need massive amounts of memory to hold the datasets that they need to work on. So, while the team here probably isn't super stoved on using up 4Us, just so they can stuff more memory into the system, until Intel Optane reaches a higher level of maturity, this is the only choice they have. Now finally we're getting to my favorite nodes, the most expensive nodes. These are the GPU nodes... and while they're actually quite similar to the base nodes, with respect to their CPU and RAM configurations, what's got the researchers in the fields of molecular dynamics, AI and machine learning all amped up about these, are the quad Nvidia Tesla P100 graphics cards that they have crammed into each one. I mean, seriously! With 1500 watts of power being consumed by each one of these is it an engineering marvel that they've crammed enough power and cooling to make this whole thing work. So, actually, now that you think about it. How exactly did they do that? So the key knight among you might have already caught a couple of hints earlier in this video But the secret lies in the rear doors on the server racks. Look how thick this is. Yes, my friends this entire door is a gigantic heat exchanger, so their servers don't actually have water blocks. That would be more expensive. What they're doing is they've just got the front of the rack all sealed up, so there's no back draft pressure And they've got normal air-cooled servers, that pass the air from the front, where they just draw in room temperature air in here and it comes out hot like 30-plus degrees and push it through the heat exchanger, where it is actually cool to my skin. That's how efficient these are. And that cooling system is massively expandable, too. You can actually see, above me I am standing where we got a blue and green cooling pipe connected to a whole bunch of quick-release fittings ready to add more racks right here. But to see what they actually do with the heat. We're actually going to have to go upstairs... ...where we'll find the final and perhaps the "coolest" stuff in our tour here. This is the mechanical room, where the pumps and these freakin' pipes take all the water from downstairs and dump it into 3 cooling towers outside the building. Now, right now the weather is favorable to cooling, the ambient temperature is quite low, so it's just operating as gigantic radiators. But get this: when the conditions become less favorable in the summer, they kick things into high gear with an automated system that sprays water onto the fins of the radiators in the cooling towers, and if you watched our bong cooling video, which you can check out right here, you'll be familiar with this concept already. But this is called evaporative cooling and, by these means, even in ambient temperatures up to 30 degrees Celsius, they can achieve the 17 degree coolant levels that they need to, without employing the massive chiller unit that they have over on the other side of the room. Squarespace is the way to build a website, whether it's for your small business or for your, you know, local freaking book club. It doesn't matter! If you want a web presence affordably and quickly Squarespace gets it done for you. It starts at just 12 bucks a month and you get a free domain if you buy Squarespace for the year, you just pick one of their templates and... Boom! you upload some pictures, you fill in some text. It's all cloud-based and your website will ...simple as that... look great on any device. Every website comes with a free online store and their cover pages feature allows you to set up a beautiful one-page online presence in just minutes. So start a trial, with no credit card required and start building your website today. Then, when you decide to sign up for Squarespace, don't forget, head over to Squarespace.com/LTT and use offer code LTT to get 10% off your first purchase. So, a massive thank you to SFU and Compute Canada for allowing us to run a mock in their data center. Thanks to you guys for watching. If you disliked this video you know what to do, but if you liked it, hit that like button, get subscribed, maybe consider checking out where to buy the stuff we feature at the link in the video description. Also, down there you'll find a link to our merch store, which has cool shirts like this one and our community forum, which you should totally join!
Incredible.
Wow, he didn't break anything.
That is beautiful.
One wonders why they didn't publish this on canada day
He needs to not touch/flick the 40 and 100 gbe fiber lines..
Yes I know, they are shocking durable. I've run plenty of fiber myself in server rooms, don't touch it unless you are moving it.
This was a great video. God that hardware is amazing.
Awesome presentation, Micheal Bolton! Loved it.
The 100Gbe stole the show for me. That is amazing.
560,000 watts of power!?