Meet the new SBC Linux Cluster King!

this is the Turning Pi 2 board and it has four of the world's fastest arm sbcs these new rk1 boards this board is the new SBC cluster King it has 32 CPU cores 24 tops of AI compute 128 gigs of RAM and I put 4 terab of fast nvme storage on the bottom this thing is so fast it even beats the pants off this six node Raspberry Pi cluster board this video is sponsored by Squarespace and I'll tell you more on them later but I already talked about the motherboard here the turp 2 back in December 2021 basically this is a mini ITX cluster board that holds up to four computers each of these slots has power IO and networking so you can build your own tiny little kubernetes cluster or use it for experimenting or storage or whatever what this doesn't do is give you one super fast computer you still get four individual computers just conveniently put together in the same space as a normal single PC but back in 2021 the main PCU use with it was this the tiny Raspberry Pi compute module 4 and the pi is a neat little s or system on module they started 35 bucks and you could get up to 8 gigs of RAM per Pi but this thing came out in 2020 these new SS called the rk1 these come with CPUs that are five times faster they can use blazing fast nvme storage at over a gigabyte per second and they have built-in npus for onchip AI oh and they start at 15 50 bucks at the low end being the fastest ain't cheap yeah it's a lot more than the compute module 4 and if you go for the Royal experience with a full 32 gigs of RAM you're looking at a whopping $300 per module that puts the total price of this board I'm testing at $1,500 and full disclosure I bought this board and 4 8 gig modules with my own money so I could make this video but then turpie sent me 432 gig modules to test since they wanted me to see how the thing performs Max out but I was still happy to spend my own money on the cluster and I plan on keeping it deployed in my studio considering what you get for some people like me it's not a terrible deal but I wouldn't just go out and buy it on a whim but let's talk about what is terrible the elephant in the room the one thing that really derails this board's potential is these two ethernet ports these are both 1 gbit and this is a massive bottleneck almost anything you want to do in a cluster environment needs to be able to move data in and out and 1 G it ain't much sure there's also some pcie expansion but the easiest way to copy things to and from the cluster is through the built-in networking and consider this these compute blades even though they use the slower pies each one has its own independent gigabit connection to your network so 10 pies 10 GBS of bandwidth on the Turing Pi 2 the internal network is crippled by just having one external gigabit Port it's an architectural Achilles heel with all these cluster boards and until one of them gets built-in 10 gig networking it really limits their potential now there are some things on this board that improved a lot since the first time I tested it the built-in BMC or board management controller has a great web UI now and you can do things like control Power to each node or reflash them straight from the web other boards like the desk pie super 6C I have here make you plug a USB cable into each one of the ports for each node so managing a board like this one remotely is almost impossible you could drop this tur pip to somewhere remote and never have to touch it again all the important stuff is managed Through The BMC kind of like ipmi on a server there's also a pretty strong Community around the board supporting it with drivers and Os releases but I can't help but feel let down when I have so much potential but I can't tap into it except for a very narrow set of use cases like it has these four nvme slots on the bottom and with these RK 1s I can write to each one of those at 1.5 gigabytes per second that's double what I can do with even a Raspberry Pi 5 but what good is that when between nodes I can only copy things at like 120 megabytes per second it's still cool to learn Seth storage clustering on a board like this but it could actually be used in production scenarios if we had 10 gigs or heck even dual 2.5 gigs to be fair the compute module 4 also suffers from that same problem on this board and it also suffers from a lack of performance these rk1 modules they don't have that problem and that's because each one has a speedy RK 3588 chip this chip has eight CPU cores an npu or neural processing unit and a little igpu built in and everything about it is faster than the compute module 4 it's even faster than the chip on the pi 5 so when a compute module 5 comes out this will still be the fastest barring some sort of Miracle to illustrate that point here's the geekbench score for each of the chips but geekbench is a synthetic Benchmark let's look at something that really matters at least if you're like me and you can pile Linux multiple times a day in both cases it's about two times faster than the pi five and about five times faster than the pi 4 then there's high performance linpack I like this Benchmark because it's what the fastest supercomputers in the world use to compare performance and here again the rk1 trounces the pi two times faster than the pi 5 and five times faster than Pi 4 but what's really crazy is the efficiency it's not only faster than the pi but unlike Intel's historically fast but and efficient CPUs it's also more efficient than the pi meaning it does more work using less power and that scales from 1 node to four the rk1 performs better in a four node cluster and it's twice as efficient now the compute module 5 hasn't been released yet but even when it is barring any Miracles the rk1 is going to stand as the SBC cluster king for a while especially considering it tops out at 32 gigs of RAM that's four times the amount of RAM on any PI right now so we've established the rk1 is fast and expensive but one of the most annoying things are into with a lot of pip projects is deployability for a lot of these projects sure you can slap together some circuit boards but putting them out in the wild means you need a case or an enclosure or just really anything to protect it the great thing about being Mini ITX is the Turing Pi 2 can fit in standard PC enclosures and over a year ago now my electronics sent me a prototype of their new dual Mini ITX rack mount enclosure you can mount two Mini ITX systems side by side in a standard Rack in two U of space but this thing has a party trick the two sides can be separated and it turns into two mini 10-in rack mount enclosures but that's not really useful without a 10-in minir rack desk pie who happens to make that super 6C board I mentioned earlier also just started making the rack mate T1 an 8U minir rack that you can stick on your desktop or take anywhere and they just sent me this one to test I've wanted to get into 10in racks for a while so this thing is perfect for me it has an open back so even though I think this my electronics case will stick out a little bit it should be fine but my goal is to put all these things together get my first mini rack set up and then deploy kubernetes using my open source pie cluster project I'm going to focus on getting this set up first so we can set this back down over here and get some things moved away and uh this is the first time I'm taking a look at this box so I hope that it includes all the accessories I mentioned that this is a this was a prototype they sent this was before they actually released the product you can buy it now you can buy this on I think Amazon even in the US but uh I'm going to try it looks like they they threw in a few parts this is the the 10-in rack face plate looks like uh and it looks like their power button is actually a keystone power button that's kind of cool I was going to say there's no place for a power button but then I noticed that this is a keystone I like that they included this little goodie bag which has a blank for the keystone jack some screws these are for ssds and hard drives uh and then some more screws don't know what those are for maybe case screws and a teeny tiny uh half height PCI Express card slot cover so this is cool if I can get it open maybe black acrylic covers so you can actually use this as a desktop enclosure too you don't have to have it rack mount cuz my ultimate goal is to maybe have uh this system be rk1 boards and this system if the compute module 5 ever comes out this could hold uh six compute module 5es and we could do some more comparisons my guess is that six compute module fives would probably be getting close to the performance of four of these RK 1s uh but I don't know until we get them and I just saw news on the Raspberry Pi EO the latest firmware for the Raspberry Pi it has support for cm5 baked in which means they must be getting closer and it had uh some configuration for a new d0o stepping so I think the pi5 has a c0 chip and d0 means that they would have made some revisions to the Silicon so maybe the new uh BCM 201712 d0 is a little faster I don't know maybe it can be clock Tire uh we'll see but I'm not going to use these for now these these little heat sinks have built-in fan so just the passive air flow in this case is probably enough but I think that there's a mounting pattern for a 60 mm fan on the front of this case and if there is I'm going to put this noctua fan in there and I believe that there's a fan header on here maybe no there is a fan header but it's only two pins so I'm guessing that's power and yeah I can plug that in so I I think that might just be power and ground I almost lost that one and yes I know I could put down a mat here I might do that someday I don't know okay so that comes right off that is the 19-in rack mount face plate so now we have two cases and there is um there are mounting holes for that fan and there's a power button and then a blank in here and I think the uh the first time I ever got something from my electronics that is using hex bits which is relatively standard uh but less standard I think at least in North America and I had suggested they use uh Phillips even though Phillips is not the best thing in the world they're so they're everywhere and everybody has one of these little Phillips screwdrivers uh since we won't have a piea Express card in this build I'm going to go ahead and install this card slot cover and uh screws are lined up and I probably should have installed the fan before this but that's okay I'm going to use it as an intake okay get out my box and there's my screws put that aside let's grab four of these guys all right there there is a little fan header right here but it's only two pins and I need to check the pin out on that because I could plug this straight in but I don't know if that's just power and ground right there or you know what's funny there's actually an unpopulated socket right here for a fan and it uh that would be perfect cuz I could just go Boop like that but I don't know why that's not populated on mine uh another thing that I need to do is get power so I have these these front panel connectors which go right here underneath all right so I found the front panel pin out on their website nicely Illustrated power LED and ground are there and if you really wanted to Cable manage I could zip tie those but I'm just going to leave it like that for now so I'm going to just plug this in and hope that it's ground in 12 volts if it's not we'll see what happens to the fan worst case we'll just have a little bit of smoke come out of here and for power since we're in this tiny case you can't really do an internal power supply even Flex ATX is too big uh so this is really designed for projects where you can use a Pico PSU I bought this one off Amazon it's the rgeek 250 wat PSU there's no more specs on it but you can find Pico psus from a lot of different places it's nice cuz it has um you know CPU pins I think it has Molex and it has SATA power and uh you can even take these off cuz I don't think I need any of those do I no I can even take this off cuz I won't need those on this build uh but it it basically breaks out an ATX 24 pin connection into a 12volt barrel plug and I can even put this on the back somewhere there's nowhere oh there is a little spot here I wonder if this will fit look at that they thought of everything in this case so this will be our power supply connection uh and then this just plugs in here one issue is uh these two leads going right into the fan let's see if I can safely Bend these out of the way we might be hitting some fan blades here but you know alls well that ends in not a fire but it' be it'd be kind of fun to see if the fan blades can slowly slice away the insulation on here uh but right now the fan blades have about 2 mm of space so I'm not too concerned I am more concerned about wrenching on those connections it it might be a good idea if you can get a low profile 60 mm fan that would solve that problem too but everything is connected now and uh it looks it looks pretty good inside that's that's pretty Sleek I I like this case more than I thought I would kind of pops into into place here it's kind of a fun look just Ram some air in there actually this this is held off by about four or five millimeters from the front of the fan so that actually will help with the noise but let's get these two screws in to hold this plate on and the fact that there's only two screws holding it in and no other real support I don't think you'd want to put a really dense dense system inside of one of these because you're holding all that weight from this uh face plate onto the body of this through these two screws it can probably hold maybe 5 or 10 lbs but if you put a pile of lead in here it would probably rip off uh but it's it's pretty solid and uh if I if I had my ATX cover on here it would look a little nicer in the back it also has four Keystone Jacks in the back so that's that's a nice touch uh but that's that's it that's our little case now we need to set up the rack to put this into I have not taken a inside of this thing yet and it's actually probably going to be easier to open it on the ground but let's see what it comes with I think I saw it was $120 on desk Pie's website but on Amazon this showing is $1.99 I don't know what the final price is going to be and I'm not sure if it includes everything that I have in this case but I'm guessing it does let's check it out okay it feels solid the top is acrylic uh with some ventilation whoops there's a box in there and the sides are acrylic with kind of a smoked smoked glass look this comes with let's see there's a panel on the bottom that looks like it has fan mounting holes then there's a blank a blank this blank is actually bent just a slight bit so I wonder if in shipping it got crushed a little bit there's a shelf and and this is a Shelf with a lot of mounting holes I think that this lets you mount two Nvidia Jetson or raspberry pies or different things and I think you can mount hard drives or ssds underneath it too that's interesting both sides have uh holes so you can actually do it from either side you could have some things come in from one side some from the other the depth of this I mentioned earlier is not quite deep enough for this but since the back is open this should just be able to go through since this is a bit heavier I'm probably going mount it down low LT you're failing me you don't have the right size hex bit I wonder if it comes with something let's see ah look at that they include the tools how nice but uh you can find out more about this dusk dusk pie rack mate T1 I'll have a link down in the description I'm going to pull this big shelf out right now the Shelf will be nice if I want to oh well that's interesting I thought this was just straight up through but now it makes sense this is this shelf goes onto this little tray that's in the back see these little these little guys are just holding that shelf on and this shelf is useful for things like a network switch or little power distribution you know what's funny is in Europe at least there's a pretty decent number of 10-in minir rack options in the US it's really hard to find them and uh that's kind of disappointing I think that this kind of rack like look at the size of this thing this would be perfect for a little home lab especially if you're in an apartment or somewhere you can grab it and take it with you uh you can fit this almost anywhere and uh a lot of things like mini PCS fit perfectly in here if you have you know I've covered many of them on my channel raspberry pies of course let's see if this fits in the bottom here uhoh oh there we go and there we have it I have not one thing I have never seen is a half rack UPS it' be cool to make a little little oneu or maybe even 2u ups for this thing then you could literally unplug it from the wall take it somewhere set it down plug it in and it would still be running that's I don't know to me that seems pretty cool I I don't know about you but uh I see this and I think of a lot of possibilities and uh I don't know what what thread pitch they use on here I'm going to have to go grab a couple of my rack screws and find that out that's solid and you can see it it comes out the back just a teeny tiny bit it's not bad at all and when we plug in power the power cable will just pop right out here like that and network goes here and we'll have a functional cluster this is an 8 amp 12volt power supply and that should be plenty for this uh this setup with the four RK 1s I'll plug networking into one of the ports on the back power can go in over here too and the board is powering on its BMC there's a few LEDs that are on another thing that could be improved with this particular setup is having uh LEDs either on the back or the front one thing I did like a lot about the desk pie super 6C is it has activity LEDs for each node on the back so if you look at the back of it you can see which ones are doing stuff or powered on uh it's a little harder to see on on this once you put it into an enclosure that's it looks like everything is up now and the fan is on so that's good I can feel the air coming out and the power came on so I guess it boots itself the first time and I see all the nodes LEDs are on so let's let's go on the network and check this out and see what we have I did check the the rack screws and they are 1032 threads so you can get standard rack screws which is nice and uh and replace the screws that come in the kit if you want uh that way for me it's nice because I can use Philips head for everything uh the other thing I did was when I pulled out the box I I took the cover off and I put in noa's low noise adapter so the fan is about half as loud um and it probably it could probably run without a fan at all uh honestly because the heat sinks have built-in fans but now that it everything's booted up it's on my network and I have this Playbook it's an anable Playbook called pie cluster and this is not raspberry pies but they are turning pies I I guess uh but they're RK 1s and this Playbook is what I use for all my cluster testing so it basically does a lot of things to it it can Benchmark the cluster it can set up storage it can set up um uh kubernetes which is what we'll use today uh to to make it so that instead of having four separate nodes you can have four nodes together in a cluster that you can do useful things with uh since I'm running Ubuntu and not Debian uh Debian or posos 12 includes network manager out of the box so with Ubuntu there's actually a step that I had to do um there's a Playbook that I I have called ubun Ubuntu prep and that sets up network manager instead of the whatever net plan that's set up in ubun Ubuntu by default uh so I did that already and the next thing that I need to do before I can deploy kubernetes to it is set up static IP address networking if you don't have static IP addresses kubernetes gets really confused uh um at least if you're like me you want to use uh fancy host names so in my case I'm using Turing 1. local turning 2. Local 3 4 and I already set those up when I flash the nodes so if I say SSH uh Ubuntu Turing one. local I can log in using my SSH key and uh that way I don't have to know what the IP address is it doesn't matter to me I use the DNS names uh but because mdns is a little funny and it might be using IPv6 like this or it might be using ipv before it's best to be very explicit about it uh so we're going to set up static networking uh but this is the inventory file that anible is going to use to know how to connect to the different nodes in my cluster and then I also have a config file that tells anable some other details like this is my Subnet 10.0.2 do whatever and the whatever is these octets the last one uh so the first node turning Pi 1 will be uh 10.0.2 at 91 and the second one will be 92 93 94 uh and so I'm going to run that Playbook the static networking which is here this is a networking Playbook that runs a few different networking tasks in this case just setting up static networking well first let's make sure we can connect all the nodes uh you can run this command anible all-m ping this is basically going to send a ping out to each node over SSH and if it can work then it shows success which it did good job thank you uh then the next step is I'm going to set up the static networking uh which is this Playbook and that hopefully will just work it should uh set up the hosts file to make sure that each each of the nodes can connect to each other and once it finishes uh setting up static networking it's best to just reboot it you can actually use it this way but I like to reboot so it only has the static IP address and doesn't have any potential collisions so I'm going to run this command it will reboot all of the nodes and wait for them to reboot once they reboot an will come back can say it's successful so the main Playbook will set up kubernetes and it will set up some configurations this particular thing has some demonstration like running Drupal inside of your kubernetes cluster which is a nice uh kind of challenge because Drupal is a fairly heavyweight PHP content management system and it requires a mySQL database which requires persistence so you're really testing the full breadth of capabilities of your cluster when you run an an actual real world application like Drupal with persistence and with uh front end and all that kind of stuff so I'm going to run this Playbook and it takes uh I don't know between 10 to 20 minutes depending on uh your internet connection and everything and I just noticed that I have another bug in here that expects the user pi to be present but when we're running on these BTU nodes Pi is not the user that's used so um I have an issue for all these little bugs and if you find if you if you want to run this Playbook on your own system and you find any problems feel free to open an issue that's why it's open source so I should just use anible user anible user okay so now let's try running that again and we'll see if it works it should use the user Ubuntu instead of Pi there it goes now it's working okay the cluster is deployed now and I'm going to log into the first node uh turning one and check if I can see kubernetes running and see where the pods are located pods are basically little units of work for kubernetes and there should be like a Drupal pod and a Maria DB pod and they should be all working together and we should be able to access Drupal we'll see if that works so logging into turning Pi 1 and if I change to the root user account following the readme that's over here I should be able to use QB cuddle or cube control to see if the Drupal pods are running so let's check that out and it says pending so that's not a great sign that means something's up uh let's get all pods and uh so a lot of pods are there it's creating the NFS pod which means there's probably an issue with the shared file system the way this Playbook is set up it's supposed to set up a shared NFS folder on node 3 I think by default and that might be failing because this is Ubuntu and I was running it on Debian and there might be some path issue or who knows what uh but I'll get that sorted really quick and uh we'll see how this works so a quick overview of kubernetes there's a there's a command line utility called Q cuddle or cube control or kubernetes control whatever you want to call it but Q cuddle is supposedly the official name for it uh but we can say CU cuttle get and then nodes uh that shows us all the nodes that are connected and it's seeing that turp one is the control plane slm node and the other ones are just work work ER nodes I can also get pods and uh if we're do if we do these things it does it in the default namespace but you can use namespaces in kubernetes to do special things like isolate apps and things like that so I isolated Drupal in the uh Drupal Nam space if we do that we can see that these two are pending and this is a common thing when you're setting up kubernetes clusters you're going to have weird issues like this one says container creating and that means it's probably having some issue uh getting set up and there might be a problem it might be something that you're using a pod that doesn't run on arm or you have a pod that requires some resource that's not available so I checked the NFS Mount uh so if I do this I can see that that that mount is here UB cut will describe pod this uh let's see it says Mount failed exit status 32 through the power of Google I found this issue and uh someone had suggested make sure that you have NFS common installed on all your nodes if I scroll down here so I will add that to the Playbook I wonder if that's actually on P by default and not on Ubuntu and that's why I didn't see this problem on my pie clusters oh there we go started container Drupal so if we go here we can see that both are running and that's saying that I could go to any of these nodes on Port 80 and it should work so let's let's try that we can go to turn turning one. loal or 10.0.2 91 if we go here here look at that there's Drupal installer save and continue let's see if it's installs with the Umami food magazine install profile which is like a demo site I also noticed this happens in Drupal sometimes this seems to be a Drupal issue but uh if you just try again A lot of times it works I don't know why it doesn't work the first time but here it goes installing Drupal and this is actually kind of fast this is faster than it is on a pi4 I haven't tested this on a pi five yet but uh installing Drupal seems to be going fairly decently on uh on the rk1 so while it's doing that we can also check where everything's deployed so if we look at these this column here which is the node you can see that there's some things on Turing one this is on turning four Maria DB is on node 4 and Drupal is on node 2 so kubernetes has done a pretty good job of spreading things out so it's running Drupal on node 2 Maria DB on node 4 and if we have scaling rule set up in kubernetes it can actually scale and run instances of Drupal on other nodes I've run through the kubernetes setup I fixed a few bugs in my pie cluster Playbook so it works with the Ubuntu and the rk1 and now I have a working cluster I could deploy really any kubernetes workloads here as long as they have arm containers and just clicking around while I'm logged into the demo site feels a lot faster than when I ran it on my piie cluster it's not perfect and the gigabit Network between the nodes leaves some performance on the table but this is definitely a step up for my pie cluster and I think the rk1 boards will definitely hold on to the SBC cluster performance crown for a long while now you might think it's fun to compile your own Linux kernel or self-host a website on a mini kubernetes cluster and I'm in that group in fact I host a bunch of websites on my own but I wanted to make a better online store for my merch since I'm switching to a way better t-shirt supplier I mean you can see this old shirt's already peeling and it's only been through like eight Washings for my new store I'm using this video's sponsor Squarespace I already mentioned how I got a full store site set up in less than an hour and I integrated it with my bank for checkout in just a few minutes Squarespace Smooths over all the hard parts of building website and they let me focus on design adding products and getting red shirt jef.com ready for a relaunch not every self-hosted app's a good fit for Squarespace I mean it can't replace jelly fin but especially if you're a small business and you want to avoid the hassle of building your own system or running your own servers give Squarespace a try go to squarespace.com for a free trial and when you're ready to launch go to squarespace.com red shirt Jeff to save 10% off your first purchase of a website or domain until next time I'm Jeff Kling

Channel: Jeff Geerling

Views: 212,656

Keywords: raspberry pi, linux, sbc, rk1, turing pi, cluster, clusterboard, single board computer, open source, kubernetes, k8s, k3s, rancher, ansible, automation, build, dual, myelectronics, rackmount, servers, livestream, live, rack, homelab, edge, compute, mpi, hpl, top500, benchmark, performance, test, deploy, application, hpc, education, rockchip, rk3588, soc, bcm2711, pi 5, cm5, cm4, compute module, deskpi, geeekpi, mini, inch, 10, half, rackmate, t1

Id: h6zt8KeXFdA

Length: 31min 5sec (1865 seconds)

Published: Fri Apr 26 2024