EPYC TrueNAS Scale Build and VM Install

one of the questions that's asked most often here in my videos is how much does it actually cost to run a full stack of servers out in my garage next question welcome back to craft computing everyone as always i'm jeff i have a problem i like servers and at this point i have seven servers running in the rack out in my garage plus a dedicated trip light 7k rm air conditioner to help keep everything cool as you might imagine those servers require a ton of power which i happily added with the help of two dedicated 20 amp circuits last year power delivery to the rack isn't the problem i have it's limited ups capacity you see when your server rack starts with just a couple physical boxes and a handful of hard drives it's pretty easy to keep the total power draw at around 300 watts in just over two years i've ballooned from two home built servers and eight hard drives to a whopping seven servers 36 hard drives 20 plus ssds and give or take about three quarters of a terabyte of memory as such my 1500 va apc ups unit just isn't holding up anymore if i have all my servers running at the same time just the idle power draw is somewhere around 1100 watts the apc is only capable of continuously delivering a thousand watts and in reality should be kept closer to 800 if possible and that 1100 watt load is only coming from the servers in the rack i have the air conditioner running on the second 20 amp circuit which pulls around 5 to 700 watts on its own during the summer so while the last year was all about expansion i think my current vibe is going to be about a little consolidation and that starts today with an upgrade that is admittedly entirely overkill but we'll take the services from four servers and cram them all into a single box one of the biggest power draws out in my server rack is my nas box which can easily pull down 500 watts on its own even when sitting at idle and while the hard drives certainly contribute to that the motherboard and cpu combo i have in there right now isn't exactly sipping on the amps either currently powering my true nas box is a super micro x9 adi dual socket motherboard and a pair of xeon e5 4627 v2 cpus along with 64 gigabytes of ddr3 1866 memory and all it's doing is running the nas it's overkill most of the time but as i edit video directly off the nas using a 10 gigabit link ram and single threaded cpu performance were pretty critical so what am i going to be replacing that hardware with well you all know this server right here this is my cloud gaming server or at least it was until i replaced it about a month ago inside this is an amd epic 7601 32 core cpu 256 gigabytes of ddr4 ecc registered memory all running on a super micro h11 ssl-i motherboard one of the goals in my server rack has always been to virtualize truenast full time not just as a one-off experiment that i've done a couple videos on and i think i finally have the hardware that i'm comfortable doing that with so today we're going to replace the power hungry dual xeon ivybridge nass motherboard with some amd epic goodness jumping to the epic hardware will allow me to allocate a full 128 gigabytes of memory to truenas while still having enough headroom to run all the other services i use here at home this will also eliminate the need for the trio of hive zeus boxes higher up on the rack that are currently running all of my vms each of those hive zoo servers has 64 gigabytes of dvr3 ecc memory and a pair of ivybridge xeon cpus while they still do an amazing job and i still do recommend picking them up if you're building out a home lab when i've got 96 cores worth of epic just sitting around doing nothing it's kind of silly to leave the ivy bridge systems in place so today i'm going to take this epic 7601 install it into my nas chassis install proxmox and get trunast scale up and running inside a virtual machine oh right by the way i'll be installing the debian based truna scale for reasons we'll get into at the end of the video but for right now let's get to swapping all this hardware around [Music] you know what's nice about taking six hours to replace all the hardware inside your server you can open a second beer welcome back it's only been about six or seven hours since i started this project uh a couple of things went wrong a couple of things went right and the server is now up and running so let's go ahead and take a look at that jumping on over into proxmox here you can see i booted this thing up about a half hour ago and finally got it up and running our epic cpu is recognizing all 32 cores and 64 threads and all 256 gigabytes of memory is showing up just fine now i am running the most up-to-date version of proxmox 6.4 that is update number 13. however i have not yet upgraded to proxmox 7 and the reason being is as far as i know the vgpu unlock script to allow me to partition my nvidia cards is not yet functioning on version 7. so i am sticking with version 6 for now as there are a couple things that i want to do inside this server that involves splitting up my gpu but that's not why we're here today today we are here to get truenas scale up and running inside of a virtual machine with actual hardware running behind it so the first thing i'm going to do is upload the iso image into my storage repository so we can get started on the install so i'm going to go ahead and grab my trunascale beta iso and get that uploaded with that done we can go ahead and create a new virtual machine so i'm going to right click on my proxmox server and go to create vm we'll go ahead and give this an id of 201 and we'll name it craft dash trunas for the cpu i'm going to go ahead and give this thing 16 cpu threads to work with and for the memory it is go big or go home that is 131 072 or 128 gigabytes and i'm going to turn off ballooning device as i want all of the memory allocated to this machine 100 of the time so it can use it for zfs cache space for our network adapter we're going to leave that alone and then finish and we'll go ahead and double click on that allow popups and start it up so we're going to go ahead and run the installer i'm gonna install it on the only hard drive that is available that is sda or the 32 gigabyte boot drive that we created and here's where we enter our root password so forgive me but i'm going to black out your screen really quick while i type that in and now we just wait for this to finish up and that did not take long at all so let's go ahead and hit ok and then we will reboot our system while that's rebooting i'm going to go ahead and unmount that iso image so it doesn't get confused in the future now while this is starting up for the first time i'm going to talk about the difference between true now scale and true nas core i've been running trunas core and true nascor is the version that you all probably know and love true now scale is a new endeavor by truenass that does away with the bsd core of the old trunas and introduces the debian core which is the same thing that we use on ubuntu popos and a number of different operating systems and that didn't take long at all and now that it's started up i'm actually going to shut it down i know that sounds weird but you'll see why in just a second now on this proxmox server i have already set up i o mu and vfio for pci express passthrough as i do that with pretty much all of my proxmox servers i'm not going to cover that process now because i've done that a number of times in the past rather if you want to click the link down in the video description i will have written instructions on how to enable iomu inside of proxmox but right now there are a couple pci express devices that i want to pass through into truenass starting with the hba controller that i have and that is a dell hba330 there it is so technically the dell hba330 is nothing more than an lsi sas 3008 controller which is an 8 port sas 3 controller and i've got it set in it mode already so i'm going to click pci express and all functions and click on add the other thing i'm going to pass through right away is one of the 10 gigabit network ports on my dual 10 gig card and that should be this guy right here the q logic number two and again pci express and we'll click on add now i didn't check all functions when i pass through the network card because i only wanted one of the ports of the network card forwarded through they show up as two separate devices but they do share a pci bus if you click all functions it forwards everything on that pci bus to the virtual machine which would mean i would lose my 10 gig network port inside of proxmox so why am i forwarding one of my 10 gigabit ports well if you remember back to my how do i backup all of my servers video i use a dedicated 10 gig port to connect to my backup server which means when i'm running backups at night it's not going to interfere on the network with any other bandwidth so all of my other services run unencumbered and my backups can still run on their own dedicated 10 gig line now that we have our devices passed through i can go ahead and start truenast back up so i'm going to double click that and click on start remember how i was talking about how i don't want to forward the port for my network card that's on proxmox because i would lose connection to proxmox apparently 60.01 was the network card for proxmox not the network card that i wanted to forward for true nasa's backup so i just killed the network connection to my proxmox server i'm gonna run out to the garage and i'll be right back hey look it works again i think we can continue now so let's go ahead and remove host pci device 61.00.1 yes and i'm going to add pci device 61.00.0 and hit add and if this doesn't work that means i can't split that network card but i'm kind of hoping i can so i'm going to hit start again and we'll see if i lose connection again okay i can't forward that network card now we know now there are some network cards that you can forward just a single port for and in this case i thought i could because it did show up as two independent devices however in this case it seems like they share an onboard chip which cannot be split and so either it's forwarding the port and then disabling the chip or the chip doesn't know what to do and just turns off itself either way i'm gonna go back out to my garage and reboot the server again all right and now we wait until it starts pinging again the problem with 256 gigabytes of ram and an epic 32 core cpu it takes like two or three minutes to post it's faster than my 64 core epic cpu which takes like five minutes to post all right we're back in business again all right so let's go ahead and remove that pci device i think i will explore that in another video on how to dedicate a network port inside of proxmox to a virtual machine without using pci express passthrough so i guess make sure you're subscribed if you want to see the finale to that whole endeavor but for right now let's go ahead and continue on with just the hba now do note i will be passing on more devices than just the hba just not today and there we go truenas scale now do note truenow scale is still in beta you should not run this in a production environment the only reason i'm running it is i have pretty darn good backups and i can run my files directly off of my backup server so there's really no such thing as downtime unless i lose both the same time if there is a problem with true nas scale i can always revert back to trunas core and my data is fine so you have been warned all right we are booted into truna scale first thing i'm going to do is give this thing a static ip address and i'm going to select network interface 1. we are not going to remove the current settings we are not going to configure for dhcp uh yes i want to configure ipv4 network name is ens18 which is the name it already had we're going to give this an ip address of 10.0.0.221 slash 24 and we do not need to configure ipv6 on my internal network next thing we're going to do is set up our default route and that is our default gateway so option four we're going to configure our ipv4 static route and that is going to be 10.0.0.1 first up is our dns domain so i'm going to enter craft.local which is what i named my so this is definitely a bug uh when you hit backspace it doesn't you remove the backspace it adds the character that you tried to backspace from uh so forgive me for this but my local dns name is craft dot local there we go uh dns name server one is 10.0.0.1 and i do not have a dns server too and if everything checks out i should be able to go to the shell so option number nine and i should be able to ping google.com hey look i have an internet connection and dns resolution that is good enough for me and that's pretty much all we need to do on the command line side of things i'm going to go ahead and jump over to the web gui and we'll get to configuring my new truenastbox so once again we set up an ip address of 10.0.0.221 and here is our new splash screen now this should look very familiar to everyone who has used truenous before however true now scale does have some very interesting new features including a pretty fully fleshed out virtualization system i've not gone through any bit of testing on this but it is using the standard kvm virtualization system which is actually what proxmox uses for virtualization so this is definitely something i'm going to be checking out in the future but i didn't want to rely on it on a primary basis quite yet but just like other truenow systems there are apps available so you can download these one-click installs of very common applications like a plex server or nexcloud and everything else is essentially in the same place just with a little bit more of a polished look to it and there's a little bit more central management to the system than there was with truenas core this is meant to be replicated across multiple enterprises or multiple locations a little bit easier than truenas core was but onto the configuration if i go on over to storage normally i would click on create pool and i would select from a subset of hard drives here now i am importing my hard drives into true nas scale from trunas core as in there's already a zfs file system on my hard drives and i kind of want to save that because my data is important so rather than creating a new pool i'm actually just going to import the old configuration from my trunas core server to do that i'm going to click on system settings go to general and then up on manage configuration pull that down and click on upload file i'm going to click on choose file and then i'm going to choose my most recent backup which surprisingly enough was today at 1 30 or about eight hours ago so click on that and click on open and upload and now all i have to do is wait and hopefully when true now scale reboots all of my drives and drive shares will automatically be reconfigured for me and after only about 10 minutes to import all of my existing storage pools you can see that truenas scale is booted back up and should hopefully have all of my existing configurations so again i'm going to log on in and there we go you can see all three of my storage pools have imported successfully if i go over to storage there they are in all of their glory with all of their data and if i go down to shares we should have all of my smb shares imported as well and indeed we do now the keynote among you have probably already noticed that all of my pools are reporting as unhealthy and there's a pretty good reason for that i imported all of my drives into trunas but i didn't import my cache disks into true now scale the reason being is number one i didn't have enough pci express slots to add all of my cache disks back into the new server and reason two is i'm actually going in a new direction for my cache drives if you remember back to a couple months ago i took a look at these these are sandisk fusion i o i o memory modules they were typically sold to first party servers like dell hp or lenovo as a flash accelerator for large format storage now in that video i went through some of the problems getting these working as in there were a lot of problems getting them working they will only run inside of windows if you want to use it as a storage disk and there's virtually no drivers available if you want to run them on linux however in the couple of months since that video i actually discovered a github repository with an open source reverse engineered set of drivers that works with both vsl3 and vsl4 drives and they work inside of linux more specifically they'll work inside of the debian core so while trunas core is not able to run these s-cache disks truenas scale running on the all-new debian core definitely can so make sure to tune into my next video and i'm going to go over how to compile your own driver and get these running as a cache drive inside of trunas scale so fusion io drives how to use nvidia vgpu profiles as an envin encoder inside of plex and more coming up very soon on the channel as always if you're interested in any of the hardware that i showed off today i will have amazon affiliate links down in the video description make sure you go give those a look on your way down there make sure to drop this video a like and subscribe to craft computing if you haven't done so already follow me on twitter craft computing to keep up with daily shenanigans like this and if you like the content you see on this channel and want to help support me on what i do head on over to craftcomputing.store get yourself some official merch like a pint glass shirts or more if you want the opportunity to chat directly with me think about joining my patreon or float plane links are also down in the video description you'll get exclusive access to my discord server where you can chat with myself as well as the other hosts from talking heads that's going to do for me in this one guys thank you so much for watching and as always i will see you in the next video cheers guys that one's not very good here for today is from matchless brewing company right here in oregon it is the plug and play hazy ipa and yes i bought this one just for the can art just like you can buy this pint glass just for the art over at craft computing.store normally i expect better for matchless this just isn't that good uh it's a hazy ipa without a lot of citrus and without a lot of hops it's kind of like a acidy multi thing i mean it's drinkable but i'm not exactly enjoying it either i think that's going to be a hard pass on the second pint beer for today is the kung fu karaoke ipa and the can art is absolutely phenomenal on this can this is from the kings county brewers collective in brooklyn new york that's not bad it's got this little little weird funk to it that malt is is a little weird i don't hate it but it's a little weird whoa i think i know why that was funky uh there's a crap ton of sediment at the bottom that just turned this into a hazy so i think i was getting half hazy yup now it's a hazy ipa you see how dark that was it's like three week old orange juice pouring into my glass ah that looked disgusting and that is not a pleasant color for a beer it's just not that did add a little bit of flavor right up front i'm not gonna say it was a good flavor it's not often i will open a second beer on the show because i don't like the look of me drinking two beers even though i rarely drink two beers in a day anyway uh except on wednesdays wednesday is different but it's even more rare that i go over two i'm i'm just not digging this one now whether it's settled and i all i was getting was the malt and then i got you know this the lactose and the acid kind of poured in afterwards this is a new england ipa it's a hazy ipa it is a very hazy not overly pleasant looking ipa this literally looks like i went to like a dirty river and scooped some mud up off the bottom and that's what it looks like i mean it's just not a good look it kind of tastes like dirt there's no real citrus or real great flavors that are poking through it's just okay ignore that take just like i kind of want to ignore that beer

Channel: Craft Computing

Views: 45,812

Keywords:

Id: Vi-ZdJOenWc

Length: 22min 46sec (1366 seconds)

Published: Thu Sep 09 2021