How I Built My $10,000 Deep Learning Workstation

hello and welcome back to another video guys I know it has been a long time since my last video sincere apologies for that in this video I want to show you how I build this $10,000 deep learning workstation which is an insane amount of money but obviously it's a very powerful deep learning workstation that I have in my hands right now and yeah let's just jump right into it before we start I want to share two things with you first showing you how I built my deep learning workstation means that there are many more videos to come in the near future so in case you're interested in a certain topic just let me know in the comments and I will look into it and second I've started a collaboration with Nvidia which is also part of the reason why I'm creating this video and yeah more on that later in the video and now let's get back to how I actually built my deep learning workstation and for this first of all a lot of planning is needed I read a bunch of blog articles and watch YouTube videos to gather some information on how to decide on a certain component or like which criteria is important for a certain component and since they're like very extensive guides out there and very extensive articles my intention with this video is to summarize that so that you get a very concise guide highlighting the most important aspects to building your own deep learning workstation for this I will first show you how to pick the right component for your setup then I will show you how I assembled my whole deep loing workstation and as a third step I will show you how I installed the software so I could start running models on my workstation when it comes to building your own deeping workstation there is no way around PC port picker because it's such a helpful tool to organize your build for example here you can see my build it shows you all the individual component needed to build your own debling workstation then it also shows you the prices and also where you can get that component for the cheapest price and one feature that I really liked about PC per Pickers you can see the compatibility of individual components so here for example you can see that I still have two fre memory slots and could increase my ram afterwards or I could add an additional SSD and on top of that you can also get inspiration by other users sharing their builds with you so it's a super valuable website to organize building your own deep loaring workstation all right let's start with the most important component the GPU while the GPU most likely will make up the majority of your cost it is also the most important component for a powerful deep learning workstation because during inference or fine-tuning or training ideally all the computation is done using the GPU but how to choose the right GPU for this from my research and understanding there are three main features to look out for which are the tensor cores the memory bandwidth and the GPU memory and let's start with the tensor cores why are they so important and I found this very nice explanation by Tim Demmer explaining tensor cores are tiny cores that perform very efficient matrix multiplication since the most expensive part of any deep neural network is matrix multiplication Tor cores are very useful and fast they are so powerful that I do not recommend any gpus that do not have tzor cores and I think all the gpus that Nvidia has released in the last four to 5 years already had tensor cores so only the really old ones don't have tensor cores and while tensor cores are very important for fast computation we still need to make sure that our data is transferred to those processing units to those tensor cores in time and for this the memory bandwidth is very important so Tim Demmer in another example described here that Tac cores during a GPD 3 siiz training have a utilization of about 45 to 65% meaning that even for the large neural network about 50% of the time hensor cores are idle and this kind of highlights why High memory bandwidth is important so that we can move our data very fast to their processing units the tenser cores and the last feature that I would recommend you watching out for is the vram or GPU memory which kind of determines what kind of AI models you are able to load with your GPU so the vrm can really be a bus kill for working with certain AI models so definitely make sure to assess beforehand how much vram you will need for your purposes and then get a GPU that at least fulfills your vram requirements and also keep in mind that in recent years scaling laws have been more and more explored so bigger corporations have used more compute and more data to train bigger models and to be able to use such models you also usually need a lot of vram and Tim demmas also had a nice horis for this which can help as a guidance so if you're planning to do image generation having a g U with at least 12 GB of vram would be recommended and if you're planning to working with Transformers even more vram is recommended having at least 24 GB of vram or more having this knowledge now it's time to actually choose a GPU and for this I have very exciting news because I started a collaboration with Nvidia and Nvidia was kind enough to support my Channel with this RTX 6,000 GPU which is really cool with that I can create many more videos in the future train models myself or fine tune large language models for example the 70 billion lmer 2 model which is very exciting so shout out to midor and many thanks for giving me this GPU and with all this excitement I almost forgot to actually mention the stats or the specs of this GPU so this GPU has 48 GB of vram it has 568 tensor cores and it has a memory bandwidth of 960 GB per second which is also very nice so this GPU is definitely really really high end pce and I'm very grateful that Nvidia sent me one to support my channel and if you want more information about this GPU I will link it down in the description box and here Tim Demmer has also showed in a very nice graph the relative performance of different gpus giving you an idea of how much weam each GPU has and their individual names as you can see he only investigated Nvidia gpus and now I'm now doing a collaboration but honestly I think all other manufacturers like AMD or Intel are maybe offering gpus that have similar or slightly worse specs than those Nvidia gpus but the major issue at the moment is that all major AI libraries rely on NVIDIA drivers to accelerate their gpus for deep learning so it's sometimes not even the hardware or the GPU itself but it's a software that makes all the Nvidia gpus outstanding and more or less the only option at the moment and that's also why Tim demos for example here only compared Nvidia gpus and yeah I would just recommend you checking out this graph and I will also link this article as well as all the other helpful sources in the description box and he also shows another graph which maybe is even more relevant for you guys so here he has the relative performance per US dollar so definitely also make sure to check out this graph and depending on your budget I would recommend you to pick the one that fits your budget but achieves the highest performance in here because I think that can definitely be a helpful source to deciding on What GPU to buy the CPU is probably a component where you will spend way more money on than you actually need to and this is because almost all the computation is done on the GPU so the cpu's job is more or less really just to transfer data to the GPU memory or the vram and to help pre-processing input data and for this reason from my understanding there are only two criteria that I would recommend you to look for and the first is that you have at least four cores per GPU so for example if you have a 4 GPU setup you would need at least a 16 core CPU and the second one is to have a sufficient amount of PCI Lanes which one GPU ideally requires 16 PC lanes and one MVM Express Drive requires four additional lines but for this I also found that Tim dmer in his blog post that we saw earlier argued that having this many PC lens is not even necessary and you only have a minor performance degradation of a few per if you have less than the ideal amount of PCI Lanes in your system so I would recommend you don't get carried away too much by the PC lanes and just get a CPU that's powerful enough also for everyday purposes to use your system but also don't spend too much money on the CPU because really the GPU is the most important piece when it comes to deep learning performance and for this reason I got this Intel I9 CPU of this 13th generation I know there also very nice AMD processors out there I had different builds and I just figured out that using this I9 processor is the cheaper option but I think as of right now now either with AMD or with Intel you can't really go wrong it seems like Intel is a little bit cheaper while I saw in the community AMD is a little bit more favored for the Ram or your memory I would give you the following advice and I forgot to mention that with the CPU once you have decided to buy a certain GPU all the other components are more or less dependent on the GPU or the gpus you've got so so you buy all the other pieces around your gpus to kind of form a well around setup and for the ram I would advise you to buy at least as much RAM as your largest GPU has vram also don't get carried away by clock rates which sometimes are used to sell RAM for higher prices and a third thing if you're unsure how much RAM you actually need just keep in mind that you can still add RAM later it's very easy to add on your motherboard at a later time so maybe first get as much as you think you need at least and if that's not enough just buy more for the ram I got this cuse a venan 64 GB hope you can see it kind of can my camera zoom in yeah here it can see it I saw in many different builds that this kind of memory was used so I also went for it and I'm pretty sure it will be perfectly fine with the motherboard compatibility is key you want to make sure that your motherboard is compatible with your CPU including the number of PC lanes that your CPU supports also obviously you want to make sure that your motherboard is compatible with the GPU or gpus that you have bought and for this just keep in mind that some gpus take up more space than you initially would think for example the the very popular RTX 390 or 4090 are really like heavy gpus let's say so they can take up two neighboring PCI slots just keep that in mind when you select the motherboard and as a last thing the motherboard obviously also should support the amount of ram that you're planning on having yeah and I actually bought this ASRock 790 how can I show it to you guys it's actually a new practice for me to show articles in my videos so I bought this ASRock z790 M Phantom gaming lightning motherboard um not much to say about it it's just compatible with the CPU with the GPU also with the NVM Express memory and with the ram I bought 64 GB of RAM and still have two empty slots so I could add up to 128 GB of RAM if I wanted and that's basically all I have to say about this motherboard when it comes to storage and drives I saw many different approaches I would recommend you to get one OS Drive which ideally is an NVM Express SSD which are just the fastest maybe go for 1 or 2 terab one so you could even store some model checkpoints or code I saw also other people getting an separate nvme Express drive for code and checkpoints which might be an Overkill and not necessary but it fastens up your workflow for sure and if you're planning on training larger models and using bigger data sets you definitely also need additional storage for example using an HD or SSD while HDs are usually half the price or definitely way cheaper Tim dear also argues in his blog post that while the GPU is Computing current batches the next batch can basically be fetched from the drive and the speed up gained by using an SSD isn't really relevant because the computation takes longer than the fetching of the data anyway so that way you can also use hcds for storing your data sets but in Practical terms I actually often saw people using ssds also for storing their data sets though I'm not 100% sure on this one I probably would get an SST just to be on the safe end and yeah as my OS Drive I got this Samsung 9 80 Pro NVM Express Drive which you can see here super fast yeah definitely will be a charm using this drive as my OS drive and then for data sets I got an 8 terb SSD which is uno flip Amazon I don't know I think this uh I think this was a scam sent me uh this Uno flip instead of the SSD so now I have to return this um yeah well I'm a little bit unlucky I guess on this end so have to uh wa to get a replacement SSD and will'll add this to my computer after I've got it to run all the components we need power and for that we need a power supply unit or short PSU and there's actually not much to say I I found a really nice rule of thumb which is helpful to find the perfect PSU for your purposes and this is 1.1 times the CPU wattage needed plus the GPU wattage needed for example for my use case the CPU needs a Max wage of 253 and the GPU Max wage of 300 so in total using this formula we get a around 68 and for this reason to leave some extra room I bought this coair RM 840xs to leas some extra room so this PSU can accommodate a total of 850 watts and actually is pretty heavy with a case you should pay attention to two things first obviously all of your components should fit in especially the gpus which tend to be the biggest component in your whole build especially if you want to accommodate multiple gpus within your case it's often a good idea to get a really big case so you have more space and potentially better air flow within your case and this also brings me to the second thing you should pay attention to which are The Thermals even though this is less important depending on what kind of case you're buying the inside temperature of your case can be decreased by a few degrees I bought this fractal mashify 2 mini case which yeah actually doesn't look all that mini in the video but comparatively is mini I wanted to just have a single GPU system so that's why I opted for a mini case which is very concise cute I can just put it under the table nothing special has a glass window I like the design it's pretty simple and doesn't take up too much space for the CPU Cooler there's really not much to say the only thing is your CPU Cooler should keep your CPU cool so you don't run in any temperature issues with your CPU and for this I bought this noctur nhu12a CPU Cooler which I checked their website is sufficient for the Intel I9 I choose and also since I have a mini case I needed to make sure that this CPU Cooler fits into my case and to allow a higher airflow inside my mini case I also bought those two case fans which are called noctura NF h12 not even sure if I need them potentially if I get issues with The Thermals inside my mini case I can even upgrade and add more to the case um but this is just for the start all right now that we have planned and bought all necessary components it's time to assemble the workstation and I've already shown you all the components that I have bought and as a first step of assembling I will install the CPU on the motherboard and for some reason this is always the part that makes me the most nervous I guess that's just because it's such a small and in a way fragile piece you really don't want to mess up the conductors on the back when you mount the CPU into the CPU socket on the motherboard next it's time to install the RAM on the motherboard for that just make sure that you plug in the ram in the right direction and that the ram Locker basically snaps back into its default position next we install the NVM Express SST which is super easy as you can see and more or less just requires to tighten a screw when I had to install the CPU Cooler I more or less messed up for the first time because I first mounted the motherboard inside the C case and then realized that I had to install a piece of the CPU Cooler at the back of the motherboard and for that I had to unscrew the whole motherboard and take it out of the case again to then install that support piece on the back of the motherboard and then again had to mount the motherboard inside the case so to summarize that I definitely would recommend you to check out beforehand if you have to install anything on the back of your motherboard for your CPU Cooler all right now that the motherboard looks pretty crowded it's time to mount it and this time I definitely didn't need to unscrew it again because that's definitely a part that also made me nervous yet to kind of make the motherboard fit into the mini case and also tighten in all screws of the motherboard because with a CPU Cooler it was already hard to reach certain points then it's time to supply our motherboard our CPU and other components with power and for that we need to install the power supply unit the PSU and here it can only give you one advice make sure to plan beforehand which components you need to supply with power width so you can already plug in all the necessary cables so once your PSU is fitted inside your case you don't need to plug in any additional cables that you haven't thought about in advance now it's time to mount the case fans which is a pretty straightforward task you just need to make sure to mount the case fans onto your case using screws and then find the right plug on your motherboard to supply the case fan with power all right time for the grand finale adding the GPU to your motherboard and that's the last piece basically that we are adding and here again just make sure to supply the GPU with power and then mount it on a PCI slot on your motherboard and that's more or less it all right time to wrap things up and close the case yeah not much more to say after we have installed all the components now it's time to close our case and start setting up the workstation all right now that we have finished as sampling our deep learning workstation it's time to install all the necessary software so we can start running our deep learning models all right for this we first need to download an Ubuntu distribution by visiting ubuntu.com download and here you can click on the download U desktop version so here we go and then I will just download the first version that shown here which is the latest LTS version of ubun 2 and just click the download button here then we also need to download bolina Atcher which allows you to flash OS images to SD cards or USB drives safely and easily which is perfect and this software is open source you can also see it supports Linux Mac OS and windows and for doing so I would recommend you to just click on this download link visit their website and already download the ater application just by clicking this button and then downloading the application by clicking clicking the download button here for your system and I will go with me OS then you can see I have downloaded both files and I will open the bolina eter application by double clicking it dragging it to the application folder and then I press command space ater click this application and now I can then select Flash from file which we already saw in the Uber to OS that's this is where I saved the ISO file the ubu OS ISO file and then I need to select the target which is an USB drive or an SD card and here I'm selecting this device and yeah that's basically it now we can press the flash button and that's all it takes to make our USB drive or SD card bootable all right and now it's time to boot the workstation for the first time a very exciting moment so all we need is a monitor and a keyboard optionally a mouse and obviously also our boot Drive which is the USB drive or SD card and with that we can install Ubuntu on the workstation all right now let's first switch on the power button on the PSU then hit the power button on our workstation then make sure to plug in your bootable OS drive and then hit the enter button to start installing Ubuntu and then I fast fored the process of installing Ubuntu which took me maybe like 10 to 15 minutes in real time then I decided to use Lambda stack to install all drivers that are necessary to run AI models using the GPU and Lambda stack makes it super easy for you because you just need to run one single command and for this I just open the Lambda website scroll down copied the command open the terminal in yunu and then just pasted the command into the terminal and just pressed enter then I had to type in my password and and that's all it took to install all necessary drivers to run AI models on your GPU once the installation process is done which took around 7 to 10 minutes for me we just need to reboot our system and this can be done by typing in P sudo reboot to verify that everything worked out we can type in Nvidia minus SMI to check if the NVIDIA drivers got installed properly and if our GPU gets listed in the driver overview then to double check we can open the python interpreter by typing Python and pressing enter then we import torch to make sure if the pytorch installation also worked and then we can type in torch. cuder do is available to check if an GPU is available and we can see it's true so our GPU got detected and then we can also type in torch cooter. get device uh get device name to see if uh to just see the name of our GPU and this the one that I installed so that's perfect and now I can use the GPU to run AI models which is very exciting all right guys and that's it for today's video I hope you learned something new I hope you enjoyed watching it and I'm really excited to now get going with a new workstation to try some new models to maybe fine-tune certain models or even train new models because it allows me to do so much more and yeah if you have any ideas of what I should look into just let me know in the comments and I'm looking forward to see you in the next video and until then have a good time bye-bye

Channel: Martin Thissen

Views: 5,836

Keywords:

Id: rCmvXCtY7eA

Length: 22min 25sec (1345 seconds)

Published: Tue Dec 05 2023