The Secret to NVIDIA GPU Success. What AMD, Tesla and Cerebras Have to Offer?

the market for crypto and gaming GPUs has cooled  anyway now everyone wants a GPU more than ever   before at the same time the supply struggles to  keep up with this HIGH demand so what's happening? ever since ChatGPT took off every company is  trying to deploy AI in their workflow and we   are talking about thousands and thousands  of companies here this obviously requires   a lot of computing resources a lot of gpus this  explains why GPU sales and Nvidia stock are Sky   High in an interview Sam Altman discussed open AI  struggles with GPU limitations they simply can't   get enough gpus creating charge GPT required tens  of thousands of gpus to train it and thousands   more to continuously run it now open AI has to  significantly scale up their data centers to   exploit multi-modal models and it seems that even  open AI can get enough gpus and this appears to   be the bottleneck for scaling their AI system this  happened due to many large companies placing huge   GPU orders and limited supply for example meta  and Microsoft they have hundreds of thousands   of gpus and at the moment they are planning to  double their capacities Tesla is striving to be   one of the leading AI companies and now they are  going to expand their AI capacity by a factor of   10. as you may know Tesla is building their own  AI accelerator in-house called Dojo but DOJO is   still a work in progress in the moment Tesla  is still employing Nvidia GP racks they are   purchasing thousands of Nvidia gpus for Tesla and  also many thousands for the new X.AI company but   that's just the part of the story in addition  to that Chinese companies are trying to buy as   many gpus as possible already in March this year  Nvidia stopped exporting their high-end a100 and   h100 gpus to China instead selling the reduced  capability versions that comply with U.S law   still China is stocking up on gpus and they have  reportedly ordered over 1 billion dollars worth of   Nvidia gpus this rash the stock up on gpus has  cost Market to go crazy and in video sales and   this reached unprecedented levels at the moment  they are shipping half a million gpus each quarter   but they are sold out till the middle of next  year right now Nvidia owns roughly 90 percent of   the GPU market and they dominate the market not  only due to their high performance Hardware but   also due to their proprietary software I would  even call it Nvidia software mode Let's Take a   Look Back Nvidia initially build gpus for gaming  and people only started to use it for Math and   Science later on over the years Nvidia evolved  particularly in the field of AI and machine   learning one of the key factors that enabled this  shift was the development of their CUDA platform   basically it is a software stack which used to  run parallel workloads between many many gpus   it was created back in 2006 and it was the first  commercial solution for running high performance   Computing on gpus basically they extended the  C language with some data parallel constructs   and this was the foundation of the modern Cuda  platform then could improved over the years more   or less it was happening in a lock step with the  development of their GPU Hardware this enabled   Nvidia to get into the new markets Beyond graphic  and gaming at the same time they heavily invested   in developing hardware for deep learning like  their dancer course now with of generative AI GPU   have become more crucial than ever and Nvidia  is benefiting from their long-term strategy   now all the companies are placing huge  huge orders for the new h100 GPU from   Nvidia so why does everyone wants this GPU  this example is a 10 million dollar server   and so 10 million dollars gets you nearly a  thousand CPU servers and to train to process   this large language model takes 11 gigawatt  hours 11 gigawatt hours so for 10 million   dollars you buy 48 GPU servers it only consumes  3.2 gigawatt hours and 44 times the performance   let me just show it to you one more time this  is before and this is after and this is like if your goal if your goal is to get the work done   and this is the work you want to get done ISO  work okay this is ISO work all right look at this you see him that's why I say trillion dollars  so compare it to a 100 GPU the h100 GPU delivers   roughly three times better performance part  of this performance comes from the improved   technology it's fabricated by tsmc in a custom  version of tsncs and for process node this is   actually an improved version of N5 with better  power efficiency speed and transistor density   the rest of the Improvement comes from the  updated tensor cores and a new Transformer   engine let's look at the die shot as you can  see this is a cheap LED design there is a GPU   in the center which is surrounded by six High  bandless memories memory and the compute tiles   are packaged together using tsmcs chip on wafer on  substrate packaging technology shortly it's called   c-o-w-oos packaging technology they integrated on  a waveform on the top of the passive interposer   this allows to make very short connections to  the memory which gives us low latency so super   fast cheap the cheap communication this packaging  is particularly beneficial for AI chips because   all the AI accelerators are using high bandwidth  memory and to pack this I bandwidth memory with   a Computing tile we need this 2.5 g packaging  technology from tsmc obviously the AI Market is   set to grow and the demand for NVIDIA gpus will  grow as well Nvidia is preparing for this and   boosting their tsmc orders and tsmc reportedly was  not able to cope with this high demand I mean tsmc   can keep up in terms of fabrication of wafers in  phone nanometer process but the co-wis packaging   capacity of the Fabs is definitely limited they  simply have not expected to get so many orders   so quickly Nvidia is now trying to get more  Wafers each month and tsmc is opening another   app to fulfill these orders you see GPU demand is  exploding and this complex supply chain just can't   keep up and this results in one of the bottlenecks  for AI development the situation got even more   attention after the recent presentation of Jensen  Huang so this is the dgx gh200 it is one giant GPU   and it will be used to train the most advanced  large language models over the next few years it   built around Nvidia latest Grace Hopper super chip  this is basically an accelerated processor Gray's   CPU plus a hopper GPU sharing a giant memory now  how it scales eight of such Super Chips connected   with Envy link to form one pot then the connected  32 Parts together into Grace Hopper supercomputer   which delivers one extra flop of compute but from  the software side it seemed like one gigantic   GPU sharing 144 terabytes of memory I would like  just to mention that this integration of the pot   into the software stack the seamless integration  is actually a big deal and Nvidia made a great   job here in general the main problem of all the  custom AI Hardware is a software building this   entire software stack is hard particularly for the  new hardware like in case of Deutsche that's why   up to now we have not seen a white usage of the  custom AI chips from various startups I'm sure you   will agree with me and VGA gpus are amazing but  very expensive now with this increased demand the   new h100 GPU cost about forty thousand dollars  however the main problem is not even the price   but the opportunity to buy it not everyone will  get it so now a good question is if there is an   alternative to Nvidia gpus in particularly  for AI application I would say yes and no   actually there are several companies first of all  Tesla it's Dojo supercomputer Tesla is willing to   invest another one billion dollars on the further  development of Dojo anyway it's not for everyone   you can just order a dojo Elon mentioned that  he will be considering giving an access but   it's not clear to whom and on each terms this  dodge story I think these are a separate video   then there is Google and it's getting even more  interesting Google has their own in-house designed   AI accelerator TPU Google loves this chip and  they're using it for internal AI applications   such as training Google's Bart and also deepmind  used it for training of their amazing models Alpha   Dev Alpha dancer Alpha zero and alphago as much as  Google loves to be yours Nvidia gpus have become   a sort of necessity because of its performance  scalability and software stack which is used by   the most of their customers of course now Google  is also stocking up on Nvidia gpus some months ago   they announced a new A3 supercomputer which is  made of 26 000 Nvidia h100 gpus and this gives   Google about 26 extra flops of AI performance the  most important that it's versatile so it can be   used in a wide range of AI applications and it  will be used by Google customers to train their   large language models in the cloud meaning they  will be renting Computing time in the cloud Google   itself will be using it to train their latest  large language models Palm tool which is powering   the Google's part I would say a sound alternative  to Nvidia gpus is cerebrus Wafer skill engine   cerebrus is a startup from California and they are  building a huge AI chip which occupies entire 300   millimeter wafer it's practically the largest chip  ever built and all of this area is dedicated to   AI cores which share a giant memory when we have  such a huge processor the data has never to go off   the die and here we save a lot of time and power  what's really interesting just like with nvidia's   Cuda platform cerebras built a custom compiler and  this compiler plays a crucial role it's basically   realizing the code between many cars and is doing  the mapping to the physical parts of the Chip and   there is a big difference between paralyzing this  work between many GPU cores or using a single huge   chip rate if we compare cerebral Surfer scale  to engine to Envision GPU they have roughly 100   times more cores thousand times more memory  and much much higher memory bandwidth than   a GPU what makes a difference here that cerebris  invented a new technique called weight streaming   so they can keep memory off cheap separated from  the wafer with that cerebras can support models   with any number of parameters to run on a single  cluster while gpus usually have a fixed memory   so if you want to work with a larger data set  you need to buy more gpus and then distribute   the work over multiple gpus and this explodes  the complexity on the software side as well   looking at the numbers published by cerebras  the costs and the performance for training of   large language models it seems comparable to  the Nvidia a100 GPU recently cerebrus trained   seven GPT based large language model us on  their Andromeda AI supercomputer those are   the models with up to 16 billion parameters and  actually they were one of the first companies who   managed to train large language models on their  own Hardware in a matter of weeks and also open   sourced the models and the results there was a  lot of excitement in the community about this   that they openly share the results at the moments  River's chips are running at the Argonne National   Lab and several other labs and also at Pittsburgh  super computer center at the same time they have   other large customers who they keep confidential  I already talked about this startup on my channel   and I think these guys got that chance to get a  piece of the pie Intel and AMD have competitive   chips as well for instance Intel Gaudi Chip have  achieved reasonable performance compared to Nvidia   h100 GPU on some of the training benchmarks in  general h100 GPU from Nvidia is at least three   times as fast as gauge 2. but other big hopes  for Geology stream the next intos AI chip which   will be taped out in five nanometer later this  year AMD also has a GPU to rival Nvidia sh-100   so with that I am super excited to show  you for the very first time Mi 300X at the moment Nvidia has a clear advantage of  our competitors not only with the hardware but   mostly with their software stack Coulda AMD is  desperately trying to catch up investing a lot   of time and money in their software stack  called rockcom and in contrast to Nvidia   they are betting on an open source  solution and I'm a huge fan of this   Rock'em is a complete set of libraries runtime  compilers and tools need to develop run and   tune AI models and algorithms a significant  portion of Rock'em stack is actually open   and I know that now there is a way to run Cuda  on AMD gpus and if they can manage to do it with   minimal changes in the code these will be the  game changer for AMD we will for sure see a lot   of competition in this field over the next years  and this is definitely exciting let me know what   you think in the comments below because I am  pretty sure that one of the companies which I   mentioned today will manage to get a piece of  the pie in this generative AI boom as you may   have guessed a big part of my portfolio is in  tech companies that's because of my experience   and my background I understand the potential of  various Technologies for example I invested quite   some money in Nvidia stock back when it was at  hundred dollars and now it's more than tripled   however most experts agree that diversifying  portfolio is useful and needed to reduce the   risks according to array Intel almost 90 percent  of financial advisors have alternative Investments   and now there is one specific alternative Fine Art  investment this video is sponsored by Masterworks   and art investment platform they reportedly  sold over 45 million dollars worth of artwork   delivering the net process to their investors  Masterworks offers the art from Legends like   Picasso and Banksy and you don't need Millions  to invest and then the resell the works with   any potential profits after Fierce distributed  according to Masterworks to date every single one   of Masterwork sales delivered the positive returns  to the investors and paintings on Masterworks site   can sell out in hours but you can get a Priority  Access with the link below thank you Masterworks   for sponsoring this video now you might like to  check out the video about Dojo AI accelerator   which is Tesla's building right now it's really  architecturally interesting I will link it below   at the same time I have a lot of fun videos  about other AI accelerators like cerebras I   have a separate video and also about the graph  core bow chip I also will be linking it below   let me know what you think about the dominance of  Nvidia in the field in the comments below thanks   for watching I wish you a beautiful summer  and I will see you in the next video ciao [Music] foreign [Music]

Channel: Anastasi In Tech

Views: 76,956

Keywords: NVIDIA GPUs, GPU boom, NVIDIA BREAKTHROUGH, NVIDIA, ticker symbol you, NVIDIA GPU alternatives

Id: oREXJ4va3F4

Length: 19min 14sec (1154 seconds)

Published: Wed Jul 26 2023