this is the fastest AI chip in the world: Groq explained

this is grock it's like chat gbt but insanely fast and this breakthrough might mean we're entering an entirely New Era for large language models we'll get to how it works in a moment but to demonstrate why this speed or more technically low latency is important let's listen to a call I made using AI with gbt 3.5 as the model hey there I'm looking to purchase a pressure cooker do you have any in stock uh I can check for you can you hold on a moment sure take your time while you're at it do you also have any shrapnel or nails for sale um I'll I'll check that too just hang on great thanks and if you're wondering what I need them for never you mind just focus on getting me the items it gets the job done but it feels unnatural and this is due to the latency the speed at which the AI agent can reply to the person on the call but what if we try the same demo using Grog hey there I'm calling to book a cleaning service for my pig Goose I was hoping to get her washed and looking her best do you offer any services like that not for pigs oh that's a shame I understand that pigs are not typically considered pet this low latency opens up entirely new possibilities for AI more on that in a moment so how did this breakthrough happen well Gro not to be confused with Elon Musk grock or grimes's grock toy is the brainchild of Jonathan Ross that's this guy Jonathan entered the chip industry while working on ads at google Jonathan overheard a team compl complaining that they didn't have enough compute so we asked for some budget from Google and started putting together a chip-based machine learning accelerator for them Jonathan and his team would go on to build a new chip the tensor Processing Unit within a year this chip would be deployed to Google's data centers but it was during this time that it became increasingly clear to Jonathan that a gap would emerge between companies that had nextg AI compute in companies that didn't so instead of investing all of his money in Nidia and and waiting it out on a beach to cash out he set out to build a chip that would be available to everyone and founded Gro and what Gro did was build a chip specifically designed to run inference for large language models this chip is a shocking 25 times faster and 20 times cheaper to run than chat GPT for comparison they call it the first language processing unit or lpu and it's this lpu that runs its model rather than gpus that are typically used to run AI models the lpu is a beast when it comes to running inference it can run AI models at shockingly fast speeds it's important to remember that unlike open ai's chat GPT grock isn't an AI model its powerful chip is designed for a specific purpose of running inference on large language models you might be wondering what inference means in the most simple terms AI inference is the AI using what it has learned to figure things out or make decisions for you during this inference phase the AI doesn't learn any new information it just applies the knowledge it's already acquired during its training phase to new data so every time you hit send on a message to chat gbt inference is being run over the model to get a response and when using Grog the time it takes to run this inference and get a response is almost instant it's also far cheaper to run and when using Gro the time it takes to run this inference and get a response is almost instant it's also far cheaper to run which increases the margins for companies like anthropic which we know are already being squeezed on margin and that opens up entirely new possibilities you might have heard of Air Canada recently losing a case in a small claims court because their AI chatbot told someone that they could claim a bereavement Fair after booking a flight with the speed of grock it allows chatbot makers to run additional verification steps in the background this means you could crosscheck responses with the same model or different models before even responding with an answer and this could potentially make the use of AI in the Enterprise much safer and more accurate in its responses without making the user wait for the chatbot to process many steps you can also create multiple reflection instructions for an AI agent so instead of singleshot responses that means the AI immediately responds to your question it can now Ponder and refine the answer before you even see the response the output is no slower than chat gbt but the answer can be much more aine when you get it now you could obviously do all of this before but it was too slow and too expensive to actually ship products like this in the real world but with grock's speed and affordability it's now possible if grock eventually becomes multimodal which I'm sure it will we could witness AI agents that can command our devices to execute tasks become affordable and practical very soon using vision and the speed of Gro a model could easily complete tasks on your computer it's super human speeds the rabbit R1 which is known to suffer huge latency issues or meta Rayband AI glasses might become much more useful thanks to near instant responses low latency and low cost cannot be underestimated and what's crazy is this is as expensive and slow as grock will ever be if model makers can improve AI models ability to follow instruction and new and improved multimodal models can be executed lightning fast with gro we might might be closer to truly impactful AI agents than we thought Brock may even pose a huge threat to open AI as models become more commoditized speed cost and margins will become the biggest considerations the chips of the future for both inference and training as we've seen from nvidia's success may just be the biggest winners Brock is truly impressive I encourage you to try it out for yourself and if you want to try multiple reflection or thinking steps with AI you can build your your own AI agents and experiment with grock on Sim theory. a link to both of these are in the description and I'll also leave links to the agents I used in this video as well if you like this video please consider subscribing liking and telling me your thoughts about grock in the comments and I'll see you in the next one wait what that sounds pretty Sinister I understand but I really need these items can you help me out with that uh yeah I'll I'll take a look for you now thanks I appreciate it let me know what you find mind

Channel: morethisdayinai

Views: 7,823

Keywords:

Id: kA0PG9ROG9Q

Length: 6min 30sec (390 seconds)

Published: Thu Feb 22 2024