Run Google Gemma 2B and 7B on your computer for free (CPU and GPU)

hello everyone welcome to this new video today I wanted to do a quick and short video and show you how to run GMA on your local machine so what is gimma first Jimma is this new model that Google released few hours ago so it it was released today it use the same architecture as jimite but it is actually the state of the art open source model so it's better than mistal uh it is better than Lama and there's like um quick shot here showing the performance of the 7B compared to the Lama 2 7B and 13B so the model should be extremely good um the good part is that Google just integrated in a lot of other Frameworks like py torch a tensor flow and it come in two different size you have the two uh billions parameter size and the 7 billion parameter size so uh the two billion parameter size you can actually run it in your computer that's a nice part um the 7 billion as well I if it is as good as the Benchmark say we will see uh by testing directly using uh local tool and open source tool to run this model so let's get started um first the tool that we will use are yeah we will use this um this uh open G GUI and we will use also ama ama is like a tool that help you to run uh open source language model on your local machine and you will see that this extremely simple to use this tool to run any open source model withraw uh dips there's a lot of upcoming video that will show you how to have your own copilot locally how to have your own AI St locally but let's first start with this GMA thing okay um yeah so GMI are online right now but the problem is to use it with you need at least this version this version is online since 2 hours ago I think yeah you can see here this 1. 0.126 is so you need to pull the image and create the image again if you had it before otherwise I'll just show you the the same process so you just need uh the the docker file the do compost file and then you're good to go so first thing you have to do is go to web UI uh and uh go here this is the CH GPT like uh interface for open source models um yeah so you can just go on this repository there's a lot of Docker compost file here you have this compos for the API you can use this to plug to any other tool like continue or code gpg that will integrate into vs code or something else and U yeah you have the GPU for GPU support you have um the do compos normally so we will you we will use some of these files right um so we'll just use the do compos fight to launch Andi and we use the GPU version to launch AMA uh using our local dpu because I have um an Nvidia G gForce LTX here so let's start by just putting this image I already have it in the repository here I have the GPU the data API and the compos so let just open the terminal in this repository and the thing we do is pretty simple we just do a doer compose uh the file that's doer compos I want to run into GPU version things the GPU The jpu The J yeah and then we say uh that D build ah yes yes we up there before the D okay um um yeah I had this eror before so I just have to um just run the normal version the to compose the cham oh sure this is because in the GPU file that you can see here there's a count this count here and this count should be something of this CL variable thing anyway um yeah you just Poli this version um and then when you will run the the GPU version it will work so let first let it download the model the the the software and everything and then we'll see that later okay that's perfect so if you go here we will see I think uh I had the web g i me before cheama here and if I check right we have the letter version that's perfect um okay but uh we have the doer compos that run in and if you go here hopefully it works yes we have an account no the email and password okay uh I just had to create something quickly okay perfect so we are here on this admin panel somehow and uh to pull the model is extremely simple just come here go to model go to click here and then uh it will bring us here we can search for the model that we want here but we want so say this model here and then we need to copy the name this this name here exactly with the tag yes and then paste it here and now Lo oh perfect so it's done come to select model we select our model and then normally it should be possible to chat with the model right now yeah it's perfect so it's working great uh this is a mod you can play around with it you can ask whatever you want and that's it however you want so uh that's perfect and you you can see here that we have theama here we have this API where some kind of API that looks like TB API where we can ask questions and thing um and uh this I think this actually using the CPU but uh the GPU part we'll talk about it in coming video I have a lot of video coming out in the channel we will see how to use this tool to build a complete local well functioning AI stack and a lot of other things so if you're interested you can subscribe to the channel and if you have any questions leave them in the comment below

Channel: LTPhen von Ulife

Views: 6,162

Keywords:

Id: 9KI1gSTDCCg

Length: 7min 2sec (422 seconds)

Published: Wed Feb 21 2024