How To Connect Local LLMs to CrewAI [Ollama, Llama2, Mistral]

hey guys in today's video I'm going to be teaching you how you can run crew AI completely for free using olama so that you don't rack up a huge open AI Bill like I just did so by the end of this video you're going to learn how to run llms locally on your machine like llama 2 and mistol and you're going to learn how to connect these lolms to your crew AI agent so that you can run crew AI completely for free and don't worry whether you're beginner or Advanced you're going to get a ton of value through this video because I'm going to be walking you guys through everything step by step and I'm also going to be giving away all the source code for this video completely for free all you have to do is just click the link down description below and if you want to see more content just like this it' mean a lot to me if you guys could also hit that like And subscribe button but enough of that let's go ahead and dive into the rest of the video so here are the three major parts of this video first we're going to do a quick recap of the four technologies that you're going to be using in this video just to make sure we're on the same page second I'm going to show you how to set up and run llama 2 and mistol on your machine using o and third we're going to connect crew aai to AMA so that you guys can see everything in action so let's go ahead and dive into the first part where we're going to cover the four core technologies that we're going to use in this tutorial so the first technology that you're going to be using in this tutorial is going to be olama and olama is a tool that makes it super easy to modify download and run lolms locally on your machine when you visit the models page on this website you'll actually be able to see that there are a ton of different llms that you can run using olom and actually in this tutorial we're going to be downloading modifying and running these two different llms that you see right here the second technology that we're going to be using in this tutorial is going to be llama 2 now llama 2 is an llm that was released by meta and as you can see this model was trained on two trillion tokens and what I think is super important for you to know as a developer is that there are three different models and each one of these models comes with different Ram requirements so in this tutorial I'm basically going to be using the 7B model 7 billion model which requires about 8 GB of RAM so that's what we're going to be using so if you have a beer machine you can go up to the 70 billion model which requires up to 64 GB of RAM but don't worry AMA will figure out what your system has and use the right model based on your system requirements and something else that's super important for you to know is that llama 2 was trained on a larger data set compared to mistel however when it comes to actually handling task I've noticed that llama 2 doesn't always perform as well as mistal so that's just something to keep in the back of your mind when you're trying to pick between which llm to use llama 2 or mistl but we'll talk about more of this later on and the third technology that we're going to be using in this tutorial is going to be mistol now mistel is another large language model that we can run locally on our computers and this thing packs a punch as you can see mistol is a 7.3 billion parameter model and you can see that it actually outperforms the Lama 2 13 billion parameter model and the Llama 134 billion parameter model so as you can see this thing packs a punch and you're going to be able to see it later off in this tutorial and the fourth technology that we're going to be using in this tutorial is crew Ai and simply put crew AI is a framework that allows us to create and manage AI agents that work towards solving complicated tasks such as analyzing stocks planning trips writing newsletters and much more so if you're looking to learn specifically more about crew AI I actually just released two tutorials that you'll definitely want to check out the first video is a complete crash course for beginners on crew AI where you'll cover the basics of working with agents task and tools and the second video is an updated video on crew AI where we cover some of the newly released features of crew AI where you'll learn how to build more reliable and faster Crews so those are the four technologies that we're going to be working with this in this tutorial so let's go ahead and dive in into setting up our local llms on our machines using AMA so the first thing we need to do so that we can run our large language models locally on our machine is to head over to ama.com so that we can download amaama so what you'll need to do is just click that download button and what it'll do is it'll take about a few seconds to download the package in application cuz it's about 400 megabytes but once it's finished downloading what you'll need to do is just double click on the application and it's going to ask you to go ahead and move AMA over to your applications folder so that's what I'm going to go ahead and do real fast now this will take a few seconds cuz I've already done it in the past but you know that's exactly what will happen on your screen once it moves over to the application folder you'll see this little intro pop up saying hey welcome to so you're just going to click next and then you're going to be prompted to install the olama to your command line so you're just going to click install so in my case it might ask you to go ahead and put in your password so do that real fast and then once it's done you'll actually see the AMA icon in the top of your computer so that's how you know it's working and then finally it's going to say hey let's go ahead and run your first model but we're going to ignore that for now because we're going to open up our terminal and in my case I'm going to use warp as my terminal we're actually going to start actually testing out AMA to make sure it works so if I type in ama right now I should actually see that it runs and what I can do is to check to see which models I have you can actually type in O Lama list and this will actually list all of your different models and large language models you have installed on your computer now yours will be empty at first because you hav installed anything yet but what I'm going to do is just go ahead and show you how you'll be able to download your first llm so what you'll do is you'll type in O llama and then what you'll do is type in pull and then you'll just type in llama 2 now how did I know to do this well if you head back over to.com and click on the model section and click on the model you want to download you'll see in the top right hand corner it'll say hey this is basically the command you'll do to run it but we're just trying to pull it right now so I'll go ahead and pull lud down and this should take um you know it's a big package almost 4 gigb so it'll take a minute to actually download but once it's actually finished downloading you can actually run it so we'll just now type in O Lama run and we'll type in llama 2 fantastic so this should say oh I misspelled it and this should actually go ahead and actually run the llm locally on your computer so you can just say you know tell me to dad jokes just to make sure that's it's running and this will take a few seconds and yeah you can see it works just like chachu BT normally would on your computer so that's awesome so now that we've went ahead and installed oama what we're going to do next is actually go ahead and start configuring our llms to get them ready to use with crew AI so before we can start connecting our large language models over to crew AI we need to do a little bit of configurations to our large language models and don't worry this is actually super simple to do what we're going to do is create a thing called a model file and let me show you what this looks like real fast it's basically just a set of instructions that say Hey I want a specific large language model to use a few specific parameters to make sure that everything runs smoothly now in our case this is actually going to be super straightforward CU if you head back over here this guide actually tells us exactly what we need to do in our case it's saying hey I need you to add the result word as a key stop word inside of our parameters now let me actually show you what this looks like in action so that you you know this will all make sense so what I've done so far is I'm actually over here inside of our GitHub code that I actually have for this project and if you remember from earlier you can actually download all the source code for this video completely for free all you need to do is just go click the link down in the description and you can have access to all this code so in our case what we're doing is we want to make a llama 2 model file and all we're going to do is say hey for my model I want to grab llama 2 and I want to add two different parameters in our case I want to like set the temperature that's just you know how consistent or creative the results are going to be and then I want to add a stop keyword so that our agents know to stop producing and executing on a specific task whenever it gets and sees that word right there result and outside that I'm just going to say hey you're starting with a blank slate for you know the initiating and starting of the conversation with the LM pretty much you can ignore that if that didn't make sense Okay cool so now that you know what it looks like to actually customize an llm with a the model file let me show you what we need to do next well if you head back over to the instructions what it says to do is just go and basically run two specific scripts the first one is just hey using AMA I need to create a new model using this specific model file and after that you can actually go off and run that model file and what I've done to make your life easier is I went ahead and actually made that script for you so what we're going to do is we are going to pull down that model in our case llama 2 and then we are going to create our new crew AI llama model using that model file so I'm going to go ahead and run it and show you what it does so all we're going to do is just type in we're just going to run this code and if this is your first time running it actually you might have to do something like this it's called chod plus X and all that does is it actually says like hey turn this into an executable but what we're going to do is just run our setup file and this is our create llama 2 llama 2 fantastic so that should make it executable and now we can run it so I'm going to do setup create llama 2 now this should go start running it'll pull down the Llama 2 model cuz that's what we said to do and then from there what it's going to do is actually go ahead and create our new file and I was actually in the wrong directory so that's why I got that error so I'm actually just going to change over to my setup directory which you can see right here so that I'm working from the same file path so that was the reason I got the erir but if I run it again what you'll notice is let's change this fantastic so if I run it again it'll actually now reull it down and now it knows how to point to the right model file so now if I run oama list you'll now see that I have my new crew AI llama 2 latest and it was made 4 seconds ago and now we're going to go ahead and do the same thing to go off and create our mistal large language model that's custom for crew AI to so once again all we need to do is just type in you know chod to make our script executable cuz we want this script to be executable and then so let's do that chod plus X and then we need to tell which file in our case it's the create mystal file fantastic so now that's executable and let's make sure our mystal file looks great I need to rename that real fast mistal fantastic so this is basically the same one by the way we just added the word stop to this file now I can go ahead and run this one but instead I'm just going to say create mistal fantastic so that should go off and create our new large language model but it's going to make sure it's using the new mistal large language model and it's going to specify it for our crew so what we're going to do now if I run AMA list again you'll now notice that we also had uh oh llama you'll now notice that we also had have a mistal large language model that we can start using in our cruise so that everything will work smoothly so I know that was a little bit of a lot but once again don't worry if you had any issues actually feel free just drop a comment down below and I'll do my best to help you guys out as quickly as possible and feel free once again to go grab all the source code for this video so that you can actually just run these scripts over here and that way everything will work smoothly okay cool well now that we have our large language model set up and ready for crew AI let's actually start diving in to connecting these over to our simple example and actually get everything running all right guys so I'm super excited to go ahead and show you our first example where we're going to connect our local language model over to a simple crew AI example now what we're going to do is I just want to show you a quick overview of the crew that we're building and then show you a quick output and then I'm going to show you how we got everything running and then we're going to repeat this for the advanced example in just a second okay so what I have done is I have actually clone down a example that was provided in the crew AI examples list this is amazing repository that actually shows you guys a few different ways that you can use crew AI so what I've done is I've pulled down the markdown validator and what you'll notice is this markdown validator basically allows you to use a crew to go and analyze a markdown file and it will provide recommended changes that you need to make to the document so it's super straightforward and what you'll notice is that like in the main.py file where we're actually you know running the crew there's really only two things that happen there's one agent and then there's one task and that's pretty much just how the entire thing runs that one task is the only thing that's executed and it just you know it Returns the final feedback that you need to make for your markdown file so what I've done is I've actually went ahead and run this for you guys using the Llama 2 that we just made it the custom one that works with crei so I just ran it and I just want to show you what the final output looks like cuz I think you guys are going to be super impressed so this took about h a few minutes to run you know running some of the local language models actually takes a little bit longer but hey it's free so that's awesome but what I did is I said hey I want you to analyze this markdown file that you see right here and provide me feedback and what you'll notice is it actually went ahead and use tools and everything to find that there was actually five errors in the document and then it tells you feedback on what you need to change in order to make this markdown file valid so it's like hey add some brackets over here add some new lines and so forth and so forth so like I said I thought this was a pretty awesome example to show you guys that like hey it actually works okay cool well now that we've like seen that these large language models that are running locally work let me show you what I had to do to get this language model running so what I'm going to do is I'm just going to go ahead and clear all this out so we're starting from a blank slate and I want to go ahead and actually show you guys what I did so what I did first is I changed my directory over to the simple example folder so that I was set up right here and just to give you guys a quick overview we'll cover a lot more of this in that crew aai Basics project tutorial that I created for you guys but just in a nutshell what I did is I in installed all the dependencies that were in my pi project. tomomo and this is super easy to do all you need to do is just type in you know poetry install D- no root and when you do that it'll go ahead and grab all these dependencies for you and it'll make a python environment that you can use to actually go ahead and use and that's super simple you would just type in poetry shell and whenever you do that you'll basically get the same thing right here where it's like oh yep you were using the markdown validation crew cool so that's what I did now let's go ahead and dive into how I actually got our local language model working with this crew well if you actually head over to the documents that crew AI provides you'll notice that it's very easy to set up what you need to do is actually update your environment variables to point to your local host that's actually running AMA and AMA actually basically sets up a server that will help facilitate all the communications between your crew and the local language models that you just set up so it's up to you to update your environment file to point to this local host and then provide your model name so let's go ahead and look at our project so you can see this working in real life so what you'll notice is if you head over to the EnV file or in your case there'll probably be a EnV I'll go ahead and create for you guys so you'll have it but you'll have a EnV example file and you'll just clone this to you know rename it to be a EnV but what you'll do and you'll notice is we copied exactly what they said we set up our base URL to point to our Local Host where AMA is running and then outside of that we pointed our opena model name to be the newly configured large language model that we just set up so that's how we were able to get it to run and it's actually very simple whenever you head back over to our main.py file where we set up our agents you'll notice that we did not set up an llm where we actually pointed to you know hey this is the large language model I want you to use by setting up our environment variables basically any new open AI chat large language model that we set up we actually just use this base configuration right here and that's how you can tell it's working and another thing that you can do just to like prove to you guys that's working if you actually go over to your root directory and go into oama what you'll notice inside of here is you actually have some logs so if I go into our logs folder let me go into our logs folder what you'll notice is that we have our server. logs and that's all the logs that are going to come from here so I'm just going to go ahead and check to see what's inside of here and our server logs and what you'll notice is yeah we have a ton of different chat requests that were just running so you know this was just literally just a few minutes ago when I was running it before recording this part of the video so yeah you can see there's a ton of stuff and it also just has some other general information about your computer as a whole but yeah so that's how you can actually get your crew to work with local language models the other thing that I want to do is just show you option number two so if you just go ahead actually going to just delete EMV file just to show you that this guys is working the other option that you can do if you want to be more explicit is you can actually create a large language model and you can say hey I want to create a new chat open Ai and then what you can do is actually provide your model name and your base URL this is basically the same information that was you know defined in our EMV file just a second ago but we can actually just go ahead and hard code in here now that we're importing everything what we can do inside of our agents is actually go ahead and point to it fantastic so now we're actually going to be using this large language model that we just defined so I'm going to go ahead and start running it again let's go ahead and CD back into our right file I'm actually just going to close out of this and open up a new one just cuz it's it's easier fantastic so now that we're back in our file let CD into our crew AI simple example and then I'm just going to run python Main and I'm just going to point to our read me file great so that's going to go ahead and kick off this should work ah I actually messed up I should have done was just you actually still need to say that your open AI key is in a that should work fantastic now it's going to go off and execute and I just want to pull up in a parallel task real fast just to show you guys that it's running in real time so if we go over to AMA once again we go over to our logs what we can do is cat oh that server file oops um sorry I'm struggling on the keyboard today what I can do is just once again cat that server log file and you'll see that things are happening in real time so this is 455 local time and if you look it is 455 so you can see that's actually running in real time cool so yeah it just running so those are the two ways that you can get your Crews to start connecting to your local language model that's been configured to work with crew AI so what we're going to do next is we're going to hop over to the more advanced example next and I'm going to provide you guys some more General tips and feedback that I have found when working with you know these local language models and we're also going to use mistal in the next example so let's go ahead and dive on over hey guys so welcome to the second example where we're going to connect up our crew AI to our local llm in this case we're going to be using mistol so let me give you a quick overview of this crew and what it's trying to do show you the outputs and then we'll actually dive into how we got everything set up okay so the first thing you'll notice is now we're in the advanced example folder and the synopsis or just quick summary of what we're doing is we're making a financial crew and the whole point of this financial crew is for us to be able to pass in a stock and then it's going to go off and research the internet find us everything it can about the stock and then return some you know hey this is our opinion on based on what you should do for that specific stock so I just ran this basically This Crew for Tesla and as you can see here's the final output that we got from running this crew using mistl locally and here's everything that you need to know so our agents came came to an answer where we say hey Tesla's got some strong growth here's some few things that we just found out about this company as we looked at you know their quarterly reports to the SEC and you know here's our summary where we say like hey you know we're cautiously optimistic about Tesla going forward so all around this is awesome work that I was able to do this this one took a little bit longer because as you can see we have a bunch of different agents going off and performing a bunch of different tasks we're looking on the internet so forth and so forth okay so once again I was able to grab this example once again from the crew AI examples link I'll have a link to this repo Down Below in the description so you can browse it for yourself but let me go ahead and show you actually what I had to do in this codebase to go ahead and get it to work with our mistal local LM so this one was once again pretty straightforward all I had to do let's just go ahead and open up a new terminal so you can see what I had to do so the first thing that I had to do was once again go ahead and install all the dependencies for this crew and what you'll notice is this one has a lot more you you know we're working with a lot of Finance data and SEC information so forth and so forth so what I had to do next Once I went ahead and you know did the normal poetry first off I had obviously C CD into this file so crew AI Advanced now once I'm in here I'm going to zoom out just to make the terminal more not so cut off for you guys but the next thing I did obviously same thing as before you know I did the Poetry install d d no roote to go grab everything no route and then from there what I did is I went ahead and actually just did poetry shell to go ahead and activate this crew Okay cool so once I had all that running what I was able to do I want to show you what I had to actually change in the codebase so the few major things is you'll notice the way that this file is structured it's very straightforward we are building a crew and as we're building a crew we need to import agents and we need to give those agents task and then we need to compile everything together in a crew okay well where are we passing in our llm well in this case where we're doing that is if you go over to the agents file what you'll notice is for each one of our different agents that we have created what we're doing is we are passing in which llm we would like to use and in this case what we're doing is just now using our crew Ai mistal llm and this one like I said basically exactly what you saw beforehand we're you know passing in the model passing in the base URL and in this case I actually just went ahead and Define API key basically all you have to do to avoid the error you saw in the last screen when we were working with the simple example is you just have to put string in here it just can't be empty we're not even going to be using it but it's just that's how you get around you know open AI giving you issues or sorry the chat open AI giving you issues so yeah this is literally all you have to do to go ahead and start working with your local llm but to actually get results that are meaningful what I have found is that you have to provide a lot more context in your task to make sure that they run so for example when you look at a task let me show you an example task so when you look at at like our research task what you'll notice is we're having to pass in our agent and our company okay so that's how we're going to make sure that when we're researching a company we're going to make sure we research for the right company well in the example that you get from this examples list you'll notice when you go look at theirs you'll notice that they only provide the company here and they were able to get away with that when setting up a crew because they were using open AI which has specifically chat gbt 4 which seems to have much larger cont text window meaning open AI is able to look at and understand a lot more data at one time I noticed whenever I was running this crew if I didn't put the company in every single task like I have set up here the crew would go off and start researching random stuff on the internet so the workaround that you need to keep in mind when working with these local llms that don't have as large of a context window you need to be very very specific in your task on what you want them to do because you know they're not going to remember things that were being being worked on a few seconds ago so that's just a tip that I'd recommend as you guys to go work on more advanced Crews and the other thing that I want to point out is as you get more involved with working with Crews you'll start working on you know newer features that are coming out with crew such as you'll you know you'll start to work with making processes that are hierarchical or you'll start working with making sure that your research task and all these other task that you're looking at you'll make sure that they're going to be asynchronous so it's important to note that you can't actually use some of these more advanced features when working with your local language models and the reason why is if you actually go over to what features are natively supported with each of the llms you'll notice trying to go down to open AI for you guys you'll notice with open AI everything is supported and by everything I mean you're able to do invoke new requests you're able to do them asynchronously you can stream back data so forth and so forth and you can't really do that using some of the local language models that we're working with such as like I said oama it really only allows for you to invoke a you know the local language model so I just wanted to provide that for you guys just you know as you go off and become experts at working with local language models I just want you to know that this is a current bottleneck of working with them but hey as time goes on I guarantee you that they're going to improve what ol can do So eventually I'm sure these will also become green check marks yeah that is everything for working with your Advanced example and if you have any questions feel free to drop a comment down below I'll be sure to get back to you guys as fast as possible so we can get you guys up and running and building your own local language models but that's a wrap for this video guys I hope youall learned a ton about crew aai o Lama llama 2 and mistro and if you did enjoy this video and want to see a bunch more AI related content just like this you'll definitely want to check out some of my other crew AI or fullstack tutorials that I have on this channel so want to definitely click out one of those videos that pops up on the screen right after that but enough of that I hope you guys have a great rest of your day and I can't wait to see yall in the next video see you [Music]

Channel: codewithbrandon

Views: 43,737

Keywords: ai agents, crew ai, crew ai tutorial, crewai langchain, crew agents, autonomous agents, autonomous ai agents, auto gen, autogen tutorial, autogen create ai agents, ai agent, autogen step by step guide, chatgpt prompts, llm tutorial, llama ai, ollama langchain, ollama tutorial, ollama api, ollama rag, ollama mistral, llama 2 tutorial, llama 2 local, llama 2 langchain, local llm python

Id: 0ai-L50VCYU

Length: 25min 6sec (1506 seconds)

Published: Tue Feb 27 2024