MemGPT + Open-Source Models Tutorial 🔥 Insane Power

the team behind mgpt have been hard at work since I published my research paper review and intro tutorial on Friday they've already released local llm support and an integration into Auto GPT today I'm going to show you how to use an open-source local model with mgpt and in the next video coming soon I'm going to show you how to use autogen with mgpt so let's go so I'm going to be doing this on run pod because I can't run inference and record video at the same time without my computer absolutely hating me but but you can do this the same exact way on your local machine what we're going to be using is text generation web UI to expose an API based on the open source model we're choosing but if you do want to run it on runpod here's a few steps to do so you're going to click on secure Cloud you're going to scroll down you're going to select a GPU I'm going to use this RTX a6000 let's click deploy I don't need to change anything but if we click customize deployment right here 78 860 that's the interface and 5000 is the port for the API so we'll just leave that the same and then we click continue and deploy and so this first step we're just going to be downloading the model and getting it set up and the API exposed so mgpt can use it once we're here we're going to wait for it to fully load then we click connect and then we click connect to http service and we're going to be using Port 7860 that's the interface because the first thing we need to do is actually download the model now the model we're going to be using today is Dolphin 2.0 mistol 7B it is a smallish model so I'm going to go ahead and click copy but I spoke to the authors and they said that the aerob boros prompt template tends to work best but for the purposes of just showing how to get it done I'm going to use the smaller model switch over to the model tab in text generation web UI we're going to paste that model card info and then we're going to click download and again all of this is the same exact thing you would do if you were installing it on your local machine I have a video going over how to install text generation web UI on your computer I'll drop that link in the description below and I also want to say a quick thank you to Charles and Vivien two of the authors from mgpt for helping me figure out a few remaining bugs this stuff is Cutting Edge this is all completely new very raw there's going to be lots of bugs but if you're watching this video and you're following along and getting this working you are on the absolute Cutting Edge of artificial intelligence but with it comes a little Fric okay it's done says done right there now what we're going to do is we're going to use the model loader Transformers cuz this is an unquantized model we're going to click this little refresh button which will load up the model that we just downloaded now it still has that bug where it selects the model but if I click on it it still selects none so you have to click none and then switch back to the new model once we do that we're going to click this load button and it's going to load the model into memory okay successfully loaded so one quick thing I want to show you is Click over to the session tab and last time when we were using runpod to power autogen with a local model I enabled this open AI flag right here and then I applied extensions we are not going to be doing that this time we're going to be using the default API provided by text generation web UI and so we're pretty much done next we're going to come up here to the URL and we're going to copy the URL next we're going to follow the same steps to get mgpt installed as we did on Friday so get clone and then the GitHub UR L hit enter and by the way the authors already put together a mgpt module so you don't technically need to clone the repository but I like doing that I feel like I have a little bit more control but if I were to use mgpt in a production level project I wouldn't do that I would just use the module and install it now we're going to change directory into mgpt CD mgpt hit enter next we're going to run the command export open aore aior base all capitals equals and then then we're going to put the URL of the runp Pod instance and we're going to change one thing rather than 7860 as the port we're going to write 5,000 and that's going to be the API and I'm going to remove that trailing slash and that's it hit enter and that's going to tell mgpt to use this URL as the API endpoint instead of open AI next we're going to set back endore type to web UI just like that export backend type web UI enter uh actually I forgot to do one thing we actually have to install the requirements so let's go ahead and spin up a cond environment and we don't need to change anything else everything else should work the ordering didn't matter so far so I'm going to say condac create DN Auto mem GPT python equals 3.1 1.3 hit enter now I already have an environment named that and I want to replace it okay done now we're going to activate the environment cond to activate autom mgpt and now we install the requirements so pip install dasr requirements.txt okay we're done all right now we just just have to load it up so Python 3 main.py D- noore verify and that nocore verify is what's going to allow us to run this local model hit enter so it says found save config file but if you're running this for the first time you won't get that and I'm going to say no cuz I'm going to set it up from scratch now this is a little bug it isn't giving me the option of selecting my local model but that's okay we're just going to select GPT 4 for now then I'm going to select Sam as the Persona and then I'm going to use the basic user and then I don't want to pre reload anything I'm just going to show you this working I'm not actually going to go into a specific use case okay then we hit enter to begin and there we go it is hitting our new runpod opsource API endpoint and that's it now let's say my name is Matt save that in your memory enter and let's make sure it works and I still need to put a deep dive together on how to really get the most out of mem GPT and I'm planning on doing that if you want me to prioritize that video let me know in the comments I have a bunch of videos videos coming and I don't know which ones to record first so if you tell me in the comments I'll try to prioritize those videos and one thing to note is they have multi-line support now so although I hit enter it's not actually done we have to hit Escape first and then enter just like that so now it's thinking all right user's name just got saved so there we go and I'll say What's My Name all right there we go your name is Matt perfect so it is using a local model absolutely amazing and that's going to be it for today right after this video I'm going to record one on on getting autogen set up with mgbt and that's going to be awesome so make sure you like this video make sure you subscribe to my channel if you're interested in seeing that video thanks see you in the next one

Channel: Matthew Berman

Views: 44,981

Keywords: memgpt, memgpt open source, open source llm, llm, ai, artificial intelligence, memgpt local, local memgpt, llm infinite memory, chatgpt

Id: QCdQe8CdWV0

Length: 6min 40sec (400 seconds)

Published: Fri Oct 27 2023