How to run a private Chroma Vector Database locally in 5 mins!

hey there Timothy kbat founder of mlex labs and today I'm actually going to be doing an updated version of all of my chroma videos because they came out with some new improvements since my last tutorials I'm going to basically delist the other videos just so you can get to them for reference um and these will be the new videos just so you know let's look at the video today is November 21st 2023 obviously this Tech moves fast and chroma is a fast working team let's go to their GitHub figure out what the current version of chroma is that we're going to be working with and that is 0418 if we are way in the future and this version is different your mileage may vary but in general we're going to start with something simple so today I'm going to show you how do you run chroma it's going to take us five minutes right how do you just get it working on your local machine and also how do you use the API key persist the storage make it to where each time you need to update to the latest version of chroma you don't blow away 100,000 of your embeddings or anything like that so this is going to be easy fast secure and it's going to work with pretty much every other integration you can either build or work with let's get into it take us maybe about 5 10 minutes so you used to have to go to the GitHub clone it build it whatever that has been solved let's actually just pull in the image from Docker run it in Docker and call it a day so of course a requirement for this is you need Docker so here I am in Docker desktop if I go to images you'll see I have chroma DB chroma which is going to be the latest version and I already have that pulled in but to do this all you would need to do is open up a fresh new terminal let's make the text a little bit bigger so everybody can see and you would do Docker pull and it's chroma DB chroma you'll see that defaults to the latest tag it says that it's already existing and it turns out that actually there's an update that can be run so okay new version got released cool so we're going to wait wait for this to complete and then once we're done we'll see that the image is updated in Docker okay so the latest Docker version just got pulled in I actually just realized that the 4.18 package got published two hours ago so this is very very fresh video right now so you're getting the best your second question is okay well we got it pulled in how do we run it that's actually really easy but also we'll get into why that's maybe not the best idea to just jump into it the easiest way to run it is to just run Docker run and then you want to expose the port 8,000 it can be any host Port you want but you want your Target Port to be 8,000 cuz that's the port chroma runs on in the docker container and then we just would specify chroma DB chroma and what this is going to do is this is going to build a chroma instance in Docker for us that we can connect to on our local machine in order to basically start saving in beddings and using all of the power that chroma comes with okay so we can see that the server is now running and it's running internally in Docker on Port 8000 and since we connected the ports I should be able to go to localhost 8000 API V1 and we return a nan heartbeat which means that our chroma instance is finally online and we are good to go we can start saving eddings and doing all of that so let's save an embedding and then I'll show you why maybe the command we just ran is not actually the best command so here I'm in a local instance of anything llm because this is just an easy way for me to upload documents and PDFs and store them as embeddings so we're going to connect to chroma we're going to go to that HTTP SL localhost 8000 we have no API key or header we didn't set one I'll show you how to do that but we're going to click save we're now saved we're going to jump in go to new workspace call this chroma go in here I already have uploaded some docs to anything LM this is just for me if you have a script to upload embeddings use that same effect and there we are and so now if we are to go to slash collections we should have one single collection named chroma so we're good we have stored an embedding if you are to go into Docker and let's go to The Container running this Docker instance and we are to stop it so stopping a container is like pausing it um the data should be persistent as long as this container does not get destroyed if there is a new version .19 of chroma comes out the problem is you can't take the data that's in this instance and easily Port it to a new container which is as you can imagine a big problem you're stuck and pinned on this version because all of your data is within this container but let's just start it up again and as we look at the docker logs we just have to wait until the initial boot script completes and you'll see that the server will then say UV UV corn uvi corn I don't know how to pronounce it to be honest I I if someone knows put it in the comments and we're booted all right so let's revisit that same URL our data should still be there and it is we've refreshed it I'll keep refreshing it you can't tell but I've refreshed it let's say that chroma comes out with a brand new version and we want to take advantage of whatever the newest stuff is in fact actually they just released a new version that now has multimodality so how would you upgrade well if you did it the way that we just did it you can't so here is actually what you should do when you run Docker so that you can actually have your data go between containers that you spin up of chroma this is a much more reliable way of doing things so you don't wind up with your data isolated and fixed on a single chroma instance to run it with a persistent storage between boots is actually let's just make a temporary folder essentially or just a storage folder on our host machine so this is going to be where our data from the docker container gets written to so that basically no matter how many containers we spin up they can all point to the same thing no regardless of how many times we do it easiest way to do this is I am in terminal I'm just on my desktop you can make a folder wherever you have write permissions the easiest way to do it is let's just do make directory and we're going to make a folder on my desktop called chroma and we'll just call it that and so we'll CD into chroma and you can see there is nothing in this directory now we're going to run a Docker command that looks very similar but is not the same and you'll see it here and so we we're going to add an argument we're going to add a volume so- v and then we're going to add an absolute path to this new folder that we just created which is on my desktop and it's called chroma and don't forget that trailing slash to denote that it's a folder then we're going to do a colon SL chroma chroma now I'm not going to get into why we're doing colon SL chroma chroma it has to do with how Docker mounts local storage to Docker storage it's in the dock it's in there Docker compose if you really want to know but honestly just know that it works so we're going to run this command and let it build the container okay and so our container was live that took like two seconds and actually I want to show you if you go into Docker desktop or do this through the CLI and you inspect the container you should see that there is a new Mount chroma chroma is mounted to users Tim desktop chroma yours will be mounted wherever you assigned it now if we go into our browser and go to the collections API endpoint it's empty because we just created a new container there's nothing to save yet so back in anything LM I'm going to take these two files move them over save them all right workspace is embedded refresh we now have our data again so let's go look at actually where it got saved because I didn't run Docker in Damian mode that tab is occupied right now so instead I'm just going to open up a new tab CD into the folder that we specified to mount to and if I do LS you'll now see there is a hash here which is not important for you to really know about and then there is the chroma SQL light file so this is our data this is our chroma data and any Docker container we start with we can just Mount this same directory and it'll preload our data on boot so we're done that's it so let me show you that deleting my Docker container doesn't actually delete my data let's go back to Docker in here I'm going to go and I'm going to what would normally blow away all my embeddings I'm just going to click delete I'm going to delete the whole thing okay so our Docker container got blown away that's it that's all of our data now if we go to the folder and I type LS the data is still there so we did save it so how do I run another Docker container that can just inherit this data because let's say I upgraded I did Docker pull chroma DB chroma and then it's and let's just say there's a new image that got pulled in and it got updated great we're on the latest image again okay cool just rerun that same command with the same mounted volume if you move the file around make sure you change where you saved it to and as long as you mount to SL chroma chroma whatever is in this directory which should be a folder of hashes and a sqlite a chroma sqlite file you will boot with your old data so let's do that okay so our container rebooted let's go back to the end point point and then just refresh it we should see a collection here it looks just like this and look at that I'm refreshing right now and we do that means that we were now able to successfully totally destroy our instance reboot it and keep the data this is an important thing to know because if you set environment variables to perhaps set an API key if you want to do that you can't change Docker container environment variables without destroying the container itself so now you see why this little scale right here is so important because we need to sometimes change our API keys or do whatever and so you can't keep a container alive for its entire life so it's important to have the data saved actually on your host machine now for the last part of this video because I promised it in the beginning I'm going to show you how to use the API key setup for you to be able to basically just pass an API key like you normally would and have a authenticated chroma version it used to be that chroma didn't have authentication built in they added it very quickly I might add and so that's why I got to redo these videos because the newest stuff has authentication built in simplifies the process dramatically so let's show you how to do it locally and then in future videos we'll do it on AWS and render so I still have my data persisted in this chroma folder what I want to do is just simply add authentication so the way you do this is actually it looks complicated but I promise you it isn't and so I'm going to bring up a note that shows you what it does okay so you can't see me on this screen but you can see that we have this Docker run command then we set our Port as we normally would and we even do our Mount to well we changed the folder so let's call it chroma right and our Mount is even in the same place and the all the commands look exactly the same but you'll notice Dash e commands are actually how you set environment variables in Docker and you're probably wondering where in the world I got these from in trom.com or docs. trom chroma dcom if you go to getting started and then oh I'm sorry if you go to usage guide and then go to authentication you can actually see that there's basic oth for using a b 64 password I don't want to do that um and then there's static off which is more like an API token and that's what we want there are a couple things that you can have here there are these different providers of HTTP TP password and then there are also these config server off credentials if you use this top example you will only be using a authorization Bearer type token which if you're watching this channel you probably know what I'm talking about I want to use a more simple method which is basically you add a header to your request that says X hyphen chroma hyphen token and you just put the token and that's authentication I want to use that so you can actually copy these environment variables from here omit the export part and this is where we land so you can see this chroma server off credential stuff I'm specifying the text of new token and I want to use the X chroma token you cannot customize this it must say xor chroma token or authorization in all caps you can't change this don't try it I already did so let's run that Docker command with those environment variables in here now what we should expect to do is boot a Docker container that when I try to visit that URL in the browser we've been visiting says unauthorized and instead we have to specify a header in the git request and the token for this is a very nons new hyphen token but let's try it anyway so as to not bore you out of your minds I have Postman pulled up so that we can actually run the post requests for both get and post and all of that and I have it pointed at the same URL The Local Host 8000 API V1 collection I go to headers nothing special in the headers our Docker container appears to be running so let's just run that same get request we've been running in the browser we get an error unauthorize a 401 which is correct the way to remedy that is to add our special token of X chroma token this text by the way is not case sensitive so if you were wondering it isn't and then our token was called new token let's send that same get request with our API token we now get our collection result and if we try to do it in a browser where we can't set a header you can see we get unauthorized which is perfect that's exactly what we wanted last little caveat before we tie up this video because all of these learnings relate to hosting on cloud as well and I'm sure your question is oh no I accidentally committed my token to GitHub my chroma access token or your API token well with Docker you actually can't set an environment variable while a container is running uh so what do we do now well we have persistent storage stored on our host machine so all we need to do is actually just delete our container go back into our terminal and then just update this the chroma server undor oore credentials my gosh um and just change new token to literally anything else our data will persist and our token to access the data will change the reason this is important is and I know we're only on Local Host but the reason this is important is because you never know when you might need to change the kind of access token you are using the header that you want to use maybe you want to switch to authorization who knows I don't you know whatever you want to do and so let's boot that container and what we should expect is this previously working Postman request to fail we have to change the token to Hunter 2 which is classic and then we should get the correct response again so let me just show you that because I like to show as opposed to tell okay so Docker server is running let's hop back into Postman unauthorized let's change our token to Hunter 2 we're done all right so that's the end of this video you now have a fully working and authenticated locally running version of cha DB this is on the version 0.418 but there will probably be another one soon and so when there is using these methods you can upgrade to the latest version of chroma as long as it's not breaking changes which they'll tell you if they are and not have any data loss or worry about any of that in the next videos I'm actually going to build off of this knowledge on how to easily Deploy on AWS I have a template for you and also how to run this kind of service exactly the way I just showed you with credentials on a service like render if you have any questions or comments or compliments that would be nice to hear you can leave them in the comments thank you very much and I'll see you in the next video

Channel: Tim Carambat

Views: 1,971

Keywords:

Id: 61kaK-e3Owc

Length: 16min 7sec (967 seconds)

Published: Wed Nov 22 2023