Open Source Multi Modal Rag with chromadb using OpenCLIP embeddings

hey everyone in this video we're going to build a multimodel rag system using chroma using all open source models and embeddings I guess this is something new so let's just dive right into the code when we uh run this code uh we have some images as you can see the computer a cat a dog and robot data when we run this it automatically scans the directory and creates a chroma DB uh database everything's done automatically and then we can query this let's first take a look we have a cat image computer image a dog image and some robot data stuff if we type in a cat uh we actually get the distances so cat is the shortest distance so actually it resulted in cat being on the top you can search for computer and as you can see computer now actually comes up on top we can search for dog you search for dog then dog comes up and search for animal and then dog and cat comes up let say technology and then computer and robot data comes out on top we can also use a image query uh this is a command that is going to use an image that you place under the query query image folder it's currently a cat and when we do that the cat comes out on top so this it means you can qu query this uh multimodel database with text and images and you can also have a combination combination of both images and text although in my experience experience when you mix them up it doesn't give a great uh results so the bottom line is you upload anything you like uh any images you like here is PNG it auto detects the IDS that goes in the chroma DV by the file names and then it will automatically load it into chroma DB if we for example delete an image and rerun this uh rerun this script it will actually remove whatever image has been deleted you can also add more images here everything will be dealt with automatically now that we search for doc doc comes up on top but as you can see the computer has been removed as we've deleted it so it's pretty automated and it works well now uh code files for this project will be a my THX Master Class along with a very detailed uh live commentary coding walkthr on how I coded this from start to finish towards the end of this video I'll actually put a preview of this part of the video uh F to the walkr if you want to check that out but thousand xas class currently has 11 live coding walkthroughs to build the universal chat app cursor Deep dive Persona generator memory extractor and building an xjs app using versal SDK so if you do become a patron you will have access to the code files for all this along with their uh video walkthroughs but now we we are actually going to review the code here in this video uh but if you like to see the coding walk through feel free to visit the Thousand x master class link will be in the description now before doing the review I do want to give props to this gentleman Edam L this is where I first heard about the M model rag he has some pretty cool videos to check him out uh so we are importing the open clip embedding function from chroma DB utils this uses the open source clip embeddings and the uh requirements for this is chroma DB and open clip torch uh chrom ODB automatically installs npy and pillow and some other stuff that you need and when you first run this script so open source models will be downloaded to your computer just so you know and we Define embedding function is open clip embedding function then we import the image loader uh and then initialize the data loader as the image loader and then we import chrom ADB nonp because we pass the image data to the our uh Vector DB as an unpi array and pillow image from pillow so we can load images we create a persist directory and we use the persistent client from chroma DB with the pur directory so this will all be saved into your local machine you define a collection name and then we uh when the script runs it checks if this collection exists if it exists it gets the collection by that name also using the same embedding function otherwise you're going to get embedding um Dimensions mismatch error if it doesn't exist we create it with the collection name using the embedding function and then the data loader and then we check we get the list of image files in the documents directory we also get the list of text files in the uh documents directory like I said this also works with text files let me actually demonstrate it as well here I have the documentation. txt uh for this particular uh code actually so when you download it you'll be able have access to that to but I just copied it here so now when I run this again uh this will automatically detect the text files in the document strory as well along with its name and it's going to use the name as the the the as the uh ID that it inserts into chrom ADB with like let's search for Doc when I search for Doc documentation comes out on top with the minim minimum distance but this important thing is that this was inserted here and the script automatically took care of it because it not only checks for image files that end with PNG only though you can modify it to deal with JPEG and other other things it also deals with text files I would like to take a moment to talk about the benefits of becoming a patron as some of you may know in the last year and a half I've spent 3,000 hours over 300 uh projects as a patron you will have access to all the code files so you can get inspiration and itate quickly another benefit is that you'll have access to all my courses and my most recent and most most proud one the THX Master Class teaching how I what I've learned on how to code fast and efficiently also the streamlit course and the fast API course in my patreon I also have tiers in which you can connect with me oneon-one check those out as well and then it extracts the IDS it creates a list of IDs to insert in Chrome with DB by the uh names of the documents so it's convenient like that and then we load and add each file if it's ID it doesn't exist in the collection so we Loop over the files to add we check if it uh if it's uh check if the file ID is not already in the collection and we get the image path and then open it with image from pillow turn it into a npy array and then add it like this as the file ID is the file names and then uh here are the images as an array this is this adds it to chroma DB if the file is a text then we deal with that the same way uh and then add it but this time is documents instead of images so we deal with image files and now we deal with text files uh otherwise if it already exists in our collection then we print a message then we actually check for deleted files uh because you can delete files from here as well uh and then when you do that we we get the IDS and then we remove them one by one uh from The Collection so you can add additional documents here you can also delete see just for testing purposes uh anyway then we start a chat Loop and we take in a user query this query can be any text but can also include a command called image query if you call this command then it will actually check the image that is in uh the query image folder and it will perform a search using that uh if it's exit break out of it um otherwise we have a query text if the query text is image query then we get the folder we get the image files from there you can actually perform multiple searches this this script is designed to do query one uh image or one text uh query at a time uh and if if there are images that you have in the query image which we have then we get the path we open it turn into an ire array and then we query with that image this time we use the query images uh as you remember before we used uh documents or images anyway uh so we query with query images otherwise no quer query image is found and if we are doing a text query then we use Query texts here and then we extract the shortest distances which are the most relevant search results along with their IDs we sort it zip it and then create a list and then using this for Loop printed uh with the ID and distance as we see right here so this is pretty cool you can actually play around with it uh chroma DB uh doesn't recommend using persistent client for uh production use cases for that you must you probably want to use just a client with uh persist through and the persist directory but this was more convenient I thought uh anyway I hope you enjoy this like I said at the end of the video there will be a preview of the uh coding walkthrough that is present in my THX master class I highly recommend it I have spent over 3,000 hours building over 300 projects and I try to teach everything I've learned on how to code fast using cursor and AI assistance these projects use cursor IDE where you can replicate the same things that I'm doing with the prompts by copy pasting with catg PT or Opus user uh web UI thank you for watching so in this episode of THX master class we're going to build a multimodel rag system I just went to track promo.com apparently they have a multi model stuff now I'm just going to go to learn and here I find myself in the docks and then from here on out we're just going to copy some of these examples I already created a new project called multimol of rack from scratch let's create a main.py file let's initialize our terminal you can start your terminal here anyway uh let's create a first of all a cond environment it's cond the activate my current environment and then let's contrl K in the terminal create a new actually I think I know the command Honda create dasn say multi model let's just name it like that and then we say python equals 3.10 I don't I just like to use a few version behind just because some stuff is not compatible with the latest version but they should create a multimodel cond environment with python 3.10 unless I'm mistaken okay it's going to ask us if you want to continue we're going to say yes okay our environment has been created we can copy this command and enter it now we are in multimodal environment also list to control shift p and then select in Python interpreter so our IDE in general is pointing to this environment if you don't do see this you can do a a sideways shareon or Arrow whatever and type in interpreter find the select python interpreter your newly created environment may not appear here so you can refresh it in my case it appeared but let's refresh anyway and select this environment and let's copy the first piece of code here so this uses open clip embedding so everything here is open source let's paste this in and we're going to see that uh chroma DP is not defined so to uh install chrom ADB let's say pip install chrom ADB let's do a Google search this is I always like to double check with piie proper way to install this is Chrome how we install chrom ADB let's go ahead and install it made a mistake paste this here and install it and also we're going to have to create requirements at txt file with chroma DB this has to match how you installed uh as with Pip let's see what version version we are installing and we can add it here once it's done okay it installed we can say p chome should show our version number we just copy this put it here and we got our first requirement set okay let's move on to what we do next here we are going to copy this these are data loaders it is an automatic image loader so that's nice so our embedding function is going to be open clip embedding function and image loader comes from chroma DB as well what else uh multimodal collections all right so this is going to create a collection let's put it here so when we run this let's quickly run this and see what happens when we run this I believe chroma is going to create a folder here I think okay all right all right so we we we see we're Miss Missing uh the open clip torch so we're going to install this by running this command we also going to put this into our requirements once it's done uh we're going to pick the PIP show this library and write its version number okay this was in we can also do pip list and just scroll and find this we are looking for open flip torch this is the version number copy it paste it here and now we're done with second requirement let's run this again and like I said this should create a new folder and stuff some kind of initial promod DB initialization running this takes a moment because it's important quite a lot of stuff okay actually nothing happened when we ran this let me ref ref fresh okay so let's just continue with code snippet so this was creating a collection adding data so we can add data multimodal collection so by specifying data modality as you can see this can add images let's let's copy this let's put it here so we have it let's let me grab an image and paste it here okay I pasted this image so now we have at least one image to work with let's rename this to one.png and here I believe okay so this accepts a list of npy arrays representing images let's do contrl a and then contrl K and ask we have 1.png and we need to add it as a npy array to our collection let's see if this will figure it out for us we are using GPT 40 okay it's going to use npy plus pill so we need to add these to our requirements let's just go ahead and pillow let's see let's see how we pip install pillow this was the end of our video but I'd like to talk quickly about my oo streamer version 3 project streamer version 3 is a pqt powered P installer package python project that I came up with uses Z API key to create course websites such as this one in real time this is also deployed at Railway and including with audio the El if clause in py so you have quite a lot of Co such as six different uh voice choices and over 50 languages that you can choose from you can choose light or dark team uh when you go here to generate courses you just enter uh a course that you would like to generate for example we just said for culture Basics and I can pick how many chapters I'd like to generate and I'm just going to go ahead and generate it real quick this should this shouldn't take too long our curriculum was created successfully I can go into view course outline and search for that permac culture and I as I can see ethics and principles design methods and tools and practical applications I can then actually uh select this uh course outline and continue to generate the course in light mode and then the website will launch automatically and will be created for you in real time and you can record it as I'm doing right now or actually live streaming it's really up to you and once it begins we'll be able to for the permaculture revolves around three core ethics one of which is the care for the Earth this ethic I'm going to go ahead and pause it I will delete this run then this entire course will be generated live and I can listen to it live I'm going to go ahead and and if I were to let this course be generated then it'll be under my view and launch generated courses for example I just created a course called Financial basics for go ahead and it's like this I can actually switch to uh light mode as well I believe uh I'm sorry dark mode and then I can re revisit the scor both I can zoom in both is in text and the importance of emergency funds yeah it has three chapters which I can easily use the benefit of this and what you'll get out of it is that instead of chatting with in a disorganized manner this allows you to create uh structured courses that you can run and listen to before you go to sleep or just fill your time when you have have just 5 minutes or 10 minutes worth you can visit these courses back whenever anytime you like so Auto streamer you can download a free demo 4 from autost streamer. live I'll put the link in the description Mech version is coming soon you can if you click on the download free demo it will take you to my Google Drive download and these are the files you'll be downloading go to streamer demo. exe is the same thing as this except with limited features and if you wanted to download the full version then clicking this will take you to my patreon shop where it's currently only for $200 instead of 300 you can read all about it in the website you do need an open API key for this to work and sometimes your this is a p installer package pqt python application so your MEF or malware bites May flag it as is not good but as a matter of fact all you have to do is just make an exception for the program and if you have any questions feel free to join our Discord or ask me a question that you have in Discord well thank you for watching and do let me know what you think of this project I was really proud of this one and like I said the code files will be available at patreon and I also have special tiers for one-on-one meetings with me if that's something you're interested in and uh thank you for watching and I'll see you in the next video

Channel: echohive

Views: 763

Keywords:

Id: gzJYsnfcuXs

Length: 19min 16sec (1156 seconds)

Published: Sun Jun 09 2024